Project STAR 

Hiring the Right
 

Evaluator
 for your
 

National Service
 

Program
 

Project STAR 

1-800-548-3656 

star@JBSinternational.com 

www.nationalserviceresources.org 

(Search: project star) 

Developed by Project STAR and the Corporation for National and Community Service

REV 6/17/05 A-EV043
 
Project STAR 

Introduction 

Program evaluation is an important aspect of every national service program. 

Through well-designed, systematic evaluation, the essential components of successful 

national service programs can be identified and replicated, and less effective elements 

can be modified. Further, evaluation results will provide stakeholders with evidence of 

the impacts of national service. 

Effective evaluation requires commitment, experience, and an investment of time. 

Maintaining objectivity is vital. Many programs find hiring an external evaluator to be an 

effective means of obtaining a valuable evaluation. An external evaluator must be chosen 

carefully, however, because program personnel can easily become dissatisfied with an 

inappropriate or inadequate evaluation and may come to question the usefulness of the 

results obtained. 

Well-planned and well-conducted evaluations are invaluable in determining 

where a national service program succeeds and how it can be improved. The following 

eight guidelines can assist you in selecting and working with an appropriate external 

evaluator. 

Guideline 1: Form an evaluation committee. 

An evaluation committee should oversee the entire evaluation process, from 

initial planning through implementation and crafting of the final report. This committee 

should be made up of persons representing the various aspects of your program, including 

service participants, participant supervisors, program managers, host agency staff, and 

other key people involved in programmatic planning. The committee’s size will depend 

upon the size of your program. A large program might support an evaluation committee 

of five to seven persons, chaired by the program director. Smaller programs, of course, 

may require no more than three people; some, only a single staff member. In other cases, 

the evaluation committee might need to include the entire program staff. 

The evaluation committee will have primary responsibility for all aspects of the 

evaluation and will receive regular reports from the external evaluator. Committee 

members must be actively involved in the evaluation process to increase their 

understanding of it and to allow them to feel ownership and make use of, evaluation 

results. 

Guideline 2: Define the evaluation. 

The evaluation committee must have a clear understanding of what it wants the 

evaluation to accomplish. In particular, the committee must identify the issues intended to 

be the focus of the evaluation. The development of those issues should be guided by the 

program’s objectives. Keep in mind that the evaluation should be decision-focused. At 

REV 6/17/05 A-EV043 1 
Project STAR 

each step of the design process, committee members should ask themselves how data 

collected will inform programmatic decisions. 

Once the key issues have been identified and clarified in writing, the committee 

should identify specific tasks for which the external evaluator will be responsible. These 

tasks should relate directly to the conduct of the evaluation and might include 

development of the evaluation plan, development of evaluation instruments, selection of 

the sampling procedures and drawing of the evaluation sample, collection of evaluation 

data, analysis of evaluation data, composition of the evaluation report, and assistance in 

presenting evaluation results. 

Once these tasks have been determined, the committee must compose a task 

description (also called a statement of work or request for proposals) including these and 

any other requirements for the position. An overall description of your national service 

project and an estimate of the evaluation funds available should also be included in this 

document. 

The task description should include detailed instructions on how potential 

candidates are to apply for the position, identifying all information and documentation 

they must provide to the committee. Most committees will want to have candidates 

submit written proposals, even if they are brief. At a minimum, the committee will want 

to review a copy of at least one evaluation report previously written by the applicant and 

interview at least two references from individuals or organizations for which the 

evaluator rendered services. 

Guideline 3: Develop a task description and solicit candidates. 

Developing a task description 

Once your committee has determined the issues and decisions that your 

evaluation should address, you need to design a statement of work to which potential 

evaluators can respond. Your statement of work can be complex or simple, depending 

upon your needs and the degree to which you want to specify how the candidates will 

respond. Here are some work elements to consider including in your task description are: 

•	 Required meetings (e.g. planning meeting, regular updates, interim report 

presentation, final debrief) 

•	 Development of an evaluation plan (or execution of your existing plan) 

•	 Development of instruments 

•	 Selection and implementation of a sampling plan 

•	 Collection of data 

•	 Analysis of data 

•	 Development of report 

•	 Presentation of results 

REV 6/17/05 A-EV043 2 
Project STAR 

Soliciting candidates 

Once the task description has been completed, it should be distributed to local and 

professional organizations such as the American Educational Research Association, the 

American Evaluation Association, and others that might have a network of experienced 

evaluators. It also should be advertised at least once in a local newspaper. Viable 

candidates with good credentials in program evaluation can usually be found in social 

science research organizations and institutions of higher education. Try using your 

existing network to identify persons who have conducted program evaluations for other 

organizations in your community or among your national service contacts. 

Send copies of the statement of work to all identified candidates. When someone 

is identified as a good program evaluator, a telephone call or letter to recruit that person 

is appropriate. If a potential candidate resides outside your immediate geographical area, 

make sure the evaluator will be able to participate in regular evaluation committee 

meetings. 

Solicitation will require some real effort, so use your evaluation committee to 

actively recruit and make your outreach effective. 

Your committee should decide how candidates will apply. Written proposals, 

interviews, or some combination of the two is most typical. If you decide to use a written 

proposal, consider how you want the candidates’ responses to be organized. Here are 

some common section headings: 

•	 Description of program: Give the candidate an overview of your program, 

or include some appended materials about it. 

•	 General goals of the evaluation: Let the respondent know what you are 

seeking. 

•	 Work elements of the task plan: Specify that the candidate must describe 

their approach to each work task. Select from those listed above under “develop a 

task description”, or add others, as appropriate. 

•	 Timeline: Have the candidate estimate the completion date of all key
 

elements.
 

•	 Funds available: Give the candidate some sense of the scale of funding you 

can support. 

•	 Qualifications: Have the candidate include relevant professional experience, 

credentials, references, work samples, etc. 

•	 Application format (e.g. length) and due date: Give the candidate 

guidance on how extensive a proposal you are seeking and provide a submission 

deadline. 

•	 Contact person: Include the name of a person in your organization to whom 

candidates may direct inquiries. 

REV 6/17/05 A-EV043 3 
Project STAR 

Guideline 4: Interview and select the evaluator. 

Although the evaluation committee will likely review a number of applicants for 

the position, it should narrow the pool down to five or fewer candidates for formal 

interviews. The committee will want to explore a number of issues during these 

interviews. The following questions—to be asked of the candidate or discussed among 

committee members—may be helpful. 

Does the candidate understand the difference between research and 

evaluation? 

The primary purpose of research is to develop a new knowledge base or expand 

on an existing one. On the other hand, the primary purpose of program evaluation is to 

provide information related to specific program improvement or program management 

decisions. It is possible, of course, for program evaluation efforts to expand on an 

existing knowledge base while simultaneously providing program personnel with the 

information they need. However, external evaluators are sometimes more interested in 

conducting research peripheral to the evaluation needs of the program—in part, because 

of their desire to publish in research journals. Such desires will frequently lead to 

ineffective evaluation results. To prevent such a situation, have the candidate describe the 

difference between research and evaluation approaches. If a candidate does not appreciate 

this difference, he or she could lean toward conducting research during your evaluation as 

a consequence of professional research interests. 

Does the candidate understand your program? 

Have the candidate describe his or her understanding of your program’s intent (as 

reflected in the information you provide to applicants) and how its goals are to be 

attained. If the candidate misunderstands program goals or operational strategies, make 

corrections at this time to give the candidate a fair opportunity to respond to subsequent 

questions. It is important that you feel comfortable about the candidate’s understanding 

of your program’s goals and strategies. 

What would the candidate’s general approach be to your evaluation? 

Have the candidate describe the general approach he or she intends to take for the 

evaluation. Pay attention to questions and issues the candidate believes should be the 

focus of the evaluation, the type of data to be collected to address those questions and 

issues, the method of data collection, and the presentation of the evaluation’s results. If 

the discussion becomes very technical and a candidate is unable to present information in 

an easily understood manner, it is unlikely that this candidate will meet your needs. A 

candidate unable to communicate effectively at this time will probably not overcome the 

problem during the evaluation. Effective communication is a key for success, and the 

interview gives committee members a good idea of how effectively a candidate can 

communicate. 

REV 6/17/05 A-EV043 4 
Project STAR 

Does the candidate believe your evaluation can be conducted for the 

available funds? 

Candidates must indicate that their proposed evaluation approach can be carried 

out for the funds you indicated would be available. You may find a proposed evaluation 

plan to be excellent but unattainable under your anticipated budget. 

All candidates who pass the initial screening process should be asked to produce a 

detailed budget for the evaluation. A detailed budget is useful for identifying the aspects 

of the evaluation that are being emphasized, as well as for providing the committee with a 

way of monitoring the overall evaluation effort. 

What is the candidate’s reaction to supervision by the evaluation 

committee? 

The evaluator should report to the evaluation committee to assist the committee’s 

supervision of the evaluation. It is reasonable to assume that a candidate unwilling to 

work under such conditions is not appropriate for the position. 

Experienced evaluators might offer suggestions to help your committee’s 

proposed project management operate more efficiently and effectively. 

What is the candidate’s prior evaluation experience? 

Experience is an important factor to consider. A candidate probably will not have 

performed exactly the same evaluation that you require, but many similarities between 

previous programs and your own can be found. The candidate’s prior experience will be 

your main opportunity to discover and weigh that person’s strengths and weaknesses. 

How useful are the candidate’s previous evaluation reports? 

Look for evaluation reports for which the candidate served as lead author. Assess 

the reports for their clarity, organization, readability, and potential usefulness for decision 

makers. Pay particular attention to how well they would help a program improve. 

Candidates providing technical, poorly written, disorganized, difficult-to-understand, or 

lengthy evaluation reports will likely compose similar reports for your evaluation. 

Evaluators with poor writing skills are all too common, and a poorly written report can 

ruin the best evaluation study. 

REV 6/17/05 A-EV043 5 
Project STAR 

Does the candidate have good references? 

Candidates should be asked to provide the names of at least two persons or 

organizations for whom they have previously conducted evaluation projects. These 

references should be contacted for objective views of the candidates. Here are some 

questions you could ask the references: 

•	 Did the evaluation approach used by the evaluator address the needs and desires 

of your organization? 

•	 Was the evaluation conducted in a timely fashion? 

•	 Was the evaluation conducted within your budget? 

•	 Was the evaluation report useful to you? 

•	 Would you hire the evaluator to conduct another evaluation for you? 

Will the candidate’s existing professional commitments interfere with the 

planned evaluation? 

Good program evaluators are usually in demand. A candidate who is engaged in 

several projects, however, may be unable to devote sufficient time to your program 

evaluation. Ask the candidate to describe current and expected professional 

commitments. If the commitments seem excessive, ask how the candidate plans to 

conduct your program evaluation along with these other tasks. If the candidate indicates 

that other persons will be used to assist with the evaluation, determine which tasks will be 

performed by whom. Also determine if these other persons are capable of performing the 

tasks assigned to them. Using a team of trained and experienced persons to perform an 

evaluation is common, but the committee must satisfy itself that the team leader (i.e., the 

candidate) will be involved in all tasks that the committee believes require this person’s 

direct involvement. 

What is your general reaction to the candidate? 

During interviews, be alert to the candidate’s ability to communicate in a 

straightforward manner, and be alert to your own expectations of how effectively you and 

your colleagues can work with this person. A clash of working styles can certainly be a 

problem, and the chemistry between a candidate and committee members should not be 

ignored. 

What is your overall rating of the candidate? 

Following the interview process, committee members should individually rate the 

candidates on all of the issues previously described. Candidates might be rated on a fivepoint scale, ranging from “Definitely hire as our evaluator” to “Definitely do not hire as 

our evaluator”. “No opinion” should be the midpoint. (See Attachment 1 for a sample 

form for rating candidates). After combining committee members’ individual scores, the 

REV 6/17/05 A-EV043 6 
Project STAR 

candidates should be ranked so that the position can be offered to the candidate most 

acceptable to the majority of committee members. 

Guideline 5: Write and negotiate the contract. 

The desired relationship between the evaluation committee and the external 

evaluator is one of partnership and should be reflected as such in the contract. The 

contract should state, in a single paragraph if possible, the evaluator’s general 

responsibilities. If you have the evaluator produce a work plan, it can be incorporated into 

your agreement. Also include a brief statement detailing your intended decision-making 

process and the authority of the evaluation committee. In another paragraph, list the 

contract deliverables and provide a timetable for them. Consider including language 

describing how changes in the scope of work or work plan will be handled. Many 

evaluation contracts also specify who owns the data gathered during the evaluation, as 

well as who has the right to publish the results of the evaluation study. Finally, indicate 

how the evaluator will bill for services rendered and a schedule of payments. Between 20 

and 30 percent of the evaluator’s fee should be with held until acceptance of the final 

report by the committee. 

The contract should also detail the evaluation committee’s responsibilities to 

provide the external evaluator with timely and appropriate guidance, to review and 

approve evaluation instruments and documents in a timely and constructive manner, and 

to assist the evaluator in solving problems that arise during the evaluation. 

Guideline 6: Interact closely with the evaluator. 

At the first meeting with the evaluator, the evaluation committee should again 

describe the overall project and express its expectations of how the evaluation should be 

conducted. Following this general discussion, the evaluator and the committee should 

schedule, and make agendas for, subsequent meetings to keep the evaluation moving in a 

timely and efficient manner. Later meetings should include a review of the sample 

selection process, if applicable, and a discussion of the data-collection plan. The more 

specific the agenda you make for subsequent meetings, the more likely that the evaluation 

will meet the needs of the program and be completed in a timely manner. Be sure to keep 

the minutes of the committee meetings that record decisions made about the evaluation 

effort. 

The committee’s involvement in the evaluation process should not be limited to 

periodic meetings with the evaluator. The committee, in part or in whole, should 

continually monitor the evaluator and the evaluation. Committee members should review 

all major work elements, including sampling plans, instrumentation, data collection plans, 

etc. The committee can be of considerable service to the evaluator by facilitating 

involvement with host sites, beneficiaries, service participants, and others from whom the 

evaluator wishes to collect information. 

Solicit feedback from your evaluator as the study progresses. Informal insights 

gained during the study period can be valuable. A committee that is intermittently 

REV 6/17/05 A-EV043 7 
Project STAR 

involved in the evaluation process may discover that the study had gone in an 

inappropriate direction after considerable time, money, and effort have been expended. 

Guideline 7: Prepare the final report and release of results. 

As your evaluation nears completion, the committee and the evaluator should 

agree on a format for the evaluation report. The evaluation report should address 

evaluation questions directly and briefly and should be understandable to the target 

audience. Any report, of course, must provide useful and direct guidance for program 

decision makers. 

The committee and the evaluator should also agree at this time on the evaluator’s 

role in the release of the evaluation’s results. The committee may request that the 

evaluator be available to meet with decision makers, conduct interviews with news 

media, and make public presentations of the results. 

The evaluator customarily submits a preliminary draft of the final report to 

committee members for review and comment. If the changes suggested by the committee 

are significant and would have the effect of changing the findings, recommendations, or 

overall focus of the report, they should be discussed at a meeting between the committee 

and the evaluator. If the evaluator does not believe that the suggested changes are 

consistent with the data, the evaluator has the right to be disassociated from the report. In 

such an instance, the evaluator may make the requested changes and assign authorship of 

the report to the committee. Such situations should be avoided, however, as they usually 

place the integrity and public acceptance of the report in jeopardy. 

Guideline 8: Use the findings of your study. 

Once your study is complete, its value will be very limited unless the results are 

used to improve your program. Use the insights gained to review your program design 

and modify, or replicate, program elements as appropriate. Use your evaluator as a 

resource in this activity. Your evaluator’s assistance in interpreting the results for their 

program management implications can be very valuable. 

Address the most compelling findings first. Don’t get overwhelmed trying to do 

everything at once. Make note of findings that suggest further areas of investigation. 

Future evaluations may shed light on unclear data. 

Use your study not only to improve operations, but to market your program. 

Potential funders and service partners will be more willing to become involved with your 

program if you can show them evidence of effectiveness and a willingness to make 

improvements based on evaluation data. 

REV 6/17/05 A-EV043 8 
Project STAR 

Conclusion 

Most evaluations of national service programs are undertaken to help decision 

makers improve the programs. The relationship between the evaluation committee and 

the external evaluator must thus be viewed as a functional partnership. An effective 

functional partnership is founded on agreement over the objectives of the evaluation, an 

understanding of the responsibilities and authority of each partner, and mutual respect for 

the contributions that each partner provides to the evaluation. Your committee must have 

ultimate responsibility for the program’s evaluation, and the evaluator, as a partner, 

should function as an advisor and staff person to you. Selecting and appropriate evaluator 

will largely depend upon the specificity of the evaluation tasks your committee identifies 

to be performed, the effort you put into the selection process, and the extent and quality 

of your interaction with the evaluator during the development and implementation of the 

evaluation. If the steps identified here are followed, the evaluation process should be 

enjoyable and productive for both the evaluation committee and the external evaluator. 

REV 6/17/05 A-EV043 9 
Project STAR 

Attachment 1 

Candidate Rating Form 

Candidate Name: __________________________ 

1. Distinction between research and evaluation 

Clearly understands the No opinion Has no understanding of 

distinction between the distinction between 

evaluation and research evaluation and research 

2. Understanding of the program. 

Has excellent understanding of No opinion Has no understanding of 

the program the program 

3. Evaluation approach 

Understandable: can clearly 

explain how evaluation approach 

addresses needs of program 

No opinion Not understandable: cannot 

clearly explain how evaluation 

approach addresses needs of 

program 

4. Evaluation costs 

Evaluation is very likely to be 

conducted with available 

resources 

No opinion Evaluation can be conducted 

with available resources 

5. Reaction to the evaluation management structure 

Supports management structure No opinion Wants sole authority 

REV 6/17/05 A-EV043 10 
Project STAR 

6. Prior evaluation experience 

Had done evaluations No opinion Has no experience with 

similar to this one evaluations such as this one 

7. Usefulness of previous evaluation reports 

Evaluation reports are No opinion Evaluation reports are not 

understandable and useful understandable and useful 

8. References 

Excellent No opinion Poor 

9. Professional commitments 

Commitments will not interfere No opinion Commitments will interfere 

with evaluation with evaluation 

10. General reaction (manner, personality) 

Very positive No opinion Very negative 

11. Overall rating 

Definitely hire as our evaluator No opinion Definitely do not hire as our 

evaluator 

REV 6/17/05 A-EV043 11