Budgeting for Evaluation

Evaluation is a powerful tool to gain critical insights into your programs, policies, or operations to support informed decision-making, learning, and improvement. When viewed as a strategic investment, understanding how and to what extent your programs and policies work can help you be more efficient and effective. 

Knowing how much an evaluation costs requires an understanding of the key factors that drive those costs. An evaluation’s budget depends on the scope and complexity of both what’s being evaluated and the evaluation itself. It should be noted that although this resource focuses on budgeting for program evaluation, the questions included apply to evaluations of policies and operations as well. 

In general, evaluation budgets should be:

· Commensurate with interested parties’ expectations and direction (including board of directors, senior leadership, program and evaluation personnel, and others);

· Appropriate for the research design used and key questions to be answered;

· Adequate for ensuring quality and rigor, and;

· In line with the level of program and organizational resources available. 

This guide presents five main questions to help with budgeting for an evaluation by determining how key factors contribute to evaluation costs. At the end of the guide is a sample template that itemizes the costs to include in a detailed budget. 

***What are the program’s characteristics and context?***

The scope and complexity of your program have a direct impact on evaluation costs. Program characteristics such as type and length of services delivered, population served, number of operating sites, and geography all influence your evaluation budget estimates. 

· **Type(s) of services delivered.** Multifaceted programs involving many, diverse components and actors are likely to be more expensive to evaluate than simple interventions. 

· **Length of service delivery.** The duration of an intervention is related to when outcomes are expected to be realized and can affect the length of an evaluation and the frequency of data collection.

· **Population(s) served.** Hard-to-reach or vulnerable populations face sensitivities that may create challenges to data collection or require specialized data collection methods.

· **Single or multi-site.** Multi-site programs need to account for potential variation in implementation, the involvement of larger numbers of program stakeholders, and higher costs resulting from more on-site data collection compared to single-site programs.  

· **Geography.** Costs can vary by geographic location and economic differences in communities.

***What does the evaluation seek to answer?***

As you begin to plan what your evaluation seeks to answer, step back and put your evaluation in the context of a long-term research agenda. Remember, you don’t have to evaluate your whole program at once, and you can “chunk out” a bigger evaluation into smaller components to maximize the resources you have on hand. Figuring out what you want to know 5 or 10 years in the future will help you spend evaluation money more strategically by laying out studies that build upon one another and will allow you to create complementary resources that can be deployed multiple times. 

You can do this by working backwards from an end goal. If you ultimately want to achieve a strong level of evidence for your program by implementing an impact evaluation, such as a randomized control trial (RCT) or a quasi-experimental design (QED), what supporting steps do you need to take ahead of time to get there? Maybe you need to begin with descriptive studies and start collecting routine data. Then you might develop and administer an annual survey, field a process evaluation, and invest in building additional data collection instruments before considering a more resource-intensive impact evaluation. 

The evaluation question(s) you want to answer also influences evaluation costs. Clearly defined research questions are essential for determining the scope and complexity of the evaluation (see also “Developing the Right Research Questions resource for program evaluation”), and consequently, the evaluation budget. Your study may focus on a single research question with short-term outcomes or multiple research questions that examine complex, interrelated outcomes over time. 

Common questions that may be pursued independently or jointly include: 

· What are the characteristics of the program participants and target population?

· Is the intervention being implemented as intended? 

· What are the experiences of program participants?

· To what extent does participation in the intervention improve outcomes for program participants? 

· Does participation in the intervention improve outcomes for participants compared to individuals not served by the program? 

· To what extent does program impact vary by participant subgroup or by implementation strategy?

· Can the program be replicated to other populations/geographic areas? 

· What are the costs and benefits of the program to participants? 

When planning for an evaluation, keep in mind that each question has an associated cost, even for studies that employ existing data. If resources are limited, think about conducting the evaluation in stages, prioritizing important evaluation questions first, and examining remaining questions later, as appropriate, to help defray costs. The bottom line is to determine what you want to learn from the evaluation, identify the key research questions, and build a realistic budget that can answer those questions. 

***Who will conduct or oversee the evaluation?***

Once you have a list of research questions for your evaluation, you will need to decide whether to leverage existing internal program staff, bring in an external evaluator, or use a combination of the two to carry out the study. Please note that AmeriCorps State and National grantees receiving annual funds of $500,000 or more are required to use an external evaluator, while grantees receiving annual funds of less than $500,000 are not required to conduct an external evaluation.[footnoteRef:1] Although the costs associated with the evaluation may be lower if a study is conducted internally, each approach represents a tradeoff. Consider the following questions when making your decision:  [1:  Large grantees have the option to request an alternative evaluation approach (AEA) for Funding Threshold. This allows grantees that receive funding between $500K and less than $1M on average per year to be exempt from the requirement to use an external evaluator to conduct an impact evaluation. More details on the Funding Threshold AEA, and other AEAs, can be found here: https://americorps.gov/sites/default/files/document/Alternative_Evaluation_Approach_2023-06_Final_508.pdf ] 

· ***Is there sufficient capacity to carry out the evaluation internally?***

· Are staff with the requisite expertise able to conduct the evaluation without external assistance or expertise? 

· What is the anticipated level of effort for internal staff to work on the evaluation?

· Can staff dedicate enough time to the evaluation, given other their other responsibilities? 

· Is additional training, new hires, or technology needed (e.g., data collection, management, and analysis)?

· ***Is an external evaluator needed to support some or all aspects of the evaluation’s requirements?***

· Is an independent, third-party evaluation a requirement? 

· Would certain activities associated with the evaluation (e.g., analysis, data collection) require an independent lens or benefit from the impression of independence that an external evaluator would provide?

· How can internal staff participation be optimized to offset costs (e.g., staff may support administrative data access while the external evaluator may lead the study design and data analysis that rely on specialized skills)?

· What type of evaluator is cost-effective for this study? Is the capacity of a large research organization needed or university-based researchers be more cost-effective?

Even when an external evaluator is retained, grantee program staff must remain engaged in monitoring the evaluation to ensure that the study is progressing as planned. The activities for internal staff to monitor the evaluation may include: regular check-in meetings with the evaluation team; providing input and feedback on instruments developed and data collection strategies; facilitating access to existing data sources or stakeholders; and reviewing reports and other materials prior to dissemination. 

***How will the evaluation be carried out?***

The scope and complexity of the evaluation design and data collection strategy used to address your research question(s) plays a significant role in the cost of your evaluation. Each of these evaluation design factors below should be considered. 

· **Evaluation design.** A range of study designs can be used to address specific research questions, including experimental (e.g., randomized controlled trials or RCTs), quasi-experimental (e.g., matched comparison group design), and non-experimental designs (e.g., outcome and process evaluations; cost studies). Answering some evaluation questions requires more complex designs and sophisticated data collection methods which can drive up costs.   

· **Data source/collection**. Primary data collection can be the most expensive cost in an evaluation budget and may include new qualitative data collection, such as key informant interviews, focus groups, or site visits, and quantitative data, such as survey data collection. One tip is to try to build data collection into routine program operations from the start rather than viewing it as a one-time exercise for a specific evaluation only. Collecting data regularly will pay dividends for years to come for both performance measurement and evaluation and for continuous program improvement. Regular data collection can be a helpful element of building a long-term research agenda.

· **Existing data and infrastructure.** Can existing data and resources be leveraged for the evaluation? To what extent do new data need to be collected and new instruments or data platforms developed? Although RCTs are believed to be the most rigorous design for determining the extent to which your program effected program outcomes or impacts, it is not necessarily the most expensive design if existing data can be used. Meanwhile, observational designs can be costly if extensive qualitative data collection and analysis is necessary.  

· It should be noted that gaining access to secondary data (such as health care claims data) can be labor intensive and may require additional fees, depending on the managing organization or institution. 

· **Study sample.**Who are the respondents for primary data collection (is there one or many diverse respondent groups)? It should be noted that complex sampling designs (e.g., propensity score matching) may require specialized expertise leading to higher costs. 

· **Method.**

· ***How will data be collected?*** In-person data collection are often more expensive than other forms of data collection, such as phone- or web-based data collection (such as for surveys). For qualitative data collection, the cost of interviews or focus groups can be reduced by using video-conferencing rather than in-person methods. The choice needs to be balanced by assessing how the different data collection modes affect response rates and data quality. It is important also to confirm which data collection methods are most effective for your target population(s). It can be useful to talk to staff working in the locations where data collection will occur, so that you can better determine what systems are in place, which new ones will need to be created, and how data can be collected most efficiently and effectively. Additional costs related to data collection may be necessary, such as Institutional Review Board (IRB) fees for research involving human subjects or purchasing datasets. 

· ***Will tokens of appreciation be provided to respondents?***In which format will tokens of appreciation be provided to respondents (e.g., cash, coupon, gift card) and how much? Alternately, are there costs related to making the data collection easier for respondents, such as providing transportation, food, or child care?

· ***Does data need to be collected in multiple languages?***How many languages do respondents speak or read that may require translation of instruments?  Do current staff have the skills to conduct interviews or focus groups in other languages or is there a need to hire interpreters or bilingual staff?

· **Frequency.** How frequently do the data need to be collected over the course of the evaluation? If data are collected more than once, will it be collected among the same or different samples?

· **Data management.** Data management may include, but is not limited to cleaning, merging, linking, formatting, and storing primary and secondary data. 

· **Data analysis.** The amount of time and level of technical expertise required to conduct data analysis and interpretation are important considerations. Analysis of data using sophisticated statistical methods (such as text analysis, natural language processing, causal forest model, Bayesian analysis) will require more technical expertise and may need appropriate training or additional guidance from subject matter experts. It may also require more labor hours to complete, leading to higher costs. 

· **Community engagement**. Higher levels of engagement with the people affected by the program will mean more time dedicated to evaluation, making it more costly. However, the greater inclusion of community members in the evaluation process may result in more representative and valid evaluation findings and bring significant value, including increased buy-in for and relevance of the study, resulting in greater use of evaluation findings in decision making. This value must be balanced with the potential cost implications not only in time, but potentially in the translation of instruments and materials, as well as hiring staff with the relevant cultural and linguistic competency. 

As you plan for your evaluation, it is also important to keep your long-term goals in mind. For example, can the collected data or developed instruments be repurposed for future program evaluations? To what extent can new data collection and management systems ease future implementation efforts and help build evaluation capacity in the long-term? All these considerations may increase the cost of an evaluation now, but may allow future evaluations to be conducted more efficiently and cost-effectively. 

***How will findings be disseminated?***

Reporting and dissemination of evaluation findings are as important as other parts of the evaluation life cycle. The cost of communicating findings typically includes: 

· Time and resources needed to document evaluation findings and prepare reports, briefs, presentations, and other evaluation deliverables. 

· Other considerations include whether findings need to be modified so they are accessible to multiple audiences. 

· Time and resources needed by internal audiences and other interested parties to discuss and reflect on evaluation findings and ensure that findings are used for informing decision making within the organization, as needed (internal utilization). 

· Time, effort, and resources required to implement (external) communications and dissemination plans around the evaluation study and its results, including sharing of lessons learned in the process (e.g., preparing abstracts/manuscripts and presenting at conferences). Consider when evaluations are due to State Commissions and the next Grant Application Review Process (GARP) deadline to allow for sufficient time to complete required evaluation reports, 

Consider whether you have existing communications channels and approaches that can be leveraged to disseminate the evaluation findings and reduce costs related to establishing new ones.

***Where can I find resources for my evaluation?***

**Final Thoughts for Developing Evaluation Budgets**

Although there are opportunities to offset the costs of an evaluation, keep in mind that failure to allocate sufficient resources for evaluations can negatively impact the quality of the evaluation and the evidence generated. An insufficiently funded evaluation may have many challenges such as lack of appropriate expertise, under-powered study design, poor communication, and unanswered evaluation questions – all of these would produce poor evidence, and ultimately waste resources.  Again, program evaluation should be viewed as a strategic investment in improving your program, and a steppingstone to more effectively meet your mission.

To help cover the cost of an evaluation, ASN grantees are encouraged to tap into local resources, such as program partners or local collaborators that might have a stake in the information you’ll collect and the results your evaluation would generate. You can approach these groups by articulating why the evaluation results will be useful for them: perhaps their own program or cause would be advanced by the questions your evaluation is asking, or maybe the data you generate would help them in their own evaluation efforts. Universities, local consultants, or cooperative extensions may be able to pitch in, and many of these groups are interested in the intellectual challenge of designing high-quality evaluations (though please ensure they can meet the external evaluator definition, if applicable). Additionally, you can ask your fellow programs how much they are spending and where they are conducting evaluations.

**Tips for Designing an Adequate Evaluation Budget**

Once you have an evaluation plan developed and have identified the resources needed to complete the study, you are ready to start preparing a budget for your evaluation. In the process of estimating costs for your evaluation activities, you will likely need to make adjustments to your plan to ensure that available funds can sufficiently address your evaluation goals. Below are some tips to consider as you balance the needs of the evaluation with your available budget:

· I

· Talk with colleagues and partners within your State Commission, organization, AmeriCorps, and in other agencies about their budgets for evaluations of similar size and scope.

· Consult with your State Commission’s Evaluation Officer, if applicable, and/or your AmeriCorps Portfolio Manager.

· Consult with budgeting experts or staff members familiar with the contracting or procurement process for program evaluation services. They often possess a wealth of knowledge about typical costs and methods for maximizing resources.

· Talk with staff working in the locations where data collection will occur so that you can better determine what systems are in place, which new ones will need to be created, and how data can be collected most efficiently and effectively.

· Discuss your budget with interested parties who have been involved in the evaluation planning process to ensure that your assumptions are reasonable and that everyone is on the same page about what the evaluation is designed to deliver.

· Conduct an evaluability assessment to both determine whether the program is ready for a rigorous evaluation and to determine what additional investments (e.g., in data systems and tools, staff, or communication) need to be made to improve the quality of the evaluation.

· Create two versions of your budget, a high and a low estimate, knowing that true costs will likely fall somewhere in the middle.  

· Consider whether the initial evaluation plan’s activities may be approached with lower cost options, if needed. For example, can in-person data collection shift to other, less expensive modes (e.g., online survey)? Can staffing be redistributed so that more junior to mid-level staff can support data collection activities and senior staff can focus on oversight?

· Consider whether the evaluation can be carried out in stages, prioritizing important evaluation questions first and examining remaining questions later. This approach will not only help defray costs, but will help ensure that each evaluation effort is sufficiently funded to deliver high-quality findings.

· Plan for contingencies and be prepared to adjust your evaluation budget along the way as needed.

**Costs to Include in a Detailed Budget**

The following table provides a sample template for identifying potential costs to include an evaluation budget. The costs are organized into four categories:

1. Labor costs: This covers staff time to conduct or support evaluation activities and includes evaluation staff and subcontractor salary and benefits and consultant time to conduct activities.

1. Travel costs: Travel costs vary from project to project. Projects evaluated across multiple sites will likely require larger travel budgets compared to those located at one site. Proximity of the evaluator can also affect travel costs. There may be travel costs associated with data collection, program support activities, communication and dissemination plans.

1. Other direct costs: Other costs directly related to evaluation that are not labor or travel, such as supplies, software, and equipment.

1. Program costs: This covers program staff time and systems/services to monitor and support evaluation activities. 

****

**Evaluation Cost Considerations**

2

# image1.png

# image2.jpeg