**AmeriCorps State and National Evaluation Plan**

Organization Name: All On Board 

Program Name: AmeriCorps Organization Building (AOB) 

Application ID: 22ND123456

1. Introductory Sections and Program Description	Comment by Megan Edgin (she/her): The purpose of this additional Process Evaluation Plan is to provide a comprehensive version that goes above and beyond what is expected of grantees. This plan is intended to demonstrate different scenarios based on the research questions/data needs.

1.1 Theory of Change	Comment by Megan Edgin (she/her): A program’s theory of change is the general underlying idea of how you believe your intervention will create change and why the desired change is expected to come about. Your theory of change articulates the assumptions underlying your choice of activities. A good program evaluation should be centered around some part of your theory of change. You can focus your evaluation on one or more components of your theory of change.

In communities across the United States, nonprofit organizations are working to address large disparities in economic mobility, health and education among its residents. AOB provides opportunities for organizations to build their capacity to deliver direct services by leveraging the communities' under-utilized human assets – adult volunteers from the very communities they are serving. During their ten months of service, AmeriCorps Members are placed at nonprofit organizations and government agencies (“partner organizations”) to build capacity in targeted areas. 	Comment by Megan Edgin (she/her): Defining the problem your program addresses helps clarify why you are implementing your intervention. Your intervention should logically follow from your problem statement; alignment between your targeted problem and intervention is critical for producing the change you desire.This section of the plan clearly describes the problem the program addresses and a high level overview of the program's goals.

AOB recently added community engagement to our logic model in order to emphasize its centrality to our approach to capacity building. This addition aligns with our application of the Asset Based Community Development (ABCD) approach, which focuses on sustainable community-driven development. All AmeriCorps members receive training in community engagement through AOB’s core curriculum, which includes sessions on ABCD, community outreach, volunteer engagement and management, and community collaboration. Members then are expected to put this training into practice as they carry out their capacity building activities at their assigned partner organization. Community engagement service activities might include recruiting community members and organization clients as volunteers, enlisting resident input into development plans, or engaging program constituents in governance groups that design and evaluate new programs. 	Comment by Megan Edgin (she/her): This section sets a common understanding of the program approach, including member training and service activities.

The short-term outcomes of the AOB which are related to community engagement are 1) organizations reporting increased community engagement and 2) partner organizations indicate that AOB has helped them expand their efforts to identify qualified volunteers and employees from diverse backgrounds. The mid-term outcome is that organizations sustain investment in authentic community engagement through organizational processes that invest efforts in the capacity building of community members, thereby contributing to AOB’s long term outcome of advancing racial equity and increasing economic opportunity in communities across the country.	Comment by Megan Edgin (she/her): This section clearly states the short, mid, and long term outcomes anticipated from the program. Throughout the Theory of Change section, there is a written description of the program’s theory of change (i.e., how a program intends to achieve its desired outcomes) with enough detail to assess its alignment with the program's logic model and proposed outcomes of interest.

1.2 Scope of the Evaluation	Comment by Megan Edgin (she/her): State concisely the goal(s) of the evaluation and specify which service activity/ies will be assessed. AmeriCorps does not require grantees to evaluate all components of their theory of change; evaluation may focus on a sub-set of program activities.

This evaluation will focus on the community engagement component of our program’s logic model, and specifically, our members’ application of our program’s training curriculum with fidelity. The overarching question guiding this proposed evaluation is: To what extent are AOB AmeriCorps members implementing the AOB program with fidelity? For the program to accomplish its goal of increasing community engagement among participating organizations, it must first establish that AmeriCorps members are able to successfully facilitate the core elements of the AOB curriculum. Accordingly, the focus of this evaluation will be on assessing how consistently AmeriCorps members demonstrate successful implementation of these core elements. 	Comment by Megan Edgin (she/her): This section explains which component(s) of the program are included in the scope of the evaluation. For this evaluation, "community engagement" is the program component that will be the focus of the evaluation, and the guiding question for the evaluation centers on fidelity of implementation. This is important because grantees are not expected to focus on all aspects of their program for evaluation.

In addition, AOB has also designed the evaluation to further exploring the evidence from a 2020 evaluation suggesting that there is a relationship between AmeriCorps member characteristics and the ways they carry out community engagement activities at our partner organizations. The 2020 study found variations in how the training curriculum were applied, so we would also like to understand how community engagement activities differ across program sites, how partner organizations enhance member leadership in community engagement, and how the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation.	Comment by Megan Edgin (she/her): This description is valuable to demonstrate how the evaluation will build upon prior findings and justify this focus of the evaluation.

1. Evaluation Outcome(s) of Interest	Comment by Megan Edgin (she/her): The evaluation outcome(s) of interest section specifies the outcome(s) the evaluation will measure; outcomes are aligned with the theory of change, logic model, scope of study and are feasible to measure based on the source(s) of data needed and level of effort required.

This evaluation builds on a 2020 impact evaluation, which examined perceived capacity growth and sustainability among organizations hosting an AmeriCorps member. In this evaluation, members who were most successful in increasing partner organization capacity received training on the AOB core curriculum, which includes ABCD, community outreach, volunteer engagement and management, and community collaboration.	Comment by Megan Edgin (she/her): This description again ties the evaluation plan to prior findings.

The proposed process evaluation specifically focuses on the inputs and activities elements of our logic model that relate to community engagement. AOB proposes to conduct a process evaluation that will deepen our understanding of how members apply their training in community engagement to their service activities, how different partner organizations are enhancing member core training at different sites, how members’ backgrounds and experiences affect their service, and what are partner organizations’ perceptions of how the members’ training contributes to project implementation. Ultimately, what we learn from the process evaluation will help us strengthen implementation of the AOB curriculum and improve the quality of outcomes related to community engagement.	Comment by Megan Edgin (she/her): The inputs and activities from the logic model that will be explored in this evaluation are clearly stated. The evaluation focus is aligned to the theory of change, scope of the evaluation, and is feasible to measure in the duration of the evaluation.

1. Research Question(s)	Comment by Megan Edgin (she/her): Written research questions are provided and are clearly connected to measurable outcomes that are aligned with the theory of change. Research questions are phrased in accordance with the research design, which is a process evaluation in this case. Process evaluations focus on understanding things such as whether programs are implemented with fidelity and supports and barriers to implementation.For more information on creating good research questions, see the evaluation core curriculum webinar on “Asking the Right Research Questions.” 	Comment by Megan Edgin (she/her): [ASN TO INSERT LINK]

The process evaluation will address the following research questions:

1. To what extent are AmeriCorps members implementing the AOB core curriculum (i.e., ABCD, community outreach, volunteer engagement and management, and community collaboration) with fidelity? 	Comment by Megan Edgin (she/her): These research questions are aligned with a process evaluation, the noted outcomes of interest, and theory of change. They clearly indicate who the question focuses on, the specific activities/characteristics that will be explored, and appear to be feasible to answer.

a. What additional supports would help AmeriCorps members better deliver AOB capacity building activities at their partner organizations? 

b. What are possible facilitators and barriers to implementation?

2. In addition to the core curriculum training, how are partner organizations enhancing member leadership in community engagement? How do community engagement activities differ across program sites?

3. From the perspectives of members and the perspective of partner organizations, how do member characteristics, identities, and proximity to the social issue(s) addressed by the partner organizations influence member approaches to service? How does this relationship affect member contributions to partner organizations, particularly when service activities involve community engagement?  

4. From the perspectives of partner organizations, how does the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation (e.g., achievement of project outcomes, project sustainability, etc.)

1. Evaluation Design

4.1 Evaluation type	Comment by Megan Edgin (she/her): This subsection should fully describe the type(s) of evaluation design that will be used, and explain why this is the most appropriate design to achieve the evaluation goal(s) in #1 and answer the research question(s) in #3.

AOB proposes a mixed methods (quantitative and qualitative) process evaluation that relies on data from existing program administrative systems (member database and exit surveys) and from new data collection activities (new community engagement survey, interviews, and focus groups). The evaluation questions posed in the plan are very timely and designed to provide findings that will be immediately useful to the development of AOB’s community engagement efforts. 	Comment by Megan Edgin (she/her): The section clearly states that this will be a mixed methods process evaluation. In this example, the plan clearly states the type of evaluation, a high-level description of data sources, and why this evaluation type will be valuable to the program. The design choice aligns with the goals for the evaluation.

4.2 Comparison Group Formation	Comment by Megan Edgin (she/her): This section is not applicable for a process evaluation since no control or comparison group will be included in the design.

Not applicable. There is no proposed control or comparison group.

1. Sampling Methods

5.1 Sample Selection	Comment by Megan Edgin (she/her): In this section, describe the population from which the sample will be drawn and how the sample will be selected. The plan should include a detailed and complete description of: 1) the sample sizes to be used; and 2) the sampling procedures, including any eligibility criteria for inclusion in the study sample (e.g.., simple random, stratified random, purposive, convenience sampling). Alternatively, the plan should clearly state that no sample will be drawn, if applicable (i.e., the entire program population will be included in the study). Specify any consent procedures (i.e., parental/guardian consent, opt-in/opt-out) or data use agreements that will be necessary to gather or obtain data. 

The population from which the evaluation sample will be drawn includes AmeriCorps members, partner organization program coordinators, and AOB program site staff engaged in the 2022-2023 program year. All partner organization program coordinators (approximately 22) and all AmeriCorps members (approximately 50)****will make up the sample for member and partner organization exit surveys. The exit surveys are existing annual data collections where all potential members and organizations are invited to participate.	Comment by Megan Edgin (she/her): This section describes the population from which the samples will be drawn, that happens to be all partner organization program coordinators and all AmeriCorps members for the exit survey. The approximate sample size for each sample is included.

In addition to the annual surveys, eight sites will be purposefully selected for additional data collection based on either their perceived success in building community engagement capacity or because building community engagement is an area of focus for the site. Within each site, we will attempt to include all members (generally 4-10 depending on the size of the site), site staff supervisors (1-2 per site), and partner organization program coordinators (generally 1 per site) to participate in individual and/or focus group interviews, depending on the number of participants. We also plan to utilize data on the eight sites from our member database and member observation data typically collected for training and staff review purposes. 	Comment by Megan Edgin (she/her): This section describes the sample for focus groups and/or interviews, as well as approximate sample sizes. Throughout this section, it is clear who will be included in each data collection effort.

5.2 *Sample Size Justification*	Comment by Megan Edgin (she/her): In this section, explain the basis for selecting the sample sizes in #5.1 and how the size will be adequate to answer the research questions. This section may be skipped for process designs. 

The evaluation will examine training and implementation more broadly across all 22 program sites, while also capturing more in-depth detail using a purposefully chosen sample of eight program sites and the partner organizations, program staff, and AmeriCorps members associated with each selected site. Each research question is best addressed by both collecting information across all members and partner organizations, while also gathering detailed, in-depth information on a smaller sub-set of program sites. Having both breadth and depth is necessary for fully addressing the research questions. Selecting eight sites for more in-depth data collection in addition to the census of program sites provides an opportunity to feasibly examine both variation and similarities across sites more thoroughly within the time and financial resources of the evaluation.	Comment by Megan Edgin (she/her): This plan includes a description of the sampling approach, why both a breadth and depth of data collection is important to the evaluation, and why a sub-set of program sites will be included for in-depth data collection.

1. Data Collection Procedures, Data Sources, and Measurement Tools	Comment by Megan Edgin (she/her): Describe each data source and measurement tool and the procedures that will be used to collect or extract data, including when, how often, and by what mode (i.e., paper/pencil, phone, or web survey; administrative data extract). Explain how the proposed data sources and tools are adequate for addressing all of the research question(s) and how the data align with the evaluation’s outcome(s) of interest. 

To answer the study’s research questions, we will rely on a number of existing data sources, including our AmeriCorps member database, member observation data, AmeriCorps Member Exit Survey, and Partner Organization Exit Survey. These existing tools will be modified, as needed, to collect all necessary data for addressing the study’s research questions. One additional survey focused on community engagement will be developed for members. Our evaluators will conduct semi-structured interviews with program site staff responsible for training and supervising AmeriCorps members and with partner organization program coordinators. Two focus groups (6-8 participants per group) with AmeriCorps members will also be held to collect additional qualitative data on member, staff, and partner organization’s perspectives on community engagement implementation. 	Comment by Megan Edgin (she/her): This section shows how the evaluation will utilize pre-existing data sources and lists each data source for the evaluation, which are then described in further detail later in the section.

The study will include ten months of data collection and ongoing analysis while members are serving (a full program year), with an additional three months of preparation and stakeholder engagement prior to members’ start of service and three months of analysis and reporting after the conclusion of the program year. The evaluation team will engage stakeholders in the preliminary phase to refine data collection tools prior to full program year data collection. 	Comment by Megan Edgin (she/her): This describes the timeline, including preparation, refining data collection tools, data collection, and analysis and reporting.

Below is a description of various data sources and tools to be used to provide data for addressing the study’s research questions. We provide a summary table at the end which maps the data sources and analysis methods to each research question. 	Comment by Megan Edgin (she/her): This section requires a description of each data source that will be collected with enough detail to ensure that the data sources are adequate for addressing all of the study's research questions.

*Structured observations of program activities staffed by AmeriCorps members*: Program staff currently complete observations of AmeriCorps members’ implementation of the core curriculum at all sites for ongoing training purposes during their first and fifth month of implementation using an existing assessment tool. We will collect and examine the completed assessment data used to evaluate all member activities at the eight sampled program sites. The tool is structured to direct the staff person using the tool to collect information on the content and structure of member activities. It also requests quality ratings on each member activity and observer notes. The tool is currently used for member development purposes, but the information collected will be compiled, stored, and analyzed across the eight selected program sites for evaluation purposes. 	Comment by Megan Edgin (she/her): This describes how the evaluation will utilize existing structured observations of AmeriCorps members' implementation of program activities. The description includes the types of information gathered from observations and key elements related to the focus of the evaluation. The reader can visualize how, when, and why these observations take place and the usefulness of the data for the objectives of the evaluation.

The instrument is designed to evaluate the program elements that members are expected to implement as part of the AOB core curriculum. The instrument allows us to rate program activities on various indicators addressing the four key program elements related to community engagement. The four elements are:

· Asset Based Community Development (ABCD);

· Community outreach;

· Volunteer engagement and management; and 

· Community collaboration

Within each element, items receive a rating from zero (the item is not evident) to three (the item is highly evident). Members also receive a quality rating on their knowledge, leadership, organization, and implementation of each element. The goal of each observation “period” is to observe and rate a sample of all four elements within a 1-5 day period. These ratings provide a systematic method for evaluators to quantify observations of the elements of quality community engagement activities.

*Member Database:* AOB maintains a database of all AmeriCorps members that includes demographic information, educational background, and employment information. This data is drawn from the member application and Member Exit Survey. 	Comment by Megan Edgin (she/her): This lists the types of information that will be gathered from the member database and source of the data.

*Member Exit Survey:* AOB currently distributes this survey to all Members via email using Survey Monkey at the end of members’ service term to gather feedback on Members’ experiences and future plans. Items pertaining to the implementation of the AOB core curriculum will be added to assess leadership training and community engagement activities across members and sites.	Comment by Megan Edgin (she/her): This describes how and when the Member Exit Survey is administered. Additionally, the types of questions included on the survey are described with enough detail to envision the resulting data and how it will contribute to answering the research questions.

*Partner Organization Exit Survey*: AOB distributes this survey via email using Survey Monkey to the single key contact at each of our partner organizations at the end of the program year to gain feedback on their experiences with the program, obtain confirmation of members’ activities, and gather opinions on the perceived impact of the members’ service on their organizations. Items pertaining to the effectiveness of the AOB core curriculum in preparing members for community engagement activities and its implementation will be added to this tool.	Comment by Megan Edgin (she/her): This describes how and when the Partner Organization Exit Survey is administered. Additionally, the types of questions included on the survey are described with enough detail to envision the resulting data and how it will contribute to answering the research questions.

*Member Community Engagement Survey:* This survey will be distributed electronically (i.e. using software such as Survey Monkey or Qualtrics) towards the end of members’ service term to gather information on members’ experiences implementing community engagement activities at their partner organizations. Within this survey, the program and evaluation team will pilot an instrument for members to assess where their community engagement service activities fall on a spectrum of community member participation and responsibilities in the organization that moves from less intensive (invited to town hall on new program or recruited volunteers) to more intensive (work with community members to co-design programming) activities. Existing public participation rubrics will be used to develop this tool. The survey will also include a number of closed- and open-ended questions that ask members if and how they incorporate their background into their approach to service, particularly around community engagement.	Comment by Megan Edgin (she/her): This describes how and when the Member Community Engagement Survey is administered. Additionally, the types of questions included on the survey are described with enough detail to envision how additional questions will be developed and piloted. It is clear that the resulting data will contribute to answering the research questions.

*Semi-structured Individual and Focus Group Interviews:*At each of the eight select sites, we will conduct semi-structured interviews and focus groups (depending on the number of potential respondents at each site) with all members, program site staff supervisors, and partner organization program coordinators. Interviews are the likely forum to be used with staff (1-2 per site) and supervisors (generally 1 per site) given the low number of people at each site, while 1-2 focus groups of 4-7 participants each would be used to gain feedback from members (4-10 per site). Separate semi-structured protocols will be developed for individual and focus group interviews with members, site staff supervisors, and partner organization program coordinators. The protocols will be designed to last between 30 and 60 minutes. To promote a more interactive, and rich dialogue between evaluators and subjects, these interviews will prioritize open-ended questions addressing key topics related to the evaluation. The research questions will inform the focus of the interviews and include topics such as perceptions of community engagement implementation (RQs#1 & 4), barriers and facilitators to community engagement implementation at partner organizations (RQ#1), the relationship between members’ identities and proximity to community served or social issues addressed (RQ#3), and community engagement enhancements and supports (RQs#1 & 2). Most interviews and focus groups will be conducted in-person when possible, but some may need to be conducted remotely using online video technology or telephone. 	Comment by Megan Edgin (she/her): This describes how and when the interviews and focus groups will be conducted, including who will participate. Separate protocols will be developed according to role, the duration of the protocols, and approach to conducting the sessions. Additionally, the topics to be discussed are clearly mapped to associated research questions. 

The program year phase of the study will span 10 months of data collection with the majority of data collection occurring in the second half of the program year, giving members the opportunity to become orientated to their work with partner organizations. 

Data Procedures for Member and Partner Organization Surveys	Comment by Megan Edgin (she/her): This section describes when data is entered into the Member Database and how surveys are sent to participants. 

Towards the beginning of the program year, the evaluator will obtain access to the Member Database to examine Members’ demographic characteristics from their program applications. The Member Exit Survey and the Partner Organization Exit Survey are both administered electronically on an annual basis by AOB at the end of each program year, and these data will be merged with the Member Database dataset. Potential respondents to the surveys receive a unique survey link via email address. All of the email addresses of respondents will have been recently updated within the past year.

2

Table 1: Evaluation Questions, Data Sources, and Analysis	Comment by Megan Edgin (she/her): While not required, tables are an excellent way to demonstrate alignment of research questions, data sources, and analysis plans. This plan clearly shows the data sources that will be used to answer each research question. Additionally, the analysis approach for each data source is specified, such as calculating descriptive statistics and identifying themes that emerge in interview/focus group responses.

Data Procedures for Select Members, Program Site staff, and Partner Organizations

For the eight selected program sites, data collection will be spread out across the final two-thirds of the program year, beginning after members have become oriented at their partner organizations and program site implementation has moved beyond the program start phase. The Member Community Engagement Survey will be developed at the beginning of the project year and administered electronically via email to all members at the eight selected program sites towards the end of the project year. All program site staff supervisors and the partner organization program coordinator at the eight sites will be invited to participate in interviews, and all members will be invited to participate in focus groups. At each site, evaluators will conduct 1-2 member focus group interviews (4-7 Members per focus group) and individual interviews with program site staff supervisors (1-2 staff members) and partner organization program coordinators (1 per site). Focus groups and interviews will be held in-person, unless a virtual video option is necessary. Interviews will last 30 minutes, and focus groups will last 60 minutes. Interviews with program site staff and partner organization program coordinators from the sample of eight sites will occur in the second half of the program year.

The external evaluator will work with their own Institutional Review Board (IRB) to determine what level of consent and other human subjects procedures will be required to obtain data. Our study will follow all required guidance for human subjects research stipulated by the evaluator’s IRB. 	Comment by Megan Edgin (she/her): Here it is helpful to include IRB procedures and plans to ensure the evaluation aligns with required human subjects procedures.

1. Analysis Plan	Comment by Megan Edgin (she/her): Describe an analysis that is appropriate for the evaluation’s design and data sources. Explain how the analysis will address all of the evaluation’s research questions.For example, descriptive analyses (e.g., frequencies, means, etc.), correlational analyses, and t-tests/chi-square tests are appropriate for quantitative data used for non-impact designs. Qualitative analysis methods (e.g., content analysis, thematic analysis) are appropriate for qualitative data.

Data will be analyzed as it is collected when appropriate with much of the analysis occurring in the final part of the program year and three months immediately following the end of the program year. 

*Qualitative Data Analysis:* Interviews and focus groups will be transcribed and, along with open-ended survey questions and reflections where appropriate, transferred into analytic software for analysis, such as MAXQDA, NVIVO or ATLAS.TI. Thematic analysis of these transcripts will be conducted to identify patterns in responses and to derive themes using the following approach with each of the key research questions: 	Comment by Megan Edgin (she/her): The plan includes a clearly described analysis approach for qualitative data. The sources of qualitative data are named, as well as potential software to be used for analysis. The type of analysis is named, along with guiding questions to ensure the thematic analysis corresponds to research questions as intended. 

· What common themes or patterns emerge in the responses on specific topics related to the research question? 

· Are there exceptions to these patterns and are there common characteristics associated with these exceptions? 

· What additional useful or interesting stories emerge from the responses that may help inform the research question?

*Quantitative Data*: Quantitative data collected through the Member Database and quantitative survey items will be analyzed and summarized using basic descriptive statistics (e.g., frequencies, means, etc.). Quantitative and qualitative data will be triangulated to get a better picture from multiple perspectives of how members are implementing community engagement, what affects members' experiences, etc. 	Comment by Megan Edgin (she/her): The plan describes how quantitative data will be analyzed via descriptive statistics. Also, noting that qualitative and quantitative data will be triangulated provides additional insight into the analysis approach.

The specific ways in which the qualitative and quantitative data will be used to inform the evaluation’s research questions is described below. 	Comment by Megan Edgin (she/her): This example provides clear description of how the analysis plan corresponds with each research question. This is helpful to clarify exactly how the approaches described above will answer each research question based on the types of data collected.

**Research Question #1**

As described above, the evaluation team will examine the data collected on members during their first and fifth month of service using a structured observation protocol at the eight selected program sites to evaluate program quality. During these observations, program staff will utilize an existing assessment tool to capture and rate observable indicators of community engagement activities. To assess member implementation of the AOB core curriculum, the mean average quality rating of members at each site will be calculated on their knowledge, leadership, organization, and implementation of each of the four major community engagement elements (ABCD, community outreach, volunteer engagement and management, and community collaboration) and compared across sites. If differences are found across sites in implementation quality, the evaluators will use additional data on AmeriCorps member and site demographics from the member database, AmeriCorps Member Exit Survey and Partner Organization Exit Survey to identify correlations and patterns between member and site characteristics and implementation quality. 

To identify what additional supports are needed for better implementation and also possible facilitators and barriers to implementation, these topics will be examined as part of the qualitative analysis of the interviews and focus group discussions with members, program site staff, and partner organization leadership. We will look for common themes and patterns in responses, as well as useful recommendations and noteworthy observations from respondents. 

**Research Question #2**

To address the second research question, exit survey data will be analyzed using descriptive statistics to summarize perceptions on leadership training and to report on community engagement activities reported by all members and partner organizations. Also, on the Member Community Engagement Survey taken by members at the eight selected sites, evaluators will examine members’ open-ended responses to reflection questions on how they are implementing community engagement activities, looking for common themes and patterns, to understand variations in how programs are enhancing member leadership in community engagement. Evaluators will calculate the average member intensity score on community engagement activities as reported on the Member Community Engagement Survey. 

**Research Question #3**

To understand how member characteristics, identities, and proximity to social issue(s) influences members’ approaches to service and member contributions to partner organizations, the evaluators will use a combination of broad-reaching survey instruments and in-depth data sources with smaller samples of respondents to better understand this topic. First, the overall member and partner organization perspective will be obtained from calculating and reporting descriptive statistics on these topics from the two member and partner organization exit surveys. The evaluators will also calculate descriptive statistics and conduct thematic, qualitative analysis on members’ answers to a series of closed- and open-ended survey questions on the Member Community Engagement Survey (distributed to members at the eight select sites) about the ways in which they bring their background into their approach to service, particularly around community engagement. Responses to the Community Engagement Survey will be merged with demographic information on member respondents from the member database and Member Exit Survey to identify associations and patterns in member characteristics and backgrounds with their approaches to community engagement. In addition to gaining the perspectives of members from the Community Engagement Survey, the evaluators will examine interview and focus group data from the member, site staff, and partner organization interviews/focus groups for themes on the relationship between members’ identities and proximity to communities served or social issues addressed.

**Research Question #4**

The final research question will focus on better understanding and documenting the perspectives of partner organizations on how the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation as measured by achievement of project outcomes, project sustainability, etc. The study will rely on two major data sources to address this research question: responses to the open-ended survey questions from the Partner Organization Exit Survey completed by all partner organizations and the semi-structured, in-depth partner interviews conducted at the eight selected program sites. Both sources will be examined for themes and patterns on how prepared Members are to implement program activities in community engagement and how well supported they are by AOB throughout their service period. The responses to the in-depth interviews may also provide useful insights and recommendations for how to improve the curriculum training and other supports provided to members, especially in terms of project sustainability. 

1. Evaluator Qualifications	Comment by Megan Edgin (she/her): Describe how the person(s) who will conduct the evaluation are sufficiently qualified to conduct the proposed evaluation (e.g., have experience and technical qualifications that align with the planned evaluation design). Include details for how the evaluator is sufficiently qualified and meets AmeriCorps evaluation requirement for the applicant (large or small). Whether the evaluator is internal or external to the program is clearly stated.If an external evaluator has been identified, the description supports the conclusions that the evaluator is independent from the program and qualified to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts). This also means that evaluators that have any affiliation to the program’s host agency or organization may not be considered external. Grantees that are uncertain of their evaluator’s affiliation, should confirm with their AmeriCorps Portfolio Manager or with their State Commission representative if a State Commission sub-grantee.If the evaluator is not yet identified or hired, describe the required and/or preferred qualifications for an evaluator. For more information on hiring an external evaluator, please see the evaluation core curriculum “Managing an External Evaluation” course.	Comment by Megan Edgin (she/her):  [ASN TO INSERT LINK]

AOB will engage an external evaluator or contractor skilled in mixed methods evaluation (quantitative and qualitative data sources) to guide, co-develop, and implement the research design and evaluation. AOB prefers an evaluator who has been active in the field of program evaluation for many years and has provided evaluation in a variety of settings, especially in organizational capacity and/or community engagement measurement. In addition to skills in mixed methods evaluation, the evaluator must also demonstrate experience with program evaluation and culturally-responsive evaluation. Particularly critical to the selection of an evaluator for this project will be their experience in collecting and synthesizing large amounts of qualitative information from multiple sources for drawing important conclusions for informing program practices and recommendations. 	Comment by Megan Edgin (she/her): The plan clearly describes the skills and qualifications that will be prioritized in finding an external evaluator who is qualified to conduct the evaluation, has relevant expertise, and will provide a culturally-responsive approach.

1. Timeline	Comment by Megan Edgin (she/her): Provide a timeline for all of the major evaluation activities (e.g., finalizing evaluation design, hiring evaluator, developing data collection instruments, collecting pre-intervention data, collecting post-intervention data, analyzing data, writing report). Delineate the timeline by month and year or on a quarterly basis (e.g., fall 2023, spring 2024). The timeline must show how all evaluation activities and a final report will be completed before your next recompete application.AmeriCorps recommends using the first program year for evaluation planning (including gaining final approval of the plan) and data collection instrument development; the second program year for data collection; and the remaining time in the third program year to analyze data and complete the evaluation report. Since grantees have unique programs and recompete application deadlines may vary by state, exact evaluation timelines may vary. 

1. Budget	Comment by Megan Edgin (she/her): Specify the overall budget allotted for the evaluation, including the cost of engaging an external evaluator, if applicable. If you will be utilizing staff time for conducting an internal evaluation, provide a description of those in-kind resources.

AOB expects to spend $60,000 for an external evaluator, including Institutional Review Board (IRB) submission and approval, modification of existing survey instruments and observation protocols, development of new interview and focus group protocols, data collection and analysis, and report writing. We anticipate some additional costs for internal staff time managing the project and for materials. Approximately 80% of our evaluation budget is from AmeriCorps funds.	Comment by Megan Edgin (she/her): The budget appears reasonable for the evaluation and shows that allocation will be made for each component of the evaluation, as well as considering additional costs that may accrue.