**AmeriCorps State and National Evaluation Plan**

Organization Name: All On Board 

Program Name: AmeriCorps Organization Building (AOB) 

Application ID: 22ND123456

1. Introductory Sections and Program Description

1.1 Theory of Change

In communities across the United States, nonprofit organizations are working to address large disparities in economic mobility, health and education among its residents. AOB provides opportunities for organizations to build their capacity to deliver direct services by leveraging the communities' under-utilized human assets – adult volunteers from the very communities they are serving. During their ten months of service, AmeriCorps Members are placed at nonprofit organizations and government agencies (“partner organizations”) to build capacity in targeted areas. 

AOB recently added community engagement to our logic model in order to emphasize its centrality to our approach to capacity building. This addition aligns with our application of the Asset Based Community Development (ABCD) approach, which focuses on sustainable community-driven development. All AmeriCorps members receive training in community engagement through AOB’s core curriculum, which includes sessions on ABCD, community outreach, volunteer engagement and management, and community collaboration. Members then are expected to put this training into practice as they carry out their capacity building activities at their assigned partner organization. Community engagement service activities might include recruiting community members and organization clients as volunteers, enlisting resident input into development plans, or engaging program constituents in governance groups that design and evaluate new programs. 

The short-term outcomes of the AOB which are related to community engagement are 1) organizations reporting increased community engagement and 2) partner organizations indicate that AOB has helped them expand their efforts to identify qualified volunteers and employees from diverse backgrounds. The mid-term outcome is that organizations sustain investment in authentic community engagement through organizational processes that invest efforts in the capacity building of community members, thereby contributing to AOB’s long term outcome of advancing racial equity and increasing economic opportunity in communities across the country.

1.2 Scope of the Evaluation

This evaluation will focus on the community engagement component of our program’s logic model, and specifically, our members’ application of our program’s training curriculum with fidelity. The overarching question guiding this proposed evaluation is: To what extent are AOB AmeriCorps members implementing the AOB program with fidelity? For the program to accomplish its goal of increasing community engagement among participating organizations, it must first establish that AmeriCorps members are able to successfully facilitate the core elements of the AOB curriculum. Accordingly, the focus of this evaluation will be on assessing how consistently AmeriCorps members demonstrate successful implementation of these core elements. 

In addition, AOB has also designed the evaluation to further exploring the evidence from a 2020 evaluation suggesting that there is a relationship between AmeriCorps member characteristics and the ways they carry out community engagement activities at our partner organizations. The 2020 study found variations in how the training curriculum were applied, so we would also like to understand how community engagement activities differ across program sites, how partner organizations enhance member leadership in community engagement, and how the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation.

1. Evaluation Outcome(s) of Interest

This evaluation builds on a 2020 impact evaluation, which examined perceived capacity growth and sustainability among organizations hosting an AmeriCorps member. In this evaluation, members who were most successful in increasing partner organization capacity received training on the AOB core curriculum, which includes ABCD, community outreach, volunteer engagement and management, and community collaboration.

The proposed process evaluation specifically focuses on the inputs and activities elements of our logic model that relate to community engagement. AOB proposes to conduct a process evaluation that will deepen our understanding of how members apply their training in community engagement to their service activities, how different partner organizations are enhancing member core training at different sites, how members’ backgrounds and experiences affect their service, and what are partner organizations’ perceptions of how the members’ training contributes to project implementation. Ultimately, what we learn from the process evaluation will help us strengthen implementation of the AOB curriculum and improve the quality of outcomes related to community engagement.

1. Research Question(s)

The process evaluation will address the following research questions:

1. To what extent are AmeriCorps members implementing the AOB core curriculum (i.e., ABCD, community outreach, volunteer engagement and management, and community collaboration) with fidelity? 

a. What additional supports would help AmeriCorps members better deliver AOB capacity building activities at their partner organizations? 

b. What are possible facilitators and barriers to implementation?

2. In addition to the core curriculum training, how are partner organizations enhancing member leadership in community engagement? How do community engagement activities differ across program sites?

3. From the perspectives of members and the perspective of partner organizations, how do member characteristics, identities, and proximity to the social issue(s) addressed by the partner organizations influence member approaches to service? How does this relationship affect member contributions to partner organizations, particularly when service activities involve community engagement?  

4. From the perspectives of partner organizations, how does the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation (e.g., achievement of project outcomes, project sustainability, etc.)

1. Evaluation Design

4.1 Evaluation type

AOB proposes a mixed methods (quantitative and qualitative) process evaluation that relies on data from existing program administrative systems (member database and exit surveys) and from new data collection activities (new community engagement survey, interviews, and focus groups). The evaluation questions posed in the plan are very timely and designed to provide findings that will be immediately useful to the development of AOB’s community engagement efforts. 

4.2 Comparison Group Formation

Not applicable. There is no proposed control or comparison group.

1. Sampling Methods

5.1 Sample Selection

The population from which the evaluation sample will be drawn includes AmeriCorps members, partner organization program coordinators, and AOB program site staff engaged in the 2022-2023 program year. All partner organization program coordinators (approximately 22) and all AmeriCorps members (approximately 50)****will make up the sample for member and partner organization exit surveys. The exit surveys are existing annual data collections where all potential members and organizations are invited to participate.

In addition to the annual surveys, eight sites will be purposefully selected for additional data collection based on either their perceived success in building community engagement capacity or because building community engagement is an area of focus for the site. Within each site, we will attempt to include all members (generally 4-10 depending on the size of the site), site staff supervisors (1-2 per site), and partner organization program coordinators (generally 1 per site) to participate in individual and/or focus group interviews, depending on the number of participants. We also plan to utilize data on the eight sites from our member database and member observation data typically collected for training and staff review purposes. 

5.2 *Sample Size Justification*

The evaluation will examine training and implementation more broadly across all 22 program sites, while also capturing more in-depth detail using a purposefully chosen sample of eight program sites and the partner organizations, program staff, and AmeriCorps members associated with each selected site. Each research question is best addressed by both collecting information across all members and partner organizations, while also gathering detailed, in-depth information on a smaller sub-set of program sites. Having both breadth and depth is necessary for fully addressing the research questions. Selecting eight sites for more in-depth data collection in addition to the census of program sites provides an opportunity to feasibly examine both variation and similarities across sites more thoroughly within the time and financial resources of the evaluation.

1. Data Collection Procedures, Data Sources, and Measurement Tools

To answer the study’s research questions, we will rely on a number of existing data sources, including our AmeriCorps member database, member observation data, AmeriCorps Member Exit Survey, and Partner Organization Exit Survey. These existing tools will be modified, as needed, to collect all necessary data for addressing the study’s research questions. One additional survey focused on community engagement will be developed for members. Our evaluators will conduct semi-structured interviews with program site staff responsible for training and supervising AmeriCorps members and with partner organization program coordinators. Two focus groups (6-8 participants per group) with AmeriCorps members will also be held to collect additional qualitative data on member, staff, and partner organization’s perspectives on community engagement implementation. 

The study will include ten months of data collection and ongoing analysis while members are serving (a full program year), with an additional three months of preparation and stakeholder engagement prior to members’ start of service and three months of analysis and reporting after the conclusion of the program year. The evaluation team will engage stakeholders in the preliminary phase to refine data collection tools prior to full program year data collection. 

Below is a description of various data sources and tools to be used to provide data for addressing the study’s research questions. We provide a summary table at the end which maps the data sources and analysis methods to each research question. 

*Structured observations of program activities staffed by AmeriCorps members*: Program staff currently complete observations of AmeriCorps members’ implementation of the core curriculum at all sites for ongoing training purposes during their first and fifth month of implementation using an existing assessment tool. We will collect and examine the completed assessment data used to evaluate all member activities at the eight sampled program sites. The tool is structured to direct the staff person using the tool to collect information on the content and structure of member activities. It also requests quality ratings on each member activity and observer notes. The tool is currently used for member development purposes, but the information collected will be compiled, stored, and analyzed across the eight selected program sites for evaluation purposes. 

The instrument is designed to evaluate the program elements that members are expected to implement as part of the AOB core curriculum. The instrument allows us to rate program activities on various indicators addressing the four key program elements related to community engagement. The four elements are:

· Asset Based Community Development (ABCD);

· Community outreach;

· Volunteer engagement and management; and 

· Community collaboration

Within each element, items receive a rating from zero (the item is not evident) to three (the item is highly evident). Members also receive a quality rating on their knowledge, leadership, organization, and implementation of each element. The goal of each observation “period” is to observe and rate a sample of all four elements within a 1-5 day period. These ratings provide a systematic method for evaluators to quantify observations of the elements of quality community engagement activities.

*Member Database:* AOB maintains a database of all AmeriCorps members that includes demographic information, educational background, and employment information. This data is drawn from the member application and Member Exit Survey. 

*Member Exit Survey:* AOB currently distributes this survey to all Members via email using Survey Monkey at the end of members’ service term to gather feedback on Members’ experiences and future plans. Items pertaining to the implementation of the AOB core curriculum will be added to assess leadership training and community engagement activities across members and sites.

*Partner Organization Exit Survey*: AOB distributes this survey via email using Survey Monkey to the single key contact at each of our partner organizations at the end of the program year to gain feedback on their experiences with the program, obtain confirmation of members’ activities, and gather opinions on the perceived impact of the members’ service on their organizations. Items pertaining to the effectiveness of the AOB core curriculum in preparing members for community engagement activities and its implementation will be added to this tool.

*Member Community Engagement Survey:* This survey will be distributed electronically (i.e. using software such as Survey Monkey or Qualtrics) towards the end of members’ service term to gather information on members’ experiences implementing community engagement activities at their partner organizations. Within this survey, the program and evaluation team will pilot an instrument for members to assess where their community engagement service activities fall on a spectrum of community member participation and responsibilities in the organization that moves from less intensive (invited to town hall on new program or recruited volunteers) to more intensive (work with community members to co-design programming) activities. Existing public participation rubrics will be used to develop this tool. The survey will also include a number of closed- and open-ended questions that ask members if and how they incorporate their background into their approach to service, particularly around community engagement.

*Semi-structured Individual and Focus Group Interviews:*At each of the eight select sites, we will conduct semi-structured interviews and focus groups (depending on the number of potential respondents at each site) with all members, program site staff supervisors, and partner organization program coordinators. Interviews are the likely forum to be used with staff (1-2 per site) and supervisors (generally 1 per site) given the low number of people at each site, while 1-2 focus groups of 4-7 participants each would be used to gain feedback from members (4-10 per site). Separate semi-structured protocols will be developed for individual and focus group interviews with members, site staff supervisors, and partner organization program coordinators. The protocols will be designed to last between 30 and 60 minutes. To promote a more interactive, and rich dialogue between evaluators and subjects, these interviews will prioritize open-ended questions addressing key topics related to the evaluation. The research questions will inform the focus of the interviews and include topics such as perceptions of community engagement implementation (RQs#1 & 4), barriers and facilitators to community engagement implementation at partner organizations (RQ#1), the relationship between members’ identities and proximity to community served or social issues addressed (RQ#3), and community engagement enhancements and supports (RQs#1 & 2). Most interviews and focus groups will be conducted in-person when possible, but some may need to be conducted remotely using online video technology or telephone. 

The program year phase of the study will span 10 months of data collection with the majority of data collection occurring in the second half of the program year, giving members the opportunity to become orientated to their work with partner organizations. 

Data Procedures for Member and Partner Organization Surveys

Towards the beginning of the program year, the evaluator will obtain access to the Member Database to examine Members’ demographic characteristics from their program applications. The Member Exit Survey and the Partner Organization Exit Survey are both administered electronically on an annual basis by AOB at the end of each program year, and these data will be merged with the Member Database dataset. Potential respondents to the surveys receive a unique survey link via email address. All of the email addresses of respondents will have been recently updated within the past year.

2

Table 1: Evaluation Questions, Data Sources, and Analysis

Data Procedures for Select Members, Program Site staff, and Partner Organizations

For the eight selected program sites, data collection will be spread out across the final two-thirds of the program year, beginning after members have become oriented at their partner organizations and program site implementation has moved beyond the program start phase. The Member Community Engagement Survey will be developed at the beginning of the project year and administered electronically via email to all members at the eight selected program sites towards the end of the project year. All program site staff supervisors and the partner organization program coordinator at the eight sites will be invited to participate in interviews, and all members will be invited to participate in focus groups. At each site, evaluators will conduct 1-2 member focus group interviews (4-7 Members per focus group) and individual interviews with program site staff supervisors (1-2 staff members) and partner organization program coordinators (1 per site). Focus groups and interviews will be held in-person, unless a virtual video option is necessary. Interviews will last 30 minutes, and focus groups will last 60 minutes. Interviews with program site staff and partner organization program coordinators from the sample of eight sites will occur in the second half of the program year.

The external evaluator will work with their own Institutional Review Board (IRB) to determine what level of consent and other human subjects procedures will be required to obtain data. Our study will follow all required guidance for human subjects research stipulated by the evaluator’s IRB. 

1. Analysis Plan

Data will be analyzed as it is collected when appropriate with much of the analysis occurring in the final part of the program year and three months immediately following the end of the program year. 

*Qualitative Data Analysis:* Interviews and focus groups will be transcribed and, along with open-ended survey questions and reflections where appropriate, transferred into analytic software for analysis, such as MAXQDA, NVIVO or ATLAS.TI. Thematic analysis of these transcripts will be conducted to identify patterns in responses and to derive themes using the following approach with each of the key research questions: 

· What common themes or patterns emerge in the responses on specific topics related to the research question? 

· Are there exceptions to these patterns and are there common characteristics associated with these exceptions? 

· What additional useful or interesting stories emerge from the responses that may help inform the research question?

*Quantitative Data*: Quantitative data collected through the Member Database and quantitative survey items will be analyzed and summarized using basic descriptive statistics (e.g., frequencies, means, etc.). Quantitative and qualitative data will be triangulated to get a better picture from multiple perspectives of how members are implementing community engagement, what affects members' experiences, etc. 

The specific ways in which the qualitative and quantitative data will be used to inform the evaluation’s research questions is described below. 

**Research Question #1**

As described above, the evaluation team will examine the data collected on members during their first and fifth month of service using a structured observation protocol at the eight selected program sites to evaluate program quality. During these observations, program staff will utilize an existing assessment tool to capture and rate observable indicators of community engagement activities. To assess member implementation of the AOB core curriculum, the mean average quality rating of members at each site will be calculated on their knowledge, leadership, organization, and implementation of each of the four major community engagement elements (ABCD, community outreach, volunteer engagement and management, and community collaboration) and compared across sites. If differences are found across sites in implementation quality, the evaluators will use additional data on AmeriCorps member and site demographics from the member database, AmeriCorps Member Exit Survey and Partner Organization Exit Survey to identify correlations and patterns between member and site characteristics and implementation quality. 

To identify what additional supports are needed for better implementation and also possible facilitators and barriers to implementation, these topics will be examined as part of the qualitative analysis of the interviews and focus group discussions with members, program site staff, and partner organization leadership. We will look for common themes and patterns in responses, as well as useful recommendations and noteworthy observations from respondents. 

**Research Question #2**

To address the second research question, exit survey data will be analyzed using descriptive statistics to summarize perceptions on leadership training and to report on community engagement activities reported by all members and partner organizations. Also, on the Member Community Engagement Survey taken by members at the eight selected sites, evaluators will examine members’ open-ended responses to reflection questions on how they are implementing community engagement activities, looking for common themes and patterns, to understand variations in how programs are enhancing member leadership in community engagement. Evaluators will calculate the average member intensity score on community engagement activities as reported on the Member Community Engagement Survey. 

**Research Question #3**

To understand how member characteristics, identities, and proximity to social issue(s) influences members’ approaches to service and member contributions to partner organizations, the evaluators will use a combination of broad-reaching survey instruments and in-depth data sources with smaller samples of respondents to better understand this topic. First, the overall member and partner organization perspective will be obtained from calculating and reporting descriptive statistics on these topics from the two member and partner organization exit surveys. The evaluators will also calculate descriptive statistics and conduct thematic, qualitative analysis on members’ answers to a series of closed- and open-ended survey questions on the Member Community Engagement Survey (distributed to members at the eight select sites) about the ways in which they bring their background into their approach to service, particularly around community engagement. Responses to the Community Engagement Survey will be merged with demographic information on member respondents from the member database and Member Exit Survey to identify associations and patterns in member characteristics and backgrounds with their approaches to community engagement. In addition to gaining the perspectives of members from the Community Engagement Survey, the evaluators will examine interview and focus group data from the member, site staff, and partner organization interviews/focus groups for themes on the relationship between members’ identities and proximity to communities served or social issues addressed.

**Research Question #4**

The final research question will focus on better understanding and documenting the perspectives of partner organizations on how the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation as measured by achievement of project outcomes, project sustainability, etc. The study will rely on two major data sources to address this research question: responses to the open-ended survey questions from the Partner Organization Exit Survey completed by all partner organizations and the semi-structured, in-depth partner interviews conducted at the eight selected program sites. Both sources will be examined for themes and patterns on how prepared Members are to implement program activities in community engagement and how well supported they are by AOB throughout their service period. The responses to the in-depth interviews may also provide useful insights and recommendations for how to improve the curriculum training and other supports provided to members, especially in terms of project sustainability. 

1. Evaluator Qualifications

AOB will engage an external evaluator or contractor skilled in mixed methods evaluation (quantitative and qualitative data sources) to guide, co-develop, and implement the research design and evaluation. AOB prefers an evaluator who has been active in the field of program evaluation for many years and has provided evaluation in a variety of settings, especially in organizational capacity and/or community engagement measurement. In addition to skills in mixed methods evaluation, the evaluator must also demonstrate experience with program evaluation and culturally-responsive evaluation. Particularly critical to the selection of an evaluator for this project will be their experience in collecting and synthesizing large amounts of qualitative information from multiple sources for drawing important conclusions for informing program practices and recommendations. 

1. Timeline

1. Budget

AOB expects to spend $60,000 for an external evaluator, including Institutional Review Board (IRB) submission and approval, modification of existing survey instruments and observation protocols, development of new interview and focus group protocols, data collection and analysis, and report writing. We anticipate some additional costs for internal staff time managing the project and for materials. Approximately 80% of our evaluation budget is from AmeriCorps funds.