**AmeriCorps State and National Evaluation Plan**

Organization Name: All On Board 

Program Name: AmeriCorps Organization Building (AOB) 

Application ID: 22ND123456

1. Introductory Sections and Program Description

1.1 Theory of Change

In communities across the United States, nonprofit organizations are working to address large disparities in economic mobility, health and education among its residents. AOB provides opportunities for organizations to build their capacity to deliver direct services by leveraging the communities' under-utilized human assets – adult volunteers from the very communities they are serving. During their ten months of service, AmeriCorps Members are placed at nonprofit organizations and government agencies (“partner organizations”) to build capacity in targeted areas. 

AOB recently added community engagement to our logic model in order to emphasize its centrality to our approach to capacity building. This addition aligns with our application of the Asset Based Community Development (ABCD) approach, which focuses on sustainable community-driven development. All AmeriCorps members receive training in community engagement through AOB’s core curriculum, which includes sessions on ABCD, community outreach, volunteer engagement and management, and community collaboration. Members then are expected to put this training into practice as they carry out their capacity building activities at their assigned partner organization. Community engagement service activities might include recruiting community members and organization clients as volunteers, enlisting resident input into development plans, or engaging program constituents in governance groups that design and evaluate new programs. 

The short-term outcomes of the AOB which are related to community engagement are 1) organizations reporting increased community engagement and 2) partner organizations indicate that AOB has helped them expand their efforts to identify qualified volunteers and employees from diverse backgrounds. The mid-term outcome is that organizations sustain investment in authentic community engagement through organizational processes that invest efforts in the capacity building of community members, thereby contributing to AOB’s long term outcome of advancing racial equity and increasing economic opportunity in communities across the country.

1.2 Scope of the Evaluation

This evaluation will focus on the community engagement component of our program’s logic model, and specifically, our members’ application of our program’s training curriculum with fidelity. The overarching question guiding this proposed evaluation is: To what extent are AOB AmeriCorps members implementing the AOB program with fidelity? For the program to accomplish its goal of increasing community engagement among participating organizations, it must first establish that AmeriCorps members are able to successfully facilitate the core elements of the AOB curriculum. Accordingly, the focus of this evaluation will be on assessing how consistently AmeriCorps members demonstrate successful implementation of these core elements. 

In addition, AOB has also designed the evaluation to further exploring the evidence from a 2020 evaluation suggesting that there is a relationship between AmeriCorps member characteristics and the ways they carry out community engagement activities at our partner organizations. The 2020 study found variations in how the training curriculum were applied, so we would also like to understand how community engagement activities differ across program sites, how partner organizations enhance member leadership in community engagement, and how the curriculum training and other supports provided to AmeriCorps members by the AOB program contribute to successful project implementation.

1. Evaluation Outcome(s) of Interest

This evaluation builds on a 2020 impact evaluation, which examined perceived capacity growth and sustainability among organizations hosting an AmeriCorps member. In this evaluation, members who were most successful in increasing partner organization capacity received training on the AOB core curriculum, which includes ABCD, community outreach, volunteer engagement and management, and community collaboration.

The proposed process evaluation specifically focuses on the inputs and activities elements of our logic model that relate to community engagement. AOB proposes to conduct a process evaluation that will deepen our understanding of how different partner organizations are enhancing member core training at different sites, how community engagement activities differ across program sites, and how members’ backgrounds and experiences affect their service. Ultimately, what we learn from the process evaluation will help us strengthen implementation of the AOB curriculum and improve the quality of outcomes related to community engagement.

1. Research Question(s)

The process evaluation will address the following research questions:

1. In addition to the AOB’s core member training, how are partner organizations enhancing member leadership in community engagement? How do community engagement activities differ across program sites?

2. From the perspectives of members and the perspective of partner organizations, how do member characteristics, identities, and proximity to the social issue(s) addressed by the partner organizations influence member approaches to service? How does this relationship affect member contributions to partner organizations, particularly when service activities involve community engagement?  

1. Evaluation Design

4.1 Evaluation type

AOB proposes a mixed methods (quantitative and qualitative) process evaluation that relies on data from existing program administrative systems (member database and exit surveys) and from new data collection activities (interviews and focus groups at select program sites). The evaluation questions posed in the plan are very timely and designed to provide findings that will be immediately useful to the development of AOB’s community engagement efforts. 

4.2 Comparison Group Formation

Not applicable. There is no proposed control or comparison group.

1. Sampling Methods

5.1 Sample Selection

The population from which the evaluation sample will be drawn includes AmeriCorps members, partner organization program coordinators, and AOB program site staff engaged in the 2022-2023 program year. All partner organization program coordinators (approximately 22) and all AmeriCorps members (approximately 50)****will make up the sample for member and partner organization exit surveys. The exit surveys are existing annual data collections where all potential members and organizations are invited to participate.

In addition to the annual surveys, four sites will be purposefully selected for additional data collection based on either their perceived success in building community engagement capacity or because building community engagement is an area of focus for the site. Within each site, we will attempt to include all members (generally 4-10 depending on the size of the site), site staff supervisors (1-2 per site), and partner organization program coordinators (generally 1 per site) to participate in individual and/or focus group interviews, depending on the number of participants. We also plan to utilize data on the four sites from our member database typically collected for training and staff review purposes. 

5.2 *Sample Size Justification*

The evaluation will examine training and implementation more broadly across all 22 program sites, while also capturing more in-depth detail using a purposefully chosen sample of four program sites and the partner organizations, program staff, and AmeriCorps members associated with each selected site. Each research question is best addressed by both collecting information across all members and partner organizations, while also gathering detailed, in-depth information on a smaller sub-set of program sites. Having both breadth and depth is necessary for fully addressing the research questions. Selecting four sites for more in-depth data collection in addition to the census of program sites provides an opportunity to feasibly examine both variation and similarities across sites more thoroughly within the time and financial resources of the evaluation.

1. Data Collection Procedures, Data Sources, and Measurement Tools

To answer the study’s research questions, we will rely on a number of existing data sources, including our AmeriCorps member database, the AmeriCorps Member Exit Survey, and the Partner Organization Exit Survey. These existing tools will be modified, as needed, to collect all necessary data for addressing the study’s research questions. Our evaluators will conduct semi-structured interviews with program site staff responsible for training and supervising AmeriCorps members and with partner organization program coordinators. Two focus groups (6-8 participants per group) with AmeriCorps members will also be held to collect additional qualitative data on member, staff, and partner organization’s perspectives on community engagement implementation. 

The study will include ten months of data collection and ongoing analysis while members are serving (a full program year), with an additional three months of preparation and stakeholder engagement prior to members’ start of service and three months of analysis and reporting after the conclusion of the program year. The evaluation team will engage stakeholders in the preliminary phase to refine data collection tools prior to full program year data collection. 

Below is a description of various data sources and tools to be used to provide data for addressing the study’s research questions. We provide a summary table at the end which maps the data sources and analysis methods to each research question. 

***Member Database****:* AOB maintains a database of all AmeriCorps members that includes demographic information, educational background, and employment information. This data is drawn from the member application and Member Exit Survey. 

***Member Exit Survey****:* AOB currently distributes this survey to all Members via email using Survey Monkey at the end of members’ service term to gather feedback on Members’ experiences and future plans. Items pertaining to the implementation of the AOB core curriculum will be added to assess leadership training and community engagement activities across members and sites. Within this survey, the program and evaluation team will pilot an instrument for members to assess where their community engagement service activities fall on a spectrum of community member participation and responsibilities in the organization that moves from less intensive (invited to town hall on new program or recruited volunteers) to more intensive (work with community members to co-design programming) activities. Existing public participation rubrics will be used to develop this tool. The survey will also include a number of closed- and open-ended questions that ask members if and how they incorporate their background into their approach to service, particularly around community engagement.

***Partner Organization Exit Survey***: AOB distributes this survey via email using Survey Monkey to the single key contact at each of our partner organizations at the end of the program year to gain feedback on their experiences with the program, obtain confirmation of members’ activities, and gather opinions on the perceived impact of the members’ service on their organizations. Items pertaining to the effectiveness of the AOB core curriculum and site enhancements of member leadership in preparing members for leadership in community engagement activities and community engagement implementation will be added to this tool. Within this survey, the program and evaluation team will pilot an instrument for partner organizations to assess where their member’s community engagement service activities fall on a spectrum of community member participation and responsibilities in the organization that moves from less intensive (invited to town hall on new program or recruited volunteers) to more intensive (work with community members to co-design programming) activities. Existing public participation rubrics will be used to develop this tool. The survey will also include a number of closed- and open-ended questions that ask partner organizations if and how they observed members incorporating their background into their approach to service, particularly around community engagement.

***Semi-structured Individual and Focus Group Interviews****:*At each of the four select sites, we will conduct semi-structured interviews and focus groups (depending on the number of potential respondents at each site) with all members, program site staff supervisors, and partner organization program coordinators. Interviews are the likely forum to be used with staff (1-2 per site) and supervisors (generally 1 per site) given the low number of people at each site, while 1-2 focus groups of 4-7 participants each would be used to gain feedback from members (4-10 per site). Separate semi-structured protocols will be developed for individual and focus group interviews with members, site staff supervisors, and partner organization program coordinators. The protocols will be designed to last between 30 and 60 minutes. To promote a more interactive, and rich dialogue between evaluators and subjects, these interviews will prioritize open-ended questions addressing key topics related to the evaluation. The research questions will inform the focus of the interviews and include topics such as core curriculum training and how program sites enhance member leadership in community engagement (RQ#1), perceptions of community engagement activities (RQ#1) and the relationship between members’ identities and proximity to community served or social issues addressed and their approaches to service (RQ#2). Most interviews and focus groups will be conducted in-person when possible, but some may need to be conducted remotely using online video technology or telephone. 

The program year phase of the study will span 10 months of data collection with the majority of data collection occurring in the second half of the program year, giving members the opportunity to become orientated to their work with partner organizations. 

Data Procedures for Member and Partner Organization Surveys

Towards the beginning of the program year, the evaluator will obtain access to the Member Database to examine Members’ demographic characteristics from their program applications. The Member Exit Survey and the Partner Organization Exit Survey are both administered electronically on an annual basis by AOB at the end of each program year, and these data will be merged with the Member Database dataset. Potential respondents to the surveys receive a unique survey link via email address. All of the email addresses of respondents will have been recently updated within the past year.

Table 1: Evaluation Questions, Data Sources, and Analysis

Data Procedures for Select Members, Program Site staff, and Partner Organizations

For the four selected program sites, data collection will be spread out across the final two-thirds of the program year, beginning after members have become oriented at their partner organizations and program site implementation has moved beyond the program start phase. All program site staff supervisors and the partner organization program coordinator at the four sites will be invited to participate in interviews, and all members will be invited to participate in focus groups. At each site, evaluators will conduct 1-2 member focus group interviews (4-7 Members per focus group) and individual interviews with program site staff supervisors (1-2 staff members) and partner organization program coordinators (1 per site). Focus groups and interviews will be held in-person, unless a virtual video option is necessary. Interviews will last 30 minutes, and focus groups will last 60 minutes. Interviews with program site staff and partner organization program coordinators from the sample of four sites will occur in the second half of the program year.

The external evaluator will work with their own Institutional Review Board (IRB) to determine what level of consent and other human subjects procedures will be required to obtain data. Our study will follow all required guidance for human subjects research stipulated by the evaluator’s IRB. 

1. Analysis Plan

Data will be analyzed as it is collected when appropriate with much of the analysis occurring in the final part of the program year and three months immediately following the end of the program year. 

*Qualitative Data Analysis:* Interviews and focus groups will be transcribed and, along with open-ended survey questions and reflections where appropriate, transferred into analytic software for analysis, such as MAXQDA, NVIVO or ATLAS.TI. Thematic analysis of these transcripts will be conducted to identify patterns in responses and to derive themes using the following approach with each of the key research questions: 

· What common themes or patterns emerge in the responses on specific topics related to the research question? 

· Are there exceptions to these patterns and are there common characteristics associated with these exceptions? 

· What additional useful or interesting stories emerge from the responses that may help inform the research question?

*Quantitative Data*: Quantitative data collected through the Member Database and quantitative survey items will be analyzed and summarized using basic descriptive statistics (e.g., frequencies, means, etc.). Quantitative and qualitative data will be triangulated to get a better picture from multiple perspectives of how members are implementing community engagement, what affects members' experiences, etc. 

The specific ways in which the qualitative and quantitative data will be used to inform the evaluation’s research questions is described below. 

**Research Question #1**

To address the second research question, exit survey data will be analyzed using descriptive statistics to summarize perceptions on leadership training and to report on community engagement activities reported by all members and partner organizations. Also, evaluators will examine members’ open-ended responses to reflection questions on how they are implementing community engagement activities, looking for common themes and patterns, to understand variations in how programs are enhancing member leadership in community engagement. Evaluators will calculate the average member intensity score on community engagement activities as reported on the Member and Partner Exit Surveys. 

**Research Question #2**

To understand how member characteristics, identities, and proximity to social issue(s) influences members’ approaches to service and member contributions to partner organizations, the evaluators will use a combination of broad-reaching survey instruments and in-depth data sources with smaller samples of respondents to better understand this topic. First, the overall member and partner organization perspective will be obtained from calculating and reporting descriptive statistics on these topics from the two member and partner organization exit surveys. The evaluators will also calculate descriptive statistics and conduct thematic, qualitative analysis on members’ answers to a series of closed- and open-ended survey questions about the ways in which members bring their background into their approach to service, particularly around community engagement. Responses to the survey items will be merged with demographic information on member respondents from the member database and Member Exit Survey to identify associations and patterns in member characteristics and backgrounds with their approaches to community engagement. In addition to gaining the perspectives of members from the Exit Surveys, the evaluators will examine interview and focus group data from the member, site staff, and partner organization interviews/focus groups for themes on the relationship between members’ identities and proximity to communities served or social issues addressed.

1. Evaluator Qualifications

AOB will engage an external evaluator or contractor skilled in mixed methods evaluation (quantitative and qualitative data sources) to guide, co-develop, and implement the research design and evaluation. AOB prefers an evaluator who has been active in the field of program evaluation for many years and has provided evaluation in a variety of settings, especially in organizational capacity and/or community engagement measurement. In addition to skills in mixed methods evaluation, the evaluator must also demonstrate experience with program evaluation and culturally-responsive evaluation. Particularly critical to the selection of an evaluator for this project will be their experience in collecting and synthesizing large amounts of qualitative information from multiple sources for drawing important conclusions for informing program practices and recommendations. 

1. Timeline

1. Budget

AOB expects to spend $60,000 for an external evaluator, including Institutional Review Board (IRB) submission and approval, modification of existing survey instruments and observation protocols, development of new interview and focus group protocols, data collection and analysis, and report writing. We anticipate some additional costs for internal staff time managing the project and for materials. Approximately 80% of our evaluation budget is from AmeriCorps funds.

1