Public Allies External Impact Evaluation

Prepared by Data You Can Use and the Urban Institute

Public Allies Evaluation Report

December 2020
2 

Public Allies External Impact Evaluation

Prepared by 

Data You Can Use and the Urban Institute

December 2020

Introduction and Background

Purpose

The report summarizes the results of an external evaluation conducted by Data You Can Use 

(DYCU) and the Urban Institute (UI) as a requirement of the AmeriCorps Public Allies program 

supported by the Corporation for National and Community Service (CNCS). It follows a 

presentation on the methodology and findings related to the impact of Public Allies on the 

organizational capacity of host agencies, as well as discussion and review by stakeholders and 

the research partners. 

Background

As a condition of funding, CNCS required an external impact evaluation of the Public Allies 

program using an experimental or quasi-experimental design (QED) to determine the impact of 

the placement on the capacity of the host organization. Impact evaluation was defined as being 

designed to provide statistical evidence of the impact of the program compared to what would 

have happened in the absence of the program. The plan, developed by third-party evaluator, 

was submitted and approved by federal administrators. Public Allies then sought an external 

evaluator to implement the approved plan, conduct the analyses, and report on the findings.

Previous evaluations of the Public Allies program had been conducted. Specifically, in 2015 

grant cycle, an “alternative evaluation” examined the relationship between Allies and Partner 

Organizations (PO) and found that the Ally’s background was not related to capacity gains in the 

partner organizations, nor the leadership growth of the Ally. This evaluation documented that 

the program was more effective in increasing certain capacities (e.g., outreach) and that hosting 

an Ally did serve to broaden the supervisors’ notions of “qualified talent.” 1

When approaching the 2018 grant cycle, Public Allies began to explore quasi-experimental 

design options to advance to the “moderate” evidence tier which required a minimum of one 

quasi-experimental design. Evaluation plans incorporating QED of organizational capacity 

building were submitted and the final design was approved by CNCS and became the basis of 

this evaluation. The goal was to build evidence of the program’s impact on increasing and 

sustaining the capacity of nonprofit organizations through assistance of an Ally placed with the 

organization for a ten-month period. 

1 Center for Urban Initiatives and Research, University of Wisconsin-Milwaukee, Public Allies External Evaluation 

Report, October 2015 authored by Josh Lang, MA and Atiera Coleman, MA 
3 

Public Allies will submit the final evaluation report to CNCS in January 2021 with the application 

for continued funding, incorporating findings and lessons learned to help the program 

continuously improve.

Partners 

This evaluation was conducted by Data You Can Use, an independent nonprofit organization 

located in Milwaukee, Wisconsin. Data You Can Use is the Milwaukee Partner in the Urban 

Institute’s National Neighborhood Indicator Partnership (NNIP). Data You Can Use 

subcontracted with the Urban Institute, a leading research organization in Washington, DC. For 

a portion of the evaluation. Researchers Daniel Teles and Benny Docter prepared a detailed 

Analysis Memo explaining the methodology and conclusions. (The full report is submitted as an 

appendix.) Additionally, the Analysis Memo was reviewed by Drs. Nathan Dietz and LesLeigh 

Ford as independent subject and methodological experts from Urban Institute. Researchers 

from Data You Can Use included Dr. Kathleen Pritchard, Carrie Koss Vallejo, and Amy Rohan. 

Credentials of the research team are included in the appendix. Erin Guest and Sean Luedke 

from Public Allies collaborated in the execution of the evaluation and were extremely helpful in 

providing background information and detailed data documentation.

Scope 

There are multiple ways that the value of Public Allies program might be assessed. In fact, an 

earlier evaluation conducted by the Center for Urban Initiatives and Research at the University 

of Wisconsin-Milwaukee established two logic models for the program—one for impact on the 

Ally and one for impact on the hosting organization. These logic models serve to demonstrate 

the recognition of multiple levels at which the program is expected to have impact. 

From an evaluation perspective, program effects are generally most obvious and demonstrable 

at the most direct level. Clearly, serving as an Ally has a direct effect on the participants’ skills 

and capacity and there are ongoing efforts to track the progress of the Allies. The theory of 

change illustrated in the logic models also posits that there are also many ways that the Public 

Allies program may affect change at the organizational level. For example, the program might 

be expected to have an impact on the clients, volunteers, and staff with whom the Ally interacts, 

an impact on the non-operational aspects of the partner organization (exposure to younger, 

more diverse staff and ideas, growth in capacity in defined areas, etc.), as well as changes at 

the community level (e.g., increased network-building and shared learning as a result of 

interaction with other host sites.) For the purpose of this study, the focus is on the growth in and 

sustainability of organizational capacity of the host sites, as directed by AmeriCorps.

Research Questions

Specifically, this evaluation called for the analysis of two questions: the effect of the Allies’ 

presence on the capacity of the organization, and the sustainability of the increased capacity.

The design called for three assessments given as a pre-assessment, post-assessment, and 

three-month follow-up assessment. The assessments were designed to address the following 

research questions: 
4 

1. Did the Public Allies program increase the capacity of partner organizations as 

compared to a group of similar non-profit organizations that did not host a Member? 

(impact)

2. Do organizations that report capacity gains sustain and leverage those gains three 

months after the member’s term of service is complete? (outcome) 

Process

To address the first question, DYCU partnered with Public Allies to get initial data and then 

worked with methodological and substantive specialists from the Urban Institute to execute the 

matching and analysis in accordance with the proposed design. Details of the survey and 

comparison are provided in the Analysis Memo (See Appendix B, page 15 of this report.). The 

data are based on the 2018-19 program year. 

Monthly meetings were held virtually throughout the period to answer questions and keep the 

project on track. Discussion of adjustments to the initial design, due primarily to low response 

rates, and to the adaptations and options for analysis were ongoing. Preliminary findings were 

presented to the Public Allies leadership team in mid-November and the final report was 

submitted in December. 

Summary of findings

Increases in Ally-host Capabilities

The Analysis Memo in the Appendix provides a detailed description of the methodology used to 

address the first question. Urban Institute researchers structured their analysis around the four 

areas that are impacted by the questions asked of host sites by Public Allies. These include: 

1. Program Development, Delivery, and Education 

2. Outreach

3. Technology and 

4. Volunteer Management. 

Using these areas of organizational capacity, Urban Institute researchers used composite 

measures to reduce the chances of asserting a spurious causal relationship in estimating the 

impact of Public Allies on the host sites. 

The responses of Ally hosts are summarized in tables 2, 3, 4, and 5 in the Analysis Memo. On 

average, the capacity growth experienced by organizations that hosted Allies grew 0.3 points on 

the 1.0 to 4.0 scale in the questions they answered. At the end of the year, the typical Ally host 

was about midway between “somewhat capable” and “very capable” in most areas. This finding 

was remarkably consistent across questions. The largest improvement was 0.58 for “organizing 

reward celebrations for volunteer services,” followed by 0.55 for both “developing a database” 

and “creating a volunteer manual or curriculum.” The largest estimated decline in capability was 

-0.06 in “training staff on a performance management system.” This is, however, statistically 

indistinguishable from no change. For the initial comparison group, the increase was 0.2 and for 

the larger retrospective comparison group, the increase was 0.6. 
5 

The key take-aways regarding the initial question of inquiry are that:

 Public Ally-hosting organizations reported moderate, but consistent increases in their 

capabilities over the course of the service year.

 Low survey response rates among comparison organizations produced data limitations 

that prevent the ability to estimate the impact of Public Allies with confidence.

The authors conclude: 

“Our analysis of these data suggests organizational capacity typically grew during the 

time organizations hosted Allies. While this is promising, we could not isolate the impact 

of Allies on partner organizations relative to growth in a comparison group of other nonprofit organizations.” (Analysis Memo, page 1)

While the Ally host sites show increased capacity, the comparison group also shows increased 

capacity and suggests that organizations that are likely to host an Ally (as indicated by their 

previous sponsorship) are also likely to perceive increases in capacity whether or not they are 

currently hosting an Ally. The data, detailed methodology and explanations are provided in the 

Analysis Memo which is included in its entirety in this report. 

Sustained change over time

In addition to the question about impact on organizational capacity, the evaluation also sought 

to explore whether the capacity gains that organizational partners experienced were sustained 

and leveraged after the Allies’ terms of service were complete. 

Based on the End of Year Assessment, of the 206 supervisors who hosted a single Ally who 

graduated, 182 of these partners (88%) responded affirmatively to a question that asked 

“...would you say the organization's capacity has increased as a result of the Ally's service?" To 

determine whether these capacity gains were sustained, the study design called for asking 

those partners who reported that there had been an increase in capacity (182) if that had been 

sustained over time. The time period was defined in the original design as three months. As 

Table 2b shows, of the 75 (41%) responses to the follow-up survey, 64 or 85% indicated that 

the increased capacity had been sustained. 

Table 2.b. Sustained Capacity

Host Assessment of Capacity Increase at 

end of service (n=206) * 

Sustained Capacity 

follow up survey response (n=75) 

Yes No Yes No

182 (88%) 24* (12%) 64 (85%) 11 (15%) 

Among those responding that the increased capacity had been sustained, two-thirds provided 

additional qualitative explanations of their response. The most common explanation was the 

ongoing benefit of systems, organizational tools, or connections that the Ally established or 

helped to create during their placement. 
6 

“During [the Ally’s] term, she created new systems and materials that our organization 

has continued to benefit from.”

“The foundation laid by our 2018-19 Public Ally - the documents she created and 

relationships she built - has been instrumental this month as we begin our 2019-20 

student recruitment and selection season.”

In addition to explaining sustained impact in terms of products and relationships, a number of 

hosts explained that staff size increased, sometimes with hiring the Ally, acquiring another Ally 

or a continued relationship. 

“We have hired full time staff to address some of the activities that [our Ally] was 

responsible for. We have also continued to evolve and develop the work that [our Ally] 

started making connections with communities in the [area].”

“Fortunately, our capacity growth corresponded with funding growth, thus we were able 

to expand efforts in the specific initiative that our Ally was serving in. In fact, we 

expanded by creating a brand-new position which that Ally was hired for.”

“We’ve been able to maintain our relationship with our Public Ally, who is now a student 

at the University and a Graduate Assistant with us. She continues to provide in a similar 

capacity, while co-teaching and leading others to provide for our program in the ways 

that she’s helped us build and more.”

Eleven (15%) of those who indicated there had been capacity gains reported that after three 

months that capacity gain had not been sustained. All of those who replied that the capacity 

gain had not been sustained provided additional qualitative information explaining their 

response. The most common explanation was that they were lacking funds or personnel to 

maintain the capacity. 

“Although we now have a plan, we have not had the staff to [carry it out.]”

“Most of our work is about people rather than systems. Without our PA physically in the 

building, our capacity has decreased back to about what it was before.” 

Learnings from the Evaluation Findings 

Questions related to increased capacity

While the statistical implications of the assessment of impact are inconclusive in establishing 

capacity growth that differs from the comparison group, the evidence of perceived capacity 

growth and sustainability among those hosting an Ally is promising. And there are other 

potentially useful observations that might suggest opportunities for both program and evaluation 

improvements. 

For example, the assessment questions identify a total of 30 areas of potential capacity-building 

for Public Allies. The responses, which are based on the area of focus in the position

descriptions, are clustered around four capacity categories: program development (14 

questions), technology (seven questions), outreach (four questions) and volunteer management

(seven questions). The fact that there are at least twice as many items in this category suggests 

it may be combining two concepts. Based on the number of questions and responses, the 

majority of work is in the program development areas with at least two times the number of 
7 

items as the other three areas. This may be obvious from other sources, but the documentation 

here is useful and suggests a conversation regarding how this distribution reflects current needs 

and demands of host sites. It may also be instructive in designing and assessing the training 

and coaching needs for Allies to maximize their impact. 

The data indicate that, among those who responded at both the beginning and end of the 

service year, most of the activity is concentrated in two areas of capacity building: 1) Program 

Development, Delivery and Education, and 2) Outreach. Interestingly, the question relating to 

“developing an outreach plan” in the outreach capacity category received the greatest number 

of responses (143), suggesting extensive inclusion of this in the position description of Public 

Allies.2 The question relating to “improved or expanded program materials” in the program 

development capacity category similarly received 119 responses suggesting that among those 

who responded a substantial portion of Allies had that in their position description. Table 2c 

highlights those questions that received the greatest response and identifies those activities 

most requested by hosts and performed by Allies.

Table 2c Most commonly rated areas of capacity 

Question Host 

Organizations

Capacity Category

Develop outreach plan for target 

beneficiaries/ participants or volunteers

143 (60%) Outreach

Improve or expand program materials 119 (50%) Program development, delivery, 

and education

Make community presentations to improve 

beneficiary recruitment

98 (41%) Outreach

Expansion of existing programs 93 (39%) Program development, delivery, 

and education

Assessment of existing program(s) to 

improve or expand delivery

85 (36%) Program development, delivery, 

and education

Engage community voices in program 

development 

80 (33%) Program development, delivery, 

and education

Develop or pilot a new program 74 (31%) Program development, delivery, 

and education

(See tables 2-5 of the Analysis Memo)

Although the individual items, for the most part are remarkably similar and do not show much 

variation, there are some areas where capacity growth was higher (or lower) than the average. 

Interestingly, the areas where responses reflected the greatest growth in capacity were not in 

the categories with the greatest demand. Rather, based on the scores, the greatest areas of 

growth are seen in the areas of technology and volunteer management where fewer than a third 

of all Allies have these tasks included in their position descriptions. (See table 2d below)

2 239 organizations that hosted an Ally.
8 

Table 2d Top five areas of capacity building

Question Responses Avg 

growth

Capacity Category

Organize reward celebrations for volunteers 22 (9%) .58 Volunteer

Management

Develop data base (volunteer, client) 41 (17%) .55 Technology

Create volunteer manual or curriculum 27 (11%) .55 Volunteer 

Management

Pilot new technology tools 18 (8%) .51 Technology

Establish partnerships for volunteer 

recruitment

39 (16%) .47 Volunteer 

Management

(See tables 2-5 of the Analysis Memo)

Another way that the data might be useful is in looking at those areas where the host agencies 

indicated that there was the least gain in capacity. Although even in the area identified as 

showing a loss in capacity (training staff on performance management), the decrease is so 

slight as to be indistinguishable from no change. Particularly in those areas where larger 

percentages of the Allies have certain tasks in their position descriptions (e.g., developing 

mechanisms for communicating with partners or developing mechanisms for constituents to give 

feedback on services, developing cultural competence training), these numbers might suggest 

increased training and coaching for Allies tasked with these roles. 

Table 2e Areas of least gain

Question Responses Avg 

growth

Capacity

Category

Train staff on performance 

management system

8 (3%) -.06 Program development, delivery, 

and education

Establish mechanisms for 

constituents to give feedback on 

services

43 (18%) .19 Program development, delivery, 

and education

Develop mechanisms for 

communicating with current partners

94 (32%) .21 Program development, delivery, 

and education

Develop, pilot or revise internal or 

external knowledge management 

system(s)

24 (10%) .23 Technology

Develop and pilot cultural competence 

training for staff

17 (7%) .24 Outreach

Oversee volunteer service projects 28 (12%) .24 Volunteer Management

(See Tables 2-5 in the Analysis Memo)

If the areas where most Allies are working are not seeing appreciable increases and conversely, 

areas where fewer Allies are working are seeing more significant gains, it may warrant 

discussion and strategic thought. 

Questions that might be raised from this data include: 

Are there particular areas where Public Allies would like to see the greatest impact? 

Are there any priorities or specialties in this list of 30? 
9 

Are training resources distributed accordingly?

Does Public Allies have the capacity to increase capacity in each of these areas?

Do these areas reflect greatest demand of nonprofits seeking to increase their capacity?

If the goal is to increase organizational capacity in specific areas, what does the Ally 

need to know? 

How can the data be used in guiding placements?

How can the data be used in establishing training for Allies? 

Are Allies sufficiently prepared in the areas where they are expected to affect capacity? 

How can the data be used to set realistic expectations about the extent of capacity 

growth? 

How can the data be used in recruiting host sites?

Is PA addressing current demand? 

Can this data be used to help predict future demand? 

How can the data be used to set realistic expectations about the extent of capacity 

growth that can be accomplished in a ten-month period?

How can the data be used to identify the core training and competencies Allies require to 

successfully build sustainable organizational capacity?

Questions related to sustained capacity

When asked directly, nearly 90% of the host agencies indicated that having a Public Ally has 

increased their capacity and 85% indicate that the capacity is sustained over the short term. 

While these numbers are impressive, they also raise consideration of strategic questions that 

might be addressed. These might include the following: 

Is three months a sufficient time to consider an increase being sustainable?

Is the hiring of an Ally by the host organization considered a success?

Is the hiring of another Ally considered a success?

If sustained capacity is the goal, what are fundamental activities that should be 

undertaken to assure this?

Should the program track policies or procedures that institutionalize capacity?
10

Possible Implications for Future Assessment methods

Based on the number of responses to questions, most Allies have position descriptions that 

define their work as building capacity in program development, delivery and education. Since 

this category also has the most questions (14), might it be helpful to refine the category and 

better capture the components to assess this change?

Questions related to ongoing and future evaluation

Selection of the Comparison Group

The comparison group for this evaluation was identified as organizations that had previously 

hosted an Ally, organizations where a former Ally was currently working, or organizations that 

had a relationship with a site. In practice, this approach was not particularly helpful in assuring a 

good response rate and had the added potential concern of possible contamination in that the 

organization had already experienced increased organizational capacity due to the previous 

placement or the presence of a former Ally on staff. Since the comparison groups consisted of 

former Ally host sites or sites where former Allies were now employed, how does the design 

control for previous capacity built by the Ally that was sustained? 

In future evaluations, with adequate time and funding, the comparison group might be 

predetermined based on matches on other variables thought to be important (revenue, salary of 

executives, service sector, age of the organization, etc.) This might also help in drawing 

distinctions between the types of nonprofits that are likely to host an Ally and those that are not. 

Avoiding Survey Fatigue

As valuable as we believe research and evaluation are to good programming, program 

providers sometimes develop “survey fatigue” and become “evaluation weary,” often seeing the 

requests for additional tracking as take-aways from service. In fact, there is evidence (low 

response rates, requests to limit surveys, failure to perform the required assessments, etc.) that 

some of the organizational partners may be experiencing this and care should be taken not to 

overburden them in the process. Strategies that may help to prevent this in the future include:

• Creating shared ownership with partner organizations through an advisory or users 

group of people who have an interest in evaluation and who might serve to test 

products, advise on and even recruit assistance from other sites.

• Clarifying the time required for partner organizations to participate in the research 

and the consequences of nonparticipation.

• Employing retrospective surveys with a single assessment of “now” and “then” rather

than discrete pre- and post-assessments. The post-then-pre design was originally 

proposed as a way to avoid the response shift that occurs when, because of the 

intervention, assessments are based on a new understanding of the question. In 

addition to avoiding response shift bias, this approach takes less time, and is less 

intrusive. (See Appendix E for additional resources discussing the comparative use 

of pre-post v. retrospective evaluation methods) 

• Simplifying and limiting research requests to focus on the more interesting and 

important data points. 
11

• Developing a plan for distribution and dissemination of research results and assuring 

that results are continuously shared with those who respond. 

Adjusting the tool to better reflect the empirical findings

The current evaluation tool for assessing increases in capacity uses a seven-point scale with 

the ends anchored in “not capable” and “very capable and the mid-point labeled “somewhat 

capable.” The discrete points were numbered at 0.5 intervals for analysis. While this approach is 

similar to other assessments in this area, the results have been shown not to reflect the 

responses of partner organizations when they are asked directly if the organization has 

experienced an increase in organizational capacity. For example, although nearly 90% of the 

partner organizations state that there is an increase in capacity, the current calculation, which 

requires a 0.5 increase to count as progress, does not support their assessment. Based on this 

evaluation, the Urban Institute analysis finds an average increase of 0.3 which is lower than the 

standard set in the current calculation. This suggests some modification of the expectation and 

the criteria may be warranted to determine whether there is shared understanding of the 

concept of increased organizational capacity. Additionally, the research found that using 

average gains on individual items (rather than individual organizational gains across items) 

yielded a sounder basis of assessment, and this too might be considered in future assessments. 

Focus 

As previously noted, the program capacity category currently has 14 points to assess. Given 

that the other capacity areas have far fewer items, it might be worth exploring the items within 

the category to breaking it down further. It might, for example, be helpful to look for clusters or 

patterns in the data itself. Another possibility would be to begin to identify areas where there is 

relatively high demand (a larger share of Allies has the task in their position descriptions) and 

higher than average improvement in capacity. The shaded items in Table 2f, for example, 

identify some of those areas which might be focused on to train Allies to respond to common 

areas of need and see greater impact. 

Table 2f Program-- Component frequency and growth

Task in position description Frequency Growth

Improve or expand program materials 119 .28

Expansion of existing program 93 .24

Assessment of existing program(s) to improve or expand delivery 85 .37

Engage community voices in program development 80 .24

Develop or pilot a new program 74 .34

Develop mechanisms for communicating with current partners 47 .21

Establish or improve tracking of program outcomes 46 .35

Establish mechanism for constituents to give feedback on services 43 .19

Assessment of new program(s) or new aspect of an expanded program 38 .30

Research evidence-based practices/model programs 35 .27

Train staff to use new materials 29 .26

Establish mechanisms for referring constituents to other organizations 23 .27

Develop, pilot, revise performance management systems 21 .42

Train staff on performance management system 8 -.06

 High demand and greater than the average 0.3 capacity gain across items.
12

Shifting to noncategorical capacity assessment

Although promotional materials for the Public Allies program (website, brochure, etc.) refer to 

increasing the capacity of the partner organization, the reference is to “greater effectiveness, 

efficiency and reach.” The claim is not to increasing capacity in specific areas such as program, 

outreach, volunteer management or technology. Many of the expected areas of change are 

technical and challenging. For example, developing a performance management system, 

training staff on performance management, assessing a program or tracking outcomes are 

specialized fields that require training and experience. An appreciable change in organizational 

capacity in these areas would require an Ally with specialized training or expertise, and probably 

more than 10 months. Although these subsets have been used to judge increased 

organizational capacity because they are in the Allies’ position descriptions, they do not directly 

address the program’s promise of increased effectiveness, efficiency, or reach. 

Suggestions to address this challenge include:

• Consider and be explicit about the various levels of expected impact (participant, 

program, agency, system, cross-system, community) of Public Allies and recognize 

the challenges, limitations, and benefits of assessing impact at the various levels as 

well as the inter-relationships among these levels. 

• Categorize position description elements by the promised impact of effectiveness, 

efficiency or reach to assess progress this way. For example, tasks associated with 

assessing programs or processes might be considered aimed at effectiveness, tasks 

associated with training might be associated with efficiency, or tasks associated with 

expanding audiences or programs might be associated with reach. Since these 

would be “after the fact” categories, it is uncertain whether there would be sufficient 

payoff to this approach. 

• Explore, possibly through qualitative interviews or focus groups, the partner 

organizations’ assessment of increases in effectiveness, increased efficiency, and 

increased reach, and the perceived value of the program from their perspective. 

Suggested areas for future research

While the main recommendation is to simplify and focus on the most direct and intentional 

effects of the Public Allies program, there are interesting ways an assessment of the impact on 

organizational capacity might be further developed in the future. Two directions stand out.

• Frame the evaluation and focus on the promised change of efficiency, effectiveness 

and reach (as opposed to organizational capacity growth areas based on position 

descriptions) and engage with host site supervisors in providing and collecting 

examples that could inform a future measurement system. 

• Explore the effects of Public Allies on the more systemic capacity of host 

organizations such as increased willingness to hire nontraditional applicants, 

increased community network-building and shared learning as a result of interaction 

with the program and other host sites.

 
13

Appendices

A. Research partners

B. Urban Institute Analysis Memo

C. Sustainability methodology 

D. Resources on retrospective analysis
14

Appendix A. Research partners

Data You Can Use is an independent non-profit organization, based in Milwaukee, Wisconsin, 

founded in 2016 to provide data services to the foundation and nonprofit community. The 

organization is comprised of a diverse group of local professionals who help people access data 

and make it useful in improving community conditions. 

Dr. Kathleen Pritchard, President of Data You Can Use, served as lead evaluator for 

this project. Her research and evaluation expertise focus on outcome measurement, the 

creation of theories of change and logic models, making data useful, and assessing 

cross-sector, collective impact initiatives. She has conducted quantitative and qualitative 

evaluations and specializes in partnering to produce the most useful evaluation products 

for clients. Over the course of the evaluation, Carrie Koss Vallejo and Amy Rohan

served on the DYCU evaluation team. Ms. Vallejo is a former Data Advocate with the 

organization and worked to refine the comparison group survey at the onset of the 

project. Ms. Rohan is a Data and Research Analyst who joined the team in June of this 

year and assisted in the final analysis. 

Urban Institute The nonprofit Urban Institute is dedicated to elevating the debate on social and 

economic policy. For nearly five decades, Urban scholars have conducted research and offered 

evidence-based solutions that improve lives and strengthen communities. Urban’s objective 

research helps expand opportunities for all, reduce hardship among the most vulnerable, and 

strengthen the effectiveness of the public sector. 

The Urban team was led by Dr. Daniel Teles, an economist and policy analyst whose

research focuses on the effects of public policies on poverty, inequality, and the

nonprofit sector. Dr. Teles has experience with the CNCS approach to evaluating public 

service programs; he was awarded a 2015 National Service and Civic Engagement 

Research Competition grant by CNCS and is currently engaged in two similar projects in 

New York. He led the quantitative analysis. Benny and assist with the review of 

adaptations to the evaluation design. Benny Docter is a research assistant at UI 

specializing in affordable housing. His work included matching of the comparison groups 

and analysis of the data. The team also benefited from the review of Dr. Nathan Dietz, a 

senior research associate in the Urban Institute’s Center on Nonprofits and Philanthropy

served as technical advisor and Dr. LesLeigh Ford, research associate, served as 

reviewer.

Public Allies

Special thanks are extended to Erin Guest, National Administrator of Program 

Operations at Public Allies who consistently asked good questions and assured the work 

was practical, and to Sean Luedke, Director, Research and Innovation who provided 

well-documented data and background during the research process. 
15

Daniel Teles, Benny Docter

December 2020* 

Public Allies works to place and support AmeriCorps service members (Allies) within 

partnering non-profits across the country. To evaluate their work, Public Allies collected 

survey data from these partner organizations. In these surveys, supervising staff at the nonprofits assessed their organization’s capabilities both before and after their corps members 

served in their organizations. Our analysis of these data suggests organizational capacity 

typically grew during the time organizations hosted Allies. While this is promising, we 

could not isolate the impact of Allies on partner organizations relative to growth in a 

comparison group of other non-profit organizations. 

KEY TAKEAWAYS

 Public Ally-hosting organizations reported moderate, but consistent increases in their 

capabilities over the course of the service year.

 Low survey response rates among comparison organizations produced data limitations that 

prevented us from estimating the impact of Public Allies with confidence

METROPOLITAN HOUSING AND COMMUNITIES POLICY CENTER APPENDIX B 

Analysis Memo: Organizational 

Capacity Among Public Allies’ Hosts

What Survey Data Shows About Corps Member’s Service 
16

 *Tables 2, 3, 4, and 5 have been updated from an earlier version of this memo.

Survey and Study Design

This evaluation focused on self-reported capacity outcomes collected in a survey. To understand the 

impact of Public Allies on host organizations, Public Allies and the research team partnered to send 

identical beginning and end of year surveys to both host organizations and a comparison group. The 

group of host organizations included all organizations who hosted an ally during the 2018-19 year. 

Supervisors from these organizations were required to complete the survey as part of their partnership 

with Public Allies. The comparison group was selected based on outreach to former ally hosts who were 

referred by Public Allies. We expect that past participation with Public Allies meant that these 

organizations have similar goals, constraints, and capacities as organizations currently in the program. 

Original Study Design

The survey was designed to be implemented immediately before and after Allies served in their 

organizations. For organizations without Allies this year, we asked these questions at the same two 

points in time, although no term of service occurred during this interval.3

 

While there were two different surveys, both sets of surveys included questions drawn from the same 

set of 30 total possible questions. Question text perfectly aligned between surveys. In the original survey 

design, the only thing that differed between the surveys administered to the host organizations and the 

comparison group was which (and how many) questions the organization received. Hosting organizations 

only received questions aligned to their pre-identified goals for their Ally’s service. This meant that 

hosting organizations responded to individualized subsets (ranging in size between 4 and 14) questions. 

These were all drawn from the 30 possible questions in the survey. Respondents in the comparison group 

were asked to respond to all 30 questions. 

Each of the questions was prefaced with the text “Given current levels of funding, staffing and 

volunteers, rate how capable your organization is at completing the following activities...”. This was then 

followed by one of 30 specific phrases, designed to measure a specific organizational capacity. 

Respondents were asked to rank their capacity on a scale between “Not Capable”, “Somewhat: 

Capable”, and “Very Capable” (figure 1). The survey included seven mutually exclusive points on the scale 

as potential responses. That is, while the scale only had those three labeled options, there were 4 

unlabeled choices, two on each side of the middle “Somewhat Capable” choice. We converted these 

options into a scale the that ran from 1.0 to 4.0. We assigned the scale so that each of the numbers 1, 1.5, 

2, (and so on) represented one of the seven choices on the survey.

3 Although this memo was authored by Daniel Teles and Benny Docter at Urban Institute, “we” refers to the full 

study team from Urban Institute and Data You Can Use. 
17

FIGURE 1

Survey Ranking Scale

Modified Study Design

Out of 437 organizations who were solicited, only 31 organizations responded to the initial comparison 

group survey request. We refer to these 31 organizations as the initial comparison group. 

To increase sample size of the comparison group, we modified the study design to include a second

comparison group. The second comparison would receive only one survey. That survey included the same 

set of 30 questions, but this version noted “the survey asks for your assessment at two points in time: 

‘Then’ (looking back at August/September of 2018) and ‘Now’ (Your assessment in August of 2019.)” Each 

question had two selection rows with identical choices beneath it. One was labeled “Then” and the other 

was labeled “Now”. 

Respondents rating their capacity both “Then” and “Now” at the same time may perceive or report 

larger gains than if they answered the questions as different points in time. To increase comparability with 

the retrospective group, we also modified the end of year surveys sent to Ally hosts and the initial 31 

comparison group organizations. The modified surveys showed respondents the results of their beginning 

of year assessment. With this modification, all groups would see their beginning- of- year assessment at 

the time that they provided their end- of-year assessment.

Table 1 summarizes responses to each phase of the survey. Of the initial comparison group, 19 

completed the end of year survey. We received 48 responses to the retrospective survey. We refer to 

these organizations as the retrospective comparison group. 

TABLE 1

Sample Size

Responses 

in 2018

Responses 

in 2019

Used in Impact 

Analysis

Ally-hosting Organizations 415 287 239 

Initial Comparison Group 31 19 17 

Retrospective Comparison Group n/a 48 38 

Combined Comparison Group 31 67 55 

Note: Only nonprofit organizations with beginning- and end-of-year responses, that also appeared in the 2015 IRS datafile on 

nonprofit entities were included in the impact analysis. 
18

Quantitative Methods

To understand the impact Public Allies had on their hosts organizational capacities, we examined 

changes in Ally hosts over the course of the year and compared changes for Public Ally hosts with 

changes in the comparison group. We estimated the impact of Public Allies using difference-indifferences with regression adjustment.

Summarizing Improvements in Host Capacities

To understand improvements in capabilities made by Ally-hosts during the service year, we first 

examined the difference between the beginning of year and end of year capacity scores reported in the 

survey. We then calculated the average improvement (or decline) in capacity for each question. Finally, 

we used t-tests to assess whether the improvement (or decline) is likely driven by a consistent increase 

across organizations or random chance. 

Estimating the Impact of Public Allies

Our goal is to understand the impact of Public Allies on their host organizations. Ideally, we would 

compare the survey responses of host organizations to the responses of the same organizations in a 

world in which everything was the same, except they did not host an Ally. The difference in responses in 

the world with and without a Public Ally would be the treatment effect of the Ally. 

Since we do not know how host organizations would have responded to the survey if they had not 

hosted an Ally, we need to estimate the treatment effect.4 We do this by calculating a difference-indifferences using regression adjustment. This is a three-step process. The first step is to calculate the “first 

difference” which is the difference in responses between the beginning of year and end of year capacity 

scores. We do this for each question and for each host organization and each organization in the 

comparison group. The next step is to estimate the relationship between a set of predictor variables and 

the outcome measures separately for Ally hosts and for the comparison group. This is the regression 

adjustment. The third step is to calculate the difference between predicted outcomes for each group. We 

implement steps two and three using Stata’s “teffects ra” command, estimating the average treatment 

effect on the treated. Teffects ra estimates standard errors that account for uncertainly in both steps. 

Additional information on this method can be found in Cattaneo (2010). 

To better account for differences between organizations we collected 2015 IRS filing data for 

nonprofit organizations that included organization’s revenue, salary of executives, service sector, and age. 

Adding these data allowed us to better account for differences in organizational capability unrelated to 

Public Allies. It also led to us to exclude government agencies and nonprofits that were less than four 

years old. This further reduced our sample to 239 Public Allies hosts and 55 comparison organizations. 

Our implementation of the regression adjustment uses the following predictor variables: the average, 

beginning-of-year score given by respondents on all of the other questions, the natural log of total 

revenue reported in 2015, the natural log of total officer compensation in 2015, a binary indicator 

4 In the original study design, we intended to estimate the treatment effect by creating a matched “control group”. 

Because of the small number of comparison organizations from which to create a control group, and the weak 

relationship between the predictor variables and the probability of treatment, we determined that this method 

would be ineffective.
19

differentiating between human services organizations and other organization types, and the year the 

organization was founded (or first filed as a nonprofit). 

Multiple Outcomes Analysis

In any program evaluation, there is some risk that outcomes attributed to the program happened by

chance (a false positive or type I error). Because our study includes many questions, there’s a 

heightened risk of a false assertion of impact. On average, with 30 total questions, and using a 95% 

confidence interval, we’d expect 1-2 of these to show up as having an impact based solely on random 

chance. To adjust for this, we follow Kling et al. (2007) and focus on summary measures of impact. 

We structured our primary analysis for the impact study around the 4 groups of survey questions: 

1. Development, Delivery, and Education

2. Outreach

3. Technology

4. Volunteer management

Using these groups, we then used composite measures of program impact to estimate Public Allies’ 

impact. By doing this, we greatly reduced our chances of asserting a spurious causal relationship. 

The composite measures are calculated as follows. First, for each question we calculate the first 

difference, the difference in responses between the beginning of year and end of year capacity scores. 

Second, we calculate the average and standard deviation of the first differences in the comparison group. 

Third, we normalize each the first differences for each respondent by subtracting comparison group 

average and dividing by the comparison group standard deviation. Fourth, for each organization, we 

calculated the average of the normalized measures in each category. Finally, we performed the two-step 

regression adjustment procedure to estimate a treatment effect. 

Increases in Ally-host Capabilities

We summarize the responses of Ally hosts in tables 2, 3, 4, and 5. On average, the capacity growth 

experienced by organizations that hosted allies grew 0.3 points on the 1.0 to 4.0 scale in the questions 

they answered. At the end of the year, the typical Ally host was about midway between somewhat 

capable and very capable in most areas. This finding was remarkably consistent across questions. The 

largest estimated decline in capability was -0.06, statistically indistinguishable from no change, in 

training staff on a performance management system. The largest improvement was 0.58 for organize 

reward celebrations for volunteer services, followed by 0.55 for both developing a data base and 

creating a volunteer manual or curriculum. 
TABLE 2

Public Ally Responses to Questions about Program Development, Delivery and Education Capacity

Question

Number of 

Responding 

Organizations

Average 

Response 

Prior to 

Hosting an 

Ally (August 

2018)

Average 

Response 

After to 

Hosting an 

Ally (August 

2019)

Average 

Growth 

Over 

Service 

Term

Confidence 

Level

(p-value)

Develop or pilot a new program 74 2.89 3.23 0.34 0.001 

Assessment of new program(s) or new aspects of an expanded 

program 

38 2.97 3.27 0.30 0.002 

Expansion of existing programs 93 2.97 3.21 0.24 0.002 

Assessment of existing program(s) to improve or expand 

delivery 

85 2.89 3.26 0.37 0.000 

Improve or expand program materials 119 2.99 3.28 0.28 0.000 

Train staff to use new materials 29 2.87 3.12 0.26 0.012 

Engage community voices in program development 80 2.91 3.15 0.24 0.005 

Establish mechanism for constituents to give feedback on 

services 

43 2.90 3.09 0.19 0.031 

Research evidence-based practices/model programs 35 2.66 2.93 0.27 0.089 

Establish or improve tracking of program outcomes 46 2.88 3.23 0.35 0.000 

Develop, pilot, revise performance management system 21 2.72 3.14 0.42 0.028 

Train staff on performance management system 8 3.03 2.97 -0.06 0.634 

Develop mechanisms for communicating with current partners 47 2.83 3.05 0.21 0.010 

Establish mechanisms for referring constituents to other 

organizations 

23 2.73 3.00 0.27 0.040 

Source: Authors’ calculations of survey data
21

TABLE 3

Public Ally Responses to Questions about Outreach Capacity

Question

Number of 

Responding 

Organizations

Average 

Response Prior 

to Hosting an 

Ally (August 

2018)

Average 

Response After 

to Hosting an 

Ally (August 

2019)

Average 

Growth Over 

Service Term

Confidence 

Level

(p-value)

Develop outreach plan for 

target beneficiaries/participants or volunteers

143 2.88 3.21 0.34 0.000 

Make community presentations to improve 

beneficiary recruitment

98 2.96 3.39 0.44 0.000 

Revise existing program to reach additional 

populations through language access

12 2.50 2.92 0.42 0.064 

Develop and pilot cultural competence training 

for staff

17 3.18 3.41 0.24 0.082 

Source: Authors’ calculations of survey data

TABLE 4

Public Ally Responses to Questions about Technology Capacity

Question

Number of 

Responding 

Organizations

Average 

Response Prior 

to Hosting an 

Ally (August 

2018)

Average 

Response After 

to Hosting an 

Ally (August 

2019)

Average 

Growth Over 

Service Term

Confidence 

Level

(p-value)

Develop database (volunteer, client) 41 2.62 3.16 0.55 0.000 

Pilot or revise database 29 2.78 3.03 0.26 0.063 

Develop, pilot or revise internal or external knowledge management

system(s) 

24 2.81 3.05 0.23 0.117 

Develop social media platforms or websites to promote volunteer o

pportuniti 

48 2.59 3.03 0.44 0.000 

Pilot new technology tools 18 2.57 3.08 0.51 0.007 

Source: Authors’ calculations of survey data
22

TABLE 5

Public Ally Responses to Questions about Volunteer Management Capacity

Question

Number of 

Responding 

Organizations

Average 

Response Prior 

to Hosting an 

Ally (August 

2018)

Average 

Response 

After to 

Hosting an 

Ally (August 

2019)

Average 

Growth Over 

Service Term

Confidence Level

(p-value)

Train volunteers 55 3.00 3.28 0.28 0.000 

Establish partnerships for volunteer recruitment 39 2.76 3.23 0.47 0.000 

Create volunteer manual or curriculum 27 2.70 3.25 0.55 0.000 

Design system to track and manage volunteers 32 2.77 3.03 0.26 0.011 

Oversee volunteer service projects 28 2.86 3.10 0.24 0.077 

Organize reward celebrations for volunteer service 22 2.52 3.10 0.58 0.001 

Evaluate volunteer impact 16 2.67 3.03 0.36 0.055 

Source: Authors’ calculations of survey data
23

Estimating Public Allies’ Impact on Host Capabilities 

Survey responses suggest significant growth was experienced by all three groups (Table 6). The hosting 

organizations, the initial comparison group, and the retrospective comparison group. On average, 

hosting organizations improved their self-reported capabilities by .3 points during the year that they 

hosted an Ally. During this period, the initial comparison group showed less growth but still grew an 

average of .2 points in each question. The largest growth was observed in the retrospective comparison 

group, which grew on average .6 points in each question response. However, our estimates of the 

impact the allies had on their hosting organizations are inconclusive. 

Differences in initial capacities emerge immediately

Reviewing the responses representing capabilities in August 2018, we observe differences across the 

three groups. The group with the highest starting capacity was the first comparison group that answered 

sequentially but did not host an ally in between. The second highest reported rating in prior capacity 

was the treated group, who hosted allies. The lowest rated capacity was the retrospective comparison 

group.

TABLE 6 

Hosting 

Organizations Non-Hosting Organizations

Treatment Group

Initial Comparison 

Group 

Retrospective 

Comparison 

Group 

Combined 

Comparison 

Group 

Number of 

Organizations 239 17 38 55 

 

Average number 

questions 

completed 5.0 29.9 28 28.6 

Average Score 

Representing 

Period Before Ally 

Cohort 2.9 3.0 2.8 2.9

Average Score 

Representing 

Period After Ally 

Cohort 3.2 3.2 3.4 3.3

Source: Authors’ tabulation of survey responses.

Notes: The Ally hosting organizations and the initial comparison group responded to the survey at the beginning and end of the 

service year. The retrospective comparison group responded to a single survey about their capabilities currently and one year prior. 

We have two primary hypotheses about why these differences were observed.

1. It is possible that the prospect of immediately ranking how much growth your organization 

had experienced caused respondents to under-rank their retrospective period.
24

2. A second possibility is that the organizations who didn’t respond to the first set invitations 

to participate, were, on average lower capacity organizations. 

While the initial capacity differences between the three groups are not large, the difference between 

the two comparison groups is greater than the difference between the Ally-hosting group and either 

comparison group. 

Assumptions Required to Make Causal Inferences 

Given the differences between the two comparison groups we had three options for estimating the 

impact of Public Allies:

1. Option 1: Use only the initial comparison group 

We eliminate the retrospective comparison group, determining that the different 

format made them incomparable with the Ally hosts whose surveys had used the 

original design. This would shrink our comparison group to 14 organizations.

2. Option 2: Assume all responses are comparable

We include all observations and assume that they are equally valid and comparable, 

regardless of how the questions were asked. This would create one comparison group 

of 55 organizations.

3. Option 3: Adjust for the retrospective format

We could include all observations and adjust our calculation of treatment effects to 

estimate and account for the extent to which taking the retrospective survey had 

changed people’s responses. This assumption would essentially be a compromise 

between Option 1 and Option 2, and would leave the comparison group at 55 

observations, but the additional model complexity reduces statistical power. 

The original study design is best aligned with option 1 as the original study design did not include a 

second comparison group. Under the modified study design, we hoped to minimize differences between 

the two comparison groups by providing the initial group with their prior answers. If the two groups 

appear otherwise similar, the analysis could proceed with option 2. However, the retrospective 

comparison group consistently gave themselves lower scores on their beginning of year capacity and 

higher scores on their end of year capacity than the initial comparison group which answered the survey 

at two points in time (table 6). despite our efforts, differences between the two comparison groups 

were larger than differences between either group and the Ally hosts. We, therefore, prefer the third 

option that accounts of differences in survey type. (Results under option 1 and option 2 appear as an 

appendix.)

Impact Estimates

To account for differences in survey format, we added a “survey format” variable to the regression 

adjustment model. We then allowed the regression model to estimate how much of the difference 

could be attributed to survey format, as opposed to the other considerations already in the model. 

Figure 2 shows the estimated treatment effect of hosting an Ally on the multiple outcomes measures 

related to each capacity area. We cannot confidently determine whether Allies had an impact on their 
25

hosting organizations. The direction of Allies impact on their hosts was mixed, and none of the differences 

between Ally-hosts and the comparison group are statistically significant with 95% confidence. 

FIGURE 2

Treatment Effects Controlling for the Differences Between Surveys

Source: Urban Institute Analysis of Public Allies and Data You Can Use survey data.

Note: Using a 95 percent confidence interval, solid bars represent statistically significant changes calculated between data that 

describe hosting organizations and weighted comparisons organizations. Outlined bars represent differences that were not found to 

be statistically significant. In order to combine questions into a single measure, the differences represented have been adjusted and 

re-weighted according each question’s standard errors. As result, they do not directly correspond to the survey scale. 

Appendix: Impact Estimates Under Alternative Assumptions

Using only the Initial comparison group

In line with the first assumption, we analyzed all the hosting organizations and the respondents to the 

original comparison-group survey. Figure A1 shows the estimated treatment effect of hosting an Ally 

question groups relating to each capacity area. The effect sizes are determined based on our regression 

adjustment model.
26

FIGURE A1 

Treatment Effects Using Only Initial Respondents

Source: Urban Institute Analysis of Public Allies and Data You Can Use survey data. 

Note: Using a 95 percent confidence interval, solid bars represent statistically significant changes calculated between data that 

describe hosting organizations and weighted comparisons organizations. Outlined bars represent differences that were not found to 

be statistically significant. To combine questions into a single measure, the differences represented have been adjusted and reweighted according each question’s standard errors. As result, they do not directly correspond to the survey scale.

As figure A1 shows, the hosting organizations compare favorably with the first comparison group. Using 

only those observations, we found a statistically significant impact on organization’s capacity to conduct 

outreach. Two of the other areas, Technology and Volunteer Management also performed better, on 

average, but the differences aren’t large enough to be considered statistically significant with 95 percent 

confidence. These results are promising, but the small sample size in the comparison group is a concern. 

Assuming all responses are Comparable

Our second possible approach was to simply use all the data we had. These results appear in Figure A2. 

Drawing conclusions from these results rests on the idea that the different survey formats had little 

impact on how people responded. However, we determined that the differences we observed between 

the responses we received from the two comparison groups are dissimilar enough to warrant that we

discard these findings. Without other information, it seems reasonable to assume that these two groups 

answered the surveys with different scores because the survey questions were written in different ways. 

Ignoring these concerns, our analysis would suggest that Public Allies led to less organizational growth 

for their hosting organizations. Ally-host organizations showed less growth, on average, in every area. Only 

in Program Development, Delivery and Education, however, was this difference statistically significant at 

the 95 percent confidence level. 
27

FIGURE A2

Treatment Effects Ignoring the Differences Between Surveys

Source: Urban Institute Analysis of Public Allies and Data You Can Use survey data.

Note: Using a 95 percent confidence interval, solid bars represent statistically significant changes calculated between data that 

describe hosting organizations and weighted comparisons organizations. Outlined bars represent differences that were not found to 

be statistically significant. To combine questions into a single measure, the differences represented have been adjusted and reweighted according each question’s standard errors. As result, they do not directly correspond to the survey scale.
28

References 

Cattaneo, M.D., 2010. Efficient Semiparametric Estimation of Multi-Valued Treatment Effects Under 

Ignorability. Journal of Econometrics, 155(2), pp.138-154.

Kling, J.R., Liebman, J.B. and Katz, L.F., 2007. Experimental analysis of neighborhood effects. 

Econometrica, 75(1), pp.83-119.
29

About the Authors

Daniel Teles is a research associate in the Metropolitan Housing and Communities Policy Center at the 

Urban Institute, where he specializes in applied microeconomic policy analysis. His research examines the 

effects of public policy on local communities. Previously, Teles worked on Louisiana’s Hazard Mitigation 

Grant Program and for the New Orleans Area Habitat for Humanity. Teles earned a bachelor’s degree from 

the George Washington University and a master’s degree and a doctorate degree in economics from 

Tulane University. At Tulane, Teles was a community-engaged graduate fellow and coprincipal investigator 

of the AmeriCorps Crowd Out Study. He has contributed to the Journal of Economic Inequality, the 

Handbook of Research on Nonprofit Economics and Management, and the Lincoln Institute for Land 

Policy’s Significant Features of the Property Tax.

Benny Docter is a research assistant in the Metropolitan Housing and Communities Policy Center at the 

Urban Institute. In both professional and academic contexts, Docter has built resident databases and used 

them to analyze long-term trends in tenancy. His research interests include affordable housing and 

neighborhood housing density. Before joining Urban, Docter worked in affordable housing development 

and community programing. He graduated with honors from Wesleyan University, majoring in economics 

and social studies.

Acknowledgments

This memo was funded by and prepared for Data You Can Use (DYCY) as a subcontract to the evaluation 

of Public of Allies funded by Public Allies.

The views expressed are those of the authors and should not be attributed to the Urban Institute, its 

trustees, or its funders. Funders do not determine research findings or the insights and recommendations 

of Urban experts. Further information on the Urban Institute’s funding principles is available at 

urban.org/fundingprinciples. 

Per subcontract agreement, DYCY can copy any part of this text or the memo in its entirety in a final 

report to Public Allies, provided they list Daniel Teles and Benny Docter as co-authors on final report and 

provide Teles and Docter a draft prior to sharing with Public Allies. 

ABOUT THE URBAN INSTITUTE

The nonprofit Urban Institute is a leading research organization dedicated to 

developing evidence-based insights that improve people’s lives and strengthen 

communities. For 50 years, Urban has been the trusted source for rigorous analysis 

of complex social and economic issues; strategic advice to policymakers, 

philanthropists, and practitioners; and new, promising ideas that expand 

opportunities for all. Our work inspires effective decisions that advance fairness 

and enhance the well-being of people and places.

Copyright © September 2020. Urban Institute. Permission is granted for 

reproduction of this file, with attribution to the Urban Institute. 

500 L’Enfant Plaza SW 

Washington, DC 20024 

www.urban.org
30

Appendix C. Sustainability Methodology Note 

There was a total of 492 supervisors in 2018-2019. Of these, 355 hosted an Ally who graduated. 

182 of those supervisors hosted a single Ally AND responded affirmatively to the question of 

their capacity increasing. There were an additional 24 supervisors that hosted more than one 

Ally and replied affirmatively that each of the Allies increased their organization's capacity. To 

avoid contamination, those hosting more than one Ally, those who graduated after October 16, 

2019 and those who did not graduate or were still serving, were not included in the survey pool. 

The question in the End of Year Assessment reads "Based upon the above and your 

observations of the Ally and your organization, would you say the organization's capacity has 

increased as a result of the Ally's service?"

In October of 2019, a follow-up survey was sent to the 182 supervisors who hosted an Ally who 

graduated before July and had indicated in the previous survey that there was an increase in 

agency capacity after the Ally was hosted. The question in the follow-up survey read: “At the 

end of [the Ally’s] term of service, you reported that your organization’s capacity had increased 

as a result of their work. Now, about three months later, would you say that the capacity gain 

has been sustained?” 

Respondents could indicate “yes” or “no” and had the opportunity to add additional comments 

responding to the question that read “If you answered "Yes" how has your organization 

continued to maintain and build upon the capacity increase? If you answered "No," what have 

been the challenges / barriers?”
31

Appendix D. Resources on Retrospective Analysis

Using a Retrospective Pre-Post Questionnaire to Determine Program ... 

www.joe.org/joe/2003august/tt4.php

Use Retrospective Surveys to Obtain Complete Data Sets and ...

www.joe.org/joe/2004april/rb2.php

The retrospective pre-post: a practical method to evaluate learning ...

www.ncbi.nlm.nih.gov/.../22320369

Capturing Change: Comparing Pretest-Posttest and Retrospective Evaluation Methods.

https://edis.ifas.ufl.edu/wc135Available at edis.ifas.ufl.edu/wc135 

Developing a Survey Instrument: Program Evaluation Resources 

njaes.rutgers.edu/evaluation/resources/survey-instrument.asp

The Retrospective PrePost: A Practical Method to Evaluate Learning ...

onlinelibrary.wiley.com/doi/10.1111/j.1553-2712.2011.../pdf 

The Retrospective Pre-test References

http://www.ksbe.edu/_assets/spi/pdfs/survey_toolkit/ks_tools/The_Retrospective_Pre.pdf

Using the Retrospective Post-then-Pre Design - University of ...

https://fyi.extension.wisc.edu/programdevelopment/files/2016/04/Tipsheet27.pdf