1 

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

External Evaluation Report 

October 2015 

Final Report 

Prepared by 

Josh Lang, MA and Atiera Coleman, MA 

Center for Urban Initiatives and Research 

University of Wisconsin-Milwaukee 

Acknowledgement 

The UWM Center for Urban Initiatives and Research wishes to express their 

sincere gratitude to Erin Guest and Sean Luedke of Public Allies for their 

assistance in data collection, data access, and survey development for this 

evaluation 

. 
2 

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Contents

I. ............................................................................................................................................... Summary4 

II. ........................................................................................................................................ Background7 

Evaluation Team ............................................................................................................................ 7 

What is Public Allies? .................................................................................................................... 9 

Ten-Month Apprenticeship Program ......................................................................................... 9 

Program Reach ........................................................................................................................... 110 

III................................................................................................................Evaluation Description13

Purpose of Evaluation ................................................................................................................. 13 

Overall Evaluation Design .......................................................................................................... 13 

Evaluation Questions .................................................................................................................. 14 

IV. .............................................................................................................Evaluation Components15

Partner Organizations .......................................................................................................... 15 

Partner Organization Outcome Measures................................................................................ 16 

Outcome Measure1 .................................................................................................................. 16 

Outcome Measure 2 ................................................................................................................ 17 

Data Collection Instruments ...................................................................................................... 17 

Data Collection Procedures ........................................................................................................ 18 

Results ........................................................................................................................................... 17 

Sample Size ............................................................................................................................... 17 

Improved Technical Capacity ................................................................................................ 22 

Capacity Change Scores .......................................................................................................... 31 

Adaptive Capacity .................................................................................................................... 33 

Implications .................................................................................................................................. 35 

Recommendations ....................................................................................................................... 36 

Limitations ................................................................................................................................... 37 

Current Ally Apprentices ..................................................................................................... 37

Current Ally Retrospective and Post Ally Survey Outcome Measures ................................. 38 

Outcome Measure 1: Increased 21stCentury Skills ............................................................. 38 

Socially Responsible Leadership Scale (SRLS-R2) ............................................................. 38 

Collaboration ............................................................................................................................ 39 

Controversy with Civility. ....................................................................................................... 39 

Leadership ................................................................................................................................ 39 

Self Confidence ........................................................................................................................ 39 

Resilience .................................................................................................................................. 41 

Outcome Measure 2: Increased Civic Knowledge ............................................................... 41 

Outcome Measure 3: Increased Interest in Higher Education ......................................... 41 

Data Collection Instruments ...................................................................................................... 42 

Data Collection Procedures ........................................................................................................ 42 

Results ........................................................................................................................................... 42 

Current Ally, Retrospective Survey Between-group Outcomes Comparison. ................. 45 
3

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Current Ally, Post Survey Between-group Outcomes Comparison. ................................. 46 

Current Allies: Disconnected Allies Outcome Comparison ............................................... 47 

Current Ally, Connected Ally Outcome Comparison .......................................................... 48 

Implications.................................................................................................................................. 53 

Public Allies Alumni .............................................................................................................. 56 

Alumni Ally Outcome Measures ................................................................................................ 57 

Outcome Measure 1: Civic engagement................................................................................ 57 

Outcome Measure 2: Increased Employment in Family Sustaining Careers.................. 57 

Outcome Measure 3: Increased Pursuit of Higher Education .......................................... 57 

Data Collection Instruments ...................................................................................................... 58 

Data Collection Procedures ........................................................................................................ 58 

Results ........................................................................................................................................... 60 

Alumni Ally Outcome Comparison. ...................................................................................... 60 

Alumni Allies, Disconnected During the Program Outcome Comparison ...................... 60 

Alumni Allies, Connected Ally Outcome Comparison. ....................................................... 62 

Alumni Pursuit of Higher Education .................................................................................... 62 

Alumni Increased Employment ............................................................................................. 63 

Implications.................................................................................................................................. 65 

Limitations ................................................................................................................................... 67 

Public Allies Site Directors ................................................................................................. 67 

Data Collection Instruments and Procedures...................................................................... 68 

Results ....................................................................................................................................... 69 

Implications.............................................................................................................................. 74 

Recommendations ................................................................................................................... 75 

Limitations ............................................................................................................................... 75 

Appendix A: CNCS Charts ......................................................................................................... 75

Appendix B: Partner Organization Theory of Change ........................................................... 76

Appendix C: Capacity Self-Assessment for Partner Organizations ..................................... 77

Appendix D: Ally-focused Theory of Change .......................................................................... 89

Appendix E - Current Public Allies Survey ............................................................................. 90

Appendix F - Public Allies Alumni Survey .............................................................................. 99 
4

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 
5

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

I. Summary

Overall Evaluation Design

This evaluation examines the four main components of the Public Allies program: 1) Partner 

Organizations (PO’s), 2) Current Ally apprentices, 3) Alumni Ally apprentices and 4) Site Directors. 

A mixed-methods approach was utilized, employing both qualitative and quantitative data 

collection techniques such as surveying, secondary data analysis, and key informant interviews. 

The evaluation of Public Allies placed a special focus on between-group comparisons of four types 

of Allies: 1) non-degreed, economically disadvantaged Allies; 2) non-degreed, non-economically 

disadvantaged Allies; 3) degreed, economically disadvantaged Allies; and 4) degreed, noneconomically disadvantaged Allies. However, due to an insufficient number of cases in each group, 

categories were collapsed into dichotomous groups for comparison purposes. The three groups 

representing diverse and uncommon Allies; 1) non-degreed, economically disadvantaged Allies, 2) 

non-degreed, non-economically disadvantaged Allies, and 3) degreed, economically disadvantaged 

Allies, were combined to form a single group representing of disconnected Allies and adults which 

will furthermore be referred to as “disconnected Allies” in this report. The comparison group was 

comprised of degreed, non-economically disadvantaged Allies and adults who are referred to as 

“connected Allies.”

Evaluation Purpose 

The evaluation is designed to provide quantitative evidence of the impact of the program. 

The goal of the evaluation is to provide empirical evidence of impact at the partner organization 

and individual Ally level, thereby demonstrating that the program is a valid intervention to effect 

community problems. In addition, the evaluation, which includes both impact and process 

measures, will serve as a vehicle to examine the Allies program more closely in order to inform PA 

leadership around program improvements. 

Major Findings and Recommendations 

Partner Organizations 

Technical Capacity 

1. Overall, the evaluation found that Allies as a whole were most successful when their service fell

into either the Program Development, Delivery, and Evaluation or Community Engagement /

Interface category and was related to helping Partner Organizations expand their reach by
6

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

creating new programs, building upon existing programs or reaching out and engaging with the 

communities served by the PO’s.

2. Connected Allies were most successful affecting positive capacity increases in activity areas

within the Program Development, Delivery, and Evaluation category where their projects were

focused on developing new programming or expanding existing programming and related

activities such as developing new materials and outreach related to new/expanded

programming.

3. Disconnected Allies exhibited the most success increasing capacity in activity areas within the

Community Engagement/Interface category. Projects in this category were focused on

engaging the community to determine their needs and utilizing that information to improve PO

programming.

4. The majority of technical change scores for both connected Allies and disconnected Allies are

positive indicating increased capacity in nearly every activity category regardless of Ally type.

Adaptive Capacity 

1. Over two-thirds (66.7%) of PO supervisors working with connected Allies strongly agreed or

agreed that their notions of what constitutes a qualified employee had expanded. Among PO

supervisors working with disconnected Allies, 67.5% reported agreeing or strongly agreeing.

2. A comparison of responses of supervisors of connected and disconnected Allies and found no

statistically significant differences indicating supervisors responses did not differ based on the

type of Ally they supervised.

Current Ally Apprentices 

1. For all current Allies, the reported results indicate statistically significant growth for all shortterm 21st Century skill outcomes.

2. The results indicate that the impact of the Public Allies program on the 21st century skills

outcome indicators does not differ between connected and disconnected Allies.

3. Public Allies program participation increases interest in higher educstion for both connected

and disconnected Allies.

PA Alumni 

1. Public Allies has no differential impacts on Alumni that were degreed and non-economically

disadvantaged in comparison to those that were not degreed or considered economically

disadvantaged at the time of participation on the short-term and intermediate 21st century skill

outcomes.
7

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

2. When discussing the outcomes of the Alumni, the results indicate statistically significant

growth for Collaboration, Controversy with Civility, Resilience, Transformational Leadership,

Self- confidence, and Civic Engagement.

3. When examining Alumni career situations post participation, a larger proportion of connected

Allies experience an increase in wages at their current employment, while a larger proportion of

disconnected Allies are employed within non-profit organizations versus other public entities

Recommendations

1. Identify additional indicators of potential positive impacts Allies may have on PO’s and the

broader community.

2. Track Ally projects beyond the initial year.

3. Compare capacity changes between PO’s that host a single Ally against those that host multiple

Allies.

4. Streamline data collection requirements.

Evaluation limitations

1. Complete data for all Partner Organizations and supervisors was not available at the time of

analysis.

2. Sample sizes limited options for statistical testing in some areas including individual activity

areas of partner organization technical capacity and the Alumni survey.
8

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

II. Background

The following presents the results of an external impact evaluation for Public Allies (PA) 

AmeriCorps program, which is funded by the Corporation for National and Community Service. 

The over-arching goal of the evaluation is to provide empirical evidence to determine the shortterm impact of the Public Allies program on developing AmeriCorps Allies’ leadership abilities 

while helping nonprofit organizations build capacity to deliver services to the communities they 

serve. Intermediate level outcome indicators corresponding to program Alumni were also included 

in the analysis to examine whether anticipated changes associated with participation in the Public 

Allies program were sustained over time. 

The Center for Urban Initiatives and Research (CUIR), an evaluation and applied research center 

located at the University of Wisconsin – Milwaukee (UWM), conducted this assessment. The 

evaluation was intended to provide a clear description of the Public Allies program by reviewing 

data collected from a variety of sources, such as original surveys and semi-structured interviews 

developed by CUIR and existing data collected by the Public Allies program following indicators 

and objectives set forth in the external evaluation plan, detailing progress as well as challenges in 

its development1. 

Evaluation Team 

About the Center 

The Center for Urban Initiatives and Research (CUIR) is an applied research center at the 

University of Wisconsin-Milwaukee with a primary focus on evaluation research. For over 25 years, 

CUIR has supported community based nonprofit and public organizations and initiatives by 

providing a range of services from simple outcome monitoring to quasi-experimental evaluation to 

assistance with theory of change-driven logic modeling and setting up data collection systems. The 

center employs five evaluation practitioners, three of whom would support this project. Collectively 

CUIR researchers have worked on dozens of evaluation projects over the years, including those in a 

variety of public health areas such as youth physical activity and behavioral and emotional health, 

domestic violence, tobacco prevention, and healthy aging. Our partners have included the 

1 The following report was made using the data provided by current Public Allies apprentices, Public Allies 

Alumni apprentices, Public Ally Site Directors and Partner Organization (PO) Supervisors. It is based on data 

collected as of August 31, 2015. 
9

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Wisconsin Department of Health Services, Milwaukee Public Schools, Healthier Wisconsin, and the 

Violence Prevention Initiative. 

In addition to our research staff’s extensive evaluation experience and skills, the center has other 

resources to support a range of applied research projects. This includes a database programmer 

who is exceptionally skilled at merging, transforming and manipulating large databases. CUIR staff 

design and conduct mail, online and telephone surveys through our Survey Center, which houses a 

15-station computer-assisted telephone interview (CATI) system for conducting cell and hard line

telephone interviews. Recently, the Survey Center completed a police satisfaction survey of 

Milwaukee citizens that included a random sample large enough to stratify by seven police districts. 

The university’s secure web-based Qualtrics system is used for online surveying, and quantitative 

analysis is conducted by researchers at CUIR using the Statistical Package for the Social Sciences 

(IBM-SPSS). 

The following are the bios of the primary researchers who conducted this evaluation. 

Josh Lang, M.A., Associate Researcher 

Josh Lang received his master’s degree in Sociology from the University of Wisconsin-Milwaukee 

(UWM). Josh also holds a graduate certificate in applied gerontology and is currently a Doctoral 

candidate in the Urban Studies Program at UWM. A recent addition to CUIR, he comes in with 

extensive prior experience conducting and supporting multifaceted, applied research projects. As 

an Assistant Researcher in the Office of Applied Gerontology at UWM, Josh was a member of 

research teams on several large-scale research and evaluation projects. These include a study of the 

impact of an intervention for informal caregivers being served by the states of Georgia and 

Washington and several for-profit and non-profit caregiver support service providers across the 

country; a network analysis of the dementia support network in Wisconsin funded by the 

Alzheimer’s Association of Wisconsin; and a study of best practices for end-of-life care funded by 

the Milwaukee County Department of Family Care. He has been co-author on peer-reviewed 

articles published in The Gerontologist, Research on Aging, and Activities, Adaptation, & Aging. 

Josh possesses a wide array of research skills including survey instrument development, data 

collection and management, sampling, cross-cultural research techniques, technical writing, 

translational research, leading focus groups, conducting key informant interviews, and other 

quantitative and qualitative research methodologies. He is also adept with several research-based 

software packages including SPSS (statistical analysis), NVivo (qualitative analysis) and Ucinet 

(network analysis). 
10

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Atiera Coleman, M.A., 

Atiera holds a Bachelor’s degree in Sociology from Beloit College, and a Master’s degree in 

Sociology from the University of Wisconsin-Milwaukee (UWM). Atiera is currently a Doctoral 

candidate in the Sociology program at UWM, with concentrations in education and inequality. Her 

research focal points are aligned in residential segregation, school level inequality, and the racial 

achievement gap. In addition to research, Atiera is also a program assistant for the University of 

Wisconsin-Milwaukee Upward Bound program, and has numerous other roles in several student 

services programs at the University of Wisconsin- Milwaukee. 

What is Public Allies?

According to their website, Public Allies is a national movement grounded in the conviction that 

everyone has the ability to be a leader in their community, can make a difference, work to inspire 

more citizens to believe in themselves, step up, and act. The Public Allies model argues that 

throughout our nation’s history, lasting social change has always resulted from the courageous acts 

of many, not just the inspiration of the few. 

As an example, Public Allies points to The Civil Rights movement, which resulted from thousands 

of individuals throughout communities taking action as much because of the leadership of Martin 

Luther King and President Lyndon Baines Johnson as because of Rosa Parks. Parks was a 

seamstress and a volunteer with the NAACP who attended training on nonviolent civil 

disobedience at The Highlander School before the fateful day she chose to spark a movement. 

Today, more citizens need to be engaged, skilled, and prepared to lead in order to solve our most 

pressing problems. Public Allies believes the leadership that's needed for this is already resident in 

communities -- it’s only a matter of finding, cultivating, and connecting these leaders2

. 

Ten-Month Apprenticeship Program 

Public Allies’ mission is to create a just and equitable society and the diverse leadership to sustain 

it. The organization aims to change the face and the practice of leadership in 23 communities 

across the U.S. by cultivating talented, diverse, “home-grown” leaders – which they call “Allies” --

and preparing them for lives and careers devoted to social change. The vast majority of Allies are 

2 http://www.publicAllies.org/site/c.liKUL3PNLvF/b.2775807/k.C8B5/About_Us.htm
11

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

from the communities in which they serve and have a passion for, and connection to, the social 

issues that their placement organizations address -- qualities which make them outstanding and 

necessary change agents. 

The organization’s signature program is the 10-month apprenticeship, a 23-year partnership with 

AmeriCorps, in which diverse and uncommon leaders are given the opportunity to serve their 

communities to create more impactful and sustainable solutions. During the apprenticeship, Allies 

build their leadership through a model which combines classroom learning, application through 

service delivery, reflection and community building. For example, Allies participate in a 200-hour 

leadership development curriculum which encompasses how to build capacity in nonprofit 

organizations; job/career readiness; local government systems and structures; movement building; 

group dynamics, conflict resolution, and cultural competence; and analysis of, and reflection on, 

the dynamics of power, privilege, and oppression. Classroom learning is reinforced through regular 

coaching and mentoring opportunities and through the feedback of the cohort. In addition, smaller 

teams of Allies immerse themselves in a local neighborhood, map its assets, and develop and 

implement a 6-month project designed to have lasting community impact. 

Public Allies reports that since 1992, about 6,000 Allies have completed the program with more 

than.80% continuing careers in nonprofit and public service. The program has a track record 

of results and impacts have led to recognition and honors from the Pew Partnership for Civic 

Change, The Bridgespan Group, McKinsey & Co, Fast Company, and others. 

Program Reach 

Public Allies currently serves 23 communities across the country, each of which offers the flagship 

ten-month AmeriCorps program as a strategy toward sustainable community-level results. Core to 

the Public Allies model is the notion that creating opportunities for diverse and uncommon leaders 

to tap their potential is the key to helping communities create more impactful and sustainable 

solutions. In 2014-15, the Public Allies cohort started out with 674 Ally apprentices, of which 574 

completed the program. Out of those 574 Ally apprentices were approximately 75% people of color, 

66% women, and 12% identified as LGBTQ. Furthermore, 36% of the class grew up in homes that 

received government assistance, and 29% were the first in their family to attend college. 
12

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Areas served by Public Allies include: 

 Arizona

 Central Florida

 Chicago, IL

 Cincinnati, OH

 Connecticut

 Delaware

 Detroit (Metro), MI

 Eagle Rock, CO

 Iowa

 Indianapolis, IN

 Los Angeles, CA

 Maryland

 Miami, FL

 Milwaukee, WI

 New Mexico

 New York

 North Carolina

 Pittsburgh, PA

 San Antonio, TX

 Silicon Valley / San Francisco, CA

 Twin Cities, MN

 Washington, D.C.
13

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

III. Evaluation Description

Purpose of Evaluation

This evaluation is designed to provide quantitative evidence of the impact of the Public Allies 

program. The goal of the evaluation is to provide empirical evidence of impact at the partner 

organization and individual Ally apprentice level. The evaluation centers around two primary 

research hypotheses: 

H1: The experience of having a Public Ally will increase the capacity of Partner 

Organizations 

H2: The experience of serving at a community organization will have a positive impact on 

the Allies themselves. 

In addition, the evaluation, which includes both impact and process measures, serves as a vehicle 

to examine the Allies program more closely in order to inform PA leadership around program 

improvements. This includes suggestions on ways to improve current data collection tools and 

practices. 

Overall Evaluation Design 

This evaluation examines the four main components of the Public Allies program: 1) Partner 

Organizations, 2) Current Ally apprentices, 3) Alumni Ally apprentices, and 4) Public Allies Site 

Directors. A mixed-methods approach was utilized, employing both qualitative and quantitative 

data collection techniques such as surveying, secondary data analysis, and key informant 

interviews. IBM SPSS Statistics 20 was used to analyze quantitative data; including descriptive 

analyses such as variable frequencies, and statistical analyses with test of significance such as chisquare cross tabulations, paired t-tests, and independent samples t-tests. Qualitative data was 

collected through key informant interviews and analyzed utilizing a Grounded Theory approach to 

identify and categorize data into themes and concepts. 

The evaluation of Public Allies placed a special focus on between-group comparisons of four types 

of Allies: 1) non-degreed, economically disadvantaged Allies; 2) non-degreed, non-economically 

disadvantaged Allies; 3) degreed, economically disadvantaged Allies; and 4) degreed, noneconomically disadvantaged Allies. However, due to an insufficient number of cases in each group, 
14

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

categories were collapsed into dichotomous groups for comparison purposes. The three groups 

representing diverse and uncommon Allies: 1) non-degreed, economically disadvantaged Allies; 2) 

non-degreed, non-economically disadvantaged Allies; and 3) degreed, economically disadvantaged 

Allies; were combined to form a single group representing disconnected youth and adults which 

will furthermore be referred to as “disconnected Allies” in this report. The comparison group was 

comprised of degreed, non-economically disadvantaged youth and adults who are referred to as 

“connected Allies.” The purpose of including between-group comparison groups was to assess 

differences in outcome achievement of Allies across these groups, as well as perceptions of 

increased capacity of organizations based on the type of Ally placed within Partner Organizations. 

For the purposes of this evaluation, the term “degreed” refers to Allies that have received an 

associates, bachelors, or graduate degree. Furthermore, the term “economically disadvantaged” 

refers to the Corporation for National and Community Service (CNCS) definition, which applies if 

Allies are currently receiving or qualified for government assistance. Based on information 

collected on Allies, there are multiple ways in which Public Allies can define an Ally as economically 

disadvantaged that is beyond the CNCS definition. 

To ensure that there were no major outcome differences between the two definitions, an 

independent samples t-test analysis was conducted on the seven major program outcomes between 

Allies that are defined as economically disadvantaged using the CNCS definition, and Allies that are 

defined as economically disadvantaged based on if the Ally replied ‘yes’ to one or more of the 

economic questions. Results from these tests can be found in Appendix A. 

Evaluation Questions 

Several key research questions related to both impact and process measures were addressed in the 

evaluation. Research questions for each component of the evaluation are listed below along with an 

indication of whether the question examined impact or process measures. 

Partner Organizations 

1. What changes are evident in Partner Organizations’ technical capacity at the end of the

program and one year after participating in the program? (impact)

2. What are the types of projects chosen for organizations and their Allies that hold the most

promise for sustainability? (process)
15

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Current Ally Apprentices 

1. What impact does serving as a Public Ally have on current program participants' education

and career intentions and choices? (impact)

a. Do these differ by Ally demographics, including socioeconomic factors?

2. What impact does serving as a Public Ally have on current program participants’ selfconfidence and 21st Century skills? (impact)

a. Do these changes differ by Ally demographics, including socioeconomic factors?

b. Do these changes differ by site? Do they differ by support provided by sites? (process)

Alumni Apprentices 

1. Is there a change in Alumni’s self-confidence and 21st Century skills one year after the

program? (impact)

a. Do these differ by Alumni demographics, including socioeconomic factors?

2. What impact does serving as a Public Ally have on Alumni’s civic engagement, and

education and career intentions and choices one year after program completion? (impact)

a. Do these differ by Alumni demographics, including socioeconomic factors?

Public Allies Site Directors 

1. What are the perceptions of Public Allies Site Directors on quality of placements that affect

Allies’ outcome achievement? (process)

IV. Evaluation Components

Partner Organizations

The evaluation of Partner Organizations centers on the Partner Organization Theory of Change 

(Appendix B) which posits that Public Allies improves the ability of Partner Organizations (PO’s) to 

meet their missions by becoming more inclusive, collaborative and effective, resulting in resilient, 

healthy, safe, and engaged communities. This is reflected in the first evaluation hypothesis which 

broadly states: 

H1: The experience of having a Public Ally will increase the capacity of Partner 

Organizations. 
16

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Short-term outcomes focused on evaluating increased technical capacity of Partner Organizations 

are the primary focus of this evaluation. Outcomes associated with “adaptive” capacity, short-term 

changes regarding supervisor attitudes toward the value of the experiences provided by 

disconnected Ally apprentices, were also examined. Available data did not allow for the evaluation 

of intermediate outcomes, however suggestions regarding future data collection and analysis are 

provided to assist Public Allies in positioning the organization to be able to evaluate intermediate 

outcomes in the future. 

As previously noted, the following two research questions were posed to address Hypothesis 1: 

1. What changes are evident in Partner Organizations’ technical capacity at the end of the

program and one year after participating in the program? (impact)

2. What are the types of projects chosen for organizations and their Allies that hold the most

promise for sustainability? (process)

To answer the above research questions and evaluate changes in Partner Organizations technical 

and adaptive capacity, two outcome measures, which are explained in greater detail below, were 

analyzed. 

Partner Organization Outcome Measures 

Outcome Measure1: Improved technical capacity to deliver services: “technical capacity” is 

directly related to the Capacity Building projects implemented by Public Allies in collaboration with 

Partner Organizations. 

Improved Technical Capacity. Improved technical capacity was measured by comparing Partner 

Organizations self-reported capacity scores in each of 16 individual activity areas at the preassessment (prior to the Ally beginning their apprenticeship) to self-reported capacity scores at the 

post assessment (at the conclusion of the Ally’s apprenticeship). Each activity area is included in 

one of the following three activity area categories: Program Development, Delivery, and 

Evaluation; Community Engagement/Interface; and Organizational Infrastructure & Resources. 

Activity areas refer to the specific type of service Ally apprentices performed within the Partner 

Organization they were assigned. A complete description of each activity area can be found in the 
17

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Capacity Self-Assessment for Partner Organizations (version 26) located in Appendix C. 

Differences in technical capacity between pre- and post - assessments are examined using paired ttests. 

Capacity Change Scores. Capacity change scores were created for each individual activity area and 

for each of the three activity area categories to examine differences in Partner Organization 

capacity before and after the introduction of Ally apprentices. Change scores were computed by 

subtracting capacity scores at the post-assessment from capacity scores at the pre-assessment. 

Positive change scores indicate an increase in capacity while negative change scores indicate a 

decrease in capacity. No change in capacity would be indicated by a change score of zero. 

Independent samples t-tests are used to examine differences between the change scores of 

disconnected Allies and connected Allies. 

Outcome Measure 2: Increased “adaptive capacity,” particularly for organizations hosting 

diverse, uncommon leaders (including disconnected Allies) 

Adaptive capacity. Question 16 on the Supervisor End of Year Survey was used as a proxy variable 

to examine Partner Organizations “adaptive capacity.” The items states, “Public Allies has helped 

me/my organization expand our notion of how to identify qualified volunteers and employees 

from diverse backgrounds.” Respondents indicate the extent to which they agreed with the 

statement by choosing one of four response categories (Strongly Disagree, Disagree, Agree, 

Strongly Agree). Surveys were completed by Ally apprentice Supervisors at the conclusion of the 

Allies apprenticeship. Descriptive statistics were used to examine differences in adaptive capacity. 

Data Collection Instruments 

Public Allies existing data collection instruments were used to collect data to examine Partner 

Organizations short and intermediate outcomes. The pre- and post- Capacity Self-Assessment for 

Partner Organizations (version 26) and the post-only Supervisor End of Year Survey on increased 

organizational capacity were completed by Partner Organization supervisors and analyzed to 

determine changes in technical capacity in activity areas and adaptive capacity. Both the Capacity 

Self-Assessment for Partner Organizations and Supervisor End of Year Survey were utilized to 

gain deeper understanding of the types of projects that hold the most promise for future 

sustainability. 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Data Collection Procedures 

As a requirement of participation in the Public Allies program, Partner Organizations complete a 

pre- and post - Capacity Self-Assessment for Partner Organizations (version 26) online using the 

Public Allies Personal Impact System Documentation (PISD) system. The post-only Supervisor 

End of Year Survey on increased organizational capacity was completed using the online survey 

tool Survey Monkey. Data from both the Capacity Self-Assessment for Partner Organizations 

(version 26) and Supervisor End of Year Survey were provided to CUIR research staff in Microsoft 

Excel format. Data were then converted and saved into IBM SPSS Statistics 20 format for analysis. 

Data were then merged with Ally apprentice demographic data to allow for comparisons on 

outcomes between connected and disconnected Allies. 

Results 

Sample Size. As of the time of this report, complete technical capacity change data (demographic, 

pre-assessment capacity scores, and post-assessment capacity scores) were available for 395 Allies 

and included in the analysis. The Supervisor End of Year Survey was completed by 315 PO 

supervisors. Of these, 302 had complete data for Question 16 and were included in the adaptive 

capacity analysis. Description of each activity area are presented in Table 3 along with overall 

sample sizes for both the technical capacity and adaptive capacity evaluation. It should be noted 

that the number of cases varies for each activity area of technical capacity based on what activities 

Allies were engaged in. 

18
19

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 1. Sample size for Capacity Change Analyses 

Connected 

Allies 

Disconnected 

Allies Total 

Overall Technical Capacity 136 (34.4%) 259 (65.6%) 395 (100%) 

Activity Area Activity Area Description 

Program Development and Delivery A 

(expand/create programming) 

Is the organization able to improve, 

expand, or create new programming? 

84 (33.2%) 169 (66.8%) 253 (100%) 

Program Development and Delivery B 

(respond to constituent needs) 

Is the program delivery model 

effective in responding to constituent 

needs? 

82 (32.5%) 170 (67.5%) 252 (100%) 

Outreach Is the organization able to effectively 

reach the populations that can benefit 

from programming? 

31 (29.5%) 74 (70.5%) 105 (100%) 

Materials Development Does the program have the appropriate 

materials/tools to effectively deliver 

programming? 

12 (22.2%) 42 (77.8%) 54 (100%) 

Performance Measurement A 

(assess impact) 

Is the organization able to assess the 

impact of its programming on the 

constituency served? 

26 (48.1%) 28 (51.9%) 54 (100%) 

Performance Measurement B 

(use performance data) 

Does the organization use performance 

data to improve program offerings? 

14 (34.1%) 27 (65.9%) 41 (100%) 

Overall Program Development, 

Delivery, and Evaluation 

117 (34.4%) 223 (65.6%) 340 (100%) 

Community Assessment A 

(understand community needs/assets) 

Does the organization understand the 

community needs to be addressed, 

community context, and the assets 

available? 

2 (8.7%) 21 (91.3%) 23 (100%) 

Community Assessment B 

(community needs drives 

programming) 

Does an understanding of the 

community being served drive program 

development and delivery modeling? 

2 (9.5%) 19 (90.5%) 21 (100%) 

Community Awareness and 

Engagement 

Is community aware of organization’s 

services such that services are utilized 

by target audience? 

35 (35.4%) 64 (64.6%) 99 (100%) 

Partnerships and Collaborations Does the organization collaborate with 

others to help improve or expand 

programming and help reduce or 

improve fragmentation of services? 

40 (44.4%) 50 (55.6%) 90 (100%) 

Overall Community Engagement 

/ Interface 

59 (35.3%) 108 (64.7%) 167 (100%) 
20

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Technology Use A 

(website in place) 

Does the organization have a website in 

place that supports and promote its 

program(s)? 

5 (33.3%) 10 (66.6%) 15 (100%) 

Technology Use B 

(technology improves efficiency) 

Are there technological systems in place 

that help improve organizational 

efficiency and effectiveness? 

7 (35%) 13 (65%) 20 (100%) 

Volunteer Generation Does the organization have the 

adequate support of community 

volunteers to help assist in the delivery 

of service? 

31 (41.9%) 43 (58.1%) 74 (100%) 

Organizational Management and 

Operation A 

(internal management policies) 

Does the organization have the policies 

and systems for effective management, 

staff development, and internal 

communications? 

9 (26.5%) 25 (73.5%) 34 (100%) 

Organizational Management and 

Operation B 

(process of project coordination) 

Does the organization have a process for 

program and/or project coordination? 

9 (26.5%) 25 (73.5%) 34 (100%) 

Financial Resources Does the organization have a sufficient 

and diversified funding stream to 

support its efforts? 

8 (53.3%) 7 (46.7%) 15 (100%) 

Overall Organizational 

Infrastructure & Resources 

47 (39.5%) 72 (60.5%) 119 (100%) 

Adaptive Capacity NA NA 302 (100%) 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 2. Overall Mean Change between Pre- and post - Capacity Assessment by Ally Type 

Connected Allies Disconnected Allies All Allies 

Activity Area Pre Post t score Pre Post t score Pre Post t score 

Program Development and Delivery A 

(expand/create programming) 3.018 3.196 -2.279** 2.988 3.021 -.495 3.00 3.08 -1.585

Program Development and Delivery B 

(respond to constituent needs) 3.037 3.183 -1.744* 2.947 3.029 -1.237 2.98 3.08 -1.965*

Outreach 2.839 3.081 -1.767* 2.818 2.939 -1.175 2.82 2.98 -1.888*

Materials Development 2.750 3.125 -2.462* 2.667 3.012 -2.770** 2.69 3.04 -3.447***

Performance Measurement A (assess impact) 2.865 2.923 -0.345 2.589 2.696 -.732 2.72 2.81 -.760 

Performance Measurement B 

(use performance data) 2.571 2.929 -1.859* 2.370 2.796 -2.442** 2.44 3.02 -3.074**

Overall Program Development, Delivery, 

and Evaluation 2.871 3.091 3.643*** 2.880 2.984 -2.222** 2.88 3.02 -3.871***

Community Assessment A 

(understand community needs/assets) 2.750 3.000 -1.000 2.595 3.119 -2.796** 2.69 3.11 -2.904**

Community Assessment B 

(community needs drives programming) 2.750 3.000 -1.000 2.789 3.211 -1.804* 2.79 3.19 -1.913*

Community Awareness and Engagement 2.767 3.083 -2.129* 2.710 2.935 -2.656** 2.73 2.98 -3.418***

Partnerships and Collaborations 3.042 3.167 -0.988 3.054 3.033 .159 3.05 3.09 -.452 

Overall Community Engagement / 

Interface 3.000 3.130 -1.424 2.904 3.023 -1.553 2.94 3.06 -2.082*

Technology Use A (website in place) 3.000 2.800 0.492 3.000 3.100 -.281 3.00 3.00 0.000 

Technology Use B 

(technology improves efficiency) 3.071 3.000 0.162 2.731 3.000 -1.133 2.85 3.00 -.698 

Volunteer Generation 2.532 2.806 -1.675* 2.570 2.663 -.667 2.55 2.72 -1.594

Organizational Management and Operation A 

(internal management policies) 2.556 3.167 -1.609 2.600 2.900 -1.867* 2.59 2.97 -2.487**

Organizational Management and Operation B 

(process of project coordination) 2.556 2.944 -1.257 2.800 3.060 -1.762* 2.74 3.03 -2.200*

Financial Resources 2.188 2.125 0.424 2.717 2.500 .493 2.43 2.30 .636 

Overall Organizational Infrastructure and 

Resources 2.635 2.797 -1.179 2.722 2.812 -.901 2.69 2.81 -1.465

***p<.001, **p<.01, *p<.05 

21
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Improved Technical Capacity. Table 2 displays Partner Organizations average mean scores on preand post - capacity assessments as well as t-statistics for all 16 individual activity areas and three 

categories of activity areas. The t statistic is a statistical examination of two population means to 

determine whether changes in mean scores between pre- and post - assessments are statistically 

significant, or the product of random chance. Means and t-scores are provided for the complete 

population and both Ally types. Statistically significant differences are noted in bold and display an 

asterisk indicating the significance level. Figure 1 displays a Venn diagram of activity areas where 

statistically significant increases in capacity were reported for disconnected and connected Allies. 

All Allies 

Examining the mean technical capacity scores for all Allies reveals increases in technical capacity 

among every activity area, with the exception of Program Development and Delivery A 

(expand/create programming) where capacity remained the same, and Financial Resources where 

a slight decrease in capacity was reported. 

Statistically significant increases in capacity were found overall for the Program Development, 

Delivery, and Evaluation category. Within that category the following individual activity areas 

were also found to have statistically significant increases: Program Development and Delivery B 

(respond to constituent needs), Outreach, Materials Development, and Performance 

Measurement B (use performance data). Increases between pre- and post - assessments were also 

significant for the Community Engagement / Interface category. Within that category, Community 

Assessment A (understand community needs/assets), Community Assessment B (community 

needs drive programming), and Community Awareness and Engagement were also statistically 

significant. The Organizational Infrastructure & Resources category was not significant; however, 

the individual activity areas Organizational Management and Operation A (internal management 

policies) and Organizational Management and Operation B (process of project coordination) 

were found to be statistically significant. 

Overall, the evaluation found that Allies as a whole were most successful when their 

service fell into either the Program Development, Delivery, and Evaluation or 

Community Engagement / Interface category and related to helping Partner 

Organizations expand their reach by creating new programs, building upon existing 

programs or reaching out and engaging with the communities served by the PO’s.

22
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Figure 1. Activity Areas with Significant Increases in Capacity by Ally Type. 

Connected 

Allies 

Disconnected 

Allies 

Program Development 

and Delivery A 

(expand/create programming) 

Program Development 

and Delivery B 

(respond to constituent needs) 

Outreach 

Volunteer Generation 

Materials Development 

Performance Measurement B 

(use performance data) 

Community Awareness and 

Engagement 

Community Assessment A 

(understand community 

needs/assets) 

Community Assessment B 

(community needs drive 

programming) 

Organizational Management and 

Operation A 

(internal management policies) 

Organizational Management and 

Operation B 

(process of project coordination) 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

23
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Connected Allies 

Focus on projects completed by connected Allies indicates that capacity increases were found in 

every activity area with the exception of Technology Use A (website in place), Technology Use B 

(technology improves efficiency), and Financial Resources which all decreased (Table 4). Data on 

Partner Organizations indicated that the category Program Development, Delivery, and 

Evaluation experienced statistically significant capacity increases along with each associated 

activity area except Performance Measurement A (assess impact). Only two additional capacity 

increases were found to be significant: Community Awareness and Engagement and Volunteer 

Generation. Based on the data, connected Allies were most successful affecting positive capacity 

increases in activity areas within the Program Development, Delivery, and Evaluation category 

where their projects were focused on developing new programming or expanding existing 

programming and related activities, such as developing new materials and outreach related to 

new/expanded programming. 

Disconnected Allies 

Partner Organizations reported capacity increases in all activity areas served by disconnected Allies 

except Partnerships and Collaborations and Financial Resources (Table 2). Statistically significant 

increases were found in the Program Development, Delivery, and Evaluation category and 

associated activity areas Materials Development and Performance Measurement B (use 

performance data) as well as Community Assessment A (understand community needs/assets), 

Community Assessment B (community needs drive programming), and Community Awareness 

and Engagement. Overall, the disconnected Allies exhibited the most success increasing capacity in 

activity areas within the Community Engagement/Interface category. Projects in this category 

were focused on engaging the community to determine their needs and utilizing that information to 

improve PO programming. 

Figures 2 through 20 display technical capacity changes for the pre- and post - assessment for all 

three activity area categories and 16 individual activity areas. Overall mean scores are displayed as 

well as mean scores of connected Allies and disconnected Allies. Trends in the direction and 

magnitude of the change in mean capacity scores between pre- and post - assessment are also 

presented in each Figure. 

24
25

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

All Program Development, Delivery, and Evaluation 
26

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 
27

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Community Engagement / Interface 
28

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 
29

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Organizational Infrastructure & Resources 
Prepared by University of Wisconsin 

– Milwaukee Center for Urban Initiatives and Research 

30
31

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Capacity Change Scores. Differences in capacity scores between connected Allies and disconnected 

Allies are presented in Table 5. Mean change scores as well as t-statistics for all 16 individual 

activity areas and three categories of activity areas are reported. The independent samples t-test is 

used to determine the probability that two populations are the same with respect to the variable 

tested. In this evaluation, the change scores of connected Allies are being compared to those of 

disconnected Allies to determine if there is a statistically significant difference in the level of 

capacity change reported by Partner Organizations based on Ally type. In other words, is the 

magnitude of capacity change being reported larger or smaller between the two groups of Allies? Is 

one group of Allies impacting capacity to a greater extent than the other group? 

Data found in Table 3 shows that the majority of change scores for both connected Allies and 

disconnected Allies are positive indicating increased capacity in nearly every activity category. The 

exceptions among connected Allies where negative capacity change scores are reported were 

Technology Use A (website in place), Technology Use B (technology improves efficiency), and 

Financial Resources. Disconnected Allies displayed negative change scores for Partnerships and 

Collaborations and Financial Resources. However, it should be noted that independent samples ttests reveal no statistically significant differences in scores between connected Allies and 

disconnected Allies, indicating that the magnitude of capacity change in each activity area did not 

differ by Ally type. 
32

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

 Table 3. Capacity change scores by Ally Type 

Connected 

Allies 

Disconnected 

Allies t score 

Program Development and Delivery A 

(expand/create programming) 0.179 0.033 1.348 

Program Development and Delivery B 

(respond to constituent needs) 0.146 0.082 0.570 

Outreach 0.242 0.122 0.658 

Materials Development 0.375 0.315 0.120 

Performance Measurement A 

(assess impact) 0.058 0.107 -0.223

Performance Measurement B 

(use performance data) 0.357 0.426 -0.246

Program Development, Delivery, and 

 Evaluation 0.220 0.105 1.474 

Connected 

Allies 

Disconnected 

Allies t score 

Community Assessment A 

(understand community needs/assets) 0.250 0.524 -0.440

Community Assessment B 

(community needs drives programming) 0.250 0.421 -0.232

Community Awareness and Engagement 0.317 0.226 0.568

Partnerships and Collaborations 0.125 -0.022 0.770

Community Engagement / Interface 0.128 0.118 0.080 

Connected 

Allies 

Disconnected 

Allies t score 

Technology Use A 

(website in place) -0.200 0.100 -0.515

Technology Use B 

(technology improves efficiency) -0.071 0.269 -0.747

Volunteer Generation 0.274 0.093 0.842

Organizational Management and Operation A 

(internal management policies) 0.611 0.300 0.890 

Organizational Management and Operation B 

(process of project management) 0.389 0.260 0.420 

Financial Resources -0.063 -0.214 0.350 

Organizational Infrastructure & 

Resources 0.163 0.090 0.439 
33

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Adaptive Capacity. Partner Organization supervisors’ responses to the survey statement “Public 

Allies has helped me/my organization expand our notion of how to identify qualified volunteers 

and employees from diverse backgrounds” are presented in Figures 21 and 22 for connected and 

disconnected Allies, respectively. Analysis of the descriptive statistics showed that over two-thirds 

(66.7%) of PO supervisors working with connected Allies strongly agreed or agreed that their 

notions of what constitutes a qualified employee had expanded. Among PO supervisors working 

with disconnected Allies, 67.5% reported agreeing or strongly agreeing. These findings indicate that 

PO supervisors’ experiences as participants in the Public Allies program had broadened their 

perspective about the characteristics and skills that future volunteers, employees and community 

leaders must possess in order to positively influence their organizations and the community. 

Conversely, 21.7% of PO supervisors of connected Allies and 23.3% of PO supervisors of 

disconnected Allies strongly disagreed or disagreed with this statement indicating that hosting an 

Ally did not change their thoughts regarding the qualifications they feel are necessary to identify 

successful employees. An additional 11.6% of connected Allies’ PO supervisors and 9.3% of 

disconnected Allies’ supervisors stated they did not know if their notions had been altered. A Chisquare test of significance was utilized to compare responses of supervisors of connected and 

disconnected Allies and found no statistically significant differences (χ2(4,N=189 ) = .320, 

p=.988.), indicating supervisors responses did not differ based on the type of Ally they supervised 

It should be noted that supervisor responses covered all Allies they managed during the year. This 

means that regardless if a supervisor was responsible for one or multiple Allies, they still only 

completed one survey. For supervisors with more than one Ally, responses reflect an overall rating 

of their experiences with all of their Allies rather than any one individual. As a result, supervisors 

with more than one Ally were excluded from this analysis. Moving forward it is suggested that PO 

supervisors complete the End of Year Supervisor Survey for each individual Ally to allow analyses 

of whether expansion of PO supervisors’ notions of how to identify qualified employees differed by 

the type of Ally they hosted. 
34

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Figure 21. Connected Allies: “Public Allies has helped me/my organization expands our notion 

of how to identify qualified volunteers and employees from diverse backgrounds.”

Figure 22. Disconnected Allies: “Public Allies has helped me/my organization expands our 

notion of how to identify qualified volunteers and employees from diverse backgrounds.”
35

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Implications 

The Partner Organization component of the evaluation centered on the hypothesis that “the 

experience of having a Public Ally will increase the capacity of Partner Organizations.” Evaluation 

results provide evidence that support this hypothesis. Analysis of Partner Organizations technical 

and adaptive capacity produced three main findings: 1) Overall, the data suggests that Partner 

Organizations experienced technical capacity increases in nearly all activity areas in which Allies 

worked, regardless of whether they hosted a connected or disconnected Ally; 2) differences in the 

magnitude of capacity change between connected Allies and disconnected Allies were not 

statistically significant, suggesting that both groups of Allies were equally successful in helping PO’s 

build capacity in their respective activity areas and 3) the percentage of PO’s who reported that 

hosting an Ally had opened their mind to the value of the diverse skills and experiences each 

individual Ally brings to the table did not differ between those hosting connected versus 

disconnected Allies. 

As noted previously, comparison of pre- and post - capacity assessment data shows increases in 

technical capacity for nearly all activity areas. Overall, evidence suggests that Allies are most able to 

positively impact PO capacity in activity areas related to Program Development, Delivery and 

Evaluation and Community Engagement/Interface. While both groups of Allies have assisted 

organizations in increasing their capacity in a variety of activity areas, the data suggests that 

connected Allies are able to affect change in more individual activity areas for Program 

Development, Delivery and Evaluation related activities while disconnected Allies have had the 

most success increasing capacity for Community Engagement/Interface activity areas (Figure 1). 

Activities classified as Organizational Infrastructure and Resources experienced the least amount 

of capacity increase overall for both Ally types and it was the only category in which capacity 

decreases were reported for individual activity areas. 

Differences in capacity changes scores between connected Allies and disconnected Allies were not 

found in this evaluation. This suggests that the magnitude of capacity change does not differ by Ally 

type. Both types of Allies were equally effective in assisting to spur capacity increases within 

Partner Organizations. This finding is important to note and indicates that regardless of the skills 

and/or background, by the completion of the Public Allies program both Ally types had assisted in 

increasing capacity within their Partner Organizations by approximately the same level. 

While the concept of adaptive capacity was noted to be largely theoretical at the time of the 
36

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

evaluation plan, the data does provide clues as to how PO supervisors perceive changes in their 

organizations adaptive capacity as a result of hosting an Ally. With over two-thirds of PO 

supervisors indicating that working with their Ally/Allies has expanded their notions of what 

constitutes a qualified volunteer or employee, the implication is that involvement in the Public 

Allies program can help breakdown prejudices and biases about the skill sets and traits individuals 

need in order to be successful in the non-profit or governmental sectors. This supports one of the 

tenets of the Public Allies program that when given the opportunity, disconnected Allies/adults 

with diverse and/or disconnected backgrounds can positively contribute in meaningful ways to 

community-based organizations and the neighborhoods and communities they operate within. 

Recommendations 

Specific recommendations regarding future data collection procedures, data analysis, and outcome 

measurement for Partner Organizations are as follows: 

1. Suggestions for Future Research. This evaluation serves as a starting point for assessing

the impact Public Allies have on Partner Organizations and the individual Allies. However, the

limited scope does not allow for a broader evaluation of Public Allies impact on PO’s and the

communities they serve. It is recommended that Public Allies identify additional indicators of

potential positive impacts Allies may have on PO’s and the broader community, keeping in

mind that a wide range of factors outside of the program can impact community wide

outcomes. Examples of potential questions include:

a. In addition to building technical and adaptive capacity, what other potential

indicators would show a “successful” partnership with Partner Organizations that

lasts beyond the program year?

b. What insights could Partner Organizations provide regarding how the strengths of

disconnected Allies benefited their organization?

2. Track Ally projects beyond the initial year. Current data collection does not support a

long term analysis of the sustainability of individual projects beyond the year of an Ally’s

apprenticeship. To examine whether capacity changes attributed to Allies remained after the

Ally exited the program, tracking would need to be developed for individual projects so that

follow up could be conducted one year after the Ally completed their apprenticeship. Potential

follow up questions could include whether the specific projects, or results of those projects were

still in existence; if supervisors felt the capacity gains achieved through said project were
37

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

maintained one year later; and if capacity was not maintained, what the potential reasons may 

have been. 

3. Require all PO supervisors to complete the End of Year Survey for each individual

Ally. This data can then be used to further examine differences in supervisor perceptions based

on the type of Ally they host. Data from technical capacity assessments could then be matched

and merged with supervisor End of Year survey responses to create dynamic maps of the

intersection of technical capacity and adaptive capacity. Individual activity areas could then be

identified for their potential to produce greater increases in both technical and adaptive

capacity.

4. Compare capacity changes between PO’s that host a single Ally against those that

host multiple Allies. Public Allies could benefit from an analysis comparing PO technical

and adaptive capacity increases among organizations with more than one Ally to those who host

a single Ally.

5. Streamline data collection requirements. Eliminate unused data collection instruments

and focus on those data elements that are going to be regularly analyzed and used to help Public

Allies program staff make decisions about program directions and effectiveness of targeted

outcomes. Eliminating non-essential areas of data collection can improve the quality and

validity of the data that is collected by allowing supervisors more time to thoroughly answer the

questions posed to them rather than rapidly completing stacks of surveys and assessments in

an effort to get them all completed on time. Public Allies requests a significant amount of data

from Partner Organizations, much of which is never utilized in a meaningful way. Public Allies

should re-evaluate which data collection tools or items are essential to tracking and measuring

Ally and PO outcomes and phase out unneeded survey and assessment instruments.

Limitations 

As with any evaluation, there are some limitations that need to be noted regarding data collection, 

analysis and interpretation. The limitations impacting this evaluation are as follows: 

1. Incomplete data. Complete data for all Partner Organizations and supervisors was not

available at the time of analyses. Data completed by August 31, 2015 was included for analyses

in this evaluation. While it is unlikely, it is possible PO’s and supervisors that did not have
38

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

completed data could have significantly changed the results and interpretation of the analyses. 

2. Small sample size. Sample sizes among some individual activity areas were small. While

studies have shown that the t-test is robust enough to handle very small sample sizes3,

interpretation and attributing statistical power in those activity areas should be done with

caution.

Current Ally Apprentices

The evaluation of Current Allies and Public Allies Alumni centers on the Ally-focused Theory of 

Change (Appendix D), which posits that Public Allies provides uncommon and diverse individuals 

with guided opportunities to develop and apply leadership and 21st Century skills through a 10- 

month apprenticeship in a community-serving nonprofit, which increases a community’s number 

and quality of civic-minded citizens and leaders. Consequently, individuals participating in Public 

Allies improve the long-term conditions of their own lives and the communities around them. This 

is reflected in the second evaluation hypothesis which broadly states: 

H2: The experience of serving at a community based organization will have a positive impact 

on Allies. 

Short-term outcomes focused on evaluating the short-term and intermediate impacts that the 

Public Allies apprenticeship program had on current Allies and Alumni Allies. Between-group 

differences in impact between Allies and Alumni based on their levels of education and economic 

backgrounds (already connected Allies vs disconnected Allies), are also examined. “Level of 

education” is defined as whether Allies have an advanced degree, including bachelor, associate and 

technical degrees. Whether an Ally is “economically disadvantaged” as defined according to the 

guidelines set forth by CNCS. For a more detailed description of the method through which Allies 

were defined as “already connected Allies” or “disconnected Allies,” refer to the Overall Evaluation 

Design section of this report (p. 6). 

Specifically, the Ally apprentice component of the evaluation examined the differences in outcomes 

for current Allies in 21st Century skills (including leadership), self-confidence, civic knowledge, and 

3 de Winter, J. C. (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research & 

Evaluation, 18(10), 1-12.
39

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

education and career goals. Outcomes of Alumni Allies included 21stCentury skills (including 

leadership), self-confidence, civic engagement activities, and the success in the pursuit of education 

and career goals. Comparisons were also made on outcome measures between connected Allies and 

disconnected Allies. For both current and Alumni Allies, changes were measured at one time point 

using a post-retrospective questionnaire design. 

As previously noted, the following research questions were posed to address Hypothesis 2: 

1. What impact does serving as a Public Ally have on current program participants' education

and career intentions and choices? (impact)

a. Do these differ by Ally demographics, including socioeconomic factors?

2. What impact does serving as a Public Ally have on current program participants’ selfconfidence and 21st Century skills? (impact)

c. Do these changes differ by Ally demographics, including socioeconomic factors?

d. Do these changes differ by site? Do they differ by support provided by sites? (process)

3. Is there a change in Alumni’s self-confidence and 21st Century skills one year after the

program? (impact)

a. Do these differ by Alumni demographics, including socioeconomic factors?

4. What impact does serving as a Public Ally have on Alumni’s civic engagement, and

education and career intentions and choices one year after program completion? (impact)

a. Do these differ by Alumni demographics, including socioeconomic factors?

To answer the above research questions, changes in the following outcome indicators were 

examined: 21st Century skills (including leadership), self-confidence, civic knowledge, and 

education and career goals and Alumni Allies’ 21st Century skills (including leadership), selfconfidence, civic engagement activities, and the success in the pursuit of education and career goals 

short the following outcomes measures, which are explained in greater detail below, were analyzed. 

Current Ally Retrospective and Post Ally Survey Outcome Measures 

Outcome Measure 1: Increased 21st Century Skills, including leadership 

Socially Responsible Leadership Scale (SRLS-R2) . Several scales were used to capture Ally 

competence in 21st Century skills. The scales used to capture “collaboration” and “controversy with 

civility” were developed as part of the larger Socially Responsible Leadership Scale (SRLS-R2) 

which itself was based on the Social Change Model of Leadership (SCM). This SRLS-is composed of 
40

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

68 items, separated into eight statistically valid and reliable scales in 6-9 Likert format questions. 

Individual subscales are explained in greater detail below. 

Collaboration. For the outcome collaboration, participants were asked to self-report on 8 

items that measured their skills with developing human relationships to achieve common 

goals, share responsibilities, authority, and accountability4

. The answer options consisted of 

a five-point scale, where 1=Strongly Disagree, 2=Disagree, 3=Neither Agree nor Disagree, 

4=Agree, and 5=Strongly Agree. 

Controversy with Civility. To measure controversy and civility skills, participants were 

asked to self-report on 11 items that measured their ability to handle differences in 

viewpoints, and remain respectful and courteous in the workplace5

. The answer options 

were consistent with the previous five-point scale where 1=Strongly Disagree, 2=Disagree, 

3=Neither Agree nor Disagree, 4=Agree, and 5=Strongly Agree. 

Leadership. The Global Transformational Leadership scale developed by Carless, Wearing, 

and Mann6 measures an individual’s ability to be self-aware, act in accordance with 

personal values and beliefs, invest time and energy in activities the individual deems as 

important, serve in diverse settings to accomplish common goals, have a sense of civic and 

social responsibility, and desires to make the world a better place7. Response options were 

adapted for this evaluation in order to capture Allies’ self-perceptions on each of the seven 

items that measured their leadership for social change. The answer options consisted of a 

five-point scale, where 1=Strongly Disagree, 2=Disagree, 3=Neither Agree nor Disagree, 

4=Agree, and 5=Strongly Agree. 

Self Confidence. The Career Engagement Scale was used to measure Allies’ self-confidence. 

Career Engagement is defined as the degree to which somebody is proactively developing 

his or her career as expressed by diverse career behaviors. Career Engagement refers not to 

attitudes or desired aspirations, but instead to the specific career behaviors someone 

exhibits to enhance his or her career development. The Career Engagement Scale is a brief 

self-report that “directly measures the general degree of engagement in self-directed career 

4 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

5 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

6 Carless, S. A., Wearing, A. J., & Mann, L. (2000). A short measure of transformational leadership. Journal of Business and 

Psychology, 14(3), 389-405.

7 Tyree, T. M. (1998). Designing an instrument to measure socially responsible leadership using the social change model of leadership 

development. Dissertation Abstracts International, 59(06), 

1945. (UMI No. 9836493) & Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.
41

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

management behaviors8”(p. 576). Allies indicated their level of confidence with the each 

statement regarding various proactive career behaviors using a scale from 1 (no confidence 

at all) to 5 (complete confidence). Higher scores indicated higher levels of confidence. 

Resilience. The two item version of the Connor-Davidson Resilience Scale (CD-RISC) was used in 

the evaluation. This scale was measured by asking participants to self-report on a two item 

condensed version of the scale (CD-RISC2), and asses their ability to handle stress, cope in certain 

situations, and be resilient9. The answer options consisted of a five-point scale, where 1=Strongly 

Disagree, 2=Disagree, 3=Neither Agree nor Disagree, 4=Agree, and 5=Strongly Agree. 

Outcome Measure 2: Increased Civic Knowledge 

Civic Knowledge. A competence for civic action scale was used to explore Allies civic behaviors. The 

9-item measure contains several items drawn from the California Civic Index (Kahne, Middaugh,

and Schutjer-Mance, 2005) and one item adapted from the Civic Engagement Questionnaire 

(Keeter, Zukin, Andolina, and Jenkins, 2012). Participants were asked to self-report on their ability 

to engage in a range of civic-minded activities (various community and political activities). 

Participants answered each of the questions using a five-point scale, where 1=I Definitely Can’t, 

2=I Probably Can’t, 3=Maybe, 4= I Probably Can, and 5= I Definitely Can. 

Outcome Measure 3: Increased Interest in Higher Education. 

Interest in Higher Education. The CUIR research team worked with Public Allies staff to create a 

series of questions to examine Allies level of interest in pursuing higher education. Questions 

reflected whether Allies intended to pursue higher education options within the next six months to 

a year. If Allies indicated they would be pursuing higher education they were asked a follow up 

question regarding the type of degree they planned to pursue. Allies were also asked a series of five 

questions regarding their knowledge about various ways they could gather information about their 

options in pursuing higher education. Response categories on these questions ranged from 1 (no 

confidence at all) to 5 (complete confidence).

8 Hirschi, A., Freund, P. A., & Herrmann, A. (2014). The Career Engagement Scale Development and Validation of a Measure of 

Proactive Career Behaviors.Journal of Career Assessment, 22(4), 575-594.

9 Vaishnavi, S., Connor, K., & Davidson, J. R. (2007). An abbreviated version of the Connor-Davidson Resilience Scale (CD-RISC), the

CD-RISC2: Psychometric properties and applications in psychopharmacological trials. Psychiatry research, 152(2), 293-297.
42

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Data Collection Instruments 

A post-test and retrospective pre-test survey (Appendix E) was designed by the CUIR research 

team to collect data on changes in current Allies short-term and intermediate outcomes. An 

extensive literature review was conducted to identify valid and reliable scales to measure each 

outcome. Survey questions measured changes in current Allies’ 21st Century skills, leadership, civic 

knowledge, self-confidence and interest in higher education. 

Data Collection Procedures 

Surveys were designed using Qualtrics, a web-based survey platform, and were distributed to 

current Ally apprentices via email. Lists of eligible Allies were provided by the Public Allies national 

office. Data collection was completed in August 2015 for post-test/retrospective surveys. 

Results 

Current Allies Sample. Table 4 gives an overview of all of the current Allies’ demographics. 

Information was received for 574 Allies. The evaluation and analysis were conducted on the Allies 

that completed both the retrospective and post surveys. Statistical tests were conducted to ensure 

that there were not any statistically significant differences in the two sample groups. On 

demographic variables, this indicates that those who completed both the retrospective and post 

survey are representative of the full 574 sample. 

Of the Allies that completed both the retrospective and post survey, just over 70% were female, the 

three most prevalent races/ethnicities were Black or African American (34.2%), White or Caucasian 

(30.2%), and Hispanic or Latino (18.3%), and over half (58.9%) obtained a Bachelor’s degree.

When asked if the Ally identifies as LGBTQ, 45% stated that they do not. The vast majority of Allies 

do not identify as having a disability or as being a veteran. When asked if the Ally’s parents 

graduated from college, almost 50% stated that they had. This indicates that 41.6% of the Allies are 

defined as first-generation. Almost 60% of Allies qualify for or receive government assistance, 

while 32.2% had previously applied for it, and 36.6% come from families where their parent or 

guardian received government assistance. 
43

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 4. Current Ally Demographics 

Retrospective & Post 

Survey All Allies 

(N=321) (N=574) 

N (%) N (%) 

Gender 

Male 103 (32.1%) 185 (32.2%) 

Female 214 (66.7%) 381 (66.4%) 

Transgender 1 (0.3%) 3 (0.5%) 

Race 10

American Indian or Alaska 

Native 2 (0.6%) 3 (.5%) 

Asian 20 (6.2%) 29 (5.1%) 

Black or African American 116 (36.1%) 221 (38.5%) 

Hispanic or Latino 69 (21.5%) 108 (18.8%) 

Native Hawaiian/ Pacific 

Islander 0 (0%) 1 (.2%) 

White or Caucasian 89 (27.7%) 142 (24.7%) 

Two or more race/ethnicity 16 (4.9%) 35 (6.0%) 

Education 

GED 3 (0.9%) 5 (.9%) 

High school graduate 54 (16.9%) 110 (19.2%) 

Associates degree 15 (4.7%) 24 (4.2%) 

Technical 

school/apprenticeship' 3 (0.9%) 5 (.9%) 

Some college 64 (20.0%) 109 (19%) 

College graduate 164 (51.2%) 294 (51.3%) 

Some graduate school 6 (1.9%) 10 (1.7%) 

Graduate degree 11 (3.4%) 16 (2.8%) 

Ally self-identified as LGBTQ 

Yes 37 (11.5%) 66 (11.5%) 

No 141 (43.9%) 237 (41.3%) 

Aly disability status 

Disability status 9 (2.8%) 24 (4.2%) 

Non disability status 312 (97.2%) 550 (95.8%) 

10 Race categories were collapsed into ‘two or more race/ethnicity’ if the participant selected more than one 

race/ethnicity category. 
44

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Ally military Veteran status 

Military Veteran 7 (2.2%) 10 (1.7%) 

Nonmilitary Veteran 312 (97.2%) 559 (97.4%) 

Ally parents/guardians 

graduated from college 

Yes 130 (40.5%) 228 (39.7%) 

No 138 (43.0%) 229 (39.9%) 

Ally member currently 

qualifies/receives gov. 

assistance 

Yes 185 (57.6%) 323 (56.3%) 

No 46 (14.3%) 89 (15.5%) 

Ally member previously 

applied for gov. assistance 

Yes 105 (32.7%) 179 (31.2%) 

No 91 (28.3%) 174 (30.3%) 

Ally parents/guardians 

received gov. assistance 

Yes 114 (35.5%) 206 (35.9%) 

No 107 (33.3%) 185 (32.2%) 

Current Ally Outcomes. The retrospective survey consisted of several questions that asked the 

current Allies to choose a response that best fit them prior to joining the Public Allies program. 

This survey is representative of the 321 Allies that completed both the retrospective and post 

surveys. The survey was split into several sections that consisted of questions that fit into the 

following short-term outcomes categories: Collaboration, Controversy with Civility, Selfconfidence, Civic Knowledge, Resilience, Transformational Leadership, and Increased Interest in 

Higher Education. The individual questions for each section can be found in Appendix E. The 

responses from each of the questions were converted into scale scores in order to more completely 

capture each outcome. Indicator comparisons between disconnected Allies and connected Allies on 

all seven sections of the survey were conducted. 

The Posttest survey was administered toward the end of the Allies’ participation in the program, 

and reflects the Allies responses after participating in the Public Allies Program. Consistent with 

the retrospective survey, the test survey consisted of several scales: Collaboration, Controversy with 

Civility, Resilience, Transformational Leadership, Self- confidence, Civic Knowledge, and Increased 

Interest in Higher Education Scale. Responses were converted into scale scores in order to fully 

capture how each ally scored under each section. 
45

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 5 demonstrates the means for the retrospective and post surveys. It shows differences in 

outcome indicators from how Allies responded prior to participating in the program to after 

participation in Public Allies. All seven outcome indicators are statistically significant, indicating 

that current Allies report growth in all outcome indicators. 

Table 5. Current Ally Outcomes 

Outcomes Retrospective 

Survey 

Scale Score 

Mean 

(N=321) 

Post 

Survey 

Scale Score 

Mean 

(N=321) 

Range of 

Scale 

Collaboration 30.64** 34.92** 0-40

Controversy with Civility 37.03** 43.78** 0-55

Resilience 7.81** 8.61** 0-10

Transformational Leadership 25.97** 29.82** 0-35

Self-confidence 20.69** 27.77** 0-45

Civic Knowledge 21.40** 29.10** 0-45

Increased Interest in Higher Education 14.73** 17.71** 0-25

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Current Ally, Retrospective Survey Between-group Outcomes Comparison. As noted previously, the 

term “connected” refers to Allies that possess a degree and are not considered economically 

disadvantaged. “Disconnected” refers to Allies that do not have a degree or/and are economically 

disadvantaged. Table 6 demonstrates that the means for both disconnected Allies and connected 

Allies are very similar. To test significance a series of t-tests were conducted to determine whether 

there were any significant differences between the means for the disconnected Allies and the 

connected Allies. There is one statistically significant difference among the outcomes: the results 

indicate that disconnected Allies have a higher controversy and civility score than connected Allies 

prior to participating in the program. This indicates that disconnected Allies report having an 

increased ability to handle differences in viewpoints, and remain respectful and courteous in the 

workplace. No other significant differences between the characteristics of disconnected Allies and 

connected Allies were found on the retrospective survey. 
46

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 6. Retrospective Survey Outcomes Comparison 

Outcomes Disconnected 

Allies 

Scale Score 

Mean 

(N=199) 

Connected 

Allies 

Scale Score 

Mean 

(N=103) 

Range of 

Scale 

Collaboration 30.86 30.25 0-40

Controversy with Civility 37.26* 36.73* 0-55

Resilience 7.97 7.57 0-10

Transformational Leadership 26.35 25.39 0-35

Self-confidence 21.11 20.17 0-45

Civic Knowledge 21.62 21.31 0-45

Increased Interest in Higher Education 14.33 15.89 0-25

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Current Ally, Post Survey Between-group Outcomes Comparison. The test survey was administered 

toward the end of the Allies’ participation in the program, and reflects the Allies responses after 

participating in the Public Allies Program. Consistent with the retrospective survey, the test survey 

consisted of several scales: Collaboration, Controversy with Civility, Resilience, Transformational 

Leadership, Self- confidence, Civic Knowledge, and Increased Interest in Higher Education Scale. 

Responses were converted into scale scores in order to fully capture how each Ally scored under 

each section. Comparisons between disconnected Allies and connected Allies on all seven sections 

of the survey were conducted. 

Table 7 exhibits the means for each outcome measure for both the disconnected and connected 

Allies. Results indicate that the overall outcomes of participating in the Public Allies program do 

not differ between disconnected Allies and connected Allies on six of the seven outcome indicators. 

The results do indicate that connected Allies report having a greater increased interest in higher 

education compared to disconnected Allies. 
47

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 7. Post Survey Outcomes Comparison 

Disconnected 

Allies 

Scale Score 

Mean 

(N=199) 

Connected 

Allies 

Scale Score 

Mean 

(N=103) 

Range 

of 

Scale 

Collaboration 34.8 34.39 0-40

Controversy with Civility 43.6 43.37 0-55

Resilience 8.6 8.36 0-10

Transformational Leadership 29.8 29.34 0-35

Self-confidence 28.17 26.44 0-45

Civic Knowledge 29.04 27.62 0-45

Increased Interest in Higher 

Education 

17.8* 18.26* 0-25

(*) A statistically significant difference at p<.05, (**) A statistically significant difference at p<.001 

Current Allies: Disconnected Allies Outcome Comparison. To analyze growth in disconnected Allies 

during the duration of the Public Allies program, a series of paired t-tests were conducted to 

examine statistically significant changes in Ally responses between the retrospective survey and the 

post survey. Table 8 shows that for all of the sections there was an increase in means. All seven 

outcome indicators were statistically significant within these sections of the survey. The largest of 

the differences are with the Civic Knowledge outcome. The overall mean score in the retrospective 

survey was 21.62, yet the overall mean increased by 7.42 points in the post survey to an overall post 

mean of 29.04. These results indicate that the Public Allies program has a significant positive 

impact on all outcomes for disconnected Allies. 

Table 8. Disconnected Allies Outcomes 

Retrospective 

Disconnected 

Allies 

Scale Score 

Mean 

(N=199) 

Post 

Disconnected 

Allies 

Scale Score 

Mean 

(N=199) 

Gain 

Score 

Range 

of 

Scale 

Collaboration 30.86** 34.8** 4.27 0-40

Controversy with Civility 37.26** 43.6** 6.8 0-55

Resilience 7.97** 8.6** 0.7 0-10

Transformational Leadership 26.35** 29.8** 3.7 0-35

Self-confidence 21.11** 28.17** 7.2 0-45

Civic Knowledge 21.62** 29.04** 8.0 0-45

Increased Interest in Higher 

Education 

14.33** 17.8** 3.3 0-25

(*) A statistically significant difference at p<.05, (**) A statistically significant difference at p<.001 
48

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Current Ally, Connected Ally Outcome Comparison. Table 9 demonstrates that connected Allies 

also reported statistically significant increases in means for all outcome indicator scores. The 

greatest increase reported is on Civic Knowledge. In the retrospective survey the overall mean was 

20.5, but increased by 7 points in the post survey. This indicates that connected Allies also reported 

positive impacts during their participation in the Public Allies program. All Allies, regardless of 

education and economic status, reported growth in these key areas. 

Table 9. Connected Allies Outcomes 

Retrospective 

Connected 

Allies 

Scale Score 

Mean 

(N=103) 

Post 

Connected 

Allies 

Scale Score 

Mean 

(N=103) 

Gain 

Score 

Range 

of 

Scale 

Collaboration 30.25** 34.39** 4.1 0-40

Controversy with Civility 36.73** 43.37** 6.7 0-55

Resilience Scale 7.57** 8.36** 0.8 0-10

Transformational Leadership 25.39** 29.34** 3.9 0-35

Self-confidence 20.17** 26.44** 6.3 0-45

Civic Knowledge 21.31** 27.62** 6.5 0-45

Increased Interest in Higher 

Education 

15.89** 18.26** 2.3 0-25

(*) A statistically significant difference at p<.05, (**) A statistically significant difference at p<.001 

Current Ally Interest in Higher Education 

When examining current Allies interest in higher education (Table 10), the reported results 

indicate that connected and disconnected Allies have statistically significant differences. Tables 10 

and 11 demonstrate that prior to joining the program, over half of disconnected Allies planned on 

obtaining a degree, in comparison to only 23.3% of connected Allies. This result is expected 

considering that all connected Allies entered the program with a degree. This further explains why 

there are statistically significant differences in the types of degrees that Allies planned to obtain. 

The majority (75%) of connected Allies who planned on obtaining a degree were seeking a graduate 

degree (Table 11). 
49

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 10. Retrospective Survey: Prior to the program, were you planning on 

obtaining your degree or completing a college course contributing toward a college 

degree? 

Disconnected 

Allies 

(N=199) 

Connected 

Allies 

(N=103) 

All Allies 

N=262 

No, I didn’t plan on attending college 14 (7.0%)** 0 (0%)** 14 (4.4%) 

No, I already had a degree 58 (29.1%)** 79 (76.7%)** 143 (44.5%) 

Yes, I was planning on attending college 127 (63.8%)** 24 (23.3%)** 164 (51.1%) 

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Table 11. Retrospective Survey: What type of degree are you interested in obtaining? 

Disconnected 

Allies 

(N=199) 

Connected 

Allies 

(N=103) 

All Allies 

N=262 

Vocational/technical 3 (2.4%)** 0 (0%)** 3 (1.8%) 

2 year college 16 (12.6%)** 0 (0%)** 19 (11.6%) 

4 year college 67 (52.8%)** 5 (20.8%)** 81 (49.4%) 

Graduate school 34(26.8%)** 18 (75%)** 53 (32.3%) 

Unsure 7 (5.5%)** 1 (4.2%)** 8 (4.9%) 

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Tables 12 and 13 demonstrate that the post survey results follow the same trend as the retrospective 

survey results. All of the results are statistically significant as a result of the majority of connected 

Allies already possessing a degree prior to participating in the Public Allies program. In comparison 

to the retrospective survey, a higher percentage of disconnected Allies are interested in obtaining 

after participation in the program than they did prior to participation. Only 63.8% of disconnected 

Allies stated that they wanted to obtain a degree prior to joining Public Allies, however, after 

participation, 74.2% of disconnected Allies were interested in obtaining a degree. When examining 

connected Allies, prior to joining the program, 23.5% of Allies stated that they were interested in 

obtaining a degree, however, in the post survey 37.8% stated that they were interested in obtaining 

a degree. 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 12. Post Survey: Are you interested in obtaining a degree or completing a 

college course contributing to a degree? 

Disconnected 

Allies 

(N=199) 

Connected 

Allies 

(N=103) 

All Allies 

N=262 

No, I didn’t plan on attending college 5 (2.5%)** 0 (0%)** 6 (1.9%) 

Yes, within 6 months 49 (24.7%)** 9 (8.7%)** 65 (20.3%) 

Yes, within the next year 62 (31.3%)** 21 (20.4%)** 87 (27.2%) 

Yes, but not within the next year 36 (18.2%)** 9 (8.7%)** 49 (15.3%) 

No, I already had a degree 46 (23.2%)** 64 (62.1%)** 113 (35.3%) 

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Table 13. Post Survey: What type of degree are you interested in obtaining? 

Disconnected 

Allies 

(N=199) 

Connected 

Allies 

(N=103) 

All Allies 

N=262 

Vocational/technical 1 (.7%)** 0 (0%)** 1 (.5%) 

2 year college 12 (8.2%)** 0 (0%)** 14 (7%) 

4 year college 69 (46.9%)** 3 (7.7%)** 81 (40.3%) 

Graduate school 55 (37.4%)** 33 (84.6%)** 90 (44.8%) 

Unsure 10 (6.8%)** 3 (7.7%)** 15 (7.5%) 

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

50
51

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Visual Comparison of all Outcome for Disconnected and Connected Allies 
Prepared by University of Wisconsin 

– Milwaukee Center for Urban Initiatives and Research 

52
53

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Implications 

The results indicate that when examing the impact of the Public Allies program on the short-term 

21st Century Skills outcome indicators, Allies’ participation results in growth on all seven outcome 

indicators. The analysis also suggests growth does not differ by being degreed or economically 

disadvantaged. When comparing disconnected Allies to connected Allies, there are small 

differences in the overall means for the outcomes, yet, the statistical tests indicate there are only 

two statistically significant differences. The findings indicate that disconnected Allies have a higher 

Controversy with Civility score when compared to connected Allies prior to participating in the 

program. This indicates that disconnected Allies report coming into the program with an increased 

ability to handle differences in viewpoints, and remain respectful and courteous in the workplace. 

The second reported significant result suggests that connected Allies report having a greater 

increase in Interest in Higher Education compared to disconnected Allies after participating in the 

program. This indicates that Allies that are degreed and not economically disadvantaged at the 

time of participation have an increased interest in pursuing additional education after participating 

in Public Allies. No other significant differences between the characteristic of disconnected Allies 

and connected Allies were found. This suggests that Allies gain skills in all seven outcome areas 

regardless of being disconnected or connected. 

For both connected and disconnected Allies the reported results indicate statistically significant 

growth for Collaboration, Controversy with Civility, Resilience, Transformational Leadership, 

Self-confidence, Civic Knowledge, and Increased Interest in Higher Education. This indicates that 

current Allies increased thier ability in Collaboration throughout participation in the Public Allies 

program, meaning that their ability to develop human relationships to achieve common goals, 

share responsibilities, authority, and accountability increased11. Allies increased their Controversy 

with Civility skills, indicating an improvement in their ability to handle differences in viewpoints, 

and remain respectful and courteous in the workplace12. Allies experienced an increase in Selfconfidence, which is demonstrated through their ability to proactively develop his or her career as 

expressed by diverse career behaviors. Leadership was also increased, which means Allies 

increased their ability to be self-aware, act in accordance with personal values and beliefs, invest 

time and energy in activities the individual deems as important, work in diverse settings to 

accomplish common goals, have a sense of civic and social responsibility, and desires to make the 

world a better place13. Allies increased their Resiliency, and their ability to handle stress, and cope 

11 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

12 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

13 Tyree, T. M. (1998). Designing an instrument to measure socially responsible leadership using the social change model of leadership 

development. Dissertation Abstracts International, 59(06), 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

in certain situations14. And finally there was a statistically significant increase in Allies Interest in 

Higher Education, meaning that participation in the Public Allies program increased Allies interest 

in pursuing higher education upon exiting the program. 

When examining the current Allies interest in higher education, chi-square test results indicate that 

there are statistically significant differences between connected and disconnected Allies. The bulk 

of these differences are a result of all connected Allies possessing a degree prior to participation in 

Public Allies. As stated previously, 63.8% of disconnected Allies stated that they wanted to obtain a 

degree prior to joining Public Allies, yet after participation 74.2% of disconnected Allies were 

interested in obtaining a degree. When examining connected Allies, prior to joining the program, 

23.5% stated that they were interested in obtaining a degree, and in the post survey 37.8% stated 

that they were interested in obtaining a degree. This suggests that Public Allies increases the 

interest in obtaining a degree for both connected and disconnected Allies. While both connected 

and disconnected Allies’ interest in higher education increased, Table 7 demonstrates that 

connected Allies experience more interest over the course of participation in Public Allies. It must 

be noted that the majority of interest for connected Allies post participation is for a graduate 

degree, while the majority of disconnected Allies have an interest in pursuing a two or four year 

undergraduate degree. 

The reported results also indicate some room for improvement with some of the outcome 

indicators. For instance, on the Self-confidence scale, Civic Knowledge, and Controversy and 

Civility, while the Allies experienced significant growth, they are all over 10 points below the 

highest level of profiency. Other outside factors may be impacting these low scale scores, such as 

age and lack of skills prior to the program, but these results should be noted. Despite low score 

ranges for some of the outcomes, the results of these surveys conclude that participating in the 

Public Allies program positively impacts participants self-confidence and other short-term 21st 

century skills. 

Current Ally Outcomes by Site. Tables 14 and 15 provide the overall retrospective and post survey 

outcome means for all 23 national Public Allies sites. The small sample sizes for each individual site 

hinder the ability to perform a statistical test to identify any significant outliers, yet one can 

examine and compare mean scores to locate differences. 

1945. (UMI No. 9836493) & Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

14 Vaishnavi, S., Connor, K., & Davidson, J. R. (2007). An abbreviated version of the Connor-Davidson Resilience Scale (CD-RISC), the 

CD-RISC2: Psychometric properties and applications in psychopharmacological trials. Psychiatry research, 152(2), 293-297.

54
55

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 14. Retrospective Survey Outcome Measure Means by Site 

Site N Collaboration 

Controversy 

with Civility Resilience 

Transformational 

Leadership 

Selfconfidence Civic 

Knowledge 

Increased Interest 

in Higher 

Education 

Arizona 28 31.21 37.25 7.70 25.32 19.21 21.42 15.71 

Central 

Florida 

11 33.72 38.18 8.45 27.63 21.36 23.54 15.81 

Chicago 31 31.96 39.40 7.86 28.10 22.17 22.58 14.50 

Cincinnati 17 27.94 36.17 7.35 24.81 18.94 18.00 14.58 

Connecticut 15 33.66 37.53 7.90 26.26 19.13 18.66 13.13 

Delaware 16 27.68 34.93 7.62 24.75 18.93 20.81 14.87 

Detroit 14 32.07 40.21 8.71 26.92 24.71 23.76 15.92 

Eagle Rock 7 31.28 36.28 8.00 29.00 22.85 26.28 18.28 

Indianapolis 17 31.00 37.05 7.64 26.58 21.35 24.47 15.23 

Iowa 12 28.75 36.41 7.25 25.25 19.66 21.00 15.33 

Los Angeles 25 29.08 33.56 7.60 24.92 19.60 19.64 14.44 

Maryland 11 29.81 33.10 7.27 25.72 19.45 22.18 16.09 

Miami 7 34.14 37.50 8.28 28.00 22.14 26.14 16.57 

Milwaukee 25 29.24 35.24 7.60 24.04 20.04 19.40 13.16 

New Mexico 13 30.41 39.30 7.61 27.53 23.84 24.25 13.92 

New York 24 29.86 37.70 7.95 26.34 21.91 22.33 14.08 

North 

Carolina

3 37.66 42.66 7.66 28.00 15.66 20.00 13.66 

Pittsburgh 7 28.42 35.71 8.85 24.71 18.00 19.00 15.00 

San Antonio 7 27.85 32.85 8.14 23.85 20.28 17.71 11.42 

Silicon 

Valley 

11 31.72 36.81 7.45 24.81 17.81 17.00 13.90 

Twin Cities 10 31.00 39.10 7.60 24.90 20.30 19.60 13.55 

Washington 

D.C

10 33.50 41.50 8.60 26.90 26.00 26.20 17.20 
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 15. Post Survey Outcome Measure Means by Site 

Site N Collaboration 

Controversy 

with Civility Resilience 

Transformational 

Leadership 

Selfconfidence Civic 

Knowledge 

Increased Interest 

in Higher 

Education 

Arizona 28 34.67 42.92 8.96 29.35 27.67 27.57 18.14 

Central 

Florida 

11 36.72 44.63 8.63 30.36 28.81 29.18 18.70 

Chicago 31 34.74 42.72 8.06 29.33 26.53 28.43 16.40 

Cincinnati 17 33.58 44.58 8.29 29.29 26.88 26.87 17.94 

Connecticut 15 36.53 44.80 8.70 31.00 28.20 29.53 18.40 

Delaware 16 37.06 45.56 9.06 30.81 29.56 31.00 19.12 

Detroit 14 34.28 43.28 8.71 29.71 29.35 29.14 16.42 

Eagle Rock 7 32.00 42.00 8.14 27.71 23.83 27.66 17.14 

Indianapolis 17 34.82 44.58 8.94 30.70 29.82 29.23 17.58 

Iowa 12 34.58 44.00 8.33 29.66 28.50 31.08 17.58 

Los Angeles 25 34.72 42.32 8.28 28.60 25.96 28.80 17.96 

Maryland 11 35.18 40.00 8.54 29.63 26.90 28.81 17.90 

Miami 7 34.57 45.00 8.14 32.00 31.71 31.28 18.71 

Milwaukee 25 34.92 44.72 8.72 29.84 28.32 30.56 17.60 

New Mexico 13 33.76 45.30 9.00 30.69 27.46 31.53 16.76 

New York 24 34.60 43.17 8.41 30.04 28.25 30.25 17.37 

North 

Carolina 

3 37.33 45.33 9.00 31.33 28.00 22.66 19.33 

Pittsburgh 7 33.57 45.42 9.14 28.85 26.28 28.42 19.57 

San Antonio 7 34.57 44.00 9.00 29.42 28.00 28.57 17.00 

Silicon 

Valley 

11 35.72 43.00 8.36 29.54 25.81 25.09 17.50 

Twin Cities 10 34.50 43.10 8.60 28.40 24.90 26.90 16.60 

Washington 

D.C

10 36.70 46.70 9.50 32.30 30.90 32.70 19.00 

56
57

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Public Allies Alumni

Alumni Ally Outcome Measures 

Outcome measures for increased 21st Century skills and self-confidence were identical to the 

measure described in the current Allies section above. Those that are different are described below. 

Outcome Measure 1: Civic engagement 

Civic Engagement. The civic engagement scale was created from scales developed during a National 

Evaluation of Youth Corps. The creation of each construct identifies the participant’s strength, 

abilities, and connection to community15. Participants were asked to self-report on their 

community based activism after participating in the Public Allies Program. This scale consisted of 6 

items where the answer options consisted of a five-point scale, where 1=Never, 2=Rarely, 

3=Sometimes, 4=Often, and 5=Always. 

Outcome Measure 2: Increased Employment in Family Sustaining Careers 

Increased Employment. The CUIR research team worked with Public Allies staff to create a series 

of questions to examine Alumni Allies current employment situation. Alumni were asked to 

compare their current employment to their employment prior to participating in the Public Allies 

program. If the Alumni reported they were currently employed, they were also asked how 

characteristics of their current job (wages, health insurance, etc.) compared to their previous job, 

and whether they were currently employed by a non-profit or government entity. 

Outcome Measure 3: Increased Pursuit of Higher Education 

Pursuit of Higher Education. The CUIR research team worked with Public Allies staff to create a 

series of questions to examine whether Alumni Allies had pursued higher education after leaving 

the Public Allies program. Questions reflected whether Allies had obtained a degree or were 

currently working on completing a degree. If Allies indicated they had a obtained or were working 

to obtain a degree they were asked a follow up question regarding the type of degree they had 

received or planned to receive. 

15 Jastrzab, J. (1996). Evaluation of National and Community Service Programs. Impacts of Service: Final Report on the Evaluation of 

American Conservation and Youth Service Corps.
58

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Data Collection Instruments 

Following a literature review of established and valid measures that reflect the outcomes of 

interest, a post-test and retrospective pre-test survey (Appendix F) was designed by the CUIR 

research team to collect data on changes in short-term and intermediate outcomes and distributed 

via email to Alumni Allies from both the 2012-2013 and 2013-2014 cohorts. Data collection was 

completed in August 2015. Survey questions measured changes in each Ally apprentice’s current 

21st Century skills, leadership, civic engagement activities, self-confidence, pursuit of higher 

education, and employment in family-sustaining careers. 

Data Collection Procedures 

The Alumni survey was administered to Alumni participants of the Public Allies program who were 

active Allies from 2012-2014. Surveys were administered via Qualtrics, a web-based survey 

platform, to Alumni Ally apprentices via email. Data collection was completed in August 2015 for 

post-test/retrospective surveys. 

Alumni Allies Sample. Table 16 gives an overview of the Alumni demographics. Surveys were 

distributed to 977 Alumni, yet only 261 (27%) of Alumni Allies completed both the retrospective 

and post survey. Of the Allies that completed both surveys, the majority (69.7%) were female. The 

three most prevalent races/ethnicities are Black or African American (36%), White or Caucasian 

(29.5%), and Hispanic or Latino (14.6%). Over 65% of the Alumni were college graduates. 

Additionally, 9.2%) of Alumni did identify as LGBTQ, 3.8% had a disability, and .8% were a 

military veteran. Finally, among the Alumni that completed both surveys, 22.2% were first 

generation college attendants. 
59

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 16. Alumni Demographics 

Completed Retrospective & 

Post Survey All Public Ally Alumni 

(N=261) (N=983) 

Gender 

Male 76 (29.2%) 300 (30.5%) 

Female 182 (69.7%) 669 (68.1%) 

Prefer Not to Respond 3 (1.1%) 13 (1.3%) 

Race 

American Indian or Alaska Native 4 (1.5%) 16 (1.6%) 

Asian 18 (6.9%) 61 (6.2%) 

Black or African American 94 (36.0%) 384 (39.1%) 

Hispanic or Latino 38 (14.6%) 161 (16.4%) 

Native Hawaiian/ Pacific Islander 1 (0.4%) 2 (0.2%) 

White or Caucasian 77 (29.5%) 257 (26.1%) 

Other race/ethnicity 4 (1.5%) 20 (2.0%) 

Level of Education 

High school graduate 1 (6.1%) 88 (9.0%) 

Associates/Technical Degree 13 (4.9%) 55 (5.6%) 

Some college 42 (16.1%) 180 (18.3%) 

College graduate 163 (62.5%) 573 (58.3%) 

Some Graduate School 6 (2.3%) 17 (1.7) 

Graduate degree 21 (8.0%) 54 (5.5%) 

Ally self-identified as LGBTQ 

Yes 24 (9.2%) 107 (10.9%) 

Ally disability status 

Yes 10 (3.8%) 29 (3.0%) 

Ally military Veteran status 

Military Veteran 2 (.8%) 10 (1.0%) 

Ally first member of family to 

attend college 

Yes 58 (22.2%) 221 (22.5%) 
60

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Results 

Alumni Ally Outcome Comparison. Table 17 demonstrates that Alumni of the Public Allies program 

reported statistically significant growth in all six outcome categories. While all outcomes were 

statistically significant, the largest difference was seen in Self-confidence. Questions about selfconfidence measured a participant’s level of confidence with actions such as actively seeking to 

design their professional future, undertaking things to achieve their career goals, and assume 

duties or positions that will help them progress professionally. The overall mean for Alumni in the 

retrospective survey was 18.5, yet this increased by 8.5 points in the post survey. 

Table 17. Alumni Outcomes-All Allies 

Alumni Retrospective 

Survey 

Alumni Post 

Survey 

Range 

Collaboration 30.4** 34.8** 0-40

Controversy with Civility 37.8** 40.4** 0-55

Resilience 7.6** 8.7** 0-10

Transformational 

Leadership 

25.4** 29.4** 0-35

Self- confidence 18.5** 27** 0-45

Civic Engagement 8.4** 12** 0-30

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Definitions of how Allies qualify as economically disadvantaged have changed over time. To keep 

the analysis consistent with the available demographic information, Alumni who are not degreed, 

or are degreed but had directly received government assistance or had their family receive 

government assistance prior to participating in Public Allies, are defined as disconnected. Allies 

who were degreed and not economically disadvantaged when they participated in the program are 

defined as connected. Tables 18 and 19 demonstrate the Alumni outcome indicators as well as the 

comparison between disconnected Allies and connected Allies. There are no statistically significant 

differences between the outcome measure means when comparing disconnected Allies to 

connected Allies. 
61

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 18. Alumni Disconnected During the Program and Connected Allies 

Retrospective Survey Comparison 

Alumni Disconnected 

During Program 

Retrospective Survey 

Alumni Connected 

During Program 

Retrospective Survey 

Range 

Collaboration 30.4 30.3 0-40

Controversy with Civility 37.8 37.6 0-55

Resilience 7.5 7.5 0-10

Transformational 

Leadership 

25.3 25.7 0-35

Self-confidence 18.2 19.5 0-45

Civic Engagement 8.9 8.2 0-30

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Table 19. Alumni Disconnected During the Program and Connected Alumni 

Post- Survey Comparison 

Alumni Disconnected 

During Program 

Post Survey 

Alumni Connected 

During Program Post 

Survey 

Range 

Collaboration 34.9 34.5 0-40

Controversy with Civility 40.2 40.3 0-55

Resilience 8.7 8.4 0-10

Transformational 

Leadership 

29.6 28.8 0-35

Self- confidence 27.1 26.9 0-45

Civic Knowledge 12.8 11.2 0-30

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Alumni Allies, Disconnected During the Program Outcome Comparison. To analyze differences 

between the retrospective and post surveys, a series of t-tests were conducted to examine 

statistically significant changes in Ally responses. Table 20 shows that there was an increase in 

means on all of the outcome sections. All seven outcomes were statistically significant within these 

sections of the survey. 

The largest of the differences is with the outcome Self-confidence. The overall mean score in the 

retrospective survey was 18.2, yet the overall mean increased by 8.4 points in the post survey to an 

overall post mean of 27.1. These results indicate that the Public Allies program has a significant 

positive impact on all outcomes for disconnected Allies. 
62

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 20. Outcomes for Alumni who were Disconnected Allies 

Disconnected Allies 

Alumni 

Retrospective Survey 

Disconnected 

Allies Alumni 

Post Survey 

Gain 

Score 

Range 

Collaboration** 30.4 34.9 4.5 0-40

Controversy with 

Civility**

37.8 40.2 2.4 0-55

Resilience** 7.5 8.7 1.2 0-10

Transformational 

Leadership**

25.3 29.6 4.3 0-35

Self- confidence** 18.2 27.1 8.9 0-45

Civic Engagement** 8.9 12.8 3.9 0-30

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p <.001 

Alumni Allies, Connected Ally Outcome Comparison. Connected Allies also reported statistically 

significant increases in means for all outcomes (Table 21), with the greatest differences reported on 

Self-confidence. In the retrospective survey, the overall mean was 19.5 but increased to 26.9 in the 

post survey. 

This indicates that connected Allies also experienced positive impacts on these outcome indicators 

during their participation in the Public Allies program. All Alumni, regardless of education and 

economic disadvantage status reported experiencing growth in these key areas. 

Table 21. Alumni Connected Allies Outcomes 

Connected Alumni 

Retrospective Survey 

Connected 

Alumni Post 

Survey 

Gain 

Score 

Range 

Collaboration 30.3 34.5 4.2 0-40

Controversy with Civility 37.6 40.3 2.7 0-55

Resilience 7.5 8.4 .9 0-10

Transformational 

Leadership 

25.7 28.8 3.1 0-35

Self- confidence 19.5 26.9 7.4 0-45

Civic Engagement 8.2 11.2 3.0 0-30

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001 

Alumni Pursuit of Higher Education. In the retrospective survey, Alumni were asked questions 

regarding their educational plans prior to participating in the program – whether they were 

planning to obtain a degree or complete a college course to contribute towards a college degree. 

Table 22 demonstrates that the majority (69.5%) had no plans to obtain a degree because they 
63

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

already possessed one. Among Alumni with no degree, 24.3% were planning on attending college, 

while only 6.2% stated that they had no plans on attending college. 

When comparing the disconnected Allies Alumni to the disconnected Alumni, there are statistically 

significant differences in who obtained a degree prior participating in Public Allies. (This result is 

expected considering the definition of disconnect Allies includes not possessing a degree.) 

Table 22. Prior to the program, were you planning on obtaining your degree or 

completing a college course contributing toward a college degree? 

Disconnected 

Allies Alumni 

N=101 

Connected 

Alumni 

N=76 

All Alumni 

N=177 

No, I didn’t plan on attending college 10 (9.9%) 1 (1.3%) 11 (6.2%) 

No, I already had a degree 56 (55.4%) 67 (88.2%) 123 (69.5%) 

Yes, I was planning on attending college 35 (34.7%) 8 (10.5%) 43 (24.3%) 

Table 23 displays the type of degrees that the Alumni were trying to obtain. Over half of the Alumni 

were attempting to obtain a degree from a 4 year college, while an additional 25.6% were 

attempting to obtain an advance graduate degree. 

Table 23.What type of degree were you planning on obtaining? 

Disconnected 

Allies Alumni 

N=35 

Connected 

Alumni 

N=8 

All Alumni 

N=43 

Vocational/technical 2 (5.7%) 0 (0%) (4.7%) 

2 year college 7 (20.0%) 0 (0%) 7 (16.3%) 

4 year college 19 (54.3%) 3 (37.5%) 22 (51.2%) 

Graduate school 6 (17.1%) 5 (62.5%) 11 (25.6%) 

Unsure 1 (2.9%) 0 (0%) 1 (2.3%) 

Since leaving the Public Allies program, 11.1% of Alumni reported they have completed a degree, 

and an additional 16.1% are currently completing a degree. Of the degrees obtained or in the 

process of being obtained, 34.7% are from 4 year colleges, while 44.9% are advanced graduate 

degrees. 
64

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 24. Since leaving the program, have you obtained your degree or completed a 

college course contributing toward a degree? 

Disconnected 

Allies Alumni 

N=104 

Connected 

Alumni 

N=76 

All Alumni 

N=180 

No, I don’t plan on attending college 4 (3.8%) 0 (0%) 4 (2.2%) 

Yes, I have completed my degree 8 (7.7%) 12 (15.8%) 20 (11.1%) 

Yes, I am currently working on my degree 23 (22.1%) 6 (7.9%) 29 (16.1%) 

No, I had a degree prior to leaving Public Allies 52 (50.0%) 57 (75.0%) 109 (60.6%) 

No, but still I still plan on attending college 17 (16.3%) 1 (1.3%) 18 (10.0%) 

Table 25. What type of degree have you obtained or are you obtaining? 

Disconnected 

Allies Alumni 

N=31 

Connected 

Alumni 

N=18 

All Alumni 

N=49 

Vocational/technical 2 (6.5%) 0 (0%) 2 (4.1%) 

2 year college 8 (25.8%) 0 (0%) 8 (16.3%) 

4 year college 13 (41.9%) 4 (22.2%) 17 (34.7%) 

Graduate school 8 (25.8%) 18 (77.8%) 22 (44.9%) 

Alumni Increased Employment. Of the Alumni that filled out the survey, 84.2% of them were 

employed at the time the survey was administered. In comparison to previous jobs, the majority of 

Alumni’s current employment provided more responsibilities, higher wages, increased access to 

health insurance, and was a stepping stone that helped the Alumni move towards their desired 

career. 

When comparing disconnected Alumni and connected Alumni both groups are employed at similar 

rates; however, a statistically significant larger percentage of connected Alumni see an increase in 

wages in their current employment in comparison to the disconnected Alumni. Table 26 

demonstrates that 73.6% of disconnected Alumni saw a wage increase in their current position, yet 

over 85% of connected Allies saw an increase in wages. A similar trend is reported with connected 

Alumni reporting higher increases in access to health insurance, and a larger proportion reported 

being on a pathway towards their desired career. 

Table 26. Are you employed? 

Disconnected 

Alumni 

N=104 

Connected Alumni 

N=76 

All Alumni 

N=180 

Yes 87 (83.7%) 64 (84.2%) 151 (83.9%) 

No 17 (15.8%) 12 (15.8%) 29 (16.1%) 
65

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Table 27 Connected and Disconnected Alumni Current Employment Comparison 

Disconnected 

Alumni 

(N=71) 

Connected 

Alumni 

(N=55) 

All Alumni 

(N=122) 

Job responsibilities increased 71 (81.6%) 51 (79.7%) 122 (80.8%) 

Wages increased 64 (73.6%) 55 (85.9%) 119 (78.8%) 

Access to health insurance 

increased 

54 (62.1%) 44 (68.8%) 98 (64.9%) 

On a pathway towards desired 

career 

65 (74.7%) 52 (81.2%) 117 (77.5%) 

Of the Alumni that were employed at the time this survey was administered, 72.8% worked for a 

non-profit organization, while 19.9% were employed by a public entity. When comparing the 

disconnected Alumni to the connected Alumni, there is a statistically significant difference in 

Alumni who are employed by non-profit organizations. Table 28 demonstrates that 77% of 

disconnected Alumni are employed by a non-profit organization, while 67.2% of connected Alumni 

are employed by a non-profit organization. 

Table 28. Connected and Disconnected organization comparison 

Disconnected 

Alumni 

(N=111) 

Connected 

Alumni 

(N=115) 

All Alumni 

(N=262) 

Employed by a non-profit organization 67 (77%) 43 (67.2%) 110 (72.8%) 

Employed by a public entity 18 (20.71%) 12 (18.8%) 30 (19.9%) 

Columns may not add up to 100% due to missing categories not displayed 

(*) A statistically significant difference at p<.05 

(**) A statistically significant difference at p<.001

Implications 

The t-tests conducted for this analysis indicate that there are no statistically significant differences 

in outcome indicator scores when comparing Alumni that were disconnected at the time of 

participation to connected Alumni. While means may vary by each group, these small differences 

are not statistically significant. This indicates that the outcomes of the program did not differ 

depending on whether the Alumni was degreed and economically disadvantaged. 

When discussing the outcomes of the Alumni, the results indicate that the Public Allies program 

had a positive impact on all outcome indicators. This indicates statistically significant growth for 

Collaboration, Controversy with Civility, Resilience, Transformational Leadership, Selfconfidence, and Civic Engagement. Alumni increased their ability in collaboration throughout 

participation in the Public Allies program, meaning that their ability to develop human 

relationships to achieve common goals, share responsibilities, authority, and accountability 
66

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

increased16. Alumni increased their Controversy with Civility skills, indicating an improvement in 

their ability to handle differences in viewpoints, and remain respectful and courteous in the 

workplace17. Alumni experienced an increase in Self-confidence, which is demonstrated through 

their ability to proactively develop his or her career as expressed by diverse career behaviors. 

Leadership was increased, which means Alumni increased their ability to be self-aware, act in 

accordance with personal values and beliefs, invest time and energy in activities the individual 

deems as important, work in diverse settings to accomplish common goals, have a sense of civic 

and social responsibility, and desires to make the world a better place18. Alumni increased their 

Resiliency, and their ability to handle stress, cope is certain situations, and be resilient19. And 

finally, Alumni reported growth in Civic Engagement, indicating that they strengthened their 

abilities to create a connection to communities20. 

Within these findings the means for disconnected and connected Alumni vary for the retrospective 

and post survey, making it appear as though connected Alumni experienced greater gains. 

However, when comparing the gain scores found in Tables 18 and 19, the t-tests indicate that there 

are no statistically significant differences. This suggests that at the time of participation, 

disconnected and connected Allies experienced similar levels of growth. This all leads to the 

implication that Public Allies has no differential impacts on Alumni that were degreed and noneconomically disadvantaged and those that were not degreed or considered economically 

disadvantaged on the short-term and intermediate 21st Century skill outcomes. 

When analyzing their current education and career situations, several chi-square tests were 

conducted, the results indicated that there were no statistically significant differences in Alumni 

who were connected in comparison to Alumni who were disconnected at the time of participation. 

However, there were descriptive differences in the results that should be noted in relation to a 

larger proportion of connected Alumni possessing a degree, a larger proportion of connected 

Alumni having higher wages at their current employment, and a larger proportion of disconnected 

Alumni being employed within non-profit organizations. 

16 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

17 Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

18 Tyree, T. M. (1998). Designing an instrument to measure socially responsible leadership using the social change model of leadership 

development. Dissertation Abstracts International, 59(06), 

1945. (UMI No. 9836493) & Astin, H. S. (1996). Leadership for social change. About Campus, 1(3), 4-10.

19 Vaishnavi, S., Connor, K., & Davidson, J. R. (2007). An abbreviated version of the Connor-Davidson Resilience Scale (CD-RISC), the 

CD-RISC2: Psychometric properties and applications in psychopharmacological trials. Psychiatry research, 152(2), 293-297.

20 Jastrzab, J. (1996). Evaluation of National and Community Service Programs. Impacts of Service: Final Report on the Evaluation of

American Conservation and Youth Service Corps.
67

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

The majority of connected Alumni possessing a degree is indicative of the fact that a degree is one 

of the main factors in identifying an Alumni as ‘connected’. One would expect this reported result 

in the findings. The results also indicated that both connected Alumni and Alumni that were 

disconnected at the time of participation experienced higher responsibility in their employment 

after Public Allies participation. However, while 85.9% of connected Alumni saw an increase in 

wages, only 73.6% of disconnected Allies saw an increased in wages. This would suggest that while 

participating in Public Allies does impact an Alumni’s job responsibilities, a higher proportion of 

connected Alumni are benefiting financially. This may speak to the impact that having a degree has 

for connected Alumni. Finally, the results show that while 67.2% of connected Alumni go on to 

work as a non-profit organization, over 77% of disconnected Alumni are employed by a non-profit 

organization. This suggests that Alumni that are disconnected at the time of participation in Public 

Allies are more prone to stay employed in the social sector. 

Limitations 

1. One limitation that should be noted is the use of a retrospective survey in the place of a

connected pre-test survey. The retrospective survey was administered with the post survey, and 

Alumni were asked to choose a response that best fit them prior to joining the Public Allies 

program. Literature suggests that retrospective pre-test administration yields comparable results to 

connectedly administered pre-tests.21

2. An additional limitation is the collapsing of the three types of disconnected Allies. For the

purposes of analysis during this evaluation, the three types of disconnected Allies, which consists 

of, non-degreed, economically disadvantaged Allies, non-degreed, non-economically disadvantaged 

Allies, and degreed, economically disadvantaged Allies were all collapsed into one category labeled 

disconnected Allies. There may be statistical differences among the three types of Allies, but that 

examination was beyond the scope of this evaluation. 

21 Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. K. (1979). Internal invalidity in pretestposttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1-23.
68

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Public Allies Site Directors

While Partner Organizations and Ally apprentices were the primary focus of this evaluation, an 

analysis of Public Allies Site Directors perceptions of the factors that contribute to successful 

placements was also conducted. Each of the 23 Public Allies locations is headed by a Site Director 

who is responsible for overseeing each sites partnerships, Ally recruitment, and Ally training 

activities. While there was no overarching hypothesis driving the evaluation of Public Allies Site 

Directors or specific outcomes to evaluate, the evaluation did seek to address the following research 

question: 

1. What are the perceptions of Public Allies Site Directors on quality of placements that affect

Allies’ outcome achievement?

The steps undertaken to evaluate this research question as well as the results are presented below. 

Data Collection Instruments and Procedures 

Unlike Partner Organizations and Ally apprentices, the Public Allies program does not regularly 

collect formal data from Site Directors. For the purposes or this analysis, the CUIR research team 

designed two data collection instruments: 1) key informant semi-structured interviews, and 2.) a 

brief online survey. 

Semi-structured interviews. Semi-structured interview questions were designed in conjunction 

with Public Allies staff and administered to four Public Allies Site Directors, chosen by Public Allies 

staff due to their familiarity with the program, and the varied experiences each of their sites 

represented. The goal of the interviews was twofold: first, to gain contextual information used to 

inform the construction of a brief online survey and second; to collect more nuanced details 

regarding Site Directors perceptions of quality Ally placements that cannot be gathered from a brief 

survey. Interview responses were entered into a Microsoft Excel spreadsheet, examined for 

patterns and themes, and coded separately by two members of the CUIR research team. Coding 

differences were resolved through discussion until a single code could be agreed upon. Coded 

themes were then quantified and analyzed, and used to create categories of Partner Organization 

characteristics that are key to successful Ally placements for use in the online survey. 
69

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Online Survey. A brief online survey was created using information gathered during the semistructured interviews and in consultation with Public Allies staff. Surveys were designed in and 

administered via Qualtrics, a web-based survey platform, and were distributed to all 23 Site 

Directors via email. Three follow up emails were sent to Site Directors encouraging their 

participation. Data collection was closed mid-September 2015. 

Results 

Sample Size. Online surveys were completed by 11 out of 23 Site Directors (48%). Of these eleven, 

10 completed the ranking of support services question. Demographic information about Site 

Directors was not available. 

Ranking of Top Three Support Services for Ally Success. Site Directors were asked to rank nine 

categories of site specific supports provided to Allies in order of their importance in supporting Ally 

development with one (1) being the most important, and nine (9) being the least important. 

Respondents also had the option to write in up to three more of their own categories and rank them 

along with the other nine. While several Site Directors did add additional categories, none was 

ranked in the top five in terms of importance. Figure 31 displays the percentage of Site Directors 

who ranked each category in their top three in terms of importance. 

The categories chosen most often as the most important Ally support were Training and Learning 

(40%), Mentoring(20%), Partnership Development (20%), Marketing PA Mission (10%) and 

Working with PO Supervisors. When combining the percentage of categories that were ranked 

first, second or third, Partnership Development was ranked the most important overall (60%). 

Alumni Engagement was not ranked in the top three by any Site Directors and only two ranked 

Coaching third. 
70

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Importance of PO Characteristics for Ally Success. Site Directors were asked to indicate the level of 

importance of 15 potential Partner Organization characteristics in terms of how they impact the 

success of Ally placements. They were encouraged to respond to the statement keeping in mind that 

the focus was not necessarily on the overall success of the organization itself, but rather on the 

organization characteristics that help to foster and grow a successful partnership with the Public 

Allies program. Response categories ranged from “Not at all Important” (1) to “Extremely 

Important” (7) and included a “Neither Important nor Unimportant” (4) as a neutral midpoint. For 

the purposes of this evaluation, the two response categories on each side of the spectrum were 

combined (Not at all or Not Very Important) and (Very or Extremely Important) to identify the 

characteristics that were deemed by Site Directors to be the most or least important for Ally 

success. Results are presented in Figure 36. 

Examining Figure 32 illustrates that 100% of Site Directors responded that they felt PO 

Accountability and whether PO’s Value being a mentor were “Very or Extremely Important” PO 

characteristics that can impact Allies’ success. Another 80% or more reported that Clear 

Expectations (of Ally skill level and responsibilities), PO Stability, and Supervisor Accessibility 

were Very or Extremely Important PO characteristics. Conversely, looking at Focus of PO and
71

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

whether the PO had Previous PA Experience there were an equal amount of Site Directors who 

reported that those characteristics were “Not at all or Not Very Important” as reported that they 

were “Very or Extremely Important.” Other categories, such as Risk Acceptance, PO Adding New 

Capacity, and Size of PO also were reported to be “Not at all or Not Very Important,” however 

more Site Directors felt they were in fact “Very or Extremely Important.”

Figure 32. Importance of PO Characteristics to Ally Success 
72

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Open Ended Survey Responses. In addition to the quantitative survey responses, Site 

Directors were also given the option to answer two open ended questions focused on current 

challenges in creating new/maintaining existing partnerships and ideas on changing current 

practices to address these challenges. Responses to each question are provided in their entirety 

below. 

Q6. Are there any other insights you would like to add regarding current challenges 

in forming successful partnerships with Partner Organizations? 

1. Melding AmeriCorps needs with Community organization needs. There is often a tension,

especially as performance measures shift frequently

2. PA is competing with other free or lower cost capacity building human capital solutions.

3. Prohibited activities are becoming more of a barrier especially the limitations in marketing

and advocacy

4. The only challenge I face is lack of staff capacity locally. With only 2 individuals, one

including myself running the program, it is sometimes difficult to manage every single task

in an effective meaningful way.

5. They have to be on board with the development part of things, if they are looking for a

cheap worker... forget about it.

6. We currently don't offer them a lot of programming, resources or networking. There's lots

of unlocked potential in these relationships

7. We talk about creating long term partnerships, but when we keep changing the dynamics of

partnership (enrollment demographics, focus, etc.), it makes it difficult for partners to

continuously rely on what it means to partner with Public Allies.
73

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Q7. Do you have any suggestions about how current partnership development 

practices could be altered to assist Sites with building/strengthening partnerships 

with new and existing Partner Organizations? 

1. I am eager to hear suggestions from other sites. It has been really helpful this year to separate

PO recruitment from Ally recruitment. We were able to engage more partners and spend the

necessary time building quality placements.

2. I think online seminars with PO sites would assist local sites. This also would strengthen

relationships locally if National Office was more involved.

3. Include partners in larger picture vision. They must get the long-range goal and be willing to

coach and train. They must believe in our values. They must be advocates for our mission.

4. Much of the challenges arise from the Allies' personal life challenges, not necessarily the

partnership development practices. Ally loss has almost always been because of insecure

housing, challenges with transportation leading to chronic lateness, behavioral/mental health

challenges, and family financial challenges. POs and Program staff can do all they want to have

a good, clear relationship, but I've never lost an Ally because of the PO/site relationship.

Additionally, it would be more helpful for PAN to help us think through restructuring the

program, etc. Connectedly, when site staff/Directors express challenges, PANO staff turns the

question around and ask where we can find the opportunity in that particular challenge. There

needs to be more shared responsibility instead of just assuming site staff will shoulder the

burden and figure it out.

5. Take some of responsibilities off Program Managers so that a few can focus on building

relationships.

6. There currently is little to no training offered for local site staff to help improve their ability to

do partner outreach and relationship building. There is little to no marketing material that is

updated or distributed to staff. Our materials often too rough around the edges. I have not seen

an updated recruitment brochure in at least 6 years, nor an annual report. It may be assumed

that sites all talk about PA and our mission and values in consistent ways, but that may need

some exploring. Interpretation may vary from site to site, as such partnerships and partnership
74

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

quality and Ally experience will vary. Our website is outdated. Other AmeriCorps programs are 

passing us by in terms of marketing/outreach/staffing. 

7. We need some good job descriptions that we can shop around to partners i.e. We can find you

an Ally who will do ABC and D for you. Are you interested? It'd be great to get a cohort of

Allies all with similar job descriptions in different orgs.

Implications 

The evaluation of the online survey data and open ended responses given by Site Directors 

regarding their perceptions of the factors that contribute to Ally success reveals some interesting 

patterns. First, and not surprisingly, both the ranking of support services, and open ended 

responses reflect Site Directors perceptions that the proper training of Allies is critical to their 

success. What is equally interesting is that Site Directors also report a need for additional training 

for Site staff, specifically around Partnership Development (Q7, 6). 

This first theme ties directly into the second theme that emerged from the evaluation; the need to 

invest additional time on Partner Development and Networking. Not only was Partner 

Development/Network ranked in the top three in importance by (60%) of site directors, but the 

topic was also mentioned several times in open ended responses both in terms of making sure the 

program is meeting PO needs, and in making sure PO’s are aligned with the broader Public Allies

mission to create leaders. As one Site Director stated, “They (PO’s) have to be on board with the 

development part of things, if they are looking for a cheap worker...forget about it” (Q6, 5). Another 

noted that the Public Allies program should, “Include partners in the larger picture vision. They 

must get the long-range goal and be willing to coach and train. They must believe in our values. 

They must be advocates for our mission” (Q7, 3). This is also reflected in the high importance Site 

Directors reported for PO’s to Value being a mentor. 

Finally, it is worth noting that Site Directors also reported a desire to work more closely with the 

Public Allies National Office to develop new techniques, resources and capabilities as well as a 

consistent overall message to engage Partner Organizations. While this question was not included 

in the quantitative part of the survey, it was expressed repeatedly in some fashion in the open 

ended responses (Q6, 1; Q6, 6; Q6, 7; Q7,2; Q7, 4; Q7,5; Q7, 6; Q7,7). 
75

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Recommendations 

1. Have all Site Directors complete the online survey. In ordered to gain a complete picture

of Site Directors perceptions and the resources being utilized to support Ally success, the remaining 

12 Site Directors should be asked to complete the survey. 

2. Tap the knowledge of local sites annually. Each Public Allies Site has valuable knowledge

about their local communities’ strengths and weaknesses. This includes insight into the unique 

challenges they face working with their local Partner Organizations and Ally apprentices. , as well 

as ideas about how to utilize local resources and potential to carry out the PA mission of creating 

community leaders, and building the capacity of local community organizations 

3. Gather information on Public Allies/Partner Organization partnerships. Partner

Organizations are asked to complete a number of surveys and assessments regarding the 

experiences they have working with Ally apprentices and capacity outcomes, however they do not 

have to provide feedback regarding their partnership with the Public Allies organization. Likewise, 

Site Directors do not have a means to evaluate the PO’s role in the partnership. One such tool that 

could be refined and used is the Scale for High Performance Partnerships developed by de Wall22

which requires both sides of the partnership to rate one another on factors such as Openness in the 

Partnership, Equality of the Partnership, and Conflict Management in the Partnership. 

Limitations 

1. Small Sample Size. Less than half of Site Directors are included in the analyses. The

experiences and perspectives of the other twelve PA Sites that did not participate in the survey are 

not represented and may be very different. 

2. Incomplete Inventory. A complete inventory of support services provided at all Public Allies

site could not be compiled due to incomplete Site Director Data. 

22 André de Waal , Robert Goedegebuure , Eveline Hinfelaar , (2015) "Developing a scale for measuring high 

performance partnerships", Journal of Strategy and Management, Vol. 8 Iss: 1, pp.87 - 108
Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Appendix A: CNCS Charts 

As seen in Tables 1 and 2 in both the post and retrospective survey the means for all seven 

outcomes are almost identical indicating that regardless of the definition of economic 

disadvantages used, outcome scores did not differ significantly. None of the seven outcomes have 

more than a .07 difference in the overall mean and as a result the CNCS definition will be used 

throughout the evaluation when referencing “disconnected Allies”.

Table 1. Retrospective Survey Pathway Allies Definition Comparison 

Disconnected Allies 

Retrospective CNCS 

Mean Scale Score 

Disconnected Allies 

Retrospective Non-CNCS 

Mean Scale Score 

Collaboration 30.80 31.84 

Controversy with Civility 37.26 37.27 

Resilience 7.97 7.90 

Transformational Leadership 26.35 26.30 

Self- confidence 21.11 21.16 

Civic Knowledge 21.62 21.70 

Increased Interest in Higher Education 14.33 14.50 

Table 2. Post Survey Disconnected Allies Definition Comparison 

Disconnected 

Allies Post CNCS 

Mean Scale Score 

Disconnected Allies 

Post Non-CNCS 

Mean Scale Score 

Collaboration 35.14 35.1 

Controversy with Civility 43.94 43.97 

Resilience 8.71 8.71

Transformational Leadership 30.01 30.03 

Self- confidence 28.3 28.37 

Civic Knowledge 29.56 29.55 

Increased Interest in Higher Education 17.62 17.68 

A detailed description of evaluation tools associated with each of the four individual evaluation 

component areas (Partner Organizations, Current Ally apprentices, Alumni Ally apprentices, and 

Public Allies Site Directors) is included within each components’ related section of the report.

76
77

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Appendix B: Partner Organization Theory of Change 
78

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Appendix C: Capacity Self-Assessment for Partner Organizations (versio 

Capacity Self-Assessment 

For Partner Organizations 

Version 26 February 2013; this page: Version 2 October 2013 

Organization Name OR Organization + Department / Division Name 

Completed By (Name & Title): 

Pre-Service Assessment Post-Service Assessment Date Completed: 

Mark the one you’ll conduct:

Assessment of the Organization Assessment of a Department / Division of the Organization 

Welcome & Instructions 

A. This tool is designed to help an organization assess its capacity to deliver better services. In assessing capacity, the users of this tool will

be able to establish goals for improvement. Users can also decide the ways in which the service of AmeriCorps members may be

leveraged to help improve or increase the organization’s capacity to provide services that improve community conditions.

B. The assessment should be completed by at least two people within the organization who represent executive-level and program/project-level

managerial perspectives. Each person should complete this tool on his/her own.

C. The assessment is in two parts and takes 10 to 15 minutes to complete.
Page 

Part One: Self-Assess 

Public Allies Partner Organization Capacity Self-Assessment 

Program Development, Delivery, and Evaluation 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level of 

capacity in place 

Exceptional level 

of capacity in place 

1. Program Development &

Delivery

(A) Is the organization able to

improve, expand, or create new

programming?

(B) Is the program delivery model

effective in responding to

constituent needs?

1 1. 

No effort to evaluate 

possibility of expanding 

existing program(s) to meet 

needs of recipients 

Limited ability to expand 

existing program(s) 

5 2 2

Limited effort to evaluate 

possibility of expanding 

existing program(s) 

Limited or no action taken to 

expand existing program(s) 

.5 3 3.

Occasional effort to evaluate 

possibility of expanding 

existing program(s) 

Action taken to expand 

existing program(s) 

5 4

Frequent evaluation of possibility 

of expanding existing program(s) 

Organization is able to efficiently 

and effectively grow existing 

program(s) to meet needs of 

recipients 

1 1. 

No evaluation of gaps in 

ability of current program(s) 

to meet recipient needs 

Limited ability to create new 

programs 

New programs created 

largely in response to 

funding 

5 2 2

Limited evaluation of gaps in 

ability of existing program(s) 

to meet recipient needs 

Some ability to modify 

existing programs and 

create new programs 

When new programs 

developed, thought given to 

organizational strategy and 

mission 

.5 3 3.

Occasional evaluation of gaps 

in ability of existing program(s) 

to meet recipient needs and 

adjustments sometimes made 

Demonstrated ability to modify 

and fine-tune existing 

programs and create new 

programs in line with 

organizational strategy and 

mission 

5 4

Regular evaluation of gaps in 

ability of existing program(s) to 

meet recipient needs and 

adjustment always made 

Ability and tendency to effectively 

create new, innovative programs 

to meet the needs of potential 

service recipients in line with 

strategy and mission 

79

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

For each row, determine the description that 

most closely describes your organization and 

circle the corresponding point on the scale. 
Page 

2. Outreach

Is the organization able to 

effectively reach the populations 

that can benefit from 

programming? 

1 1.

No outreach to populations 

that can benefit from 

programming 

5 2 2 

Organization conducts some 

outreach to populations that 

can benefit from 

programming, but no 

outreach plan is in place 

.5 3 3.

Organization conducts 

outreach to populations that 

can benefit from programming 

Organization is in the early 

stages of creating an outreach 

plan

Some attempts to conduct 

outreach in various languages 

5 4

Outreach is conducted to include 

populations speaking various 

languages 

Organization has a solid plan to 

effectively reach populations that 

can benefit from programming and 

conducts regular outreach to 

populations that can benefit from 

programming 

80

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 
81

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Program Development, Delivery, and Evaluation 

Page 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level of 

capacity in place 

Exceptional level 

of capacity in place 

3. Materials Development

Does the program have the 

appropriate materials/tools to 

effectively deliver programming? 

Note: AmeriCorps member 

activity must promote the 

organization’s programming and 

cannot promote the organization 

in general.

1 1. 

Organization has no formal 

materials or tools for 

program delivery 

Programs are delivered 

largely through individuals 

who share learning through 

word-of-mouth 

5 2 2

Organization has a few 

materials or tools that assist 

with program delivery 

Materials exist but are outof-date and need to be 

updated 

Organization is in the early 

stages of planning to 

document materials/tools 

that can deliver effective 

programming 

.5 3 3.

Organization has a number of 

materials or tools that assist 

with program delivery 

Organization has some form of 

plan/process to document 

materials/tools that can deliver 

effective programming 

5 4

Organization has a high quality, 

comprehensive set of materials or 

tools that assist with program 

delivery 

Materials and tools are continually 

updated as needed 

Organization has a 

comprehensive plan/process for 

documenting materials/tools and 

uses these tools to consistently 

deliver effective programming 

4. Performance Measurement

(A) Is the organization able to

assess the impact of its

programming on the constituency

served?

(B) Does the organization use

performance data to improve

program offerings?

1 1.

No attempt to track program 

outcomes 

All or most evaluation based 

on anecdotal evidence 

5 2 2

Some attempt to track 

program outputs ( e.g., 

number of children served) 

Limited attempt to track 

program outcomes (e.g., 

drop-out rate lowered) 

.5 3 3. 

Program outcomes tracked 

regularly 

Tracking outcomes not integral 

part of organizational strategy 

5 4

Well-developed comprehensive 

system used to track program 

outcomes on continual basis 

Tracking program outcomes part 

of organizational strategy 

1 1. 

Little or no analysis of 

program performance 

measurement and tracking 

data 

Data rarely used to improve 

program(s) 

5 2 2

Some efforts made to 

analyze performance 

measurement and tracking 

data 

Performance data used 

occasionally to improve 

program performance 

.5 3 3. 

Effective analysis of 

performance measurement 

and tracking data, but driven 

largely by top management 

What is learned is distributed 

throughout organization 

Often used to make 

adjustments and 

improvements

5 4

Comprehensive analysis of data 

part of the organizational culture 

Used by staff in target-setting and 

daily operations 

Systematic practice of making 

adjustments and improvements on 

basis of consistent analysis 
82

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Community Engagement / Interface 

Page 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level 

of capacity in place 

Exceptional level 

of capacity in place 

1. Community Assessment

(A) Does the organization

understand the community needs

to be addressed, community

context, and the assets available?

(B) Does an understanding of the

community being served drive

program development and delivery

modeling?

1 1. 

Organization plans new 

programs on an ad hoc 

basis with no external 

research 

New program neither 

necessarily based on 

available data nor linked to 

mission, vision, strategic 

plan

No pre-thought to 

organization’s capacity

5 2 2

New programs planned 

based on some effort to do 

external needs assessment 

New programs somewhat 

linked to to mission, vision or 

strategic plan 

Some thought given to 

organization’s internal 

capacity 

.5 3 3. 

New programs planned after 

needs assessment 

New program linked to either 

mission, vision, or based on 

strategic plan 

Thought given to internal 

capacity 

5 4

Assessment made of existing 

programs and other community 

resources to meet need and 

reduce fragmentation & 

duplication of services 

New programs linked to mission 

and based on strategic plan 

Thought given to internal capacity 

and program funding 

1 1.

Minimal knowledge and 

understanding of other 

players and alternative 

models in program area 

5 2 2 

Basic knowledge of players 

and alternative models in 

program area 

Limited ability to adapt 

decision-making based on 

understanding 

.5 3 3. 

Solid knowledge of players 

and alternative models in 

program area 

Good ability to adapt decisionmaking based on acquired 

understanding at times 

5 4

Extensive knowledge of players 

and alternative models in program 

area 

Refined ability and systematic 

tendency to adapt decisionmaking based on understanding 

2. Community Awareness and

Engagement

Is community aware of 

organization’s services such that 

services are utilized by target 

audience? 

Note: AmeriCorps member 

activity must promote the 

organization’s programming and 

cannot promote the organization 

in general.

1 1.

Presence of organization’s 

program(s) either not 

recognized, understood, or 

generally not regarded as 

positive 

Few members of the 

community constructively 

involved in the organization 

5 2 2 

Presence of organization’s 

program(s) somewhat 

recognized, understood, or 

generally regarded as 

positive within the 

community 

Members of the community 

and some members of the 

broader community (e.g., 

other nonprofit leaders, 

academics, business 

leaders) engaged 

.5 3 3. 

Organization’s program(s) 

reasonably well known within 

community, and perceived as 

open and responsive to 

community needs 

Members of broader 

community (including a few 

prominent ones) constructively 

involved in organization 

5 4

Organization’s program(s) widely 

known within community, and 

perceived as actively engaged 

with and extremely responsive to 

it

Many community members and 

members of the broader 

community (including many 

prominent members) actively and 

constructively involved in 

organization 
83

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Community Engagement / Interface 

Page 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level 

of capacity in place 

Exceptional level 

of capacity in place 

3. Partnerships and

Collaboration

Does the organization collaborate 

with others to help improve or 

expand programming and help 

reduce or improve fragmentation of 

services? 

1 1. 

No use or very limited use of 

partnerships and alliances 

with nonprofit, government, 

or for-profit entities 

5 2 2 

Early stages of building 

relationships and 

collaborating with nonprofit, 

government, or for-profit 

entities 

.5 3 3.

Effectively build and leverage 

some key relationships with a 

few types of relevant nonprofit, 

government, or for-profit 

entities 

Some relationships may not be 

fully “win-win” or mutually 

beneficial 

5 4

Build, leverage, and maintain 

strong relationships with variety of 

relevant nonprofit, government, or 

for-profit entities 

Relationships deeply anchored in 

stable, long-term, mutually 

beneficial collaboration 
84

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Organizational Infrastructure & Resources 

Page 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level 

of capacity in place 

Exceptional level 

of capacity in place 

1. Technology Use

(A) Does the organization have a

website in place that supports and

promote its program(s)?

(B) Are there technological systems

in place that help improve

organizational efficiency and

effectiveness?

1 1.

Organization does not have 

a website for a program 

5 2 2

Program has a website that 

conveys basic information 

about its program(s) but it is 

rarely updated 

.5 3 3.

Organization has or is planning 

a website that supports 

program(s) 

Website is/will be regularly 

updated 

5 4

Organization has a regularly 

updated program website that is 

interactive and well planned 

Website promotes/supports the 

organization’s program(s)

1 1. 

No or limited systems in 

place for tracking contacts 

(e.g. clients, members, 

volunteers, funders, or 

prospects) or program 

activities and outcomes 

5 2 2

Electronic databases and 

management reporting 

systems exist only in a few 

areas 

Systems perform only basic 

features and do not fully 

meet organizational needs 

.5 3 3. 

Electronic database and 

management reporting 

systems exist in most areas for 

tracking contacts and program 

activities/outcomes 

Commonly used and help 

increase information sharing 

and efficiency

5 4

Sophisticated, comprehensive 

database management reporting 

systems exist for tracking contacts 

and program activities/outcomes 

Systems are widely used and 

essential in increasing information 

sharing and efficiency and data 

supplied by systems are used to 

make decisions 

2. Volunteer Generation

Does the organization have the 

adequate support of community 

volunteers to help assist in the 

delivery of service? 

1 1.

No or few volunteers with 

limited skills 

May be unreliable or have 

low commitment 

Volunteers are poorly 

managed and trained 

5 2 2

Volunteers have good skills 

Mostly reliable, loyal and 

committed to the success of 

the organization and its 

programs 

Volunteers trained and 

managed but without 

standards and little 

accountability 

.5 3 3.

Very capable set of volunteers, 

bring necessary skills to the 

organization and its programs 

Reliable, loyal and highly 

committed to organization’s 

success and to “making things 

happen”

Work easily with most staff, but 

do not generally play core 

roles without substantial staff 

supervision 

Volunteers are managed and 

contribute to the overall 

success of the organization 

5 4

Extremely capable set of 

volunteers, bring complementary 

skills to the organization and its 

programs 

Reliable, loyal, highly committed 

to organization’s success, often 

go “beyond call of duty”

Able to work easily with wide 

range of staff and play core roles 

without special supervision 

Volunteers managed very well, 

significantly contribute to overall 

success of organization, and 

contributions are recognized by 

the organization 
85

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Organizational Infrastructure & Resources 

Page 

Activity Area Clear need 

for increased capacity 

Basic level 

of capacity in place 

High level 

of capacity in place 

Exceptional level 

of capacity in place 

3. Organizational Management

and Operations

(A) Does the organization have the

policies and systems for effective

management, staff development,

and internal communications?

(B) Does the organization have a

process for program and/or project

coordination?

1 1.

Limited set of processes 

(e.g., decision making, 

planning, reviews) for 

ensuring effective 

functioning of organization 

and its programs 

Use of processes is variable, 

or processes are seen as ad 

hoc requirements 

(“paperwork exercises")

No monitoring or 

assessment of processes 

5 2 2 

Basic set of processes in 

core areas for ensuring 

efficient functioning of the 

organization and its 

programs 

Processes known, used, and 

truly accepted by only 

portion of staff 

Limited monitoring and 

assessment of processes, 

with few improvements 

made as a result 

.5 3 3.

Solid, well designed set of 

processes in place in core 

areas to ensure smooth, 

effective functioning of the 

organization and its programs 

Processes known and 

accepted by many, often used 

and contribute to increased 

impact 

Occasional monitoring and 

assessment of processes, with 

some improvements made 

5 4

Lean, and well designed set of 

processes (e.g., decision making, 

planning, reviews) in place in all 

areas to ensure effective and 

efficient functioning of 

organization and its programs 

Processes are widely known, 

used and accepted, and are key 

to ensuring full impact of 

organization 

Continual assessment and 

systematic improvements made 

1 1. 

Different programs within 

organization operate 

independently with little or 

no coordination between 

them 

No thought given to possible 

shared resources 

5 2 2

Interactions between 

different programs are 

generally good, though 

coordination issues do exist 

Some pooling of resources 

.5 3 3.

All programs within 

organization function together 

effectively with sharing of 

information and resources 

where appropriate 

Few coordination issues 

5 4

Constant and seamless 

integration between different 

programs with few coordination 

issues 

Relationships are dictated by 

organizational needs (rather than 

hierarchy or politics) 

4. Financial Resources

Does the organization have a 

sufficient and diversified funding 

stream to support its efforts? 

Note: Limited to 10% of 

AmeriCorps Members’ time and 

must be directly related to 

programming. Allies must not 

raise funds for general 

operations.

1 1.

Organization has no overall 

funding plan for its 

program(s) 

Program(s) is/are highly 

dependent on a few funders 

Largely of same type (e.g., 

government, foundation, 

corporation or private 

Individuals) 

5 2 2 

Organization has some 

overall funding plan for its 

program(s) 

Program has access to 

multiple funding types ( e.g., 

government, foundations, 

corporations, individuals) 

with a few funders in each 

type OR Has many funders 

within only one or two types 

.5 3 3.

Organization has solid overall 

funding plan for its program(s) 

Program(s) has/have solid 

base of funders in most types 

of funding sources 

Has developed some 

sustainable revenue 

generating activity for 

program(s) 

5 4

Organization has solid funding 

plan in place for its program(s) 

Program(s) has/have funding that 

is highly diversified with funding 

across multiple source types 

Organization has developed 

sustainable revenue generation 

activities for program(s) 
Page 8 

Part Two: Reflect 

Public Allies Partner Organization Capacity Self-Assessment 

Understanding Our Shared Values 

 We believe that through the practice of a set of core values in our work, service, and interactions

with others, we can sustainably improve community conditions.

 As we continue to deepen our understanding of your organizational capacity, we would like to

know more about your organizational culture and the role values play in your organization.

 As in the previous section, read through each row and circle the point on the scale that best

describes your organization.

86

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 
87

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Organizational Culture 

Shared Beliefs and 

Values 

Does the organization 

have a common set of 

basic beliefs and values 

that guide individual 

behavior and support 

achievement of 

organizational mission? 

1 1.

No common set of basic beliefs 

and values exists within the 

organization 

5 2 2

A common set of basic beliefs/values 

exists and are written, but those 

beliefs are not embraced throughout 

the organization 

Written beliefs are applied only in 

pockets of the organization 

.5 3 3

A common set of beliefs/values 

are written and embraced 

throughout the organization 

Organizational values are used 

to help stakeholders (staff, 

board, collaborators, etc.) think 

about how to deliver 

programming 

.5 4

A common set of values guides the 

organization at all levels and 

values are consistently reflected in 

the operational and policy-related 

decisions made by staff and 

decision-making bodies. 

Engaging Community 

Assets 

Does the organization 

view and engage 

community members, 

including current & 

potential clients, as 

assets that can 

enhance the 

organization’s impact?

1 1.

Community members are 

“clients” to be served. Client 

feedback rarely sought. 

5 2 2

Clients and constituent community 

members’ feedback is sought about 

the organization’s services and used 

occasionally to improve services 

.5 3 3

Clients and representatives of 

the constituent community are 

routinely asked to provide 

feedback that is used to 

improve services 

Clients and constituents are 

viewed as assets and are hired 

and sought as volunteers 

.5 4

Organization highly values the 

perspective of its clients/ 

constituents; client input is given 

on the development of policies 

affecting them

Clients/constituents are 

represented at the highest 

decision-making levels of the 

organization 

Inclusiveness 

Are the organization’s 

board, staff, and 

volunteers reflective of 

diverse backgrounds 

and experiences of the 

community it serves? 

1 1.

Staff, board and volunteers 

drawn from a narrow range of 

backgrounds and experiences 

No representation of the 

community/clients among the 

organization’s staff, board or 

volunteers 

5 2 2

Some variety of backgrounds and 

experiences among board, staff and 

volunteers 

Sparse community/client 

representation among staff, board or 

volunteers, primarily among 

volunteers 

.5 3 3

Staff, board and volunteers 

drawn from diverse backgrounds 

and experiences reflective of the 

community’s diversity and bring 

a broad range of skills to the 

organization 

Community/clients represented 

among staff and volunteers, 

primarily in front-line or direct 

service roles with little to no 

influence over decision-making 

.5 4

Staff, board and volunteers drawn 

from extraordinarily diverse 

backgrounds and experiences and 

bring a broad range of skills 

Community/clients are 

represented throughout the 

organization, including roles with 

influence and decision-making 

authority 

For each row, determine the description that 

most closely describes your organization and 

circle the corresponding point on the scale. 
88

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Organizational Culture 

Disconnected Youth 

How does the 

organization view and 

engage disconnected 

youth (young adults, 

ages 16 -24, with no 

college degree or 

limited employment 

experience or 

unemployed) 

1 1.

Disconnected young adults are 

not engaged in the organization 

5 2 2

Organization sees disconnected 

young adults as assets and delivers 

programming to this group 

.5 3 3

Organization delivers 

programming to address 

disconnected youth and 

attempts to engage them as 

interns, volunteers, employees, 

and committee members 

5 4

Organization regularly employs 

formerly disconnected youth or 

engages them as leaders on the 

executive level and in highest 

decision-making bodies 
89

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Page 11 

Part Three: Choose the Focus & Finalize Scores 

Public Allies Partner Organization Capacity Self-Assessment 

Focusing the Partnership 

1. Gather those who complete self-assessments and discuss the scores.

2. Are there differences / similarities in how you scored the items? What does that reveal?

For Part I 

3. Using what you discussed, agree on an activity area or the areas that you would like to focus on for this partnership.

4. In the area or areas you chose, agree upon a final score for the area or scores for those areas. These are not grades. Rather, they are

pre-assessment ratings to help develop position descriptions.

For Part II 

5. For all four questions, agree upon final scores.
90

Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research 

Appendix D: Ally-focused Theory of Change 
91

Appendix E - Current Public Allies Survey 

This survey is part of a larger evaluation of the Public Allies program that is being conducted by the 

University of Wisconsin-Milwaukee’s Center for Urban Initiatives and Research (UWM/CUIR). As a current 

Public Ally, your insights and opinions are extremely important!

As the program’s external evaluators, please be assured that protecting your privacy is our highest priority 

at CUIR and we promise to maintain confidentiality for you at all times. At no time will we reveal your 

participation or personally identify you in any oral or written reports. We will be providing a final report of the 

evaluation findings to Public Allies, but will report answers only in aggregate, as a group.

The findings from the evaluation will be used to improve the Public Allies program and to better understand 

how the apprenticeship advances leaders committed to a just and equitable society. The data you provide 

through this survey will go directly to UWM/CUIR’s secure website.

This survey should take about 15 minutes of your time to complete and your participation is voluntary, of 

course. However, your insights and opinions are extremely important, so thank you for agreeing to help with 

this important survey!

Please read each statement carefully and choose the response that best fits you. 

21st Century Skills 

Q1: SRLS-R2: Collaboration Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I am seen as someone who

works well with others.     

2. I can make a difference

when I work with others on

a task.

    

3. I actively listen to what

others have to say.     

4. I enjoy working with others

toward common goals.     

5. Others would describe me

as a cooperative group

member. 

    

6. Collaboration produces

better results.     

7. My contributions are

recognized by others in the

groups I belong to. 

    

8. I am able to trust the people

with whom I work.     
92

Q2: Controversy with Civility Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I am open to others’ ideas.     

2. Creativity can come from

conflict.     

3. I value differences in others.     

4. Hearing differences in

opinions enriches my thinking.     

5. I struggle when group

members have ideas that are

different from mine.



Reverse    

6. Greater harmony can come

out of disagreement.     

7. I respect opinions other

than my own     

8. I am uncomfortable when

someone disagrees with me.



Reverse    

9. When there is a conflict

between two people, one will

win and the other will lose.



Reverse    

10. I am comfortable with

conflict.     

11. I share my ideas with

others.     

Q3: Resilience Scale (CDRISC) 

Strongly 

Disagree Disagree 

Neither 

Agree nor 

Disagree 

Agree Strongly 

Agree 

1. I am able to adapt to

change.     

2. I tend to bounce back after

I’ve experienced a hardship.     

Transformational 

Leadership Please read each statement carefully and choose the response that best fits 

you. 

Q4: Global Transformational 

Leadership Scale Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I communicate a clear and

positive vision of the future     
93

2. I treat others as individuals and

support and encourage their

development

    

3. I give encouragement and

recognition to staff     

4. I foster trust, involvement and

cooperation among team

members

    

5. I encourage thinking about

problems in new ways and

questions assumptions

    

6. I am clear about my values and

practices what I preach     

7. I instill pride and respect in

others and inspire them by being

highly competent

    

Self- confidence Thinking of your current situation, how much confidence do you have that you could 

accomplish the following tasks? 

Q5: Career Engagement 

Scale 

(Modified) 

No 

confidence at 

all 

Very little 

confidence 

Moderate 

confidence

Much 

confidence 

Complete 

confidence 

1. Actively seek to design your

professional future     

2. Undertake things to achieve

your career goals     

3. Care for the development of

your career.     

4. Develop plans and goals for

your future career     

5. Sincerely think about personal

values, interests, abilities, and

weaknesses

    

6. Collect information about

employers, professional

development opportunities, or

the job market in your desired

area

    

7. Establish or maintain

contacts with people who can

help you professionally

    

8. Voluntarily participate in

further education, training, or

other events to support your

career

    

9. Assume duties or positions

that will help you progress

professionally

    
94

Civic Knowledge If you found out about a problem in your community that you wanted to do 

something about (for example, police relations with your community are strained), 

how well do you think you would be able to do each of the following? 

Q6: Competence for 

Civic Action Scale 

I definitely 

can’t

I probably 

can’t Maybe I probably can 

I definitely 

can 

1. Create a plan to address the

problem.     

2. Get other people to care

about the problem.     

3. Organize and run a meeting.     

4. Express your views in front

of a group of people.     

5. Identify individuals or

groups who could help you

with the problem.

    

6. Write an opinion letter to a

local newspaper.     

7. Call someone on the phone

that you had never met before

to get their help with the

problem.

    

8. Contact an elected official

about the problem     

9. Organize a Petition.     

Increased Interest in 

Higher Education 

Yes, within 6 

months 

Yes, within 

the next 

year 

Yes, but not 

within the 

next year 

No, I already 

have a degree 

No, I don’t 

plan on 

attending 

college 

Q7a: Are you interested in 

obtaining your degree or 

completing a college course 

contributing toward a college 

degree?

    

Vocational/ 

Technical 

2 year 

college 

4 year 

college

Graduate 

School 

Does not 

apply 

Q7b:. If yes, what type of degree 

are you interested in obtaining?     

How confident are you in your ability to...
No 

confidence at 

all

Very little 

confidence

Moderate 

confidence

Much 

confidence

Complete 

confidence

Q8_1. ...collect information 

about colleges or other 

educational opportunities that are 

available beyond high school.

    

Q8_2. ...talk to a school 

counselor and/or other mentors 

about educational options beyond 

high school.

    

Q8_3. ...request information 

(financial aid, admission 

requirements, and deadlines, etc) 

and/or applications for admission 

to college.

    

Q8_4. ...develop plans and goals 

for your educational future?     

Q8_5. ...research academic 

majors that might be a good fit 

with your interests and goals.

    

Retrospective items 

In the following section you will be asked a series of questions that refer to yourself during the time prior to when 

you joined the Public Allies program. To the best of your ability, please think back and choose the response that best 

fit you prior to joining the Public Allies program. 

21st Century Skills Thinking about yourself prior to joining Public Allies, please read each 

statement carefully and choose the response that best fit you at that time.

Q1r: SRLS-R2: Collaboration Strongly 

disagree Disagree

Neither 

Agree nor 

Disagree Agree

Strongly 

agree

1. I was seen as someone who

worked well with others.     

2. I could make a difference

when I worked with others

on a task.

    

3. I actively listened to what

others have to say.     

4. I enjoyed working with

others toward

common goals.

    

5. Others described me as a

cooperative group member.     

6. I felt collaboration produced

better results.     

95
96

7. My contributions were

recognized by others in the

groups I belonged to. 

    

8. I was able to trust the

people with whom I worked.     

Q2r: Controversy with Civility 

Strongly 

disagree 

Disagree Neither 

Agree nor 

Disagree 

Agree 

Strongly 

agree 

1. I was open to others’ ideas.     

2. I believed creativity could

come from conflict.

    

3. I valued differences in others.     

4. I believed hearing

differences in opinions enriched

my thinking.

    

5. I struggled when group

members had ideas that were

different from mine.



Reverse 

   

6. I felt that greater harmony

could come out of

disagreement.

    

7. I respected opinions other

than my own

    

8. I was uncomfortable when

someone disagreed with me.



Reverse 

   

9. When there was a conflict

between two people, I believed

one would win and the other

would lose.



Reverse 

   

10. I was comfortable with

conflict.

    

11. I shared my ideas with

others.

    

Q3r: Resilience Scale (CDRISC) 

Strongly 

Disagree 

Disagree Neither 

Agree nor 

Disagree 

Agree Strongly 

Agree 

1. I was able to adapt to change.     

2. I tended to bounce back after

I’d experienced a hardship.

    
97

Transformational 

Leadership Thinking about your life before you joined Public Allies, 

please choose the response that best fit you at that time. 

Q4r: Global 

Transformational 

Leadership Scale 

Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I communicate a clear and

positive vision of the future     

2. I treat others as individuals and

support and encourage their

development

    

3. I give encouragement and

recognition to staff     

4. I foster trust, involvement and

cooperation among team

members

    

5. I encourage thinking about

problems in new ways and

questions assumptions

    

6. I am clear about my values and

practices what I preach     

7. I instill pride and respect in

others and inspire them by being

highly competent

    

Civic Knowledge 

Thinking about your life before you joined Public Allies, please respond how you 

would have reacted if you found out about a problem in your community that you 

wanted to do something about (for example, police relations with your community 

are strained), how well do you think you would have be able to do each of the 

following? 

Q5: Competence for 

Civic Action Scale 

I definitely 

couldn’t 

have 

I probably 

couldn’t 

have 

Maybe I 

could have 

I probably 

could have 

I definitely 

could have 

1. Create a plan to address the

problem.     

2. Get other people to care

about the problem.     

3. Organize and run a meeting.     

4. Express your views in front

of a group of people.     

5. Identify individuals or

groups who could help you

with the problem.

    

6. Write an opinion letter to a

local newspaper.     
98

7. Call someone on the phone

that you had never met before

to get their help with the

problem.

    

8. Contact an elected official

about the problem     

9. Organize a petition.     

Self- confidence Prior to joining Public Allies, how much confidence did you have that you could: 

Q6r: Career Engagement 

Scale 

(Modified) 

No 

confidence 

at all 

Very little 

confidence 

Moderate 

confidence

Much 

confidence 

Complete 

confidence 

1. Actively seek to design your

professional future     

2. Undertake things to achieve

your career goals     

3. Care for the development of

your career.     

4. Develop plans and goals for

your future career     

5. Sincerely think about

personal values, interests,

abilities, and weaknesses

    

6. Collect information about

employers, professional

development opportunities, or

the job market in your desired

area

    

7. Establish or maintain

contacts with people who can

help you professionally

    

8. Voluntarily participate in

further education, training, or

other events to support your

career

    

9. Assume duties or positions

that will help you progress

professionally

    
99

Increased Interest in 

Higher Education 

Thinking about your life before you joined Public Allies, 

please choose the response that best fit you at that time. 

Yes, I was 

planning on 

attending 

college 

No, I already 

had a degree 

No, I didn’t 

plan on 

attending 

college 

Q7a: Were you interested in 

obtaining your degree or 

completing a college course 

contributing toward a college 

degree?

  

Vocational/ 

Technical 

2 year 

college 

4 year 

college 

Graduate 

School 

Does not 

apply 

Q7b: If yes, what type of 

degree were you interested in 

obtaining?

    

How confident were you in your ability to...

No confidence 

at all 

Very little 

confidence 

Moderate 

confidence

Much 

confidence 

Complete 

confidence 

Q8_1. ...collect information 

about colleges or other 

educational opportunities 

that were available beyond 

high school. 

    

Q8_2. ...talk to your school 

counselor and/or other 

mentors about educational 

options beyond high school. 

    

Q8_3. ...request information 

(financial aid, admission 

requirements, and deadlines, 

etc) and/or applications for 

admission to college. 

    

Q8_4. ...develop plans and 

goals for your educational 

future? 

    

Q8_5. ...research academic 

majors that might have been a 

good fit with your interests and 

goals. 

    
100

Appendix F- Public Allies Alumni Survey 

This survey is part of a larger evaluation of the Public Allies program that is being conducted by the 

University of Wisconsin-Milwaukee’s Center for Urban Initiatives and Research (UWM/CUIR). As a Public 

Allies Alum, your insights and opinions are extremely important!

As the program’s external evaluators, please be assured that protecting your privacy is our highest priority 

at CUIR and we promise to maintain confidentiality for you at all times. At no time will we reveal your 

participation or personally identify you in any oral or written reports. We will be providing a final report of the 

evaluation findings to Public Allies, but will report answers only in aggregate, as a group and individual 

responses will not be identifiable.

The findings from the evaluation will be used to improve the Public Allies program and to better understand 

how the apprenticeship advances leaders committed to a just and equitable society. The data you provide 

through this survey will go directly to UWM/CUIR’s secure website.

This survey should take about 15 minutes of your time to complete and your participation is voluntary, of 

course. However, your insights and opinions are extremely important, so thank you for agreeing to help with 

this important survey!

21st Century Skills Please read each statement carefully and choose the response that best fits you 

Q1: SRLS-R2: Collaboration Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

9. I am seen as someone who

works well with others.     

10. I can make a difference

when I work with others on

a task.

    

11. I actively listen to what

others have to say.     

12. I enjoy working with others

toward common goals.     

13. Others would describe me

as a cooperative group

member.

    

14. Collaboration produces

better results.     

15. My contributions are

recognized by others in the

groups I belong to.

    

16. I am able to trust the people

with whom I work.     
101

Q2: Controversy with Civility Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I am open to others’ ideas.

    

2. Creativity can come from

conflict.     

3. I value differences in others.

    

4. Hearing differences in

opinions enriches my thinking.     

5. I struggle when group

members have ideas that are

different from mine.

    

6. Greater harmony can come

out of disagreement.     

7. I respect opinions other

than my own     

8. I am uncomfortable when

someone disagrees with me.     

9. When there is a conflict

between two people, one will

win and the other will lose.

    

10. I am comfortable with

conflict.     

11. I share my ideas with

others.     

Q3: Resilience Scale (CDRISC) 

Strongly 

Disagree Disagree 

Neither 

Agree nor 

Disagree 

Agree Strongly 

Agree 

2. I am able to adapt to

change.     

2. I tend to bounce back after

I’ve experienced a hardship.     

Transformational 

Leadership Please read each statement carefully and choose the response that best fits 

you. 

Q4: Global Transformational 

Leadership Scale Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I communicate a clear and

positive vision of the future     
102

2. I treat others as individuals and

support and encourage their

development

    

3. I give encouragement and

recognition to staff     

4. I foster trust, involvement and

cooperation among others.     

5. I encourage thinking about

problems in new ways and

questions assumptions

    

6. I am clear about my values and

practices what I preach     

7. I instill pride and respect in

others and inspire them by being

highly competent

    

Self- confidence Thinking of your current situation, how much confidence do you have that you can: 

Q5: Career Engagement 

Scale (Modified) 

No 

confidence at 

all 

Very little 

confidence 

Moderate 

confidence

Much 

confidence 

Complete 

confidence 

1. Actively seek to design your

professional future     

2. Undertake things to achieve

your career goals     

3. Care for the development of

your career.     

4. Develop plans and goals for

your future career     

5. Sincerely think about personal

values, interests, abilities, and

weaknesses

    

6. Collect information about

employers, professional

development opportunities, or

the job market in your desired

area

    

7. Establish or maintain

contacts with people who can

help you professionally

    

8. Voluntarily participate in

further education, training, or

other events to support your

career

    

9. Assume duties or positions

that will help you progress

professionally

    
103

Civic Engagement 

Activities Since graduating from Public Allies, how often have you done the following activities? 

Q6: Community-based 

Activism Scale Never Rarely Sometimes Often Always 

1. Worked with other people in

your neighborhood to fix or

improve something

    

2. Attended any public meeting

where there was a discussion of

community affairs.

    

3. Attended any club or

organizational meeting   

 

4. Participated in community

events such as community

meetings, celebrations, or

activities.

    

5. Joined organizations that

support issues that are important

to you.

    

6. Wrote or emailed newspaper or

organizations to voice your view

on an issue.

    

Increased Employment Comparing your current employment to the job you had prior to 

joining Public Allies...

Yes No

Q7a: Are you currently 

employed?  

Q7b: If yes... Yes No About the 

same 

1. ...have your job

responsibilities increased in a

significant way?

  

2. ...have your wages increased?   

3. ...has your access to employer

sponsored health benefits (i.e.

health insurance, retirement, etc.)

increased?

  

4. ...do you feel like you are now

on a pathway towards your

desired career?

  

Yes No Currently 

unemployed 
104

Q8a: Are you currently 

employed by a non-profit 

organization? 

  

Q8b: Are you currently 

employed by a public entity (i.e. 

city, county, state government)? 

  

Increased Higher 

Education 

Yes, I have 

completed my 

degree 

Yes, I am 

currently 

working on 

my degree. 

No, I already 

had a degree 

No, but I still 

plan on 

attending 

college 

No, and I 

don’t plan 

on 

attending 

college 

Q9a: Since leaving the program, 

have you obtained your degree or 

completed a college course 

contributing toward a college 

degree?

    

Vocational/ 

Technical 

2 year 

college 4 year college Graduate 

School 

Not at this 

time

Q9b: If yes, what type of degree 

have you obtained or are 

obtaining? 

    

In the following section you will be asked a series of questions asking you reflect back to the time before you joined the Public Allies 

program. To the best of your ability, please think back and choose the response that best fit you prior to your 

participation in the Public Allies program. 

Retrospective items 

21st Century Skills Thinking about yourself prior to joining Public Allies, please read each 

statement carefully and choose the response that best fit you at that time. 

Q1r: SRLS-R2: Collaboration Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

9. I was seen as someone who

worked well with others.     

10. I could make a difference

when I worked with others

on a task.

    

11. I actively listened to what

others have to say.     

12. I enjoyed working with

others toward

common goals.

    
105

13. Others described me as a

cooperative group member.     

14. I felt collaboration produced

better results.     

15. My contributions were

recognized by others in the

groups I belong to.

    

16. I was able to trust the

people with whom I worked.     

Q2r: Controversy with Civility 

Strongly 

disagree 

Disagree Neither 

Agree nor 

Disagree 

Agree 

Strongly 

agree 

1. I was open to others’ ideas.     

2. I believed creativity could

come from conflict.

    

3. I valued differences in others.     

4. I believed hearing

differences in opinions enriched

my thinking.

    

5. I struggled when group

members had ideas that were

different from mine.

    

6. I felt that greater harmony

could come out of

disagreement.

    

7. I respected opinions other

than my own

    

8. I was uncomfortable when

someone disagreed with me.

    

9. When there was a conflict

between two people, I believed

one would win and the other

would lose.

    

10. I was comfortable with

conflict.

    

11. I shared my ideas with

others.

    

Q3r: Resilience Scale (CDRISC) 

Strongly 

Disagree 

Disagree Neither 

Agree nor 

Disagree 

Agree Strongly 

Agree 

1. I was able to adapt to change.     

2. I tended to bounce back after

I’d experienced a hardship.

    
106

Transformational Leadership Thinking about your life before you joined Public Allies, 

please choose the response that best fit you at that time. 

Q4r: Global Transformational 

Leadership Scale Strongly 

disagree Disagree 

Neither 

Agree nor 

Disagree Agree 

Strongly 

agree 

1. I communicated a clear and

positive vision of the future     

2. I treated others as individuals

and supported and encouraged

their development

    

3. I gave encouragement and

recognition to others     

4. I fostered trust, involvement and

cooperation among others.     

5. I encouraged thinking about

problems in new ways and

questioned assumptions

    

6. I was clear about my values and

practiced what I preach     

7.I instilled pride and respect in

others and inspired them by being

highly competent

    

Self- confidence Prior to joining Public Allies, how much confidence did you have that you could: 

Q5r: Career Engagement 

Scale (Modified) 

No 

confidenc 

e at all 

Very little 

confidence 

Moderate 

confidence

Much 

confidence 

Complete 

confidence 

1. Actively seek to design your

professional future     

2. Undertake things to achieve

your career goals     

3. Care for the development of

your career.     

4. Develop plans and goals for

your future career     

5. Sincerely think about personal

values, interests, abilities, and

weaknesses

    

6. Collect information about

employers, professional

development opportunities, or

the job market in your desired

area

    

7. Establish or maintain contacts

with people who can help you

professionally

    
107

8. Voluntarily participate in

further education, training, or

other events to support your

career

    

9. Assume duties or positions

that will help you progress

professionally

    

Civic Engagement 

Activities 

Thinking back to the 12 months before you joined Public Allies, how often 

did you participate in the following activities? 

Q6r: Community-based 

Activism Scale Never Rarely Sometimes Often Always 

1. Worked with other people in

your neighborhood to fix or

improve something

    

2. Attended any public meeting

where there was a discussion of

community affairs.

    

3. Attended any club or

organizational meeting     

4. Participated in community

events such as community

meetings, celebrations, or

activities.

    

5. Joined organizations that

support issues that were important

to you.

    

6. Wrote or emailed newspaper or

organizations to voice your view on

an issue.

    

Increased Interest in 

Higher Education 

Thinking about your life before you joined Public Allies, 

please choose the response that best fit you at that time. 

Yes, I was 

planning on 

attending 

college 

No, I 

already had 

a degree 

No, I didn’t 

plan on 

attending 

college 

Q7ar: Were you planning on 

obtaining your degree or 

completing a college course 

contributing toward a college 

degree?

  

Vocational/ 

Technical 

2 year 

college 

4 year 

college 

Graduate 

School 

Does not 

apply 

Q7br: If yes, what type of degree 

were you planning on obtaining?     