AmeriCorps VIP 

Volunteer Capacity Study

November 2012 

Prepared for: 

Julie McClure & Sara F. Sitch 

CalSERVES-NCOE 

5789 State Farm Dr., Suite 230 

Rohnert Park, CA 94928
2

Contents 

Introduction ................................................................................................................................. 3

Description of Intervention ......................................................................................................... 3

Research Questions ..................................................................................................................... 4

Description of Programs ............................................................................................................. 4

Sampling & Data Collection ....................................................................................................... 8

Volunteer Capacity Assessment Instrument ........................................................................... 8

Data Collection Worksheet Instrument ................................................................................... 9

Methods....................................................................................................................................... 9

Propensity Score Matching ..................................................................................................... 9

Comparing Data Points from Partner and Comparison Group Sites .................................... 10

Differences between AmeriCorps VIP Sites and Non-participating Sites (Unmatched) ......... 11

Differences between AmeriCorps VIP Sites and Non-participating Sites (PSM) .................... 14

Reliability and Validity of the VCA Instrument ....................................................................... 18

Internal and External Validity ................................................................................................... 18

Discussion/Findings .................................................................................................................. 20

Conclusions ............................................................................................................................... 21

Appendix A: VCA Instrument ...................................................................................................... 23

Appendix B: DCA Form ............................................................................................................... 24

Appendix C: Test-Retest Reliability of the VCA Instrument ....................................................... 32
3

Introduction 

The AmeriCorps Volunteer Infrastructure Program (VIP) was developed to build the volunteer 

capacity of non-profit and educational organizations serving California communities. These 

communities are facing challenges such as poverty, hunger, homelessness, unemployment, 

limited education and English Language ability. Often accompanying these issues are ills such as 

low student academic achievement and test scores; high dropout rates; and high rates of justice 

system involvement. Non-profit and educational agencies struggle to solve these problems at a 

time when their budgets have been severely diminished by the recession. The need for outreach, 

service, and support for these communities is high and climbing. Community volunteers could 

help meet these needs, but organizations often lack the infrastructure to effectively use them. 

Since 2009, AmeriCorps VIP has placed over 400 members in community organizations with the 

specific targeted goal of systematically increasing their capacity to recruit, train, support, and

utilize volunteers in order to more effectively serve their clients. Since 2009, CalSERVES, which 

operates AmeriCorps VIP, has tracked the number and proportion of organizations where 

members were placed that increased their volunteer capacity. During the 2011-12 program year, 

79% of participating organizations were able to increase their capacity to work with volunteers 

by at least 20%. 

The purpose of this report is to build on the outcome information available by providing 

evidence of the value-added impact of AmeriCorps VIP, to answer the question “What difference 

did AmeriCorps VIP make in volunteer capacity?” To accomplish this, CalSERVES engaged an 

external evaluation firm, JBS International Inc., to develop and implement an evaluation that 

included a comparison of organizations participating in the VIP program with a matched group 

of organizations without AmeriCorps VIP members. This comparison allowed for the 

determination of what amount of change in volunteer capacity would be expected without any 

intervention by AmeriCorps VIP, and correspondingly, what change can be attributed to 

AmeriCorps VIP. 

Description of Intervention 

Through the Volunteer Infrastructure Project, teams of AmeriCorps members are placed in 

communities to provide infrastructure support for volunteer programs in non-profit and 

educational organizations across California. Each VIP Fellow provides 1,700 hours of service to 

build the systems necessary for organizations to structure, implement, increase and maintain 

volunteer programs. All Fellows have a recruitment goal of 100 volunteers, specifically 50 longterm volunteers (over 8 hours of service) and 50 short-term volunteers (between 2 and 8 hours of 

service). The full range of specific member activiites varies by partner site but generally includes 

working with organization staff to: 

 Develop volunteer roles/position descriptions;

 Develop and implement systems and proceedures for volunteer recruitment, screening,

placement and scheduling;

 Provide volunteer training and volunteer training curriculum development;

 Provide volunteer supervision and supervision system development;

 Develop and implement recognition and reward systems; and,

 Clarify and promote the role of volunteers within the organization.
4

Fellows are provided with 255 hours of training, as well as support and supervision through 

supervising organizations, who oversee their placement at the individual partner sites. 

Research Questions 

The goal of the evaluation of the CalSERVES VIP program is to assess the impact of 

program participation on non-profit and educational organizations’ volunteer capabilities and 

capacities. Specifically, the evaluation investigates: 

1. Does participation in the CalSERVES VIP program increase organizational capacity to

utilize volunteers compared to organizations that did not take part in the CalSERVES

VIP program?

2. Does participation in the CalSERVES VIP program improve organizational capacity

concerning volunteer recruitment, training, and retention compared to organizations that

did not take part in the CalSERVES VIP program?

3. Does participation in the CalSERVES VIP program improve organizational capacity to

create and sustain successful volunteer programs compared to organizations that did not

take part in the CalSERVES VIP program?

Description of Programs 

The AmeriCorps VIP program worked with 8 supervising organizations that funded 133 

partner sites in 2011-2012. The partner sites were located throughout the state of California, 

with 37 partner sites in northern California, 70 in central California, and 26 sites in southern 

California. In addition to the 133 partner sites that received services, data were collected on 

59 comparison sites that did not receive services to assist in organizational development and 

volunteer capacity. This group was composed of like organizations in California that were 

recruited to participate in the evaluation study by the supervising organizations.

Organizations that were included in the comparison group have never taken part in the 

AmeriCorps VIP program. 

As can be seen from the descriptive data below, without any matching, the two groups of 

programs had many similarities, although the comparison sites were generally smaller, newer 

and less well-funded. Creating a propensity score-matched subset of programs for 

comparison can help rule out concerns that the differences in volunteer capacity are due to 

the programs rather than the VIP Fellows’ service. 
5

Figure 1: Program Service Areas 

0%

10%

20%

30%

40%

50%

60%

70%

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one service area, so columns do not sum to 100%.

The types of services delivered ranged widely, but many organizations offered tutoring or 

support to K-12 students, workshops, trainings, or classes to adults with a wide range of 

information needs.

Figure 2: Types of Services Delivered 

0%

10%

20%

30%

40%

50%

60%

70%

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one service area, so columns do not sum to 100%.
6

Service delivery was primarily through one-on-one or group activities, although partner 

organizations also offered material support to clients as well.

Figure 3: Manner of Service Delivery 

0%

10%

20%

30%

40%

50%

60%

70%

80%

One On One Group Train the

trainers

Provide

Material

Support

Other

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one service area, so columns do not sum to 100%. 

Partner sites provided their services at a variety of locations, while comparison organizations 

were slightly more likely to state that they offered their services either at their program site or 

in the community. 

Figure 4: Site of Service Delivery 

0%

10%

20%

30%

40%

50%

60%

70%

80%

School Program Site People's Home Community

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one site for delivery of services, so columns do not sum to 100% 
7

While organizations often target specific groups of people for their services, it is difficult to 

group together and classify such a diverse group of programs. However, many organizations 

focused their services on youth or low-income individuals and families. 

Figure 5: Special Target Populations 

0%

5%

10%

15%

20%

25%

30%

35%

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one site for delivery of services, so columns do not sum to 100% 

Figure 6: Target Demographic Populations 

0%

5%

10%

15%

20%

25%

30%

35%

40%

Youth Seniors Race/Ethnicity

Group

Poor Near Poor

Partner Sites

Comparison Sites

*Organizations could list themselves as having more than one site for delivery of services, so columns do not sum to 100% 
8

Table 1 shows that the organizations varied in size and characteristics greatly. However, on 

average, the partner sites that participated in the CalSERVES VIP program tended to be 

larger, to be older, and to have bigger budgets. 

Table 1: Characteristics of Organizations 

Partner Sites Comparison Sites

Average Number of Staff (Total) 30 18

Average Number of Administrators 6 3

Average Number of Direct Service Staff 7 2

Average Number of Volunteers 1,300 260

Average Budget per Year $3,100,000 $1,500,000

Average Number of Years in Existence 31 24

Average Number of People Served per Year 82,000 44,000

Sampling & Data Collection 

The evaluation of the AmeriCorps VIP program examines the effect of the intervention on 

organizational development and volunteer capacity among non-profit and educational 

organizations. 

CalSERVES funds eight supervising organizations throughout the state, which, in turn, select 

partner sites to receive an AmeriCorps member. Supervising organizations provide member 

training and supervision following a core curriculum from CalSERVES. To ensure some 

degree of similarity in the groups, comparison sites were also recruited through the 

supervising organizations, sometimes through their local knowledge of the community, and 

sometimes by engaging partner sites that had applied for, but had not received, an 

AmeriCorps VIP Fellow. 

JBS and CalSERVES collected data on both volunteer capacity and organizational 

characteristics to allow for matching of participating and non-participating sites. JBS then 

used this data to create a matched comparison group.

Volunteer Capacity Assessment Instrument 

To measure the effect of program participation on organizational and volunteer capacity over 

time, data were collected using the Volunteer Capacity Assessment (VCA), a survey that 

includes three sections with items focused on: organizational capacity, volunteer recruitment, 

and elements of a successful volunteer program. The VCA survey is attached as Appendix A 

of this report. 

The VCA is designed to assess how well a non-profit or educational organization is prepared 

for recruiting, training, and utilizing volunteers to achieve its mission and goals. The 

instrument asks for organizations to report how well their practices align with the best 

practices listed on the survey. 

AmeriCorps VIP partner sites provided information on the VCA three times, once each 

during the fall, winter, and summer, generating data on organizational and volunteer capacity 

prior to taking part in the program, mid-way through taking part in the program, and at the 

end of program participation. Data were collected in August or September of 2011, 
9

December of 2011 or January of 2012, and in July of 2012 for organizations that took part in 

the program. Comparison group organizations completed the survey twice; once in April of 

2012 and again in August of 2012.While the two groups did not complete the VCA 

instruments over exactly comparable periods of time, the use of propensity score matching 

helps diminish concerns about the effects of this discrepancy. (The methods and analysis 

sections below describe how data were used and adjusted as necessary to make the time 

periods comparable.) 

Data Collection Worksheet Instrument 

In addition to completing the VCA, all partner and comparison sites completed a Data 

Collection Worksheet (DCW). The primary purpose of this survey was to provide data for 

use in propensity score matching of partner sites with comparison sites for use in the 

analysis. This worksheet was completed electronically as a MS Word document, filled in by 

hand, or completed as an online survey using SurveyMonkey.com. In some cases, missing 

data was supplemented by phone contact with respondents or by web research on the 

organization (Appendix B includes a copy of the DCW). This survey asked questions about 

staff and volunteers, organizational characteristics including: budget, years in existence, 

services provided, and populations served. The DCW was collected for both partner and 

comparison organizations in the spring of 2012. 

Methods 

To evaluate the CalSERVES VIP program, data were collected from participating program 

sites and from comparable organizations; the data were used to form a counterfactual 

comparison group. Outcome data were collected using the VCA. Data were collected in the 

fall of 2011, in the winter of 2011-2012, in the early spring of 2012, and in the summer of 

2012 for all partner sites. Comparison sites provided data in mid-spring of 2012 and in late 

summer of 2012. The comparison group was formed post hoc rather than at baseline because 

the methodology for the study and hence the comparison group sites were selected after the 

baseline data collection for participating programs. The different time points of data 

collection are adjusted for in the analysis (see below). 

The evaluation of the AmeriCorps VIP program uses propensity score matching to match 

comparison group sites with partner sites to ensure a more robust assessment of program 

impact. A fuller description of the propensity score matching is included below. 

Propensity Score Matching 

To assess the impact of the AmeriCorps VIP program, the evaluation uses a matched 

comparison group. Comparing the outcomes of two groups – one of which received services 

and one which did not – in a statistically robust manner provides evidence that program 

participation is the likely reason for any observed changes on key outcome measures rather 

than other possible causes. In an ideal research scenario, organizations could be randomly 

assigned to receive an AmeriCorps VIP Fellow or not (eliminating any differences in the 

partner organizations that was not due to pure chance). However, this was not possible due to 

program constraints. Propensity score matching is a way of simulating an experimental 

design, although it does not do so perfectly. 

In a propensity score matched design, the two groups are not formed by random assignment. 

Instead, the group that participates in the program is selected non-randomly as is the 
10

comparison group. Because of this, the two groups may, before matching, differ in both 

observed and unobserved ways that could be the explanation for any changes observed in the 

outcomes. Propensity score matching minimizes the differences between the two groups 

statistically. The statistical procedure used in propensity score matching matches program 

participant organizations with non-participant organizations based on a wide range of 

characteristics (see below, Figure 7, for the characteristics used in this evaluation). The 

match is even more robust than simply pairing participants with non-participants because it is 

based on the statistical likelihood that a program that did not participate would have 

participated based on the characteristics of all organizations (both participants and nonparticipants). Thus, each participating organization is matched with a non-participating 

organization, based on the congruent characteristics of both organizations. 

For this evaluation, data collected on the DCW were used to create a dataset of 

characteristics of all partner and comparison group sites. A review of the literature provided 

guidance concerning which characteristics were used to match the programs. Figure 7

includes all of the characteristics of the organizations that were used to match programs. In 

addition to those characteristics, the site’s VCA score from either the mid-point data 

collection (for partner sites) or from the first data collection time point (comparison group 

sites) was also included in the model. 

Figure 7: List of Characteristics Used in Propensity Score Matching 

Organization 

Characteristics

Staff/volunteers’ 

Characteristics

Program Focus

 Region of California

 If the program is housed 

within a larger organization 

(such as a university)

 If the program is part of a 

larger (regional, state, or 

national) organization

 Number of people served by 

organization per year

 Annual budget

 Total number of staff

 Total number of volunteers

 Most senior staff members’

number of years at 

organization 

 If the program is youthfocused

 If the program is seniorfocused

 If the program is povertyfocused

The propensity scores were generated in SAS using a logistic regression model. Matches 

were made to the eighth decimal point. Because the ratio of participating organizations to 

non-participating organizations was nearly two to one, comparison group members were 

matched to participating organizations more than once. The propensity score matching 

process yielded matches for 75 participating program sites and for 18 non-participating sites. 

Comparing Data Points from Partner Sites and Comparison Group Sites 

To best understand the impact of the CalSERVES VIP program on partner sites, data were 

collected at three points throughout the program year to capture organizational capacity prior 

to taking part in the CalSERVES VIP program, approximately 4 to 6 months after baseline 

data collection, and once more approximately 10 to 12 months later. This data collection 
11

strategy enables the assessment of change over time. Ideally, data would have been collected 

at the exact same time points for the comparison group sites. However, the comparison group 

sites were recruited mid-way through the program year, and data collection started 

significantly later than for partner sites. While partner sites had 10 to 12 months between the 

initial data collection and the final data collection, because of the later start with data 

collection for comparison sites, only approximately 4 months separated the two data 

collection periods. 

Because of this, the analysis presented in this report compares partner site data from the midpoint data collection period and final data collection period with the comparison group sites 

initial and final data. Comparing these two time periods is appropriate because the amount of 

time between the mid-point and the final point for the partner sites is similar to that between 

the initial and final data points for the comparison group. The two windows are also more 

similar in terms of when the data collection occurred in the calendar year (both were in the 

spring and summer, respectively). 

Differences between AmeriCorps VIP Sites and Non-participating Sites 

(Unmatched) 

The VCA instrument includes items that measure a variety of organizational and volunteer 

capacity characteristics. The VCA items all asked respondents respond on a scale of zero to 

four, indicating “none”, “little,” “some,” “much,” or “completed,” respectively. All programs 

were asked to rate how well their organization fulfills the statements on the form (in Tables 

2-4 below). The goal of the AmeriCorps VIP program is to see 20 percent, or approximately 

0.8, improvement in ratings from the first time point to the second or final time point. 

Figure 8 and Tables 2 through 4 show the differences between partner sites and comparison 

sites without matching sites between the two groups. The differences are instructive because 

they are taken from all data available from each group. However, these numbers cannot be 

used to understand the impact of the CalSERVES VIP program because the differences 

between the two groups may be due to other factors than program participation. In the section 

below on “Threats to Validity,” the main concerns are explained in greater detail, but they 

are also briefly addressed here. When two unmatched groups are compared, other factors, 

such as organization characteristics (including, but not limited to, staff characteristics, budget 

size, or affiliation with a national organization, for example), might explain why a program 

has changed over time. The main worry is that these factors, rather than program 

participation, underlie the changes seen between the two data collection points. In the next 

section, the groups are matched and compared, which will allow for assessment of program 

impact net of organizational characteristics. 

Even so, there are advantages in examining the unmatched data. The primary advantage is 

that these tables include all organizations for which data were collected. These data also 

show that even without matching the general trends remain the same, lending credibility to 

the impact illustrated in the matched data tables. 

Figure 8 shows the overall average for partner sites and comparison sites. This mean is 

calculated by taking all 23 items on the VCA instrument and averaging them. This average 

gives a high-level, general assessment of organizational capacity related to volunteers. While 

both groups showed similar levels of overall organizational capacity at the first data 
12

collection time point for the comparison group (the second time point for the treatment 

group), the organizations that took part in the CalSERVES VIP program appeared to 

experience much greater positive change over time. 

Figure 8: Average Pre/Mid VCA Score and Average Post VCA Scores, Unmatched 

Comparison 

0

0.5

1

1.5

2

2.5

3

3.5

Pre (Placement) Pre (Comparison)/Mid Post

Placement

Comparison

*Not all 133 partner sites completed both mid-point and final data collection so the total number of partner sites is 117. 

Table 2 includes the first four questions on organizational capacity on the VCA instrument. 

While the overall difference between the two groups was, on average, not very large, many 

of the individual items in this table, and in subsequent tables, illustrate the considerable 

differences. The change from the first time point to the second time point was positive and 

relatively large for the partner sites, but was very small (and sometimes negative) for the 

comparison sites. 
13

Table 2: Organizational Capacity, differences between time points 1 and 2 

Organizational Capacity Partner Difference 

between Mid and 

Post VCA (N=117)

Comparison Difference 

between Pre and Post VCA 

(N=58)

The organization has a vision for what the volunteers can do for 

the organization and for the people it serves. 0.67 -0.07

Volunteers within the organization represent the diversity within 

the community. 0.76 0.09

The organization has strategically thought about the benefits and 

challenges related to volunteer involvement within the 

organization.

0.86 -0.10

The organization has developed a written statement of philosophy 

as to why the organization welcomes volunteers. 0.96 -0.07

Table 3 presents the second section of the VCA instrument. As with the items on 

organizational capacity, partner sites that participated in the CalSERVES VIP saw greater 

changes over time compared to the comparison sites. 

Table 3: Volunteer Recruitment, differences between time points 1 and 2 

Volunteer Recruitment Partner Difference 

between Mid and 

Post VCA (N=117)

Comparison Difference 

between Pre and Post 

VCA (N=58)

Targeted recruitment efforts based on each volunteers’ job 

description instead of generic volunteer recruitment campaigns. 0.98 0.09

There are strong publicity, public relations, and marketing 

campaigns in place that have built a positive image of the 

organization within the community.

0.86 0.14

The organization has a clear understanding of why people would 

want to volunteer. 0.89 -0.02

The organization is prepared to accept applications and is 

welcoming to prospective volunteers. 0.82 -0.07

Table 4 shows the change in responses to the items on the VCA that asked about the 

characteristics of successful volunteer programs. As with the previous sections of the VCA, 

CalSERVES VIP partner sites reported far greater and overwhelming positive changes over 

time. 
14

Table 4: Elements of Successful Volunteer Program, differences between time points 1 

and 2 

Elements of Successful Volunteer Program Partner Difference 

between Mid and 

Post VCA (N=117)

Comparison Difference 

between Pre and Post 

VCA (N=58)

The organization has planned for the resources necessary to 

support volunteers. 0.78 0.02

Training and supervision resources for volunteers have been 

identified. 0.87 0.07

Job descriptions have been developed for volunteer positions. 1.00 0.10

Flexibility has been built into volunteer positions to accommodate 

different skills and schedules. 0.80 0.09

The community understands what the organization does and is

eager to support its efforts. 0.83 -0.03

There is a screening and selection process in place to aid in 

matching new volunteers to appropriate positions. 0.90 0.12

All volunteers participate in an orientation session that provides 

them with an understanding of policies, procedures, rights, and 

responsibilities.

0.96 0.10

Volunteers receive start up and ongoing training. 0.86 0.00

Positive volunteer/employee relationships are nurtured and 

problems are dealt with quickly. 0.85 -0.09

There is a clear leader within the organization that is seen as 

having the responsibility for coordinating and staffing volunteer 

programs.

0.96 0.04

Supervision is provided to all volunteers for support, 

communication, and accountability. 0.75 0.00

The work of volunteers and the impact of their activities are 

evaluated on a regular basis. 0.90 -0.09

Volunteers receive formal and informal recognition for their 

contributions. 0.85 0.03

Records are kept of what volunteers are doing, and results and 

reported and shared with the volunteers, administration, 

community, and funding sources.

0.81 0.02

The organization regularly seeks input from volunteers. 0.92 0.12

Differences between AmeriCorps VIP Sites and Non-participating Sites (PSM) 

To assess program impact, it is essential to have a counterfactual comparison that helps 

diminish the likelihood of factors other than participation in the CalSERVES VIP program 

affecting the organizations during the 12-month data collection period. In the previous 

section, all programs from which data were collected were included in the tables. In this 

section, only data from programs that were matched are presented. 

The propensity score process creates two groups of sites that are alike in many ways through 

the statistical matching procedure (see the section above for greater details regarding 

propensity score matching). Sites were matched on a variety of organizational and staff 

characteristics as well their initial VCA average score. Because of the diversity among 

partner sites and the smaller overall number of comparison group sites, not all organizations 

from which data were collected were included in the analysis using matched sites. Only 

partner and comparison sites that had very similar propensity scores were included in the 
15

matched analysis. This matching procedure yielded a set of programs with very similar 

scores on the initial Pre-test VCA (1.4 for the treatment group, and 1.7 for the comparison) 

although by the Mid-test VCA, the treatment group had already increased in their average 

score to 2.2.

Even though the total number of sites included in the tables below is smaller, the comparison 

between the two groups is stronger than the comparison between all sites in the section 

above. This strength stems from the statistical matching that diminishes the likelihood that 

any differences between the two groups in the tables are due to characteristics of the 

organizations or their staff. The PSM procedure approximates the experimental design – the 

gold standard in evaluation research when trying to understand the causal impact of a 

program – and permits assessment of program impact. The tables below show that the 

CalSERVES VIP program has a strong impact on partner sites. 

Figure 9 shows that when matched groups of partner sites and comparison group sites are 

compared, the overall average scores for the VCA are considerably higher for the partner 

sites, and increase significantly more, than for those of the comparison group sites. 

Figure 9: Average Pre/Mid VCA Score and Average Post VCA Scores, Matched 

Comparison 

0

0.5

1

1.5

2

2.5

3

3.5

Pre (Placement) Pre (Comparison)/Mid Post

Placement

Comparison

*All differences between partner sites and comparison sites are statistically significant (p=0.01). 

As was shown in the comparison of all sites, the differences between CalSERVES VIP 

partner sites and comparison sites are significant and positive. In the matched comparison 

analysis, the change over time for partner sites is larger than for the entire group of partner 

sites (in Figure 8 above). 
16

The CalSERVES VIP program appears to positively affect the partner sites’ organizational 

capacity to host volunteers. In particular, the program significantly impacts the partner sites’ 

integration of volunteers through the organizations’ strategic thinking about volunteer 

involvement benefit and challenges, and written statement of philosophy for welcoming 

volunteers. 

Table 5: Organizational Capacity, differences between time points 1 and 2* 

Organizational Capacity Partner Difference 

between Mid and 

Post VCA (N=75)

Comparison Difference 

between Pre and Post 

VCA (N=18)

The organization has a vision for what the volunteers can do for the 

organization and for the people it serves. 0.56 0.00**

Volunteers within the organization represent the diversity within the 

community. 0.75 0.17

The organization has strategically thought about the benefits and 

challenges related to volunteer involvement within the organization. 0.76 -0.06

The organization has developed a written statement of philosophy as 

to why the organization welcomes volunteers. 0.81 0.06

*All differences between partner sites and comparison sites are statistically significant (p=0.005) 

**Because of the small number of cases used in the comparison group and the small range of values, mathematically it is not unlikely that 

the average could equal 0.

The CalSERVES VIP program also has a positive and significant effect on volunteer 

recruitment as measured by the VCA instrument. Program participation appears most 

strongly to affect targeted recruitment efforts among partner sites. 

Table 6: Volunteer Recruitment, differences between time points 1 and 2* 

Volunteer Recruitment Partner Difference 

between Mid and 

Post VCA (N=75)

Comparison Difference 

between Pre and Post 

VCA (N=18)

Targeted recruitment efforts based on each volunteers’ job 

description instead of generic volunteer recruitment campaigns. 0.94 0.06

There are strong publicity, public relations, and marketing 

campaigns in place that have built a positive image of the 

organization within the community.

0.84 0.28

The organization has a clear understanding of why people would 

want to volunteer. 0.82 0.11

The organization is prepared to accept applications and is 

welcoming to prospective volunteers. 0.74 -0.12

*All differences between partner sites and comparison sites are statistically significant (p=0.05) 

While all of the organizations included in the matched comparison saw positive changes as 

measured by the elements of successful volunteer programs on the VCA, the CalSERVES 

VIP partner sites all experienced considerably larger changes over time. In particular, the 

partner sites reported the greatest change concerning the development of job descriptions for 

volunteer positions, the establishment of screening and selection processes for volunteers, 

and the organizations’ regular requests for input from volunteers. 
17

Table 7: Elements of Successful Volunteer Program, differences between time points 1 

and 2 

Elements of Successful Volunteer Program Partner Difference 

between Mid and 

Post VCA

Comparison Difference 

between Pre and Post 

VCA

The organization has planned for the resources necessary to 

support volunteers. 0.74 0.17

Training and supervision resources for volunteers have been 

identified. 0.78 0.22

Job descriptions have been developed for volunteer positions. 0.92 0.00**

Flexibility has been built into volunteer positions to accommodate 

different skills and schedules. 0.79 0.11

The community understands what the organization does and is

eager to support its efforts. 0.75 0.11

There is a screening and selection process in place to aid in 

matching new volunteers to appropriate positions. 0.92 0.11

All volunteers participate in an orientation session that provides 

them with an understanding of policies, procedures, rights, and 

responsibilities.

0.89 0.28

Volunteers receive start up and ongoing training. 0.73 0.17

Positive volunteer/employee relationships are nurtured and 

problems are dealt with quickly. 0.87 0.17

There is a clear leader within the organization that is seen as 

having the responsibility for coordinating and staffing volunteer 

programs.

0.84 0.11

Supervision is provided to all volunteers for support, 

communication, and accountability. 0.76 0.17

The work of volunteers and the impact of their activities are 

evaluated on a regular basis. 0.89 0.11

Volunteers receive formal and informal recognition for their 

contributions. 0.89 0.00**

Records are kept of what volunteers are doing, and results and 

reported and shared with the volunteers, administration, 

community, and funding sources.

0.83 0.33

The organization regularly seeks input from volunteers. 0.96 0.17

*All differences between partner sites and comparison sites are statistically significant (p=0.01) 

**Because of the small number of cases in the comparison group and the small range of values, mathematically it is not unlikely that the 

average could equal 0. 

Overall, the partner sites that participated in the CalSERVES VIP program reported strong, 

positive changes over time on the items included in the VCA instrument. Organizations that 

did not participate in the CalSERVES VIP program reported much smaller (and sometimes 

negative) changes on the VCA items. The analysis of the propensity score matched groups 

confirms the initial results presented in the tables that included all partner and comparison 

group sites. Because the matching procedure diminishes the likelihood that factors other than 

participation in the CalSERVES VIP program influenced the change in reported scores over 

time, the evidence strongly suggests that the CalSERVES VIP program positively impacts 

organizations’ capacity concerning volunteer programs.
18

Reliability and Validity of the VCA Instrument 

Reliability and validity are key to understanding evaluation research. The VCA instrument 

does not have a long history of use with information about its reliability or validity, so these 

two concerns are addressed here using the data available. 

Reliability refers to how likely it is that an instrument, or other data collection technique, 

could be used across populations, places, and time, and generate similar results given similar 

inputs. It refers to the extent that measures are repeatable and that any random influence 

which tends to make measurements different from occasion to occasion or circumstance to 

circumstance is a source of measurement error. To establish the reliability of the VCA 

instrument, test-retest reliability – or a measure of the degree to which responses are 

consistent over time – is used here. Because people who completed VCA forms for 

comparison group organizations did not (at least in theory) receive any sort of services or 

treatment during the time that they completed the VCA forms initially and subsequently, the 

data collected are suitable for test-retest reliability assessment. 

Using a weighted kappa statistic to assess the test-retest reliability of the VCA instrument, 

the results show that overall the reliability of items on the VCA are considered “good” to 

“very good” using a commonly accepted scale (see Appendix C for tables with weighted 

kappa statistics and more detailed information on the weighted kappa statistic). Weighted 

kappas for the VCA items ranged from 0.61 to 0.86. 

Validity refers to how well a particular measure captures the intended concept or idea. There 

are many dimensions to validity: content validity, face validity, predictive validity, 

concurrent validity, and construct validity. The VCA instrument has strong content validity, 

in that the items included on it are appropriate for understanding organizational capacity 

related to volunteering, and strong face validity, in that the items appear to measure what 

they purport to measure. Further, the VCA is content-aligned with the training curriculum 

that CalSERVES VIP utilizes. This curriculum and the instrument are based on materials 

from the Michigan State University Extension Services “Achieving Success Through 

Volunteers.” 

Internal and External Validity 

To assess program impact, it is essential to address concerns about internal and external 

validity of a study. Internal validity refers to certainty that any changes because of program 

participation are due solely to the program and not to other characteristics of the individuals 

or organizations, factors external to the program (such as historical events), or things that 

happen during the process of evaluating the program (e.g.- errors in measurement). External 

validity refers to the generalizability of the findings from the program to other groups of 

people or organizations in different places and at different points in time. Conventionally, 

internal threats to validity are generally said to be mitigated or minimized by a good 

evaluation design, while external validity is generally said to be strengthened in a good 

evaluation design. Tables 8 and 9 list the common threats to internal validity and 

characteristics of strong external validity. 

The evaluation of the CalSERVES VIP program attempts to minimize threats to internal 

validity in several different ways. First, data were collected before, during, and towards the 

end of program participation for partner sites. Second, a pre-existing instrument was used. 
19

Third, data were collected from a comparison group of organizations that were similar to the 

partner sites. Finally, the comparison group was matched to the partner sites group using 

propensity score matching. 

Tables 8 and 9 indicate how each of these aspects of the evaluation design worked to 

minimize particular threats to internal validity and to strengthen external validity. Overall, 

the evaluation of the CalSERVES VIP program addresses several key threats to internal 

validity by (a) collecting data over time and (b) using a propensity score matched comparison 

group. 

Table 8: Potential Internal Validity Concerns 

Threat Addressed? Reason

Selection Bias 

Propensity score matching the comparison group minimized the 

likelihood that characteristics of comparison site organizations were

responsible for the differences seen between partner sites’ and 

comparison sites’ reported changes on the VCA forms.

Selection Additive Effects 

Propensity score matching the comparison group minimized the 

likelihood that characteristics of comparison site organizations were

responsible for the differences seen between partner sites’ and 

comparison sites’ reported changes on the VCA forms.

Mortality 

By collecting data over time, mortality, or the having sites no longer 

take part in the program, was minimized.

Regression to the Mean

This threat was not addressed by this design and could not be 

adjust for because (a) the mean value for the VCA items among all 

organizations is unknown and (b) it is possible that if more data 

were collected from both partner and comparison group sites, each 

set of sites would eventually regress to the mean value.

History 

By collecting data over time and comparing similar time frames for 

both program sites and comparison site groups, history effects, or 

the likelihood that some external, large scale experience would 

affect sites differentially, was minimized.

Maturation 

By collecting data over time for both program sites and comparison 

group sites, maturation, or the likelihood that the passage of time 

might cause changes, was minimized.

Testing These threats to validity are difficult to minimize in evaluations that 

use surveys that measure change over time. These four threats to 

internal validity all center on the effects of taking part in a program

or an evaluation – that is, by simply receiving program services or 

taking part in evaluation activities an organization may do 

something differently than it would have otherwise. The evaluation 

protocol did, however use the same instrumentation for both 

groups, and both treatment and comparison programs were 

instructed to look at their pre-measure scores when completing 

their post measures.

Novelty

John Henry

Expectancy Effects

Additionally, because of the diversity of programs included in both the partner site and 

comparison site groups, the external validity of the evaluation design is strengthened. 

However, the results are likely most generalizable within the state of California; additional 

research in other states would be necessary to understand if the results would likely be 

replicated elsewhere. 
20

Table 9: External Validity Characteristics 

Concern Strengthened? Reason

Applicability to other 

populations 

Because the partner and comparison site groups

include a wide variety of programs working with 

different populations using different modalities, this 

evaluation likely has some applicability to other 

groups of programs.

Applicability to other 

settings/locations 

Because of the geographic distribution of the 

partner and comparison group sites across 

California, this evaluation likely has applicability

across the state (but more research would need to 

be conducted to determine its applicability in other 

states).

Discussion/Findings 

The evaluation of the CalSERVES VIP program’s impact on organizational capacity 

concerning volunteer operations shows that this program has a strong positive impact on 

participating sites. The CalSERVES VIP program partner sites reported positive changes on 

all measures of the VCA instrument. These changes were much larger and statistically 

significantly different from the changes that were reported by the comparison sites. The 

difference between the partner sties and the comparison group sites was evident both in the 

unmatched and in the propensity score-matched comparisons of VCA items. 

The VCA instrument used in this evaluation appears to be both reliable and valid based on 

initial analyses. The evaluation minimizes many threats to internal validity and has some 

external validity due to the diversity of programs and sites. The findings from this study are 

likely generalizable to other social service programs that use volunteers across the state of 

California, although more research would be needed to know if the findings are applicable to 

other states. 

The items that partner sites reported the largest change over time for were (with change 

between time point one and time point two for all partner sites reported in parentheses):

 Job descriptions have been developed for volunteer positions. (1.0) 

 Targeted recruitment efforts based on each volunteers’ job description instead of generic 

volunteer recruitment campaigns.(0.98) 

 The organization has developed a written statement of philosophy as to why the 

organization welcomes volunteers. (0.96) 

 All volunteers participate in an orientation session that provides them with an 

understanding of policies, procedures, rights, and responsibilities. (0.96) 

 There is a clear leader within the organization that is seen as having the responsibility for 

coordinating and staffing volunteer programs. (0.96) 

 The organization regularly seeks input from volunteers. (0.92) 

 There is a screening and selection process in place to aid in matching new volunteers to 

appropriate positions. (0.90) 
21

 The work of volunteers and the impact of their activities are evaluated on a regular basis. 

(0.90) 

These items center on ways in which partner sites improved how volunteers join and become 

part of the organization. The CalSERVES VIP program appears to impact these types of 

actions or policies in partner sites to the largest extent. 

 The organization has a vision for what the volunteers can do for the organization and for 

the people it serves. (0.67) 

This lowest recorded change (which, at 0.67 is still a significant and positive change) may 

indicate that CalSERVES VIP may wish to offer more guidance to partner sties regarding 

establishing a vision for what organizations might do with volunteers. 

Conclusion 

While there are a number of program practices that could contribute to these significant 

evaluation findings, CalSERVES VIP’s utilization of research-based best practices likely 

plays an important role. In terms of professional development, the CalSERVES VIP program 

places a significant focus on growing AmeriCorps Fellows’ professional and capacity 

building skills. To achieve this, the program uses a diverse set of methods. 

AmeriCorps Fellows attend a statewide training conference at the beginning of their service 

term. This conference includes relevant workshops led by trainers who are experts in the 

field of volunteer program management. Additionally, fellows are given the opportunity to 

learn from and network with their peers from around the state. While at their partner sites, 

AmeriCorps Fellows attend biweekly team trainings with the other members serving with 

their supervising organization, which gives them the opportunity to develop and receive 

support on an ongoing basis. Unique to the CalSERVES VIP program, online training is 

utilized to efficiently disseminate research-based practices in volunteer program capacity 

building. Finally, the CalSERVES VIP supervisors at each supervising organization receive 

training on how to best support and grow their AmeriCorps Fellows as well as how to select, 

maintain, and provide support for their partner sites. By providing this high-quality 

professional development through a diversity of mediums, CalSERVES VIP effectively 

increases the likelihood of success for AmeriCorps members and the program in building 

volunteer capacity at their partner organizations. 

In addition to utilizing best practices in AmeriCorps member development, CalSERVES VIP 

also incorporates many research-based volunteer management best practices proven to be 

successful in other programs. From targeted recruitment campaigns to volunteer evaluations, 

CalSERVES VIP uses models for building volunteer infrastructure at partner sites that have 

track records of success across the United States. These models and methods increase the 

likelihood that the partner sites will report growth and success in their volunteer program 

capacity building efforts. 

An AmeriCorps Fellow, reporting back to her supervising organization and to CalSERVES, 

provides a useful illustration of how the training and best practices come together to build 

volunteer capacity. She notes: 
22

"I accomplished two of the chief goals my Partner Site set for me over the last few months. 

The first was to create a volunteer tracking system that could be used campus wide. A few 

months ago, I was invited to be a member of the software development task force and we 

have selected a software company that will track our volunteers across departments and 

throughout the campus. More significantly, I learned about co-curricular transcripts at the 

last AmeriCorps regional training, and plan to incorporate that knowledge into the software 

that is being developed. The other big accomplishment this quarter was the creation of a cocurricular community service position. This week, I learned that the University is ready to 

hire its first Community Service Center Coordinator. This position came about due to my 

service as a VIP. I am so excited to see that my work to help build the program’s 

sustainability is taking root!" 

In review, the evaluation of the CalSERVES VIP program indicates that the program impacts 

the organizations in which Fellows serve. The CalSERVES VIP program has a measurable 

impact on capacity related to the organizations’ abilities to prepare for volunteers, to recruit 

volunteers, to train volunteers, and to establish positive on-going relationships with 

volunteers. The evaluation further demonstrates that these impacts are likely caused by the 

program as shown through the comparison of program participating partner sites with nonparticipating comparison sites matched by propensity score. 

While the current evaluation uses a counterfactual design to assess impact, future evaluations 

could greatly strengthen the evidence of program efficacy through a more robust research 

design. Several aspects of the evaluation could be improved. Selection of comparison groups 

at baseline would improve baseline comparability between the comparison group and the 

partner sites that received program services. On that same note, larger comparison groups 

would increase the certainty in the results by providing a stronger counterfactual. 

Additionally, it would be useful to develop a stronger set of measures developed more 

closely in line with the actual practices of the program and reflect programmatic activities 

more strongly, including more direct/observational measurement of treatment and 

comparison group volunteer capacity (e.g. numbers of volunteers, examination of volunteer 

job descriptions, recruitment materials and recognition programs), rather than program self 

report. These steps would assist in ensuring that future evaluation efforts would produce 

higher-level evidence to continue expanding the knowledge base of programs designed to 

develop volunteer program organizational capacity. 
23

Appendix A: VCA Instrument 
24

Appendix B: DCA Form 

AmeriCorps VIP Data Collection Worksheet 

General Information

Organization Name

Address

If you are housed in another larger 

organization, such as a university or 

non-profit, please list:

Staffing and Budget

Number of staff and volunteers Staff: Volunteers:

Background and characteristics of 

staff; enter information below for 

up to ten key staff members 

(include: name; position; number of 

years at organization; number years 

working in the field; and highest 

level of education. Use the back of 

1.
25

the form as necessary.)

2.

3.

Annual budget (in dollars)

Source(s) of funding
26

General Organization Information

Service Areas (i.e. what issues does the 

organization address or deal with? For 

example, childhood literacy, housing, 

substance use, youth development, etc.)

Services (i.e. how does the organization 

actually work with people? What are the 

types of activities that are done in the 

organization?)

How are organization services delivered?
27

Where are organization services 

delivered? (i.e. at the office, at a local 

school, community center, people’s 

homes, etc.)

Who delivers the organization’s services? 

(i.e. staff, volunteers?)
28

Organization History

How long has the organization existed?

Has the organization always been at this site? If not, when did it 

move here?

Is this part of a larger organization? If so, please describe (i.e. is 

it part of a university center, a larger non-profit, etc.? Is it an offshoot of another organization?)

Organization Participants

Number of people served annually

Geographic catchment area for people served (i.e., how far away 

do participants come from to take part?)
29

Demographic characteristics of people served: (race/ethnicity, age, gender, income levels)

Race/ethnicity (circle all that apply)

Caucasian 

African American 

Hispanic or Latino 

Asian 

American Indian or Alaska Native

Native Hawaiian or Other Pacific Islander

Multi-racial/Ethnic 

Other (specify below) 

____________________________________________

____________________________________________

Gender (circle all that apply)

Female

Male

Age (circle all that apply)

Under 18

18-24

25-34

35-44

45-54

55-64

65 and over

Income levels (circle all that 

apply)

Below $10,000

$10,001 to $20,000

$20,001 to $30,000

$30,001 to $40,000

$40,001 to $50,000

Above $50,001
30

Organization Participants (continued)

Describe any other demographic or personal characteristics 

the organization’s services are specifically targeted at

Other characteristics of population served (i.e. anything else 

about participants that the program targets?)

Organization Efficacy

Has there been an evaluation of the program? If so, when, and 

by whom?
31

What did the evaluation show?

Thank you! 
32

Appendix C: Test-Retest Reliability of the VCA Instrument 

To assess test-retest reliability, a weighted kappa statistic is used.1 The kappa statistic is 

calculated based on the distribution of responses of a given item across two (or more) points 

in time. The VCA items that were completed by comparison sites are used here. Table C.1 

provides a guideline for assessing weighted kappa values. 

Table C.1: Kappa Values 

Value of K Strength of agreement

< 0.20 Poor

0.21 - 0.40 Fair

0.41 - 0.60 Moderate

0.61 - 0.80 Good

0.81 - 1.00 Very good

(Adapted from Altman 1991 – reference below) 

Tables C.2-C.4 present the weighted kappa values for all items on the VCA. The weighted 

kappa values range from 0.61 to 0.86, or from good to very good, indicating that the items on 

the VCA have test-retest validity. 

Table C.2: Organizational Capacity, Test-Retest Reliability 

Organizational Capacity Weighted Kappa

The organization has a vision for what the volunteers 

can do for the organization and for the people it 

serves.

0.74 (p <0.0001)

Volunteers within the organization represent the 

diversity within the community.

0.76 (p <0.0001)

The organization has strategically thought about the 

benefits and challenges related to volunteer 

involvement within the organization.

0.77 (p <0.0001)

The organization has developed a written statement of 

philosophy as to why the organization welcomes 

volunteers.

0.73 (p <0.0001)

Table C.3: Volunteer Recruitment, Test-Retest Reliability 

Volunteer Recruitment Weighted Kappa

Targeted recruitment efforts based on each volunteers’ 

job description instead of generic volunteer 

recruitment campaigns.

0.65 (p <0.0001)

There are strong publicity, public relations, and 

marketing campaigns in place that have built a positive 

image of the organization within the community.

0.66 (p <0.0001)

The organization has a clear understanding of why 0.72 (p<0.0001)

 

1 Altman, DG. (1991). Practical Statistics for Medical Research, Chapman & Hall, London. 
33

people would want to volunteer.

The organization is prepared to accept applications 

and is welcoming to prospective volunteers.

0.78 (p <0.0001)

Table C.4: Elements of Successful Volunteer Program, differences between time points 1 and 

2

Elements of Successful Volunteer Program Weighted Kappa

The organization has planned for the resources 

necessary to support volunteers.

0.71 (p <0.0001)

Training and supervision resources for volunteers have 

been identified.

0.73 (p <0.0001)

Job descriptions have been developed for volunteer 

positions.

0.75 (p <0.0001)

Flexibility has been built into volunteer positions to 

accommodate different skills and schedules.

0.86 (p <0.0001)

The community understands what the organization 

does and is eager to support its efforts.

0.61 (p <0.0001)

There is a screening and selection process in place to 

aid in matching new volunteers to appropriate 

positions.

0.72 (p <0.0001)

All volunteers participate in an orientation session that 

provides them with an understanding of policies, 

procedures, rights, and responsibilities.

0.62 (p <0.0001)

Volunteers receive start up and ongoing training. 0.72 (p <0.0001)

Positive volunteer/employee relationships are nurtured 

and problems are dealt with quickly.

0.75 (p <0.0001)

There is a clear leader within the organization that is 

seen as having the responsibility for coordinating and 

staffing volunteer programs.

0.74 (p <0.0001)

Supervision is provided to all volunteers for support, 

communication, and accountability.

0.67 (p <0.0001)

The work of volunteers and the impact of their 

activities are evaluated on a regular basis.

0.62 (p <0.0001)

Volunteers receive formal and informal recognition for 

their contributions.

0.71 (p <0.0001)

Records are kept of what volunteers are doing, and 

results and reported and shared with the volunteers, 

administration, community, and funding sources.

0.64 (p <0.0001)

The organization regularly seeks input from 

volunteers.

0.71 (p <0.0001)