REVISED Evaluation Report

Appalachian Forest Heritage Area AmeriCorps:

Enhancing Assets to Benefit Communities

Building Volunteer Management Capacity

For Community Organizations

Appalachian Forest Heritage Area 

Phyllis Baxter, Executive Director 

Logan Smith, Deputy Director

External Evaluator

Maryann Durland, Ph.D 

Durland Consulting

December 2020

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building i 
Table of Contents

Executive Summary..................................................................................................................................................... 1 

Introduction ................................................................................................................................................................... 2 

Program Background and Problem Definition................................................................................................. 2 

Overview of Prior Research.............................................................................................................................. 3 

Program Theory of Change and Outcome of Interest .................................................................................... 6 

AFNHA AmeriCorps Volunteer Capacity Theory of Change ............................................................................. 6 

Evaluation Overview................................................................................................................................................... 8 

Evaluation Scope and Purpose......................................................................................................................... 8 

Evaluation Background..................................................................................................................................... 8 

Evaluation Research Questions........................................................................................................................ 9 

Evaluation Design............................................................................................................................................. 9 

Data Collection................................................................................................................................................. 9 

Data Analysis................................................................................................................................................. 12

Miscellaneous Data and Reporting Notes......................................................................................................14

Evaluation Findings...................................................................................................................................................15

Conclusions...................................................................................................................................................................21

Recommendations .....................................................................................................................................................22

Future Evaluation Opportunities.........................................................................................................................23

Appendix A: Volunteer Management Capacity Survey................................................................................24

Appendix B: References...........................................................................................................................................31

Appendix C: Evaluation Team ...............................................................................................................................33

Dr. Maryann Durland Ph.D – Senior Evaluator, Study Analysis..................................................................... 33

AFNHA Staff................................................................................................................................................... 33

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building ii
Executive Summary

Appalachian Forest National Heritage Area AmeriCorps (AFNHA): Enhancing Assets to 

Benefit Communities, a multi-focus intermediary program which provides direct member 

services related to environmental stewardship and community improvement goals, also 

provides volunteer management and capacity building for the organizations where members 

serve. The 2017-2020 AFNHA external impact evaluation aimed to assess the effectiveness of 

our volunteer management program interventions, primarily using a volunteer capacity survey 

completed by site supervisors of program sites, and by representatives of comparison sites of 

similar size and purpose but who did not have AmeriCorps members assigned.

Results from the survey and member performance data found AmeriCorps members were 

useful in managing volunteers for their sites, but their impact on the long-term capacity of the 

sites was less clear. Results were indicative, but not statistically significant. Large agency 

conservation sites with AmeriCorps members providing volunteer management showed 

somewhat more frequent use of volunteer management practices among large AFNHA 

organizations than the comparison group in 2017-18, but less so in 2018-19. Small non-profit 

community sites, who generally had lower capacity to start with, did not show stronger capacity 

than comparison sites. For the small program sites where AmeriCorps members had significant 

responsibility for volunteer management, they rated higher in 4 of the 6 practices studied than 

the comparison sites.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 1 
Introduction

Appalachian Forest National Heritage Area AmeriCorps (AFNHA): Enhancing Assets to 

Benefit Communities is a multi-focus intermediary program which provides direct member 

service related to environmental and cultural heritage stewardship. Appalachian Forest 

Heritage Area (AFHA) AmeriCorps has been serving in 16 West Virginia counties since 2007. 

Now designated as Appalachian Forest National Heritage Area (AFNHA), our AmeriCorps 

members serve in a variety of positions at partner sites, providing environmental stewardship 

through lands and assets improved, stewardship education and outreach, and community 

capacity building. Members serve directly with assigned sites, in a variety of locations across 

our counties. Members with conservation background serve with federal or state agencies or 

large non-profit organizations to conserve natural resources through ecosystem restoration 

and land improvement, habitat monitoring, and environmental education, awareness, and 

outreach. Members with cultural heritage background serve with small non-profits or local 

agencies including historical societies and museums, community groups, and arts organizations, 

to conserve cultural heritage assets including local history, arts, and community environment 

for rural communities including education to encourage stewardship of these resources. In 

addition, members also provide indirect service activities to improve volunteer management 

and build capacity for the organizations they serve.

Program Background and Problem Definition

Appalachian Forest National Heritage Area (AFNHA) AmeriCorps’ Enhancing Assets to 

Benefit Communities multi-focus intermediary program that has been providing service 

through West Virginia since 2007. Members are full-time and generally placed individually 

with sub-sponsor organizations.

Members provide direct services related to environmental stewardship and community 

Improvement. In these direct service activities, conservation members provide environmental 

stewardship services to protect and improve ecosystems in the central Appalachian forest of 

our area of West Virginia including hands-on improvements and monitoring for public forest 

lands, as well as improving community environments through public parks improvements. 

Members provide environmental education, outreach and raising stewardship awareness. 

AFNHA community heritage members serve struggling rural Appalachian communities 

throughout our rural service area. Members help communities move forward with community 

improvement goals, historic preservation of public historic buildings, enhancing and 

interpreting local cultural and heritage assets, and providing education to increase 

stewardship of community heritage.

In addition to their direct service activities, another role of AFNHA AmeriCorps members is 

to leverage community volunteers and build capacity for organizations and programs in small, 

rural communities. Members are providing service to improve volunteer management 

practices, build volunteer capacity, and improve organizational effectiveness. Volunteer 

Management is the common service activity across all our sites, with approximately 30% of

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 2 
member service related to volunteer management, although the degree of emphasis varies 

from site to site. The purpose of this study is to assess the impact of these capacity building 

services by AmeriCorps members on the volunteer management practices of the 

organizations where they serve.

(Note: AFNHA was designated as a National Heritage Area in 2019. Prior to that time, 

the organizational name was Appalachian Forest Heritage Area (AFHA). Both names were 

used during the period covered by the evaluation. However, to avoid confusion, AFNHA 

will be used throughout the report.)

Overview of Prior Research

The importance of volunteering was underscored by a seminal 1998 study on volunteerism, 

(UPS 1998, p1) which stated “America is a nation of volunteers. And not-for-profit organizations 

have an unending need for the time and talents of volunteers. Organizations that are able to 

recruit and make effective use of volunteer resources are positioned to make the greatest 

contributions to those who need help.”

Across the United States, 25.3% of residents volunteer, and 36.29% participate in groups or 

organizations. Even more, 62.5%, engage in “informal volunteering” such as doing favors for 

neighbors. (Volunteering, 2015) A study of national charities found that over 80% of charities 

with receipts over $25,000 use volunteers 2010. (Urban, Charities p 4) This rate will be even 

higher for smaller organizations with little or no paid staff, such as are prevalent in rural West 

Virginia.

In a study of “Volunteer Management Practices during Challenging Economic Times” 

(Knepper, p1) the authors summarized “Nonprofit organizations rely upon volunteers to 

facilitate their missions of meeting critical community needs.” When non-profit organizations 

are stressed by federal funding cuts and reduced donor support, “nonprofits will increasingly 

be compelled to cope in new ways to achieve their missions.”

Management of volunteers is an ongoing challenge for most organizations. As explained in 

“Why is Volunteer Management Important?” (Idealist.org) “Without a volunteer management 

professional on staff – whether they be paid or volunteers themselves – volunteers often fall 

between the cracks of already busy organizations and are left to fend for themselves or 

forgotten altogether. In either case, the end result is a volunteer who doesn't feel valued or 

engaged. And that is a volunteer who will leave, taking with them the energy and time they 

could have contributed to the mission as well as, potentially, ill will towards the organization for 

how they were treated (think of it as negative PR).” The UPS study recognized that “managing 

volunteers effectively is a problem for many not-for-profit organizations. UPS recognizes that 

these organizations could accomplish even more if they could better recruit, manage and 

recognize the work of volunteers. These organizations often lack the resources to put systems in 

place that will allow them to get the most out of volunteers and retain them as committed 

workers.” (UPS 1998, p1) Their research documented this need in asking why volunteers

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 3 
stopped helping at an organization. While some reasons were personal and not related to the 

organization, 40% cited a shortcoming of the organization volunteer management practices. 

“The organization is not well managed” is the most popular specific reason given for stopping 

work as a volunteer, mentioned by 65% of those who stopped volunteering for poor volunteer 

management reasons. The other most cited reasons were “The organization did not make good 

use of time” 57%, “the organization did not make good use of talents, skills or expertise” 45%, 

and “volunteer tasks were not clearly defined” 40%. 23% stopped volunteering because they 

“were not thanked and recognized for their efforts.” (UPS 1998, pp 14-15)

Volunteer management includes a complex range of activities, such as “recruiting, 

coordinating, leading, supporting, administering and organizing volunteers as well as strategic 

oversight and management of volunteer programs.” (Knepper p3) For the purposes of this 

study, the researchers defined volunteer management practices as including “regular 

supervision and communication with volunteers, liability coverage or insurance protection for 

volunteers, regular collection of information on volunteer numbers and hours, screening and 

matching of volunteers to assignments, written policies and job description for volunteers, 

recognition activities, annual measurement of impacts of volunteers, training and professional 

development opportunities for volunteers, and training for paid staff working with volunteers; 

and the utilization of a volunteer coordinator, including a paid staff person who can spend a 

substantial portion of their time on volunteer management duties.” (Knepper p 5) The Urban 

Institute Study on Volunteer Practices found that those practices that focused more directly on 

enriching the volunteer experience were more helpful for volunteer recruitment and 

retention, while organizational practices like tracking or policies provide other benefits such as 

generating resources. (Urban, Vol Practices, p 1) This study also discusses the need for 

practices outside of the more traditional “volunteer management practices” that “support 

volunteer involvement, cultivate an organizational climate that is welcoming to volunteers, 

give their volunteers an experience worth sharing, and enlist volunteers in recruiting other 

volunteers one-on-one.” (Urban, Vol Practices, p 12)

AFNHA AmeriCorps is taking a broad definition of “volunteer management practices” to 

include the traditional best practice recommendations, as well as additional support for 

volunteer involvement. In addition to recognized best practices, AFNHA members are trained 

with tools such as those from the Rural Volunteerism Initiative that are based on experiences 

with small, rural organizations. (ruralvolunteer.org) Members also learn from the real life 

experiences of the organizations in their communities and share their challenges and successes 

with other members of their AmeriCorps team. We consider this broad range of support to 

contribute to volunteer management for their organizations.

AmeriCorps members can be effective in volunteer management and increasing volunteer 

capacity. “The most popular capacity-building option among . . . charities . . . is the addition of a 

one-year, full-time volunteer with a living stipend (like an AmeriCorps member), with 

responsibility for volunteer recruitment and management.” (Urban, Charities p 5) “AmeriCorps

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 4 
was designed to work from the bottom up, (with) groups meeting locally-determined needs in 

locally-determined ways. AmeriCorps Members help expand these groups’ reach and impact”, 

states the 2010 Transforming Communities through Service Report. They continue to say, “A 

very critical element of this approach is the design of projects that purposely engages the 

talents and abilities of community volunteers so they can also contribute to the overall success 

of these endeavors”. This report quotes President Barack Obama on the signing of the Serve 

America Act ““Programs like these are a force multiplier; they leverage small numbers of 

members into thousands of volunteers.” (Transforming, 2010, p iv)

AmeriCorps programs are placing members (stipended volunteers) in communities across 

the United States in an effort to help organizations build capacity, especially through volunteer 

management. Volunteer Maryland (volunteer.maryland.gov) is an AmeriCorps program which 

places full-time AmeriCorps members as Volunteer Coordinators with community sites. In the 

2012-2013 service year they reported:

• “96 percent of Service Sites reported that the organization does a better job of 

managing and supervising volunteers as a result of the Volunteer Maryland partnership.

• 92 percent of Service Sites reported that volunteers have enhanced the effectiveness of 

the organization as a result of the Volunteer Maryland partnership.

• 87 percent of Service Sites achieved some or all of their measurable objectives 

for volunteer impact.”

CalSERVES Volunteer Infrastructure program places members in community organizations 

across California to systematically increase their capacity to recruit, train, support, and utilize 

volunteers in order to be more effective. In this program’s quasi-experimental impact 

evaluation, sites with VIP members were compared to comparison sites of like organizations, 

with propensity score matching between the two groups. Data was collected by a Volunteer 

Capacity Assessment pre-post survey completed by all sites at least two times with a six month 

interval. The study concluded that “CalSERVES VIP program appears to positively affect the 

partner sites’ organizational capacity to host volunteers” compared to the control 

organizations. (CalSERVES 2012)

Methodology of using an assessment survey and matched control groups can be adapted 

from the CalSERVES study for use in evaluating the AFHA program. While AFNHA AmeriCorps 

members provide volunteer management service for only about 1/3 of their service time, 

rather than as their primary role in the Volunteer Maryland or CalSERVES program, we will 

assess evaluation of these member impacts to see if they still show improvement in the 

volunteer management capacity and effectiveness of the organizations they serve, showing 

improvement of capacity due to AFNHA AmeriCorps member assistance, and showing stronger 

capacity than for the control organizations without AmeriCorps assistance. Additionally, we 

will aim to compare to a matched set of control organizations.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 5 
Increased volunteer capacity can lead to more effective organizations, reporting better 

results and outcomes. “(Many) organizations lack the capacity or the expertise to measure 

volunteer impact. . . . . The most common measurement of volunteer involvement was the 

number of hours worked and the monetary value of those hours. Reporting only the numbers 

of volunteers and hours engaged does not say anything about volunteer impact on the 

organization’s mission or its clients.” (United Way King Co p 6) One study that did look at the 

results of volunteering on organizations, determined that “Of the charities surveyed, more than 

60% reported significant benefits, and more than 90% reported moderate benefits, in the 

following categories: Increases in the quality of services or programs you provide; Cost savings 

to your organization; increased public support for your programs, or improved community 

relations; Services or levels of service you otherwise could not provide.” (Urban, Charities p 14) 

In addition, many reported benefits of “access to specialized skills possessed by volunteers.” 

This study also found that smaller charities received higher levels of benefits from volunteers.

AFNHA AmeriCorps volunteer management support is intended to build volunteer capacity 

resulting in better outcomes and increased effectiveness for the rural community organizations 

which our members serve. The AFNHA AmeriCorps program creates real and lasting differences 

to our communities and sites. Rigorous evaluation of volunteer practices data will inform our 

decisions going forward as to how AFNHA AmeriCorps members can be most effectively 

deployed to support volunteers and the organizations they serve.

Program Theory of Change and Outcome of Interest

AFNHA AmeriCorps Volunteer Capacity Theory of Change

Need: AFNHA area rural communities are economically depressed and lack resources and 

capacity to effect meaningful change. Rural community and environmental organizations have 

inadequate professional staff to accomplish development goals and are often dependent on 

volunteer assistance to deliver program services. Yet they lack staff and capacity to effectively 

manage volunteers to accomplish needed projects.

This need for volunteers for non-profit organizations is as important in rural West Virginia 

as it is nationally. In our area, small communities and organizations often have little or no staff, 

and rely heavily on volunteers to accomplish their mission. While the 2014 formal volunteer 

rate in West Virginia was lower than national average at 20.8%, those who participate in 

organizations was almost even at 34.4%, and informal volunteering was significantly higher at 

74.4% (Volunteering, 2015). Some factors that seem correlated with lower levels of 

volunteerism, such as higher poverty and lower educational levels might help explain the 

lower volunteering rates in West Virginia. Yet some factors that correlate with higher 

volunteering, such as commitment to the community and large numbers of non-profit 

organizations (Volunteering Issue Brief, 2010), are prevalent in West Virginia. These factors, 

added to the high rate of informal volunteering, could indicate that West Virginians offer the 

potential for significantly more volunteer activity than is currently being utilized.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 6 
Volunteerism is essential for mission delivery for many small non-profit organizations. 

Management of volunteers is an ongoing challenge, as it takes time and effort to recruit, 

manage, and coordinate volunteer service. AmeriCorps members can be effective in providing 

this volunteer management and building volunteer capacity for their organizations. Increased 

volunteer capacity can lead to more effective organizations, giving better results and outcomes.

Interventions: Full-time AFNHA AmeriCorps members provide volunteer support and 

capacity building for a portion of their time for the sites where they are assigned for direct 

service. These include programs of large environmental and land management agencies and 

organizations, and small community organizations including museums, historical and cultural 

sites, and community improvement groups. While members provide valuable direct service for 

about 2/3 of their time implementing environmental and community projects, they also show 

significant impact by improving volunteer capacity to increase effectiveness of their organization 

to accomplish program goals, accounting for about 1/3 of member time. Members are trained in 

volunteer management by the program, with curriculum including stages of volunteer program 

development and steps of volunteer management developed from “Volunteering Reinvented: 

Human Capital Solutions for the Nonprofit Sector” (Corporation for National and Community 

Service, 2007), and a variety of other research and experiential sources. Members implement 

volunteer management best practices that support at least one of these eight functions for 

effectively managing volunteers Our training and interventions are based on these 

recommended functions and they provide the outline for the survey questions:

1. Market Research and Community Needs Assessments

2. Strategic Planning to Maximize Volunteer Impact

3. Recruiting and Marketing to Prospective Volunteers

4. Interviewing, Screening, and Selecting Volunteers

5. Orienting and Training Volunteers

6. Ongoing Supervision and Management

7. Recognition and Volunteer Development

8. Measuring Outcomes and Evaluating the Process

Members use that training to develop volunteer programs, improve volunteer management 

practices, recruit and manage volunteers, and help assess & build volunteer capacity for the 

organizations. Activities needed by sites vary depending on the organizational capacity and the 

development level of their volunteer program. For some sites members help with assessment 

and program policy development, while others focus on recruitment, selection and training of 

volunteers. For sites that focus on one-day projects and events, members help organize 

volunteer tasks, recruit volunteers, and manage them during the activity. For some sites with 

ongoing volunteers, members provide support for these volunteers to help their volunteer 

service be more effective. Members may have a role in thanking and tracking volunteers. AFNHA 

AmeriCorps members and the volunteers they manage conduct activities that support

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 7 
planning, design, and implementation of projects identified by each conservation or 

community organization as contributing toward that organization’s mission.

Outcomes: To the extent that members manage and support volunteers in these activities, 

improve volunteer management, and help to improve organizational volunteer policies and 

practice, then those members are providing volunteer capacity building service. In general, 

implementing these activities is theorized to result in improved capacity for the organization to 

recruit and use volunteers and increased effectiveness for the organization in program delivery. 

This leads to stronger organizations with increased efficiency, effectiveness, and program reach 

in delivering their mission objectives.

Evaluation Overview

Evaluation Scope and Purpose

The purpose of the 2017-20 external impact evaluation study was to assess the 

effectiveness of the volunteer management capacity building services provided by AFNHA 

AmeriCorps members for their site. Since our program is multi-focused and direct service 

activities vary from site to site, the part-time capacity building services were selected as the 

focus for the evaluation as these activities were more consistently provided across all of our 

service organizations.

In addition to meeting the external impact evaluation requirements for a program 

receiving over $500,000 in federal funds annually, the results this evaluation have been 

used to assess the consistency and effectiveness with which our AmeriCorps members are 

implementing volunteer management assistance across our service organizations and 

provide us with information for continuous improvement.

Evaluation Background

The 2017-20 impact evaluation built upon a pilot survey of volunteer management capacity 

conducted by an external evaluator for AFNHA organizations hosting the 26 members serving 

2015-16 (September to August) during the last 3-year grant cycle and a group of comparable 

organizations that had had no AmeriCorps member. Overall, the previous study found some 

differences between program organizations with volunteer management support provided by 

AFNHA AmeriCorps members and comparison organizations without AmeriCorps member 

assistance. Significant differences were found in six areas related to volunteer capacity of the 

organization. One set of questions related to improvement in volunteer capacity over time, and 

though not statistically significant, results were positive and indicative of potential effectiveness 

of AFNHA AmeriCorps member activities.

This 2017-20 cycle built on that first effort with a more extensive survey over multiple years. 

The first report from that data, submitted in 2019, compared all program sites as a group with 

all comparison sites, over the full range of questions. This REVISED report contains the results of 

Phase II data analyses conducted based on the Volunteer Management Capacity Survey data

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 8 
collected for 2017-18 and 2018-19. The new analyses were undertaken to see if utilization of 

survey data only for practices most applicable to AmeriCorps member service, and analyzing by 

large or small site types, would provide any new, more accurate insights to the effectiveness of 

member engagement in volunteer management that could be used for continuous program 

improvement.

Evaluation Research Questions

The 2017-2020 external impact evaluation was originally designed to answer the following two

research questions.

1. To what extent do community organizations with volunteer management support 

by AFNHA AmeriCorps members show differences in volunteer capacity compared 

with organizations that have not had assistance from AmeriCorps members?

2. To what extent do community organizations with volunteer management support by 

AFNHA AmeriCorps members show improvements in volunteer capacity over time?

The re-analysis of the survey data presented in this report ONLY focused on the first question 

due to small sample sizes and inconsistency in organization participation across multiple years. 

Therefore, there was insufficient high quality data available to adequately answer the second 

question.

Evaluation Design

The evaluation for the 2017-2020 grant cycle was an external impact evaluation using a

quasi-experimental design.

The AFNHA evaluation design built upon the quasi-experimental impact evaluation 

conducted by CalSERVES AmeriCorps Volunteer Infrastructure Program , which found 

significantly stronger volunteer capacity in organizations served by AmeriCorps VIP 

members than in comparison organizations without such service.

1

Data Collection

The primary sources of data used in the evaluation was the Volunteer Management 

Capacity Survey. The Volunteer Capacity Survey was designed using questions developed for 

the CalSERVES study and adapted for use to fit AFNHA parameters and focus. The survey 

consisted of 45 questions focused on volunteer management practices which are intended to 

build volunteer capacity resulting in better outcomes and increased effectiveness for the 

organizations. There were multiple sets of questions on the survey including demographic 

and matching information (use of and types of service members (stipend volunteers), use of 

volunteers (non-stipend volunteers), and hours service members and volunteers served.)

1 AmeriCorps VIP Volunteer Capacity Study, JBS International, Inc, Nov 2012 ,

www.nationalservice.gov/organizations/default/files/ evidenceexchange/FR_CaliforniaVolunteers_CalSERVESNCOE_AmeriCorpsVIP.pdf)

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 9 
Volunteer management practices questions were grouped according to the AmeriCorps 

recommended functions for effectively managing volunteers, with multiple questions 

asked for each function.

The Survey was administered by the external evaluator using a digital survey in the 

fall in 2017-19 and 2018-19 to both AFNHA organizations and comparison organizations. 

All AFNHA organizations (ranging from 24 to 33 depending on year) and approximately 

25 total comparison groups were invited via email to participate in the survey. 

The evaluation sample included organizations hosting AFNHA AmeriCorps members to 

provide volunteer assistance and a comparison group of organizations similar in type to 

the AFNHA organizations that had not had assistance from AmeriCorps members. 

Comparison Organizations are matched with program organizations, as closely as possible 

to the following characteristics:

1. Type of organization (government agency, non-profit, etc)

2. Mission-focus of organization

3. Size of organization site (generally based on staff size)

The list of organizations for the comparison groups was constructed from references from 

AFNHA organizations, online searches, and general knowledge. The organizations were selected 

to be as similar to the ones with AmeriCorps members as possible. The list expanded or 

contracted as we updated emails and utilized other data verification and cleaning processes.

The external evaluator was responsible for survey development, survey administration, 

data cleaning and storage, statistical analyses, report findings, conclusions, and the draft 

and final reports. Program staff helped with identification of survey questions, identification 

of program and comparison organizations, providing program performance measure data, 

and input on report draft.

Overview of Survey Respondents

A total of 39 organizations completed a survey at least once during the two years of 

the study: 25 AFNHA program organizations and 14 comparison organizations. Program 

organizations with AFNHA assigned members studied in the volunteer capacity survey 

evaluation included:

• 7 environmental stewardship conservation organizations, most of which were 

distinct program organizations within a larger federal agency or were field offices 

of large national non-profits. These were large, conservation organizations 

included in the evaluation.

• 18 community heritage organizations which were mostly small non-profit 

organizations with only one or a few staff, or all volunteer run. These included 

museums, historical and cultural groups, and community improvement organizations. 

These were the small, community organizations included in the evaluation.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 10
Comparison organizations were selected to be similar to program organizations, but 

without having had significant AmeriCorps service. Similarities were confirmed as measured on 

several demographic variables including organizational type, site description which asked about 

the specific organizational structure of a site such as a branch or a larger organization, or a nonprofit. Comparisons were also selected based on program emphasis and on the number of full 

and part-time paid staff and how many hours they worked on average.

Comparison organizations, which did not have significant AmeriCorps or national 

service members, included:

• 5 large conservation comparison organizations, selected as equivalent organizations 

in West Virginia or adjoining states

• 9 small community comparison organizations selected from similar organizations in 

West Virginia, of similar size and mission to the AFNHA site communities.

Table 1 provides a summary of the organizations completing the Volunteer 

Management Capacity Survey each year.

Table 1. Summary of Survey Respondents by Group, Size, and Year

Group

Number of Responses

All Small Large

17-18 18-19 17-18 18-19 17-18 18-19

AFNHA 22 17 16 9 6 6

Comparison 10 8 8 3 2 4

Approximately half as many comparison organizations responded to the survey each 

year as program organizations.

In 2017-18, the number of organizations completing the survey were as follows:

• 22 AFNHA organizations and 10 comparison organizations for a total 

of 32 respondents

• 16 small AFNHA organizations and 8 small comparison organizations for a total of 

24 small respondents

• 6 large AFNHA organizations and 2 large comparison organizations for a total 

of 8 large respondents

In 2018-19, the number of organizations completing the survey were as follows:

• 17 AFNHA organizations and 8 comparison organizations for a total 

of 25 respondents

• 9 small AFNHA organizations and 3 comparison organizations for a total of 12 

small respondents

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 11
• 6 large AFNHA organizations and 4 comparison organizations for a total of 10 

large respondents

Data Analysis

The analysis of collected Volunteer Capacity Survey data was conducted in two phases.

• Phase I was conducted on all survey data collected over three years and 

summarized in an initial evaluation report completed in December 2019 and 

submitted to Volunteer West Virginia.

• Phase II was conducted on only the survey data most aligned with member 

service activities and is summarized in this revised report.

The following is a description of the analyses completed during each 

phase. Phase I

The first phase of data analysis included analyses of all 45 survey question responses for 

both AFNHA organizations and comparison group organizations across three years – partial 

pilot data (2016-17) and data from the next 3-year grant cycle (2017-18 and 2018-19.) The 

analyses of results included descriptive statistics, including frequency and percent for each 

questions and subpart, and inferential statistics where appropriate. T-tests were performed 

between the control and comparison organizations, and ANOVAs when categories of 

organizations were identified across the control and comparison organizations, for example, 

museums. Four measures and tests were used to determine the significance of differences, if 

found. Because we used a population, the response rate was important. We calculated other 

measures as appropriate.

a. sample size

b. effect size “Effect size is a simple way of quantifying the difference between two 

groups. It has importance over the use of tests of statistical significance alone. It 

emphasizes the size of the difference and is not based on sample size.” In

addition, there are ways to interpret effect size so that comparisons are clearer 

and easily visualized.

2

c. significance level = P(Type I error) = probability of finding an effect that is not there (.05)

d. power = 1 - P (Type II error) = probability of finding an effect that is there. “Power 

analysis is normally conducted before the data collection. The main purpose underlying 

power analysis is to determine the smallest sample size that will detect the effect of a 

given test at the desired level of significance.” (70 as sample = .8358) 3

2 https://www.leeds.ac.uk/educol/documents/00002182.htm

3 http://www.statisticssolutions.com/statistical-power-analysis/

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 12
All calculations were meant to determine if there were differences, beyond 

significance testing, between the AFNHA organizations and the comparison organizations, 

as illustrated in the hypotheses table with possible actions.

Phase II

An in depth follow up review of the original evaluation report and its findings, the Volunteer 

Capacity Survey questions, three years of raw data, and the results of statistical tests run during 

the initial analysis were undertaken in October-November 2020. The review revealed four key 

limitations of the data collection and Phase I analyses which distorted the reported results.

1. Many of the survey questions about volunteer management capacity did not align with 

the service activities members were tasked to do at their host organizations. Therefore, 

those questions were not valid measures for assessing member effectiveness.

2. Both the AFNHA and comparison group sample sizes were very small (between n=2 

and n=16) plus many organizations did not completed the survey for all three years 

– therefore the planned look at capacity building over time was not possible.

3. The analyses of the responses for the AFNHA and comparison organizations did not 

differentiate by organization size – small and large organizations were aggregated 

within each group. However, for example, the capacity and practices used by small 

community based organizations with no paid staff is quite different than a large 

governmental organization such as the WV Department of Forestry with multiple staff 

associated with the volunteer program.

4. The role of AmeriCorps members in volunteer management at AFNHA organizations 

was not factored into the analysis. At some organizations, members had primary 

responsibility for volunteer management, at others they assisted staff, and at a few 

their role was negligible.

In light of these limitations, it was decided to undertake a second round of analyses focused 

on answering the research question: To what extent do community organizations with volunteer 

management support by AFNHA AmeriCorps members show differences in volunteer capacity 

compared with organizations that have not had assistance from AmeriCorps members?

In Phase II of the analysis on which this report is based, six volunteer management practice 

survey questions were identified as being directly related to member service activities. The 

questions that were not included covered activities members would be unable to directly 

influence such as organizational risk management, funding sources, board administration, etc.

Collected data for the six key practices were then analyzed in order to provide a more 

accurate understanding of the impact of AFNHA member service on host organizations as 

compared to comparable size organizations that had no member.

Means for reported frequencies for each of the six practices were calculated five ways:

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 13
1. All AFNHA organization and all comparison organizations

2. All small organizations: AFNHA and comparisons

3. Small organizations where members played a reported role in volunteer 

management: AFNHA and comparisons

4. All large organizations: AFNHA and comparisons

5. Large organizations where members played a reported role in volunteer 

management: AFNHA and comparisons

Miscellaneous Data and Reporting Notes

• Every attempt was made to ensure the accuracy of the data provided in tables and the 

narratives accompanying them. However, despite our best efforts, there may still be a 

few small inadvertent typos or calculation errors for which we apologize in advance.

• All calculated means in tables were rounded to the nearest full number or tenth for 

ease in interpretation. We believe trends are more visible to the reader using the 

shorter numeric form.

• Tables were designed to present data as consistently and clearly as possible. 

Narrative describing the contents of each table are also provided.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 14
Evaluation Findings

Survey respondents were asked to rate the frequency with which their organization 

engaged in six volunteer management practices using a 5-point Likert scale (5=Consistently, 

4=Usually, 3=Sometimes, 2=Rarely, 1=Never.) Each practice was selected from a longer list 

of items to reflect the specific activities in which AFNHA members were most likely involved 

at the majority of their host site organizations. The purpose was to determine whether 

organizations that had AFNHA members more frequently engaged in the six practices than 

comparable organizations that were not involved in the program.

Table 2 provides a summary of all AFNHA and comparison organization 

responses concerning volunteer management practice frequency.

Table 2. Volunteer Management Practices: AFNHA vs. Comparison Organizations

Volunteer Management 

Practice

All Organizations

2017-18 2018-19

AFNHA 

Organizations

Comparison 

Organizations

AFNHA 

Organizations

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site actively recruits volunteers. 22 2.9 10 3.2 17 3.2 8 4.8

Volunteers are provided 

with specific training for 

their position.

22 3.4 10 3.4 17 3.5 8 3.5

Site has regular supervision and 

communication with volunteers. 22 3.8 10 3.8 17 3.6 8 3.9

Volunteer projects are well 

planned, and adequate 

resources provided to 

support the volunteer.

22 3.3 10 3.5 17 3.6 8 4.3

Volunteers are publicly 

recognized for their service. 22 3.3 10 3.3 17 3.6 8 4.0

Site collects and tracks

volunteer hours and numbers 

of volunteers.

22 3.5 10 3.2 17 3.5 8 4.6

Among all organizations responding to the survey in 2017-18, the following differences 

between AFNHA and comparison organizations among the six volunteer management 

practices were noted.

• Volunteer hour and number tracking were undertaken more frequently by 

AFNHA organizations than comparison organizations.

• Volunteer training, supervision and communication, and recognition frequency 

were comparable for both AFNHA and comparison organizations.

• Comparison organizations reported they more frequently engaged in two of the 

six volunteer management practices than AFNHA organizations (recruitment and 

project planning) based on mean responses.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 15
Among all organizations responding to the survey in 2018-19, the following 

differences among the six volunteer management practices were noted.

• No practice was undertaken more frequently by AFNHA than comparison organizations.

• Volunteer training was comparable for both AFNHA and comparison organizations.

• Comparison organizations reported they more frequently engaged in five of the six 

volunteer management practices than AFNHA organizations (recruitment, 

supervision/communication, project planning, recognition, and data tracking) 

based on mean responses.

Looking at both years, only one practice in the first of the two years was found to be done 

more frequently at AFNHA organizations than comparison organizations. Four practices were 

comparable between both groups over the two years. Comparison organizations more 

frequently engaged in seven of the practices over two years. Therefore, it appears having an 

AFNHA member was not a strong indicator of improved capacity as measured by frequency of 

volunteer management practice use.

However, since the context and capacity of the small community-based organizations, some 

with no paid staff is very different from that of large governmental organizations, an additional 

analysis was done breaking out the responses by size of organization. The following sections 

provide the results of the analysis based on organization size - large and small.

Large Organizations

The survey data for all large AFNHA and comparison organizations was analyzed and mean 

frequencies for the six practices calculated. Table 5 provides a summary of the results.

Table 3. Volunteer Management Practices: Large AFNHA vs. Comparison Organizations

Volunteer Management 

Practice

Large Organizations

2017-18 2018-19

AFNHA 

Organizations

Comparison 

Organizations

AFNHA 

Organizations

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site actively recruits volunteers. 6 3.8 2 3.0 7 3.3 4 3.3

Volunteers are provided 

with specific training for 

their position.

6 4.5 2 3.5 7 3.8 4 3.8

Site has regular supervision and 

communication with volunteers. 6 4.7 2 3.5 7 3.8 4 3.5

Volunteer projects are well 

planned and adequate 

resources provided to 

support the volunteer.

6 4.2 2 3.5 7 4.0 4 4.0

Volunteers are publicly 

recognized for their service. 6 3.2 2 3.0 7 3.5 4 3.8

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 16
Volunteer Management 

Practice

Large Organizations

2017-18 2018-19

AFNHA 

Organizations

Comparison 

Organizations

AFNHA 

Organizations

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site collects and tracks

volunteer hours and numbers of 

volunteers.

6 4.7 2 3.5 7 4.3 4 4.8

Among large organizations responding to the survey in 2017-18, the following differences 

between AFNHA and comparison organizations among the six volunteer management practices 

were noted.

• All six volunteer management practices were done more frequently by AFNHA 

organizations than comparison organizations based on mean responses

Among large organizations responding to the survey in 2018-19, the following 

differences among the six volunteer management practices were noted.

• Volunteer supervision and communication was undertaken slightly more 

frequently by AFNHA organizations than comparison organizations.

• Volunteer recruitment, training, and project planning mean frequencies 

were comparable for both large AFNHA and comparison organizations.

• Comparison organizations reported they more frequently engaged in for two of 

the six volunteer management practices than AFNHA organizations (recognition 

and data tracking) based on mean responses.

In summary, the results for volunteer management practice frequency at large AFNHA 

organizations compared to the large non-AFNHA organizations did not show consistency across 

the two years of data. In the first year, AFNHA organizations more frequently engaged in all six 

practices but in the second year only in one practice. Therefore, it is difficult to make definitive 

conclusions about the effectiveness of the AFNHA model for capacity building, especially due to 

small sample sizes.

To take a deeper look at the potential differences in volunteer management practices 

among large organizations where members were specifically identified as having the primary 

role for volunteer management and/or assisted staff, a secondary analysis of the six practices 

was undertaken. Table 6 below provides a summary of the mean frequencies with which large 

organizations engaged in the six practices comparing AFNHA organizations clearly using 

members to build their capacity versus the comparison organizations.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 17
Table 4. Volunteer Management Practices: Large AFNHA vs. Comparison Organization Size by Member Role

Volunteer Management 

Practice

Large Organizations

2017-18 2018-19

AFNHA 

Organizations*

Comparison 

Organizations

AFNHA 

Organizations*

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site actively recruits volunteers. 6 3.8 2 3.0 6 3.3 4 3.3

Volunteers are provided 

with specific training for 

their position.

6 4.5 2 3.5 6 3.8 4 3.8

Site has regular supervision and 

communication with volunteers. 6 4.7 2 3.5 6 3.8 4 3.5

Volunteer projects are well 

planned and adequate 

resources provided to 

support the volunteer.

6 4.2 2 3.5 6 4.0 4 4.0

Volunteers are publicly 

recognized for their service. 6 3.2 2 3.0 6 3.5 4 3.8

Site collects and tracks

volunteer hours and numbers 

of volunteers.

6 4.7 2 3.5 6 4.3 4 4.8

* Only includes AFNHA host organizations where members were reported to be either directly responsible for 

or assisted staff with volunteers.

Among large organizations responding to the survey in 2017-18 that clearly used members 

to build their volunteer management capacity versus comparison organizations, the following 

differences among the six volunteer management practices were noted.

• Large AFNHA organizations using members to manage volunteers more frequently 

engaged in all six practices compared to comparison organizations. Based on mean 

responses

Among large organizations responding to the survey in 2018-19 that used members to build 

their volunteer management capacity versus comparison organizations, the differences noted 

were less clear.

• Large AFNHA organizations using members to manage volunteers more frequently 

engaged in supervision and communication than comparison organization.

• Volunteer recruitment, training and project planning frequency were comparable for 

both large AFNHA and comparison organizations.

• Large comparison organizations reported they more frequently engaged in volunteer 

recruitment than large AFNHA organizations based on mean responses for two 

practices: recognition and volunteer tracking.

The secondary analysis revealed somewhat more frequent use of volunteer management 

practices among large AFNHA organizations than the comparison group in 2017-18, but less so 

in 2018-19.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 18
Small Organizations

The survey data for all small AFNHA and comparison organizations was analyzed and 

mean frequencies for the six practices calculated. Table 3 provides a summary of the results.

Table 5. Volunteer Management Practices: Small AFNHA vs. Comparison Organizations

Volunteer Management 

Practice

Small Organizations

2017-18 2018-19

AFNHA 

Organizations

Comparison 

Organizations

AFNHA 

Organizations

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site actively recruits volunteers. 16 2.6 8 3.3 9 3.4 3 4.0

Volunteers are provided 

with specific training for 

their position.

16 3.0 8 3.3 9 4.0 3 3.3

Site has regular supervision and 

communication with volunteers. 16 3.4 8 3.9 9 4.0 3 4.3

Volunteer projects are well 

planned, and adequate 

resources provided to support 

the volunteer.

16 3.0 8 3.5 9 4.0 3 4.3

Volunteers are publicly 

recognized for their service. 16 3.4 8 3.4 9 4.1 3 4.7

Site collects and tracks

volunteer hours and numbers of 

volunteers.

16 3.1 8 3.0 9 4.0 3 4.3

Among small organizations responding to the survey in 2017-18, the following differences 

between AFNHA and comparison organizations among the six volunteer management practices 

were noted.

• Volunteer hour and number tracking were undertaken slightly more frequently 

by AFNHA organizations than comparison organizations.

• Volunteer recognition frequency was comparable for both AFNHA and 

comparison organizations.

• Comparison organizations reported they more frequently engaged in four of the six 

volunteer management practices than AFNHA organizations (recruitment, training, 

supervision/communication, and project planning) based on mean responses.

Among small organizations responding to the survey in 2018-19, the following 

differences among the six volunteer management practices were noted.

• Volunteer training was undertaken slightly more frequently by AFNHA 

organizations than comparison organizations.

• Comparison organizations reported they more frequently engaged in five of the 

six volunteer management practices than AFNHA organizations (recruitment,

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 19
supervision/communication, project planning, recognition, and data tracking) 

based on mean responses.

In summary, there was only a negligible positive difference in volunteer 

management practice frequency at small AFNHA organizations compared to the small 

non-AFNHA organizations.

Possible explanations for this finding include volunteer management at all 8 small 

comparison organizations were the responsibility of organization staff which likely implies 

a more consistent approach to volunteer use generally. Organizations invited to be part 

of the comparison group that had no staff or without strong practices may have selfselected out of responding to the survey. Also, it is also likely the small organizations that 

hosted AFNHA members were aware they had limited capacity for volunteer 

management which motivated them to apply for AmeriCorps support.

As the survey did not specifically ask AFNHA organizations to consider the effects of 

their member(s) while responding how often they engaged in various practices, it is 

difficult to know how which areas members made the most significant contributions. Also, 

because the survey did not ask about improvement, only current status.

To take a deeper look at the potential differences in volunteer management practices 

among small organizations where members were specifically identified as having the primary 

role for volunteer management and/or assisted staff, a secondary analysis of the six practices 

was undertaken. Table 4 below provides a summary of the mean frequencies with which small 

organizations engaged in the six practices comparing AFNHA organizations clearly using 

members to build their capacity versus the comparison organizations.

Table 6. Volunteer Management Practices: Small AFNHA vs. Comparison Organizations by Member Role

Volunteer Management 

Practice

Small Organizations

2017-18 2018-19

AFNHA 

Organizations*

Comparison 

Organizations

AFNHA 

Organizations*

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site actively recruits volunteers. 10 3.1 8 3.3 9 3.4 3 4.0

Volunteers are provided 

with specific training for 

their position.

10 3.7 8 3.3 9 4.0 3 3.3

Site has regular supervision and 

communication with volunteers. 10 3.9 8 3.9 9 4.0 3 4.3

Volunteer projects are well 

planned and adequate 

resources provided to support 

the volunteer.

10 3.8 8 3.5 9 4.0 3 4.3

Volunteers are publicly 

recognized for their service. 10 3.6 8 3.4 9 4.1 3 4.7

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 20
Volunteer Management 

Practice

Small Organizations

2017-18 2018-19

AFNHA 

Organizations*

Comparison 

Organizations

AFNHA 

Organizations*

Comparison 

Organizations

Number Mean Number Mean Number Mean Number Mean

Site collects and tracks

volunteer hours and numbers of 

volunteers.

10 4.3 8 3.0 9 4.0 3 4.3

* Only includes AFNHA host organizations where members were reported to be either directly responsible for 

or assisted staff with volunteers.

Among small organizations responding to the survey in 2017-18 that clearly used members 

to build their volunteer management capacity versus comparison organizations, the following 

differences among the six volunteer management practices were noted.

• Small AFNHA organizations using members to manage volunteers more frequently 

engaged in four of the six practices compared to comparison organizations: 

volunteer training, project planning, recognition, and data collection.

• Volunteer supervision and communication frequency was comparable for both small 

AFNHA and comparison organizations.

• Small comparison organizations reported they more frequently engaged in volunteer 

recruitment than small AFNHA organizations based on mean responses.

Among small organizations responding to the survey in 2018-19 that used members to build 

their volunteer management capacity versus comparison organizations, the differences noted 

were less clear.

• Small AFNHA organizations using members to manage volunteers more 

frequently engaged in member training than comparison organization.

• Small comparison organizations reported they more frequently engaged in volunteer 

recruitment than small AFNHA organizations based on mean responses for the other 

five practices.

The secondary analysis revealed somewhat more frequent use of volunteer 

management practices among AFNHA organizations than the comparison group in 2017-18, 

but less so in 2018-19.

Conclusions

The effects of AFNHA members on increasing the capacity of host organizations as 

compared to organizations without an AFNHA member was unclear based on the collected 

data. In some analyses, AFNHA organizations reported more frequent engagement in selected 

practices; in another year, the same advantage was not noted.

Of the AFNHA program sites, 20 sites in 2017-18 and 19 in 2018-19 reported for 

performance measurement that members at their sites had implemented at least three 

volunteer management practices. Yet the survey data is not showing clear benefit of longer2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 21
term use of these practices. In comparing this to the clear benefit found in the CalSERVES 

study for members focusing entirely on volunteer management, this may have implications 

for AFNHA program model expectations. These results indicate that while AFNHA member 

service can be useful in managing volunteers, the amount of time they spend on volunteer 

management compared to direct service is less helpful in achieving long-term capacity 

building for their organization.

There were also major limitations in the implementation of this study. One of the greatest 

explanatory factors for the small effects of AFNHA members at their host organizations 

compared to other organizations is likely the limitation of the data collection tool used. When 

the survey was administered, the instructions for both groups was the same to eliminate 

potential differences. There were no additional questions asked which would have allowed 

AFNHA organizations to indicate the greatest impact their member had on volunteer 

management practices. Since the survey did not specifically ask AFNHA organizations to 

consider the effects of their member(s) while responding to how often they engaged in various 

practices, it is difficult to know how which areas members made the most significant 

contributions. Also, the survey did not ask about improvement, only current status. There was 

no way for host organizations to indicate which practices the member had been assigned and 

had actively participated in. Any assumption that hosting a member improves volunteer 

program practices across the board did not bear out.

Another time, a survey of host organizations could be more useful if respondents were 

asked to indicate which practice(s) they felt their member had positively impacted. Rating 

overall organizational capacity in a multitude of areas that were not part of a member’s 

service description could not provide an accurate picture of the value AFNHA provided to 

building an organization’s capacity to effectively manage volunteers.

Other possible explanations for the inconclusive findings include volunteer management at 

all 8 small comparison organizations being the responsibility of organization staff which likely 

implies a more consistent approach to volunteer use generally. Organizations invited to be part 

of the comparison group that had no staff or without strong practices may have self-selected 

out of responding to the survey. Also, it is also likely the small organizations that hosted AFNHA 

members were aware they had limited capacity for volunteer management which motivated 

them to apply for AmeriCorps support.

All large organizations regardless of group had many more resources and more 

established volunteer programs. It is unlikely that an ANFHA member would be able to 

make much significant change in the volunteer program of a large bureaucracy.

Recommendations

The inconclusiveness of the evaluation results has implications for future directions and 

priorities for the AFNHA program. AFNHA interventions up to now have included volunteer 

management capacity building as a partial, but significant, portion of the duties for all

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 22
members, regardless of their primary direct service assignment. While the evaluation results 

suggest the value of member support for volunteer management, especially for a number of 

small community organizations with no or minimal staff, the results also reveal opportunities 

for continuous improvement of this portion of the AFNHA program moving forward.

1. Reduce the level of priority of capacity building and volunteer management within 

our program model. This recommendation is reflected in the change in logic model 

for 2021 grant application to 15% of member time on capacity building, rather than 

the previous goal of 30%. Remove Capacity Building as a Performance Measure for 

the program, instead focusing more on member environmental stewardship direct 

service. AFNHA will continue to collect volunteer data, and member and site reports 

on volunteer management practices.

2. Work with site supervisors to continue assignment of volunteer management, and 

organizational capacity building, only for those organizations that identify the need 

for these services from their AmeriCorps members. Other organizations will prioritize 

the members’ direct service activities if that is needed more for that site.

3. Continue to work to strengthen training in volunteer management and volunteer 

management practices for the members who have these duties assigned, so that they 

will be able to follow through with them effectively. Encourage site representatives 

who also work with volunteers to attend this training as well.

Future Evaluation Opportunities

When measuring the outcomes of member volunteer management activities in the future, 

revisions to the survey (e.g., the number of questions, content, and wording) would allow 

AFNHA to more accurately assess the benefits that are directly related to AmeriCorps member 

service. It would be useful to have feedback from members and supervisors about what 

specific activities were most feasible for members to assist with, as well as the effectiveness of 

those efforts.

If still of interest, another evaluation could revisit the second evaluation question that was 

not included in the current analysis and report: To what extent do community organizations 

with volunteer management support by AFNHA AmeriCorps members show improvements in 

volunteer capacity over time? This would require a different approach to collecting baseline 

data and tracking improvements over time than was included in the current study.

For the next round of AFNHA AmeriCorps program evaluation, we plan to look at elements 

of our member direct service activities, rather than capacity building. We are adapting our 

program model to focus more on stewardship education, which is appropriate for many of our 

organizations. Our next evaluation is intended to help us shape this shift in our program focus 

and give us the tools to effectively measure outcomes, and in the future to measure impact, of 

stewardship education delivered by AmeriCorps members.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 23
Appendix A: Volunteer Management Capacity Survey

AFNHA Volunteer Management Survey

INTRODUCTION

Thank you for your participation in the Appalachian Forest Heritage Area (AFHA) volunteer 

management survey. This survey asks about your organization and the site where service members 

and/or volunteers work.

The purpose of the survey is to understand how volunteers overall are coordinated and managed in 

order to contribute to the work of sites. The survey will also help AFHA understand how we can 

support our sites with volunteer management and coordination.

We will ask about two types of help, service members and volunteers. Service members are 

individuals who are usually paid through a stipend (such as AmeriCorps members or interns). 

Volunteers are unpaid sources of help who may have many roles in your program (including helping 

with one-time events, or ongoing roles such as board members, committee members, docents, etc.)

For some questions we ask you to think back over the past year, generally between September 1, 

2017, and September 1, 2018. For other questions think about your organization in general.

We use the term "site" to mean the location where service members and/or volunteers are 

supervised, work, and/or work out of (office, museum, etc.)

If you think there is someone else who should also be completing the survey, please feel free to 

forward the link. We want survey completed from one person with each site who has substantial 

interaction with service members and volunteers.

We appreciate your taking the time to complete the survey, which should take 15 to 30 minutes. If 

you have any questions please feel free to contact our researcher, Maryann Durland at 

mdurland@durlandconsulting.com or 630.650.9944

GENERAL INFORMATION

1. Location Information

Name of Person completing survey:

Email _________________ 

Phone number _________________ 

Name of site: Town nearest site location

2. What type of an organization is your site a part of?

[--Please Select--] 

• Federal Government

• State Government

• Local Government

• Legal Non-profit

• Unaffiliated/ friends of/ etc

• Other

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 24
3. Which best describes your site?

[--Please Select--] 

• Subsidiary, branch, or office of a larger organization

• Program focusing on specific purposes within a larger organization

• Commission/Authority or other quasi-governmental organization

• Single stand-alone organization

• Other (describe) ________________________________ 

4. Estimate the percentage of program emphasis for each relevant service focus area for your site.

(To equal 100%) The sum of all entered values must be 100.

Community Development

Cultural Networking & Arts

Tourism

Environment/natural resources conservation

Environmental education & outreach

Heritage development & interpretation

Historic Preservation

Other

5. During the past year (September - September), what was the number of full-time paid staff for 

your site?

6. During the past year (September - September) what was the number of part-time or 

seasonal paid staff for your site?

7. Overall, how important are volunteers to your site?

Extremely Important

Very Important 

Moderately Important 

Slightly Important 

Not at All Important

8. Additional Comments

SERVICE MEMBERS & VOLUNTEERS

9. Over the past year, what was the total number of service members (stipended or paid) 

who helped your site; and how many total hours did they work, per type.

Numbe

r

Hours

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 25
AFHA Americorps _ _ 

VISTA _ _ 

Other National Service Member _ _ 

Student Conservation Association 

(SCA) with stipend _ _ 

College Intern with stipend _ _

Other _ _ 

10. Over the past year, what is the total number of volunteers (unpaid) who helped your site, 

and approximately how many total hours did they work, per type.

11. Over the past year, did you have a process to track hours for volunteers?

0. If you selected yes, could you briefly describe your process?

1. Who at your site had primary responsibility for coordinating volunteers (including; 

planning, recruiting, managing, training, recognizing, etc.)

[--Please Select One--] 

 No-one

 Exec Director or similar leadership position

 Volunteer Coordinator or similar staff position

 Other staff as part of their position

 AmeriCorps service member

 Other stipended service members

 Volunteer assigned as volunteer coordinator

 Other

 If other, please specify_________________________________ 

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 26

Number Hours

Episodic (1-3 times a year) volunteers

Ongoing volunteers (board members, docents, 

committee members, etc) _ _

Number Hours

Episodic (1-3 times a year) volunteers _ _

Ongoing volunteers (board members, docents, 

committee members, etc) _ _
14. Approximately how many total hours did they spend coordinating volunteers?

15. Who else at your site assists with coordinating volunteers (Including: planning, 

recruiting, managing, training, recognizing, etc)? (Select as many as applicable)

 N o -one

 Exec Director or similar leadership position

 Volunteer Coordinator or similar staff position

 Other staff as part of their position

 AmeriCorps service member

 Other stipended service members

 Volunteer assigned as volunteer coordinator

 Other

 If other, please specify__________________ 

16. Approximately how many total hours, altogether, did they spend coordinating volunteers?

17. What was your role regarding supervising and working with volunteers?

18. Your volunteer coordinator, staff, and/or board receive specific training in working 

with volunteers effectively.

Always Often Sometimes Rarely Never

Volunteer coordinator O O O O O 

Staff O O O O O 

Supervisors O O O O O 

Administration O O O O O 

Board O O O O O 

VOLUNTEER MANAGEMENT

The next sets of questions apply to your overall site and volunteers in general. Volunteers are 

unpaid, and may include board members and other ongoing volunteers, as well as episodic or 

occasional volunteers.

The sections will include:

• Volunteer Planning

• Volunteer Recruiting

• Volunteer Training

• Volunteer Management

• Volunteer Recognition

• Volunteer Data & Evaluation

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 27
19. At what level does your site have a process or plan that methodically and strategically 

addresses volunteer involvement in the organization.

Process is in Process is in We are We do not A process is

place and 

consistently used

place but not 

consistently used

developing a 

process

have a process, 

but is on agenda

not required or 

needed

If other, please specify

20. Your site plans and allocates resources for the support and management of volunteers.

Consistently Usually Sometimes Rarely Never

21. Your site has and applies written policies and procedures for volunteer management.

Consistently Usually Sometimes Rarely Never

0. Volunteer policies for your site adequately and appropriately address risk management 

and safety issues.

Consistently Usually Sometimes Rarely Never

0. 

1. 

2. 

3. 

4. 

Your site actively recruits volunteers.

Consistently Usually Sometimes Rarely Never

Your site conducts targeted volunteer recruitment efforts based on specific needs or jobs.

Consistently Usually Sometimes Rarely Never

Job or task descriptions for volunteers have been developed.

Consistently Usually Sometimes Rarely Never

Volunteers are matched to positions based on their interests and skills

Consistently Usually Sometimes Rarely Never

Flexibility has been built into the positions to suit varying volunteer skills, schedules 

and interests.

Consistently Usually Sometimes Rarely Never

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 28
28. Your volunteers undergo orientation training which provides them with a clear 

understanding of the site and its policies.

Consistently Usually Sometimes Rarely Never

29. Your volunteers are provided with specific training for their position.

Consistently Usually Sometimes Rarely Never

30. Your organization provides additional skill training and/or professional development for 

your volunteers.

Consistently Usually Sometimes Rarely Never

31 Your site has regular supervision and communication with volunteers.

Consistently Usually Sometimes Rarely Never

32 Volunteer projects are well planned and adequate resources provided to support the

volunteer.

Consistently Usually Sometimes Rarely Never

33 Each volunteer is provided with team leader, mentoring or individual support that they need

to serve effectively.

Consistently Usually Sometimes Rarely Never

34 Volunteers are encouraged to get more involved with the organization, and 

offered opportunities for additional or different tasks when they are ready

Consistently Usually Sometimes Rarely Never

35

36.

Your volunteers receive informal or personal thanks for their service.

Consistently Usually Sometimes Rarely Never

Your volunteers are publicly recognized for their service.

Consistently Usually Sometimes Rarely Never
2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 29
37. Volunteer performance is regularly evaluated and feedback given to volunteers.

Consistently Usually Sometimes Rarely Never

38. Your site collects and tracks volunteer hours and numbers of volunteers.

Consistently Usually Sometimes Rarely Never

39. Your organization collects information and data on volunteer accomplishments, results 

and outcomes of their activities.

Consistently Usually Sometimes Rarely Never

0. Evaluation results of volunteer activities and outcomes are used for continual improvement 

of our site and volunteer program.

Consistently Usually Sometimes Rarely Never

1. Your site regularly and actively seeks advice or input from its volunteers.

Consistently Usually Sometimes Rarely Never

40. Data and evaluation results of volunteer activities and impacts are shared with:

Always Often Sometimes Rarely Never

No one O O O O O 

Volunteers O O O O O 

Service members O O O O O 

Board and 

Administration O O O O O 

Community or public O O O O O 

Funding sources/donors O O O O O 

Main office O O O O O 

others O O O O O 

41. If you would like to add anything to help us understand volunteer management at 

your organization, please do so here. Thank you.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 30
Appendix B: References

(UPS 1998) “Managing Volunteers, A Report From: United Parcel Service.” 1998. 

http://academic.regis.edu/volunteer/specialcollection/specfiles/1998upssurvey.pdf 

(Volunteering 2015) “Volunteering and Civic Life in America 2015”, Corporation for 

National and Community Service & National Conference on Citizenship. 

(www.VolunteeringInAmerica.gov), 

(Urban, Charities 2004) “Volunteer Management Capacity in America’s Charities 

and Congregations” Mark A Hagar, The Urban Institute 2004 

http://www.urban.org/research/publication/volunteer-management-capacityamericas-charities-and-congregations 

(Knepper 2015) "Volunteer Management Practices during Challenging Economic Times," 

Knepper, Hillary; D'Agostino, Maria J.; and Levine, Helisse (2015) Journal of Public Management 

& Social Policy: Vol. 20: No. 2, Article 7. Http://digitalscholarship.tsu.edu/jpmsp/vol20/iss2/7/ 

(Volunteering: Issue Brief 2010). “Volunteering in America 2010: Issue Brief. National, State, 

and City Information”, Corporation for National and Community Service, Office of Research 

and Policy Development Washington, DC. 2010, June. 

http://www.nationalservice.gov/organizations/default/files/documents/10_0614_via_final 

issue_brief.pdf 

(Idealist.org) “Why is Volunteer Management Important?” 

Idealist.org. http://www.idealist.org/info/VolunteerMgmt/Important 

(Urban, Volunteers 2004) “Volunteer Management Practices and Retention of 

Volunteers”. The Urban Institute. June 2004. 

http://www.urban.org/organizations/default/files/alfresco/publication-pdfs/411005-

Volunteer-Management-Practices-and-Retention-of-Volunteers.PDF 

(ruralvolunteer.org) “Toolkit for Working with Rural Volunteers”. Appalachian Coal 

Country Team and Western Hardrock Watershed Team http://ruralvolunteer.org/ 

(Transforming, 2010) “Transforming Communities Through Service: A Collection of the 

Most Innovative AmeriCorps State and Volunteer Generation Fund Programs in the United 

States” A report by America’s Service Commissions and Innovations in Civic Participation, 

April 2010, http://www.icicp.org/wp-content/uploads/2013/11/TransformingCommunities-Through-Service.pdf 

(volunteer.maryland.gov) Web site of Volunteer Maryland AmeriCorps program, Our 

Impact tab. (http://volunteer.maryland.gov/volunteer-maryland-results/) 

(CalSERVES 2012) “AmeriCorps VIP Volunteer Capacity Study”, JBS International, Inc., 

November 2012 , 

http://www.nationalservice.gov/organizations/default/files/evidenceexchange/FR_CaliforniaVo

lunteers_CalSERVES-NCOE_AmeriCorpsVIP.pdf

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 31
(United Way, King County 2009) “Volunteer Management: Challenges and Opportunities 

Facing Non-Profits”. Rick Lynch, Nikki Russell, United Way, King County. 2009. 

http://www.thenonprofitpartnership.org/files/volunteer-management-uw-king.pdf 

Building Volunteer Management Capacity for Community Organizations, Outcome 

Evaluation Study, Appalachian Forest Heritage Area AmeriCorps: Enhancing Assets to 

Benefit Communities, Durland, Maryann, AFNHA staff, January 2017.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 32
Appendix C: Evaluation Team

Dr. Maryann Durland Ph.D – Senior Evaluator, Study Analysis

Dr. Durland began her work in evaluation in the mid-1970’s with Title I monitoring and evaluation 

at the Louisiana State Department of Education. She received her Doctorate in Educational Research 

Methodology, August 1996, from Louisiana State University and holds a Class A Evaluation Certificate 

from Louisiana. She specializes in social network analysis and mixed methods and models evaluations 

and is currently completing the Field Guide to Social Network Analysis in Evaluation, for Stanford 

University Press (2016). She was one of the founding members of the American Evaluation Association 

TIG, Social Network Analysis. She has conducted evaluations for non-profits, schools and school 

systems, and businesses. She has very strong communication and facilitation skills as well as problem 

solving skills. In addition, she has strong research skills including both quantitative and qualitative data 

collection and analysis and social network analysis. She has presented consistently conferences since 

1996.

Durland Consulting started in 2001 and since then has conducted over 40 mixed methods and 

models evaluations. Durland Consulting’s clients have included WESTAT, University of Washington, 

Community Science/ Robert Woods Johnson, NDI, Price-Waterhouse-Coopers, Battelle Energy Alliance, 

Arthur Andersen, Rockford School District, North Cook Intermediate Service Center, South Cook 

Education Consortium, the Knight Foundation, National Lewis University, West Virginia University at 

Parkersburg, as well as consultation on social network analysis to many independent consultants.

Dr. Durland conducted statistical analysis of the Volunteer Capacity Survey results and developed 

findings from that analysis.

AFNHA Staff

Phyllis Baxter, AFNHA Executive Director has worked with AFNHA since 2001 and has been Program 

Director for AFNHA AmeriCorps from the start of the program in 2007. She has a Masters degree in 

History with a post-graduate certificate in Public History. She has extensive experience with grant 

writing, project management, and administration over a 25-year career in historic preservation and nonprofit management. She attended the CNCS Research Summit in December 2015 and has participated in 

Volunteer West Virginia evaluation trainings. Baxter had primary responsibility for overseeing this 

program evaluation.

Logan Smith, AFNHA Financial and Program Administrator, is responsible for administration and 

bookkeeping for AFNHA and the AmeriCorps grant, and assists with program supervision. Smith has 

attended CNCS specific trainings such as the Leadership Forum for Non-Profit Executives held at the 

University of Washington, which included a week-long evaluation module, CNCS sponsored financial 

management training held in Wisconsin, and CNCS Regional Conferences. He assisted with evaluation 

and survey design and data collection.

2017-20 REVISED AFNHA Evaluation Report on Volunteer Management Capacity Building 33