Evaluation of the Appalachian 

Regional Commission’s Education and 

Workforce Development Projects: 

2000–2008 

Authors 

 

Gary Silverstein

Melissa Bryce

Westat

Diana Long 

Rahall Transportation Institute

Steve Landau 

Economic Development Research Group

December 2012 

Prepared for:

The Appalachian Regional 

Commission

1666 Connecticut Avenue, NW

Suite 700

Washington, DC 20009-1068

Prepared by:

Westat

1600 Research Boulevard

Rockville, Maryland 20850-3129

(301) 251-1500

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 iii 

Contents

Chapter Page

Executive Summary ......................................................................................... vii

1 Introduction ...................................................................................................... 1-1 

1.1 Overview of the Appalachian Regional Commission’s 

Education and Workforce Development Projects ............................... 1-1 

1.2 Overview of the Study ........................................................................ 1-2 

1.3 Structure of the Report ........................................................................ 1-6 

2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects .................................................................... 2-1 

2.1 Characteristics of ARC Education and Workforce 

Development Projects ......................................................................... 2-1 

2.2 Information About Students and Workers/Trainees Served 

and Improved ...................................................................................... 2-7 

3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects ................................ 3-1 

3.1 Extent to Which ARC Education and Workforce 

Development Projects Are Tracking Long-Term Participant 

Outcomes ............................................................................................ 3-4 

3.2 Other Items in ARC.net that Might be Reported for 

Education and Workforce Development Projects ............................... 3-7 

3.3 Other Potential Employment and Private Sector Impacts 

(Not Currently in ARC.net) ................................................................ 3-11 

4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development 

Projects............................................................................................................. 4-1 

4.1 Summary ............................................................................................. 4-1 

4.2 Recommendations ............................................................................... 4-2 
 iv Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Appendix Page

A Case Study Findings ........................................................................................ A-1 

B Opportunities and Challenges Regarding the Collection and Analysis 

of Economic Outcomes for the ARC’s Education and Workforce 

Development Projects ...................................................................................... B-1 

C Proposed Addendum to the Application Form for ARC Education 

and Workforce Development Projects ............................................................. C-1 

D Proposed Supplement to the Annual Report for ARC Education and 

Workforce Development Projects .................................................................... D-1 

E Online Questionnaire Administered With ARC Education and 

Workforce Development Projects .................................................................... E-1 

F Protocol for Telephone Interviews With a Sample of ARC Education 

and Workforce Development Projects That Collected Follow-up 

Data From Former Participants ........................................................................ F-1 

G Protocol for Telephone Interviews With a Sample of ARC Education 

and Workforce Development Projects That Did Not Collect Followup Data From Former Participants ................................................................... G-1 

Table

1-1 Questionnaire response rate, by year in which ARC education and 

workforce development projects were first funded .......................................... 1-5 

2-1 Total and average funds allocated for ARC education and workforce 

development projects between FY 2000 and FY 2008, for all projects 

and projects that completed a questionnaire .................................................... 2-2 

2-2 Populations served by ARC education and workforce development 

projects ............................................................................................................. 2-3 

2-3 Types of populations targeted by ARC education and workforce 

development projects ....................................................................................... 2-4 

2-4 General activities supported by ARC education and workforce 

development projects ....................................................................................... 2-5 

2-5 Specific activities supported by ARC education and workforce 

development projects ....................................................................................... 2-6 

2-6 Proportion of ARC education and workforce development projects 

providing multiple services .............................................................................. 2-7 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 v 

Table Page

2-7 Proportion of ARC education and workforce development projects 

that provided information about students and workers/trainees served 

and improved ................................................................................................... 2-8 

2-8 Number of students and workers/trainees that received services and 

benefited as a result of ARC education and workforce development 

projects ............................................................................................................. 2-11 

2-9 Projects with corresponding output and outcome records in both the 

questionnaire and ARC.net .............................................................................. 2-11 

2-10 How students were improved by their participation in ARC 

education and workforce development projects............................................... 2-13 

2-11 How workers/trainees were improved by their participation in ARC 

education and workforce development projects............................................... 2-13 

3-1 Length of time that ARC education and workforce development 

projects tracked participants after program completion ................................... 3-5 

3-2 Whether ARC education and workforce development projects 

provided information about other performance measures in ARC.net: 

2000–2008 ....................................................................................................... 3-7 

3-3 Whether ARC education and workforce development projects 

provided information about other ARC.net performance measures on 

the Westat questionnaire .................................................................................. 3-9 

3-4 Number of participants who received services and benefited as a 

result of ARC education and workforce development projects ....................... 3-10 

3-5 Economic benefits that occurred as a result of ARC education and 

workforce development projects ...................................................................... 3-11 

3-6 Whether ARC education and workforce development projects 

provided information about other potential employment-related 

outcomes .......................................................................................................... 3-13 

3-7 Employment status of individuals before and after participation in an 

ARC education and workforce development project ....................................... 3-14 

3-8 Residential status of individuals after they exited an ARC education 

and workforce development project ................................................................. 3-14 

3-9 Sector in which individuals were employed after they exited an ARC 

education and workforce development project ................................................ 3-15 

3-10 Wage rates for individuals who found work after they exited an ARC 

education and workforce development project ................................................ 3-16 
 vi Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table Page

4-1a Output and outcome comparison between questionnaire and ARC.net 

data from 2006–2008: Data for projects that provided outputs or 

outcomes in the questionnaire .......................................................................... 4-10 

4-1b Output and outcome comparison between questionnaire and ARC.net 

data from 2006–2008: Projects with corresponding output and 

outcome records in both the questionnaire and ARC.net ................................. 4-11 

Exhibit

3-1 Examples of outcomes for other education and workforce 

development projects ....................................................................................... 3-3 

4-1 Summary of findings from the online questionnaire ....................................... 4-11 

4-2a Potential response options for coding ARC.net narratives on how 

students were improved ................................................................................... 4-13 

4-2b Potential response options for coding ARC.net narratives on how 

workers/trainees were improved ...................................................................... 4-14 

4-3 Example of items for an intake form for workforce development 

projects ............................................................................................................. 4-15 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 vii 

Executive Summary

Between 2000 and 2008, the Appalachian Regional Commission (ARC) invested a total of $65 million in 

386 education and workforce development projects.1

 This report documents the characteristics and 

outcomes of a subset of these projects and provides recommendations for enhancing the Commission’s 

collection and utilization of performance measurement data. It is based on data from an online 

questionnaire administered to all 386 education and workforce development projects funded by the ARC 

between 2000 and 2008, as well as telephone interviews with 23 of the projects that completed the 

questionnaire. The data presented in this report are intended to document the activities and 

accomplishments of projects that responded to these data requests only; findings should not be 

generalized to the ARC education and workforce development program as a whole. 

Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

The online questionnaire opened on June 28, 2011, and closed on October 28, 2011. Throughout the 

four-month data collection period, Westat and the ARC took a series of steps to increase the overall 

response rate. A total of 222 projects (57.5 percent) ultimately completed the questionnaire. Of the 222 

projects, 145 (65.3 percent) were education and 77 (34.7 percent) were workforce development grants. 

Almost half (49 percent) of projects that responded to the questionnaire provided at least some of their 

services to adults (not including postsecondary students or teachers), while 44 percent served K–12 

students and 36 percent served postsecondary students. One-fourth (24 percent) provided services to 

teachers, while a smaller proportion targeted services to preschool children (14 percent) or out-of-school 

youth (10 percent). 

Types of education and workforce development services provided. Projects were designed to provide 

a wide range of educational and workforce development services. The majority (56 percent) were 

providing educational attainment or achievement services that were specifically targeted to students 

pursuing a high school diploma or postsecondary degree, while 43 percent were providing career and 

technical education services and 41 percent were providing workforce training skills. Over half of the 

projects provided at least one of the following services: skills/training that enhanced employability 

 

1 When including matching funds from other sources (e.g., other Federal agencies, state or local agencies, private entities), the aggregate funding 

levels for these 386 projects was $142 million.
Executive Summary

 viii Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

(59 percent), special use classroom equipment such as computers or science labs (58 percent), and/or 

occupational/job skills training or instruction (53 percent). One-third (34 percent) provided computer 

skills instruction, while 23 percent provided or improved a physical structure. 

Number of students and workers/trainees served and benefited. At the time the questionnaire was 

administered (summer 2011), the 222 ARC education and workforce development projects funded 

between 2000 and 2008 reported that they had served a total of 414,296 students and workers/trainees—

including 334,803 students and 79,493 workers/trainees. Of the total served, 99,809 individuals were 

reported to have benefited from their participation—including 81,974 students and 17,835 

workers/trainees. Thus, approximately 25 percent of the students and 22 percent of the workers/trainees 

served were reported to have ultimately benefited from their participation in an ARC education and 

workforce development activity. However, the use of a single generic category (i.e., students and 

workers/trainees improved) to document the range of potential improvements for education and 

workforce development participants makes it difficult to interpret these findings in a meaningful manner. 

It also potentially fails to capture information about tangible benefits (e.g., attainment of a new skill) that 

went undocumented in projects that were solely focused on reporting data for a specific improvement 

(e.g., receiving a career credential). Therefore, we also compared the number of individuals that projects 

expected to serve and improve (as reported in ARC.net at the time projects applied for ARC funding) with 

the number that were reported as being served and improved on the Westat questionnaire. This 

comparison reveals that projects served more students and workers/trainees than initially anticipated 

(141,037, compared with a projected total of 77,606). In addition, they reported improving substantially 

more students and workers/trainees combined (41,481) than originally projected (27,502). 

Among students, the most common benefits that projects reported were primarily focused on teenagers 

and young adults—e.g., obtaining vocational and technical skills (36 percent), enrolling in a college or 

postsecondary program (31 percent), obtaining basic or academic skills in a specific subject (30 percent), 

or obtaining a postsecondary degree, credential, or certification (28 percent). Among workers/trainees, 

the most common benefit that projects reported was learning skills in a new area—including vocational 

and technical skills (37 percent), employability skills (36 percent), and basic or academic skills 

(25 percent). A smaller proportion of projects indicated that their efforts resulted in improved work 

status—with 27 percent indicating that workers/trainees obtained new full-time employment and 

21 percent reporting that workers increased their job status and/or earned increased wages.

Other outcomes reported by ARC education and workforce development projects. A total of 44 

projects reported creating 2,625 new jobs, of which more than half (1,484) were accounted for by 24 
Executive Summary

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 ix 

education projects. Thirty-five projects reported retaining 57,014 jobs—most of which (52,684) were 

reported by 17 workforce development projects. In addition: 

• Nine projects created a total of 75 businesses and 13 projects retained 133 businesses. 

• Five workforce development projects created 69 of the businesses, while eight education 

projects accounted for 53 of the businesses that were retained.

• Twenty-one projects leveraged a total of $120 million in private investments, of which 

$45 million was reported by 10 education projects and $75 million was reported by 11 

workforce projects. Two workforce and one education projects increased their nonexport 

revenues by $545,000, while one workforce project increased its export revenues by 

$10,000. Three workforce and one education projects reduced their costs by $2.2 million. 

• There was an 18 percentage point decline in the proportion of participants who were 

unemployed before and after participating in an ARC education and/or workforce 

development initiative. In addition, the proportion of participants who were employed on a 

part-time or full-time basis increased by 13 percentage points (from 21.4 percent to 

34.4 percent).

• Just over 27 percent of all former participants were making less than $40,000 per year after 

exiting their programs—while 5 percent were making between $40,000 and $69,999 per 

year. Projects were not able to provide wage information for 63 percent of all former 

participants.

Recommendations for Enhancing the ARC’s Collection and 

Utilization of Performance Measurement Data

Beyond the ARC.net requirements, almost 40 percent of projects tracked participants after program 

completion, with 18 percent tracking participants for one year and 19 percent tracking participants for two 

or more years. This finding suggests that a significant number of projects are making at least some effort 

to obtain longer-term data for a number of outcomes that are typically associated with education and 

workforce development projects.

Given the narrow focus of this study, we have limited our recommendations to four broad steps that 

would enhance the ARC’s capacity to document the full range of outcomes associated with its education 

and workforce development projects. These recommendations are designed to help the ARC maximize 

its use of the data that are already submitted into ARC.net, as well as to extend the reach of the ARC.net 

database to include ARC-supported outcomes that occur after a project’s grant has ended. 
Executive Summary

 x Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

• Recommendation 1: Develop a coding scheme in ARC.net to more precisely report on the 

accomplishments of education and workforce development projects. The ARC.net 

database currently requires that education and workforce development projects provide data 

on the number of students and/or workers/trainees served and improved by their efforts. In 

recent years, projects have also been asked to provide a narrative that describes how students 

and/or workers/trainees improved. While this information is useful, the ARC does not 

currently have procedures in place to codify and analyze the descriptions that are provided 

by projects. Such a process would enhance the Commission’s capacity to use this 

information to shed light on the range of outcomes associated with its portfolio of education 

and workforce development initiatives. It would also increase the likelihood that ARC 

personnel are able to make use of the narrative information supplied by projects as part of 

the existing ARC.net requirement. One option would be for the ARC to supplement 

ARC.net by providing a close-ended coding scheme that projects would use to classify their 

narrative response. Such a coding scheme could build upon the response options that were 

used in the online questionnaire.

• Recommendation 2: Expand the definition of what constitutes project success for the 

ARC’s education and workforce development projects. In addition to students and 

workers/trainees improved, the ARC.net database contains other performance measures that 

could be used to demonstrate the impact of the Commission’s education and workforce 

development activities—including participants improved, jobs created, jobs retained, 

businesses created, and businesses retained. In addition, other federal workforce 

development programs collect data on such outcomes as employment status and wages. 

Measuring these additional outcomes would enable the ARC to demonstrate a fuller range of 

outcomes associated with its education and workforce development projects.

In addition, the collection of data on employment status and wages would potentially allow 

the ARC to measure the economic impact of its education and workforce development 

projects on participating counties. An exploratory economic analysis conducted as part of 

this study found that the ARC’s education and workforce development projects provide 

benefits across the Appalachian region by substantially increasing participants’ income and 

employment opportunities. This analysis is an example of the type of analyses that could be 

conducted if more complete economic data were routinely obtained from ARC grant 

recipients. A more rigorous data reporting system would enable a more in-depth analysis, 

thereby allowing the ARC to differentiate among ARC states, local development districts, 

and specific program strategies (see Recommendation 3 below).

• Recommendation 3: Provide tools that education and workforce development projects can 

use to efficiently document an expanded set of outcomes. The data collection tools could 

be used to obtain basic information about participants’ current employment status, number of 

hours currently worked in an average week, current salary, and type of industry in which 

participants are currently employed. Collecting this information as part of their intake 

process would enable projects to assemble baseline characteristics of their participants. By 

continuing to collect this information over time (e.g., annually), projects would be able to 

document the number of participants that are enhancing their employment status and 

increasing their incomes.

• Develop procedures that encourage projects to report outcomes to the ARC after their 

grant awards have expired. Aside from validation visits conducted with a small sample of 

projects, the Commission currently has no way to systematically document any education 

and workforce development outcomes that occur after the ARC grant has expired. In order 
Executive Summary

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 xi 

to document these long-term outcomes, we recommend that the ARC continue to administer 

an annual data collection tool for up to five years after a project’s grant expires. Obtaining 

these data over time will enable the Commission to document the outcomes that are being 

leveraged through the long-term continuation of strategies that were created with the ARC’s 

seed money.
Executive Summary

 xii Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 1-1 

This report presents findings from an evaluation of the Appalachian Regional Commission’s (ARC’s) 

education and workforce development projects and provides recommendations for enhancing the 

Commission’s performance measurement process. It is based on an online questionnaire administered to 

386 education and workforce development projects funded between 2000 and 2008, as well as follow-up 

telephone interviews conducted with 23 of the projects that completed a questionnaire. 

1.1 Overview of the Appalachian Regional Commission’s 

Education and Workforce Development Projects

The ARC was established in 1965 as a regional economic development agency. The agency’s purpose is 

to improve social and economic conditions in the states and counties that make up the Appalachian 

region. The ARC strives to accomplish these improvements through federal-state-local partnerships that 

emphasize funding of grassroots initiatives, often providing initial project funding that would otherwise 

be unavailable. Despite the ARC’s efforts and substantial progress over the last four decades, Appalachia 

continues to battle economic distress, high poverty, unemployment, and educational disparity. The fiscal 

year (FY) 2013 Performance Budget Justification for the Appalachian Regional Commission (February 

2012) outlines the myriad of challenges facing students and workers in Appalachia:

In order to compete in the twenty-first century economy, the people of 

Appalachia must have skills and knowledge required to develop, staff, and 

manage globally competitive businesses. In addition, the Region’s communities 

must provide adequate healthcare in order to keep existing businesses and 

develop new ones. 

During the last decade, one-third of the region’s 420 counties lost population—

mostly in the northern and central counties, as well as in parts of Alabama and 

Mississippi. Additionally, the proportion of adults in Appalachia with a college 

degree is about two-thirds that of the nation, and the gap is widening. This 

phenomenon is partly due to college-educated young adults not returning to or 

settling in the region, and partly due to lower college-going rate among high 

school graduates. Roughly 12 percent of Central Appalachian residents ages 

25 and over have a completed a bachelor’s degree or more, compared to 27.5 

percent for the entire United States. 

Introduction 1
1 Introduction

 1-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

To address these challenges, the ARC has long funded education and workforce development projects 

that are designed to enhance residents’ educational attainment and workforce skills. These projects 

support a wide range of activities and services—including providing GED preparation training, providing 

computer training, purchasing equipment (e.g., computers), renovating structures, developing teacher 

training materials, providing job skills training, providing literacy training, providing job search 

assistance and career counseling, and providing social support services. The goals of these projects are to 

prepare workers and trainees to compete in the global economy, raise educational achievement and 

attainment levels for students, increase college-going rates, improve access to early childhood education 

programs, and prevent students from dropping out of school. 

Previous evaluations conducted in 1999 and 2001 provided considerable evidence that ARC-funded 

education and workforce development projects succeeded in bringing about a series of educational and 

employment gains throughout Appalachia. For example, a sample of 84 education projects funded during 

the 1990s enhanced learning opportunities, increased educational outcomes, and reached segments of 

Appalachia that were most in need—including persons in extreme poverty, persons who were 

geographically isolated, those who were unemployed or underemployed, youth at risk of dropping out of 

school, and persons with disabilities.2

 Additionally, a sample of 67 vocational education and workforce 

training projects funded between 1995 and 1999 provided anecdotal and quantitative confirmation that 

participants went on to improve their aptitudes and advance their employment status.3

1.2 Overview of the Study

In 2010, the ARC commissioned an evaluation of all education and workforce development projects 

funded between 2000 and 2008. The purposes of the study were to (1) obtain information about the 

characteristics of the Commission’s education and workforce development projects; (2) update 

performance measurement fields in the ARC.net database for the number of students and workers/trainees 

served and improved; (3) document the range of outcomes associated with these projects; (4) assess the 

feasibility of encouraging education and workforce development projects to report additional performance 

measures not currently recorded in the ARC.net database; and (5) develop recommendations for 

 

2 Silverstein, G., Plishker, L., Frechtling, J., Bartfai, N., and Snow, K. (2001). Evaluation of the Appalachian Regional Commission’s Education 

Projects (Prepared under contract to the Appalachian Regional Commission). Rockville, MD: Westat.

3 Silverstein, G., Plishker, L., and Frechtling, J. (2002). Evaluation of the Appalachian Regional Commission’s Vocational Education and 

Workforce Training Projects (Prepared under contract to the Appalachian Regional Commission). Rockville, MD: Westat.
Introduction 1 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 1-3 

enhancing the ARC’s collection and utilization of data from its education and workforce development 

projects.

This report presents findings from this evaluation and provides recommendations for enhancing the 

ARC’s performance measurement procedures. It is based on an online questionnaire administered to 386 

education and workforce development projects funded between 2000 and 2008, as well as follow-up 

telephone interviews conducted with 23 of the projects that completed a questionnaire. As part of the 

study, we also reviewed the procedures of comparable federal programs to learn more about the data 

collection and utilization practices of other education and workforce development initiatives. Information 

about each of these study methods is presented below. 

1.2.1 Document Review

At the outset of the study, we conducted a comprehensive review of the outputs and outcomes that have 

been used by the ARC and other agencies to measure the impact of education and vocational education 

initiatives. This review was primarily used to identify the range of outputs and outcomes that might be 

included on the online questionnaire that was to be administered with the ARC education and workforce 

development projects funded between 2000 and 2008. A secondary purpose was to identify potential 

outcomes that the ARC might use to define and track the progress of future program participants. 

As part of this review, we also examined the data collection practices of eight federal programs and one 

foundation program that are designed to provide education and workforce development services to similar 

populations. The primary purpose was to provide the ARC with information about the practices that other 

programs are using to collect data from former participants, as well as the level of effort required to 

collect and validate these data. 

Finally, we conducted a series of analyses of the data that ARC projects submitted into ARC.net, a grant 

management database that is used to obtain performance results from grantees at different stages 

throughout the project life cycle.4

 The primary purpose of this review was to assess the extent to which 

education and workforce development projects were able to provide information for a range of ARC 

 

4 Specifically, grantees provide estimates of projected outputs and outcomes during the grant approval process. At the end of the grant, actual 

project outputs and outcomes are submitted in final reports to the ARC.
1 Introduction

 1-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

performance measures. A secondary purpose was to allow for a comparison of the outputs and outcomes 

reported by projects in ARC.net and the online questionnaire. 

1.2.2 Online Questionnaire

Information from previous evaluations of ARC’s education and workforce development projects, as well 

as the review of other agencies, was used to develop an online questionnaire that could be administered 

with the study population. This questionnaire was designed to obtain descriptive statistics about projects’ 

characteristics and accomplishments. Data from the questionnaire were also used to conduct an 

exploratory economic analysis that focused on whether the ARC’s education and workforce development 

projects increased participants’ income and employment opportunities. Descriptive statistics are provided 

in Chapters 2 and 3, while the economic analysis is summarized in Appendix B.

In preparation for the administration of the questionnaire, we conducted a comprehensive review of the 

540 ARC projects in ARC.net that were categorized as education or workforce development between 

2000 and 2008. Following consultation with ARC staff, we identified grant recipients that received 

extensions to continue work initiated during their initial grant so the projects would only receive a single 

survey. By consolidating these grants, the number of education and workforce development projects that 

were identified as needing to be included in the questionnaire sample was reduced by 123 (from 540 to 

417). In addition, ARC staff identified 31 projects that should not complete the questionnaire (e.g., 

because they were cancelled or provided services that were not directly aligned with the outcomes of 

education and workforce development projects). As a result, a total of 386 education and workforce 

development projects were identified as needing to be included in the questionnaire sample. Of this 

number: 

• 26 (7 percent) were still open, while 360 (93 percent) were closed at the time the questionnaire 

was administered.

• 141 (37 percent) had actual data in ARC.net for students improved, 79 (21 percent) had actual 

data for workers/trainees improved, and 14 (4 percent) had actual data for participants improved. 

Very few projects had actual data for the remaining outcome fields in ARC.net.

• 68 education and workforce development projects did not have either projected or actual data in 

ARC.net for any of the following fields: students improved, workers/trainees improved, and 

participants improved. An additional 120 projects did not have actual data in ARC.net for any of 

the following fields: students improved, workers/trainees improved, and participants improved. 
Introduction 1 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 1-5 

• 37 individuals were responsible for two or more ARC education and workforce development 

awards between FY 2000 and FY 2008. These respondents were asked to complete between two

and nine questionnaires.

The online questionnaire opened on June 28, 2011, and closed on October 28, 2011. Throughout the 

four-month data collection period, Westat and the ARC took a series of steps to increase the overall 

response rate. For example, in early July, the ARC contacted the 12 individuals that had three or more 

questionnaires to complete. In addition, Westat field staff began contacting nonrespondents by phone in 

late July to urge their cooperation in the data collection effort.5

 A total of 222 projects (57.5 percent) 

ultimately completed the questionnaire. Not surprisingly, the response rate was lowest for projects that 

received their initial funding in 2000 or 2001 and highest for projects funded in 2005 or later (Table 1-1). 

Table 1-1. Questionnaire response rate, by year in which ARC education and 

workforce development projects were first funded

Year

Number of 

projects that 

received a 

questionnaire

Number of 

projects that 

completed a 

questionnaire

Response

rate

2000.................................................................. 55 22 40.0

2001.................................................................. 45 18 40.0

2002.................................................................. 39 25 64.1

2003.................................................................. 46 23 50.0

2004.................................................................. 42 23 54.8

2005.................................................................. 39 27 69.2

2006.................................................................. 35 24 68.6

2007.................................................................. 41 30 73.2

2008.................................................................. 44 30 68.2

Total ................................................................. 386 222 57.5

1.2.3 Telephone Interviews With a Sample of Questionnaire 

Respondents

Following the administration of the online questionnaire, a series of follow-up interviews was conducted 

with a purposeful sample of 23 ARC education and workforce development projects. These interviews 

were used to obtain more detailed information about the accomplishments associated with these 23 

projects, as well as to learn more about their efforts to collect outcome data from and about former 

participants. A summary of findings from the interviews is available in Appendix A.

 

5 Overall, Westat was unable to reach a total of 40 respondents representing 45 projects.
1 Introduction

 1-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Thirteen of the projects included in the follow-up interviews were selected because they reported on the 

online questionnaire that they had collected at least some outcome data from former participants. The 

interviews with these projects included questions about the type of data that were collected, the methods 

used to obtain and validate these data, and the ways in which projects made use of these data. The 

remaining 10 projects were selected because they indicated on the online questionnaire that they had not 

collected any follow-up data with former participants. The interviews with these projects included 

questions about the types of data that they would have liked to have collected from former participants, as 

well as the factors that prevented the collections of such data.

1.3 Structure of the Report

The remainder of this report presents substantive findings from the evaluation. Chapter 2 provides 

background information on the characteristics of the ARC’s education and workforce development 

projects, as well as data on the number of students and workers/trainees served and improved. Chapter 3 

examines the feasibility and value of looking beyond students and workers/trainees as measures of project 

success for the ARC’s education and workforce development projects. Chapter 4 provides a series of 

recommendations on how the ARC might enhance its capacity to document the range of outcomes 

associated with its education and workforce development projects.

Appendix A provides information about the practices of a sample of projects, their efforts to collect and 

utilize program participation/outcome data, and lessons learned that may be of interest to future ARC 

education and workforce development projects. Appendix B uses data from the online questionnaire to 

provide an example of the type of economic impact analyses that could be conducted should the ARC 

choose to collect additional outcome data from its education and workforce development projects. 

Appendices C and D provide our recommended format for new forms that could be used to collect 

baseline and annual data from ARC education and workforce development projects. Appendix E 

provides the online questionnaire that was administered to the 386 education and workforce development 

projects that received ARC funding between 2000 and 2008. Appendices F and G provide the telephone 

protocols that were used to collect supplemental information from those education and workforce 

development projects that indicated on the questionnaire that they did or did not collect data from former 

participants, respectively.
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-1 

Between 2000 and 2008, the ARC invested a total of $65 million in education and workforce 

development projects. In an effort to obtain information about the characteristics and outcomes of these 

projects, we administered an online questionnaire with the 386 projects that received ARC funding during 

the time period. A total of 222 projects (57.5 percent) completed the questionnaire.

6 Based on project 

type designations in ARC.net, 145 were education grants and 77 were workforce development grants.

7

This chapter provides information about these 222 ARC education and workforce development projects, 

with a special emphasis on the number of students and worker/trainees that were served and improved as 

a result of ARC-supported activities. Unless otherwise noted, the source for all figures and tables in this 

chapter is the Westat questionnaire of education and workforce development projects. 

2.1 Characteristics of ARC Education and Workforce 

Development Projects

The ARC funded a total of 386 education and workforce development projects between FY 2000 and 

FY 2008. As shown in Table 2-1, ARC invested a total of $65 million in these 386 projects during this 

time period—with projects raising an additional $77.4 million in matching funds ($3.7 million from other 

federal agencies and $73.7 million from state and local agencies) for a total of $142 million.8

 The 

average project received a total of $368,730—with $168,212 coming from the ARC, $9,529 from another 

federal agency, and $190,989 from state and local agencies. 

The 222 projects that completed a questionnaire represented approximately half of the total ARC 

expenditures for education and workforce development projects between FY 2000 and FY 2008. 

Specifically, the projects that completed a questionnaire received a total of $32.6 million (or an average of 

$146,819) from the ARC during this time period. These projects raised an average of $181,881 from 

other sources—including $7,091 from another federal agency and $174,790 from other state, local, and 

 

6 The online questionnaire opened on June 28, 2011, and closed on October 28, 2011. 

7 Education projects are generally designed to assist participants’ efforts to attain a diploma, degree, or certificate, while workforce development 

projects are generally designed to help participants attain employment or improve their job status.

8 The figures presented in this section only pertain to expenditures that occurred between FY 2000 and FY 2008, even though some of these 

projects received ARC funding prior to 2000 and/or after 2008.

Characteristics and Outcomes 

of the ARC’s Education and 

Workforce Development Projects 2
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

private agencies—for a total of $328,700. As such, the average total amount received by projects that 

completed the online questionnaire was $40,030 less than the amount received by all ARC education and 

workforce development projects for the same time period.

Table 2-1. Total and average funds allocated for ARC education and workforce 

development projects between FY 2000 and FY 2008, for all projects and 

projects that completed a questionnaire

Project characteristic

Number of 

ARC 

projects ARC funds

Matching funds

Federal State/local/private Total funds

Total expenditures

All ARC education and 

workforce development 

projects 386 $64,929,749 $3,678,001 $73,721,931 $142,329,681

Education and workforce 

development projects 

that completed a 

questionnaire 222 $32,593,802 $1,574,203 $38,803,366 $72,971,371

Mean expenditures

All ARC education and 

workforce development 

projects 386 $168,212 $9,529 $190,989 $368,730

Education and workforce 

development projects 

that completed a 

questionnaire 222 $146,819 $7,091 $174,790 $328,700

Difference between all 

projects and those that 

completed a 

questionnaire NA $21,393 $2,438 $16,199 $40,030

NA = not applicable.

SOURCE: ARC.net database.
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-3 

2.1.1 Populations Targeted by ARC Education and Workforce 

Development Projects

Of the 222 projects that completed a questionnaire, 65.3 percent were classified in the ARC.net database 

as being education, while 34.7 percent were classified as being workforce development. However, as 

shown in Table 2-2, 35 percent of all projects, 36 percent of education projects, and 31 percent of 

education projects were providing services to two or more target populations. For example, 12 percent of 

all projects were serving both youth and adults, while 9 percent were serving youth, adults, and teachers. 

Not surprisingly, education projects were more likely than workforce development projects to only serve 

youth (37 percent and 8 percent, respectively)—while workforce development projects were more likely 

than education projects to only serve adults (60 percent and 25 percent, respectively). Only a small 

proportion (2 percent) of all education and workforce development projects were designed to only provide 

services to teachers.

Table 2-2. Population served by ARC education and workforce development projects

Population served

Overall

(n=217)

Education 

(n=143)

Workforce 

development

(n=74)

Youth only............................................................... 27.2% 37.1% 8.1%

Adults only .............................................................. 36.4 24.5 59.5

Teachers only ......................................................... 1.8 2.1 1.4

Adults and teachers ................................................ 4.6 4.2 5.4

Youth and teachers................................................. 8.8 12.6 1.4

Youth and adults ..................................................... 12.4 9.1 18.9

Youth, adults, and teachers .................................... 8.8 10.5 5.4

NOTE: Totals may not sum to 100 due to rounding.

Overall, just under half (49 percent) of all the projects that responded to the questionnaire provided at 

least some of their services to adults (not including postsecondary students or teachers), while 44 percent 

served K–12 students and 36 percent served postsecondary students (Table 2-3). Approximately onefourth (24 percent) provided services to teachers, while a smaller proportion targeted services to 

preschool children (14 percent) or out-of-school youth (10 percent). In addition: 

• As would be expected, education projects were more likely than workforce development projects 

to be serving K–12 students and teachers. For example, over half (56 percent) of education 

projects were serving K–12 students, compared with 20 percent of workforce development 

projects.

• Conversely, workforce development projects were more likely than education projects to be 

serving out-of-school youth, postsecondary students, and adults. However, it should be noted that 
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

a substantial proportion of education projects were serving postsecondary students (29 percent) 

and adults (37 percent). 

• Education projects were more likely than workforce development projects to be serving teachers 

(29 percent and 14 percent, respectively).

Table 2-3. Types of populations targeted by ARC education and workforce 

development projects

Population type

Overall

(n=217)

Education 

(n=143)

Workforce 

development

(n=74)

Preschool children .................................................................. 14.3% 18.9% 5.4%

K–12 students......................................................................... 43.8 55.9 20.3

Out-of-school youth................................................................. 10.1 4.9 20.3

Postsecondary students.......................................................... 36.4 28.7 51.4

Adults (not including postsecondary students or teachers)..... 48.8 37.1 71.6

Teachers................................................................................. 24.0 29.4 13.5

Although education and workforce projects tended to serve specific populations types, no population, or 

combination of populations, was served exclusively by one project type or another. As such, one 

recommendation would be to revise ARC.net so that education and workforce development projects can 

be coded as having multiple target audiences (e.g., youth and adults, education and workforce 

development). 

2.1.2 Types of Services Provided by ARC Education and 

Workforce Development Projects

The ARC projects included in the study were designed to provide a wide range of educational and 

workforce development services. Over half (56 percent) of all projects were providing educational 

attainment or achievement services that were specifically targeted to students pursuing a high school 

diploma or postsecondary degree, while 43 percent were providing career and technical education 

services and 41 percent were providing workforce training skills (Table 2-4). Once again, data from the 

questionnaire suggests that some of the activities spanned the two project type designations. For 

example, over two-fifths (44 percent) of workforce development projects conducted educational 

attainment or achievement activities, while approximately one-third of education projects conducted 

career and technical education (40 percent) or workforce training (29 percent) activities.
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-5 

Table 2-4. General activities supported by ARC education and workforce development 

projects

Activity

Overall

(n=208)

Education 

(n=136)

Workforce 

development

(n=72)

Educational attainment or achievement (e.g., for students 

pursuing a high school diploma or postsecondary degree) ..... 56.3% 62.5% 44.4% 

Career and technical education (e.g., skills and job training 

that lead to a career credential or certification)........................ 43.3 39.7 50.0

Workforce training (e.g., skills and job training that do not lead 

to a diploma or degree) ........................................................... 41.3 29.4 63.9

Teacher training (e.g., skills enhancement for inservice and/or 

preservice teachers)................................................................ 29.8 36.0 18.1

Adult education (e.g., literacy/numeracy, GED, basic work 

skills, etc.)................................................................................ 19.7 16.9 25.0

Child development (e.g., child care, early childhood 

education)................................................................................ 14.9 19.9 5.6

In addition, as shown in Table 2-5, over half of all projects provided at least one of the following services: 

skills/training that enhanced employability (59 percent), special use classroom equipment such as 

computers or science labs (58 percent), and/or occupational/job skills training or instruction (53 percent). 

One-third (34 percent) provided computer skills instruction, while 23 percent provided or improved a 

physical structure. Only a few projects provided GED instruction (10 percent), provided referrals to other 

agencies for job assistance/career counseling (10 percent), or provided courses in parenting skills 

(4 percent). At the project type level, there was considerable overlap in supported activities. For 

example, a significant proportion of education projects provided skills/training designed to enhance 

employability (51 percent) or provided specific occupational/job skills training/instruction (42 percent), 

while 51 percent of workforce development projects provided special use classroom equipment.
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 2-5. Specific activities supported by ARC education and workforce 

development projects

Activity

Overall

(n=210)

Education 

(n=138)

Workforce 

development

(n=72)

Provided skills/training that enhanced employability ............ 58.6% 51.4% 72.2%

Provided special use classroom equipment 

(e.g., computers, networks, tools, equipment, science 

lab, etc.)............................................................................ 57.6 60.9 51.4

Provided specific occupational/job skills training or 

instruction......................................................................... 53.3 42.0 75.0

Developed or purchased educational materials 

(e.g., manuals, books, software, etc.)............................... 43.3 47.8 34.7

Established community or business partnerships ................ 36.7 34.1 41.7

Conducted community outreach activities............................ 36.2 39.1 30.6

Developed or purchased curriculum or instructional 

program............................................................................ 34.3 33.3 36.1

Provided computer skills training/instruction ........................ 33.8 31.9 37.5

Provided or improved physical structures (e.g., buildings, 

renovation, equipment, furniture, etc.).............................. 22.9 21.0 26.4

Provided distance learning infrastructure (e.g., software, 

equipment, technology).................................................... 20.5 20.3 20.8

Provided pedagogy or teaching skills training for 

teachers............................................................................ 19.5 24.6 9.7

Provided career counseling (e.g., discussions, 

diagnostic/aptitude testing)............................................... 17.1 18.8 13.9

Provided apprenticeship opportunities ................................. 14.8 10.1 23.6

Provided social support services.......................................... 12.4 13.8 9.7

Provided college counseling ................................................ 11.0 13.0 6.9

Provided instruction in business management..................... 10.5 8.0 15.3

Provided job search/placement assistance (e.g., job 

bank, employer outreach)................................................. 10.5 5.1 20.8

Distributed funds/mini-grants/stipends ................................. 10.0 12.3 5.6

Conducted GED preparation................................................ 9.5 6.5 15.3

Provided referrals to other agencies for job assistance/

career counseling............................................................. 9.5 5.8 16.7

Conducted course in parenting skills.................................... 4.3 5.1 2.8

Overall, most (92 percent) projects were providing at least two of the service types presented in 

Table 2-5—with the majority (79 percent) providing three or more services (Table 2-6). Only 8 percent 

of all projects reported that they only provided one service as part of their ARC education and workforce 

development grant. These patterns remained consistent across project types. This finding, compounded 

by the overlap in general and specific activities conducted by education and workforce development 

projects (Tables 2-4 and 2-5), reinforces the multifaceted focus of ARC education and workforce 

development projects that was noted in findings corresponding to Tables 2-2 and 2-3. 
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-7 

Table 2-6. Proportion of ARC education and workforce development projects 

providing multiple services

Activity

Overall

(n=210)

Education 

(n=138)

Workforce 

development

(n=72)

Provided one service.............................................................. 8.0% 9.4% 5.6%

Provided two services ............................................................ 13.0 13.0 15.3

Provided three or more services ............................................ 79.0 77.5 79.2

NOTE: Totals may not sum to 100 due to rounding.

2.2 Information About Students and Workers/Trainees 

Served and Improved

As part of the ARC application process, education and workforce development projects are expected to 

provide an estimate of the number of students and/or workers/trainees that will ultimately be served and 

improved by their efforts. In an effort to update the information in ARC.net, project grantees were asked 

to provide the cumulative number of students and workers/trainees served and improved since the 

inception of their ARC grant. These terms were defined as follows:

• Students served—the cumulative total number of students who participated in your ARC project, 

from project inception until now. For the purposes of this questionnaire, students include children 

and youth in prekindergarten programs through 12th grade, as well as adults in postsecondary 

educational programs. 

• Students improved—the cumulative total number of student participants who (1) enhanced their 

knowledge or skills; (2) passed or graduated to the next grade or level necessary to continue their 

education; (3) received a career credential; (4) made progress toward a degree, diploma, or 

certification; and/or (5) obtained a job in the field for which they were specifically trained.

• Workers/trainees served—the cumulative total number of individuals that received training or 

participated in an activity designed to enhance their employability, but not necessarily leading to 

a certification, diploma, or degree. 

• Workers/trainees improved—the cumulative total number of participants (1) with new/ 

improved knowledge or skills; (2) that received an educational credential; (3) that received a 

career credential; and/or (4) that obtained/enhanced their employment status (e.g., received higher 

pay or better positions). 
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Not surprisingly, projects were more likely to be able to provide information about the number served 

(i.e., participated in an activity) than the number improved (i.e., benefited from participating in an 

activity). This difference likely reflects the ease of counting the number served compared with the added 

burden associated with tracking participants’ progress after they completed an activity to determine the 

number improved. Specifically, as shown in Table 2-7: 

• Almost two-thirds (64 percent) of all projects provided information about the number of students 

that participated in an ARC-supported activity. Only 8 percent indicated that they served students 

but did not track this output. However, fewer than half (49 percent) provided information on the 

number of students improved and 32 percent skipped this item (suggesting they were not able to 

provide information about the number of students that benefited as a result of their ARC project). 

• Approximately one-third (32 percent) of all projects provided information about the number of 

workers/trainees served, while 12 percent did not track this output and 16 percent indicated that 

their projects were not designed to serve workers/trainees. Only 22 percent provided information 

on the number of workers/trainees improved as a result of their ARC project, while 46 percent 

skipped this item. 

• Education projects were more likely than workforce development projects to provide 

questionnaire data for the number of students served and improved. Conversely, workforce 

development projects were more likely than education projects to provide questionnaire data for 

the number of workers/trainees served and improved.

Table 2-7. Proportion of ARC education and workforce development projects that 

provided information about students and workers/trainees served and 

improved

Output or outcome

Provided a 

number 

Output or 

outcome not 

applicable

Did not track 

information

Skipped 

item

Overall 

(n=222)

Students served ..................... 63.5% 7.2% 7.7% 21.6%

Students improved ................. 48.6 9.0 10.4 32.0

Workers/trainees served......... 32.0 16.2 12.2 39.6 

Workers/trainees improved..... 22.1 17.1 15.3 45.5 

Education 

(n=145)

Students served ..................... 71.7 6.2 8.3 13.1

Students improved ................. 51.0 9.7 12.4 26.2

Workers/trainees served......... 29.0 18.6 11.7 40.7

Workers/trainees improved..... 16.6 20.0 16.6 46.2

Workforce 

development 

(n=77)

Students served ..................... 48.1 9.1 6.5 36.4

Students improved ................. 44.2 7.8 6.5 41.6

Workers/trainees served......... 37.7 11.7 13.0 37.7

Workers/trainees improved..... 32.5 11.7 13.0 42.9

NOTE: Total for each output or outcome may not sum to 100 due to rounding.
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-9 

2.2.1 Number of Individuals Served and Improved

At the time the questionnaire was administered (summer 2011), the 222 ARC education and workforce 

development projects funded between 2000 and 2008 reported that they had served a total of 414,296 

students and workers/trainees—including 334,803 students and 79,493 workers/trainees (Table 2-8).9

 

Education projects served the vast majority of students. Of the 334,803 total students served, 322,204 (96 

percent) were from education projects and 12,599 (4 percent) were from workforce development projects. 

In addition, education projects served almost half (47 percent) of the 79,493 workers/trainees reported on 

the questionnaire—with workforce development projects serving 53 percent.

Of the total served, 99,809 individuals were reported to have “improved’ from their participation—

including 81,974 students and 17,835 workers/trainees. Thus, approximately 25 percent of the students 

and 22 percent of the workers/trainees served were reported by projects to have benefited from their 

participation in an ARC education and workforce development activity. Notably, 40 percent of 

individuals served by workforce development projects were reported to have improved; compared to 22 

percent of individuals reported by education projects. However, the use of a single generic category (i.e., 

students and workers/trainees improved) to document the range of potential improvements for education 

and workforce development participants makes it difficult to interpret these findings in a meaningful 

manner. This is because some projects may have used a singular or rigorous definition as to what 

constituted an improvement (e.g., completion of a degree or new employment status), while others may 

have used multiple measures—or relied on a more imprecise standard (e.g., made progress toward a 

degree, gained a new skill). Combining these diverse improvements into a single statistic (e.g., students 

improved) makes it difficult to portray the nature of these improvements or make meaningful crossproject comparisons of the proportion of participants that were ultimately improved. It also potentially 

fails to capture information about tangible benefits (e.g., attainment of a new skill) that went 

undocumented in projects that were solely focused on reporting data for a specific improvement (e.g., 

receiving a career credential).

In an effort to benchmark these findings, we compared the number of individuals that projects expected to 

serve and improve (as reported in ARC.net at the time projects applied for ARC funding) with the number 

 

9 For the purposes of the questionnaire, students included children and youth in prekindergarten programs through 12th grade, as well as adults in 

postsecondary educational programs. Workers/trainees included individuals who received training or participated in an activity designed to 

enhance their employability.
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

that were reported as being served and improved on the Westat questionnaire.10 As shown in Table 2.9, 

the ARC projects served more students and workers/trainees than initially anticipated (141,037, compared 

with a projected total of 77,606). In addition, projects reported improving substantially more students and 

workers/trainees combined (41,481) than originally projected (27,502). These overall statistics mask 

several noteworthy trends: 

• Education projects exceeded their anticipated targets for the number of students and workers/ 

trainees served. They also reported improving substantially more students (33,105) than 

originally anticipated (17,298). 

• Workforce development projects served and improved fewer students than projected—serving 

7,105 (80 percent) of their expected target of 8,897 students, and improving 3,871 (55 percent) of 

their expected target of 7,007 students. However, they served and improved more workers/ 

trainees than was originally anticipated—serving three times as many workers/trainees and 

improving 1,310 more workers/trainees than expected. 

While these findings suggest that education and workforce development projects differed in their ability 

to anticipate the number of students and workers/trainees they would serve and improve, these trends may 

also reflect the small number of projects for which some of these comparisons could be made. For 

example, only three education projects and thirteen workforce development projects provided both 

projected and actual data for workers/trainees improved. As such, the small number of workforce 

development projects for which projected and actual data are available makes it difficult to interpret the 

comparisons of numbers served and improved. As is discussed in Chapter 4, collecting data on discrete 

categories (e.g., anticipated and actual number of students that attain a degree) would increase the 

capacity of the ARC to compare baseline projections with actual data in a valid and reliable manner.

 

10Two important caveats regarding this comparison are worth noting. First, this comparison excludes the 32 ARC education and workforce 

development projects with multiple entries in ARC.net (e.g., due to a contract extension or revision), as well as those projects that did not 

provide data in both ARC.net and the Westat questionnaire for a given output or outcome. Second, any differences between projected and 

actual data may reflect the differing timeframes for which information was requested. Specifically, the ARC.net baseline projections cover the 

life of the ARC grant, while the questionnaire data also include those served and improved since the expiration of the ARC grant.
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-11 

Table 2-8. Number of students and workers/trainees that received services and 

benefited as a result of ARC education and workforce development 

projects

Target population Number served

Improved

Number

Percent of those 

served that were 

documented as 

having improved

Overall 

(n=222)

Students..................................... 334,803 81,974 24.5

Workers/trainees........................ 79,493 17,835 22.4

Total........................................... 414,296 99,809 24.1

Education 

(n=145)

Students..................................... 322,204 75,085 23.3

Workers/trainees........................ 37,608 3,040 8.1

Total........................................... 359,812 78,125 21.7

Workforce 

development 

(n=77)

Students..................................... 12,599 6,889 54.7

Workers/trainees........................ 41,885 14,795 35.3

Total........................................... 54,484 21,684 39.8

Table 2-9. Projects with corresponding output and outcome records in both the 

questionnaire and ARC.net

Output or outcome

Projected 

ARC.net total

Actual 

questionnaire 

total Total ratio

Overall

Students served (n=96) .................. 66,097 110,020 1.66

Students improved (n=67) .............. 24,305 36,976 1.52

Workers/trainees served (n=20)...... 11,509 31,017 2.70

Workers/trainees improved (n=16).. 3,197 4,505 1.41

Education

Students served (n=77) .................. 57,200 102,915 1.80

Students improved (n=50) .............. 17,298 33,105 1.91

Workers/trainees served (n=6)........ 1,925 2,157 1.12

Workers/trainees improved (n=3).... 935 933 1.00

Workforce 

development

Students served (n=19) .................. 8,897 7,105 0.80

Students improved (n=17) .............. 7,007 3,871 0.55

Workers/trainees served (n=14)...... 9,584 28,860 3.01

Workers/trainees improved (n=13).. 2,262 3,572 1.58

NOTE: This table excludes the 32 ARC education and workforce development projects with multiple entries in ARC.net (e.g., due to 

a contract extension or revision), as well as those projects that did not provide data in both ARC.net and the Westat questionnaire 

for a given output or outcome. 

SOURCE: ARC.net database and Westat questionnaire of ARC education and workforce development projects.
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

2.2.2 How Individuals Were Served and Improved

In light of the diverse ways in which students and workers/trainees could be counted as having 

“improved,” the questionnaire was also designed to obtain information about how students and 

workers/trainees were benefited by their ARC project.11 Among students, the most common benefits that 

all projects reported were primarily focused on teenagers and young adults—e.g., obtaining vocational 

and technical skills (36 percent); enrolling in a college or postsecondary program (31 percent); obtaining 

basic or academic skills in a specific subject (30 percent); or obtaining a postsecondary degree, credential, 

or certification (28 percent) (Table 2-10). Only a few projects reported that their efforts led to students 

obtaining a high school diploma, GED, or equivalent (14 percent); increasing scores on a statewide 

assessment (14 percent); decreasing suspensions or other behavioral problems (12 percent); or returning 

to school after dropping out (7 percent). In addition: 

• Student outcomes most commonly reported by education projects were improved school 

readiness (34 percent), obtaining basic or academic skills in a specific subject (33 percent), 

enrolling in a more challenging class (31 percent), and enrolling in college or a post-secondary 

program (30 percent). 

• Student outcomes most commonly reported by workforce projects were obtaining vocational and 

technical skills (50 percent), obtaining a postsecondary degree, credential, or certification (39 

percent), and enrolling in college or a postsecondary program (33 percent). 

• Thirty percent of workforce projects and 14 percent of education projects reported that they had 

no student outcomes. 

As shown in Table 2-11, at least one-fourth of all projects reported that their workers/trainees learned new 

skills in a new area—including vocational and technical skills (37 percent), employability skills (36 

percent), and basic or academic skills (25 percent). A smaller proportion indicated that their efforts 

resulted in improved work status—with 27 percent indicating that workers/trainees obtained new full-time 

employment and 21 percent reporting that workers increased their job status and/or earned increased 

wages. A minority of projects indicated that the workers/trainees they served reduced their dependence 

on public assistance (14 percent) or obtained/maintained employer-provided benefits (6 percent). In 

addition: 

 

11Because projects could report multiple ways in which their students and workers/trainees were improved, it is not possible to reliably apportion 

the number of students and workers/trainees that were improved across the categories in Tables 2-10 and 2-11. 
Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 2 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 2-13 

• Over half of workforce projects reported that workers/trainees obtained vocational and technical 

skills (55 percent) and other employability skills (55 percent). Approximately one-quarter of 

education projects reported the same impacts. 

• Forty-one percent of education projects and 22 percent of workforce projects reported that they 

had no student outcomes. 

Table 2-10. How students were improved by their participation in ARC education and 

workforce development projects

Student improvement Overall (n=194)

Education 

(n=130)

Workforce 

development 

(n=64)

Obtained vocational and technical skills ...................................... 36.1% 29.2% 50.0%

Enrolled in college or postsecondary program............................. 30.9 30.0 32.8

Obtained basic or academic skills in a specific subject................ 29.9 33.1 23.4

Obtained a postsecondary degree, credential, or certification ..... 28.4 23.1 39.1

Improved school readiness .......................................................... 27.3 33.8 14.1

Enrolled in more challenging classes........................................... 26.3 30.8 17.2

Remained in school when at risk of dropping out......................... 22.7 26.2 15.6

Improved attendance on a daily/regular basis.............................. 22.2 26.2 14.1

Obtained a high school diploma, GED, or equivalent................... 14.4 16.2 10.9

Increased scores on statewide assessments............................... 14.4 17.7 7.8

Advanced to next grade level....................................................... 12.9 14.6 9.4

Decreased suspensions and other problem behaviors ................ 12.4 16.2 4.7

Tested at or above grade level..................................................... 8.8 11.5 3.1

Re-entered an educational program after dropping out ............... 7.2 6.9 7.8

None—project did not have any student outcomes...................... 19.1 13.8 29.7

Table 2-11. How workers/trainees were improved by their participation in ARC 

education and workforce development projects

Worker/trainee improvement

Overall 

(n=192)

Education 

(n=123)

Workforce 

development 

(n=69)

Obtained vocational and technical skills .......................................... 36.5% 26.0% 55.1%

Obtained other employability skills (e.g., work attitudes/habits)....... 35.9 25.2 55.1

Obtained new full-time employment................................................. 27.1 17.1 44.9

Obtained basic or academic skills.................................................... 25.0 18.7 36.2

Maintained current employment....................................................... 20.3 12.2 34.8

Obtained new part-time employment ............................................... 19.8 13.0 31.9

Increased job status and/or earned increased wages...................... 21.4 16.3 30.4

Reduced dependence on public assistance..................................... 14.1 8.1 24.6

Teachers or instructors enhanced their classroom practices ........... 29.7 34.1 21.7

Retrained in another field and obtained new employment ............... 10.9 5.7 20.3

Obtained/maintained employer-provided health benefits ................. 5.7 2.4 11.6

None—project did not have any worker/trainee outcomes............... 33.9 40.7 21.7
2 Characteristics and Outcomes of the ARC’s Education and 

Workforce Development Projects 

 2-14 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-1 

The impact of the ARC’s education and workforce development projects is primarily documented through 

two aggregate measures: students improved and workers/trainees improved. The numbers obtained 

through these two measures contribute to the Commission’s long-term 12-year performance goal that 

240,000 citizens of the Region will benefit from enhanced education and job-related skills (Moving 

Appalachia Forward, the Commission’s strategic plan for 2011–16). Although ARC.net includes a 

narrative field that captures information on how students and workers were improved, this information 

was only available for a small number of the education and workforce development projects that were 

funded between 2000 and 2008. As discussed in Chapter 2, findings from the online questionnaire 

suggest that the improvements experienced by students and workers/trainees included such tangible 

outcomes as obtaining a high school diploma or postsecondary degree, increasing scores on statewide 

assessments, decreasing suspensions or other behavioral problems, learning new skills, improving work 

status and/or wages, reducing dependence on public assistance, and obtaining/maintaining employerprovided benefits. 

Additional information on the accomplishments of education and workforce development projects was 

obtained through a series of follow-up interviews with a purposeful sample of 23 projects that completed 

the questionnaire (see Appendix A for a more detailed discussion of the findings from these follow-up 

interviews). The most commonly cited project accomplishment was improved access to educational 

services for participating students (13 projects) through which distressed and previously unserved rural 

communities received services that would otherwise have been unavailable.12 Additional 

accomplishments involved improving the relationship between industry and academia (five projects), 

workers being trained and hired in industry (five projects), securing additional funding to continue efforts 

begun with ARC seed money (four projects), and effective grant management (two projects). Notably, 

four of the 23 projects indicated that their efforts resulted in workers being trained and hired for industry. 

 

12Three of these 13 projects established a link between secondary schools and higher education institutions, resulting in increased standards for 

participating students. Specifically, one project focused on reducing the high school dropout rate by rewarding attendance, exposing students to 

postsecondary education opportunities, providing rewards for good grades, and helping students stay on track to graduate. The second project 

raised standards by implementing dual-credit courses that allowed students to earn college credit in the medical assisting health field while 

completing their high school diplomas. The third project increased the number of Advanced Placement (AP) courses offered, the number of 

students enrolled in AP courses, and the number of students earning quality scores on AP exams. According to the project director, this raised 

standards because “the inclusion of AP courses ...added more rigor into the typical course offerings and ... increased the quality of teachers in 

those schools.”

Examination of Other Potential Project 

Accomplishments for the ARC’s Education 

and Workforce Development Projects 3
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Two projects reported job placement data for all of their program completers across all curricula 

offerings, including Adult Basic Education. A third project placed workers in construction trades, 

HVAC, welding, and building construction. The fourth project placed students in the health care sector. 

The range of improvements reported by these projects matches the types of performance measures that 

are collected by other federal and foundation programs that support similar education and workforce 

development initiatives. However, as shown in Exhibit 3-1, other federal programs have adopted specific 

performance measures for their grant recipients that focus on specific and unambiguous outcomes— 

e.g., number of high school graduates that enroll in college, number of postsecondary students who 

advance from remedial to credit-bearing courses, number of adults who attain a GED, and number of 

adults/families with increased wages. These programs are typically larger and address a more narrow set 

of outcomes than the ARC’s education and workforce development projects. Nonetheless, the data 

collected by these programs suggest some additional performance measures that the ARC might want to 

use to demonstrate the impact of its portfolio on a range of educational and economic outcomes. 

This chapter examines the feasibility and potential value of expanding the number of outcome measures 

for the ARC’s education and workforce development projects. The purpose is to provide the Commission 

with guidance on whether there are more precise indicators that could be used to document the benefits 

associated with the strategies used by education and workforce development projects, as well as to 

explore ways in which these specific indicators could be used to calculate the impact of these projects on 

the economies of participating communities. 
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-3 

Exhibit 3-1. Examples of outcomes for other education and workforce development 

projects

Category Output/outcome Source

Secondary 

school 

students

Number of secondary school dropouts who re-enter a secondary education 

program

TS

Number of secondary school students who exhibit a decrease in suspensions 

and other problem behaviors

GEAR-UP 

(interim)

Number of high school students enrolled in at least one Advanced 

Placement and/or International Baccalaureate course Lumina

Number of secondary school students who graduate from high school DoL ETA, TS, 

Perkins, Lumina

Number of high school students who intend to enroll in college TS, Lumina

Number of high school graduates that enroll in college

TS, GEAR-UP 

(long-term), 

Perkins, Lumina

Number of low income and first generation high school students who intend 

to enroll in college Lumina

Number of low income and first generation high school students who enroll 

in college Lumina

Number of high school graduates that apply for financial aid for 

postsecondary education (e.g., submission of FAFSA) TS, Lumina

Number of high school graduates that receive financial aid for 

postsecondary education (e.g., receipt of FAFSA) Lumina

Postsecondary 

students

Number of postsecondary “stopouts” who re-enter a postsecondary 

education program TS

Number of postsecondary students who advance from remedial to creditbearing courses Lumina

Number of postsecondary students who successfully complete gatekeeper 

courses

Lumina

Number of postsecondary students who earn an associate’s, bachelor’s, or 

higher degree or certificate

SSS, Perkins, 

Lumina

Number of postsecondary students who attain career and technical skill 

proficiencies Perkins

Number of postsecondary students who complete a postsecondary skills 

training program Perkins

Number of first-generation and/or low-income adults who apply to a 

postsecondary education program EOC

Number of postsecondary students who transfer from 2- to 4-year 

institutions SSS, Perkins

Number of postsecondary students who enroll in graduate school McNair
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Exhibit 3-1. Examples of outcomes for other education and workforce development 

projects—Continued

Category Output/outcome Source

Adults

Number of adults in career and technical education programs that obtain 

employment in nontraditional fields Perkins

Number of adults who attain a GED DoL ETA,

HC & HG

Number of adults who attain a vocational degree/credential/certificate DoL ETA, 

HC & HG

Number of adults who obtain employment DoL ETA, 

HC & HG

Number of adults who retain employment DoL ETA, 

HC & HG

Number of adults/families with increased wages DoL ETA

SOURCES:

DoL ETA: Common Measures Policy for the Employment and Training Administration’s (ETA) Performance Accountability System 

and Related Performance Issues (2006, TEGL 17-05). 

GEAR-UP: Early Outcomes of the GEAR UP Program (2008, Westat). 

HC & HG: American Recovery and Reinvestment Act of 2009: Healthcare and Other High Growth Emerging Industries Grants (no 

bib info). 

McNair: Ronald E. McNair Postbaccalaureate Achievement Program: 2002-05 Facts and Figures at a Glance (2007). 

Perkins: The Carl D. Perkins Career and Technical Education Act of 2006: Report to Congress on State Performance Program Year 

2007-08 (2010). 

SSS: National Evaluation of Student Support Services: Examination of Student Outcomes After Six Years (2010, Westat). 

TS: An Interim Report on the Talent Search Program: 2001-03 and 2003-04, with Select Data from 2000-02 (2006, American 

Institute for Research). 

Lumina: Indicators and Internal Data Sources for Lumina’s Key Strategic Areas (2010, Lumina Foundation for Education). 

EOC: A Report on the Educational Opportunity Centers Program: 2007-2008, With Select Comparative Data, 2002-07. 

3.1 Extent to Which ARC Education and Workforce 

Development Projects Are Tracking Long-Term 

Participant Outcomes

An important purpose of the study was to determine whether education and workforce development 

projects were obtaining longer-term outcome data about their participants and, if so, the procedures they 

were using to obtain and validate these data. As shown in Table 3-1, almost two-fifths of all projects 

tracked participants after program completion, with 18 percent tracking participants for one year and 

19 percent tracking participants for two or more years. The remaining projects either did not track 

program participants (29 percent) or indicated that such tracking was not applicable to the types of 
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-5 

services they provided (33 percent). Workforce development projects were more likely than education 

projects to have tracked participants after program completion—with 47 percent of workforce 

development projects tracking participants for at least one year (compared with 32 percent for education 

projects). 

Table 3-1. Length of time that ARC education and workforce development projects 

tracked participants after program completion

Length of time

Overall

(n=204)

Education 

(n=136)

Workforce 

development

(n=68)

Did not track the progress of program participants after 

completing the program........................................................... 29.4% 31.6% 25.0%

For up to 1 year after completing the program............................ 18.1 14.0 26.5

For up to 2 years after completing the program .......................... 7.8 7.4 8.8

For more than 2 years after completing the program.................. 11.3 11.0 11.8

Not applicable ............................................................................. 33.3 36.0 27.9

NOTE: Totals may not sum to 100 due to rounding. 

This finding suggests that a significant number of projects are making at least some effort to obtain 

longer-term data for a number of outcomes that are typically associated with education and workforce 

development projects. This is noteworthy, given that the ARC does not require collection of longer-term 

outcome data from former participants or from projects that are no longer funded. To understand the 

conditions under which these data were collected, we conducted follow-up interviews with a small sample 

of questionnaire respondents to determine why projects chose to collect outcome data from former 

participants when they were not required to do so. Almost all (12 of 13 projects) indicated that they 

collected data from former participants because of a requirement of an agency other than the ARC. 

Several projects indicated that they collected outcome data to document the project’s effectiveness and 

efficiency for local stakeholders.13 While some projects used participant surveys to obtain their outcome 

data, others relied on reviews of extant databases:

• As shown in Exhibit A-1 (at the end of Appendix A), four projects were collecting information 

about former participants’ employment and wage status. Two of these projects were using 

surveys to obtain these data directly from students. A third used faculty members’ unofficial 

knowledge about the employment status of their former students to prepare an annual report for 

the state department of education, while a fourth project used a database maintained by a state 

agency (established to align universities, government, and industry resources) to mine existing 

data related to individual taxes, unemployment, and education enrollment.

 

13For example, two projects kept data to document the effectiveness (or lack thereof) of interventions (one to increase the college-going rate of 

students and the other to measure the impact of technology on instruction). A third project was of particular interest to the governor’s office, 

and staff wanted to document the initiative’s impact on adult literacy levels. A fourth project kept data on spending to document fiscal 

responsibility to stakeholders and political watchdogs. 
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

• Three projects tracked the numbers of high school graduates who attended a postsecondary 

institution. Two of these projects conducted surveys of former participants to obtain this 

information—with both projects reporting that low response rates and the lack of validation 

methods minimized the value of the data for project monitoring and program planning. A third 

project used the National Student Clearinghouse (NSC) to examine students’ enrollment in 

postsecondary education.

• Finally, three projects used existing databases to obtain information about the number of former 

participants that had taken a high school AP course/exam, passed a licensing or certification 

exam, or passed their GED exam. 

Regardless of the method used, projects generally acknowledged that their efforts were often hindered by 

low response rates, lack of staff time to conduct follow-up with nonrespondents, and a reliance on selfreported data on employment status that were difficult to validate. In addition, projects generally 

indicated that while they would have liked to collect wage data from (or about) their former participants, 

most respondents were likely unwilling to provide such information about themselves. Three projects 

indicated they saw a privacy issue, reflected by students’ “none of your business” answers on surveys. 

Another project indicated they could not ask for any personal data from students because of privacy laws. 

Projects also discussed the difficulty of using third-party systems to obtain information about former 

participants. For example, two projects indicated that they were unable to obtain data about participants 

that had moved to another state—with one project indicating that it was only able to track students’ 

enrollment at in-state universities and colleges.

Projects made multiple uses of data about former participants, including marketing, operations, 

documentation for accreditation, and as a basis for generating additional funding. For example, four of 

the 13 projects were using data for program improvement initiatives, with two projects using the data to 

evaluate the efficacy of training programs (e.g., by demonstrating the skills that students had learned). 

One project used participant feedback to improve campus services and another used student achievement 

data to research the effectiveness of traditional classroom instruction as compared to distance learning.

Appendix A provides additional information about the conditions under which projects collected and 

utilized outcome data. 
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-7 

3.2 Other Items in ARC.net that Might be Reported for 

Education and Workforce Development Projects

In addition to students improved and workers/trainees improved, the ARC has developed other 

performance measures that are used to document the Commission’s progress toward its long-term goals. 

Some of these performance measures potentially pertain to education and workforce development 

activities. Nonetheless, a review of ARC.net reveals that very few education and workforce development 

projects provided data for any of these measures. For example, 11 percent of education and workforce 

development projects had data in ARC.net for participants served, while 4 percent had data for 

participants improved and 4 percent had data for jobs created (Table 3-2). This finding that few projects 

had any entries in ARC.net for these performance measures is not surprising, given that the ARC does not 

formally track these outcomes as part of their performance goal pertaining to citizens benefiting from 

enhanced education and job-related skills. 

Table 3-2. Whether ARC education and workforce development projects provided 

information about other performance measures in ARC.net: 2000–2008

Output or outcome

Provided a number in ARC.net

Number Percent

Overall 

(n=190)

Participants served................................... 20 10.5 

Participants improved............................... 7 3.7 

Jobs created............................................. 7 3.7 

Jobs retained............................................ 4 2.1 

Businesses created.................................. 3 1.6 

Businesses retained ................................. 0 0.0 

Leveraged private investments................. 0 0.0 

Revenues increased (nonexport) ............. 0 0.0 

Revenues increased (export) ................... 0 0.0 

Costs reduced .......................................... 0 0.0 

Education 

(n=121)

Participants served................................... 12 9.9 

Participants improved............................... 6 5.0 

Jobs created............................................. 5 4.1 

Jobs retained............................................ 3 2.5 

Businesses created.................................. 2 1.7 

Businesses retained ................................. 0 0.0 

Leveraged private investments................. 0 0.0 

Revenues increased (nonexport) ............. 0 0.0 

Revenues increased (export) ................... 0 0.0 

Costs reduced .......................................... 0 0.0 
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 3-2. Whether ARC education and workforce development projects provided 

information about other performance measures in ARC.net: 2000–2008—

Continued

Output or outcome

Provided a number in ARC.net

Number Percent

Workforce 

development 

(n=69)

Participants served................................... 8 11.6 

Participants improved............................... 1 1.4 

Jobs created............................................. 2 2.9 

Jobs retained............................................ 1 1.4 

Businesses created.................................. 1 1.4 

Businesses retained ................................. 0 0.0 

Leveraged private investments................. 0 0.0 

Revenues increased (nonexport) ............. 0 0.0 

Revenues increased (export) ................... 0 0.0 

Costs reduced .......................................... 0 0.0 

NOTE: This table excludes the 32 ARC education and workforce development projects with multiple entries in ARC.net (e.g., due to 

a contract extension or revision). 

In an effort to assess whether projects were, in fact, able to provide data for these additional ARC.net 

measures, we included items in the online questionnaire designed to document the number of projects that 

had data for each of these outcomes. As shown in Table 3-3, at least 5 percent of all projects provided 

numbers for each of these measures—with 42 percent providing data for participants served, 25 percent 

for participants improved, 23 percent for jobs created, and 19 percent for jobs retained. This finding that 

some projects were able to provide data for these other categories in ARC.net suggests that the ARC 

should consider collecting these data and documenting these outcomes from future education and 

workforce development projects. Specific numbers reported for these measures are summarized below:

• The total number of participants served by all projects was 295,277, of which 40,209 (14 percent) 

were reported to have improved (Table 3-4). However, this finding reflecting the small 

proportion of participants that were improved may reflect that projects were unable to fully 

account for the diverse ways in which participants benefited as a result of their involvement in an 

ARC-supported activity.

• Education projects accounted for the majority of participants served (281,382) and improved 

(28,488). Although workforce development projects reported serving significantly fewer 

participants (13,895), they reported that a far greater proportion of those served were improved 

(11,721)—that is, 84 percent of workforce development participants were reported to have 

improved compared with only 10 percent of education participants. This difference between the 

two project types may reflect the relative ease with which workforce development projects were 

able to obtain follow-up data on their participants. 

• A total of 44 projects reported creating 2,625 new jobs, of which more than half (1,484) were 

reported by 24 education projects (Table 3-5). 
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-9 

• A total of 35 projects reported retaining 57,014 jobs—most of which (52,684) were reported by 

17 workforce development projects. 

• Nine projects created a total of 75 businesses and 13 projects retained 133 businesses. Five 

workforce development projects created 69 of the businesses, while eight education projects 

accounted for two-fifths (53) of the businesses that were retained. 

• Twenty-one projects leveraged a total of $120 million in private investments, of which 

$46 million was reported by 10 education projects and $75 million was reported by 11 workforce 

projects.

• Two workforce and one education projects together increased their nonexport revenues by 

$545,000, while one workforce project increased its export revenues by $10,000. 

• Three workforce and one education projects reduced their costs by $2.2 million. 

Table 3-3. Whether ARC education and workforce development projects provided 

information about other ARC.net performance measures on the Westat 

questionnaire

Output or outcome

Provided a 

number 

Output or 

outcome not 

applicable

Did not track 

information

Skipped 

item

Overall 

(n=222)

Participants served................. 41.9% 14.9% 9.5% 33.8%

Participants improved............. 25.2 16.7 14.9 43.2

Jobs created........................... 23.0 27.9 21.2 27.9

Jobs retained.......................... 18.5 24.3 18.5 38.7

Businesses created................ 9.5 28.8 17.6 44.1

Businesses retained ............... 10.8 27.9 17.6 43.7 

Leveraged private 

investments......................... 12.2 31.1 14.4 42.3

Revenues increased 

(nonexport).......................... 5.0 33.8 16.7 44.6 

Revenues increased 

(export)................................ 4.5 33.8 16.7 45.0

Costs reduced ........................ 5.9 32.4 17.6 44.1

Education 

(n=145)

Participants served................. 42.8 15.2 8.3 33.8

Participants improved............. 23.4 17.2 15.9 43.4

Jobs created........................... 20.0 31.7 21.4 26.9

Jobs retained.......................... 15.2 26.9 18.6 39.3

Businesses created................ 9.0 31.7 16.6 42.8

Businesses retained ............... 11.0 30.3 16.6 42.1

Leveraged private 

investments......................... 10.3 33.8 15.9 40.0

Revenues increased 

(nonexport).......................... 4.8 34.5 17.9 42.8

Revenues increased 

(export)................................ 4.1 34.5 17.9 43.4

Costs reduced ........................ 4.1 33.1 19.3 43.4
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 3-3. Whether ARC education and workforce development projects provided 

information about other ARC.net performance measures on the Westat 

questionnaire—Continued

Output or outcome

Provided a 

number 

Output or 

outcome not 

applicable

Did not track 

information

Skipped 

item

Workforce 

development 

(n=77)

Participants served................. 40.3 14.3 11.7 33.8

Participants improved............. 28.6 15.6 13.0 42.9

Jobs created........................... 28.6 20.8 20.8 29.9

Jobs retained.......................... 24.7 19.5 18.2 37.7

Businesses created................ 10.4 23.4 19.5 46.8

Businesses retained ............... 10.4 23.4 19.5 46.8

Leveraged private 

investments......................... 15.6 26.0 11.7 46.8

Revenues increased 

(nonexport).......................... 5.2 32.5 14.3 48.1

Revenues increased 

(export)................................ 5.2 32.5 14.3 48.1

Costs reduced ........................ 9.1 31.2 14.3 45.5

NOTE: Total for each output or outcome may not sum to 100 due to rounding.

Table 3-4. Number of participants who received services and benefited as a result of 

ARC education and workforce development projects

Level

Number of 

participants served

Participants improved

Number

Percent of those 

served that 

improved

Overall ............................................................. 295,277 40,209 13.6

Education......................................................... 281,382 28,488 10.1

Workforce development ................................... 13,895 11,721 84.4

NOTE: The number for participants served excludes one outlier project that reported serving a total of 597,221 participants; no data 

were provided for number of participants improved.
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-11 

Table 3-5. Economic benefits that occurred as a result of ARC education and 

workforce development projects

Economic benefit Number or dollar amount

Overall

Jobs created (n=44) ..................................................... 2,625

Jobs retained (n=35) .................................................... 57,014

Businesses created (n=9) ............................................ 75

Businesses retained (n=13) ......................................... 133

Leveraged private investments (n=21) ......................... $120,472,112 

Revenues increased (nonexport) (n=3) ....................... $545,000 

Revenues increased (export) (n=1) ............................. $10,000 

Costs reduced (n=4) .................................................... $2,230,500 

Education

Jobs created (n=24) ..................................................... 1,484

Jobs retained (n=18) .................................................... 4,330

Businesses created (n=4) ............................................ 6

Businesses retained (n=8) ........................................... 53

Leveraged private investments (n=10) ......................... $45,674,775 

Revenues increased (nonexport) (n=1) ....................... $10,000 

Revenues increased (export) (n=0) ............................. $0 

Costs reduced (n=1) .................................................... $85,000 

Workforce 

development

Jobs created (n=20) ..................................................... 1,141 

Jobs retained (n=17) .................................................... 52,684 

Businesses created (n=5) ............................................ 69 

Businesses retained (n=5) ........................................... 80 

Leveraged private investments (n=11) ......................... $74,797,337 

Revenues increased (nonexport) (n=2) ....................... $535,000 

Revenues increased (export) (n=1) ............................. $10,000 

Costs reduced (n=3) .................................................... $2,145,500 

3.3 Other Potential Employment and Private Sector 

Impacts (Not Currently in ARC.net) 

In an effort to examine other potential outcomes not currently collected in ARC.net, the online 

questionnaire examined whether projects were collecting data for such measures as pre/post program 

employment status, the sector in which participants were employed, and residential status and wage rates 

after program participation. The purpose was to examine whether it would be feasible to obtain these data 

and, if so, how they might be used to examine community impacts.

As shown in Table 3-6, at least 34 percent of all projects provided data for each of these measures—with 

45 percent providing numbers for employment status before program participation, 42 percent providing 

numbers for employment status after program participation, 43 percent providing numbers on the sector 
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

in which participants were employed after program participation, 36 percent providing numbers on 

participants’ wage rates after program participation, and 34 percent providing numbers on participants’ 

residential status after program participation. On all of these items, a greater proportion of workforce than 

education projects was able to provide numbers. Among the projects that provided data for these 

measures: 

• There was an 18 percentage point decline in the proportion of all participants who were 

unemployed before and after participating in an ARC education and/or workforce development 

initiative (Table 3-7). In addition, the proportion of participants who were employed on a parttime or full-time basis increased by 13 percentage points (from 21.4 percent to 34.4 percent). 

This pattern was similar for education and workforce development projects. Specifically, the 

proportion of participants that were unemployed after participation decreased 17 percentage 

points for education projects and 19 percentage points for workforce development projects; parttime or full-time employment increased by 14 percentage points for education projects and 12 

percentage points for workforce development projects.

• Projects were also asked to provide information about the residential status of individuals after 

they exited the program. The purpose was to determine whether program participants had to 

relocate to find employment. As shown in Table 3-8, 29 percent of all program participants 

remained in the county in which they originally resided after completing the program, while 3 

percent were known to have relocated to another county. The percentage of participants that were 

known to remain in their county was higher for workforce development projects (39 percent) than 

for education projects (22 percent), although both project types reported that 3 percent of 

participants relocated. Across project types, respondents were not able to provide any 

information about the remaining 68 percent of participants. 

• As shown in Table 3-9, the percentage of all participants employed in the manufacturing sector 

after they exited their program was 12 percent, while 9 percent found employment in the services 

sector. Education projects were most likely to employ participants in other sectors (9 percent) or 

services (8 percent), and workforce projects were most likely to employ participants in 

manufacturing (24 percent) or services (10 percent). Overall, projects were unable to provide any 

employment information for 63 percent of former participants. 

• Just over one-fourth (27 percent) of all former participants were making less than $40,000 per 

year after exiting their programs—while 5 percent were making between $40,000 to $69,999 per 

year (Table 3-10). Projects were not able to provide wage information for 68 percent of all 

former participants. 
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-13 

Table 3-6. Whether ARC education and workforce development projects provided 

information about other potential employment-related outcomes

Outcome

Percent that 

provided a number 

in the questionnaire

Percent that did not 

provide a number

in the questionnaire

(i.e., reported 100% 

for “Don’t know”)

Overall

Employment status before program participation 

(n=120) .............................................................. 45.0 55.0 

Employment status after program participation 

(n=119) ............................................................. 42.0 58.0 

Residential status after program participation 

(n=119) .............................................................. 33.6 66.4 

Sector in which individuals were employed after 

program participation (n=121) ........................... 43.0 57.0 

Wage rates after program participation (n=117) ... 35.9 64.1 

Education

Employment status before program participation 

(n=71) ................................................................ 32.4 67.6 

Employment status after program participation 

(n=71) ............................................................... 29.6 70.4 

Residential status after program participation 

(n=71) ................................................................ 26.8 73.2 

Sector in which individuals were employed after 

program participation (n=72) ............................. 30.6 69.4 

Wage rates after program participation (n=70) ..... 22.9 77.1 

Workforce 

development

Employment status before program participation 

(n=49) ................................................................ 63.3 36.7 

Employment status after program participation 

(n=48) ............................................................... 60.4 39.6 

Residential status after program participation 

(n=48) ................................................................ 43.8 56.3 

Sector in which individuals were employed after 

program participation (n=49) ............................. 61.2 38.8 

Wage rates after program participation (n=47) ..... 55.3 44.7 

NOTE: Excludes 65 projects that reported on Question 22 that they had no worker/trainee outcomes, as well as the 30 projects that 

skipped Question 22. Total for each output or outcome may not sum to 100 due to rounding.
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-14 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 3-7. Employment status of individuals before and after participation in an ARC 

education and workforce development project

Employment status Before program 

participation

After program 

participation

Overall

(n=120) (n=119)

Unemployed ................................................................. 22.3% 4.4%

Employed on a part-time basis ..................................... 8.1 8.9

Employed on a full-time basis ....................................... 13.3 25.5

Don’t know .................................................................... 56.4 61.1

Education

(n=71) (n=71)

Unemployed ................................................................. 20.1 2.8

Employed on a part-time basis ..................................... 5.6 7.6

Employed on a full-time basis ....................................... 6.0 18.1

Don’t know .................................................................... 68.2 71.5

Workforce 

development 

(n=49) (n=48)

Unemployed ................................................................. 25.4 6.8

Employed on a part-time basis ..................................... 11.7 10.9

Employed on a full-time basis ....................................... 23.7 36.6

Don’t know .................................................................... 39.2 45.7

NOTE: Excludes 65 projects that reported on Question 22 that they had no worker/trainee outcomes, as well as the 30 projects that 

skipped Question 22. Overall unemployed before program participation includes 11.8 percent that had no prior work experience 

when they entered the program and 10.5 percent that had some previous work experience but were unemployed when they entered 

the program. Education unemployed before program participation includes 12.5 percent that had no prior work experience when 

they entered the program and 7.6 percent that had some previous work experience but were unemployed when they entered the 

program. Workforce development unemployed before program participation includes 10.7 percent that had no prior work 

experience when they entered the program and 14.7 percent that had some previous work experience but were unemployed when 

they entered the program. Totals may not sum to 100 due to rounding.

Table 3-8. Residential status of individuals after they exited an ARC education and 

workforce development project

Residential status Average 

percent

Overall (n=119)

Remained in the county in which they resided after completing the program .. 29.0

Relocated to another county after completing the program.............................. 3.3

Don’t know ....................................................................................................... 67.7

Education (n=71)

Remained in the county in which they resided after completing the program .. 22.4

Relocated to another county after completing the program.............................. 3.3

Don’t know ....................................................................................................... 74.4

Workforce 

development 

(n=48)

Remained in the county in which they resided after completing the program .. 38.9

Relocated to another county after completing the program.............................. 3.2

Don’t know ....................................................................................................... 57.8

NOTE: Excludes 65 projects that reported on Question 22 that they had no worker/trainee outcomes, as well as the 30 projects that 

skipped Question 22. Totals may not sum to 100 due to rounding.
Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects 3 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 3-15 

Table 3-9. Sector in which individuals were employed after they exited an ARC 

education and workforce development project

Sector Average 

percent

Overall 

(n=121)

Manufacturing............................................................................................................ 11.8

Services..................................................................................................................... 8.7

Government............................................................................................................... 4.3

Retail ......................................................................................................................... 1.8

Agriculture ................................................................................................................. 1.2

Mining........................................................................................................................ 0.2

Other ......................................................................................................................... 9.0

Don’t know................................................................................................................. 63.0

Education 

(n=72)

Manufacturing............................................................................................................ 3.9

Services..................................................................................................................... 7.8

Government............................................................................................................... 2.8

Retail ......................................................................................................................... 0.9

Agriculture ................................................................................................................. 1.5

Mining........................................................................................................................ 0.1

Other ......................................................................................................................... 9.0

Don’t know................................................................................................................. 74.1

Workforce 

development 

(n=49)

Manufacturing............................................................................................................ 23.6

Services..................................................................................................................... 10.3

Government............................................................................................................... 6.5

Retail ......................................................................................................................... 3.0

Agriculture ................................................................................................................. 0.7

Mining........................................................................................................................ 0.4

Other ......................................................................................................................... 8.9

Don’t know................................................................................................................. 46.7

NOTE: Excludes 65 projects that reported on Question 22 that they had no worker/trainee outcomes, as well as the 30 projects that 

skipped Question 22. Totals may not sum to 100 due to rounding.
3 Examination of Other Potential Project Accomplishments for the 

ARC’s Education and Workforce Development Projects

 3-16 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 3-10. Wage rates for individuals who found work after they exited an ARC 

education and workforce development project

Wage rate Average 

percent

Overall 

(n=117)

Less than $20,000 per year....................................................................................... 7.3

$20,001–$39,999 ...................................................................................................... 20.0

$40,000–$69,999 ...................................................................................................... 4.7

$70,000–$99,999 ...................................................................................................... 0.2

$100,000 or more...................................................................................................... 0.1

Don’t know ................................................................................................................ 67.7

Education 

(n=70)

Less than $20,000 per year....................................................................................... 6.8

$20,001–$39,999 ...................................................................................................... 11.7

$40,000–$69,999 ...................................................................................................... 2.9

$70,000–$99,999 ...................................................................................................... 0.2

$100,000 or more...................................................................................................... 0.1

Don’t know ................................................................................................................ 78.3

Workforce 

development 

(n=47)

Less than $20,000 per year....................................................................................... 8.0

$20,001–$39,999 ...................................................................................................... 32.3

$40,000–$69,999 ...................................................................................................... 7.3

$70,000–$99,999 ...................................................................................................... 0.3

$100,000 or more...................................................................................................... 0.1

Don’t know ................................................................................................................ 52.0

NOTE: Excludes 65 projects that reported on Question 22 that they had no worker/trainee outcomes, as well as the 30 projects that 

skipped Question 22. Totals may not sum to 100 due to rounding.
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-1 

4.1 Summary

Between 2000 and 2008, the ARC invested a total of $65 million across 386 education and workforce 

development projects. This study was designed to document the range of activities that these projects 

conducted and examine the outcomes that resulted from their efforts. Special emphasis was placed on 

updating the outputs and outcomes that projects provided into ARC.net, the Commission’s performance 

measurement database.

The primary data collection activity conducted as part of this study was an online questionnaire that was 

administered to these 386 projects. The 222 projects that completed a questionnaire represented 

approximately half of the total ARC expenditures for education and workforce development projects 

between FY 2000 and FY 2008. Among these 222 projects, 65.3 percent were classified in the ARC.net 

database as being education, while 34.7 percent were classified as being workforce development. A 

summary of findings from the questionnaire is provided in Exhibit 4-1 at the end of this chapter. 

As part of the evaluation, we also documented the extent to which projects were able to provide data for 

the performance measures in ARC.net that pertain to education and workforce development initiatives. 

Not surprisingly, projects were more likely to be able to provide information about the number of students 

or workers/trainees served (i.e., participated in an activity) than the number improved (i.e., benefited from 

participating in an activity).

Beyond the ARC.net requirements, almost two-fifths of projects tracked participants after program 

completion, with 18 percent tracking participants for one year and 19 percent tracking participants for two 

or more years. The remaining projects either did not track program participants (29 percent) or indicated

that such tracking was not applicable to the types of services they provided (33 percent). This finding 

suggests that a significant number of projects are making at least some effort to obtain longer-term data 

for a number of outcomes that are typically associated with education and workforce development 

projects. This is noteworthy, given that the ARC does not require collection of longer-term outcome data 

from former participants or from projects that are no longer funded. However, during follow-up 

Recommendations for 

Enhancing the ARC’s Collection 

and Utilization of Data About Education 

and Workforce Development Projects 4
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

telephone interviews, a small sample of questionnaire respondents generally acknowledged that their 

efforts to collect follow-up data were often hindered by low response rates, lack of staff time to conduct 

follow-up with nonrespondents, and a reliance on self-reported data on employment status that were 

difficult to validate.

4.2 Recommendations

In previous studies, we have offered recommendations for enhancing the ARC’s procedures for 

measuring, tracking, and reporting the outcomes associated with specific program areas (e.g., education, 

vocational education, telecommunications, and community capacity-building). Specifically, we have 

advised that the Commission develop separate guidelines (or supplemental materials) that provide 

customized application and reporting instructions, as well as examples of outcomes for a particular issue 

area. We have also suggested that the Commission facilitate projects’ access to information about how to 

collect and analyze data.

Given the narrow focus of this study, we have limited our recommendations to four broad steps that 

would enhance the ARC’s capacity to document the full range of outcomes associated with its education 

and workforce development projects. These recommendations are designed to help the ARC maximize 

its use of the data that are already submitted into ARC.net, as well as to extend the reach of the ARC.net 

database to include ARC-supported outcomes that occur after a project’s grant has ended. 

Recommendation 1: Develop a coding scheme in ARC.net to more precisely report on the 

accomplishments of education and workforce development projects.

The ARC.net database currently requires that education and workforce development projects provide data 

on the number of students and/or workers/trainees served and improved by their efforts. However, as 

described in Chapter 2, the diverse ways in which students and worker/trainees could be categorized as 

having been served and improved makes it difficult to meaningfully aggregate this information across 

projects. In recent years, projects have also been asked to provide a narrative that describes how students 

and/or workers/trainees improved. While this information is useful, the ARC does not currently have 

procedures in place to codify and analyze the descriptions that are provided by projects. Such a process 

would enhance the Commission’s capacity to use this information to shed light on the range of outcomes 

associated with its portfolio of education and workforce development initiatives. It would also increase 
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-3 

the likelihood that ARC personnel are able to make use of the narrative information supplied by projects 

as part of the existing ARC.net requirement. 

One option would be for the ARC to supplement ARC.net by providing a close-ended coding scheme that 

projects would use to classify their narrative response. As shown in Exhibits 4-2a and 4-2b at the end of 

this chapter, this coding scheme could build upon the response options that were used in the online 

questionnaire. Projects could be asked to select the response option that best describes how students 

and/or workers/trainees were improved, as well as any secondary options that can be used to capture the 

full extent of the improvements. ARC staff could then use these codes to classify projects’ narratives 

about how their students and workers/trainers improved, as well as quickly identify projects that are 

associated with a given type of outcome.

A second option would be to allow projects to select multiple categories and provide a number for each of 

the benefit types that are selected. Thus, for example, instead of providing data on the number of students 

and workers/trainees with improvements, projects could report anticipated and actual data for each of the 

applicable categories in Exhibits 4-2a and 4-2b, respectively. While this would potentially require that 

projects invest more resources into tracking services and outcomes, the resulting data would enhance the 

ARC’s capacity to compare projected and actual outcomes. It would also allow the ARC to more 

accurately report across projects on the number of individuals that attained a specific type of 

improvement, thereby allowing for more valid and reliable comparisons of baseline projections and actual 

results. These data could also be aggregated to provide stakeholders with annual data on the number of 

students and workers/trainees served and improved. 

Recommendation 2: Expand the definition of what constitutes project success for the ARC’s education 

and workforce development projects.

The Commission’s performance goal for education and workforce development projects is that 240,000 

citizens of the Appalachian region will benefit from enhanced education and job-related skills by 

2022.14 Progress toward this goal is only measured through two ARC.net data fields: number of students 

and workers/trainees improved. However, the ARC.net database contains other performance measures 

that could be used to demonstrate the impact of the Commission’s education and workforce development 

activities—including participants improved, jobs created, jobs retained, businesses created, and 

businesses retained. In fact, evidence from the online questionnaire suggests that a significant number of 

 

14 Appalachian Regional Commission. (2010). Moving Appalachia Forward. Appalachian Regional Commission Strategic Plan 2011–2016.
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

education and workforce development projects are able to report data for these measures. Specifically, 42 

percent provided data for participants served, 25 percent for participants improved, 23 percent for jobs 

created, and 19 percent for jobs retained. This finding suggests that the Commission could encourage 

future education and workforce development projects to collect and report information about these 

additional data elements in ARC.net. The purpose would be to document and report the full range of 

outcomes for education and workforce development projects that are currently covered by ARC.net. 

In addition to the performance measures that are currently included in ARC.net, there are additional 

outcomes that education and workforce development project might be expected to attain. For example, as 

discussed in Chapter 3, other federal workforce development programs collect data on such outcomes as 

employment status and wages. In an effort to examine other potential outcomes not currently collected in 

ARC.net, the online questionnaire examined whether ARC projects were able to report data for such 

measures as pre/post-program employment status, the sector in which participants were employed, and 

residential status and wage rates after program participation. Once again, evidence from the questionnaire 

suggests that a significant number of projects are able to report these data—with 45 percent providing 

numbers for employment status before program participation, 42 providing numbers for employment 

status after program participation, 43 percent providing numbers on the sector in which participants were 

employed after program participation, 36 percent providing numbers on participants’ wage rates after 

program participation, and 34 percent providing numbers on participants’ residential status after program 

participation. Not surprisingly, it was easier to collect these data for workforce projects than education 

projects.

Measuring how workforce and education projects change the work status of participants would enable the 

ARC to demonstrate a fuller range of outcomes associated with its education and workforce development 

projects. In addition, the collection of data on employment status and wages would potentially allow the 

ARC to measure the economic impact of its education and workforce development projects on 

participating counties. An exploratory economic analysis conducted as part of this study found that the 

ARC’s education and workforce development projects provide benefits across the Appalachian region by 

substantially increasing participants’ income and employment opportunities. This analysis, summarized 

in Appendix B, is an example of the type of analyses that could be conducted if more complete economic 

data were routinely obtained from ARC grant recipients. 

As discussed in Appendix B, the findings obtained through the supplemental items in the online 

questionnaire were sufficient to show systematic patterns of program impacts on both jobs and 

income. These impacts vary depending on the employment status of participants and the economic and 
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-5 

density patterns of the counties in which these projects occur. The limited number of complete responses 

to economic impact Questions 30 to 35 of the questionnaire prevented us from differentiating results 

among portions of the region in terms of location and economic distress levels. As such, the analysis was 

conducted for the entire Appalachian region. Moreover, the effects of different types of ARC education 

and workforce training programs could not be characterized. A higher response rate for these items in the 

questionnaire would have allowed for an examination of the impact of individual education and 

workforce development strategies for such specific variables as population density (i.e., urban versus rural 

counties) and county designation (e.g., distressed, at-risk, transitional, competitive, and attainment). 

For this study, respondents were asked to provide data about long-term outcomes that they were never 

specifically asked to collect when their programs were initiated. In this respect, it is impressive that 

nearly 20 percent of respondents tracked sufficient information that made an initial ARC-wide analysis 

possible. It is a tribute to program managers that 39 grantees were in a position to provide enough data to 

allow for an employment analysis and 25 were able to provide data for an income analysis. A more 

rigorous data reporting system would enable for a more in-depth analysis, thereby allowing the ARC to 

differentiate among ARC states, local development districts, and specific program strategies (see 

Recommendation 3 below). 

Recommendation 3: Provide tools that education and workforce development projects can use to 

efficiently document an expanded set of outcomes. 

The ARC does not currently require applicants or projects to use a standard reporting format to submit 

their baseline and annual data. In addition, because the same forms are used to obtain data from all nonconstruction projects, there is no opportunity for the ARC to collect more comprehensive information 

about the characteristics and accomplishments of projects in a specific program area. This lack of indepth information ultimately hinders the Commission’s capacity to fully document its outcomes or 

examine the relationship between projects’ characteristics, strategies, and accomplishments.

The questionnaire administered as part of this study was designed to address this limitation by developing 

a standardized protocol that could be used to obtain information about the full range of activities, outputs, 

outcomes, and impacts that are typically associated with education and workforce development projects. 

By developing a questionnaire that was purposely tailored to a particular program area, we were able to 

focus on specific attributes and accomplishments that would not have been possible with a more generic 

survey designed to collect follow-up data across all program areas. Now that this questionnaire has been 
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

tested, we recommend that the ARC use a modified version of this instrument to obtain baseline and 

annual data from its education and workforce development applicants and projects. Specifically:

• Collect baseline data through an addendum to the standard application. As part of the 

application, we are suggesting that the ARC obtain baseline information about project 

characteristics and anticipated outcomes. Appendix C provides an example of the types of data 

that might be collected from education and workforce development projects through an 

addendum to the standard application for nonconstruction projects—including (1) counties, age 

groups, and populations to be targeted; (2) education and workforce development activities to be 

conducted with these populations; and (3) the educational, employment, and economic outcomes 

that are expected to occur as a result of the project. By collecting this information as part of the 

application process, the ARC will ensure that it obtains a response from all funded projects 

without having to conduct extensive follow-up. Prior to a grant being approved, we would 

further recommend that staff in the Program Operations Division (POD) review applicants’ 

estimates for project outcomes to assure that all required data fields are adequately addressed and 

that estimates are in line with the ARC’s expectations (e.g., in terms of the number expected to be 

served and improved). 

• Collect annual data through a supplement to the standard annual and closeout reports. We 

also recommend that the ARC use a similar supplement to the annual and closeout reports to 

obtain valid and reliable output and outcome data from education and workforce development 

projects. The purpose would be to obtain cumulative data on the outputs, outcomes, and impacts 

that have occurred over the life of their grant. A prototype of this form, presented in Appendix D, 

provides an example of the types of annual information that might be collected from education 

and workforce development projects—including (1) whether or not the ARC project provided any 

services in the past 12 months; (2) the counties, age groups, and populations targeted by the 

project; (3) the education and workforce development activities conducted over the life of the 

project; (4) the educational, employment, and economic outcomes that occurred over the life of 

the project; (5) how students and workers/trainees have been improved by the project; and (6) the 

employment status of former participants. For each annual report, we recommend that the time 

frame for which annual data are provided be cumulative—that is, projects would provide updated 

information on the number of individuals served and improved since the beginning of their grant. 

This approach would ensure that projects do not double count participants and beneficiaries. It 

would also allow the ARC to systematically and accurately determine the proportion of 

individuals receiving services that are ultimately improved (since the numerators and 

denominators would be aligned over time). Finally, this approach would enable the ARC to more 

accurately document progress toward the annual goals outlined in the Commission’s strategic 

plan. 

Information obtained through the use of the addendums to the application, annual, and closeout reports 

could be entered into ARC.net and used to help the ARC develop a comprehensive inventory of the 

educational and workforce development strategies that projects are using (as well as the populations with 

which those strategies are being applied). By using protocols that specifically pertain to education and 

workforce development projects, the ARC would have an opportunity to test a data collection and 

utilization strategy that might ultimately be used with other program areas. In addition, the information 
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-7 

on actual outputs and outcomes could be compared with data obtained during the application process to 

determine whether education and workforce development projects (both individually and in the 

aggregate) are meeting or exceeding their expectations. 

As shown in Appendices B and C, we are proposing that these forms include detailed definitions for key 

terms, thereby ensuring that applicants will use common assumptions and criteria when providing their 

information. For example, the forms (and the coding scheme in ARC.net) should allow for such common 

conditions as “don’t know,” and “not applicable.” We would also recommend that the items include 

instructions about the time frame for which estimates about anticipated outputs and outcomes are to be 

provided (e.g., for up to three years after project closeout in the application addendum, since the 

beginning of the grant award for the annual supplement). 

It should be noted that several projects used the telephone interviews to request that the ARC provide 

them with tools that could be used to collect participant data. We therefore recommend that the ARC 

provide education and workforce development projects with a standard form that can be used to obtain 

valid and reliable participant intake and follow-up data. The example in Exhibit 4-3 (at the end of this 

chapter) demonstrates how such a form could be used to obtain basic information about participants’ 

current employment status, number of hours currently worked in an average week, current salary, and 

type of industry in which participants are currently employed (a separate form could be developed for 

education projects that primarily serve students). Collecting this information as part of the intake process 

would enable projects to assemble data on the baseline characteristics of their participants. By continuing 

to collect this information over time (e.g., annually), projects would be able to document the number of 

participants that are enhancing their educational and employment status, increasing their incomes, and 

improving their quality of life due to their participation in ARC projects. This information could then be 

reported to the ARC in the aggregate in the annual and follow-up forms in Appendix D. 

Recommendation 4: Develop procedures that encourage projects to report outcomes to the ARC after 

their grant awards have expired. 

ARC projects are strongly encouraged to secure other sources of support so that they can sustain their 

activities beyond the life of their ARC grant. However, aside from the validation visits that are conducted 

with a small sample of projects, the Commission currently has no way to systematically document any 

education and workforce development outcomes that occur after the ARC grant has expired. As such, a 

purpose of this study was to collect long-term data from former projects so as to update the educational 

and employment performance measures in the ARC.net database.
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Since projects were not expected to report these data, many respondents were not able to provide any 

information about the educational and employment status of their former participants. For those projects 

that were able to report these data, the outputs and outcomes reported on the questionnaire were 

substantially higher than those reported in the closeout fields in ARC.net.15 For example, as shown in 

Table 4-1a, the 55 projects that provided a number for students served in the questionnaire reported 

serving a total of 87,693 students; only 32 of the same 55 projects had a record for students served 

recorded in ARC.net (at closeout), the total of which equaled 22,715—a difference of 64,978. In Table 4-

1b, numbers for specific output or outcome types were only compared for projects with a corresponding 

record in both the questionnaire and ARC.net (at closeout). In the majority of these cases, numbers 

reported in the questionnaire remained higher than those reported in the ARC.net closeout fields, although 

the differences were significantly less than in Table 4-1a. As such, when comparing the 32 projects that 

had records in both the questionnaire and ARC.net (closeout) for students served, the total reported in the 

questionnaire was 32,576 and the ARC.net total was 22,715—a difference of only 9,861. 

One interpretation of these findings is that the ARC’s education and workforce development projects 

significantly exceeded their initial estimates of the number of individuals at closeout that would be served 

and improved by their efforts. Additionally, a significant portion of projects that did not have data 

recorded in ARC.net either are not being asked to provide long-term information on their outputs and 

outcomes or are unable to provide the information in their closeout reports. However, it is also possible 

that at least some of these differences were the result of other factors—most notably the longer time 

period covered by the questionnaire. In order to better understand the differences between numbers 

reported in the questionnaire and closeout numbers recorded in ARC.net, we contacted the 28 projects 

whose numbers of students and workers improved differed by 30 percent or more between the two 

sources to ask why the number reported in the questionnaire for students or workers served was greater 

than or lesser than the number recorded by the ARC at each project’s close. Most (22 of the 28) of the 

projects responded, with the follow-up revealing four distinct explanations for the discrepancies: 

• The program continued after the grant period ended, which led to an increase in the numbers 

reported in the questionnaire when compared with those recorded in ARC.net (10 projects). 

 

15 In an effort to assess the accuracy of the data contained in the ARC.net database for education and workforce development projects, we 

compared the projected and closeout data in ARC.net with the output and outcome data obtained through the questionnaire. This assessment 

was complicated by multiple entries in ARC.net for 32 of the 222 projects that were included in the questionnaire sample (these 32 multiple 

entries represent a total of 94 entries in ARC.net, from which 62 duplicates were removed). We are therefore only providing a comparison of 

questionnaire and projected/closeout ARC.net data for the 190 projects that did not have multiple entries in ARC.net (which allows for a more 

accurate assessment of the ARC.net data for projected and closeout outputs and outcomes). In addition, we did not compare ARC.net and 

questionnaire data for the 266 projects that were funded prior to 2006, since those projects were less likely to have been encouraged by the 

ARC to maintain these data beyond the life of this grant. 
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-9 

• Different individuals entered data into ARC.net and the Westat questionnaire—with each 

individual potentially using different assumptions regarding project scope or timeframe (four 

projects).

• Human error in entering data into ARC.net (three projects). 

• More individuals were served or improved than originally anticipated (two projects).

The number of projects that tracked outcomes after their project ended reveals an opportunity for the 

ARC to conduct a more comprehensive closeout questionnaire with projects that obtains follow-up 

information about the post-program status of project participants. Specifically, in an effort to count 

longer-term outcomes toward the Commission’s goal regarding enhanced education and job-related skills, 

the ARC could continue to administer the annual supplement described in Recommendation 3 for up to 

five years after a project’s grant expires. Using the same form that was administered during the grant 

award would ensure that projects are familiar with the procedures required to obtain and tally the 

requested data. Obtaining these data over time will enable the Commission to document the outcomes 

that are being leveraged through the long-term continuation of strategies that were created with the 

ARC’s seed money. This, in turn, would enable the ARC to document its ongoing progress in meeting the 

annual goals outlined in its strategic plan. 
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Table 4-1a. Output and outcome comparison between questionnaire and ARC.net data

from 2006–2008: Data for projects that provided outputs or outcomes in the 

questionnaire

Output or outcome

Questionnaire

ARC.net closeout

(Includes projects that provided 

no data in ARC.net)

Number 

of 

projects 

with a 

record

Total 

output or 

outcome

Number 

or 

projects 

with a 

record

Number 

of 

projects 

without a 

record

Total 

output or 

outcome

Total 

output or

outcome 

ratio

Overall

Students served..... 55 87,693 32 23 22,715 3.86

Students improved. 43 36,172 19 24 10,282 3.52

Workers/trainees 

served................ 24 7,638 6 18 2,087 3.66

Workers/trainees 

improved............ 17 4,295 4 13 703 6.11

Education

Students served..... 39 81,378 27 12 22,090 3.68

Students improved. 28 33,080 17 11 10,101 3.27

Workers/trainees 

served................ 16 2,669 1 15 23 116.04

Workers/trainees 

improved............ 10 1,696 1 9 23 73.74

Workforce 

development

Students served..... 16 6,315 5 11 625 10.1

Students improved. 15 3,092 2 13 181 17.08

Workers/trainees 

served................ 8 4,969 5 3 2,064 2.41

Workers/trainees 

improved............ 7 2,599 3 4 680 3.82

NOTE: This table excludes 32 ARC education and workforce development projects with multiple entries in ARC.net (e.g., due to a 

contract extension or revision) that were funded between 2006 and 2008.

SOURCE: ARC.net database and Westat questionnaire of ARC education and workforce development projects.
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-11 

Table 4-1b. Output and outcome comparison between questionnaire and ARC.net data 

from 2006–2008: Projects with corresponding output and outcome records 

in both the questionnaire and ARC.net

Output or outcome

Questionnaire 

total 

(Excludes 

projects 

without a 

corresponding 

output or 

outcome record 

in ARC.net)

ARC.net 

closeout total

(Excludes 

projects 

without a 

corresponding 

output or 

outcome record 

in the 

questionnaire) Total ratio

Overall

Students served (n=32) .................. 32,576 22,715 1.43

Students improved (n=19) .............. 19,301 10,282 1.88

Workers/trainees served (n=6)........ 4,656 2,087 2.23

Workers/trainees improved (n=4).... 2,338 703 3.33

Education

Students served (n=27) .................. 31,382 22,090 1.42

Students improved (n=17) .............. 19,159 10,101 1.90

Workers/trainees served (n=1)........ 23 23 1.00

Workers/trainees improved (n=1).... 23 23 1.00

Workforce 

development

Students served (n=5) .................... 1,194 625 1.91

Students improved (n=2) ................ 142 181 0.78

Workers/trainees served (n=5)........ 4,633 2,064 2.24

Workers/trainees improved (n=3).... 2,315 680 3.40

NOTE: This table excludes the 32 ARC education and workforce development projects with multiple entries in ARC.net (e.g., due to 

a contract extension or revision). 

SOURCE: ARC.net database and Westat questionnaire of ARC education and workforce development projects.
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Exhibit 4-1.—Summary of findings from the online questionnaire

Who received ARC education and workforce development services? Almost half (49 percent) of projects that 

responded to the questionnaire provided at least some of their services to adults (not including postsecondary students or 

teachers), while 44 percent served K–12 students and 36 percent served postsecondary students. One-fourth (24 percent) 

provided services to teachers, while a smaller proportion targeted services to preschool children (14 percent) or out-ofschool youth (10 percent).

What types of education and workforce development services did ARC projects provide? Projects were designed to 

provide a wide range of educational and workforce development services. The majority (56 percent) were providing 

educational attainment or achievement services that were specifically targeted to students pursuing a high school diploma 

or postsecondary degree, while 43 percent were providing career and technical education services and 41 percent were 

providing workforce training skills. Over half of the projects provided at least one of the following services: skills/training 

that enhanced employability (59 percent), special use classroom equipment such as computers or science labs (58 percent), 

and/or occupational/job skills training or instruction (53 percent). One-third (34 percent) provided computer skills 

instruction, while 23 percent provided or improved a physical structure.

How many students and workers/trainees were served and improved by ARC education and workforce development 

projects? At the time the questionnaire was administered (summer 2011), the 222 ARC education and workforce 

development projects funded between 2000 and 2008 reported that they had served a total of 414,296 students and 

workers/trainees—including 334,803 students and 79,493 workers/trainees. Of the total served, 99,809 individuals were 

reported to have benefited from their participation—including 81,974 students and 17,835 workers/trainees. We also 

compared the baseline estimates of the number of individuals that projects expected to serve and improve with the number 

that were reported as being served and improved on the Westat questionnaire. Projects served more students and 

workers/trainees than initially anticipated (141,037, compared with a projected total of 77,606). They also reported 

improving substantially more students and workers/trainees combined (41,481) than originally projected (27,502).

What types of benefits did ARC projects report for students that were improved? Among students, the most common 

benefits that projects reported were primarily focused on teenagers and young adults—e.g., obtaining vocational and 

technical skills (36 percent), enrolling in a college or postsecondary program (31 percent), obtaining basic or academic 

skills in a specific subject (30 percent), or obtaining a postsecondary degree, credential, or certification (28 percent).

What types of benefits did ARC projects report for workers/trainees that were improved? The most common benefit 

that projects reported for workers/trainees were learning skills in a new area—including vocational and technical skills 

(37 percent), employability skills (36 percent), and basic or academic skills (25 percent). A smaller proportion of projects 

indicated that their efforts resulted in improved work status—with 27 percent indicating that workers/trainees obtained new 

full-time employment and 21 percent reporting that workers increased their job status and/or earned increased wages.

What other outcomes did ARC education and workforce development projects report? A total of 44 projects reported 

creating 2,625 new jobs, of which more than half (1,484) were accounted for by 24 education projects. Thirty-five projects 

reported retaining 57,014 jobs—most of which (52,684) were reported by 17 workforce development projects. In addition, 

nine projects created a total of 75 businesses and 13 projects retained 133 businesses. Five workforce development projects 

created 69 of the businesses, while eight education projects accounted for 53 of the businesses that were retained.

Twenty-one projects leveraged a total of $120 million in private investments, of which $45 million was reported by 10 

education projects and $75 million was reported by 11 workforce projects. Two workforce and one education projects 

increased their nonexport revenues by $545,000, while one workforce project increased its export revenues by $10,000. 

Three workforce and one education projects reduced their costs by $2.2 million.

There was an 18 percentage point decline in the proportion of participants who were unemployed before and after 

participating in an ARC education and/or workforce development initiative. In addition, the proportion of participants who 

were employed on a part-time or full-time basis increased by 13 percentage points (from 21.4 percent to 34.4 percent).

Just over 27 percent of all former participants were making less than $40,000 per year after exiting their programs—while 5 

percent were making between $40,000 and $69,999 per year. Projects were not able to provide wage information for 

63 percent of all former participants.

SOURCE: Westat questionnaire of ARC education and workforce development projects funded between FY 2000 and FY 2008.
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-13 

Exhibit 4-2a. Potential response options for coding ARC.net narratives on how students were 

improved 

Please use the following list to indicate how students were improved by your ARC project 

Primary

(select one)

Secondary

(select all that apply)

Obtained vocational and technical skills  

Enrolled in college or postsecondary program  

Obtained basic or academic skills in a specific subject  

Obtained a postsecondary degree, credential, or certification  

Improved school readiness  

Enrolled in more challenging classes  

Remained in school when at risk of dropping out  

Improved attendance on a daily/regular basis  

Obtained a high school diploma, GED, or equivalent  

Increased scores on statewide assessments  

Advanced to next grade level  

Decreased suspensions and other problem behaviors  

Tested at or above grade level  

Re-entered an educational program after dropping out  
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-14 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Exhibit 4-2b. Potential response options for coding ARC.net narratives on how workers/trainees 

were improved

Please use the following list to indicate how workers/trainees were improved by your ARC project 

Primary

(select one)

Secondary

(select all that apply)

Obtained vocational and technical skills  

Obtained other employability skills (e.g., work attitudes/habits)  

Obtained new full-time employment  

Obtained basic or academic skills  

Maintained current employments  

Obtained new part-time employment  

Increased job status and/or earned increased wages  

Reduced dependence on public assistance  

Teachers or instructors enhanced their classroom practices  

Retrained in another field and obtained new employment  

Obtained/maintained employer-provided health benefits  
Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects 4 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 4-15 

Exhibit 4-3. Example of items for an intake form for workforce development projects

The following information will only be used to report on the overall success of this employment 

and training program. None of this information will be used or reported at the individual level. 

1. Name (for tracking purposes): ________________________________________________ 

2. Home zip code: ___________________________________________________________ 

3. In what year were you born: _________________________________________________ 

4. Employment status (check one): 

 Never employed (end survey)

 Currently unemployed—but previously employed (end survey)

 Currently employed (answer questions 5-8)

5. How many hours do you currently work in an average week: ______ hours per week 

6. What is your current salary (provide information for one of the following): 

$ ______ per hour 

$ ______ per week 

$ ______ per year 

7. Which of the following benefits does your current employer provide (check all that apply): 

 Health

 Disability

 Life insurance 

 Education/professional development 

 Matching 401(k)/retirement 

 Other (specify): _________________________________________________________

8. In what type of industry are you currently employed (check one): 

 Agriculture 

 Government 

 Manufacturing

 Mining 

 Retail

 Services

 Other (specify): _________________________________________________________
4 Recommendations for Enhancing the ARC’s Collection and 

Utilization of Data About Education and Workforce Development Projects

 4-16 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-1 

Appendix A: 

Case Study Findings
 

A2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-3 

A.1 Introduction

Following the administration of the online questionnaire, a series of follow-up interviews were conducted 

with a purposeful sample of 23 ARC education and workforce development projects. These interviews 

were used to obtain more detailed information about the accomplishments associated with these 23 

projects, as well as to learn more about their efforts to collect outcome data from and about former 

participants.

Thirteen of the projects included in the follow-up interviews were selected because they reported on the 

online questionnaire that they had collected at least some outcome data from former participants. The 

remaining 10 projects were selected because they indicated on the online questionnaire that they had not 

collected any follow-up data with former participants. The 23 projects represented all but three (New 

York, North Carolina, and South Carolina) of the 13 states that are covered by the ARC. Just over half 

(12) of the projects had targeted a single population (e.g., K–12 students only), while 11 had served 

multiple populations (e.g., both K–12 students and adults). Specifically, as shown in Figure A-1, 11 of 

the 23 projects had served K–12 students, 10 had served postsecondary students, and 10 had served 

adults. The most prominent strategies being used by these 23 projects were educational attainment (17 

projects), workforce training (14 projects), and career and technical education (12 projects) (Figure A-2). 

Case Study Findings A
A Case Study Findings

 A-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Figure A-1. Target populations of projects included in the follow-up telephone 

interviews

Figure A-2. General activities conducted or supported by the projects included in the 

follow-up telephone interviews

1 

3 

4 

10

10

11

0 2 4 6 8 10 12

Preservice

Teachers

Out-of-school youth

Postsecondary youth

Adults

K–12 students

Number of projects

3 

6 

6 

12

14

17

0 2 4 6 8 10 12 14 16 18

Child development

Adult education

Teacher training

Career and technical education

Workforce training

Educational attainment

Number of projects
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-5 

A.2 Implementation and Accomplishments of Case Study 

Projects 

This section summarizes the information that projects provided in the telephone interviews about the 

implementation and outcomes associated with their ARC education and workforce development grants. 

Specifically, it addresses projects’ assessment of their most significant accomplishments, identifies 

factors that facilitated or hindered their capacity to successfully implement the range of services covered 

by their ARC grant, and describes lessons learned in implementing their projects that they thought would 

be of interest to future ARC education and workforce development projects. 

A.2.1 Projects’ Assessment of Their Most Significant 

Accomplishment

During the interviews, respondents were asked to describe what they thought was their project’s most 

significant accomplishment. All of the participants named at least one significant accomplishment, with 

most identifying two or three achievements that they considered noteworthy. These accomplishments can 

be categorized into the following broad categories: improved access to educational services, enhanced 

educational or job placement accomplishments, improved relationship between academia and industry, 

and improved project management and sustainability. Each of these is discussed below.

A.2.1.1 Improved Access to Educational Services

The most frequently cited accomplishment (described by 13 of 23 projects) was improved educational 

access for participating students. These projects indicated that absent the ARC’s support, funding would 

not have been available for a wide range of services in their distressed and previously unserved rural 

communities. Most notably, the establishment of an ARC-supported facility or program resulted in 

reduced driving times for accessing services or resources. For example:

• One project indicated that prior to an ARC-funded renovation, the city had no training facilities 

that were in close proximity to the corporate business district. As a result of the project, local 

businesses and industries were able to take advantage of training programs offered, reducing lost 

time from travel. The central location of the training facility also resulted in a growth in demand 

for college credit courses.
A Case Study Findings

 A-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

• Another projected, situated in a rural region with high unemployment, established a distance 

learning facility. Because the region was not served by a higher education facility, the project 

director concluded, “If we were not here, the students would not be taking courses anywhere. 

Drive time and access to personal transportation and the perceived geographical boundary prevent 

students from accessing education in other areas.”

Other tangible outcomes cited by projects included increased program availability, increased number of 

parents taking advantage of early childhood education opportunities for their children, and enhanced 

capability of students to use telecommunications to connect with other students or their professors.

A.2.1.2 Enhanced Educational or Job Placement Accomplishments

Nineteen of the 23 projects specified education-related accomplishments such as numbers served, raised 

standards, or increased college-going rates. For example, 11 of the 13 projects that reported improved 

access to educational services indicated that these expanded services resulted in an increase in the number 

of individuals served. In fact, five of these projects indicated that they exceeded their goal for the 

numbers they expected to serve in a given time frame. For example, one project director stated, “We 

anticipated serving 300, we served 1,000.” For two of these projects, this increase in numbers served 

resulted in more investment by higher education agencies for project-related activities. For example, one 

state agency committed to expanding the project site into a full-service facility that will provide the types 

of basic student services that can be found on most college campuses (including a library and cafeteria). 

Within five years, the project also expects to have additional personnel for financial aid, admissions, and 

counseling. A second project reported that their welding program’s increased application, enrollment, 

and completion rates justified the establishment of a new welding associate’s degree (above and beyond 

the existing welding certificate).

Three projects cited “raising standards” as a significant outcome of their programs. These programs 

established a link between secondary schools and higher education institutions. One project focused on 

reducing the high school dropout rate by rewarding attendance, exposing students to postsecondary 

education opportunities, providing rewards for good grades, and helping students stay on track to 

graduate. The second project raised standards by implementing dual-credit courses that allowed students 

to earn college credit in the medical assisting health field while completing their high school diplomas. 

The third project increased the number of Advanced Placement (AP) courses offered, the number of 

students enrolled in AP courses, and the number of students earning quality scores on AP exams. 
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-7 

According to the project director, this raised standards because “the inclusion of AP courses ...added 

more rigor into the typical course offerings and ... increased the quality of teachers in those schools.”

Three projects reported that their most significant accomplishment was an increase in youth’s interest in 

attending college. One director indicated that the college-going rate in counties served by the program 

increased, while another reported an increase in the numbers of students taking dual-credit courses—i.e., 

as a result of the project, over 100 students earned between 12 and 16 hours of college credit before they 

graduated from high school. Although the third project could not quantify their increased college-going 

rate, the project director explained that “the notion of ‘our most significant accomplishment’ has been to 

change the culture regarding postsecondary education.... we are succeeding in moving the finish line 

from a high school diploma to the completion of a postsecondary education.” 

Finally, four projects indicated that their efforts resulted in workers being trained and hired for industry. 

Two projects reported job placement data for all of their program completers across all curricula 

offerings, including Adult Basic Education. A third project placed workers in construction trades, 

HVAC, welding, and building construction. The fourth project placed students in the health care sector. 

A.2.1.3 Improved Relationship Between Industry and Academia

Five projects described their most significant accomplishment as improving their relationship with 

industry. This improved relationship resulted in reciprocal, positive outcomes. For example, three 

projects reported an increase of industry service on their program advisory committees. Typical industry 

involvement included providing oversight on curriculum content, guidance on the types of equipment that 

should be used by the program, and the identification of credentials that would be recognized and/or 

required by industry. In return, participating industries gained access to applicants trained in the content 

and skills that they desired.

Two projects reported that their improved relationships with industry led to an increase in private sector 

support for field trips. For example, one dropout prevention project reported that a local company 

provided funding to transport students from ARC programs to industrial sites. At these sites, students 

were exposed to work processes, equipment, and career opportunities. The project representative stated, 

“It was great for the kids to see how interested the company is in seeing them graduate and stay in the 

area.”
A Case Study Findings

 A-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

A.2.1.4 Improved Project Management and Sustainability

Six projects described accomplishments regarding project and grant management. For example, one 

construction project felt it was significant that they finished “on time and on budget.” Another project that 

focused on education indicated that their effective fund management allowed them to extend their 

program and offer additional services to students. In addition, four education and training projects 

identified significant accomplishments related to sustainability, reporting that they used ARC funds as 

seed money and subsequently secured other funding to continue their efforts. 

A.2.2 Projects’ Assessment of Factors That Facilitated and 

Hindered Their Success

Projects were asked to describe factors that facilitated or hindered the scope and impact of their project 

efforts. Projects reported one or more facilitators of success related to stakeholder involvement, the use 

of personnel, and previous experience with the ARC. Hindrances were related to ARC requirements, 

institutional barriers, the lack of stakeholder involvement, the unintended consequences of successful 

training, and technology literacy. While many of the facilitating factors were within the control of project 

staff, some of the hindrances were beyond their control. 

A.2.2.1 Facilitating Factors

Stakeholder buy-in. Twelve projects discussed the importance of stakeholder buy-in as a crucial factor 

that facilitated their project’s success. The identity of the stakeholders, the role that they played, and their 

level of involvement varied from project to project, but their buy-in was crucial for project success. 

The stakeholders were described as representatives from either community-based organizations 

(e.g., economic development agencies in the area, local elected officials and other government agencies 

which could or would be affected by the project), or business and industry. 

The buy-in of community-based organizations was cultivated by purposefully communicating program 

goals, costs, and expected outcomes. This communication began early in the planning and budget phase 

and helped encourage support throughout all phases of the project. Projects indicated that if issues and 

obstacles surfaced, stakeholders were more likely to support, rather than block, project director decisions 

if those stakeholders had been actively involved in the project. For example, one project manager 
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-9 

indicated that efforts to increase stakeholders’ understanding of the program (e.g., goals, costs, and 

expected outcomes) resulted in enhanced support that, in turn, has allowed for greater opportunities for 

sustained funding beyond the ARC grant. 

Representatives from workforce development projects said that local business and industry buy-in was a 

crucial factor. Three projects indicated that they were more successful when they engaged stakeholders 

from different industry sectors with similar needs. For example, one project was responsible for the 

education and placement of welders. The stakeholders they engaged included representatives from local 

mining, manufacturing, and construction companies who became involved in the project because they all 

needed welders. They each cooperated during the development of the program and later competed for 

program graduates. This multisector stakeholder group was advantageous for the training provider for 

two reasons. First, having multiple stakeholders contributed to the long-term viability of the program; if 

one industry had a downturn and needed to discontinue hiring new workers, the training program was still 

able to place graduates in other job sectors. Second, because the companies were competing for 

graduates, they were more likely to become involved in the program’s advisory committee and provide 

the program with more equipment, consumable supplies, and monetary scholarships and awards, and 

eventually hire the program’s graduates.

Strategic use of personnel and resources. Several projects attributed their success to the voluntary and 

flexible nature of their initiatives. For example, one project indicated that its success was enhanced by a 

decision to provide grants only to teachers who volunteered, with the respondent indicating that the 

project would not have been as successful if participation been mandatory. A second project indicated 

that its success was tied to a decision that allowed teachers the flexibility to shape the project to fit the 

unique needs of their school—i.e., their success was linked to a decision to take a “one size does not fit 

all” approach.

Other projects described how their use of technological resources excited staff and enhanced 

communications and access. For example, one project indicated that its faculty’s eagerness to obtain and 

use the ARC-provided technology was contagious, which helped the “technology avoidant” faculty to 

embrace and use the technology within their own courses. Another project that used distance learning 

reported that the use of technology enabled face-to-face contact between professors and their students, 

thereby increasing the student retention rate.

Previous experience with the ARC. Three projects identified previous experience with the ARC as a 

factor that facilitated their success. These projects stressed the benefits of having attended ARC-
A Case Study Findings

 A-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

sponsored conferences and meetings and developing relationships with ARC staff (in both Washington, 

DC, and their own states)—indicating that these contacts enhanced their ability to be successful in their 

reporting and financial accountability and in obtaining additional ARC grants. 

A.2.2.2 Hindrances

ARC requirements. Three of the education projects indicated that the timing of the ARC’s grant cycle 

represented an immediate obstacle. Specifically, when a one-year grant award is made in October, the 

project only has a few months to sign agency agreements, obtain equipment, and recruit staff. Even then, 

the project must implement its strategy and accomplish its goals in a single school semester (i.e., between 

January and June). Under this scenario, projects indicated that they require more time to recruit and 

engage teachers.

Another education project indicated that it tried to expand into another county to serve economically 

disadvantaged students but could not due to differences between the ARC’s definition of a distressed 

county and the definition used by the state. As a result, the project’s ability to make use of its ARC funds 

was restricted to students living in a distressed county (the project ultimately obtained funding from 

another source to serve students in other parts of the state).

Institutional barriers. Several education agencies indicated that they found it difficult to recruit skilled 

instructors because they were unable to offer salaries comparable to those in private industry. One project 

reported that the potential scope of its impact was limited because the lead organization had a policy 

against providing assistance with transportation and child care costs, thereby limiting the number of 

students that were able to participate in project-related activities.

Lack of stakeholder involvement. While some projects identified stakeholder support as a facilitating 

factor, three projects indicated that lack of stakeholder involvement was a major hindrance. Specifically:

• In one state, ARC funds were awarded to build and design a facility under the direction of a 

county system. The staff who were assigned to move in and manage the facility were unaware 

the project existed and they were not consulted for input into the design of the facility. Because 

of this, the building lacked features (e.g., storage space) that could have enhanced everyday 

operations and functions. In addition, staff members have not been trained on how to use the 

video-conferencing equipment, so it is not used as intended. The director believes that if staff had 

been consulted during the planning phase, the building would be more functional and funding 

would have been allocated for staff training on how to use the equipment. 
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-11 

• One project described the difficulty of getting political officials to support education projects—

with the respondent noting that public officials were more supportive of projects that were visible 

to the public (e.g., water and infrastructure) than they were for education projects. 

• A third project was closed after one year because they did not completely analyze and identify a 

critical stakeholder. Even though their first-year goals were met and they were successful, the 

second-year funding was blocked by a person with the authority and position to make decisions 

over project priorities. Project staff believed that had this person been recognized as a critical 

stakeholder in the beginning of the project and fully engaged in the first year, that decision might 

have been avoided.

As a result of these experiences, respondents suggested that future education and workforce development 

projects take steps to publically offer their support for ARC-supported activities as often as possible. 

Examples of recognition opportunities include newsletter features and photos, certificates at graduation 

and award ceremonies, and opportunities for program participants to meet and speak with public officials. 

One respondent pointed out that in addition to benefiting staff and participants, the inclusion of project 

and community leaders can increase the likelihood that elected officials will view education as a 

component of economic development (and therefore offer continued support for the project). 

Unintended consequences of successful training. One project indicated that it was not as successful as 

it might have been because students dropped out of its program as soon as they had obtained the skills 

needed by a local company. The respondent noted that if a project provides exceptional job training, it is 

common for students to drop out of the academic program to enter the workforce. Education providers 

are then charged with a negative outcome because they must record those individuals as “noncompleters.” The project director indicated that he learned to communicate to industry that early exits 

were seen as a “negative” by his funding source, and he began encouraging industry representatives to 

agree not to hire participants on a full-time basis before they finish the entire program. If a business does 

hire students before they complete the program, they are asked that the students not be assigned overtime 

so that they can complete the program on a part-time basis. As a result of this experience, the project 

suggested that future projects take steps to ensure that industry stakeholders understand the operating 

parameters of education and training programs.

Lack of technology literacy. Project managers in three sites were hindered by their own lack of 

knowledge of the technology they were attempting to integrate within their programs. They indicated that 

this lack of access to a technology specialist meant that they were at the mercy of single product vendors, 

were unable to easily navigate complex Federal Communication Commission rules and regulations, and 

were not able to provide technology training to their staff. 
A Case Study Findings

 A-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

A.3 Projects’ Effort to Collect and Utilize Outcome Data

As discussed in Chapter 3, almost 40 percent of the 222 projects that completed an education and 

workforce development questionnaire tracked participants after program completion, with 18 percent 

tracking participants for one year and 19 percent tracking participants for two or more years. The 

remaining projects either did not track program participants (29 percent) or indicated that such tracking 

was not applicable to the types of services they provided (33 percent).

A.3.1 Factors That Prompted Projects to Collect Outcome Data

The ARC currently has no requirements that education and workforce development projects collect 

longer-term outcome data from former participants. Nor are there any systematic requirements that 

projects provide the ARC with outcome data after their grants have ended. As such, a purpose of the 

telephone interviews was to determine why projects chose to collect outcome data from former 

participants when they were not required to do so. 

Almost all (12 of 13 projects) indicated that they collected data from former participants because of a 

requirement of an agency other than the ARC. Other agencies requesting data from or about former 

participants included state departments of education or workforce development, state higher education 

commissions, a county education department, the U.S. Economic Development Administration (EDA), an 

accrediting agency (e.g., a nursing licensing board), another federal government agency (e.g., the National 

Institute of Standards and Technology), or the National Mathematics and Science Initiative. It should be 

noted that even though these data elements were collected, the ARC projects frequently did not have 

immediate access to the data (although they could access the data if necessary). In addition, because there 

is no effort to utilize standardized definitions or collection procedures, it would be difficult to compare or 

aggregate these outcome data across ARC projects.

Several projects indicated that they collected outcome data to document the project’s effectiveness and 

efficiency for local stakeholders. Two projects kept data to document the effectiveness (or lack thereof) 

of interventions (one to increase the college-going rate of students and the other to measure the impact of 

technology on instruction). The third project was of particular interest to the governor’s office, and staff 

wanted to document the initiative’s impact on adult literacy levels. A fourth project kept data on 

spending to document fiscal responsibility to stakeholders and political watchdogs. 
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-13 

A.3.2 Types of Data Collected From and About Former Participants

During the telephone interviews, the 13 projects were asked to describe the types of data that they were 

collecting from former participants. Although the intent of the question was to learn about methods used 

to collect data directly from participants, many of the respondents provided information about steps being 

taken to obtain data from existing sources about former participants.

As shown in Exhibit A-1 (at the end of the appendix), four projects were collecting information about 

former participants’ employment and wage status. Specifically:

• Two of these projects were using surveys to obtain these data directly from students. For 

example, one project had schools mail surveys to a student’s last known address, with some 

faculty conducting follow-up with nonresponders (the survey is issued annually for up to five 

years after students complete the program). Both projects were challenged by low response rates, 

lack of staff time to conduct follow-up with nonrespondents, and a reliance on self-reported data 

on employment status that are difficult to validate. 

• A third project used faculty members’ unofficial knowledge about the employment status of their 

former students to prepare an annual report for the state department of education. However, this 

method relied on anecdotal information that was not subsequently verified, and no effort was 

made to obtain information about those students for whom such information was not easily 

obtainable. 

• A fourth project used a database maintained by a state agency (established to align universities, 

government, and industry resources) to mine existing data related to individual taxes, 

unemployment, and education enrollment. Using these existing data, the project was able to 

determine if a particular student was in school, was earning a paycheck, was drawing 

unemployment, or a combination thereof. While the data were unobtrusive and reliable, they 

were not collected directly from former participants. As such, the information was only made 

available in the aggregate and could not be linked to the characteristics and experiences of 

individual students.

Three projects tracked the numbers of high school graduates who attended a postsecondary institution. 

Two of these projects conducted surveys of former participants to obtain this information—with both 

projects reporting that low response rates and the lack of validation methods minimized the value of the 

data for project monitoring and program planning. As such, they did not believe that the data they had 

collected could be used to reliably help the ARC staff assess emerging trends or consider steps that might 
A Case Study Findings

 A-14 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

be taken to improve their programs A third project used the National Student Clearinghouse (NSC) to 

examine students’ enrollment in postsecondary education.1

Finally, three projects used existing databases to obtain information about the number of former 

participants that had taken a high school AP course/exam, passed a licensing or certification exam, or 

passed their GED exam. However, one project indicated that it was only able to obtain program data for 

those program graduates that actually attempted the licensing exam, making it difficult to calculate a valid 

and reliable “success rate.”

A.3.3 Steps Taken to Enhance Response Rates and Validate Data

The collection of valid and reliable outcome data requires that steps be taken to minimize the potential for 

nonresponse bias and respondent errors. As such, organizations responsible for collecting and using 

outcome data are often required to implement steps designed to conduct follow-up with nonrespondents, 

as well as with respondents that appear to have provided erroneous information (e.g., inconsistent 

information across survey items). This section provides information on the extent to which the 13 

projects that collected data took steps to enhance their response rates and validate outcome data.

Steps taken to enhance response rates. During the follow-up interviews, projects were asked to discuss 

response rates to their attempts to collect follow-up data. The 13 projects interviewed had no 

standardized or operational definition of the term “response rate.” Participants used words such as “low,” 

“very high,” or “I am not sure, another department keeps that information.” Others had specific numbers, 

although they indicated those numbers were estimates. It is important to note that only seven of the 13 

projects actually attempted to collect data from and about participants. 

The only two projects that attempted to collect follow-up data directly from former participants reported a 

low response rate. In addition, they made little effort to follow up with participants to gain missing 

information. One program indicated that although their staff tries to follow up with phone calls, project 

personnel have no idea how much additional data are actually collected as a result of these efforts. None 

of the projects knew exactly how many individuals were tracked over time—with lack of time and 

resources being cited as the reason for not trying to reach nonrespondents. 

 

1

 The NSC is a subscription service for degree and enrollment verification. The NSC tracks the student’s college attendance status for six years. 

Schools and colleges upload information to the database annually and the project is able to document the education status (enrollment, course 

work) for the student for up to six years.
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-15 

Steps taken by projects to validate data about former respondents. None of the projects that 

collected data directly from former participants reported any action to validate the data they received. 

Two projects accepted information on student submitted surveys, while a third accepted data submitted by 

staff. (The remaining projects used third-party systems to collect data, and two accepted reports by a 

testing center and licensing board.) The two projects that obtained data from student surveys indicated 

that their projects struggled to find staff and time to simply collect the data. As such, they lacked the 

resources to subsequently validate the data.

A.3.4 Types of Data Projects Were Unable to Collect From Former 

Participants

Projects indicated that while they would have liked to collect wage data from (or about) their former 

participants, most respondents were likely unwilling to provide such information about themselves. 

Three projects indicated they saw a privacy issue, reflected by students’ “none of your business” answers 

on surveys. Another project indicated they could not ask for any personal data from students because of 

privacy laws.

Projects also discussed the difficulty of using third-party systems to obtain information about former 

participants. For example, two projects indicated that they were unable to obtain data about participants 

that had moved to another state—with one project indicating that it was only able to track students’ 

enrollment in in-state universities and colleges.

A.3.5 How Projects Made Use of Data About Former Participants

Projects made multiple uses of data about former participants—including marketing, operations, 

documentation for accreditation, and as a basis for generating additional funding. For example: 

• Four projects discussed using data for program improvement initiatives, with two projects using 

the data to evaluate the efficacy of training programs (e.g., by demonstrating the skills that 

students had learned). One project used participant feedback to improve campus services and 

another used student achievement data to research the effectiveness of traditional classroom 

instruction as compared to distance learning.

• Three projects used data to support their marketing efforts in newsletters, brochures, and 

websites. The ultimate goal of the marketing efforts was to increase future enrollments and 
A Case Study Findings

 A-16 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

secure additional funding. One project used its data on enrollment and job placement to justify 

additional staff for student services.

• Three projects used data to provide documentation for accreditation and licensing boards to 

obtain or maintain necessary credentials. The most common data elements included completion 

rates, employment status, testing results, and enrollment figures.

A.3.6 Information About Projects That Did Not Collect Follow-up 

Data From Former Participants

In an effort to understand factors that prevented projects from collecting follow-up data, we also 

conducted telephone interviews with 10 projects that reported on the educational and vocational education 

questionnaire that they did not collect any follow-up data from participants. The purpose was to ascertain 

(1) the types of outcome data they would have liked to collect from program participants, (2) the factors 

that prevented them from collecting these data, (3) how their project would have been able to make use of 

these data, and (4) any technical assistance that the ARC could have provided that would have made it 

possible to collect these data.

As shown in Exhibit A-2 (at the end of the appendix), the most prominent data that projects would have 

liked to collect were college-going rates (three projects) and the number of students employed in the field 

for which they were trained (three projects). Most of the other data elements suggested by projects would 

have required sophisticated techniques to accurately measure economic impact, employer satisfaction, 

new business start-ups, and productivity improvement. 

The potential uses for these data primarily focused on marketing and recruitment, documentation that 

could have been used to raise additional funding, validation of industry needs, stakeholder 

communication, and a demonstration of their return on investment. Projects provided a common set of 

reasons for not collecting the data—including lack of staff and resources, lack of access to an existing 

database, lack of assistance by a school or college, and an inability to track students who leave the 

community. In addition, five of the 10 projects indicated they did not have to collect data because they 

were technology or construction projects and the collection of outcome data was not required by the 

ARC. For example, one respondent did not see the need to document the outcomes of all the participants 

going through his school to justify construction of a computer lab. He said he did not see the connection 

between an agency that funded bricks and mortar for a training center and the effectiveness of the 

program housed inside the new building. 
Case Study Findings A 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-17 

A.3.7 Lessons Learned and Opportunities for Technical Assistance 

From ARC

During the interviews, projects were asked if they had any suggestions that they would share with future 

ARC education and workforce development projects regarding the collection of outcome data. The most 

prominent recommendations involved the development of common data collection instruments and 

practices (e.g., frequency of data collection, methods for reporting data) that could be shared across 

projects. Other projects offered suggestions for dealing with specific data collection challenges issues. 

For example, one respondent advised projects to use emails and cell phone numbers (instead of land lines 

and street addresses) to contact former participants, since individuals are more likely to maintain their 

email addresses and cell phone numbers when they move. Another respondent suggested that graduating 

students be informed about any upcoming surveys (including how the resulting data will benefit the 

school and the program) so that they are more likely to provide any requested information in future years. 

A third project suggested that data be maintained in a single repository so that they are easily accessible 

and can be accessed by multiple individuals.

Projects also offered suggestions for how the ARC could assist projects that want to collect data from 

former participants. Since the ARC does not currently require projects to collect such follow-up data, the 

recommendations generally came with the following caveat: “if they start requiring projects to collect 

data it would be helpful if...” Given this caveat, the most common request was that the ARC provide 

projects with (1) operational definitions for such key terms as college-going rate, full-time student, and 

full-time equivalency; (2) guidelines about how to address specific challenges (e.g., whether or not to 

collect data from participants that leave a program before they graduate); (3) templates and other specific 

guidelines as to how they wanted data to be presented; (4) suggestions for third-party data collection 

services that could be used to obtain valid and reliable data; and (5) access to experts that could provide 

project-specific guidance on how to collect and report outcome data. In addition, one project suggested 

that the ARC develop a portal on its website that could be used to obtain data from individual projects—

indicating that such a portal would compel projects to utilize standardized definitions and provide projects 

with a means of comparing their results with those of other similar initiatives.
Education 

Evaluation of the Appalachian Regional Commission’s 

and Workforce Development Projects: 2000–2008 A-18 

 

A Case Study Findings

Exhibit A-1. Information about data that projects collected from or about former participants

Data element Number of 

projects Data collection method Frequency of data collection Issues

Post-program 

employment and 

wage status

4 

Teachers use a telephone or email survey 

to collect data directly from former 

students.

At a minimum, teachers report students’ 

status at graduation and three years after 

graduation. Some programs track 

students for a longer period (based on 

program accreditation requirements).

Low response rate.

Self-reported data with no validation.

No time and staff to follow up with all 

nonrespondents.

Students cautious about reporting 

income data.

Data collection initiated by a survey 

administered with students prior to 

graduation. After program completion, 

the state agency sends a survey “alert 

card” to the students followed by a 

survey. Schools are responsible for 

follow-up, although only a few actually 

make phone calls.

Students complete a survey before 

graduation, three months after 

graduation (usually September), and 

annually for five years.

Low response rate (20 percent).

Self-reported data with no validation.

No time and staff to follow up with all 

nonrespondents.

Students cautious about reporting 

income data.

Instructors are provided a list of students 

who had completed the program. 

Teachers are asked to report the current 

status of those students based on 

anecdotal information from the 

community grapevine. 

Data are compiled on an annual basis 

and submitted to the state department of 

education.

Anecdotal information not always valid.

Data only collected from students for a 

single year.

No effort made to determine status of 

students not known by teachers.

Data collected by a third party (not 

directly from former participants). The 

state established an office charged to 

collect data on workforce programs.

Students are issued a state ID number 

that is used to track their participation in 

training programs, employment status, 

and wages on a shared database.

Training programs generate reports on a 

quarterly basis and submit data at the 

end of the year to the state office. The 

state office reports outcomes in an 

annual report.

Students are tracked for two years after 

completing training. Students that do 

not show up in the database for two 

quarters are classified as unemployed.

The system is unobtrusive, tracks a high 

volume of former students, and obtains 

reliable data as long as students are still 

living in the state.

Projects do not receive data about 

individual students (only aggregate data 

about previous participants).
A-19 Evaluation of the Appalachian Regional Commission’s Education and Workforce Development Projects: 2000–2008 

 

Case Study Findings A 

Exhibit A-1. Information about data that projects collected from or about former participants—continued

Data element Number of 

projects Data collection method Frequency of data collection Issues

Number of high 

school graduates 

that transition to 

postsecondary 

education

3 

Data collected directly from students. 

Teachers use a standard survey to 

administer a telephone or email survey 

with their former students.

At a minimum, teachers report students’ 

status at graduation and three years after 

graduation. Some programs track 

students for a longer period (based on 

program accreditation requirements).

Low response rate.

Self-reported data with no validation.

No time and staff to follow up with 

nonrespondents.

Data collected directly from students 

using a survey prior to graduation. After 

program completion, the state agency 

sends a survey “alert card” to the 

students followed by a survey. Schools 

are responsible for follow-up, although 

only a few made phone calls.

Students complete a survey before 

graduation, three months after 

graduation (usually September), and 

annually for five years.

Low response rate (20 percent).

Self-reported data with no validation.

No time and staff to follow up with 

nonrespondents.

The state subscribes to the National 

Student Clearinghouse (NSC) to track 

the students’ status after graduation. 

Schools upload the information to the 

database on an annual basis.

The NSC collects and reports data on 

students for six years after graduation 

and reports data on an annual basis.

NSC depends on schools submitting 

information on students.

Subscription to the NSC is expensive.

There is no incentive for schools that 

drop out of the project to continue 

uploading the student data.

Number of AP 

courses and exams 

taken

1 

Data collected from a third party. The 

College Board issues a report to the 

school, which is included in the National 

Math and Science Initiative report. 

Annually The public school has the ability to track 

students’ college attendance status if 

they attend college in their home state, 

but the project does not keep this 

information.

Status of people 

passing licensing/ 

certification exams

1 

Data collected from a third party. State 

Licensing Nursing Board reports the 

names of those passing the exam and 

those who have not passed.

Students have the opportunity to sit for 

an exam multiple times of the year. The 

board only reports the status of the 

student after they have passed the exam. 

Those students that are retested are 

tracked until they pass the exam. 

Program only has data for students who 

sit for the test; there is no accounting for 

students who do not graduate, who 

move, or who choose not to sit for the 

exam. 
Education 

Evaluation of the Appalachian Regional Commission’s 

and Workforce Development Projects: 2000–2008 A-20 

 

A Case Study Findings

Exhibit A-1. Information about data that projects collected from or about former participants—continued

Data element Number of 

projects Data collection method Frequency of data collection Issues

Status of people 

who passed the 

GED

1 

Data collected by a third party. Test 

scores are released by the GED test 

centers. Teachers in the centers report 

any known information about the 

students the year following passage. 

Information is compiled into a report and 

submitted to the state department of 

education. 

Results are reported after each test and 

compiled into an annual report each 

year.

Student placement is based on program 

staff knowledge of the students. As 

such, the report may not contain 

information about the same students over 

time. 

NOTE: Information only collected from projects that obtained at least some data from or about former participants.
A-21 Evaluation of the Appalachian Regional Commission’s Education and Workforce Development Projects: 2000–2008 

 

Case Study Findings A 

Exhibit A-2. Information about data elements that projects would have liked to collect from former participants

Data elements

Number of

projects How data would have been used by projects Reasons data were not collected by project

College-going rate 3 

Recruitment of more students in the program and to 

recruit more teachers to teach dual credit courses.

No existing mechanism to track students, nor any knowledge 

about how to initiate such a process.

Lack of assistance from colleges.

No way to track student who leave the community.

Document merit of the grant. Do not know how to track to get accurate data.

Lack of an existing database. 

Document program’s success as a basis to apply and 

receive future grants.

No way to track student who leave the community.

Cannot supervise guidance counselors to standardize the 

definition of “college-going rate,” so you aggregate across 

schools.

Number of students 

employed in the field 

for which they were 

trained

3 

Basis for program improvement.

Document program’s success as a basis to apply and 

receive future grants.

It was a construction project, and data collection was not 

required.

Grant ended after one year; did not report after the project 

ended.

Economic impact 1 Recruitment of more students in the programs offered in 

the facility. Technology upgrades, no follow-up data required.

Employer satisfaction 1 

Update and recruit advisory committee members.

Make a case for sustainability.

Identify what employers need to modify programs.

Lack of staff and resources.

Number of students 

starting new 

businesses 

1 

Justification to continue the program.

Document project’s success as a basis to apply and 

receive future grants.

Lack of staff and resources.

Number of students 

that gained a desired 

skill

1 

Document number completing the training for use by 

economic developers.

Attract new industries to the region.

Not required because it was a technology grant.

Lack of staff and resources.

No access to monitor teachers’ classrooms.

Productivity 

improvement 1 

Document project’s success as a basis to recruit more 

local businesses to use the facility to upgrade training of 

their employees.

Technology upgrades, no follow-up data required

NOTE: Information only collected from projects that did not attempt to obtain any data from or about former participants.
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 A-22 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-1 

Appendix B: 

Opportunities and Challenges Regarding the Collection 

and Analysis of Economic Outcomes for the ARC’s Education 

and Workforce Development Projects
 B-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-3 

Opportunities and Challenges

Regarding the Collection and Analysis of Economic Outcomes

for the ARC’s Education and Workforce Development Projects

This appendix describes results of an analysis of the economic impact of participation in ARC education 

and workforce development programs. Specifically, the analyses reported in this appendix utilize 

findings the questionnaire that Westat administered with the 386 education and workforce development 

projects that were funded between 2000 and 2008. The purpose was to extract available information on 

the effect of participation on worker/trainees’ employment status and income level.

The use of the questionnaire to collect enhanced outcome data is central to this analysis of economic 

impacts. Taken together, the questionnaire and the resulting analysis was part of an effort to examine the 

feasibility and value of expanding the unified ARC.net database of outcomes associated with grant 

programs. The purpose was to assess whether the collection of enhanced outcomes data would allow for 

the measurement of the benefits of ARC programs on its constituent counties. Due to low response rates 

on the questionnaire items pertinent to the economic status, the analyses presented here can only provide 

examples of the types of economic impacts that the ARC’s education and workforce development 

programs generate.1

In interpreting the findings in this appendix, it is important to keep in mind that the collection of outcome 

data was not originally required at the time that these grants were awarded, and post-project monitoring 

and record-keeping are not currently required (outside of the validation visits conducted with a sample of 

projects). Therefore, only a portion of projects kept records of participants’ post-program outcomes and 

were able to provide this information. The analysis results reported in this appendix are thus useful in 

highlighting both potential benefits and current limitations pertaining to the ways that ARC education and 

workforce development grantees are asked to maintain data and track project outcomes. 

Methodological Approach 

This analysis utilizes projects’ responses from the following sections in the questionnaire: Background 

Information, Project-Related Activities, and Economic Impacts. The analysis discussed in this appendix

is centered on questions in the sections Project-Related Activities (Questions 14–16) and Economic 

Impacts (Questions 31–35).

To maximize the number of projects that could be included in the analysis, we first considered the larger 

of the two (improved or served categories) summations of workers/trainees and students, filtering for ages 

19 and over for each questionnaire. If the number was zero but the participants’ served or improved 

fields were positive, then the smaller of the two participant fields (participants improved or participants 

served) was used to fill in the blank. As shown in Table B-1, this approach resulted in a total of 134 

 

1

 As reported in Chapter 2, the questionnaire received responses from 222 of the 386 education and workforce development projects that received 

funding between 2000 and 2008. However, as is discussed throughout this appendix, only a small portion of grantees formally tracked how 

clients fared after completing the programs supported by ARC grants.
B Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects

 B-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

observations as a baseline (as opposed to the 78 that resulted from removing projects that only served 

youth ages 18 and under). These 134 projects were matched to the percentage fields of pre- and postemployment status. The lower percentage of pre- or post-status participation was used in cases where the 

observations had some percentage of unknown employment and pre- and post-reporting did not match. 

As discussed below, this multistage process resulted in a total of 39 observations: 

• First, projects that did not provide a number greater than zero for any of the categories on 

Question 14 were removed from the analysis. These categories include students served, students 

improved, workers/trainees served, workers/trainees improved, participants served, and 

participants improved. As shown in Table B-1, this resulted in the removal of 43 projects—with 

179 projects providing a number greater than zero for at least one of these categories.

• Second, projects that only served youth (i.e., persons age 18 and below) were removed from the 

analysis since they were not expected to have any employment outcomes. As shown in Table 

B-1, 78 of the 166 projects that served at least one person age 19 and over provided a number 

greater than zero for at least one of the categories in Question 14 (without reporting “don’t 

know”) and provided at least some of their education and workforce development services to 

individuals age 19 and older. When allowing for the reporting of “don’t know” for one or more 

categories in Question 14, 134 of the 166 projects provided a number greater than zero for at 

least one of the categories and provided at least some of the their education and workforce 

development services to individuals age 19 and over. For the purposes of this analysis, we added 

56 projects that indicated “don’t know” in their response to Question 14. 

• Third, projects that did not provide data on employment outcomes were removed from the 

analysis. As shown in Table B-1, 55 of the 179 projects that reported a number for Question 14 

provided data on the pre-employment status of their participants—and 35 of these 55 projects 

also reported on post-employment status. A smaller number (13) also provided information on 

both the industry type and pay levels for program participants’ jobs. It is important to note that 

even though only a fraction of respondents were able to provide complete information, they still 

covered training for a large number of participants. The 35 projects that had pre- and postproject employment data represent roughly 441,800 participants; the 13 projects that also had 

information on job characteristics served more than 7,800 participants. 

• Finally, by using a strict interpretation of the questionnaire responses regarding ages served, the 

number of projects with trainees 19 or older that reported data on post-employment and salary 

were 22 and 11, respectively. To maximize the number of surveys that could be used for this 

economic analysis, we allowed for the inclusion of (1) “don’t know;” (2) either pre-employment 

or post-employment data (increasing the number of usable projects from 22 to 39 for the analysis 

of pre- and post-employment); and (3) either industry or salary data (increasing the number of 

usable projects from 11 to 25 for the analysis of industry and salary).
Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects B 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-5 

Table B-1. Number of questionnaire responses used in the analysis of economic and income 

outcomes

Nesting criterion

Responses 

based on 

nesting criteria

Nesting criteria 

screened for 

age (19+)

Adjustments to 

maximize usable 

questionnaires for 

economic analysis

Total number of questionnaires that met the initial 

screening criteria 222 166 166

Projects that reported a number greater than 0 for 

one or more participant categories on Question 141 179 782 1343

Projects that provided data on participants’ preemployment status 55 28

394

Projects that provided data on participants’ postemployment status 35 22

Projects that provided data on industries in which 

former participants were employed 15 13

255

Projects that provided data on former participants’ 

salaries 13 11

1

Observations for which the sum of Question 14 categories was greater than 0: students served, students 

improved, workers/trainees served, workers/trainees improved, participants served and participants improved. 

Blanks or nonnumerical notations (such as question marks) were treated as 0. Following this tabulation, in 

subsequent parts of this analysis the category of “participants” was redefined to maximize usable questionnaires 

and responses for the analysis of economic impact (see numbers under Adjustments to maximize usable 

questionnaires for economic analysis). 

2

Does not include records where “don’t know” was reported for any of the Question 14 categories.

3

Allows for “don’t know” in one or more of the Question 14 categories.

4

Pre-employment and post-employment was greater than 0, and the smaller of the two answers was used if the 

answers did not agree.

5

Either salary or industry was answered. If salary did not account for 100 percent of participants, industry-level 

data from the Bureau of Labor Statistics were used to fit to industry responses.

NOTE: One ARC project was not included because the data contained made it an extreme exception, although 

using this project would bring the total usable employment questionnaires to 40 and salary questionnaires to 26. 
B Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects

 B-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Several caveats are worth noting. First, the decision rules described above were adopted to maximize the 

number of questionnaires that could be used to estimate job and income impacts. This decision allowed 

for the largest sampling possible, given the data set, for each metric. As a result, more projects were 

available for the employment analysis than the income analysis. In both cases, though, the number of 

valid questionnaires allowed for normal distributions for each metric. 

Second, the numbers presented in this discussion do not match the data provided in Chapter 3. Even 

though both sets of analyses are based on the same questionnaires, different assumptions regarding item 

nonresponse and the handling of don’t know were used in the frequencies presented in Chapter 3 and the 

economic impacts derived in this appendix. In addition, unlike the procedures used to calculate 

frequencies in Chapter 3, many of the calculations in this discussion were dependent on prior calculations 

(as illustrated by the nesting criteria in Table B-1, which are used to show the inter-relationships of 

multiple responses). 

Findings for the Projects That Completed a Questionnaire

Jobs. As shown on Table B-2, the 39 job training projects that tracked both pre-program and postprogram employment status led to the following: 

• Full-time or part-time jobs for 3,339 previously unemployed people (accounting for 72 percent of 

participants who entered the programs while unemployed); and 

• 4,351 workers securing full-time jobs who were previously unemployed or underemployed 

(working part time), accounting for 47 percent of program entrants who were unemployed or 

working part time.

Since these 39 projects represent approximately one-sixth of the total projects, it might be inferred that 

total program impacts on employment status are likely to be roughly four times the values shown in Table 

B-2 (when limited to clients 19 years or older who are legitimate workforce candidates). 

Table B-2. Summary of participants’ employment status changes for the 39 projects that had 

pre/ post-employment status data

Employment status When entering 

program

When exiting 

program

Difference

Not employed 4,646 1,307 -3,339

Employed part time 4,544 3,532 -1,012

Employed full time 13,031 17,382 4,351

Total 22,221 22,221 0 
Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects B 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-7 

It should be noted that it is not possible to confirm the extent to which individual participants moved 

among the employment status categories shown in Table B-2 because the questionnaire obtained data on 

aggregate program totals (i.e., there were no participant-level data that allowed for the matching of preand post-project employment status). Nonetheless, the aggregate results clearly show an aggregate 

reduction in part-time workers and an even larger reduction in unemployed persons, with a net gain in 

full-time workers. In addition, by applying a set of generally reasonable calculation rules2

 to the results 

of the 39 projects with pre- and post-project employment data, it is possible to develop a rough 

characterization of the ways in which workers are likely to have moved among employment categories as 

a result of ARC-funded workforce development projects. Thus, as shown in Table B-3, 282 individuals 

transitioned from being unemployed to part-time employment, while 3,057 transitioned from 

unemployment to full-time employment. Using these assumptions, we calculate that 4,633 out of 22,221 

participants aged 19 and older improved their job employment status (at least in part) due to the 39 ARC 

education and workforce training projects that provided pre- and post-project employment status.

Table B-3. Estimated patterns of change in employment status following ARC workforce 

training programs for 39 projects that had pre/post employment status data

Pre-participation 

status

Number of 

participants

Post-participation status

Unemployed Part time Full time Total

Unemployed 4,646 1,307 282 3,057 4,646

Part time 4,544 3,250 1,294 4,544

Full time 13,031 13,031 13,031

Totals 22,221 1,307 3,532 17,382 22,221

NOTE: These patterns of employment change were estimated based on assumptions that participants did not move 

backwards in job status after completing their programs.

Income. Of the 39 projects that provided pre- and post-project employment status, approximately onethird could not be used to calculate improved income because incomplete data (missing income or 

employment industry data) or errors of addition (either data fields did not sum to 100 percent or included 

some percentage of participants as unknown income brackets). For a record that was missing relevant 

income information but contained complete industry information, the average industry specific income 

bracket data from the Bureau of Labor Statistics were used to plug the missing observation's values. As a 

result of this hybridization, 25 of the 39 records (64 percent) were able to be preserved so that we could 

generate income effects, whereas a more absolute methodology would have done well to preserve less 

than half as many.3

 

2

 The rules that were adopted are summarized at the end of this appendix. 

3 There were 26 validated responses, but one project was not counted because it was considered an extreme outlier.
B Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects

 B-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Altogether, 25 of the projects had at least partial information on the post-project industries and salary 

levels of 10,759 participants (Table B-4).4

 The results indicated the following outcomes:

• 6,368 participants maintained their job status and increased their income (i.e., remained part time 

or remained full time);

• 3,559 participants improved their job status and increased their income (i.e., moved from 

unemployed to part time, unemployed to full time, or part time to full time); and 

• 832 participants remained unemployed. 

The total income change for the participants in these 25 projects is estimated at roughly $104 million in 

annual personal income, or an annual increase of $9,700 per participant. Over an assumed 20-year work 

history, this increase would total $194,000 in undiscounted value per participant and $121,000 when 

discounted over time at 5 percent per year.5

 For participants who improved their employment status 

(moved from unemployed to full- or part-time employment, or moved from part-time to full-time 

employment), annual personal income increases averaged more than $19,700. ARC investments in these 

25 projects totaled $5.4 million, indicating a return on investment per participant exceeding $19 in annual 

income per dollar of one-time ARC investment. Total program costs, including ARC and other funding 

sources, were $12.9 million, which indicates an $8 return of annual income for one-time total project 

costs. Changes in annual personal income are shown in Table B-4. 

Table B-4. Changes in annual personal income for the 25 ARC project with complete economic 

impact responses

Change in status Incremental 

income

Incremental income 

per participant

Number of 

participants

Unemployed to full time $55,236,000 $24,700 2,234

Unemployed to part time 3,908,000 15,200 257

Part time to full time 11,118,000 10,400 1,068

Remained full time 21,623,000 6,300 3,410

Remained part time 12,124,000 4,100 2,958

Remained unemployed 0 0 832

Totals 104,009,000 9,700 10,759

NOTE: Dollars are rounded to the nearest hundred.

 

4

 This information was analyzed, and supplemented when needed with data on the distribution of full-time and part-time industry salary levels 

drawn from the U.S. Bureau of Economic Analysis based in part on answers to Question 33 on industries in which participants were employed.

5 A 5 percent discount rate was assumed in the Economic Impact Study of Completing the Appalachian Development Highway System Final 

Report. June, 2008: Appalachian Regional Commission. Prepared by Cambridge Systematics, EDR Group, and HDR Decision Economics.
Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects B 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-9 

Expected Program Outcomes

One purpose of this analysis was to examine the feasibility of building a model that tempers project 

performance (number of clients who improve job status by program investment and increased wages by 

program investment) by program context (urban/rural counties, counties that are economically distressed 

or not distressed). This exploratory analysis should be viewed as an example of the types of modeling 

that could have been performed with rigor had there been a higher response rate for the economic items 

on the questionnaire. Necessarily, the analyses below were narrowed from 39 responses (for employment 

changes) and 25 responses (personal income improved) to 18 and 21, respectively, to account for 

statistical outliers and nonqualifiers (e.g., counties that are equivalent to national unemployment rates and 

therefore cannot be considered distressed or nondistressed).

Income gains due to improved job status. As a more narrowly focused investigation, we focused on 

just those participants who improve their employment status in terms of investment by ARC in job 

training programs. For the purpose of this analysis, improved status includes moving from part-time to 

full-time status or from unemployed to either part-time or full-time employment. Figure B-1 uses a 

simple linear relationship to illustrate the number of people who can be expected to improve their job 

status based on ARC program investments—i.e., the post-project relationship of total income of workers 

with improved employment status and the number of participants per project who have improved their 

status.

Figure B-1. Aggregate income per program by workers with improved 

employment status

$0

$50

$100

$150

$200

$250

$300

$350

$400

$450

$500

0 200 400 600 800 1000 1200

Aggregate income generated 

per program

Thousands

Number of paticipants with improved job status
B Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects

 B-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Effects of project location characteristics. We also examined the effects of participants’ geographic 

location on their income level. Specifically, we examined the effect on income level of participants’ 

location in economically distressed versus nondistressed counties and in counties that are urban versus 

rural. For the income level, unemployment rates of counties were compared to the national average. For

urbanicity, a measure of population density was used.

Counties with unemployment rates higher than 105 percent of the national rate were considered 

distressed, while counties with unemployment rates less than 95 percent of the national average were not 

considered distressed. Counties with unemployment rates falling between 95 percent and 105 percent are 

considered consistent with the national average.6

 Per participant, mean income ranged from $33,000 per 

participant in nondistressed counties to $21,000 in distressed counties (Table B-5). 

Table B-5. Income per participant with improved job status based on local distress 

Distress factor Mean income

Distressed counties $20,938

Nondistressed counties $33,082

Income per population density also shows a strong linear trend, indicating a greater likelihood that a 

program will successfully raise participants’ income level in urbanized areas that provide multifaceted 

economies and, therefore, greater job opportunities (Figure B-2). On average, we found that an additional 

person per square mile in program counties resulted in an expected increase of total income generated of 

$2,700 by participants who improve their job status.7

Figure B-2. Program income as a function of population density

 

6

 Source of unemployment data is the ARC-LEAP database.

7 Source of population density data is the ARC-LEAP database.

$0

$1,000

$2,000

$3,000

$4,000

0 500 1000 1500 2000 2500

Aggregate income by 

participanrts who improve 

job status

Thousands

Population density per county
Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects B 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 B-11 

Calculation Rules to Characterize How Participants Are Likely to Move Among 

Employment Categories as a Result of ARC-Funded Workforce Development Projects 

(Based on Pre- and Post-Employment Status Responses)

Still Unemployed: This value was calculated by taking the minimum value of the pre- and post-project 

unemployed responses8 under the assumption that the unemployment count could not increase during the 

duration of the program (i.e., someone employed becomes unemployed). 

Unemployed to Part Time: This value was calculated by taking the post-project employment response 

for employed part time and subtracting the calculated number of workers who remained part time. 

Still Full Time: This value was calculated by taking the minimum value of the pre- and post-project 

participation full-time worker responses, under the assumption that workers would not willingly go to 

part-time status and could obviously not advance to a status beyond full time.

Still Part Time: This value was calculated by taking the minimum value of the pre- and post-project 

participant part-time worker responses, under the assumption that workers would not decrease their level 

of employment, so logically the smaller of the two values must represent the population of workers who 

remained in their employment bracket.

Part Time to Full Time: This value was calculated by summing the pre-project participant part-time 

worker and full-time worker fields, and subtracting from the resulting value the calculations of those 

workers who remained in either their full time or part time positions.

Unemployed to Full Time: This value was calculated by subtracting from the number of post-project 

full-time workers the sum of workers who advanced from part-time to full-time positions or remained in 

their full-time position. 

 

8

 For pre-program, unemployment was the summation of “never previously employed” and “previously employed, but not when entered.” 
B Opportunities and Challenges Regarding the Collection and Analysis of

Economic Outcomes for the ARC’s Education and Workforce Development Projects

 B-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 C-1 

Appendix C: 

Proposed Addendum to the Application Form for

ARC Education and Workforce Development Projects
 C-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 C-1 

Section A: Background Information

1. In what counties will your ARC project be implemented—e.g., where will the training be 

provided and/or where will participants primarily be reached?

County State

County #1

County #2

County #3

County #4

County #5

County #6

County #7

County #8

County #9

County #10

2. Which of the following age groups will your project be targeting (check all that apply):

 0–18 

 19–24 

 25–29 

 30–39 

 40–49 

 50+ 

3. Which of the following populations will be targeted by your ARC project: (check all that apply):

 Pre-school children 

 K–12 students

 Out-of-school youth 

 Postsecondary students

 Adults (not including postsecondary students or teachers) 

 Teachers

 Other (specify): _________________________________________________________________ 
C Proposed Addendum to the Application Form for

ARC Education and Workforce Development Projects

 C-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

4. Which of the following general activities will be conducted or supported by your ARC project

(check all that apply):

 Educational attainment or achievement (e.g., for students pursuing a high school diploma or postsecondary degree)

 Adult education (e.g., literacy/numeracy, GED, basic work skills, etc.) 

 Career and technical education (e.g., skills and job training that leads to a career credential or 

certification)

 Workforce training (e.g., skills and job training that does not lead to a diploma or degree)

 Child development (e.g., child care, early childhood education) 

 Teacher training (e.g., skills enhancement for in-service and/or pre-service teachers)

5. Which of the following specific activities will be conducted or supported by your ARC project

(check all that apply):

 Skills/training that enhanced employability

 Specific occupational/job skills training or instruction 

 GED preparation 

 Computer skills training/instruction

 Apprenticeship or other job exposure opportunities 

 Instruction in business management 

 Instruction in knowledge/skills for a specific academic subject

 Develop or purchased educational materials (e.g., manuals, books, software, etc. 

 Develop or purchased curriculum, instructional program, or course 

 Provide special use classroom equipment (e.g., computers, networks, tools, science lab, etc.) 

 Provide distance learning infrastructure (e.g., software, equipment, technology) 

 Provide or improved physical structures (e.g., buildings, renovation, equipment, furniture, etc.) 

 Provide course in parenting skills 

 Pedagogy or teaching skills training for teachers or instructors 

 Conduct community outreach activities 

 Establish community or business partnerships 

 Provide social support services

 Distribute funds/mini-grants/stipend

 Provide or arranged for child care services

 Provide college counseling 

 Provide career counseling (e.g., discussions, diagnostic/aptitude testing) 

 Provide/develop job search/placement assistance (e.g., job bank, employer outreach) 

 Provide referrals to other agencies for job assistance /career counseling

 Other (specify): _________________________________________________________________
Proposed Addendum to the Application Form for

ARC Education and Workforce Development Projects C 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 C-3 

Section B: Anticipated Outputs, Outcomes, and Impacts

6. What specific outputs and outcomes do you expect to achieve as a result of your ARC project?

• Depending on your ARC project timeframe, include cumulative outputs and outcomes for up to 3 

years after project closeout.

• Please provide a numerical answer where appropriate. 

• Enter “DK” if you do not have information for a relevant measure (e.g., you are unable to 

anticipate the number of trainees/participants that will be served by your ARC project). 

• Enter “NA” if a measure is not applicable (e.g., your ARC project will not provide any services to 

students). 

Participant type

Anticipated 

number served 

or improved

Students served—the number of children and youth that will enroll in prekindergarten 

programs through 12th grade, as well as the number of adults that will enroll in 

postsecondary educational programs

Students improved—the number of student participants that will (1) enhance their 

knowledge or skills; (2) pass or graduate to the next grade or level necessary to continue 

their education; (3) receive an educational credential; (4) receive a career credential; (5) 

make progress toward a degree, diploma, or certification; and/or (6) obtain a job in the field 

for which they were specifically trained

Workers/trainees served—the number of individuals that will receive training or 

participate in an activity designed to enhance their employability, but not necessarily 

leading to a certification, diploma, or degree

Workers/trainees improved—the number of participants that will (1) obtain new/ 

improved knowledge or skills; (2) receive an educational credential; (3) receive a career 

credential; and/or (4) obtain/enhance their employment status (e.g., received higher pay or 

better positions)

Participants served—the number of individuals that will attend an educational 

presentation or program (e.g., a community-wide computer skills training course)

Participants improved—the number of individuals that will benefit from their 

participation in an educational presentation or program (e.g., the number of individuals 

that will enhance their computer skills and/or increase their Internet usage as a result of 

their participation in an educational presentation or program)

7. If applicable, list any additional outputs or outcomes that you expect to occur as a result of your 

ARC project:
C Proposed Addendum to the Application Form for

ARC Education and Workforce Development Projects

 C-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

8. What specific economic benefits do you expect to achieve as a result of your ARC project?

• Depending on your ARC project timeframe, include economic benefits for up to 3 years after 

project closeout.

• Please provide a numerical answer where appropriate.

• Enter “DK” if you do not have information for a relevant measure (e.g., you are unable to 

anticipate the number of trainees/participants that will be served by your ARC project). 

• Enter “NA” if a measure is not applicable (e.g., your ARC project will not create any jobs). 

Economic benefit Anticipated 

impact

Jobs created—the total number of (1) direct hires you expect to make as a result of the 

project’s operation (e.g. teachers, public safety, information services, etc.); and (2) private 

sector jobs you expect to create in the three years following the delivery of ARC-funded 

services or project completion. Does not include construction jobs for buildings funded by 

this ARC grant. In the case of part-time jobs, please convert these to full-time equivalent 

and round up to report whole numbers.

Jobs retained—the total number of jobs that will be retained because of an ARC 

investment needed to keep the business and jobs in continued operations in the area (e.g., 

training workers to use new machinery).

Business created—the total number of businesses that will locate in the region as a 

direct/indirect result of ARC-supported workforce training.

Businesses retained—the total number of existing regional businesses that will improve 

their competitiveness because they gained access to a more skilled labor force and/or 

because their existing workforce was upgraded.

Leveraged private investments—the total dollar amount for private sector financial 

commitments that will not be part of the project funding, but will follow as a result of the 

completion of your ARC project (such as an infrastructure project) or the delivery of 

services (e.g., worker training, marketing campaign, export promotion program).

Revenues increased (nonexport)—the total dollar amount for any increase in nonexport 

sales that will occur among participating businesses as a result of your ARC project.

Revenues increased (export)—the total dollar amount for any increase in export sales that 

will occur among participating businesses as a result of your ARC project.

Costs reduced—the total dollar amount for any cost reductions among participating 

organizations and businesses that will occur as a result of your ARC project.

9. If applicable, list any additional economic impacts that you expect to occur as a result of your 

ARC project:
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 D-1 

Appendix D: 

Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects
 D-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Report Name D-3 

Section A: Background Information

1. Did your ARC project provide any services in the past 12 months?

 Yes (answer questions 2-6) 

 No (skip to Section B)

2. In what counties has your ARC project been implemented—e.g., where was training provided 

and/or where were participants primarily reached?

County State

County #1

County #2

County #3

County #4

County #5

County #6

County #7

County #8

County #9

County #10
D Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects

 D-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

3. Which of the following age groups has your project targeted (check all that apply):

 0–18 

 19–24 

 25–29 

 30–39 

 40–49 

 50+ 

4. Which of the following populations have been targeted by your ARC project (check all that 

apply):

 Pre-school children 

 K–12 students

 Out-of-school youth 

 Postsecondary students

 Adults (not including postsecondary students or teachers) 

 Teachers

 Other (specify): _________________________________________________________________ 

5. Which of the following general activities has your ARC project conducted or supported (check 

all that apply):

 Educational attainment or achievement (e.g., for students pursuing a high school diploma or postsecondary degree)

 Adult education (e.g., literacy/numeracy, GED, basic work skills, etc.) 

 Career and technical education (e.g., skills and job training that leads to a career credential or 

certification)

 Workforce training (e.g., skills and job training that does not lead to a diploma or degree)

 Child development (e.g., child care, early childhood education) 

 Teacher training (e.g., skills enhancement for in-service and/or pre-service teachers)
Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects D 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 D-5 

6. Which of the following specific activities has your ARC project conducted or supported (check 

all that apply):

 Skills/training that enhanced employability

 Specific occupational/job skills training or instruction 

 GED preparation 

 Computer skills training/instruction

 Apprenticeship or other job exposure opportunities 

 Instruction in business management 

 Instruction in knowledge/skills for a specific academic subject

 Develop or purchased educational materials (e.g., manuals, books, software, etc. 

 Develop or purchased curriculum, instructional program, or course 

 Provide special use classroom equipment (e.g., computers, networks, tools, science lab, etc.) 

 Provide distance learning infrastructure (e.g., software, equipment, technology) 

 Provide or improved physical structures (e.g., buildings, renovation, equipment, furniture, etc.)

 Provide course in parenting skills 

 Pedagogy or teaching skills training for teachers or instructors 

 Conduct community outreach activities 

 Establish community or business partnerships 

 Provide social support services

 Distribute funds/mini-grants/stipend

 Provide or arranged for child care services

 Provide college counseling 

 Provide career counseling (e.g., discussions, diagnostic/aptitude testing) 

 Provide/develop job search/placement assistance (e.g., job bank, employer outreach) 

 Provide referrals to other agencies for job assistance /career counseling

 Other (specify): _________________________________________________________________
D Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects

 D-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Section B: Outputs and Outcomes for the Past Year

7. What specific outputs and outcomes have been achieved since the beginning of your ARC 

project?

• Provide cumulative data since the beginning of your project. 

• Please provide a numerical answer where appropriate.

• Enter “DK” if you do not have information for a relevant measure (e.g., you are unable to 

anticipate the number of trainees/participants that will be served by your ARC project). 

• Enter “NA” if a measure is not applicable (e.g., your ARC project did not provide any services to 

students). 

Participant type

Actual number 

served or 

improved

Students served—the number of children and youth that enrolled in prekindergarten 

programs through 12th grade, as well as the number of adults that enrolled in 

postsecondary educational programs

Students improved—the number of student participants that (1) enhanced their knowledge 

or skills; (2) passed or graduated to the next grade or level necessary to continue their 

education; (3) received an educational credential; (4) received a career credential; (5) made 

progress toward a degree, diploma, or certification; and/or (6) obtained a job in the field for 

which they were specifically trained

Workers/trainees served—the number of individuals that received training or participated 

in an activity designed to enhance their employability, but not necessarily leading to a 

certification, diploma, or degree

Workers/trainees improved—the number of participants that (1) obtained new/ improved 

knowledge or skills; (2) received an educational credential; (3) received a career credential;

and/or (4) obtained/enhanced their employment status (e.g., received higher pay or better 

positions)

Participants served—the number of individuals that attended an educational presentation 

or program (e.g., a community-wide computer skills training course)

Participants improved—the number of individuals that benefited from their 

participation in an educational presentation or program (e.g., the number of individuals 

that enhanced their computer skills and/or increased their Internet usage as a result of 

their participation in an educational presentation or program)

8. If applicable, list any additional outputs or outcomes that have occurred as a result of your 

ARC project:
Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects D 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 D-7 

9. (if students improved >0) Please indicate how students have been improved by your ARC 

project (check all that apply):

 Improved attendance on a daily/regular basis 

 decreased suspensions and other problem behaviors

 Enrolled in more challenging classes

 Remained in school when at risk of dropping out 

 Re-entered an educational program after dropping out 

 Improved school readiness

 Obtained basic or academic skills in a specific subject

 Obtained vocational and technical skills

 Obtained a high school diploma, GED, or equivalent 

 Tested at or above grade level

 Advanced to next grade level

 Increased scores on statewide assessments or other standardized tests

 Enrolled in college or postsecondary program 

 Obtained a postsecondary degree, credential, or certification

10. (if students improved >0) If applicable, list any additional student outcomes that have occurred 

since the beginning of your ARC project:

11. (if workers/trainees improved >0) Please indicate how workers/trainees have been improved by 

your ARC project (check all that apply):

 Obtained vocational and technical skills

 Obtained basic or academic skills

 Obtained other employability skills (e.g., work attitudes/habits) 

 Obtained a credential or certification

 Teachers or instructors enhanced their classroom practices

 Maintained current employment 

 Obtained new part-time employment 

 Obtained new full-time employment

 Increased job status and/or earned increased wages

 Retrained in another field and obtained new employment 

 Obtained/maintained employer-provided health benefits 

 Reduced dependence on public assistance

12. (if workers/trainees improved >0) If applicable, list any additional worker/trainee outcomes that 

have occurred as a result of your ARC project:
D Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects

 D-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Section C: Worker/Trainee Outcomes Over the Life of the Project

13. (if workers/trainees improved >0 in ANY year) How many of the workers./trainees that your 

ARC project has served since the beginning of your ARC project:

Had never been employed when they entered your program 

Had previously been employed—but were unemployed when they entered your program

Were employed on a part-time basis when they entered you program

Were employed on a full-time basis when they entered your program

Don’t know

Total

14. For those who were unemployed when they entered your program, how many:

Were still unemployed after completing your program

Obtained part-time employment after completing your program

Obtained full-time employment after completing your program

Don’t know

Total

15. For those who were employed on a part-time or full-time basis when they entered your 

program, how many:

Were unemployed after completing your program

Were employed at a lower level or at the same level (e.g., pay, skills, hours) after completing 

your program

Enhanced their employment status (e.g., pay, skills, hours) after completing your program

Don’t know

Total
Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects D 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 D-9 

16. How many of the workers/trainees served by your ARC project found work in the following 

industries:

Agriculture

Government

Manufacturing

Mining

Retail

Services

Other (specify)

Don’t know

Total

17. How many of the individuals served by your ARC project found work at the following annual 

wage rates (excluding benefits):

Less than $20,000/year

$20,001 - $39,999

$40,000 - $69,999

$70,000 - $99,999

$100,000 or more

Don’t know

Total
D Proposed Supplement to the Annual Report for

ARC Education and Workforce Development Projects

 D-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Section D: Economic Benefits for the Past Year

18. What specific economic benefits have been achieved since the beginning of your ARC project?

• Provide cumulative data since the beginning of your project. 

• Please provide a numerical answer where appropriate.

• Enter “DK” if you do not have information for a relevant measure (e.g., you are unable to 

anticipate the number of trainees/participants that will be served by your ARC project).

• Enter “NA” if a measure is not applicable (e.g., your ARC project has not created any jobs).

Economic benefit Impact

Jobs created—the total number of (1) direct hires you made as a result of the project’s 

operation (e.g. teachers, public safety, information services, etc.); and (2) private sector 

jobs created. Does not include construction jobs for buildings funded by this ARC grant. 

In the case of part-time jobs, please convert these to full-time equivalent and round up to 

report whole numbers.

Jobs retained—the total number of jobs retained because of an ARC investment and jobs 

in continued operations in the area (e.g., training workers to use new machinery).

Business created—the total number of businesses that located in the region as a 

direct/indirect result of ARC-supported workforce training.

Businesses retained—the total number of existing regional businesses that improved their 

competitiveness because they gained access to a more skilled labor force and/or because 

their existing workforce was upgraded.

Leveraged private investments—the total dollar amount for private sector financial 

commitments that was not be part of the project funding, but followed as a result of the 

completion of your ARC project (such as an infrastructure project) or the delivery of 

services (e.g., worker training, marketing campaign, export promotion program).

Revenues increased (nonexport)—the total dollar amount for any increase in nonexport 

sales that occurred among participating businesses as a result of your ARC project.

Revenues increased (export)—the total dollar amount for any increase in export sales that 

occurred among participating businesses as a result of your ARC project.

Costs reduced—the total dollar amount for any cost reductions among participating 

organizations and businesses that occurred as a result of your ARC project.

19. If applicable, list any additional economic impacts that occurred since the beginning of your 

ARC project:
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-1 

Appendix E: 

Online Questionnaire Administered With 

ARC Education and Workforce Development Projects
 E-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-3 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-5 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-6 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-7 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-8 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-9 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-10 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-11 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-12 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-13 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-14 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-15 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-16 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-17 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-18 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Online Questionnaire Administered With

ARC Education and Workforce Development Projects E 

Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 E-19 
E Online Questionnaire Administered With

ARC Education and Workforce Development Projects

 E-20 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 F-1 

Appendix F:

Protocol for Telephone Interviews With a Sample of ARC 

Education and Workforce Development Projects That 

Collected Follow-up Data From Former Participants
 F-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 F-3 

Telephone Interviews With a Sample of ARC Education and Workforce Development Projects That 

Collected Follow-up Data From Former Participants

Project name: _________________________________________________________________ 

Project ID: _________________________________________________________________ 

Respondent name: _________________________________________________________________ 

Date: _________________________________________________________________ 

Thank you for taking the time to talk to me today about your experience as an ARC grantee. The results 

of this questionnaire will help the ARC fulfill its mission to promote economic development in the region, 

and assist current and future grantees with project performance measurement.

This interview will take about 20 minutes to complete. There are no correct or incorrect answers to the 

questions I’m going to ask you. The most important thing is that you feel comfortable giving me your 

honest opinions. Everything we discuss today will be strictly confidential. Information from this 

interview will be summarized and presented in the aggregate in our final report; your name will not be 

used. 

Part A Project Efforts to Obtain Follow-up Data on Program Participants 

You indicated on your questionnaire that your project collected follow-up data from participants. The 

ARC is interested in learning more about these efforts and obtaining additional information about the 

outcomes associated with your project. 

1. What prompted your project to collect outcome data from participants after they completed your 

program? 

2. What type of data did your project collect from program participants? 

3. How often did your project collect these data?

4. What methods did your project use to collect these data? 

5. What steps did your project take to validate the accuracy of these data?

6. From how many individuals did your project collect outcome data? What was your project’s 

response rate? What steps did your project take to obtain data from nonrespondents? 

7. How did your project make use of these data?

8. What problems did your project encounter when collecting these data? What steps did your project 

take to address these problems?
F Protocol for Telephone Interviews With a Sample of

ARC Education and Workforce Development Projects 

That Collected Follow-up Data From Former Participants

 F-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

9. Are there any data that your project was unable to collect? What factors prevented your project 

from collecting these data?

10. What recommendations would you have for other ARC projects that want to collect and make use 

of similar data?

11. Is there any technical assistance that the ARC could have provided your project that would have 

made it easier to collect these data (e.g., ideas about how to collect data, sample protocols and 

reporting formats)?

Part B Information About the Overall Project 

The ARC is also looking for information about potentially promising practices that might be adapted in 

other communities. 

12. What was your project’s most significant accomplishment? What made this accomplishment 

significant?

13. What was your project’s greatest challenge? How did you overcome this challenge?

14. What promising approaches emerged from your project that you would recommend be adopted in 

other communities? What made these approaches promising? What conditions would need to be 

in place to adopt these approaches elsewhere?

15. What difficulties emerged from your project that could be avoided in other communities? What 

caused these difficulties? What did you learn from these difficulties that might be useful to other 

communities? 

16. What factors led to the success of your ARC education and workforce development project? Are 

there any steps that the ARC (or other communities) can take to maximize these enabling factors?

17. What obstacles did you encounter that limited the success of your ARC education and workforce 

development project? Are there any steps that the ARC (or other communities) can take to avoid 

these limiting factors?

18. What lessons did you learn in implementing your project that might be of interest to future ARC 

grant recipients? 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 G-1 

Appendix G:

Protocol for Telephone Interviews With a Sample of ARC 

Education and Workforce Development Projects That Did 

Not Collect Follow-up Data From Former Participants
 G-2 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 
Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 G-3 

Telephone Interviews With a Sample of ARC Education and Workforce Development Projects That Did 

Not Collect Follow-up Data From Former Participants

Project name: _________________________________________________________________ 

Project ID: _________________________________________________________________ 

Respondent name: _________________________________________________________________ 

Date: _________________________________________________________________ 

Thank you for taking the time to talk to me today about your experience as an ARC grantee. The results 

of this survey will help the ARC fulfill its mission to promote economic development in the region, and 

assist current and future grantees with project performance measurement.

This interview will take about 20 minutes to complete. There are no correct or incorrect answers to the 

questions I’m going to ask you. The most important thing is that you feel comfortable giving me your 

honest opinions. Everything we discuss today will be strictly confidential. Information from this 

interview will be summarized and presented in the aggregate in our final report; your name will not be 

used. 

Part A Project Efforts to Obtain Follow-up Data on Program Participants 

You indicated on your survey that your project did not collect data about outcomes. The ARC is 

interested in learning more about the factors that prevented you from obtaining information about the 

outcomes associated with your project. 

1. What type of data would you have liked to collect from program participants? What factors 

prevented you from collecting these data? 

2. How would your project have been able to make use of these data? 

3. Is there any technical assistance that the ARC could have provided your project that would have 

made it possible to collect these data (e.g., ideas about how to collect data, sample protocols and 

reporting formats)?
G 

Protocol for Telephone Interviews With a Sample of

ARC Education and Workforce Development Projects 

That Did Not Collect Follow-up Data From Former Participants

 G-4 Evaluation of the Appalachian Regional Commission’s 

Education and Workforce Development Projects: 2000–2008 

Part B Information about the Overall Project

The ARC is also looking for information about potentially promising practices that might be adapted in 

other communities. 

4. What was your project’s most significant accomplishment? What made this accomplishment 

significant?

5. What was your project’s greatest challenge? How did you overcome this challenge? 

6. What promising approaches emerged from your project that you would recommend be adopted in 

other communities? What made these approaches promising? What conditions would need to be 

in place to adopt these approaches elsewhere?

7. What difficulties emerged from your project that could be avoided in other communities? What 

caused these difficulties? What did you learn from these difficulties that might be useful to other 

communities? 

8. What factors led to the success of your ARC education and workforce development project? Are 

there any steps that the ARC (or other communities) can take to maximize these enabling factors?

9. What obstacles did you encounter that limited the success of your ARC education and workforce 

development project? Are there any steps that the ARC (or other communities) can take to avoid 

these limiting factors?

10. What lessons did you learn in implementing your project that might be of interest to future ARC 

grant recipients?