Creating an Entrepreneurial Appalachian Region: 

Findings and Lessons from an Evaluation 

of the Appalachian Regional Commission’s 

Entrepreneurship Initiative 1997 – 2005 

Evaluation Team: 

Deborah Markley, RUPRI Center for Rural Entrepreneurship 

Erik Pages, EntreWorks Consulting 

Brian Dabson, RUPRI 

Thomas Johnson, RUPRI 

Sara Lawrence, RTI 

Sara Yanosy, RTI 

Karen Dabson, RUPRI Center for Rural Entrepreneurship 

April 2008 
i 

ACKNOWLEDGEMENTS 

The Evaluation Team – Deborah Markley, Erik Pages, Brian Dabson, Thomas 

Johnson, Sara Lawrence, Sara Yanosy, and Karen Dabson – would like to 

acknowledge the important role played by Catherine Renault on this project. 

Cathy was involved in developing the proposal and early project design, before 

returning to her roots in Maine as the Director of the Office of Innovation, 

Department of Economic and Community Development. She also provided input 

on the final report. The team would also like to acknowledge the input of the 

advisory group – Thomas Lyons, Edward Malecki, and Jonathan Potter. Their 

keen insights and input throughout the project helped to strengthen project 

design and the final report. The team also benefited from discussions held with 

ARC staff throughout the project, particularly Greg Bischak and Ray Daffner, as 

well as a group of advisors who discussed a summary report on key findings. 

The senior members of the evaluation team would also like to acknowledge the 

heroic efforts of two members of the team, Sara Lawrence and Sara Yanosy, to 

develop and complete the database for the sample projects. They diligently 

tracked down project leaders to discuss projects that, in some cases, were 

completed almost 10 years earlier. They shared data and insights that greatly 

enhanced this evaluation. 

Finally, this evaluation would not have been possible without the many 

individuals throughout the ARC region who shared project data, recollections and 

insights into the importance of the ARC Entrepreneurship Initiative to their 

individual rural places and to the broader region. We acknowledge their important 

contribution and greatly appreciate the time they took to talk with team members 

about their past and ongoing efforts to build a more entrepreneurial Appalachian 

region. 
ii 

EXECUTIVE SUMMARY 

From 1997 through 2005, the Appalachian Regional Commission (ARC) invested 

nearly $43 million in a ground-breaking program to stimulate and support 

entrepreneurship across Appalachia. The Entrepreneurship Initiative (EI) was 

the first large scale attempt to give greater focus to homegrown business 

development as a regional economic development strategy. 

The Rural Policy Research Institute (RUPRI), the RUPRI Center for Rural 

Entrepreneurship, EntreWorks Consulting, and RTI were commissioned by ARC 

in 2006 to conduct an evaluation of EI in terms both of outcomes achieved by a 

sample of funded projects and of broader policy impacts across the region. The 

evaluation team undertook literature reviews, reviewed project files for a sample 

of 114 projects, conducted phone interviews with 36 stakeholders and experts, 

developed a metrics framework, completed interviews with project staff 

associated with 88 projects, made four site visits, and conducted a meta-analysis 

of the outcomes and impacts. The team’s work was informed by a three-person 

advisory committee of leading academic experts on entrepreneurship – Dr. 

Thomas Lyons, Dr. Edward Malecki, and Dr. Jonathan Potter. 

A review of entrepreneurship trends in the region during the EI provided the 

context and backdrop for ARC’s investments. Over the period of the EI, trends in 

nonfarm proprietor and microenterprise employment in Appalachia showed 

increases in line with the nation as a whole, but trends in nonfarm proprietor 

income showed the region lagging behind the nation and slipping further behind 

by 2005. Data on the impact of entrepreneurship on the local economy showed 

that only 15 percent of Appalachian counties saw income increases associated 

with entrepreneurial activity that were higher than the national rate. It appears 

that entrepreneurship had greater impact in terms of both employment and 

income in the southern tier of Appalachian states. This evaluation was not 

designed to discern cause and effect between EI investments and these trends; 

however, the context is important for interpreting evaluation findings and 

understanding the resulting recommendations. 

 

Also of contextual importance is the rapid growth in interest in and adoption of 

entrepreneurship development policies and programs since the EI began in 

1997. A review of current literature provides many insights on the linkages 

between entrepreneurship and regional development and on the efficacy and 

impact of different types of entrepreneurship programs. This body of research 

work was not available to the designers of the EI but was particularly helpful in 

conducting the evaluation and determining appropriate performance metrics. 

The evaluation team identified three goals that were at the core of the EI – to 

increase the number of entrepreneurs establishing businesses in the region, to 

increase the survival rate of such ventures, and to increase the proportion that 

develop into high growth businesses that create jobs and wealth in Appalachia. 
iii 

These goals were operationalized through five program categories – 

entrepreneurship education, access to capital, business incubators, sector 

interventions, and technical assistance and training. There was also a sixth 

cross-cutting category of community capacity-building. 

As identified through the final reports submitted to ARC, the EI led to the creation 

of at least 9,156 jobs, the retention of a further 3,022 jobs, the formation of 1,787 

new businesses, and the provision of services to 8,242 businesses. The cost per 

job created was $4,693, which compares favorably with other economic 

development efforts. ARC investments were made in 340 unique projects across 

the region at an average investment per state of $3.3 million and investment per 

capita of $1.82. The total ARC investment has leveraged an additional $72.8 

million in private investment for those projects that have been closed, a figure 

that is projected to rise to $109.9 million when all projects in the portfolio have 

been completed. 

Other metrics identified through in-depth investigation of outcomes from the 

sample of projects expand on this picture. In the 88 projects included in the 

sample, over 11,500 students and teachers participated in or received training in 

entrepreneurship education projects, 1,500 entrepreneurs took part in sectorfocused activities and another 1,620 received training and technical assistance. 

The evaluation team’s assessment of qualitative impacts were drawn from 

interviews with project leaders most familiar with the investments and regional 

stakeholders and entrepreneurship experts with deep experience both in the 

region and with entrepreneurship development – key informants. Common 

themes identified were that ARC investments: 

 Raised the profile of entrepreneurship as a development strategy, helping 

to change the mindset within the region 

 Represented “but for” money in the region, providing start-up funding for 

innovative projects 

 Leveraged additional resources that helped some projects achieve scale 

and impact 

 Facilitated networking and collaboration among practitioners 

 Helped to change people’s attitudes, particularly among youth and their 

teachers. 

There were also a number of lessons gleaned from the many interviews 

conducted across the region, some of which will benefit those who are actively 

engaged in implementing entrepreneurship programs – the practitioners – and 

others which will guide future programs either of ARC or agencies across the 

country pursuing similar efforts. For the practitioners, the lessons were of three 

kinds: 
iv 

 Lessons for Program Leadership 

o Successful entrepreneurship initiatives had sparkplugs or local 

champions that provided leadership for these efforts. 

o Local capacity was a key to success. 

 Lessons for Program Management 

o Program self-sufficiency (sustainability) and success went hand in 

hand. 

o Entrepreneurship development was recognized to be a long-term 

process. 

o Successful projects altered their goals and approaches as 

conditions warranted. 

 Lessons for Program Outreach 

o Partnerships and collaborations were important to success. 

o Successful projects celebrated and shared the story of their 

success. 

For program designers and implementers, again the lessons were of three kinds: 

 Lessons for Program Design 

o Practitioners and entrepreneurs have unique local knowledge that 

can be applied to program design and subsequent program 

refinements. 

o Successful initiatives brought together related investments, in this 

case, other regional economic development or entrepreneurshiprelated investments. 

 Lessons for Program Implementation 

o Getting EI funds to local partners was dependent upon state 

leaders, such as governors and program managers, and varied 

based on the importance assigned to the initiative. 

o The size of ARC grants placed limits on regional impacts. 

 Lessons for Program Impacts 

o Building a broader base of support for entrepreneurship 

investments requires continued efforts to “make the case” to local 

leaders. 

o Programs can be improved by embracing long-term and locallydriven evaluation of program outcomes and impacts. 

Finally, the evaluation team offered three sets of recommendations to ARC. 

Regarding investments in entrepreneurship development: 
v 

 Entrepreneurship development initiatives should include assessment of 

existing capacity and capacity-building activities as part of the project 

design. 

 Entrepreneurship development initiatives should be made with a focus on 

the long term. 

 Entrepreneurship development initiatives receiving investments should be 

market-driven and practice continuous improvement. 

 Emphasis should be placed on investing in initiatives that demonstrate the 

ability to form regional partnerships and collaborations. 

The second set of recommendations is for creating a “best in class” metrics 

system: 

 “Job creation” is an overused metric, paints an incomplete picture of the 

outcomes of entrepreneurship development investments, and should be 

replaced by an “entrepreneurship development metrics portfolio.” In 

addition to jobs created/retained and new business starts, this system 

should include outcome measures such as: 

o Change in business profitability (performance) following a capital 

investment 

o Number of youth considering business creation as a career option 

after participation in an education program 

o Percent of incubator tenants who graduate and remain in the region 

o Change in total sector sales over time as a result of investment to 

encourage sector development 

o Number of customers still in business after receiving technical 

assistance 

o Positive change in perceived community support for

entrepreneurship as measured by community pre- and postsurveys. 

 A “best in class” metrics system requires investment in a “best in class” 

evaluation system. 

The third set of recommendations focus on program design and management: 

 ARC’s initiative process should be regularized so that state program 

managers can more effectively plan for and promote the use of the 

resources. 

 ARC’s proven experience can be applied to developing and delivering 

effective, region-wide education programs that help make the case for 

entrepreneurship as a core economic development strategy for the 

Appalachian region. 

 To build on the momentum created by the EI, ARC should create a Next 

Generation Entrepreneurship Innovation Initiative that will be 

groundbreaking in its design. A long-term investment is recommended 
vi 

that incorporates all the learning from the EI and the emerging 

entrepreneurship development field. 
vii 

Table of Contents 

PAGE

CHAPTER 1 INTRODUCTION 1 

CHAPTER 2 REVIEW OF THE ENTREPRENEURSHIP 

 DEVELOPMENT FIELD 5 

CHAPTER 3 CONCEPTUAL FRAMEWORK AND REVIEW 

 OF THE LITERATURE 14 

CHAPTER 4 APPROACH TO THE EVALUATION 31 

CHAPTER 5 THE WIDER REGIONAL ECONOMY – 

 ENTREPRENEURSHIP TRENDS IN APPALACHIA 44 

CHAPTER 6 ARC INVESTMENT PORTFOLIO – PROJECT 

 IMPLEMENTATION AND REPORTED METRICS 58 

CHAPTER 7 SAMPLE PROJECT OUTCOMES AND 

 BROADER POLICY IMPACTS 75 

CHAPTER 8 LESSONS LEARNED FROM THE ARC 

 ENTREPRENEURSHIP INITIATIVE EXPERIENCE 88 

CHAPTER 9 CONCLUSIONS AND RECOMMENDATIONS 106 

BIBLIOGRAPHY 118 

APPENDIX A LITERATURE REVIEW 130 

APPENDIX B NON-PROJECT STAKEHOLDER INTERVIEWS 159 

APPENDIX C PROTOCOL FOR PROJECT LEADERS 160 

APPENDIX D PROTOCOL FOR NON-PROJECT STAKEHOLDERS 167 

 
1 

CHAPTER 1 

INTRODUCTION 

In 1997, the Appalachian Regional Commission (ARC) began a multi-year 

initiative to invest in projects designed to build entrepreneurial economies across 

the region – the Entrepreneurship Initiative (EI). Since that time, ARC has 

invested almost $43 million (composed of EI funds and dollars from other ARC 

accounts) in various entrepreneurship development projects. Over 10 years, 

these projects have created jobs and businesses, supported partnerships and 

collaborations, and helped leaders at the community and state levels recognize 

the value of entrepreneurship as an economic development strategy. While ARC 

has collected select data to describe the outcomes of these investments, these 

data do not begin to tell the story about the extent to which and how the EI has 

had an impact on the region. This evaluation was designed to provide a more 

detailed and nuanced description of the impact of the EI, both in terms of the 

outcomes achieved by the portfolio of projects and the broader policy impacts 

across the region. This evaluation, however, must first be placed within the 

context of the initiative’s history. 

HISTORY OF THE PROGRAM 

The EI emerged in 1997 as a special initiative under the leadership of thenFederal Co-Chairman Jesse White. White had long pushed for economic 

development strategies focused more on home-grown business development as 

opposed to business recruitment and attraction. As he told a Federal Reserve 

Bank of Kansas City symposium in 2000, ”[W]e’ve got to re-instill in rural America 

the idea, particularly in Appalachia and the Rural South, that job creation, 

business creation, and, most importantly, wealth creation, occurs as a result of 

local indigenous business creation.”1

 The EI was designed as a test of public 

policy approaches that sought to achieve this objective. 

The genesis for the EI was in the belief that entrepreneurial activity could be 

encouraged through strategic investments in education, business assistance, 

and capacity building projects. Areas for strategic investment identified prior to 

the launch of the EI included: 

 Access to capital and financial assistance 

 Technical and managerial assistance 

 Technology transfer 

 Entrepreneurial education and training 

 Entrepreneurial networks. 

 

1

 Jesse White, “Overview Panel Comments,” Proceedings of Federal Reserve Bank of Kansas 

City Conference on Beyond Agriculture: New Policies for Rural America (April 2000):193. 
2 

The EI was originally funded with $15 million over three years, with additional 

investments being provided beyond this original total. Through 2000, ARC had 

invested $17.6 million in 169 projects.2

 Through 2003, ARC had invested $31.4 

million in 368 projects. An additional amount of approximately $11 million was 

invested in subsequent years through the EI or via the use of Area Development 

funds. Table 1.1 shows ARC investment in entrepreneurship-related activities by 

source of funds for the period of this evaluation. While all ARC investments in 

entrepreneurship were not specifically drawn from EI funds, the rationale for 

making these investments was clearly driven by the goals associated with the EI. 

Today, ARC continues to fund entrepreneurship development-related projects 

under the Asset-Based Development Initiative launched in 2005, although the EI 

is no longer operating as a stand alone initiative. The Asset-Based Development 

Initiative is building on the foundation laid by the EI, leveraging the new 

businesses and additional capacity created by EI investments. 

Table 1.1 ARC Funds Invested in Entrepreneurship-Related Projects, 1997-

2005, by Source of Funds 

Source of funds $ invested

Entrepreneurship Initiative 26,546,366

Area Development 6,698,724

Commission’s EI 3,206,803

Distressed Counties 3,052,109

CoChair Fund 2,432,440

Regional Initiatives 868,673

Goal Fund 114,000

New Markets Fund 52,574

TOTAL 42,971,688

ORGANIZATION OF THE EI 

ARC’s special initiatives are traditionally designed to support innovative local 

projects and also to serve as a demonstration of new regional development 

strategies and approaches. ARC officials were very explicit about the importance 

of the EI as a means to demonstrate the viability of entrepreneurship as an 

economic development strategy. Interviewed experts also emphasized this 

aspect of the EI. 

The EI’s organization reflects this dual focus on educating local leaders and on 

maximizing the effect of local project investments. In pursuing its wider 

educational goals, ARC recognized the need to form partnerships and engage 

other institutions in order to achieve sustainable impacts in the region. Through 

the formation of four advisory committees with significant private sector 

 

2

 For an early review of these investments, see Regional Technology Strategies, Inc., Evaluation 

of the Early Stages of the Appalachian Regional Commission’s Entrepreneurship Initiative, A 

Report to the Appalachian Regional Commission, December 2001. 
3 

participation, ARC was able to tap into national expertise and “best practices” to 

guide the initiative. Each advisory committee was charged with providing input to 

ARC in a particular program area – entrepreneurship education, technical 

assistance, capital access, and sectorally targeted strategies. In particular, these 

advisory committees helped to organize a region-wide educational effort that 

augmented the specific project investments made by ARC. Elements of this 

educational effort included, for example: 

 Scholarships to the 16th Annual Entrepreneurship Education Forum and 

support for three regional entrepreneurship conferences; creation of the 

Springboard Youth Entrepreneurship Education Awards; creation of the 

Entrepreneurship Everywhere web-based resource guide

3

 Support for conferences on sector-based development; a competitive 

grant program for “strategic sectoral interventions” in the region 

 Sponsorship of a workshop on community development venture capital; 

follow-up regional workshops on equity capital in rural communities; 

publications on developmental venture capital in the region; creation of an 

opportunity fund to leverage private investment using the New Markets 

Tax Credit program 

 Funding for four workshops on business incubation “best practices” in the 

region; a survey on business incubators in Appalachia; creation of a 

business incubation mentor program. 

These region-wide educational efforts were, for the most part, funded outside the 

EI, using other ARC dollars, such as Commission or CoChair funds. While using 

distinct funding sources, these projects also contributed to the mission of the EI – 

to encourage the development of an entrepreneurial economy in the region. In 

fact, it is not possible to separate the impact of broader ARC investments in 

entrepreneurship from those specifically identified with the EI. Most of ARC’s 

investments through the EI resulted in project-specific outcomes; the outcomes 

of their educational efforts were much broader, serving to raise the overall level 

of regional awareness about specific aspects of entrepreneurship development – 

education, sector approaches, capital and incubation. The impacts, in most 

cases, were not confined to particular communities or even states. Instead, the 

impacts could be measured in terms of increased understanding of several 

aspects of entrepreneurship development and enhanced capacity to design and 

implement entrepreneurship activities within the region. 

While this evaluation did not seek to quantify ARC’s capacity building and 

education impacts related to entrepreneurship development, this effect was cited 

in nearly all of the interviews with outside experts and state level officials. 

During interviews, the evaluation team regularly heard comments such as the EI 

“opened people’s eyes to other possibilities” or “the program got people talking 

about entrepreneurship.” Several other programs, such as the New Markets 

 

3

 Available at <http://www.entre-ed.org/_arc/home1.htm>. 
4 

Venture Capital program, built on momentum generated by the EI. In fact, two of 

the first six New Markets Venture Capital companies operate in Appalachia. 

While the EI’s impact in terms of building better awareness of entrepreneurship 

cannot be quantified, it should be included in any full accounting of the EI’s 

outcomes. The EI was designed as a demonstration, i.e., to test new policy 

models and to encourage Appalachian communities to focus within on nurturing 

home-grown businesses. Today, nearly all of the Appalachian states are involved 

in some form of organized efforts to promote entrepreneurial development. 

While the ARC cannot claim sole credit for this shift in thinking, based on 

feedback from key stakeholders it is clear that the EI played a role in convincing 

policy makers that supporting entrepreneurs is good economic development 

policy. 

STRUCTURE OF THIS REPORT 

This report presents the findings from an evaluation of ARC’s investments in 

entrepreneurship development from 1997 through 2005. Based on analysis of 

data collected by ARC, in-depth research into the impacts associated with a 

sample of ARC-funded projects, and interviews with a broad range of 

stakeholders, this evaluation provides key insights into the value of the EI to the 

region. Chapter 2 provides an overview of entrepreneurship development, 

particularly in terms of the state of the field in 1997, the beginning of the EI. This 

context is important for understanding how innovative ARC was in launching the 

EI, plowing new ground in terms of economic development. Chapter 3 lays out 

the conceptual framework developed to guide this evaluation process and 

reviews relevant evaluation literature related to entrepreneurship development. 

This review of the literature served as the basis for developing the metrics 

framework that would guide in-depth evaluation of sample projects. Chapter 4 

describes this metrics framework and the overall research approach used in this 

study. 

Chapter 5 describes key aspects of the Appalachian region’s business and 

economic environment that created the context for the EI. It is this context that 

created the need for and gave rise to the projects that became part of the EI 

portfolio. Changes over the ten year time horizon of the EI are also presented. 

Chapters 6 and 7 provide the key findings from this evaluation. Chapter 6 

describes the types of projects in which ARC invested and presents data on key 

metrics collected by ARC for all projects. Chapter 7 provides a detailed 

discussion of the impacts, quantitative and qualitative, associated with the 

sample of projects considered for this evaluation. Chapter 8 presents the 

lessons learned from the EI experience for both practitioners of entrepreneurship 

development and policy makers who may be considering similar types of 

investment programs in the future. Finally, Chapter 9 provides recommendations, 

based on evaluation findings, for the field and for ARC going forward. 
5 

CHAPTER 2 

REVIEW OF THE ENTREPRENEURSHIP DEVELOPMENT FIELD 

While entrepreneurship has always been a feature of the American (and 

Appalachian) landscape, the history of explicit public support for regional 

entrepreneurial development is quite brief. Indeed, ARC’s Entrepreneurship 

Initiative serves as one of the first such large-scale efforts, and the absolute first 

such effort investing Federal dollars in regional entrepreneurship strategies. 

At the time of the Entrepreneurship Initiative’s unveiling in 1997, ARC’s 

leadership had grown increasingly concerned about Appalachia’s future vitality. 

The region was especially hard hit by cutbacks in declining sectors such as 

timber, textiles, tobacco, and was seeking new approaches to jump start 

development. Entrepreneurship-focused economic development strategies were 

viewed by ARC as “. . . a critical element in the establishment of self-sustaining 

communities that create jobs, build local wealth, and contribute broadly to 

economic and community development.”4

By investing in the Entrepreneurship Initiative in 1997, the ARC was something of 

an “early adopter.” Up to that stage, economic developers had focused almost 

exclusively on industrial recruitment as a core strategy. The industrial 

restructuring of the 1970s and the 1980s had stimulated some interest in 

business retention and technology development strategies, but entrepreneurial 

development strategies represented something of a “new thing” in 1997.5

Nevertheless, ARC was not promoting entrepreneurship without some evidence 

that it would make a significant contribution to economic opportunity in 

Appalachia. In the late 1980s, Eisinger discussed the limited returns to traditional 

economic development activities and suggested a more important role for 

entrepreneurial, growth from within, economic development strategies.6

 There 

was some research that questioned the efficacy of traditional “smokestack 

chasing” and some that provided insights into the impact of specific programs 

such as business incubation, youth entrepreneurship, and targeted technical 

assistance. What was missing at that time was the evidence to link 

entrepreneurship to regional and community development, and to identify which 

policy interventions, and in what combination, would lead to increased 

entrepreneurial activity. Over the past decade, there has been rapidly growing 

interest among researchers and policy analysts in entrepreneurship as a core 

 

4 Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and 

Approved Projects, September 2003, 9 August 2007 

<http://www.arc.gov/index.do?nodeId=1970>. 

5

 For background, see Erik R. Pages, Doris Freedman, and Patrick Von Bargen , 

“Entrepreneurship as a State and Local Economic Development Strategy,” The Emergence of 

Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S. Knowledge Economy, 

Ed. David Hart, Cambridge, MA: Cambridge University Press, 2003. 

6

 Peter K. Eisinger, The Rise of the Entrepreneurial State: State and Local Economic 

Development Policy in the United States, Madison, WI: University of Wisconsin Press, 1988.
6 

economic development strategy, and more evidence has been found to support 

this approach. It was important to the evaluation to understand the implications of 

this expanded understanding in order to appropriately assess the value and 

impact of the EI investment. 

This chapter provides a brief overview of research, policy and practice that 

addresses two primary questions: 

 What makes a region (or community) entrepreneurial? 

 What policy or community interventions, if any, can help support 

entrepreneurial development? 

WHAT MAKES A REGION ENTREPRENEURIAL? 

Researchers have examined a host of both macroeconomic and microeconomic 

factors that help explain the innovation or entrepreneurial capacities of a region. 

At the broadest level, recent research has reviewed how leading demographic 

trends are correlated with new firm births. For example, in a 1994 review article, 

Reynolds, Storey and Westhead describe key factors that are associated with 

higher levels of new firm starts.7

 These include net population growth, increases 

in personal or household income or regional gross product, high population 

density, high educational levels among population, and a high percentage of 

population between the ages of 25 and 44. Acs and Armington similarly find that 

higher relative education levels strongly affect new firm formation rates, 

especially among service businesses.8

 These same factors have been 

highlighted in the cross-national research studies conducted under the auspices 

of the Global Entrepreneurship Monitor (GEM) research consortium.9

While this literature contains something of a consensus about several key factors 

(e.g., population growth, higher education levels) that are correlated with higher 

new firm formation rates, some researchers caution that the causal chain still 

remains unclear. For example, Feldman notes that many key factors associated 

with entrepreneurship, such as the presence of local venture capital firms, may 

actually lag instead of lead entrepreneurial growth.

10 In other words, these 

 

7

 Paul Reynolds, D.J. Storey, and P. Westhead. “Crossnational Comparisons of the Variation in 

New Firm Formation Rates, Regional Studies 28.4 (1994):443-456. 

8

 Zoltan Acs and Catherine Armington, “Using Census BITS to Explore, Entrepreneurship, 

Geography, and Economic Growth,” Research Summary No. 248, Small Business Administration 

Office of Advocacy, February 2005. Similar findings that emphasize the importance of local 

educational levels as well as local employment growth and productivity rates can be found in 

Advanced Research Technologies, “The Innovation-Entrepreneurship NEXUS,” Research 

Summary No. 256, Small Business Administration Office of Advocacy, April 2005. 

9

 Begun in 1999, the Global Entrepreneurship Monitor project is a cross-national effort to assess 

national rates of entrepreneurship. Begun with ten countries, the research program now tracks 

entrepreneurial activity in forty-two countries. The project’s research reports can be accessed at 

<www.gemconsortium.org>. 

10 Maryann Feldman, “The Entrepreneurial Event Revisited: Firm Formation in a Regional 

Context,” Industrial and Corporate Change 10.4 (2001):861-891. 
7 

regional assets emerge as a result of strong local entrepreneurial activity. They 

are a by-product, as opposed to a trigger, for high firm formation rates. Feldman 

concludes by cautioning that each region’s entrepreneurial development activities 

generally emerge from a unique and idiosyncratic mix of historical factors, local 

resources, and business conditions. 

PUBLIC POLICY AND ENTREPRENEURSHIP 

While many economists and researchers continue to assess how various 

demographic and macroeconomic factors help drive regional entrepreneurial 

activity, policy makers, including the ARC’s leadership, are more concerned with 

questions of how (and whether) policy interventions can affect a region’s 

entrepreneurial propensities and development patterns. A large and growing 

literature examines this issue,11 and much of this work has helped guide the 

strategic direction of ARC’s own Entrepreneurship Initiative. 

While individual analysts may differ on the relative importance of certain regional 

factors, there is relatively strong consensus that five factors are especially 

important: 

 Access to Capital 

 Enabling Culture 

 Local Networks 

 Supportive Infrastructure 

 Supportive Government Policies. 

A more detailed description of each factor and the public policy approaches used 

to address them is provided below. 

Access to Capital 

Successful entrepreneurial regions tend to enjoy a wide range of options for 

financing businesses at different stages of the business life cycle. Successful 

regions host a variety of financial institutions that can provide businesses with a 

range of both equity and debt financing options. Rural communities may be 

especially challenged on this front. Recent US Department of Agriculturesponsored research has found that rural areas tend to have fewer lenders and 

less diverse markets. While rural areas do have less bank competition, the study 

found few rural-urban differences between the cost and availability of debt 

 

11 For an excellent comprehensive literature review, see Jill S. Taylor, “What Makes a Region 

Entrepreneurial? A Review of the Literature,” Monograph, Cleveland State University, Center for 

Economic Development, September 2006. Other good sources include Brian Dabson, et al., 

Mapping Rural Entrepreneurship, Washington, D.C.: CFED, August 2003; Deborah Markley, Don 

Macke and Vicki Luther, Energizing Entrepreneurs: Charting a Course for Rural Communities, 

Lincoln, NE: Heartland Center for Leadership Development, 2005, and OECD, Entrepreneurship 

and Local Economic Development, Paris: OECD, 2003. 
8 

financing.12 In fact, a recent study of capital access in the Appalachian region 

found that banks in the region had higher small business loan to deposit ratios as 

compared to national figures.13

 

However, businesses in rural regions experience capital access problems when 

their needs fall outside the type of loans traditionally made by banks – e.g., 

microloans (to fund very small enterprises with limited collateral) and equity 

finance. Microenterprise development was just emerging in the early to mid1990s. A recent study of the industry noted that only about one-quarter of 

microenterprise programs listed in a 2001 directory existed before 1991.14 The 

importance of providing both training and microfinancing to entrepreneurs was 

just being viewed as a development strategy as ARC invested in the EI. 

Appalachia also suffers in terms of access to equity finance. For example, a 2000 

ARC study found that only 1/3 of 1% of all venture capital ($117 million) was 

invested in rural regions of Appalachia.15 A more recent study of capital access in 

the Appalachian region shows improved access to equity capital.16 Through 

2004, the study shows that ARC had invested in 11 funds in seven states and 

these funds collectively had invested $13.6 million in regional businesses. If all of 

this investment is considered new to the region, it represents a 12% increase in 

equity capital as compared to that found in 2000. However, the later study also 

identified a need to continue to expand the capacity of community development 

financial institutions in the region by broadening sources of funds, increasing selfsufficiency, and expanding products available to businesses. 

As a result of these market gaps, policy makers in Appalachia and elsewhere 

have supported a host of initiatives to develop new sources of microcredit and 

equity and equity-like capital.17 Within the region, ARC funded microenterprise 

development initiatives and revolving loan funds, as well as others that created a 

venture capital industry. At the Federal level, these initiatives include the Small 

Business Administration’s (SBA) Microenterprise Loan and PRIME (Program for 

 

12 Ray Collender, et al., “Financial Markets Serve Rural Areas Fairly Well,” Rural Development 

Perspectives 14.1 (May 1999):28-35. 

13 National Community Reinvestment Coalition, Access to Capital and Credit for Small 

Businesses in Appalachia, Washington, D.C.: National Community Reinvestment Coalition, April 

2007. 

14 Elaine L. Edgcomb and Joyce A. Klein, Opening Doors, Building Ownership: Fulfilling the 

Promise of Microenterprise in the United States, Washington, D.C.: FIELD, A Program of the 

Aspen Institute, 2005 <http://www.fieldus.org/Publications/FulfillingthePromise.pdf> 28 January 

2008. 

15 Appalachian Regional Commission, Capitalizing on Rural Communities, Washington, DC: 

Appalachian Regional Commission, 2000, 8. 

16 National Community Reinvestment Coalition, April 2007. 

17 Deborah Markley, et al. Rural Equity Capital Initiative Study of Nontraditional Venture Capital 

Institutions, RUPRI PB2001-11A-D, 2001, 

<http://www.energizingentrepreneurs.org/content/cr.php?id=4&sel=2>. An excellent summary of 

these efforts can be found in a special issue of the Federal Reserve Bank of San Francisco’s 

Community Development Review 3.2 (2006). 
9 

Investment in Micro-Entrepreneurs) programs, the Small Business Investment 

Company (SBIC) program, the New Markets Tax Credit initiative, the CDFI Fund, 

and other efforts. State and local initiatives have been more far-reaching and 

comprehensive, including Nebraska’s Microenterprise Partnership Fund, the 

Oklahoma Center for the Advancement of Science and Technology (OCAST) 

and the Pappajohn Entrepreneurship Center at North Iowa Area Community 

College (NIACC). Nebraska provides state support for microenterprise 

development through a public-private partnership that channels investment to 

microenterprise development programs across the state through a competitive 

process. OCAST has developed a comprehensive set of financing tools for 

technology businesses in the state that includes pre-seed financing of up to 

$100,000 for tech start-up companies, a seed capital program with equity 

investments up to $750,000, and a number of technical assistance and sectorspecific programs.18 NIACC’s capital programs are designed to work hand-inhand with technical assistance and education programs available at the 

community college. Capital programs include a revolving loan fund providing debt 

capital, a nanoloan program providing debt capital for microenterprises, and 

access to both formal venture capital and angel investors.19 Finally, private 

programs, such as the creation of local angel investor networks, are also being 

introduced across the US. Angel investor groups have grown from 50 formal 

groups in 1997 to an estimated 170 formal and informal groups in 2002;20 there 

were an estimated 200,000 individual angel investors active in 2002 and 234,000 

in 2006.21

 

Enabling Culture 

The need for an “enabling culture” is widely recognized in the literature, but the 

details of what constitutes such a culture are expressed in different ways by 

researchers and practitioners. At the most basic level, an enabling culture is 

one that understands, recognizes, and honors the importance of local 

entrepreneurs. These three terms – understanding, recognizing, and honoring –

also connote three different sets of potential policy interventions. 

 

18 More information is available about OCAST on their website, <http://www.ocast.state.ok.us/>. 

19 More information about NIACC can be found on their website, 

<http://www.niacc.edu/pappajohn/> and in Deborah Markley and Karen Dabson, Innovative 

Approaches to Entrepreneurial Development: Cases from the Northwest Region, RUPRI Center 

for Rural Entrepreneurship, 2006, 32-41. 

20 “Business Angel Investing Groups Growing in North America,” Ewing Marion Kauffman 

Foundation, October 2002, 4 December 2007 

<http://www.angelcapitaleducation.org/dir_downloads/resources/BestPractices_Summit1.pdf>; 

21 “Full Year 2002 Angel Market Analysis Report,” Center for Venture Research, Whittemore 

School of Business and Economics, University of New Hampshire, 4 December 2007 

<http://wsbe2.unh.edu/files/Center%20for%20Venture%20Research%20Press%20Release%20J

une%202003%20-%20The%20Angel%20Investor%20Market%20in%202002.pdf>; Full Year 

2006 Angel Market Analysis Report,” Center for Venture Research, Whittemore School of 

Business and Economics, University of New Hampshire, 4 December 2007 

<http://wsbe2.unh.edu/files/Full%20Year%202006%20Analysis%20Report%20-

%20March%202007.pdf>. 
10 

To promote understanding, analysts recommend the introduction of 

entrepreneurship education to all parts of the population. The introduction of 

entrepreneurship education at the college and university level has been a 

resounding success. In 1979, only 127 schools offered courses in small 

business and entrepreneurship. Today, more than 1600 schools offer this 

training.22 Community colleges have also witnessed major growth in 

entrepreneurship training. The introduction of entrepreneurship education at the 

primary and secondary school levels has been less smooth, but innovative 

programs and curricula are widely available across the US.23

To promote recognition, analysts have advocated for several ideas, including 

more active entrepreneur involvement in the policy-making process. Given the 

time demands of running a business, few entrepreneurs have stepped up to this 

challenge. However, several states have created advisory bodies, such as New 

York’s Small Business Advisory Board, to provide opportunities for this input. 

The creation of regional business plan competitions is another widely used tool 

that helps publicize local entrepreneurs. These competitions exist across the 

US, with a scale that can range from major national/international competitions to 

smaller local efforts focused on youth or specific market segments. Several of 

the ARC’s Entrepreneurship Initiative grants funded efforts of this sort. 

To honor entrepreneurs, many analysts recommend the creation of awards 

programs such as local Entrepreneur of the Year Awards.24 ARC’s own 

Springboard Awards, designed to honor innovations by entrepreneurship 

educators, was a particularly effective design for a regional awards program. 

These education and recognition efforts are critical to improving the local climate 

for entrepreneurs, but existing cultural attitudes also come into play. Extensive 

research indicates that cultural factors play an important role in explaining 

differing entrepreneurship rates across countries and even within countries. For 

example, Giannetti and Simonov find that the presence of local entrepreneurial 

role models helps explain differences in regional entrepreneurship rates.25 A 

recent US study also stresses the importance of role models.26 Research 

sponsored by the Global Entrepreneurship Monitor project notes that differences 

 

22 Jerome Katz, And Another Thing. . .,”2006 Coleman Foundation White Paper on 

Entrepreneurship Education, 2006 Annual Meeting of US Association of Small Business and 

Entrepreneurship, January 13, 2006, 9 August 2007 

<http://www.usasbe.org/knowledge/whitepapers/Katz%20White%20Paper-Final.pdf>. 

23 For background, see National Governors Association, A Governor’s Guide to Strengthening 

State Entrepreneurship Policy, Washington, DC: National Governors Association, 2004. 

24 National Governors Association, 2004. 

25 Mariassunta Gianetti and Andrei Simonov, On the Determinants of Entrepreneurial Activity: 

Individual Characteristics, Economic Environment, and Social Norms, White Paper, Stockholm 

School of Economics, June 2004, 9 August 2007, 

<http://papers.ssrn.com/sol3/papers.cfm?abstract_id=554511#PaperDownload>. 

26 Edward J. Malecki, “Geographical Environments for Entrepreneurship,” International Journal of 

Entrepreneurship and Small Business, Forthcoming 2008. 
11 

in cross-national entrepreneurship rates are influenced by local cultural attitudes 

toward risk-taking and fear of failure.27 Like many rural regions, some 

Appalachian communities perform poorly on various measures of cultural 

attitudes toward entrepreneurship. Indeed, the ARC’s own materials note that 

“the culture of entrepreneurship is neither wide nor deep throughout 

Appalachia.”28

From the perspective of ARC’s Entrepreneurship Initiative, recognition of the 

importance of an enabling culture is reflected in the number of program 

investments that can be described as “community capacity building” – facilitating 

visioning, leadership development, asset-mapping and community engagement 

activities intended to make the community more supportive of and attractive to 

entrepreneurs. In many cases, initial ARC investments were designed to build 

capacity for supporting entrepreneurs, with follow on investments supporting 

program implementation. These capacity building investments were designed, in 

essence, to enhance community social capital. An extensive literature measuring 

the importance of social capital for economic development, and the impact of 

enhanced social capital on community development outcomes, has been 

developed by researchers, particularly Cornelia Flora.29

Local Networks 

Networks refer to local locations (both virtual and physical locations) where 

entrepreneurs can gain access to peers and others with expertise or knowledge 

about the processes of starting and growing a business. Entrepreneurs 

regularly report that such networks are a critical component in helping them learn 

the ins and outs of business and gain easier access to needed support 

services.30 Such networks are commonplace in major urban areas, especially in 

technology hot spots such as Silicon Valley or Boston. They are less commonly 

found in rural areas as they typically depend on a critical mass of local business 

owners with an interest in networking. Because these dense concentrations of 

business owners and service providers do not exist in many rural regions, 

networking opportunities are often lacking. 

 

27 See, for example, Maria Minniti with William Bygrave and Erkko Autio, Global Entrepreneurship 

Monitor 2005 Executive Report,

<http://www.gemconsortium.org/about.aspx?page=global_reports_2005>. 

28 Appalachian Regional Commission, Entrepreneurship Initiative: Program Summary and 

Approved Projects, September 2003, 9 August 2007 

<http://www.arc.gov/index.do?nodeId=1970>. 

29 See for example, Mary Emery and Cornelia Flora, “Spiraling Up: Mapping Community 

Transformation with Community Capitals Framework,” Journal of the Community Development 

Society 37.1 (Spring 2006):19-35. 

30 See, for example, Erik R. Pages and Shari Garmise, “The Power of Entrepreneurial Networks," 

The Economic Development Journal 2.3 (Summer 2003):20-30. For a local example of network 

building efforts, see North Carolina Rural Economic Development Center, Hello, My Business 

Name is W: A Guide to Building Entrepreneurial Networks in North Carolina, Raleigh, NC: North 

Carolina Rural Economic Development Center, 2007. 
12 

Many regional organizations – both public and private – have jumped in to help 

seed new networks. By providing staff support and limited funding, economic 

developers hope to jump start local networks that can then operate on their own. 

The networks can operate in a manner that is open to all entrepreneurs, such as 

a regional network, or they can focus on the issues and challenges facing a 

specific industry sector or cluster, such as ceramics and related industries. 

Supportive Infrastructure 

Entrepreneurs are no different from other kinds of business owners in their need 

for a supportive and reliable physical infrastructure including transportation, water 

and sewer. Building such infrastructure has been a traditional focus of ARC, 

through its highway program and other investments. More recently, ARC’s 

broadband initiative has invested in providing critical IT-related infrastructure for 

the region’s businesses. 

ARC-sponsored research clearly indicates the powerful impacts of investments in 

traditional infrastructure such as highways, water and sewer facilities, and 

industrial parks.31 Investments in broadband infrastructure can also have 

beneficial economic development impacts. Research from Minnesota indicates 

that the presence of high speed Internet access was a major factor in explaining 

the presence of local gazelle (fast-growing) businesses in regions across the 

state.32 Other research indicates that communities with extensive broadband 

access outperform comparable regions without such amenities in terms of job 

growth and the number of businesses.33

Supportive Government Policies 

While entrepreneurs may succeed anywhere, they are more likely to flourish in a 

region where government and community leaders are “entrepreneur-friendly.” 

This can take two forms: 

 The creation and nurturing of a supportive tax and regulatory climate 

 The provision of a wide range of private and publicly-funded business 

services, including technical assistance, incubators, and other forms of 

specialized support. 

 

31 The Brandow Company and Economic Development Research Group, Evaluation of the 

Appalachian Regional Commission’s Infrastructure and Public Works Program Projects, 

Washington, D.C.: Appalachian Regional Commission, 2000. 

32 Thu-Mai Ho-Kim and Ernesto Venegas, “Starting Up Economic Engines: Key Factors for 

Growing Successful Start-Ups,” Minnesota Department of Employment and Economic 

Development Issue Brief, October 2003. 

33 Sharon E. Gillett, et al, Measuring the Economic Impact of Broadband Deployment,

Washington, D.C.: Economic Development Administration, February 28, 2006. 
13 

An early study by the National Commission on Entrepreneurship documented 

federal policy in support of entrepreneurship from 1958-1998.34 Since that time, 

much of the policy innovation surrounding entrepreneurship has occurred at the 

state level. For example, in 2004, the Kansas legislature passed a 

comprehensive package of legislative support for entrepreneurship development 

including the creation of an entrepreneurship center and implementation of a tax 

credit program to spur investment in new ventures.35 As the focus has shifted 

from federal to state policy in recent years, the impact analysis of these policies 

remains to be done. 

THE STATE OF THE FIELD AND THE EI 

Much of the research and policy innovation described above has happened in the 

years since ARC’s EI was launched. For example, the Global Entrepreneurship 

Monitor project was initiated first in 1999 with only 10 countries; GEM 2007 

research is being conducted in 42 countries. Most state innovations have 

occurred since 2000. ARC was clearly leading the way in encouraging the 

practice of entrepreneurship as an economic development strategy. As such, 

ARC investments were designed to demonstrate the potential associated with a 

wide range of strategies, from youth education to venture capital development. 

Given that the entrepreneurship development field was in its early innovation 

stage, such an approach, referred to by one stakeholder as “let a thousand 

flowers bloom,” was strategic in its design and appropriate for the time. 

 

34 National Commission on Entrepreneurship, American Formula For Growth, Washington, D.C.: 

National Commission on Entrepreneurship, October 2002. 

35 The Kansas Economic Growth Act of 2004 established the Kansas Center for Entrepreneurship 

and introduced several capital initiatives including StartUp Kansas – a seed capital fund, 

authorization for regional foundations, and angel networks. A review of other state efforts can be 

found in National Governors Association, 2004. 
14 

CHAPTER 3 

CONCEPTUAL FRAMEWORK AND REVIEW OF THE 

LITERATURE 

Given the breadth and diversity of investment activity undertaken by ARC as part 

of the EI, the evaluation team began by reviewing the actual operating realities of 

the initiative. The ARC set general guidelines for programs, but each state 

responded to these guidelines in a unique manner. Similarly, each project was 

quite unique. It is easy to understand that a grant for high school 

entrepreneurship summer camps will have different metrics than those of a 

revolving loan fund. Yet, great diversity also existed within single program 

categories. For example, investments in business incubation might fund a 

feasibility study, build a new incubator, or provide technical assistance to new 

sets of customers. As will be seen in Chapter 4, building a set of consensus 

metrics for such a diverse set of program interventions proved quite challenging. 

At the outset of the EI, ARC convened four advisory committees charged with 

identifying best practices and key insights into four areas ARC had identified as 

critical for building an entrepreneurial region – access to capital, 

entrepreneurship education, technical assistance, and sectoral strategies. As an 

outgrowth of the work undertaken by these committees, ARC eventually 

identified five strategic areas for investment - access to capital and financial 

assistance, technical and managerial assistance, technology transfer, 

entrepreneurial education and training, and entrepreneurial networks. However, 

after an initial review of the portfolio of projects receiving ARC investments from 

1997-2005, the evaluation team identified five program categories that captured 

the actual range of projects implemented: 

 Capital Access – provision of services and technical assistance to 

connect entrepreneurs and businesses to appropriate levels and types of 

debt and equity capital 

 Entrepreneurship Education – structured experiential opportunities in 

and out of school for young people (K-16) to learn entrepreneurial skills 

and attributes and to understand business basics with the aim of 

encouraging young people to consider entrepreneurship as a career 

sooner or later 

 Incubators – provision of opportunities for acquiring information, skills, 

resources within (often subsidized) workspace settings 

 Technical Assistance and Training – provision of expert and/or peer 

one-on-one mentoring and consulting services on technical and 

managerial matters and provision of information and skills in formal 

classroom or laboratory settings to adult entrepreneurs 

 Sectors – packaging of some combination of training, incubating, 

technical assistance, and capital access services targeted at a single, 

specific business sector. 
15 

Another category, Community Capacity, facilitating visioning, leadership 

development, asset-mapping and community engagement activities intended to 

make the community more supportive of and attractive to entrepreneurs, was 

also identified. This category actually cuts across the other programmatic areas 

as initial ARC investments were often designed to build capacity for supporting 

entrepreneurs, with follow on investments supporting program implementation. 

The program categories defined by the evaluation team differ somewhat from the 

five strategic areas originally defined by the Appalachian Regional Commission. 

After reviewing actual investments made by the EI, the team determined that 

investments had not been made in the technology transfer area, while actual 

investments in entrepreneurial networks were more accurately described as 

investments in either sector-specific activities or incubation activities, such as 

investment in Virginia’s sustainable wood products industry or Ohio’s food sector. 

As an example, ARC's investments in commercial kitchen incubators, while 

focused on a specific sector, were coded as incubators. Table 3.1 lists ARC’s 

strategic areas and describes how and why they were modified and repackaged 

into the five program categories used as the basis for this evaluation. All of these 

changes were made after a review of the actual projects in which ARC invested 

and represent the adaptation of the ARC investment process to the needs of 

communities as reflected by their proposed project activities. 

Table 3.1: Comparison of ARC Strategic Areas and Evaluation Program 

Categories 

ARC Strategic Areas Comments

Access to capital and financial assistance Capital Access – no change 

Technical and managerial assistance Split into Technical Assistance and Training

and Incubators to reflect different modes of 

entrepreneurial support 

Technology transfer Not implemented 

Entrepreneurial education and training Split between Entrepreneurship Education

for youth and Training (in Technical 

Assistance) for adults 

Entrepreneurial networks Actually implemented as Sector or Incubator 

initiatives 

Community Capacity – added to capture the 

place-based and cultural dimensions of the 

initiative; a category that cuts across program 

categories and strategic areas 

DEVELOPMENT OF A CONCEPTUAL FRAMEWORK 

The origins of the Appalachian Regional Commission’s Entrepreneurship 

Initiative were rooted in the challenge for states and communities throughout the 

region – a challenge that is shared across rural America – of how to foster the 

economic and cultural conditions that give birth to entrepreneurs, support 

innovation, and assist in the development and expansion of successful 

enterprises. These desired outcomes were, and still are, considered critical to 
16 

Appalachia’s future economic vitality. Moreover, they were seen as essential to 

reducing the region’s dependence upon extractive industries and branch plant 

manufacturing, to replacing the long-term practice of asset-stripping with asset 

accumulation and value-addition, and to creating a more entrepreneurial and 

outward-looking culture. 

Given these expectations, the evaluation team chose to approach its charge by 

establishing at the outset a conceptual framework that required a sharp focus on 

the essence of the initiative and reduced the temptation to explore interesting but 

tangential avenues that would perhaps make the evaluation richer but less useful 

as a guide to future policy and action. Three fundamental goals were identified 

which best reflect the primary purposes of the EI. While the following goals were 

not articulated specifically by ARC, they reflect the mission and purpose of the EI 

as understood through ARC publications and interviews with key leaders: 

 More entrepreneurs – To increase the number of entrepreneurs 

establishing businesses in Appalachia 

 Stronger entrepreneurs – To increase the survival rate of entrepreneurial 

ventures in Appalachia 

 More high growth entrepreneurs – To increase the proportion of 

entrepreneurial ventures that achieve rapid growth rates, thus providing 

jobs and wealth within and increasing the competitiveness of Appalachia. 

These fundamental goals then were operationalized by the evaluation team into 

six programmatic goals: 

 More entrepreneurs in the pipeline – increasing the number of people, 

youth and adults, who are actively considering setting up their own 

businesses 

 More entrepreneurs staying – creating the conditions in which 

entrepreneurs wish to stay and grow their businesses in their community 

 Better informed entrepreneurs – providing entrepreneurs with the 

information and tools they need to establish and grow their businesses 

 Better skilled entrepreneurs – providing entrepreneurs with the technical 

and managerial skills they need to sustain and grow their businesses 

 More job creating businesses – providing the tools and resources to 

encourage entrepreneurs to expand and employ others

 Greater business productivity – providing the tools and resources to 

enable entrepreneurs to operate efficient and competitive businesses. 

While this conceptual framework proved useful as an organizing framework for 

the evaluation, the team needed to align this framework with the stated goals and 

operational practice of the EI. The program categories that grew out of the 

evaluation team’s review of ARC’s investment portfolio map well onto the set of 

programmatic goals articulated in the conceptual model. Each category was 

identified as contributing to the attainment of at least one of these programmatic 
17 

goals. Table 3.2 shows how the program categories and programmatic goals 

intersect. It is certainly possible to argue for additional Xs in this table. Indeed, 

one might argue that all project types could have a long-term impact on every 

programmatic goal. However, we have chosen to focus on the primary goal(s) of 

each program type. For example, while entrepreneurship education programs 

ideally, in the long run, lead to more youth creating businesses and/or staying in 

the region, the primary goal of these educational efforts is to expose young 

people to entrepreneurial concepts and possibilities, with the ultimate goal of 

having more youth pursue entrepreneurship as a career path, leading to more 

entrepreneurs in the pipeline. 

The conceptual framework developed to provide context for the ARC 

Entrepreneurship Initiative and to ground the evaluation was also used to inform 

the review of the literature related to entrepreneurship development. 

REVIEW OF THE LITERATURE 

Entrepreneurship and efforts to promote entrepreneurship as an economic 

development strategy have emerged as important topics for both research and 

policy discussion. The literature is vast and seems to be growing larger every 

day. The majority of this research focuses on questions directly related to 

entrepreneurs and their companies by examining questions such as appropriate 

business development strategies, the qualities of successful entrepreneurs, or 

the role of various management practices in producing business success or 

failure. Very few studies address the measurement of program impacts or 

community outcomes. This section presents a review of both current trends in 

economic development performance measurement and of the limited evaluation 

literature. 

Evaluation Research by Program Category and Objective 

The following sections summarize recent research and thinking about 

performance measurement in the five ARC program evaluation categories: 

Capital Access, Sectors, Entrepreneurship Education, Business Incubation, and 

Technical Assistance and Training. Appendix A contains a more complete review 

of the literature. As Table 3.2 indicates, each of the five program categories 

seeks to achieve multiple goals. Yet, many of the efforts can be categorized 

according to a predominant objective. For example, entrepreneurship education, 

with its heavy emphasis on youth empowerment, is most concerned with the 

objective of creating more entrepreneurs. Meanwhile, capital access programs, 

especially those equity programs emphasized by the ARC, are largely focused 

on creating more high-growth entrepreneurs. 
18 

Table 3.2: Intersection of Program Categories and Programmatic Goals 

Capital 

Access 

Entrepreneurial 

Education 

Incubators Sectors Technical 

Assistance and 

Training 

Community 

Capacity 

More entrepreneurs in pipeline X 

X

X 

More entrepreneurs staying X X X X Better informed entrepreneurs X X X X Better skilled entrepreneurs X X X X More job creating businesses X X X X Greater business productivity X X X X X 
19

Creating More Entrepreneurs

Within ARC’s program categories, entrepreneurship education is most concerned 

with feeding the pipeline with new entrepreneurs – exposing youth to 

entrepreneurship as a potential career path and to entrepreneurial ways of 

thinking that can be applied to working for oneself or for someone else. An 

ultimate or long-term goal is to have these new entrepreneurs create new 

enterprises that remain in the region and produce jobs and wealth. 

Entrepreneurship Education. Evaluations of entrepreneurship education 

efforts vary across different levels of the educational system. For programs 

targeting youth, a host of different evaluation methodologies have been 

deployed. In general, most of these measures stress student performance and 

outcomes as opposed to community outcomes. The programs operate on an 

implicit assumption that by empowering and providing skills to young people 

(especially at-risk youth), positive community outcomes will emerge over the 

long-term. 

Studies of youth entrepreneurship programs indicate that these efforts have a 

strong impact on program participants. While some programs, including ARCfunded efforts, provide training for primary school students, most existing 

programs targeted middle and high school students. Studies sponsored by the 

National Foundation for Teaching Entrepreneurship (NFTE) find that NFTE 

program participants had more interest in attending college and had more 

ambitious career aspirations.36 A study of the EnterprisePrep curriculum used in 

Philadelphia found that program participants had lower drop out rates and 

improved performance in science, math and English. Studies of the Junior 

Achievement curriculum identify similar positive outcomes in terms of youth 

attitudes toward entrepreneurship.37

Evaluations of entrepreneurship training at the college and graduate level place 

more emphasis on outcomes in terms of new business formation. A detailed 

study of the University of Arizona’s Berger Entrepreneurship program found that 

program graduates were three times more likely to start a business than their 

student counterparts. Program graduates also enjoyed higher average 

incomes.38 A whole host of other studies provide similar results.39

 

36 Michael Nakkula, Expanded Explorations into the Psychology of Entrepreneurship: Findings

form the 2001-2002 Study of NFTE in two Boston Public High Schools, Working Paper, Harvard 

University Graduate School of Education, 2003. 

37 Junior Achievement, The Impact on Students of Participation in Junior Achievement: Selected 

Cumulative and Longitudinal Findings, Monograph, January 26, 2004. 

38 Alberta Charney and Gary Libecap, The Impact of Entrepreneurship Education: An Evaluation 

of the Berger Entrepreneurship Program at the University of Arizona, 1985-1999, Final Report to 

the Kauffman Center for Entrepreneurial Leadership, 2000. 

39 Lena Lee, Entrepreneurship Education: A Compendium of Related Issues, Working Paper, 

NUS Entrepreneurship Centre, National University of Singapore, 2005. 
20

In general, entrepreneurship education programs are not expected to provide 

direct community impacts in terms of new job or wealth creation. Instead, they 

help produce intermediate outcomes, which may then translate into positive 

economic development outcomes. Analysts contend that entrepreneurship 

education efforts change attitudes toward entrepreneurship, increase awareness 

of key business concepts, and build necessary skills for starting and operating a 

business. These claims are generally confirmed in surveys of participants in 

major programs. These attitudinal and skill changes may lead to new business 

starts, which are then expected to generate positive community outcomes. 

Based on this logic chain, most entrepreneurship education programs emphasize 

program output and student performance measures. While most programs do 

track new business starts by students, they place greater emphasis on more 

short-term measures of student performance and attitudinal change. In contrast, 

many technical assistance programs, such as trainings sponsored by Small 

Business Development Center programs, are evaluated according to their 

capacity to generate new business starts. 

Creating Stronger Entrepreneurs 

Three of the ARC’s program interventions – technical assistance and training, 

business incubation, and sectors/networks – were primarily concerned with 

supporting the mission of stronger entrepreneurs, i.e. to increase the survival rate 

of local entrepreneurial ventures. These three initiatives operate according to 

similar rationales. They seek to create more skilled and better informed 

entrepreneurs through the provision of: 

 New information (via trainings and workshops) 

 Access to mentors and peer support (via networks) 

 Access to subsidized facilities or equipment (via incubators) 

 Access to potential new markets (via networks or training). 

The research literature for each of the program categories emphasizes different 

evaluation methodologies, but they all share some common characteristics. 

First, they seek to assess general customer satisfaction with the provided 

assistance. Second, they seek to assess whether this support led to improved 

knowledge or skills for the entrepreneur. Finally, they seek to assess whether 

this new knowledge has led to changes in behavior; specifically, has the 

intervention led to improved performance by the company or its management 

team? 

Technical and Managerial Assistance. Since business incubators often 

provide technical and managerial assistance as part of their program operations, 

incubators and technical assistance often use similar measurement tools and 

methodologies. Efforts to track and measure the effects of business assistance 

programs have been underway for a long time. Many Federal programs, such as 
21

the Small Business Development Center program and the Manufacturing 

Extension Partnerships, have invested significant resources into evaluation 

efforts. 

Most technical assistance evaluations begin with calculations of traditional 

economic development impacts such as job creation or the leveraging of outside 

investments. As noted earlier, many state-level programs have become quite 

sophisticated in their evaluation efforts. For example, Pennsylvania’s Ben 

Franklin Technology Partners (BFTP) tracks its customers by company size and 

sector; they also compare their client companies to state averages in terms of 

average wages. This latter metric offers a useful measure of job quality. BFTP 

tracks the public return on investment and impact on state gross product via 

calculations of job creation, new investment, and newly generated tax 

revenues.40 BFTP is also one of the few programs to utilize a control group 

methodology in its assessments. This costly but effective method compares the 

performance of program clients to comparable firms who did not utilize BFTP 

services. This comparison indicates that BFTP-supported firms employed three 

more people in each year after the program investment. 

In addition to basic measures of job creation and customer satisfaction, the 

Oklahoma Center for the Advancement of Science and Technology (OCAST) 

tracks outside research investments, Small Business Innovation Research 

(SBIR) funds per capita (compared to national benchmarks), and company 

financing by stage. OCAST’s partner, I2E, utilizes some other specialized 

measures such as growth in number of companies positioned for financing, 

creation of commercialization road maps for customers, and a host of activity 

measures, such as creation of new partnerships, number of events, and so on. 

Both OCAST and I2E also provide breakouts (via pie charts) of customers by 

industrial sector and by region.41

The Maine Technology Institute (MTI) utilizes university researchers to assess its 

program operations and the impact of its grants.42 The MTI’s performance has 

been tracked using four categories of measures: economic impact, effects on 

company finances, intellectual property development, and relationships. Within 

these categories, several unique measures are used. These include sources of 

material and service inputs (used to assess in-state purchasing) and 

relationships. This latter measure tracks usage of other service providers, such 

as SBDCs, and measures related to customer satisfaction and impact. 

 

40 Nexus Associates, A Continuing Record of Achievement: The Economic Impact of the Ben 

Franklin Technology Partners, Final Report, March 2003. 

41 Oklahoma Center for the Advancement of Science and Technology, Impact Report 2006

(Oklahoma City, Oklahoma: OCAST, January 2006) 10 November 2006 

<http://www.ocast.state.ok.us/Portals/0/docs/brochures/2006-ImpactReport.pdf>. 

42 Charles S. Colgan and Bruce Andrews, Evaluation of Maine Technology Institute Programs,

University of Southern Maine Center for Business and Economic Research, December 2004, 10 

November 2006, 

<http://www.usm.maine.edu/cber/activities/MTI%20Final%20Report%202004.pdf>. 
22

Business Incubation. Because business incubation has been a core economic 

development strategy for several decades, both practitioners and analysts have 

developed an array of metrics and tools for measuring the effectiveness of 

business incubation programs. The National Business Incubation Association 

(NBIA) has led many of these efforts and was an important ARC partner during 

this initiative. In addition, ARC has funded incubator programs throughout its 

existence and has funded several useful studies of the field.43

Most studies of business incubation provide reviews of best practices that 

typically cover the details of program management and facility operations.44

 

Many of these reports also include suggestions for assessing the impact of 

business incubation programs. Lichtenstein and Lyons place a heavy emphasis 

on effective evaluations that capture both process and outcome measures.45

 

Most analysts recommend that traditional economic development metrics, such 

as job creation, be supplemented with other measures that capture unique 

aspects of the business incubation process. For example, most business 

incubators regularly track graduated companies, i.e., firms that have moved from 

subsidized incubator space to owning or leasing space at market rates. 

Customer satisfaction surveys are also frequently used. Business performance 

measures – both during and after residence in the incubator – also offer useful 

data on an incubator’s regional impacts. For example, a firm’s post-graduation 

ability to continue growth, to access outside financing, and to enter new markets 

are all important measures of community economic impacts from business 

incubation. 

Sectors/Networks. Economic developers’ thinking about sector/network 

strategies has undergone an interesting evolution over the past twenty years. 

Beginning in the 1980s and 1990s, a number of fledgling programs sought to 

stimulate the development of sector-based networks in industries such as wood 

products and manufacturing. Much of this initial work was based on successful 

experiments in Denmark and Italy, with overseas lessons applied to US 

experience. Several of the first such US-based programs were located in 

Appalachia. The North Carolina Rural Economic Development Center supported 

a networks initiative in Western North Carolina, and ACEnet sponsored a similar 

effort in Appalachian Ohio. 

 

43 For example, Greenwood Consulting Group, A Survey of Business Incubators in Appalachia

(Washington, DC: ARC, July 2005) 10 November 2006 

<http://www.arc.gov/images/programs/entrep/survey2005.pdf>. 

44 See, for example, Louis G. Tornatzky et al., Incubating Technology Businesses: A National 

Benchmarking Study (Athens, Ohio: National Business Incubation Association, 2003). 

45 Gregg A. Lichtenstein and Thomas S. Lyons, Incubating New Enterprises: A Guide to 

Successful Practice (Washington, DC: Aspen Institute, 1996). 
23

These initial network building efforts remained limited to networks based on an 

industrial cluster or sector.46 More recently, policy makers have sought to 

support the creation of broader entrepreneurial networks that include participants 

from a variety of sectors and disciplines. ARC has invested in both types of 

networks via the Entrepreneurship Initiative. For example, the Team 

Pennsylvania Foundation sought to use ARC funds to stimulate the creation of 

several regional entrepreneurship networks across the state. Other ARC 

investments sought to build sector-based networks in ceramics (New York), arts 

(Ohio), and aquaculture (Georgia). 

While the operations of cluster-based and entrepreneurial networks may differ 

slightly, both types are evaluated using a similar methodology. Evaluators 

regularly seek hard quantitative data on the impact of networks, but qualitative 

measures are also necessary as much of the network impact is qualitative. 

Relationship building, trust, and increased knowledge are all important outcomes 

of network activities. In general, research strongly indicates that more networked 

firms tend to perform better than firms with weak networks and limited strategic 

alliances.47

Networks are typically evaluated using customer surveys, interviews, and case 

studies.48 Typical qualitative outcomes would be high rates of customer 

satisfaction, better awareness of resources and networks, and more openness 

toward collaboration. Quantitative measures examine the network’s ability to 

help produce changes in business outcomes, such as entry into new markets, 

revenue growth, job creation, and the ability to access outside financing. 

As networking becomes a more important part of a typical economic 

development portfolio, analysts are searching for new tools that can better 

assess the role of business support providers as network builders. Social 

network analysis seeks to map and evaluate the power of networking activities. 

In the 1980s and 1990s, researchers sought to simply count the number of 

outside alliances and assess their strength. Today, new software tools allow 

users to map social networks and assess them according to their diversity, 

strength, and influence. These maps can be used to diagnose network 

 

46 For background, see Stuart A. Rosenfeld, “Networks and Clusters: The Yin and Yang of Rural 

Development,” Proceedings of Federal Reserve Bank of Kansas City Conference on Exploring 

Policy Options for a New Rural America, (September 2001):103-120. 

47 An excellent review of this literature can be found in Luke Pittaway, Maxine Robertson, Kamal 

Munir, and David Denyer, Networking and Innovation: A Systematic Review of the Evidence, 

University of Lancaster Institute for Entrepreneurship and Enterprise Development, Working 

Paper 016, 2004, 10 November 2006 <www.lums.lancs.ac.uk>. See also Christopher T. Street 

and Ann-Frances Cameron, “External Relationships and Small Business: A Review of Small 

Business Alliance and Network Research,” Journal of Small Business Management 45.2 

(2007):239-266. 

48 For example, see Peter Witt, “Entrepreneurs’ Networks and the Success of Start-Ups,” 

Entrepreneurship and Regional Development 16 (2004):391-412; Philip Shapira, The Evaluation 

of USNet: Overview of Methods, Results and Implications – Final Report, Georgia Tech Policy 

Project on Industrial Modernization Working Paper 9805, 1998. 
24

weaknesses and design interventions to strengthen the network and the quality 

of company network ties.49 Social network analysis has also been used as a 

means to better describe and understand company and industry supply chains. 

Creating More High-Growth Entrepreneurs 

While all program interventions share the goal of creating high-growth 

entrepreneurs, very few of the Entrepreneurship Initiative’s grantees focused 

exclusively on this objective. However, a number of grant recipients in the 

capital category did embrace a mission of supporting gazelle businesses. 

Because its metrics rely on quantifiable financial measures, evaluations of capital 

access programs tend to be much more rigorous than other types of 

entrepreneurship-related policy interventions. 

Capital. ARC’s capital projects tend to fall into two broad categories – support for 

microenterprise initiatives and investments in more specialized types of financial 

assistance, such as specialized loan funds or new sources of equity capital. 

Many of these investment vehicles promote what they refer to as a “double 

bottom line.” In other words, the funds seek to build strong businesses but also 

promote other social goals, such as community development or environmental 

sustainability. In addition, microenterprise programs measure progress along 

two fronts – traditional business outcomes and measures that capture the 

empowerment of individual clients. These latter metrics could include movement 

off of welfare, increases in family income, and length of self-employment periods. 

Since ARC’s Entrepreneurship Initiative investments have focused on regional 

economic development, this literature review is concerned primarily with how to 

measure the economic impacts of investments in capital programs. Since the EI 

first began in 1997, capacity to measure these impacts has greatly improved. At 

that time, policy makers had very limited experience in creating and operating 

publicly-sponsored seed or equity capital programs. And, many early programs 

were shut down due to political controversies.50

The intervening decade has been one of great experimentation and innovation in 

the development of new capital access initiatives and associated evaluation 

tools. Since 1997, the New Markets Venture Capital Program, the Community 

Development Financial Institutions program, and revisions to the Small Business 

Investment Company (SBIC) program have all been put into place. In addition, 

several new trade associations, including the Community Development Venture 

Capital Association (CDVCA), the National Association of Seed and Venture 

 

49 For example, see <www.networkweaving.com>.

50 An example is Mississippi’s Magnolia Fund which operated for only 2 1⁄2 years before being 

closed due to misappropriation of funds. David L. Barkley, et al., Establishing Nontraditional 

Venture Capital Institutions: Lessons Learned, Rural Equity Capital Initiative Study of 

Nontraditional Venture Capital Institutions, RUPRI PB2001-11A, 2001, 

<http://www.energizingentrepreneurs.org/content/cr_2/2_000026.pdf>. 
25

Funds, and the Angel Capital Association, have all been established to help 

professionalize the field. 

Meanwhile, more established disciplines, like microenterprise and community 

development corporations, have become much more rigorous in strengthening 

professional development programs and creating common performance 

measurement systems. In the microenterprise field, the Aspen Institute’s FIELD 

(Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination) 

program has played a critical role in collecting, developing and disseminating 

new program ideas and new performance measures and tools.51 Its MicroTest 

program provides a useful framework for assessing both program performance 

and client outcomes. Similar comprehensive efforts are underway at leading 

organizations in the field. These include the CDFI Data Project, the Opportunity 

Finance Network’s CDFI Assessment and Rating System (CARS), and the 

CDVCA’s Return on Investment Project.52

All of these efforts, and other outside analysis, reach a similar conclusion – 

publicly-sponsored investment programs need to be managed and measured just 

like private investments. While programs may pursue multiple goals, the 

programs must be managed and assessed on their capacity to make good 

business decisions. As a 2000 National Governor’s Association guide put it, 

“The best programs are not afraid to make money.”53 This same study noted: “In 

the best cases, state leaders take the initiative in getting programs launched and 

setting long-term direction. They rely on experienced, private-sector managers to 

make the day-to-day investment decisions.”54

As seed and equity capital programs have moved in this direction, issues of 

performance measurement have become more straightforward. At the most 

basic level, most funds should be expected to produce a reasonable rate of 

return. A reasonable rate will differ depending on the types of companies in a 

fund’s portfolio. For example, institutional venture capital funds have averaged a 

20.3% annual return over the past ten years, but only a 1% return over the past 

five years. A 2002 study of venture capital rates of return in five countries 

identified average rates of return that fell anywhere between 26% and 45%.55

 

Meanwhile, the CDVCA estimates that its members have enjoyed an average 

 

51 An excellent review of effects of microenterprise programs can be found in Signe-Mary 

McKernan and Henry Chen, Small Business and Microenterprise as an Opportunity and AssetBuilding Strategy, Urban Institute Issue Brief No. 3, June 2005. 

52 Community Development Venture Capital Association, Measuring Impacts Toolkit (New York: 

Community Development Venture Capital Alliance, 2005). 

53 Robert Heard and John Sibert, Growing New Businesses with Seed and Venture Capital: State 

Experiences and Options (Washington, DC: National Governors Association, 2000) 18. 

54 Heard and Sibert, 17. 

55 Sophie Manigart, et al, “Determinants of Required Return in Venture Capital Investments: A 

Five Country Study,” Journal of Business Venturing 16.6 (July 2002):291-312. 
26

annual return of 15.5% over a twenty-five year period.56 Programs that invest in 

microenterprises or slower growth businesses should be expected to post rates 

of return much lower than these benchmarks. 

In addition to rates of return, funds can also be assessed in terms of program 

outputs and economic development outcomes. Program outputs refer directly to 

the activities of the fund or its related programs. They are activity measures that 

track data such as the number and amount of loans, average loan size or 

number/amount of financing provided to certain target customer sets. 

Outcome measures seek to assess a fund’s business and community impacts. 

In this case, most analysts recognize that many traditional economic 

development metrics do a good job of measuring community outcomes. Thus, 

most studies continue to recommend tracking job creation and retention, 

business performance of portfolio companies, and the leveraging of outside 

investments. 

IMPLICATIONS FOR PERFORMANCE MEASUREMENT 

The conceptual framework presented above seeks to portray entrepreneurial 

development as an array of programs that serve entrepreneurs at various points 

in the lifecycle of their business. This “pipeline” model of entrepreneurial 

development, first described by Lichtenstein and Lyons,57 recommends multiple 

policy interventions that help achieve the three broad purposes of creating more 

entrepreneurs, stronger entrepreneurs, and more high-growth entrepreneurs. 

In practice, few program managers have the scope to manage programs that 

cover the whole pipeline of entrepreneurial development. Instead, they typically 

manage a single program or a single type of policy intervention, such as training 

or business financing. This restricted span of control over business outcomes 

makes it difficult for individual programs to introduce more sophisticated tools for 

measuring program performance. 

In the field, entrepreneurship program managers are beginning to consider 

alternative performance measurement systems, but they still feel pressured by 

funders and elected officials to utilize traditional measures of job and firm 

creation outcomes.58 By using traditional economic development metrics to 

 

56 Amy Simpkins, “Community Development Venture Capital: Producing Results for 

Entrepreneurs, Investors, and Communities,” Bridges Summer 2006, 10 November 2006 

<http://stlouisfed.org/publications/br/2006/b/pages/1-article.html>. 

57 Gregg A. Lichtenstein and Thomas S. Lyons, “The Entrepreneurial Development System: 

Transforming Business Talent and Community Economies,” Economic Development Quarterly

15.1 (2001):3-20. 

58 Laura Czohara and Julia Melkers, Performance Measurement in State Economic Development 

Agencies: Lessons and Next Steps for GDITT, Georgia State University Fiscal Research Center 

Report No. 92, February 2004. See also, Erik R. Pages and Kenneth A. Poole, 

“Entrepreneurship Promotion as an Economic Development Strategy: Next Steps in 
27

assess entrepreneurial and innovation-based development, program managers 

are employing inappropriate performance measures. Building businesses takes 

patience and resilience. It is unrealistic to expect quick results in terms of 

traditional economic development outcomes. Program managers need a more 

thoughtful approach to tracking the performance of their efforts. Moreover, new 

measures for entrepreneurial development need to be devised that recognize a 

stream of benefits over an extended period of time. Measuring entrepreneurial 

development using annual job creation impacts alone is like measuring the 

success of a loan program solely by the ability of the borrower to repay on a 

monthly basis. Short-term job creation is simply not the purpose of these 

programs. 

In their quest to “tell a better story,” program managers are considering a host of 

other metrics, which tend to fall into the following categories: 

 Activity/Output Measures (e.g., number of customers served) 

 Customer Satisfaction Surveys 

 Input Measures (e.g., increase in budget) 

 Outcome Measures (e.g., increase in business starts) 

 Cost Efficiency (e.g., return on investment). 

While some programs use a full range of measures, most economic developers, 

including ARC grant recipients, use a more limited menu of metrics that is 

generally limited to job creation and retention and the leveraging of outside 

investments. These limited metrics can provide a much skewed picture of the 

impact of entrepreneurial development efforts. 

Economic development programs have responded to this challenge in a number 

of ways. The metrics used by Small Business Development Centers (SBDCs) 

offer one model. For example, the North Carolina Small Business and 

Technology Development Centers track outside financing (loans, equity, and 

SBIR/Small Business Technology Transfer Program funds), state and federal 

contract awards, customer satisfaction, job creation, firm sales growth, and 

incremental taxes generated. 59 Florida combines capital formation, business 

start-ups, jobs created/retained, sales growth, and contract awards to calculate 

the return on investment (ROI) for its SBDC system.60 A slightly more 

sophisticated set of metrics is used by various statewide technology development 

organizations such as Pennsylvania’s Ben Franklin Partners, the Oklahoma 

Center for the Advancement of Science and Technology, and the Maine 

Technology Institute. 

 

Institutionalizing the Field,” Applied Research in Economic Development 3.2 (December 

2006):.10-27. 

59 North Carolina Small Business and Technology Development Center, 2005 Annual Report, 

<http://www.sbtdc.org/pdf/annual_report.pdf>. 

60 See <www.floridasbdc.com>. 
28

Useful benchmarks for metrics can also be found in the work of various trade 

associations working in the fields of microenterprise and community 

development. These groups are seeking to develop industry-wide metrics that 

can help improve management practices and provide industry benchmarks for 

effective practice. 

Local groups are also experimenting with new methods. For example, Maine’s 

Coastal Enterprises, Inc. has developed a very rigorous and comprehensive set 

of program measures.61 Its Social Information System combines a host of 

measures that provide internal feedback to management and employees, permit 

assessment of program outcomes, and also generate data and case studies that 

can be communicated to an outside audience. 

These varied initiatives seek similar types of information on client companies. 

They ask for information on firm growth, performance, and local economic 

impact. In keeping with their social missions, the organizations also tend to track 

information on employee benefits, wages, and community and environmental 

impacts. 

At the federal level, the US Department of Agriculture’s Rural BusinessCooperative Services (RBS) is implementing a Socio-Economic Benefits 

Assessment System (SEBAS) developed at the University of MissouriColumbia.62 SEBAS provides a means of evaluating performance and 

effectiveness of RBS’ loan and grant programs by measuring the economic and 

social impacts that these have on rural community environments. Using a multiregional social accounting matrix model, SEBAS is able to measure both direct 

and indirect effects of the loans and grants, such as business sales, income, 

indirect business taxes, employment, household income, public revenues, and 

distribution of household income and occupations. 

Finally, many organizations are seeking new ways to measure “innovation 

impacts.” This work remains relevant to the existing literature on 

entrepreneurship as many entrepreneurial development programs seek to 

stimulate innovation-related outcomes, such as improved productivity rates, 

wider diffusion of new technologies, and improvements in human capital.63 The 

US Department of Commerce has created a new “Measuring Innovation in the 

21st Century Economy Advisory Committee.” Commerce’s Manufacturing 

Extension Partnerships and the Advanced Technology Program have probably 

faced the most rigorous scrutiny of any publicly-funded technology program in 

the country. The National Institutes of Standards and Technology (NIST) have a 

 

61 Coastal Enterprises, Inc., Measuring Impact in Practice (Wiscasset, ME: Coastal Enterprises 

Inc., February 2006). 

62 Robinson, Dennis and Zuoming Liu, A User Guide to the Socio-Economic Benefits Assessment 

System: A Rural Business-Cooperative Services Assessment Tool for Economic Development. 

Community Policy Analysis Center, University of Missouri-Columbia, December 2004. 

63 B.K. Atrostic, “Measuring U.S. Innovative Activity,” US Census Bureau Center for Economic 

Studies Working Paper (07-11), March 2007. 
29

huge library of resources on this work.64 Studies of the MEP have generally 

used industry surveys to assess whether client firms have introduced new 

processes (such as Total Quality Management or a reconfigured plant layout) as 

a result of MEP’s technical assistance.65

Overseas, the European Union, the OECD, and many national and regional 

programs are developing interesting new ways to measure progress in innovation 

policy. They have also made major efforts to go beyond studies and to get 

practitioners to use these tools in the field. For example, the OECD has 

produced the very detailed Oslo Manual for collecting and interpreting innovation 

data. The European Union has its own PAXIS Manual that profiles hundreds of 

effective measurement tools and practices. Many national governments are also 

doing good work in this area. Britain’s Department of Trade and Industry has 

recently published a useful study of UK innovation indicators.66 These efforts all 

share a commitment to regular comprehensive performance measurements that 

capture both program outputs as well as community outcomes. 

These efforts generally propose the use of detailed company surveys to capture 

data on a wider range of variables related to innovation. In addition to traditional 

measures such as new patents or licenses, the new survey tools ask questions 

about education backgrounds of new hires and existing personnel, development 

of new R&D projects, joint ventures, training expenditures, and the deployment of 

“high-impact” human resource practices (e.g., equity sharing, team building). 

The UK study captures some interesting innovation information via firm 

surveys.67 These data are organized around three categories, with several submeasures under each category: 

 Product-Orientated Effects: increased range of goods or services; entry 

into new markets; improved quality of goods and services 

 Process-Orientated Effects: improved flexibility of production or service 

provision; increased capacity for production or service provision; reduced 

unit costs 

 Other Effects: reduced environmental impact or improved health/safety; 

met regulatory requirements; improved value added. 

 

64 These reports are available at <http://www.atp.nist.gov/eao/eao_pubs.htm>. 

65 See, for example, Eric Oldsman, “Do Manufacturing Extension Programs Matter?” Research 

Policy 25.2 (March 1996):215-232. 

66 European Commission, Directorate General, Enterprise and Industry, The PAXIS Manual for 

Innovation Policy Makers and Practitioners (Brussels: European Commission, 2006); 

Organization for Economic Cooperation and Development, Oslo Manual: Guidelines for Collecting 

and Interpreting Innovation Data Third Edition (Paris: Organization for Economic Cooperation and 

Development, 2005); United Kingdom Department of Trade and Industry, Innovation in the UK: 

Indicators and Insights, DTI Occasional Paper No. 6, July 2006. 

67 United Kingdom Department of Trade and Industry (2006). 
30

These surveys also ask companies to comment on innovation constraints they 

face in their region or industry. These data help program managers anticipate 

future needs or service requirements. 

Drawing on this review of literature and insights into performance measurement 

gained, the evaluation team developed a metrics framework to guide the 

evaluation, particularly the data collection for the sample of ARC projects. 

Chapter 4 describes the metrics framework and the overall approach to the 

evaluation. 
31

CHAPTER 4 

APPROACH TO THE EVALUATION 

In designing the approach to this evaluation, the team was conscious of the 

charge articulated by ARC in the Request for Proposals for the evaluation. 

ARC was seeking a “policy impact and program evaluation”68 of the 

Entrepreneurship Initiative that would include: 

 An examination of the project outcomes of a sample of projects closed 

since 1997 

 An assessment of wider policy impacts informed by input from 

stakeholders within the region. 

Through its Entrepreneurship Initiative, the ARC has invested in a diverse set of 

projects in a diverse set of communities. Thus, a “one-size-fits-all” approach 

would not produce an effective and comprehensive evaluation. The evaluation 

team’s approach to this project was to combine a detailed and rigorous review of 

a sample set of projects with extensive interviews of key players on the project 

teams and in the targeted communities. This approach provided us with a strong 

set of collective program metrics as well as rich detailed case-study-like 

information on key projects and their outcomes. 

The ideal evaluation would measure project outcomes relative to the sample 

programs’ ultimate goals.69 The ARC evaluation is complicated by the fact that

individual projects (those receiving ARC investments) had a set of specific goals 

each was trying to achieve, i.e., “local” impacts, such as increasing access to 

equity capital or incorporating entrepreneurship education into high school 

classrooms. In addition, through investments made in the totality of EI projects, 

ARC was trying to provide communities with the tools they needed to support 

homegrown entrepreneurs because they “play an important role in creating selfsustaining local economies and improving the quality of life in Appalachia”70 – a 

broader set of regional impacts. While “local” and regional impacts were 

complementary in most cases, individual projects were evaluated based on their 

ability to achieve locally articulated goals and not these broader regional impacts. 

To distinguish between these two levels of impact, the evaluation team chose to 

couple an assessment of sample projects’ outcomes, as measured by data 

reported by project leaders, with identification of wider policy impacts as reported 

 

68 Request for Proposals for A Program Evaluation of ARC’s Entrepreneurship Initiative, 

Appalachian Regional Commission, July 28, 2006. 

69 David J. Storey, Six Steps to Heaven: Evaluating the Impact of Public Policies to Support Small 

Businesses in Developed Economies, Working paper No. 59, Small and Medium Sized Enterprise 

Centre, Warwick Business School, University of Warwick, September 1998, 3 December 2007 

<http://www2.warwick.ac.uk/fac/soc/wbs/research/csme/research/working_papers/wp59-

six_steps.pdf>. 

70 Appalachian Regional Commission, Entrepreneurship Initiative, 9 August 2006 

<http://www.arc.gov/index.do?nodeId=19#toc>. 
32

by key informants – both project leaders and regional stakeholders. A list of nonproject stakeholders is included as Appendix B. 

Program outcomes for the EI investments can be measured in two ways. One is 

to compare the levels of outcome measures before and after the ARC 

investment. To be accurate, one must be sure that the metric or outcome 

measure would not have changed without the ARC-funded program (i.e., the “but 

for” criterion). Alternatively, one could conduct a with and without comparison, 

estimating what the outcome would be without ARC investment and comparing 

this measure to the outcome with ARC investment. To be accurate using this 

strategy, one would need to employ a control group of communities, identical in 

every way to the communities that housed an ARC-funded program. While 

employing a control group may make sense when looking at firm-level impacts of 

public policy, as suggested by Storey, it is more difficult to consider identifying 

control groups for public policies aimed at bringing about changes in firms, 

students, educators and even communities. As a result, the with and without

approach was deemed impractical for this evaluation as it would require 

identifying at least 88 “control” places – some individual communities, some 

multi-county regions, some multi-state regions – and collecting comparable data 

through interviews and secondary data. 

Given the scope of the evaluation as articulated by ARC, this latter approach was 

not used in this evaluation. While neither approach is without criticism (see 

caveats at the end of this chapter), the evaluation team chose the before and 

after strategy and relied on those administering the programs and respondents to 

stakeholder interviews to indicate if the “but for” criterion was satisfied. 

 

RESEARCH QUESTIONS 

A number of research questions guided this evaluation effort. 

 Have sample projects achieved their stated project objectives? 

 How does the performance of ARC projects compare to the performance 

of similar types of projects in other regions or countries? 

 How well do existing performance metrics capture the impact of ARC 

projects and what additional metrics would improve the usefulness and 

integrity of evaluation results? 

 What broader policy impacts are associated with ARC project 

investments? 

 What innovative practices or lessons learned have relevance for other 

national and international economic development efforts? 

ARC’s Entrepreneurship Initiative is unique in that investments have been made 

in a diverse project portfolio, varying widely by geography, program type (i.e., 

access to capital, technical assistance and training, incubators, entrepreneurship 

education, and sectors), type of lead institution and identified output and outcome 
33

measures. As such, no single performance measurement or metric would 

adequately capture the impact of ARC entrepreneurship investments on the 

region. For example, the performance measures associated with a youth 

entrepreneurship education project are likely to vary significantly from those 

associated with a community development venture capital project. The evaluation 

methodology recognized this uniqueness and created a rigorous and broad set of 

project metrics that would demonstrate the impact of a wide range of project 

types on the region. 

METRICS FRAMEWORK 

The review of the literature on the state of the art in evaluating entrepreneurial 

development programs yields one important conclusion: the field has a long way 

to go in terms of creating rigorous, compelling, and effective techniques and 

strategies for evaluating programs and communicating their effectiveness to 

policy makers and community stakeholders. Indeed, much of the literature 

reflects what Storey would describe as “monitoring” as opposed to rigorous 

evaluation of performance as compared to the objectives established for a 

program or investment.71 Drawing on the lessons from this review and based on 

the conceptual framework developed to guide the evaluation process (described 

in Chapter 3), performance measures were defined for each of the program 

categories that, as far as is practical from the available data, reflect the 

appropriate programmatic goals. The evaluation team developed a list of metrics 

that captures key outputs and outcomes for each of the five program categories. 

This framework is depicted in Table 4.1. 

The development of this metrics framework was guided by the literature review 

and a critical discussion of the need to link outcomes to program goals. For 

example, while the literature on entrepreneurship education offers a range of 

metrics related to student outcomes (e.g., higher test scores), the evaluation 

team chose to focus on those metrics that relate directly to the programmatic 

goals of the ARC Entrepreneurship Initiative – in the case of entrepreneurship 

education, the goal is to create more entrepreneurs in the pipeline. 

These metrics allow researchers to obtain a good picture of the progress of the 

various projects funded through the ARC’s Entrepreneurship Initiative. They 

allow an understanding of the impacts of each program type, and also help 

identify exemplary practices and programs. They do not yet give a good picture 

of the overall effects of varied entrepreneurship initiatives across a specific 

region. Achieving this objective requires that researchers assess programs 

operated by numerous different organizations (many outside of ARC’s purview) 

across all types of policy interventions along the business life cycle. This 

important effort, which falls outside scope of this evaluation, presents an 

interesting challenge for future research. 

 

71 Storey (1998):12. 
34

TABLE 4.1. PROPOSED METRICS FRAMEWORK

TYPE OF PROJECT OUTPUT INDICATORS OUTCOME INDICATORS

Capital Access  Number of loans/year 

 Amount ($)/year 

 Number of funds created 

 Fund size ($) 

 Average loan size 

Percent sectoral distribution 

of loans ($) 

 Amount ($) funds leveraged 

(public, private or other) 

 Number of jobs (FTEs) 

created/retained 

 Percent of funded firms still in 

business 

 Annual income and benefits/job or 

average wage/job 

Sectors  Number of participants in 

networking meetings 

 Number of members 

(change over time) 

 Increase in inter-firm 

collaborations 

 Number of partnerships created 

 Amount ($) of increased sales 

from network participation 

 Number of jobs (FTEs) 

created/retained 

 Change in total sector sales 

 Number of business starts in 

targeted sector 

Incubators  Number of current clients 

 Number of clients served 

 Number of graduated firms 

 Number of clients still in 

business 

 Amount ($) leveraged by 

incubator (other 

public/private money) 

 % businesses retained in service 

area 

 # of jobs (FTEs) created/retained 

while in incubator 

 # of jobs (FTEs) created/retained 

post-graduation 

 Amount ($) of capital raised by 

tenants 

Entrepreneurship 

Education 

 Number of participants 

enrolled in the program 

 % of local schools offering 

(pre and post investment) 

 % of participants completing 

the program 

 Increase in awareness of 

business concepts (pre vs. post) 

 Increase in number of participants 

considering business creation as 

a career option (pre vs. post) 

 Change in student performance 

before and after program 

 Number of students starting 

businesses 

 Number of students that stay 

within the service area 

Technical 

Assistance 

And Training 

 Number of business starts 

 Number of business 

expansions 

 Number of clients 

 Number of clients still in business 

 Number of jobs (FTEs) 

created/retained 

 Private $ raised by client firm 

 Annual income and benefits/job or 

average wage/job 

Jobs (FTE) = wage earners and proprietors 
35

While this broad set of performance metrics provided a useful framework for the 

evaluation, a “best in class” metrics system as proposed later in this report would 

require refinement of this list down to those outcome measures that best capture 

the impacts of entrepreneurship development investments across a range of 

project types. Recommendations in Chapter 9 include the identification of a set of 

outcome measures that could result in an operationalized “best in class” metrics 

system to guide future investments by ARC. 

DATA COLLECTION 

 

To gather the data identified in the metrics framework, the evaluation team used 

a four-part approach: 

1. Data for each sample project were gathered from the ARC project file – 

the program category, funding level, goals (or need addressed), and how 

the program was implemented. 

2. Additional data on project performance were gathered through phone 

interviews with project directors – outcomes produced (both quantitative 

and qualitative), value attributed to the project, success in achieving 

objectives. 

3. Data on broader capacity and policy impacts were gathered through 

phone interviews with both project and non-project stakeholders – policy 

impacts in the region, other qualitative and quantitative impacts on 

capacity in the region. 

4. Data on place-based and broad policy impacts were gathered through 

selected site visits in geographic areas where investments in a number of 

program categories had been made, determined after the previous parts 

of the data collection process were completed. 

To facilitate the collection of data in part two above, a draft protocol for the followup phone interviews was developed (see Appendix C).

DESCRIPTION OF EVALUATION TASKS 

A number of specific evaluation tasks were identified and completed. To provide 

context for the evaluation and to guide the development of the metrics 

framework, the team completed a literature review of entrepreneurship 

development project evaluations conducted by organizations such as 

Organisation for Economic Co-operation and Development (OECD), Small 

Business Administration (SBA), Economic Development Administration (EDA), 

US Department of Agriculture (USDA), and National Institute of Standards and 

Technology (NIST) – Manufacturing Extension Program (MEP) and Advanced 

Technology Program (ATP), as well as evaluations by private organizations such 
36

as Community Development Venture Capital Alliance (CDVCA), National 

Business Incubation Association (NBIA) and other local and state organizations 

(see Chapter 3 and Appendix A). In addition to guiding development of the 

metrics framework, this review helped the evaluation team determine whether a 

comparative analysis of ARC performance relative to other similar types of 

initiatives was feasible. 

Members of the evaluation team conducted phone interviews with non-project 

stakeholders to identify the broader impacts of ARC Entrepreneurship Initiative 

projects. These individuals tended to be national or regional experts on 

economic development and the specific issues and challenges facing the 

Appalachian region or specific states and localities within the ARC service area 

(see Appendix B for a list of stakeholders interviewed and Appendix D for the 

protocol used.) Interviews were also conducted with state program managers to 

develop an understanding of how the EI was implemented in each state and with 

other key leaders who had deep experience in the region. Recurring themes from 

these key informant interviews were identified and reported as qualitative impacts 

of ARC program investments. 

The most significant evaluation task was the team’s review of project files 

provided by ARC for a sample of 114 projects to understand the purpose, goals, 

objectives, and identified outputs and outcomes. ARC’s performance 

measurement system tracked a number of key performance metrics across 

projects that were relevant to this evaluation: 

 Businesses served 

 Jobs created 

 Jobs retained 

 Project participants 

 New businesses created 

 Private investment leveraged. 

While this common set of metrics facilitates assessment of the impacts of ARC’s 

project portfolio, the evaluation team also used the metrics framework developed 

from the literature review to collect additional data on the sample projects. The 

team conducted phone interviews with key staff of each project’s lead institution 

to review and/or collect information and data on both the performance metrics 

reported to ARC and the broader set of performance metrics identified by the 

project team in consultation with the advisory group. It is important to note that 

few project leaders were able to report data for this broad set of metrics. Since 

these metrics were not necessarily identified as outcomes by the grantees, it was 

not expected that they would be able to report on all of these metrics. However, 

the evaluation team wanted to be able to capture as broad a set of performance 

metrics as possible during these interviews, so the metrics framework was used 

as a guide. 
37

Using data obtained from final project reports submitted to ARC and data 

collected during the phone interviews, the team prepared a descriptive statistical 

analysis of performance metrics. This analysis allowed the team to evaluate how 

sample projects had achieved their stated objectives and how additional metrics 

contribute to understanding the outcomes and impacts of project investment. 

These interviews provided useful data on project outcomes. As part of the effort 

to gain a sense of impact beyond individual projects, the project team also 

conducted site visits to four locations with the region. These site visits were used 

to conduct personal interviews and/or focus groups with project teams and other 

community stakeholders. The insights gained through these site visits helped to 

confirm what the team learned through other interviews and to provide case 

examples to illustrate specific lessons learned associated with this evaluation. 

A final task was to conduct a meta-analysis of the outcomes and impacts 

associated with ARC projects as compared to any benchmarks identified through 

the literature review. The purpose of this analysis was to assess the impacts of 

ARC programs relative to similar investments in economic development 

activities. As described in Chapter 3, the literature review identified few other 

evaluations that sought to measure the outcomes of entrepreneurship 

development investments like ARC – initiatives that are designed to change the 

culture and economic development direction of a region. Assessments were most 

often completed for particular programs, such as SBDCs, with a focus on 

measuring impacts at the individual entrepreneur (customer) level. Most of these 

assessments have measured outputs from program activities, e.g., number of 

clients served, dollars invested, as opposed to outcomes on either individual 

businesses or the overall economy. 

It was difficult to use information from these previous assessments for a 

comparative analysis of the ARC EI since ARC investments were strategically 

made to demonstrate the potential for entrepreneurship development and to 

change attitudes about economic development within communities and the 

broader region. Specific projects were not designed simply to create jobs, but 

had a broader set of goals including, for example, to: 

 Expand use of e-commerce 

 Create an angel network 

 Double the number of high schools teaching entrepreneurship 

 Train teachers to use entrepreneurship curriculum 

 Prepare 100 new business plans as part of a competition 

 Complete a business incubator feasibility study and strategic plan 

 Attract 10,000 visitors in first year of a heritage tourism development. 

Even detailed metrics for these types of broader goals tell only part of the story. It 

is not possible to talk about entrepreneurship development by reporting on any 

single metric. However, in lieu of a broader meta analysis, it was possible to 

calculate for the sample projects and for the entire portfolio an estimate of public 

cost per job created that could be compared to estimates for other similar 

programs. These comparisons are provided in Chapter 6. 
38

As a result of these challenges, the evaluation team chose to recognize the 

embryonic nature of entrepreneurship development evaluation research and to 

use what had been learned through both the literature review and in-depth 

assessment of sample projects to suggest what performance metrics would best 

capture the essence of initiatives like ARC’s EI – a “best in class” metrics system. 

This system is discussed in detail in Chapter 9. 

SAMPLE SELECTION PROCESS 

ARC made 448 grants to entrepreneurship-related projects from 1997 through 

2005. This evaluation focused on projects whose commitment from ARC had 

been completed, i.e., “closed” projects between 1997 and 2005; a total of 354 

grants were closed at the time of the evaluation. However, since a number of 

grants were for follow-on investments, the evaluation team further narrowed the 

universe of eligible projects to a set of 229 unique, closed projects. While most 

projects were designed to address the specific issues or needs within a 

community, region or state, some of these investments were in region-wide 

projects, primarily designed to raise the level of awareness of entrepreneurship 

generally or to provide an opportunity to explore a specific issue with applicability 

across the region, such as the role of business incubators or issues related to 

access to capital for entrepreneurs. 

This group of 229 projects represented investments in all types of projects (Table 

4.2) and in all states in the region (Table 4.4). It was from this universe of 

projects that the evaluation sample was drawn. All grants related to a specific 

project were reviewed as part of the sample analysis. 

Each of the closed, unique projects was assigned to one of five categories by 

reviewing the title of the project and the description of the project contained in the 

September 2003 ARC publication, Entrepreneurship Initiative: Program Summary 

and Approved Projects. Some projects in the database were not in this 

publication, notably those coded as “Commission” rather than coded by a specific 

state. These Commission projects were coded according to the titles where 

possible. The five categories used were: 

 Capital Access – any project where loans, grants or equity investments 

were made in companies 

 Entrepreneurship Education– all projects for youth 

 Incubator – any project to study the feasibility of, plan, or operate an 

incubator; virtual incubators included under technical assistance 

 Sectors – any project whose aim was to support entrepreneurs in a single 

sector or type of business; excluding incubator projects 

 Technical assistance and training – any project where assistance was 

given directly to individual entrepreneurs. 
39

Since community capacity was a cross-cutting theme, the evaluation team 

determined that all projects in the sample would be evaluated for community 

capacity building. A total of 28 projects were not coded because they did not 

have a description, the title was vague, or the project was for a conference or 

similar activity. The distribution of closed, unique projects is shown in Table 4.2.

Based on this distribution, the team randomly selected a sample of 114 projects. 

Then, three projects were discarded and three were added to make the state 

distribution more even.

Table 4.2. Distribution by Program Category – Universe of Unique Projects 

Category Number Percent

Capital Access 18 7.9 

Entrepreneurship Education 42 18.3 

Incubators 30 13.1 

Sectors 40 17.5 

Technical Assistance and Training 71 31.0 

Other/Not Coded 28 12.2 

Total 229 100 

The objective was to choose a representative sample of projects, defined as 

having the same distribution of projects by category and by state as the universe 

of unique projects. In addition, the team sought to have a sample that was 

representative in terms of size of investment by ARC. The team concluded that 

the sample was representative, but not strictly random, as described below. 

Table 4.3 shows the number and percent of projects in each program category in 

the original sample of 114 and in the final sample of 88.72 Based on the 

distribution of the sample projects by program category, the final evaluation 

sample appears to be representative of ARC’s EI projects as a whole. 

The geographic distribution of the sample projects as compared to the 

distribution by state for the universe of unique projects is shown in Table 4.4 and 

in Figure 4.1.73 Note that three states, Georgia, North Carolina and South 

Carolina, appear in the sample at a rate slightly higher than their actual number 

 

72 Although 114 projects were included in the original sample, completed interviews were 

obtained for 88 projects. Project interviews were not completed for a variety of reasons including 

loss of institutional memory of the project because project leaders had left the organization, 

inability to schedule interviews after repeated attempts, and the closure of lead organizations. 

73 The map in Figure 4.1 indicates that some grants went to organizations outside the region. This 

apparent anomaly occurs because, in some cases, the lead organization was located outside the 

ARC region, although the project was implemented within the region. 
40

of projects. These three states had fewer total projects and, in order to include 

more than one project from each of these states in the sample, the team chose to 

over sample to insure more representative findings from these states. The 

sample appears to be representative of the geographic diversity of EI projects. 

In terms of ARC investment, the sample again appears to be representative of 

the project universe. Average ARC investment per project was $126,387, while 

the average amount of ARC investment per project in the sample was $145,997. 

The range of ARC investments in the universe was $2,000 to $2.2 million; within 

the sample, the range was $3,500 to $2.2 million. The total investment in the 

ARC projects was $42,971,688; the total in the sample was $12,847,733. 

Table 4.3. Distribution by Program Category – Original and Final Sample of 

Projects 

CATEGORY ORIGINAL ORIGINAL 

% 

FINAL FINAL 

% 

Capital Access 10 8.8 8 9.1 

Entrepreneurship 

Education 

23 20.2 17 19.3 

Incubators 17 14.9 12 13.6 

Sectors 23 20.2 17 19.3 

Technical assistance and 

Training 

41 36.0 34 38.6 

Other/not coded --- --- --- --- 

Total 114 100% 88 100% 

CAVEATS TO THE EVALUATION APPROACH 

Any evaluation must address the potential caveats associated with the proposed 

methodology. While it is important to acknowledge and, to the extent possible, 

mitigate these shortcomings, the methodology used in this project was as 

rigorous as possible given the diverse set of individual projects, each with its own 

set of self-defined performance measures that characterize the ARC 

Entrepreneurship Initiative. 

The first caveat to the selected methodology was its reliance on self-reporting of 

outcomes and other performance metrics. In many circumstances, this would be 

a severe constraint on the integrity and objectivity of the data and conclusions. In 

this case, however, the grantee organizations were required to submit final 

reports as part of their contractual agreement with ARC and these reports could 

be, and often were, subject to audit. In addition, ARC’s Regional Planning and 

Research division made quasi-random validation field visits to project leaders. 
41

The goal of these visits was to validate that the activities proposed under the 

grant were actually taking place. The evaluation team took the view that the 

project reports could legitimately be regarded as binding and accurate depictions 

of the impacts associated with each project. It was beyond the scope of this 

evaluation to independently verify the data reported by individual projects as part 

of their reporting to ARC or as part of the data collection process associated with 

this evaluation. 

To develop insights into the broader policy impacts associated with ARC 

investments, we chose to rely on interviews with key stakeholders, an adaptation 

of the key informant research approach. Since these broader impacts related to 

such things as changes in attitudes toward entrepreneurship and economic 

development in the region, renewed hope for the future, and the elevation of a 

new set of leaders, the best means of gaining insight into these changes in 

regional or community capacity was to question those who had deep 

appreciation for the culture of Appalachia, extensive experience working in 

economic development in the region, and expertise in particular project areas in 

which ARC invested. Recognizing the inherent bias in relying on interviews with 

individuals to assess overall change in a region, the evaluation team chose to 

identify a broad group of stakeholders and to report recurring themes and 

observations that were widely held within this group. 

Another caveat relates to what might be called a “bias toward success.” If the 

universe of closed projects does not include both “successful” and “failed” 

projects, the evaluation results will be biased toward success, i.e., the evaluation 

will not capture the insights and outcomes (or lack thereof) associated with 

projects that were not successful. Indeed, failed projects may provide insights 

that are helpful in addressing some of the research questions articulated in this 

proposal. Based on the outcome of the project interviews, it was clear that the 

sample included projects that were successful and sustainable, as well as those 

that were not. 

The final caveat relates to the rigor of statistical analysis permitted as part of this 

evaluation. The small sample size and the diverse types of projects included in 

the sample prevent the use of sophisticated statistical models or methods. 

However, the descriptive statistical methods used allowed us to address the 

critical research questions outlined above and to provide recommendations for 

ARC’s consideration in guiding the entrepreneurship investments in the future.
42

Table 4.4: Distribution of ARC Projects by State – Universe of Unique Projects, Original Sample and Final Sample* 

# UNIVERSE # ORIGINAL 

SAMPLE 

# FINAL 

SAMPLE 

% UNIVERSE % ORIGINAL 

SAMPLE 

% FINAL 

SAMPLE 

Alabama 20 11 8 10 10 9 Georgia 7 5 4 3 4 4 Kentucky 14 8 6 7 7 7 Maryland 11 5 5 5 4 6 Mississippi 14 8 5 7 7 6 New York 16 9 7 8 8 8 North Carolina 3 3 3 1 3 3 Ohio 26 14 12 13 12 14 Pennsylvania 12 7 6 6 6 7 South Carolina 3 3 3 1 3 3 Tennessee 10 6 4 5 5 4 

Virginia 38 21 14 19 18 16 

West Virginia 21 11 10 10 10 11 ARC Commission 6 3 1 3 3 1 

201 114 88 100 100 100 

*

 A total of 28 projects were not coded as described in the text. 
43

Figure 4.1. Map of Sample and Total EI Projects, by State 

 
44

CHAPTER 5 

THE WIDER REGIONAL ECONOMY – ENTREPRENEURSHIP 

TRENDS IN APPALACHIA 

INTRODUCTION 

The Entrepreneurship Initiative was a bold attempt to bring about an economic 

and attitudinal transformation across the Appalachian region. It is therefore 

appropriate to step back from the specifics of the initiative – the individual 

projects and programs funded and the communities in which they are located – 

to consider at a more general level entrepreneurship trends across the region. In 

particular, there is value in looking at the state of entrepreneurship in Appalachia 

at the beginning of the initiative, in comparison with the nation as a whole, and 

whether there has been any discernible change in the distribution of 

entrepreneurial activity within the region. The initiative was launched in 1997 and 

this evaluation considered projects closed through 2005, so these are the two 

comparison dates used to assemble data in the following analysis. No attempt is 

being made to establish cause and effect; the analysis is intended to 

demonstrate the context within which the initiative was implemented. 

THE CHALLENGE OF MEASUREMENT 

Entrepreneurship is a difficult concept to measure, and many aspects of 

entrepreneurship success are intangible.74 However, several data sources allow 

an examination of employment and income indicators at the county level to 

assess general entrepreneurship trends. This analysis relies upon three data 

sets: 

 Bureau of Economic Analysis’ Regional Economic Information System 

 U.S. Census Bureau’s Nonemployer Statistics 

 U.S. Census Bureau’s County Business Patterns. 

The Center for the Study of Rural America at the Federal Reserve Bank of 

Kansas City developed a series of indicators on the breadth and depth of 

entrepreneurship at a regional level.75 These measures use data from the 

Bureau of Economic Analysis’ Regional Economic Information System and the 

U.S. Census Bureau’s Nonemployer Statistics. The “breadth” of 

entrepreneurship is measured by nonfarm proprietors as a proportion of total 

nonfarm employment, and the “depth” of entrepreneurship is a measure of 

 

74 Stephan J. Goetz and David Freshwater, “State-Level Determinants of Entrepreneurship and a 

Preliminary Measure of Entrepreneurial Culture,” Economic Development Quarterly 15.1 

(February 2001):58-70.

75 Sarah Low, “Regional Asset Indicators: Entrepreneurship Breadth and Depth,” The Main Street 

Economist (September 2004):1-4. 
45

nonfarm proprietors’ income per total nonfarm proprietors. These indicators have 

been applied to all the counties in the Appalachian region for 1997 and 2005. 

The second set of indicators follows from work of the Association for Enterprise 

Opportunity (AEO). This organization uses the U.S. Census Bureau’s 

Nonemployer Statistics and County Business Patterns data to estimate the 

number of microenterprise businesses and the number of microenterprise 

employees.76 These datasets have been used in this analysis to provide 

estimates for the number of microenterprises and the amount of microenterprise 

employment for Appalachian counties for 1997 and 2005.77

 

Additionally, a measure was developed to provide a sense of how important 

entrepreneurship is to the local economy. This measure presents both 

proprietors’ income as a proportion of total personal income in each county in 

Appalachia and – in order to distinguish the effects of entrepreneurship from 

broader trends – shows how these two components have themselves changed 

over time. 

It should be noted that ARC used a broad definition for the EI that includes smalland medium-sized enterprises with fewer than 200 employees. This has 

particular relevance for the capital access programs and for some of the 

technical assistance activities, but for the purposes of this analysis data on 

proprietors and microenterprises have been used as they serve as better (but far 

from perfect) proxies for entrepreneurial activity. To provide a context, Table 5.1 

shows growth rates in the number of new businesses in different size categories 

between 1997 and 2005. 

Table 5.1: Change in the Number of Small- and Medium-Sized Enterprises 

410 ARC COUNTIES UNITED STATES

Size 

# Employees 

1997 2005 % 

Change 

1997 2005 % 

Change 

1-4 274,877 208,856 2.2 3,757,627 4,119,363 9.6 

5-9 106,893 110,852 3.7 1,354,488 1,411,199 4.2 

10-19 64,319 70,180 9.1 856,118 937,617 9.5 

20-49 41,708 46,164 10.7 572,437 636,625 11.2 

50-99 13,773 15,258 10.8 194,068 219,324 13.0 

100-249 8,413 8,750 4.0 113,832 125,027 9.8 

Total 509,983 532,060 4.3 6,848,570 7,449,155 8.8

Source: U.S. Census Bureau, County Business Patterns – Establishments by Size Class 

 

76 For more information, see <www.microenterpriseworks.org>. 

77 The number of microenterprise businesses is the number of nonemployer establishments plus 

the number of establishments with 1-4 employees. The estimate of microenterprise employment 

is the number of nonemployers plus an average of 2.5 employees per establishment 

(establishments with 1-4 employees). This methodology differs from the AEO methodology. 
46

Overall, the growth rate of business establishments with fewer than 250 

employees in the ARC region was less than half of that of the nation as a whole, 

with the greatest divergence in those establishments with fewer than five 

employees and those in the range of 100-249 employees. In no size category 

does the ARC region outperform the national rate. Given the comparatively low 

growth rates for small- and medium-sized enterprises in the region, the EI’s 

inclusion of these firms as a target for support appears to be strategic. 

UNDERSTANDING THE REGIONAL CONTEXT 

Three main observations derive from the following analysis of regional 

entrepreneurship data and trends:

 Trends in nonfarm proprietor and microenterprise employment in 

Appalachia showed an increase over the 1997-2005 period consistent with 

the nation as a whole. The counties benefiting most from above national 

average increases were in the southern tier of Appalachia. 

 Trends in nonfarm proprietor income showed the Appalachian region 

lagging behind U.S. levels in 1997 and slipping further behind by 2005. 

Forty-two percent of Appalachian counties saw a decrease in income 

levels, primarily in the northern and central tiers, whereas 25% saw 

increases in nonfarm proprietor income levels higher than the national 

rate, the majority of which are located in the southern tier. 

 Trends in the impact of entrepreneurship on the local economy, as might 

be expected, generally echo the trends in nonfarm proprietor income, with 

similar levels in Appalachia and the U.S. in 1997, but with Appalachia 

substantially falling behind by 2005. This decline occurred in spite of the 

fact that almost all Appalachian counties experienced increases in total 

personal income. Only 15% of counties benefited from levels higher than 

the national rate in 2005, and these were evenly distributed across the 

region. 

A detailed presentation of the data underlying these observations follows. 

Entrepreneurship Employment 

As there are many ways to determine or interpret what constitutes 

entrepreneurship employment, two methods are presented here, each of which 

gives a slightly different picture of entrepreneurship employment in the region. 

Table 5.2 provides a summary of data for the U.S. and the 410 counties that 

comprise the Appalachian Region. 

As the table shows, trends in entrepreneurship employment in Appalachia are 

broadly similar to those for the nation as a whole. In 1997, nonfarm proprietor 

employment comprised 15.5% of total nonfarm employment in the U.S. and 
47

15.6% in Appalachia. By 2005, these values had increased to 18.2% in the U.S. 

and 18.0% in Appalachia. 

Table 5.2. Indicators of Entrepreneurship Employment 

INDICATOR / MEASURE 1997 VALUE 2005 VALUE

Nonfarm Proprietors Employment 

U.S. 

Appalachia 

23,648,200

1,744,246

31,147,600

2,202,141

Nonfarm Proprietors Employment/Total Nonfarm Employment 

U.S. 

Appalachia 

15.5%

15.6%

18.2%

18.0%

Number of Microenterprise Businesses 

U.S. 

Appalachia 

19,197,236

1,432,733

24,511,431

1,757,105

Estimate of Microenterprise Employment 

U.S. 

Appalachia 

24,833,677

1,845,049

31,690,476

2,178,389

Microenterprise Employment Estimate/Total Nonfarm 

Employment 

U.S. 

Appalachia 

16.3%

16.5%

17.9%

17.8%

Sources: U.S. Census Bureau Nonemployer Statistics and County Business Patterns; 

Bureau of Economic Analysis, Regional Economic Information System 

 

The following maps show nonfarm proprietors’ employment as a proportion of 

total nonfarm employment in the Appalachian region. In 1997, there were 39 

counties in Appalachia in which nonfarm proprietors accounted for 30 percent or 

more of total nonfarm employment (Figure 5.1); this number almost doubled to 

78 counties by 2005 (Figure 5.2). 

Figure 5.1 

Entrepreneurship Employment in 

Appalachia:

Nonfarm Proprietors as a Percent of Total 

Nonfarm Employment, 1997

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

7.96% to 19.99% (243)

20% to 29.99% (128)

30% to 60.15% (39)
48

Figure 5.2 

Entrepreneurship Employment in 

Appalachia:

Nonfarm Proprietors as a Percent of Total 

Nonfarm Employment, 2005

4.46% to 19.99% (187)

20% to 29.99% (145)

30% to 66.16% (78)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

Nonfarm proprietor employment increased 31.7% from 1997 to 2005 in the U.S. 

and 26.3% in Appalachia. Levels increased in all but 70 Appalachian counties 

from 1997 through 2005, and grew at a rate exceeding the national average in 

141 Appalachian counties – more than doubling in 21 counties (Figure 5.3). 

Figure 5.3

Entrepreneurship Employment 

Change in Appalachia:

Percent Change in Nonfarm Proprietor 

Employment, 1997-2005

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

Decrease or No Change (70)

Increase 0.1%-31.7% (199) 

Increase 31.8%-271.7% (141)
49

As Figure 5.3 shows, although the distribution of growth in entrepreneurship 

employment extends to all parts of the region, there is a concentration of 

counties with growth above the national average in eastern Tennessee, western 

North Carolina, northern Georgia and northern Alabama. 

Examining the number of microenterprise employees shows a similar picture of 

Appalachia tracking the national levels. In 1997, microenterprise employment as 

a percent of total nonfarm employment was 16.3% in the U.S. and 16.5% in 

Appalachia. By 2005, these levels had increased to 17.9% and 17.8% 

respectively. In Appalachia in 1997, there were 25 counties in which 

microenterprise employment accounted for 30% or more of total nonfarm 

employment (Figure 5.4); by 2005 there were 48 such counties (Figure 5.5). 

Figure 5.4 

Microenterprise Employment in 

Appalachia:

Microbusiness Employment* as a Percent of 

Total Nonfarm Employment, 1997

10.0% to 19.9% (215)

20% to 29.9% (170)

30% to 42.9% (25)

*Microbusiness employment equals the number of 

nonemployers plus the estimated number of employees in 

establishments of employment size class 1-4 (2.5 

employees per establishment). 

Data Source: U.S. Census Bureau County Business 

Patterns and Nonemployer Statistics; Bureau of 

Economic Analysis, Regional Economic Information 

System

Map Prepared by RUPRI

October 2007

From 1997 to 2005, the number of microenterprise businesses increased 27.7% 

in the U.S. and 22.6% in Appalachia. Microenterprise employment increased 

23.6% in the U.S. and 18.1% in Appalachia over this same period. Only 24 

counties in Appalachia had a decrease or no change in the number of 

microenterprise businesses (Figure 5.6), and only 44 counties saw a decrease or 

no change in microenterprise employment from 1997 to 2005 (Figure 5.7). 
50

Figure 5.5

Microenterprise Employment in 

Appalachia:

Microbusiness Employment* as a Percent of 

Total Nonfarm Employment, 2005

8.27% to 19.9% (196)

20% to 29.9% (166)

30% to 45.7% (48)

Data Source: U.S. Census Bureau County Business 

Patterns and Nonemployer Statistics; Bureau of 

Economic Analysis, Regional Economic Information 

System

*Microbusiness employment equals the number of 

nonemployers plus the estimated number of employees

in establishments of employment size class 1-4 (2.5

employees per establishment). 

Map Prepared by RUPRI

October 2007

Figure 5.6 

Microenterprise in Appalachia:

Percent Change in the Number of 

Microenterprise Businesses*, 1997-2005

*Microbusiness business are the number of 

nonemployers plus the number establishments of 

employment size class 1-4.

Data Source: U.S. Census Bureau County Business 

Patterns and Nonemployer Statistics; Bureau of 

Economic Analysis, Regional Economic Information 

System

Decrease or No Change (24)

Increase 0.1%-27.7% (272) 

Increase 27.8%-124.7% (114)

Map Prepared by RUPRI

October 2007
51

Figure 5.7 

Microenterprise in Appalachia:

Percent Change in the Number of 

Microenterprise Employees*, 1997-2005

*Microbusiness employment equals the number of 

nonemployers plus the estimated number of employees

in establishments of employment size class 1-4 (2.5

employees per establishment). 

Data Source: U.S. Census Bureau County Business 

Patterns and Nonemployer Statistics; Bureau of 

Economic Analysis, Regional Economic Information 

System

Decrease or No Change (44)

Increase 0.1%-23.7% (258) 

Increase 23.8%-118.8% (108)

Map Prepared by RUPRI

October 2007

Entrepreneurship Income 

A measure of entrepreneurship income is nonfarm proprietors’ income per 

nonfarm proprietor employment. Table 5.3 shows this indicator for 1997 and 

2005, along with other indicators of income and earnings as a basis for 

comparison. 

Table 5.3. Indicators of Entrepreneurship Income 

INDICATOR / MEASURE 1997 VALUE

(ADJUSTED 

TO $05) 

2005 VALUE

Nonfarm Proprietors’ Income / Nonfarm Proprietor 

Employment 

U.S. 

Appalachia 

Appalachia as a % of U.S. Value 

$27,881

$23,094

82.8%

$30,193

$23,218

76.9%

Per Capita Income 

U.S. 

Appalachia 

Appalachia as a % of U.S. Value 

$30,827

$25,687

83.3%

$34,471

$28,336

82.2%

Wage & Salary Disbursements / Wage & Salary Employment 

U.S. 

Appalachia 

Appalachia as a % of U.S. Value 

$36,331

$30,706

84.5%

$40,146

$32,779

81.6%

Source: Bureau of Economic Analysis, Regional Economic Information System 

On all measures of income, Appalachia lags somewhat behind the national 

figures and fell further behind over the period 1997-2005. In 1997, the three 
52

income indicators for Appalachia were between 82% and 85% of the U.S. value. 

By 2005, nonfarm proprietor income per nonfarm proprietor had fallen to 76.9% 

of the U.S. value and wage and salary disbursements per wage and salary job 

had slipped to 81.6% of the U.S. level. 

In 1997, nonfarm proprietor income per nonfarm proprietor (adjusted to 2005 

dollars) ranged from $5,263 to $54,117 in Appalachia (Figure 5.8). In 26 

counties in Appalachia, the value exceeded the U.S. value of $27,881 for this 

indicator. In 2005, nonfarm proprietor income per nonfarm proprietor in 

Appalachia ranged from $7,190 to $64,689, and 28 counties had values 

exceeding the U.S. figure of $30,193 (Figure 5.9).

Figure 5.8 

$5,263 - $16,572 (135)

$16,573 - $27,881 (249)

$27,882 - $54,117 (26)

Entrepreneurship Income in 

Appalachia:

Nonfarm Proprietors Income per Nonfarm 

Proprietor, 1997

(adjusted to 2005$)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

Total nonfarm proprietors’ income (real $2005) increased by 42.6% in the U.S. 

and 26.9% in Appalachia from 1997 to 2005 (Figure 5.10). Within Appalachia, 

total nonfarm proprietors’ income decreased in 172 counties, noticeably in New 

York, Pennsylvania, eastern Kentucky, western West Virginia and southwestern 

Virginia. Increases were experienced in 238 counties – 102 counties in 

Appalachia had increases in nonfarm proprietors’ income exceeding the national 

rate, and in 22 counties income more than doubled during this time period. The 

greatest gains were to be found in eastern West Virginia, eastern Tennessee, 

western North Carolina, and northern Alabama. 
53

Figure 5.9 

Entrepreneurship Income in 

Appalachia:

Nonfarm Proprietors Income per Nonfarm 

Proprietor, 2005

Data Source: Bureau of Economic Analysis, 

Regional Economic Information System

$7,189 - $18,641 (284)

$18,642 – $30,193 (98)

$30,194 - $64,686 (28)

Map Prepared by RUPRI

October 2007

Figure 5.10 

Entrepreneurship Income in 

Appalachia:

Percent Change in Total Nonfarm 

Proprietors’ Income, 1997-2005

(1997 adjusted to 2005$)

Decrease (172)

Increase 0.1%-42.6% (136) 

Increase 42.7%-188.7% (102)

Data Source: Bureau of Economic Analysis, 

Regional Economic Information System

Map Prepared by RUPRI

October 2007
54

 

Entrepreneurship’s Contribution to the Local Economy 

One method of assessing the importance of entrepreneurship to the local 

economy is to examine the share of nonfarm proprietors’ income to total county 

personal income in 1997 and 2005. Table 5.4 summarizes this indicator for the 

U.S. and Appalachia. 

Table 5.4. Indicators of Entrepreneurial Contribution 

INDICATOR/MEASURE 1997 VALUE 2005 VALUE

Nonfarm Proprietor Income/Total Personal Income 

U.S. 

Appalachia 

7.8%

7.0%

9.2%

7.6%

As Table 5.4 shows, Appalachia has fallen substantially behind the U.S. on this 

measure of entrepreneurial contribution. Within Appalachia in 1997, 90 counties 

had shares greater than the national average of 7.8% (Figure 5.11), but by 2005, 

there were only 63 counties in Appalachia with a share greater than the national 

average of 9.2% (Figure 5.12). From 1997 to 2005, the nonfarm proprietors’ 

income share of total county personal income increased in 170 Appalachian 

counties, and decreased in 240 Appalachian counties (Figure 5.13). 

Before drawing any conclusions about these increasing or decreasing shares, it 

is first necessary to look at the trends in actual nonfarm proprietors’ income and 

total personal income. In some cases, proprietors’ income share may be 

increasing as a share because total county income is on the decline, resulting in 

a higher share even if actual nonfarm proprietors’ income did not rise. The map 

in Figure 5.14 compares nonfarm proprietors’ income change to total county 

personal income change. 

In 238 Appalachian counties, nonfarm proprietors’ income (in real $2005) 

increased from 1997 to 2005, and total county personal income decreased in 

only 4 of these counties. Nonfarm proprietors’ income decreased in 172 

Appalachian counties; in 15 of these counties, total county personal income also 

decreased. The conclusion to be drawn here is that whether or not counties 

have an increasing or decreasing share of nonfarm proprietor income does not 

generally appear to be impacted by increases or decreases in total personal 

income. 
55

Figure 5.11 

Entrepreneurship and the Local 

Economy:

Nonfarm Proprietors’ Income as a Share of 

Total Personal Income (TPI), 1997

At or below U.S. Share (320)

Above U.S. Share (90)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

U.S. Share = 7.84%

Figure 5.12 

Entrepreneurship and the Local 

Economy:

Nonfarm Proprietors’ Income as a Share of 

Total Personal Income (TPI), 2005

At or below U.S. Share (347)

Above U.S. Share (63)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

U.S. Share = 9.20%
56

Figure 5.13 

Entrepreneurship and the Local 

Economy:

Change in the Share of Nonfarm 

Proprietors’ Income to Total Personal 

Income (TPI), 1997-2005

Share Increased (170)

Share Decreased (240)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007

EI PERFORMANCE WITHIN THE REGIONAL CONTEXT 

What do these findings suggest about the impact of ARC’s EI on the region? 

While it is difficult to draw any causal connections between ARC investments and 

these regional data, several trends are evident. Entrepreneurship, when defined 

broadly as nonfarm proprietor and microenterprise employment, is an 

increasingly important component of the Appalachian regional economy. For 

some parts of the region – the southern tier in particular – rates of employment 

associated with entrepreneurship are greater than for the country as a whole. 

This observation confirms the importance of investments in entrepreneurship 

development as a key piece of a regional economic development strategy. In 

essence, entrepreneurship represents an asset upon which the region can build. 

However, in terms of income associated with entrepreneurship, the picture 

unfortunately mirrors the past – a region continuing to lag behind the rest of the 

nation in terms of nonfarm proprietors’ income. This observation suggests that in 

spite of ARC’s investment, the income generated through entrepreneurial activity 

is not growing in such a way as to allow the region to catch up with the rest of the 

country. This analysis also suggests that ARC’s investments have had an impact 

on getting more entrepreneurs into the pipeline, but have had a lesser impact on 

creating stronger and higher growth entrepreneurs, at least as measured by 

income trends. To better understand the reason behind this disparity, a more 

detailed investigation of entrepreneurial income and wage and salary levels 

associated with entrepreneurship in the region should be undertaken. It was not 

possible in the current evaluation to gain insights into the income and wealth 

creating potential of entrepreneurial ventures or into the quality of jobs created 

for others by these entrepreneurs. 
57

Have ARC investments contributed to creating a more entrepreneurial region? It 

is difficult to say from analysis of secondary data. Entrepreneurial activity has 

increased over the period of investment and some parts of the region have 

performed even better than the nation. It is possible that the Appalachian region 

would have fallen even further behind the nation in terms of income from 

entrepreneurship without EI investments. Insights from more detailed analysis of 

sample projects and key stakeholders help answer this question in a way that 

secondary data analysis cannot.

Figure 5.14 

Entrepreneurship and the Local 

Economy:

Change in Nonfarm Proprietors’ Income and 

Total Personal Income (TPI), 1997-2005 

(Values adjusted to $05)

TPI Increased (234)

TPI Decreased (4)

Nonfarm Proprietors Income Increased

Nonfarm Proprietors Income Decreased

TPI Increased (157)

TPI Decreased (15)

Data Source: Bureau of Economic Analysis, Regional 

Economic Information System

Map Prepared by RUPRI

October 2007
58

CHAPTER 6 

ARC INVESTMENT PORTFOLIO – 

PROJECT IMPLEMENTATION AND REPORTED METRICS 

 

Because ARC is built around a federal-state-local partnership, few of its 

programs operate according to a single cookie-cutter model. Instead, each state 

and each local program tend to take a distinctive approach to ARC funding 

programs and opportunities. Differences in program management from state to 

state and differences in the types of projects funded make generalizing impacts 

across the EI portfolio difficult. This chapter is designed to provide insights into 

project implementation so that later discussions of lessons learned and 

recommendations are understood within this overall context.78

ARC required the reporting of several metrics for all project investments, 

including job creation and retention, business creation, businesses served, and 

private investment leveraged. Data from the final project reports submitted to 

ARC by all grant recipients provide useful insights into the size and scope of the 

initiative, as well as these specific, but limited, impacts of project investments. 

Deeper understanding of project impacts and the broader policy implications of 

ARC investments gained through analysis of the sample projects is presented in 

Chapter 7. 

PROJECT IMPLEMENTATION 

As with most ARC initiatives, the approach to using the EI funds varied widely 

across the states in the region. To better understand these differences, and what 

they might mean in terms of program outcomes, the evaluation team conducted 

phone interviews with state program managers and with the director of the 

Entrepreneurship Initiative in Washington, D.C. The purpose of these interviews 

was to develop a sense of how each state approached the EI and to get a view of 

the initiative from the state level. 

Each governor and state program manager approached the Entrepreneurship 

Initiative with a different level of commitment to entrepreneurship as an economic 

development strategy and a different strategy for distributing what in many cases 

was described as a relatively small resource pool. Table 6.1 describes the 

strategy or approach used to distribute funds in each state while Table 6.2 shows 

the distribution of investments across the region over the 1997 to 2005 period. In 

one state, Pennsylvania, a large portion of resources was used to support a 

specific intervention – the creation of entrepreneurial networks. In other states, 

local development districts were responsible for promoting the initiative and 

encouraging local organizations to submit grant proposals. In most cases, states 

 

78 Insights and observations presented in the first section of this chapter are drawn from 

interviews with regional stakeholders and from project leaders. The evaluation team reported 

insights that were widely held and recurring across the range of interviews conducted. 
59

Table 6.1. Description of State Approaches to EI Investments* 

STATE APPROACH

Alabama Entrepreneurship included in Appalachian Development Plan; provided impetus 

for the state to support diverse homegrown initiatives, including entrepreneurship 

education and incubators 

Georgia State program manager’s office identified local and regional projects to be 

funded; relied less on Local Development Districts (LDDs) to take the lead on 

developing projects 

Kentucky State created a government-sponsored commission whose chair was the state 

alternate; Chair lived in region and selected both local and regional projects; also 

funded other state organizations to expand activities into region; identified 

technical assistance gap and tried to select projects to fill it 

Maryland Initially tried to start new programs in the region, providing several years of 

funding; partnered with WV on projects; efforts were not able to create 

sustainable new institutions; later stages, plugged EI into ongoing activities in the 

counties, like Main Street program 

Mississippi State program manager and alternate active in identifying projects; tried to work 

with local philanthropy organizations 

New York Worked at the state level to fund some large programs to support 

entrepreneurship education – some local and some regional; supported some 

other local initiatives 

North 

Carolina 

Rolled money over until state had a pool of money; contracted with state 

Department of Commerce for multi-year program; efforts were not able to create 

sustainability; locally driven projects were also funded, with some more success 

Ohio Leadership came from Governor’s Office of Appalachia; active in identifying 

projects with both Local Development Districts (LDDs) and nonprofits; joint 

funding of projects with the Foundation for Appalachian Ohio; open process for 

soliciting proposals; multi-year funding of regional institutions and one time 

projects 

Pennsylvania Most of funds were used to support local networking activities; also put money 

into region-wide youth entrepreneurship program; state program manager and 

alternate worked out of the Office of Appalachian Development in state 

Department of Community and Economic Development 

South 

Carolina 

Most projects identified by state program managers; relied less on Local 

Development Districts (LDDs) 

Tennessee Worked with some Local Development Districts (LDDs); major investment made 

in the development of one organization, leading to the creation of a sustainable 

organization with a broad reach 

Virginia High priority for the state; spread grants around to as many communities as 

possible; saw EI funds as being part of a set of resources, including ARC 

telecommunications and Main Street programs, that communities could tap; used 

a competitive RFP process with an open workshop planning process leading to a 

diverse set of project proposals; funded multi-year projects; drew in other funding 

resources, particularly from foundations 

West Virginia State highlighted entrepreneurship funds as one of many tools available; nonprofit organizations stepped up and provided most of the impetus for developing 

and implementing grant projects; used a competitive RFP process with an open 

workshop planning process leading to a diverse set of project proposals; funded 

multi-year projects; drew in other funding resources, particularly other state funds 

*Interviews were attempted but not completed with state program managers in Mississippi and 

Ohio due to illness and staff turnover, respectively, at the time the interviews were undertaken. 

Information about their approach to the EI was obtained through an interview with the director of 

the EI. 
60

used their annual allocations to fund a range of projects; in one case, North 

Carolina, funds were rolled over until the state had a pool of resources to support 

a larger project. 

Several themes were identified through the program manager interviews. From 

the perspective of local communities, ARC Entrepreneurship Initiative resources 

were described as “catalytic” and important to demonstrating new approaches to 

entrepreneurship development. In many cases, these projects would not have 

gotten started without ARC investment. As one manager described it, these were 

“but for” resources. At the same time, program managers admitted that the 

relatively small pool of resources attached to the initiative made it harder to make 

entrepreneurship investments a priority, particularly if local development districts 

were not focusing in this area and encouraging proposals from their regions. 

It was also clear that the state leaders, including governors and program 

managers, played a “gate keeping” role in terms of ARC initiatives, including 

entrepreneurship. Strong support for entrepreneurship from state leaders often 

translated into more aggressive promotion of the program within the state, such 

as was seen in Virginia. In addition, since the innovation in entrepreneurship 

programs was primarily coming from the local rather than the state level, the 

degree to which the state program manager was networked with local economic 

development organizations, especially non-profits, could impact the ability to 

effectively implement the initiative. Much of the investment in West Virginia, for 

example, was driven by community non-profit organizations. 

In addition to the diverse approaches used by the states in allocating resources, 

ARC investments were spread across a portfolio of program categories. Prior to 

selecting the evaluation sample, Entrepreneurship Initiative projects were 

classified into program categories based on a review of project descriptions, 

including the goals of each project, and follow-up interviews with project 

leadership. Project goals differed even within program categories. Metrics used 

to report on project outcomes varied by program category, and it was clear from 

the interviews with project leaders that job creation and business creation were 

not the only metrics considered in articulating project goals and reporting 

outcome measures. 

To put the findings of this evaluation into better perspective, a description of the 

range of projects found within these program types is provided here. These 

descriptions should provide readers with a sense of the diversity within each 

project type and the breadth of investments undertaken by ARC in support of 

entrepreneurship development in the region. 

Entrepreneurship Education Projects 

Entrepreneurship education projects were designed to introduce 

entrepreneurship concepts and curricula into the schools, kindergarten through 
61

community college and university. In the language of the programmatic goals set 

out in Chapter 3, these programs sought to help create more entrepreneurs in 

the pipeline, and were targeted primarily to students as opposed to adult 

learners. (Programs for adult learners were included in the technical assistance 

and training category.) A profile of the sample entrepreneurship education 

projects appears below. 

Sample Entrepreneurship Education Projects

Number of projects = 17 

Total ARC investment = $2,199,207 

Average ARC investment = $129,365 

Minimum ARC investment = $10,000 

Maximum ARC investment = $784,517 

Range of years for project investment = 1 – 4 

The range of entrepreneurship education projects supported by the EI was quite 

diverse. For example, funds were used to: 

 Support outreach into rural communities for an existing business plan 

competition 

 Create an internship program and place youth in entrepreneurial 

companies 

 Organize summer youth entrepreneurship camps, using REAL (Rural 

Entrepreneurship through Action Learning) and other curriculum 

 Train teachers in the REAL curriculum and bring it into the classroom 

 Train middle school students using Junior Achievement, with follow-up 

training using a virtual business simulation model 

 Create student run, school-based enterprises. 

Some projects were designed as pilots to test an entrepreneurship education 

approach in a community or region. Others were designed to build 

entrepreneurship education capacity by training teachers and incorporating new 

curriculum into school systems, or by investing in the creation of school-based 

enterprises that could be passed down to future classes of students. About half 

the sample projects were defined by project leaders as being sustainable, i.e., 

they continued beyond the ARC grant with support from other funders or, in 

some cases, with an additional ARC grant. 

With the exception of those projects that created school-based enterprises, most 

were not designed to create or retain jobs or to build new businesses. Rather, 

they were designed to expose youth to entrepreneurship with an expectation that 

these entrepreneurial skills would serve to prepare them to become better 

employees in the future and/or to motivate them to create their own economic 

futures through enterprise development. Given their goals, traditional economic 

development metrics provide an incomplete story of the impact of these projects. 
62

Table 6.2. Annual and Total ARC Entrepreneurship Investments, 1997-2005, by State ($)* 

STATE 1997 1998 1999 2000 2001 2002 2003 2004 2005 TOTAL

Alabama 0 159,000 381,663 537,187 198,850 405,000 92,000 518,892 112,675 2,405,267Georgia 110,000 0 441,826 0 581,529 349,442 83,000 723,747 334,118 2,623,662Kentucky 92,769 226,000 787,500 290,532 510,500 225,362 657,880 383,794 587,198 3,761,535Maryland 0 207,000 455,200 548,918 312,379 139,000 99,000 130,000 45,000 1,936,497Mississippi 0 220,577 30,000 164,970 594,835 157,000 0 200,000 226,000 1,593,382New York 0 243,140 187,654 920,000 75,000 25,000 145,000 265,000 35,000 1,895,794North Carolina 0 0 0 1,050,000 0 444,908 200,000 157,500 673,673 2,526,081Ohio 0 153,940 406,213 451,921 613,700 244,500 1,673,330 15,000 50,000 3,608,604Pennsylvania 313,397 1,072,808 1,099,000 199,900 659,636 383,750 362,000 460,000 125,000 4,675,491South Carolina 0 500,000 0 750,000 119,105 752,000 500,000 891,382 58,750 3,571,237Tennessee 0 451,748 325,000 390,000 410,185 185,000 93,000 201,133 35,000 2,091,066

Virginia 374,500 0 495,380 416,527 476,630 433,000 181,200 280,391 90,000 2,747,628

West Virginia 142,150 752,948 1,075,916 300,537 1,673,748 1,042,835 582,000 458,400 1,105,474 7,134,008Regional 14,000 332,503 124,623 264,901 169,162 339,610 973,574 108,063 75,000 2,401,436Total ARC 1,046,816 4,319,664 5,809,975 6,285,393 6,395,259 5,126,407 5,641,984 4,793,302 3,552,888 42,971,688

*

Figures include all projects (closed and open) and investments from all funding sources within ARC.
63

Several examples help to demonstrate the range of entrepreneurship education 

projects funded by ARC and the type of outcomes they achieved. 

In 1999, the Kentucky Science and Technology Council (KSTC) received funding 

from ARC to establish next generation entrepreneurial schools in two pilot 

districts. The goal of the project was to create learning communities where 

students could embrace an entrepreneurial frame of mind and, in turn, to help all 

students generate ideas for and start new ventures in their home communities. 

Using initial and follow-on ARC grants, KSTC supported the creation of 99 

EdVentures – school-based enterprises – including 31 in high schools, 25 in 

middle schools, 37 in elementary schools, and 6 district-wide. KSTC has 

documented the outcomes of these activities in two ways – through metrics 

including the number of school-based enterprises started (99) and the number of 

participants in these ventures (27,000) as well as through stories of each 

individual EdVenture that describe the start-up process, funds raised, students 

involved, etc. 

In West Virginia, ARC funds were used to bring an entrepreneurship focus to an 

existing summer camp program, using the REAL curriculum. The Ohio-West 

Virginia YMCA provided leadership for this initiative which secured four years of 

ARC funding. In addition to serving the 465 student participants, the initiative has 

generated additional interest in youth entrepreneurship in the state, attracting 

funding to make the program sustainable and engaging youth in several ways. 

The statewide economic development planning process, Vision Shared, 

partnered with the Ohio-West Virginia YMCA to provide a teen perspective on the 

future of the state. The YMCA held a Teen Forum on West Virginia’s Future in 

2002 to develop this perspective. The Department of Education created a fulltime entrepreneurship coordinator position and hired a certified REAL instructor 

as the coordinator. 

ACEnet in Ohio used ARC funding to extend into three high schools a computer 

opportunities program designed to provide training to students in setting up 

computer-based enterprises. The program created three new businesses and 14 

new jobs. In addition, however, it expanded the computer technology available in 

these schools and exposed students to entrepreneurship concepts. Another 

outcome was the attraction of additional federal support to continue the program 

beyond ARC funding. 

Technical Assistance and Training Projects 

Technical assistance and training projects were designed to build the skill sets of 

individual entrepreneurs by offering them access to one-on-one counseling and 

workshops related to starting and growing their businesses. The programmatic 

goals of these programs were to create more entrepreneurs in the pipeline, and 

better informed and skilled entrepreneurs whose businesses were more 

productive and created more jobs in the region. Most of these programs were 
64

targeted at adult entrepreneurs as distinguished from the entrepreneurship 

education projects that primarily targeted youth. A profile of sample technical 

assistance and training projects appears below. 

Sample Technical Assistance and Training Projects

Number of projects = 34 

Total ARC investment = $4,933,997 

Average ARC investment = $145,118 

Minimum ARC investment = $10,000 

Maximum ARC investment = $2,088,961 

Range of years for project investment = 1 – 6 

One large grant to Pennsylvania to support a statewide technical 

assistance/networking initiative skews average investment. If the 

Pennsylvania project is removed from the sample, the range of 

investments is $10,000 to $650,615 and average investment 

per technical assistance project is $86,213. 

As with education projects, the range of technical assistance projects was quite 

wide. Investments were used to fund such activities as: 

 Grants to individual entrepreneurs to purchase private sector services, 

such as accounting or legal assistance 

 Funding for “how to start a business” and other workshops for 

entrepreneurs and potential entrepreneurs 

 Providing technical assistance in conjunction with a microenterprise 

program 

 Developing export assistance programs for entrepreneurs 

 Offering one-on-one assistance to entrepreneurs in accounting, marketing, 

and other business management areas 

 Delivering marketing assistance to clients of a food incubator 

 Developing computer-based entrepreneurship training programs 

 Providing managerial and technical assistance to high-tech start ups 

 Creating a virtual business assistance center. 

In general, these projects provided entrepreneurs with access to business 

support resources, both customized and in workshop settings. Often, these 

services represented additions or expansions to other programs offered by the 

local organizations. However, in some cases, these investments were used to 

create new capacity in the region, such as through the creation of a virtual 

assistance center. About three quarters of projects were defined as sustainable 

by project leaders. Several examples provide insights into these technical 

assistance and training projects. 

Kentucky Highlands Investment Corporation (KHIC) received funding from ARC 

for its business support center, The Launching Pad. These funds were used to 

provide technical assistance to entrepreneurs, some of whom were customers 
65

(or future customers) who had received capital through KHIC’s range of financial 

instruments. The Launching Pad provided a place for entrepreneurs to access 

computers and other technology, and to obtain services from KHIC’s skilled staff. 

This assistance served to formalize and build the capacity of KHIC to support the 

non-financial needs of its business customers. Previous assessment of KHIC’s 

programs found that this technical assistance or managerial support was as 

important to entrepreneurs as the capital that KHIC provided.79

The Ohio Valley Regional Development Commission used ARC funds to provide 

small grants (up to $5,000) for entrepreneurs to purchase business services from 

private sector providers. This program was designed in response to the 

expressed needs of the Commission’s revolving loan customers who sought 

assistance early in the life of their businesses, before they had the sales and 

resources to pay for services. The project was successful in meeting the needs 

of these entrepreneurs and connecting them to appropriate service providers. 

The Commission was able to help almost twice as many entrepreneurs as initially 

projected because entrepreneurs often required very small grants to get the help 

they needed. 

Mississippi State University received support to develop training programs to 

make entrepreneurs aware of the power of marketing via the Internet. In 1998, 

when the ARC investment was made, the concept and value of e-commerce was 

not universally recognized. Mississippi State’s programs exposed home-based 

business owners, farmers and youth to Internet marketing concepts. While the 

program helped individual business owners, it also launched regional and 

national efforts to develop e-commerce as an effective business strategy. The 

curriculum developed at Mississippi State is used throughout the country and a 

national e-commerce education effort through Cooperative Extension continues 

to support education and training in this area. 

Sector Projects 

Projects funded in the sectors category were focused on supporting 

entrepreneurs in a particular sector, such as food, wood products, ceramics, 

often by building networks among entrepreneurs. In terms of programmatic 

goals, these projects were designed to create more informed and skilled 

entrepreneurs, operating more productive businesses that create jobs in the 

region. The projects included in the sample, however, were as diverse as the 

sectors that make up the economy of the Appalachian region, including: 

 Creation of a consignment gift shop to support local artisans 

 Development of an aquaculture program including entrepreneurship 

training 

 

79 David Barkley and Deborah Markley, The Development of an Entrepreneurial Support 

Organization: The Case of the Kentucky Highlands Investment Corporation, RUPRI Center for 

Rural Entrepreneurship, Research Case Study Series Number 1, March 2003. 
66

 Support for a ceramics corridor cluster in collaboration with a university 

 Operational support for the Ralph Stanley museum and music center 

 Development of a driving tour to cultural and heritage attractions and 

artisan businesses 

 Funding for a market study of the cut flowers industry in Mississippi 

 Support for workshops on development of a craft woodworking industry. 

A profile of sample sector projects appears below. Less than half of project 

leaders defined their efforts as being sustainable over time. Several examples 

serve to illustrate the high degree of diversity in these sector projects. 

Sample Sector Projects

Number of projects = 17 

Total ARC investment = $1,496,585 

Average ARC investment = $90,976 

Minimum ARC investment = $1,400 

Maximum ARC investment = $393,711 

Range of years for project investment = 1 – 4

The Kentucky Artisan Trails Project was designed to create a gateway attraction 

on I-75 that would promote a driving tour of regional cultural and heritage sites as 

well as artisan businesses along the route. Through development of an 

information kiosk on the interstate, a website and hard copy maps, the project 

helped to encourage tourism development by networking individual sites into a 

destination tour. The website features products produced by regional 

entrepreneurs, places for tourists to visit, and events that are planned throughout 

the region. The project focused on the heritage tourism sector, including 

entrepreneurs, and developed a network of people, places and organizations that 

worked to build the sector in the region. 

Advantage West, a regional economic development organization in western 

North Carolina, used ARC investment to help introduce and develop a biotech 

sector in the region. There was much interest in biotech in the region, but limited 

staff capacity to move the concept forward. ARC funds were used to hire a 

biotech coordinator and to develop wet lab space at the community college – the 

first in the region. Advantage West also established a steering committee to 

consider strategic opportunities for advancing the sector. These initial 

investments have helped create a biotech sector focused on the natural products 

available in the region. The region has attracted additional local government and 

regional university investments to continue to build this sector. 

Appalachian by Design, a non-profit rural economic development organization, 

wanted to help create a knitting industry in rural Appalachia by connecting female 

artisans with broader markets. ARC funding was used to help individual rural 

women develop a cottage knitting industry. The project was initially quite 
67

successful, producing high quality products and tapping new markets. Since its 

initial success, the project has scaled back as resources became constrained, 

forcing the organization to focus on a more limited market. 

Incubator Projects 

Incubators projects in the region were designed to provide a supportive 

environment for entrepreneurs to hatch and grow new enterprises. Incubator 

programmatic goals usually related to creating more informed and better skilled 

entrepreneurs, helping them create more productive businesses and ultimately 

encouraging more entrepreneurs to stay in the region. ARC investments in 

incubator projects, however, were generally of three types: 

 Investments to determine the feasibility of an incubator in a particular 

community or region 

 Investments to support the building and/or operation of an incubator 

facility 

 Investments to develop programs offered through the incubator. 

The sample of 12 incubator investments was equally split across these three 

types. A profile of sample incubator projects appears below. 

Sample Incubator Projects

Number of projects = 12 

Total ARC investment = $913,291 

Average ARC investment = $76,108 

Minimum ARC investment = $10,000 

Maximum ARC investment = $388,084 

Range of years for project investment = .5 – 1 

Projects tended to be much more short term than the other program types, 

ranging from six months to one year. Nine of the 12 incubator projects were 

defined as being sustainable. Examples of incubator projects follow. 

ARC funds were used to enhance the services provided by the Clinch Powell 

Community Kitchen in east Tennessee. While the community kitchen already 

provided shared space for food entrepreneurs in the area to develop and 

produce their goods, ARC funds allowed the incubator to purchase new 

equipment and expand the training support offered to entrepreneurs, particularly 

in the areas of product development, marketing and general business training. 

Incubator staff was able to devote more time to exploring marketing channels 

that would add value to products, for example, by combining gift baskets with 

specialty food products produced in the region. The incubator was successful in 

expanding the number of clients served and was able to leverage additional 

funds to become sustainable. 
68

In western North Carolina, ARC investment was used to create Blue Ridge Food 

Ventures, a kitchen incubator set up to help farmers and others develop valueadded agricultural products. ARC funding provided most of their operating 

budget, with other income coming from users of the facility. This relatively new 

incubator (at the time of the interview) developed its capacity to serve its clients 

by partnering with service providers in the area, such as a microenterprise 

development organization, to become a “one stop shop.” The incubator staff 

spent most of their time working with clients on product development, specifically 

how to put a new twist on a food product. 

The city of Gadsden, Alabama received ARC funds to conduct an incubator 

feasibility study. The city lost a major employer, a steel mill, in 2000 – a 

devastating blow to the economy. The city’s community development director 

was looking for alternatives and the ARC funds provided an opportunity to 

determine whether it would be feasible for the city to develop an incubator. The 

study concluded that an incubator was feasible and that there were residents in 

the community who had an interest in starting their own businesses. What the 

study did, according to the development director, was to give the city information 

and options – they could begin to think about what kind of incubator facility might 

be most appropriate, where it would be located and what funding they would 

need to develop the incubator. The attitude in the city changed when they saw 

the possibilities associated with an incubator strategy. 

Capital Access Projects 

Projects in this category, in general, were designed to enhance access to capital 

for entrepreneurs in the region. The programmatic goals were to create more 

productive and job creating businesses and entrepreneurs who could remain in 

the region. Based on ARC investment in this area, access to capital was 

considered to be critical to helping entrepreneurs start and grow businesses in 

the region. In addition to the EI investments in capital projects, ARC invested an 

additional almost $4 million from area development, regional initiatives, 

commission and co-chair funds. However, the projects were as diverse as the 

sources of capital. A profile of capital access projects appears below. 

Sample Capital Access Projects

Number of projects = 8 

Total ARC investment = $901,340 

Average ARC investment = $112,668 

Minimum ARC investment = $30,000 

Maximum ARC investment = $447,440 

Range of years for project investment = 1 – 6 
69

The geographic regions served by these projects ranged from individual 

communities to multi-county and even multi-state service areas. While some 

projects were one-time investments by ARC, others represented multi-year 

commitments, with additional investments phased in as the project progressed. 

All of the capital projects in the sample were judged to be sustainable by project 

leaders, i.e., the projects continued beyond the period of the ARC grant. 

ARC investments were used in diverse ways, including: 

 Support for research and planning to explore the creation of alternative 

financial institutions, such as New Market Venture funds, Community 

Development Financial Institutions (CDFIs), and angel capital networks 

 Support for technical assistance and outreach associated with revolving 

loan funds 

 Investment in capital programs ranging from revolving loan funds and 

microenterprise programs to equity funds. 

Several examples serve to illustrate the ways in which many of these capital 

projects were supported by ARC. Appalachian Community Enterprises (ACE), a 

non-profit microenterprise program started in northeast Georgia in 1997, received 

initial funding from ARC in 1999 to support a loan fund, including funds for 

technical assistance and training. Similar follow-on grants were made to ACE in 

subsequent years. Building on this base of activity and their track record of 

lending, in 2004, ACE received CDFI certification and significant funding from 

ARC’s Area Development program for their microenterprise loan program. 

In 1998, the South Carolina Appalachian Council of Governments received a 

modest initial investment from the EI to support the development of a loan fund 

and technical assistance and outreach to rural entrepreneurs. Complementing 

this EI investment was significant funding in 1998 under ARC’s Regional 

Initiatives to recapitalize the loan fund. Additional support for the revolving loan 

fund through ARC’s Area Development program was received in 2004. 

In 2003, Advantage West, a regional economic development organization in 

western North Carolina, received funding from ARC’s EI to develop a regional 

entrepreneurship network. Funding was used to put on workshops throughout the 

region, to develop a website, and to complete a market assessment for a 

regional investment fund. During this time, Advantage West assisted a number of 

entrepreneurs in acquiring capital from angel investors. A second grant was used 

to develop a regional angel investor network that continues to operate in the 

region. 

REPORTED METRICS 

From its inception in 1997 through 2005, ARC made almost $43 million in 

entrepreneurship-related investments in the region, including investments made 
70

specifically by the Entrepreneurship Initiative. As Table 6.2 shows, annual 

investments increased from inception in 1997 to a peak in 2001, with gradual 

reductions in annual investments since then. This pattern of investment 

represents the phasing in and out of the EI and, in more recent years, the 

inclusion of entrepreneurship investments in ARC’s Asset-Based Initiative. Total 

investments were made from a number of sources within ARC, with the most 

significant investment coming through the EI (Table 6.3). Total ARC investment, 

in turn, leveraged an additional $72.8 million in private investment (for those 

projects that have closed) with projected total private leveraged investment of 

$109.9 million once all projects in the portfolio have closed. 

Table 6.3. ARC Investments from All Sources – Universe of Projects, 1997-

2005 

SOURCE OF FUNDS ARC $

Entrepreneurship Initiative $22,519,996

Area Development 11,603,420

Distressed Counties 3,722,992

Commission EI 2,126,380

CoChair Fund 1,553,887

Regional Initiatives 1,368,439

New Markets 52,574

Goal Fund 24,000

Total $42,971,688

 

ARC investments were made in 340 unique projects across the region (Table 

6.4). The distribution of both dollars and projects varied across the region, 

ranging from a high of 48 projects funded in Virginia to a low of nine projects 

funded in South Carolina. On average, investment per state was $3.3 million and 

investment per capita was $1.82 from 1997 through 2005. 

In addition to projects funded in individual states, ARC made investments in a 

number of projects that had a region-wide focus. For example, regional projects 

included support for: 

 A microenterprise conference 

 The development of regional community development venture capital 

funds, including support for technical assistance 

 A survey of Appalachian business incubators 

 The creation of entrepreneurship education materials, including working 

with REAL and Junior Achievement 
71

 A regional conference on entrepreneurship education and training 

 A regional technology commercialization initiative

 The development of sector-based entrepreneurship initiatives. 

Table 6.4. Distribution of Entrepreneurship Projects Funded from All ARC 

Sources, 1997-2005* 

STATE # CLOSED PROJECTS # OPEN PROJECTS # TOTAL 

PROJECTS 

Alabama 23 5 28 

Georgia 7 3 10 

Kentucky 22 9 31 

Maryland 11 3 14 

Mississippi 15 2 17 

New York 17 4 21 

North Carolina 5 6 11 

Ohio 37 3 40 

Pennsylvania 15 6 21 

South Carolina 4 5 9 

Tennessee 11 2 13 

Virginia 42 6 48 

West Virginia 24 13 37 

Region 30 10 40 

Total 263 77 340 

*These totals include unique projects only. 

 

Table 6.5 shows total and average investment in the universe of ARC projects 

(both open and closed projects) as well as in the evaluation sample. Average 

investment per project for the universe was $126,387, and projects ranged in 

size from $2,000 to almost $2.2 million.80 ARC invested almost $13 million in the 

specific projects included in the evaluation sample, with average investment per 

project of $145,997. Sample projects were drawn from those projects that were 

primarily funded with Entrepreneurship Initiative dollars, as opposed to 

 

80 The $2.2 million project investment was made in Pennsylvania, where the state chose to invest 

in building regional assistance networks across the state. While this project was treated as a 

single unique project, the funds flowed to regions across the state. 
72

Commission EI or CoChair funds. As a result, some of the smaller projects, such 

as Springboard Awards for Youth Entrepreneurship, were excluded from sample 

selection providing a slight upward bias on project size. However, the range of 

projects shows that the sample includes both small and large projects. 

Table 6.5. Dollars Invested in Entrepreneurship Initiative Projects, from all 

sources – Universe and Sample 

ARC $

Universe of Projects 

Total 42,971,688

Mean 126,387

Min 2,000

Max 2,177,326

Sample Projects 

Total 12,847,733

Mean 145,997

Min 10,000

Max 2,177,326

ARC collected outcome data as part of the final reporting requirements for each 

project. Table 6.6 shows actual and projected jobs created, jobs retained, new 

businesses created, and businesses served for the universe of unique projects, 

as reported in the close out documents submitted to ARC and included in the 

database provided by ARC. In total, 9,156 jobs were created, 3,022 jobs 

retained, 1,787 new businesses created, and 8,242 businesses served across 

the region between 1997 and 2005. On average, projects created almost 27 

jobs, retained almost 9 jobs, created 5 new businesses, and served 24 

businesses. The 9,156 jobs were created at a cost, in terms of ARC funds 

invested, of $4,693. As discussed in more detail in Chapter 7, this figure 

compares favorably to other economic development efforts. 

Table 6.7 shows these same metrics for those projects included in the evaluation 

sample. Collectively, sample projects created 4,332 jobs, retained 1,351, created 

1,083 new businesses, and served 2,957 businesses. On average, sample 

projects achieved greater outcomes than the universe of projects in terms of job 

creation (49), job retention (15), new business creation (12), and businesses 

served (almost 34). This variation can be explained by two factors. First, the 

universe of projects includes some of the smaller projects funded from ARC 

sources other than the EI, likely reducing the overall impact numbers for the 
73

universe. Second, the evaluation team was able to capture through follow-up 

interviews those ongoing impacts achieved by projects that were sustainable 

beyond the ARC investments. These job and business creation numbers, 

therefore, represent a more accurate view of the impact of ARC investment than 

those developed strictly from close out reports submitted to ARC. 

CONCLUSIONS ABOUT OVERALL EI PORTFOLIO PERFORMANCE 

The outcome data described in this chapter suggest that ARC investments have 

been successful in generating jobs and businesses within the region. Actual jobs 

created and retained and number of businesses served exceeded the projections 

or goals established by the projects in their funding applications to ARC for the 

sample of projects; only new business creation numbers fell short of 

projections.81 Sample data provide the most accurate view of the impact of ARC 

investments since they reflect ongoing impacts associated with the projects. 

Even considering all closed projects (where data are reported at project close out 

only), ARC investments have created/retained more jobs and served more 

businesses than projected.

 

81 It is not possible within the scope of this evaluation to determine whether the initial business 

creation goals for the universe of projects were, in fact, realistic. It is possible that projects 

identified business creation as a goal because it was a reporting metric for ARC, and not because 

it was a realistic outcome of program investment. A deeper understanding of specific projects 

would be required to address this issue. However, insights based on our analysis of the sample 

projects are provided in Chapter 7. 
74

Table 6.6 Job Creation and Retention, New Business Creation, and Businesses Served for the Universe of Projects* 

JOBS 

CREATED 

ACTUAL 

JOBS 

CREATED 

 PROJECTED

JOBS 

RETAINED 

ACTUAL 

JOBS 

RETAINED 

PROJECTED

NEW 

BUSINESSES 

CREATED 

ACTUAL 

NEW 

BUSINESSES 

CREATED 

PROJECTED 

BUSINESSES 

SERVED 

ACTUAL 

BUSINESSES 

SERVED 

PROJECTED 

Universe (340) Total 9,156 16,196 3,022 2,385 1,787 3,207 8,242 12,025Mean 26.9 47.6 8.9 7.0 5.3 9.4 24.2 35.4Min 0 0 0 0 0 0 0 0Max 528 1,000 430 372 264 1,000 2,717 2,500*These data are drawn from the ARC database and do not reflect additional job creation/retention, business creation and businesses served that 

may have occurred as projects continued beyond the period of the ARC grant. 

Table 6.7. Job Creation and Retention, New Business Creation, and Businesses Served for the Sample of Projects* 

JOBS 

CREATED 

ACTUAL 

JOBS 

CREATED 

 PROJECTED

JOBS 

RETAINED 

ACTUAL 

JOBS 

RETAINED 

PROJECTED

NEW 

BUSINESSES 

CREATED 

ACTUAL 

NEW 

BUSINESSES 

CREATED 

PROJECTED 

BUSINESSES 

SERVED 

ACTUAL 

BUSINESSES 

SERVED 

PROJECTED 

Sample (88) Total 4,332 2,937 1,351 443 1,083 981 2,957 2,757Mean 49.2 33.4 15.4 5.0 12.3 11.2 33.6 31.3Min 0 0 0 0 0 0 0 0Max 758 4 344 100 316 251 643 695*These data are drawn from follow-up interviews with project leaders and do reflect additional job creation/retention, business creation and 

businesses served that occurred as projects continued beyond the period of the ARC grant. 
75

CHAPTER 7 

SAMPLE PROJECT OUTCOMES AND 

BROADER POLICY IMPACTS 

Chapter 6 reported on select metrics for the entire portfolio of projects in which 

ARC invested; these data describe one level of impact the EI had on the region. 

However, entrepreneurship development strategies, including those funded by 

ARC, are often designed to achieve goals that extend beyond job and business 

creation. In Chapter 6, the evaluation team identified an underlying set of 

programmatic goals related to the conceptual model that were attributed to each 

project type. In reality, each project was designed to address a unique set of 

goals that were relevant to a particular rural place. To understand the full range 

of project impacts, and to assess whether these projects were achieving their 

goals, it was necessary to do a more in-depth evaluation of a sample of projects. 

This part of the evaluation of ARC’s EI has generated a series of findings that fall 

into two main categories – those related to the outcomes achieved by the 

representative sample of projects in which ARC invested and those related to the 

broader policy impacts on the Appalachian region associated with EI’s entire 

portfolio of investments. 

IMPACTS OF EI FUNDED PROJECTS 

These findings are based on detailed analysis of project outcomes associated 

with a sample of 88 projects that reflects the diversity, geographic reach, and 

scope of the EI. As described in Chapter 4, the sample was representative of the 

universe of projects receiving ARC investments between 1997 and 2005 in terms 

of both program type and state.82

Quantitative Impacts 

To understand the quantitative impacts of ARC investments, the evaluation team 

began by identifying three classes of metrics that were common to most projects 

within each program category (Table 7.1). One class included common goals that 

were articulated in project proposals but for which no outcome measures were 

provided by project leaders. These are metrics that were believed to be important 

during the design of the project but, for some reason, no outcome data were 

collected for most projects in the category. In some cases, it may have been a 

matter of definition – data collected as part of the project were not defined using 

the same terminology as project goals. For example, technical assistance 

providers often stated a goal of “number of businesses served” but actually 

 

82 While the sample was judged to be representative, the evaluation team is not suggesting that 

the quantitative results of this evaluation should be extrapolated to provide an estimate of overall 

quantitative impacts of the EI. Rather, outcomes associated with these specific projects should be 

viewed as reflective of what other projects might produce given similar capacity, assets, 

leadership, etc. 
76

reported “number of clients”. However, in some cases, it appears that data 

considered relevant when the project was designed were not collected. For 

example, entrepreneurship education providers often stated a goal of “number of 

jobs created” but did not report such data as an outcome measure. While these 

metrics do not provide insight into the overall performance of the EI, they were 

useful in considering the development of the “best in class” metrics system 

described in Chapter 9. 

Table 7.1. Metrics Common within Program Categories 

PROGRAM 

CATEGORY 

PROJECT 

METRICS 

INCLUDED AS 

STATED 

PROJECT GOALS 

ONLY 

PROJECT METRICS 

INCLUDED AS 

MEASURED 

PROJECT 

OUTCOMES ONLY 

PROJECT METRICS 

INCLUDED AS BOTH 

STATED PROJECT 

GOALS AND 

MEASURED PROJECT 

OUTCOMES 

Capital Access - Businesses 

served 

- Businesses 

created 

- Businesses 

created/expanded 

- Jobs created/retained 

Entrepreneurship 

Education 

- Jobs created - Number of 

participants/trainees (both 

students and teachers) 

- Number of business 

starts (student or adult) 

Sectors - Businesses 

served 

- Number of 

participants/trainees 

- Businesses 

created/expanded 

- Jobs created/retained 

Incubators - Businesses 

created 

- Businesses 

served 

- Number of current 

clients 

- Total clients served 

- Number of graduated 

firms 

- Jobs created/retained 

- Complete incubator 

feasibility study 

Technical 

Assistance and 

Training 

- Businesses 

served 

- Number of 

business plans 

created/assisted 

- Number of 

trainings, seminars, 

conferences 

- Businesses expanded 

- Number of clients 

- Number of clients 

retained in service area 

- Business created 

- Jobs created/retained 

A second class of metrics included measured and reported project outcomes that 

did not appear among the goals articulated for most projects in the category. 

These are metrics that likely appeared relevant only after a project was 

implemented or that were measured in a way that was different from the original 

project goals. For example, most capital projects reported the number of 

“businesses created and expanded” but had set a goal that related only to 

“businesses created”. And, incubator projects generally reported “graduated 

firms” as an outcome measure, but did not always include this metric as a project 

goal. These measures help describe the impact of ARC investments, but are not 

useful in evaluating the success of projects in achieving stated goals. 
77

The third class of metrics included those that were both stated project goals and 

measured and reported project outcomes. These metrics provide a means of 

measuring the “success” of the portfolio of projects included within each program 

category as will be described in more detail below.

Table 7.283 provides initial quantitative results for metrics that were common 

within program categories. These results provide insight into the range of 

measurable impacts associated with EI investments. Entrepreneurship education 

projects, targeted to youth, exposed 11,634 students and teachers to 

entrepreneurship principles. Almost 1,500 entrepreneurs participated in sectorspecific activities. Incubators served 475 clients and graduated at least 15 firms. 

Training and technical assistance were provided to 1,620 entrepreneurs in the 

region. While these data suggest positive outcomes for EI investments, it is 

difficult to quantify these results for the entire EI portfolio of investments since 

metrics varied across project types. 

Some measures were found to be common across categories – jobs 

created/retained and businesses created/expanded.84 Table 7.3 provides initial 

quantitative results for these metrics. These results show that EI investments did 

produce positive quantifiable results in terms of both job and business creation – 

metrics most often reported in evaluations of economic development projects. EI 

investments in sample projects created or retained 5,339 jobs, created 248, 

expanded 39, and created or expanded 324 businesses. 

While the evaluation team cautions against using single performance measures 

to gauge the success and impact of ARC’s EI, there is value in placing these 

entrepreneurship development investments within the context of more traditional 

economic development metrics – specifically, cost per job created/retained and 

cost per business created/expanded. Table 7.4 presents estimates of these costs 

for the ARC sample projects as well as for similar types of business development 

programs. ARC public costs per job or business created compare favorably with 

program investments made in a variety of similar types of programs. With the 

 

83 Job creation/retention and business creation numbers included in Tables 7.2 and 7.3 in this 

chapter differ slightly from those reported in Chapter 6, Table 6.7. Data in this chapter are more 

conservative because some individual projects were excluded from this analysis if they did not 

report both measured and projected outcomes. In Chapter 6, totals were calculated across the 

entire sample as a collective and, therefore, no observations were dropped. 

84 As with any data collection effort, there are caveats that must be stated. In conducting the 

follow-up interviews with project leaders, most reported jobs created and retained as a single 

category rather than separate categories. The evaluation team has chosen to report this 

combined category since it was not possible for most respondents to distinguish between new 

and retained jobs. In addition, one can consider the impact of a job retained within a community 

or region as being equivalent, from an economic development perspective, to a new job created. 

A similar caveat applies to businesses created and businesses expanded. Depending upon 

program type, some projects reported combined metrics – businesses created and/or expanded – 

while others, such as incubators, reported only business creation numbers. We have chosen to 

report these categories as they were reported during interviews with project leaders. 
78

exception of incubator costs, ARC investments produced jobs and businesses at 

lower public cost than other types of investments. However, it is important to 

recognize that these ARC figures reflect the cost in terms of ARC dollars and not 

total dollars invested in these projects. Given that sample projects leveraged 

$1.20 in private investments per $1 of ARC investments, these costs are 

understated by about half. However, even inflating these cost figures to reflect 

total investments shows that ARC investments compare well against similar 

investments. As importantly, these job creation cost figures are relatively small 

compared to the average cost per job created via industrial recruitment strategies 

which can range anywhere from $7,000-$15,000 per job on an annual basis.85

While positive, these metrics alone cannot answer the question of whether EI 

projects have achieved their objectives. The evaluation team considered two 

factors in evaluating the overall success of the initiative: 

 Whether project leaders considered the project to be successful 

 Whether measured outcomes exceeded stated goals (for quantitative 

measures). 

To assess the first factor, project leaders were asked whether they considered 

the project a success, whether the project had met stated objectives, and 

whether the project was sustainable over time, i.e., beyond the ARC grant period. 

The evaluation team felt justified in using this key informant information since 

there was no compelling reason why project leaders would not be objective in 

evaluating success. For most projects, ARC funding had ended and project 

leaders had nothing to gain by being overly optimistic about project results. In 

addition, ARC has not continued to fund the EI as a separate initiative, so project 

leaders had no incentive to overestimate impacts. Finally, project leaders were 

informed that the evaluation was about the overall impacts of EI projects and not 

about the individual performance of any single project. In addition, during the 

interviews, project leaders shared information about both the successes and 

failures associated with their activities. 

Table 7.5 shows that, overwhelmingly, project leaders considered projects to be 

successful and to have achieved program objectives, as determined through 

follow-up interviews. Project leaders generally defined success in terms of 

accomplishing the goals set out at the beginning of the project. However, they 

were also likely to consider a project that had achieved some, but not all, of its 

objectives as being successful. And, they were less concerned about the extent 

to which a goal was met, as long as they saw some accomplishment toward the 

goal. For example, oftentimes a project manager would consider a goal of 

“creating 50 jobs” as being met if they had created some but not all of those jobs. 

 

85 Peter Fisher, “The Fiscal Consequences of Competition for Capital,” Reining in the Competition 

for Capital, Ed. Ann Markusen (Kalamazoo, MI: WE Upjohn Institute, 2007). 
79

If project goals are used as a benchmark for assessing project success, it 

becomes important to assess whether project goals, as initially defined, were 

attainable. That is, were project goals reasonable and realistic? Unfortunately, it 

is not possible through this ex post evaluation to determine on a project by 

project basis whether goals were attainable. However, interviews with project 

leaders suggest that project goals did change over time, often being revised as 

the project was implemented and the overall direction refined. For instance, 

some project leaders indicated that they learned over time that their goals had 

been too ambitious and that they did not realize how difficult implementing the 

project would be. In addition, there were qualitative results that contributed to the 

project leader’s view that the project was successful, but which did not relate to 

the original goals of the project. In both cases, project leaders had legitimate 

reasons for declaring the project a success, based on their personal experience 

and understanding of the broad set of impacts achieved. 

 

Project leaders were generally objective in identifying the sustainability of their 

projects – defined simply as whether or not the project continued beyond the 

period of the ARC grant. It is important to recognize that sustainability was 

defined in terms of the project itself and not in terms of the businesses or jobs 

that were created as a result. Again, there appeared to be no incentive for these 

leaders to overestimate sustainability and the responses to these questions 

suggest that project leaders evaluated sustainability more critically than they did 

success. Even when some part of a project proved to be sustainable, project 

leaders most often defined the project as “not sustainable” if the most substantive 

parts of the project did not continue.86

 

While the perception of project leaders is an important factor to consider in 

evaluating the success of the EI, follow-up interviews also generated data on 

both stated goals and measured outcomes for a set of metrics. These metrics 

provide quantitative information to assess the performance of the initiative. While 

individual projects experienced varying levels of success in achieving stated 

goals, the evaluation team chose to view the success of the EI from a portfolio 

perspective, i.e., data are reported for each program category rather than for 

each individual project within that category. From this portfolio perspective, it is 

clear that the initiative was successful in achieving most of the common goals 

identified for each program category. 

Table 7.6 shows that for six of nine metrics, the stated goal was exceeded or 

met. For the other three variables, the stated goal was not met. However, it is 

useful to consider each of these variables in more detail. For entrepreneurship 

education projects, a stated goal was new business starts. Given our original 

conceptual model, the primary programmatic goal for these types of projects was 

to get more entrepreneurs into the pipeline – to build entrepreneurial skills in 

 

86 It is important to note that most of the projects that were sustainable beyond the ARC grant 

continued to rely on local, state, foundation and other support. Very few of these projects were 

sustainable defined as producing income sufficient to cover the operating costs of a program. 
80

young people and expose them to entrepreneurship as a potential career path. 

The literature is quite clear that entrepreneurship education for youth is not about 

creating businesses in the short run. It is about inspiring young people to develop 

skills that can lead them toward entrepreneurship in the future. For ARC’s EI, 

metrics were reported for entrepreneurship education projects that lasted from 

one to four years, as described in Chapter 6. Creating fewer new business starts 

than projected may have more to do with a misalignment of goals and metrics 

than with lack of success for this set of programs. Following students who 

participated in these ARC funded projects as they advance through school and 

into a career path would likely generate more accurate metrics on business 

creation than could be expected from this initial, short-term glimpse of project 

impacts. Unfortunately, none of these projects provided that long-term follow up. 

 

In terms of incubator performance, one can argue that incubators are established 

to help create more informed and more highly skilled entrepreneurs whose 

businesses, as a result are more productive and more likely to remain in the 

region. These goals could very well be achieved with no impact on job creation, 

at least in the short run. In addition, the range of activities included in the 

incubator program category – from grants for feasibility studies to implementation 

grants – may skew job creation figures. Capacity building projects such as an 

incubator feasibility study are unlikely to have any measurable impact on job and 

business creation. 

Similarly, one can argue that technical assistance and training projects serve to 

create better skilled and informed entrepreneurs who, in turn, improve the 

performance of existing businesses. While some aspiring entrepreneurs may 

create businesses as a result of the technical assistance and training they 

receive, new business creation is not the primary goal of many technical 

assistance providers, such as some Small Business Development Centers that 

work mostly with existing business owners. 

The leveraging of private investment is a final quantitative measure that tracks 

local support of entrepreneurship. ARC investments were described by key 

stakeholders as catalytic. The ability to attract private investment, i.e., leverage, 

as a result of ARC investment is one measure of the impact that these funded 

projects have had on the region. Table 7.7 presents actual and projected total 

private leverage for all ARC projects and sample projects. Projected leverage 

includes the private investment associated with both closed and open projects, 

suggesting the longer term impact associated with ARC investments. For the 

ARC portfolio as a whole, private leverage rates range from 1.7:1 (actual) to 

2.6:1 (projected). For the sample, leverage rates range from 1.2:1 (actual) to 

1.4:1 (projected).87 Collectively, ARC and leveraged private investments have 

 

87 The discrepancy in leveraging rates between the universe of projects and our sample results 

from four revolving loan fund projects that were not included in the sample but leveraged almost 

$32 million in private investment. 
81

had an impact by creating the quantitative and qualitative impacts described in 

this chapter. 

Conclusions Regarding Quantitative Impacts 

Based on analysis of quantitative metrics for the evaluation sample, ARC’s 

Entrepreneurship Initiative has had an impact on the region. Collectively, sample 

projects created or retained over 5,300 jobs and created or expanded over 600 

businesses. While it is not possible to determine definitively whether these jobs 

would have been created without ARC investment, project leaders reported these 

outcomes and indicated that ARC investment was critical to achieving these 

impacts. 

Business and job creation numbers tell only part of the story. Over 11,500 

students and teachers participated in or received training from the sample 

entrepreneurship education projects in which ARC invested. Almost 1,500 

entrepreneurs participated in sector-focused activities. Another 1,620 received 

training and technical assistance, while 475 were served by incubators. Every 

dollar invested by ARC in these sample projects leveraged $1.20 in actual private 

investment and is projected to leverage $1.40 private dollars for every ARC dollar 

invested when project investments are complete. And, project leaders and others 

in the region identified a number of qualitative impacts from these investments 

that are having far reaching consequences for the Appalachian region. In 

addition to supporting the conclusion that the EI has had an important positive 

impact on the region, these observations also suggest that a portfolio of 

programs like ARC’s EI requires a “portfolio of metrics” to accurately tell the story 

of program impact. 
82

Table 7.2. Quantitative Results for Metrics Common within Program Categories88

CAPITAL ACCESS ENTREPRENEURSHIP 

EDUCATION 

SECTORS INCUBATORS TECHNICAL 

ASSISTANCE AND TRAINING 

Metric # Metric # Metric # Metric # Metric #- Businesses created/expanded 223 - Jobs created/ retained 1,229

- # of participants/ 

trainees (students or teachers) 11,634 - New business starts (student or adult) 85 

- # of participants/ 

trainees 1,497 - Business created/ expanded 101 - Jobs created/ retained 994 

- # current clients 114 

- Total clients served 475 - # graduated firms 15 - Jobs created/ retained 130

-Complete incubator feasibility study 3 

- # business 

expanded 39

- # of clients 1,620 - # of clients remaining in service area 114 - Businesses created 163- Jobs created/ retained 2,986 

Table 7.3. Quantitative Results for Metrics Common across Program Categories PROGRAM CATEGORY JOBS 

CREATED/RETAINED 

BUSINESSES 

CREATED 

BUSINESSES 

EXPANDED 

BUSINESSES 

CREATED/EXPANDED 

Capital Access 1229 --- --- 223 Entrepreneurship Education --- 85 --- --- Sectors 994 --- --- 101 Incubators 130 --- --- --- Technical Assistance and Training 

2986 163 39 --- 

TOTAL 5339 248 39 32488 Only those metrics that were reported as project outcomes are included in this table. 
83

TABLE 7.4. COMPARISON OF COST PER JOB CREATED/RETAINED AND BUSINESS CREATED/EXPANDED – ARC SAMPLE PROJECTS AND OTHER REPORTED PROJECTS 

ARC SAMPLE 

PROJECTSA 

EDA PROJECTS PENNSYLVANIA 

SMALL BUSINESS 

DEVELOPMENT 

CENTERSG 

INCUBATORSH ENTERPRISE 

FACILITATION IN 

KANSASI 

SBA MICROLOAN 

PROGRAMJ 

$ per job created/retained: Capital Access - 624 Sectors - 732 Technical  Assistance - 579 Incubators - 3,994 $ per business created/expanded: Capital Access - 2,988 Sectors - 3,759 Technical  Assistance - 7,818 

$ per job created: Revolving Loan Fundsb - 936 

Rural Projectsc - 

6,157 Urban Projectsd - 

2,982 Public Workse -

4,857 Defense 

Adjustmentf - 

12,000 

$ per job created - 929 

$ per business created - 3,300 

$ per job created -1,100 

$ per job created -2,400 – 7,800 

$ per job created -4,568 

aFigures are public cost per job, counting ARC project investments only, not total project investment including leveraged private funds. bFigures are for EDA project investments only, not total project investment. Source: Robert W. Burchell, EDA RLFs: Performance Evaluation

(Washington, DC: U.S. Economic Development Administration, 2002). 

cSource: Amy K. Glasmeier, Cost per Job Associated with EDA Investments in Urban and Rural Areas (Washington, DC: U.S. Economic 

Development Administration, 2002). dSource: Glasmeier (2002). 

eSource: Robert W. Burchell, EDA Public Works Program: Performance Evaluation (Washington, DC: U.S. Economic Development Administration, 

1997). fSource: Robert W. Burchell, EDA Defense Adjustment Program: Performance Evaluation (Washington, DC: U.S. Economic Development 

Administration, 1997). 

gSource: Pennsylvania Small Business Development Centers, “Return on Investment: 2005” 13 December 2007

<http://www.pasbdc.org/downloads/pdf/impact.pdf>. hSource: Lawrence Molnar, et al., Impact of Incubator Investments (Athens, OH: National Business Incubation Association, 1997). iSource: Don Macke and Deborah Markley and Erik Pages, “Enterprise Faciliation® in Kansas: Lessons and Recommendations,” RUPRI Center 

for Rural Entrepreneurship, August 2005. jFigures include actual dollars loaned plus cost of technical assistance per job created. Source: Association for Enterprise Opportunity, Testimony 

by Kevin Kelly, Managing Director for Policy and Advocacy, before the House Committee on Small Business, September 27, 2007. 
84

Table 7.5. Project Leaders’ Perceptions of Project Success and 

Sustainability 

QUESTION # RESPONDING 

“YES” 

% RESPONDING 

“YES” 

Did you think the project was a success? 87 89.7 

Do you feel you achieved the objectives set 

forth for this project? 85 92.9 

Has the project continued after ARC funding 

ended? 86 79.1 

Table 7.6. Measured Success of ARC’s Entrepreneurship Initiative Portfolio 

PROGRAM 

CATEGORY 

STATED GOAL MEASURED 

OUTCOME 

PERFORMANCE

(=/-/+) 

Capital Access 79 jobs created/ 

retained 

1,229 jobs 

created/retained 

+ 

Entrepreneurship 

Education 

4,483 

participants/trainees 

(students or teachers) 

141 new business 

starts 

11,634 

participants/trainees 

(students or teachers) 

85 new business 

starts 

+ 

- 

Sectors 38 businesses 

created/expanded 

438 jobs 

created/retained 

101 businesses 

created/expanded 

994 jobs 

created/retained 

+ 

+ 

Incubators 162 jobs 

created/retained 

3 completed incubator 

feasibility studies 

130 jobs 

created/retained 

3 completed incubator 

feasibility studies 

- 

= 

Technical 

assistance and 

training 

177 businesses 

created 

1,295 jobs 

created/retained 

163 businesses 

created 

2,986 jobs 

created/retained 

- 

+ 

Table 7.7. Actual and Projected Leveraged Private Investment – Universe 

and Sample Projects 

ARC $ ACTUAL 

LEVERAGED 

PRIVATE $ 

PROJECTED 

LEVERAGED 

PRIVATE $ 

Universe 42,971,688 72,802,868 109,879,064

Sample 12,847,733 15,856,275 18,596,174
85

QUALITATIVE IMPACTS 

There is an emerging consensus among practitioners, particularly in rural places, 

that entrepreneurship development is a long-term economic development 

strategy. It requires a cultural shift from a mindset of dependency to one of selfsufficiency. In other words, this requires a change in the view that growth will only 

come from decisions and investments that are controlled from outside the 

community to one that growth will come from encouraging homegrown 

entrepreneurs and the businesses they create. ARC’s Entrepreneurship Initiative 

was designed to encourage this culture change and to build capacity within the 

region to become more entrepreneurial. One might expect, therefore, that the 

outcomes achieved through EI investments would extend beyond job and 

business creation. Interviews with project leaders and stakeholders across the 

region identified a set of cross-cutting qualitative outcomes, described below, that 

suggest a much broader impact on the region than that described through the 

analysis of quantitative metrics. 

 ARC investments raised the profile of entrepreneurship as a 

development strategy, helping to change the mindset within the 

region. The investments made by the EI helped to raise awareness of 

entrepreneurship and to give credibility to entrepreneurship as an 

economic development approach. The EI projects “opened people’s eyes 

to other possibilities” and showed that there was “more to economic 

development than infrastructure”. According to one interviewee, the “most 

effective part was the fact that ARC recognized entrepreneurship as 

economic development. That, in itself, was a major step.” 

 ARC investments represented “but for” money in the region, 

providing start-up funding for innovative projects. The EI projects 

often represented “outside the box” thinking or demonstration projects that 

would not have gotten off the ground “but for” the ARC investment. In 

some places, these projects served to prove a concept or approach that 

then attracted additional investment. ARC investments were variously 

described as being “catalytic” or “foundational” to the efforts to encourage 

entrepreneurship in the region. 

 ARC investments leveraged additional resources that helped some 

projects achieve scale and impact. As demonstrated in Table 7.7, 

ARC’s EI portfolio leveraged significant private investment. To the extent 

that these private funds represented resources new to the region, the 

impact would indeed be positive. It is possible, however, that some of 

these investments represent a reallocation of capital from one use to 

another in the region. However, in the case of capital access, ARC 

investment leveraged additional resources that helped to create an 

industry, in this case, the venture capital industry in the region. 

 ARC investments facilitated networking and collaboration among 

practitioners. The increased focus on entrepreneurship helped to 

reinforce that practitioners were doing “good work” and served to connect 
86

both people and organizations in broader regional partnerships. The 

investment provided support and encouragement to what one interviewee 

described as a “fragile community” of practitioners engaged in 

entrepreneurship development. Practitioners realized that they had assets, 

skills and opportunities to combine that would help them achieve selfsufficiency. As one interviewee noted, “It’s amazing what we can get done 

here.” 

 ARC investments helped to change people’s attitudes, particularly 

among youth and their teachers. Project leaders noted increased 

enthusiasm and a change in attitude among young people in particular. 

These changes were described in a number of ways including increased 

self esteem, improved performance in school, and a new “entrepreneurial 

mindset” for students; increased enthusiasm and interest in pursuing their 

own entrepreneurial aspirations among teachers. 

These qualitative insights are drawn from interviews with key stakeholders in the 

region. It was beyond the scope of this evaluation project to conduct the in-depth 

field work required to accurately verify these outcomes. However, these recurring 

themes were heard across our interviews, from people representing different 

states and organizations – non-profit and governmental – and engaged in 

different aspects of entrepreneurship development – from providing capital to 

educating youth. A more rigorous assessment of these qualitative impacts would 

be possible by designing a participatory evaluation approach as part of project 

design. The challenges and opportunities associated with this approach are 

discussed in Chapter 9 as part of the “best in class” metric system. 

Conclusions Regarding Qualitative Impacts 

By identifying the qualitative outcomes associated with ARC’s EI investments, a 

more nuanced, in-depth picture of the impact on the region is obtained. The EI 

has served to “change the conversation” – to elevate entrepreneurship as a key 

component of economic development in the region. The beginning of a culture 

shift is evident at the community level, where EI projects have been having 

demonstrable quantitative and qualitative impacts. There is less evidence that 

these impacts are translating into policy change at the state level, suggesting the 

need to explore the lessons learned from the EI experience for both 

entrepreneurship development, generally, and the design and management of 

future ARC entrepreneurship investment activities. 

What do these qualitative impacts suggest about the metrics needed to measure 

the broad set of outcomes entrepreneurship investments may have? The 

observation of these impacts supports the need for using a broad set of metrics 

to accurately depict project outcomes. These qualitative outcome measures 

could include: 
87

 Public investment ($) in entrepreneurship development activities (pre- vs. 

post project investment) 

 Private investment leveraged ($) as a result of project investment in 

entrepreneurship development 

 Perceived change in community/regional support for entrepreneurship 

development (as measured through pre- and post-investment community 

surveys) 

 Increased collaboration among support providers (as measured by the 

number of partners contributing resources to entrepreneurship 

development). 
88

CHAPTER 8 

LESSONS LEARNED FROM THE ARC ENTREPRENEURSHIP INITIATIVE 

EXPERIENCE 

When ARC first announced the Entrepreneurship Initiative in 1997, ARC staff 

and its state and local partners had a very limited base of experience and 

effective practices upon which they could build. Yet the EI investments were not 

a completely new thing. In fact, many EI projects had similarities with earlier 

ARC investments in capacity building or sector development strategies. In 

addition, other small business development programs (such as the national Small 

Business Development Center initiative) or state business support programs 

(such as Pennsylvania’s Ben Franklin Partners effort) offered useful ideas and 

lessons learned. 

While these earlier programs provided some useful lessons, none of these 

predecessors sought to combine the objectives of supporting traditional 

economic development goals, such as new business starts, with a wider mission 

of knitting entrepreneurial development into the mainstream of economic 

development thinking and practice in Appalachia. As a result, the ARC team 

was, in some sense, building the road as they traveled it. 

Through the course of the EI, ARC and its program partners learned a great deal 

about how to do entrepreneurial development right. These lessons were 

sometimes learned through the school of hard knocks, as once promising 

initiatives failed to pay the expected dividends. In other cases, successful pilot 

projects were replicated throughout Appalachia and throughout the US. For 

example, ARC’s early investments in developing alternative equity capital 

sources were one stimulus for creation of the New Markets Venture Capital 

initiative and New Markets Tax Credit program which now supports more than 

$19 billion in investments in low-income communities. 

Throughout the evaluation process, the research team has focused on gathering 

these “lessons learned” as means to capture established best practices, 

exemplary program models, as well as informal and tacit learning that has 

occurred through the life of the EI program. These lessons learned should inform 

future ARC investments (in entrepreneurial development and elsewhere) as well 

as other federal, state, and local efforts to promote entrepreneurship. 

The compiled “lessons learned” generally fall into two broad categories. One set 

of lessons applies to those actively engaged in the practice of entrepreneurship 

development – people who are implementing entrepreneurship education, 

training and technical assistance, capital, incubator, and networking or sector 

specific initiatives in their communities, regions or states. The other set of 

lessons applies more directly to the design and implementation of ARC’s 

Entrepreneurship Initiative and would be most useful to those seeking to create 

similar or additional region-wide initiatives. 
89

LESSONS FOR PRACTITIONERS – WHAT WORKS IN ENTREPRENEURIAL 

DEVELOPMENT 

Nearly all of the interviewees offered thoughtful lessons learned based on their 

participation in EI. Because EI was something of a “new thing” for many state 

and local partners, it forced many project leaders to think differently and move 

out of their traditional comfort zones. As the projects progressed, the project 

teams evolved in their thinking and became more sophisticated in understanding 

the key ingredients for a successful regional entrepreneurship strategy. 

Project leaders and other regional stakeholders emphasized several key lessons 

learned. If one were concocting a recipe for successful regional 

entrepreneurship projects, the following ingredients would be required. 

Organizations with these attributes tended to be more successful than their 

counterparts who lacked some or all of the necessary key ingredients. These 

exemplary practices related to program and community leadership, a program’s 

management, goals and objectives, and a program’s outreach efforts (Table 8.1). 

Additional information on each of these factors is provided below. 

 

Table 8.1. Lessons Learned for Practitioners from Evaluation of ARC’s 

Entrepreneurship Initiative 

LESSONS FOR PROGRAM 

LEADERSHIP 

LESSONS FOR PROGRAM 

MANAGEMENT 

LESSONS FOR PROGRAM 

OUTREACH 

Successful entrepreneurship 

initiatives had sparkplugs or 

local champions that provided 

leadership for these efforts. 

Local capacity was a key to 

success. 

Program self-sufficiency 

(sustainability) and success 

went hand in hand. 

Entrepreneurship 

development was recognized 

to be a long-term process. 

Successful projects altered 

their goals and approaches as 

conditions warranted. 

Partnerships and 

collaborations were important 

to success. 

Successful projects 

celebrated and shared the 

story of their success. 

Lessons for Program Leadership 

Successful entrepreneurship initiatives had sparkplugs or local champions 

that provided leadership for these efforts. 

The need for committed local leadership is a critical requirement for success with 

regional entrepreneurship efforts – just as it is with other forms of economic 

development. The role of this local “sparkplug” became especially important in 

the EI since it was, in part, designed as a response to the decreasing dividends 

generated by traditional economic development strategies based on business 

recruitment and attraction. Changing thirty years of economic development 
90

practice would not occur overnight. In many communities, the concepts of 

entrepreneurial development were not well understood or were resisted by those 

who were comfortable with the status quo as the way things had always been 

done. 

In the midst of this environment where the new concepts were poorly understood 

or discounted, strong leadership was required. Entrepreneurial development 

needed a strong “brand” and a compelling vision that would capture the 

imagination of local leaders and residents. In most cases, this vision was 

generated by a local champion who came to embody the new approaches and 

the new vision for the region. 

These leaders came from non-profit organizations, community colleges, schools, 

economic development organizations, and other institutions. They included 

people who were visionaries and saw the potential for entrepreneurship to be a 

force for change in their communities. They embodied the characteristics of 

entrepreneurs themselves – they saw opportunities, marshaled resources, were 

flexible, and determined and committed to creating a new economic development 

path in their part of Appalachia. 

Abingdon, Virginia’s Appalachian Sustainable Development (ASD) and its leader, 

Anthony Flaccavento, exemplify this pattern.89 Founded in 1995, ASD seeks to 

promote sustainable farming and forestry in Appalachian regions of Virginia and 

Tennessee. Southwest Virginia had always been a center of agriculture and 

forestry, but these industries, especially forestry, had never previously focused 

on sustainable practices. And, for many local leaders, environmentalism and 

economic development were opposing forces. In their view, environmentalists 

were opposed to business and insufficiently concerned about strengthening the 

local economy. 

Flaccavento and ASD sought to redefine the debate by highlighting the 

tremendous entrepreneurial opportunities presented by value-added agriculture 

and forestry. They educated farmers about sustainable forestry practices and 

enlisted their support, so that instead of simply harvesting logs, local 

entrepreneurs could sustainably gather the wood, process the lumber close to 

home and use it in value-added products like flooring and wood trim. The 

Appalachian Sustainable Woods Processing Center, created with EI funds, 

became a local symbol for home-grown initiatives. It combined exciting 

entrepreneurial opportunities with a respect for home-grown traditions and 

industries, and fostered a growing belief throughout the region that sustainable 

business practices can and do work. In 2001, ASD created and is actively 

building its own Sustainable Woods product line90 to sell wood products that are 

environmentally friendly. 

 

89 An interview with Anthony Flaccavento can be found at PBS NOW Enterprising Ideas website, 

<http://www.pbs.org/now/enterprisingideas/asd.html>. 

90 To learn more, visit <http://www.asdevelop.org/sustainablewoods2.html>. 
91

Local capacity was a key to success. 

ARC has a long history of investing in local community capacity building,91 and a 

number of the EI investments sought to develop local capacity for supporting 

entrepreneurs. For example, TEAM Pennsylvania sought to seed 

entrepreneurial assistance networks across the state. These networks would be 

affiliated with the seven economic development districts located within ARC’s 

jurisdiction. This effort ultimately generated mixed results. Several of the 

networks jelled, and one effort (the Northeast Pennsylvania Entrepreneurial 

Alliance)92 received a national award. Yet, few of these networks are now in 

operation and their role in supporting local entrepreneurs was limited. 

The TEAM Pennsylvania experience was fairly typical for EI. Because of limited 

resources, EI faced significant challenges in creating new community capacity 

where none had previously existed. However, one case where EI contributed to 

building capacity is Tech 2020 in Tennessee. Tech 2020 was initiated in 1993 to 

build on the unique regional assets in eastern Tennessee and create a high tech 

industry. ARC made a series of investments totaling $1.2 million over five years 

to build the capacity of Tech 2020, primarily in the area of venture investing. 

Tech 2020 has grown over time into a significant economic development 

organization in the state and the region, playing a major role in establishing the 

Southern Appalachian Fund, one of the first New Markets Venture Capital 

Companies making investments in Tennessee, Kentucky and Appalachian 

Georgia and Mississippi. 

When strong organizations with existing capacity were already in place, EI 

investments had a catalytic effect. In most of the successful cases, a community 

was home to an organization with a strong track record in other related fields of 

activity and this capacity was leveraged in support of entrepreneurship 

development. For example, the Kentucky Highlands Investment Corporation 

(KHIC) had been supporting community development efforts in Eastern Kentucky 

since 1968.93 KHIC began as a traditional community development organization 

but its mission has evolved over time. Over the years, it has become an 

intermediary for many Federal lending programs, such as SBA and USDA, and 

had thus developed extensive in-house financial expertise and capacity. At the 

time of the EI’s introduction, KHIC was seeking to increase the supply of seed-

 

91 For an evaluation of these projects, see Brian Kleiner, et al., Evaluation of The Appalachian 

Regional Commission’s Community Capacity-Building Projects, Final Report to the Appalachian 

Regional Commission, July 2004, xi. 

92 National Association of Development Organizations (NADO) Research Foundation, Business 

Not as Usual: Regional Development Organizations Promote Rural Entrepreneurship

(Washington, DC: NADO, 2002), <http://www.nado.org/pubs/pioneer02.pdf>. 

93 To learn more, see Deborah Markley and David Barkley, Development of an Entrepreneur 

Support Organization: The Case of Kentucky Highlands Investment Corporation, RUPRI Center 

for Rural Entrepreneurship, Research Case Studies Series No. 1, March 2003, 

<http://www.ruraleship.org/content/pdf/KHICfinalstudy.pdf>. 
92

stage equity in the region. Thanks to EI investments, KHIC was able to branch 

out into these new related markets. Today, the Southern Appalachian Fund, 

backed in part with EI dollars, manages $12.5 million that can be invested in local 

firms seeking early stage equity capital. The success with which the Southern 

Appalachian Fund emerged as a developmental venture capital force in the 

region was due, in part, to the strong KHIC base upon which it was built. 

Lessons for Program Management 

Program self-sufficiency (sustainability) and success went hand in hand. 

While all ARC grantees seek to be self-sustaining, effective EI projects viewed 

ARC funds as start-up investments that were not an end in themselves. Instead, 

ARC dollars were used to jump-start programs that would rise or fall based on 

how they performed in the marketplace – just like any other entrepreneurial 

venture. Projects that had a goal of becoming self-sufficient appeared to create 

better outcomes and stronger sustained efforts. Leaders of these projects 

pursued sustainability by creating partnerships and finding resources to continue 

to build the program beyond the ARC grant. Creating sustainable economic 

development programs takes time; this lesson suggests that making selfsufficiency an explicit project goal may result in greater priority being placed on 

its achievement. 

As a group, the EI’s Capital Access projects performed best in terms of 

leveraging outside investments to have a sustainable impact on the region. As 

described above, a developmental venture capital industry was spawned through 

the efforts of organizations funded, in part, by ARC. KHIC and Tech 2020 were 

able to launch the Southern Appalachian Fund (SAF) as one of six New Markets 

Venture Capital companies in the country through initial support provided by 

ARC. SAF received $2 million in New Market Tax Credits in 2002, and was able 

to raise a total of $12.5 million from investors, including the Tennessee Valley 

Authority, foundations and a number of banking institutions. The principals 

behind SAF credit ARC with its creation. One noted that if EI’s director “had not 

been creative, Southern Appalachian Fund would not have been created.” ARC 

was also an early investor in Meritus Ventures, a $36 million Rural Business 

Investment Company established in 2002. As with SAF, Meritus brings a much 

needed source of equity capital to support expanding companies in the region. 

Among Meritus’ investors are regional entities, such as TVA and the University of 

Kentucky, private financial institutions, and some high net worth individuals. Both 

SAF and Meritus are combining access to capital with business support services, 

funded in part by the U.S. Small Business Administration and U.S. Department of 

Agriculture. 

Yet attracting new money is not the only measure of an effective and sustainable 

program. Sustainability can also be generated when programs succeed in 

attracting new partners, building local collaboration, and generating energy and 
93

buzz about local community-building efforts. The development of Athens, Ohio’s 

Dairy Barn Arts Center reflects this pattern. Thanks to a $50,000 EI grant, the 

Dairy Barn Southeast Ohio Arts Center was able to build a shop to sell artwork 

and crafts produced by local craftspeople. The shop soon turned a profit, 

becoming self-sustaining and providing valuable income to approximately 100 

local artisans. The shop has also stimulated new partnerships with local schools, 

and has generated buzz about the Arts Center’s other projects. From an 

entrepreneurship standpoint, the project has also helped educate local people 

about the potential for “self-sufficiency through art.” 

The Clinch-Powell Community Kitchens (in Treadway, TN) pursued a similar path 

to sustainability. ARC’s investments aided the kitchen in purchasing some new 

equipment, but more importantly, program managers invested these dollars to 

improve marketing capabilities for incubator clients. As the Community Kitchen 

has grown, it has developed close partnerships with the Clinch Appalachian 

Artists Cooperative. As artists and food producers have built partnerships, they 

have entered new markets with new combined products such as gift baskets. 

Many of these products are now sold via the Appalachian Spring Cooperative, a 

joint marketing effort designed to promote family farms and local artisans. 

Entrepreneurship development was recognized to be a long-term process. 

It takes years to produce the culture change that is a desired outcome of many 

entrepreneurship development efforts. Successful project leaders recognized the 

long-term nature of their endeavors and concentrated on developing the staying 

power – resources, leadership, organizational capacity, community support – 

needed. As one interviewee noted, “It usually takes longer to reach the critical 

mass (and resultant job and business creation) than you would expect.” And, 

another noted the need to “be patient, education projects don’t have immediate 

results.” 

By definition, the EI’s entire portfolio of youth entrepreneurship education 

projects reflects this perspective. These projects rarely assessed their 

performance based on traditional economic development measures of job 

creation or leveraging of new investments. Appropriately, they measured 

progress according to unique measures, such as the number of schools offering 

entrepreneurship training or students’ increased awareness of business 

concepts. 

ARC sought to publicize the best regional youth entrepreneurship initiatives 

through its sponsorship of the 2002 and 2003 Appalachian Youth 

Entrepreneurship Springboard Awards (Table 8.2).94 Twelve different programs 

were honored for their ability to teach youth about the key components of 

entrepreneurship, to develop clear and measurable outcomes that provided value 

 

94 See Appalachian Regional Commission, Appalachian Youth Entrepreneurship Springboard 

Award: 2002 and 2003 Winners, 2004, <http://www.arc.gov/index.do?nodeId=1994>. 
94

to the local community, and to create models that could be sustained at home 

and replicated elsewhere. 

The Springboard Award winners were located across the region, and were based 

at a variety of institutions including public schools, non-profits, and vocational 

training centers. Walhalla High School (in Walhalla, SC) was one of the first 

Springboard awardees. The Walhalla School District requires all ninth graders to 

be exposed to entrepreneurship training in their social studies class. When these 

students enter high school, they can take two separate entrepreneurship-related 

courses where they are introduced to basic business concepts and move on to 

start their own businesses. Each year, Walhalla High School students start 

dozens of new school-based companies. In 2007, a local student won the 

prestigious International Young Entrepreneur of the Year Award from the 

National Foundation for Teaching Entrepreneurship. 

Other Springboard winners sought to use entrepreneurship education as an effort 

to slow the out-migration of area youth. Virginia’s Lonesome Pine Office of Youth 

used ARC funds to support the Stay for Life Project at Bush Mill. This effort 

linked local youth with area volunteers who rebuilt and refurbished a local stone 

mill that is now used to produce flour and corn meal. Area youth continue to 

operate the mill and sell its products, and point to the effort as a great means to 

learn about entrepreneurship and about the history of their community. More 

importantly, they were exposed to the idea that they can enjoy a successful 

economic future without having to leave their hometowns. 

Table 8.2. Springboard Award Winners – 2002 and 2003 

2002 WINNERS 2003 WINNERS

Estill County High School, 

Irvine, Kentucky 

Tupelo Middle School, Tupelo, Mississippi 

ACEnet, Athens, Ohio 

Walhalla High School, Walhalla, South 

Carolina 

Lonesome Pine Office on Youth, Big Stone 

Gap, Virginia 

Randolph County Vocational Technical 

Center, Elkins, West Virginia 

Hale County Technology Center, 

Greensboro, Alabama 

Monroe County High School, 

Tompkinsville, Kentucky 

Ripley Union Lewis Huntington High 

School, Ripley, Ohio 

East Stroudsburg High School-North, 

Dingmans Ferry, Pennsylvania 

Carroll County Public Schools, Hillsville, 

Virginia 

United Technical Center, Clarksburg, West 

Virginia 

Additional information about the Springboard Awards can be found on the ARC website at 

http://www.arc.gov/index.do?nodeId=1994. 
95

In addition to the Springboard Awards, ARC made investments in other youth 

education programs that have proven to be sustainable. ARC invested in a 

program run by the University of Alabama to bring entrepreneurship education to 

a 10-county region in the state. The program was successful in using the REAL 

curriculum to train students and to build school-based enterprises. The program 

continued beyond ARC funding, expanding to 14 counties. Many of the students 

were at risk and according to the project leader some businesses were started as 

a result of the training. In another case, ARC funded the Ohio-West Virginia 

YMCA to use REAL training in a summer camp for youth. Two of the important 

outcomes identified were the engagement of schools, communities and an 

economic development district in the project and the change in attitudes seen in 

the young people for participated in the camps. 

While ARC’s investment in youth entrepreneurship programs and approaches 

reflects a concern for changing the culture of the region, it was less apparent that 

portfolio investments were made with explicit attention to the development of 

entrepreneurs over time. Many of the incubation and TA projects measured 

outputs such as businesses served rather than measuring the transformation of 

entrepreneurial skills or the outcomes associated with the projects, such as 

improved business performance. Insights gained from the implementation of 

Lyons and Lichtenstein’s Entrepreneurial League System® approach suggest that 

the long-term transformative impact of entrepreneurship investments depends, in 

part, on the development of entrepreneurs over time.

95

 

 

Successful projects altered their goals and approaches as conditions 

warranted. 

The EI was the first regional effort to encourage entrepreneurship development 

and investments were made when the field was relatively new. Project leaders 

were plowing new economic development territory and many projects were 

considered demonstrations. Effective project leaders adjusted their approaches 

to reflect changing demand and to overcome unexpected obstacles. As a result, 

these projects achieved outcomes that were positive but, in many cases, different 

from what was originally intended. 

Georgia’s Appalachian Community Enterprises’ (ACE) experience during the EI 

may be instructive. ACE was established to help address a pressing capital gap 

facing many small businesses in North Georgia. As ACE developed its microloan 

program, it soon discovered that simply making new loan funds available was an 

insufficient response to the challenges facing the region’s small firms and 

aspiring entrepreneurs. Many local residents lacked money skills or suffered from 

poor credit ratings that made it nearly impossible for them to do business with 

local banks. As a result, ACE quickly geared up its financial literacy education 

efforts. Programs such as ACE’s Money Camp for Grown-Ups are now used to 

help residents learn personal money management skills. 

 

95 Gregg A. Lichtenstein and Thomas S. Lyons (2001). 
96

As ACE has introduced new programs, it has also begun to enter new markets. 

North Georgia is in the midst of a massive influx of Latino immigrants who are 

changing the face of many of the area’s small towns. These new residents are 

also very interested in starting new businesses, and ACE has now begun an 

aggressive effort to provide services to the region’s Hispanic entrepreneurs. 

In Coeburn, Virginia, the initial intent behind an EI grant was to provide training to 

low-income people to become crafters. However, according to the project 

designer, “We learned that you can’t teach people to be artists and crafters 

unless they want to be.” But through the exploration associated with the ARC 

grant, they learned that a significant number of musicians were seeking ways to 

expand their music businesses. The project leader credits the EI with helping the 

local area spawn a popular spot for bluegrass and country music. Now people of 

all ages are participating in the Friday night concerts and Thursday evening jam 

sessions. Crowds are coming from miles around to dance, eat, and enjoy 

affordable, excellent entertainment. The community center originally started 

through the ARC grant is a vintage downtown building that formerly housed Lay’s 

Hardware. It is being conserved through income generated and utilized as a 

mecca for music, the arts, and various art-related training sessions. They are a 

site for, and one of the progenitors of, The Crooked Road, a 13-county effort to 

highlight and market country music venues throughout Southwestern Virginia. 

Lessons for Program Outreach 

Partnerships and collaborations were important to success. 

Successful projects marshaled resources by forming partnerships and 

collaborating with other organizations to share resources and build capacity. 

They leveraged assets and avoided duplication of efforts. These partnerships 

also facilitated networking among service providers, creating a better 

environment for entrepreneurs. 

Nearly all of the EI projects were built on partnerships of some sort. In fact, 

partnerships – at least in the form of matching dollars from state or local sources 

– were a requirement of all EI grants. But, several EI projects developed 

exemplary collaborations that still exist today. For example, ARC invested 

$100,000 to support a regional biotechnology initiative in western North Carolina. 

This project linked Advantage West (the regional development agency), NC Bio, 

Buncombe County, Western Carolina University, and several other key players. 

These funds allowed Advantage West to hire a full-time regional biotech 

coordinator, and to also develop biotech incubator space at Ashe-Buncombe 

Technical Community College. They also developed a region-wide biotech 

steering committee that still operates today. This effort is now an integral part of 

a statewide North BioNetwork, with Ashe-Buncombe College serving as 

headquarters for the state’s BioNetwork BioBusiness Center. 
97

Another example of regional collaboration can be found in the Start Smart 

initiative funded by the EI in a nine-county region of Appalachian Tennessee. 

Funding was provided to the Southeast Local Development Corporation to 

provide technical assistance and training to entrepreneurs in these rural counties 

– a region of the state that did not have a critical mass of business resources. 

This initiative built on prior successful work by the Southeast Women’s Business 

Center and project leaders were able to leverage this reputation to get buy-in for 

the ARC-funded project. However, collaboration was not achieved through 

reputation alone. The Women’s Center program director identified all the 

business resource providers who served the region and personally met with each 

of them. By understanding who the potential partners were – their strengths and 

potential weaknesses – the director leveraged significant resources in support of 

entrepreneurs throughout the region and designed a value-added program. 

Successful projects celebrated and shared the story of their success. 

Many projects engaged the media to help build community support as well as to 

publicize their activities as part of a broader marketing campaign. Some 

communities held up their successful entrepreneurs as role models. ARC was 

also instrumental in sharing success through their region-wide education efforts, 

e.g., conferences, and programs like the Appalachian Youth Entrepreneurship 

Education Springboard award. 

Kentucky’s Artisan Heritage Trails Program, headquartered in Richmond, 

Kentucky, was especially savvy in its media outreach strategies. Program 

leaders faced two challenges in terms of media outreach. First, they needed to 

engage local artisans to participate in the program. Second, they needed to 

market outside of the region to attract tourists to use the trails and visit local 

crafts people. In terms of engaging local artisans, project leaders focused first on 

using the Internet as a marketing tool. As local crafts people began profiting from 

online sales, word of mouth did the rest of the work. More artisans signed up and 

the project was off and running. The Trails program began with a small base of 

82 artisans. Today, more than 600 artisans sell their wares through the program. 

The Trails Programs has also been very savvy and fortunate in its efforts to 

attract tourists to the region. The most recent innovation is an online trail system 

that allows visitors to map out their travel plans (along 17 different trails) and also 

learn more about sites and shops long the trail. But, a more important partnership 

has been developed with National Geographic. The site was featured in National 

Geographic Traveler magazine and its online maps can be accessed at the 

National Geographic Traveler website. 
98

LESSONS RELATED TO EI PROGRAM – WHAT WORKS IN PROGRAM 

DESIGN AND IMPLEMENTATION 

While this evaluation uncovered significant praise for ARC’s leadership in 

entrepreneurship development and the value of the investments made from 

1997-2005, challenges experienced with the program provide some lessons that 

may contribute to the continuous improvement of this or other initiatives in the 

future. Interviewees cited several key challenges that relate to the program’s 

initial design, its structure for implementation, and efforts to track and 

communicate its impacts (Table 8.3). 

Table 8.3. Lessons Learned for Program Design and Implementation from 

Evaluation of ARC’s Entrepreneurship Initiative 

LESSONS FOR PROGRAM 

DESIGN 

LESSONS FOR PROGRAM 

IMPLEMENTATION 

LESSONS FOR PROGRAM 

IMPACTS 

Practitioners and 

entrepreneurs have unique 

local knowledge that can be 

applied to program design and 

subsequent program 

refinements. 

Successful initiatives brought 

together related investments, 

in this case, other regional 

economic development or 

entrepreneurship-related 

investments. 

Getting EI funds to local 

partners was dependent upon 

state program managers and 

varied based on the 

importance assigned to the 

initiative. 

The size of the ARC EI grants 

placed limits on regional 

impacts. 

Building a broader base of 

support for entrepreneurship 

investments requires 

continued efforts to “make the 

case” to local leaders. 

Programs can be improved by 

embracing long-term and 

locally-driven evaluation of 

program outcomes and 

impacts. 

Lessons for Program Design 

Practitioners and entrepreneurs have unique local knowledge that can be 

applied to program design and subsequent program refinements. 

Nearly every interviewee felt that the EI was an idea whose time had come. The 

initiative was appropriately tailored for local needs, and helped introduce new 

economic development concepts into the region. Nonetheless, two primary 

issues emerged when program managers and other stakeholders commented on 

the initial design of the EI. First, some practitioners felt they could have been 

more engaged in the design of the Entrepreneurship Initiative. People in the field 

felt they had limited input on how the EI was designed or on helping ARC 

understand what was needed in the region. And, they felt there was no explicit 

approach to engaging entrepreneurs – the customers – in program design. 

Second, they also suggested that ARC encourage a more holistic approach to 

entrepreneurship development. Successful programs weaved together the 

various ARC initiatives into a more holistic, systems approach to development, 

combining EI investment with broadband, etc. This lesson could guide ARC in 

designing future programs. 
99

Critics recognized that the EI, and other ARC special initiatives, do not simply 

emerge out of thin air. ARC staff and state program managers are responding to 

trends in the field and to their own analyses of local economic development 

needs. Indeed, ARC engaged advisory groups with good representation from the 

private and non-profit sectors to inform the design and implementation of the EI. 

However, as ARC continues to develop future special initiatives or other 

entrepreneurship-related investments, it might consider supplementing these 

approaches with other methods that capture local input in a broader and more 

systematic manner. Multiple methods could be employed. These could include 

online surveys, public forums (e.g., listening tours or town hall meetings) that 

discuss potential new program ideas, or even a process of formal petitions or 

suggestions provided by state or local program partners. Of paramount 

importance in any approach taken by ARC, however, is to engage the 

entrepreneurs in decision making about entrepreneurship development program 

design. 

These techniques are all designed to create a more transparent process of 

communication between local partner organizations and ARC staff. This more 

open process will not only help to improve local satisfaction with ARC programs, 

it will also improve their effectiveness as they become more responsive to local 

markets and more cognizant of local economic development capacities. 

Successful initiatives brought together related investments, in this case, 

other regional economic development or entrepreneurship-related 

investments. 

A second challenge concerns the relationship between EI and other regional 

economic development investments from ARC and other local, state or federal 

partners. For some communities, EI investments served as a one-time 

investment to support a youth entrepreneurship summer camp or to fund the 

start-up costs of an incubator. When these projects lacked a sustainability plan 

or were poorly coordinated with other regional development efforts, they tended 

to fizzle out once ARC funding ceased. 

Virginia’s use of EI funds presents a compelling contrast, offering a model of how 

to link ARC funding into a broader regional regeneration strategy. Southwest 

Virginia’s economy had long depended on mining and manufacturing as regional 

drivers. At the time of EI’s inception, state and local leaders were focused on 

developing new strategies for regional reconstruction. They viewed EI funds as 

seed investments to support strategies around heritage tourism, sustainable 

agriculture and forestry, and Main Street development. As such, they sought to 

tie EI projects to the Virginia Main Street program, and also used EI dollars to 

start-up marquee projects like The Crooked Road heritage music trail and the 

Round the Mountain artisans’ network. In fact, six of the eight venues on The 

Crooked Road tour have received ARC funds. Virginia has invested dollars from 

subsequent ARC special initiatives (in broadband and asset-based development) 
100

to further support this larger region-wide strategy. As one statewide observer put 

it, “We see ARC as part of a mosaic, not as projects. . . . ARC programs are 

tools, not ends in themselves.” 

A similar process occurred in Western Maryland’s Garrett County where local 

leaders linked the EI to an ongoing effort to revitalize the county’s downtown 

areas. EI investments were coordinated with Main Street programs. Main Street 

programs focused on issues of streetscapes, beautification, and supporting local 

retail; EI dollars supported expanded business technical assistance, a new 

microloan program, and creation of a business incubator. Together, these efforts 

have helped revitalize downtown Oakland and the county’s other town centers. 

Lessons for Program Implementation 

Getting EI funds to local partners was dependent upon state leadership and 

varied based on the importance assigned to the initiative. 

State governments are the ARC’s key regional partners, so it is not surprising 

that several dimensions of state government helped shape the EI process. The 

differences between state approaches to the EI are quite striking. As we saw in 

the case of Virginia, several states used EI funds to support components of a 

wider regional development strategy. Other states, such as Pennsylvania, 

steered investments toward certain types of programs or strategies (e.g., 

entrepreneurial assistance through networking). Finally, other states supported 

grass-roots innovation and funded a host of local pilot projects. 

In addition to pursuing different strategies, states also differed in terms of their 

commitment to promoting entrepreneurship as a regional development strategy. 

Some governors and state program managers were cool to the new approach, 

while others viewed it as a critical component for community transformation. Not 

surprisingly, the level of “buy-in” by state leaders had a large impact on how the 

deployment, outcomes and effectiveness of the program varied by state. If the EI 

was not a priority for state leaders, it was not likely to be promoted within their 

state. Statewide performance of EI projects also seemed to improve when state 

program managers had close ties with local community practitioners. 

When it comes to state program management, there is no one best approach. 

Each strategy has its pros and cons, and each strategy may also exclude some 

programs from potential consideration for ARC funding. These programs may be 

located in regions or may be pursuing approaches that fall outside of the state’s 

primary strategic focus. State leaders will retain a central role in future ARC 

project decisions, but, in the case of special initiatives, ARC might consider 

setting aside some portion of funds that are open to direct application from 

localities or non-profits. Further details of this proposed Innovation Fund are 

presented in Chapter 9. 
101

The size of ARC grants placed limits on regional impacts. 

Practitioners, and state program managers, were challenged by the relatively 

small pool of money allocated to the EI. State program managers noted that it 

was difficult to figure out what to do with small amounts of money and it was hard 

to get recognition for the EI as a result. Most of those interviewed recognized the 

political necessity for ARC to spread resources throughout the region but noted 

that this often resulted in too little money to achieve significant impacts. As one 

stakeholder noted, the initiative sowed many seeds but the ground was not fertile 

enough to grow sustainability for most of these efforts.

 

Table 8.4 provides a summary of state-level EI investments on an annual basis. 

They indicate that ARC faced serious obstacles when seeking to transform 

regions or even a region’s thinking about entrepreneurial development. ARC’s 

investments were generally dwarfed by other economic development spending in 

the region. For example, Kentucky spent $808 million on economic development 

in 2004,96 dwarfing ARC’s 2004 Kentucky EI investments of nearly $384,000. 

Because of these disparities, ARC investments must be tightly focused on 

programs that can build scale, be sustainable, and have major impact. 

One interviewee referred to the “sprinkling effect” of EI investments. ARC spent 

enough dollars to meet one of the EI’s key objectives – to educate state and local 

leaders about the potential of entrepreneurial development as a regional 

development strategy. Unfortunately, the investments were not sufficiently large 

to meet ARC’s other objectives of fostering “systemic change in the economic 

development landscape of the region.”97 With average per project investments of 

$126,387, the EI served mainly as seed stage funding for regional 

transformation. There are some noteworthy exceptions that prove the value of 

more sustained investments in program development. ARC made a series of 

grants to a number of programs throughout the region that have demonstrated 

sustainability after the EI investments were completed. The Shoals 

Entrepreneurial Center incubator in Alabama and the Natural Capital Investment 

Fund (NCIF) in West Virginia are two examples of multi-year grant making that 

helped build institutional capacity that ultimately led to sustainable organizations 

that continue to have an impact on their states and, in the case of NCIF, the 

region. 

What this lesson learned suggests for the structure of any future 

entrepreneurship investments is that “next level” investments for local initiatives 

that the EI has spawned are needed. Interviewed project managers remained 

strongly committed to building on the momentum generated by EI. They noted 

 

96 Mountain Association for Community Economic Development, Accounting for Impact: 

Economic Development Spending in Kentucky (Berea, KY: Mountain Association for Community 

Economic Development, 2005). 

97 Appalachian Regional Commission, Entrepreneurship Initiative Program Summary, September 

30, 2003, http://www.arc.gov/index.do?nodeId=1970. 
102

that the EI had “kick-started the conversation” and had “got people thinking” 

about new ways to build community prosperity. The cultural shift sought by EI’s 

designers is starting to take hold. One community leader put it well, “I don’t hear 

the phrase, ‘this isn’t gonna work’ anymore.” The region’s residents – and its 

economic development organizations – are increasingly aware of local economic 

assets and their ability to support home-grown economic development 

approaches. 

ARC has sought to build on this momentum through its regular programs and 

ongoing Chairman’s initiatives in broadband, energy, and asset-based 

development. But, it should not abandon its previous focus on local 

entrepreneurial development. Indeed, as demonstrated in Chapter 5, 

entrepreneurship is a key asset in the region and can be a driving force within 

ARC’s asset-based development initiative. ARC was instrumental in seeding 

entrepreneurship activity in the region, and it should consider some way to 

continue these investments. Emerging successful programs would benefit from 

“next level” funding that would allow project leaders to move these initiatives to a 

level that has the potential for transformative impact. 

Lessons for Program Impacts 

Building a broader base of support for entrepreneurship investments 

requires continued efforts to “make the case” to local leaders. 

While the EI has been successful in beginning to change attitudes toward 

entrepreneurship, this cultural shift is by no means universal throughout the 

region. Particularly among local elected officials, capacity building and efforts to 

make the case are needed. 

Interviewees were quite positive about ARC’s ongoing education efforts. Under 

EI, the agency sponsored several well-attended conferences on entrepreneurial 

development in general, and on specific topics such as business incubation and 

support for creative industries. All of these events were well attended and 

received high marks for attendees. Although ARC issued specific invitations to 

local elected officials in an attempt to get them to these events, for the most part, 

attendees were already engaged with ARC as grantees and partners in the EI 

program. ARC should consider additional outreach efforts, spread across the 

region to encourage participation, as well as investments to provide workshops 

or training to the region’s elected officials or those who work in government 

agencies or cooperatives focused on other rural development issues (e.g., rural 

telecommunications cooperatives, USDA Cooperative Extension officials, 

Conservation Districts, etc.) These programs need not be funded solely by ARC 

but could be developed in cooperation with relevant trade associations. Through 

these partnerships, ARC could not only spread the message about the 

importance of entrepreneurial development. It would also reinforce an important 

message that effective entrepreneurial development requires a holistic and 
103

collaborative approach to economic development. Entrepreneurial development 

is not the sole province of economic developers. It requires partnerships with 

elected officials, educators, local business owners, and other key stakeholders in 

the region. 

Programs can be improved by embracing long-term and locally-driven 

evaluation of program outcomes and impacts. 

Practitioners would have benefited from ongoing assessment of project 

outcomes and follow up from ARC to share lessons learned and support midcourse changes as needed. It was clear that a better metrics system, both in 

terms of defining relevant outcomes and collecting and reporting data, was 

needed. 

This finding echoes the results of Westat’s 2004 evaluation of ARC’s community 

capacity-building projects which found that community program managers were 

ill-equipped to track, measure, and publicize the outcomes of their programs and 

ARC investments.98 In EI’s case, few projects tracked metrics beyond those 

required as part of the ARC grant. This measurement challenge was identified as 

part of the initial evaluation of ARC’s EI investments, completed early in the life of 

the initiative.99 This early study observed that the internal evaluation and 

monitoring systems used in the sample projects lacked “specific outcome 

measures.” While most projects they studied had created an evaluation system, 

“43 percent of all projects cited ‘monitoring outcomes’ as a problem.” The 

persistence of this measurement challenge suggests the need for corrective 

action for future entrepreneurship investments. 

As noted in Chapters 3 and 4, these shortcomings are not unique to ARC 

projects or to the general field of entrepreneurial development. In fact, because 

of their short life spans, few entrepreneurial development programs have been 

subjected to rigorous performance measurement and assessment.100 Moreover, 

most ARC grantees had limited budgets and limited staff who were primarily 

focused on delivering programs and serving customers. As a result, few 

programs could devote sufficient attention to performance measurement. 

Given the resource and time pressures facing program managers, ARC cannot 

expect them to embrace rigorous performance measurement and should instead 

seek to create incentives for more effective program assessment. ARC could set 

aside some portion of all grant funds (1-2%) to support program measurement. 

ARC could also develop a program measurement “toolkit” that helps walk all 

 

98 Brian Kleiner, et al., Evaluation of The Appalachian Regional Commission’s Community 

Capacity-Building Projects, Report to the Appalachian Regional Commission, July 2004. 

99 Regional Technology Strategies, Inc., Evaluation of the Early Stages of the Appalachian 

Regional Commission’s Entrepreneurship Initiative, Report to the Appalachian Regional 

Commission, December 2001. 

100 Organization for Economic Cooperation and Development, OECD Framework for the 

Evaluation of SME and Entrepreneurship Policies and Programmes (Paris: OECD, forthcoming). 
104

grantees through the evaluation process. Finally, ARC could designate resource 

experts among its own staff (or partner organizations) who can provide technical 

assistance (when needed) on pressing performance measurement issues. These 

recommendations are further discussed in the next chapter. 
105

Table 8.4. Annual and Total ARC Entrepreneurship Investments, 1997-2005, by State ($) 

STATE 1997 1998 1999 2000 2001 2002 2003 2004 2005 TOTAL

Alabama 0 159,000 381,663 537,187 198,850 405,000 92,000 518,892 112,675 2,405,267Georgia 110,000 0 441,826 0 581,529 349,442 83,000 723,747 334,118 2,623,662Kentucky 92,769 226,000 787,500 290,532 510,500 225,362 657,880 383,794 587,198 3,761,536Maryland 0 207,000 455,200 548,918 312,379 139,000 99,000 130,000 45,000 1,936,497Mississippi 0 220,577 30,000 164,970 594,835 157,000 0 200,000 226,000 1,593,382New York 0 243,140 187,654 920,000 75,000 25,000 145,000 265,000 35,000 1,895,794North Carolina 

0 0 0 1,050,000 0 444,908 200,000 157,500 673,673 2,526,081

Ohio 0 153,940 406,213 451,921 613,700 244,500 1,673,330 15,000 50,000 3,608,604Pennsylvania 313,397 1,072,808 1,099,000 199,900 659,636 383,750 362,000 460,000 125,000 4,675,491South Carolina 

0 500,000 0 750,000 119,105 752,000 500,000 891,382 58,750 3,571,237

Tennessee 0 451,748 325,000 390,000 410,185 185,000 93,000 201,133 35,000 2,091,066

Virginia 374,500 0 495,380 416,527 476,630 433,000 181,200 280,391 90,000 2,747,628

West Virginia 142,150 752,948 1,075,916 300,537 1,673,748 1,042,835 582,000 458,400 1,105,474 7,134,008Regional 14,000 332,503 124,623 264,901 169,162 79,610 973,574 108,063 75,000 2,401,436Total ARC 1,046,816 4,319,664 5,809,975 6,285,393 6,395,259 5,126,407 5,641,984 4,793,302 3,552,888 42,971,688*

Figures include all projects (closed and open) and investments from all funding sources within ARC.
106

CHAPTER 9 

CONCLUSIONS AND RECOMMENDATIONS 

CONCLUSIONS 

This evaluation was designed to assess the outcomes achieved by the portfolio 

of projects that received investments from ARC’s Entrepreneurship Initiative from 

1997-2005, as well as the broader policy impacts that accrued to the region as a 

result of this initiative. By considering a limited set of performance measures 

collected for the universe of closed projects and a more detailed set of measures 

and insights gained from in-depth study of a sample of projects, the evaluation 

team was able to develop a thorough understanding of what EI investments have 

meant to individual projects and the region as a whole. Job and business 

creation have been important and significant outcomes of EI investments, as 

summarized in Table 9.1 below. Over 12,000 jobs have been created or retained 

and over 1,700 new businesses created, at a cost that suggests an efficient 

allocation of resources relative to other similar types of economic development 

programs. However, these measures tell only part of the story of impacts. EI 

investments have helped to train teachers and expose students to 

entrepreneurship concepts, almost 12,000 throughout the region. These 

investments have been instrumental in attracting almost $73 million in private 

investment to support entrepreneurship development in the region. And, through 

their educational investments and the demonstration effect of the projects 

funded, ARC has made entrepreneurship a legitimate and desirable economic 

development activity in local communities and raised awareness about the 

importance of the entrepreneurial and small business sector to the region’s 

economy. 

Based on the results of this evaluation, the evaluation team concludes that the 

ARC Entrepreneurship Initiative has had an impact in the region by creating more 

entrepreneurs in the pipeline (through its entrepreneurship education 

investments), better informed entrepreneurs and better skilled entrepreneurs 

(through its technical assistance and training, incubator and sector investments). 

These investments have resulted in more job creating businesses. EI 

investments have also helped to create and enhance capacity to support 

entrepreneurship development in the region, most prominently through 

investments in equity capital funds that seeded and facilitated the creation of a 

developmental venture capital industry in the region. In addition, ARC 

investments have created the beginning of culture change in the region – 

increased recognition of the importance of entrepreneurship as an economic 

development strategy and increased support for those people and organizations 

that are committing their talents and resources to pursuing an entrepreneurial 

path. ARC investments have energized and empowered a new set of actors in 

the region, especially non-profit organizations, who continue to provide 

innovative, entrepreneurial leadership in the region. 
107

Table 9.1. Summary of Quantitative and Qualitative Evaluation Findings 

QUANTITATIVE OUTCOMES FOR THE UNIVERSE OF PROJECTS

Jobs created 9,156 

Jobs retained 3,022 

New businesses created 1,787 

Businesses served 8,242 

Actual private $ leveraged $72,802,868 

Public cost / job created or retained $579 - $3,994

Public cost / business created or expanded $2,988 - $7,818 

QUANTITATIVE OUTCOMES FOR THE SAMPLE PROJECTS

Jobs created 4,332 

Jobs retained 1,351 

New businesses created 1,083 

Businesses served 2,957 

Incubator clients served 475 

Students or teachers trained 11,634 

Actual private $ leveraged $15,856,275 

QUALITATIVE OUTCOMES OF ARC EI INVESTMENTS

Raised the profile of entrepreneurship as a development strategy and helped change the mindset 

within the region 

Provided start-up funding for innovative projects that would not have happened “but for” ARC 

investment 

Leveraged additional resources that helped some projects achieve scale and impact 

Facilitated networking and collaboration among practitioners 

Helped change people’s attitudes, particularly among youth and their teachers 

RECOMMENDATIONS 

ARC’s Entrepreneurship Initiative was plowing new ground when first conceived 

in 1997. At that time, only a handful of states and localities were experimenting 

with approaches that placed entrepreneurs and their companies at the center of 

economic development thinking and strategy. Today, the economic development 

landscape is quite different. Entrepreneurship has become a mainstream 

component of local economic development strategies and has been making 

strong inroads in education at the primary, secondary, and post-secondary 

levels. 

In the decade since EI’s inception, the field of entrepreneurship development has 

been created. An important body of knowledge about effective strategies and 

programs now exists. Yet, the field must still be defined as “emerging.” Lists of 

effective practices and programs have been developed,101 but these program 

ideas have not been accompanied with a rigorous approach to program design, 

management, and evaluation.102

 

101 For example, see Markley, Macke and Luther (2005); Dabson, et al. (2003); OECD (2003). 

102 OECD (Forthcoming). 
108

Because it was a forerunner of today’s programmatic innovations, ARC’s EI 

experience can and should be tapped for useful ideas on how to improve the field 

of entrepreneurial development. With a decade of experience under its belt, 

ARC staff and partners can offer invaluable guidance to other federal, state, and 

local policy makers. For example, the Department of Labor’s new WIRED 

(Workforce Innovation in Regional Economic Development) initiative has 

invested more than $300 million in regional projects designed in part to stimulate 

local entrepreneurial activity. These program managers could learn a great deal 

from the EI experience. 

In an effort to strengthen future entrepreneurial development programs, the team 

has developed a set of recommendations that fall into three categories: 

 Recommendations for entrepreneurship development investments 

 Recommendations for creating a “best in class” metrics system 

 Recommendations for program design and management.

Recommendations for Entrepreneurship Development Investments 

Entrepreneurship development initiatives should include assessment of 

existing capacity and capacity building activities as part of project design. 

The evaluation of ARC’s EI investments highlighted the value of capacity building 

– visioning, leadership development (youth and adult), asset mapping, 

community engagement, and strategy development. Successful entrepreneurship 

development initiatives build on existing capabilities, as was seen with Kentucky 

Highlands Investment Corporation, Appalachian Community Enterprise in 

Georgia, and the Shoals Entrepreneurial Center in Alabama, or create new 

capacity , as evidenced by the PACERS youth entrepreneurship program in 

Alabama and the Tech 2020 program in Tennessee. Therefore, including 

capacity building as an integral part of entrepreneurship investments should 

result in stronger, more effective initiatives. 

ARC (and other federal, state or local program managers) could address this 

finding in several ways. At the most basic level, it could require applicants to 

provide an assessment of existing community capacity during the grant 

application process. Such an assessment would be more than a simple listing of 

assets. Rather, project leaders should identify their leadership team, the partner 

organizations, their level of resource commitment to the initiative, and how this 

existing capacity will be used in substantive ways to support regional 

entrepreneurship initiatives. They could also be required to outline a plan for 

enhancing resource or leadership capacity if necessary. 

A second approach would create closer links between existing community 

capacity building programs and entrepreneurial development investments. 

Many of the goals pursued through ARC’s community capacity investments, such 
109

as increased organizational capacity, enhanced skills for individuals, and 

improved economic development outcomes,103 are similar or identical to those 

pursued via an entrepreneurial development strategy. Many program 

interventions, such as enhanced technical assistance to business, also share 

similarities. Building these linkages will be eased thanks to these close 

connections. Indeed, ARC has already experimented with this approach under its 

2005 Asset-Based Development Initiative which explicitly identified promotion of 

civic entrepreneurship as a primary program goal. ARC might consider more 

direct linkages between entrepreneurial development and other existing 

programs. 

Finally, ARC might consider future programs that operate according to a staged 

process where initial investments are made in a host of community capacity 

building efforts. Entrepreneurship development would be considered a “secondstage” investment for communities and programs that had relatively robust local 

capacity in place. This staging of investments would likely produce better 

community outcomes. Not surprisingly, the EI experience indicates that the 

presence of strong local capacity has a significant effect in contributing to better 

economic development outcomes. 

 

Entrepreneurship development investments should be made with a focus 

on the long term. 

The long-term nature of entrepreneurship development requires a long-term 

approach to investment. If a goal of these initiatives is to transform the culture of 

Appalachia, or another region, then a 10-year or longer time horizon is more 

appropriate than the more typical one to three year grant cycle. Throughout the 

interviews, local program managers commented that ARC investments were too 

short-lived to generate sustainable local impacts. Programs could be initiated and 

local momentum could be generated. Yet, in most cases, ARC funds ran out at 

this critical point of impact, two to three years after program initiation. 

This pattern creates a dilemma for program managers. Like a start-up 

entrepreneur, they are intensely focused on scaling up their new programs. 

Because entrepreneurial companies take time to generate outcomes (in terms of 

new products or job creation), support programs are unlikely to be able to boast 

of major economic development impacts during the start-up phase. They may 

have only a few isolated success stories after the first year or two of operations. 

Without these large (and quick) community impacts, program managers are then 

handicapped in their ability to identify other sources for program funds. 

ARC should consider giving preference to multi-year funding commitments that 

would provide a flow of resources over a longer time horizon, assuming 

performance on the part of the local partner and availability of federal funding on 

the part of ARC. These dynamics do not imply that projects should have an 

 

103 For a review, see Brian Kleiner, et al., July 2004, xi. 
110

indefinite period of funding. It is appropriate that project leaders articulate a plan 

for sustainability and show progress toward reaching that goal as a condition of 

long-term investment. But, multi-year funding commitments would provide a 

stable base from which these local projects could develop. 

Entrepreneurship development initiatives receiving investments should be 

market driven and practice continuous improvement. 

The needs of the customer – defined as youth, entrepreneurs or communities – 

must be the key drivers of entrepreneurship development. Initiatives should be 

designed to include mechanisms that obtain performance feedback, allow for 

mid-course corrections, and in some cases, redefine project goals, based on 

what project leaders are learning from their markets. While the field of 

entrepreneurship development has come a long way since the EI began in 1997, 

it is still in an experimentation stage. Organizations throughout Appalachia and 

the country are continually evolving new ways of working with entrepreneurs and 

communities that can inform both the design of new and the improvement of 

existing efforts to encourage entrepreneurship as a core economic development 

strategy. 

ARC should consider requiring project leaders to conduct annual performance 

reviews. These reviews could be conducted as part of the ongoing evaluation 

that is recommended below. And, they should include some assessment of the 

project’s market – for example, assessing the experience of entrepreneurs 

participating in a particular training or technical assistance program through focus 

groups or customer surveys. It is recommended that ARC participate in these 

reviews, providing constructive feedback and suggesting resources that local 

partners might use to improve program performance. ARC could use its network 

of regional and national partners to link a local project with a more experienced 

practitioner elsewhere to address a specific performance issue or concern. 

Emphasis should be placed on investing in initiatives that demonstrate the 

ability to form regional partnerships and collaborations. 

With capacity and resource limitations a reality for most organizations, the ability 

to form dynamic and effective partnerships that share resources becomes 

paramount to success. These cross-organizational and cross-regional 

collaborations should be emphasized in the design of entrepreneurship 

initiatives, and effective partnerships should be rewarded as part of the 

investment process. ARC should require extensive community partnerships for 

all of its future entrepreneurial development investments. 

However, ARC must recognize that it take resources to facilitate collaboration. 

While local projects should be required to demonstrate true collaboration across 

geography (e.g., multi-county projects) and organizations (e.g., public, private 

and non-profit partners), ARC should consider requiring that project leaders 
111

demonstrate budgetary commitment to cover the costs of collaboration. This 

resource commitment could come through allocating part of ARC funds to 

support collaborative activities or through the allocation of local matching dollars 

to this process. ARC can help further true collaboration by making it a priority in 

grant making and by educating local partners to the resource realities associated 

with collaboration. 

Recommendations for Creating a “Best in Class” Metrics System 

Job creation is an overused metric, paints an incomplete picture of the 

outcomes of entrepreneurship development investments, and should be 

replaced by an “entrepreneurship development metrics portfolio.” 

ARC’s existing performance measurement system requires all projects to report 

the following relevant measures as part of its final project close-out process: 

 Businesses Served 

 Jobs Created 

 Jobs Retained 

 Project Participants 

 New Businesses Created. 

These metrics provide some useful insights, but most project managers and 

outside experts felt they were poorly suited to providing a complete picture of the 

EI’s impact. They were better tailored to measure the impact of more traditional 

economic development investments in physical infrastructure. In recognition of 

these shortcomings, nearly all local projects devised their own performance 

indicators which ranged from simple additions such as use of customer 

satisfaction surveys to more extensive systems that tracked the financial 

performance of assisted companies. 

While job creation is reported as a result of ARC’s EI investments, a much richer 

understanding of the initiative’s impact has come through efforts to define and 

capture outcomes as measured by a broad set of performance metrics. These 

metrics include both quantitative output measures, e.g., students trained, as well 

as more qualitative measures, such as enhanced leadership capacity and 

resources leveraged through partnerships. What is clearly needed is the creation 

of a portfolio of metrics linked to particular types of entrepreneurship 

development programs. These metrics would be outcome measures that are 

clearly linked to the goals of particular projects. For example, to evaluate the 

impacts of entrepreneurship education efforts, metrics would focus on measuring 

the outcomes of training – whether young people are more likely to consider 

entrepreneurship as a career path, whether they go on to create businesses in 

the future, whether they return home to the region, etc. 
112

Earlier in this report, Table 4.1 provided a portfolio of performance or outcome 

metrics that can be used to measure the impacts of entrepreneurship 

development investments across the program types that are part of ARC’s EI 

portfolio. These outcome measures were drawn from a review of the literature 

and formed a framework for this evaluation. While this framework could be used 

as a basis for discussion by ARC and its grantees as part of the evaluation 

partnership approach described below, Table 9.2 provides a more limited set of 

metrics that might be operationalized into a “best in class” metrics system. The 

proposed metrics include quantitative measures that would be collected from 

project leaders, who in turn would use survey and other tools to gather data from 

their customers. In addition, there are a number of metrics that will capture the 

qualitative impacts that were observed through this evaluation of the EI 

investments. Outcome measures to capture these qualitative changes in cultural 

attitudes would need to become part of the evaluation system adopted by 

grantees, using tools and techniques developed with support of ARC or another 

funding organization. 

 

Table 9.2. Proposed “Best in Class” Outcome Measures 

Capital Access – Projects designed to provide 

access to a range of capital resources to help 

businesses start and grow and, in the process, 

become stronger competitors in local, regional, 

national and/or international markets.

 Number of new businesses financed 

(measure of business starts) 

 Number of jobs (FTEs) created/retained 

(measure of business growth) 

 Percent of funded firms still in business 

(measure of business performance) 

 Average wage/job created (measure of 

business performance) 

 Percent change profitability (measure of 

business performance) 

Sectors – Projects focused on improving the 

start up, growth, and performance of 

businesses in a particular sector and on 

growing particular sectors of the local or 

regional economy. Projects included 

networking activities designed to improve 

business performance.

 Amount of increased sales ($) attributed to 

network or sector participation (measure 

of business performance) 

 Number of jobs (FTEs) created/retained 

(measure of business growth) 

 Increase in number of business starts in 

targeted sector (measure of business start 

up) 

 Change in total sector sales over time 

(measure of growth in the sector) 

Incubators – Projects focused on creating a 

physical space for businesses to start up and 

grow, with the goal of graduating these firms 

into the local or regional economy.

 Number jobs (FTEs) created/retained 

while in the incubator (measure of 

business starts/growth) 

 Number jobs (FTEs) created/retained after 

graduation (measure of business growth) 

 Amount of capital ($) raised by tenants 

(measure of business growth) 

 Percent of business tenants retained in 

the service area (measure of local/regional 

economic impact) 
113

Entrepreneurship Education* – Projects 

focused on exposing young people to 

entrepreneurial concepts and experiences to 

enhance their understanding of 

entrepreneurship as a career option and to 

encourage youth retention through 

entrepreneurship.

 Increase in awareness of business 

concepts (measure of exposure to 

entrepreneurship concepts) 

 Increase in number of participants 

considering business creation as a career 

option (measure of exposure to 

entrepreneurship concepts) 

 Number of students that stay or return to 

the service area (measure of impact on 

youth retention) 

Technical Assistance and Training –

Projects designed to build the skills of 

individual entrepreneurs so that they can start 

and grow their businesses and create stronger 

enterprises in the local and regional economy. 

 Number jobs (FTEs) created/retained 

(measure of business growth) 

 Number of clients still in business 

(measure of business performance) 

 Private capital ($) raised by clients 

(measure of business growth) 

 Average wage/job created (measure of 

business performance) 

 Percent change in profitability (measure of 

business performance) 

Culture Change – Projects often achieve 

qualitative impacts (both intended and 

unintended) that relate to changes in people’s 

attitudes toward entrepreneurship, their view of 

the importance of entrepreneurs to the 

local/regional economy, and the value placed 

on collaborative decision making and 

partnerships to create a more supportive 

environment for entrepreneurs. 

 Public investment ($) in entrepreneurship 

development activities (pre vs. post project 

investment) 

 Private investment leveraged ($) as a 

result of project investment in 

entrepreneurship development 

 Perceived change in community/regional 

support for entrepreneurship development 

(as measure through pre and postinvestment community surveys) 

 Increased collaboration among support 

providers (as measured by the number of 

partners contributing resources to 

entrepreneurship development) 

* The metrics developed for entrepreneurship education projects refer to potential outcomes of 

these projects as economic development initiatives. Therefore, metrics focus on outcomes that 

have potential impacts on the community and not just the individual young person. These 

individual outcomes have been measured through metrics related to changes in student 

performance, e.g., increased test scores, increased applications to college, improved reading, 

and increased leadership activities in school/community. 

A “best in class” metrics system requires investment in a “best in class” 

evaluation system. 

Performance measurement should be viewed as an integral part of program 

development – from the perspective of funding agencies like ARC and project 

leaders. One of the first steps in developing any initiative needs to be an 

articulation of program goals – what are you trying to achieve – followed by 

identification of how success or performance will be measured. These inputs 

form a performance measurement system that can be used by local project 

leaders to report on success, broaden support, and attract additional resources 

and partners to the effort. From ARC’s perspective, developing the evaluation 
114

framework before investment will help to insure that individual projects contribute 

to the overall goals set forth by ARC and that the agency will have appropriate 

metrics to use to report on the performance of the overall initiative. 

 

Ex post evaluations of major investments like ARC’s EI face serious challenges 

in terms of the collection and integrity of data; recalling accurately the impacts of 

a project that ended 7-10 years ago was a challenge for most project leaders. 

Greater investment must be made in establishing criteria and providing funding 

for an ongoing evaluation of entrepreneurship development initiatives as they 

unfold. As noted above, local grantees regularly developed their own in-house 

metrics to supplement those required as part of ARC’s grants process. While 

these in-house efforts generated a fair amount of useful data, they provided no 

means to aggregate data across program types, across regions, or across the 

entire EI spectrum. 

ARC should support the creation of a performance measurement system for 

future investments by developing a participatory evaluation system in partnership 

with grantees. This measurement system would be developed by grantees and 

their customers, i.e., the entrepreneurs, with support from ARC, and would be 

designed to provide project leaders with useful information that can be used to 

adapt programs to changing circumstances as well as to report to ARC on project 

performance. The evaluation framework should be built into the program from the 

beginning, and project leaders would be expected to sign off on that evaluation 

system as part of a grant agreement. By taking this partnership approach to 

evaluation, ARC would be in a stronger position to hold project leaders 

accountable for generating the outcome metrics identified for the project, and to 

provide them with feedback on performance. 

The evaluation team also recommends that ARC consider two sets of outcome 

metrics: a base set of metrics for all programs, and a tailored set of metrics for 

each specific type of program intervention, as laid out in Table 9.2. For example, 

an entrepreneurship training program and a new business incubator might both 

be assessed according to traditional metrics of job creation or new business 

starts. Beyond these base measures, incubators might be assessed according to 

the number of firms graduated from the facility, the incubator’s annual revenue, 

and a range of financial metrics for businesses served by the incubator. Training 

programs might use a common customer satisfaction survey or other tools that 

assess whether participants gained new skills or knowledge, supplemented by 

follow-up surveys to collect financial performance metrics for business 

customers. Youth entrepreneurship education programs might develop metrics 

related to measuring the entrepreneurial skills acquired by young people who 

participate in these programs. 

ARC has an opportunity to take the lead, among any newly authorized regional 

authorities and other federal agencies, in developing and standardizing 

assessment criteria and methods that can have an impact on how 
115

entrepreneurship development investments are evaluated and, more broadly, 

how the impacts of economic development programs are measured. 

Recommendations for Program Design and Management 

ARC’s initiative process should be regularized so that state program 

managers can more effectively plan for and promote use of the resources. 

Interviews indicate that state program managers and local grantees like the ARC 

initiative process. They like the focus on specialized issues and concerns, and 

they like the learning opportunities provided by ARC through various sponsored 

conferences and reports. They are less enamored with the episodic nature of the 

initiative process. They would like more input into the discussions about new 

initiative topics, and more information on initiatives being considered in the 

future. 

Given the gate keeping role of state leaders, ARC should provide structure and 

consistency to the initiatives to encourage the active buy-in of people at the state 

level. A more transparent and open process for community input on project 

design and implementation would also help ARC create more effective initiatives 

and empower local people and organizations to actively participate in these 

efforts.

As noted in Chapter 8, ARC went through an extensive process of getting input 

into the design of the EI. Advisory groups of state leaders and private/non-profit 

sector practitioners and others with experience in entrepreneurship development 

helped to inform the decision making behind the EI. Meetings and educational 

events throughout the region provided opportunities for state and local leaders to 

provide input to the process. In spite of these significant steps, there remains a 

sense that local practitioners had limited input into the design and 

implementation of the EI. One way to address this perception might work as 

follows. As ARC’s leadership considers new topics for potential initiatives, it can 

open a process for outside input. Ideally, suggestions should be provided in 

multiple formats from a formal request for comment in the Federal Register to the 

use of blogs as a means to generate online discussion. In-person sessions, such 

as town hall meetings held in widely dispersed locations throughout the region, 

should also be considered. ARC should consider tapping into the ever growing 

infrastructure of entrepreneurship service providers in the region, using online 

surveys to get their input on what is needed in the region. The US Department of 

Agriculture’s series of sponsored Farm Bill Forums, to discuss key sections of the 

2007 Farm Bill, offers one excellent model for organizing public outreach and 

discussion.104

 

104 Summaries of the Farm Bill Forums, held throughout 2006, can be found at 

<http://www.usda.gov/wps/portal/usdafarmbill?contentidonly=true&contentid=2006/03/0106.xml>. 
116

ARC’s proven experience can be applied to developing and delivering 

effective, region-wide education programs that help make the case for 

entrepreneurship as a core economic development strategy for the 

Appalachian region. 

There continues to be a strong need to make the case for entrepreneurship, 

particularly among local elected officials and traditional economic developers. 

Throughout the EI, ARC has organized educational opportunities to share 

information about topics such as capital access, business incubation, and 

entrepreneurship education. What is needed now, however, is a broader effort to 

provide community leaders, elected and others, with the understanding and tools 

they need to embrace entrepreneurship as part of an economic development 

strategy. 

ARC has developed a reputation as a trusted authority in this field and also has 

the lessons learned from ten years of entrepreneurship development investments 

to share throughout the region. In addition, ARC’s partnership approach in the 

beginning of the EI process can be brought to bear on this educational effort, 

drawing on resources and experiences of other organizations working on 

entrepreneurship development throughout the country.

A Next Generation Entrepreneurship Initiative 

This evaluation has generated a host of new ideas and lessons learned, but one 

prevailing idea has emerged throughout the evaluation process – additional 

investments in entrepreneurship development throughout Appalachia are still in 

significant demand. Given the success and capacity that the ARC EI has already 

been building, the evidence suggests that continued ARC investment in 

entrepreneurship development in the region is a compellingly logical and vital 

next step. 

To build on this momentum, ARC should create a Next Generation 

Entrepreneurship Innovation Initiative that will be groundbreaking in its 

design. A long-term investment is recommended that incorporates all the 

learning from the EI and the emerging entrepreneurship development field. 

It will include four critical elements: 

 The Entrepreneurship Innovation Fund would provide selective, 

competitive, strategic investments in “next level” entrepreneurship 

development activities throughout the region. The Entrepreneurship 

Innovation Fund would not be tied to individual states, but would be 

competitively awarded across the region. Investments would be made in 

initiatives that demonstrate a holistic, systems approach to 

entrepreneurship development, with an emphasis on those initiatives that 

have the potential to be transformational and sustainable. It is 

recommended that ARC take a portfolio approach to these investments – 
117

investing in more proven innovations as well as those that offer promise 

but are still early stage innovations. 

 The second element would be a pool of funds distributed to the states for 

investments in first tier entrepreneurship projects at the ground level. 

Similar to the EI, these projects would build capacity and fulfill distinct 

entrepreneurship metrics that are developed by the communities and ARC 

working together. 

 In the interest of capacity building, ARC should fund the development of 

“Entrepreneurship Innovation – Guidelines for the Future” – a framework 

for communities to use based on what ARC has learned from 10 years of 

investment in this field and what its partner organizations across the 

country have learned through their various activities. 

 The fourth element would be a built-in evaluation system that is initiated 

from the beginning of the Entrepreneurship Innovation Initiative. It would 

incorporate the “best in class” metrics derived from this evaluation, 

discussions with ARC, and input from the field. This evaluation system will 

be essential to making the case as well as measuring and ensuring 

impact. 

This capstone recommendation is based on the recognition that, while the 

Entrepreneurship Initiative has achieved important impacts at the community 

level, the region has not seen widespread or significant policy change at the state 

level. The entrepreneurship context assessment in Chapter 5 suggests that many 

parts of the region continue to lag the nation, particularly in terms of income 

generated by entrepreneurial activities. The Entrepreneurship Innovation 

Initiative would give ARC an opportunity to make investments that are deeper 

and more transformational, generating impacts that are influential in achieving 

policy change at the state, as well as local, level throughout the region. This 

initiative would also provide an opportunity to implement a participatory 

evaluation system that can generate the data and insights that will provide a 

deeper understanding of how ARC investments help to change the economic 

outlook and performance of the region. 
118

BIBLIOGRAPHY 

Acs, Zoltan J. “The State of the Literature on Small to Medium Sized Enterprises 

and Entrepreneurship in Low-Income Communities.” Federal Reserve 

Bank of Kansas City and Ewing Marion Kauffman Foundation, 2006. 

Acs, Zoltan J. and Catherine Armington. Using Census BITS to Explore, 

Entrepreneurship, Geography, and Economic Growth. Washington, DC: 

US Small Business Administration Office of Advocacy, 2005. 

Advanced Research Technologies. The Innovation-Entrepreneurship Nexus. 

Washington, DC: US Small Business Administration Office of Advocacy, 

2005. 

Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business 

Economics. Vol. 23, No. 2, September 2004. 127-135. 

Alliance for Regional Stewardship. Regional Indicators: Telling Stories, 

Measuring Trends, Inspiring Actions. ARS Monograph Series No. 10. 

November 2005. 

Appalachian Regional Commission. Appalachian Youth Entrepreneurship 

Springboard Award: 2002 and 2003 Winners. Washington, DC: 

Appalachian Regional Commission, 2004. 

Appalachian Regional Commission. Capitalizing on Rural Communities. 

Washington, DC: Appalachian Regional Commission, 2000.

Appalachian Regional Commission. Entrepreneurship Initiative: Program 

Summary and Approved Projects. Washington, DC: Appalachian 

Regional Commission, 2003. 

Atkinson, Robert D. The Past and Future of America’s Economy. 

Northhampton, MA: Edward Elgar, 2004. 

Atrostic, B.K. “Measuring U.S. Innovative Activity.” US Census Bureau Center 

for Economic Studies Working Paper (07-11). March 2007. 

Association of University Technology Managers. AUTM U.S. Licensing Survey, 

FY 2004: A Survey Summary of Technology Licensing (and Related) 

Performance for U.S. Academic and Nonprofit Institutions, and 

Technology Investment Firms. Washington, DC: AUTM, 2004. 

Audretsch, David, et al. “The Knowledge Filter and Economic Growth: The Role 

of Scientist Entrepreneurship.” Ewing Marion Kauffman Foundation, 

2006.
119

Barkley, David L. et al. Establishing Nontraditional Venture Capital Institutions: 

Lessons Learned. Rural Equity Capital Initiative Study of Nontraditional 

Venture Capital Institutions. Columbia, MO: Rural Policy Research 

Institute, 2001. 

Barkley, David L. and Deborah Markley. The Development of an Entrepreneurial 

Support Organization: The Case of the Kentucky Highlands Investment 

Corporation. RUPRI Center for Rural Entrepreneurship Research Case 

Study Series Number 1. March 2003. 

Bartik, Timothy J. Evaluating the Impacts of Economic Development Programs 

on Local Economic Outcomes: What Has Been Done and What is 

Doable? Upjohn Institute Staff Working Paper (No. 03-89). Kalamazoo, 

MI: Upjohn Institute, 2002. 

Bartik, Timothy J. and Richard D. Bingham. "Can Economic Development 

Programs be Evaluated?" Dilemmas of Urban Economic Development. 

Eds. R.D. Bingham and Robert Mier. Thousand Oaks, CA: Sage, 1997. 

246-277. 

Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience 

and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L. 

Sexton and J. D. Kasarda. Boston, MA: PWS-Kent Publishing, 1992. 17-

45. 

Bollingtoft, Anne and John P. Ulhoi. “The Networked Business Incubator – 

Leveraging Entrepreneurial Agency?” Journal of Business Venturing Vol. 

20, 2005: 265-290. 

Brandow Company and Economic Development Research Group. Evaluation of 

the Appalachian Regional Commission’s Infrastructure and Public Works 

Program Projects, Final Report to the Appalachian Regional Commission. 

Washington, DC: Appalachian Regional Commission, 2000. 

Brown, J. David, John S. Earle, and Dana Lup. “What Makes Small Firms Grow? 

Finance, Human Capital, Technical Assistance, and the Business 

Environment in Romania.” Economic Development and Cultural Change

Vol. 54, No. 1, 2005: 33-70. 

Burchell, Robert W. EDA RLFs: Performance Evaluation. Washington, DC: U.S. 

Economic Development Administration, 2002. 

Burchell, Robert W. EDA Defense Adjustment Program: Performance 

Evaluation. Washington, DC: U.S. Economic Development 

Administration, 1997.
120

Burchell, Robert W. EDA Public Works Program: Performance Evaluation. 

Washington, DC: U.S. Economic Development Administration, 1997. 

Carter, Sara, et al. “Beyond Portfolio Entrepreneurship: Multiple Income Sources 

in Small Firms.” Entrepreneurship and Regional Development Vol. 16, 

2004: 481-499. 

Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship Education: 

An Evaluation of the Berger Entrepreneurship Program at the University of 

Arizona, 1985-1999. Kauffman Center for Entrepreneurial Leadership, 

2000. 

Chukumba, Celestine and Richard Jensen. Invention, Entrepreneurship, and 

Start-Ups. National Bureau of Economic Research Working Paper 

(#11475). July 2005. 

Coalition of Community Development Financial Institutions. Providing Capital 

Building Communities Creating Impact. Washington, DC: CDFI Coalition, 

2004. 

Coastal Enterprises, Inc. Measuring Impact in Practice. Wiscasset, ME: Coastal 

Enterprises Inc., February 2006. 

Colgan, Charles S. and Bruce Andrews. Evaluation of Maine Technology 

Institute Programs. University of Southern Maine Center for Business and 

Economic Research. December 2004. 

Collender, Ray et al. “Financial Markets Serve Rural Areas Fairly Well.” Rural 

Development Perspectives Vol. 14, No. 1, May 1999: 28-35. 

Community Development Venture Capital Association. Measuring Impacts 

Toolkit. New York: Community Development Venture Capital Alliance, 

2005.

Cooke, Phil. Regional Innovation Systems as Public Goods.” Working Paper. 

United Nations Industrial Development Organization, 2005. 

Council on Competitiveness. Measuring Regional Innovation. Washington, DC: 

Council on Competitiveness, 2006. 

Council on Competitiveness. Innovate America. Washington, DC: Council on 

Competitiveness, December 2004

Czohara, Laura and Julia Melkers. Performance Measurement in State Economic 

Development Agencies: Lessons and Next Steps for GDITT. Georgia 

State University Fiscal Research Center Report No. 92. February 2004. 
121

Dabson, Brian, et al. Mapping Rural Entrepreneurship. Washington, D.C.: CFED, 

August 2003. 

Drabenstott, Mark. A Review of the Federal Role in Regional Economic 

Development. Working Paper. Center for the Study of Rural America, 

Federal Reserve Bank of Kansas City, May 2005. 

Eberts, Randall W., George A. Erickcek, and Jack Kleinhenz. “Development of 

Regional Economic Dashboard.” Employment Research July 2006: 3-5. 

Edgcomb, Elaine L. and Joyce A. Klein. Opening Doors, Building Ownership: 

Fulfilling the Promise of Microenterprise in the United States. Washington, 

D.C.: FIELD, A Program of the Aspen Institute, 2005. 

Eisinger, Peter K. The Rise of the Entrepreneurial State: State and Local 

Economic Development Policy in the United States. Madison, WI: 

University of Wisconsin Press, 1988. 

Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case of 

High-technology Firms.” Small Business Economics Vol. 21, No. 4, 2003: 

409-422. 

Emery, Mary and Cornelia Flora. “Spiraling Up: Mapping Community 

Transformation with Community Capitals Framework.” Journal of the 

Community Development Society Vol. 37, No. 1, Spring 2006: 19-35. 

Enterprise Corporation for the Delta. Enterprise Corporation for the Delta 

Program Monitoring Report – Business Technical Assistance 2003. 

Jackson, MS: Enterprise Corporation for the Delta, 2003.

European Commission, Directorate General, Enterprise and Industry. The 

PAXIS Manual for Innovation Policy Makers and Practitioners. Brussels: 

European Commission, 2006. 

Fairlie, Robert W. Kaufffman Index of Entrepreneurial Activity, State Report 2005. 

Kansas City: Ewing Marion Kauffman Foundation, 2006. 

Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior Performing 

or Intention Increasing?” International Journal of Entrepreneurship and 

Small Business Vol. 2, No.1, 2005: 89-98. 

Feldman, Maryann. “The Entrepreneurial Event Revisited: Firm Formation in a 

Regional Context.” Industrial and Corporate Change Vol. 10, No. 4, 2001: 

861-891. 
122

Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship and 

Access to Capital in Peripheral Locations: An Empirical Analysis.” Growth 

and Change Vol. 33, 2002: 196-215. 

Fisher, Peter. “The Fiscal Consequences of Competition for Capital.” Reining in 

the Competition for Capital. Ed. Ann Markusen. Kalamazoo, MI: WE 

Upjohn Institute, 2007. 

Foster, Lucia. Reallocation and Productivity Dynamics in the Appalachian 

Region. US Census Bureau Center for Economic Studies Working Paper 

(06-03). January 2006. 

Giannetti, Mariassunta and Andrei Simonov. On the Determinants of 

Entrepreneurial Activity: Individual Characteristics, Economic 

Environment, and Social Norms. White Paper. Stockholm School of 

Economics, June 2004. 

Gilbert, Brett Anitra, David B. Audretsch, and Patricia P. McDougall. “The 

Emergence of Entrepreneurship Policy.” Small Business Economics Vol. 

22, 2004: 313-323. 

Gillett, Sharon E., et al. Measuring the Economic Impact of Broadband 

Deployment, Final Report to the Economic Development Administration. 

Washington, DC: Economic Development Administration, February 2006. 

Glasmeier, Amy K. Cost per Job Associated with EDA Investments in Urban and 

Rural Areas. Washington, DC: U.S. Economic Development 

Administration, 2002. 

Goetz, Stephan J. and David Freshwater. “State-Level Determinants of 

Entrepreneurship and a Preliminary Measure of Entrepreneurial Culture.” 

Economic Development Quarterly Vol. 15, No. 1, February 2001: 58-70.

Greenburg, Elizabeth and Richard Reeder. Who Benefits from Business 

Assistance Programs? Results of the ERS Rural Manufacturing Survey. 

United States Department of Agriculture Information Bulletin Number 736-

04. 1998. 

Greene, F.J. and D.J. Storey. “An Assessment of a Venture Capital Creation 

Programme: the Case of Shell LiveWIRE.” Entrepreneurship and

Regional Development Vol. 16, No. 2, March 2004: 145-159. 

Greenwood Consulting Group. A Survey of Business Incubators in Appalachia. 

Washington, DC: Appalachian Regional Commission, July 2005. 
123

Grimaldi, Rosa and Alessandro Grandi. “Business Incubators and New Venture 

Creation: An Assessment of Incubating Models.” Technovation Vol. 25, 

2005: 111-121. 

Heard, Robert and John Sibert. Growing New Businesses with Seed and Venture 

Capital: State Experiences and Options. Washington, DC: National 

Governors Association, 2000. 

Ho-Kim, Thu-Mai and Ernesto Venegas. Starting Up Economic Engines: Key 

Factors for Growing Successful Start-Ups. Minnesota Department of 

Employment and Economic Development Issue Brief. October 2003. 

Huggins, R. “The Success and Failure of Policy-Implanted Inter-firm Network 

Initiatives: Motivations, Processes and Structure.” Entrepreneurship and 

Regional Development Vol. 12, 2000: 111-135. 

Innovation Associates Inc. Accelerating Economic Development Through 

University Technology Transfer. February 2005.

Institute for Competitiveness and Prosperity, Measuring Ontario’s Prosperity: 

Developing an Economic Indicator System. Working Paper No. 2. 

Ontario, Canada: Institute for Competitiveness and Prosperity, 2002. 

Junior Achievement. The Impact on Students of Participation in Junior 

Achievement: Selected Cumulative and Longitudinal Findings. 

Monograph. January 26, 2004. 

Katz, Jerome. And Another Thing. . . 2006 Coleman Foundation White Paper on 

Entrepreneurship Education. 2006 Annual Meeting of US Association of 

Small Business and Entrepreneurship, January 13, 2006. 

Katz, Jerome A. “The Chronology and Intellectual Trajectory of American 

Entrepreneurship Education, 1876-1999.” Journal of Business Venturing

Vol. 18, 2003: 283-300. 

Kingsley, Gordon and Edward J. Malecki. “Networking for Competitiveness.” 

Small Business Economics Vol. 23, 2004: 71–84.

Kleiner, Brian, et al. Evaluation of The Appalachian Regional Commission’s 

Community Capacity-Building Projects. Washington, D.C.: Appalachian 

Regional Commission, July 2004. 

Kourilsky, Marilyn, and William B. Walstad. The E Generation. Dubuque, IA: 

Kendall-Hunt, 2000. 
124

Lambrecht, Johan and Fabrice Pirnay. “An Evaluation of Public Support 

Measures for Private External Consultancies to SMEs in the Walloon 

Region of Belgium.” Entrepreneurship and Regional Development Vol. 

17, 2005: 89-108.

Lee, Lena. Entrepreneurship Education: A Compendium of Related Issues. 

Working Paper. National University of Singapore Entrepreneurship Centre, 

2005. 

Lichtenstein, Gregg A. and Thomas S. Lyons. “The Entrepreneurial Development 

System: Transforming Business Talent and Community Economies.” 

Economic Development Quarterly Vol. 15, No. 1, 2001: 3-20. 

Lichtenstein, Gregg A. and Thomas S. Lyons, Incubating New Enterprises: A 

Guide to Successful Practice. Washington, DC: Aspen Institute, 1996. 

Low, Sarah. “Regional Asset Indicators: Entrepreneurship Breadth and Depth.” 

The Main Street Economist September 2004:1-4. 

Low, Sarah, Jason Henderson, and Stephan Weiler. “Gauging a Region’s 

Entrepreneurial Potential. “Economic Review Third Quarter, 2005. 

Edward Lowe Foundation, et al. Michigan: Toward an Entrepreneurial Economy, 

2005-2006. Cassopolis, MI; Edward Lowe Foundation, 2006. 

Malecki, Edward J. “Geographical Environments for Entrepreneurship.” 

International Journal of Entrepreneurship and Small Business

Forthcoming 2008. 

Manigart, Sophie, et al. “Determinants of Required Return in Venture Capital 

Investments: A Five Country Study.” Journal of Business Venturing Vol. 

16, No. 6, July 2002: 291-312. 

Maunula, Mari. The Perceived Value-Added of Venture Capital Investors. 

Discussion Paper No. 1030. The Research Institute of the Finnish 

Economy, March 2006. 

Markley, Deborah, et al. Rural Equity Capital Initiative Study of Nontraditional 

Venture Capital Institutions. Working Paper PB2001-11A-D. Rural Policy 

Research Institute, 2001. 

Markley, Deborah and Karen Dabson. Innovative Approaches to Entrepreneurial 

Development: Cases from the Northwest Region. RUPRI Center for Rural 

Entrepreneurship, 2006. 32-41. 
125

Markley, Deborah, Don Macke and Vicki Luther. Energizing Entrepreneurs: 

Charting a Course for Rural Communities. Lincoln, NE: Heartland Center 

for Leadership Development, 2005.

Markman, Gideon, et al. “Entrepreneurship and University-based Technology 

Transfer.” Journal of Business Venturing Vol. 20, No. 2, 2005: 241-263. 

McKernan, Signe-Mary and Henry Chen. Small Business and Microenterprise as 

an Opportunity and Asset-Building Strategy. Urban Institute Issue Brief 

No. 3. June 2005. 

McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.” 

Journal of Business Venturing Vol. 18, No. 2, 1987: 261-275. 

Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination. 

Improving Microenterprise Training and Technical Assistance: Findings for 

Program Managers. Washington, DC: Aspen Institute, 2002. 

Microenterprise Fund for Innovation, Effectiveness, Learning and Dissemination. 

Assessing the Effectiveness of Training and Technical Assistance. Field 

Forum Issue 1. Washington, DC: Aspen Institute, 1999. 

Minniti, Maria, with William Bygrave and Erkko Autio. Global Entrepreneurship 

Monitor 2005 Executive Report. London: London Business School, 2005. 

Molnar, Lawrence, et al. Impact of Incubator Investments. Athens, OH: National 

Business Incubation Association, 1997. 

Mountain Association for Community Economic Development. Accounting for 

Impact: Economic Development Spending in Kentucky. Berea, KY: 

Mountain Association for Community Economic Development, 2005.

Nakkula, Michael. Expanded Explorations into the Psychology of 

Entrepreneurship: Findings form the 2001-2002 Study of NFTE in two 

Boston Public High Schools. Working Paper. Harvard University Graduate 

School of Education, 2003. 

National Association of Development Organizations Research Foundation. 

Business Not as Usual: Regional Development Organizations Promote 

Rural Entrepreneurship. Washington, DC: NADO, 2002. 

The National Association of Seed and Venture Funds. Seed and Venture 

Capital: State Experiences and Options. NASVF White Paper. May 2006. 
126

National Commission on Entrepreneurship. American Formula For Growth. 

Washington, D.C.: National Commission on Entrepreneurship, October 

2002. 

National Community Reinvestment Coalition. Access to Capital and Credit for 

Small Businesses in Appalachia. Washington, DC: National Community 

Reinvestment Coalition, 2007. 

National Governors Association. A Governor’s Guide to Strengthening State 

Entrepreneurship Policy. Washington, DC: National Governors 

Association, 2004. 

Nexus Associates. A Continuing Record of Achievement: The Economic Impact 

of the Ben Franklin Technology Partners, Final Report. March 2003. 

North Carolina Rural Economic Development Center. Hello, My Business Name 

is W: A Guide to Building Entrepreneurial Networks in North Carolina. 

Raleigh, NC: North Carolina Rural Economic Development Center, 2007. 

Oklahoma Center for the Advancement of Science and Technology, Impact 

Report 2006. Oklahoma City, Oklahoma: OCAST, January 2006. 

Oldsman, Eric. Evaluation as an Effective Management Tool. Nexus Associates 

Inc., 2003. 

Oldsman, Eric. “Do Manufacturing Extension Programs Matter?” Research Policy

Vol. 25, No. 2, March 1996: 215-232. 

Organization for Economic Cooperation and Development. OECD Framework for 

the Evaluation of SME and Entrepreneurship Policies and Programmes. 

Paris: Organization for Economic Cooperation and Development, 

Forthcoming. 

Organization for Economic Cooperation and Development. Oslo Manual: 

Guidelines for Collecting and Interpreting Innovation Data Third Edition. 

Paris: Organization for Economic Cooperation and Development, 2005.

Organization for Economic Cooperation and Development. Entrepreneurship and 

Local Economic Development. Paris: Organization for Economic 

Cooperation and Development, 2003. 

Pages, Erik R., Doris Freedman, and Patrick Von Bargen. “Entrepreneurship as a 

State and Local Economic Development Strategy.” The Emergence of 

Entrepreneurship Policy: Governance, Start-Ups and Growth in the U.S. 

Knowledge Economy. Ed. David Hart. Cambridge, MA: Cambridge 

University Press, 2003. 
127

Pages, Erik R. and Shari Garmise. “The Power of Entrepreneurial Networks." The 

Economic Development Journal Vol. 2, No. 3, Summer 2003: 20-30. 

Pages, Erik R. and Kenneth A. Poole. “Entrepreneurship Promotion as an 

Economic Development Strategy: Next Steps in Institutionalizing the 

Field.” Applied Research in Economic Development Vol. 3, No. 2, 

December 2006: 10-27.

Phan, Phillip and Siegel, Donald. The Effectiveness of University Technology 

Transfer: Lessons Learned from Quantitative and Qualitative Research in 

the U.S. and the U.K. Rensselaer Working Papers in Economics. April 

2006. 

Pittaway, Luke, Maxine Robertson, Kamal Munir, and David Denyer. Networking 

and Innovation: A Systematic Review of the Evidence. University of 

Lancaster Institute for Entrepreneurship and Enterprise Development 

Working Paper 016. 2004. 

Reamer, Andrew, Larry Icerman, and Jan Youtie. Technology Transfer and 

Commercialization: Their Role in Economic Development. Washington, 

D.C.: US Economic Development Administration, August 2003. 

Reese, Laura and David Fasenfast. "Critical Perspectives on Local Development 

Policy Evaluations." Economic Development Quarterly Vol. 13, No. 1, 

1999: 3-7. 

Regional Technology Strategies, Inc. Evaluation of the Early Stages of the 

Appalachian Regional Commission’s Entrepreneurship Initiative. 

Washington, D.C.: Appalachian Regional Commission, December 2001. 

Reynolds, Paul, D.J. Storey, and P. Westhead. “Crossnational Comparisons of 

the Variation in New Firm Formation Rates.” Regional Studies Vol. 28, No. 

4, 1994: 443-456. 

Rice, Mark P. “Co-production of Business Assistance in Business Incubators: An 

Exploratory Study.” Journal of Business Venturing Vol. 17, No. 2, 2002: 

163-187. 

Robinson, Dennis and Zuoming Liu. A User Guide to the Socio-Economic 

Benefits Assessment System: A Rural Business-Cooperative Services 

Assessment Tool for Economic Development. Community Policy Analysis 

Center, University of Missouri-Columbia, December 2004. 

Romanelli, Elaine and C.B. Schoonhoven, eds. The Entrepreneurship Dynamic. 

Stanford, CA: Stanford University Press, 2001. 
128

Rosenfeld, Stuart A. “Networks and Clusters: The Yin and Yang of Rural 

Development.” Proceedings of Federal Reserve Bank of Kansas City 

Conference on Exploring Policy Options for a New Rural America. 

September 2001: 103-120. 

Sandler, David. The Effective Use of Tax Credits in State Venture Capital 

Programs. Ottawa: Canadian Tax Foundation, 2004.

Schlough, Charles and Streeter, Deborah. Cornell University’s Entrepreneurship 

Education and Outreach Program: An Evaluation and Proposal. 

Department of Agricultural, Resource, and Managerial Economics 

Working Paper. Cornell University, 1999. 

Shane, Scott. Academic Entrepreneurship: University Spin-offs and Wealth 

Creation. Cheltenham: Edward Elgar, 2004. 

Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and 

Implications Final Report submitted to USNet Partners. August 1998.

Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results and 

Implications – Final Report. Georgia Tech Policy Project on Industrial 

Modernization Working Paper 9805. 1998. 

Simpkins, Amy. “Community Development Venture Capital: Producing Results 

for Entrepreneurs, Investors, and Communities.” Bridges Vol. 10, 

November 2006. 

Smits, Ruud and Stefan Kuhlmann. “The Rise of Systemic Instruments in 

Innovation Policy.” International Journal of Foresight and Innovation

Policy. Vol. 1, Nos. 1/ 2, 2004. 

Solomon, George G., S. Duffy and A. Tarabishy. “The State of Entrepreneurship 

Education in the United States: A Nationwide Survey and Analysis.” 

International Journal of Entrepreneurship Education Vol. 1, No. 1, 2002: 

65-86.

Storey, David J. Six Steps to Heaven: Evaluating the Impact of Public Policies to 

Support Small Businesses in Developed Economies. Warwick Business 

School Small and Medium Sized Enterprise Centre Working Paper No. 59. 

University of Warwick, September 1998. 

Street, Christopher T. and Ann-Frances Cameron. “External Relationships and 

Small Business: A Review of Small Business Alliance and Network 

Research.” Journal of Small Business Management Vol. 45, No. 2, 2007: 

239-266. 
129

Taylor, Jill S. What Makes a Region Entrepreneurial? A Review of the Literature. 

Cleveland State University Center for Economic Development Monograph. 

September 2006. 

Tornatzky, Louis G. et al. Incubating Technology Businesses: A National 

Benchmarking Study. Athens, Ohio: National Business Incubation 

Association, 2003. 

Tornatzky, Louis G. et al. Innovation U.: New University Roles in a Knowledge 

Economy. Durham, NC: Southern Growth Policies Board, 2002.

United Kingdom Department of Trade and Industry. Innovation in the UK: 

Indicators and Insights. DTI Occasional Paper No. 6. July 2006. 

U.S. Small Business Administration Office of Advocacy and the Ewing Marion 

Kauffman Foundation. Entrepreneurship in the 21st Century. Conference 

Proceedings. March 26, 2004. 

Vesper, Karl and William Gartner. “Measuring Progress in Entrepreneurship 

Education.” Journal of Business Venturing Vol. 12, No. 5, May 1997: 403-

421. 

Welch, Doug, et al. Net Benefits: An Assessment of a Set of Manufacturing 

Business Networks and their Impacts on Member Companies. Report 

Prepared for USNet and Regional Technology Strategies, Inc. 1997. 

Westat. Evaluation of The Appalachian Regional Commission’s Community 

Capacity-Building Projects. Washington, D.C.: Appalachian Regional 

Commission, July 2004. 

White, Jesse. “Overview Panel Comments.” Proceedings of Federal Reserve 

Bank of Kansas City Conference on Beyond Agriculture: New Policies for 

Rural America. April 2000. 

Witt, Peter. “Entrepreneurs’ Networks and the Success of Start-Ups.” 

Entrepreneurship and Regional Development Vol. 16, 2004: 391-412. 

Zider, B. “How Venture Capital Works.” Harvard Business Review Vol. 76, No. 6, 

1998: 131-139.
130

APPENDIX A 

LITERATURE REVIEW 

A review of literature was completed for works involving evaluation of 

entrepreneurship initiatives in five areas – access to capital, entrepreneurial 

networks, entrepreneurship education, technical assistance (including incubator 

projects), and technology transfer. These five program areas correspond to the 

original five areas identified by ARC as requiring investment in order to create a 

more entrepreneurial region. The literature review included primarily works that 

reported on more comprehensive evaluations of entrepreneurship efforts. Not all 

of the publications cited in this report were considered as candidates for the 

literature review. A citation and brief summary are provided for each publication 

reviewed and the metrics used or suggested are highlighted. Works appear 

alphabetically within each program section. 

Access to Capital 

Colgan, Charles D. and Bruce H. Andrews. Evaluation of Maine Technology 

Institute Programs. Maine Institute for Technology and the University of 

Southern Maine Center for Business and Economic Research. 2004. 

Summary

This presentation evaluates the states investments in technology by looking at 

the impacts for institutional grant recipients, the economic impact, effects on 

company finances, intellectual property development, relationships cultivated, 

and quality of assistance programs. 

The evaluation finds that assistance recipients have had significant success in 

developing new products leading to intellectual property protection and that they 

are likely to have an economic impact on the state. Grant assistance has served 

as a much needed catalyst for external financing, and relationships have been 

cultivated to enhance the economic cluster networks in the state. 

Metrics Suggested

 Employment growth at assisted companies 

 Matching funds received (federal grants, external debt, external equity and 

grantee match) 

 Number of firms indicating new product for sale 

 Sources of firm revenue 

 Distribution of company sales by geography and technology sector 

 Expected raw materials purchased within the state 

 Sources and amount of debt and equity capital 

 Patent activity 

 Other intellectual protection activity 

 Number of company respondents that received assistance from the 

identified program (compared to other assistance programs). 
131

 Mean rating of organizations consulted and percent of respondents who 

claimed relationship was critical to success 

 Client assessments of interactions with program in relation to percent 

satisfaction to all state R&D assistance 

Community Development Financial Institutions Data Project. Providing 

Capital, Building Communities, Creating Impact. Fourth Edition. 

Washington, D.C. 2004. 10 November 2006 

<http://www.cdfi.org/downloads/CDP_fy2004_complete.pdf>. 

Summary 

Data from 517 Community Development Financial Institutions was analyzed to 

demonstrate the impact of CDFIs on emerging domestic markets throughout the 

U.S. Some of the results include that almost 7,000 businesses received 

assistance from CDFIs resulting in 28,330 jobs. The financial assistance served 

niche domestic markets that are underserved by traditional lending institutions 

and the transactions were prudent and effective. CDFIs were also shown to be 

flexible and timely enough to grow and change with the dynamics of the market 

and respond effectively to large-scale disasters such as hurricane Katrina. 

Metrics Suggested 

 Number of CDFIs 

 Total Assets 

 Average Assets 

 Total FTEs 

 Total Direct Financing Outstanding 

 Average Direct Financing Outstanding 

 Percent of Direct Financing Outstanding to specific sectors 

 Net Charge-Off Ratio 

 Delinquency Rate > 90 days 

 Delinquency Rate > 2 months 

 Average Capital 

 Percent of Debt from different types of institutions 

 Markets served: rural vs. urban and regional markets 

 Types of community betterment programs assisted, such as childcare, 

affordable housing, service organizations, educational slots, payday loan 

alternatives 

Felsenstein, Daniel and Aliza Fleischer. “Small-Scale Entrepreneurship 

and Access to Capital in Peripheral Locations: An Empirical Analysis.” 

Growth and Change 33 (2002):196-215. 

Summary

This paper analyses public assistance programs for small-scale entrepreneurship 

programs in rural areas. The authors use data from Israel to establish that 

lending institutions perceive a high risk when lending to areas where there is little 
132

information. The study has three findings: 1. Location matters in determining risk 

profile, 2. Location-oriented programs can improve information asymmetries that 

can be a risk factor and 3. These programs can create positive welfare effects. 

The study also asserts that there is speculation about the ability of information 

technology to increase visibility of small rural firms and enhancing information 

flow. 

Metrics Suggested

An estimation framework using 

 Net employment 

 Total employment 

 Monthly wage for an individual income groups 

 Revenue 

 Loan value 

 Guarantee Value 

Greene, F.J. and D.J. Storey. “An Assessment of Venture Capital Creation 

Programme: The Case of Shell LiveWIRE.” Entrepreneurship and Regional 

Development 16.2 (March 2004):145-159. 

Summary 

This paper suggests there are two areas when considering the problems inherent 

to assessing venture capital creation programmes: assessment is contingent 

upon the evaluation context and an input-output analysis is inadequate. The 

researchers create a new instrument to assess the value of these programs. 

They find that “soft” forms of support were of little value and that the more likely 

an individual was to engage in entrepreneurial activity the less likely they would 

be to seek the venture capital program’s services. 

Points to Consider for appropriate metrics

 What is the purpose of the evaluation? Evaluate the efficacy of the 

program or the internal efficiencies? 

 Be cognizant of the operational issues involved with a program. 

Manigart, Sophie et al. “Determinants of Required Return in Venture 

Capital Investments: A Five Country Study.” Journal of Business 

Venturing 16.6 (July 2002):291-312. 

Summary

The authors use two theoretical perspectives (resource theory and financial 

theory) to develop hypotheses about the determinants of the return required by 

venture capitalists. They test them on over 200 companies in 5 countries. They 

find that acquisition and buyout specialists require a significantly lower return 

than other venture capital companies (VCCs). Also, highly stage-diversified 

VCCs, independent VCCs, and VCCs providing more intensity of involvement all 

expect higher return rates. 
133

Metrics Suggested for Average Rate of Return for Venture Capital Companies

 36%-45% for early stage investments 

 26%-30% for expansion investments, acquisitions, buy-outs and other 

later stage categories 

 42% for early stage, stage specific rates 

 33% for later stage, stage specific 

National Association of Seed and Venture Funds. Seed and Venture 

Capital: State Experiences and Options. May 2006. 10 November 2006 

<http://www.nasvf.org/nasvf/web.nsf/fbaad5956b2928b086256efa005c5f78/1

412e8744c1c500c862572ad00019ab5/$FILE/Seed%20and%20Venture%20Ca

pital%20Report%20-%20Final.pdf>. 

Summary

The report includes a survey of the 50 states and their status in seed and venture 

capital. The report concludes there are 10 lessons learned. One of these 

lessons in that each state should develop a “System of Evaluation.” It claims that 

the best programs establish outcome measures from the beginning, keep track of 

the program results and evolve according to changing conditions. 

The report mentions that state funds can have different objectives. It notes that if 

funds have a more economic or social development target than on should not 

expect the same rate of return as private funds. 

The report establishes a starting point for more in-depth evaluation and design 

prospects for state venture capital funds. 

Metrics Suggested

 Number of jobs created or retained 

 Geographic reach 

 Industry reach 

Sandler, Daniel. “The Effective Use of Tax Credits in State Venture Capital 

Programs.” Tax Paper 108, Canadian Tax Foundation. 2004. 

Summary 

In his paper, Sandler states how the Venture Capital Industry in the U.S. is highly 

localized and for many states to encourage the geographic dispersal of venture 

funds, they are using tax credits. He offers several suggestions on how to 

evaluate these tax programs. 

Metrics Suggested 

 Economic growth in the state generated by the SMEs that are funded. 

 The amount of capital raised through the tax credits 
134

 All other capital the business raised 

 Number of new employees 

 Wage rates 

 Capital expenditures 
135

Entrepreneurial Networks 

Elfring, Tom and Willem Hulsink. “Networks in Entrepreneurship: The Case 

of High-technology Firms.” Small Business Economics 21.4 (2003):409-422. 

Summary 

The value of networks as an integral part of the explanation of entrepreneurial 

success is widely acknowledged. However, the network perspective does not 

specify the role of networks in the emergence and early growth of a venture. We 

have distinguished three entrepreneurial processes in new venture development, 

i.e. discovery of opportunities, securing resources, and obtaining legitimacy, 

which are of importance for survival and performance. This paper examines how 

these processes are influenced by strong and/or weak ties and whether the 

degree of innovation (incremental versus radical) acts as a contingency factor in 

the way network ties support entrepreneurial processes. In this explorative study 

three cases on high technology firms in The Netherlands provide empirical 

material enabling us to develop a number of propositions on the network effect, 

in particular the mix of strong and weak ties, on the three entrepreneurial 

processes. 

Metrics Suggested 

 Network effects

 Types of network ties

 Motives for getting involved in networks 

Kingsley, Gordon and Edward J. Malecki. “Networking for 

Competitiveness.” Small Business Economics 23 (2004):71–84. 

Summary 

A policy innovation that has achieved widespread diffusion across national and 

sub national governments in industrialized countries is the promotion of networks 

among small manufacturers as a means of promoting competitiveness. 

However, research and evaluations of formal networks formed in response to 

policy initiatives tend not account for the informal networks that small 

manufacturers routinely use in gathering information and business resources. 

This study examines the use of informal networks by 50 small manufacturing 

firms in rural and urban regions of northern Florida. The analysis is inductive and 

designed to provide a point of comparison to the growing literature on formal 

small manufacturing networks. Unlike formal networks, the links that comprise 

informal networks tend to be geographically and socially mixed. Small firms use 

informal networks to gather information on a mix of issues. Urban and rural firms 

have similar patterns of network use on issues affecting product development 

and competitiveness. But they have different patterns of network usage for 

issues associated with exporting and labor problems. 
136

Informal networks draw upon local and non-local information resources that do 

not require significant amounts of interpersonal contact across actors. Proximity 

is not a factor in the effective use of informal networks for information purposes. 

Rather the emphasis is upon locating “tried-and-true” solutions that solve the 

business needs. There is little evidence suggesting that the informal networks 

that these small manufacturers use are gravitating towards or seeking the 

development of formal networks. Thus, policy prescriptions identifying barriers to 

networking among small manufacturers are borne out in this study. 

Metrics Suggested 

 Business relationships between participating firms

 Needs of firms that prompted them to seek out networks (were those 

needs met?) 

Information sources used (did network participation change these?)

Pages, Erik and Shari Garmise. “The Power of Entrepreneurial 

Networking.” Economic Development Journal 2.3 (Summer 2003):20-30. 

Summary 

In 2000 the National Commission of Entrepreneurship helped convene a national 

series of focus groups to find out what makes a community entrepreneurial. 

They found that strong universities, access to venture capital, and good physical 

infrastructure matter, but so do soft, people-based assets. The opportunity to 

network with other entrepreneurs was an important factor in the success of 

entrepreneurs in regions. 

This article was designed to offer strategies for creating and nurturing networks. 

Metrics Suggested 

 Learning between individuals (rather than exclusively inter-firm) 

 Linkages created between entrepreneurs (brokering)

 Creation of common perspectives. 

 Cultural change 

 Creation of civic leaders 

 Branding of a region 

 Enhancing regional competitiveness 

 Networks spun off from parent network 

Regional Technology Strategies, Inc. Evaluation of the Early Stages of the 

Appalachian Regional Commission’s Entrepreneurship Initiative. A Report 

to the Appalachian Regional Commission. December 2001. 

Summary 

This is the evaluation of the Appalachian Regional Commission’s $17.6 million 

effort, begun in 1997, to bolster the entrepreneurial infrastructure of Appalachia. 
137

ARC awarded Regional Technology Strategies, Inc. (RTS) the contract to 

undertake the assessment and analysis based on a sample of 23 - 25 projects. 

To collect the necessary data, the research team (1) conducted a careful review 

of information and documentation supplied to the ARC by grant recipients, (2) 

conducted a survey of the sample of projects supplemented by selected 

telephone interviews with project staff, (3) conducted telephone interviews with 

randomly selected clients and partner organizations from among names 

submitted by projects, and (4) visited two project sites for a more in-depth 

perspective. 

The evaluation uses a variety of measures to assess outcomes and impacts on 

individuals and the local economy. The first set of measures assesses relative 

degrees of satisfaction of clients with the intervention. The second important set 

evaluates the economic results of the intervention on the client such as new 

enterprises started, new markets, or new products lines. The third captures the 

more general impacts on the economy in terms of jobs created by the new 

enterprises, growth potential of the enterprises, and potential sustainability of the 

program. 

Clients of the programs generally expressed high levels of satisfaction with the 

support they received. Three-quarters of projects reported businesses developed 

new products, 55 percent indicated that firms upgraded technologies or 

management methods, and half reported starting new businesses. 

About 52 percent of the sample projects reported creating jobs in existing firms 

and 39 percent reported saving jobs that would have otherwise been lost. Adults 

created 214 new firms—33 new firms with1 and 181 without employees. Based 

on those projects that were able to report hard numbers in the survey, 356 new 

jobs were created—54 in new firms, 121 in existing firms, and 181 jobs through 

self-employment. Another 85 jobs were saved from extinction in existing firms. In 

addition, surveyed projects reported 46 new businesses created by youth or 

students as class projects, some of which could become self-sustaining 

businesses after graduation. 

Metrics Suggested 

 Client levels of satisfaction 

 New product development in existing businesses 

 Technology or management method improvement in existing businesses 

 New businesses created 

 Jobs created 

 Jobs retained that would have been lost without the project 

 Businesses started by youths 

 Income of businesses created 
138

Rosenfeld, Stuart. “Networks and Clusters: The Yin and Yang of Rural 

Development.” Proceedings of Federal Reserve Bank of Kansas City 

Conference on Exploring Policy Options for a New Rural America. 

(September 2001):103-120. 

Summary 

Beginning in the mid-1980s, policymakers, particularly in rural areas, realized 

that they could no longer rely only on attracting large branch plants to sustain 

their economies. Increasing competition from newly developed and less 

developed low-wage nations was erasing their cost advantages. The more 

creative development agencies began to rechannel efforts towards stimulating 

entrepreneurial activity and strengthening indigenous businesses. Among the 

many policies discovered and promulgated by various experts and advocates in 

the late 1980s was interfirm collaboration, i.e., the widespread formation of 

formal and informal alliances, or networks, among groups of companies for 

mutual competitive advantage. 

The policy levers to convert networks from a practice considered by many to be 

uniquely linked to northern Italy’s social and business culture into a more 

universal practice were first formulated in Denmark in 1989. This approach to 

interfirm cooperation, designed in Denmark with advice from American consultant 

Richard Hatch, became the U.S. and international standard. With an allocation of 

$25 million from the Ministry of Trade and Industry, the scheme consisted mainly 

of training people, called brokers, to create networks and then offering groups of 

three or more companies sequentially phased grants for conceptualization, 

planning, and implementation. Eligible network activities included joint marketing, 

production, problem solving, research and development, and purchasing. 

The first and only, national effort to move the numbers of networks in the U.S. to 

a scale that could have impact began in 1993 under a grant from the National 

Institute for Standards and Technology to Regional Technology Strategies, Inc. 

The project, called USNet, was based on the documented value of networks to 

industrial modernization and technology adoption in Europe and involved 15 

state partners working together to build statewide network programs for SMEs. It 

relied heavily on the Danish model of training brokers and multipliers and on a 

process for informing companies and development organizations. But it 

depended on the individual partner states to provide the financial incentives. 

While not a rural development initiative per se, many of the early adopting areas 

were rural. 

Metrics Suggested

 Improved quality of products 

 New customers 

 New suppliers outside network 

 Increased sales 

 Increased profitability 
139

 Improved existing process 

 Improved relationships with customers 

 Adopted new technologies 

 Improved supplier quality 

 Savings by group purchasing or shared resources 

 Developed new product 

 Increased exports 

 Established new company 

**Also suggests that learning is a sufficient outcome for businesses to network.

Shapira, Philip. The Evaluation of USNet: Overview of Methods, Results 

and Implications. Final Report to USNet Partners. August 1998. 

Summary 

This report summarizes the aims, methods, and principal findings from the 

evaluation of USNet. The USNet project was a pilot initiative to build the capacity 

of its partners to promote inter-firm collaboration, with the ultimate aim of 

enhancing the competitiveness of small and mid-sized manufacturing 

enterprises. 

The evaluation discusses the findings of the major USNet evaluation studies 

undertaken by members of the evaluation team. The principal findings of the 

evaluation are: 

 Firms who collaborate in inter-firm networks report positive net benefits, 

while greatest private impacts are associated with strong industry 

leadership of networks 

 USNet’s original network promotion goals were too ambitious, given the 

resources available; judged against more realistic expectations, USNet 

has performed well 

 USNet’s training programs have generated widespread awareness about 

inter-firm collaboration 

 USNet special projects demonstrate the value of explicit follow-on 

initiatives to promote inter-firm collaboration 

 USNet policy and organizational impacts at the state level were modest 

 Federal support can strengthen efforts to promote inter-firm collaboration 

at the state level and aid shared learning 

Metrics Suggested 

 Impacts of networks on states-number of firms involved in networks, 

attitudes of organizations toward inter-firm collaboration, learning new 

information, making new contacts, learned about other states’ practices, 

getting ideas to resolve problems, becoming aware of tools. 

 Net economic benefits accruing to participating firms 

 Generating awareness of inter-firm collaboration 
140

Welch, Doug, et al. “Net benefits: An Assessment of a Set of Manufacturing 

Business Networks and their Impacts on Member Companies.” USNet 

Evaluation Working Paper 9701. 1997. 

Summary 

During the 1990s, a growing number of US companies became involved in 

collaborative interfirm partnerships, flexible business networks and other 

organized collaborative efforts to aid business performance. Drawing on survey 

data from industrial companies and network brokers in five US states, this study 

identifies and measures a range of hard and soft impacts on firms resulting from 

their participation in interfirm networks. The measures include effects on firms’ 

activities, business strategy, relationships, trust, confidence, technology use, 

know-how, employment and economic benefits and costs. 

USNet, a federal-state initiative to strengthen interfirm collaboration, sponsored 

this study. In order to assess the impacts of network participation on member 

companies, two types of surveys were administered at the beginning of 1997, the 

“1997 National Benchmark Survey of Industrial Network Companies” and the 

“Survey of Network Presidents/Coordinators.” This study presents findings from 

descriptive and comparative analyses of data from these surveys. 

A total of ninety-nine members of 13 separate business networks responded to 

the survey. Principal findings from this dual survey effort include the following: 

 Most of the thirteen surveyed networks are young, urban organizations 

with limited staff resources.

 With a few exceptions, the network members responding to the survey 

were small manufacturers. 

 The networks’ most common primary objective is information sharing. 

 Companies of different sizes report different network activities. 

 Overall, most respondents are satisfied with the networks. Thirty-one 

 Most companies report positive effects to date and expect even larger 

future effects.

 The average total net benefits per firm of network participation are 

positive. 

 On average, companies experienced a net increase in their employment 

levels as a result of network participation.

 A few more ‘intensive’ network activities are associated with stronger 

overall impacts.

 Companies that share sales leads report higher net benefits of network 

participation.

 Companies that have been in networks longer are more likely to report 

sharing technical capabilities with other network members.

Metrics Suggested 

 Network objectives 

 Differential network activities depending on client characteristics 
141

 Satisfaction levels 

 Client experiences of positive effects on strength of positive effects 

 Total net benefit per firm ($) 

 Change in employment levels of participating companies 

 Intensity of network activities 

 Improvement in employee and management skills due to network 

participation 

 Knowledge sharing with other companies 
142

Entrepreneurship Education 

Charney, Alberta and Gary Libecap. The Impact of Entrepreneurship 

Education: An Evaluation of the Berger Entrepreneurship Program at the 

University of Arizona, 1985-1999. Final Report to Kauffman Center for 

Entrepreneurial Leadership. May 2000. 

Summary 

The report details the results of a series of surveys on the effect that the Berger 

Entrepreneurship Program at the University of Arizona has had on its graduates, 

the university’s technology transfer program, financial contributions to the college 

of business, and the degree to which entrepreneurship has been incorporated 

into curriculums in other departments. The Berger Program was a good 

candidate for such an analysis because it had been in existence for sixteen years 

at the time of the report. A sample of program alumni and other university 

graduates were surveyed to assess the program’s effect on students. According 

to the results, program graduates were three times more likely to start a new 

business than non-entrepreneurship graduates. Additionally, program graduates’ 

average annual income was found to be twenty-seven percent higher than the 

average annual income of general university graduates. However, there was 

little evidence that program participation led to higher levels of job satisfaction. 

With regard to technology transfer, the report concludes that entrepreneurship 

education promotes technology transfer as graduates were more likely to be 

involved with firms that use licensed technologies or with firms that license 

technologies to others. A separate survey of University of Arizona administrators 

revealed that entrepreneurship education had reportedly led to increased 

financial contributions to the university and respondents felt that other 

curriculums had been enriched due to pedagogical innovations in the 

entrepreneurship program. 

Metrics Suggested 

 Number of business ventures started 

 Number of workers employed in venture 

 Amount of sales/revenue 

 Graduate job satisfaction 

 Type of venture started 

 Whether or not venture is “high-tech” 

Fayolle, Alain. “Evaluation of Entrepreneurship Education: Behavior 

Performing or Intention Increasing?” International Journal of 

Entrepreneurship and Small Business 2.1 (2005):89-98. 

Summary

Fayolle says there is a growing interest around the question of how to evaluate 

entrepreneurship education programs. He then suggests that the standard 

metrics such as direct and indirect job creation are insufficient measuring sticks. 
143

Fayolle points out that there are significant timing issues associated with using 

job creation as the primary criteria, i.e. he says the entrepreneurial process is a 

nonlinear one and there is no definitive way to know at what point employment 

opportunities may be created. Accordingly, he advocates for the inclusion of 

some more intangible metrics. Fayolle believes that entrepreneurship programs 

should also be evaluated on the attitudes, mindsets, and intentions of students. 

He also discusses how different teaching strategies can influence student 

behavior. Fayolle notes that entrepreneurship training may enable students to 

start their own enterprise, but it can at the same time dent their desire to do so – 

consequently, teaching style matters. In essence, Fayolle says that because 

entrepreneurship is so connected to personal desire and motivation that it is very 

important to evaluate the student psyche throughout the educational process. 

Bottom line being that evaluating programs on job creation alone is difficult and 

incomplete; therefore, that analysis should be supplemented with some 

assessment of how the program nurtures the entrepreneurial spirit. 

Metrics Suggested

 Teaching strategies 

 Student attitudes and intentions throughout the process 

 An evaluation of overall program goal compared to student goals 

 Knowledge acquisition 

Nakkula, Michael. “Expanded Explorations into the Psychology of 

Entrepreneurship: Findings from the 2001-2002 Study of the NFTE in two 

Boston Public High Schools.” Working Paper, Harvard University Graduate 

School of Education. 2003. 

Summary 

This reports details the results of a National Foundation for Teaching 

Entrepreneurship (NFTE) program that was taught in two Boston public schools 

during the 2001-2002 school year. In the Boston public school system every 

student is required to take a school to career “pathway” class that forces them to 

think about their plans after graduation. In concert with that, a study was set up 

where some students were enrolled in a special “pathway” class sponsored by 

NFTE. Students were assessed at the beginning of their “pathway” classes and 

then again upon completion regarding questions like desire to go to college. 

Students taking NFTE’s entrepreneurship “pathway” course were found to have 

experienced considerably higher increases in college interest and occupational 

aspirations over the duration of the class than students enrolled in non-NFTE 

“pathway” courses. 

Metrics Suggested 

 Level of college interest, pre and posttest 

 Occupational interest, pre and posttest 

 Overall school engagement 

 Amount of independent reading 
144

Schlough, Charles and Deborah Streeter. “Cornell University’s 

Entrepreneurship Education and Outreach Program: An Evaluation and 

Proposal.” Working Paper, Department of Agricultural, Resource, and 

Managerial Economics, Cornell University. 1999. 

Summary 

The first part of the report evaluates Cornell’s Entrepreneurship Education and 

Outreach (EEO) program. The authors discuss the challenges that the program 

has encountered during its first two years including a lack of widespread 

cooperative support from state and local-level stakeholders. Consequently, the 

second part of the report is a proposal to develop a statewide network of 

supportive partnerships in order to improve Cornell’s entrepreneurship education 

mission. Included in that plan is a set of “criteria for success” and a 

corresponding list of metrics. The authors suggest evaluating the proposed 

program by measuring the number of small businesses started by graduates, the 

number of jobs created within those ventures, the number of decisions by 

graduates to not proceed with a business concept, and annual sustained 

enrollment levels in program courses. 

Metrics Suggested 

 Number of businesses started 

 Number of people employed within those businesses 

 Number of decisions by graduates to not pursue a business concept 

 Annual sustained enrollment in program courses 

Soloman, G.T., et al. “The State of Entrepreneurship Education in the 

United States: A Nationwide Survey and Analysis.” International Journal of 

Entrepreneurship Education 1.1 (2002):65-86. 

Summary 

This paper offers an assessment of entrepreneurship education both 

domestically and abroad as of 2000. Included in the article is a brief literature 

review on entrepreneurship education. The following is a summary of two 

articles discussed in that literature review for which full-text versions were not 

readily available. Citations for both works are listed below. 

Block and Stumpf (1992) and McMullan and Long (1987) both assert that 

traditional measures of program effectiveness such as number of graduates are 

insufficient indicators of success. Instead they agree that programs should be 

evaluated according to their socioeconomic impacts. Accordingly, Block and 

Stumpf (1992) suggest using number of jobs created by graduates and overall 

job satisfaction as relevant measures. Similarly, McMullan and Long (1987) 

advocate assessing the number, type, and growth of companies created by 

entrepreneurship graduates as a better gauge of success. 
145

Metrics Suggested 

 Number of jobs created 

 Level of job satisfaction 

 Number of businesses created 

 Types of companies created 

 Growth of companies created 

Sources 

Block, Z. and S. A. Stumpf. “Entrepreneurship Education Research: Experience 

and Challenge.” The State of the Art of Entrepreneurship. Eds. D. L. Sexton and 

J. D. Kasarda. Boston: PWS-Kent Publishing S.A., 1999. 17-45. 

McMullan, W. E. and W.A. Long. “Entrepreneurship Education in the Nineties.” 

Journal of Business Venturing 2 (1987):261-275. 

Vesper, Karl and William Gartner. “Measuring Progress in 

Entrepreneurship Education.” Journal of Business Venturing 12 

(September 1997): 403-421. 

Summary 

In this article, Vesper and Gartner report on the results of a 1994 survey which 

attempted to rank university-level entrepreneurship education programs. The 

survey was mailed to 941 business school deans both domestically and abroad. 

Respondents were asked to rank the top programs and to list the most important 

criteria considered when doing so. Among the possible 18 criterion listed, 

courses offered was ranked number one followed by faculty publications, impact 

on community, exploits of alumni, and innovations. Faculty start-ups and location 

were listed as the bottom two. However, Vesper and Gartner quickly caution the 

reader about the results of the survey and similar rankings in general. The 

authors state that the survey did not tie the program rank to specific criteria nor 

did it evaluate respondent knowledge of the other programs in question. With 

respect to other popular rankings, for example, program ratings published by 

Business Week, Vesper and Gartner argue that traditional metrics used in those 

rankings – GMAT scores, computers per capita, etc. – may not be appropriate 

predictors of entrepreneurial success. Instead, they suggest employing the set of 

criteria used in awarding the Malcolm Baldrige National Quality Award (MBNQA), 

a well-established quality improvement program. Using the MBNQA framework, 

Vesper and Gartner stress the importance of evaluating entrepreneurship 

programs based on the following seven factors: leadership, information and 

analysis, strategic and operational planning, human resource development and 

management, educational and business process management, school 

performance results, and student focus and student and stakeholder satisfaction. 

Metrics Suggested 

 Student performance (in specific classes or as demonstrated through a 

portfolio) 
146

 Student improvement throughout program 

 Student satisfaction 

 Impact on the community (number of start-ups, students employed in new 

firms, students working in positions assisting new firms) 
147

Technical Assistance 

Aernoudt, Rudy. “Incubators: Tool for Entrepreneurship?” Small Business 

Economics 23.2 (2004):127-135. 

Summary 

This paper examines U.S. and European experiences with business incubators 

and stresses the need for accurate evaluations of their impact relative to their 

different types. The author finds that lack of entrepreneurship and the 

underdevelopment of seed financing and business angel networks are some of 

the biggest barriers to success. He asserts that seed financing, links with 

business angels and business angel networks as well as involvement in second 

round financing and IPO assistance should be integrated into the business 

incubation concept. 

Metrics Suggested 

 Survival rate 

 Tenants by incubator 

 Employment by tenants 

 Employment created by graduates 

 Graduates remaining in the community 

Brown, J. David, John S. Earle and Dana Lup. “What Makes Small Firms 

Grow? Finance, Human Capital, Technical Assistance, and the Business 

Environment in Romania.” Economic Development and Cultural Change 54 

(October 2005):33-70. 

Summary 

This paper sought to explore new ground by looking at the policy-relevant factors 

that may stimulate or hinder small start-up companies, and fill the need for 

quantitative studies using panel data to analyze statistical relationships between 

firm growth and objective measures of factors related to policies. They found 

that availability of loans is a factor while internal finance and trade credit tend to 

be unimportant. They also found that high school education raises growth but 

university education and worker training are not necessarily a factor. Technical 

assistance was also considered a weak factor for small business growth 

success. 

Metrics Suggested

 Retained earnings, 

 conventional bank lending, 

 informal credit markets, 

 tax credits offered by the state 

 Membership in business association 

 Membership in consultancy programs 
148

 Rate of usefulness of assistance programs 

Community Development Financial Institutions Data Project. Providing 

Capital, Building Communities, Creating Impact. Fourth Edition. 

Washington, D.C. 2004. 10 November 2006 

<http://www.cdfi.org/downloads/CDP_fy2004_complete.pdf>. 

Summary 

Data from 517 Community Development Financial Institutions was analyzed to 

demonstrate the impact of CDFIs on emerging domestic markets throughout the 

U.S. Some of the results include that almost 7,000 businesses received 

assistance from CDFIs resulting in 28,330 jobs. The financial assistance served 

niche domestic markets that are underserved by traditional lending institutions 

and the transactions were prudent and effective. CDFIs were also shown to be 

flexible and timely enough to grow and change with the dynamics of the market 

and respond effectively to large-scale disasters such as hurricane Katrina. 

Metrics Suggested 

 Number of people receiving group-based training 

 Number of people receiving one on one technical assistance 

 Number of organizations receiving training 

 Number of jobs created overall from the project 

 Number of business receiving training 

Enterprise Corporation for the Delta. Enterprise Corporation for the Delta 

Program Monitoring Report—Business Technical Assistance. 2003. 10 

November 2006 

<http://www.ecd.org/Documents/Evaluation/2003TAMonitoringReport.pdf>. 

Summary

This report provides baseline information about the Enterprise Corporation for the 

Delta’s technical Assistance activities associated with their Community 

Development Financial Institution. This baseline is established to measure the 

progress of their efforts to address the non-financial needs of potential and 

existing customers and lay the groundwork for more in-depth analysis in the 

future. 

The ECD has three technical assistance activities: FastTrac entrepreneurial 

training, brokered TA and Business LINC mentor/protégé program. ECD learned 

that very few graduates from FastTrac attained loan financing, but many 

graduates demonstrated more sound decision making. The Brokered TA mostly 

focused on establishing the foundation for future analysis. The mentoring 

program of the ECD found that creating a mentoring environment for small startups works, but the conditions for its success is very different than many urban 

programs mostly because there are not large established companies to become 

a cadre of mentors. They also found that protégés were more successful if they 
149

had several years of experience and if they were in close proximity to their 

mentor. 

Metrics Suggested

 Number of mentor relationships established 

 Finance seeking intentions of graduates, before and after course 

completion 

 Number of technical assistance engagements 

 Types of TA services requested 

 Loan ratings of TA graduates and loan ratings of control group 

 Satisfaction level with mentor programs or other TA initiatives. 

Greenburg, Elizabeth and Richard Reeder. “Who Benefits from Business 

Assistance Programs? Results of the ERS Rural Manufacturing Survey.” 

Agriculture Information Bulletin Number 736-04. United States Department 

of Agriculture. 1998. 

Summary

The authors sought to find out how much government programs helped rural 

manufacturers and who exactly is benefiting from these programs. They 

discovered that over 60 percent of manufacturing establishments benefited from 

the programs and 28 percent of these firms found these programs to be very 

important to their operations over the last three years. They discovered that 

manufacturers using advanced technologies benefited more than other 

manufacturers. Also, large businesses were more likely to benefit than small 

ones, although small firms seemed to have more problems and benefit the most 

once assistance was administered. 

Business assistance programs were identified as: tax incentives, loans (direct, 

indirect/guaranteed, and revolving), industrial parks and enterprise zones, and 

training and technical assistance. 

Manufacturing establishments were characterized by the following: metro/non 

metro, geographic region in the U.S., employment size, type (branch plant and 

high tech), and distressed (high poverty rates, high unemployment rates and 

population loss). 

The study further analyzed which types of firms benefited from the specific types 

of assistance programs. State and local tax breaks benefited the largest 

proportion of nonmetro establishments (46 percent), training and technical 

assistance (29 percent), industrial parks/enterprise zones (21 percent), direct 

loans (15 percent), guaranteed loans (13 percent), revolving loan funds (9 

percent). 

Metrics Suggested
150

 Survey of how important each assistance program was to their business. 

This metric can be cross-analyzed by 

o Geographic location 

o Size 

o Type of business 

o Type of distressed area 

o Rural/Urban 

Lambrecht, Johan and Fabrice Pinray. “An Evaluation of Public Support 

Measures for Private External Consultancies to SMEs in the Walloon 

Region of Belgium.” Entrepreneurship and Regional Development. 17.2 

(March 2005):89-108. 

Summary 

This paper evaluated public support measures for private consultancies to SMEs 

in the Walloon region of Belgium. It presents an analysis of the supply and 

demand, an evaluation of the efficiency and the effectiveness of policy measures, 

and real policy recommendations. The paper recommends that the real needs of 

the entrepreneur and of the SME determine the publicly financed advisory 

process. It also recommends a “one stop shop” for private external consultants 

that help SMEs. 

Metrics Suggested

 Profit 

 Sales 

 Market Share 

 Employment 

Microenterprise Fund for Innovation, Effectiveness, Learning and 

Dissemination (FIELD). Improving Microenterprise Training and Technical 

Assistance: Findings for Program Managers. 2002. 10 November 2006 

<http://www.fieldus.org/Publications/improvingmicro.pdf>. 

One of the first tasks FIELD set for itself and for the organizations who received 

awards from their RFPs was to answer “What makes for effective training and 

technical assistance?” In support of this question they also asked “What are the 

appropriate indicators, which intermediate measures are better indicators of 

financial impact on clients, and what practical approaches can programs used to 

document and track outcomes? 

They found that many entrepreneurs sustained and grew their businesses, while 

some other businesses floundered. This was mostly due to family or personal 

reasons. The analysis found that effective training programs acknowledge the 

importance of client readiness for business and offer a range of services to help 

clients meet these needs. It also found that adult learning theory is an effective 

method for designing and delivering training. Training must include soft skills 
151

and basic competencies as well as key financial and marketing skills. Finally, 

FIELD learned TA should be offered within a structure that keeps clients 

connected to a larger program, but places the initiative on the client to receive 

services. 

Metrics Suggested

 Completion of Training 

 Development of Business Plan 

 Progress in Seeking Business Financing 

 Business Starts 

 Business Expansions 

 Business Stabilizations 

 Is there an improvement in income reporting 

 Increase in business assets 

 Increase in personal assets, such as cars, savings, homeownership. 

Microenterprise Fund for Innovation, effectiveness, Learning and 

Dissemination. “Assessing the Effectiveness of Training and Technical 

Assistance.” FIELD Forum Issue 1. 1999. 10 November 2006 

<http://www.fieldus.org/Publications/Field_Forum1.pdf>. 

Summary 

FIELD recognizes that training and technical assistance to small businesses has 

a significant impact on their success, but there is little data to substantiate this to 

policymakers and funders. Therefore they sought to identify models and other 

metrics to show what makes for effective training and how that equates to lower 

costs. FIELD asked practioners to propose strategies and identify indicators that 

could establish a link between service and outcomes. Some intermediate and 

final outcomes were suggested. 

Suggested Metrics

 Course completion and graduation 

 Pre and post knowledge testing 

 Business plan completion 

 Satisfaction 

 Hours of Training and TA 

 Sequence of Training and TA 

 Business Skills acquired 

 Personal effectiveness skills 

 Established networks/relationships 

 Business start-ups/survivals 

 Sales and profit/loss 

 Number of employees 

 Employee wages and benefits 

 Household income assets and net worth 

 Change in public assistance 
152

Oldsman, Eric. “Evaluation as an Effective Management Tool.” Nexus 

Associates, Inc. 2003. 10 November 2006 <http://www.nexusassociates.com/hanoi.pdf>. 

Summary

This paper critiques the Performance Measurement Framework established by 

the Committee of Donor Agencies for Small Enterprise Development to learn 

more about the performance of business development services. The paper 

suggests that the PMF framework can be useful as a tool for managers to 

improve their performance but considerable caution should be taken when using 

common performance standards for such diverse programs. The author 

recommends that evaluations be grounded in explicit theories of the particular 

initiatives. 

Things to consider when developing metrics

 Characterize conditions within markets to learn more about their structure 

and performance 

 Determine specific needs within a target population of firms in order to 

design new programs 

 Establish whether existing programs are being implemented as intended 

 Find out whether existing programs are achieving their objectives 

 Compare existing programs to judge the relative merits of different 

approaches to addressing specific needs. 

 Examine operations in great detail (aggregate data can mask a lot). 

Oldsman, Eric. “Do Manufacturing Extension Programs Matter?” 

Research Policy 25.2 (March 1996):215-232. 

Summary

Based on his evaluation of the New York based Industrial Technology Extension 

Service Program, Oldsman finds that manufacturing extension programs can 

have a favorable impact on participating companies. Because of the expertise of 

field agents, firms have been able to reduce costs, particularly with respect to 

direct and indirect labor, and in some instances increase revenue. 

He found that MEP programs should be designed to focus on adding value rather 

than cutting costs; pay attention to direct, long-term assistance; and foster 

cooperation to compensate for the lack of internal economies of scale. 

Metrics Suggested

 Refining layout of operations 

 Purchased or developed new software 

 Ask entrepreneurs if they had not received assistance would they have 

stayed in operation and in the state? 

 Cost savings 
153

 Direct labor productivity 

 Reduction in inventory 

 Reductions in manufacturing lead time 

 Reductions in direct labor costs per unit 

 Reductions in material costs per unit 

 Reductions in energy costs per unit 

 Reductions in indirect labor costs per unit 

 Reductions in other overhead costs 
154

Technology Transfer

Association of University Technology Managers. AUTM U.S. Licensing 

Survey: FY 2004 Survey Summary. 2004. 10 November 2006 

<http://www.autm.net/events/File/Surveys/03_Abridged_Survey.pdf>. 

Summary 

This year’s Licensing Survey shows a continued steady growth in the 6 percent 

range for most of the performance measures that are considered meaningful 

indicators within the profession: 

 Products available to the public 

 Invention disclosures received 

 Licenses and options executed 

 Licenses and options active 

 Licenses and options generating income 

 Licenses and options generating running royalties 

 Net income 

One or two important performance measures, specifically U.S. patents issued, 

though down from fiscal year 2003, appeared to be consistent with long-term 

growth trends. However, the most dramatic results were the clear evidence of a 

recovery from the very difficult market conditions for new company startups 

reported in the fiscal years 2002 and 2003 Licensing Surveys. Institutions 

launched 23.5 percent more new startups in fiscal year 2004 than in fiscal year 

2003, and the number of existing startup companies that went out of business 

declined more than 30 percent. 

The new startup company activity reflects the changed circumstances in capital 

markets. The second half of 2003 will be remembered as the end of the venture 

industry’s hemorrhaging that followed the nearly simultaneous collapse of the ecommerce, telecommunications and biotechnology markets. 

The second half of 2003 also saw the first revitalization of the initial public 

offering market since 2000. The number of venture-backed IPOs began 

increasing in the second half of 2003, with 20 of the 22 venture backed IPOs for 

2003 occurring in the third and fourth quarters of the year. The first two quarters 

of 2004 saw 34 venture-backed IPOs, and the year ended with 67 venturebacked IPOs raising $4.98 billion vs. the $1.4 billion raised in 2003. 

Metrics Suggested 

 Products available to the public 

 Invention disclosures received 

 Licenses and options executed 

 Licenses and options active 

 Licenses and options generating income 

 Licenses and options generating running royalties 
155

 Net income 

 New Startups 

 Number of startups going out of business 

 University equity interests in their startups 

 IPOs 

Audretsch, David B., Taylor Aldridge, and Alexander Oettl. The Knowledge 

Filter and Economic Growth: The Role of Scientist Entrepreneurship. 

Ewing Marion Kauffman Foundation. 2006. 10 November 2006 

<http://www.kauffman.org/pdf/scientist_entrepreneurs_audretsch.pdf>. 

Summary 

This study examines the prevalence and determinants of the commercialization 

of research by the top twenty percent of university scientists funded by grants 

from the National Cancer Institute (NCI). Because the two publicly available 

modes of scientist commercialization – patents and Small Business Innovation 

Research (SBIR) grants – do not cover the full spectrum of commercializing 

activities undertaken by university scientists, the study also includes two 

additional measures obtained from detailed scientist interviews: licensing of 

intellectual property and starting a new firm. These measures are used to assess 

both the prevalence and determinants of scientist commercialization of research. 

In particular, two distinct routes for commercializing scientist research are 

identified, the Technology Transfer Office (TTO) route and the entrepreneurial 

route, which does not involve assigning a patent to the university. This study in 

no way provides an assessment or judgment about the efficacy of the TTO. 

Rather, this study highlights the extent to which additional commercialization of 

research takes place, suggesting that the contribution of universities to U.S. 

innovation and ultimately economic growth may be greater than had previously 

been believed. Relevant findings include: 

 Two paths for commercialization of scientist research are identified - the 

TTO route and the entrepreneurial route. Scientists who select the TTO 

route by commercializing their research through assigning all patents to 

their university TTO account for 70 percent of NCI patenting scientists. 

Scientists who choose the entrepreneurial route to commercialize their 

research, in that they do not assign patents to their university TTO, 

comprise 30 percent of patenting NCI scientists. 

 Social capital enhances the propensity for scientists to commercialize their 

research. The impact of social capital is particularly high for the 

commercialization mode of scientist entrepreneurship. 

 Scientists choosing the entrepreneurial route to commercialize their 

research, by not assigning patents to their university to commercialize 

research, tend to rely on the commercialization mode of entrepreneurship. 

By contrast, scientists who select the TTO route by assigning their patents 

to the university tend to rely on the commercialization mode of licensing. 

Metrics suggested 
156

 Research grants to university researchers (i.e., SBIR) 

 Patents 

 Licenses of intellectual property 

 University researchers starting new firms 

 Route by which commercialization happens- entrepreneurial or through 

tech transfer office 

Chukumba, Celestine and Richard Jensen. “University Invention, 

Entrepreneurship, and Start-Ups.” National Bureau of Economic Research 

Working Paper No. 11475. July 2005. 

Summary 

This study examines the commercialization of university inventions in licensing to 

both start-up firms and established firms, and seek to determine when licensing 

to start-ups is more likely. They construct a theoretical model that predicts startups are more likely if their opportunity cost of development and 

commercialization is lower or if the technology transfer officer’s (TTO) opportunity 

cost of searching for a partner among established firms is higher. Using data 

from the Association of University Technology Managers, the National Venture 

Capital Association Yearbook, and the National Research Council, the study 

finds that inventor quality and measures of past TTO success (age, the number 

of disclosures, gross royalties) are all positively and significantly related to the 

number of licenses to both start-ups and established firms. However, it also finds 

that start-up activity is positively and significantly related to the S&P 500, but 

negatively and significantly related to the interest rate and rate of return to 

venture capital. 

Metrics suggested 

 Licensing to start-ups and existing firms 

 Start-ups created 

 Partnering with outside entities by tech transfer offices 

 Venture capital secured by start-ups 

Markman, Gideon, et al. “Entrepreneurship and University-based 

Technology Transfer.” Journal of Business Venturing 20.2 (2005):241-263. 

Summary 

The success of business incubators and technology parks in university settings is 

often determined by how well technology is transferred from the labs to their 

startup firms. University technology transfer offices (UTTOs) function as 

‘‘technology intermediaries’’ in fulfilling this role. This article builds a framework to 

address two questions: (a) Which UTTOs’ structures and licensing strategies are 

most conducive to new venture formation; and (b) how are the various UTTOs’ 

structures and licensing strategies correlated with each other. The findings reveal 

a complex set of relationships between UTTO structure and strategies, new 

venture formation, and business incubation. 
157

Based on interviews with 128 UTTO directors, findings show that whereas forprofit UTTO structures are positively related to new venture formation, traditional 

university and nonprofit UTTO structures are more likely to correlate with the 

presence of university-based business incubators. Licensing-for equity strategy is 

positively related to new venture formation while sponsored research licensing 

strategy is negatively related. The licensing-for-cash strategy, the most prevalent 

transfer strategy, is least correlated to new venture formation. A content analysis 

of UTTO mission statements also revealed an overemphasis on royalty income 

and an under emphasis on entrepreneurship. 

Metrics Suggested 

 Structure of tech transfer office (for- or non-profit) 

 Incubators 

 Technology parks 

 Startup firms 

 Business incubation 

 Licensing 

 University equity shares in startups 

 Applied v. basic research at university where tech transfer office is located 

Oklahoma Center for the Advancement of Science and Technology. Impact 

Report 2006. Oklahoma City, Oklahoma. January 2006. 10 November 2006 

<http://www.ocast.state.ok.us/Portals/0/docs/brochures/2006-

ImpactReport.pdf>. 

Summary 

The report summarized OCAST’s program impacts in terms of award amounts 

and leveraged private and federal funds. It also gives some detail about each 

program, including the Oklahoma Technology Commercialization Center, whose 

progress is measured in terms of facilitation of capital acquisition, jobs created, 

and companies served. 

Metrics suggested 

 Amount of capital acquisition facilitated

 Jobs created

 Number of technology companies served

Palmintera, Diane. Accelerating Economic Development Through 

University Technology Transfer. Reston, VA: Innovation Associates Inc., 

February 2005. 

Summary 

This report highlights models of university tech transfer and commercialization, 

related efforts like entrepreneurship programs, and the infrastructure and 

environment needed to support commercialization efforts. It includes case 
158

studies of university-based tech transfer and related economic development 

initiatives that lay the groundwork for state, university, and corporate actions to 

leverage university resources. 

Practices in tech transfer at 10 universities were examined, along with related 

entrepreneurship programs and other programs. These case studies were 

analyzed to extract best practices and recommendations. 

Suggested Metrics 

 Corporate sponsored research

 Levels of government funding

 Seed capital and source (i.e., university-created funds or private funds)

 Innovation centers

 Number of start-ups created

 Number of start-ups assisted

 Incubators

 Research Parks

 Employment

Phan, Phillip and Donald Siegel. “The Effectiveness of University 

Technology Transfer: Lessons Learned from Quantitative and Qualitative 

Research in the U.S. and the U.K.” Rensselaer Working Papers in 

Economics Number 0609. Rensselaer Polytechnic Institute. April 2006. 

Summary 

In recent years, there have been numerous studies of the effectiveness of 

university technology transfer. Such technology transfer mechanisms include 

licensing agreements between the university and private firms, science parks, 

incubators, and university-based startups. This study reviews and synthesizes 

these papers and presents some recommendations on how to enhance 

effectiveness. Implementation of these recommendations will depend on the 

mechanisms that universities choose to stress, based on their technology 

transfer “strategy.” For example, institutions that emphasize the entrepreneurial 

dimension of technology transfer must address skill deficiencies in technology 

transfer offices, reward systems that are inconsistent with enhanced 

entrepreneurial activity and the lack of training for faculty members, post-docs, 

and graduate students in starting new ventures or interacting with entrepreneurs. 

Metrics Suggested 

 Licensing

 Science parks

 Incubators

 University-based startups

 Training provided to employees in working with entrepreneurs/start ups
159

APPENDIX B 

NON-PROJECT* STAKEHOLDER INTERVIEWS 

REGIONAL 

STAKEHOLDERs

REGIONAL STAKEHOLDER ORGANIZATIONs

Dinah Adkins President, National Business Incubation Association (OH) 

Cathy Ashmore Executive Director, Consortium for Entrepreneurship Education 

Bill Campbell Director, Alabama Small Business Development Centers 

Caroline Carpenter Program Director, W.K. Kellogg Foundation 

Dale Carroll President, Advantage West (NC) 

Pam Curry Executive Director, Center for Economic Options 

Eleanor Herndon Executive Director, North Carolina REAL 

June Holley Consultant, Network Weaving (OH) 

Mary Hunt-Lieving Program Officer, Benedum Foundation 

Lisa Ison President, New Century Venture Center (VA) 

Kris Kimmel President, Kentucky Science and Technology Corporation 

Bill Loope New River Community and Technical College (WV) 

Justin Maxson President, MACED 

Ray Moncrief Vice President, Kentucky Highlands Investment Corporation 

Welthy Soni Myers Managing Director for Special Initiatives, Association for Enterprise 

Opportunity (VT) 

Becky Naugle Director, Kentucky Small Business Development Centers 

Kim Pate Vice President, CFED 

Stuart Rosenfeld Principal, Regional Technology Strategies (NC) 

Greg Rutherford President, York Technical College (SC) 

Jeff Spencer Executive Director, Ohio Valley Regional Development Commission 

Kerwin Tesdell President, Community Development Venture Capital Alliance (NY) 

Jesse White Director, Office of Economic and Business Development, University of 

North Carolina 

PROGRAM 

LEADER 

STATE

Denise Ambrose Program Manager, Virginia Department of Housing and Community 

Development 

Bonnie Ammons Senior Program Manager, Office of Community Grant Programs (SC) 

Todd Christiansen Associate Director, Virginia Department of Housing and Community 

Development 

Olivia Collier ARC Program Manager, North Carolina 

Bonnie Durham ARC Program Manager, Alabama 

Al Feldstein ARC Program Manager, Maryland 

Neil Fowler ARC Program Manager, Pennsylvania 

Ralph Goolsby ARC Program Manager, West Virginia 

Elisabeth Kovacs ARC Program Manager, South Carolina 

Rick Meredith Assistant Commissioner, Tennessee Department of Economic and 

Community Development 

Peggy Satterly ARC Program Manager, Kentucky 

Sara Stuckey Retired ARC Program Manager, North Carolina 

James Thompson ARC Program Manager, Georgia 

Kyle Wilbur ARC Program Manager, New York 

*Some stakeholders were from organizations that received ARC funding for entrepreneurship 

projects but were included because they (1) had broad and unique knowledge of some program 

area and/or entrepreneurship, (2) the organization’s project was not included in the sample, 

and/or (3) the individual was not interviewed as follow up with grantees included in the sample. 
160

Appendix C 

PROTOCOL FOR PROJECT LEADERS 

Enter the following data from the project folder: 

Project Number: __________________ Project Status (circle one): I 

C1 C2 

Project Title: _______________________ Project Type (circle one): C N I 

E TA 

Grantee: _________________________ 

Year Project Initiated:_____________ Number of Years of ARC Funding: 

_____ 

Name of Person Interviewed: _______________________Phone/Email: 

___________ 

Organization of Person Interviewed:______________________ 

ARC Funds Invested:_____________________ Leveraged

Funds:_______________ 

Project Summary: 

Stated goals of the project: 

Number of Businesses Served: 

Number of jobs created 

Jobs retained: 

Amount of leveraged private investment: 

For all calls: 

1. Were you involved with this project and/or are you knowledgeable about 

it? (If no, get referral.) If yes, in what capacity did you work with the 

project? 

2. What is your background? (Try to understand the importance of their 

leadership.) Alternative: Tell me a little about your background and 

experience with projects like this in the past. How did you get involved in 

this project? How long have you been involved with the relevant 

community? If not long have you had similar experiences in your 

previous communities? 
161

3. What was the problem that you were trying to address with this project? 

How were you trying to address this problem? 

4. Did you think the project was a success? Why or why not? What were the 

elements of success/failure? 

5. What specific results or outcomes were achieved through this project 

during the period of ARC investment? 

a. Are there specific results that you believe were particularly 

important? If so, why? 

b. Are there specific results that were unintended or unexpected? 

c. Do you feel you achieved the objectives set forth for this project? 

6. Has the project continued after ARC funding ended? 

a. If yes, how was the project funded after ARC? 

b. What outcomes have you experienced post-ARC? 

7. What value did this project create in your service area/community?

a. Are there specific quantifiable changes that you have seen in the 

community because of this project? 

b. Are there specific qualitative changes that you have seen in the 

community because of this project?

8. Is there anything you learned from this project that would be useful to 

others who are attempting to do something similar in their communities? 

9. Is there anything else you would like to add about the project and its 

implementation in your service area/community?

10. Is there anyone else who is familiar with this project and its broad impacts 

on the community and/or the region who might provide us with useful 

insights? (If yes, collect contact information.) 

GO TO SPECFIC QUESTIONS FOR EACH TYPE OF PROJECT 
162

Capital 

At ARC Project End After ARC Funding

Total number of loans 

Number of years 

operated 

 

Total $ amount of loans 

Number of funds created 

Size of Fund(s) 

Distribution of loans by 

sector 

 

$ funds leveraged 

Jobs created 

Jobs retained 

Percent of portfolio 

companies still in 

business 

 

*Wages, income per job 

or total 

 

Comments: 
163

Sectors 

At ARC Project End After ARC Funding

Number of participants 

Number of members 

Jobs created 

Jobs retained 

*Increase in interfirm 

collaboration 

 

*Change in total sector 

sales 

 

*Number of business 

start-ups in targeted 

sector 

 

*Number of participants 

retained in service area 

 

*Number of participants 

still in business 

 

Comments: 
164

Incubators 

At ARC Project End After ARC Funding

Number of current clients 

Number of clients served 

Number of graduated 

firms 

 

Number of clients still in 

business 

 

Amount $ leveraged by 

incubator 

 

*Number of graduates 

retained in service area 

 

Jobs created while in 

incubator 

 

Jobs created after 

graduation 

 

Jobs retained while in 

incubator 

 

Jobs retained postgraduation  

$ capital raised by 

tenants 

 

Comments: 
165

Education 

At ARC Project End After ARC Funding

Number of participants 

enrolled 

 

Number of participants 

completing program 

 

Number of schools 

offering entrepreneurship 

education pre and post 

 

Number of schools in 

service area, pre and 

post 

 

Change in student 

performance pre and 

post 

 

Number of student 

businesses started 

 

Number of students that 

stay within the service 

area 

 

*Increase in awareness 

of business concepts 

 

*Increase in number 

considering business 

creation as a career 

option 

 

* Change in community 

attitudes toward 

entrepreneurship 

 

Comments: 
166

Technical Assistance 

At ARC Project End After ARC Funding

Number of business 

starts 

 

Number of business 

expansions 

 

Number of clients 

*Number of discouraged 

clients 

 

Number of clients still in 

business 

 

Number of jobs created 

Number of jobs retained 

*Firm performance ($ 

capital raised) 

 

*Number of clients 

retained in service area 

 

*Number of clients still in 

business 

 

Comments: 
167

APPENDIX D 

PROTOCOL FOR NON-PROJECT STAKEHOLDERS 

1. Background information on the stakeholder 

 Name: 

Title: 

Organization: 

2. In what ways are you (or have you been) involved with entrepreneurship 

and/or economic development in the Appalachian Region? 

3. Were you directly involved in any specific projects funded by the ARC 

Entrepreneurship Initiative? If yes, please identify the project(s): 

4. [For State Alternates Only] What was your state’s strategy for engaging 

with ARC’s Entrepreneurship Initiative? (Was this initiative a priority in 

your state? Was there a single statewide project approach or did individual 

communities/organizations propose projects?) 

5. [For State Alternative Only] Did your state’s participation in ARC’s 

Entrepreneurship Initiative lead to any change in the policy environment 

for supporting entrepreneurship? If so, please explain. 

6. What do you think are the most significant region-wide impacts associated 

with ARC Entrepreneurship Initiative projects generally? Were there 

specific community or project impacts that you can identify? 

7. How have these broader impacts been measured? 

8. Would you provide some concrete examples of these broader impacts? 

9. In your view, what has limited the broader impacts associated with the 

ARC projects? 

10. In your view, what has contributed to the broader impacts associated with 

the ARC projects? 

11. Do you think the ARC Entrepreneurship Initiative projects have had an 

impact on creating a more supportive climate for entrepreneurs in the 

region? Why or why not? 
168

12. Considering the broad impacts associated with the ARC Entrepreneurship 

Initiative you have identified above, what do you think are some of the 

most valuable or effective performance metrics for capturing these 

impacts?