UNITED

MERICA 

DEPAR

ERCE 

NISTIR 05–7174 

Evaluation Best Practices and Results: 

The Advanced Technology Program 

Stephanie Shipp 

Director, Economic Assessment Office 

Connie Chang 

Supervisory Economist, EAO 

Lorel Wisniewski 

Deputy Director, ATP 

Economic Assessment Office (EAO) 

Advanced Technology Program (ATP) 

U.S. National Institute of Standards and Technology 

May 2005 

TMENT OF COMM

STATES OF A

U.S. DEPARTMENT OF COMMERCE 

Carlos M. Gutierrez, Secretary 

TECHNOLOGY ADMINISTRATION 

Phillip J. Bond, Under Secretary of Commerce for Technology 

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY 

Hratch G. Semerjian, Acting Director 

ADVANCED TECHNOLOGY PROGRAM 

Marc G. Stanley, Director 
Abstract 

The Economic Assessment Office of the Advanced Technology Program (ATP) measures the 

economic impact of co-funding projects with the private sector to develop high-risk, enabling 

technologies, and seeks to increase understanding of the underlying relationships between 

technological innovation and economic phenomena. The National Academy of Sciences has 

praised ATP’s evaluation program as “one of the most rigorous and intensive efforts of any 

U.S. technology program” (National Research Council, 2001). ATP’s evaluation efforts were 

instituted to meet external requests for ATP program results, to use evaluation as a management 

tool to meet program goals and improve program effectiveness, to understand ATP’s contribution 

to the U.S. innovation system, and to develop innovative methodologies to measure the impact 

of public R&D investment. To do this, ATP economists track progress throughout each project’s 

life and into the post-project period by conducting surveys, compiling data, producing statistical 

analyses, undertaking economic studies, and commissioning studies with consultants and research 

economists. This paper describes the evolution of ATP’s evaluation activities, its evaluation best 

practices based on ATP’s experience since 1990, and findings from ATP studies. 

ii 
Acknowledgements 

The authors would like to thank the assistance of EAO staff who create, maintain, and update 

our numerous studies, statistics, and products. We would also like to thank our reviewers, 

Romain Tweedy from the NIST Manufacturing Extension Partnership, Robert Sienkiewicz, 

Kathleen McTigue, and Karian Forsha in the ATP Economic Assessment Office, and Brian 

Belanger, former ATP Deputy Director. 

iii 

Table of Contents 

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii 

Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii 

ATP Mission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 

Measuring Against Mission: An Evaluation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 

ATP’s Evaluation Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 

ATP Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 

Examples of ATP’s Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 

Recommendations for Future Directions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 17 

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 

About the Economic Assessment Office . . . . . . . . . . . . . . . . . . . . . . . . . .Inside front cover 

About the Advanced Technology Program . . . . . . . . . . . . . . . . . . . . . . . . Inside back cover 

v 

ATP Mission 

The mission of the Advanced Technology Program (ATP) is to accelerate the development of 

innovative technologies for broad national benefit through partnerships with the private sector. 

ATP began in 1990 in response to the erosion of U.S industry’s international competitiveness 

in strategic markets and the relative slowness of U.S. firms in translating inventions created 

in universities, national laboratories, and corporate laboratories into innovative products and 

processes (National Research Council, 1999; Ruegg and Feller, 2003, U.S. Congress 1988). ATP 

provides cost-shared funding to industry to accelerate the development and broad dissemination 

of challenging, high-risk technologies that promise significant commercial payoffs and widespread benefits for the nation. This unique government-industry partnership helps companies 

accelerate the development of emerging or enabling technologies. Those technologies, in turn, 

lead to revolutionary new products and to new industrial processes and services that can compete in rapidly changing world markets. 

ATP challenges industry to take on projects that have higher technical risk, and commensurately 

higher potential payoffs for the nation, than they otherwise would pursue. The ATP project 

selection criteria reflect this philosophy. Half of the criteria are based on scientific and technological merit and include an explanation of the innovation, a detailed research plan, and 

justification that the approach is feasible and has the potential to overcome the technical 

hurdles. The other half of the criteria are based on the potential for broad-based economic 

benefits, including benefits to the economy and society that would result from developing the 

new technology, justification for the need for ATP funding, and a plan for how the technology, 

once developed, will be commercialized. Proposals that are submitted in response to ATP’s 

announced competitions are peer-reviewed against the published selection criteria. On average, 

one out of eight proposals meet ATP’s criteria for funding. 

ATP accepts applications from single companies and joint ventures. For-profit companies may 

apply as single applicants to receive an award for up to $2 million over three years to cover 

project costs. Single-company applicants are required to cover their indirect costs—this requirement encourages the participation of small firms that have low overhead costs. In fact, small 

businesses are thriving in the program, (nearly half of all ATP-funded small firms have fewer 

than 20 employees) and lead two out of three of all projects. Single-company applicants often 

bring in subcontractors (universities or other companies) to participate in the project. Large, 

Fortune 500 companies applying as single-company applicants must cover at least 60 percent 

of total project costs—this requirement encourages large firms to formally collaborate with 

others and apply as a joint venture. 

At least two separately owned for-profit companies may apply as a joint venture, with both 

companies substantially contributing to the research effort and to the requirement to cover at 

1 
Evaluation Best Practices and Results: The Advanced Technology Program 

least half of the total project costs. Additional organizations (universities, nonprofits, or other 

companies) may join the joint venture either as formal participants or as subcontractors. Joint 

ventures can receive ATP funding for up to five years, with no funding limitation other than the 

announced availability of funds and the organizations’ ability to cover half the total project costs. 

ATP announces competitions through the Federal Register and held 44 competitions between 

1990 and September 2004. ATP has provided $2.2 billion in awards, and industry has provided 

an additional $2.1 billion as cost share, for a total of $4.3 billion for high-risk research. Of 

the 768 projects awarded to date, 550 are to single-company applicants and 218 are to joint 

ventures. More than 165 universities and 30 national laboratories participate in ATP projects, 

reflecting the collaborative nature and the diversity of the projects’ participants. Projects that 

are awarded are organized into four broad technology areas: advanced materials and chemistry, 

information technology, electronics and photonics, and biotechnology. Manufacturing is a subset 

in all four categories. Technical topics under each of the broad categories are broad and diverse. 

2 
Measuring Against Mission: 

An Evaluation Framework 

Evaluation has been an integral part of program operations from the outset. To learn about the 

program’s impact, ATP set aside a small amount of their initial budget in 1990 to fund rudimentary evaluation activities. Since then, the budget for program evaluation has grown significantly, 

as has interest in evaluation. With a professional staff of economists, statisticians, information 

specialists, social scientists, business liaison specialists, and administrative support, the ATP’s 

Economic Assessment Office (EAO) is charged with carrying out ATP’s evaluation activities. 

EAO aims to measure the economic impacts of ATP’s funding of high-risk, enabling technologies, 

and also to increase understanding of underlying relationships between technological change and 

economic phenomena. EAO also provides business and economic expertise for ATP selection 

boards and locates expert business reviewers to review proposals. 

ATP’s evaluation program has four main goals: 

1. To meet external requests for ATP program results, 

2. To use evaluation as a management tool to meet program goals and to improve program 

effectiveness, 

3. To understand ATP’s contribution to the U.S. innovation system, and 

4. To develop innovative methodologies to measure the impact of public R&D investment. 

ATP’s Economic Assessment Office tracks progress throughout the life of funded projects and 

for several years after the ATP funding ends. Evaluation work consists of conducting surveys, 

compiling data, producing statistical analyses, undertaking economic and policy research studies, 

and commissioning studies by consultants and research economists. 

Evaluation works best when it is closely mapped to a program’s mission. ATP’s legislative mandate is to increase the prosperity of the United States by funding the development of high risk 

technologies through a public-private partnership. ATP’s goals are to add to the nation’s scientific 

and technical knowledge base, to foster accelerated technology development and commercialization, to promote collaborative R&D, to refine manufacturing processes, to ensure small business 

participation, to increase the competitiveness of U.S. firms, and to generate broad economic and 

social benefits (Ruegg and Feller, 2003). 

ATP award recipients deliver benefits directly and indirectly. Direct benefits are achieved when 

technology development and commercialization is accelerated, which leads to private returns 

and market spillovers. Indirect benefits are delivered through publications, conference presentations, patents, and other ways in which knowledge is disseminated. From program purpose and 

design to final outputs, outcomes, and impacts, ATP’s evaluation program measures these direct 

and indirect benefits. 

3 

ATP’s Evaluation Best Practices 

ATP’s experience in funding early-stage technologies and evaluating the impact of its awarded 

projects has resulted in many best practices. These best practices may prove useful to similar 

government programs in the early stages of their operations or to government programs that 

must meet external performance reporting requirements. 

COMMITTING TO PERFORMANCE EVALUATION 

One of the most important best practices is to establish the practice of evaluation and to sustain 

those activities despite budgetary pressures. ATP allocates funding for a staff dedicated to evaluation activities—the Economic Assessment Office—and for carrying out evaluation activities 

using internal and external resources. It is important for public research and development programs to treat evaluation as a core activity and to pursue evaluation within a framework that 

measures the program against its stated objectives. Having a dedicated staff with appropriate 

backgrounds, capabilities, and experience is essential. Having a dedicated budget for evaluation 

activities is critical. 

USING A MULTIFACETED APPROACH TO EVALUATION 

Evaluation of early stage technologies and their longer run impacts requires a multi-faceted 

approach to evaluation because developing new technologies is complex and can take several 

years. Our evaluation tools assess commercialization as well as knowledge creation and dissemination. These methods must accommodate the measuring of inputs, outputs, outcomes, and 

impacts over the life cycle of a project. Research and development take place in the short to mid 

term, commercialization in the mid to longer term, and widespread diffusion of the technology 

over a longer time horizon. This time frame varies by technology area—shorter for information 

technology projects and much longer for biotech projects (Powell and Moris, 2002). It also 

accounts for why multiple evaluation approaches are needed to capture the status of projects 

at various stages of their life cycle. 

COMMISSIONING EXTERNAL STUDIES BY EXPERTS 

ATP contracts with experts to conduct economic analysis of individual projects, clusters of projects, or concepts underlying the economic principles of the program. ATP’s Economic Assessment 

Office works with well-known researchers to shape, manage, and produce many of our reports. 

In the early years, ATP’s Economic Assessment Office worked with economists affiliated with the 

5 
Evaluation Best Practices and Results: The Advanced Technology Program 

National Bureau of Economic Research to help lay a strong foundation for evaluating the program. Zvi Griliches, Edwin Mansfield, Adam Jaffe, Bronwyn Hall, and others collaborated with 

ATP on important research to explore how to measure and track key economic concepts that 

apply to government support for the development of high-risk, enabling technologies carried 

out by the private sector. They studied concepts such as spillovers (knowledge, network, and 

market spillovers—see Jaffe, 1997), return on investment (social, private, and public rates of 

return), (see Mansfield, 1996) and research productivity (see Darby et al., 2002 and Sakakibara 

and Branstetter, 2003). 

EVALUATING UNSUCCESSFUL PROJECTS 

Another best practice is evaluating unsuccessful projects along with successful ones. There is 

a great deal to learn from projects that failed to complete their goals or to deliver promised 

benefits. ATP has analyzed the reasons behind projects terminating early (see ATP, 2001, 

Appendix B). The knowledge generated by examining the reasons why projects fail can enhance 

project selection and project management. 

Almost 10 percent of projects terminate early. A project can end early or not start for participant-initiated reasons, such as a change in goals, financial distress, lack of technical progress, 

or the inability of a joint venture project to reach an agreement on rights to intellectual property. 

A project can also end for ATP-initiated reasons, such as the project no longer meets ATP 

project selection criteria or it shifts away from the pursuit of high-risk research. In a very few 

cases, early success was the cause for early termination. 

STRATEGICALLY PRESENTING RESULTS 

Results have more effect if they are presented so that a nontechnical person can understand 

the science and commercialization. Results are presented in multiple ways—a brief abstract 

enabling someone to quickly grasp the key findings, an executive summary for someone who 

wants an overview of key highlights, and the full report. Quantitative findings are presented in 

tabular form, with graphics, and with accompanying qualitative analyses. We release many of 

our findings in fact sheets and make them available on ATP’s website. We have also published 

three special topic brochures that highlight projects in the health care, energy, and manufacturing 

sectors. (See ATP, 2003b, 2003c, 2005.) 

Another way that results and data are summarized is in the form of a ‘statistical abstract,’ an 

idea borrowed from the U.S. Census Bureau’s annual Statistical Abstract. Plans are to publish 

the ATP statistical abstract every other year—the first was released in September 2004, in a 

report called Measuring ATP Impact, 2004 Report on Economic Progress. The report describes 

ATP, using findings and data from recent reports and statistics. It also provides summaries of 

recent studies and ten detailed statistical tables that provide data on number and distributions 

by types of awards, technology areas, geographic regions, university participation, number of 

patents, commercialization, and post-award attraction of external funding. The data are presented for all projects and by characteristic of the project. 

6 
ATP’s Evaluation Best Practices 

DEVELOPING INNOVATIVE METHODS TO EVALUATE ATP’S EFFECTIVENESS 

Evaluation of emerging technologies is a relatively new field. While traditional economic and 

social science methods can be employed to assess program success, the existing tools are often 

insufficient to describe the nuances and input of public-private investments. It is appropriate to 

modify existing tools, develop exciting new tools, or combine existing methods in ways never 

before explored. 

For example, one of the more difficult concepts to measure is social return resulting from an 

ATP project. Social return includes private returns to the participating company in the project, 

and public returns, including knowledge, network, and market spillover benefits to that company’s customers or to other firms, and a variety of indirect benefits to other companies and 

their customers as a result of the diffusion of knowledge created from the project. (See Jaffe, 

1997; Chang, Shipp, and Wang, 2002, for an historical description of this issue.) 

Despite the difficulties in measuring social return, ATP has pursued a greater understanding 

of this concept by collaborating with consultants, professional economists, and academicians. 

Together, we carry out prospective benefit-cost studies of a range of technologies and projects 

to test and stretch various methodological approaches. These studies include case studies of 

projects that developed closed-cycle air refrigeration technology (Pelsoci, 2001), flow-control 

machining technology (Ehlen, 1999), and technologies that reduced the dimensional variation 

of U.S. motor vehicles (Polenske et al., 2004). By supplementing core in-house evaluation capability with expertise provided by outside contractors, ATP has pursued a balanced approach to 

evaluation and has welcomed new ideas and approaches—another example of a best practice. 

In measuring spillovers, for example, we have used various approaches and means of illustration. 

To capture knowledge spillovers, for the status reports of completed projects—a portfolio-wide, 

mini–case study tool—we developed patent trees, which illustrate multi-tiered citations of patents 

that were issued for ATP-funded technologies. In addition, we commissioned a study to examine 

knowledge spillovers using social network analysis. This emerging method uses fuzzy logic and 

systems analysis to examine knowledge spillovers from research and development projects within 

networks of participating organizations. (See the discussion in Ruegg and Feller, 2003, pp. 

271–75; Fogarty et al., 2005.) 

To study market spillovers, we have recently begun exploring the use of the U.S. Department 

of Commerce’s Bureau of Economic Analysis input-output tables. Specifically, we have taken 

the first 50 completed ATP projects and mapped their make-and-use industries to trace where 

the new technologies began and where they have since ended up (Popkin, 2003). We are also 

exploring other emerging methods to measure spillovers and the impact of ATP funding, 

including coding potential commercial applications identified by ATP project participants using 

NAICs (North American Industry Classification) codes to identify make-and-use industries 

that illuminate the spillover path (Nail and Brown, 2005, forthcoming). 

7 
USING SYSTEMATIC DATA COLLECTION 

Perhaps the cornerstone of ATP’s evaluation program is its comprehensive survey and data 

collection system. Our survey collection efforts are structured to align with our overall evaluation goals, which in turn are crafted to optimize the performance of ATP. As part of an ongoing 

survey and database assessment effort, we have identified five broad-based goals that form the 

conceptual basis of our surveys: (1) opportunities for national economic benefits, (2) acceleration 

of R&D, (3) increased investment in high-risk, long-term technology, (4) stimulation of collaboration, and (5) progress in commercialization of technology. These five goals define how ATP 

projects affect the economy and society. 
ATP Surveys 

As mentioned above, systematic data collection is one component of ATP’s best practices. This 

section provides an in-depth explanation of how ATP surveys have evolved and continue to 

improve. 

CONCEPTUAL BASIS OF ATP SURVEYS 

Initiated in ATP’s early years, the survey program has continually incorporated refinements in 

structure, content, and form to ensure that they remain effective and informative tools for the 

assessment of ATP progress toward its goals. 

From their inception in 1991 as telephone surveys, ATP’s surveys have evolved through several 

distinct stages of development. This evolution has occurred as we (1) have gained a deeper 

understanding of the economic, social, business, and technological processes in which ATP 

participates; (2) incorporated feedback from prior evaluation efforts, including surveys, economic 

and policy research studies, and benefit-cost analyses; and (3) integrated state-of-the-art survey 

methodologies and techniques into our analytical and administrative structures. 

ATP surveys can be viewed as a microcosm of our overall evaluation program at ATP. The 

survey system is a multifaceted effort that is designed to meet multiple (but complementary) 

program goals while balancing efficiency in operation with excellence in results. This is achieved 

by identifying and leveraging both internal and external resources and harnessing the benefits 

of collaboration with survey experts (a tactic learned through our evaluation of ATP), and by 

relying on continual self-assessment and feedback. We do not lose sight of our goal to measure 

against mission, in the short, medium, and long term. 

To design our survey content, economic and organizational changes we would expect to observe 

in our award recipient population if progress were made toward those goals are identified. 

We then crafted survey questions to capture the observed behaviors and outcomes. The themes 

and topics defined by the goals are reflected in multiple lines of questions that vary in a logical 

progression over the survey lifetime. Baseline information is collected on the initial survey, 

and follow-up questions in each area are included at the appropriate anniversary, closeout, or 

post-project surveys. Several variants of the surveys for different types of organizations in the 

population are used. For example, participating nonprofit organizations or universities are given 

a slightly different survey than companies to reflect their unique roles in the project and different 

organizational structures. 

9 
Evaluation Best Practices and Results: The Advanced Technology Program 

Diffusion of knowledge is captured by tracking patents, papers, presentations, and other information about intellectual property. Measures of social and environmental effects, or spillovers, 

of the technologies are captured. 

Acceleration of R&D is measured in terms of reduced time to achieve technical progress. 

Instances in which the technology would not have been developed at all, or perhaps on a lesser 

scale or scope, without ATP funding are identified. This captures the key counterfactual question: what is the difference made by ATP? Reductions in time-to-market are also tracked. 

The goal of increased investment in high-risk, long-term technologies is an indicator of the 

extent to which the work of ATP results in sustained technological development. The level of 

R&D risk is collected by seeking estimators of probability of success, ambitiousness of goals, 

technical difficulty, degree of innovation, and project duration. The “halo effect,” the extent to 

which the initial ATP award has stimulated additional funding, either from within the firm or 

from external sources is also captured. (Survey of ATP Applicants 2000, (2003a); Feldman and 

Kelley, 2001; Solomon Associates, 1993.) 

The right mix of collaboration can lead to positive outcomes for technology creation, development, and commercialization, and for the national benefits that arise from knowledge diffusion 

and new commercial products and processes. To determine whether and how collaboration is a 

factor in creating such impacts among our award recipients, we survey to learn about the structure and nature of collaborative arrangements. And because our primary goal in the Economic 

Assessment Office is to measure the impact of ATP, we are interested in whether the collaboration would have occurred at all without ATP funding. 

The centerpiece of our surveys is the Business Reporting System, or BRS, which superseded the 

original telephone surveys in the mid-1990s (Anderson et al., 2003). As did its predecessor, the 

first BRS addressed the dual objectives of ATP project management and program evaluation. It 

also incorporated lessons learned, most notably, use of the counterfactual approach to measure 

the overall impact of ATP. The BRS is not a single survey instrument, but rather a system of 

surveys administered to all ATP participants throughout the project’s duration and beyond. 

Then, as now, the BRS consisted of a series of surveys conducted at baseline; annually upon 

each project anniversary (up to five years, depending on the type of project award); and two, 

four, and six years after project closeout. The first BRS was sent to participants on a diskette. 

Following developments in survey design and methodology, we transitioned to a web-based 

application and revised the survey design in 1999. 

National economic benefits are measured by gathering information about business growth, 

the development of business relationships and networks, and the diversity of commercial 

applications arising from the technology that ATP has funded. 

Finally, ATP surveys capture commercialization progress, results, and expectations, or specific 

aspects of it. Although, ATP does not fund the commercialization phase of projects. Our mission 

at ATP is not simply to fund high-risk technologies, but to fund high-risk technologies that 

have a strong potential to enhance economic growth. Economic growth can only be achieved 

when the technology enters the marketplace. To measure this impact, information is collected 

10 
ATP Surveys 

from firms on current and expected economic value achieved through revenues from commercial 

applications of the technology, licensing, and cost savings. 

SURVEY AND DATABASE REFINEMENTS AND INTEGRATION 

Today, ATP is in the midst of a conversion to another iteration of our surveys. The current 

transition involves three components: (1) refinements to the content and structure of the survey 

instruments; (2) collaboration with an external professional survey services firm for programming, administration, and support for survey design and data collection; and (3) integration 

into the BRS of special surveys that we have designed and conducted in recent years. 

Refinements to the survey content and structure are being made by incorporating feedback from 

prior survey and evaluation efforts, then by assessing proposed survey revisions against the five 

conceptual goals outlined above. An internal survey design team, working in conjunction with 

an external survey design firm, made initial revisions. The in-house team provides expertise in 

subject matter content, past practice, program goals, and overall direction, and the external 

contractor provides professional guidance in survey methodology. The contractor will ultimately 

program and conduct the revamped system of surveys, lending efficiency and consistency in 

the long-term administration of the surveys, enabling us to focus on analysis and results. The 

survey organization will follow up with respondents who do not answer the survey questions 

or answer only some of the questions. They will also edit the data for consistency and to reduce 

item non-response. 

Assessment of the revised BRS and other surveys against evaluation program goals is a longterm process being conducted by the full economic assessment staff, led by the survey design 

team. As the survey design team completes a draft of each phase of the revised survey instrument, the team submits the draft to the staff for review against program goals. Staff members 

provide input and comments, focusing on their assigned areas of expertise, enabling the survey 

team to further refine the draft. 

Because the BRS is a unified system, changes to later stages of the survey—such as the project 

closeout or post-project surveys—affect the baseline or anniversary surveys, necessitating further 

modifications to earlier drafts. Although we expect the bulk of the survey redesign work and 

transition to the contractor for administration to be completed by late-2005, we additionally 

expect the survey evaluation and revision mechanism to be an ongoing process. 

Though not a large survey in terms of population size, the BRS has a complex structure. 

Including 32 ATP awards made in September 2004, ATP has now funded 768 projects involving 

more than 1,500 participants. The population of the survey is diverse. About one-third of the 

projects are joint ventures, and over half include universities. All projects involve one or more 

for-profit firms, and these firms vary in size from tiny start-ups to large Fortune 500 companies. 

Furthermore, although the survey is administered to firms (or other business or research organizations), it is the project rather than the organization that is the focal point. We are interested in 

following the technology, rather than the firm per se, although we are interested in the economic 

11 
Evaluation Best Practices and Results: The Advanced Technology Program 

and institutional effects and interactions between the technology and the firm. Thus, this is not 

strictly an establishment survey. 

A layer of complexity stems from the survey time frame. ATP funding is awarded through 

competitions, but the competitions do not occur at the same time each year and, especially in 

recent years, competitions may consist of several batches. This raises the question as to how to 

define a cohort and complicates survey administration and analysis. For example, the 27 projects 

that were awarded funding in May 2004 were actually the last batch of award recipients from 

our 2002 competition. They were the first group to begin with the newly redesigned baseline 

survey in October 2004. Furthermore, project duration may vary from about two to five years, 

and individual projects may be terminated early or suspended temporarily, requiring an adjustment to survey cycle times. 

A component of the current survey conversion effort is the integration of special surveys into 

the BRS. In particular, ATP conducted two separate surveys in recent years: a Survey of ATP 

Joint Ventures and a Survey of ATP Applicants. The Survey of ATP Joint Ventures addresses 

the structure, nature, and impact of collaborative relationships. The Survey of ATP Applicants 

covers the full population of all firms that applied for ATP funding in a given competition, 

enabling analysis of the question of additionality (or the difference that the ATP award made) 

by studying both firms that do receive ATP funding and firms that do not. Incorporating concepts and questions from both surveys into the BRS will improve efficiency in administration, 

facilitate analysis, and reduce respondent burden. Nonawardees will continue to receive a separate survey, since the BRS is designed to collect data from awardees. 

Finally, we are embarking on a redesign and integration of the databases that underlie our 

multifaceted surveys. We also hope to integrate data from external sources, such as data from 

Compustat and Dun & Bradstreet, into our internal databases. The database integration effort 

complements our survey redesign work, and it is also necessitated by the growth of ATP. 

Database methods and structures created in the early years are no longer adequate for an 

expanding survey population, especially in light of the complex structure of the survey cycles. 

Because our ultimate goal is the collection and production of data to serve the multiple functions 

of program evaluation, the analysis of R&D, and the identification and development of new 

research and evaluation methodologies, survey and database development will remain on the 

agenda for continuous process improvement. 

12 
Examples of ATP’s Studies 

ATP uses a multi-faceted approach to evaluation. Reflecting the richness and diversity of studies, 

several examples of studies, plus highlights of findings from those studies, are presented below. 

These examples include policy studies, Status Reports of completed ATP-funded projects, selected 

survey results, and benefit-cost studies. These examples are provided to show the depth and 

breadth of our assessment work. 

DEFINING ATP’S ROLE IN THE POLICY ARENA 

In a report commissioned by ATP’s Economic Assessment Office, Branscomb and Auerswald 

(2002) address the question: how important is the public role in funding early-stage technology 

development? They estimate that between $5 billion (2 percent) and $36 billion (14 percent) of 

overall national R&D spending in 1998 was devoted to early-stage technology development. 

The relatively small share of total national R&D spending devoted to early-stage technology 

development, which arguably drives future economic growth, supports the view that there is a 

“funding gap” in which the amount of funding currently available is less than what is socially 

optimal or desirable. 

Of particular interest is the finding that the federal government, and not organized venture 

capital, is a major funding source for early-stage technology development. The federal government provided approximately 20 percent to 25 percent of funding toward early-stage technology 

development, with the Advanced Technology Program as one of the principal federal programs 

focused in this critical area. ATP targets technology development that venture capitalists do 

not address. Venture capitalists, state governments, and universities contribute only between 

8 percent and 16 percent toward early-stage technology development. 

PROJECT AND PORTFOLIO ASSESSMENT 

Status Reports are descriptive mini–case studies for each completed ATP project, written several 

years after ATP funding ends. Status Reports address how well the project performed against 

ATP’s mission objectives. 

A performance rating for each project (zero to four stars) is computed using a uniform set of data. 

The aggregation of stars provides a portfolio view of ATP performance. For example, aggregating 

the performance ratings for the first 120 ATP projects shows the following distribution: 

13 
Evaluation Best Practices and Results: The Advanced Technology Program 

• 16 % 0 stars 

• 14 % 1 star 

• 29 % 2 stars 

• 31 % 3 stars 

• 10 % 4 stars 

The largest group of projects, 31 percent, fell into the three-star category. These projects show 

strong progress. Combining these with the 10 percent rated outstanding shows an impressive 

41 percent of projects performing at a high level. Twenty-nine percent fell into the two-star 

category. These projects show moderate progress but are not particularly robust overall. Thirty 

percent of the projects scored one star or less, which is not surprising, given that ATP projects 

are high-risk R&D and not all projects are expected to succeed. Projects may fail for technical 

reasons, business reasons, or a combination of both. 

Each status report also includes a patent tree for each patent filed during or after the ATP 

project ends to show the citation of the patent in subsequent patents. This is one way to portray 

knowledge spillovers. 

ANSWERING THE COUNTERFACTUAL QUESTION: WHAT HAPPENS WITHOUT ATP? 

The Survey of ATP Applicants was conducted in 1998, 2000, and 2002. This survey was administered to all applicants in the previous competition year in order to compare the company and 

project characteristics of awardee and non-awardee companies soon after awards are announced. 

It addresses the counterfactual question, what happens when a project does not receive ATP 

funding. The Survey of ATP Applicants asks organizations that did not receive awards what 

happened to their projects once they did not receive funding from ATP. The survey results find 

that 41 percent of these projects are not pursued, 40 percent are pursued on a smaller scale, and 

of those pursued on a smaller scale, four out of five reports that the project scope is reduced to 

below 40 percent of the proposed ATP project. 

Evidence from the Survey of ATP Applicants shows that ATP is successful in directing funding 

to projects that have higher technical risk and longer time horizons than projects proposed by 

nonawardees (ATP, 2003a). A measure of technical risk is the probability that a project will 

not achieve its technical goals. 

• Among ATP awardees, the average estimate for the probability of not fully achieving 

technical goals is 45 percent, compared to nonawardees’ estimated probability of 31 

percent. 

• About half (54 percent) of ATP awardees expect a time horizon of 4 years or more on 

their proposed ATP projects compared to one-third of nonawardees. 

Proposed ATP projects for both awardees and nonawardees are higher risk and have a longer 

time horizon than to ‘typical’ R&D projects. ATP awardees report a greater contrast between 

their proposed and typical R&D projects, compared to nonawardees. 

14 
ATP Mission 

A key finding is that ATP awardees attract additional funding since submitting their ATP 

proposal. This phenomenon is referred to as the halo effect. For example, three out of four 

awardees report increased funding, while one out of four non-awardees reports increased funding from internal company sources. ATP awardees are also more likely to receive funding from 

external sources. One out of three awardees reports increased funding, and only one out of 

five non-awardees report increased funding from external sources. 

MEASURING ACCELERATION EFFECTS 

The Business Reporting System (BRS) allows an examination of ATP awardees from a longitudinal perspective and from a cross-sectional perspective. Responses to the BRS surveys indicate 

that ATP funding accelerated R&D in 9 out of 10 organizations. Of those organizations that 

indicated they were ahead in their R&D cycle: 

• 13 percent indicate they are ahead by one year. 

• 53 percent indicate that they are ahead by one to three years. 

• 7 percent indicate that they are ahead by more than three years. 

ATP participants report that the acceleration of R&D reduces the time it will take to bring 

products to market or to implement new production processes. Reduction in time-to-market by 

two years or more is anticipated for about three out of five planned commercial applications.1 

MEASURING OUTCOMES—BENEFIT-COST STUDIES 

Benefit-cost studies are one of the primary ways to measure outcomes quantitatively. Outcomes 

are difficult to measure because one must make assumptions about the impact of the new technology and acceptance by buyers. The results of individual benefit-cost studies can be aggregated 

to look at the impact (usually prospective estimates) across ATP. The net social benefits from 

about 40 ATP projects are estimated to be $18 billion. This is a more than adequate return 

on an investment of $2.2 billion that ATP has made between 1990 and September 2004. ATP 

projects produce large benefits. 

One example of a benefit-cost study is Low-Cost Manufacturing Technology for Amorphous 

Silicon Detectors, a joint venture project funded by ATP in 1995 (Pelsoci, 2003). Digital 

mammography and radiography systems are innovative technology solutions to the diagnostic 

and productivity limitations of conventional x-ray systems. The new process, which was 

implemented in 2004, reduces fabrication costs by approximately 25 percent without 

compromising performance. On the basis of 33 million mammography and 68 million chest 

x-rays per year, estimated benefits are $125 to $193 for every $1 that ATP has spent. Societal 

benefits include avoidance of unnecessary medical procedures as a result of lower false-positive 

rates, improved breast cancer detection, reduced patient exposure to radiation, and reduced 

examination time. 

1. Based on Business Reporting System (BRS) survey data from 673 organizations in 347 ATP projects funded from 

1993 to 1998, for projects with one or more years of ATP funding. 

15 

Recommendations for Future 

Directions and Conclusion 

ATP is continuously improving its evaluation program. In A Toolkit for Evaluating Public R&D 

Investment, a list of recommendations for guiding these improvements is provided (Ruegg and 

Feller, 2003). 

• Increase retrospective analyses based on market data. 

• Incorporate both direct- and indirect-path analysis in benefit-cost case studies, 

including estimates of both market and knowledge spillovers. 

• Continue status reports of completed projects and conduct periodic updates for 

a sampling of projects. 

• Update information on state and foreign counterpart programs. 

• Further develop several of the promising new evaluation techniques. 

• Deepen analysis of knowledge spillovers beyond patent studies. 

• Identify and address new questions that arise as ATP is modified. 

• Pursue analysis of failures and successes. 

• Continue an effective mix of in-house and external evaluation studies. 

• Take greater advantage of evaluation results in decision-making processes. 

ATP is beginning to incorporate these recommendations in planning future work. 

ATP’s multifaceted and integrated evaluation program uses widely accepted approaches to 

economic and statistical analysis to develop estimates of impacts of ATP funding on project 

timing and success. In this paper, we have discussed the best practices learned from our 15 

years of evaluation. Through the use of these evaluation best practices, our surveys, studies, 

and reports show that ATP is indeed meeting its mission to accelerate the development of 

high-risk, enabling technologies. 

17 

References 

Advanced Technology Program (ATP) (2001), Performance of 50 Completed ATP Projects, 

Status Report Number 2, NIST Special Publication 950–2, ATP, National Institute of Standards 

and Technology, U.S. Government Printing Office, Washington, D.C. 

Advanced Technology Program, (2003a), Survey of ATP Applicants 2000, NIST GCR 03-847 

(Gaithersburg, MD: National Institute of Standards and Technology). 

Advanced Technology Program (2003b), Powering Our High-Speed Economy, A Profile of ATP 

Energy Investments, (Gaithersburg, MD: National Institute of Standards and Technology). 

Advanced Technology Program (2003c), Beyond Measure: A Profile of ATP Health Care 

Investments, (Gaithersburg, MD: National Institute of Standards and Technology). 

Advanced Technology Program (2004), Measuring ATP Impact, 2004 Report on Economic 

Progress, (Gaithersburg, MD: National Institute of Standards and Technology). 

Advanced Technology Program (2005), A Profile of ATP Manufacturing Investments, Inspiring 

Innovations in Industry, (Gaithersburg, MD: National Institute of Standards and Technology). 

Anderson, Gary, Jeanne Powell, and Stephanie Shipp (2003), “Improving the Advanced 

Technology Program’s Business Reporting System—A Firm-Level R&D Survey,” 2003 

Proceedings of the American Statistical Association, Government Statistics Section, 

[CD-ROM], Alexandria, VA, American Statistical Association. 

Branscomb, Lewis, and P. Auerswald (2002), Between Invention and Innovation: An Analysis 

of Early-Stage Technology Development, NIST GCR 02–841 (Gaithersburg, MD: National 

Institute of Standards and Technology). 

Chang, Connie, Stephanie Shipp, and Andrew Wang (2002), “The Advanced Technology 

Program: A Public-Private Partnership for Early Stage Technology Development,” Venture 

Capital 4(4): 363–70. 

Darby, Michael, Lynne Zucker, and Andrew Wang (2002), Program Design and Firm Success in 

the Advanced Technology Program: Project Structure and Innovation Outcomes, NISTIR 6943 

(Gaithersburg, MD: National Institute of Standards and Technology. 

19 
Evaluation Best Practices and Results: The Advanced Technology Program 

Ehlen, Mark A. (1999), Economic Impacts of Flow-Control Machining Technologies: Early 

Applications in the Automobile Industry, NISTIR 6373 (Gaithersburg, MD: National Institute 

of Standards and Technology). 

Feldman, Maryann P., and Maryellen R. Kelley (2001), Winning an Award from the Advanced 

Technology Program, NISTIR–6577 (Gaithersburg, MD: National Institute of Standards and 

Technology). 

Fogarty, Michael S., Amit K. Sinha, and Adam B. Jaffe, (forthcoming, 2005) ATP and the U.S. 

Innovation System—A Methodology for Identifying Enabling R&D Spillover Networks with 

Applications to Micro-electromechanical Systems (MEMS) and Optical Recording, draft report 

submitted to ATP. 

Jaffe, Adam B. (1997), Economic Analysis of Research Spillovers: Implications for the Advanced 

Technology Program, NIST GCR 97–708 (Gaithersburg, MD: National Institute of Standards 

and Technology). 

Mansfield, Edwin, (1996), Estimating Social and Private Returns from Innovations Based on 

the Advanced Technology Program: Problems and Opportunities, GCR 99–780 (Gaithersburg, 

MD: National Institute of Standards and Technology). 

Nail, John and Hayden Brown (2005 forthcoming), Classifying and Analyzing ATP Projects 

Using the North American Industry Classification System (NAICS), NISTIR 2005– 

(Gaithersburg, MD: National Institute of Standards and Technology). 

National Research Council (1999), The Advanced Technology Program: Challenges and 

Opportunities, Charles W. Wessner, ed., National Academy Press, Washington, D.C. 

National Research Council, (2001), The Advanced Technology Program: Assessing Outcomes, 

Charles W. Wessner, ed., National Academy Press, Washington, D.C. 

Pelsoci, Thomas (2001), Closed-Cycle Air Refrigeration Technology for Cross-Cutting 

Applications in Food Processing, Volatile Organic Compound Recovery and Liquid Natural 

Gas Industries, NIST GCR 01-819 (Gaithersburg, MD: National Institute of Standards and 

Technology). 

Pelsoci, Thomas (2003), Low-Cost Manufacturing Process Technology for Amorphous Silicon 

Detector Panels: Applications in Digital Mammography and Radiography, NIST GCR 03–844 

(Gaithersburg, MD: National Institute of Standards and Technology). 

Polenske, Karen, Nicolas Rockler, and others (2004), Closing the Competitive Gap: A 

Retrospective Analysis of the ATP 2mm Project, NIST GCR 03–856 (Gaithersburg, MD: 

National Institute of Standards and Technology). 

Popkin, Joel (2003), Inter-Industry Diffusion of Technology That Results from ATP Projects: 

Preliminary Research of the Potential Impacts of ATP Funding, NIST GCR 03–848 

(Gaithersburg, MD: National Institute of Standards and Technology). 

20 
References 

Powell, Jeanne, and Francisco Moris (2002), Different Timelines for Different Technologies, 

NISTIR 6917 (Gaithersburg, MD: National Institute of Standards and Technology). 

Ruegg, Rosalie, and Irwin Feller (July 2003), A Toolkit for Evaluating Public R&D Investment: 

Models, Methods, and Findings from ATP’s First Decade, NIST GCR 03–857 (Gaithersburg, 

MD: National Institute of Standards and Technology). 

Sakakibara, Mariko, and Lee Branstetter (2002), Measuring the Impact of ATP-Funded Research 

Consortia on Research Productivity of Participating Firms, NIST GCR 02–830 (Gaithersburg, 

MD: National Institute of Standards and Technology). 

Solomon Associates (1993), The Advanced Technology Program, An Assessment of Short-Term 

Impacts: First Competition Participants, (Gaithersburg, MD: National Institute of Standards 

and Technology). 

U.S. Congress, Omnibus Trade and Competitiveness Act of 1988 (P.L. 100–418, codified in 

15 U.S.C. 278n. and later amended by the American Technology Preeminence Act of 1991 

(P.L. 102–245, codified in 15 U.S.C. 3701). 

21