Performance Measurement: Efforts to Evaluate the Advanced Technology Program
Released: 1995-06-15T00:00:00

Letter

Background

ATP was established by the Omnibus Trade and Competitiveness Act of 1988 (P.L. 100-418). The program is intended to assist U.S. businesses in creating and applying the generic technology and research results necessary to (1) commercialize significant new scientific discoveries and technologies rapidly and (2) refine manufacturing technologies. Funding for ATP is awarded through announced competitions. Single companies that receive awards are reimbursed for the direct costs of their proposed research but must pay for all overhead costs. Joint ventures, which consist of two or more companies, are reimbursed for both their direct and overhead costs but must provide more than 50 percent of the total funding for their project. ATP supports high-risk projects that have the potential for eventual substantial widespread commercial application. Since the first competition in fiscal year 1990, NIST has funded 177 ATP projects. As of April 1995, 12 projects had been completed.

NIST summarized the results of its initial evaluation efforts and future plans in Setting Priorities, which received wide distribution. NIST distributed 3,800 copies of this document to the Congress, administration officials, and industry. NIST officials also submitted the document to the Congress during the fiscal year 1995 appropriations hearings.

Evaluating ATP poses many challenges. For example, ATP research projects are intended not only to be technical successes but also to have commercial results. The linkage between technical work and commercial results may not always be direct and may be subject to interpretation.

Also, several years can elapse between the end of technical work and the realization of commercial results.

ATP’S Results Reported in Setting Priorities

NIST cites six examples of ATP’s short-term results in Setting Priorities. While all six have limitations, four overstate ATP’s success or lack adequate support. Specifically, NIST reported that as a result of ATP total U.S. research on advanced technologies for printed wiring boards has quadrupled, participants have pursued research they otherwise could not have participants have forged new relationships with companies and government or academic laboratories, and the number of joint R&D ventures in private industry has increased.

ATP’s Impact on Printed Wiring Board Industry Overstated

“of the 29 major project areas under investigation , the participants reported that on average only 6.5 projects would have been started in the absence of the ATP award. In fact, a number of critical projects would not have been attempted in the absence of the joint venture.”

According to an industry association representative, however, there are approximately 800 merchant manufacturers in the printed wiring board industry, many of which are active in R&D. Discussions with the industry association indicate that the industry spent at least $26.5 million on R&D in 1992. In comparison, the ATP-supported joint venture spends $5.7 million per year, on average, on printed wiring board research.

We believe that NIST’s conclusion that total U.S. R&D on printed wiring boards quadrupled as a result of ATP was an overstatement because the third-party study was limited to a single five-member joint venture in an industry that contains over 800 merchant manufacturers, many of which engage in R&D. The evidence presented by NIST supports statements only about the five-member joint venture, not about the entire U.S. industry.

Claims of Increased High-Risk Research and New Relationships Not Adequately Supported by Survey Data

In reporting the most important effects of ATP on the basis of a survey of early award recipients, NIST was selective in its use of the survey data. NIST reported that the most important effect cited by award recipients was “the ability to pursue promising lines of research that they otherwise could not have followed.” According to NIST officials, this statement is based on responses to the survey’s question “What would you say is the single most important effect that the ATP award has had on your organization thus far?” Fifteen out of 28 responses in the study were categorized as saying “the ability to afford and engage in this kind of high-risk, long-term research.”

However, responses to another question in the same survey provided conflicting information. When asked “In the absence of this ATP award, would your organization have pursued the development of this technology?” nearly as many respondents—14 out of 26—responded affirmatively. Four respondents said they definitely would have; 10 said they probably would have; 7 said they probably would not have; and 5 said they definitely would not have. In response to a subsequent question, the 14 respondents said they would have pursued the development of the technology at a different level of effort. Thus, although 15 participants believed that ATP enabled them to pursue this kind of high-risk, long-term research, 14 participants in the same study believed that they would at least probably have pursued the technology even without the ATP award, although at a different level of effort.

On the basis of the same survey, NIST reported that the second most important effect cited by early ATP participants was “forging new relationships between companies, and between companies and government or academic labs.” However, several discrepancies exist. First, the survey results do not refer to a second most important effect, nor were the participants asked to identify a second most important effect. NIST officials said this statement was supported by the second most frequent response to the question about the “single most important effect” of the ATP award. However, the second most important effect cannot be inferred from the second most frequent response because the frequency of the response to this question does not say anything about the relative importance that individual respondents would have ascribed to this effect. The survey would have to have asked the participants specifically what they believed the second most important effect of ATP was in order to make that determination.

Another discrepancy is that the second most frequent responses were categorized as “the benefits that can flow from industry-industry collaboration,” but made no mention of “government or academic labs,” as NIST reported. To support its conclusion that the second most important effect was “forging new relationships between companies, and between companies and government or academic labs” NIST officials said that they had also included information from responses to another survey question. The other question asked participants to rate a list of potential effects of ATP. According to NIST officials, the item rated second highest on the list of potential effects for this question was the basis for NIST’s statement about the “second most important effect.” This item was “enhanced the technology infrastructure by strengthening linkages between sectors (industry-government, industry-university) and/or within sectors (industry-industry).”

However, NIST did not base its “most important effect” on the same question. Since NIST based its “second most important effect” on this question (respondents were asked to rate a list of potential effects of ATP), in order to be consistent, the highest rated response to the same question should have been the “most important effect.” However, the highest rated item on this list is “collaboration and strategic alliances.” This conflicts with the responses mentioned previously, which said that the “single most important effect” was “the ability to afford and engage in this kind of high-risk, long-term research.” This inconsistent methodology casts doubt on NIST’s reporting of ATP’s most important effects.

Relationship Between ATP and Increase in Number of Joint Ventures Not Adequately Supported

“Profiles also suggest that the ATP has led—as desired—to an increase in joint R&D ventures in private industry. In the first four competitions, approximately 125 joint ventures involving over 800 organizations were formed to apply to the ATP.”

However, the only support NIST gave us for this statement is the fact that 125 joint ventures submitted proposals to ATP. Although the number of joint R&D ventures has increased, there are several reasons to question a direct relationship between this increase and ATP. The number of joint R&D ventures has steadily increased since 1986—years before NIST made its first ATP award. Some explanations for the causes of this trend and for the formation of joint ventures are unrelated to ATP. For example, the National Science Foundation cites the passage of the National Cooperative Research Act of 1984 as one reason for this growth. The Foundation explains that this act encourages research collaboration among industry competitors by better defining joint R&D ventures and protecting them from antitrust suits by limiting potential liability. We believe that NIST’s conclusion about the causal relationship between ATP and an increase in the number of joint ventures lacks adequate support.

NIST’s Future Plans for Evaluating ATP in Setting Priorities

The evaluation plan, as presented in Setting Priorities, includes several measures that NIST expects will indicate the long-term economic success of ATP projects. However, some of these measures may not indicate the economic success of ATP.

One of the measures that NIST believes will indicate the long-term economic success of ATP projects is “straightforward tracking of technical milestones.” However, achieving technical milestones may not be a valid indicator of the economic success of ATP projects because technical advancement does not always lead to economic success. For example, earlier versions of the ATP evaluation plan pointed to one ATP project that was achieving all of its technical milestones as evidence of the project’s likely success in stimulating economic growth. The lead company involved in this joint venture, however, went bankrupt before the project was completed. Although the other company in the joint venture has stated its intention to continue the joint venture’s commercialization plan, the lead company’s bankruptcy reduces the likelihood of future economic effects being realized from this ATP project.

Tracking the completion of technical milestones for ATP projects provides helpful information to ATP managers who need to know whether this vital step in the commercialization process is being achieved. However, using this information as an indicator of “long-term economic success” may create the false expectation that technical success will result in commercial success. “Increased collaborations and strategic alliances ” is another measure that NIST expects to indicate long-term economic success. However, the number of collaborations and strategic alliances may not indicate ATP’s economic success. A joint venture is one form of collaboration or strategic alliance that can occur between companies. As the previously cited example of the bankrupt company and its collaboration shows, the use of this measure to indicate “long-term economic success” may create the false expectation that collaboration will lead to commercial success.

The ATP evaluation plan summarized in Setting Priorities shows that NIST intends to continue providing descriptive data on the program and its operations. Two of the five major components of the evaluation plan focus on obtaining this type of information and are descriptive in nature: (1) assessing ATP’s operational activities and (2) profiling applicants, recipients, technologies, and projects. This information will include descriptive data about the program’s operations, participants, and monitoring activities as integral parts.

This information is helpful to ATP officials in managing the program. Collection of these data, however, does not provide the Congress with information about the program’s impact and economic success.

Additional ATP Evaluation Efforts

According to NIST officials, program evaluation has been a part of ATP from its beginning, and the development of a long-term evaluation strategy is an ongoing NIST process. NIST says that at this point in ATP’s history, its approach to evaluating ATP is to lay the groundwork to provide metrics for the program’s results at the earliest possible time.

NIST’s evaluation efforts include engaging the advice and services of the nation’s leading economists in impact assessment and evaluation. NIST has also put in place an extensive data collection system to support its ATP evaluation efforts. In addition, NIST says it is conducting microeconomic case studies and supporting the development and use of economic models for projecting outcomes of ATP.

Conclusions

It is too early to determine ATP’s long-term economic impact; therefore, there has not been a complete assessment of ATP. Evaluating ATP will be challenging. For example, ATP research projects are intended not only to be technically successful but also to have a commercial impact. The linkage between technical work and commercial results may not always be direct and may be subject to interpretation.

NIST has reported short-term results in Setting Priorities that, it says, indicate that the program is making an impact. However, our analysis indicates that these results are overstated or lack adequate support. Thus, judgments about the economic success of ATP should not be based solely on the information in Setting Priorities. In addition, some of the indicators contained in Setting Priorities that NIST proposes to track for future evaluations of ATP, namely technical milestones and the number of collaborations and strategic alliances, may create false expectations of the program’s economic success. Neither of these indicators necessarily reflects the long-term economic success of the program. According to NIST, other efforts are under way that will support studies of the program’s long-term outcomes as soon as such studies are feasible.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Commerce for written comments. These comments, along with our detailed responses, are provided in appendix I. In addition, at the Secretary of Commerce’s request, we met with the Under Secretary for Technology and the Director of NIST to discuss the draft report in more detail. Specifically, NIST made the following observations about our draft report: It overestimated the amount of “advanced” R&D in the printed wiring board industry by citing industry figures that include R&D that is not “advanced.” However, we point out in our response that the broader industry figure is appropriate to use for comparison purposes because the ATP project also includes R&D that is not “advanced.”

It introduced a negative bias to our conclusions by including only partial responses to a survey question. We have added language to the final report reflecting the additional information contained in responses to this survey question. However, this does not change our analysis or conclusions because the information still suggests that as many as half of the ATP projects would have been undertaken even without ATP support, although at a different level of funding. Moreover, a Congressional Budget Office study reached a conclusion similar to ours on the basis of the same data. It overlooked evidence and made it appear that NIST’s conclusions about ATP’s impact on forging relationships between companies and government or academic labs lacked support. Our review of this evidence is included in this report and shows that NIST’s conclusions are based on an inconsistent methodology used in analyzing the evidence. It overlooked evidence supporting NIST’s statement that ATP has led to an increase in the number of joint ventures. However, the evidence provided by NIST still does not demonstrate that ATP has caused an increase in the number of joint ventures for primarily two reasons. First, the National Cooperative Research Act was influencing the number of joint ventures over the same time period. Second, NIST has no evidence that shows why joint ventures that applied to ATP formed in the first place. NIST currently has a study under way to determine that information. The fact that the joint ventures registered with the Federal Trade Commission or the Department of Justice when applying to ATP is irrelevant because joint ventures are not required to register with these agencies when they form.

NIST’s comments on our draft report also include important qualifications that help dispel false expectations about the indicators of long-term economic success in Setting Priorities. Had these qualifications appeared in Setting Priorities, one would have been less likely to arrive at false conclusions about the program’s impact.

Our draft report contained a proposed recommendation that the Secretary of Commerce direct NIST officials to develop an evaluation strategy that includes measures of the program’s outcomes. In commenting on our draft report, NIST said that it intends to continue to refine the ATP evaluation plan through the use of microeconomic case studies and economic models for projecting ATP’s outcomes. Therefore, we have withdrawn that proposed recommendation.

Scope and Methodology

In conducting our analysis, we interviewed the NIST senior economist responsible for evaluating ATP and examined Setting Priorities. The NIST economist said that this document summarizes the evaluations of the program conducted to date, as well as the plan for evaluating ATP in the future. We investigated all statements about ATP in this document by reviewing the supporting studies and data to determine their consistency with NIST’s reported statements. We analyzed the ATP evaluation plan’s “indicators of future economic success” but were unable to analyze the usefulness of those indicators that were too general for understanding the effects of ATP. For example, indicators that include terms such as “technological infrastructure” and “enabling technologies” do not clearly identify what they measure or how they are related to the economic success of ATP. In addition, NIST supports some of its statements about ATP’s effects with references to two NIST-supported studies. Although we examined these studies, we did not evaluate them for their validity. NIST’s evaluation of ATP is an ongoing process. When we had nearly completed our work, NIST provided us with a copy of NIST Industrial Impacts: A Sampling of Successful Partnerships, which contains anecdotes about ATP awards. We did not evaluate this document. We also consulted economists, the R&D evaluation literature, and a trade association representative. We conducted our review from January 1994 to April 1995 in accordance with generally accepted government auditing standards.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to the Secretary of Commerce; the Director, NIST; the Director, ATP; the Inspector General, Department of Commerce; the Director, Office of Management and Budget; and other interested parties. We will make copies available to others on request.

Please contact me at (202) 512-3841 if you or your staff have any questions. Major contributors to this report are listed in appendix II.

Comments From the Department of Commerce

The following are GAO’s comments on the Department of Commerce’s letter dated January 27, 1995.

GAO’s Comments

1. Our work focused specifically on the information provided by the National Institute of Standards and Technology (NIST) in its document entitled Setting Priorities and Measuring Results at the National Institute of Standards and Technology. We examined (1) the short-term results that NIST says indicate the impact of the Advanced Technology Program (ATP) and (2) the measures that NIST expects will indicate ATP’s long-term economic success. We did not address the progress made by NIST in implementing its evaluation plan. After reviewing additional support provided by NIST, we maintain that our original assessment of NIST’s conclusions about short-term results is valid. None of the information provided in the comments refutes our original conclusions. We have included additional information about NIST’s ongoing evaluation efforts beyond those cited in Setting Priorities. We also maintain that the indicators of long-term economic success included in Setting Priorities may create false expectations. Although NIST’s comments on our draft report included important qualifications that help dispel false expectations (e.g., “accomplishment of technical milestones is a necessary, but insufficient, condition for the achievement of long-run economic success”), Setting Priorities did not include any of these qualifications. 2. The draft report said “ATP evaluations would better assist the Congress in making budget decisions if the evaluations focused more on outcomes, which reflect the impacts of the program, than on outputs, which describe the activities of the program.” This statement was not intended to be a broad conclusion about the plan. The report now discusses this topic within the context of the descriptive information that ATP collects. As the report notes, this information does not necessarily provide the Congress with information about the program’s impact and economic success. 3. We continue to maintain that “tracking technical milestones” and “increased industrial collaborations and strategic alliances,” when presented as indicators of long-term economic success, may create false expectations. Presenting additional information, such as NIST provides in its comments, would help avoid creating false expectations. 4. Since we focused our work specifically on the statements about ATP in Setting Priorities, we reviewed the studies and data supporting these statements to determine their consistency. We did not ignore any information provided that was relevant to this work. In addition, we have added information about NIST’s evaluation efforts that does not appear in Setting Priorities. 5. We maintain that our analysis and estimates are appropriate and accurate. The details are provided in the body of the final report and in comments 13 and 14. Moreover, the evidence NIST provided—a report on a single five-member joint venture—cannot be extrapolated to the entire U.S. printed wiring board industry. 6. See comment 15. 7. We included all of the relevant information in our analysis and maintain that NIST’s statements lack support and are based on a selective use of data. See comment 19. 8. Comment 23 summarizes our rationale for questioning NIST’s statement about ATP’s impact on the formation of joint ventures. 9. We have withdrawn the proposed recommendation in light of additional information about plans to refine the ATP evaluation plan. 10. We agree that our draft report did not lay out the full scope of the ATP evaluation plan. That was not our intent. Our work focused on the information contained in Setting Priorities, which states on page 15 that “A number of measurable short-term effects are expected to provide indicators of long-term economic success. In addition to straightforward tracking of technical milestones, these indicators include: . . . increased industrial collaborations and strategic alliances; . . .” 11. We have not changed our position. Our 1993 report did not evaluate NIST’s ATP evaluation strategy but did present a NIST-provided summary of the ATP evaluation strategy. In addition, the current report contains similar language concerning the barriers and challenges facing NIST in evaluating ATP. For example, our report states, “Evaluating ATP poses many challenges. For example, ATP research projects are intended not only to be technical successes but also to have commercial results. The linkage between technical work and commercial results may not always be direct and may be subject to interpretation. Also, several years can elapse between the end of technical work and the realization of commercial results.” 12. According to NIST, the statement about ATP’s impact on the printed wiring board industry is based on a study that is limited to a single five-member joint venture. We maintain that it is an overstatement to project the impact of this joint venture to the entire industry of over 800 manufacturers. For purposes of comparison, in the draft report we estimated spending for research and development (R&D) for the entire printed wiring board industry. We still maintain that these estimates are the correct figures to use for such purposes and that the figures further reinforce our conclusion. 13. Our analysis is based on the industry’s overall expenditures for R&D for the following reason. As NIST comments, only a portion of the industry’s R&D is focused on advanced technology. But similarly, only a portion of the joint venture’s R&D is focused on advanced technology, and the larger balance is devoted to incremental improvements in existing technology. 14. NIST’s suggested analysis still does not demonstrate that ATP has quadrupled total U.S. R&D work on advanced technologies for printed wiring boards. Taking into account the Department of Energy’s contribution of $5.2 million, or $1.04 million annually, the total annual spending on R&D by the ATP-supported joint venture is that amount plus the joint venture’s original annual expenditure of $5.7 million, for a total of $6.74 million. Since less than half of that total is spent for R&D on advanced technology, $3.37 million, or a half, is a high estimate of the amount spent annually by the joint venture on advanced technology R&D. NIST’s claim remains overstated because the joint venture’s annual expenditure of $3.37 million still does not quadruple the industry’s expenditure of $2.65 million per year for advanced technology research on printed wiring boards. 15. Language has been added to the final report reflecting the information provided by responses to this question. This information, however, does not change our analysis or our conclusions. Our conclusions are reinforced by a Congressional Budget Office (CBO) report, which arrives at a similar conclusion about this evidence. According to the CBO report, “One privately funded study of the 11 projects supported by the first [ATP] competition in 1990 suggests that as many as half of them would probably have been undertaken even without ATP support, although at a lower level of funding.”16. Language has been added to the body of the final report reflecting the information provided by these responses. This information does not change our conclusions. 17. We included all of the relevant information in our analysis, and our conclusions remain unchanged. See comment 19. 18. At the exit conference we said we understood NIST’s logic and rationale for making the statement. We did not say we accepted it as reasonable. 19. As stated in our final report, this is an inconsistent use of survey data. NIST’s synthesis of the results of two different questions, one open-ended and one closed-ended, does not adequately support NIST’s statements, nor does the information provided by another question in the survey (section L.). 20. We included all of the relevant information in our analysis, and our conclusions remain unchanged. We agree with NIST’s comment on the National Cooperative Research Act and feel that NIST should have included references such as this in Setting Priorities to avoid overstating any potential effects of ATP on the formation of joint ventures. As we pointed out, the effects of the National Cooperative Research Act make it difficult to determine the effects of ATP on the number of joint ventures during ATP’s first four competitions. 21. Joint R&D ventures exist throughout industry and are not required to register with the Department of Justice or the Federal Trade Commission. Therefore, the joint ventures may have been formed before applying to ATP and may never have applied to the Department of Justice or the Federal Trade Commission. 22. None of this information shows that ATP caused 125 joint ventures to form. As shown in comment 21, registration with the Department of Justice or the Federal Trade Commission does not mean a joint venture is new and did not exist before the time of registration. 23. As noted in comments 20, 21, and 22, the current evidence supporting ATP’s impact on the formation of joint ventures is inconclusive and anecdotal. Moreover, the NIST statement says that “approximately 125 joint ventures,” i.e., all of the joint ventures that sent in a proposal to ATP, were formed because of ATP. We look forward to the completion of NIST’s new survey, which is under way, for more definitive information on “whether or not the ATP had any influence on [ATP award recipients’] decision to collaborate.”

Major Contributors to This Report

Resources, Community, and Economic Development Division, Washington, D.C.

Office of the Chief Economist

Loren Yager, Assistant Director The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.