Measuring Performance: Challenges in Evaluating Research and Development
Published: 1997-04-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO discussed its work related to the National Institute of Standards and Technology's Advanced Technology Program (ATP), Manufacturing Extension Partnership (MEP) Program, and research and development (R&D) performance measures.
section_title: What GAO Found
paragraphs: GAO noted: (1) ATP has funded research projects that would have been funded by the private sector as well as those that would not; (2) the award recipients were nearly evenly divided when asked if they would have pursued their projects if they had not received such funding; (3) GAO also found that in most cases, the participants in its survey did not look for funding from other sources, private or public, before trying to obtain funding from ATP; (4) about half of the 45 applicants that tried to find funding elsewhere before turning to ATP were told by prospective funders that their projects were either too risky or "precompetitive", characteristics that fulfill the aims of ATP; (5) manufacturers viewed the manufacturing extension programs' services positively, as was demonstrated in GAO's national survey of manufacturers who had received substantive services from the programs in 1993; (6) most manufacturers responding to GAO's questionnaire, about 73 percent, reported that they believed that the type of assistance they had received from these programs had positively affected their overall business performance; (7) about 15 percent of the respondents reported that they believed the programs' assistance had not affected their overall business performance; (8) the amount of money spent on R&D, the primary indicator of research investment, is useful as an input measure of how much research is being performed; (9) however, the level of spending is not a reliable indicator of research results; (10) GAO found that there is no primary indicator of R&D results; (11) the companies that GAO spoke with collect data on various output indicators, such as return on investment and patents granted, but in general make limited use of them in their investment decisions; (12) instead, they emphasized that R&D contributes directly to the bottom line; (13) because companies are profit oriented, many of the indicators tracked by the private sector cannot be directly applied to the federal government; (14) determining the specific outcomes resulting from federal R&D is a challenge that will not be easily resolved; and (15) however, in response to recent legislation requiring agencies to report on program results, some progress is being made in measuring the impacts of research.

section_title: Letter
paragraphs: We are pleased to be here today to discuss our work related to two programs at the National Institute of Standards and Technology (NIST): the Advanced Technology Program (ATP) and the Manufacturing Extension Partnership (MEP) Program. In addition, we will be discussing our work performed in response to the Subcommittee’s request regarding research and development (R&D) performance measures. This work resulted in our report entitled Performance Measurement: Strengths and Limitations of Research Indicators (GAO/RCED-97-91), which is being released today in conjunction with this hearing.
Our report on measuring R&D performance highlights the difficulty in measuring the impact of technology programs such as ATP and MEP. Our work on these programs was performed in response to specific congressional concerns. In the case of ATP, we were asked to assess the impact of the program. In the case of MEP, we obtained manufacturers’ views on (1) the impact of the services provided by the program, as well as other manufacturing extension programs, on their business performance and (2) the factors affecting the impact of these services. In the case of our report on performance measurement, this Subcommittee asked us to evaluate various indicators that are used to measure the results of R&D.
In summary, our work showed that:
The Advanced Technology Program has funded research projects that would have been funded by the private sector as well as those that would not. The award recipients were nearly evenly divided when asked if they would have pursued their projects if they had not received such funding. We also found that in most cases, the participants in our survey did not look for funding from other sources, private or public, before trying to obtain funding from the Advanced Technology Program. About half of the 45 applicants that tried to find funding elsewhere before turning to the Advanced Technology Program were told by prospective funders that their projects were either too risky or “precompetitive”—characteristics that fulfill the aims of the Advanced Technology Program.
Manufacturers viewed the manufacturing extension programs’ services positively, as was demonstrated in our national survey of manufacturers who had received substantive services from the programs in 1993. Most manufacturers responding to our questionnaire—about 73 percent—reported that they believed that the type of assistance they had received from these programs had positively affected their overall business performance. About 15 percent of the respondents reported that they believed the programs’ assistance had not affected their overall business performance.
The amount of money spent on research and development, the primary indicator of research investment, is useful as an input measure of how much research is being performed. However, the level of spending is not a reliable indicator of research results. We found that there is no primary indicator of research and development results. The companies that we spoke with collect data on various output indicators, such as return on investment and patents granted, but in general make limited use of them in their investment decisions. Instead, they emphasized that research and development contributes directly to the bottom line. Because companies are profit oriented, many of the indicators tracked by the private sector cannot be directly applied to the federal government. Determining the specific outcomes resulting from federal research and development is a challenge that will not be easily resolved. However, in response to recent legislation requiring agencies to report on program results, some progress is being made in measuring the impacts of research.
subsections: section_title: Background
paragraphs: ATP’s mission is to stimulate economic growth in the United States through technology development. The program seeks to accomplish that mission by sharing the cost of R&D projects with private industry. The projects selected by ATP for funding are characterized by the program as having “a potential broad-based economic impact but a relatively high technical risk and a long time horizon.”
ATP’s program guidance has stated that if the technical risk associated with a project is very low, federal funding should not be necessary. In addition, when submitting a research proposal, applicants must sign a form stating that “this proposal is not requesting funding for existing or planned research programs that would be conducted in the same time period in the absence of financial assistance under the ATP.” This wording suggests that ATP should not fund projects that other sources would have funded or, when ATP does fund such projects, that ATP funds should enable applicants to complete their projects in a shorter time.
Manufacturing extension programs offer manufacturers assistance in modernizing or upgrading their operations, often with state and federal funding. NIST manages federal funding of this type of program through its Manufacturing Extension Partnership Program, or MEP. In our prior reports, we used MEP to collectively refer to all state, federal, and university manufacturing extension programs.
The primary mission of manufacturing extension programs is to give “hands-on” technical assistance to small- and medium-sized manufacturers trying to improve their operations through the use of appropriate technologies. These programs engage in a variety of activities to assist small- and medium-sized manufacturers, often in partnership with other business assistance providers, such as Small Business Development Centers, community colleges, and federal laboratories. The programs offer a wide range of business services, including helping companies (1) solve individual manufacturing problems, (2) obtain training for their workers, (3) create marketing plans, and (4) upgrade their equipment and computers. The assistance focuses on small- and medium-sized manufacturers because research by the National Research Council and others has indicated that these companies lack the resources necessary to improve their manufacturing performance.
section_title: The Advanced Technology Program
paragraphs: In our work on ATP, our objective was to examine, as one way to assess the program’s impact, whether research projects would have been funded by the private sector if they had not received funds from ATP. We also examined ATP’s impact in terms of other goals of the program, such as aiding the formation of joint ventures.
We focused on two groups of ATP applicants, which we called “winners” and “near winners.” Both groups submitted proposals that were rated highest during ATP’s review, but the near winners did not ultimately receive ATP funding. We surveyed all applicants that qualified as winners or near winners during ATP’s first 4 years (1990-93). We achieved a 100-percent response rate from the 123 respondents that we included in our analysis (89 winners and 34 near winners).
We found that ATP had funded research projects that would have been funded by the private sector as well as those that would not. The winners were nearly evenly divided when asked if they would have pursued their projects even if they had not received ATP funding. Half of the near winners continued their projects without relying on ATP funding, while the other half discontinued their projects for various reasons. Almost all the near winners that continued their projects did so on a modified schedule, meeting the projects’ milestones later than they had scheduled in their proposals to ATP.
Of the 123 applicants we surveyed, 77, or 63 percent, did not look for funding from other sources before requesting it from ATP. Those applicants that did look for funding looked for a long time and made many attempts to find funding, on average. Seven applicants turned down offers from private sources because they could not reach an acceptable funding arrangement.
We also found that ATP had other effects. More than three-fourths of the joint-venture applicants indicated that they had come together solely to pursue an ATP project, thus satisfying ATP’s goal of serving as a catalyst for the formation of joint ventures. Furthermore, of the 45 applicants that tried to find funding elsewhere before turning to ATP, about half were told by prospective funders that their projects were either too risky or precompetitive—characteristics that fulfill the aims of ATP funding.
section_title: Manufacturing Extension Program Services
paragraphs: We surveyed 766 U.S. manufacturers that had completed at least 40 hours of manufacturing extension program assistance, including NIST’s MEP, and received 551 responses. We obtained respondents’ views on the impact of these services on their business performance and on the factors affecting the impact of these services. We did not verify either the positive or negative impacts reported by manufacturers, nor did we evaluate the operations or management of specific federal or state programs. We also obtained the views of other manufacturers that had little or no experience with these programs to determine why they made little or no use of them.
Most manufacturers responding to our questionnaire—about 73 percent—reported that they believed the programs’ assistance had positively affected their overall business performance. About 15 percent of the respondents reported that they believed the assistance had not affected their overall business performance. Approximately 8 percent said that it was too early to determine the effect, and another 4 percent said they had had no basis to estimate the effect.
In addition, most respondents reported that the assistance had positively affected their use of technology in the workplace (about 63 percent), the quality of their product (about 61 percent), and the productivity of their workers (about 56 percent). Between about 44 percent and 63 percent of the respondents reported that the programs’ assistance had positively affected certain specific indicators of their business performance, such as customer satisfaction, profits, and the ability to meet production schedules. Of those respondents not reporting a positive impact on specific indicators of their business performance, most said the programs’ assistance had not had any impact. Two percent or fewer of the respondents reported a negative impact on each specific performance indicator.
Among the factors that manufacturers said had affected the impact of MEP services was their own companies’ input. The companies that had committed their own financial resources to implement the programs’ recommendations reported greater benefits from the assistance relative to other survey respondents. Of those 322 respondents who had made a financial investment, 86 percent said that the programs’ assistance had positively affected their business performance. However, 54 percent of those who had not made a financial investment also reported an overall positive impact. Other factors, according to the respondents, that influenced the effectiveness of the programs’ services were the expertise and experience of the programs’ staff and the affordability of the assistance.
In our related telephone survey of 200 additional manufacturers who were not extensive users of the programs’ services, about 82 percent reported that they had not used the services because they were unaware of these programs. About 10 percent said that although they knew about these programs, they had not used them because they believed the assistance would not be necessary. The companies we interviewed said that other sources of modernization assistance besides these programs were their customers, vendors and/or suppliers, industry associations, and consultants.
section_title: Measuring the Impacts of Federal Research Programs
paragraphs: The report we are releasing today on performance measurement shows that there is no single indicator or evaluation method that adequately captures the results of R&D. However, indicators do provide helpful information for making decisions about R&D. Whether the focus is on basic research, applied research, or development, the amount of money spent in that area is taken as an indication of how much research is being performed. The major advantages of using expenditure data as an indicator are that they are easily understandable, readily available, and have been, in general, consistently gathered over time. In addition, spending on different projects in different research areas can be measured according to the same unit, dollars, making comparisons between projects straightforward.
The amount of funding, however, does not provide a good indication of research results. Companies told us that they are switching their spending to more short-term R&D projects rather than long-term projects. However, the impacts of that change are unclear. The reduced funding levels for long-term projects may not reflect the fact that the R&D efforts can be performed with greater efficiency. For example, one way in which the federal government and the private sector have tried to use R&D resources more efficiently and effectively is through consortia with universities or other companies. By combining their research activities, companies attempt to avoid expensive duplication and learn from each other.
We also found that because of the difficulties in identifying the impacts of research, quantitative and qualitative indicators have been developed as proxies to assess R&D results. The strengths and limitations are evident in both types of indicators. Quantitative indicators focus mainly on return on investment, patenting rates, and bibliometrics—the study of publication-based data. While implying a degree of precision, these indicators were not originally intended to measure long-term R&D results. Qualitative assessment such as peer review provides detailed information, but it relies on the judgments of experts and may be expensive.
Because of these difficulties, the companies we interviewed stressed marketplace results rather than R&D output indicators. While varying in the types of indicators they collect, they emphasized the difficulties in measuring R&D’s specific contribution to a company’s overall performance. For example, one company stated that because so many people have been involved in a product’s evolution, it is difficult to separate the contribution of the research unit from that of other units. All of the companies interviewed have increased their expectation that R&D contribute directly to their profitability. However, instead of increasing their efforts at measuring R&D results, they have shifted the responsibility for making R&D decisions to the business units. For example, if the business units believe that a particular R&D project would increase their profits, the firm would budget for that R&D.
Many of the R&D output measures tracked by the private sector do not apply directly to the federal government. In particular, while facing the same increasing cost pressures as the private sector, the federal government cannot rely on the profit motive to guide its decisions.
This discussion of performance measures for R&D is particularly relevant because of the current emphasis on the Government Performance and Results Act (GPRA). In response to questions about the value and effectiveness of federal programs, GPRA seeks to shift federal agencies’ focus away from such traditional concerns as staffing, activity levels, and tasks completed toward a focus on program outcomes. GPRA incorporates performance measurement as one of its most important features. Under this act, executive branch agencies are required to develop annual performance plans that use performance measurement to reinforce the connection between the long-term strategic goals outlined in their strategic plans and the day-to-day activities of their managers and staff. However, the very nature of the innovative process makes measuring the performance of science-related projects difficult. For example, a wide range of factors determines if and when a particular R&D project will result in commercial or other benefits. It can also take many years for a research project to achieve results.
Experiences from recent GPRA pilot efforts reinforce the fact that output measures are highly specific to the management and mission of each federal agency and that no single indicator exists to measure the results of research. The Army Research Laboratory, which was designated as a pilot project for performance measurement under the act, has developed a multifaceted approach using quantitative indicators, peer review, and customer feedback to evaluate the results of R&D. Although this is not the only approach that can be taken, this response to the challenges in measuring the impacts of research shows that some progress is being made in response to GPRA.
Madame Chair, this concludes my prepared remarks. I would be happy to respond to any questions you may have.
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.