Measuring the Effectiveness of R&D: R&D Metrics Continue to Be an Important Topic for Measuring the Effectiveness of R&D. Practitioners Share Their Issues and Recommendations

The creation of a set of metrics to measure the effectiveness of R&D has been a major need for research managers for some time. In recent surveys of Industrial Research Institute (IRI) participants, the need for metrics has ranked in the top three for the past three years (Cosner 2010). The enhanced importance of reliable metrics is being driven by several forces: the need to justify the investment in R&D to senior management, the desire to improve efficiency in the use of R&D resources, and the need for a means to estimate the value of the R&D investment for the future growth of the company. Because R&D tends to be both longer terra and more subjective than a sales or manufacturing target, effective metrics must encompass the broad influence R&D has over an organization, including such diverse elements as new products, cost savings, customer support, competitive evaluations, and IP protection. The difficulty in creating metrics is further complicated by the need for changing measures of R&D effectiveness as business conditions and technology strategies change. Over the years, there have been many attempts to create metrics to quantify the success or failure of R&D organizations. The result has been a proliferation of metrics, ranging from financial measures, such as R&D spending as a percent of sales (Andrew et al. 2008, Hauser 1996), to more complex measures, such as strategic alignment (Roussel, Saad, and Erickson 1991). A 2004 study by the European Industrial Research Management Association (EIRMA) lists over 250 potential R&D metrics (EIRMA 2004). While metrics clearly draw a high level of interest from R&D leaders, it isn't clear whether this continued focus is because of the inadequacy of existing choices for metrics or a lack of knowledge of the topic. To address these questions, the IRI formed the Measuring the Effectiveness of R&D Research-on-Research (ROR) Working Group in 2005 as a follow-up to previous ROR groups on the topic. After a review of the literature on R&D metrics, the ROR group polled members on the metrics they currently used. The subcommittee further attempted to evaluate whether metric choices were dependent on the type or industry of the organization. Using the Technology Value Pyramid (TVP), a rediscovered and modernized model created by a 1994 ROR group, the group classified and organized existing metrics, providing a guide for choosing relevant, useful metrics, and identified gaps in available metrics. The Technology Value Pyramid One of the key challenges of implementing R&D metrics is matching metrics to the various levels and functions of the R&D organization, so that metrics are meaningful to the appropriate personnel. A bench scientist's metrics, for instance, should be related to his or her accomplishments, while the CEO or general manager should have more overarching metrics related to the performance of the organization under his or her responsibility. A metric for financial return makes sense for a CEO or CTO but has little direct meaning for a bench scientist. Conversely, the number of patents issued may make sense as a measure of performance for the bench scientist but is not directly meaningful for the CEO. Categorizing metrics by their relevance to the particular components that make up the eventual value of the R&D investment addresses this by allowing metrics to be targeted to the most relevant levels of the organization. In this way, bench scientists can access relevant measures of their group's performance, while managers get the quantitative measures they need--not just to rate the return on the R&D investment but to answer the key question, "Are we doing the right things? " An earlier ROR group developed the TVP (Figure 1), a model that takes this approach to categorizing metrics. The TVP provides a hierarchy of metrics based on the fundamental elements of R&D value and the relationships of those elements to business results in the long and short term (Tipping, Zeffren, and Fusfeld 1995). …