Dynamics of Disruption in Science and Technology

Although the number of new scientific discoveries and technological inventions has increased dramatically over the past century, there have also been concerns of a slowdown in the progress of science and technology. We analyze 25 million papers and 4 million patents across 6 decades and find that science and technology are becoming less disruptive of existing knowledge, a pattern that holds nearly universally across fields. We link this decline in disruptiveness to a narrowing in the utilization of existing knowledge. Diminishing quality of published science and changes in citation practices are unlikely to be responsible for this trend, suggesting that this pattern represents a fundamental shift in science and technology. ∗E-mail park1892@umn.edu, leahey@arizona.edu, or rfunk@umn.edu. We thank the National Science Foundation for financial support of work related to this project (grants 1829168 and 1932596) and the participants of the CADRE workshop for their feedback. 1 ar X iv :2 10 6. 11 18 4v 1 [ ec on . G N ] 2 1 Ju n 20 21 While the past century witnessed an unprecedented expansion in scientific and technological knowledge, there are growing concerns about a possible slowing of innovative activity [Jones, 2009, Gordon, 2016, Cowen and Southwood, 2019, Bhattacharya and Packalen, 2020]. Studies have documented declining research productivity in semiconductors, pharmaceuticals, agriculture, and other fields [Pammolli et al., 2011, Bloom et al., 2020]. Over time, papers, patents, and even grant applications have become less novel and less likely to connect disparate areas of knowledge, both of which are important precursors of innovation [Foster et al., 2015, Mukherjee et al., 2016, Packalen and Bhattacharya, 2020, Jaffe and Lerner, 2011]. The gap between the year of discovery and the awarding of a Nobel Prize has also increased [e.g., Horgan, 2015, Jones and Weinberg, 2011, Collison and Nielsen, 2018], suggesting that today’s contributions may not measure up to those of the past. Numerous explanations for this apparent slowdown have been proposed. Some point to a dearth of “low hanging fruit,” as the easier and more obvious innovations have already been produced [Arbesman, 2011, Cowen, 2011, Gordon, 2016]. Others suggest that the decline may be due to an increasing burden of knowledge; scientists and technologists require more training to reach the frontiers of their fields, leaving less time for making breakthroughs [e.g., Einstein, 1949, Jones, 2009, Chu and Evans, 2018]. Yet much remains unknown, not merely about the causes of slowing innovative activity, but also the depth and breadth of the phenomenon. To date, the evidence pointing to a slowdown is based largely on studies of particular fields, using disparate and domain-specific metrics [Pammolli et al., 2011, Bloom et al., 2020], making it difficult to know whether the observed changes are happening at a similar rate across areas of science and technology. Little is also known about whether the patterns seen in aggregate indicators may mask important differences in the degree to which individual works push the boundaries of science and technology. In this study, we address these gaps in knowledge by analyzing 25 million research papers from 1945 to 2010 in the Web of Science database (“WoS data”) and 4 million utility patents from 1976 to 2010 in the United States Patent and Trademark Office’s Patents View database (“USPTO data”). The WoS data include 159 million citations and the text of 28 million paper titles and abstracts. The USPTO data include 18 million citations and the text of 6 million patent titles and abstracts. Using these data, we join a novel citation-based measure [Funk and Owen-Smith, 2017] with textual analyses of titles and abstracts to understand whether papers and patents forge new directions in science and technology over time and across fields. To characterize the nature of innovation, we draw on insights from foundational theories of scientific and technological change [Arthur, 2007, Schumpeter, 1947], which distinguish between two types of breakthroughs. First, some contributions improve and refine existing streams of knowledge, and therefore consolidate the scientific or technological status quo. For example, the Nobel Prize winning paper1 Kohn and Sham [1965] (“KS”) utilized established theorems to develop a method for calculating the structure of electrons, which cemented the value of prior research. Second, some contributions disrupt existing streams of knowledge, rendering them obsolete, and propelling future scientific or technological work in new directions. As an illustration, Watson and Crick [1953] (“WC”), also a Nobel winner, introduced a new (and better) model of the structure of DNA, which superseded previous approaches and detracted attention away from that prior work (e.g., Pauling’s triple helix model). KS and WC were both important papers, but their implications for scientific and technological change were dramatically different. To quantify this distinction, we utilize a novel measure—the CD index—which characterizes the consolidating/disruptive nature of science and technology based on the citation networks that form around papers or patents (fig. 1). The intuition underpinning this measure is that if a paper or patent is disruptive, the subsequent work that cites it is less likely to also cite its predecessors; for future scientists and technologists, the ideas that went into its production are less relevant (e.g., Pauling’s triple helix model). If a paper or patent is consolidating, subsequent work that cites it is also more likely to cite its predecessors; for future researchers, the prior knowledge upon which the paper or patent builds is still (and perhaps more) relevant (e.g., the existing theorems KS used to develop their method). The CD index ranges from -1 (most consolidating) to 1 (most disruptive). For our study, we measure the CD index five years after the year of publication (indicated by CD5). As an illustrative example, the WC paper and KS paper both received over a hundred citations within five years of publication, indicating major attention from subsequent work. However, measured within the same window, the KS paper has a CD5 index value of -0.22 (indicating consolidation), whereas the WC paper has a CD5 index value of 0.62 (indicating disruption). The CD5 index has been validated extensively 1Data on Nobel-Prize-winning papers are taken from Li et al. [2019].