NIST Economic Analysis Briefs 6 

NIST Impact on Patenting 

Gary Anderson (gary.anderson@nist.gov) 

Technology Partnerships Office, Innovation & Industry Services 

October 2016 Key Findings 

•	 NIST has a large and growing impact on invention - two orders of magnitude greater than indicated by 

NIST-invented patents alone. 

•	 Laboratory practices that prioritize only one type of technical output could negatively impact NIST’s 

ability to effectively transfer technology. 

•	 Technology transfer policy focused primarily on patented inventions will miss opportunities to 

enhance impacts through alternative outputs. 

•	 Growth in citations to NIST laboratory outputs exceeds comparison groups. 

•	 Past NIST extramural programs have long-lasting and increasing impact on private sector invention. 

Introduction 

Called to action by a 2011 Presidential Memorandum and recognizing the varied approaches the 

institute takes to ensuring that knowledge, capabilities and facilities produced and managed by the 

National Institute of Standards and Technology (NIST) are disseminated to its stakeholders, NIST 

adopted a new and broad definition of technology transfer. Technology transfer is the overall 

process by which NIST knowledge, facilities, or capabilities in measurement science, standards and 

technology promote U.S. innovation and industrial competitiveness in order to enhance economic 

security and improve quality of life. This encompasses knowledge transfer to individuals and 

organizations as well as commercialization or adoption of NIST research outputs by businesses and 

other organizations. Under this broad definition, previous technology transfer metrics do not capture 

the totality of NIST impacts on invention. NIST recently commissioned an analysis to 

comprehensively examine NIST impact on inventive activity. 

NIST and Inventive Activity 

Historically, NIST has measured its impact on inventive activity by counting the number of patents 

issued each year to NIST-employed inventors. Annually, the number of NIST-assigned patents varies 

but is rarely over twenty. Given that inventors refer to these patents as prior art, reference NIST 

publications, use NIST data and reference materials to calibrate their scientific equipment, reference 

NIST software and algorithms and cite less formal outputs such as NIST workshop presentations, it is 

likely that the count of NIST-assigned patents understates the true impact of NIST on inventive 

activity. 

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards 1 

and Technology and Director 
 

 

Figure 1: NIST-Assigned Patents 

0 

5 

10 

15 

20 

25 

30 

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 

Role of Citation Analysis 

While inventors are required to cite prior art in their patent applications, researchers have found 

that these references are an important indicator of knowledge flow.1 Even if these references are a 

“bit noisy,”2 studies show there is a strong positive correlation between citations and technological 

importance3 as indicated by awards,4 expert judgement5 and continued payment of maintenance 

fees.6 Research has also found a positive relationship between patent citation indicators and 

business financial performance.7 Further, patent to publication citations have been used to indicate 

application of science to technology8 as well as to measure knowledge transfer from public9 and 

federal scientific institutions.10 Finally, patent references have been used to demonstrate knowledge 

transfer from voluntary consensus standards groups.11 

Methodology 

To identify the range of citations to NIST, the analysis first assembles data on an array of outputs of 

NIST research. The report leverages existing data on NIST-invented intellectual property maintained 

by the Technology Partnerships Office, employs a novel database containing all NIST peer-reviewed 

Table 1. NIST Technical Outputs 

Traditional Technical Outputs Grey Literature 

1. NIST-Assigned Patents 4. Educational Networking 

2. NIST Government Interest Patents 5. Software/Standard Reference Databases/Algorithms 

3. NIST-Authored Peer-Reviewed 

Publications 

6. Standard Reference Materials/Resource 

Materials/General Information 

7. Official NIST Publications (not peer-reviewed journals) 

8. Joint Partnership Publications (not peer-reviewed 

journals). 

9. Other NIST (non peer-reviewed) publications 

10. Communication / Correspondence / Inquiries 

11. “Other” 

2 
 

 

 

 

 

 

 

publications since 1900 constructed using NIST’s subscription to Thompson-Reuter’s Web of 

Science,12,13 and develops search algorithms to identify a set of patents that cite NIST in the 

Government Interest Section of patents.

14 References to these NIST outputs indicate that businesses 

and other organizations have adopted and used NIST knowledge, facilities, or capabilities in their 

own inventions. Additionally, the analysis develops algorithms to identify any other references to 

NIST technical outputs cited in the non-patent references section of patents. Together, this array of 

additional outputs is referred to as “grey literature.” The analysis analyzes the role of grey literature 

as a whole and considers the impact of distinct types of technical outputs such as Standard 

Reference Data, Standard Reference Materials, official NIST publications and less formal outputs 

such as conference and workshop presentations. 

Results 

Figures 1 and 2 below show the magnitude of NIST impact on inventive activity and highlight the 

diverse channels through which these impacts are felt. In total, there are over 34,000 patent 

references to NIST research outputs between 1970 and 2014. The plurality of these references are to 

NIST peer-reviewed publications but references to NIST grey literature and government interest 

patents15 each account for over 25% of the references. Nearly all of the Government Interest 

Patents are inventions created by companies funded through NIST’s Advanced Technology Program 

and Technology Innovation Program. Over time, the number of annual citations has risen. In 2014, 

there were more than 4,500 references in more than 3,000 unique patents. Given overall increases 

in patenting and natural growth in the collection of NIST outputs, additional analysis is needed to 

identify the extent to which these trends indicate an increase in impact or relevance rather than 

general trends in patenting and an accumulation of NIST outputs. Breitzman and Thomas (2016) use 

standard normalization techniques to examine citations to NIST patents. While NIST assigned patents 

underperform patents with a similar technological focus and age, the limited number of patents – 

just 122 patents between 2005 and 2014 – diminishes the significance of this result. On the other 

hand, NIST Government Interest Patents are cited over 60% more frequently than expected. Due to 

the novelty of the approach used in this paper, there are no established normalization techniques for 

Figure 1: Citations to NIST Technical Outputs 1975-2014 

2081, 6% 

9154, 27% 

14538, 42% 

8468, 25% 

NIST Assigned Patents 

NIST Government Interest 

Patents 

NIST Peer-Reviewed 

Publications 

NIST Grey Literature 

3 
 

 

 

 

 

 

 

 

patent to publication citations. Therefore, patent to NIST paper and patent to NIST grey literature 

trends were compared to publications from a selective group of journals and a leading academic 

institution respectively. The growth in patent citations to NIST publications has outpaced the growth 

in patent citations from the 40 most active patenting companies to papers in leading journals. 

Additionally, the growth in patent references to NIST Grey Literature has marginally exceeded the 

growth in references to grey literature from the Massachusetts Institute of Technology.16 Breitzman 

and Thomas [7] find that the performance of NIST technical outputs in generating subsequent 

citations is “impressive.” 

Figure 2: Patent Citations to NIST Technical Outputs 

0 

500 

1000 

1500 

2000 

2500 

3000 

3500 

4000 

4500 

5000 

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 

Number of Citations 

NIST Assigned Patent NIST Government Interest Patent 

Peer-Reviewed Publication NIST Grey Literature 

The report also examined how different companies, industries, and technology sectors cite various 

types of NIST technical outputs. While the full report looks at both grey literature as a whole and 

examines the more detailed categorization identified in Table 1, for brevity Figure 3 only shows the 

results by industry and detailed classification scheme. The results clearly show both that particular 

industries rely on a variety of NIST technology transfer tools and that there is significant variation 

across inventors with respect to the type of NIST technical output cited. So while industrial 

equipment relies heavily on peer-reviewed publications, grey literature is the most frequently relied 

upon technology transfer tool within the software industry. Different types of NIST technical outputs 

- different technology transfer tools - are used by different sectors and companies. 

Findings 

The variety of NIST technical outputs considered in this analysis closely corresponds to the 

technology transfer tools identified in NIST’s response to the 2011 Presidential Memorandum on 

Technology Transfer. The data presented in Figures 1 and 2 show that NIST has a large and growing 

impact on invention. NIST impact on invention is at least two orders of magnitude greater than 

indicated by counting NIST-assigned patents. Counting patents attributed to NIST-employed 

4 
 

 

 

 

Figure 3: Usage of NIST Technical outputs by Industry 

inventors neither indicates that the invention is used by other companies or organizations nor 

accounts for the variety of channels through which NIST impacts inventive activity. As discussed 

above, citations are a noisy but important indicator of knowledge transfer. Identifying the magnitude 

and varied usage of different types of NIST outputs represents a significant improvement in NIST 

impact metrics. 

Further, the results clearly indicate that different companies, industries and technologies rely on 

different types of NIST technical outputs. That is to say different stakeholder groups rely in different 

modes of technology transfer. Because of this variation in stakeholder usage of types of NIST 

technical outputs and technology transfer tools, any policy that prioritizes or encourages one type of 

NIST technical output over another could negatively impact NIST’s ability to transfer technology. 

Similarly, technology transfer policy that focuses primarily on increasing the number of NIST 

inventions will miss significant opportunities to enhance NIST impacts on invention through other 

channels. 

The results also indicate the high quality of NIST scientific and technical outputs. The growth rate in 

citations to NIST publications and grey literature outpaces prestigious and highly selective 

comparison groups like MIT. Finally, Figure 2 shows that NIST extramural programs have long-lasting 

and increasing impact on private sector invention. The number of citations to patents resulting from 

the NIST’s Advanced Technology Program and Technology Innovation Program continue to increase 

even though these programs have been discontinued. NIST government-interest patents have over 

60% more citations that patents of identical age and technology area. Further, citation indices show 

that the magnitude and breadth of the impact of these patents is highest for more recent time 

periods. 

5 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

References 

[1] Albert M, Avery D, McAllister P Narin F (1991) Direct Validation of Citation Counts as Indicators of 

Industrially Important Patents Research Policy 20: 251-259. 

[2] Anderson G (2016) NIST Impact on Patenting. (NIST, Gaithersburg, MD) NIST Economic Analysis 

Briefs 6. 

[3] Azoulay P, Graff Zivin JS, Sampat BN (2011) The Diffusion of Scientific Knowledge Across Time and 

Space: Evidence from Professional Transitions for the Superstars of Medicine National Bureau of 

Economic Research, Working Paper 16683. 

[4] Breitzman A, Mogee M (2002) The Many Applications of Patent Analysis Journal of Information 

Science 28 (3), 187–205. 

[5] Breitzman A (2009) Analysis of Patent Referencing to IEEE Papers, Conferences, and Standards 

1997-2008, 1790 Analytics, LLC. 

[6] Breitzman A (2012) Analysis of Patent Referencing to IEEE Papers, Conferences, and Standards 

1997-2012, 1790 Analytics, LLC. 

[7] Breitzman A, Thomas P (2016) Analysis of References from US Patents to NIST-Supported 

Technical Outputs. (NIST, Gaithersburg, MD) NIST GCR 16-009. 

[8] Carpenter M, Narin F, Woolf P (1981) Citation Rates to Technologically Important Patents World 

Patent Information 4, 160-163. 

[9] Zhen D, Lev B, Narin F (1999). Science and technology as predictors of stock performance 

Financial Analysts Journal, 55(3), 20–32. 

[10] Harhoff D, Narin F, Scherer FM, Vopel K (1999) Citation frequency and the value of patented 

inventions The Review of Economics and Statistics 81(3), 511 - 515. 

[11] Jaffe A, Lerner J (2004) Innovation And Its Discontents : How Our Broken Patent System Is 

Endangering Innovation And Progress, And What To Do About It (Princeton University Press, 

Princeton, NJ, USA). 

[12] Jaffe A, Lerner J (2001). Reinventing public R&D: Patent policy and the commercialization of 

national laboratory technologies The RAND Journal of Economics 32(1) 167-198. 

[13] Jaffe A, de Rassenfosse G (2016) Patent citation data in social science research: Overview and 

best practices NBER Working Paper No. 21868. 

[14] Jaffe A, Trajtenberg M (2002) Patents, Citations and Innovations: A Window on the Knowledge 

Economy, (M.I.T. Press, Boston, MA, USA). 

[15] Jaffe AB, Trajtenberg M, Fogarty MS (2000) Knowledge spillovers and patent citations: Evidence 

from a survey of inventors American Economic Review 90(2) 215-218. 

[19] Narin F, Noma E, Perry R (1987) Patents as indicators of corporate technological strength 

Research Policy 16 143-155. 

[20] Narin F, Hamilton KS, Olivastro D (1997) The increasing linkage between US technology and 

public science Research Policy 26 (3) 317-330 

[21] Popp D (2016) Economic analysis of scientific publications and implications for energy research 

and development Nature Energy, Article number: 16020 doi:10.1038/nenergy.2016.20. 

[22] Roach M, Cohen W (2013) Lens or prism? Patent citations as a measure of knowledge flows 

from public research Management Science 59(2) 504–525. 

[23] Ruegg R, Thomas P (2008) Linkages of DOE's Energy Storage R&D to Batteries and 

Ultracapacitors for Hybrid, Plug-In Hybrid, and Electric Vehicles, Office of Energy Efficiency and 

Renewable Energy, United States Department of Energy. 

6 
 

 

 

 

 

 

 

 

 

 

 

 

[24] Ruegg R, Thomas P (2009) Linkages from DOE's Wind Energy Program to Commercial 

Renewable Power Generation, Office of Energy Efficiency and Renewable Energy, United States 

Department of Energy. 

[25] Ruegg R, Thomas P (2011) Linkages from DOE's Solar Photovoltaic R&D to Commercial 

Renewable Power Generation from Solar Energy, Office of Energy Efficiency and Renewable 

Energy, United States Department of Energy. 

[26] Ruegg R, Thomas P (2011) Linkages from DOE's Geothermal R&D to Commercial Power 

generation, Office of Energy Efficiency and Renewable Energy, United States Department of 

Energy. 

[27] Ruegg R, Thomas P (2011) Linkages from DOE's Vehicle Technologies R&D in Advanced 

Combustion to More Efficient, Cleaner-Burning Engines Office of Energy Efficiency and 

Renewable Energy, United States Department of Energy. 

[28] Sheft J (2008) Technology transfer and idea commercialization Nature Biotechnology 26 711 – 

712. 

[29] Thomas, Patrick and Francis Narin (2004) System and Method for Producing Technology-Based 

Price Targets for a Company Stock. US Patent #6,832,211. 

1 See Jaffe and Lerner [11] and Jaffe and Trajtenberg [14] for an overview and critique.
 2 See Jaffe et al. [15].
 3 See Breitzman and Mogee [4].
 4 See Carpenter et al. [8].
 5 See Albert et al. [1].
 6 See Haroff et al. [10].
 7 Research has examined correlation with market valuations (Zhen et al. [9]), stock price movements
 

(Thomas and Narin [29]), as well as sales and profitability (Narin et al. [19]).
 8 For recent examples, see Breitzman [5,6]and Azoulay at al. [3].
 9 Roach and Cohen [22]
 10 See Narin et al. [20] and Reugg and Thomas [23], [24], [25] and [26].
 11 See Breitzman [5] and [6].
 12 A commercially available product is identified in this paper in order to specify the experimental
 

procedure adequately. Such identification is not intended to imply recommendation or endorsement by 

NIST, nor is it intended to imply that the materials or equipment identified are necessarily the best
 

available for the purpose.
 13 NIST subscribes to the Science Citation Index Expanded and Conference Proceedings Citation IndexScience databases.
 14 Jaffe and Lerner [12] demonstrate both assignees and government interest sections need to be
 

searched to identify the complete set of government inventions.
 15 The data demonstrate that over 90% of the patents identified through the Government Interest search
 

are attributable to NIST extramural programs. Citations to these patents are a NIST impact but not
 

indicative of technology transferred from NIST laboratory research programs.
 16 Due to a lack of established benchmarks and normalization approaches for publications and grey 

literature, growth rates in publication and grey literature citation rates were compared to previous work 

performed by 1790 Analytics. For more details, please see Breitzman and Thomas [7].
 

7