NIST Economic Analysis Briefs 7 

The Impact of NIST Laboratory Outputs on Innovation
 Gary Anderson (gary.anderson@nist.gov) 

Technology Partnerships Office, Innovation & Industry Services 

October 2016 

Key Findings 

•	 Patent citations indicate that over 90% of NIST’s impact on invention and innovation occurs through 

NIST peer-reviewed and non-academic publications. 

•	 Increased focus on the dissemination of peer-reviewed and non-academic publications can increase 

NIST’s impact on invention and innovation from current outputs. 

•	 Performance measurement focused on scientific impact alone may not ensure that NIST research has 

the technological impact adequate to meet its mission to promote U.S. innovation and industrial 

competitiveness. 

In response to the 2011 Presidential Memorandum -- Accelerating Technology Transfer and 

Commercialization of Federal Research in Support of High-Growth Businesses, the National Institute 

of Standards and Technology (NIST) adopted a new and broad definition of technology transfer. 

Technology transfer is the overall process by which NIST knowledge, facilities, or capabilities in 

measurement science, standards and technology promote U.S. innovation and industrial 

competitiveness in order to enhance economic security and improve quality of life. This 

encompasses knowledge transfer to individuals and organizations as well as commercialization or 

adoption of NIST research outputs by businesses and other organizations. This richer definition 

recognizes the varied approach NIST takes to ensure that knowledge, capabilities and facilities 

produced and managed by NIST are used by stakeholders. 

Evaluating the broader definition of technology transfer demands new metrics and approaches to 

measure total NIST impact on invention. A key goal of these novel approaches is to move from 

counting scientific outputs (e.g., papers and patents) produced by NIST to measuring usage of NIST 

science by NIST stakeholders. NIST recently commissioned analysis to increase the breadth of its 

technology transfer metrics by creating a dataset containing each instance when patents issued 

between 1970 and 2014 referred to a NIST technical output as prior art.1 The results of this earlier 

work show2: 

•	 NIST has a large and growing impact on innovation with growth in citations to NIST
 

laboratory outputs exceeding that in stringent comparison groups;
 

•	 Prioritization of a single type of technical output can negatively impact NIST’s ability to 

transfer technology; and 

•	 Narrowly focused technology transfer policy limited to patented inventions will miss 

opportunities to enhance impacts through alternative outputs. 

National Institute of Standards and Technology 

Willie May, Under Secretary of Commerce for Standards 1 

and Technology and Director 
 

 

 

 

-

This analysis leverages the data created in the earlier project with the aim to develop a better 

understanding of how NIST might engage in activities to increase those impacts. 

Results - Citations to NIST Research Outputs 

The number of patents issued each year to NIST-employed inventors understates NIST’s true impact 

on invention and does not indicate usage of NIST scientific output by other inventors. In contrast, 

economic research finds that patent references to prior art and publications are an important 

indicator of knowledge flow.3 While prior research analyzed the role of patent citations to other 

patents and peer-reviewed publications in technology transfer, a significant portion of NIST scientific 

outputs lie outside these two categories. NIST produces a wide variety of research outputs such as 

Standard Reference Materials, Standard Reference Data, NIST special publications, workshop 

presentations, and other informal scientific outputs. For the purpose of this analysis, these outputs 

are defined as “Grey Literature.” Because different NIST stakeholders rely on different types of 

scientific outputs, 4 it is important to capture this broad set of NIST outputs. Table 15 shows the total 

number of patents issued to NIST inventors6 and the total number of peer-reviewed publications 

authored by NIST researchers, as well as an indicator of the volume of grey literature outputs. 

Table 1: Patent Citations to NIST Laboratory Technical Outputs 1970-2015 

NIST Laboratory 

Output 

Total 

Output 

Total 

Patent 

Citations 

% Cited by 

Patents 

Cited NIST Outputs 

Median 

(Maximum) 

Number of 

Citations 

Median 

(Maximum) Years 

Prior to Initial 

Citation 

NIST Assigned 

Patents 

197 2,081 63.5% 6 (233) 0 (11) 

NIST Peer 

Reviewed 

Publications 

54,066 14,538 6.6% 1 (277) 8 (106) 

NIST Grey 

Literature 

N/A7 8,468 N/A 1 (336) 8 (84) 

The data show that the 1978 patents assigned to NIST have been cited over 2,000 times. Similarly, 

the 54,066 papers authored by NIST researchers have been cited 14,538 times as prior art in patents. 

In total, NIST grey literature generated 8,468 patent citations. These citations indicate substantial 

usage of NIST science. However, the data in Table 1 also make it clear that the rate and frequency 

with which technical outputs are cited in patents varies greatly. While 63.5% of NIST inventions are 

subsequently cited by patents, only 6.6% of NIST-authored papers are ever cited as prior art. 

Similarly, there is significant variation in the time before NIST outputs are first cited and the total 

number of citations that laboratory outputs receive. While there are highly cited NIST scientific 

outputs, the median number of citations indicates that even if scientific outputs are cited, they are 

typically cited by only a few patents. NIST assigned patents are often cited as prior art in the year the 

patent is issued, but NIST peer-reviewed publications and grey literature are typically cited for the 

first time eight years after publication. Figures 1 and 2 take a more in-depth look at the distribution 

of initial citation lags and total citations for NIST scientific outputs that have been cited by patents. 

2 
 

 

 

 

 

 

Not surprisingly, the initial lag of patent to patent citations is shorter than patent to publication 

citations. Notably, nearly 40% of NIST publications are published 10 or more years prior to receiving 

their initial patent citation. Among cited publications, 9% are published 25 or more years prior to 

their initial patent citation. Figure 2 confirms that when NIST scientific outputs are cited by patents, 

they are generally cited by a limited number of patents. This is particularly true for NIST publications. 

While 43% of NIST-assigned patents are cited more than 10 times, just 7% of cited NIST peerreviewed publications and 4% of cited NIST grey literature publications receive more than 10 

citations. 

0% 

20% 

40% 

60% 

80% 

100% 

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50+ 

% Cited Outputs 

Years 

Figure 1: Time to First Patent Citation 

NIST Assigned Patents Peer-Reviewed Publications Grey Literature Publications 

0% 

10% 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

100% 

1-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 51+ 

% Cited Outputs 

Number of Patent Citations 

Figure 2: Total Patent Citations per NIST Scientific Output 

NIST Assigned Patents Peer-Reviewed Publications Grey Literature Publications 

Figure 3 examines citations to NIST peer-reviewed publications. The chart9 shows how many times 

each publication was cited by a patent and how many times it was cited by a publication. For 

example, the marker near the lower right-hand corner shows a NIST article that was cited 4500 

times by other articles, but has been referenced by one patent as prior art. The figure shows there is 

generally a positive correlation between patent and paper citations. Of the 3582 papers that are 

cited as prior art in patents, 93.4% are cited by at least one paper. However, the figure also makes it 

3 
180 Patents 

160 

ts

140 

aten 120 

100

Citations by P

80 

60 

40 

20 

0 

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 

Citations by Publications 

Findings

very clear that there are papers that are highly cited by other publications but not cited as prior art 

in patents. Similarly, there are NIST papers that have a notable number of patent references, but no 

citations within the peer-reviewed literature. 

Figure 3: Peer-Reviewed Publications: Citations by Publications and Figure 3: NIST Peer-Reviewed Publications – Citations by Publications and Patents

Patent citations, or references by inventors to previous NIST laboratory scientific outputs, are an 

important indicator of knowledge transfer and indicate usage of NIST scientific outputs. Citations 

demonstrate the breadth of means by which NIST impacts invention and innovation. Significantly, 

over 90% of NIST’s impact on invention and innovation occurs through NIST peer-reviewed and grey 

literature publications.

Perhaps more importantly, the results demonstrate that these impacts on invention and innovation 

do not arise quickly or easily. Practitioners frequently refer to technology transfer as a “contact 

sport” that requires interaction between developers and users of scientific knowledge.10

Implementing a broad definition of technology transfer that recognizes the breadth of channels 

through which NIST knowledge, capabilities and facilities impact stakeholders demands increased 

participation in and attention to the dissemination and transfer of NIST science. Indeed, the Federal 

Technology Transfer Act of 1986 declared “Technology transfer, consistent with mission 

responsibilities, is a responsibility of each laboratory science and engineering professional.” Such 

efforts may decrease the time until inventors cite NIST science and increase the likelihood and 

frequency that NIST outputs are cited. Increased focus on the dissemination of peer-reviewed and 

non-academic publications can increase NIST’s impact on invention and innovation from the

scientific and technical outputs that NIST currently produces. 

Finally, the results indicate that there are different drivers for scientific and technological impact. 

Citations within peer-reviewed literature are a common indicator of scientific impact, and patent 

citations are a commonly used indicator of technological impact. The results above demonstrate that 

there are NIST publications that are highly cited by either patents or publications but not both. In 

Figure 3 above, there is a concentration of publications along each axis. Given this distinction 

between scientific and technological impact, measuring NIST impact exclusively though indicators of 

scientific impact may not ensure that NIST research has the technological impact needed to meet its 

mission to promote U.S. innovation and industrial competitiveness.

4
 

 

 

 

 

 

 

 

 

 

References 

[1] Anderson G (2016) NIST Impact on Patenting. (NIST, Gaithersburg, MD) NIST Economic Analysis 

Briefs 6. 

[2] Anderson, Gary and Anthony Breitzman (2016). “Identifying NIST Impacts on Patenting: A Novel 

Dataset and Potential Uses,” Journal of Research of NIST (under review) 

[3] Breitzman A, Thomas P (2016) Analysis of References from US Patents to NIST-Supported 

Technical Outputs. (NIST, Gaithersburg, MD) NIST GCR 16-009. 

[4] Jaffe A, Lerner J (2004) Innovation And Its Discontents: How Our Broken Patent System Is 

Endangering Innovation And Progress, And What To Do About It (Princeton University Press, 

Princeton, NJ, USA). 

[5] Jaffe A, Lerner J (2001). Reinventing public R&D: Patent policy and the commercialization of 

national laboratory technologies The RAND Journal of Economics 32(1) 167-198. 

[6] Jaffe A, Trajtenberg M (2002) Patents, Citations and Innovations: A Window on the Knowledge 

Economy, (M.I.T. Press, Boston, MA, USA). 

[7] Sheft J (2008) Technology transfer and idea commercialization Nature Biotechnology 26 711–712. 

1 See Breitzman and Thomas [3].
 2 See Anderson [1]. 3 See Anderson and Breitzman [2] for a more thorough discussion of these issues. See Jaffe and Lerner [4]
 

and Jaffe and Trajtenberg [6] for an overview and critique.
 4 See Breitzman and Thomas [3] and Anderson [1].
 5 For more details regarding the construction, usage and limitations of the data discussed in this analysis
 

brief please see Anderson and Breitzman [2].
 6 The US Patent and Trademark Office keeps data on both patent inventors and patent ownership,
 

assignment. Typically, NIST retains or shares ownership of intellectually property but at times through 

prior or subsequent agreement ownership of NIST invented IP is assigned to a third party. For simplicity,
 

we refer to all NIST-Invented IP as NIST-Assigned IP.
 7 The total output of grey literature and citation rates are not applicable metrics. Because grey literature 

includes personal communication, workshop presentations and other informal scientific communication it
 

is impossible to assemble a complete list or even count of such outputs.
 8 Add note about lack of pre-1997 data.
 9 Two outlier publications are not presented in this chart. The patent and publication citations (# patent
 

citations, # paper citations) for these two papers were (277, 3715) and (85,13151) respectively. The first
 

was the paper with the greatest number of patent citations and the latter had the greatest number of
 

paper citations.
 10 For example, see Sheft [7].
 

5