INTELLECTUAL 

PROPERTY

Patent Office Should 

Strengthen Search 

Capabilities and 

Better Monitor 

Examiners' Work

Report to the Chairman, Committee on 

the Judiciary, House of Representatives

June 2016

GAO-16-479

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-479, a report to the 

Chairman, Committee on the Judiciary, House 

of Representatives

June 2016

INTELLECTUAL PROPERTY

Patent Office Should Strengthen Search Capabilities

and Better Monitor Examiners’ Work

Why GAO Did This Study

USPTO examines patent applications 

to ensure that inventions are, among 

other requirements, novel and not 

obvious. USPTO patent examiners 

accomplish this by comparing 

applications to “prior art”—existing 

patents and applications in the United 

States and abroad, and nonpatent 

literature, such as scientific articles.

Thorough prior art searches help 

ensure the validity of granted patents. 

GAO was asked to identify ways to 

improve patent quality through use of 

the best available prior art. This report

(1) describes the challenges examiners 

face in identifying relevant prior art, (2) 

describes how selected foreign patent

offices have addressed challenges in 

identifying relevant prior art, and (3)

assesses the extent to which USPTO 

has taken steps to address challenges 

in identifying relevant prior art. GAO 

surveyed a generalizable stratified 

random sample of USPTO examiners

with an 80 percent response rate; 

interviewed experts active in the field,

including patent holders, attorneys, 

and academics; interviewed officials 

from USPTO and similarly sized 

foreign patent offices, and other 

knowledgeable stakeholders; and 

reviewed USPTO documents and 

relevant laws.

What GAO Recommends

GAO is making seven 

recommendations, among them, that 

USPTO develop a strategy to identify 

key sources of nonpatent literature,

establish goals and indicators for prior 

art search quality, and collect sufficient 

information to assess prior art search 

quality. USPTO concurred with GAO’s

recommendations.

What GAO Found

Experts and U.S. Patent and Trademark Office (USPTO) examiners described a 

variety of challenges in identifying information relevant to a claimed invention—or 

“prior art”—that can affect examiners’ ability to complete a thorough prior art 

search in the time allotted and their confidence in their search efforts. These 

challenges include, among others, the quantity and availability of prior art, the 

clarity of patent applications, and USPTO’s policies and search tools. 

Selected Challenges Experts and Examiners Cited in Identifying Relevant Prior Art

 

The European Patent Office and Japan Patent Office face similar challenges to 

USPTO in identifying prior art and use various approaches to help address them, 

such as leveraging work of other patent offices on related patent applications and 

integrating nonpatent literature into their search tools. In some cases, these

approaches are coordinated with, similar to, or could inform USPTO actions.

USPTO has taken actions to address challenges in identifying prior art, but some

actions have limitations. For example, USPTO is in the process of upgrading its

search tools. However, examiners will still need to access a variety of external 

sources to meet USPTO’s requirement to consider nonpatent literature. Federal 

internal control standards call for controls to evolve to remain effective and 

USPTO officials noted that the new search system can be expanded to include 

more nonpatent literature as the European and Japan patent offices have done.

However, USPTO does not have a documented strategy for identifying additional 

sources. Without such a strategy, USPTO cannot be assured that its information 

technology investment will improve examiners’ searches. USPTO is also taking 

steps to augment the number of, and consistency with which, reviews of 

examiners’ work are conducted and documented, which could improve USPTO’s 

monitoring of examiners’ work. However, USPTO still faces limitations in 

assessing the thoroughness of examiners’ prior art searches, because, for 

example, the agency has not established goals or indicators for search quality 

and may not be collecting sufficient information on examiners’ searches to 

assess prior art search quality. Without monitoring examiners’ prior art searches,

the agency cannot be assured that examiners are searching all relevant sources 

of prior art and may not be able to develop appropriate responses as called for 

by federal internal control standards. View GAO-16-479. To view an e-supplement 

with data from a survey of patent examiners, 

see GAO-16-478SP. For more information, 

contact John Neumann at (202) 512-3841 or 

neumannj@gao.gov.
Page i GAO-16-479 Intellectual Property

Letter 1

Background 6

Patent Examiners Face a Variety of Challenges in Identifying 

Relevant Prior Art 13

Patent Offices for Europe and Japan Use Various Approaches to 

Help Address Challenges in Identifying Relevant Prior Art 30

USPTO Is Taking Actions to Address Challenges in Identifying 

Prior Art, but Some Actions Have Limitations 34

Conclusions 56

Recommendations for Executive Action 59

Agency Comments and Our Evaluation 60

Appendix I Objectives, Scope, and Methodology 63

Appendix II Patent Examination Process at the U.S. Patent and Trademark 

Office (USPTO) (Corresponds to Fig. 1) 71

Appendix III Selected Survey Results by Technology Center or Examiners’ Level 73

Appendix IV Comments from the U.S. Department of Commerce 80

Appendix V GAO Contact and Staff Acknowledgments 84

Related GAO Products 85

Tables

Table 1: Comparison of Workload and Workforce of Selected 

Patent Offices 12

Table 2: Estimated Difficulty and Frequency of Searching for 

Various Types of Prior Art 18

Table 3: Estimated Occurrence of Clarity Issues in Patent 

Applications and Their Effects on Prior Art Searches 20

Contents
Page ii GAO-16-479 Intellectual Property

Table 4: Estimated Effects of the Cooperative Patent 

Classification System on Completing a Thorough Prior Art 

Search in the Time Allotted 38

Table 5: Voluntary Technical Training Programs for U.S. Patent 

and Trademark Office (USPTO) Examiners and Their 

Fiscal Year 2015 Utilization 54

Table 6: Strata of United States Patent and Trademark Office 

(USPTO) Examiners Surveyed 65

Table 7: Descriptions of Selected Steps in the Patent Examination 

Process at the U.S. Patent and Trademark Office 

(USPTO) 71

Table 8: Estimated Percentage of Examiners Who Find the 

Quantity of Art Makes Prior Art Searching Easier or More 

Difficult 73

Table 9: Estimated Percentage of Examiners Who Always Search 

for Certain Types of Prior Art in a Typical Patent 

Examination 74

Table 10: Estimated Percentage of Examiners Who Find It Very 

Difficult to Obtain Relevant Prior Art from Different Types 

of Sources 75

Table 11: Estimated Sufficiency of Patent Examiners’ Time for 

Completing a Thorough Prior Art Search 76

Table 12: Estimated Time Examiners Spent per Application on 

Prior Art Search and Each First Action on the Merits in the 

Past Quarter 77

Table 13: Estimated Percentage of Examiners Who Worked 

Biweekly Voluntary/Uncompensated Overtime during the 

Last 6 Months, by Examiner General Schedule Level 78

Table 14: Estimated Percentage of Examiners Who Encountered 

Misclassified Patent Applications in the Past Quarter 78

Table 15: Examiners’ Views on the Usefulness of Continuing 

Education in Their Art Area, by Technology Center 79

Figures

Figure 1: Selected Steps in the Patent Examination Process at the 

U.S. Patent and Trademark Office (USPTO) 7

Figure 2: Number of Patent Examiners in Eight U.S. Patent and 

Trademark Office Technology Centers GAO Reviewed, 

as of May 2015 10
Page iii GAO-16-479 Intellectual Property

Figure 3: General Schedule (GS) Levels of U.S. Patent and 

Trademark Office Patent Examiners in Technology 

Centers GAO Reviewed, as of May 2015 11

Figure 4: Challenges in Identifying Relevant Prior Art 13

Figure 5: Estimated Sufficiency of Patent Examiners’ Time for 

Completing Thorough Prior Art Searches 23

Figure 6: Estimated Biweekly Voluntary/Uncompensated Overtime 

Worked by Patent Examiners in the Past 6 Months 24

Figure 7: Patent Examiners’ Confidence That They Find the Most 

Relevant Prior Art during Examination 51

Abbreviations

CMH Cochran-Mantel-Haenszel

CPC Cooperative Patent Classification

EAST Examiner’s Automated Search Tool

EPO European Patent Office

GS General Schedule

JPO Japan Patent Office

OPQA Office of Patent Quality Assurance

STIC Scientific and Technical Information Center

TC technology center

USPC U.S. Patent Classification

USPTO U.S. Patent and Trademark Office

WEST Web-Based Examiner Search Tool

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-479 Intellectual Property

441 G St. N.W.

Washington, DC 20548

June 30, 2016

The Honorable Bob Goodlatte

Chairman

Committee on the Judiciary

House of Representatives

Dear Mr. Chairman:

Scientific and technological innovations provide a foundation for American 

economic growth and competitiveness and help to support a high 

standard of living. The Constitution grants Congress the power to give 

inventors a patent providing exclusive rights for a limited time to their 

inventions to “promote the Progress of Science and useful Arts,” thereby 

encouraging innovation.1 In 1836, Congress established the U.S. Patent 

Office, later renamed the U.S. Patent and Trademark Office (USPTO), to 

administer the patent system in accordance with patent laws.2 A patent 

from USPTO gives its holder the ability to generally exclude others from 

making, using, selling, or importing a patented invention in the United 

States for up to 20 years from the date on which a successful patent 

application was filed. Patent holders may enforce these rights through 

patent infringement lawsuits.3 Third parties may challenge the validity of 

an issued patent through USPTO proceedings or in federal courts. A 

patent from USPTO does not offer protections abroad, but applicants may 

seek patents for the same invention from foreign patent offices.

USPTO receives over half a million applications each year from inventors 

seeking patents to protect the intellectual property stemming from their 

work. When examining a patent application, a USPTO patent examiner 

determines whether the claimed invention meets the legal requirements 

for patentability as set forth in patent laws and federal case law. For 

 1

The Constitution grants to Congress the power “[t]o promote the Progress of Science and 

useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to 

their respective Writings and Discoveries.” U.S. Const., Art. I, § 8, cl. 8.

2

Patent Act of 1836, ch. 357, 5 Stat. 117.

3

Anyone who makes, sells, offers to sell, uses, or imports the patented invention during 

the term of the patent without the patent owner’s permission infringes the patent. 

Letter
Page 2 GAO-16-479 Intellectual Property

example, examiners must determine whether the invention is novel and 

not obvious, as those concepts have been defined and interpreted by the 

agency and the courts.4 USPTO examiners make such determinations by 

comparing information in the application to other information relevant to 

the claimed invention. Such information, generally known as prior art, 

may include prior patents, patent applications, or nonpatent publications 

describing a technology, among other things. Finding the most relevant 

prior art when examining a patent application reduces the chance that 

USPTO will grant a patent for something previously invented or for an 

obvious combination of prior inventions.

Recent trends in disputes over patent validity in USPTO proceedings and 

in federal courts have raised concerns about the overall validity of patents 

issued by USPTO. Some observers, citing the extent to which the validity 

of granted patents has been successfully challenged, believe that USPTO 

examiners do not always identify the most relevant prior art, which has 

resulted in granting some patents that may not meet the statutory 

requirements for patentable inventions. Specifically, under the 2011 

Leahy-Smith America Invents Act, a party other than the patentee may 

bring a proceeding before the Patent Trial and Appeal Board to challenge 

the validity of a patent.5 From September 2012 through February 2016, 

719 patents have been partially or wholly invalidated based on prior art 

submitted through one type of these proceedings.6 As a result of these 

concerns and others about whether the U.S. patent system continues to 

support innovation and competitiveness, Congress has continued to 

consider patent reform legislation in recent years.

 4

An invention is considered to be novel if it has not been previously patented; described in 

a printed publication; or in public use, on sale, or otherwise available to the public. 35 

U.S.C. § 102. An invention is considered to be obvious, and therefore not patentable, if 

the differences between the claimed invention and the prior art are such that the claimed 

invention as a whole would have been obvious to a person having ordinary skill in the art 

to which the claimed invention pertains. 35 U.S.C. § 103.

5

The Leahy-Smith America Invents Act created the Patent Trial and Appeal Board within 

USPTO and established three new proceedings for challenging issued patents. Pub. L. 

No.112-29, §§ 6, 7, and 18 (2011). 

6

This proceeding, the Inter Partes review, is limited to challenges alleging that the patent 

does not meet the legal standards for novelty or nonobviousness. 35 U.S.C. § 311. In this 

period, the Patent Trial and Appeal Board received 4,181 petitions for Inter Partes review, 

denied 832 petitions, accepted 1,359 petitions, and completed 828 trials.
Page 3 GAO-16-479 Intellectual Property

You asked us to identify ways to improve patent quality through the use of 

the best available prior art during USPTO’s examination of patent 

applications. This report (1) describes the challenges examiners face in 

identifying relevant prior art, (2) describes how selected foreign patent 

offices have addressed challenges in identifying relevant prior art, and (3) 

assesses the extent to which USPTO has taken steps to address any 

challenges in identifying relevant prior art. Our work for this report was 

coordinated with work for our report on patent quality being issued today.7

To describe the challenges examiners face in identifying relevant prior art, 

we reviewed relevant laws and USPTO documents, and interviewed 

USPTO officials, supervisory examiners, officials of the examiners’ union 

(the Patent Office Professional Association), and a group of five 

examiners serving as union representatives. We also conducted 

semistructured interviews with 18 subject matter experts (experts) active 

in the intellectual property field and knowledgeable about the subject of 

prior art. To do so, we first identified the following five stakeholder groups 

to obtain different views: intellectual property academics, patent holders, 

patent attorneys, nongovernmental organizations, and patent data 

experts. We then selected experts based on criteria we developed for 

each group. When describing the views of these experts, “most”

represents more than 10 of the 18 experts we interviewed.

In addition, we conducted a web-based survey of a stratified random 

sample of 3,336 USPTO patent examiners from across 8 of the 11 

technology-based subject matter groups (referred to as technology 

centers) into which USPTO examiners are divided.8 For one technology 

center, we separated certain art units—subunits of a technology center—

 7

GAO, Intellectual Property: Patent Office Should Define Quality, Reassess Incentives,

and Improve Clarity, GAO-16-490 (Washington, D.C.: June 30, 2016).

8

We surveyed examiners in the following eight technology centers: (1) Biotechnology and 

Organic Chemistry; (2) Chemical and Materials Engineering; (3) Computer Architecture, 

Software, and Information Security; (4) Computer Networks, Multiplex Communication, 

Video Distribution, and Security; (5) Communications; (6) Semiconductors, Electrical and 

Optical Systems and Components; (7) Transportation, Construction, Electronic 

Commerce, Agriculture, National Security and License and Review; and (8) Mechanical 

Engineering, Manufacturing, and Products. We excluded examiners in the three 

technology centers for design patents, reexamination of patent applications, and training 

of new examiners.
Page 4 GAO-16-479 Intellectual Property

into a separate category, for a total of nine strata.9 We excluded 

examiners employed at USPTO for less than 1 year, as of May 2015. The 

survey collected information on examiners’ experience with prior art 

searches and their opinions on challenges and options for improving 

USPTO’s prior art search procedures and capabilities. The survey was 

designed and administered in conjunction with our patent quality report 

(GAO-16-490); therefore, questions also addressed USPTO’s approach 

to patent quality and how the agency might improve patent quality. 

Overall, we received responses from 80 percent of the examiners in our 

survey sample, which we designed to produce estimates that are 

generalizable to the population of patent examiners in our study overall, 

as well as within each technology center.10 The survey questions and 

detailed data on the results can be viewed on our website.11

To describe how selected foreign patent offices have addressed 

challenges in identifying relevant prior art, we interviewed officials at the 

European Patent Office (EPO) and the Japan Patent Office (JPO), which 

we selected based on comments observers made about the quality of 

their work and because they are similar in size to USPTO. To assess the 

extent to which USPTO has taken steps to address any challenges in 

identifying relevant prior art, we interviewed USPTO officials and 

reviewed agency documents related to prior art search procedures and 

capabilities, including ongoing and planned efforts in the areas of 

information technology resources, human capital management, and 

 9

The Transportation, Construction, Electronic Commerce, Agriculture, National Security 

and License and Review technology center includes a diverse set of technologies, 

including transportation, construction, agriculture, and business methods. In our review, 

we separated the art units focused on electronic commerce and business methods 

(collectively referred to as business methods) in light of recent legislation and court 

decisions related to business methods.

10Because we followed a probability procedure based on random selections, our sample is 

only one of a large number of samples that we might have drawn. Since each sample 

could have provided different estimates, we express our confidence in the precision of our 

particular sample’s results as a 95 percent confidence interval. This is the interval that 

would contain the actual population value for 95 percent of the samples we could have 

drawn. All percentage estimates from our survey have margins of error at the 95 percent 

confidence level of plus or minus 6 or fewer percentage points.

11GAO, Intellectual Property: Survey of U.S. Patent Examiners, an E-supplement to 

GAO-16-479 and GAO-16-490, GAO-16-478SP (Washington, D.C.: June 30, 2016). This 

e-supplement pertains to both this report and our patent quality report.
Page 5 GAO-16-479 Intellectual Property

international cooperation. In addition, we reviewed relevant results from 

our survey of examiners. In assessing USPTO’s efforts, we identified 

criteria in the federal standards for internal control;12 the Government 

Performance and Results Act of 1993, as amended;13 USPTO’s manual

for patent examiners;

14 and USPTO’s strategic plan. Appendix I provides 

more detail on our scope and methodology.

We conducted this performance audit from November 2014 to June 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

 12GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999). Revised standards became effective beginning with 

fiscal year 2016 after our work began. See Standards for Internal Control in the Federal 

Government, GAO-14-704G (Washington, D.C.: September 2014).

13Pub. L. No. 103-62 (1993), amended by Government Performance and Results Act

Modernization Act of 2010, Pub. L. No. 111-352 (2011). The Act sought to improve the 

effectiveness and accountability of federal programs by requiring federal agencies to set 

goals for program performance, measure results, and report on annual performance 

compared with the goals. Although the Act’s requirements apply at the agency level, we 

have previously reported that these practices can serve as leading practices within an 

organization, such as with individual programs or initiatives. See GAO, Environmental 

Justice: EPA Needs to Take Additional Actions to Help Ensure Effective Implementation, 

GAO-12-77 (Washington, D.C.: Oct. 6, 2011), and Great Lakes Restoration Initiative: 

Further Actions Would Result in More Useful Assessments and Help Address Factors 

That Limit Progress, GAO-13-797 (Washington, D.C.: Sept. 27, 2013).

14U.S. Patent and Trademark Office, Manual of Patent Examining Procedure (November 

2015). The manual is available electronically at www.uspto.gov/web/offices/pac/mpep/. 
Page 6 GAO-16-479 Intellectual Property

The focus of patent examination is to determine whether the invention in 

a patent application satisfies the legal requirements for a patent, including 

that the invention be novel and not obvious. As shown in figure 1, 

USPTO’s patent examination process involves a variety of steps, at least 

one of which includes prior art searches that examiners use to determine 

whether an invention is novel and not obvious.

Background

Patent Examination 

Process and Prior Art 

Searches
Page 7 GAO-16-479 Intellectual Property

Figure 1: Selected Steps in the Patent Examination Process at the U.S. Patent and Trademark Office 

(USPTO) Interactive Graphic

Instructions: Hover over each step in the patent examination process below to see more information. 

To print a version containing text, see appendix II, page 71.

Note: See app. II for the text underlying this figure.
Page 8 GAO-16-479 Intellectual Property

USPTO’s manual for patent examiners establishes certain requirements 

and processes that examiners must follow in examining patent 

applications, including performing prior art searches. There are generally 

two types of prior art—patent literature or nonpatent literature. Patent 

literature consists of previously issued U.S. or foreign patents and 

published patent applications.15 Nonpatent literature consists of other 

publicly available documents and can include such things as product 

manuals, standards established by international organizations, textbooks, 

periodicals, or conference presentations. Both patent and nonpatent 

literature may be written in a language other than English, referred to as 

foreign-language art in this report. USPTO’s manual for patent examiners 

requires them to conduct a thorough prior art search and directs them to 

consider U.S. patents, foreign patents, and nonpatent literature unless 

they can justify with reasonable certainty that no more pertinent prior art 

references can be found.

Examiners are expected to complete their examination of an application 

in a certain number of hours. The time allotted varies depending on 

factors such as the technology and the seniority of the examiner. For 

example, an examiner reviewing an application related to wire fabrics and 

structure may be allotted about 14 hours for examination, while an 

examiner at the same experience level would be allotted about 32 hours 

for an application related to data processing, such as database and file 

management. The allotted time includes the time needed to review the 

application, perform a search for prior art, and complete all office actions. 

Examiners have minimum production goals, based on the time allotted, 

for the number of office actions they must complete, and examiners may 

earn bonuses for exceeding these minimum production goals.

USPTO uses several different information technology systems to assist 

examiners in conducting prior art searches. For example, to search U.S. 

patent literature, examiners use two systems, the Examiner’s Automated 

Search Tool (EAST) and the Web-Based Examiner Search Tool (WEST), 

to search the full texts of published patent applications since 2001, 

patents granted since 1970, and optically scanned U.S. patents granted 

 15USPTO is required to keep patent applications confidential for a period of 18 months 

from the earliest filing date for which benefit is sought under title 35 of the United States 

Code. At that time, USPTO is generally required to publish the application. USPTO may 

publish the application earlier at the request of the applicant. 35 U.S.C. § 122(a) and (b).
Page 9 GAO-16-479 Intellectual Property

from 1920 through 1970. These systems also include abstracts of some 

foreign patents. Examiners may access additional foreign patent 

documents through other web-based tools. In addition, USPTO’s 

Scientific and Technical Information Center (STIC) operates systems that 

can search U.S. patent literature and that store some nonpatent literature. 

STIC also provides subscriptions to various web-based sources of 

nonpatent literature. According to a USPTO document, the agency had 

subscriptions to 119 different journals or external databases in 2014. 

USPTO’s current search tools do not provide examiners with immediate 

access to computer-generated translations, known as machine 

translations; however, examiners can request human and machine 

translation services from STIC and three contracted translation vendors 

that currently cover 35 languages.16

As of May 2015, USPTO had nearly 8,300 patent examiners across the 

eight technology centers we reviewed (see fig. 2). USPTO uses the 

General Schedule (GS) classification system for patent examiners, whose 

levels range from GS-5 to GS-15.17 Examiners at the GS-14 level or 

above (44 percent of the examiners in the technology centers we 

reviewed, as shown in fig. 3) are generally primary examiners. Primary 

examiners may accept or reject a patent application without additional 

review. This level of authority is in contrast to junior examiners—most 

examiners below the GS-14 level—whose work must first be reviewed by 

a primary examiner before an office action can be sent to the applicant. At 

the GS-13 level, some examiners are in the process of becoming primary 

examiners. Supervisory patent examiners are at the GS-15 level and are 

responsible for the day-to-day management of examiners.

 16According to a USPTO official, STIC processed more than 3,000 human translations for 

patent examiners, and the commercial vendors provided more than 3,000 additional 

human translations in fiscal year 2015. The most translated languages for these requests 

were Japanese, German, Chinese, Korean, French, and Russian.

17The GS system is the federal government’s classification system for defining and 

organizing federal positions, primarily to assign rates of pay, based on a position’s duties, 

responsibilities, and qualification requirements, among other things. The GS system 

includes 15 statutorily defined grade levels, numerical designations based on the 

complexity of the work and knowledge required to do the job, from GS-1 to GS-15. 

USPTO’s Patent Examiner 

Workforce
Page 10 GAO-16-479 Intellectual Property

Figure 2: Number of Patent Examiners in Eight U.S. Patent and Trademark Office Technology Centers GAO Reviewed, as of 

May 2015
Page 11 GAO-16-479 Intellectual Property

Figure 3: General Schedule (GS) Levels of U.S. Patent and Trademark Office Patent 

Examiners in Technology Centers GAO Reviewed, as of May 2015

Note: Some examiners at the GS-13 level are in the process of becoming primary examiners; these 

examiners have the authority to communicate certain decisions to patent applicants, but they cannot 

issue final decisions on patent applications without review by a primary examiner. Examiners at the 

GS-14 level are generally primary examiners.

In addition to applying for a patent from USPTO, an inventor may also 

seek patent protection in other countries for the same invention by filing in 

multiple patent offices. Such interrelated patent applications are 

described as a patent family. The World Intellectual Property Organization 

has estimated that approximately half of all applications worldwide are 

Foreign Patent Offices
Page 12 GAO-16-479 Intellectual Property

repetitive filings in a patent family, and the rest are initial filings.18

According to the World Intellectual Property Organization, around 2.7 

million patent applications were filed worldwide in 2014, of which 2.2 

million applications were filed with patent offices in the United States, 

China, Japan, South Korea, and Europe—known as the IP5. These five 

offices, including EPO and JPO, each receive hundreds of thousands of 

patent applications each year. EPO issues patents that cover 42 

countries, most of which are member countries that are party to the 

European Patent Convention. Applicants may apply to national patent 

offices or apply through EPO for coverage in some or all of these 42 

countries. EPO has three official languages, one of which must be used 

for processing an application. These three languages are English (about 

80 percent of applications), French (about 5 percent), and German (about 

15 percent), according to EPO officials. EPO and JPO have about onehalf and one-fifth as many examiners as USPTO, respectively. According 

to JPO officials, there is a limit on the number of federal employees JPO 

can have; therefore, 494 of its 1,702 examiners are fixed-term rather than 

permanent employees, and JPO has begun outsourcing aspects of prior 

art search. JPO officials stated that outsourcing some aspects of prior art 

searches based on instructions from a government examiner frees up the 

examiners’ time and allows the office to review more applications. Table 1 

describes the workload and workforce of USPTO, EPO, and JPO.

Table 1: Comparison of Workload and Workforce of Selected Patent Offices

Patent office

Number of applications 

received, 2014

Number of patents 

granted, 2014

Number of patent 

examiners

U.S. Patent and 

Trademark Office 

(USPTO) 578,802 300,678 8,279a

European Patent 

Office (EPO) 151,981 64,613 4,126b

Japan Patent 

Office (JPO) 325,989 227,142 1,702c

Sources: IP5 Statistical Report 2014 Edition, GAO analysis of May 2015 USPTO examiner data, EPO Social Report for the year 2014, 

and patent office officials. | GAO-16-479

 18The World Intellectual Property Organization is a self-funding agency of the United 

Nations, with 188 member states. The organization’s mission is to lead the development 

of a balanced and effective international intellectual property system that enables 

innovation and creativity for the benefit of all.
Page 13 GAO-16-479 Intellectual Property

a

The number of USPTO patent examiners does not include 491 examiners in the technology centers 

that GAO did not review, such as examiners in the Designs technology center.

b

The number of EPO examiners does not include contract staff examiners.

c

The number of JPO examiners does not include nongovernment employees performing outsourced 

aspects of prior art searches.

USPTO examiners face a variety of challenges in identifying relevant prior 

art during patent examination. As shown in figure 4, the experts we 

interviewed and examiners we surveyed cited challenges related to 

certain attributes of prior art and patent applications and USPTO 

examination policies, search tools, and human capital management. Our 

survey results show the extent of the challenges may vary by technology 

center or examiners’ GS level (see app. III). Because we surveyed a 

generalizable stratified random sample of examiners and adjusted for 

nonresponse, our results provide estimates for the entire population of 

examiners in our study and, when reported by technology center, for each 

of the technology centers we reviewed.19

Figure 4: Challenges in Identifying Relevant Prior Art

 19Survey results may not always total 100 percent because of rounding. 

Patent Examiners 

Face a Variety of 

Challenges in 

Identifying Relevant 

Prior Art
Page 14 GAO-16-479 Intellectual Property

Several attributes of prior art and patent applications present challenges 

for examiners in identifying relevant prior art, including the quantity of 

prior art, amount and relevance of prior art cited by applicants, availability 

of prior art, and clarity of patent applications.

Quantity of prior art. The large volume of prior art available from 

multiple sources makes searching for relevant prior art challenging, 

according to most experts we interviewed as well as examiners 

responding to our survey. For example, one expert noted that the amount 

of patents, publications, and other nonpatent literature has grown 

exponentially, making it harder to find relevant prior art in the time 

allotted. Another expert said that technological innovations are occurring 

at a tremendous rate, and that the growing volume of prior art 

domestically and worldwide can be overwhelming. Based on our survey, 

we estimate that 45 percent of all examiners in the eight technology 

centers we reviewed find that the large quantity of art makes it somewhat 

or much more difficult to complete a thorough prior art search in the time 

allotted, while fewer examiners—34 percent—find that the quantity makes 

it somewhat or much easier.20 However, responses varied among 

technology centers, as shown in appendix III. For example, 30 percent of 

examiners in the Computer Networks, Multiplex Communication, Video 

Distribution, and Security technology center find that the quantity of art 

makes it somewhat or much more difficult to complete a thorough prior art 

search in the time allotted, compared to 60 percent of examiners in the 

Mechanical Engineering, Manufacturing, and Products technology center 

and 60 percent of examiners in the Chemical and Materials Engineering 

technology center.

Amount and relevance of prior art cited by applicants. Examiners we 

surveyed reported difficulties with the amount and relevance of prior art 

references provided by applicants.21 USPTO requires applicants and 

 20In our survey, we asked examiners how much easier or more difficult the quantity of 

prior art available (i.e., the sheer number of possibly relevant references across all 

sources) makes it to complete a thorough prior art search in the time allotted. Based on 

our survey, we estimate all examiners’ responses to be as follows: much easier, 10 

percent; somewhat easier, 24 percent; neither easier nor more difficult, 20 percent; 

somewhat more difficult, 25 percent; much more difficult, 19 percent; don’t know, 1 

percent; and no response, 1 percent. Throughout our report, estimates may not add to 

100 percent or sum to the amount reported above because of rounding.

21We did not ask experts about this issue. 

Challenges Related to 

Attributes of Prior Art and 

Patent Applications
Page 15 GAO-16-479 Intellectual Property

others assisting in filing an application to submit all information known to 

be material to patentability. This information may include search results 

from foreign patent offices or publications known to the individual.22

According to USPTO policy, examiners will consider this information 

when reviewing a patent application, which may require reviewing 

numerous prior art references submitted by the applicant. Based on our 

survey, we estimate that 82 percent of examiners sometimes, often, or 

always encountered applications with what they considered an excessive 

number of submitted art references in the past quarter.23 We estimate that 

for most examiners (64 percent), excessive references make it somewhat 

or much more difficult to complete a thorough prior art search in the time 

allotted.24

Considering all of the prior art references submitted by applicants can be 

particularly challenging for examiners because applicants are generally 

not required to explain the relevance of the references or to point 

examiners to the particular portions of references that are relevant. In 

commenting on this issue in our survey, one examiner recalled often 

receiving information disclosure statements from applicants with 

numerous prior art references, of which only a handful were relevant. 

Based on our survey, we estimate that 88 percent of examiners 

sometimes, often, or always encountered applications with irrelevant 

 22USPTO regulations provide that each individual associated with the filing and 

prosecution of a patent application has a duty of candor and good faith in dealing with 

USPTO, which includes a duty to disclose all known information material to patentability. 

37 C.F.R. § 1.56(a).

23We estimate that examiners encountered such applications in the past quarter as 

follows: always, 3 percent of examiners; often, 29 percent; sometimes, 50 percent; rarely, 

17 percent; never, 1 percent; don’t know, less than 1 percent; and no response, less than 

1 percent. 

24We asked examiners how much easier or more difficult certain factors of an application 

make it to complete a thorough prior art search in the time allotted. For an excessive 

number of art references provided by the applicant, we estimate all examiners’ responses 

to be as follows: much easier, less than 1 percent; somewhat easier, 4 percent; neither 

easier nor more difficult, 32 percent; somewhat more difficult, 40 percent; much more 

difficult, 24 percent; don’t know, 1 percent; and no response, 1 percent. 
Page 16 GAO-16-479 Intellectual Property

references in the past quarter.25 Moreover, most examiners (57 percent) 

find that irrelevant references make it somewhat or much more difficult to 

complete a thorough prior art search in the time allotted.26 In contrast, 87 

percent of examiners find that an application with relevant references 

makes it somewhat or much easier to complete thorough prior art 

searches in the time allotted.27

Availability of prior art. The availability of prior art and difficulties 

obtaining certain types of prior art are also challenges, according to most 

experts we interviewed as well as examiners we surveyed. For example, 

some relevant prior art may require a fee to access, may not be in a textsearchable format, may not be in a database, or may otherwise be 

difficult to access. In particular, experts told us that certain general types 

of prior art are more difficult to find, such as nonpatent literature overall 

and software-related prior art (most experts) and foreign-language prior 

art (6 of the 18 experts). Specific types of nonpatent literature that experts 

identified as difficult to find include product documentation and 

summaries, product databases, and user manuals; offers for sale and 

public use; and information from standards-setting organizations. Prior art 

related to computer technologies has also been difficult to find, according 

to one expert we interviewed, in part because a lot of information is 

commonly known in this field but may not be found by searching a public 

database. In addition, one expert we interviewed suggested that 

 25We estimate that examiners encountered such applications in the past quarter as 

follows: always, 7 percent of examiners; often, 42 percent; sometimes, 40 percent; rarely, 

11 percent; never, 1 percent; don’t know, less than 1 percent; and no response, less than 

1 percent. Survey results reported here do not sum to the result provided above because 

of rounding.

26We asked examiners how much easier or more difficult certain factors of an application 

make it to complete a thorough prior art search in the time allotted. For irrelevant art 

references provided by the applicant, we estimate all examiners’ responses to be as 

follows: much easier, less than 1 percent; somewhat easier, 1 percent; neither easier nor 

more difficult, 41 percent; somewhat more difficult, 38 percent; much more difficult, 18 

percent; don’t know, less than 1 percent; and no response, 1 percent. Survey results 

reported here do not sum to the result provided above because of rounding.

27We asked examiners how much easier or more difficult certain factors of an application 

make it to complete a thorough prior art search in the time allotted. For information 

disclosure statements with relevant references, we estimate all examiners’ responses to 

be as follows: much easier, 32 percent; somewhat easier, 55 percent; neither easier nor 

more difficult, 11 percent; somewhat more difficult, 1 percent; much more difficult, less 

than 1 percent; don’t know, less than 1 percent; and no response, less than 1 percent.
Page 17 GAO-16-479 Intellectual Property

examiners do not have ready access to textbooks that are a good source 

of prior art.28 Another suggested that prior art from before the mid-1970s 

is difficult to find because patents issued before then have not been fully 

digitized. This expert stated that this is a particularly challenging issue for 

examiners in the mechanical technology centers because those 

examiners tend to use older art more often than other examiners.

USPTO examiners we surveyed also reported difficulties obtaining 

relevant prior art from searches for certain types of prior art more than 

other types. In particular, on the basis of our survey, we estimate that 51 

percent of all examiners find it somewhat or very difficult to obtain 

relevant art from searches for foreign-language nonpatent literature.29

Difficulties obtaining certain types of prior art may influence how often 

patent examiners search for them. For example, 8 of the 18 experts we 

interviewed suggested that examiners focus on searching patent literature 

and may not thoroughly search nonpatent literature. Similarly, our survey 

results in table 2 show that nearly all examiners always or often search 

for U.S. patents and applications (an estimated 99 percent); we also 

found that nearly all examiners always or often view this as the most 

relevant type of art they consider (an estimated 98 percent of 

examiners).30 In contrast, we estimate that 67 percent of examiners 

always or often search for foreign patents, and 20 percent of examiners 

always or often search for foreign-language nonpatent literature. More 

examiners also find that foreign patents and foreign-language nonpatent 

literature are difficult to obtain, compared to those that find U.S. patent 

literature difficult to obtain. In analyzing our survey results, we found that 

the difficulty examiners ascribed to finding foreign patent literature and 

foreign-language nonpatent literature was statistically associated with 

 28In its technical comments on a draft of this report, USPTO noted that some textbooks 

are available through STIC. We did not evaluate the extent of this collection.

29According to USPTO officials, the foreign languages most relevant for examiners’ prior 

art searches are Japanese, German, French, Korean, Russian, and Chinese.

30We asked examiners how often prior art from various types of references is the most 

relevant prior art they consider. For prior U.S. patents and applications, we estimate all 

examiners’ responses to be as follows: always, 63 percent; often, 35 percent, and 

sometimes, 1 percent. For each of the following, we estimate responses to be less than 1 

percent of all examiners: rarely, never, don’t use, don’t know, and no response.
Page 18 GAO-16-479 Intellectual Property

how often they reported searching for these types of prior art.31 How often

examiners search for certain types of prior art, and how difficult 

examiners find those searches, also varies by technology center (see 

app. III).

Table 2: Estimated Difficulty and Frequency of Searching for Various Types of Prior Art

Estimated percentage of examiners

Difficulty of obtaining 

relevant prior art Frequency of searching

Type of prior art

Very 

difficult

Somewhat 

difficult Always Often Sometimes Rarely

Never or 

don’t use

Nonpatent 

literature 

Foreign-language nonpatent 

literature 24 28 8 13 27 31 21

Textbooks 14 28 4 9 29 40 17

Industry-related nonpatent 

literature (e.g., manuals or 

company websites) 10 27 9 20 37 26 7

Scientific articles or 

presentations 9 26 19 22 28 24 8

Software-related nonpatent 

literature 7 19 13 17 23 19 27

Patent literature

Foreign patents 7 25 35 32 24 8 1

Prior U.S. patents and 

applications 1 4 95 4 1 <1 <1

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Note: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” Survey results reported here may not sum to the results provided in the text 

above because of rounding. All estimates have 95 percent confidence intervals of within +/- 5 or fewer 

percentage points. See the related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

Clarity of patent applications. According to most of the experts we 

interviewed and examiners we surveyed, the clarity of applications can 

pose a challenge to finding relevant prior art. For example, as one expert 

noted, there often are no standard terms to describe technologies, and 

 31We performed a test of association at the 5 percent level of significance on examiners’

responses to questions about how often they search for certain types of prior art and how 

difficult it is to obtain relevant art from these searches. Additional information on these 

tests may be found in app. I.
Page 19 GAO-16-479 Intellectual Property

different applications may use different terms to describe the same thing. 

According to four of the experts we interviewed, the absence of standard 

terms is particularly a challenge for software-related applications. 

Inconsistent terminology can make it more difficult for examiners to find 

relevant prior art because searching for one term using keyword 

searches—one common method of searching for prior art described by 

USPTO officials—will not identify documents that use a different term. As 

shown in table 3, examiners reported that issues with the clarity of the 

application make it more difficult to complete thorough prior art searches 

in the time allotted. In addition, based on comments examiners made in 

our survey, examiners may face difficulties associated with applications 

that have been translated from a foreign language.32 For example, one 

examiner stated that “translation quality is often poor, and claims routinely 

contain non-standard industry terms. Issued patents and publications 

containing these non-standard terms also make searching in foreign 

collections exceptionally challenging because it is not possible to 

anticipate which synonyms to use.” In our patent quality report 

(GAO-16-490), we provide additional information on how the clarity of 

applications affects patent quality.

 32Our survey did not ask examiners how often they encountered applications translated 

from foreign languages or whether these applications make it easier or more difficult to 

complete a thorough prior art search in the time allotted. However, examiners raised 

concerns about the difficulties associated with poor quality translations of applications in 

response to our open-ended survey questions asking examiners for the following: (1) 

other factors of an application not identified in our survey that make it easier or more 

difficult to complete a thorough prior art search in the time allotted and (2) their 

suggestions for improving any factors that they indicated as making prior art searches 

more difficult. Because we did not conduct a systematic review of all open-ended 

responses to our survey, we are not reporting the exact number of examiners who 

provided a response on this topic. Responses to open-ended questions are not 

generalizable to other examiners.
Page 20 GAO-16-479 Intellectual Property

Table 3: Estimated Occurrence of Clarity Issues in Patent Applications and Their Effects on Prior Art Searches 

Estimated percentage of examiners

Occurrence in new patent applications over 

the past quarter

Effect on the difficulty of completing 

prior art searches in the time allotteda

Issue with clarity of patent application Always Often Sometimes Rarely Never Much more 

difficult

Somewhat more 

difficult

Broadly worded claims 40 47 11 1 <1 28 29

Vague and indefinite claims 14 47 34 4 <1 36 41

Terms that are not well defined in the 

specification

8 37 42 13 <1 37 45

Nonstandard use of terms of art 7 29 44 19 1 32 49

Poor quality drawings or images 1 12 48 36 2 18 52

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 5 or fewer 

percentage points. See the related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

a

Our survey asked examiners how much easier or more difficult these factors make it to complete a 

thorough prior art search in the time allotted.

The effect of unclear applications may be exacerbated by USPTO’s 

practice that examiners should attempt to identify all applicable grounds 

for rejecting a claim or claims during their first review—a practice called 

compact prosecution. According to USPTO’s manual for patent 

examiners, this practice aims to avoid unnecessary delays. Such delays 

could be caused by examiners waiting to continue examining an 

application until previously identified issues are resolved. However, the 

practice of compact prosecution may discourage examiners from 

resolving any issues of clarity or ambiguity before conducting their initial 

searches for relevant prior art. For example, as one examiner commented 

in our survey, in order to follow the compact prosecution practice, an 

examiner must guess what unclear claims mean in order to search for 

prior art related to the claims. If examiners do not have a clear 

understanding of the scope and claims of the invention, they may not 

choose the most appropriate keywords for conducting their prior art 

searches, which may lead examiners to miss relevant prior art that could 

be found with more relevant keywords.
Page 21 GAO-16-479 Intellectual Property

Several aspects of USPTO’s patent examination policies, prior art search 

tools, and human capital management present challenges for examiners 

in identifying relevant prior art, including the time pressures examiners 

experience for prior art searches, USPTO search tools and capabilities, 

the misclassification of patent applications, and examiners’ technical 

competence.

Time pressures for prior art searches. According to most of the experts 

we interviewed and examiners we surveyed, time pressures may reduce 

examiners’ ability to conduct thorough prior art searches. These 

pressures relate to USPTO’s system for allotting an expected amount of 

time for examiners to complete an examination. For example, one expert 

noted that the amount of time examiners are allotted decreases as they 

become more experienced, and this may lead more senior examiners to 

increasingly rely on art they know well instead of searching for new art. 

As figure 5 shows, we estimate on the basis of our survey that 67 percent 

of examiners find they have somewhat or much less time than needed to 

complete thorough prior art searches given a typical workload. Our survey 

also found that examiners’ perception of the sufficiency of time for 

completing thorough prior art searches varies by technology center (see 

app. III). For example, an estimated 37 percent of examiners in the 

Mechanical Engineering, Manufacturing, and Products technology center 

reported having much less time than needed to complete a thorough prior 

art search, compared to 20 percent of examiners in the Biotechnology 

and Organic Chemistry technology center.

In analyzing our survey results, we found that how often examiners 

searched for foreign patent literature, scientific articles or presentations, 

or foreign-language nonpatent literature was statistically associated with 

their description of the sufficiency of time they had to complete a thorough 

Challenges Related to 

USPTO’s Patent 

Examination Policies, 

Search Tools, and Human 

Capital Management
Page 22 GAO-16-479 Intellectual Property

prior art search.33 Further, we asked examiners about overtime worked to 

meet their minimum production goals. A majority of examiners (an 

estimated 72 percent) worked voluntary/uncompensated overtime in the 

past 6 months to meet their goals, as shown in figure 6.34 An estimated 

30 percent of examiners worked an average of more than 10 hours of 

voluntary/uncompensated overtime per biweekly period, although 

examiners’ overtime varied by GS level (see app. III). However, on the 

basis of our survey, an estimated 56 percent of examiners experience no 

pressure to work overtime.35 In a 2007 report on USPTO’s efforts to hire 

and retain an adequate workforce, we found that an estimated 70 percent 

of examiners worked voluntary/uncompensated overtime in the previous 

year.36 Our patent quality report (GAO-16-490) provides additional 

information on how time pressures may affect patent quality.

 33We performed a test of association at the 5 percent level of significance on examiners’

responses to questions on how often they search for each of these types of prior art and 

whether they have sufficient time to complete a thorough prior art search. We also 

performed these tests conditional on the examiners’ technology center and found an 

association for foreign patent literature and foreign-language nonpatent literature. For 

scientific articles and presentations, this test could not be performed conditional on 

technology center because of insufficient data. We also tested the association for industryrelated nonpatent literature (e.g., manuals or company websites) and found no association 

for examiners overall, though results overall were marginally insignificant (p-value = 

0.05071), and we did find an association conditional on technology center. More 

information on these tests may be found in app. I.

34Examiners may have worked further overtime not included in their responses.

35We estimate that all examiners experience the following pressure to work overtime: a lot 

of pressure, 10 percent; moderate pressure, 9 percent; some pressure, 11 percent; a little 

pressure, 10 percent; no pressure, 56 percent; don’t know, 2 percent; and no response, 3 

percent. 

36We surveyed 1,420 patent examiners and received an 80 percent response rate. See 

GAO, U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to Reduce the 

Patent Application Backlog, GAO-07-1102 (Washington, D.C.: Sept. 4, 2007).
Page 23 GAO-16-479 Intellectual Property

Figure 5: Estimated Sufficiency of Patent Examiners’ Time for Completing 

Thorough Prior Art Searches

Note: See the related e-supplement, GAO-16-478SP, for the original survey language and more 

detailed results.
Page 24 GAO-16-479 Intellectual Property

Figure 6: Estimated Biweekly Voluntary/Uncompensated Overtime Worked by 

Patent Examiners in the Past 6 Months

Notes: We asked examiners how much voluntary/uncompensated overtime they worked per biweekly 

period, on average, to meet their minimum production goals in the past 6 months. Examiners may 

have worked further overtime not included in their responses. See the related e-supplement, 

GAO-16-478SP, for the original survey language and more detailed results.

Search tools and capabilities. Experts we interviewed and examiners 

we surveyed had mixed opinions on USPTO’s search tools and 

capabilities. Of the 18 experts, 10 agreed that the search tools and 

capabilities available to examiners create challenges to conducting 

thorough prior art searches.37 In general, these experts characterized 

USPTO’s search tools as being less advanced and more heavily reliant 

on keyword searching than other available tools. For example, one expert 

said that examiners’ electronic prior art searches predominantly rely on 

 37Two experts disagreed that search tools and capabilities are a challenge, and the 

remainder did not know or did not express opinions about it.
Page 25 GAO-16-479 Intellectual Property

using keywords, and this is a problem if the examiners do not know the 

most appropriate keywords. Another expert suggested that USPTO needs 

more comprehensive databases of prior art.

Based on our survey, examiners generally find that the search tools 

available to them from USPTO and from third parties make it easier to 

complete prior art searches,38 but they also find that other tools would 

help. Specifically, we estimate that a majority of examiners agree that 

certain search tools not currently available would make prior art searches 

somewhat or much easier, including a search engine that (1) can 

automatically search for concepts and synonyms related to the search 

terms entered by the examiner (an estimated 76 percent of examiners),

39

and (2) automatically generates relevant art, without keyword entry, 

based on all the claims in an application (69 percent)40 or an application’s 

specification (69 percent).41 In addition, a group of four supervisory patent 

examiners we interviewed said that it would be more efficient to search 

for prior art in one tool, with a single search method that covered multiple 

sources of prior art, including nonpatent literature. Currently, relevant 

 38We asked examiners how much easier or more difficult the search tools available to 

them from USPTO and from third parties make it to complete a thorough prior art search 

in the time allotted. For the USPTO search tools (e.g., EAST/WEST), based on our 

survey, we estimate all examiners’ responses to be as follows: much easier, 48 percent; 

somewhat easier, 34 percent; neither easier nor more difficult, 14 percent; somewhat 

more difficult, 3 percent; much more difficult, 1 percent; don’t know, less than 1 percent; 

and no response, less than 1 percent. For the third-party search tools (e.g., Google 

Patents and Derwent World Patents Index), we estimate all examiners’ responses to be as 

follows: much easier, 20 percent; somewhat easier, 48 percent; neither easier nor more 

difficult, 24 percent; somewhat more difficult, 3 percent; much more difficult, 1 percent; 

don’t know, 2 percent; and no response, 1 percent.

39Based on our survey, we estimate all examiners’ responses to be as follows: much 

easier, 35 percent; somewhat easier, 41 percent; neither easier nor more difficult, 13 

percent; somewhat more difficult, 3 percent; much more difficult, 1 percent; don’t know, 6 

percent; and no response, 1 percent. 

40We estimate all examiners’ responses to be as follows: much easier, 32 percent; 

somewhat easier, 37 percent; neither easier nor more difficult, 15 percent; somewhat 

more difficult, 4 percent; much more difficult, 2 percent; don’t know, 9 percent; and no 

response, 1 percent. 

41We estimate all examiners’ responses to be as follows: much easier, 31 percent; 

somewhat easier, 37 percent; neither easier nor more difficult, 15 percent; somewhat 

more difficult, 5 percent; much more difficult, 3 percent; don’t know, 8 percent; and no 

response, 1 percent.
Page 26 GAO-16-479 Intellectual Property

nonpatent literature may appear in different journals or databases that 

cannot be searched with a single search function. Similarly, examiners 

commented in our survey about the difficulty of searching nonpatent 

literature or requested an easier way to search for it. For example, 

examiners requested a method for searching nonpatent literature that is 

integrated with their systems for searching patent literature.42 In some 

cases, USPTO examiners may not be able to use public search engines 

because searching for specific terms or concepts from an unpublished

application may put the confidentiality of the application at risk.

Examiners also requested improvements to USPTO’s current translation 

capabilities. On the basis of our survey, a majority of examiners (an 

estimated 86 percent) agree that having access to immediate machine 

translation of foreign-language documents would make it somewhat or 

much easier to complete a thorough prior art search in the time allotted,43

while about half of examiners (an estimated 50 percent) find that 

additional translators would make it somewhat or much easier.44

Misclassification of patent applications. According to examiners, 

another challenge they face is misclassification of patent applications. 

Misclassification occurs when an application is not classified into the 

group most closely associated with the invention. This can result in 

applications being routed to the wrong USPTO technology center or art 

 42Examiners made such comments in response to our open-ended survey question asking 

examiners for their suggestions for improving any factors that they indicated made prior 

art searches more difficult. Because we did not conduct a systematic review of all openended responses to our survey, we are not reporting the exact number of examiners who 

provided a response on this topic. Responses to open-ended questions are not 

generalizable to other examiners. 

43We estimate all examiners’ responses to be as follows: much easier, 46 percent; 

somewhat easier, 41 percent; neither easier nor more difficult, 11 percent; somewhat 

more difficult, 1 percent; much more difficult, less than 1 percent; don’t know, 2 percent; 

and no response, 1 percent. Survey results reported here do not sum to the result 

provided above because of rounding.

44We estimate all examiners’ responses to be as follows: much easier, 18 percent; 

somewhat easier, 31 percent; neither easier nor more difficult, 33 percent; somewhat 

more difficult, less than 1 percent; much more difficult, less than 1 percent; don’t know, 13 

percent; and no response, 3 percent. Survey results reported here do not sum to the result 

provided above because of rounding.
Page 27 GAO-16-479 Intellectual Property

unit for examination.45 Although only 2 of the 18 experts we interviewed 

raised concerns about the misclassification of patent applications, on the 

basis of our survey, an estimated 75 percent of examiners encountered 

misclassified applications sometimes, often, or always in the past 

quarter,46 and an estimated 76 percent of examiners find that the 

misclassification of patent applications makes it somewhat or much more 

difficult to complete a thorough prior art search in the time allotted.47

According to our survey, how often examiners encountered misclassified 

applications in the past quarter varied by technology center (see app. III).

Misclassified applications pose particular challenges for examiners, 

according to examiners’ comments to our survey. For example, while 

examiners receiving misclassified applications can request that an 

application be transferred to a different art unit, they may encounter 

difficulties doing so, according to examiners’ comments in our survey.48

Misclassified applications that are not transferred can negatively affect 

 45Although USPTO moved to the Cooperative Patent Classification (CPC) system as of 

January 2015, according to officials, the agency is still directing its classification contractor 

to also assign classifications using the previous system, the United States Patent 

Classification (USPC) system. As of March 2016, USPTO continued to route applications 

to art units on the basis of their USPC classifications. It is not clear how use of the CPC 

system will affect the routing of applications once USPTO begins using CPC classification 

results for routing.

46We estimate that examiners encountered misclassified patent applications (e.g., routed 

to the wrong art area) as follows: always, 3 percent; often, 21 percent; sometimes, 51 

percent; rarely, 23 percent; never, 1 percent; don’t know, 1 percent; and no response, 1 

percent.

47We estimate all examiners’ responses to be as follows: much easier, less than 1 

percent; somewhat easier, 1 percent; neither easier nor more difficult, 16 percent; 

somewhat more difficult, 33 percent; much more difficult, 44 percent; don’t know, 3 

percent; and no response, 3 percent. Survey results reported here do not sum to the result 

provided above because of rounding.

48Our survey did not ask examiners about the ease or difficulty of transferring applications. 

However, examiners raised concerns about the difficulty of transferring applications in 

response to our open-ended survey questions asking examiners for the following: other 

factors not identified in our survey that make it easier or more difficult to conduct prior art 

searches in the time allowed; their suggestions for improving any factors that they 

indicated as making prior art searches more difficult; and other issues related to USPTO’s 

ability to identify relevant prior art or patent quality that we should be aware of. Because 

we did not conduct a systematic review of all open-ended responses to our survey, we are 

not reporting the exact number of examiners who provided a response on this topic. 

Responses to open-ended questions are not generalizable to other examiners.
Page 28 GAO-16-479 Intellectual Property

examiners’ confidence in their work and the quality of the examination, 

based on examiners’ comments in our survey. This is because when 

applications are routed to the wrong art areas, they may be reviewed by 

examiners who do not have appropriate knowledge and experience to 

understand the invention or relevant prior art and determine appropriate 

search strategies and terms. Additionally, misclassified applications that 

are never corrected could be difficult for examiners to subsequently use 

as prior art in later examinations. This is because misclassified 

applications may be difficult to find when examiners search for patent 

literature using technology-specific patent classification categories.

Ensuring examiners’ technical competence. According to most experts 

we interviewed, ensuring that examiners have sufficient and appropriate 

technical backgrounds, knowledge, or skills for conducting thorough prior 

art searches is also a challenge. The role of patent examiner is a difficult 

one that can take years to learn, and examiners with less education or 

work experience, or who are not abreast of advances in a particular 

technology area, may not have the technical knowledge necessary to 

identify relevant prior art. According to USPTO officials, an examiner’s 

technical knowledge enables him or her to understand the invention being 

searched, and if the examiner does not understand the invention, he or 

she may not know what to search for, where to look, and when to stop

searching. In addition, we estimate on the basis of our survey that 82 

percent of examiners find it somewhat or much easier to complete a 

thorough prior art search in the time allotted for applications with a subject 

matter in which they have knowledge of existing prior art based on their 

education or previous work experience.49

As of May 2015, approximately 39 percent of all examiners in the 

technology centers we reviewed had been at the agency for less than 5 

years, and USPTO has historically faced challenges in retaining 

examiners.50 In addition, as of September 2015, a majority of examiners 

 49We estimate all examiners’ responses to be as follows: much easier, 37 percent; 

somewhat easier, 45 percent; neither easier nor more difficult, 12 percent; somewhat 

more difficult, 1 percent; much more difficult, less than 1 percent; don’t know, 3 percent; 

and no response, 2 percent.

50In 2007, we reported that attrition was significantly offsetting the agency’s hiring efforts. 

Specifically, from 2002 through 2006, one patent examiner left USPTO for nearly every 

two hired, and 70 percent of those who left had been at the agency for less than 5 years. 

See GAO-07-1102.
Page 29 GAO-16-479 Intellectual Property

in the technology centers we reviewed (61 percent) did not have a degree 

beyond a bachelor’s degree when hired. USPTO officials told us that the 

agency has aimed to match new hires’ previous work experiences and 

educational backgrounds to technology centers. When we asked USPTO 

how often examiners in each technology center have technical work 

experience or education relevant to their art unit, agency officials 

answered “always” for five centers, “often” for three centers, and 

“sometimes” for one center. However, our survey found that in the past 

quarter, less than half of examiners—an estimated 42 percent—always or 

often encountered applications with a subject matter in which they have 

knowledge of existing prior art based on their education or previous work 

experience.51

The training examiners receive—particularly continuing training in 

evolving technologies—may also affect their ability to maintain the 

technical competence they need to effectively identify relevant prior art. 

Overall, an estimated 60 percent of examiners found that the continuing 

education they received from USPTO in their art area in the past year 

was at least somewhat useful. However, 22 percent of examiners had not 

taken such training or had not been offered such training,52 with 

responses by technology center ranging from an estimated 10 percent to 

34 percent of examiners (see app. III). We found in 2005 that examiners 

were reluctant to attend voluntary training given the time demands 

involved.53 Similarly, according to the group of five examiners we 

interviewed in 2015, USPTO did not offer enough ongoing technical 

training and did not always give examiners sufficient time to complete 

training.

 51We estimate that examiners encountered such patent applications as follows: always, 7 

percent; often, 35 percent; sometimes, 40 percent; rarely, 13 percent; never, 3 percent; 

don’t know, 1 percent; and no response, 1 percent. 

52We estimate all examiners’ responses to be as follows: very useful, 20 percent; 

moderately useful, 22 percent; somewhat useful, 18 percent; slightly useful, 10 percent; 

not at all useful, 4 percent; not taken, 12 percent; not offered, 10 percent; and no 

response, 4 percent. 

53As part of the 2005 review, we conducted 11 focus groups with a total of 91 patent 

examiners and supervisory examiners. See GAO, Intellectual Property: USPTO Has Made 

Progress in Hiring Examiners, but Challenges to Retention Remain, GAO-05-720 

(Washington, D.C.: June 17, 2005). 
Page 30 GAO-16-479 Intellectual Property

EPO and JPO use several approaches that may help their examiners 

address challenges in identifying prior art similar to those that experts and 

respondents to our survey of patent examiners cited. These approaches 

include worksharing, creating internal databases of nonpatent literature, 

using patent classification systems, adopting advanced search tools, 

hiring and training of examiners to promote technical expertise, and 

incorporating review and audit procedures.54

Worksharing. EPO and JPO have systems to share search and 

examination results. The exchange of prior art searches or patent 

examination results among patent offices—referred to as worksharing—

facilitates more efficient prior art searches for applications filed in multiple 

countries, according to foreign officials. As we discussed above, 

approximately half of all patent applications worldwide are part of patent 

families filed in multiple offices. To gain patent protection in multiple 

countries, inventors may file applications for the same invention in several 

different offices. Because the underlying invention is the same, the search 

and examination results at one office may be useful to examiners in 

another office. Therefore, worksharing can ameliorate the challenges 

posed by the quantity of potential prior art that examiners must search 

and the time pressure examiners face by allowing patent offices to 

leverage others’ work and knowledge. The IP5, a multilateral forum of the 

five largest patent offices, identified worksharing as the main tool for 

addressing an increasing number of patent applications, while helping the 

patent offices conduct timely, quality examinations. JPO officials noted 

that worksharing systems allow more efficient access to foreign patent 

literature and help JPO examiners more easily see foreign examiners’

work. JPO examiners still have to review the relevant literature 

themselves, but access to foreign examination results may help JPO 

examiners identify relevant art or sources of prior art. An EPO official also 

noted that worksharing could give offices access to search reports of 

unpublished applications at other offices, and that these search reports 

may include prior art relevant to similar applications filed with EPO.

Creating internal databases of nonpatent literature. EPO and JPO 

incorporate specific nonpatent literature resources into their offices’ main 

 54We interviewed EPO and JPO officials to identify approaches the offices use to address 

challenges in identifying relevant prior art; however, we did not assess the effectiveness of 

these approaches. The approaches used by USPTO are discussed below.

Patent Offices for 

Europe and Japan 

Use Various 

Approaches to Help 

Address Challenges 

in Identifying 

Relevant Prior Art
Page 31 GAO-16-479 Intellectual Property

search tools, which may allow examiners to more efficiently search both 

patent and nonpatent literature by keyword, rather than searching multiple 

sources individually. Officials at these offices said that incorporating these 

resources into their main search tools allows examiners to consider a 

wide array of nonpatent literature sources.

According to EPO officials, EPO’s primary search tool allows examiners 

to use a single interface to search through all of EPO’s internal databases 

as well as some external databases of prior art.55 In addition to creating 

an internal database of nonpatent literature documents to facilitate 

examiners’ searches, EPO also added descriptive metadata to index 

these documents to help examiners find the most relevant documents. 

EPO’s system also incorporates features that make it user-friendly—for 

example, grouping documents to avoid examiners facing dozens of 

copies of the same article from different sources.56 EPO officials stated 

that examiners find that these groupings enhance their ability to search 

through larger quantities of prior art. Overall, EPO officials identified their 

search engine and the large number of indexed prior art resources 

available within the search engine as strengths of the office in addressing 

the quantity of prior art available. EPO officials said that the office prefers 

to have internal collections of these resources so they may be integrated 

into its search tools, but some sources remain external and separate from 

EPO’s search systems. Because the cost of bringing sources into EPO’s 

search systems is high, the office needs a strong business case to justify 

the expense. According to these officials, publishers of academic journals 

and other nonpatent literature often raise subscription prices annually, so 

the office also must decide what to exclude, because there is simply too 

much nonpatent literature for the office to include everything.

According to JPO officials, JPO has internal databases of both patent and 

nonpatent literature and also subscribes to external databases for 

nonpatent literature. JPO’s internal databases include JPO patent 

 55EPO officials identified several types of nonpatent literature that EPO took special effort 

to obtain, including documentation from standards development organizations, nonpatent 

publications or disclosures by companies, and open-source journals. These officials said 

that standards documents are particularly relevant for examiners in telecommunications, 

computer science, and audio/visual fields.

56EPO’s system groups documents by patent family and through bibliographic information, 

such as author name. 
Page 32 GAO-16-479 Intellectual Property

literature, some foreign patent literature, and selected nonpatent 

literature. JPO annually selects useful nonpatent literature to add to its 

database based on recommendations from an internal committee of 

examiners and officials. In addition, JPO uses commercial databases of 

nonpatent literature to augment its internal database and annually 

evaluates whether each external database is useful for prior art search.

Using patent classification systems. EPO and JPO use classification 

systems to help examiners narrow their searches and find relevant patent 

literature even if the patent application uses inconsistent terms or is 

written in a foreign language. EPO and JPO classify applications and 

patents into groups that describe specific components of the invention. 

Examiners may then look at other applications and patents within the 

same group to find similar inventions. A system that has more groups—

sometimes called classes—allows for finer distinctions, and prior art 

searches within a class will produce fewer, more focused results. 

Reducing the number of results an examiner needs to consider 

decreases the time needed for the prior art search, thereby improving 

efficiency. Furthermore, by grouping similar patents into classes that 

describe details of the inventions, classification systems allow examiners 

to conduct prior art searches based on the classes assigned to an 

application, independent of its language and wording. This can improve 

search quality by producing results that would not be found from a 

specific keyword search but are nonetheless closely related. According to 

EPO officials, this can be particularly helpful for finding documents in 

Asian languages that are otherwise difficult for EPO examiners to search, 

and it facilitates searches that are less dependent on the examiner’s 

choice of keywords.

Adopting advanced search tools. EPO and JPO officials told us that 

examiners in both offices primarily use classification and keyword 

searches instead of more complex search tools. However, EPO uses 

some automated tools that assist in keyword-based searches or provide 

automated search results. One EPO tool provides recommended search 

terms in its three official languages to examiners, based on a database of 

examiners’ past searches. This tool can improve search efficiency by 

providing additional search terms, such as synonyms or translations, that 

the examiner may not have initially considered. Another EPO tool 

provides examiners with the results of an automated search based on the

concepts and words drawn from the claims in applications. The results 

are in the form of a ranked list of potentially related documents, which 

provides a starting point for examiners before they begin a manual 

search. 
Page 33 GAO-16-479 Intellectual Property

Hiring and training examiners to promote technical expertise. EPO 

and JPO hire examiners with technical expertise and provide several 

years of training, according to officials from these offices. EPO only 

recruits students with at least a master’s degree and almost all JPO 

examiners join the office after graduate school, according to EPO and 

JPO officials. In addition, JPO’s outsourced aspects of prior art searches 

are often carried out by retired engineers or technical experts, and 

according to JPO officials, their experience may enhance the quality of 

patent application examinations. At EPO, examiners must also meet a 

requirement that they understand English, German, and French. EPO 

examiners spend their first 2 years in training, alternating periods of 

classroom training with periods of on-the-job training. JPO examiners 

spend their first 2 to 4 years (depending on their academic degree and 

work experience) as assistant examiners and receive practical training by 

a supervising examiner. Thereafter, examiners in both offices have 

opportunities for additional technical training. According to EPO officials, 

EPO examiners need 3 to 4 years before they are fully trained, and 4 to 5 

years in certain fields, such as biotechnology. At both offices, examiners 

also tend to spend their entire career with the office. JPO officials 

identified the longevity of examiners as a strength, stating that examiners 

are well trained and have experience that helps ensure examination 

consistency.

Incorporating review and audit procedures. EPO and JPO have 

procedures to review examiners’ work before issuing or denying a patent. 

At EPO, applications are examined by a panel of three examiners: a first 

examiner, second examiner, and chair. According to EPO officials, the 

first examiner generally performs the bulk of the search and 

administrative tasks, with the other two examiners actively discussing and 

approving office actions. Examiners from a particular technology area are 

assigned to each panel randomly. An individual examiner may serve as 

first examiner on some applications and second examiner or chair on 

others. EPO’s quality management system also calls for random audits, 

including a minimum of two prior art search audits for each examiner per 

year, and chairs of the examination committees are required to record 

data on the quality of their examinations.

JPO officials also reported taking steps to ensure that the patents it 

issues are of high quality. These steps include a director quality check, 

consultations with other examiners, and quality audits of a sample of 

examinations. According to JPO’s quality manual, JPO directors conduct 

a quality check on examiners’ decisions to grant or reject applications 

before the office actions are sent to the applicants. The manual also 
Page 34 GAO-16-479 Intellectual Property

encourages examiners to consult with their directors or other examiners 

who may be able to provide guidance in examining an application. In 

2014, examiners recorded approximately 83,000 consultations and 

received approximately 243,000 new applications, according to JPO 

officials. Experienced managers or examiners who serve as quality 

management officers also perform quality audits on randomly selected 

applications. According to JPO officials, the office has around 90 such 

officers in particular technology areas who review entire examinations 

and complete independent searches, and 4 wide-area officers who focus 

on the appropriateness of examiners’ decisions but do not conduct 

additional searches. According to these officials, while the number of 

audits is not large enough to allow tracking of individual examiner 

performance, it allows for statistical monitoring of examination quality 

overall.

USPTO has taken or begun planning various actions that may help 

address challenges in identifying relevant prior art, but some of these 

actions have limitations that may hinder their effectiveness. USPTO’s 

actions span the following areas: (1) leveraging the work of foreign patent 

offices, (2) encouraging submission of prior art from third parties, (3) 

improving prior art search tools, (4) monitoring examiners’ prior art 

searches, (5) evaluating the agency’s system for determining the amount 

of time examiners are allotted to examine patent applications, and (6) 

strengthening the technical competence of the examiner corps.57 These 

actions may help address challenges related to prior art and applications 

and to examination policies, search tools, and human capital 

management. In some cases, these actions are coordinated with, similar 

to, or could be informed by approaches taken by EPO or JPO.

 57Our patent quality report provides additional information on steps that USPTO is taking 

to address patent quality (see GAO-16-490).

USPTO Is Taking 

Actions to Address 

Challenges in 

Identifying Prior Art, 

but Some Actions 

Have Limitations
Page 35 GAO-16-479 Intellectual Property

USPTO has taken actions to leverage the work of foreign patent offices 

just as EPO and JPO have done, but the agency has experienced 

challenges working with EPO on a new patent classification system. 

USPTO is collaborating with foreign patent offices to give examiners 

access to examination files from those offices. Specifically, USPTO has 

(1) contributed data to information technology systems that share 

published information among patent offices; (2) engaged in pilot programs 

to collaborate with other offices during search and examination of related 

patent applications; and (3) adopted a new, joint classification system 

with EPO. These efforts may help address several of the challenges we 

identified relating to prior art and applications, including the large quantity 

of art available and the availability of prior art, particularly foreign patents 

and art written in foreign languages.58

First, USPTO contributes data to three worksharing systems that make 

applications, prior art citations, and examination results from foreign 

patent offices available to examiners and the public. One system, the 

Common Citation Document, provides bibliographic information, such as 

title and source, of patent and nonpatent prior art citations.59 USPTO, 

EPO, and JPO launched this system in 2011, and have since expanded it 

to include information from additional patent offices. Another system, 

Global Dossier, provides access to the examination history of 

applications, such as office actions and other correspondence.60 USPTO 

examiners gained access to Global Dossier in 2015. USPTO officials 

described Global Dossier as a system that allows examiners to quickly 

and easily view office actions from participating foreign patent offices. 

EPO officials described these two systems as complimentary, with each 

system providing advantages based on the specific search task involved. 

A third system, PATENTSCOPE, run by the World Intellectual Property 

Organization, provides access to applications filed with multiple offices 

under the Patent Cooperation Treaty as well as patents granted by 

 58Foreign patent documents may be in English or other languages. For example, about 80 

percent of EPO applications are in English, and EPO publishes an English-language 

version of the claims of all patents it grants. 

59For more information on Common Citation Document, see 

http://ccd.fiveipoffices.org/CCD-2.0.8/, accessed April 4, 2016.

60For more information on Global Dossier, see http://globaldossier.uspto.gov/#/, accessed 

April 4, 2016.

USPTO Has Taken Actions 

to Leverage Related Work 

by Foreign Patent Offices, 

but Has Experienced 

Challenges Working with

EPO to Adopt a New 

Patent Classification 

System
Page 36 GAO-16-479 Intellectual Property

several regional and national offices.61 By allowing examiners to review 

prior art that examiners from other offices found useful in examining 

patent applications, these systems may help address the challenges 

related to managing the large quantity of prior art and accessing prior art 

that would otherwise be difficult to obtain.

Second, to enhance collaboration on related patent applications filed in 

multiple countries, USPTO has entered into two pilot programs to jointly 

examine certain applications: one with JPO and one with the Korean 

Intellectual Property Office. Applicants who file related patent applications 

with USPTO and either JPO or the Korean Intellectual Property Office 

may request to enter the pilot—known as the Collaborative Search Pilot 

Program—and receive expedited review of their applications. In the pilot 

with JPO, USPTO examiners consider prior art from both JPO’s search 

results and their own search results before responding to the applicant 

with the first office action. In the pilot with the Korean Intellectual Property 

Office, examiners perform independent searches and examinations, then 

compare results prior to final office actions. According to USPTO officials, 

these collaborative efforts could improve searches for patent and 

nonpatent literature in foreign languages, as examiners from multiple 

countries will perform searches in their native languages and share the 

results.

Third, USPTO, in collaboration with EPO, designed and implemented a 

new, joint patent classification system meant to improve examiners’ ability 

to find relevant patent literature from the United States, Europe, or other 

offices that adopt the system. However, USPTO and patent examiners 

have experienced some challenges with this effort. As discussed earlier, 

by grouping similar patents, classification systems allow examiners to 

conduct prior art searches independent of an application’s language and 

 61The Patent Cooperation Treaty established “a Union for cooperation in the filing, 

searching, and examination of applications for the protection of inventions and for 

rendering special technical services.” In its 2015 World Intellectual Property Indicators, the 

World Intellectual Property Organization reported that applications under the treaty 

accounted for about 214,000 of the approximately 2.7 million patent applications filed 

worldwide in 2014. According to a USPTO official, USPTO examiners who receive these 

applications continue to perform their own prior art searches, but may benefit from the 

international search report created as part of the Patent Cooperation Treaty application 

process. For more information on PATENTSCOPE, see https://patentscope.wipo.int/, 

accessed April 4, 2016. 
Page 37 GAO-16-479 Intellectual Property

wording and can improve search quality by producing results that would 

not be found from a specific keyword search. According to USPTO 

officials, the U.S. Patent Classification system became outdated following 

budget constraints in the early 2000s that undermined the agency’s 

efforts to update the classification system in response to technological 

change. After identifying a need to revise its classification system, 

USPTO partnered with EPO to create a new system to be jointly used by 

both offices. USPTO officially adopted the Cooperative Patent 

Classification (CPC) system in January 2015. According to USPTO 

officials, CPC gives examiners greater access to foreign patent literature 

compared to the prior system and can be expanded more easily to 

include new technologies.

USPTO has found that classifications for some U.S. patent applications 

under the CPC system are inconsistent with EPO’s classifications, and 

USPTO examiners have reported difficulty with the system. Officials from 

both USPTO and EPO told us that there were disagreements between 

the two offices on how to classify some applications, such as on which 

categories to use or the number of categories to which applications are 

assigned. Additionally, USPTO examiners in certain technology centers 

reported difficulty with the CPC system in our survey. On the basis of 

survey responses from our random probability sample of examiners, we

found that some examiners find that the CPC classifications make it 

somewhat or much more difficult to complete a thorough prior art search 

in the time allotted, as shown in table 4. For example, on the basis of our 

survey, CPC classifications make it somewhat or much more difficult to 

complete a thorough prior art search in the time allotted for an estimated 

53 percent of the examiners in the Mechanical Engineering, 

Manufacturing, and Products technology center. On the other hand, an 

estimated 42 percent of examiners in the Computer Networks, Multiplex 

Communication, Video Distribution, and Security technology center find 

CPC classifications make it somewhat or much easier to complete a 

thorough prior art search in the time allotted. In response to an openended survey question, one examiner described their difficulty, saying 

that proper classification of prior art is essential to quality and efficiency of 

examination, that classification of prior art in the CPC system is 

inconsistent, and that many CPC categories still contain large amounts of 
Page 38 GAO-16-479 Intellectual Property

prior art despite having more detailed technology categories.62 USPTO 

officials we interviewed also told us that the number of documents in 

some CPC categories was greater than those in comparable categories 

under USPTO’s previous classification system.

Table 4: Estimated Effects of the Cooperative Patent Classification System on Completing a Thorough Prior Art Search in the 

Time Allotted 

Estimated percentage of examiners

U.S. Patent and Trademark Office technology 

center Much easier

Somewhat 

easier

Neither easier 

nor more difficult

Somewhat 

more difficult

Much more 

difficult

Biotechnology and Organic Chemistry 2 11 60 16 4

Chemical and Materials Engineering 6 25 28 25 16

Computer Architecture, Software, and Information 

Security 3 26 42 19 10

Computer Networks, Multiplex Communication, 

Video Distribution, and Security 10 32 38 14 6

Communications 7 30 37 18 7

Semiconductors, Electrical and Optical Systems 

and Components 5 29 32 20 13

Transportation, Construction, 

Electronic Commerce, 

Agriculture, National Security 

and License and Reviewa

Business methods 3 16 52 19 7

Not business 

methods 3 16 25 31 25

Mechanical Engineering, Manufacturing, and 

Products 6 17 22 28 25

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 5 or fewer 

percentage points. See the related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.

 62Examiners made such comments in response to our open-ended survey question asking 

examiners for their suggestions for improving any factors that they indicated make prior art 

searches more difficult. Because we did not conduct a systematic review of all openended responses to our survey, we are not reporting the exact number of examiners who 

provided a response on this topic. Responses to open-ended questions are not 

generalizable to other examiners.
Page 39 GAO-16-479 Intellectual Property

According to USPTO officials, adopting the CPC system helps examiners 

give greater consideration to foreign patent documentation. As we note in 

this report, foreign patent and foreign-language nonpatent literature is 

sometimes difficult for examiners to find, according to the experts we 

interviewed and our survey of examiners. However, the CPC system’s 

usefulness to examiners depends, in part, on the consistency with which 

the system is applied by its users. Inconsistently applying the 

classification system to applications undermines its usefulness because 

similar applications may not be placed in the same category by USPTO 

and EPO. Also, misclassified applications may require examiners to 

familiarize themselves with new technologies, which increases the 

amount of time it takes for them to conduct their prior art searches and 

may negatively affect their confidence in the results. At USPTO, 

classification contractors apply an initial classification to applications 

before they are routed to an examiner. The examiner may revise the 

classification during examination. To help transition to the new system, 

USPTO provided examiners and classification contractors with training on 

how to classify applications and how to search for references using the 

CPC system, assigned lead quality experts to each technology center to 

provide training and assistance, and established a feedback tool for 

examiners to report issues with the new system.63 According to USPTO 

officials, the agency has not identified specific reasons that account for 

differences between its classifications and EPO’s. However, USPTO is 

updating its contract for initial classification services to require a higher 

level of expertise and is drafting a plan to assess performance under the 

new contract.

The agreement between USPTO and EPO that supports the CPC system 

commits both offices to exchange best practices on classifications and 

aim for a high level of consistency, but it does not include a specific target 

or method for assessing consistency.64 USPTO officials told us that they 

are actively working with EPO to identify ways to measure and improve 

consistency, and that both offices have developed their own methods for 

 63In its technical comments on a draft of this report, USPTO stated that feedback from this 

tool led the agency to give some examiners additional time to complete their examinations 

using CPC.

64Memorandum of Understanding between the United States Patent and Trademark 

Office and the European Patent Office on the Terms relating to the Cooperative Patent 

Classification (“CPC Terms”), MoU No. 15111120-0000-000. 
Page 40 GAO-16-479 Intellectual Property

measuring the consistency of CPC classifications. However, as of March 

2016, EPO and USPTO had not agreed upon a measure of consistency 

between the offices. In the context of interagency collaborations, we have 

previously reported that agencies can enhance and sustain their 

collaborative efforts by, among other things, defining and articulating a 

common outcome and establishing mutually reinforcing or joint 

strategies.65 Establishing agreement with EPO on a target of consistency 

in classifying patent applications and a plan for monitoring consistency 

would help to implement this practice and would establish an internal 

control to support the reliability and usefulness of the CPC system.66

Since 2012, USPTO has had authority to receive submissions from third 

parties that may help identify potentially relevant prior art for published 

patent applications. This process, sometimes described as 

crowdsourcing, allows experts and other interested parties to share 

documents that they believe will aid USPTO’s examination of an 

application. Prior art submitted by third parties can help identify prior art 

that is not immediately available to examiners (e.g., because it is not 

available electronically or without payment) or that may be difficult to find. 

By potentially reducing the need to search for this prior art, these 

submissions may address challenges related to the quantity and 

availability of prior art. According to information from USPTO, third parties 

submitted prior art for less than 1 percent of the approximately 600,000 

patent applications received per year since third-party submissions began 

in 2012.

In addition, based on our survey, we estimate that 83 percent of 

examiners have rarely or never seen an application with a third-party 

submission.67 However, USPTO evaluations of third-party submissions 

 65GAO, Managing for Results: Key Considerations for Implementing Interagency 

Collaborative Mechanisms, GAO-12-1022 (Washington, D.C.: Sept. 27, 2012). 

66GAO/AIMD-00-21.3.1. Internal control comprises the plans, methods, and procedures 

used to meet agencies’ missions, goals, and objectives. Internal control activities are the 

policies, procedures, techniques, and mechanisms that enforce management’s directives. 

67We estimate that examiners encountered such applications in the past quarter as 

follows: always, less than 1 percent; often, 1 percent; sometimes, 9 percent; rarely, 36 

percent; never, 47 percent; don’t know, 5 percent; and no response, 2 percent. 

USPTO Has Taken Actions 

to Encourage the 

Submission of Prior Art by 

Third Parties
Page 41 GAO-16-479 Intellectual Property

found that they were generally useful to examiners and that the 

documents were sometimes not available to examiners through USPTO 

sources. According to USPTO officials we interviewed, an internal review 

of 300 randomly selected applications for which a third party submitted 

prior art found that examiners used a submission to reject a patent 

application in 20 to 25 percent of such cases. Furthermore, USPTO 

reviewed approximately 2,500 submissions of nonpatent literature prior 

art it received since September 2012, and found 753 unique, Englishlanguage prior art documents (30 percent of the submissions received) 

that were otherwise not available to examiners.68 These documents were 

mostly journal or other scholarly articles, but also included other types of 

nonpatent literature, such as marketing materials, book chapters, 

conference proceedings, presentations, and manuals that could not be 

found in the agency’s internal or external nonpatent literature sources.

Because potential third-party submitters have to know about an 

application before they can submit potentially related art, USPTO has 

taken steps to allow third parties to more easily monitor published 

applications. The agency created a subscription service, whereby users 

will be notified when applications are published that contain user-chosen 

keywords. Beyond these efforts, USPTO also considered expanding use 

of third-party submissions of prior art by allowing examiners to use 

crowdsourcing to solicit such art for specific applications or topics. 

According to USPTO officials, the agency decided not to pursue this 

option, at least in part, because the agency is uncertain about its legal 

authority to do so. Specifically, while statute allows USPTO to collect 

third-party submissions, the law requires the agency to establish 

procedures to ensure that no protest or opposition is initiated during 

examination of a patent application.69 According to USPTO officials, it is 

unclear if requests for third-party submissions on a specific application 

could be viewed as allowing opposition.

 68We did not assess the methodology used by USPTO in its evaluations. 

6935 U.S.C. § 122(c) and (e).
Page 42 GAO-16-479 Intellectual Property

USPTO is making improvements to its prior art search tools that may help 

address some of the challenges examiners face in identifying relevant 

prior art, but USPTO has not developed a strategy to assess 

incorporating new sources of art into these tools over time.

As part of its Enhanced Patent Quality Initiative, USPTO is procuring an 

automated prior art search capability that could enhance examiners’

ability to identify relevant prior art.70 In June 2015, USPTO requested 

information on a search system that uses the claims and specification 

within a patent application to search for patent and nonpatent literature 

automatically, without human involvement. In its request for information, 

USPTO described the intent of the system as providing a useful prior art 

baseline for patent examiners to begin their own searches. Such a 

system could improve the search tools available to examiners and help 

address the challenge of managing the quantity of potentially relevant 

prior art. USPTO anticipates that a pilot system will be available to a 

limited number of examiners, and agency officials told us that they 

anticipate awarding a contract in late summer 2016.

The new system could expand upon the capabilities of a currently 

available system called the Patent Linguistic Utility Service. According to 

USPTO officials, the Patent Linguistic Utility Service has been in use for 

20 years and has limitations that prevent it from meeting the agency’s 

search needs. For example, the Patent Linguistic Utility Service only 

searches U.S. patent literature and cannot perform the advanced search 

techniques contemplated for the new system. USPTO uses the current 

system to perform searches for about one-tenth of the agency’s incoming 

applications, whereas USPTO expects to use the new system for every 

application. Based on our survey, an estimated 12 percent of examiners 

find that the current system makes it somewhat or much easier to 

 70USPTO began the Enhanced Patent Quality Initiative in February 2015 to build 

confidence in the patent system by enhancing patent quality, to make the system 

understandable and usable by all inventors, and to ensure that each customer is treated 

fairly and professionally throughout the patent application process. We discuss this 

initiative further in GAO-16-490.

USPTO Is Making 

Improvements to Prior Art 

Search Tools but Has Not 

Developed a Strategy to 

Assess New Sources of 

Art
Page 43 GAO-16-479 Intellectual Property

complete a thorough prior art search in the time allotted,71 whereas 52 

percent of examiners expect that an automated pre-examination search 

would make it somewhat or much easier to complete a thorough prior art 

search in the time allotted.72

Moreover, USPTO’s 2014-2018 Strategic Plan includes an objective to 

ensure optimal information technology services, including upgrading 

search systems and prior art access. Toward this objective, USPTO is in 

the process of a major, multiyear $405 million effort to upgrade its 

information technology tools to provide examiners with a new system to 

manage all aspects of patent examination, including certain aspects of 

their prior art searches. The new system, called Patents End-to-End, will, 

according to USPTO’s Strategic Information Technology Plan for fiscal 

years 2015 through 2018, replace nearly 20 systems currently used to 

search patent applications. The new system will initially replicate the prior 

art search capabilities of USPTO’s current systems, such as EAST and 

WEST, which focus on U.S. patent literature and include only one source 

of nonpatent literature. Although searching for nonpatent literature is 

required by the agency’s manual for patent examiners, under the current 

and planned systems, examiners need to individually access and search 

a variety of external sources to look for nonpatent literature during their 

examinations. Consequently, neither the current nor planned systems 

provide USPTO examiners with the capability to efficiently search for prior 

art using a single, integrated search that includes both patent literature 

and multiple sources of nonpatent literature.

 71We asked examiners how much easier or more difficult automated search results from 

the Patent Linguistic Utility Service, the current USPTO system, make it to complete a 

thorough prior art search in the time allotted. We estimate all examiners’ responses to be 

as follows: much easier, 2 percent; somewhat easier, 11 percent; neither easier nor more 

difficult, 47 percent; somewhat more difficult, 5 percent; much more difficult, 4 percent; 

don’t know, 24 percent; and no response, 9 percent. Survey results reported here do not 

sum to the result provided above because of rounding.

72We asked examiners if automated, pre-examination search with results provided by 

advanced natural language and linguistic technologies would make it easier or more 

difficult to complete a thorough prior art search in the time allotted. We estimate all 

examiners’ responses to be as follows: much easier, 16 percent; somewhat easier, 37 

percent; neither easier nor more difficult, 25 percent; somewhat more difficult, 6 percent; 

much more difficult, 3 percent; don’t know, 12 percent; and no response, 1 percent. 

Survey results reported here do not sum to the result provided above because of 

rounding.
Page 44 GAO-16-479 Intellectual Property

The time it takes examiners to search the large and increasing volume of 

nonpatent literature and the inefficiency of having to search many 

different sources individually may lead examiners to conduct less 

thorough searches of nonpatent literature, potentially missing relevant 

prior art. As discussed earlier, our analysis of examiners’ survey 

responses and experts’ statements suggests that examiners are less 

likely to search for certain types of prior art, particularly foreign-language 

patents and nonpatent literature, from which it is more difficult to find 

relevant prior art. Integrating additional sources of prior art into USPTO’s 

search tools is one way USPTO could increase the types and sources of 

prior art that examiners consider, and would be similar to approaches that 

EPO and JPO cited as helping examiners consider a wide array of 

nonpatent literature sources. According to USPTO officials, the 

capabilities of the new Patents End-to-End system can be expanded in 

the future to include additional nonpatent literature sources. However, 

USPTO officials told us that, as of March 2016, the agency does not 

currently have specific plans to add additional nonpatent literature 

sources to its new system because of its initial focus on developing parity 

with the existing system. In addition, as of March 2016, USPTO had not 

established a documented strategy to identify and assess new sources in 

the future or the most optimal means of providing access to them.

According to federal standards for internal control, control activities are 

the policies, procedures, techniques, and mechanisms that help ensure 

that actions are taken to address risks. These activities are an integral 

part of an agency’s planning to achieve effective results and efficiently 

manage government resources, including the development and 

maintenance of information systems. Because information technology 

changes rapidly, the internal control standards note that controls must 

evolve to remain effective. These standards also highlight the importance 

of clearly documenting internal controls.73 Without a documented, 

periodically updated strategy to evaluate new sources of prior art to 

include in the Patents End-to-End system and a process to periodically 

assess this strategy, USPTO will not have the assurance that it is taking 

full advantage of its information technology investment to help examiners 

more efficiently access a variety of resources for their prior art searches.

 73GAO/AIMD-00-21.3.1 .
Page 45 GAO-16-479 Intellectual Property

USPTO is taking steps to strengthen monitoring of examiners’ work; 

however, these efforts may not provide USPTO with adequate data to 

identify and address shortcomings with examiners’ prior art searches 

specific to individual technology centers and monitor search 

thoroughness over time. USPTO uses two methods to review 

examinations, which may help the agency monitor the effects of the 

challenges described above as well as the thoroughness of searches. 

First, USPTO’s Office of Patent Quality Assurance (OPQA) conducts 

audits of a random sample of office actions, and some of these audits will 

review examiners’ prior art searches. Prior to recent changes in their 

audits, reviewers in OPQA performed about 400 audits per year that 

focused on assessments of examiners’ prior art searches. Second, 

supervisory patent examiners review examiners’ work products as part of 

each examiner’s annual performance appraisal. USPTO supervisory 

patent examiners are required to review at least four office actions of 

each of their primary examiners per year—with additional reviews for 

junior examiners—and to evaluate the thoroughness of examiners’ prior 

art searches during these reviews.

However, in recent years, the number and consistency of OPQA staff and 

supervisory patent examiners’ reviews of examiners’ prior art searches 

have not been sufficient to examine trends in the thoroughness of prior art 

searches at the technology center or art unit level. Using the 400 audits 

OPQA performed annually, USPTO officials said that they could perform 

statistically valid assessments of prior art searches only for the examiner 

corps as a whole. Further, because the supervisory patent examiners’

reviews have not been conducted or documented in a consistent manner, 

USPTO could not examine trends at the technology center or art unit level 

by combining supervisory reviews with OPQA’s reviews. Specifically, past 

OPQA audits have evaluated prior art searches by, for example, 

considering if the examiner used reasonable search terms and synonyms, 

and may have included an independent search to discover prior art 

missed by the examiner. Auditors in OPQA documented the results of 

their reviews, and the office used them to evaluate the examiner corps 

from year to year. In comparison, according to a USPTO training 

document, supervisory patent examiners should check that a thorough 

search was conducted, but should also adjust their reviews based on an 

examiner’s skills, abilities, and performance history. Supervisory 

examiners do not need to perform an independent search for prior art. 

USPTO’s Monitoring of 

Prior Art Searches Cannot 

Be Used to Identify 

Technology-Specific 

Trends or Measure 

Changes in Search Quality 

over Time
Page 46 GAO-16-479 Intellectual Property

Supervisory examiners may document errors found during their reviews 

to inform individual employee performance assessments but are 

otherwise not required to record their reviews.74 Because USPTO does 

not require supervisory examiners to document these reviews in a 

consistent form, the agency cannot analyze the data to examine issues 

with prior art search quality in specific technology centers and art units.

Early in fiscal year 2016, USPTO took two steps that if finalized could 

enable monitoring of prior art search trends at the technology center or art 

unit level, according to agency officials. Beginning in November 2015, 

OPQA made changes to its review processes that along with an increase 

in staff, should allow the office to perform about 12,000 audits in 2016, 

according to OPQA officials. In addition, in 2015, USPTO drafted a 

master review form that could standardize OPQA and supervisory 

examiner reviews with a single, consistent approach and documentation. 

OPQA began using a draft version of the form in November 2015, and 

USPTO began to pilot the form with some supervisory patent examiners 

in 2016. As of March 2016, USPTO had not made a decision about the 

final content of the review form, when supervisory patent examiners might 

begin to use the new form, or how the data from OPQA and supervisory 

examiner reviews would be used to assess examiners’ prior art 

searches.75

Furthermore, despite potential improvements in how OPQA and 

supervisory patent examiners conduct and document reviews of 

examiners’ prior art searches, USPTO’s ability to use these data to 

monitor prior art searches may be limited because USPTO (1) does not 

have a clear definition of what constitutes a thorough prior art search, (2) 

may not collect sufficient information to assess examiners’ search 

strategies or the sources of prior art they consider, and (3) has not 

established goals and indicators for improving prior art searches.

 74In 2015, the Department of Commerce Office of Inspector General recommended, 

among other things, that USPTO collect information on these reviews and errors found to 

improve the agency’s ability to identify quality trends. Department of Commerce, Office of 

Inspector General, U.S. Patent and Trademark Office: USPTO Needs to Strengthen 

Patent Quality Assurance Practices, OIG-15-026-A (Washington, D.C., Apr. 10, 2015).

75USPTO officials cited concerns about the potential workload the new form might place 

on supervisory patent examiners as a key factor in determining how supervisors might use 

this form.
Page 47 GAO-16-479 Intellectual Property

• USPTO requires examiners to perform a thorough prior art search and 

record their search results or search history, but the agency has not 

clearly defined what constitutes a thorough prior art search. USPTO’s 

manual for patent examiners requires examiners to conduct a 

thorough prior art search by identifying the field of search, the 

appropriate tools, and a search strategy. It also requires examiners to 

consider U.S. patents, foreign patents, and nonpatent literature, 

unless they can justify with reasonable certainty that no more 

pertinent references can be found. USPTO has no single method for 

assessing the thoroughness of prior art searches because, as 

supervisory patent examiners told us and our survey results show, 

examiners’ search strategies and the sources they use will differ 

based on the technology (see app. III). Examiners have access to 

guidance and training on classification-based searches, but USPTO 

has not documented technology-specific guidance—such as by 

technology center or art unit—on what constitutes a thorough prior art 

search. Specifically, in its definitions of CPC classes, USPTO 

provides suggested search areas of related classes, but these 

suggestions do not specify what USPTO would consider a thorough 

prior art search for each class or describe the sources of prior art the 

examiner should consider.

• As of March 2016, USPTO’s draft of the new master review form did 

not require OPQA or supervisory examiners to evaluate the

thoroughness of an examiner’s search. Instead, the form asks if the 

examiner (1) searched for prior art associated with the inventor’s 

name, (2) searched for prior art using classification results for the 

application, and (3) recorded his or her search strategy. If a reviewer 

finds that the examiner should have made a rejection but did not, the 

form asks the reviewer to identify the source of prior art needed for 

the missed rejection. The March 2016 draft of the form does not 

include, as an October 2015 draft we reviewed did, questions 

assessing if search queries were likely to result in identification of 

relevant prior art. The March 2016 draft of the form also does not 

include questions that address whether the examiner searched 

foreign patent literature and U.S. and foreign nonpatent literature, as 

is required by the agency’s manual for patent examiners.

• While USPTO has taken steps to improve the quality of prior art 

searches, the agency has not yet established performance goals and 

indicators for improving prior art searches. One of the stated 

objectives in USPTO’s 2014-2018 Strategic Plan is to enhance 

accurate and consistent results in examination quality, and USPTO’s 

Enhanced Patent Quality Initiative also aims to achieve excellence in 
Page 48 GAO-16-479 Intellectual Property

measuring patent quality. However, USPTO’s goals for patent quality 

in the strategic plan and for the quality initiative do not currently 

include goals or indicators assessing the thoroughness of prior art 

searches. Although USPTO is not required to establish goals or 

indicators for improving prior art searches, we have previously 

reported that establishing program goals and associated indicators 

constitutes a leading practice for planning within federal agencies.76

Among other things, the Government Performance and Results Act

Modernization Act of 2010 requires that agencies (1) establish 

objective, quantifiable, and measurable performance goals and (2) 

establish performance indicators to measure progress toward each 

performance goal.

The limitations to USPTO’s ability to monitor examiners’ prior art 

searches at a technology center or art unit level may hinder the agency’s 

ability to identify and address issues in the thoroughness of examiners’

prior art searches and quality of the agency’s examinations. Federal 

standards for internal control state that agencies’ monitoring of their 

internal controls should assess the quality of performance over time, and 

that internal controls should generally be designed to assure that ongoing 

monitoring occurs in the course of normal operations.77 Without 

consistently collecting the information needed to assess the thoroughness 

of prior art searches and monitoring at a technology center or art unit 

level, the agency cannot identify and address issues that are more 

prevalent in certain technology centers or art areas, such as variations in 

the extent to which examiners in certain areas search for foreign patents 

or nonpatent literature. Similarly, without greater clarity on what 

constitutes a thorough prior art search in different technologies, it will be 

difficult for USPTO to assess the adequacy of examiners’ searches, and 

examiners may vary in their thoroughness. Moreover, without goals and 

indicators for assessing the thoroughness of prior art searches, USPTO 

cannot reliably assess the thoroughness of its searches or improvement 

in searches over time.

 76GAO-12-77 and GAO-13-797.

77GAO/AIMD-00-21.3.1. 
Page 49 GAO-16-479 Intellectual Property

USPTO plans to evaluate changes to the agency’s system for 

determining the number of applications a patent examiner is expected to 

review within a specified period of time; however, the agency has not 

identified plans to assess the amount of time examiners in different 

technologies need to perform thorough prior art searches. According to 

USPTO officials we interviewed and a document we reviewed, the time 

allotted for each individual technology was determined when the 

examiner performance and production system was first created in the 

1970s or when subsequent art units were added. USPTO adjusted the 

time allotted to examiners between fiscal years 2010 and 2012 and gave 

all patent examiners a total of 2.5 additional hours per application. 

However, according to USPTO officials, the agency did not evaluate art 

unit or technology-specific factors prior to making this change. USPTO 

also adjusted the time allotted in April 2016, when approximately 1,000 

examiners received an additional 2.7 hours for examinations of certain 

technologies to address concerns related to the transition to the CPC 

system. According to technical comments USPTO provided on a draft of 

this report, these changes were based on an initial investigation of time 

needed to perform a thorough prior art search in these technologies.

The time pressures created by USPTO’s system for determining allotted 

examination time have important implications for examiners’ ability to 

conduct thorough prior art searches. As noted above, about two-thirds of 

examiners responding to our survey reported having insufficient time to 

complete a thorough prior art search given their typical workload. 

Furthermore, examiners in different technologies may need different 

amounts of time to conduct thorough prior art searches. For example, we 

estimate based on our survey that a higher percentage of examiners in 

the Mechanical Engineering technology center find that they have much 

less time than needed, compared to examiners in other technology 

centers (see app. III).

Supervisory patent examiners we interviewed told us that prior art 

searching is the most time-consuming aspect of patent examination. 

Based on our survey, an estimated 80 percent of examiners spent, on 

average, from 6 to 20 hours initially examining patent applications and 

USPTO Intends to 

Evaluate the Time Allotted 

for Patent Examinations 

but Has Not Planned to 

Assess the Time Needed 

for Thorough Prior Art 

Searches for Different 

Technologies
Page 50 GAO-16-479 Intellectual Property

completing a first office action.78 For the prior art search portion of these 

efforts, we estimate that examiners spent an average of 8 hours per 

application during the past quarter.79 If examiners feel pressure to 

complete their examinations quickly rather than thoroughly, it could have 

important implications for their prior art search efforts. In fact, as shown in 

figure 7, less than half of the examiners—an estimated 46 percent—were 

moderately or very confident that they found the most relevant prior art 

during the time allotted. However, when they included their additional 

voluntary overtime, the number of examiners who were moderately or 

very confident increased to an estimated 69 percent of examiners. Even 

with the inclusion of their overtime, though, an estimated 23 percent of 

examiners remained not at all confident to somewhat confident.

 78Specifically, we asked examiners about the amount of time spent, on average, for each 

First Action on the Merits in the past quarter. A First Action on the Merits is typically the 

first substantive examination of the application. We estimate all examiners’ responses to 

be as follows: 0 to 5 hours, 2 percent; 6 to 10 hours, 24 percent; 11 to 15 hours, 31 

percent; 16 to 20 hours, 24 percent; more than 20 hours, 15 percent; don’t know, 2 

percent; and no response, 1 percent. Survey results reported here do not sum to the result 

provided above because of rounding.

79The estimated amount of time examiners spent on prior art searching varies by 

technology center, from a low of about 5 hours per application in one technology center to 

a high of about 11 hours in another technology center, as detailed in app. III.
Page 51 GAO-16-479 Intellectual Property

Figure 7: Patent Examiners’ Confidence That They Find the Most Relevant Prior Art 

during Examination

Note: See the related e-supplement, GAO-16-478SP, for the original survey language and more 

detailed results.

Federal standards for internal control specify that agencies should assess 

the risks the agency faces, including identifying relevant risks associated 

with achieving the agency’s objectives, assessing a risk’s significance 

and likelihood of its occurrence, and deciding what actions should be 

taken to manage the risk. In addition, these standards note that an 

agency’s operational success requires providing personnel with the right 

incentives for the job. In USPTO’s 2014-2018 Strategic Plan, the agency 

indicated its intent to evaluate changes to the its system for evaluating if 

examiners are completing office actions in the time allotted and to make 

additional modifications as needed. In November 2015, USPTO’s 

Commissioner for Patents affirmed the agency’s intent to examine this 

system and told us that he had committed to the examiners’ union to do 

so. USPTO has historically faced a backlog of applications, and changing 

the time allotted may make it more difficult for the agency to address this 

backlog. However, changing the time allotted to examiners may address 

the challenge of time pressures on examiners that we discuss above. As 
Page 52 GAO-16-479 Intellectual Property

of May 2016, USPTO had not clarified the extent to which this evaluation 

will specifically consider the time needed for a thorough prior art search 

for different technologies. Without evaluating the technology-specific time 

pressures examiners face in their efforts to identify prior art in patent 

examination, USPTO does not have assurance that the time allotted to 

examiners reflects the evolving complexity of different technologies and 

their associated prior art.

USPTO has taken several actions related to the challenge of ensuring 

that the agency has an examiner workforce with the technical 

competence—backgrounds, knowledge, or skills—needed for identifying 

relevant prior art; however, USPTO does not have a process to assess 

and measure each technology center’s progress toward closing any gaps 

in examiners’ technical knowledge and skills. Effective management of an 

organization’s workforce—its human capital—is essential; in particular, 

identifying critical occupations, skills, and competencies and analyzing 

workforce gaps are leading principles in workforce planning, as we and 

the Office of Personnel Management have previously identified.80 Further, 

as described in the federal standards for internal control, all personnel 

need to possess and maintain a level of competence that allows them to 

accomplish their assigned duties.81 Management needs to ensure that 

skill needs are continually assessed and that the organization is able to 

obtain a workforce with the skills required to achieve organizational goals. 

Management also needs to provide employees the right training to 

develop and retain skill levels to meet changing organizational needs.

Accordingly, USPTO developed a human capital strategy, its 2015-2018 

People Plan, which replaced its 2011-2015 Strategic Human Capital Plan. 

The agency also identified the role of patent examiner as a missioncritical occupation and identified the competencies needed for this 

occupation. These competencies include technical work 

experience/education and technical competence, which we refer to 

 80GAO, Workforce Planning: Interior, EPA, and the Forest Service Should Strengthen 

Linkages to Their Strategic Plans and Improve Evaluation, GAO-10-413 (Washington, 

D.C.: Mar. 31, 2010).

81GAO/AIMD-00-21.3.1.

USPTO Has Taken Actions 

to Develop and Maintain 

Examiners’ Technical 

Competence but Has Not 

Assessed Competence 

Gaps in Each Technology 

Center
Page 53 GAO-16-479 Intellectual Property

collectively as technical competence.82 USPTO defines technical 

competence as the ability to analyze and interpret written technical 

materials, rules, regulations, instructions, and reports.83 According to 

USPTO officials, specific technical competencies vary depending on the 

technology examined by each technology center and art unit. USPTO has 

further documented specific technical competencies for examiners in a 

series of “job analysis worksheets,” which describe the knowledge, skills, 

and abilities needed for examiner positions.84 Agency officials said these 

were based on the hiring needs derived from a hiring model created by 

senior management. For example, the job analysis worksheet for 

electrical engineering patent examiners considers their education, 

training, and experience in researching, analyzing, and applying scientific 

principles in specified technical areas, such as image analysis, power 

systems, and computer architecture.

To develop and maintain examiners’ technical competence, USPTO 

offers three programs, all of which are voluntary for examiners (see table 

5).85 In addition, USPTO officials said that technology centers provide 

technology-specific training through paid vendors, periodic internal 

 82USPTO refers to “technical work experience/education” for entry-level patent examiners 

at the GS-5/7/9 levels and to both “technical work experience/education” and “technical 

competence” for experienced examiners at the GS-11 level. The other competencies 

USPTO identified for examiners at the GS-5/7/9 levels are oral communication, written 

communication, problem-solving, self-management, conflict management, and 

influencing/negotiating. Competencies for examiners at the GS-11 level include those 

outlined above and knowledge of legal matters associated with patent examination 

practices, processes, and procedures.

83This does not include foreign-language capabilities. Because they are not a requirement 

of the patent examining position, according to agency officials, USPTO has not assessed 

the need for examiners with foreign-language capabilities, does not track the foreignlanguage capabilities of its examiners, and does not consider the foreign-language 

capabilities of new hires when making technology center or art unit assignments.

84USPTO officials told us that the agency has prepared these worksheets for the following 

examiner disciplines: biological sciences; chemistry; computer science; design; 

pharmacology; physics; textile technology; and biomedical, chemical, civil, computer, 

electrical, industrial, mechanical, and petroleum engineering. Officials said that these 

worksheets encompass all examiner positions in all USPTO technology centers.

85In addition, the Office of Patent Training offers voluntary training opportunities through 

the agency’s “Refresher” and “Masters” level training programs. We reviewed the lists of 

offerings under these programs and found that they generally did not pertain to technical 

competence. 
Page 54 GAO-16-479 Intellectual Property

meetings to discuss technical and quality issues, and technology fairs. 

USPTO does not require examiners to complete a minimum amount of 

ongoing technical training; however, based on information from USPTO 

officials, 74 percent of examiners participated in at least one of the 153 

technical training events held through the agency’s Patent Examiner 

Technical Training Program in fiscal year 2015, with an average of 6.7 

training hours per participant. The extent to which examiners participated 

in this technical training program varied by technology center, from a low 

of 48 percent to a high of 96 percent in fiscal year 2015, as well as by art 

unit, according to information from USPTO officials.86

Table 5: Voluntary Technical Training Programs for U.S. Patent and Trademark 

Office (USPTO) Examiners and Their Fiscal Year 2015 Utilization

Program Description 

Fiscal year 2015 

participants

Patent Examiner 

Technical Training 

Program 

External technology experts provide 

seminars and workshops, with an 

annual average of 6.7 training hours 

per participant

6,265 examiners 

participated in at 

least one technical 

training event 

Site Experience 

Education Program

USPTO funds travel costs to send 

examiners to technology sites and 

allows examiners and innovators to 

interact while viewing technology 

firsthand 737 examiners

Non-Duty Hours 

Technical Training 

Program

USPTO reimburses examiners for 

college technical courses, outside of 

duty hours, that directly relate to 

technology examined by USPTO 65 examiners

Source: USPTO officials. | GAO-16-479

While these are important steps that can help develop and maintain 

examiners’ technical competence, USPTO officials told us that the 

agency has not conducted an analysis to identify any gaps in examiners’

competence, either for the agency as a whole or for each technology 

center. USPTO strategic or human capital plans since 2007 have called 

for measuring the agency’s performance on closing competency/skill 

gaps for mission-critical occupations.87 USPTO officials told us that the 

 86We did not assess the methodology used by USPTO to determine participation.

87Both the 2007-2012 Strategic Plan and the 2011-2015 Strategic Human Capital Plan 

included this as a performance measure.
Page 55 GAO-16-479 Intellectual Property

technology centers use performance appraisal plans to individually 

assess examiners’ skills and competency gaps. However, doing so does 

not address the question of whether broader competency gaps exist at 

the technology center level or how any gaps can be addressed.

Further, using USPTO’s job analysis worksheets in hiring does not 

eliminate the need for ongoing workforce assessment. In particular, as 

technologies change, the knowledge and skills required of examiners may 

evolve accordingly, and examiners may move from one technology center 

or art unit to another during their careers. According to USPTO officials, 

there are also times when patent examiners are ultimately assigned to a 

different technology center upon completion of their initial training than 

the assignment given to them when they were hired. In addition, USPTO 

has reported that examiner attrition has increased over the past 5 years.88

Historically, attrition has been highest amongst examiners at USPTO for 

less than 5 years. As we found in 2007, attrition of less experienced 

examiners is a significant loss for the agency, in part because examiners 

require 4 to 6 years of on-the-job experience before they become fully 

proficient in conducting patent application reviews. When these staff 

leave USPTO, the agency loses as much as 5 years of training 

investment in them, and continuing turnover of many new patent 

examiners makes the overall workforce less experienced.89

Moreover, USPTO’s 2015-2018 People Plan notes that that the agency’s 

ability to retain experienced and high-performing employees may prove 

difficult with continued improvements to the economy.

90 Because of these 

factors, the technical knowledge and skills needed in a technology center 

 88According to USPTO’s performance and accountability reports, the attrition rate for 

patent examiners decreased from 7.83 percent in fiscal year 2008 to 2.96 percent in fiscal 

year 2011. Since fiscal year 2011, attrition has generally been increasing annually, with an 

attrition rate of 4.32 percent in fiscal year 2015.

89GAO-07-1102. 

90Similarly, as we reported in 2005, USPTO officials and examiners previously told us that 

the economy has a greater effect on recruitment and retention than any actions the 

agency may take. Both agency officials and examiners told us that when the economy 

picks up, more examiners tend to leave USPTO and fewer qualified candidates are 

attracted to the agency. On the other hand, when there is a downturn in the economy, 

USPTO’s ability to attract and retain qualified examiners increases because of perceived 

job security and competitive pay. See GAO-05-720. 
Page 56 GAO-16-479 Intellectual Property

may differ from the knowledge and skills of the individual examiners 

assigned to that center. Without periodically evaluating gaps between the 

technical competence of examiners and the knowledge and skills needed 

in each technology center, USPTO cannot ensure that it has appropriate 

strategies, such as training or other efforts, to close any gaps. Moreover, 

as specified in the federal standards for internal control, ongoing internal 

control monitoring should assess the quality of performance over time. 

The agency’s 2015-2018 People Plan, however, does not include 

measures for monitoring progress in closing gaps. Without monitoring and 

evaluating progress toward closing any identified gaps in technical 

competence, USPTO may not have reasonable assurance that examiners 

in all technology centers have the skills and knowledge to identify relevant 

prior art during patent examination.

Examiners face a number of challenges in their efforts to search for prior 

art, including the large volume of prior art from multiple sources to 

consider, unclear patent applications, difficulties identifying or accessing 

relevant nonpatent literature and prior art in foreign languages, and limits 

on the time available to search for relevant prior art, among others. Some 

of these challenges may reinforce one another or affect examiners in 

some technology centers more than others. USPTO, recognizing the 

importance of issuing quality patents, is taking a number of steps to 

improve patent quality, as we discuss in our patent quality report

(GAO-16-490). USPTO is also is taking a number of steps to help 

address challenges to completing a thorough prior art search.

However, while USPTO’s steps to address prior art search challenges are 

promising, opportunities exist for USPTO to address limitations that may 

hinder the effectiveness of some of its efforts. For example, USPTO has

experienced some challenges working with EPO to adopt and 

consistently implement a new patent classification system. Because 

USPTO and EPO have not identified a target level of consistency or a 

plan to monitor consistency between USPTO contractor and EPO 

classifications—important aspects of interagency collaboration—the 

potential benefits of adopting the new classification system may be 

reduced as inconsistently classified patents make it more difficult for 

examiners to identify relevant U.S. and foreign patent literature.

Similarly, USPTO is undertaking a major investment in information 

technology tools for examiners to manage their work and search for prior 

art. The new search system USPTO plans has the potential to integrate 

additional sources of nonpatent literature, an approach that EPO and JPO 

Conclusions
Page 57 GAO-16-479 Intellectual Property

have taken and cited as helping examiners consider a wide array of 

nonpatent literature sources. However, USPTO has not developed and 

documented a strategy to identify and assess the optimal means of 

incorporating new sources of art into these tools. Such a strategy would 

be consistent with federal standards for internal control. Developing and 

periodically updating such a strategy could help USPTO take full 

advantage of its investment in its new information technology tools to 

address some of the challenges examiners face in identifying relevant 

prior art.

USPTO is also taking steps to augment the number and consistency with 

which reviews of examiners’ prior art searches are conducted and 

documented that could help to address a weakness of its past efforts—an 

insufficient number of consistently performed reviews to enable USPTO 

to identify issues with examiners’ prior art searches within individual 

technology centers. However, USPTO still faces limitations in its ability to 

use the data to monitor the thoroughness of examiners’ prior art 

searches. First, USPTO has not clearly defined what constitutes a 

thorough prior art search for different technology centers, which is 

important given differences in the appropriate search strategies and prior 

art resources across technology areas. Second, USPTO’s draft review 

form, as of March 2016, does not include questions that are important to 

assess the thoroughness of examiners’ prior art searches, such as the 

extent to which an examiner searched for foreign patent and nonpatent 

literature. These are aspects of prior art search that are generally 

required by USPTO’s manual for patent examiners but that our survey 

results showed many examiners may not perform. Third, the agency has 

not established goals or indicators covering prior art search performance. 

Monitoring the thoroughness of examiners’ prior art searches at a level of 

rigor and consistency to assess trends within individual technology 

centers would better enable USPTO to identify issues that vary by 

technology center and develop appropriate responses as called for by 

federal internal control standards. Such monitoring would also allow 

USPTO to have greater assurance that examiners are searching all 

relevant sources of prior art, including foreign patents and nonpatent 

literature. Furthermore, establishing goals and indicators for prior art 

search would help USPTO reliably assess the thoroughness of its 

searches and improvements over time.

Additionally, while monitoring the thoroughness of examiners’ prior art 

searches is important, it is essential that examiners have sufficient time to 

perform high-quality work. USPTO expects examiners to examine an 

application within a specific amount of time. This means that examiners 
Page 58 GAO-16-479 Intellectual Property

may need to make trade-offs among the amount and types of prior art 

sources they consider because of the limited time available to perform 

their searches. However, as we also note in our patent quality report 

(GAO-16-490), USPTO has not recently assessed the time needed for 

examinations in different technologies. Without this information, the 

agency does not know if it is allotting examiners appropriate amounts of 

time to complete their work. Based on our survey, most examiners find 

that they do not have sufficient time for thorough prior art searches, and 

many were not confident in their ability to identify the most relevant prior 

art without working voluntary overtime. USPTO’s planned effort to 

evaluate the agency’s system for determining the time allotted for 

examination could help the agency better understand the time pressures 

examiners face; however, USPTO has not yet clarified the extent to which 

the evaluation will look at any differences among art units and 

technologies in the time needed for thorough prior art searches. Given the 

importance of thorough prior art searches to USPTO’s patent quality 

objectives, specifically assessing the time needed for a thorough prior 

search for different technologies would provide USPTO with greater 

assurance that it will address the risks posed by the time pressures 

examiners experience, consistent with federal internal control standards.

Finally, examiners’ technical competence is an important factor in 

understanding patent applications and locating relevant prior art in the 

time allotted. USPTO has taken steps to identify the technical 

competencies of various examiner positions and to offer voluntary 

technical training opportunities to examiners. However, the agency has 

not conducted an overall analysis of the technical competence of its 

examiners to identify potential competency gaps for each technology 

center. Periodically conducting such analyses to reflect evolving 

technologies and workforce changes, as called for by federal standards 

for internal control, would give USPTO greater assurance that examiners 

in all technology centers have the technical skills and knowledge to 

identify the most relevant prior art during patent examination. 

Furthermore, analyzing the results of such analyses would help the 

agency define and prioritize its strategies for closing any competency 

gaps, such as through training or other efforts. Lastly, these steps, in 

conjunction with developing measures to monitor progress toward closing 

any gaps, would help USPTO address the effects of evolving 

technologies and workforce changes over time.
Page 59 GAO-16-479 Intellectual Property

To enhance USPTO’s ability to identify relevant prior art, we recommend 

that the Secretary of Commerce direct the Director of USPTO to take the 

following seven actions as the agency continues to implement its efforts 

to improve prior art searches.

To ensure that USPTO’s collaborative efforts on classification help 

examiners find relevant prior art, USPTO should work with EPO to identify 

a target level of consistency of Cooperative Patent Classification 

decisions between USPTO and EPO and develop a plan to monitor 

consistency to achieve the target.

To ensure that USPTO is able to take full advantage of its investment in 

new information technology tools and capabilities, USPTO should 

develop and periodically update a documented strategy to identify key 

sources of nonpatent literature for individual technology centers and to 

assess the optimal means of providing access to these sources, such as 

including them in USPTO’s search system.

To improve its monitoring of prior art searches and provide USPTO the 

ability to examine and address trends in prior art search quality at the 

technology center level, USPTO should take the following three actions:

• Develop written guidance on what constitutes a thorough prior art 

search within each technology field (i.e., mechanical, chemical, 

electrical), technology center, art area, or art unit, as appropriate, and 

establish goals and indicators for improving prior art searches.

• Ensure that sufficient information is collected in reviews of prior art 

searches to assess the quality of searches at the technology center 

level, including how often examiners search for U.S. patents, foreign 

patents, and nonpatent literature.

• Use the audits and supervisory reviews to monitor the thoroughness 

of examiners’ prior art searches and improvements over time.

To ensure that examiners have sufficient time to conduct a thorough prior 

art search, USPTO should, in conjunction with implementing the 

recommendation from our patent quality report to analyze the time 

examiners need to perform a thorough examination, specifically assess 

the time examiners need to conduct a thorough prior art search for 

different technologies.

To ensure that examiners have the technical competence needed to 

complete thorough prior art searches, USPTO should assess whether the 

Recommendations for 

Executive Action
Page 60 GAO-16-479 Intellectual Property

technical competencies of examiners in each technology center match 

those necessary; develop strategies to address any gaps identified, such 

as a technical training strategy; and establish measures to monitor 

progress toward closing any gaps.

We provided a draft of this report to the Department of Commerce for its 

review and comment. In the Department of Commerce’s written 

comments, reproduced in appendix IV, the Department concurred with 

our recommendations. USPTO provided additional technical comments 

that we incorporated, as appropriate. We also provided a draft of this 

report to EPO and JPO for their views and comment. EPO indicated it 

had no comments on the draft, and JPO provided technical comments 

that we incorporated, as appropriate. 

In its written comments, the Department concurred with our 

recommendation to work with EPO to identify a target level of consistency 

for CPC decisions and develop a plan to monitor consistency. The 

Department noted ongoing efforts to work with EPO and other patent 

offices to develop objective metrics to measure the level of consistency. If 

finalized, such metrics and a plan to monitor consistency could allow 

USPTO to more fully benefit from adopting the new CPC system.

In regard to our recommendation to develop and periodically update a 

strategy related to key sources of nonpatent literature, the Department 

concurred and stated that USPTO’s STIC analyzes nonpatent literature 

sources used by examiners and assesses them for incorporation into 

USPTO’s search system. While we did not receive evidence of such 

assessments during our review, they may be a useful step toward a 

strategy to periodically assess the optimal means of providing examiners 

access to key nonpatent literature sources and to ensure the 

effectiveness of USPTO’s planned Patents End-to-End search system. 

In regard to our recommendation to develop written guidance on what 

constitutes a thorough prior art search and establish goals and indicators 

for improving prior art searches, the Department concurred and said 

USPTO would develop technology-based search training guidance and 

establish enhanced goals and indicators for improving prior art searches. 

Specifically in regard to developing guidance on what constitutes 

thorough prior art searches, USPTO’s technical comments noted that our 

report did not recognize that USPTO’s definitions of CPC classification 

areas include suggestions for related classes for search. We adjusted our 

report to acknowledge the information USPTO provides on these 

Agency Comments 

and Our Evaluation
Page 61 GAO-16-479 Intellectual Property

suggested search areas. However, these suggested search areas do not 

fully establish what constitutes a thorough prior art search for each CPC 

class, and they do not provide guidance on examiners’ searches for 

nonpatent literature. 

In regard to our recommendations to ensure that reviews of examiners’ 

prior art searches collect sufficient information to assess search quality at 

the technology center level and use these reviews to monitor examiners’ 

prior art searches over time, the Department concurred. The Department 

said it would ensure adequate search data are collected to assess the 

quality of searches at the technology center level, and would investigate 

using audits and reviews to monitor the thoroughness of examiners’ prior 

art searches over time. In its written comments, the Department noted 

that a great deal of information on examiners’ prior art searches is 

potentially available from examiners’ records and reviews by primary 

examiners, supervisors, OPQA, and others. As we describe in our report, 

these reviews have not been carried out or recorded in a consistent 

manner in the past. Therefore, as a whole, these data do not provide 

USPTO with reliable information to assess trends in prior art search 

quality at the technology center level. USPTO’s recent effort to improve 

the consistency of these reviews is an important step to address this 

issue. However, additional steps are needed to ensure that USPTO’s 

reviews of examiners’ prior art searches collect information on whether, 

for example, examiners are searching patent literature, foreign patents, 

and nonpatent literature, as is required by the manual for patent 

examiners. Without collecting such information at the technology center 

level, USPTO may be challenged to use its reviews to monitor the 

thoroughness of examiners’ prior art searches and improvements over 

time.

In regard to our recommendation on assessing the time examiners need 

to conduct a thorough prior art search for different technologies, the 

Department concurred and stated that USPTO intends to further

investigate the time needed. The Department also mentioned the April 

2016 changes to the time allotted for approximately 1,000 examiners, 

noting that they were based on an initial evaluation of the time examiners 

need to perform a thorough prior art search. This initial evaluation 

stemmed from feedback from examiners during the transition to CPC. We 

did not receive details of this evaluation during our review, but recognize 

that this action may have provided the agency with important information 

about the time needed for prior art search in some technologies. We 

continue to believe that the agency should review the time allotted for all 
Page 62 GAO-16-479 Intellectual Property

technologies, and not only those identified through feedback related to 

CPC. 

Finally, the Department concurred with our recommendation to assess 

whether the technical competence of examiners in each technology 

center matches those necessary and develop strategies to address gaps 

if identified.

As agreed with your office, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies to the appropriate 

congressional committees, the Secretary of Commerce, the Director of 

the USPTO, the Commissioner for Patents, and other interested parties. 

In addition, the report will be available at no charge on the GAO website 

at http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-3841 or neumannj@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made key contributions to this 

report are listed in appendix V.

Sincerely yours,

John Neumann

Director, Natural Resources and Environment
Appendix I: Objectives, Scope, and 

Methodology

Page 63 GAO-16-479 Intellectual Property

This report (1) describes the challenges examiners face in identifying 

relevant prior art, (2) describes how selected foreign patent offices have 

addressed challenges in identifying relevant prior art, and (3) assesses 

the extent to which the U.S. Patent and Trademark Office (USPTO) has 

taken steps to address any challenges in identifying relevant prior art. Our 

work for this report was coordinated with our report on patent quality.1

To describe the challenges examiners face in identifying relevant prior art, 

we reviewed relevant laws and USPTO documents and interviewed 

USPTO officials; a group of four supervisory patent examiners; officials of 

the examiners’ union, the Patent Office Professional Association; and a 

group of five examiners serving as union representatives. We also 

conducted semistructured interviews with subject matter experts (experts) 

active in the intellectual property field and knowledgeable about the 

subject of prior art. We selected and interviewed 18 experts to obtain 

views of different stakeholder communities, including

• four academic experts on the basis of a literature search we 

conducted for articles related to USPTO and prior art;

• four patent holders from the following technology fields: chemical 

technologies, electrical technologies, and mechanical technologies, 

based on data from USPTO that identified the companies receiving 

patents in these fields from 2010 through 2014;2

• five attorneys based on their leadership (e.g., as chairs or co-chairs) 

of patent-related committees or sections of various bar associations;3

 1

GAO, Intellectual Property: Patent Office Should Define Quality, Reassess Incentives, 

and Improve Clarity, GAO-16-490 (Washington, D.C.: June 30, 2016).

2

We interviewed one patent holder each for the chemical and electrical technology fields. 

Within the mechanical technology field, the Transportation, Construction, Electronic 

Commerce, Agriculture, National Security and License and Review technology center 

includes art units—subunits of a technology center—focused on electronic commerce and 

business methods (collectively referred to as business methods). In light of recent 

legislation and court decisions related to business methods, we separated the business 

methods art units throughout our review. Therefore, we interviewed two patent holders for 

the mechanical technology field, with one of those selected from the business methods art 

units.

3

We selected a sixth attorney but were unable to schedule an interview with the individual.

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 64 GAO-16-479 Intellectual Property

• representatives of three nongovernmental organizations we identified 

during background research that had activities or publications related 

to patents; and

• two patent data experts who were listed in a 2015 USPTO analysis 

evaluating search technologies.4

In addition, we conducted a web-based survey of a stratified random 

sample of 3,336 eligible USPTO patent examiners from across 8 of the 11 

technology-based subject matter groups (referred to as technology 

centers) into which USPTO examiners are divided. Fielded between 

August and November 2015, the survey was designed to collect 

information on challenges USPTO faces in finding relevant prior art and 

how USPTO might improve its prior art search capabilities.5 To identify 

our survey population, we obtained from USPTO a list of patent 

examiners as of May 2015. We excluded examiners from the following 3 

technology centers:

• We excluded the Designs technology center because these 

examiners work on design patents instead of utility patents; design 

patents are outside the scope of this engagement and have different 

statutory and administrative requirements than utility patents.

• We excluded examiners who perform “reexamination” work and not 

initial patent examinations.

• We excluded examiners in the patent training academy because 

these examiners are recent hires who are in a 12-month training 

program.

We also excluded examiners employed at USPTO for less than 1 year. 

We then defined nine strata by technology center, with one technology 

center separated into two strata, as described in table 6. Specifically, the 

Transportation, Construction, Electronic Commerce, Agriculture, National 

Security and License and Review technology center includes a diverse 

 4

We also selected two additional patent data experts using this source but were unable to 

schedule interviews with them. 

5

We administered the survey jointly with our review focused on patent quality issues. 

Therefore, the survey was also designed to examine USPTO’s approach to patent quality 

and how the agency might improve patent quality. For this review, see GAO-16-490.
Appendix I: Objectives, Scope, and 

Methodology

Page 65 GAO-16-479 Intellectual Property

set of technologies, including transportation, construction, agriculture, and 

business methods. In our review, we separated the art units—subunits of 

a technology center—focused on electronic commerce and business 

methods (collectively referred to as business methods) in light of recent 

legislation and court decisions related to business methods. This resulted 

in nine strata with a target survey population totaling 7,825 eligible 

examiners. From this list, we drew our stratified random sample of 3,336 

eligible USPTO patent examiners. We received responses from 2,669 

eligible examiners for an 80 percent response rate.

6

Table 6: Strata of United States Patent and Trademark Office (USPTO) Examiners 

Surveyed 

Name Technology center (TC) Technology fielda

Biotechnology and Organic 

Chemistry TC 1600 Chemical

Chemical and Materials Engineering TC 1700 Chemical

Computer Architecture, Software, 

and Information Security TC 2100 Electrical

Computer Networks, Multiplex 

Communication, Video Distribution, 

and Security TC 2400 Electrical

Communications TC 2600 Electrical

Semiconductors, Electrical and 

Optical Systems and Components TC 2800 Electrical

Transportation, Construction, 

Electronic Commerce, Agriculture, 

National Security and License and 

Reviewb

TC 3600 (business 

methods only) Mechanical

TC 3600 (not business

methods) Mechanical

Mechanical Engineering, 

Manufacturing, and Products TC 3700 Mechanical

Source: GAO review of USPTO information. | GAO-16-479

a

We defined these technologies according to how USPTO defined them in its United States Patent 

and Trademark Office Quality Survey, Summer 2015, OMB No. 0651-0057.

b

TC 3600 includes a diverse set of technologies, including transportation, construction, agriculture, 

and business methods. In our review, we separated the art units focused on electronic commerce and 

business methods (collectively referred to as business methods) in light of recent legislation and court 

decisions related to business methods.

 6

After the survey was fielded, we determined that 11 survey recipients were not eligible 

because they told us that they are no longer examiners. Those survey recipients are not 

included in the target population, sample, or respondent totals presented in this report.
Appendix I: Objectives, Scope, and 

Methodology

Page 66 GAO-16-479 Intellectual Property

Because we used a probability procedure based on random selections, 

our sample is only one of a large number of samples that we might have 

drawn. Since each sample could have provided different estimates, we 

quantified the sampling error and express our confidence in the precision 

of our particular sample’s results at a 95 percent confidence interval. This

is the interval that would contain the actual population value for 95 

percent of the samples we could have drawn. Within each stratum and 

overall, the confidence intervals for survey results for percentages are 

generally within +/- 5 percentage points. The only estimates for which the 

confidence intervals exceed 5 percentage points are certain results for 

the business methods stratum. In these instances, the confidence 

intervals are from 5 to 6 percentage points. In this report, our figures 

containing survey results show the upper and lower bounds for estimates 

at the 95 percent confidence interval. For other estimates in the report, 

we have not provided the upper and lower bounds in the text or tables; 

however, those details for all survey results are available in the esupplement related to this report.7

The quality of survey data can also be affected by nonsampling error, 

which includes, for example, variations in how respondents interpret 

questions, respondents’ willingness to offer accurate responses, 

nonresponse errors, and data collection and processing errors. To 

minimize nonsampling error, we took several steps in developing the 

survey and in collecting and analyzing survey data. Specifically, in 

developing the survey, we worked with our survey professionals, among 

other things, to draft questions that were clear and unbiased. We 

pretested the survey in person with five USPTO staff: three examiners 

who are also representatives to the examiners’ union, a supervisory 

patent examiner, and a quality assurance specialist. We used these 

pretests to check that the questions were clear and unambiguous, used 

correct terminology, requested information that could be feasibly 

obtained, and were comprehensive and unbiased. We also obtained 

comments on the survey from USPTO management and leadership from 

the examiners’ union. In addition, we obtained a quality review by a 

separate GAO survey methodologist. Based on these activities, we made 

changes to the survey before administering it. Further, using a web-based 

 7

GAO, Intellectual Property: Survey of U.S. Patent Examiners, an E-supplement to 

GAO-16-479 and GAO-16-490, GAO-16-478SP (Washington, D.C.: June 30, 2016).
Appendix I: Objectives, Scope, and 

Methodology

Page 67 GAO-16-479 Intellectual Property

survey provided several advantages, including allowing examiners to 

enter their responses into an electronic instrument that created an 

automatic record for each respondent. This eliminated the potential for 

errors that could have resulted if we had used a manual process to enter 

respondents’ data from paper surveys. In addition, to account for the 

complex sample design, we used survey software in our analyses to 

produce appropriate estimates and confidence intervals, and the 

programs we used to process and analyze the survey data were 

independently verified to ensure the accuracy of this work.

To minimize nonresponse error, we made a variety of contacts with the 

sample of examiners during the survey, including follow-up e-mails to 

encourage responses. In addition, from October 20 through 23, 2015, we 

attempted to follow up via telephone calls to all 1,102 examiners who had 

neither completed the survey nor told us that they were no longer 

examiners. We also analyzed nonresponse bias to (1) assess whether 

any factors were associated with examiners’ propensity to respond and 

(2) allow our analysis of respondents to properly reflect the sampling 

universe of eligible examiners. To adjust the sampling weight for potential 

nonresponse bias, we used standard weighting class adjustments based 

on the sampling strata and the examiners’ years of experience at USPTO. 

The weighted response rate was also 80 percent. In this report and in the 

related e-supplement, we present the survey results using the 

nonresponse adjusted weights, which are generalizable to the eligible 

population of examiners.

We analyzed the responses to the survey for all examiners, as well as 

responses by technology center and by the General Schedule (GS) level 

of the examiners. We selected three categories of GS levels—less than 

GS-13, GS-13, and greater than GS-13—because examiners at these 

levels have different responsibilities and authorities when examining 

patent applications. Specifically, examiners at the GS-14 level or above 

generally may grant a patent or reject a patent application without 

additional review, most examiners below the GS-14 level must have their 

actions reviewed by a more senior examiner, and some GS-13 examiners 

are in the process of becoming GS-14 examiners.

Question 3 in the survey asked examiners approximately how many 

hours they spent per application on prior art searches in the past quarter, 

when preparing the original First Action on the Merits, the means by 

which examiners initially notify applicants about the patentability of their 

inventions. Examiners provided a variety of open-ended responses. 
Appendix I: Objectives, Scope, and 

Methodology

Page 68 GAO-16-479 Intellectual Property

Some respondents chose to provide a range, such as “5 to 10,” while 

others provided a single number for hours spent per application on prior 

art searches. Some examiners also provided responses that did not 

clearly indicate the approximate hours per application spent on prior art 

searches in the past quarter; we excluded those responses. Where 

possible, we coded responses to reflect a range of numbers by assigning 

a low and a high value; when a single number was provided, we coded 

that number as both the low and the high value. A second analyst verified 

the initial analyst’s coding. We checked the sensitivity of results overall, 

within strata, and within GS level categories; results were not statistically 

different when using the low value, midpoint (average between low and 

high values), or high value of a range. As a result, we present results for 

question 3 based on the midpoint. The 95 percent confidence intervals for 

these results are within +/- 10 percent of the estimates themselves, 

except for the estimate for technology center 2100 (Computer 

Architecture, Software, and Information Security). The estimate for 

technology center 2100 has a 95 percent confidence interval of within +/-

18 percent of the estimate itself.

For some other survey questions, we also reviewed examiners’ openended responses on selected topics. We selected those topics based on 

our interviews with experts and USPTO officials as well as our analysis of 

closed-ended survey responses. We selected the questions for which 

examiners’ responses most frequently included keywords we identified for 

each topic. An analyst conducted a keyword search of all responses to 

the selected open-ended questions and coded responses containing the 

keywords. A second analyst verified the initial analyst’s coding. Our report 

provides some examples or summaries of examiners’ comments based 

on this review. Examiners’ responses to open-ended questions are not 

generalizable to other examiners. In addition, because we did not conduct 

a systematic review of all open-ended responses to our survey, we do not 

report the exact number of examiners who provided responses on the 

topics we reviewed.

In addition, we conducted statistical tests of association on the results of 

certain survey questions; all tests were independently verified to ensure 

their accuracy. All tests of association were carried out at the 5 percent 

level of significance and were Cochran-Mantel-Haenszel (CMH) Chisquare tests of general association. The testing was carried out in 

SUDAAN, which is statistical software appropriate for the analysis of 

survey data. The null hypothesis was that there is no association between 

the two tested variables. When the association between two variables, 
Appendix I: Objectives, Scope, and 

Methodology

Page 69 GAO-16-479 Intellectual Property

conditional on a third variable, is of interest, this is referred to as the 

stratum-adjusted CMH test. The test statistic is Wald Chi-Square. 

Specifically, among other survey questions, we performed CMH tests on 

examiners’ responses to questions on how often they search for foreign 

patent literature or foreign-language nonpatent literature, and how difficult 

it is to obtain relevant art from these searches. We also performed CMH 

tests on responses to questions on how often examiners search for 

certain types of prior art and whether they have sufficient time to 

complete a thorough prior art search. The types of art included were 

foreign patent literature, scientific articles or presentations, foreignlanguage nonpatent literature, and industry-related nonpatent literature. 

We performed these tests on responses overall as well as conditional on 

the examiners’ technology center.

To describe how selected foreign patent offices have addressed 

challenges in identifying relevant prior art, we first reviewed information 

identifying the world’s five largest patent offices. From the list of the other 

four offices similar in size to USPTO, we selected the European Patent 

Office (EPO) and the Japan Patent Office (JPO) based on comments 

observers made about the quality of their work. We conducted site visits 

with EPO and JPO, during which we interviewed officials from these 

offices and reviewed documents they provided. We also interviewed 

stakeholders knowledgeable about the offices’ practices, such as 

academics who study these offices. Because these individuals were not 

necessarily knowledgeable about USPTO practices, they are not included 

among the 18 experts discussed above.

To assess the extent to which USPTO has taken steps to address any 

challenges in identifying relevant prior art, we reviewed documents from 

the agency related to prior art search procedures and capabilities, 

including ongoing and planned initiatives related to information 

technology resources or capabilities, the examiner workforce and related 

human capital management efforts, training practices, and international 

cooperation. We also interviewed or obtained written responses to our 

questions from officials from USPTO’s Office of the Commissioner for 

Patents, Office of Patent Examination Policy, Office of Patent Information 

Management, Office of International Patent Cooperation, Office of Human 

Resources, and Office of Patent Quality Assurance, among others, and 

conducted interviews with technology center directors, supervisory patent 

examiners, and representatives from the USPTO examiners’ union. In 

addition, we reviewed the results of the survey of examiners. In assessing 

USPTO’s efforts, we identified criteria in the federal standards for internal 
Appendix I: Objectives, Scope, and 

Methodology

Page 70 GAO-16-479 Intellectual Property

control;8 the Government Performance and Results Act of 1993, as 

amended;9 USPTO’s Manual of Patent Examining Procedure; and 

USPTO’s strategic plan.

We conducted this performance audit from November 2014 to June 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

 8

GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 

(Washington, D.C.: November 1999). Revised standards became effective beginning with 

fiscal year 2016 after our work began. See Standards for Internal Control in the Federal 

Government, GAO-14-704G (Washington, D.C.: September 2014).

9

Pub. L. No. 103-62 (1993), amended by Government Performance and Results Act 

Modernization Act of 2010, Pub. L. No. 111-352 (2011). The Act sought to improve the 

effectiveness and accountability of federal programs by requiring federal agencies to set 

goals for program performance, measure results, and report on annual performance 

compared with the goals. Although the Act’s requirements apply at the agency level, we 

have previously reported that these practices can serve as leading practices within an 

organization, such as with individual programs or initiatives. See GAO, Environmental 

Justice: EPA Needs to Take Additional Actions to Help Ensure Effective Implementation, 

GAO-12-77 (Washington, D.C.: Oct. 6, 2011), and Great Lakes Restoration Initiative: 

Further Actions Would Result in More Useful Assessments and Help Address Factors 

That Limit Progress, GAO-13-797 (Washington, D.C.: Sept. 27, 2013).
Appendix II: Patent Examination Process at the 

U.S. Patent and Trademark Office (USPTO) 

(Corresponds to Fig. 1)

Page 71 GAO-16-479 Intellectual Property

This appendix provides details on steps in the patent examination 

process, including rollover information, depicted in figure 1.

Table 7: Descriptions of Selected Steps in the Patent Examination Process at the U.S. Patent and Trademark Office (USPTO)

Step in the patent examination process Description

Receipt of patent application

When USPTO receives a patent application, it first goes through a review to determine 

that it is complete. A patent application includes certain key elements, particularly the 

specification and the claims. The specification contains a written description of the 

invention, and the claims define the scope of the invention for which the applicant seeks 

patent protection. Applications can also include an abstract, drawings, and an 

information disclosure statement of prior art references—sometimes referred to as 

citations.

Classification

Because of the significant number of patent applications and issued patents in 

thousands of different technologies, USPTO, using a contractor, classifies patent 

applications according to their technical subject matter. Patent classification is useful for 

prior art searches because it allows an examiner to review a class of applications and 

issued patents to identify prior art in a specific area of technology. 

Assignment to patent examiner

On the basis of its classification, an application is then routed to one of USPTO’s 

technology centers for examination. These technology centers together cover electrical, 

chemical, and mechanical technology fields, and each center is organized into smaller 

“art units”—clusters of examiners that focus on related technologies. Once an application 

is routed to a technology center and art unit, USPTO assigns it to an examiner. 

According to USPTO officials, supervisory patent examiners assign applications to 

examiners based on each examiner’s workload, not the application’s complexity or the 

examiner’s skills.

Prior art search and examination

During patent examination, the examiner determines, among other things, whether the 

invention is novel and not obvious by comparing its claims to relevant prior art—prior 

patents, patent applications, or nonpatent literature describing a technology, among 

other things. To do so, the examiner conducts a search to find prior art that, if found, 

would help the examiner determine that the invention is not novel or is obvious. In 

addition, the examiner is to determine whether an application’s claims clearly and 

precisely inform persons skilled in the art of the boundaries of the patent requested. 

According to USPTO guidance, the scope of the claims must be clear in order to inform 

the public of the boundaries of the patent, and to clearly indicate what the applicant 

regards as the invention.

Examiner’s first decision on application

In determining if a patent is warranted, USPTO examiners use what are referred to as 

office actions to convey the outcomes of the patent examination process. Through what 

is called a first office action, or a First Action on the Merits, examiners initially notify 

applicants about the patentability of their inventions and are to fully communicate any 

deficiencies in the applications. Applicants may then amend their applications or 

otherwise address any deficiencies in order to continue the examination process. 

Supervisory examiners may also review some office actions.

Examiner’s final decision on application 

At the conclusion of the examination, examiners determine whether a patent will be 

granted in a final office action. If the application is rejected, the applicant may request 

additional examination. If a patent is to be granted, USPTO issues the patent to the 

applicant once certain fees are paid. The patent can generally remain in effect for up to 

20 years. Supervisory examiners may also review some final office actions.

Office of Patent Quality Assurance reviews

Quality assurance specialists review a subset of first and final decisions after the 

examiner’s work is complete.

Appendix II: Patent Examination Process at 

the U.S. Patent and Trademark Office 

(USPTO) (Corresponds to Fig. 1)
Appendix II: Patent Examination Process at the 

U.S. Patent and Trademark Office (USPTO) 

(Corresponds to Fig. 1)

Page 72 GAO-16-479 Intellectual Property

Step in the patent examination process Description

Appeals

Applicants and others can appeal an examiner’s decisions—including issued patents—

before USPTO’s Patent Trial and Appeal Board. In addition, USPTO decisions may be 

appealed in federal courts.

Time from receipt to decisions

Historically, USPTO has faced a backlog of unexamined applications and a lengthy 

pendency for applications—the amount of time between receipt of an application and 

USPTO’s first or final decision on it. From fiscal years 2011 through 2015, USPTO 

reduced the average pendency for first office actions from 28 months to 17 months, and 

for total pendency from 34 months to 27 months, according to the agency’s fiscal year 

2015 performance and accountability report. 

Source: GAO analysis of information from U.S. Patent and Trademark Office documents and officials. | GAO-16-479
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 73 GAO-16-479 Intellectual Property

Based on our survey results, U.S. Patent and Trademark Office (USPTO) 

patent examiners in some technology centers or at some General 

Schedule (GS) levels face certain challenges in identifying relevant prior 

art more than examiners in other technology centers or at other GS 

levels. Tables 8 through 15 provide information related to the effects of 

the quantity of prior art, how often examiners search for certain types of 

prior art and how difficult they find it to obtain relevant art from certain 

types, the sufficiency of examiners’ time to complete a thorough prior art 

search, average time examiners spent on prior art searching and each 

First Action on the Merits, examiners’ overtime, how often examiners 

encountered misclassified patent applications, and continuing education 

provided by USPTO in examiners’ art areas. Because we surveyed a 

generalizable stratified random sample of examiners, our results provide 

estimates for each technology center included in our study. See the 

related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

Table 8: Estimated Percentage of Examiners Who Find the Quantity of Art Makes Prior Art Searching Easier or More Difficult

Estimated percentage of examiners

U.S. Patent and Trademark Office technology center

Much 

easier

Somewhat 

easier

Neither easier nor 

more difficult

Somewhat 

more difficult

Much more 

difficult

Biotechnology and Organic Chemistry 10 20 17 25 27

Chemical and Materials Engineering 8 19 13 33 27

Computer Architecture, Software, and Information Security 13 28 24 21 13

Computer Networks, Multiplex Communication, Video 

Distribution, and Security 15 28 26 21 9

Communications 10 29 20 26 13

Semiconductors, Electrical and Optical Systems and 

Components 11 27 22 23 15

Transportation, Construction, 

Electronic Commerce, Agriculture, 

National Security and License and 

Reviewa

Business methods 11 26 23 25 13

Not business 

methods 9 19 16 27 28

Mechanical Engineering, Manufacturing, and Products 6 16 16 28 32

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 6 or fewer 

percentage points. We asked examiners how much easier or more difficult the quantity of art 

available makes it to complete a thorough prior art search in the time allotted. See the related esupplement, GAO-16-478SP, for the original survey language and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.

Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 74 GAO-16-479 Intellectual Property

Table 9: Estimated Percentage of Examiners Who Always Search for Certain Types of Prior Art in a Typical Patent 

Examination

U.S. Patent and Trademark Office 

technology center 

Image-based 

rather than 

text-based art

Industryrelated 

nonpatent 

literature (e.g., 

manuals or 

company 

websites)

Nonpatent 

literature in a 

foreign 

language

Scientific 

articles or 

presentations

Foreign 

patents

Biotechnology and Organic Chemistry 8 21 20 81 56

Chemical and Materials Engineering 17 5 10 20 58

Computer Architecture, Software, and 

Information Security 17 16 4 24 21

Computer Networks, Multiplex Communication, 

Video Distribution, and Security 12 11 6 15 19

Communications 21 10 10 19 27

Semiconductors, Electrical and Optical Systems 

and Components 30 7 8 9 36

Transportation, 

Construction, Electronic 

Commerce, Agriculture, 

National Security and 

License and Reviewa

Business methods 6 12 2 7 10

Not business 

methods 38 5 4 5 45

Mechanical Engineering, Manufacturing, and 

Products 39 3 5 6 35

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories. 

All estimates have 95 percent confidence intervals of within +/- 6 or fewer percentage points. See the 

related e-supplement, GAO-16-478SP, for the original survey language and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 75 GAO-16-479 Intellectual Property

Table 10: Estimated Percentage of Examiners Who Find It Very Difficult to Obtain Relevant Prior Art from Different Types of 

Sources

U.S. Patent and Trademark Office technology center Foreign patents

Nonpatent literature in a 

foreign language

Scientific articles or 

presentations

Biotechnology and Organic Chemistry 6 17 2

Chemical and Materials Engineering 5 27 5

Computer Architecture, Software, and Information Security 14 30 8

Computer Networks, Multiplex Communication, Video 

Distribution, and Security 11 23 6

Communications 9 20 10

Semiconductors, Electrical and Optical Systems and 

Components 7 25 8

Transportation, Construction, 

Electronic Commerce, 

Agriculture, National Security 

and License and Reviewa

Business methods 11 23 9

Not business methods 3 23 13

Mechanical Engineering, Manufacturing, and Products 3 22 14

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories. 

All estimates have 95 percent confidence intervals of within +/- 6 or fewer percentage points. We 

asked examiners how easy or difficult it is to obtain relevant art from searches for these types of prior 

art references. See the related e-supplement, GAO-16-478SP, for the original survey language and 

more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 76 GAO-16-479 Intellectual Property

Table 11: Estimated Sufficiency of Patent Examiners’ Time for Completing a Thorough Prior Art Search

Estimated percentage of examiners

U.S. Patent and Trademark Office technology center

Much more 

time than 

needed 

Somewhat 

more time 

than needed

About as 

much time 

as needed

Somewhat 

less time 

than needed

Much less 

time than 

needed

Biotechnology and Organic Chemistry 2 5 23 49 20

Chemical and Materials Engineering 1 4 23 46 24

Computer Architecture, Software, and Information 

Security 3 7 23 44 21

Computer Networks, Multiplex Communication, Video 

Distribution, and Security 5 10 23 36 24

Communications 6 11 21 39 23

Semiconductors, Electrical and Optical Systems and 

Components 5 8 22 42 22

Transportation, Construction, 

Electronic Commerce, 

Agriculture, National Security 

and License and Reviewa

Business methods 4 3 29 42 21

Not business methods 2 4 22 46 26

Mechanical Engineering, Manufacturing, and Products 1 5 16 40 37

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 6 or fewer 

percentage points. We asked examiners if given a typical workload they have sufficient time to 

complete a thorough prior art search. See the related e-supplement, GAO-16-478SP, for the original 

survey language and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 77 GAO-16-479 Intellectual Property

Table 12: Estimated Time Examiners Spent per Application on Prior Art Search and Each First Action on the Merits in the Past 

Quarter 

U.S. Patent and Trademark Office technology 

center

Prior art search 

(estimated mean 

hours)a

First Action on the Merits (estimated percentage of 

examiners)b

0-5 

hours

6-10 

hours

11-15 

hours

16-20 

hours

More than 

20 hours

Biotechnology and Organic Chemistry 5.2 3 29 39 21 6

Chemical and Materials Engineering 6.7 2 33 39 17 5

Computer Architecture, Software, and Information 

Security 10.9c 1 10 22 31 34

Computer Networks, Multiplex Communication, 

Video Distribution, and Security 8.8 2 14 25 31 25

Communications 8.9 1 17 28 30 21

Semiconductors, Electrical and Optical Systems 

and Components 7.4 2 31 31 22 9

Transportation, 

Construction, Electronic 

Commerce, Agriculture, 

National Security and 

License and Reviewd

Business methods 7.3 3 14 28 33 22

Not business methods 7.1 4 33 32 19 10

Mechanical Engineering, Manufacturing, and 

Products 7.9 3 29 38 18 10

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” See the related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

a

These estimates have 95 percent confidence intervals of within +/- 10 percent of the estimates 

themselves, unless otherwise noted.

b

These estimates have 95 percent confidence intervals of within +/- 6 or fewer percentage points. A 

First Action on the Merits is the means by which examiners initially notify applicants about the 

patentability of their inventions.

c

This estimate has a 95 percent confidence interval of within +/- 18 percent of the estimate itself.

d

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 78 GAO-16-479 Intellectual Property

Table 13: Estimated Percentage of Examiners Who Worked Biweekly 

Voluntary/Uncompensated Overtime during the Last 6 Months, by Examiner 

General Schedule Level

Hours

General Schedule 

(GS) level None 1-5 6-10 11-15 16-20 21-30

More 

than 30 

Below GS-13 16 19 20 13 11 6 7

GS-13 15 21 20 16 10 7 6

Above GS-13 24 23 22 11 6 2 3

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 5 or fewer 

percentage points. We asked examiners about how much voluntary/uncompensated overtime they 

worked per biweekly period in the past 6 months, on average, to meet their minimum production 

goals. See the related e-supplement, GAO-16-478SP, for the original survey language and more 

detailed results.

Table 14: Estimated Percentage of Examiners Who Encountered Misclassified Patent Applications in the Past Quarter

U.S. Patent and Trademark Office technology center Always Often Sometimes Rarely Never

Biotechnology and Organic Chemistry 2 15 40 34 4

Chemical and Materials Engineering 2 19 56 21 1

Computer Architecture, Software, and Information 

Security 3 24 51 21 1

Computer Networks, Multiplex Communication, 

Video Distribution, and Security 4 16 49 29 <1

Communications 4 21 55 18 1

Semiconductors, Electrical and Optical Systems 

and Components 4 23 52 19 1

Transportation, Construction, 

Electronic Commerce, 

Agriculture, National Security 

and License and Reviewa

Business methods 3 17 46 32 1

Not business methods 4 23 48 25 <1

Mechanical Engineering, Manufacturing, and Products 2 28 50 18 1

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 6 or fewer 

percentage points. See the related e-supplement, GAO-16-478SP, for the original survey language 

and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix III: Selected Survey Results by 

Technology Center or Examiners’ Level

Page 79 GAO-16-479 Intellectual Property

Table 15: Examiners’ Views on the Usefulness of Continuing Education in Their Art Area, by Technology Center

Estimated percentage of examiners

U.S. Patent and Trademark Office technology 

center 

Very 

useful 

Moderately 

useful 

Somewhat 

useful

Slightly 

useful

Not at all 

useful

Not 

taken

Not 

offered 

Biotechnology and Organic Chemistry 21 24 16 6 3 12 14

Chemical and Materials Engineering 19 19 21 10 3 18 8

Computer Architecture, Software, and Information 

Security 17 24 23 11 5 9 8

Computer Networks, Multiplex Communication, 

Video Distribution, and Security 26 24 20 11 6 6 4

Communications 27 25 16 11 3 8 6

Semiconductors, Electrical and Optical Systems 

and Components 20 22 16 9 5 12 12

Transportation, Construction, 

Electronic Commerce, 

Agriculture, National Security 

and License and Reviewa

Business methods 18 22 19 9 5 12 13

Not business 

methods 13 17 16 10 5 16 18

Mechanical Engineering, Manufacturing, and 

Products 14 22 19 11 5 15 11

Source: GAO survey of U.S. Patent and Trademark Office patent examiners. | GAO-16-479

Notes: Rows may not total 100 percent because the table does not include all response categories, 

such as “don’t know.” All estimates have 95 percent confidence intervals of within +/- 6 or fewer 

percentage points. We asked examiners how useful, if at all, to their jobs was the continuing 

education in their art area provided by the U.S. Patent and Trademark Office in the past year. See the 

related e-supplement, GAO-16-478SP, for the original survey language and more detailed results.

a

The Transportation, Construction, Electronic Commerce, Agriculture, National Security and License 

and Review technology center includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated the art units focused on 

electronic commerce and business methods (collectively referred to as business methods) in light of 

recent legislation and court decisions related to business methods.
Appendix IV: Comments from the U.S. 

Department of Commerce

Page 80 GAO-16-479 Intellectual Property

Appendix IV: Comments from the U.S. 

Department of Commerce
Appendix IV: Comments from the U.S. 

Department of Commerce

Page 81 GAO-16-479 Intellectual Property
Appendix IV: Comments from the U.S. 

Department of Commerce

Page 82 GAO-16-479 Intellectual Property
Appendix IV: Comments from the U.S. 

Department of Commerce

Page 83 GAO-16-479 Intellectual Property
Appendix V: GAO Contact and Staff 

Acknowledgments

Page 84 GAO-16-479 Intellectual Property

John Neumann, (202) 512-3841 or neumannj@gao.gov

In addition to the contact named above, Chris Murray (Assistant Director), 

Tind Shepper Ryen, Krista Breen Anderson, Hilary Benedict, Richard 

Burkard, John Delicath, Alice Feldesman, Armetha Liles, Rebecca Makar, 

Rob Marek, Eleni Orphanides, Kelly Rubin, Monica Savoy, Ardith Spence, 

and Sonya Vartivarian made key contributions to this report.

Appendix V: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments
Related GAO Products

Page 85 GAO-16-479 Intellectual Property

Intellectual Property: Assessing Factors That Affect Patent Infringement 

Litigation Could Help Improve Patent Quality. GAO-13-465. Washington, 

D.C.: August 22, 2013.

U.S. Patent and Trademark Office: Performance Management Processes. 

GAO-10-946R. Washington, D.C.: September 24, 2010.

Intellectual Property: Enhanced Planning by U.S. Personnel Overseas 

Could Strengthen Efforts. GAO-09-863. Washington, D.C.: September 30, 

2009.

U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to 

Reduce the Patent Application Backlog. GAO-08-527T. Washington, 

D.C.: February 27, 2008.

U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to 

Reduce the Patent Application Backlog. GAO-07-1102. Washington, 

D.C.: September 4, 2007.

Intellectual Property: Improvements Needed to Better Manage Patent 

Office Automation and Address Workforce Challenges. GAO-05-1008T. 

Washington, D.C.: September 8, 2005.

Intellectual Property: Key Processes for Managing Patent Automation 

Strategy Need Strengthening. GAO-05-336. Washington, D.C.: June 17, 

2005.

Intellectual Property: USPTO Has Made Progress in Hiring Examiners, 

but Challenges to Retention Remain. GAO-05-720. Washington, D.C.: 

June 17, 2005.

Related GAO Products

(361612)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.