INTELLECTUAL 

PROPERTY 

Patent Office Should 

Define Quality, 

Reassess Incentives, 

and Improve Clarity

Report to the Chairman, Committee on 

the Judiciary, House of Representatives

June 2016

GAO-16-490

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-490, a report to the 

Chairman, Committee on the Judiciary, House 

of Representatives

June 2016

INTELLECTUAL PROPERTY

Patent Office Should Define Quality, Reassess 

Incentives, and Improve Clarity 

Why GAO Did This Study

Resolving disputes over patent 

infringement and validity in court often 

costs millions of dollars. Legal scholars 

and economists have raised concerns 

about an increase in the numbers of 

low quality patents—such as those that 

are unclear and overly broad—which 

may lead to an increase in patent 

infringement suits and can hinder 

innovation by blocking new ideas from

entering the marketplace.

GAO was asked to review issues 

related to patent quality. GAO 

examined (1) recent trends in patent 

infringement litigation and (2) what 

additional opportunities exist, if any, to 

improve patent quality. GAO reviewed 

relevant laws and agency documents;

analyzed patent infringement litigation 

data from 2007 through 2015;

conducted a survey of a generalizable 

sample of USPTO examiners; and 

interviewed officials from USPTO and 

knowledgeable stakeholders, including 

legal scholars, technology companies, 

and patent attorneys, among others. 

What GAO Recommends

GAO makes seven recommendations, 

including that USPTO more 

consistently define patent quality and 

articulate that definition in agency 

documents and guidance, reassess the 

time allotted for examination, analyze 

the effects of incentives on patent 

quality, and consider requiring

applicants to use additional clarity 

tools. USPTO generally agreed with 

GAO’s findings, concurred with the

recommendations, and provided

information on steps officials plan to 

take to implement the 

recommendations.

What GAO Found

GAO found that district court filings of new patent infringement lawsuits increased 

from about 2,000 in 2007 to more than 5,000 in 2015, while the number of 

defendants named in these lawsuits increased from 5,000 to 8,000 over the 

same period. In 2007, about 20 percent of all defendants named in new patent 

infringement lawsuits were sued in the Eastern District of Texas, and by 2015 

this had risen to almost 50 percent. According to stakeholders, patent 

infringement suits are increasingly being tried in the predominantly rural Eastern

District of Texas, likely due to recent practices in that district that are favorable to 

the patent owners who bring these infringement suits. GAO also found that most 

patent suits involve software-related patents and computer and communications 

technologies. Several stakeholders told GAO that it is easy to unintentionally 

infringe on patents associated with these technologies because the patents can 

be unclear and overly broad, which several stakeholders believe is a 

characteristic of low patent quality.

The U.S. Patent and Trademark Office (USPTO) has taken actions to address 

patent quality, most notably through its Enhanced Patent Quality Initiative, but

there are additional opportunities for the agency to improve patent quality. For 

example, USPTO does not currently have a consistent definition for patent 

quality articulated in agency documents and guidance, which would be in line 

with federal internal-control standards and best practices for organizational 

performance. Most stakeholders GAO interviewed said they would define a 

quality patent as one that would meet the statutory requirements for novelty and 

clarity, among others, and would be upheld if challenged in a lawsuit or other

proceeding. Without a consistent definition, USPTO is unable to fully measure

progress toward meeting its patent quality goals. Additionally, USPTO has not 

fully assessed the effects of the time allotted for application examinations or

monetary incentives for examiners on patent quality. Specifically, most 

stakeholders GAO interviewed said that time pressures on examiners are a

central challenge for patent quality. Based on GAO’s survey of patent examiners, 

GAO estimates that 70 percent of the population of examiners say they do not 

have enough time to complete a thorough examination given a typical workload. 

According to federal standards for internal control, agencies should provide staff 

with the right structure, incentives, and responsibilities to make operational

success possible. Without assessing the effects of current incentives for 

examiners or the time allotted for examination, USPTO cannot be assured that its

time allotments and incentives support the agency’s patent quality goals. Finally, 

USPTO does not currently require applicants to define key terms or make use of 

additional tools to ensure patent clarity. Federal statutes require that patent 

applications use clear, concise, and exact terms. Based on a survey of patent 

examiners, GAO estimates that nearly 90 percent of examiners always or often 

encountered broadly worded patent applications, and nearly two-thirds of 

examiners said that this made it difficult to complete a thorough examination. 

Without making use of additional tools, such as a glossary of key terms, to 

improve the clarity of patent applications, USPTO is at risk of issuing patents that 

do not meet statutory requirements. View GAO-16-490. For more information, 

contact Frank Rusco at (202) 512-3841 or 

ruscof@gao.gov. 
Page i GAO-16-490 Intellectual Property

Letter 1

Background 4

Patent Infringement Litigation Has Increased in Recent Years, and 

the Majority of Lawsuits Involved Patents Related to Computers 

and Computer Software 14

Additional Opportunities Exist to Improve Patent Quality at 

USPTO 21

Conclusions 37

Recommendations for Executive Action 38

Agency Comments 39

Appendix I Objectives, Scope, and Methodology 41

Appendix II Patent Examination Process at the U.S. Patent and Trademark 

Office (USPTO) (Corresponds to Fig. 1) 47

Appendix III Comments from the U.S. Patent and Trademark Office 49

Appendix IV GAO Contact and Staff Acknowledgments 53

Related GAO Products 54

Tables

Table 1: Example of a “Claim Chart” 36

Table 2: Strata of United States Patent and Trademark Office 

(USPTO) Examiners Surveyed 43

Table 3: Descriptions of Selected Steps in the Patent Examination 

Process at the U.S. Patent and Trademark Office 

(USPTO) 47

Contents
Page ii GAO-16-490 Intellectual Property

Figures

Figure 1: Selected Steps in the Patent Examination Process at the 

U.S. Patent and Trademark Office Interactive Graphic 

(USPTO) 6

Figure 2: Number of Patent Infringement Suits and Defendants, 

2007 through 2015 15

Figure 3: Number of Defendants in Patent Infringement Suits by 

District Courts for Patent Suits, 2007 through 2015 17

Figure 4: Number of Defendants in Patent Infringement Lawsuits 

by Technology Area, 2007 through 2015 19

Figure 5: Number of Defendants in Patent Infringement Lawsuits 

for Software-Related and Non-Software-Related Patents, 

2007 through 2015 21

Figure 6: U.S. Patent and Trademark Office (USPTO) Strategic 

Goal Related to Patent Quality and Timeliness 24

Figure 7: Estimated Bi-weekly Overtime Worked by Patent 

Examiners in the Past 6 Months 27

Abbreviations

GS General Schedule 

GPRAMA Government Performance and Results Modernization Act of

2010 

PTAB Patent Trial and Appeal Board 

USPTO U.S. Patent and Trademark Office 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-490 Intellectual Property

441 G St. N.W.

Washington, DC 20548

June 30, 2016

The Honorable Bob Goodlatte

Chairman

Committee on the Judiciary

House of Representatives

Dear Mr. Chairman:

Intellectual property-intensive industries such as electronics and 

pharmaceuticals support at least 40-million jobs in the U.S. and contribute 

more than $5 trillion annually—or nearly 35 percent—to the nation’s gross 

domestic product. While most patents are never the subject of a lawsuit, 

resolving a dispute over patent infringement and validity in courts can 

cost millions of dollars. Legal scholars and economists have raised 

concerns about an increase in the number of low quality patents—such 

as those that are unclear and overly broad—which may lead to an 

increase in patent infringement suits and hinder innovation by blocking 

new ideas from entering the marketplace.1 Further, in our 2013 report, we 

found that low quality patents were likely a key factor in many patent 

infringement lawsuits between 2007 and 2011, because their unclear 

boundaries make it easy to unintentionally infringe these patents, among 

other factors.2

The U.S. patent system—authorized by the U.S. Constitution—aims, in 

part, to make innovation more profitable.3 For example, a patent owner 

 1

Anyone who makes, sells, offers to sell, uses, or imports the patented invention during 

the term of the patent without the patent owner’s permission infringes the patent.

2

GAO, Intellectual Property: Assessing Factors That Affect Patent Infringement Litigation 

Could Help Improve Patent Quality, GAO-13-465 (Washington, D.C.: Aug. 22, 2013).

3

The Constitution grants to Congress the power “[t]o promote the Progress of Science and 

useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to 

their respective Writings and Discoveries.” U.S. Const., art. I, § 8, cl. 8. Because a patent 

expires after a fixed period, patent law also “ultimately provid[es] the public with the 

benefit of lower price through unfettered competition.” Biotechnology Indus. Org. v. District 

of Columbia, 496 F.3d 1362, 1373 (Fed. Cir. 2007) (stating that “these two objectives—to 

reward innovators with higher profits and to keep prices reasonable for consumers—are in 

dialectic tension).”

Letter
Page 2 GAO-16-490 Intellectual Property

can generally exclude others from making, using, selling, or importing the 

patented technology for up to 20 years from the date on which the 

application for the patent was filed. By restricting competition, patents can 

allow their owners to earn greater profits on their patented technologies 

than they could earn if these technologies could be freely imitated. Patent 

owners sometimes need to assert their patent rights by filing a lawsuit if 

competitors infringe their patent. The exclusive rights provided by patents 

can help their owners recoup the costs of the research and development 

of new technologies. On the other hand, any reduction in competition 

caused by the exclusive nature of patents may result in higher prices for 

products having patented technologies. The patent system, therefore, 

gives rise to complex trade-offs involving innovation and competition. 

Some researchers have suggested that the balance between these tradeoffs has tipped and that some patents now are limiting innovation, 

especially in areas such as software and computer technologies.

We found in 2013 that some stakeholders, including technology 

companies, judges, and legal scholars, said that the U.S. Patent and 

Trademark Office (USPTO) should never have issued some patents 

because they believe they do not meet all of the legal standards required. 

Patents must meet standards set forth in law and interpreted by the 

courts. For example, they must cover eligible subject matter, be novel, not 

obvious, and they must clearly describe the invention.4 In this report, we 

consider patents that fully meet federal patentability standards to be high 

quality.

5 USPTO has acknowledged the need to focus additional attention 

on patent quality and, in 2015, began its Enhanced Patent Quality 

Initiative, aimed at helping to ensure that the thousands of patents the 

agency issues each year are of higher quality. In a Department of 

Commerce Inspector General’s report, USPTO’s deputy director said an 

increased focus on patent quality would play a significant role in curtailing 

patent litigation.6

 4

See 35 U.S.C. §§ 101, 102, 103 and 112. A patent that is unclear or overly broad, for 

example, could be found to be invalid under section 112 of title 35. 

5

These legal standards and the judicial interpretation of them could change after a patent 

has been granted.

6

U.S. Department of Commerce, Office of Inspector General. USPTO Needs to 

Strengthen Patent Quality Assurance Practices, Final Report Number OIG-15-026-A (Apr. 

10, 2015)
Page 3 GAO-16-490 Intellectual Property

You asked us to review issues related to patent quality. This report 

examines: (1) recent trends in patent infringement litigation, and (2) what 

additional opportunities exist, if any, to improve patent quality. Our work 

for this report was coordinated with our report on a part of the patent 

examination process known as “prior art” searches.7

To examine recent trends in patent infringement litigation, we obtained 

patent infringement litigation data from two companies—RPX Inc. and 

Lex Machina—that included all of the patent infringement lawsuits filed in 

all 94 federal district courts between 2007 and 2015. These data included 

information on the patents asserted in each suit, the defendants involved, 

and the federal district court where the suit was filed. We conducted data 

quality testing on the data from RPX and Lex Machina, interviewed 

relevant officials, and reviewed relevant documentation for the data and 

found the data to be sufficiently reliable to determine recent trends in 

patent infringement lawsuits. We also conducted 11 semi-structured 

interviews with stakeholders from technology companies, venture capital 

investors, and others knowledgeable about recent patent infringement 

litigation. In addition, we reviewed published literature on the patent 

system, including reports from patent researchers, the Federal Trade 

Commission, and the Congressional Budget Office.

To examine what additional opportunities exist, if any, to improve patent 

quality, we reviewed relevant laws and USPTO documents and 

interviewed USPTO officials and representatives of the examiners’

union—the Patent Office Professional Association. We interviewed four 

supervisory patent examiners and six patent examiners from a variety of 

technology areas. We also conducted 11 semi-structured interviews with 

patent stakeholders who were knowledgeable about patent quality and 

USPTO, including legal scholars, former high-ranking USPTO officials, 

and representatives from public interest non-governmental organizations. 

In addition, we conducted a web-based survey of a stratified random 

sample of 3,336 USPTO patent examiners from across eight of the 

technology-based subject matter groups (referred to as technology 

centers) into which USPTO examiners are assigned. We excluded 

examiners employed at USPTO for less than one year, as of May 2015. 

 7

GAO, Intellectual Property: Patent Office Should Strengthen Search Capabilities and 

Better Monitor Examiners’ Work, GAO-16-479 (Washington, D.C.: June 30, 2016).
Page 4 GAO-16-490 Intellectual Property

The survey collected information on examiners’ views on USPTO’s 

approach to patent quality and how the agency might improve its patent 

quality efforts. The survey was designed and administered in conjunction 

with a concurrent report on patent prior art.8 Overall, we received 

responses from 80 percent of the examiners in our sample. Our 

methodology was designed to produce estimates that are generalizable to 

the population of patent examiners in our study overall, as well as within 

each technology center.9 The survey questions and detailed data on the 

results can be viewed on our website.10 Appendix I provides more detail 

on our scope and methodology.

We conducted this performance audit from November 2014 to June 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

This section provides an overview of USPTO’s patent examination 

process, patent infringement litigation and challenges, and USPTO’s 

Enhanced Patent Quality Initiative. A list of our prior work related to 

patents and intellectual property is included at the end of this report.

 8

Survey questions also addressed concerns regarding prior art searches. See: GAO, 

Intellectual Property: Patent Office Should Strengthen Search Capabilities and Better 

Monitor Examiners’ Work, GAO-16-479 (Washington, D.C.: June 30, 2016).

9

Because we followed a probability procedure based on random selections, our sample is 

only one of a large number of samples that we might have drawn. Since each sample 

could have provided different estimates, we express our confidence in the precision of our 

particular sample’s results as a 95 percent confidence interval. This is the interval that 

would contain the actual population value for 95 percent of the samples we could have 

drawn. All percentage estimates from our survey have margins of error at the 95 percent 

confidence level of plus or minus 6 percentage points or less, unless otherwise noted.

10GAO, Survey of USPTO Patent Examiners, GAO-16-478SP (Washington, D.C.: June 

30, 2016).

Background
Page 5 GAO-16-490 Intellectual Property

When USPTO receives a patent application, the agency assigns it to a 

division of patent examiners with relevant technology expertise called a 

technology center. There are 11 technology centers focusing on 

everything from biotechnology to mechanical inventions. After the 

application is assigned to a technology center, it is then assigned to an 

individual examiner who is responsible for the examination, or 

prosecution, of the application. Figure 1 shows the key steps in the patent 

prosecution process.

Patent Examination 

Process
Figure 1: Selected Steps in the Patent Examination Process at the U.S. Patent and Trademark Office 

(USPTO) Interactive Graphic

Instructions: Hover over each step in the patent examination process below to see more information. 

 To print a version containing text, see appendix II, page 47.

Page 6 GAO-16-490 Intellectual Property

Note: See app. II for the text underlying this figure.
Page 7 GAO-16-490 Intellectual Property

The focus of patent examination is determining whether the invention in a 

patent application satisfies the statutory requirements for a patent, 

including that the invention be novel, useful, not obvious, and clearly 

described. Generally, prior patents, patent applications, or publications 

describing an invention, among other things, are known as prior art. 

During patent examination, the examiner, among other things, compares 

an application to the prior art to determine whether the invention is novel 

and not obvious.11 Finding prior art is the most time consuming part of 

patent examination, according to our report on prior art.12 Applicants are 

not required to search for prior art before submitting their application, 

although they are required to notify examiners of material prior art they 

know about.

A patent application includes the “specification” and at least one “claim.”

By statute, the specification is to contain “a written description of the 

invention, and of the manner and process of making it, in such full, clear, 

concise, and exact terms as to enable any person skilled in the art to 

which it pertains to make and use the invention.”13 The law requires 

further that the specification “shall conclude with one or more claims 

particularly pointing out and distinctly claiming the subject matter which 

the inventor regards as the invention.14 A patent application’s claims 

define the legal boundaries of the invention for which patent protection is 

sought.

USPTO’s data show that, as of April 2016, the average time between 

filing an application and an examiner’s initial decision on the application 

was about 16 months, and it takes an average of 26 months after an 

application is submitted for USPTO to complete the examination of an 

 11During the patent examination, the applicant and the examiner communicate about the 

application, including aspects that might be deficient. For example, the examiner may 

inform the applicant that the patent is not novel because of prior art, and the applicant 

might revise it to distinguish it from the prior art the examiner found. For more information 

on prior art and searching for prior art, see GAO-16-479.

12GAO-16-479.

1335 U.S.C. § 112(a). 

1435 U.S.C. § 112(b). 
Page 8 GAO-16-490 Intellectual Property

application.15 Due to the current inventory of applications awaiting 

examination, USPTO is not able to begin examining patent applications 

upon receiving them—as of April 2016, USPTO had a backlog of about 

550,000 unexamined applications.

Through the initial decision, or first action on the merits, examiners initially 

notify applicants about the patentability of their inventions. To avoid 

lengthy back and forth exchanges, the examiner is encouraged to identify 

all of the problems with the application under the patent statutes and rules 

during the first “office action”—a USPTO policy called “compact 

prosecution.”16 The applicant is able to respond to the examiner after the 

first office action, and interviews are often used so that the examiner and 

applicant can clarify their respective positions, as well as the scope of the 

claimed invention. Ultimately, and sometimes after many months, the 

examiner then issues a final rejection or allows the claims in the 

application. If applicants receive a final rejection, they may file a request 

for continued examination, which requires a new submission and the 

payment of additional fees, and the examiner will continue examining the 

application. There is no limit to the number of times an applicant may 

request continued examination for an application, but USPTO officials say 

it is rare to have three or more requests for continuing examination filed 

for a single application. Examination ends with an issued patent, or when 

the applicant abandons the application—there are no terminal rejections.

There are often a dozen or more claims per patent, and they can often be 

difficult for a layperson to understand, according to legal researchers. For 

example, one claim for a cardboard coffee cup insulator begins by 

referring to “a recyclable, insulating beverage container holder, 

comprising a corrugated tubular member comprising cellulosic material 

and at least a first opening therein for receiving and retaining a beverage 

 15Between fiscal years 2011 and 2015, USPTO reduced the average pendency from 28 

months to 17 months, and for total pendency from 34 months to 27 months, according to 

the agency’s fiscal year 2015 performance and accountability report. 

16According to USPTO’s Manual of Patent Examining Procedure 2173.06, “Under the 

principles of compact prosecution, the examiner should review each claim for compliance 

with every statutory requirement for patentability in the initial review of the application and 

identify all of the applicable grounds of rejection in the first office action to avoid 

unnecessary delays in the prosecution of the application.”
Page 9 GAO-16-490 Intellectual Property

container.”17 A patent application’s claims can be written broadly or be 

more narrowly defined, according to legal researchers, and applicants 

can change the wording of claims—which can affect their scope—based 

on examiner feedback during examination. Examiners can suggest that 

applicants amend their claims by adding words to their claims to address 

statutory impediments to issuance. For example, adding the word 

“corrugated” to modify “tubular member” in the example claim above 

narrows the scope of the claimed invention. Companies often prefer 

broader patent claims that make it less likely a competitor would be able 

to make a small change to its invention to avoid infringement. Patents are 

a property right and their claims define their boundaries.

In some cases, patent claims define the scope of the invention by 

encompassing an entire function––like sending an e-mail––rather than 

the specific means of performing that function. While “functional claiming”

is permitted by statute, we reported in 2013 that patents that include 

functional claiming language were more likely to be unclear and to be 

disputed in court.18 For example, if the pencil was patented as a

“mechanism for writing,” the owner of the patent could theoretically sue 

manufacturers of different technologies for infringement, including pens 

and markers.

As of May 2015, USPTO had nearly 8,300 patent examiners across the 

eight technology centers that we reviewed. The agency uses the General 

Schedule (GS) classification system for patent examiners, whose levels 

range from GS-5 to GS-15.19 Examiners at the GS-14 level or above (44 

percent of the examiners in the technology centers we reviewed) are 

referred to as “primary examiners” who may accept or reject a patent 

 17See US patent 5205473.

18An element in a claim for a combination may be expressed as a means or step for 

performing a specified function without the recital of structure, material, or acts in support 

thereof, and such claim shall be construed to cover the corresponding structure, material, 

or acts described in the specification or equivalents thereof. 35 U.S.C. § 112(f).

19The GS system is the federal government’s classification system for defining and 

organizing federal positions, primarily to assign rates of pay, based on a position’s duties, 

responsibilities, and qualification requirements, among other things. The GS system 

includes 15 statutorily-defined grade levels, which are numerical designations based on 

the complexity of the work and knowledge required to do the job, and are on a scale of 

GS-1 to GS-15.
Page 10 GAO-16-490 Intellectual Property

application without additional review. This level of authority is in contrast 

to junior examiners—most examiners below the GS-14 level— whose 

work must first be reviewed by a supervisory or primary patent examiner 

before it can be sent to the applicant. 

Examiners are rated based on their production, or the number of 

examination tasks they perform, among other factors. The number of 

examination tasks that an examiner is expected to perform is set based 

on the examiner’s technology area and experience level. USPTO allots 

more time to review applications to examiners that deal with more 

complex technologies. For example examiners working on artificial 

intelligence patent applications are given an average of about 31 hours to 

complete an examination, while those working on patent applications for 

exercise devices are given an average of about 17 hours. As examiners 

are promoted on the GS-scale, the average number of hours they are 

allotted to work on each application and the level of review from their 

supervisors declines. Primary examiners have the least amount of time to 

examine patent applications and the applications they review undergo the 

least amount of supervisory review, in part because their experience 

allows them to work more efficiently and effectively, according to USPTO 

officials. As examiners rise from junior to primary status, their examination 

time is roughly cut in half. According to USPTO officials, on average, 

examiners spend about 22 hours total on average on each application 

from start to final determination, with a low of about 11 hours on average 

for some primary examiners in the least complex technologies and a high 

of about 60 hours on average for an entry-level junior examiner in more 

complex technologies.

A few studies have shown that there are differences in issued patents 

depending on how much time examiners are given to complete their 

examination. For example, an academic paper from 2012 found that more 

experienced examiners cite less prior art, are more likely to grant patents 

rather than reject them, and are more likely to grant patents without any 

preliminary rejections.20 Similarly, other researchers have found that 

examiners put less effort into searching for prior art when they are given 

less time to review an application. For example, the National Bureau of 

 20Lemley, M. and Sampat, B. Examiner Characteristics and Patent Office Outcomes. 

Review of Economics and Statistics 2012. 94(3): 817–827.
Page 11 GAO-16-490 Intellectual Property

Economic Research published a paper in 2014 that found when 

examiners were allotted less time to conduct a patent examination, they 

were less likely to make time-intensive prior art rejections and more likely 

to grant a patent.21

When a patent right is not clearly defined, it can lead to boundary 

disputes, often in the form of infringement lawsuits. Although litigated 

patents are a small percentage of issued patents, low quality patents are 

more likely to be asserted in patent infringement lawsuits because, 

according to some economists, the less clear the claim boundaries are, 

the more likely that others will infringe the patent or will continue to 

infringe when confronted by the patent owner.22 Patent owners can bring 

infringement lawsuits against anyone who uses, makes, sells, offers to 

sell, or imports the patented invention without authorization. If the court 

finds that infringement has occurred, it must award the patent owner 

damages adequate to compensate for the infringement. During an 

infringement case, the accused infringer may seek to have the 

infringement lawsuit dismissed by showing, by clear and convincing 

evidence, that the patent at issue is invalid. However, because most of 

the roughly 4,000 patent infringement lawsuits filed each year settle 

before the court ever makes a determination of patent validity some 

patent owners asserting low quality patents may not be concerned with 

the risk the courts will invalidate their patents.

23 When the courts do rule 

on validity, they generally invalidate almost half of the patents that are 

challenged, according to academic research.24

 21Frakes, M. and Wasserman, M. Is Time Allocated to Review Patent Applications 

Inducing Examiners to Grant Invalid Patents: Evidence from Micro-Level Application Data, 

National Bureau of Economic Research, 2014. Obviousness rejections often require an 

examiner to combine multiple disparate prior art documents.

22According to some scholars, the most litigated patents are among the most valuable 

patents and contain more claims, cite more prior art, and are more often cited by other 

patents as prior art. See John R. Allison et al., Valuable Patents, 92 Geo. L.J. 435 (2004).

23We reported in 2013 that patent infringement lawsuits often settle before trial because 

parties want to avoid the high cost of litigation. See GAO-13-465.

24For example, see Allison, J., Lemley, M., and Schwartz, D. Understanding the Realities 

of Modern Patent Litigation. Hoover Institution Working Group Paper, 2014.

Patent Infringement 

Litigation and Challenges
Page 12 GAO-16-490 Intellectual Property

Accused infringers can also challenge a patent’s validity outside of an 

infringement lawsuit in administrative proceedings at USPTO’s Patent 

Trial and Appeal Board (PTAB).25 Patent challenges at PTAB are often 

initiated by individuals or firms that have been sued for infringing a 

particular patent, so the patents that appear in these proceedings are 

often the same ones that appear in patent infringement suits. The 

challengers in these proceedings seek to present evidence that shows 

that the patent claims should not have been granted because they failed

to meet a statutory patentability requirement. The PTAB proceedings are 

a lower cost alternative to the federal courts where infringement suits are 

often very expensive.26 As of March 2016, there have been around 4,700 

patent challenges filed with the PTAB since its inception in 2012 and 

about 60 percent of these challenged patents are related to computers 

and software. About 30 percent of the PTAB proceedings have reached a 

final decision, and nearly 75 percent of those final decisions have resulted 

in all of the challenged claims being held unpatentable.27

In February 2015, USPTO launched an Enhanced Patent Quality Initiative 

designed to improve the quality of patents. According to USPTO, it

started its quality initiative because the agency had successfully reduced 

its backlog of patent applications, and had the financial resources to 

consider longer-term improvements to patent quality due to fee setting 

authority provided by the America Invents Act.

As part of its initiative, the agency has taken a number of actions, 

including:

• Creating a new leadership role and Patent Quality Office: The agency 

created a new senior position for overseeing patent quality—the 

 25The Leahy-Smith America Invents Act created the PTAB within the USPTO and 

established three new proceedings for challenging issued patents. Pub. L. No. 112-29, §§ 

6, 7, and 18 (2011).

26In 2013, industry stakeholders told us that challenges at PTAB cost upwards of 

$400,000 including filing and legal fees, while federal court proceedings can cost between 

$650,000 and $5 million. See GAO-13-465.

27Data from PTAB. The PTAB’s Inter Partes review, which is limited to challenges alleging 

that the patent does not meet the legal standards for novelty or non-obviousness, 

comprise nearly 90 percent of all PTAB post-grant decisions. 

USPTO’s Enhanced 

Patent Quality Initiative
Page 13 GAO-16-490 Intellectual Property

Deputy Commissioner for Patent Quality—in January 2015 to provide 

a dedicated focus on the agency’s patent quality efforts.

• Patent Quality Summit: USPTO held its first ever Patent Quality 

Summit in March 2015. The Summit was designed for the public, 

including internal and external stakeholders, to provide input to 

USPTO about patent quality, specifically how the agency could 

guarantee the most efficient process to review applications and to 

ensure the issuance of the highest quality patents, according to the 

USPTO Summit website. According to USPTO, the agency received 

1,206 ideas for improvement for patent quality from all sources, 

including the Patent Quality Summit and examiner forums.

• Evolving programs of the Enhanced Patent Quality Initiative: USPTO 

established 11 initiatives based on feedback from internal and 

external stakeholders to help achieve its goals to enhance patent 

quality, according to its Enhanced Patent Quality Initiative website. 

These initiatives are in various stages, ranging from early 

development to having been completed, according to a senior USPTO 

official. Some of the evolving programs include:

• Clarity of the Record Pilot Program: USPTO began this pilot in 

February 2016 with about 130 examiners, and the pilot is expected to 

end in August 2016. This pilot seeks to develop best practices for 

examiners to enhance the clarity of all aspects of the prosecution 

record, and to study the effect of implementing these best practices 

during examination.

• Clarity and Correctness Data Capture (Master Review Form): This 

effort is expected to replace USPTO’s current quality assurance and 

supervisory approaches to reviewing examiners’ work and will allow 

the agency, for the first time, to collect consistent data across all the 

reviews. Quality assurance officials also told us that using the form 

would allow their office to have data on 50 percent more reviews this 

year than in the past, resulting in a total of about 12,000 reviews of 

examiners’ work this year compared with about 8,000 reviews per 

year in the past. USPTO expects to implement the form agency-wide 

in fiscal year 2017.28

 28In our report on prior art searching, we provide additional information on how the master 

review form could affect prior art searching and the deficiencies of the form to review prior 

art searches, and make a recommendation to address the deficiencies. See GAO-16-479.
Page 14 GAO-16-490 Intellectual Property

In addition to the actions taken under its Enhanced Patent Quality 

Initiative, the USPTO’s Office of Patent Training develops and 

implements training for examiners on a variety of topics, with a focus on 

legal and policy matters.29 For example, the USPTO recently provided 

examiners with training on functional claiming.

The number of federal district court filings of new patent infringement 

lawsuits has generally increased between 2007 and 2015, from more 

than 2,000 suits in 2007 to more than 5,000 suits in 2015 (see fig. 2). 

Because lawsuits can include multiple defendants, we also analyzed data 

on federal district court filings at the defendant level to account for 

lawsuits in which a patent was asserted against multiple defendants.30

Looking at the data in this way, we found that the number of defendants 

in new patent infringement suits filed in federal district courts increased 

from about 5,000 defendants in 2007 to more than 8,000 defendants in 

2015, as shown in figure 2.

 29USPTO officials told us that in fiscal year 2015, USPTO provided 340 examiners with 

new examiner training and more than 8,500 examiners with refresher and advanced 

examiner training. In GAO-16-479, we also discuss issues related to technical training 

provided to examiners.

30As we found in our 2013 report (GAO-13-465), some of the increase in the number of 

patent infringement lawsuits filed starting in 2011 was likely due to anticipated changes in 

the Leahy-Smith America Invents Act. The Act made it more difficult for plaintiffs to sue 

multiple defendants in a lawsuit, causing some plaintiffs that would have previously filed a 

single lawsuit with multiple defendants to break the lawsuit into multiple lawsuits. 

Patent Infringement 

Litigation Has 

Increased in Recent 

Years, and the 

Majority of Lawsuits 

Involved Patents 

Related to Computers 

and Computer 

Software
Page 15 GAO-16-490 Intellectual Property

Figure 2: Number of Patent Infringement Suits and Defendants, 2007 through 2015

According to some stakeholders with whom we spoke, the decreases in 

litigation that occurred in 2014, both in the number of suits and 

defendants, were likely due to a key Supreme Court decision in that year, 

Alice Corp. Pty. Ltd. v. CLS Bank Int’l.

31 The Court in the Alice decision 

held that where a patent claim is based on an abstract idea, which is not 

patentable, merely using generic computer implementation does not 

transform that idea into a patent-eligible invention. This decision has 

limited the validity of what some stakeholders considered to be overly 

broad and low quality patents thus preventing them from being used to 

file infringement lawsuits. In addition, the ability of potential defendants in 

patent infringement suits to file inter partes challenges to the validity of a 

patent at the PTAB beginning in 2012 may have made some patent 

owners reluctant to bring infringement suits, which could have contributed 

 31134 S. Ct. 2347, 189 L.Ed. 2d 296 (2014).
Page 16 GAO-16-490 Intellectual Property

to the decline in the number of suits according to some stakeholders we 

spoke with.32

Most patent infringement suits are filed in just a few of the 94 federal 

district courts, and these suits may generally be brought in any district in 

the country where the allegedly infringing products are sold.33 Patent 

infringement suits are increasingly being filed in the predominantly rural 

Eastern District of Texas (see fig. 3). In 2007, about 20 percent of all 

patent infringement defendants were named in cases filed in the Eastern 

District of Texas, and this percentage increased to almost 50 percent in 

2015.

 32The Leahy-Smith America Invents Act, Pub. L. No. 112-29, (2011), created, among 

other things, fast track proceedings for challenging issued patents, called Inter Partes 

Review. Under these proceedings, petitioners may challenge the patentability of claims 

under sections 102 and 103 of Title 35, which require novelty and non-obviousness. 35 

U.S.C. § 311(b). These proceedings became available on September 16, 2012.Pub. L.

No. 112-29, § 7(e).

33See Beverly Hills Fan Co. v. Royal Sovereign Corp., 21 F.3d 1558, (Fed. Cir. 1994), and 

VE Holding v. Johnson Gas Appliance, 917 F.2d 1574 (Fed. Cir. 1990).
Page 17 GAO-16-490 Intellectual Property

Figure 3: Number of Defendants in Patent Infringement Suits by District Courts for 

Patent Suits, 2007 through 2015

Historically, the Eastern District of Texas has been attractive to patent 

owners filing infringement lawsuits because of the speed at which suits 

moved to trial, and the perception of plaintiff-friendly juries, as we found in 

2013.34 In addition, according to one published paper we reviewed, 

judges in the Eastern District of Texas have implemented a number of 

court rules and practices to attract patent infringement suits to their 

district.35 For example, according to the paper and to some stakeholders 

 34GAO-13-465.

35Klerman D. and Reilly G. Forum Selling, 89 S. Cal L. Rev. 241 (Jan. 2016). 
Page 18 GAO-16-490 Intellectual Property

we interviewed, judges in the Eastern District of Texas have not been 

suspending infringement lawsuits pending patent validity challenges at 

the PTAB, as some had anticipated.36 In addition, most recently, 

according to some stakeholders, some judges in Eastern Texas have 

been reluctant to dismiss infringement allegations at early stages of the 

litigation based on the Alice decision that calls into question the validity of 

certain types of software-related patents. Our analysis shows a sharp 

increase from 2014 to 2015 in the number of defendants that were 

accused of infringing software-related patents in suits filed in Eastern 

Texas, which have averaged about 85 percent of the defendants in that 

district each year between 2007 and 2015. None of the other top six 

district courts saw any large increases in patent infringement suits in 

2015, including suits involving patents related to software or otherwise. 

This indicates that the increase in suits in Eastern Texas drove the overall 

increase in suits filed nationwide in 2015.

The number of defendants in patent infringement litigation was relatively 

stable between 2007 and 2015, with the exception of lawsuits involving 

patents for computers and communications technologies (see fig. 4).37

This area includes technologies related to computer hardware and 

software. The percentage of defendants in lawsuits involving patents in 

this technology area increased from 38 percent in 2007 to 62 percent in 

2015.

 36Courts may stay infringement litigation brought by a patent owner pending the outcome 

of a PTAB proceeding. One stakeholder mentioned Delaware as another district where it 

was less likely for judges to grant a stay for a PTAB proceeding. 

37These technology categories are used by USPTO and are described in Hall, Jaffe, 

Trajtenberg, The NBER Patent Citations Data File: Lessons, Insights and Methodological 

Tools. National Bureau of Economic Research 2001.
Page 19 GAO-16-490 Intellectual Property

Figure 4: Number of Defendants in Patent Infringement Lawsuits by Technology 

Area, 2007 through 2015

Note: This represents the number of defendants in suits where there was at least one patent in the 

given technology area; therefore, suits involving patents from multiple technology areas would be 

represented in each of these technology areas.

Patents related to computer and communications technologies are easier 

to unintentionally infringe because they are more likely to be unclear and 

overly broad, according to some stakeholders we interviewed and some 

of the published research we reviewed. In addition, according to a 2003 

Federal Trade Commission study, in industries such as computers and 

communications, firms need to avoid infringing dozens, hundreds, or even 

thousands of patents to produce just one commercial product.38

 38Federal Trade Commission, To Promote Innovation: The Proper Balance of Competition 

and Patent Law and Policy (2003).
Page 20 GAO-16-490 Intellectual Property

Therefore, it is particularly challenging to develop innovative new 

products where there are thousands of interrelated patents covering 

similar technologies making it nearly impossible to avoid infringement, 

according to some stakeholders we interviewed. These stakeholders also 

noted that unclear and overly broad patents—an indicator of lower 

quality—can harm innovation regardless of whether the patent owner files 

an infringement lawsuit because just the threat of an infringement suit can 

deter the development of new products.39

Further, the majority of defendants in patent infringement suits were 

involved in suits with software-related patents each year from 2009 

through 2015 (see fig. 5).40 The number of defendants involved in 

infringement lawsuits where software-related patents were asserted 

generally increased through 2013, then dropped in 2014. As with 

computer and communications technologies, some stakeholders told us 

that software-related patents are easier to infringe because they also 

often have overly broad claims, an indicator of low quality patents.

 39The author of one paper argues that mere possession of a patent that would likely not 

survive a legal challenge can help maintain an illegitimate monopoly even if the patentholder does not use the patent to file an infringement suit. See Leslie, Christopher. The 

Anticompetitive Effects of Unenforced Invalid Patents, 91 Minn. L. Rev. 101 (Nov. 2006)

40Software-related patents occur in a variety of technologies containing at least some 

element of software, and cover things like sending messages or conducting business over 

the Internet (e.g., e-commerce). According to USPTO officials, USPTO does not have a 

definition of or collect information on “software-related patents.” Our definition of 

“software-related” is based on the patent classes listed in S. Graham, and S. 

Vishnubhakat, Of Smart Phone Wars and Software Patents, Journal of Economic 

Perspectives, v. 27 no.1 (2013), pp. 67-86.
Page 21 GAO-16-490 Intellectual Property

Figure 5: Number of Defendants in Patent Infringement Lawsuits for SoftwareRelated and Non-Software-Related Patents, 2007 through 2015

USPTO has taken actions to address patent quality—most notably 

through its Enhanced Patent Quality Initiative—but there are additional 

opportunities for USPTO to improve patent quality. Specifically, USPTO 

does not have a consistent definition of patent quality and has not fully

developed specific performance measures to assess whether its efforts 

are affecting patent quality. Further, USPTO has not fully assessed 

whether the time it allots for examination and the monetary incentives it 

gives examiners for completing patent examinations faster affect patent 

quality. In addition, USPTO has limited data available on PTAB decisions. 

Finally, USPTO has not fully evaluated the effects of other policies and 

procedures on patent quality.

USPTO does not currently have a consistent definition for patent quality, 

which may limit its ability to assess the effects of its examination policies 

and review processes—as well as its Enhanced Patent Quality Initiative—

on patent quality. Several high level USPTO officials and the four 

supervisory patent examiners that we interviewed told us there is no 

consistent definition of patent quality at USPTO to guide the agency in its 

overall operations. One examiner wrote in our open-ended survey 

Additional 

Opportunities Exist to 

Improve Patent 

Quality at USPTO

USPTO Has Not 

Consistently Defined 

Patent Quality or 

Established Specific 

Performance Measures
Page 22 GAO-16-490 Intellectual Property

response that the USPTO appears to have no definition of patent quality 

and that without a working definition, management’s focus on patent 

quality is meaningless.41 Most of the stakeholders that we spoke with—

including former high ranking USPTO officials, academics, and nongovernmental organizations—said that it is important for USPTO to 

develop a consistent definition of patent quality.

According to USPTO officials, one challenge in developing a consistent 

definition is that the patent community holds varying definitions of patent 

quality. For example, patent attorneys who defend companies against 

patent infringement generally tend to favor clearly defined patents that are 

easily understood in patent disputes, while patent owners tend to prefer 

less well defined patents that may offer broader coverage for their 

invention. USPTO officials offered a variety of definitions of patent quality. 

Several officials focused on validity, or patentability (i.e., meeting 

statutory requirements) and clarity, although we could not find this in 

agency guidance or documents as USPTO’s definition. Some officials 

also included aspects of patent examination, such as an application that 

requires a limited amount of time for review by examiners in their 

definition. While USPTO has found it difficult to clearly define patent 

quality, most of the stakeholders we spoke with told us that they would 

define patent quality as patent validity—that is, a quality patent would 

meet all the statutory requirements for patentability and would be upheld 

if challenged in a lawsuit or PTAB proceeding.

According to federal standards for internal control, one important internal 

control activity is that management has established clear, consistent 

agency objectives that allow the agency to assess the risks the agency 

faces from external and internal sources.42 However, USPTO’s objectives 

 41Our survey did not ask examiners about how well USPTO had defined patent quality. 

However, examiners were free to bring up any issue in open-ended responses and a few 

raised concerns about the USPTO’s not having a definition for patent quality, saying it 

makes it difficult for examiners to understand what is meant by patent quality or to 

understand how to change their work behavior to improve patent quality. Because we did 

not conduct a systematic review of all open-ended responses to our survey, we are not 

reporting the exact number of examiners who provided a response on this topic. 

Responses to open-ended questions are not generalizable to other examiners.

42GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999). GAO has revised and reissued Standards for 

Internal Control in the Federal Government, with the new revision effective as of October 

1, 2015. GAO-14-704G (Washington, D.C.: September 2014).
Page 23 GAO-16-490 Intellectual Property

may not be clear because it does not have a consistent definition of 

patent quality. Four supervisory patent examiners we interviewed told us 

that without a consistent definition of patent quality, USPTO is unable to 

standardize practices to improve patent quality. As a result, it is hard for 

USPTO to define, measure, and work toward quality goals, according to 

these supervisors. Moreover, without a definition for patent quality, 

USPTO is at risk of having agency officials work inconsistently or at 

cross-purposes in their attempts to fulfill the call to improve patent quality, 

based on individual understandings of what patent quality means to each 

staff person.

Further, although USPTO is taking steps to improve patent quality metrics 

as part of the Enhanced Patent Quality Initiative, it has not established 

specific goals or performance measures related to its strategic goal to 

optimize patent quality and timeliness, which may limit its ability to assess 

potential effects of its efforts on patent quality. USPTO’s 2014-2018 

strategic plan includes the goal to “optimize patent quality and timeliness,”

but the patent quality objective does not include specific performance 

measures that fully assess progress towards the goal.43 For example, 

USPTO names seven objectives to achieve this goal, but six of the seven 

objectives focus on timeliness, customer service, and process or 

production goals rather than patent quality. For the one patent quality 

objective, USPTO cites improving patent quality data and maximizing the 

use of such data as two of its four performance measures (see fig. 6).

 43USPTO 2014-2018 Strategic Plan, US Patent and Trademark Office.
Page 24 GAO-16-490 Intellectual Property

Figure 6: U.S. Patent and Trademark Office (USPTO) Strategic Goal Related to 

Patent Quality and Timeliness

The Government Performance and Results Modernization Act of 2010 

(GPRAMA) requires, among other things, that agencies establish 

objective, quantifiable, and measurable performance goals, and establish 

performance indicators to measure progress toward each performance 

goal.44 Although USPTO is not required to comply with GPRAMA, we 

have previously reported the practices established by the law, such as 

establishing goals and performance indicators, can serve as a leading 

practice for organizational performance management at lower levels 

within federal agencies, such as individual programs or initiatives.45

However, USPTO has not established such goals and indicators because 

the office uses general terms to describe these two performance 

 44Pub. L. No. 103-62 (1993), amended by GPRA Modernization Act of 2010, Pub. L. No. 

111-352 (2011). These laws require federal agencies to set goals for program 

performance, measure results, and report on annual performance compared with the 

goals.

45See, for example, GAO, Environmental Justice: EPA Needs to Take Additional Actions 

to Help Ensure Effective Implementation, GAO-12-77 (Washington, D.C.: Oct. 6, 2011) 

and GAO, Great Lakes Restoration Initiative: Further Actions would Result in More Useful 

Assessments and Help Address Factors that Limit Progress, GAO 13-797 (Washington, 

D.C.: Sep 27, 2013).
Page 25 GAO-16-490 Intellectual Property

measures and does not include measurable, quantifiable goals or 

performance indicators on how the agency plans to define or collect 

patent quality data or how, specifically, it will maximize use of such data. 

Without such goals and indicators, USPTO cannot determine whether it is 

meeting its goal of enhancing patent quality.

USPTO has not fully assessed the time allotted for patent examinations 

or its monetary incentive system, which may be at odds with patent 

quality. Patent examiners are rated annually on their production and 

docket management, among other elements.46 USPTO provides 

examiners with monetary incentives, or bonuses, for timeliness and 

production, but does not offer a bonus for producing high-quality work.47

Three USPTO officials told us that there are trade-offs between timeliness 

and patent quality, explaining that examiners cannot examine patents 

quickly and, at the same time, grant patents that are of the highest 

quality. One of these officials told us that the office’s focus on timeliness 

currently trumps high quality work at the agency, potentially increasing 

the tension between the goals of timeliness and quality.

Time allotments and incentives can lead to pressure for examiners to 

complete their work quickly. Most of the stakeholders we spoke with told 

us that examiners’ time pressures are one of the central challenges for 

patent quality.48 The results of our survey of patent examiners confirm 

that examiners are experiencing time pressures. Specifically, on the basis 

of our survey, we estimate that, given a typical workload, about 70 

percent of examiners have less time than needed to complete a thorough 

 46Factors related to quality constitute 35 percent of an examiner’s overall rating.

47Between fiscal years 2009 and 2013, examiners received, on average, more than 

$6,000 per year in monetary awards for meeting production and timeliness goals, 

according to a 2015 Inspector General report. Office of the Inspector General, U.S. 

Department of Commerce: U.S. Patent and Trademark Office: USPTO Needs to 

Strengthen Patent Quality Assurance Practices, OIG-15-026-A. (Washington, D.C.: April 

2015).

48In our prior art report, we provide additional information on how the time pressures that 

examiners face are a challenge for conducting thorough prior art searches. See 

GAO-16-479.

USPTO Has Not Fully 

Assessed the Time 

Allotted for Examinations 

or Monetary Incentives
Page 26 GAO-16-490 Intellectual Property

examination.49 In addition, we estimate that more than 70 percent of 

examiners worked voluntary or uncompensated overtime in the past 6 

months to meet their minimum production goals (see fig. 7).50 Further, 

incentives appear to motivate examiners to complete their work quickly. 

For example, on the basis of our survey, we estimate that for 93 percent 

of examiners, receiving bonuses for achieving production goals motivates 

them to go above and beyond their base level of performance.51

 49Based on our survey, we estimate all examiners’ responses would be as follows: much 

more time than needed, 4 percent; somewhat more time than needed, 7 percent; about as 

much time as needed, 19 percent; somewhat less time than needed, 43 percent; much 

less time than needed, 27 percent; don’t know, less than 1 percent; no response, 1 

percent.

50Examiners may have worked further overtime not included in their responses.

51We asked examiners how motivated they are by receiving bonuses for achieving 

production goals—specifically, their motivation to go above and beyond a “fully 

successful” rating on their Performance Appraisal Plans. Based on our survey, we 

estimate all examiners’ responses would be as follows: very motivated, 53 percent; 

moderately motivated, 19 percent; somewhat motivated, 13 percent; slightly motivated, 8 

percent; not at all motivated, 5 percent; don’t know, less than 1 percent; no response, 1 

percent. 
Page 27 GAO-16-490 Intellectual Property

Figure 7: Estimated Bi-weekly Overtime Worked by Patent Examiners in the Past 6 

Months

Notes: We asked examiners about voluntary/uncompensated overtime worked to meet their minimum 

production goals. Examiners may have worked further overtime not included in their responses.

Because we surveyed a generalizable stratified random sample of examiners, our results provide 

estimates for the entire population of examiners included in our study. See the related e-supplement, 

GAO-16-478SP, for the original survey language and more detailed results.

Additionally, on the basis of our survey, we estimate that nearly 70 

percent of examiners experience pressure to avoid time-consuming office 

actions.52 In addition, a few examiners we interviewed said that the 

system as currently designed incentivizes an examiner to issue a patent 

instead of issuing a final rejection, suggesting that when pressed for time 

examiners tend toward granting patents. For example, USPTO rules and 

 52Based on our survey, we estimate all examiners’ responses would be as follows: a lot of 

pressure, 17 percent; moderate pressure, 18 percent; some pressure, 20 percent; a little 

pressure, 13 percent; no pressure, 26 percent; don’t know, 4 percent; and no response, 2 

percent.
Page 28 GAO-16-490 Intellectual Property

procedures do not restrict the number of requests for continued 

examination an applicant may file, so issuing the patent is the only means 

USPTO has to end the examination process. This creates an environment 

where patents may be granted that do not fully meet patentability 

standards. In a 2015 study, USPTO economists found that patents 

examined by primary examiners had 26 percent better odds of appearing 

in a patent infringement lawsuit compared to similar patents examined by 

junior examiners, which suggests that primary examiners, despite their 

experience, may not have adequate time to ensure that the patents they

issue are always high quality.53 USPTO officials acknowledged that this 

difference was likely due to the additional time and supervisory review 

that junior examiners receive.

The precise effects of the time allotted for examinations and incentives on 

quality are unclear because USPTO has not fully analyzed the effects of 

current time allotments or incentives on an examiner’s ability to perform a 

thorough examination. Since creating the time allotments in the 1970s, 

USPTO adjusted the time allotted to examiners between fiscal years 2010 

and 2012 and gave all patent examiners a total of 2.5 additional hours per 

application.54 According to federal standards for internal control, agencies 

should provide staff with the right structure, incentives, and 

responsibilities to make operational success possible.55 The right 

incentives and structure allow staff to be aligned with the agency’s 

objectives. Without analyzing the time and incentives needed for 

examiners to complete thorough examinations, USPTO cannot be 

assured that its current time allotments and incentives support the 

agency’s goal to optimize patent quality.

 53These were patents that were assigned to primary examiners at the time of allowance. 

See: USPTO Office of the Chief Economist, Patent Litigation and USPTO Trials: 

Implications for Patent Examination Quality (January 2015). USPTO officials told us that 

applications of varying levels of difficulty or complexity are assigned without regard to the 

examiner’s experience, and that there should not be any correlation between patent 

examination factors and the incidence of litigation.

54In addition, in April 2016, USPTO also adjusted the time allotted for approximately 1,000 

examiners, who received an additional 2.7 hours for examinations of certain technologies 

to address concerns related to a transition to a new classification system.

55GAO/AIMD-00-21.3.1.
Page 29 GAO-16-490 Intellectual Property

The PTAB has been in operation since 2012 with hundreds of decisions 

made to date on challenges to existing patents; however, statistics kept 

by PTAB staff about the results of PTAB decisions are limited.56 For 

example, the data available as of March 2016, did not specify which 

precise claims in a patent were found to be unpatentable and why, or 

which sources of prior art were used in the proceeding, both of which are 

key data fields for potential analysis.57 Moreover, the data have not been 

widely shared within USPTO. According to a PTAB official familiar with 

the data, only a few USPTO staff in the Office of Patent Legal 

Administration have asked for and received the data that PTAB staff have 

compiled to date. Information on the results of the PTAB proceedings is 

not regularly provided to USPTO managers and supervisory patent 

examiners in the Technology Centers, according to PTAB staff.

According to federal standards for internal control, information from 

internal and external sources should be obtained and provided to 

management as a part of the agency’s reporting on operational 

performance relative to established objectives. In addition, pertinent 

information should be identified, captured, and distributed in a form and 

time frame that permits people to perform their duties efficiently.58

However, USPTO managers and staff in the technology centers do not 

have data on trends in the outcomes of PTAB’s post-grant proceedings 

because USPTO has not systematically pulled information from PTAB 

decisions, has not widely shared PTAB information within the agency, 

and has not analyzed the data for trends in patent invalidations to identify 

whether additional training, guidance, or other actions are needed to 

address issues related to quality. Without a process for USPTO 

managers and staff to readily access the reasoning underlying PTAB’s 

invalidation of patents or information on possible trends that may exist 

with those problem patents, USPTO may be overlooking critical 

information on problems the PTAB has found with patents issued by the 

agency. As a result, USPTO may be missing opportunities for using this 

 56PTAB data and some statistics summarizing the decisions (holdings) on these cases are 

available on the USPTO website. See: http://www.uspto.gov/patents-applicationprocess/appealing-patent-decisions/statistics/aia-trial-statistics

57Outcome information is available on the PTAB website, but it requires reading individual 

decisions one at a time to determine which claims were found to be unpatentable.

58GAO/AIMD-00-21.3.1.

USPTO Has Limited Data 

on PTAB Decisions
Page 30 GAO-16-490 Intellectual Property

information to make decisions that could help to improve patent quality, 

such as improving guidance or providing additional training on potential 

patent quality issues, and USPTO’s examiners are unable to learn 

lessons from the PTAB decisions.

USPTO has not fully evaluated the effects of other policies and 

procedures on patent quality, which may also affect their ability to issue 

high quality patents. Through interviews with USPTO officials and 

stakeholders and our examiner survey, we identified several policies and 

procedures that could affect patent quality:

• Compact prosecution: This USPTO policy encourages examiners to 

complete an examination within two office actions and to address all 

statutory issues of an application in the first office action. One 

examiner commented in our survey that compact prosecution compels 

examiners to guess what unclear claims mean in order to search for 

prior art.59 Further, based on our survey, we estimate that a change in 

policy that would not require examiners to address every issue in the 

first office action would help about half of USPTO examiners perform 

examinations more effectively.60 In addition, some stakeholders raised 

concerns about the effect of compact prosecution on patent quality, 

with one stakeholder emphasizing that the policy does not work well in 

an environment where patent applications are increasingly complex.

• Unlimited number of patent claims: Applicants are allowed to include 

any number of claims in a patent application.61 According to USPTO 

 59Examiners were free to bring up any issue in their open-ended responses and a few

raised concerns about compact prosecution and the effects on patent quality. Because we 

did not conduct a systematic review of all open-ended responses to our survey, we are not 

reporting the exact number of examiners who provided a response on this topic. 

Responses to open-ended questions are not generalizable to other examiners.

60Based on our survey, we estimate all examiners’ responses would be as follows: much 

more effectively, 24 percent; somewhat more effectively, 24 percent; neither more or less 

effectively, 22 percent; somewhat less effectively, 14 percent; much less effectively, 8 

percent; don’t know, 6 percent; and no response, 2 percent. 

61Applicants are charged additional fees for more claims, but examiners are given the 

same amount of time for each examination, regardless of the number of claims. 

Supervisory patent examiners have the authority to grant other time to examiners for 

reviewing applications that have a certain level of complexity, including applications with a 

large number of claims.

USPTO Has Not Fully 

Evaluated the Effects of 

Other Policies and 

Procedures on Patent 

Quality
Page 31 GAO-16-490 Intellectual Property

officials, applicants’ ability to file unlimited claims can have a negative 

effect on quality, because it is more difficult for examiners to fully 

review an application with numerous claims in the time allotted; this 

was also supported by our survey results.62 The 2015 study by 

USPTO economists also supported this finding; it found that patents 

that had more claims had better odds of showing up in patent 

infringement lawsuits.63

• Requests for continued examination: Applicants are currently allowed 

to file an unlimited number of requests for continued examination, 

which is a request by an applicant to reopen examination of the patent 

application after the prosecution of the application has been closed.64

Applicants request continued examination most often after final 

rejection of an application, according to USPTO officials. Such 

requests provide applicants with virtually unlimited attempts to secure 

a patent, which is problematic for patent quality, according to some 

 62Based on our survey, we estimate all examiners’ responses to the question: “[h]ow 

much easier or more difficult do [a] larger number of dependent claims than needed to 

describe the invention make it to complete a thorough examination?” would be as follows: 

much easier, less than 1 percent; somewhat easier, 1 percent; neither easier nor more 

difficult, 5 percent; somewhat more difficult, 34 percent; much more difficult, 60 percent; 

don’t know, less than 1 percent; and no response, less than 1 percent. Additionally, based 

on our survey, we estimate all examiners’ responses to the question “Would a change to 

limit the number of dependent claims in an application result in you performing 

examinations more or less effectively?” would be as follows: much more effectively, 59 

percent; somewhat more effectively, 33 percent; neither more nor less effectively, 5 

percent; somewhat less effectively, 1 percent; much less effectively, less than 1 percent; 

don’t know, 1 percent; and no response, less than 1 percent.

63There was a 6 percent increase in the odds of a patent appearing in an infringement suit 

for each additional independent claim. USPTO, Office of the Chief Economist, Patent 

Litigation and USPTO Trials: Implications for Patent Examination Quality (January 2015).

64Applicants must make a new submission and pay a fee to USPTO for filing a request for 

continued examination. 37 C.F.R. § 1.114. USPTO received about 111,000 requests for 

continued examination in fiscal year 2015; about 11,000 of those applications were the 

third such request for continued examination, according to data provided by USPTO 

officials.
Page 32 GAO-16-490 Intellectual Property

stakeholders.65 Some stakeholders also told us that such unlimited 

requests can wear down examiners, making them more likely to 

eventually grant the patent.66 While USPTO has reached out to patent 

applicants to learn why they were using such continued examinations, 

the number of requests for continued examination continues to pose a 

burden for the agency’s examiners.67

Federal standards for internal control direct agencies to comprehensively 

identify risks and consider all significant interactions between the entity 

and other parties. Once these risks have been identified, they should be 

evaluated for their potential effects, including the significance of the risks 

and the likelihood of their occurrence.68 USPTO officials acknowledged 

that some of the agency policies discussed above could affect patent 

quality; however, officials did not know the extent of the effects because 

USPTO has not done an evaluation to determine the potential effects. 

Without evaluating the effects of these policies on patent quality, USPTO 

is at risk of continuing practices that may adversely affect patent quality.

Additionally, USPTO policies and procedures generally require clarity in 

issued patents. On the basis of our survey, we estimate that additional 

claim clarity requirements to applicants would help more than 80 percent 

 65In 2007, USPTO amended the regulations to revise the rules of practice in patent cases 

relating to continuing applications and requests for continued examination practices and 

for examination of claims for applications containing more than 25 claims, among other 

things. Changes to Practice for Continued Examination Filings, Patent Applications 

Containing Patently Indistinct Claims, and Examinations of Claims in Patent Applications, 

72 Fed. Reg. 46,716 (Aug. 21, 2007). A number of large patent holders challenged the 

new regulations in court. A panel of the U.S. Court of Appeals upheld portions of the 

regulation, and struck down the portion of the regulation relating to continuing applications 

and requests for continued examination. See Tafas v. Doll, 559 F.3d 1345 (Fed.Cir.2009). 

Following the granting of an en banc rehearing, the opinion was vacated and PTO 

withdrew the regulation. 

66In addition, some stakeholders told us that applicants can broaden the scope of their 

patent by using continuing examination to encompass new technologies that they did not 

actually invent.

67Request for Comments on Request for Continued Examination (RCE) Practice, 77 Fed. 

Reg. 72830 (Dec. 6, 2012).

68GAO/AIMD-00-21.3.1.
Page 33 GAO-16-490 Intellectual Property

of examiners do their jobs more effectively.69 We identified two areas 

where USPTO has encountered challenges with claim clarity in patent 

applications that can affect patent quality: (1) unclear terms, and (2) 

unclear or broadly worded claims including the use of functional claim 

language.

First, patent applications that include unclear terms contribute to 

challenges with patent quality. On the basis of our survey, we estimate 

that 45 percent of examiners always or often encounter terms that are not 

well defined in the patent applications’ specifications.70 Most of the 

stakeholders we interviewed—including legal scholars and former highranking USPTO officials—as well as four supervisory patent examiners 

and the majority of examiners responding to our survey—indicated that 

requiring applicants to provide a glossary and define their terms would 

help to improve patent quality.71 Further, USPTO officials said that 

examiners and applicants that participated in USPTO’s Glossary Pilot 

 69Based on our survey, we estimate all examiners’ responses would be as follows: much 

more effectively, 41 percent; somewhat more effectively, 41 percent; neither more nor less 

effectively, 12 percent; somewhat less effectively, 1 percent; much less effectively, less 

than 1 percent; don’t know, 4 percent; and no response, 1 percent. 

70Based on our survey, we estimate all examiners’ responses would be as follows: 

always, 8 percent; often, 37 percent; sometimes, 42 percent; rarely, 13 percent; never, 

less than 1 percent; don’t know, less than 1 percent; and no response, less than 1 

percent. These percentages were even higher for the portion of the technology center that 

focuses on business methods, an area which includes software-related patents. In that 

technology center, 67 percent of examiners always or often encounter terms that are not 

well-defined in the specification. Based on our survey, we estimate all examiners’

responses would be as follows: always, 18 percent; often, 49 percent; sometimes, 30 

percent; rarely, 3 percent; never, 0 percent; don’t know, 1 percent; and no response, 0 

percent.

71Based on our survey, we estimate all examiners’ responses would be as follows: much 

more effectively, 21 percent; somewhat more effectively, 38 percent; neither more nor less 

effectively, 29 percent; somewhat less effectively, 3 percent; much less effectively, 2 

percent; don’t know, 6 percent; and no response, 1 percent.
Page 34 GAO-16-490 Intellectual Property

Program generally indicated benefits to including a glossary and that the 

glossary improved claim clarity.72

Second, patent applications that include unclear or broadly worded 

claims, including those that use functional claim language, contribute to 

challenges with patent quality.73 For example, on the basis of our survey, 

we estimate that nearly 90 percent of examiners always or often 

encountered broadly worded claims in applications they reviewed,74 and 

for nearly two-thirds of examiners, applications with broadly worded 

claims make completing a thorough examination more difficult.75 In 

addition, dealing with functional claims, especially for software-related 

patents, can be time consuming and difficult especially if examiners are 

not aware of the applicant’s intent to use functional claiming language 

under section 112, according to a few examiners we interviewed. On the 

basis of our survey, we estimate that more than 40 percent of examiners 

experience pressure to avoid making rejections that relate to claim clarity 

(section 112).76 Also, according to our survey data, we estimate that for a 

smaller percentage of patent examiners (41%) correctly applying 

 72In response to a series of executive actions to improve claim clarity, USPTO launched a 

Glossary Pilot Program, which was held from June 2014 to November 2015. As part of the 

voluntary program, USPTO provided guidance that applicants should include definitions 

that assist in clarifying the claimed invention, and allowed participants to select which 

terms to define and how best to define the selected terms. In March 2016, USPTO officials 

said that their analysis thus far of the pilot program did not find significant differences in 

pilot and non-pilot applications’ initial quality review scores, but that this analysis would 

continue as the glossary pilot applications progress through the USPTO examination 

process.

73On the basis of our survey, we found a statistically significant relationship between the 

amount of time examiners needed to do a thorough examination and the extent to which 

applications they received were clearly written and understandable.

74Based on our survey, we estimate all examiners’ responses would be as follows: 

always, 40 percent; often, 47 percent; sometimes, 11 percent; rarely, 1 percent; never, 0 

percent; don’t know, less than 1 percent; and no response, less than 1 percent.

75Based on our survey, we estimate all examiners’ responses would be as follows: much 

easier, 2 percent; somewhat easier, 14 percent; neither easier nor more difficult, 20 

percent; somewhat more difficult, 33 percent; much more difficult, 30 percent; don’t know, 

less than 1 percent; and no response, less than 1 percent.

76Based on our survey, we estimate all examiners’ responses would be as follows: a lot of 

pressure, 7 percent; moderate pressure, 9 percent; some pressure, 14 percent; little 

pressure, 13 percent; no pressure, 53 percent; don’t know, 3 percent; and no response, 2 

percent. 
Page 35 GAO-16-490 Intellectual Property

patentability standards for dealing with functional claims (section 112(f)) 

was very important compared to the percentage who reported adhering to 

novelty and nonobviousness standards (sections 102 and 103) was very 

important (81%).77 In a 2015 study, the USPTO economists found that 

patents with broader claims—patents with fewer words in their claims—

had better odds of appearing in patent infringement lawsuits than similar 

patents.78 Further, that analysis found that patents that contained 

functional claim language had 40 percent better odds of appearing in 

patent infringement lawsuits than similar patents that did not contain this 

language.

Some stakeholders we interviewed said requiring “claim charts” in patent 

applications could help improve patent quality. Claim charts use one 

column to present the claim and another column to present the limitations 

and boundaries of that claim (see table 1). These charts, which are 

commonly used in the federal courts and in PTAB’s post-grant 

proceedings, provide additional information on the boundaries of a patent 

claim. Similarly, according to some stakeholders, having applicants 

include a functional claim check box to indicate whether they were using 

functional claim language under Section 112(f) could help to improve 

 77These responses were consistent across technology centers. Based on our survey, we 

estimate all examiners’ responses to the question:”[h]ow important is the proper 

application of Section 112(f) to an examiner’s advancement at USPTO?” would be as 

follows: very important, 41 percent; moderately important, 25 percent; somewhat 

important, 17 percent; slightly important, 9 percent; not important, 3 percent; don’t know, 5 

percent; and no response, 2 percent. Based on our survey, we estimate all examiners’

responses to the question: “[h]ow important is the proper application of Section 102 to an 

examiner’s advancement at USPTO?” would be as follows: very important, 81 percent; 

moderately important, 11 percent; somewhat important, 4 percent; slightly important, 1 

percent; not important, 1 percent; don’t know, 2 percent; and no response, 1 percent. 

Based on our survey, we estimate all examiners’ responses to the question: “[h]ow 

important is the proper application of Section 103 to an examiner’s advancement at 

USPTO?” would be as follows: very important, 81 percent; moderately important, 11 

percent; somewhat important, 4 percent; slightly important, 1 percent; not important, 1 

percent; don’t know, 2 percent; and no response, 1 percent.

78For example, patents with 50 fewer words had 10 percent greater odds of appearing in 

an infringement suit than otherwise similar patents. According to the study, as the number 

of words per claim decreases, the boundary of the patent’s property right is less well 

defined. The less clear the claim boundaries are, the more likely that others will infringe 

the patent or will continue to infringe when confronted by the patent owner. Uncertainty 

about the outcome of a trial occurs for many other reasons than just claim interpretation. 

See: USPTO Office of the Chief Economist, Patent Litigation and USPTO Trials: 

Implications for Patent Examination Quality (January 2015). 
Page 36 GAO-16-490 Intellectual Property

patent quality.79 If the box is checked it indicates that the examiner should 

make sure that the functional claim is supported in the patent 

specification.80

Table 1: Example of a “Claim Chart”

Patent #1,234,567

Text from Claim 1 Interpretation

“a unitary molded plastic

package-saving device”

A “unitary molded” device “is a device that consists of a 

part or parts that were formed from a mold, and it 

functions as a unit or single piece.”

Source: Adapted from “Anatomy of a Patent Case” Federal Judicial Center, 2009. I GAO-16-490

By statute, an application for a patent must particularly point out and 

distinctly claim the subject matter of the invention.81 USPTO regulations 

require that the application include a description of the process of making 

and using the invention in such full, clear, concise, and exact terms as to 

enable a person skilled in the art to use and make the invention.82

However, examiners continue to encounter problems with patent 

application clarity because USPTO does not specifically require patent 

applicants to clearly define the terms used in their applications, provide 

additional means to clearly describe claim boundaries, or clearly identify 

when they are using functional claiming language. Without making use of 

tools to improve the clarity of patent applications, such as by having 

applicants include a glossary to define the terms used in the application, 

provide a claim chart, or indicate the use of functional claims through a 

checkbox, the agency is at risk of issuing unclear patents that may not 

comply with statutory requirements.

 79See Menell, Peter S. Promoting Patent Claim Clarity. UC Berkeley Public Law Research 

Paper No. 2171287. Nov. 2012. 

80Further, a former high ranking USPTO official said that requiring the use of tools such as 

glossaries and check boxes would improve the quality and clarity of patents, and would 

also clarify the meaning of the patent if it ever appeared in court. In addition, a senior 

USPTO official told us in March 2016 that having applicants include a glossary, a claim 

chart, and a claim check box with each patent application would be beneficial for 

examiners to do their work and would help to improve patent clarity. 

8135 U.S.C. § 112(b).

8237 C.F.R. § 1.71.
Page 37 GAO-16-490 Intellectual Property

The U.S. patent system plays a vital role in our nation’s economy by 

promoting innovation and supporting millions of jobs in innovation-rich 

sectors. However, the quality of patents issued by the USPTO has come 

under scrutiny in recent years, and our analysis suggests that some 

agency policies and practices may negatively affect the quality of the 

patents USPTO awards. It has taken several actions to elevate the 

importance of patent quality within the agency, but additional 

opportunities exist to improve the effectiveness of the agency’s patent 

quality efforts.

USPTO does not have a consistent definition of patent quality that is 

clearly articulated in agency guidance or fully developed measurable 

goals and performance indicators to guide and evaluate work towards the 

agency’s quality goals. Without a consistent definition of patent quality, 

USPTO is at risk of having its staff work at cross purposes to improve 

patent quality based on their individual definitions of patent quality. 

Further, without improvements to measurable goals and performance 

indicators, USPTO is at risk of not being able to fully measure and 

capture key performance data on whether the agency is meeting its 

strategic goal to optimize patent quality.

USPTO’s policies regarding the time allotted to complete patent 

application reviews and monetary incentives that are based on the 

quantity of the work examiners complete, not the quality of their work may 

negatively affect the quality of issued patents. However, USPTO has not 

analyzed its policies regarding examiner performance incentives and has 

assessed the time allotted for patent examination only minimally since the

1970s. Without analyzing whether time allotments to complete a thorough 

examination are sufficient, USPTO is at risk of issuing lower quality 

patents due to examiners’ not having enough time to complete their work. 

Our report on prior art searching recommended that USPTO reassess the 

time allotted to perform a thorough prior art search.83 Without analyzing 

the current incentive structure, USPTO cannot ensure that its incentives 

are aligned with high-quality work.

Further, USPTO has limited data available on decisions from the PTAB to 

assess its patent quality initiatives. The PTAB has reviewed patents and 

 83GAO-16-479.

Conclusions
Page 38 GAO-16-490 Intellectual Property

invalidated some, or all, of the claims for hundreds of patents issued by 

the USPTO. However, the PTAB data and analysis of these decisions are 

limited in specificity as well as in distribution. Without identifying all of the 

data fields that would be useful to track, establishing a means for USPTO 

managers and examiners to easily access information—including 

information such as why claims were found to be unpatentable—and 

analyzing the data to identify potential trends, USPTO officials may be 

overlooking key information that could help them provide additional 

training or guidance or take other actions to address recurring issues.

Further, other USPTO policies and procedures—namely, compact 

prosecution, unlimited numbers of patent claims, and requests for 

continued examination—may affect patent quality, but the agency has not 

evaluated the effects of these policies. Without evaluating the possible 

effects of these policies on patent quality, USPTO is at risk of continuing 

practices that could potentially affect patent quality.

Finally, without requiring greater clarity in applications—through the use 

of glossaries, check boxes to signal functional claiming language, or claim 

charts—USPTO is at risk of issuing patents that are overly broad and not 

clearly worded—and may not comply with statutory requirements—

thereby increasing the likelihood that the patent becomes the subject of 

litigation.

We recommend that the Secretary of Commerce direct the Director of the 

USPTO to take the following seven actions to help improve patent quality:

• Develop a consistent definition of patent quality, and clearly articulate 

this definition in agency documents and other guidance.

• Further develop measurable, quantifiable goals and performance 

indicators related to patent quality as part of the agency’s strategic 

plan.

• Analyze the time examiners need to perform a thorough patent 

examination. This action could be taken in conjunction with the 

recommendation in our report on USPTO’s prior art search 

capabilities (GAO-16-479).

• Analyze how current performance incentives affect the extent to which 

examiners perform thorough examinations of patent applications.

• Establish a process to provide data on the results of the PTAB 

proceedings to managers and staff in the USPTO’s Technology 

Recommendations for 

Executive Action
Page 39 GAO-16-490 Intellectual Property

Centers, and analyze PTAB data for trends in patent quality issues to 

identify whether additional training, guidance, or other actions are 

needed to address trends.

• Evaluate the effects of compact prosecution and other agency 

application and examination policies on patent quality. In doing so, 

USPTO should determine if any changes are needed to ensure that 

the policies are not adversely affecting patent quality.

• Consider whether to require patent applicants to include claim clarity 

tools—such as a glossary of terms, a check box to signal functional 

claim language, or claim charts—in each patent application.

We provided a copy of our draft report to USPTO for review and 

comment. In its written comments, which are reproduced in appendix III, 

USPTO generally agreed with our findings, concurred with our 

recommendations, and provided information on steps officials plan to take 

to implement the recommendations. USPTO also provided additional 

technical comments, which we incorporated, as appropriate.

USPTO stated in its response to the first recommendation that USPTO 

already has a consistent definition for patent quality, specifically that a 

quality patent is one that is correctly issued in compliance with all of the 

requirements of Title 35 as well as relevant case law at the time of 

issuance, which is consistent with how we define the term for this report. 

However, in our audit work, we did not find evidence that this definition 

was clearly articulated in agency documents and guidance or used in its 

performance indicators and goals. We revised this recommendation to 

clarify that USPTO should not only define the term, but also make this

definition clear in relevant documents. 

In response to our second recommendation, USPTO said that it has 

taken some steps to update and improve its performance indicators and 

goals related to patent quality. In its technical comments, USPTO 

suggested that we revise the report to recommend that USPTO further 

develop its goals and performance indicators. We agree with this 

suggestion and made the change. As USPTO further develops its goals 

and performance indicators, we encourage the agency to more clearly 

link these goals and indicators to its definition of patent quality.

In response to our seventh recommendation, USPTO said that, contrary

to the draft report’s findings, USPTO’s initial conclusion was that a 

glossary did not make a meaningful difference in quality during the 

prosecution of an application, though the USPTO is still analyzing 

Agency Comments
Page 40 GAO-16-490 Intellectual Property

whether the use of a glossary has a long-term impact on a patent. In 

response to this comment, we revised the statement in the report to more 

closely align with the information that USPTO officials presented at its 

March 8, 2016, Patent Quality Chat on the issue.

As agreed with your office, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies to the appropriate 

congressional committees, the Secretary of Commerce, the Director of 

the USPTO, the Commissioner for Patents, and other interested parties. 

In addition, the report will be available at no charge on the GAO website 

at http://www.gao.gov.

If you or your staff members have any questions about this report, please 

contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made major contributions to 

this report are listed in appendix IV.

Sincerely yours,

Frank Rusco

Director, Natural Resources and Environment
Appendix I: Objectives, Scope, and 

Methodology

Page 41 GAO-16-490 Intellectual Property

To examine recent trends in patent infringement litigation, we obtained 

patent infringement litigation data from two companies—RPX Inc. and 

Lex Machina—that included all of the patent infringement lawsuits filed in 

all 94 federal district courts between 2007 and 2015. These data included 

information on the patents asserted in each suit, the defendants involved, 

and the federal district court where the suit was filed.1 We conducted data 

quality testing on the data from RPX and Lex Machina to look for missing 

or out of range values, interviewed relevant officials, and reviewed 

relevant documentation for the data and found the data to be sufficiently 

reliable to determine recent trends in patent infringement lawsuits. We 

also conducted 11 semi-structured interviews with stakeholders from 

technology companies, venture capital investors, and others 

knowledgeable about recent patent infringement litigation. In addition, we 

reviewed published literature on the patent system, including reports from 

patent researchers, the Federal Trade Commission, and the 

Congressional Budget Office.

To examine what additional opportunities exist, if any, to improve patent 

quality, we reviewed relevant laws and USPTO documents and 

interviewed USPTO officials and representatives of the examiners’

union—the Patent Office Professional Association. We interviewed four 

supervisory patent examiners and six patent examiners from a variety of 

technology areas. We also conducted 11 semi-structured interviews with 

patent stakeholders who were knowledgeable about patent quality and 

USPTO, including legal scholars, former high-ranking USPTO officials, 

and representatives from public interest non-governmental organizations. 

 1

We also used U.S. patent classification data from Reed Tech and USPTO to ensure that

the 3-digit U.S. patent classes were current. We discarded selected observations from our 

analysis. For example, we did not include declaratory judgment suits in our analysis. In 

addition, we excluded lawsuits for which the filing date was missing from the suit-level 

data. We also excluded lawsuits for which the suit-level data did not report any named 

defendants or any patents asserted. We also discarded patents for which both the filing 

date and the issue date were missing. We used information about the lawsuits, 

defendants, and patents to identify likely transfers, or suits that are likely to have been 

moved to a district court other than the court in which the suit was originally filed. To avoid 

double-counting the defendants and patents in these suits, we discarded observations 

pertaining to the court where the case was originally filed. We also used information about 

the lawsuits, defendants, and patents to identify likely consolidations, or suits that were 

originally filed separately against more than one defendant but later consolidated as a 

single suit. To avoid double-counting the defendants and patents in these suits, we kept 

only the most recently filed (consolidated) suits for our analysis. In our analysis, we 

assume that, within each lawsuit, every patent is asserted against every defendant.

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 42 GAO-16-490 Intellectual Property

Two of the 11 stakeholders we interviewed are currently serving in 

leadership roles as board members of the American Intellectual Property 

Law Association. In assessing USPTO’s efforts, we identified criteria in 

the federal standards for internal control and USPTO’s strategic plan.

In addition, we conducted a web-based survey of a stratified random 

sample of 3,336 eligible USPTO patent examiners from across 8 of the 11 

the technology-based subject matter groups (referred to as technology 

centers) into which USPTO examiners are divided. Fielded between 

August and November 2015, the survey was designed to collect 

information on USPTO’s approach to patent quality and how USPTO 

might improve its patent quality efforts.2 To identify our survey population, 

we obtained from USPTO a list of patent examiners as of May 2015. We 

excluded examiners from three technology centers, as follows:

• We excluded the Designs technology center because these 

examiners work on design patents instead of utility patents; design 

patents are outside the scope of this engagement and have different 

statutory and administrative requirements than utility patents.

• We excluded examiners who perform “re-examination” work and not 

initial patent examination.

• We excluded examiners in the patent training academy because 

these examiners are recent hires who are in a 12-month training 

program.

We also excluded examiners employed at USPTO for less than one year. 

We then defined nine strata by technology center, with one technology 

center separated into two strata, as described in table 2. Specifically, the 

Transportation, Construction, Electronic Commerce, Agriculture, National 

Security and License & Review technology center includes a diverse set 

of technologies, including transportation, construction, agriculture, and 

business methods. In our review, we separated the art units—subunits of 

a technology center—focused on electronic commerce and business 

methods (collectively referred to as business methods) in light of recent 

 2

We administered the survey jointly with a second review focused on prior art issues; 

therefore, it also addressed challenges that USPTO examiners face in finding relevant 

prior art and how USPTO could improve in this area. For this review, see GAO, Intellectual 

Property: Patent Office Should Strengthen Search Capabilities and Better Monitor 

Examiners’ Work, GAO-16-479 (Washington, D.C.: June 30, 2016).
Appendix I: Objectives, Scope, and 

Methodology

Page 43 GAO-16-490 Intellectual Property

legislation and court decisions related to business methods. This resulted 

in 9 strata with a target survey population totaling 7,825 eligible 

examiners. From this list, we drew our stratified random sample of 3,336 

eligible USPTO patent examiners. We received responses from 2,669 

eligible examiners for an 80 percent response rate.3

Table 2: Strata of United States Patent and Trademark Office (USPTO) Examiners 

Surveyed

Name Technology Center (TC) Technology fielda

Biotechnology and Organic 

Chemistry

TC 1600 Chemical

Chemical and Materials Engineering TC 1700 Chemical

Computer Architecture, Software, 

and Information Security 

TC 2100 Electrical

Computer Networks, Multiplex 

communication, Video Distribution, 

and Security 

TC 2400 Electrical

Communications TC 2600 Electrical

Semiconductors, Electrical and 

Optical Systems and Components 

TC 2800 Electrical

Transportation, Construction, 

Electronic Commerce, Agriculture, 

National Security and License and 

Reviewb

parts of TC 3600 (business 

methods only)

Mechanical

parts of TC 3600 (not 

business methods)

Mechanical

Mechanical Engineering, 

Manufacturing, Products 

TC 3700 Mechanical

Source: GAO review of USPTO information. | GAO-16-490

a

We defined these technologies according to how USPTO defined them in its United States Patent 

and Trademark Office Quality Survey, Summer 2015, OMB No: 0651-0057.

b

Technology center 3600 includes a diverse set of technologies, including transportation, 

construction, agriculture, and business methods. In our review, we separated out the art units focused 

on electronic commerce and business methods (collectively referred to as business methods) in light 

of recent legislation and court decisions related to business methods.

 3

After the survey was fielded, we determined that 11 survey recipients were not eligible 

because they told us that they are no longer examiners. Those survey recipients are not 

included in the target population, sample, or respondent totals presented in this report.
Appendix I: Objectives, Scope, and 

Methodology

Page 44 GAO-16-490 Intellectual Property

Because we used a probability procedure based on random selections, 

our sample is only one of a large number of samples that we might have 

drawn. Since each sample could have provided different estimates, we 

quantified the sampling error and express our confidence in the precision 

of our particular sample’s results at a 95 percent confidence interval. This 

is the interval that would contain the actual population value for 95 

percent of the samples we could have drawn. We designed our sample to 

provide percentage estimates with 95 percent confidence intervals that 

are within upper and lower bounds of 5 percentage points, within each 

stratum. We oversampled based on an expected response rate of 70 

percent; however, because we achieved a higher than expected response 

rate, the upper and lower bounds for survey results within each stratum 

are generally less than 5 percentage points. The only estimates for which 

the upper and lower bounds exceed 5 percentage points are certain 

results for the business methods stratum. In these instances, the upper 

and lower bounds are between 5 and 6 percentage points. In this report, 

our figures containing survey results show the upper and lower bounds 

for results at the 95 percent confidence interval. For other estimates in the 

report, we have not provided the upper and lower bounds in the text or 

tables; however, those details for all survey results are available in the esupplement related to this report, GAO-16-478SP.

The quality of survey data can also be affected by nonsampling error, 

which includes, for example, variations in how respondents interpret 

questions, respondents’ willingness to offer accurate responses, 

nonresponse errors, and data collection and processing errors. To 

minimize nonsampling error, we took several steps in developing the 

survey and in collecting and analyzing survey data. Specifically, in 

developing the survey, we worked with our survey professionals to, 

among other things, draft questions that were clear and unbiased. We 

pre-tested the survey in person with five USPTO staff: three examiners 

who are also representatives to the examiners’ union, a supervisory 

patent examiner, and a quality assurance specialist. We used these pretests to check that the questions were clear and unambiguous, used 

correct terminology, requested information that could be feasibly 

obtained, and were comprehensive and unbiased. We also obtained 

comments on the survey from USPTO management and leadership from 

the examiners’ union. In addition, we obtained a quality review by a 

separate GAO survey methodologist. Based on these activities, we made 

changes to the survey before administering it. Further, using a web-based 

survey and allowing examiners to enter their responses into an electronic 

instrument created an automatic record for each respondent. This 

eliminated the potential for errors that could have resulted if we had used 
Appendix I: Objectives, Scope, and 

Methodology

Page 45 GAO-16-490 Intellectual Property

a manual process to enter respondents’ data from paper surveys. In 

addition, to account for the complex sample design, we used survey 

software in our analyses to produce appropriate estimates and 

confidence intervals, and the programs we used to process and analyze 

the survey data were independently verified to ensure the accuracy of this 

work.

To minimize nonresponse error, we made a variety of contacts with the 

sample of examiners during the survey, including follow-up e-mails to 

encourage responses. In addition, between October 20 and 23, 2015, we 

attempted to follow-up via telephone calls to all 1,102 examiners who had 

neither completed the survey nor told us that they were no longer 

examiners. We also analyzed nonresponse bias to (1) assess whether 

any factors were associated with examiners’ propensity to respond and 

(2) to allow our analysis of respondents to properly reflect the sampling 

universe of eligible examiners. To adjust the sampling weight for potential 

nonresponse bias, we used standard weighting class adjustments based 

on the sampling strata and the examiners’ years of experience at USPTO. 

In this report and in the related e-supplement at GAO-16-478SP, we 

present the survey results using the nonresponse adjusted weights, which 

are generalizable to the eligible population of examiners.

We analyzed the responses to the survey for all examiners, as well as 

responses by technology center and by the General Schedule (GS) level 

of the examiners. We selected three categories of GS levels—less than 

GS-13, GS-13, and greater than GS-13—because examiners at these 

levels have different responsibilities and authorities when examining 

patent applications. Specifically, examiners above the GS-13 level may 

grant a patent or reject a patent application without additional review; 

examiners below the GS-13 level must have their actions reviewed and 

signed by a more senior examiner; and some GS-13 examiners are 

transitioning from one GS level to the other.

For some other survey questions, we also reviewed examiners’ openended responses on selected topics. We selected those topics based on 

our interviews with experts and USPTO officials as well as our analysis of 

closed-ended survey responses. We selected the questions for which 

examiners’ responses most frequently included keywords we identified for 

each topic. An analyst conducted a keyword search of all responses to 

the selected open-ended questions and coded responses containing the 

keywords. A second analyst verified the initial analyst’s coding. Our report 

provides some examples of examiners’ comments based on this review. 

Examiners’ responses to open-ended questions are not generalizable to 
Appendix I: Objectives, Scope, and 

Methodology

Page 46 GAO-16-490 Intellectual Property

other examiners. In addition, because we did not conduct a systematic 

review of all open-ended responses to our survey, we do not report the 

exact number of examiners who provided responses on the topics we 

reviewed.

In addition, we conducted statistical tests of association on the results of 

certain survey questions; all tests were independently verified to ensure 

their accuracy. All tests of association were carried out at the 5 percent 

level of significance and were Cochran-Mantel-Haenszel (CMH) Chisquare tests of general association. The testing was carried out in 

SUDAAN, which is statistical software appropriate for the analysis of 

survey data. The null hypothesis was that there is no association between 

the two tested variables. When the association between two variables, 

conditional on a third variable, is of interest, this relationship is referred to 

as the stratum-adjusted CMH test. The test statistic is Wald Chi-Square.

We conducted this performance audit from November 2014 to June 2016 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Patent Examination Process at the 

U.S. Patent and Trademark Office (USPTO) 

(Corresponds to Fig. 1)

Page 47 GAO-16-490 Intellectual Property

This appendix provides details on steps in the patent examination 

process, including rollover information, depicted in figure 1.

Table 3: Descriptions of Selected Steps in the Patent Examination Process at the U.S. Patent and Trademark Office (USPTO)

Step in the Patent 

Examination Process Description

Receipt of patent 

application

When the United States Patent and Trademark (USPTO) receives a patent application, it first goes 

through an extensive review to determine that it is complete. A patent application includes certain key 

elements, particularly the “specification” and the “claims.” The specification contains a description of the 

invention, and the claims define the scope of the invention for which the applicant seeks patent protection. 

Applications can also include an abstract, drawings, and an information disclosure statement of 

references to “prior art”—sometimes referred to as citations.

Classification Because of the significant number of patent applications and issued patents in thousands of different 

technologies, USPTO, using a contractor, classifies patent applications in categories according to their 

technical subject matter. Patent classification is useful for prior art searches because it allows an 

examiner to review a class of applications and issued patents to identify prior art in a specific area of 

technology that the examiner is familiar with. 

Assignment to patent 

examiner

On the basis of its classification, an application is then routed to one of USPTO’s technology centers for 

examination. These technology centers together cover electrical, chemical, and mechanical technology 

fields, and each center is organized into smaller art units—clusters of examiners who focus on related 

technologies. Once an application is routed to a technology center and art unit, USPTO assigns it to an 

examiner. Then, within the art units, according to USPTO officials, supervisory patent examiners assign 

applications to examiners based on each examiner’s workload, not the application’s complexity or the 

examiner’s skills.

Prior art search and 

examination

During patent examination, the examiner determines whether the invention is novel and not obvious by 

comparing its claims to relevant prior art—prior patents, patent applications, or nonpatent literature 

describing a technology, among other things. To do so, the examiner conducts a search to find prior art 

that, if found, would help the examiner determine that the invention is not novel or is obvious. In addition, 

the examiner is to determine whether an application’s claims clearly and precisely inform persons skilled 

in the art of the boundaries of the patent requested. According to USPTO guidance, the scope of the 

claims must be clear in order to inform the public of the boundaries of the patent, and to clearly indicate 

what the applicant regards as the invention

Examiner’s first decision 

on application

In determining if a patent is warranted, USPTO examiners use what are referred to as office actions to 

convey the outcomes of the patent examination process. Through what is called a first office action, or a 

First Action on the Merits, examiners initially notify applicants about the patentability of their inventions 

and are to fully communicate any deficiencies in the applications. The applicant may then amend their 

applications or otherwise address any deficiencies in order to continue the examination process. 

Supervisory examiners may also review some office actions.

Examiner’s final decision 

on application 

At the conclusion of the examination, examiners determine whether a patent will be granted in a final 

office action. If the application is rejected, the applicant may request additional examination. If a patent is 

to be granted, USPTO issues the patent to the applicant once certain fees are paid. The patent can

generally remain in effect for up to 20 years. Supervisory examiners may also review some final office 

actions.

Office of Patent Quality 

Assurance reviews

Quality assurance specialists review a subset of first and final decisions after the examiner’s work is 

complete.

Appeals Applicants and others can appeal an examiner’s decisions—including issued patents—before the 

USPTO’s Patent Trial and Appeal Board. In addition, USPTO decisions may be appealed in federal 

courts.

Appendix II: Patent Examination Process at 

the U.S. Patent and Trademark Office 

(USPTO) (Corresponds to Fig. 1)
Appendix II: Patent Examination Process at the 

U.S. Patent and Trademark Office (USPTO) 

(Corresponds to Fig. 1)

Page 48 GAO-16-490 Intellectual Property

Step in the Patent 

Examination Process Description

Time from receipt to 

decisions

Historically, USPTO has faced a backlog of unexamined applications and a lengthy pendency for 

applications—the amount of time between receipt of an application and USPTO’s preliminary or final 

decision on it. From fiscal years 2011 through 2015, USPTO reduced the average pendency for first office 

actions from 28 months to 17 months, and for final office actions from 34 months to 27 months, according 

to the agency’s fiscal year 2015 performance and accountability report. 

Source: GAO analysis of information from U.S. Patent and Trademark documents and officials. | GAO-16-490
Appendix III: Comments from the U.S. Patent 

and Trademark Office

Page 49 GAO-16-490 Intellectual Property

Appendix III: Comments from the U.S. 

Patent and Trademark Office
Appendix III: Comments from the U.S. Patent 

and Trademark Office

Page 50 GAO-16-490 Intellectual Property
Appendix III: Comments from the U.S. Patent 

and Trademark Office

Page 51 GAO-16-490 Intellectual Property
Appendix III: Comments from the U.S. Patent 

and Trademark Office

Page 52 GAO-16-490 Intellectual Property
Appendix IV: GAO Contact and Staff 

Acknowledgments

Page 53 GAO-16-490 Intellectual Property

Frank Rusco, (202) 512-3841, ruscof@gao.gov

In addition to the contact named above, the following individuals made 

contributions to this report: Hilary Benedict (Assistant Director), Krista 

Breen Anderson, Richard Burkard, John Delicath, Cindy Gilbert, Shilpa 

Grover, Rob Letzler, Rebecca Makar, Rob Marek, Chris Murray, Eleni 

Orphanides, Shep Ryen, Kelly Rubin, Ardith Spence, Sara Sullivan, and 

Sonya Vartivarian.

Appendix IV: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments
Related GAO Products

Page 54 GAO-16-490 Intellectual Property

Intellectual Property: Assessing Factors That Affect Patent Infringement 

Litigation Could Help Improve Patent Quality. GAO-13-465. Washington, 

D.C.: Aug. 22, 2013.

U.S. Patent and Trademark Office: Performance Management Processes. 

GAO-10-946R. Washington, D.C.: Sept. 24, 2010.

Intellectual Property: Enhanced Planning by U.S. Personnel Overseas 

Could Strengthen Efforts. GAO-09-863. Washington, D.C.: Sept. 30, 

2009.

U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to 

Reduce the Patent Application Backlog GAO-08-527T. Washington, D.C.: 

Feb. 27, 2008.

U.S. Patent And Trademark Office: Hiring Efforts Are Not Sufficient to 

Reduce the Patent Application Backlog. GAO-07-1102. (Washington, 

D.C.: Sept. 4, 2007.

Intellectual Property: Improvements Needed to Better Manage Patent 

Office Automation and Address Workforce Challenges. GAO-05-1008T. 

Washington, D.C.: Sept. 8, 2005.

Intellectual Property: Key Processes for Managing Patent Automation 

Strategy Need Strengthening. GAO-05-336. Washington, D.C.: June 17, 

2005.

Intellectual Property: USPTO Has Made Progress in Hiring Examiners, 

but Challenges to Retention Remain. GAO-05-720. Washington, D.C.: 

June 17, 2005.

Related GAO Products

(361611)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.