U.S. PATENT AND 

TRADEMARK OFFICE

USPTO Needs to 

Strengthen Patent Quality 

Assurance Practices

FINAL REPORT NO. OIG-15-026-A

APRIL 10, 2015

U.S. Department of Commerce

Office of Inspector General

Office of Audit and Evaluation

FOR PUBLIC RELEASE
April 10, 2015

MEMORANDUM FOR: Michelle K. Lee

Under Secretary of Commerce for Intellectual Property and

Director of the United States Patent and Trademark Office

FROM: Andrew Katsaros

Principal Assistant Inspector General for Audit and Evaluation

SUBJECT: USPTO Needs to Strengthen Patent Quality Assurance Practices—Final 

Report no. OIG-15-026-A

We are providing our final report for our review of the U.S. Patent and Trademark Office’s 

(USPTO’s) patent quality assurance practices. This audit was conducted to (1) determine the 

sufficiency of the USPTO’s quality assurance program’s processes to prevent the issuance of 

low-quality patents and (2) assess the additional quality reviews performed to measure 

examiner performance and ensure that examiners are fully qualified to issue patent 

determinations without supervisory review.

We identified four areas of concern:

1. USPTO’s performance appraisal plan and related policies are ineffective at measuring 

whether examiners are issuing high-quality patents.

2. USPTO’s official quality metrics may underrepresent the true error rate.

3. USPTO is not collecting data that could improve patent quality.

4. USPTO’s response to patent mortgaging may not discourage abuse.

In response to our draft report, the bureau agreed with our four recommendations and noted 

that it has made meaningful progress with respect to strengthening patent quality since 

enactment of the America Invents Act.

USPTO submitted technical comments to the draft report. Where appropriate, we made 

changes to the final report based on these comments and suggestions. USPTO’s formal 

response is included as appendix D of this report.

USPTO concurred with recommendations 2, 3, and 4. For recommendation 1, USPTO agreed 

with the overall goals of the recommendation to refine supervisory guidance, processes, and 

performance appraisal plans to effectively measure patent examiner quality. USPTO stated it 

will evaluate how supervisors are currently administering the quality element of the examiner 

performance appraisal plan; however, USPTO stated that it will consider options for providing 

training and guidance to supervisors on how best to administer the quality element based upon 

the results of their evaluation. We will determine whether USPTO meets the intent of our four 

recommendations when it submits its action plan to us for review.
2

The final report will be posted on the OIG’s website pursuant to section 8M of the Inspector 

General Act as of 1978, as amended. In accordance with Department Administrative Order 

213-5, within 60 days of the date of this memorandum please provide us with an action plan 

that responds to all of the report’s recommendations.

We thank USPTO personnel for the courtesies shown to us during this review. Please direct 

any questions about the report to Carol Rice, Assistant Inspector General for Economic and 

Statistical Program Assessment, at (202) 482-6020, or David Smith, Assistant Inspector General 

for Intellectual Property and Special Program Audits, at (571) 272-5561.

Attachment

cc: Margaret A. Focarino, Commissioner for Patents, USPTO

Anthony P. Scardino, Chief Financial Officer, USPTO

Welton Lloyd, Audit Liaison, USPTO

Katrina Anwar, Audit Liaison, USPTO
U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Contents

Introduction .......................................................................................................................................................1

Objectives, Findings, and Recommendations .............................................................................................3

I. USPTO’s Performance Appraisal Plan and Related Policies Are Ineffective at Measuring 

Whether Examiners Are Issuing High-Quality Patents........................................................................4

II. USPTO’s Official Quality Metrics May Underrepresent the True Error Rate ....................... 10

A. OPQA Reviewers Do Not Always Charge Errors for 6 to 12 Months after New 

Patent Guidance Has Been Issued ..................................................................................................... 11

B. OPQA’s Periodic Internal Audits Have Shortcomings......................................................... 14

III. USPTO Is Not Collecting Data That Could Improve Patent Quality...................................... 14

A. USPTO Has Not Required Supervisors, Until Recently, to Spend Time Recording 

Errors in a Database............................................................................................................................. 15

IV. USPTO’s Response to Patent Mortgaging May Not Discourage Abuse................................. 19

Recommendations..................................................................................................................................... 22

Summary of Agency Response and OIG Comments................................................................................... 23

Appendix A: Objectives, Scope, and Methodology....................................................................................... 24

Appendix B: Overview of USPTO’s Patent Examination and Quality Assurance Processes ....... 26

Appendix C: Signatory Authority Program.............................................................................................. 27

Appendix D: Agency Response .................................................................................................................. 28

COVER: Detail of fisheries pediment,

U.S. Department of Commerce headquarters,

by sculptor James Earle Fraser, 1934
U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Acronyms and Abbreviations

FY fiscal year

GAO Government Accountability Office

GDP gross domestic product

ISO International Organization for Standardization

IT information technology

MPEP Manual of Patent Examining Procedure

OIG Office of Inspector General

OPQA Office of Patent Quality Assurance

PALM Patent Application Location and Monitoring System

POPA Patent Office Professional Association

QIR Quality Index Report

SAWS Sensitive Application Warning System

SMD Supervisory Patent Examiner Management Database

SOP standard operating procedure

SPE supervisory patent examiner

USPTO U.S. Patent and Trademark Office
FINAL REPORT NO. OIG-15-026-A 1

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Introduction

The White House called innovation “the lifeblood of our economy,”1and the U.S Patent and 

Trademark Office (USPTO) serves a specific role in protecting new ideas through the issuance 

of patents and trademarks. Patents and trademarks establish a legal foundation for ownership 

rights to inventions and ideas, affecting a substantial portion of the U.S. economy. In fiscal year 

(FY) 2010, the Department of Commerce estimated that industries that rely on patent, 

trademarks, and copyrights contributed over $5 trillion to the U.S. gross domestic product 

(GDP) and over 40 million jobs.2 The Secretary of Commerce has also made it a stated goal for 

Commerce to foster a more innovative U.S. economy—one that is better at inventing, 

improving, and commercializing products and technologies that lead to higher productivity and 

competitiveness.

Ensuring the issuance of high-quality patents has been a USPTO strategic initiative for many 

years. High-quality patents are generally considered to be those whose claims clearly define and 

provide clear notice of their boundaries, while low-quality patents are those that contain 

unclear property rights, overly broad claims, or both.

3

Increasing concerns regarding abusive 

patent litigation and ambiguous patents heightens the need for USPTO to ensure adequate 

processes are in place to promote issuing high-quality patents.

Our audit reviewed the three USPTO programs that assess patent examination quality:

(1) performance appraisal reviews, (2) independent reviews by the Office of Patent Quality 

Assurance (OPQA), and (3) signatory authority reviews4(see figure 1). During the course of 

this audit, the Inspector General testified before the U.S. House of Representatives regarding 

USPTO’s telework program and its potential impact on quality. The OIG team responsible for 

this report prepared materials to support the IG’s November 18, 2014 testimony.

5

USPTO’s performance appraisal reviews and the independent reviews by the OPQA 

experienced significant changes in FY 2011 and FY 2012. In response to stakeholder concerns 

about patent quality, OPQA in FY 2011 expanded the previous procedures it used to include 

assessments of the quality of an examiner’s initial search and whether the preliminary decisions

conformed to best practices. In the same fiscal year, USPTO also revised how it measured the 

quality of each examiner’s work product through annual performance assessments. And while 

 

1 The White House, Office of the Press Secretary, February 20, 2014, “Fact Sheet—Executive Actions: Answering 

the President’s Call to Strengthen Our Patent System and Foster Innovation.”

2 The United States Copyright Office resides within the Library of Congress.

3 U.S. Government Accountability Office, August 22, 2013. Intellectual Property: Assessing Factors That Affect Patent 

Infringement Litigation Could Help Improve Patent Quality, report no. GAO-13-465. Washington, DC: GAO.

4 The Signatory Authority Program evaluates the quality of examiner decisions over a period of time to provide 

either partial or full signatory authority to examiners. Partial signatory authority allows examiners to approve or 

disapprove non-final actions where full signatory authority allows examiners to approve or disapprove final actions 

on applications.

5 U.S. House of Representatives, Committee on Oversight and Government Reform and Committee on the 

Judiciary, “Joint Hearing—Abuse of USPTO’s Telework Program: Ensuring Oversight, Accountability, and Quality,” 

November 18, 2014, testimony of Todd Zinser, Inspector General, U.S. Department of Commerce.
2 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

the Signatory Authority Program did not undergo significant changes, in FY 2012 it began to 

more systematically collect information about the decisions or cases reviewed by the program. 

USPTO has recently taken important steps to address patent quality. In January 2015, just prior 

to OIG issuing this report, USPTO announced the creation of a new position, the Deputy 

Commissioner for Patent Quality.6In addition, on February 5, 2015, USPTO published in the 

Federal Register a request for comments, seeking public input and guidance on ways of enhancing 

patent quality.7 However, while the Federal Register notice does seek comment on issues related 

to the OPQA, the notice does not specifically seek input on issues recently examined by 

Congress during the November 2014 hearing—namely, the issues of end loading and patent 

mortgaging. The hearing’s discussions on both issues raise concerns about patent quality and 

are examined in this report. As such, the Federal Register notice misses the opportunity to seek 

wider public comment on particularly problematic quality issues.

Figure 1. USPTO Programs That Review the Quality of Patent Examiner Decisions

Source: OIG, based on USPTO data.

a

For each examiner, supervisors are required to conduct an in-depth review of a minimum of 4 written 

decisions each year as part of the performance appraisal process. (The average annual number of written 

decisions completed by a patent examiner is 73.) 

b USPTO was unable to provide an estimate for the number of reviews it conducted. Therefore, OIG derived 

an estimate by multiplying the prescribed minimum of four reviews per year by the number of patent 

examiners (excluding first-year examiners who are not subject to corps-wide performance standards); the 

range represents annual calculations for FY 2011 through FY 2013.

c This represents a sample of less than 1 percent of all patent determinations in any given year.

 

6

For a description of the new position of Deputy Commissioner for Patent Quality, see USPTO, “Patent Quality 

Initiative,” http://www.uspto.gov/patents/init_events/Patent-Quality-Initiative.jsp (accessed January 30, 2015).

7

“Request for Comments on Enhancing Patent Quality,” 80 FR 6475 (February 5, 2015)

Performance 

Appraisal Reviews

•Supervisory review of 

written decisions 

performed as part of the 

performance appraisal 

process.a

•Measures the quality of 

an individual examiner’s 

work products.

•Reviewed at least 

23,000–30,000 patent 

determinations per year 

between FY 2011 and FY 

2013.b

OPQA 

Independent Reviews

•Independent quality 

reviews performed by 

the Office of Patent 

Quality Assurance 

(OPQA).

•Generates a corps-wide 

measurement of patent 

quality and identifies 

areas for additional 

training.

•Reviewed 6,000–8,000 

patent determinations

per year between FY 

2009 and FY 2013.c

Signatory Authority 

Reviews

•Multi-level reviews to 

assess if examiners are 

qualified to issue their 

own patent 

determinations.

•Examines the quality of 

an individual examiner’s 

work products.

•Reviewed over 35,000 

patent determinations 

between FY 2012 and 

March 31, 2014.
FINAL REPORT NO. OIG-15-026-A 3

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Objectives, Findings, and Recommendations

We conducted this audit to assess USPTO’s quality assurance programs. Our audit objectives 

were to:

1. determine the sufficiency of USPTO’s quality assurance program’s processes to prevent 

the issuance of low-quality patents8and

2. assess the additional quality reviews performed to measure examiner performance and 

ensure that examiners are fully qualified to issue patent determinations without 

supervisory review.

To perform this assessment, we conducted interviews with staff and collected quantitative data 

when available. We interviewed staff involved in the OPQA, performance appraisal, and 

signatory review programs at USPTO’s offices in Alexandria, Virginia. We also collected data 

from OPQA independent quality reviews, performance ratings, and performance warnings 

issued from FY 2011 to FY 2013. Finally, we reviewed a sample of 60 individuals who 

participated in the Signatory Authority Program. However, USPTO could only provide data for 

the Signatory Authority Program from FY 2012 through the second quarter of FY 2014. 

According to USPTO, prior to FY 2012, it had not systematically collected information about 

the quality issues found during supervisory review of patent applications. Table 1 shows a 

summary of our audit findings as they relate to each audit objective.

Table 1. Audit Objectives and Results

Objective What We Found

Determine the sufficiency of USPTO’s quality 

assurance processes to prevent the issuance of 

low-quality patents.

• USPTO’s official quality metrics may 

underrepresent the true error rate.

Assess the additional quality reviews performed 

to measure examiner performance and ensure 

examiners are fully qualified to issue patent 

determinations without supervisory review.

• USPTO’s performance appraisal plan and related 

policies are ineffective at measuring whether 

examiners are issuing high-quality patents.

• USPTO is not collecting data that could improve 

quality.

• USPTO’s response to patent mortgaging may not 

discourage abuse.

Source: OIG analysis

We also reviewed USPTO’s performance appraisal system to determine whether it was an 

effective measure for determining whether examiners are issuing high-quality patents, and found 

that errors can be found in 75—and even 100—percent of an examiner’s cases reviewed, yet 

an examiner could still obtain a rating of fully successful or higher on the quality performance 

element. 

 

8 USPTO stated that inadequate examiner performance—such as including claims in patent determinations that 

should have been rejected and rejecting claims that should have been included—leads to low-quality patents.
4 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

During our review USPTO supervisors raised the issue of “patent mortgaging” (i.e. an 

employee knowingly submitting incomplete work to receive credit). We found that although 

USPTO treats patent mortgaging as an act of misconduct, provides distinct guidelines for 

punishment, and receives reports that allow it to potentially identify instances of abuse, the 

controls in place to detect and monitor the practice of patent mortgaging responses may not 

adequately discourage the practice.

Also, although we identified concerns with the collection of data within the Signatory Authority 

Program, we did not note significant instances of noncompliance with USPTO’s own internal 

policies for designating partial and full signatory authority. However, USPTO does not 

systematically collect data on which decisions contain errors and if errors are charged during 

this process.

I. USPTO’s Performance Appraisal Plan and Related Policies Are Ineffective at 

Measuring Whether Examiners Are Issuing High-Quality Patents

USPTO supervisors are responsible for measuring whether individual examiners are issuing 

high-quality patents. Their assessments are reflected in the examiners’ annual performance 

appraisals. During the course of the annual performance period, supervisors are required to 

conduct an in-depth review of a minimum of four patent determinations completed by the 

examiner, regardless of the total number of determinations completed. Although USPTO 

does not have records to confirm if these in-depth reviews actually occurred, OIG 

estimates that from FY 2011 to FY 2013 at least 23,000–30,000 reviews per year (out of a 

total of approximately 1.4 million patent determinations completed each year during this 

period9) would have been completed (see figure 1). This is more than three to four times 

the number of applications that were reviewed by OPQA between FY 2009 and FY 2013. In 

addition, until recently USPTO senior management did not require supervisors to 

systematically track when they found errors during these reviews. Thus, although 

supervisors conduct substantially more reviews than the official quality assurance 

program—that is, OPQA’s independent reviews—USPTO has less reliable data about these 

supervisory reviews.

As discussed below, we identified concerns with USPTO’s policies for measuring the quality 

of examiners’ written decisions. We found that USPTO’s performance appraisal plans and 

related policies cannot distinguish between examiners who issue high-quality decisions 

versus those who issue low-quality decisions. Rather, most examiners are rated on their 

annual performance appraisals as “above average.” This meant that, between FY 2009 and

FY 2013, 99 percent of examiners received quality ratings that made them eligible for 

almost $145 million in monetary awards.

10 This averaged to more than $6,000 in awards per 

examiner per year.

 

9

Excluding first-year examiners, who are not subject to corps-wide performance standards.

10 USPTO has awards for the production and docket management elements, but there are no awards specific to 

the quality element. Examiners can become ineligible for some awards if they are issued a marginal or 

unsatisfactory quality rating.
FINAL REPORT NO. OIG-15-026-A 5

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

USPTO’s supervisors rate patent examiners on four performance elements, which are 

graded on a five-point scale, outlined in the examiner’s performance appraisal plans. The 

four performance elements (and their respective weights) for each examiner are:

 Production: examiners issue determinations on patentability within the assigned 

time frames (35 percent);

 Quality: examiners correctly determine whether a patent application should be 

approved or rejected (35 percent);

 Docket management: examiners manage respective caseloads and properly select 

cases for review per USPTO policies (20 percent); and

 Stakeholder interaction: examiners provide appropriate service to stakeholders 

(10 percent).

In descending order, the five rating grades are “outstanding,” “commendable,” “fully 

successful,” “marginal,” and “unacceptable.” As noted above, an examiner’s quality rating 

makes up 35 percent of his or her total score. 

During the period of FY 2011 through FY 2013, over 95 percent of all patent examiners 

received outstanding or commendable ratings for the quality element of their annual 

performance evaluations (see figure 2

11) even though the Department defines commendable 

performance as “unusually good” and outstanding performance as “rare, high-quality 

 

11 Examiners also received high ratings on the quality-related metrics in FY 2009 and FY 2010. During this period,

there were multiple performance metrics that addressed quality.

Figure 2: Distribution of Patent Examiners’

Annual Quality Ratings, FY 2011–FY 2013

FY2011-2013

Source: OIG analysis of USPTO data

51.62% 57.02% 57.24%

44.28% 39.87% 38.09%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2011 2012 2013

Outstanding Commendable

Fully Successful Marginal and Unacceptable
6 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

performance” that “rarely leaves room for improvement.” This distribution of scores does 

not align with the ratings descriptions contained in the Commerce Department’s guidelines 

for performance appraisals.12

As shown in figure 2, over 50 percent of patent examiners received outstanding quality 

scores in FY 2011 through FY 2013.13 Furthermore, although the Department defines fully 

successful performance as the “level of accomplishment expected of the great majority of 

employees,” USPTO supervisors and Technology Center quality assurance specialists we 

interviewed indicated it is often difficult to justify not giving an examiner an outstanding 

performance rating.

For the purpose of evaluating examiner performance, errors are clear instances in which 

the examiner did not comply with examining standards set forth in the performance 

appraisal plan. Errors are categorized from 1 to 3 based upon severity and examiner’s grade 

level. An examiner’s error rate is the key driver in determining his or her quality rating (see 

table 2). If a supervisor finds that an examiner committed an error, the supervisor can 

potentially charge it to the examiner’s error rate. When this happens, the examiner may 

rebut the error, either orally or in writing. The supervisor can either remove the error or 

inform the examiner why his/her rebuttal was unpersuasive. If this process does not resolve 

the disagreement, the examiner may raise the disagreement with the Technology Center 

Director and, if unsatisfied, file a grievance, in accordance with USPTO’s agreement with 

the union that represents patent examiners, the Patent Office Professional Association 

(POPA).

14 In lieu of charging an error, a supervisor may choose to coach an examiner,

which will not impact an examiner’s error rate. By not charging errors, the rebuttal 

process—which many supervisors referred to as time-consuming and laborious—is avoided.

Table 2. How USPTO Measures Quality in Performance Plans

Rating

Error Rate 

(percent )

Additional 

Factors Considered

Outstanding 0–4.49 • Clear and concise communication

• Early patentable determinations during prosecution 

• Patentable subject matter allowed early in the review Commendable 4.50–5.49

Fully Successful 5.50–6.49 None

Marginal 6.50–7.49 None

Unacceptable 7.5 or higher None

Source: OIG analysis of the error rate from the GS-15 examiner performance appraisal plan

 

12 Generic performance standards are contained in appendix A of the Department’s FY 2014 Performance 

Management Record (form CD-516B LF).

13 Figure 2 excludes examiners hired within the previous 12 months. Thus, the figure excludes 902 examiners in FY 

2011, 698 in FY 2012, and 493 in FY 2013.

14 See memorandum of understanding between USPTO and POPA dated October 22, 2010.
FINAL REPORT NO. OIG-15-026-A 7

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

USPTO management claims that supervisors review more than the one case per quarter; 

however, there is no way to verify this because supervisors currently do not document 

which cases they review. In addition, USPTO supervisors and other Technology Center 

quality assurance staff we interviewed indicated that there is an incentive to not charge

errors in order to avoid the potential time-intensive error rebuttal process. Furthermore, 

and as discussed in more detail below, the current standards often make it difficult to justify 

giving an examiner a rating other than “outstanding.”

According to USPTO’s performance appraisal plan,

15 an examiner’s error rate is defined as 

the number of charged errors found during the supervisor’s review divided by the total 

number of patent determinations the examiner completed during the review period. 

However, multiple errors are counted as one if in the same determination, thus 

underrepresenting the error rate.

One reason that examiners rarely receive average or poor ratings is due to the relationship 

between the limited number of reviews required of supervisors and how USPTO measures 

the error rate. Supervisors currently only need to review one decision per quarter, or four 

per year. In FY 2013, USPTO examiners averaged 73 final determinations per year.

16

Table 3. How Patent Examiner Error Rates are Calculated

Calculation

Corresponding Ratinga

Fully Successful Commendable Outstanding

Number of cases reviewed with errors 4 4 3

÷ Number of determinations ÷ 62 ÷ 73 ÷ 68

= Error Rate (converted to percent) 6.45% 5.47% 4.41%

Source: OIG analysis, based upon four reviews annually

a

See table 2 for a breakdown of error rates and ratings.

To provide context, table 3 illustrates that errors can be found in 75—and even 100—

percent of the cases reviewed, yet an examiner could still obtain a rating of fully successful 

or higher on the quality performance element. If a supervisor reviews the minimum of four 

determinations during the annual performance period, as required, and errors are found in 

all four, the examiner will still achieve a rating of fully successful if at least 62 determinations

were completed during the year and a commendable rating as long as the average number 

of 73 determinations was completed. Thus, every patent examiner meeting USPTO’s

average case workload can receive, at a minimum, a rating of commendable. If 75 percent of 

the four determinations reviewed contain errors, an examiner completing 67 or more 

determinations during the year would qualify for an outstanding rating.

 

15 U.S. Patent and Trademark Office, “Classification and Performance Management Record,” FY 2014.

16 In FY 2013, 8,051 examiners issued 605,994 final determinations on patent applications. The number of 

determinations an examiner must issue is based on their grade and the technology they review.
8 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Supervisors can downgrade examiners from outstanding to commendable based upon nonerror factors known as indicia, but as indicated in table 2, this only applies to examiners 

receiving outstanding or commendable quality ratings. These indicia include the quality of 

the examiner’s communications with the applicant and the examiner’s ability to identify 

patentable claims early in the application process (see the full list in table 2). However, 

supervisors cannot assign examiners a rating of less than “fully successful” based on these 

factors.

Although USPTO implemented changes in FY 2011 to examiner performance appraisals to 

“align the patent examiner performance appraisal plans to organizational goals, and ensure 

strategic alignment at all levels”

17—some of the changes have made it more difficult to tie 

examiner performance to the issuance of high-quality patents. For example, USPTO relaxed 

the error rate for some examining activities by eliminating or combining multiple metrics 

into one quality error rate. Additionally, the new plan required some types of errors to 

have occurred multiple times before a supervisor could charge them to an examiner’s error 

rate.

Another impact on measuring examiner quality occurred prior to the introduction of 

changes to the performance appraisal system in FY 2011. We were informed that the 

Commissioner for Patents verbally announced that errors found by OPQA could not be 

used to calculate an examiner’s error rate. This change became effective in FY 2010 and 

may have contributed to inflated performance ratings. We confirmed that from FY 2011 to 

FY 2013, examiners with an error identified by an OPQA independent reviewer still 

received an outstanding or commendable quality rating over 95 percent of the time.

18 With 

the inclusion of OPQA-identified errors, examiner error rates and quality ratings would 

have been more accurate reflections of their performance.

One indicator that the performance appraisal process is an inadequate means for assessing 

the quality of examiner’s work is that USPTO issues far fewer written warnings for quality

compared to those issued for two other performance elements—docket management and

production. Underperforming examiners receive a series of escalating warnings before 

receiving a written warning.19 Examiners who receive three written warnings in 5 years can 

be terminated. As shown in figure 3 (see next page), during the period of FY 2011 through

FY 2013, of the approximately 6,000 to 8,000 patent examiners employed by USPTO during 

this time, 264 examiners received at least one written warning for production problems, 

and 233 received warnings for the docket management problems. However, only 7

examiners received written warnings for low-quality decisions. Of note, an individual who 

 

17 USPTO, “Briefing to Patent Professionals,” internal training document, October 2010.

18 We limited our analysis to instances in which both OPQA and the Technology Centers agree that the examiner 

committed an error.

19 An October 2010 memorandum of understanding between USPTO and POPA outlines USPTO’s current system 

of escalating warnings for underperforming examiners. Examiners first receive “safety zone” and then oral warnings 

before they receive a written warning. A safety zone warning is designed to give examiners whose performance is 

just below the acceptable range an opportunity to improve before they are issued an oral warning.
FINAL REPORT NO. OIG-15-026-A 9

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

received a written warning under the quality element still received an overall rating of

commendable in the end-of-year rating.20

Overall, we found that since the 

introduction of the new performance 

appraisal standards in FY 2011, an examiner 

is more likely to obtain an outstanding or 

commendable quality rating. To determine 

this, we analyzed examiner quality metrics

before and after the implementation of the 

FY 2011 standards. Our analysis found that 

a GS-13 examiner with partial signatory 

authority had an 89 percent chance of 

obtaining a commendable or outstanding 

rating under the previous plan. Under the 

new performance appraisal plan, a GS-13 

examiner with partial signatory authority

now has a 96 percent chance of obtaining a 

commendable or outstanding quality rating, 

holding all other variables constant.21

These quantitative results support the 

information we gained from interviews with 

representatives from USPTO’s nine Technology Centers. Seven of the nine individuals we 

interviewed indicated that USPTO’s performance appraisal policies do not allow for a 

sufficient understanding of an examiner’s patent quality; six felt that the current system of 

performance evaluation was insufficient to gain an understanding of an examiner’s overall 

performance. The interviewees cited various issues, such as the calculation of the error 

rate, policies regarding the charging of errors, and the time-consuming and onerous nature 

of justifying a less-than-outstanding quality rating for a less-than-outstanding examiner.

The weaknesses we identified with the current performance plan make it difficult to 

distinguish between patent examiners who are issuing high-quality patents and those who 

are not. In July 2014, the Deputy Director of USPTO stated that “issuing high-quality 

patents not only improves the overall vitality of our patent system but can also play a 

significant role in curtailing abusive patent litigation.” We are concerned with USPTO’s

inability to distinguish and reward examiners performing at a truly outstanding level of 

performance versus those who are not.

 

20 USPTO noted that quality issues may be masked in production warning data, because re-work of low-quality 

applications may affect an examiner’s ability to meet production goals. While this may be true for examiners at a 

GS-11 level or lower, examiners at higher grades do not need a supervisor’s approval before they receive 

production credit for some of their work. Thus, this concern would not apply to them.

21 Our logistic regression model controlled for such variables as the time period, the Technology Center of the 

employee, his or her grade, and the signatory authority level.

Figure 3. Number of Examiners Issued 

Warnings, by Performance Element,

FY 2011–FY 2013

Source: OIG analysis of USPTO data

264

233

7

0

50

100

150

200

250

300

Production Docket

Management

Quality
10 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

II. USPTO’s Official Quality Metrics May Underrepresent the True Error Rate

Although different offices within USPTO check whether examiners’ written determinations 

have errors, OPQA is the official quality assurance program within USPTO. OPQA is 

independent of the management chain to which an examiner reports, and its independent 

reviewers assess whether randomly selected patent decisions contain any errors. It is 

important to note that, on average, OPQA annually reviews less than 1 percent of all office 

actions. OPQA management indicated that it was their role to generate statistically 

significant results of the quality of determinations, not to catch or prevent the issuance of all 

low-quality patents. The office also performs reviews to assess whether examiners’ 

decisions conform to examination best practices and whether examiners perform a 

thorough search of technology relevant to the patent application. For all reviews, OPQA 

systematically records why a determination had an error or did not conform to best 

practices. For example, OPQA determines which area of patent law the examiner did not 

follow in a decision. OPQA reviews a statistical sample of both preliminary and final 

examiner decisions to generate a corps-wide measurement of patent examination quality at 

USPTO. In its review, OPQA looks for errors, which are defined as unreasonable failures by 

the patent examiner to reject patent claims for one or more reasons provided in the patent 

laws. Errors also occur when an examiner incorrectly rejects patent claims. Within USPTO, 

all of these mistakes are known as clear errors.

The results of OPQA’s analysis feed into several components of USPTO’s official quality 

metrics,

22 but these results are not used to assess the quality of particular offices within 

USPTO, nor are they used to assess the performance of individual examiners. Rather, the 

results are used to generate USPTO’s official quality metrics and provide corps-wide 

accuracy rates that affect the bonuses awarded to the supervisors of patent examiners. In 

addition, the potential areas of training needs are identified. USPTO’s February 5, 2015, 

Federal Register notice seeks public input about a proposed mechanism that would allow an 

applicant to request OPQA to review a particular application. Such targeted reviews could 

add to the data OPQA collects for identifying and targeting specific examiner training needs.

Table 4 shows the accuracy rate estimates generated from OPQA reviews for FY 2009 through 

FY 2013. Over this period, OPQA performed between 6,000 and 8,000 reviews each year. 

(USPTO issued over 1.2 million preliminary and final decisions each year during that same time 

period.)

 

22 OPQA’s official quality metrics are combined into a composite that “compares current performance against 

historical achievement and charts USPTO’s progression towards desired levels of performance,” according to 

Commissioner for Patents Robert Stoll, “August Patents Dashboard Overview,” Director’s Forum (blog post), U.S. 

Patent and Trademark Office, September 23, 2011.
FINAL REPORT NO. OIG-15-026-A 11

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Table 4. Results of OPQA’s Accuracy Rate Tests, FY 2009–FY 2013 (percent)

Quality Assurance Tests FY 2009 FY 2010 FY 2011 FY 2012 FY 2013

Accuracy rate for final decisions 94.4 96.3 95.4 96.6 96.2

Accuracy rate for cases still being 

processed 93.6 94.9 95.2 95.9 96.3

Source: USPTO

We identified concerns both with USPTO’s policies for charging errors and with OPQA’s 

internal auditing procedures. We found that some OPQA independent reviewers do not

always record errors on the patent determinations that they review when new examination 

guidance is issued. Every year, USPTO issues new guidance in response to executive actions, 

Patent Trial and Appeal Board decisions, or federal court decisions. By ignoring some 

errors, OPQA may have underrepresented the true error rate when it calculated official 

quality metrics. We also determined that OPQA’s internal audits are not structured to 

assess unwritten or informal policies and procedures—such as not charging errors for 6 to 

12 months after new patent guidance has been issued. 

A. OPQA Reviewers Do Not Always Charge Errors for 6 to 12 Months after New Patent Guidance 

Has Been Issued

OPQA is composed of reviewers and supervisors who are independent of the patent 

examiners’ chain of command. The duties of these independent reviewers and 

supervisors include providing ad hoc advice to examiners, teaching examiners in one-onone and group settings, and reviewing a sample of patent determinations issued by 

examiners.23 OPQA performed 6,484 reviews of preliminary and final decisions out of 

the 1.2 million patent determinations issued by USPTO in FY 2013. For the times when 

OPQA chooses its sample, it randomly selects from any office action made in a given FY 

for its review sample. Although OPQA samples less than 1 percent of determinations, 

the office still generates statistically significant results for estimating the accuracy rates 

of the examiner corps. As mentioned earlier in this finding, OPQA management

indicated that it was their role to generate statistically significant results of the quality of 

determinations, not to catch or prevent the issuance of all low-quality patents.

OPQA’s independent reviewers look at a number of factors when they judge the quality 

of examiners’ work. For example, they assess whether examiners properly rejected the 

individual claims in a patent application and identified appropriate reasons to approve a 

patent application. The reviews also check whether examiners’ written decisions 

conform to best practices. As illustrated in table 5, these independent reviews make up 

the first four of seven components that compose USPTO’s official composite quality 

metric.

 

23 Patent determinations are known within USPTO as patent actions.
12 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Based on interviews conducted with OPQA staff, we were informed that OPQA 

reviewers may identify, but not record, some errors. This practice is not based on 

written policy direction. There are two categories of instances where errors can be 

made by examiners. One is when an examiner omits a rejection in a non-final, final or

allowed action, where there are reasons provided in the patent laws to make the 

rejection. The other category is when, in a final or non-final action, there is an 

unreasonable rejection or restriction, or in the case of a final action, when the action is 

improperly made final.

Table 5. Seven Components of USPTO’s Composite Quality Metric

Source: OIG interviews and USPTO policies

a

Examiners search for prior art to determine whether an application is novel.

b This data comes directly from the Patent Application Location and Monitoring System

(PALM), which is the IT tool that tracks events related to patent processing. For example, one 

factor included in this data is the frequency with which applications are reopened after an 

examiner closes the review by issuing a rejection.

USPTO periodically issues new patent guidance, such as it did early in 2014 in response 

to a Supreme Court decision involving patents that covered DNA sequences owned by 

one company.

24 OPQA staff reported that whenever new patent examination guidance 

is issued by USPTO they normally allow patent examiners a grace period of 6 to 12 

months before charging applicable errors. If they find an error during this grace period, 

instead of recording an error OPQA reviewers flag applications about which they have

 

24 Association for Molecular Pathology v. Myriad Genetics, 133 S.Ct. 2107 (2013).

Component Description Weight

1. Final Disposition 

Compliance Rate Percent of final decisions without errors 20%

2. In-process 

Compliance Rate

Percent of preliminary decisions without 

errors

15%

3. First Action on 

the Merits Search 

Review 

Score given for the quality of the patent 

examiners’ initial search for related 

technology, known in USPTO as “prior art”

a

10%

4. Complete First 

Action on the Merits 

Review 

Score given for the degree to which the 

preliminary decision conforms to USPTO best 

practices

10%

5. Quality Index 

Report

Statistical representation of different qualityrelated issues during the patent examination 

process b

20%

6. External Quality 

Survey

Survey that measures quality concerns 

expressed by patent attorneys and applicants 15%

7. Internal Quality 

Survey

Survey that measures quality concerns 

expressed by USPTO’s patent examiners 10%
FINAL REPORT NO. OIG-15-026-A 13

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

concerns with the label “Needs Attention.” This label is applied in lieu of an error and is 

not an official error as defined in OPQA policy.

“Needs Attention” may be used by OPQA reviewers to identify issues with patent 

examination that do not rise to the level of an error, such as pointing out an unchecked 

box on a patent examination form that should have been checked; making note of what 

would otherwise be an error were it not for the new patent examination guidance grace 

period; or acknowledging exceptional examination—use of best practices—of a patent 

determination. The percentage of applications flagged with “Needs Attention” has risen 

in the past 5 years. In FY 2009, the rate was around 15 percent; for the past 3 fiscal 

years it has hovered around 21 percent (or about 1,200–1,300 decisions per year).

25 The 

OPQA director explained that this rise may be because, in recent years, OPQA 

reviewers have been encouraged to use the “Needs Attention” flag to highlight issues 

that are not errors.

Interviews with OPQA personnel brought to our attention the practice of not always 

charging errors until new examination guidance has been issued and in effect for 6 to 12 

months. This practice reduces our confidence in the accuracy of USPTO’s official quality 

metric. Components 1 and 2 (see table 5) together comprise 35 percent of USPTO’s

official metric. That 35 percent is based on OPQA’s review of examiner decisions, which 

in turn is dependent on the number of errors identified by reviewers. For those patent

actions examined by OPQA, USPTO was unable to provide an estimate on the number 

of errors that were recorded as “Needs Attention” instead of as an error. However, 

since FY 2009, USPTO has issued new examination guidance 19 times in response to 

federal court decisions and Patent Trial and Appeal Board decisions. Thus, the practice 

of not appropriately recording errors has had an ongoing impact on OPQA reviews and 

the official quality metric. 

The decision to not charge errors occurs when case law behind the new patent 

examination guidance is, in OPQA management’s opinion, unsettled—meaning it may be 

subject to change in the foreseeable future. We found that OPQA reviewers do not 

apply a consistent standard in response to instances of “unsettled” law. In our 

interviews, some reviewers noted that they wait 6 months before recording errors,

while others wait 12 months. If OPQA issued written policies to assist reviewers, 

OPQA could avoid inconsistency about when to start recording errors.

Accurate inputs in this process are imperative, since the composite quality metric that 

OPQA calculates from them is meant to accurately assess the overall quality of patent 

examination by USPTO’s examiner corps. Errors identified during OPQA reviews do 

not affect the performance ratings of an individual patent examiner; rather, the quality 

metric derived from them is used by stakeholders to assess the quality of decisions 

issued by USPTO. Thus, once USPTO has issued examination guidance to interpret 

changes in case law, OPQA should assess whether examiners are following this 

guidance.

 

25 We have excluded instances where OPQA used the “Needs Attention” field to identify best practices.
14 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Beyond the potential impact on the composite quality metric, there is also a negative 

secondary effect related to OPQA’s practice of recording errors as “Needs Attention”

rather than as errors. When OPQA reviewers identify an error, the Technology 

Centers must provide a response to this determination. Although OPQA feels that 

decisions flagged as “Needs Attention” are ones that could also serve as teaching 

moments for examiners, the Technology Centers are not required to provide a 

response in these instances. Furthermore, because some errors were not flagged, 

Technology Centers were not fully informed about the number of errors occurring and 

therefore may not have been taking the necessary corrective actions.

B. OPQA’s Periodic Internal Audits Have Shortcomings

We also identified an area where OPQA could improve its internal auditing procedures. 

OPQA follows the quality management system standards set by the International 

Organization for Standardization (ISO). ISO standards require OPQA to conduct 

internal audits of its operations and to also periodically have an external entity conduct 

such a review. Eight of the 10 OPQA staff members we interviewed participated in such

internal audits. These individuals stated that the internal audits reviewed OPQA’s 

standard operating procedure (SOP) and asked OPQA staff to demonstrate how they 

follow them. Yet, not every OPQA procedure is included in the SOP. 

We previously mentioned the example of the 6- to 12-month grace period for charging 

errors regarding new patent examination guidance. This example is not covered in an 

SOP and therefore represents an attribute that could not be tested during any internal 

examination or audit. The SOP also fails to specify the procedure that OPQA’s

reviewers should follow when they need to recuse themselves due to a conflict of 

interest. Thus, audit procedures that follow the SOP may not identify a lack of 

compliance with some policies and the informal practices followed by some OPQA 

reviewers. It is possible that, if the audit were structured in this way, OPQA could have 

already identified the fact that some of its reviewers were allowing a 6-month grace 

period after new examination guidance had been issued, while others were allowing a 

12-month grace period. Furthermore, audits that follow the same routine from period 

to period, as described by the OPQA staff, may not be effective if the auditee knows 

and anticipates the audit steps beforehand and learns only those aspects best, rather 

than focusing on learning all aspects of the job.

III. USPTO Is Not Collecting Data That Could Improve Patent Quality

USPTO’s FY 2014–2018 strategic plan states that USPTO will maximize the use of quality 

data it obtains from examiner reviews to ensure the issuance of high-quality patents. 

However, we learned that USPTO supervisors have not systematically collected information 

about the quality issues that were found during the supervisory review of patent 

applications. Furthermore, USPTO is not collecting valuable information from the program 

that assesses whether examiners are qualified to issue patent decisions without supervisory 
FINAL REPORT NO. OIG-15-026-A 15

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

review.26 (This latter program is known as the Signatory Authority Program.) As a result, 

data that could be used to improve examiner quality is not being collected and analyzed.

A. USPTO Has Not Required Supervisors, Until Recently, to Spend Time Recording Errors in a 

Database

USPTO created a database called the Supervisory Patent Examiner (SPE) Management 

Database (SMD) to serve the needs of supervisors and USPTO management. The 

system allows supervisors to record performance ratings and bonuses, record which 

cases were reviewed as part of the Signatory Authority Program, and track when 

supervisors find errors in examiner decisions. SMD was available in May 2011 to track 

examiner errors found in written decisions. However, it wasn’t until after we initiated 

our audit that USPTO mandated (in April 2014) that all supervisors record errors in the 

system.

Although USPTO is using SMD to collect some quality data, we identified additional 

information that is not collected which could be used to measure the quality of 

examiner decisions (see table 6, next page). For example, USPTO cannot determine 

what percentage of decisions has certain types of errors because supervisors still do not 

record which cases they review. Furthermore, USPTO cannot evaluate the types of 

errors and error rates that were upheld during the Signatory Authority Program. For 

example, USPTO could not confirm whether a sample, selected by OIG, of 60 

examiners participating in signatory reviews had attained error rates of 6.49 percent or 

less, the level needed to obtain a performance rating of fully successful. Thus, USPTO

cannot conduct a systematic analysis of errors and trends found from partial and full 

signatory reviews.

 

26 Supervisors review and sign all patent applications reviewed by examiners who have not passed what is known 

as the Signatory Authority Program. Supervisors only review selected applications of senior examiners who have 

obtained partial and full signatory authority.
16 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Table 6. USPTO Data Not Collected by Supervisors from 

Performance Reviews and Signatory Reviews

Program

Data Not 

Collected/Tracked Why Data Collection Might Be Helpful

Performance 

Review Cases reviewed

• Ensures supervisors reviewed the required 

number of cases for examiner performance 

appraisals.

• Allows supervisors access to the error data 

USPTO started to collect in April 2014.

Signatory 

Authority 

Review

Cases with errors • Allows Technology Centers to identify trends 

in errors committed by senior examiners.

Error rate achieved

by examiners

• Allows USPTO to determine whether the 

error rates of examiners completing the partial 

and full signatory authority are rising or not, and 

to document the error rate.

Source: OIG Analysis of USPTO documents

USPTO also checks for errors in examiners’ determinations during the Signatory 

Authority Program. This program is a two-stage process that assesses whether 

examiners are qualified to issue patent determinations without additional supervisory 

review. (See appendix C for a description of the steps in the program.) The first stage of 

the process is the partial signatory authority. It occurs after an examiner has reached 

the GS-13 level. The second stage of the process is full signatory authority (see table 7, 

next page, for a breakdown of the examiner corps by grade and signatory authority). 

Once the examiner obtains full signatory authority, he or she does not need to be 

recertified to maintain this authority. During both stages, supervisors and Technology 

Center quality assurance specialists review at least 17 decisions per examiner for 

errors.27 Each decision is reviewed by two different individuals who sit on a panel. The 

panelists identify potential errors, but ultimately the Technology Center director 

decides whether these errors should be considered when deciding whether the 

examiner receives partial or full signatory authority. Between FY 2012 and the second 

quarter of FY 2014, 1,040 examiners received full signatory authority. During the same 

period, 1,063 examiners received partial signatory authority. Unfortunately, USPTO 

does not systematically collect data on which decisions contain errors and if errors are 

charged during this process. Therefore, USPTO lacks key data to improve examiners’ 

quality performance.

In contrast to its failure to collect information related to the quality element of 

examiner performance, USPTO has invested significant time to measure examiner 

 

27 The number of examiner decisions reviewed accounts for about one-half of the examiner decisions during the 6-

month review timeframe.
FINAL REPORT NO. OIG-15-026-A 17

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

productivity.28 To track each examiner’s productivity, supervisors receive automated 

biweekly reports from USPTO’s Patent Application Location and Monitoring (PALM) 

System. In addition, USPTO created its Supervisory Patent Examiner (SPE) Dashboard, 

which includes numerous metrics to track examiner productivity and docket 

management compliance. 

Table 7. Patent Examiner Corps, by Grade and

Signatory Authority, as of June 5, 2014

Source: USPTO data

USPTO has used some of the case tracking and productivity data found in PALM to help

identify potential quality issues. USPTO refers to this subset of PALM information as 

Quality Index Report (QIR) data. QIR data tallies how often certain events occur that 

could indicate a potential quality concern. For example, the data tracks when an 

examiner reopened a case that had been closed and the number of determinations an 

examiner made on an application.29 USPTO supervisors stated that they rely heavily on 

QIR data to identify potential productivity and quality issues. USPTO managers we 

interviewed indicated that this data could allow supervisors to focus their reviews on

examiners whose examining behavior differs from their peers. We agree that this data is 

a valuable means for identifying examiners whose behavior deviates from the norm or 

expectations. However, this data does not track why certain types of errors, and issues

related to them, are matters of concern across the entire examiner corps.

USPTO misses an opportunity to monitor the impact of specific actions or examiner 

behavior by not collecting information from supervisory reviews. For example, during 

our interviews with USPTO staff, some individuals identified the practice of “end 

 

28 Productivity goals are expressed as the average number of hours spent by the examiner on each application. The 

number varies by grade level and subject matter. On average, examiners review 73 patents per year.

29 Typically, an examiner would issue two determinations on a patent application.

Grade Number

GS-05 4

GS-07 442

GS-09 737

GS-11 862

GS-12 1,091

GS-13 (no signatory authority) 969

GS-13 (partial signatory authority) 539

GS-14 and higher (full signatory authority) 3,342

Total 7,986
18 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

loading” as a matter of concern. End loading refers to the practice engaged in by some 

patent examiners of submitting a high volume of written decisions at the end of each 

quarter. This practice allows examiners to meet quarterly production goals for written 

decisions, but presents an appearance that the quality of the deteminations could be 

compromised. We interviewed three supervisors who noted that end loading made it 

difficult to monitor the quality of the submissions. Others indicated that they enlist the 

assistance of Technology Center quality assurance specialists to review cases submitted 

by examiners who end loaded. Without knowing which cases a supervisor or other 

quality specialist has reviewed, USPTO cannot analyze what impact end loading had on 

the quality of determinations issued to applicants. We found that between October 1, 

2009, and March 31, 2014, in any given quarter, between 543 and 1,767 (up to 20 

percent) of examiners submitted more than 50 percent of their work in the last four 

weeks of the quarter (see figure 4). 

Figure 4. Percent of Patent Examiners Who Submitted More than Half of Their 

Work in the Last Two Biweekly Periods of a Quarter, FY 2009–FY 2014

Source: OIG analysis of USPTO data

USPTO recently investigated end loading, interviewing a random sample of 49 

supervisory patent examiners and all 26 Technology Center directors.30 Of those 

employees, 71 percent of supervisors and 90 percent of the directors felt that end 

loading could negatively affect the quality of examiner work. As one supervisor 

explained, since end loaders provide the bulk of their work before deadlines, “There is 

pressure to review and approve, and you cannot spend enough time reviewing.” 

Between FY 2009 and the second quarter of FY 2014, OPQA found more than 150 

errors in decisions issued by examiners who end loaded. Given that end loading is not a 

rare occurrence at USPTO, continually monitoring the practice and collecting 

information on it would help USPTO to identify and address quality concerns.

 

30 Memo from Chief Administrative Officer, U.S. Patent and Trademark Office, to Office of Inspector General, 

August 10, 2014, “Inspector General Referral No. PPC-CI-12-1196-H; Re: Abuse of Telework Program at 

USPTO.” This memorandum discusses USPTO’s concerns about various time-and-attendance abuses, mortgaging, 

and end loading.

0%

10%

20%

30%

40%

50%
FINAL REPORT NO. OIG-15-026-A 19

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

IV. USPTO’s Response to Patent Mortgaging May Not Discourage Abuse

One type of work credit abuse, referred to as “patent mortgaging,” was mentioned in three

of the nine interviews we conducted with Technology Center representatives. Patent 

mortgaging occurs when an employee knowingly submits incomplete work for credit in 

order to, for example, receive an award or avoid a performance warning. USPTO treats 

patent mortgaging as an act of misconduct and follows guidelines for punishment that are 

different from the escalating warning process for performance we described in finding I.

31

Although supervisors receive reports that allow them to potentially identify instances of this 

abuse, we found that USPTO’s response to allegations of patent mortgaging varied widely 

and may not discourage the practice.

The practice of patent mortgaging occurs, in part, because some examiners can receive 

credit for their work without supervisory review. All decisions made by an examiner who is 

below the GS-12 level are reviewed by a supervisor or senior examiner, making this 

practice less likely for them. However, examiners at the GS-12 level and above are able to 

receive credit for non-final decisions before undergoing supervisory review.32 Furthermore, 

examiners who have achieved full signatory authority are able to issue decisions (and thus 

receive credit for their work) without any supervisory review. The credit that examiners 

receive allows them to help meet their, and USPTO’s, quarterly productivity expectations.

Although examiners earn credit for incomplete work, the incomplete decisions are not 

mailed to applicants. Rather, administrative staff review specific aspects of all determinations 

and return incomplete decisions to examiners for correction before they are mailed.

It may be difficult for supervisors to monitor patent mortgaging. Supervisors can use the 

SPE Dashboard to monitor the number of administrative returns per examiner; however, 

there may be legitimate reasons why an examiner received a returned application. For 

example, in FY 2012 and FY 2013, 83 percent and 91 percent of examiners respectively had 

at least one decision returned by administrative staff for corrections (see table 8, next 

page).

 

31 A memorandum of understanding issued on October 22, 2010, between USPTO and POPA outlines USPTO’s 

current system of escalating warnings for underperforming examiners. Examiners first receive a “safety zone”

warning and then an oral warning before they receive a written warning. A safety zone warning is designed to give 

examiners whose performance is just below the acceptable range an opportunity to improve their performance 

before they are issued an oral warning.

32 The October 22, 2010, memorandum of understanding between USPTO and POPA established the policy that 

GS-12 and GS-13 examiners who had not yet achieved partial signatory authority would be eligible to receive 

credit for preliminary decisions before they were reviewed by a supervisor. As of FY 2014, approximately 74 

percent of examiners are at the GS-12 level or higher.
20 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Table 8. Number of Examiners with at Least One Decision Returned by 

Administrative Staff, FY 2012 and FY 2013

FY 2012 FY 2013

Number of examiners with at least one decision returned 

by administrative staff 6,570 7,329

Total number of examiners as of the end of the fiscal year 7,935 8,051

Source: OIG analysis of USPTO data

The number of patent mortgaging allegations has increased in recent years. Between 

October 2008 and August 2014, 121 examiners were accused of patent mortgaging. More 

than one-half of these allegations occurred over the 11-month period from October 1, 

2013, to August 29, 2014 (see table 9). We cannot determine from this data whether the 

practice of patent mortgaging is increasing or whether USPTO has become more focused 

on identifying instances of patent mortgaging.

To determine what level of disciplinary action is appropriate to impose on an employee for 

an act of misconduct, USPTO policies identify the factors that supervisors must consider in 

determining the severity of discipline. Examples of these factors include the effect upon “the 

supervisors’ confidence in the employee’s work ability to perform assigned duties,” “the 

notoriety of the offense or its impact upon the reputation of the agency,” and “the 

consistency of the penalty with those imposed upon other employees for the same or 

similar offenses.”33

Table 9. Allegations of Patent Mortgaging, FY 2009–FY 2014

Fiscal Year 

(FY)

Allegations of 

Mortgaging

Total Patent

Examiner Corps

2009 2 6,243

2010 5 6,225

2011 11 6,780

2012 18 7,935

2013 21 8,051

2014 64 8,611

Source: OIG analysis of USPTO data received between October 1, 2009, and 

August 29, 2014

USPTO’s responses to allegations of patent mortgaging have varied widely, and in 

accordance with the factors referred to above. Between October 2009 and August 2014, 

 

33 POPA, “Work Credit Abuse,” June 12, 2013, http://popa.org/2165/ (accessed January 30, 2015).
FINAL REPORT NO. OIG-15-026-A 21

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

the penalty for a first occurrence ranged from counseling to as much as a 14-day suspension

(see table 10). During this same time period, we found that the redress most commonly

used by USPTO was an “abeyance agreement.” An abeyance agreement means that USPTO 

and the employee accused of misconduct have entered into an agreement whereby the 

agency determines the appropriate discipline for the employee’s misconduct—for example,

a suspension—but does not actually impose the discipline, provided that the employee 

fulfills his or her obligations under the agreement (typically, to commit no further act of 

misconduct for a specific number of years). If the employee does not fulfill his or her 

obligations under the agreement, and does commit a further act of misconduct, then the 

original sanction is automatically imposed.

The second and third most common responses were, respectively, “no action taken” and a 

suspension of 7 days or less. A total of 27 patent examiners were suspended for some 

period of time, and 7 resigned, retired, or were terminated.

Table 10. Actions Taken in Response to Allegations

of Patent Mortgaging, October 2009–August 2014

Action Frequency

Abeyance Agreement 15

No Action Taken 13

Suspension (7 days or less) 13

Suspension (8–14 days) 12

Oral Counseling 7

Pending 7

Reprimand 7

Counseling Letter 6

Resignation 5

Employee Exonerated 2

Settlement 2

Suspension (more than 14 days) 2

Discharge during Probationary Period 1

Factual Situation Unproven 1

Insufficient Evidence 1

Rescinded Action 1

Retirement (Voluntary) 1

Source: OIG Analysis of USPTO data

Note: USPTO had reached a final decision for 96 of the 121 patent 

mortgaging cases.

Work credit abuse can negatively impact the reputation of USPTO and unfairly affect

employees who complete their work in a timely manner. In our interviews, USPTO staff 
22 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

commented that they had heard about patent mortgaging, but were unable to identify

specific instances. While USPTO has applied disciplinary actions for patent mortgaging 

offenses, the disciplinary actions we reviewed appeared to be inconsistently applied and 

could result in real (or perceived) instances of unfair treatment.34 USPTO should, as part of 

its supervisory oversight, ensure that adequate controls are in place to detect and monitor 

the practice of patent mortgaging and that penalties are consistent with those imposed upon 

other employees for the same or similar offense. 

Recommendations

We recommend that the Undersecretary of Commerce for Intellectual Property and Director 

of the United States Patent and Trademark Office:

1. Refine supervisory guidance, processes, and performance appraisal plans to effectively

measure patent examiner quality efforts and distinguish levels of performance—including 

taking steps to avoid disincentives for supervisors to charge errors to examiners when 

assessing performance under the requirements set forth in the examiner performance 

appraisal plans.

2. Strengthen OPQA’s (a) independent quality review procedures to ensure their 

consistent application, particularly with respect to the application of new case law and

how errors are categorized, and (b) internal audit process, by minimizing the predictable 

nature of the audit steps and allowing for the identification of the informal practices 

followed by some OPQA reviewers.

3. Use available databases and systems to collect information on patent applications 

reviewed and errors found, to improve USPTO’s ability to identify quality trends.

4. Develop and document additional controls to better detect and monitor the practice of 

patent mortgaging and continue to ensure consistent application of USPTO disciplinary 

policies that address instances of it.

 

34 Obtaining the documentation and conducting a legal review of USPTO disciplinary actions was outside the scope 

of this audit. Because we do not have sufficient information, we are not making any determination as to the extent 

that USPTO has been, or currently is, consistently adjudicating instances of patent mortgaging.
FINAL REPORT NO. OIG-15-026-A 23

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Summary of Agency Response and OIG 

Comments

In response to our draft report, the Under Secretary of Commerce for Intellectual Property 

and Director of the United States Patent and Trademark Office concurred with our four 

recommendations and noted that the bureau had begun to make progress on improving patent 

quality, including creating and filling a new, full-time senior executive position, the Deputy 

Commissioner for Patent Quality. USPTO’s response is included in appendix D of this final 

report.

With respect to the first recommendation, we advise USPTO to make changes to its guidance, 

processes, and performance appraisal plans for both supervisors and employees, where 

appropriate. We look forward to reviewing the actions USPTO plans to take to meet the intent 

of our recommendation. 

USPTO also submitted technical comments to the draft report. We made some changes to the 

final report based on these comments and suggestions. In particular, we clarified that OPQA’s

practice of allowing a 6- to12-month grace period in response to new patent examination 

guidance before charging errors was an informal practice, and not an approved policy.
24 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Appendix A: Objectives, Scope, and Methodology

The objective our audit was to (1) determine the sufficiency of USPTO’s quality assurance 

program’s processes to prevent the issuance of low-quality patents and (2) assess the additional 

quality reviews performed to measure examiner performance and ensure that examiners are 

fully qualified to issue patent determinations without supervisory review.

To accomplish our objectives we:

 Reviewed public comments submitted to USPTO concerning the agency’s quality 

metrics and practices.

 Reviewed patent laws and regulations, guidance, performance appraisal standards, and 

memorandums of understanding with the Patent Office Professional Association (POPA) 

that apply to examiner performance evaluations and signatory authority reviews.

 Collected data on examiner performance appraisals, oral and written warnings 

submitted to the USPTO’s Office of Human Resource Management, and error records 

from the Office of Patent Quality Assurance (OPQA), between FY 2009 and the second 

quarter of FY 2014 (where available). We tested the reliability of the data through 

electronic testing and, in some cases, manual matching.

 Interviewed nine supervisory patent examiners or training quality assurance specialists 

drawn from different Technology Centers to (a) determine how examiner performance 

evaluations and signatory authority reviews are conducted and (b) assess the 

implementation of internal controls.

 Interviewed six review quality assurance specialists and four supervisors in OPQA to 

determine common themes or significant issues related to patent quality assurance.

 Interviewed USPTO management and union officials to gain an understanding of the 

agency’s efforts to prevent the issuance of low-quality patents.

 Assessed whether USPTO followed its criteria for designating signatory authority for a 

random sample of 60 USPTO patent examiners participating in the partial and full 

Signatory Authority Programs during FY 2012 and FY 2013 and determined whether 

USPTO has followed its procedures for rescinding examiners’ partial and full signatory 

authority for inadequate performance.

 Reviewed an internal USPTO report and interviews pertaining to USPTO time-andattendance issues, patent mortgaging, and end loading. 

 Prepared the Inspector General’s testimony of November 18, 2014, before a joint 

hearing of the U.S. House of Representatives’ Committee on Oversight and 

Government Reform and Committee on the Judiciary on “Abuse of USPTO’s Telework 

Program: Ensuring Oversight, Accountability and Quality.”

Further, we gained an understanding of the internal controls related to independent review of 

patent decisions, OPQA’s quality management system, USPTO’s program for conferring
FINAL REPORT NO. OIG-15-026-A 25

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

signatory authority, and USPTO’s policies for preventing favoritism and abuse in the Signatory 

Authority Program and performance appraisal process. 

There are additional quality-related processes and programs that were out of the scope of this 

audit. For example, our audit did not focus on the new examiner training, nor did we focus on 

whether supervisors and senior examiners provided adequate coaching and oversight to junior 

examiners. The programs we focused on address how USPTO ensures high-quality patents 

after completion of the initial examiner onboarding processes.

USPTO’s Sensitive Application Warning System (SAWS) has been identified as a process for the 

additional examination of patent applications meeting certain identified criteria. Because SAWS 

is not incorporated into USPTO’s Manual of Patent Examining Procedure (MPEP), its impact is not 

considered in this report.35

As described in finding II, we found that OPQA lacks adequate procedures to capture 

unwritten policies and procedures that could affect the accuracy of USPTO’s quality metric. 

However, we also found that USPTO has mostly sufficient internal controls regarding the 

approval of patent examiners into the Signatory Authority Program, but we lacked sufficient 

data to test some controls. We also found that USPTO had established reasonable controls to 

prevent most instances of favoritism, retaliation, and abuse of position in the performance 

appraisal and signatory review program. We did not identify any incidence of fraud, illegal acts, 

violations of laws, or abuse in our audit. We reviewed USPTO’s monitoring and responses to 

work credit abuse committed by examiners because this concern was raised during our 

interviews. 

Because USPTO does not know what cases a supervisor or other person has reviewed, USPTO 

cannot analyze the reviews it has conducted to determine the impact of end loading has on the 

quality of determinations issued to applicants.

For our quantitative analysis, we relied on computer-generated data. We found the data 

sufficiently reliable to identify trends and create statistical models. 

We conducted this audit from April 2014 through October 2014. The audit was conducted 

under the authority of the Inspector General Act of 1978, as amended, and Department 

Organizational Order 10-13, dated April 26, 2013, at USPTO offices located in Alexandria, 

Virginia. We conducted this performance audit in accordance with generally accepted 

government auditing standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence obtained provides a 

reasonable basis for our findings and conclusions based on our audit objectives.

 

35 U.S. Patent and Trademark Office, 2014. Manual of Patent Examining Procedure, 9th ed. Alexandria, VA: USPTO, 

March 2014.
26 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Appendix B: Overview of USPTO’s Patent 

Examination and Quality Assurance Processes

Examiner researches whether applicant’s claims are patentable and drafts a preliminary determination.

Examiner issues a preliminary determination

Patent Application Approved

Reject Approve

Applicant decides next option Approve

START Key Patent examination process

Has the examiner been granted partial or full signatory authority?

Has the examiner been granted full signatory authority?

Supervisor or a senior examiner reviews determination before it is issued to the applicant.

Supervisor or a senior examiner reviews determination before it is issued to the applicant

The examiner considers any amendment submitted by the applicant and drafts a final determination

Examiner issues a final determination

Yes

Yes

No

No

Reject

OPQA Independent Quality Reviews Several thousand preliminary and final determinations are independently reviewed by OPQA each year. OPQA tests for clear errors and the extent to which the determination conforms with best practices. A supervisor must review at least four determinations made by examiners each fiscal year to assess the quality of examiner’s decisions.

Performance Plan Quality Reviews

IMPACT: The results of the independent reviews affect: 1. the official quality metric 2. training 3. supervisor bonuses

IMPACT: The results of the perfor- mance plan reviews affect: 1. examiners’ quality rating 2. examiners’ awards (examiners must achieve a “fully satisfactory” quality rating to receive an award)

Performance plan quality reviews OPQA independent quality review Patent examination processes affected by signatory authority status Signatory Authority Program

Less than 1% of determinations

Unknown number of determinations

Less than 1% of determinations Unknown number of determinations

Examiners who have passed the two-stage signatory authority program no longer need to have another examiner review their determinations before the applicant receives them.

Source: OIG analysis of USPTO policies
FINAL REPORT NO. OIG-15-026-A 27

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Appendix C: Signatory Authority Program

Source: OIG analysis of USPTO policies

Step 1: Eligibility Period (Partial Signatory Authority)

An examiner who wishes to obtain partial signatory authority must become a 

GS-13 and perform at least at the “fully successful” level in their performance 

appraisal plan for 10 consecutive pay periods.

Step 2: Grant Temporary Partial Signatory Authority

Examiner receives temporary partial signatory authority.

Step 3: Enter Trial Period

• Perform at least 700 examining hours in 13 pay periods.

• Issue at least 17 preliminary decisions, which are reviewed by a minimum

of two supervisory patent examiners.

• Achieve an error rate of 6.49 percent or less during this period.

Step 4: Grant Permanent Partial Signatory Authority

Examiner receives permanent partial signatory authority.

Step 5: Eligibility Period (Full Signatory Authority)

An examiner who wishes to obtain full signatory authority must

perform at least at a fully successful level in their performance 

appraisal plan for 10 consecutive pay periods after receiving 

permanent partial signatory authority.

Step 6: Grant Temporary Full Signatory Authority

Examiner receives temporary full signatory authority.

Step 7: Enter Trial Period

• Perform at least 700 examining hours in 13 pay periods.

• Have at least 17 final determinations (allowances and rejections) reviewed 

by a minimum of two supervisory patent examiners.

• Achieve an error rate of 6.49 percent or less during this period.

Step 8: Grant Permanent Full Signatory Authority

Examiner receives permanent full signatory authority.
28 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

Appendix D: Agency Response
FINAL REPORT NO. OIG-15-026-A 29

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL
30 FINAL REPORT NO. OIG-15-026-A

U.S. DEPARTMENT OF COMMERCE OFFICE OF INSPECTOR GENERAL

10USPTO00187