Intellectual Property: Patent Office Should Define Quality, Reassess Incentives, and Improve Clarity
Published: 2016-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Resolving disputes over patent infringement and validity in court often costs millions of dollars. Legal scholars and economists have raised concerns about an increase in the numbers of low quality patents—such as those that are unclear and overly broad—which may lead to an increase in patent infringement suits and can hinder innovation by blocking new ideas from entering the marketplace.
GAO was asked to review issues related to patent quality. GAO examined (1) recent trends in patent infringement litigation and (2) what additional opportunities exist, if any, to improve patent quality. GAO reviewed relevant laws and agency documents; analyzed patent infringement litigation data from 2007 through 2015; conducted a survey of a generalizable sample of USPTO examiners; and interviewed officials from USPTO and knowledgeable stakeholders, including legal scholars, technology companies, and patent attorneys, among others.
section_title: What GAO Found
paragraphs: GAO found that district court filings of new patent infringement lawsuits increased from about 2,000 in 2007 to more than 5,000 in 2015, while the number of defendants named in these lawsuits increased from 5,000 to 8,000 over the same period. In 2007, about 20 percent of all defendants named in new patent infringement lawsuits were sued in the Eastern District of Texas, and by 2015 this had risen to almost 50 percent. According to stakeholders, patent infringement suits are increasingly being tried in the predominantly rural Eastern District of Texas, likely due to recent practices in that district that are favorable to the patent owners who bring these infringement suits. GAO also found that most patent suits involve software-related patents and computer and communications technologies. Several stakeholders told GAO that it is easy to unintentionally infringe on patents associated with these technologies because the patents can be unclear and overly broad, which several stakeholders believe is a characteristic of low patent quality.
The U.S. Patent and Trademark Office (USPTO) has taken actions to address patent quality, most notably through its Enhanced Patent Quality Initiative, but there are additional opportunities for the agency to improve patent quality. For example, USPTO does not currently have a consistent definition for patent quality articulated in agency documents and guidance, which would be in line with federal internal-control standards and best practices for organizational performance. Most stakeholders GAO interviewed said they would define a quality patent as one that would meet the statutory requirements for novelty and clarity, among others, and would be upheld if challenged in a lawsuit or other proceeding. Without a consistent definition, USPTO is unable to fully measure progress toward meeting its patent quality goals. Additionally, USPTO has not fully assessed the effects of the time allotted for application examinations or monetary incentives for examiners on patent quality. Specifically, most stakeholders GAO interviewed said that time pressures on examiners are a central challenge for patent quality. Based on GAO's survey of patent examiners, GAO estimates that 70 percent of the population of examiners say they do not have enough time to complete a thorough examination given a typical workload. According to federal standards for internal control, agencies should provide staff with the right structure, incentives, and responsibilities to make operational success possible. Without assessing the effects of current incentives for examiners or the time allotted for examination, USPTO cannot be assured that its time allotments and incentives support the agency's patent quality goals. Finally, USPTO does not currently require applicants to define key terms or make use of additional tools to ensure patent clarity. Federal statutes require that patent applications use clear, concise, and exact terms. Based on a survey of patent examiners, GAO estimates that nearly 90 percent of examiners always or often encountered broadly worded patent applications, and nearly two-thirds of examiners said that this made it difficult to complete a thorough examination. Without making use of additional tools, such as a glossary of key terms, to improve the clarity of patent applications, USPTO is at risk of issuing patents that do not meet statutory requirements.
section_title: What GAO Recommends
paragraphs: GAO makes seven recommendations, including that USPTO more consistently define patent quality and articulate that definition in agency documents and guidance, reassess the time allotted for examination, analyze the effects of incentives on patent quality, and consider requiring applicants to use additional clarity tools. USPTO generally agreed with GAO's findings, concurred with the recommendations, and provided information on steps officials plan to take to implement the recommendations.

section_title: Letter
paragraphs: Intellectual property-intensive industries such as electronics and  pharmaceuticals support at least 40-million jobs in the U.S. and contribute  more than $5 trillion annually—or nearly 35 percent—to the nation’s gross  domestic product. While most patents are never the subject of a lawsuit,  resolving a dispute over patent infringement and validity in courts can  cost millions of dollars. Legal scholars and economists have raised  concerns about an increase in the number of low quality patents—such  as those that are unclear and overly broad—which may lead to an  increase in patent infringement suits and hinder innovation by blocking  new ideas from entering the marketplace. Further, in our 2013 report, we  found that low quality patents were likely a key factor in many patent  infringement lawsuits between 2007 and 2011, because their unclear  boundaries make it easy to unintentionally infringe these patents, among  other factors.
The U.S. patent system—authorized by the U.S. Constitution—aims, in  part, to make innovation more profitable. For example, a patent owner  can generally exclude others from making, using, selling, or importing the  patented technology for up to 20 years from the date on which the  application for the patent was filed. By restricting competition, patents can  allow their owners to earn greater profits on their patented technologies  than they could earn if these technologies could be freely imitated. Patent  owners sometimes need to assert their patent rights by filing a lawsuit if  competitors infringe their patent. The exclusive rights provided by patents  can help their owners recoup the costs of the research and development  of new technologies. On the other hand, any reduction in competition  caused by the exclusive nature of patents may result in higher prices for  products having patented technologies. The patent system, therefore,  gives rise to complex trade-offs involving innovation and competition.  Some researchers have suggested that the balance between these trade- offs has tipped and that some patents now are limiting innovation,  especially in areas such as software and computer technologies.
We found in 2013 that some stakeholders, including technology  companies, judges, and legal scholars, said that the U.S. Patent and  Trademark Office (USPTO) should never have issued some patents  because they believe they do not meet all of the legal standards required.  Patents must meet standards set forth in law and interpreted by the  courts. For example, they must cover eligible subject matter, be novel, not  obvious, and they must clearly describe the invention. In this report, we  consider patents that fully meet federal patentability standards to be high  quality. USPTO has acknowledged the need to focus additional attention  on patent quality and, in 2015, began its Enhanced Patent Quality  Initiative, aimed at helping to ensure that the thousands of patents the  agency issues each year are of higher quality. In a Department of  Commerce Inspector General’s report, USPTO’s deputy director said an  increased focus on patent quality would play a significant role in curtailing  patent litigation.
You asked us to review issues related to patent quality. This report  examines: (1) recent trends in patent infringement litigation, and (2) what  additional opportunities exist, if any, to improve patent quality. Our work  for this report was coordinated with our report on a part of the patent  examination process known as “prior art” searches.
To examine recent trends in patent infringement litigation, we obtained  patent infringement litigation data from two companies—RPX Inc. and  Lex Machina—that included all of the patent infringement lawsuits filed in  all 94 federal district courts between 2007 and 2015. These data included  information on the patents asserted in each suit, the defendants involved,  and the federal district court where the suit was filed. We conducted data  quality testing on the data from RPX and Lex Machina, interviewed  relevant officials, and reviewed relevant documentation for the data and  found the data to be sufficiently reliable to determine recent trends in  patent infringement lawsuits. We also conducted 11 semi-structured  interviews with stakeholders from technology companies, venture capital  investors, and others knowledgeable about recent patent infringement  litigation. In addition, we reviewed published literature on the patent  system, including reports from patent researchers, the Federal Trade  Commission, and the Congressional Budget Office.
To examine what additional opportunities exist, if any, to improve patent  quality, we reviewed relevant laws and USPTO documents and  interviewed USPTO officials and representatives of the examiners’  union—the Patent Office Professional Association. We interviewed four  supervisory patent examiners and six patent examiners from a variety of  technology areas. We also conducted 11 semi-structured interviews with  patent stakeholders who were knowledgeable about patent quality and  USPTO, including legal scholars, former high-ranking USPTO officials,  and representatives from public interest non-governmental organizations.  In addition, we conducted a web-based survey of a stratified random  sample of 3,336 USPTO patent examiners from across eight of the  technology-based subject matter groups (referred to as technology  centers) into which USPTO examiners are assigned. We excluded  examiners employed at USPTO for less than one year, as of May 2015.
The survey collected information on examiners’ views on USPTO’s  approach to patent quality and how the agency might improve its patent  quality efforts. The survey was designed and administered in conjunction  with a concurrent report on patent prior art. Overall, we received  responses from 80 percent of the examiners in our sample. Our  methodology was designed to produce estimates that are generalizable to  the population of patent examiners in our study overall, as well as within  each technology center. The survey questions and detailed data on the  results can be viewed on our website. Appendix I provides more detail  on our scope and methodology.
We conducted this performance audit from November 2014 to June 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: This section provides an overview of USPTO’s patent examination  process, patent infringement litigation and challenges, and USPTO’s  Enhanced Patent Quality Initiative. A list of our prior work related to  patents and intellectual property is included at the end of this report.
subsections: section_title: Patent Examination Process
paragraphs: When USPTO receives a patent application, the agency assigns it to a  division of patent examiners with relevant technology expertise called a  technology center. There are 11 technology centers focusing on  everything from biotechnology to mechanical inventions. After the  application is assigned to a technology center, it is then assigned to an  individual examiner who is responsible for the examination, or  prosecution, of the application. Figure 1 shows the key steps in the patent  prosecution process.
The focus of patent examination is determining whether the invention in a  patent application satisfies the statutory requirements for a patent,  including that the invention be novel, useful, not obvious, and clearly  described. Generally, prior patents, patent applications, or publications  describing an invention, among other things, are known as prior art.  During patent examination, the examiner, among other things, compares  an application to the prior art to determine whether the invention is novel  and not obvious. Finding prior art is the most time consuming part of  patent examination, according to our report on prior art. Applicants are  not required to search for prior art before submitting their application,  although they are required to notify examiners of material prior art they  know about.
A patent application includes the “specification” and at least one “claim.”  By statute, the specification is to contain “a written description of the  invention, and of the manner and process of making it, in such full, clear,  concise, and exact terms as to enable any person skilled in the art to  which it pertains to make and use the invention.” The law requires  further that the specification “shall conclude with one or more claims  particularly pointing out and distinctly claiming the subject matter which  the inventor regards as the invention. A patent application’s claims  define the legal boundaries of the invention for which patent protection is  sought.
USPTO’s data show that, as of April 2016, the average time between  filing an application and an examiner’s initial decision on the application  was about 16 months, and it takes an average of 26 months after an  application is submitted for USPTO to complete the examination of an  application. Due to the current inventory of applications awaiting  examination, USPTO is not able to begin examining patent applications  upon receiving them—as of April 2016, USPTO had a backlog of about  550,000 unexamined applications.
Through the initial decision, or first action on the merits, examiners initially  notify applicants about the patentability of their inventions. To avoid  lengthy back and forth exchanges, the examiner is encouraged to identify  all of the problems with the application under the patent statutes and rules  during the first “office action”—a USPTO policy called “compact  prosecution.” The applicant is able to respond to the examiner after the  first office action, and interviews are often used so that the examiner and  applicant can clarify their respective positions, as well as the scope of the  claimed invention. Ultimately, and sometimes after many months, the  examiner then issues a final rejection or allows the claims in the  application. If applicants receive a final rejection, they may file a request  for continued examination, which requires a new submission and the  payment of additional fees, and the examiner will continue examining the  application. There is no limit to the number of times an applicant may  request continued examination for an application, but USPTO officials say  it is rare to have three or more requests for continuing examination filed  for a single application. Examination ends with an issued patent, or when  the applicant abandons the application—there are no terminal rejections.
There are often a dozen or more claims per patent, and they can often be  difficult for a layperson to understand, according to legal researchers. For  example, one claim for a cardboard coffee cup insulator begins by  referring to “a recyclable, insulating beverage container holder,  comprising a corrugated tubular member comprising cellulosic material  and at least a first opening therein for receiving and retaining a beverage  container.” A patent application’s claims can be written broadly or be  more narrowly defined, according to legal researchers, and applicants  can change the wording of claims—which can affect their scope—based  on examiner feedback during examination. Examiners can suggest that  applicants amend their claims by adding words to their claims to address  statutory impediments to issuance. For example, adding the word  “corrugated” to modify “tubular member” in the example claim above  narrows the scope of the claimed invention. Companies often prefer  broader patent claims that make it less likely a competitor would be able  to make a small change to its invention to avoid infringement. Patents are  a property right and their claims define their boundaries.
In some cases, patent claims define the scope of the invention by  encompassing an entire function––like sending an e-mail––rather than  the specific means of performing that function. While “functional claiming”  is permitted by statute, we reported in 2013 that patents that include  functional claiming language were more likely to be unclear and to be  disputed in court. For example, if the pencil was patented as a  “mechanism for writing,” the owner of the patent could theoretically sue  manufacturers of different technologies for infringement, including pens  and markers.
As of May 2015, USPTO had nearly 8,300 patent examiners across the  eight technology centers that we reviewed. The agency uses the General  Schedule (GS) classification system for patent examiners, whose levels  range from GS-5 to GS-15. Examiners at the GS-14 level or above (44  percent of the examiners in the technology centers we reviewed) are  referred to as “primary examiners” who may accept or reject a patent  application without additional review. This level of authority is in contrast  to junior examiners—most examiners below the GS-14 level— whose  work must first be reviewed by a supervisory or primary patent examiner  before it can be sent to the applicant.
Examiners are rated based on their production, or the number of  examination tasks they perform, among other factors. The number of  examination tasks that an examiner is expected to perform is set based  on the examiner’s technology area and experience level. USPTO allots  more time to review applications to examiners that deal with more  complex technologies. For example examiners working on artificial  intelligence patent applications are given an average of about 31 hours to  complete an examination, while those working on patent applications for  exercise devices are given an average of about 17 hours. As examiners  are promoted on the GS-scale, the average number of hours they are  allotted to work on each application and the level of review from their  supervisors declines. Primary examiners have the least amount of time to  examine patent applications and the applications they review undergo the  least amount of supervisory review, in part because their experience  allows them to work more efficiently and effectively, according to USPTO  officials. As examiners rise from junior to primary status, their examination  time is roughly cut in half. According to USPTO officials, on average,  examiners spend about 22 hours total on average on each application  from start to final determination, with a low of about 11 hours on average  for some primary examiners in the least complex technologies and a high  of about 60 hours on average for an entry-level junior examiner in more  complex technologies.
A few studies have shown that there are differences in issued patents  depending on how much time examiners are given to complete their  examination. For example, an academic paper from 2012 found that more  experienced examiners cite less prior art, are more likely to grant patents  rather than reject them, and are more likely to grant patents without any  preliminary rejections. Similarly, other researchers have found that  examiners put less effort into searching for prior art when they are given  less time to review an application. For example, the National Bureau of  Economic Research published a paper in 2014 that found when  examiners were allotted less time to conduct a patent examination, they  were less likely to make time-intensive prior art rejections and more likely  to grant a patent.
section_title: Patent Infringement Litigation and Challenges
paragraphs: When a patent right is not clearly defined, it can lead to boundary  disputes, often in the form of infringement lawsuits. Although litigated  patents are a small percentage of issued patents, low quality patents are  more likely to be asserted in patent infringement lawsuits because,  according to some economists, the less clear the claim boundaries are,  the more likely that others will infringe the patent or will continue to  infringe when confronted by the patent owner. Patent owners can bring  infringement lawsuits against anyone who uses, makes, sells, offers to  sell, or imports the patented invention without authorization. If the court  finds that infringement has occurred, it must award the patent owner  damages adequate to compensate for the infringement. During an  infringement case, the accused infringer may seek to have the  infringement lawsuit dismissed by showing, by clear and convincing  evidence, that the patent at issue is invalid. However, because most of  the roughly 4,000 patent infringement lawsuits filed each year settle  before the court ever makes a determination of patent validity some  patent owners asserting low quality patents may not be concerned with  the risk the courts will invalidate their patents. When the courts do rule  on validity, they generally invalidate almost half of the patents that are  challenged, according to academic research.
Accused infringers can also challenge a patent’s validity outside of an  infringement lawsuit in administrative proceedings at USPTO’s Patent  Trial and Appeal Board (PTAB). Patent challenges at PTAB are often  initiated by individuals or firms that have been sued for infringing a  particular patent, so the patents that appear in these proceedings are  often the same ones that appear in patent infringement suits. The  challengers in these proceedings seek to present evidence that shows  that the patent claims should not have been granted because they failed  to meet a statutory patentability requirement. The PTAB proceedings are  a lower cost alternative to the federal courts where infringement suits are  often very expensive. As of March 2016, there have been around 4,700  patent challenges filed with the PTAB since its inception in 2012 and  about 60 percent of these challenged patents are related to computers  and software. About 30 percent of the PTAB proceedings have reached a  final decision, and nearly 75 percent of those final decisions have resulted  in all of the challenged claims being held unpatentable.
section_title: USPTO’s Enhanced Patent Quality Initiative
paragraphs: In February 2015, USPTO launched an Enhanced Patent Quality Initiative  designed to improve the quality of patents. According to USPTO, it  started its quality initiative because the agency had successfully reduced  its backlog of patent applications, and had the financial resources to  consider longer-term improvements to patent quality due to fee setting  authority provided by the America Invents Act.
As part of its initiative, the agency has taken a number of actions,  including:
Creating a new leadership role and Patent Quality Office: The agency  created a new senior position for overseeing patent quality—the  Deputy Commissioner for Patent Quality—in January 2015 to provide  a dedicated focus on the agency’s patent quality efforts.
Patent Quality Summit: USPTO held its first ever Patent Quality  Summit in March 2015. The Summit was designed for the public,  including internal and external stakeholders, to provide input to  USPTO about patent quality, specifically how the agency could  guarantee the most efficient process to review applications and to  ensure the issuance of the highest quality patents, according to the  USPTO Summit website. According to USPTO, the agency received  1,206 ideas for improvement for patent quality from all sources,  including the Patent Quality Summit and examiner forums.
Evolving programs of the Enhanced Patent Quality Initiative: USPTO  established 11 initiatives based on feedback from internal and  external stakeholders to help achieve its goals to enhance patent  quality, according to its Enhanced Patent Quality Initiative website.  These initiatives are in various stages, ranging from early  development to having been completed, according to a senior USPTO  official. Some of the evolving programs include:
Clarity of the Record Pilot Program: USPTO began this pilot in  February 2016 with about 130 examiners, and the pilot is expected to  end in August 2016. This pilot seeks to develop best practices for  examiners to enhance the clarity of all aspects of the prosecution  record, and to study the effect of implementing these best practices  during examination.
Clarity and Correctness Data Capture (Master Review Form): This  effort is expected to replace USPTO’s current quality assurance and  supervisory approaches to reviewing examiners’ work and will allow  the agency, for the first time, to collect consistent data across all the  reviews. Quality assurance officials also told us that using the form  would allow their office to have data on 50 percent more reviews this  year than in the past, resulting in a total of about 12,000 reviews of  examiners’ work this year compared with about 8,000 reviews per  year in the past. USPTO expects to implement the form agency-wide  in fiscal year 2017.
In addition to the actions taken under its Enhanced Patent Quality  Initiative, the USPTO’s Office of Patent Training develops and  implements training for examiners on a variety of topics, with a focus on  legal and policy matters. For example, the USPTO recently provided  examiners with training on functional claiming.
section_title: Patent Infringement Litigation Has Increased in Recent Years, and the Majority of Lawsuits Involved Patents Related to Computers and Computer Software
paragraphs: The number of federal district court filings of new patent infringement  lawsuits has generally increased between 2007 and 2015, from more  than 2,000 suits in 2007 to more than 5,000 suits in 2015 (see fig. 2).  Because lawsuits can include multiple defendants, we also analyzed data  on federal district court filings at the defendant level to account for  lawsuits in which a patent was asserted against multiple defendants.  Looking at the data in this way, we found that the number of defendants  in new patent infringement suits filed in federal district courts increased  from about 5,000 defendants in 2007 to more than 8,000 defendants in  2015, as shown in figure 2.
According to some stakeholders with whom we spoke, the decreases in  litigation that occurred in 2014, both in the number of suits and  defendants, were likely due to a key Supreme Court decision in that year,  Alice Corp. Pty. Ltd. v. CLS Bank Int’l. The Court in the Alice decision  held that where a patent claim is based on an abstract idea, which is not  patentable, merely using generic computer implementation does not  transform that idea into a patent-eligible invention. This decision has  limited the validity of what some stakeholders considered to be overly  broad and low quality patents thus preventing them from being used to  file infringement lawsuits. In addition, the ability of potential defendants in  patent infringement suits to file inter partes challenges to the validity of a  patent at the PTAB beginning in 2012 may have made some patent  owners reluctant to bring infringement suits, which could have contributed  to the decline in the number of suits according to some stakeholders we  spoke with.
Most patent infringement suits are filed in just a few of the 94 federal  district courts, and these suits may generally be brought in any district in  the country where the allegedly infringing products are sold. Patent  infringement suits are increasingly being filed in the predominantly rural  Eastern District of Texas (see fig. 3). In 2007, about 20 percent of all  patent infringement defendants were named in cases filed in the Eastern  District of Texas, and this percentage increased to almost 50 percent in  2015.
Historically, the Eastern District of Texas has been attractive to patent  owners filing infringement lawsuits because of the speed at which suits  moved to trial, and the perception of plaintiff-friendly juries, as we found in  2013. In addition, according to one published paper we reviewed,  judges in the Eastern District of Texas have implemented a number of  court rules and practices to attract patent infringement suits to their  district. For example, according to the paper and to some stakeholders  we interviewed, judges in the Eastern District of Texas have not been  suspending infringement lawsuits pending patent validity challenges at  the PTAB, as some had anticipated. In addition, most recently,  according to some stakeholders, some judges in Eastern Texas have  been reluctant to dismiss infringement allegations at early stages of the  litigation based on the Alice decision that calls into question the validity of  certain types of software-related patents. Our analysis shows a sharp  increase from 2014 to 2015 in the number of defendants that were  accused of infringing software-related patents in suits filed in Eastern  Texas, which have averaged about 85 percent of the defendants in that  district each year between 2007 and 2015. None of the other top six  district courts saw any large increases in patent infringement suits in  2015, including suits involving patents related to software or otherwise.  This indicates that the increase in suits in Eastern Texas drove the overall  increase in suits filed nationwide in 2015.
The number of defendants in patent infringement litigation was relatively  stable between 2007 and 2015, with the exception of lawsuits involving  patents for computers and communications technologies (see fig. 4).  This area includes technologies related to computer hardware and  software. The percentage of defendants in lawsuits involving patents in  this technology area increased from 38 percent in 2007 to 62 percent in  2015.
Patents related to computer and communications technologies are easier  to unintentionally infringe because they are more likely to be unclear and  overly broad, according to some stakeholders we interviewed and some  of the published research we reviewed. In addition, according to a 2003  Federal Trade Commission study, in industries such as computers and  communications, firms need to avoid infringing dozens, hundreds, or even  thousands of patents to produce just one commercial product.
Therefore, it is particularly challenging to develop innovative new  products where there are thousands of interrelated patents covering  similar technologies making it nearly impossible to avoid infringement,  according to some stakeholders we interviewed. These stakeholders also  noted that unclear and overly broad patents—an indicator of lower  quality—can harm innovation regardless of whether the patent owner files  an infringement lawsuit because just the threat of an infringement suit can  deter the development of new products.
Further, the majority of defendants in patent infringement suits were  involved in suits with software-related patents each year from 2009  through 2015 (see fig. 5). The number of defendants involved in  infringement lawsuits where software-related patents were asserted  generally increased through 2013, then dropped in 2014. As with  computer and communications technologies, some stakeholders told us  that software-related patents are easier to infringe because they also  often have overly broad claims, an indicator of low quality patents.
section_title: Additional Opportunities Exist to Improve Patent Quality at USPTO
paragraphs: USPTO has taken actions to address patent quality—most notably  through its Enhanced Patent Quality Initiative—but there are additional  opportunities for USPTO to improve patent quality. Specifically, USPTO  does not have a consistent definition of patent quality and has not fully  developed specific performance measures to assess whether its efforts  are affecting patent quality. Further, USPTO has not fully assessed  whether the time it allots for examination and the monetary incentives it  gives examiners for completing patent examinations faster affect patent  quality. In addition, USPTO has limited data available on PTAB decisions.  Finally, USPTO has not fully evaluated the effects of other policies and  procedures on patent quality.
subsections: section_title: USPTO Has Not Consistently Defined Patent Quality or Established Specific Performance Measures
paragraphs: USPTO does not currently have a consistent definition for patent quality,  which may limit its ability to assess the effects of its examination policies  and review processes—as well as its Enhanced Patent Quality Initiative— on patent quality. Several high level USPTO officials and the four  supervisory patent examiners that we interviewed told us there is no  consistent definition of patent quality at USPTO to guide the agency in its  overall operations. One examiner wrote in our open-ended survey  response that the USPTO appears to have no definition of patent quality  and that without a working definition, management’s focus on patent  quality is meaningless. Most of the stakeholders that we spoke with— including former high ranking USPTO officials, academics, and non- governmental organizations—said that it is important for USPTO to  develop a consistent definition of patent quality.
According to USPTO officials, one challenge in developing a consistent  definition is that the patent community holds varying definitions of patent  quality. For example, patent attorneys who defend companies against  patent infringement generally tend to favor clearly defined patents that are  easily understood in patent disputes, while patent owners tend to prefer  less well defined patents that may offer broader coverage for their  invention. USPTO officials offered a variety of definitions of patent quality.  Several officials focused on validity, or patentability (i.e., meeting  statutory requirements) and clarity, although we could not find this in  agency guidance or documents as USPTO’s definition. Some officials  also included aspects of patent examination, such as an application that  requires a limited amount of time for review by examiners in their  definition. While USPTO has found it difficult to clearly define patent  quality, most of the stakeholders we spoke with told us that they would  define patent quality as patent validity—that is, a quality patent would  meet all the statutory requirements for patentability and would be upheld  if challenged in a lawsuit or PTAB proceeding.
According to federal standards for internal control, one important internal  control activity is that management has established clear, consistent  agency objectives that allow the agency to assess the risks the agency  faces from external and internal sources. However, USPTO’s objectives  may not be clear because it does not have a consistent definition of  patent quality. Four supervisory patent examiners we interviewed told us  that without a consistent definition of patent quality, USPTO is unable to  standardize practices to improve patent quality. As a result, it is hard for  USPTO to define, measure, and work toward quality goals, according to  these supervisors. Moreover, without a definition for patent quality,  USPTO is at risk of having agency officials work inconsistently or at  cross-purposes in their attempts to fulfill the call to improve patent quality,  based on individual understandings of what patent quality means to each  staff person.
Further, although USPTO is taking steps to improve patent quality metrics  as part of the Enhanced Patent Quality Initiative, it has not established  specific goals or performance measures related to its strategic goal to  optimize patent quality and timeliness, which may limit its ability to assess  potential effects of its efforts on patent quality. USPTO’s 2014-2018  strategic plan includes the goal to “optimize patent quality and timeliness,”  but the patent quality objective does not include specific performance  measures that fully assess progress towards the goal. For example,  USPTO names seven objectives to achieve this goal, but six of the seven  objectives focus on timeliness, customer service, and process or  production goals rather than patent quality. For the one patent quality  objective, USPTO cites improving patent quality data and maximizing the  use of such data as two of its four performance measures (see fig. 6).
The Government Performance and Results Modernization Act of 2010  (GPRAMA) requires, among other things, that agencies establish  objective, quantifiable, and measurable performance goals, and establish  performance indicators to measure progress toward each performance  goal. Although USPTO is not required to comply with GPRAMA, we  have previously reported the practices established by the law, such as  establishing goals and performance indicators, can serve as a leading  practice for organizational performance management at lower levels  within federal agencies, such as individual programs or initiatives.  However, USPTO has not established such goals and indicators because  the office uses general terms to describe these two performance  measures and does not include measurable, quantifiable goals or  performance indicators on how the agency plans to define or collect  patent quality data or how, specifically, it will maximize use of such data.  Without such goals and indicators, USPTO cannot determine whether it is  meeting its goal of enhancing patent quality.
section_title: USPTO Has Not Fully Assessed the Time Allotted for Examinations or Monetary Incentives
paragraphs: USPTO has not fully assessed the time allotted for patent examinations  or its monetary incentive system, which may be at odds with patent  quality. Patent examiners are rated annually on their production and  docket management, among other elements. USPTO provides  examiners with monetary incentives, or bonuses, for timeliness and  production, but does not offer a bonus for producing high-quality work.  Three USPTO officials told us that there are trade-offs between timeliness  and patent quality, explaining that examiners cannot examine patents  quickly and, at the same time, grant patents that are of the highest  quality. One of these officials told us that the office’s focus on timeliness  currently trumps high quality work at the agency, potentially increasing  the tension between the goals of timeliness and quality.
Time allotments and incentives can lead to pressure for examiners to  complete their work quickly. Most of the stakeholders we spoke with told  us that examiners’ time pressures are one of the central challenges for  patent quality. The results of our survey of patent examiners confirm  that examiners are experiencing time pressures. Specifically, on the basis  of our survey, we estimate that, given a typical workload, about 70  percent of examiners have less time than needed to complete a thorough  examination. In addition, we estimate that more than 70 percent of  examiners worked voluntary or uncompensated overtime in the past 6  months to meet their minimum production goals (see fig. 7). Further,  incentives appear to motivate examiners to complete their work quickly.  For example, on the basis of our survey, we estimate that for 93 percent  of examiners, receiving bonuses for achieving production goals motivates  them to go above and beyond their base level of performance.
Additionally, on the basis of our survey, we estimate that nearly 70  percent of examiners experience pressure to avoid time-consuming office  actions. In addition, a few examiners we interviewed said that the  system as currently designed incentivizes an examiner to issue a patent  instead of issuing a final rejection, suggesting that when pressed for time  examiners tend toward granting patents. For example, USPTO rules and  procedures do not restrict the number of requests for continued  examination an applicant may file, so issuing the patent is the only means  USPTO has to end the examination process. This creates an environment  where patents may be granted that do not fully meet patentability  standards. In a 2015 study, USPTO economists found that patents  examined by primary examiners had 26 percent better odds of appearing  in a patent infringement lawsuit compared to similar patents examined by  junior examiners, which suggests that primary examiners, despite their  experience, may not have adequate time to ensure that the patents they  issue are always high quality. USPTO officials acknowledged that this  difference was likely due to the additional time and supervisory review  that junior examiners receive.
The precise effects of the time allotted for examinations and incentives on  quality are unclear because USPTO has not fully analyzed the effects of  current time allotments or incentives on an examiner’s ability to perform a  thorough examination. Since creating the time allotments in the 1970s,  USPTO adjusted the time allotted to examiners between fiscal years 2010  and 2012 and gave all patent examiners a total of 2.5 additional hours per  application. According to federal standards for internal control, agencies  should provide staff with the right structure, incentives, and  responsibilities to make operational success possible. The right  incentives and structure allow staff to be aligned with the agency’s  objectives. Without analyzing the time and incentives needed for  examiners to complete thorough examinations, USPTO cannot be  assured that its current time allotments and incentives support the  agency’s goal to optimize patent quality.
section_title: USPTO Has Limited Data on PTAB Decisions
paragraphs: The PTAB has been in operation since 2012 with hundreds of decisions  made to date on challenges to existing patents; however, statistics kept  by PTAB staff about the results of PTAB decisions are limited. For  example, the data available as of March 2016, did not specify which  precise claims in a patent were found to be unpatentable and why, or  which sources of prior art were used in the proceeding, both of which are  key data fields for potential analysis. Moreover, the data have not been  widely shared within USPTO. According to a PTAB official familiar with  the data, only a few USPTO staff in the Office of Patent Legal  Administration have asked for and received the data that PTAB staff have  compiled to date. Information on the results of the PTAB proceedings is  not regularly provided to USPTO managers and supervisory patent  examiners in the Technology Centers, according to PTAB staff.
According to federal standards for internal control, information from  internal and external sources should be obtained and provided to  management as a part of the agency’s reporting on operational  performance relative to established objectives. In addition, pertinent  information should be identified, captured, and distributed in a form and  time frame that permits people to perform their duties efficiently.  However, USPTO managers and staff in the technology centers do not  have data on trends in the outcomes of PTAB’s post-grant proceedings  because USPTO has not systematically pulled information from PTAB  decisions, has not widely shared PTAB information within the agency,  and has not analyzed the data for trends in patent invalidations to identify  whether additional training, guidance, or other actions are needed to  address issues related to quality. Without a process for USPTO  managers and staff to readily access the reasoning underlying PTAB’s  invalidation of patents or information on possible trends that may exist  with those problem patents, USPTO may be overlooking critical  information on problems the PTAB has found with patents issued by the  agency. As a result, USPTO may be missing opportunities for using this  information to make decisions that could help to improve patent quality,  such as improving guidance or providing additional training on potential  patent quality issues, and USPTO’s examiners are unable to learn  lessons from the PTAB decisions.
section_title: USPTO Has Not Fully Evaluated the Effects of Other Policies and Procedures on Patent Quality
paragraphs: USPTO has not fully evaluated the effects of other policies and  procedures on patent quality, which may also affect their ability to issue  high quality patents. Through interviews with USPTO officials and  stakeholders and our examiner survey, we identified several policies and  procedures that could affect patent quality:
Compact prosecution: This USPTO policy encourages examiners to  complete an examination within two office actions and to address all  statutory issues of an application in the first office action. One  examiner commented in our survey that compact prosecution compels  examiners to guess what unclear claims mean in order to search for  prior art. Further, based on our survey, we estimate that a change in  policy that would not require examiners to address every issue in the  first office action would help about half of USPTO examiners perform  examinations more effectively. In addition, some stakeholders raised  concerns about the effect of compact prosecution on patent quality,  with one stakeholder emphasizing that the policy does not work well in  an environment where patent applications are increasingly complex.
Unlimited number of patent claims: Applicants are allowed to include  any number of claims in a patent application. According to USPTO  officials, applicants’ ability to file unlimited claims can have a negative  effect on quality, because it is more difficult for examiners to fully  review an application with numerous claims in the time allotted; this  was also supported by our survey results. The 2015 study by  USPTO economists also supported this finding; it found that patents  that had more claims had better odds of showing up in patent  infringement lawsuits.
Requests for continued examination: Applicants are currently allowed  to file an unlimited number of requests for continued examination,  which is a request by an applicant to reopen examination of the patent  application after the prosecution of the application has been closed.  Applicants request continued examination most often after final  rejection of an application, according to USPTO officials. Such  requests provide applicants with virtually unlimited attempts to secure  a patent, which is problematic for patent quality, according to some  stakeholders. Some stakeholders also told us that such unlimited  requests can wear down examiners, making them more likely to  eventually grant the patent. While USPTO has reached out to patent  applicants to learn why they were using such continued examinations,  the number of requests for continued examination continues to pose a  burden for the agency’s examiners.
Federal standards for internal control direct agencies to comprehensively  identify risks and consider all significant interactions between the entity  and other parties. Once these risks have been identified, they should be  evaluated for their potential effects, including the significance of the risks  and the likelihood of their occurrence. USPTO officials acknowledged  that some of the agency policies discussed above could affect patent  quality; however, officials did not know the extent of the effects because  USPTO has not done an evaluation to determine the potential effects.  Without evaluating the effects of these policies on patent quality, USPTO  is at risk of continuing practices that may adversely affect patent quality.
Additionally, USPTO policies and procedures generally require clarity in  issued patents. On the basis of our survey, we estimate that additional  claim clarity requirements to applicants would help more than 80 percent  of examiners do their jobs more effectively. We identified two areas  where USPTO has encountered challenges with claim clarity in patent  applications that can affect patent quality: (1) unclear terms, and (2)  unclear or broadly worded claims including the use of functional claim  language.
First, patent applications that include unclear terms contribute to  challenges with patent quality. On the basis of our survey, we estimate  that 45 percent of examiners always or often encounter terms that are not  well defined in the patent applications’ specifications. Most of the  stakeholders we interviewed—including legal scholars and former high- ranking USPTO officials—as well as four supervisory patent examiners  and the majority of examiners responding to our survey—indicated that  requiring applicants to provide a glossary and define their terms would  help to improve patent quality. Further, USPTO officials said that  examiners and applicants that participated in USPTO’s Glossary Pilot  Program generally indicated benefits to including a glossary and that the  glossary improved claim clarity.
Second, patent applications that include unclear or broadly worded  claims, including those that use functional claim language, contribute to  challenges with patent quality. For example, on the basis of our survey,  we estimate that nearly 90 percent of examiners always or often  encountered broadly worded claims in applications they reviewed, and  for nearly two-thirds of examiners, applications with broadly worded  claims make completing a thorough examination more difficult. In  addition, dealing with functional claims, especially for software-related  patents, can be time consuming and difficult especially if examiners are  not aware of the applicant’s intent to use functional claiming language  under section 112, according to a few examiners we interviewed. On the  basis of our survey, we estimate that more than 40 percent of examiners  experience pressure to avoid making rejections that relate to claim clarity  (section 112). Also, according to our survey data, we estimate that for a  smaller percentage of patent examiners (41%) correctly applying  patentability standards for dealing with functional claims (section 112(f))  was very important compared to the percentage who reported adhering to  novelty and nonobviousness standards (sections 102 and 103) was very  important (81%). In a 2015 study, the USPTO economists found that  patents with broader claims—patents with fewer words in their claims— had better odds of appearing in patent infringement lawsuits than similar  patents. Further, that analysis found that patents that contained  functional claim language had 40 percent better odds of appearing in  patent infringement lawsuits than similar patents that did not contain this  language.
Some stakeholders we interviewed said requiring “claim charts” in patent  applications could help improve patent quality. Claim charts use one  column to present the claim and another column to present the limitations  and boundaries of that claim (see table 1). These charts, which are  commonly used in the federal courts and in PTAB’s post-grant  proceedings, provide additional information on the boundaries of a patent  claim. Similarly, according to some stakeholders, having applicants  include a functional claim check box to indicate whether they were using  functional claim language under Section 112(f) could help to improve  patent quality. If the box is checked it indicates that the examiner should  make sure that the functional claim is supported in the patent  specification.
By statute, an application for a patent must particularly point out and  distinctly claim the subject matter of the invention. USPTO regulations  require that the application include a description of the process of making  and using the invention in such full, clear, concise, and exact terms as to  enable a person skilled in the art to use and make the invention.  However, examiners continue to encounter problems with patent  application clarity because USPTO does not specifically require patent  applicants to clearly define the terms used in their applications, provide  additional means to clearly describe claim boundaries, or clearly identify  when they are using functional claiming language. Without making use of  tools to improve the clarity of patent applications, such as by having  applicants include a glossary to define the terms used in the application,  provide a claim chart, or indicate the use of functional claims through a  checkbox, the agency is at risk of issuing unclear patents that may not  comply with statutory requirements.
section_title: Conclusions
paragraphs: The U.S. patent system plays a vital role in our nation’s economy by  promoting innovation and supporting millions of jobs in innovation-rich  sectors. However, the quality of patents issued by the USPTO has come  under scrutiny in recent years, and our analysis suggests that some  agency policies and practices may negatively affect the quality of the  patents USPTO awards. It has taken several actions to elevate the  importance of patent quality within the agency, but additional  opportunities exist to improve the effectiveness of the agency’s patent  quality efforts.
USPTO does not have a consistent definition of patent quality that is  clearly articulated in agency guidance or fully developed measurable  goals and performance indicators to guide and evaluate work towards the  agency’s quality goals. Without a consistent definition of patent quality,  USPTO is at risk of having its staff work at cross purposes to improve  patent quality based on their individual definitions of patent quality.  Further, without improvements to measurable goals and performance  indicators, USPTO is at risk of not being able to fully measure and  capture key performance data on whether the agency is meeting its  strategic goal to optimize patent quality.
USPTO’s policies regarding the time allotted to complete patent  application reviews and monetary incentives that are based on the  quantity of the work examiners complete, not the quality of their work may  negatively affect the quality of issued patents. However, USPTO has not  analyzed its policies regarding examiner performance incentives and has  assessed the time allotted for patent examination only minimally since the  1970s. Without analyzing whether time allotments to complete a thorough  examination are sufficient, USPTO is at risk of issuing lower quality  patents due to examiners’ not having enough time to complete their work.  Our report on prior art searching recommended that USPTO reassess the  time allotted to perform a thorough prior art search. Without analyzing  the current incentive structure, USPTO cannot ensure that its incentives  are aligned with high-quality work.
Further, USPTO has limited data available on decisions from the PTAB to  assess its patent quality initiatives. The PTAB has reviewed patents and  invalidated some, or all, of the claims for hundreds of patents issued by  the USPTO. However, the PTAB data and analysis of these decisions are  limited in specificity as well as in distribution. Without identifying all of the  data fields that would be useful to track, establishing a means for USPTO  managers and examiners to easily access information—including  information such as why claims were found to be unpatentable—and  analyzing the data to identify potential trends, USPTO officials may be  overlooking key information that could help them provide additional  training or guidance or take other actions to address recurring issues.
Further, other USPTO policies and procedures—namely, compact  prosecution, unlimited numbers of patent claims, and requests for  continued examination—may affect patent quality, but the agency has not  evaluated the effects of these policies. Without evaluating the possible  effects of these policies on patent quality, USPTO is at risk of continuing  practices that could potentially affect patent quality.
Finally, without requiring greater clarity in applications—through the use  of glossaries, check boxes to signal functional claiming language, or claim  charts—USPTO is at risk of issuing patents that are overly broad and not  clearly worded—and may not comply with statutory requirements— thereby increasing the likelihood that the patent becomes the subject of  litigation.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce direct the Director of the  USPTO to take the following seven actions to help improve patent quality:
Develop a consistent definition of patent quality, and clearly articulate  this definition in agency documents and other guidance.
Further develop measurable, quantifiable goals and performance  indicators related to patent quality as part of the agency’s strategic  plan.
Analyze the time examiners need to perform a thorough patent  examination. This action could be taken in conjunction with the  recommendation in our report on USPTO’s prior art search  capabilities (GAO-16-479).
Analyze how current performance incentives affect the extent to which  examiners perform thorough examinations of patent applications.
Establish a process to provide data on the results of the PTAB  proceedings to managers and staff in the USPTO’s Technology  Centers, and analyze PTAB data for trends in patent quality issues to  identify whether additional training, guidance, or other actions are  needed to address trends.
Evaluate the effects of compact prosecution and other agency  application and examination policies on patent quality. In doing so,  USPTO should determine if any changes are needed to ensure that  the policies are not adversely affecting patent quality.
Consider whether to require patent applicants to include claim clarity  tools—such as a glossary of terms, a check box to signal functional  claim language, or claim charts—in each patent application.
section_title: Agency Comments
paragraphs: We provided a copy of our draft report to USPTO for review and  comment. In its written comments, which are reproduced in appendix III,  USPTO generally agreed with our findings, concurred with our  recommendations, and provided information on steps officials plan to take  to implement the recommendations. USPTO also provided additional  technical comments, which we incorporated, as appropriate.
USPTO stated in its response to the first recommendation that USPTO  already has a consistent definition for patent quality, specifically that a  quality patent is one that is correctly issued in compliance with all of the  requirements of Title 35 as well as relevant case law at the time of  issuance, which is consistent with how we define the term for this report.  However, in our audit work, we did not find evidence that this definition  was clearly articulated in agency documents and guidance or used in its  performance indicators and goals. We revised this recommendation to  clarify that USPTO should not only define the term, but also make this  definition clear in relevant documents.
In response to our second recommendation, USPTO said that it has  taken some steps to update and improve its performance indicators and  goals related to patent quality. In its technical comments, USPTO  suggested that we revise the report to recommend that USPTO further  develop its goals and performance indicators. We agree with this  suggestion and made the change. As USPTO further develops its goals  and performance indicators, we encourage the agency to more clearly  link these goals and indicators to its definition of patent quality.
In response to our seventh recommendation, USPTO said that, contrary  to the draft report’s findings, USPTO’s initial conclusion was that a  glossary did not make a meaningful difference in quality during the  prosecution of an application, though the USPTO is still analyzing  whether the use of a glossary has a long-term impact on a patent. In  response to this comment, we revised the statement in the report to more  closely align with the information that USPTO officials presented at its  March 8, 2016, Patent Quality Chat on the issue.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Commerce, the Director of  the USPTO, the Commissioner for Patents, and other interested parties.  In addition, the report will be available at no charge on the GAO website  at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To examine recent trends in patent infringement litigation, we obtained  patent infringement litigation data from two companies—RPX Inc. and  Lex Machina—that included all of the patent infringement lawsuits filed in  all 94 federal district courts between 2007 and 2015. These data included  information on the patents asserted in each suit, the defendants involved,  and the federal district court where the suit was filed. We conducted data  quality testing on the data from RPX and Lex Machina to look for missing  or out of range values, interviewed relevant officials, and reviewed  relevant documentation for the data and found the data to be sufficiently  reliable to determine recent trends in patent infringement lawsuits. We  also conducted 11 semi-structured interviews with stakeholders from  technology companies, venture capital investors, and others  knowledgeable about recent patent infringement litigation. In addition, we  reviewed published literature on the patent system, including reports from  patent researchers, the Federal Trade Commission, and the  Congressional Budget Office.
To examine what additional opportunities exist, if any, to improve patent  quality, we reviewed relevant laws and USPTO documents and  interviewed USPTO officials and representatives of the examiners’  union—the Patent Office Professional Association. We interviewed four  supervisory patent examiners and six patent examiners from a variety of  technology areas. We also conducted 11 semi-structured interviews with  patent stakeholders who were knowledgeable about patent quality and  USPTO, including legal scholars, former high-ranking USPTO officials,  and representatives from public interest non-governmental organizations.
Two of the 11 stakeholders we interviewed are currently serving in  leadership roles as board members of the American Intellectual Property  Law Association. In assessing USPTO’s efforts, we identified criteria in  the federal standards for internal control and USPTO’s strategic plan.
In addition, we conducted a web-based survey of a stratified random  sample of 3,336 eligible USPTO patent examiners from across 8 of the 11  the technology-based subject matter groups (referred to as technology  centers) into which USPTO examiners are divided. Fielded between  August and November 2015, the survey was designed to collect  information on USPTO’s approach to patent quality and how USPTO  might improve its patent quality efforts. To identify our survey population,  we obtained from USPTO a list of patent examiners as of May 2015. We  excluded examiners from three technology centers, as follows:
We excluded the Designs technology center because these  examiners work on design patents instead of utility patents; design  patents are outside the scope of this engagement and have different  statutory and administrative requirements than utility patents.
We excluded examiners who perform “re-examination” work and not  initial patent examination.
We excluded examiners in the patent training academy because  these examiners are recent hires who are in a 12-month training  program.
We also excluded examiners employed at USPTO for less than one year.  We then defined nine strata by technology center, with one technology  center separated into two strata, as described in table 2. Specifically, the  Transportation, Construction, Electronic Commerce, Agriculture, National  Security and License & Review technology center includes a diverse set  of technologies, including transportation, construction, agriculture, and  business methods. In our review, we separated the art units—subunits of  a technology center—focused on electronic commerce and business  methods (collectively referred to as business methods) in light of recent  legislation and court decisions related to business methods. This resulted  in 9 strata with a target survey population totaling 7,825 eligible  examiners. From this list, we drew our stratified random sample of 3,336  eligible USPTO patent examiners. We received responses from 2,669  eligible examiners for an 80 percent response rate.
Because we used a probability procedure based on random selections,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  quantified the sampling error and express our confidence in the precision  of our particular sample’s results at a 95 percent confidence interval. This  is the interval that would contain the actual population value for 95  percent of the samples we could have drawn. We designed our sample to  provide percentage estimates with 95 percent confidence intervals that  are within upper and lower bounds of 5 percentage points, within each  stratum. We oversampled based on an expected response rate of 70  percent; however, because we achieved a higher than expected response  rate, the upper and lower bounds for survey results within each stratum  are generally less than 5 percentage points. The only estimates for which  the upper and lower bounds exceed 5 percentage points are certain  results for the business methods stratum. In these instances, the upper  and lower bounds are between 5 and 6 percentage points. In this report,  our figures containing survey results show the upper and lower bounds  for results at the 95 percent confidence interval. For other estimates in the  report, we have not provided the upper and lower bounds in the text or  tables; however, those details for all survey results are available in the e- supplement related to this report, GAO-16-478SP.
The quality of survey data can also be affected by nonsampling error,  which includes, for example, variations in how respondents interpret  questions, respondents’ willingness to offer accurate responses,  nonresponse errors, and data collection and processing errors. To  minimize nonsampling error, we took several steps in developing the  survey and in collecting and analyzing survey data. Specifically, in  developing the survey, we worked with our survey professionals to,  among other things, draft questions that were clear and unbiased. We  pre-tested the survey in person with five USPTO staff: three examiners  who are also representatives to the examiners’ union, a supervisory  patent examiner, and a quality assurance specialist. We used these pre- tests to check that the questions were clear and unambiguous, used  correct terminology, requested information that could be feasibly  obtained, and were comprehensive and unbiased. We also obtained  comments on the survey from USPTO management and leadership from  the examiners’ union. In addition, we obtained a quality review by a  separate GAO survey methodologist. Based on these activities, we made  changes to the survey before administering it. Further, using a web-based  survey and allowing examiners to enter their responses into an electronic  instrument created an automatic record for each respondent. This  eliminated the potential for errors that could have resulted if we had used  a manual process to enter respondents’ data from paper surveys. In  addition, to account for the complex sample design, we used survey  software in our analyses to produce appropriate estimates and  confidence intervals, and the programs we used to process and analyze  the survey data were independently verified to ensure the accuracy of this  work.
To minimize nonresponse error, we made a variety of contacts with the  sample of examiners during the survey, including follow-up e-mails to  encourage responses. In addition, between October 20 and 23, 2015, we  attempted to follow-up via telephone calls to all 1,102 examiners who had  neither completed the survey nor told us that they were no longer  examiners. We also analyzed nonresponse bias to (1) assess whether  any factors were associated with examiners’ propensity to respond and  (2) to allow our analysis of respondents to properly reflect the sampling  universe of eligible examiners. To adjust the sampling weight for potential  nonresponse bias, we used standard weighting class adjustments based  on the sampling strata and the examiners’ years of experience at USPTO.  In this report and in the related e-supplement at GAO-16-478SP, we  present the survey results using the nonresponse adjusted weights, which  are generalizable to the eligible population of examiners.
We analyzed the responses to the survey for all examiners, as well as  responses by technology center and by the General Schedule (GS) level  of the examiners. We selected three categories of GS levels—less than  GS-13, GS-13, and greater than GS-13—because examiners at these  levels have different responsibilities and authorities when examining  patent applications. Specifically, examiners above the GS-13 level may  grant a patent or reject a patent application without additional review;  examiners below the GS-13 level must have their actions reviewed and  signed by a more senior examiner; and some GS-13 examiners are  transitioning from one GS level to the other.
For some other survey questions, we also reviewed examiners’ open- ended responses on selected topics. We selected those topics based on  our interviews with experts and USPTO officials as well as our analysis of  closed-ended survey responses. We selected the questions for which  examiners’ responses most frequently included keywords we identified for  each topic. An analyst conducted a keyword search of all responses to  the selected open-ended questions and coded responses containing the  keywords. A second analyst verified the initial analyst’s coding. Our report  provides some examples of examiners’ comments based on this review.  Examiners’ responses to open-ended questions are not generalizable to  other examiners. In addition, because we did not conduct a systematic  review of all open-ended responses to our survey, we do not report the  exact number of examiners who provided responses on the topics we  reviewed.
In addition, we conducted statistical tests of association on the results of  certain survey questions; all tests were independently verified to ensure  their accuracy. All tests of association were carried out at the 5 percent  level of significance and were Cochran-Mantel-Haenszel (CMH) Chi- square tests of general association. The testing was carried out in  SUDAAN, which is statistical software appropriate for the analysis of  survey data. The null hypothesis was that there is no association between  the two tested variables. When the association between two variables,  conditional on a third variable, is of interest, this relationship is referred to  as the stratum-adjusted CMH test. The test statistic is Wald Chi-Square.
We conducted this performance audit from November 2014 to June 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Patent Examination Process at the U.S. Patent and Trademark Office (USPTO) (Corresponds to Fig. 1)
paragraphs: Appendix II: Patent Examination Process at the  U.S. Patent and Trademark Office (USPTO)  (Corresponds to Fig. 1)
This appendix provides details on steps in the patent examination  process, including rollover information, depicted in figure 1.
section_title: Appendix III: Comments from the U.S. Patent and Trademark Office
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following individuals made  contributions to this report: Hilary Benedict (Assistant Director), Krista  Breen Anderson, Richard Burkard, John Delicath, Cindy Gilbert, Shilpa  Grover, Rob Letzler, Rebecca Makar, Rob Marek, Chris Murray, Eleni  Orphanides, Shep Ryen, Kelly Rubin, Ardith Spence, Sara Sullivan, and  Sonya Vartivarian.
section_title: Related GAO Products
paragraphs: Intellectual Property: Assessing Factors That Affect Patent Infringement  Litigation Could Help Improve Patent Quality. GAO-13-465. Washington,  D.C.: Aug. 22, 2013.
U.S. Patent and Trademark Office: Performance Management Processes.  GAO-10-946R. Washington, D.C.: Sept. 24, 2010.
Intellectual Property: Enhanced Planning by U.S. Personnel Overseas  Could Strengthen Efforts. GAO-09-863. Washington, D.C.: Sept. 30,  2009.
U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to  Reduce the Patent Application Backlog GAO-08-527T. Washington, D.C.:  Feb. 27, 2008.
U.S. Patent And Trademark Office: Hiring Efforts Are Not Sufficient to  Reduce the Patent Application Backlog. GAO-07-1102. (Washington,  D.C.: Sept. 4, 2007.
Intellectual Property: Improvements Needed to Better Manage Patent  Office Automation and Address Workforce Challenges. GAO-05-1008T.  Washington, D.C.: Sept. 8, 2005.
Intellectual Property: Key Processes for Managing Patent Automation  Strategy Need Strengthening. GAO-05-336. Washington, D.C.: June 17,  2005.
Intellectual Property: USPTO Has Made Progress in Hiring Examiners,  but Challenges to Retention Remain. GAO-05-720. Washington, D.C.:  June 17, 2005.