Intellectual Property: Patent Office Should Strengthen Search Capabilities and Better Monitor Examiners' Work
Published: 2016-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: USPTO examines patent applications to ensure that inventions are, among other requirements, novel and not obvious. USPTO patent examiners accomplish this by comparing applications to “prior art”—existing patents and applications in the United States and abroad, and nonpatent literature, such as scientific articles. Thorough prior art searches help ensure the validity of granted patents.
GAO was asked to identify ways to improve patent quality through use of the best available prior art. This report (1) describes the challenges examiners face in identifying relevant prior art, (2) describes how selected foreign patent offices have addressed challenges in identifying relevant prior art, and (3) assesses the extent to which USPTO has taken steps to address challenges in identifying relevant prior art. GAO surveyed a generalizable stratified random sample of USPTO examiners with an 80 percent response rate; interviewed experts active in the field, including patent holders, attorneys, and academics; interviewed officials from USPTO and similarly sized foreign patent offices, and other knowledgeable stakeholders; and reviewed USPTO documents and relevant laws.
section_title: What GAO Found
paragraphs: Experts and U.S. Patent and Trademark Office (USPTO) examiners described a variety of challenges in identifying information relevant to a claimed invention—or “prior art”—that can affect examiners' ability to complete a thorough prior art search in the time allotted and their confidence in their search efforts. These challenges include, among others, the quantity and availability of prior art, the clarity of patent applications, and USPTO's policies and search tools.
The European Patent Office and Japan Patent Office face similar challenges to USPTO in identifying prior art and use various approaches to help address them, such as leveraging work of other patent offices on related patent applications and integrating nonpatent literature into their search tools. In some cases, these approaches are coordinated with, similar to, or could inform USPTO actions.
USPTO has taken actions to address challenges in identifying prior art, but some actions have limitations. For example, USPTO is in the process of upgrading its search tools. However, examiners will still need to access a variety of external sources to meet USPTO's requirement to consider nonpatent literature. Federal internal control standards call for controls to evolve to remain effective and USPTO officials noted that the new search system can be expanded to include more nonpatent literature as the European and Japan patent offices have done. However, USPTO does not have a documented strategy for identifying additional sources. Without such a strategy, USPTO cannot be assured that its information technology investment will improve examiners' searches. USPTO is also taking steps to augment the number of, and consistency with which, reviews of examiners' work are conducted and documented, which could improve USPTO's monitoring of examiners' work. However, USPTO still faces limitations in assessing the thoroughness of examiners' prior art searches, because, for example, the agency has not established goals or indicators for search quality and may not be collecting sufficient information on examiners' searches to assess prior art search quality. Without monitoring examiners' prior art searches, the agency cannot be assured that examiners are searching all relevant sources of prior art and may not be able to develop appropriate responses as called for by federal internal control standards.
section_title: What GAO Recommends
paragraphs: GAO is making seven recommendations, among them, that USPTO develop a strategy to identify key sources of nonpatent literature, establish goals and indicators for prior art search quality, and collect sufficient information to assess prior art search quality. USPTO concurred with GAO's recommendations.

section_title: Letter
paragraphs: Scientific and technological innovations provide a foundation for American  economic growth and competitiveness and help to support a high  standard of living. The Constitution grants Congress the power to give  inventors a patent providing exclusive rights for a limited time to their  inventions to “promote the Progress of Science and useful Arts,” thereby  encouraging innovation. In 1836, Congress established the U.S. Patent  Office, later renamed the U.S. Patent and Trademark Office (USPTO), to  administer the patent system in accordance with patent laws. A patent  from USPTO gives its holder the ability to generally exclude others from  making, using, selling, or importing a patented invention in the United  States for up to 20 years from the date on which a successful patent  application was filed. Patent holders may enforce these rights through  patent infringement lawsuits. Third parties may challenge the validity of  an issued patent through USPTO proceedings or in federal courts. A  patent from USPTO does not offer protections abroad, but applicants may  seek patents for the same invention from foreign patent offices.
USPTO receives over half a million applications each year from inventors  seeking patents to protect the intellectual property stemming from their  work. When examining a patent application, a USPTO patent examiner  determines whether the claimed invention meets the legal requirements  for patentability as set forth in patent laws and federal case law. For  example, examiners must determine whether the invention is novel and  not obvious, as those concepts have been defined and interpreted by the  agency and the courts. USPTO examiners make such determinations by  comparing information in the application to other information relevant to  the claimed invention. Such information, generally known as prior art,  may include prior patents, patent applications, or nonpatent publications  describing a technology, among other things. Finding the most relevant  prior art when examining a patent application reduces the chance that  USPTO will grant a patent for something previously invented or for an  obvious combination of prior inventions.
Recent trends in disputes over patent validity in USPTO proceedings and  in federal courts have raised concerns about the overall validity of patents  issued by USPTO. Some observers, citing the extent to which the validity  of granted patents has been successfully challenged, believe that USPTO  examiners do not always identify the most relevant prior art, which has  resulted in granting some patents that may not meet the statutory  requirements for patentable inventions. Specifically, under the 2011  Leahy-Smith America Invents Act, a party other than the patentee may  bring a proceeding before the Patent Trial and Appeal Board to challenge  the validity of a patent. From September 2012 through February 2016,  719 patents have been partially or wholly invalidated based on prior art  submitted through one type of these proceedings. As a result of these  concerns and others about whether the U.S. patent system continues to  support innovation and competitiveness, Congress has continued to  consider patent reform legislation in recent years.
You asked us to identify ways to improve patent quality through the use of  the best available prior art during USPTO’s examination of patent  applications. This report (1) describes the challenges examiners face in  identifying relevant prior art, (2) describes how selected foreign patent  offices have addressed challenges in identifying relevant prior art, and (3)  assesses the extent to which USPTO has taken steps to address any  challenges in identifying relevant prior art. Our work for this report was  coordinated with work for our report on patent quality being issued today.
To describe the challenges examiners face in identifying relevant prior art,  we reviewed relevant laws and USPTO documents, and interviewed  USPTO officials, supervisory examiners, officials of the examiners’ union  (the Patent Office Professional Association), and a group of five  examiners serving as union representatives. We also conducted  semistructured interviews with 18 subject matter experts (experts) active  in the intellectual property field and knowledgeable about the subject of  prior art. To do so, we first identified the following five stakeholder groups  to obtain different views: intellectual property academics, patent holders,  patent attorneys, nongovernmental organizations, and patent data  experts. We then selected experts based on criteria we developed for  each group. When describing the views of these experts, “most”  represents more than 10 of the 18 experts we interviewed.
In addition, we conducted a web-based survey of a stratified random  sample of 3,336 USPTO patent examiners from across 8 of the 11  technology-based subject matter groups (referred to as technology  centers) into which USPTO examiners are divided. For one technology  center, we separated certain art units—subunits of a technology center— into a separate category, for a total of nine strata. We excluded  examiners employed at USPTO for less than 1 year, as of May 2015. The  survey collected information on examiners’ experience with prior art  searches and their opinions on challenges and options for improving  USPTO’s prior art search procedures and capabilities. The survey was  designed and administered in conjunction with our patent quality report  (GAO-16-490); therefore, questions also addressed USPTO’s approach  to patent quality and how the agency might improve patent quality.  Overall, we received responses from 80 percent of the examiners in our  survey sample, which we designed to produce estimates that are  generalizable to the population of patent examiners in our study overall,  as well as within each technology center. The survey questions and  detailed data on the results can be viewed on our website.
To describe how selected foreign patent offices have addressed  challenges in identifying relevant prior art, we interviewed officials at the  European Patent Office (EPO) and the Japan Patent Office (JPO), which  we selected based on comments observers made about the quality of  their work and because they are similar in size to USPTO. To assess the  extent to which USPTO has taken steps to address any challenges in  identifying relevant prior art, we interviewed USPTO officials and  reviewed agency documents related to prior art search procedures and  capabilities, including ongoing and planned efforts in the areas of  information technology resources, human capital management, and  international cooperation. In addition, we reviewed relevant results from  our survey of examiners. In assessing USPTO’s efforts, we identified  criteria in the federal standards for internal control; the Government  Performance and Results Act of 1993, as amended; USPTO’s manual  for patent examiners; and USPTO’s strategic plan. Appendix I provides  more detail on our scope and methodology.
We conducted this performance audit from November 2014 to June 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Patent Examination Process and Prior Art Searches
paragraphs: The focus of patent examination is to determine whether the invention in  a patent application satisfies the legal requirements for a patent, including  that the invention be novel and not obvious. As shown in figure 1,  USPTO’s patent examination process involves a variety of steps, at least  one of which includes prior art searches that examiners use to determine  whether an invention is novel and not obvious.
USPTO’s manual for patent examiners establishes certain requirements  and processes that examiners must follow in examining patent  applications, including performing prior art searches. There are generally  two types of prior art—patent literature or nonpatent literature. Patent  literature consists of previously issued U.S. or foreign patents and  published patent applications. Nonpatent literature consists of other  publicly available documents and can include such things as product  manuals, standards established by international organizations, textbooks,  periodicals, or conference presentations. Both patent and nonpatent  literature may be written in a language other than English, referred to as  foreign-language art in this report. USPTO’s manual for patent examiners  requires them to conduct a thorough prior art search and directs them to  consider U.S. patents, foreign patents, and nonpatent literature unless  they can justify with reasonable certainty that no more pertinent prior art  references can be found.
Examiners are expected to complete their examination of an application  in a certain number of hours. The time allotted varies depending on  factors such as the technology and the seniority of the examiner. For  example, an examiner reviewing an application related to wire fabrics and  structure may be allotted about 14 hours for examination, while an  examiner at the same experience level would be allotted about 32 hours  for an application related to data processing, such as database and file  management. The allotted time includes the time needed to review the  application, perform a search for prior art, and complete all office actions.  Examiners have minimum production goals, based on the time allotted,  for the number of office actions they must complete, and examiners may  earn bonuses for exceeding these minimum production goals.
USPTO uses several different information technology systems to assist  examiners in conducting prior art searches. For example, to search U.S.  patent literature, examiners use two systems, the Examiner’s Automated  Search Tool (EAST) and the Web-Based Examiner Search Tool (WEST),  to search the full texts of published patent applications since 2001,  patents granted since 1970, and optically scanned U.S. patents granted  from 1920 through 1970. These systems also include abstracts of some  foreign patents. Examiners may access additional foreign patent  documents through other web-based tools. In addition, USPTO’s  Scientific and Technical Information Center (STIC) operates systems that  can search U.S. patent literature and that store some nonpatent literature.  STIC also provides subscriptions to various web-based sources of  nonpatent literature. According to a USPTO document, the agency had  subscriptions to 119 different journals or external databases in 2014.  USPTO’s current search tools do not provide examiners with immediate  access to computer-generated translations, known as machine  translations; however, examiners can request human and machine  translation services from STIC and three contracted translation vendors  that currently cover 35 languages.
section_title: USPTO’s Patent Examiner Workforce
paragraphs: As of May 2015, USPTO had nearly 8,300 patent examiners across the  eight technology centers we reviewed (see fig. 2). USPTO uses the  General Schedule (GS) classification system for patent examiners, whose  levels range from GS-5 to GS-15. Examiners at the GS-14 level or  above (44 percent of the examiners in the technology centers we  reviewed, as shown in fig. 3) are generally primary examiners. Primary  examiners may accept or reject a patent application without additional  review. This level of authority is in contrast to junior examiners—most  examiners below the GS-14 level—whose work must first be reviewed by  a primary examiner before an office action can be sent to the applicant. At  the GS-13 level, some examiners are in the process of becoming primary  examiners. Supervisory patent examiners are at the GS-15 level and are  responsible for the day-to-day management of examiners.
section_title: Foreign Patent Offices
paragraphs: In addition to applying for a patent from USPTO, an inventor may also  seek patent protection in other countries for the same invention by filing in  multiple patent offices. Such interrelated patent applications are  described as a patent family. The World Intellectual Property Organization  has estimated that approximately half of all applications worldwide are  repetitive filings in a patent family, and the rest are initial filings.  According to the World Intellectual Property Organization, around 2.7  million patent applications were filed worldwide in 2014, of which 2.2  million applications were filed with patent offices in the United States,  China, Japan, South Korea, and Europe—known as the IP5. These five  offices, including EPO and JPO, each receive hundreds of thousands of  patent applications each year. EPO issues patents that cover 42  countries, most of which are member countries that are party to the  European Patent Convention. Applicants may apply to national patent  offices or apply through EPO for coverage in some or all of these 42  countries. EPO has three official languages, one of which must be used  for processing an application. These three languages are English (about  80 percent of applications), French (about 5 percent), and German (about  15 percent), according to EPO officials. EPO and JPO have about one- half and one-fifth as many examiners as USPTO, respectively. According  to JPO officials, there is a limit on the number of federal employees JPO  can have; therefore, 494 of its 1,702 examiners are fixed-term rather than  permanent employees, and JPO has begun outsourcing aspects of prior  art search. JPO officials stated that outsourcing some aspects of prior art  searches based on instructions from a government examiner frees up the  examiners’ time and allows the office to review more applications. Table 1  describes the workload and workforce of USPTO, EPO, and JPO.
section_title: Patent Examiners Face a Variety of Challenges in Identifying Relevant Prior Art
paragraphs: USPTO examiners face a variety of challenges in identifying relevant prior  art during patent examination. As shown in figure 4, the experts we  interviewed and examiners we surveyed cited challenges related to  certain attributes of prior art and patent applications and USPTO  examination policies, search tools, and human capital management. Our  survey results show the extent of the challenges may vary by technology  center or examiners’ GS level (see app. III). Because we surveyed a  generalizable stratified random sample of examiners and adjusted for  nonresponse, our results provide estimates for the entire population of  examiners in our study and, when reported by technology center, for each  of the technology centers we reviewed.
subsections: section_title: Challenges Related to Attributes of Prior Art and Patent Applications
paragraphs: Several attributes of prior art and patent applications present challenges  for examiners in identifying relevant prior art, including the quantity of  prior art, amount and relevance of prior art cited by applicants, availability  of prior art, and clarity of patent applications.
Quantity of prior art. The large volume of prior art available from  multiple sources makes searching for relevant prior art challenging,  according to most experts we interviewed as well as examiners  responding to our survey. For example, one expert noted that the amount  of patents, publications, and other nonpatent literature has grown  exponentially, making it harder to find relevant prior art in the time  allotted. Another expert said that technological innovations are occurring  at a tremendous rate, and that the growing volume of prior art  domestically and worldwide can be overwhelming. Based on our survey,  we estimate that 45 percent of all examiners in the eight technology  centers we reviewed find that the large quantity of art makes it somewhat  or much more difficult to complete a thorough prior art search in the time  allotted, while fewer examiners—34 percent—find that the quantity makes  it somewhat or much easier. However, responses varied among  technology centers, as shown in appendix III. For example, 30 percent of  examiners in the Computer Networks, Multiplex Communication, Video  Distribution, and Security technology center find that the quantity of art  makes it somewhat or much more difficult to complete a thorough prior art  search in the time allotted, compared to 60 percent of examiners in the  Mechanical Engineering, Manufacturing, and Products technology center  and 60 percent of examiners in the Chemical and Materials Engineering  technology center.
Amount and relevance of prior art cited by applicants. Examiners we  surveyed reported difficulties with the amount and relevance of prior art  references provided by applicants. USPTO requires applicants and  others assisting in filing an application to submit all information known to  be material to patentability. This information may include search results  from foreign patent offices or publications known to the individual.  According to USPTO policy, examiners will consider this information  when reviewing a patent application, which may require reviewing  numerous prior art references submitted by the applicant. Based on our  survey, we estimate that 82 percent of examiners sometimes, often, or  always encountered applications with what they considered an excessive  number of submitted art references in the past quarter. We estimate that  for most examiners (64 percent), excessive references make it somewhat  or much more difficult to complete a thorough prior art search in the time  allotted.
Considering all of the prior art references submitted by applicants can be  particularly challenging for examiners because applicants are generally  not required to explain the relevance of the references or to point  examiners to the particular portions of references that are relevant. In  commenting on this issue in our survey, one examiner recalled often  receiving information disclosure statements from applicants with  numerous prior art references, of which only a handful were relevant.  Based on our survey, we estimate that 88 percent of examiners  sometimes, often, or always encountered applications with irrelevant  references in the past quarter. Moreover, most examiners (57 percent)  find that irrelevant references make it somewhat or much more difficult to  complete a thorough prior art search in the time allotted. In contrast, 87  percent of examiners find that an application with relevant references  makes it somewhat or much easier to complete thorough prior art  searches in the time allotted.
Availability of prior art. The availability of prior art and difficulties  obtaining certain types of prior art are also challenges, according to most  experts we interviewed as well as examiners we surveyed. For example,  some relevant prior art may require a fee to access, may not be in a text- searchable format, may not be in a database, or may otherwise be  difficult to access. In particular, experts told us that certain general types  of prior art are more difficult to find, such as nonpatent literature overall  and software-related prior art (most experts) and foreign-language prior  art (6 of the 18 experts). Specific types of nonpatent literature that experts  identified as difficult to find include product documentation and  summaries, product databases, and user manuals; offers for sale and  public use; and information from standards-setting organizations. Prior art  related to computer technologies has also been difficult to find, according  to one expert we interviewed, in part because a lot of information is  commonly known in this field but may not be found by searching a public  database. In addition, one expert we interviewed suggested that  examiners do not have ready access to textbooks that are a good source  of prior art. Another suggested that prior art from before the mid-1970s  is difficult to find because patents issued before then have not been fully  digitized. This expert stated that this is a particularly challenging issue for  examiners in the mechanical technology centers because those  examiners tend to use older art more often than other examiners.
USPTO examiners we surveyed also reported difficulties obtaining  relevant prior art from searches for certain types of prior art more than  other types. In particular, on the basis of our survey, we estimate that 51  percent of all examiners find it somewhat or very difficult to obtain  relevant art from searches for foreign-language nonpatent literature.  Difficulties obtaining certain types of prior art may influence how often  patent examiners search for them. For example, 8 of the 18 experts we  interviewed suggested that examiners focus on searching patent literature  and may not thoroughly search nonpatent literature. Similarly, our survey  results in table 2 show that nearly all examiners always or often search  for U.S. patents and applications (an estimated 99 percent); we also  found that nearly all examiners always or often view this as the most  relevant type of art they consider (an estimated 98 percent of  examiners). In contrast, we estimate that 67 percent of examiners  always or often search for foreign patents, and 20 percent of examiners  always or often search for foreign-language nonpatent literature. More  examiners also find that foreign patents and foreign-language nonpatent  literature are difficult to obtain, compared to those that find U.S. patent  literature difficult to obtain. In analyzing our survey results, we found that  the difficulty examiners ascribed to finding foreign patent literature and  foreign-language nonpatent literature was statistically associated with  how often they reported searching for these types of prior art. How often  examiners search for certain types of prior art, and how difficult  examiners find those searches, also varies by technology center (see  app. III).
Clarity of patent applications. According to most of the experts we  interviewed and examiners we surveyed, the clarity of applications can  pose a challenge to finding relevant prior art. For example, as one expert  noted, there often are no standard terms to describe technologies, and  different applications may use different terms to describe the same thing.  According to four of the experts we interviewed, the absence of standard  terms is particularly a challenge for software-related applications.  Inconsistent terminology can make it more difficult for examiners to find  relevant prior art because searching for one term using keyword  searches—one common method of searching for prior art described by  USPTO officials—will not identify documents that use a different term. As  shown in table 3, examiners reported that issues with the clarity of the  application make it more difficult to complete thorough prior art searches  in the time allotted. In addition, based on comments examiners made in  our survey, examiners may face difficulties associated with applications  that have been translated from a foreign language. For example, one  examiner stated that “translation quality is often poor, and claims routinely  contain non-standard industry terms. Issued patents and publications  containing these non-standard terms also make searching in foreign  collections exceptionally challenging because it is not possible to  anticipate which synonyms to use.” In our patent quality report  (GAO-16-490), we provide additional information on how the clarity of  applications affects patent quality.
The effect of unclear applications may be exacerbated by USPTO’s  practice that examiners should attempt to identify all applicable grounds  for rejecting a claim or claims during their first review—a practice called  compact prosecution. According to USPTO’s manual for patent  examiners, this practice aims to avoid unnecessary delays. Such delays  could be caused by examiners waiting to continue examining an  application until previously identified issues are resolved. However, the  practice of compact prosecution may discourage examiners from  resolving any issues of clarity or ambiguity before conducting their initial  searches for relevant prior art. For example, as one examiner commented  in our survey, in order to follow the compact prosecution practice, an  examiner must guess what unclear claims mean in order to search for  prior art related to the claims. If examiners do not have a clear  understanding of the scope and claims of the invention, they may not  choose the most appropriate keywords for conducting their prior art  searches, which may lead examiners to miss relevant prior art that could  be found with more relevant keywords.
section_title: Challenges Related to USPTO’s Patent Examination Policies, Search Tools, and Human Capital Management
paragraphs: Several aspects of USPTO’s patent examination policies, prior art search  tools, and human capital management present challenges for examiners  in identifying relevant prior art, including the time pressures examiners  experience for prior art searches, USPTO search tools and capabilities,  the misclassification of patent applications, and examiners’ technical  competence.
Time pressures for prior art searches. According to most of the experts  we interviewed and examiners we surveyed, time pressures may reduce  examiners’ ability to conduct thorough prior art searches. These  pressures relate to USPTO’s system for allotting an expected amount of  time for examiners to complete an examination. For example, one expert  noted that the amount of time examiners are allotted decreases as they  become more experienced, and this may lead more senior examiners to  increasingly rely on art they know well instead of searching for new art.  As figure 5 shows, we estimate on the basis of our survey that 67 percent  of examiners find they have somewhat or much less time than needed to  complete thorough prior art searches given a typical workload. Our survey  also found that examiners’ perception of the sufficiency of time for  completing thorough prior art searches varies by technology center (see  app. III). For example, an estimated 37 percent of examiners in the  Mechanical Engineering, Manufacturing, and Products technology center  reported having much less time than needed to complete a thorough prior  art search, compared to 20 percent of examiners in the Biotechnology  and Organic Chemistry technology center.
In analyzing our survey results, we found that how often examiners  searched for foreign patent literature, scientific articles or presentations,  or foreign-language nonpatent literature was statistically associated with  their description of the sufficiency of time they had to complete a thorough  prior art search. Further, we asked examiners about overtime worked to  meet their minimum production goals. A majority of examiners (an  estimated 72 percent) worked voluntary/uncompensated overtime in the  past 6 months to meet their goals, as shown in figure 6. An estimated  30 percent of examiners worked an average of more than 10 hours of  voluntary/uncompensated overtime per biweekly period, although  examiners’ overtime varied by GS level (see app. III). However, on the  basis of our survey, an estimated 56 percent of examiners experience no  pressure to work overtime. In a 2007 report on USPTO’s efforts to hire  and retain an adequate workforce, we found that an estimated 70 percent  of examiners worked voluntary/uncompensated overtime in the previous  year. Our patent quality report (GAO-16-490) provides additional  information on how time pressures may affect patent quality.
Search tools and capabilities. Experts we interviewed and examiners  we surveyed had mixed opinions on USPTO’s search tools and  capabilities. Of the 18 experts, 10 agreed that the search tools and  capabilities available to examiners create challenges to conducting  thorough prior art searches. In general, these experts characterized  USPTO’s search tools as being less advanced and more heavily reliant  on keyword searching than other available tools. For example, one expert  said that examiners’ electronic prior art searches predominantly rely on  using keywords, and this is a problem if the examiners do not know the  most appropriate keywords. Another expert suggested that USPTO needs  more comprehensive databases of prior art.
Based on our survey, examiners generally find that the search tools  available to them from USPTO and from third parties make it easier to  complete prior art searches, but they also find that other tools would  help. Specifically, we estimate that a majority of examiners agree that  certain search tools not currently available would make prior art searches  somewhat or much easier, including a search engine that (1) can  automatically search for concepts and synonyms related to the search  terms entered by the examiner (an estimated 76 percent of examiners),  and (2) automatically generates relevant art, without keyword entry,  based on all the claims in an application (69 percent) or an application’s  specification (69 percent). In addition, a group of four supervisory patent  examiners we interviewed said that it would be more efficient to search  for prior art in one tool, with a single search method that covered multiple  sources of prior art, including nonpatent literature. Currently, relevant  nonpatent literature may appear in different journals or databases that  cannot be searched with a single search function. Similarly, examiners  commented in our survey about the difficulty of searching nonpatent  literature or requested an easier way to search for it. For example,  examiners requested a method for searching nonpatent literature that is  integrated with their systems for searching patent literature. In some  cases, USPTO examiners may not be able to use public search engines  because searching for specific terms or concepts from an unpublished  application may put the confidentiality of the application at risk.
Examiners also requested improvements to USPTO’s current translation  capabilities. On the basis of our survey, a majority of examiners (an  estimated 86 percent) agree that having access to immediate machine  translation of foreign-language documents would make it somewhat or  much easier to complete a thorough prior art search in the time allotted,  while about half of examiners (an estimated 50 percent) find that  additional translators would make it somewhat or much easier.
Misclassification of patent applications. According to examiners,  another challenge they face is misclassification of patent applications.  Misclassification occurs when an application is not classified into the  group most closely associated with the invention. This can result in  applications being routed to the wrong USPTO technology center or art  unit for examination. Although only 2 of the 18 experts we interviewed  raised concerns about the misclassification of patent applications, on the  basis of our survey, an estimated 75 percent of examiners encountered  misclassified applications sometimes, often, or always in the past  quarter, and an estimated 76 percent of examiners find that the  misclassification of patent applications makes it somewhat or much more  difficult to complete a thorough prior art search in the time allotted.  According to our survey, how often examiners encountered misclassified  applications in the past quarter varied by technology center (see app. III).
Misclassified applications pose particular challenges for examiners,  according to examiners’ comments to our survey. For example, while  examiners receiving misclassified applications can request that an  application be transferred to a different art unit, they may encounter  difficulties doing so, according to examiners’ comments in our survey.  Misclassified applications that are not transferred can negatively affect  examiners’ confidence in their work and the quality of the examination,  based on examiners’ comments in our survey. This is because when  applications are routed to the wrong art areas, they may be reviewed by  examiners who do not have appropriate knowledge and experience to  understand the invention or relevant prior art and determine appropriate  search strategies and terms. Additionally, misclassified applications that  are never corrected could be difficult for examiners to subsequently use  as prior art in later examinations. This is because misclassified  applications may be difficult to find when examiners search for patent  literature using technology-specific patent classification categories.
Ensuring examiners’ technical competence. According to most experts  we interviewed, ensuring that examiners have sufficient and appropriate  technical backgrounds, knowledge, or skills for conducting thorough prior  art searches is also a challenge. The role of patent examiner is a difficult  one that can take years to learn, and examiners with less education or  work experience, or who are not abreast of advances in a particular  technology area, may not have the technical knowledge necessary to  identify relevant prior art. According to USPTO officials, an examiner’s  technical knowledge enables him or her to understand the invention being  searched, and if the examiner does not understand the invention, he or  she may not know what to search for, where to look, and when to stop  searching. In addition, we estimate on the basis of our survey that 82  percent of examiners find it somewhat or much easier to complete a  thorough prior art search in the time allotted for applications with a subject  matter in which they have knowledge of existing prior art based on their  education or previous work experience.
As of May 2015, approximately 39 percent of all examiners in the  technology centers we reviewed had been at the agency for less than 5  years, and USPTO has historically faced challenges in retaining  examiners. In addition, as of September 2015, a majority of examiners  in the technology centers we reviewed (61 percent) did not have a degree  beyond a bachelor’s degree when hired. USPTO officials told us that the  agency has aimed to match new hires’ previous work experiences and  educational backgrounds to technology centers. When we asked USPTO  how often examiners in each technology center have technical work  experience or education relevant to their art unit, agency officials  answered “always” for five centers, “often” for three centers, and  “sometimes” for one center. However, our survey found that in the past  quarter, less than half of examiners—an estimated 42 percent—always or  often encountered applications with a subject matter in which they have  knowledge of existing prior art based on their education or previous work  experience.
The training examiners receive—particularly continuing training in  evolving technologies—may also affect their ability to maintain the  technical competence they need to effectively identify relevant prior art.  Overall, an estimated 60 percent of examiners found that the continuing  education they received from USPTO in their art area in the past year  was at least somewhat useful. However, 22 percent of examiners had not  taken such training or had not been offered such training, with  responses by technology center ranging from an estimated 10 percent to  34 percent of examiners (see app. III). We found in 2005 that examiners  were reluctant to attend voluntary training given the time demands  involved. Similarly, according to the group of five examiners we  interviewed in 2015, USPTO did not offer enough ongoing technical  training and did not always give examiners sufficient time to complete  training.
section_title: Patent Offices for Europe and Japan Use Various Approaches to Help Address Challenges in Identifying Relevant Prior Art
paragraphs: EPO and JPO use several approaches that may help their examiners  address challenges in identifying prior art similar to those that experts and  respondents to our survey of patent examiners cited. These approaches  include worksharing, creating internal databases of nonpatent literature,  using patent classification systems, adopting advanced search tools,  hiring and training of examiners to promote technical expertise, and  incorporating review and audit procedures.
Worksharing. EPO and JPO have systems to share search and  examination results. The exchange of prior art searches or patent  examination results among patent offices—referred to as worksharing— facilitates more efficient prior art searches for applications filed in multiple  countries, according to foreign officials. As we discussed above,  approximately half of all patent applications worldwide are part of patent  families filed in multiple offices. To gain patent protection in multiple  countries, inventors may file applications for the same invention in several  different offices. Because the underlying invention is the same, the search  and examination results at one office may be useful to examiners in  another office. Therefore, worksharing can ameliorate the challenges  posed by the quantity of potential prior art that examiners must search  and the time pressure examiners face by allowing patent offices to  leverage others’ work and knowledge. The IP5, a multilateral forum of the  five largest patent offices, identified worksharing as the main tool for  addressing an increasing number of patent applications, while helping the  patent offices conduct timely, quality examinations. JPO officials noted  that worksharing systems allow more efficient access to foreign patent  literature and help JPO examiners more easily see foreign examiners’  work. JPO examiners still have to review the relevant literature  themselves, but access to foreign examination results may help JPO  examiners identify relevant art or sources of prior art. An EPO official also  noted that worksharing could give offices access to search reports of  unpublished applications at other offices, and that these search reports  may include prior art relevant to similar applications filed with EPO.
Creating internal databases of nonpatent literature. EPO and JPO  incorporate specific nonpatent literature resources into their offices’ main  search tools, which may allow examiners to more efficiently search both  patent and nonpatent literature by keyword, rather than searching multiple  sources individually. Officials at these offices said that incorporating these  resources into their main search tools allows examiners to consider a  wide array of nonpatent literature sources.
According to EPO officials, EPO’s primary search tool allows examiners  to use a single interface to search through all of EPO’s internal databases  as well as some external databases of prior art. In addition to creating  an internal database of nonpatent literature documents to facilitate  examiners’ searches, EPO also added descriptive metadata to index  these documents to help examiners find the most relevant documents.  EPO’s system also incorporates features that make it user-friendly—for  example, grouping documents to avoid examiners facing dozens of  copies of the same article from different sources. EPO officials stated  that examiners find that these groupings enhance their ability to search  through larger quantities of prior art. Overall, EPO officials identified their  search engine and the large number of indexed prior art resources  available within the search engine as strengths of the office in addressing  the quantity of prior art available. EPO officials said that the office prefers  to have internal collections of these resources so they may be integrated  into its search tools, but some sources remain external and separate from  EPO’s search systems. Because the cost of bringing sources into EPO’s  search systems is high, the office needs a strong business case to justify  the expense. According to these officials, publishers of academic journals  and other nonpatent literature often raise subscription prices annually, so  the office also must decide what to exclude, because there is simply too  much nonpatent literature for the office to include everything.
According to JPO officials, JPO has internal databases of both patent and  nonpatent literature and also subscribes to external databases for  nonpatent literature. JPO’s internal databases include JPO patent  literature, some foreign patent literature, and selected nonpatent  literature. JPO annually selects useful nonpatent literature to add to its  database based on recommendations from an internal committee of  examiners and officials. In addition, JPO uses commercial databases of  nonpatent literature to augment its internal database and annually  evaluates whether each external database is useful for prior art search.
Using patent classification systems. EPO and JPO use classification  systems to help examiners narrow their searches and find relevant patent  literature even if the patent application uses inconsistent terms or is  written in a foreign language. EPO and JPO classify applications and  patents into groups that describe specific components of the invention.  Examiners may then look at other applications and patents within the  same group to find similar inventions. A system that has more groups— sometimes called classes—allows for finer distinctions, and prior art  searches within a class will produce fewer, more focused results.  Reducing the number of results an examiner needs to consider  decreases the time needed for the prior art search, thereby improving  efficiency. Furthermore, by grouping similar patents into classes that  describe details of the inventions, classification systems allow examiners  to conduct prior art searches based on the classes assigned to an  application, independent of its language and wording. This can improve  search quality by producing results that would not be found from a  specific keyword search but are nonetheless closely related. According to  EPO officials, this can be particularly helpful for finding documents in  Asian languages that are otherwise difficult for EPO examiners to search,  and it facilitates searches that are less dependent on the examiner’s  choice of keywords.
Adopting advanced search tools. EPO and JPO officials told us that  examiners in both offices primarily use classification and keyword  searches instead of more complex search tools. However, EPO uses  some automated tools that assist in keyword-based searches or provide  automated search results. One EPO tool provides recommended search  terms in its three official languages to examiners, based on a database of  examiners’ past searches. This tool can improve search efficiency by  providing additional search terms, such as synonyms or translations, that  the examiner may not have initially considered. Another EPO tool  provides examiners with the results of an automated search based on the  concepts and words drawn from the claims in applications. The results  are in the form of a ranked list of potentially related documents, which  provides a starting point for examiners before they begin a manual  search.
Hiring and training examiners to promote technical expertise. EPO  and JPO hire examiners with technical expertise and provide several  years of training, according to officials from these offices. EPO only  recruits students with at least a master’s degree and almost all JPO  examiners join the office after graduate school, according to EPO and  JPO officials. In addition, JPO’s outsourced aspects of prior art searches  are often carried out by retired engineers or technical experts, and  according to JPO officials, their experience may enhance the quality of  patent application examinations. At EPO, examiners must also meet a  requirement that they understand English, German, and French. EPO  examiners spend their first 2 years in training, alternating periods of  classroom training with periods of on-the-job training. JPO examiners  spend their first 2 to 4 years (depending on their academic degree and  work experience) as assistant examiners and receive practical training by  a supervising examiner. Thereafter, examiners in both offices have  opportunities for additional technical training. According to EPO officials,  EPO examiners need 3 to 4 years before they are fully trained, and 4 to 5  years in certain fields, such as biotechnology. At both offices, examiners  also tend to spend their entire career with the office. JPO officials  identified the longevity of examiners as a strength, stating that examiners  are well trained and have experience that helps ensure examination  consistency.
Incorporating review and audit procedures. EPO and JPO have  procedures to review examiners’ work before issuing or denying a patent.  At EPO, applications are examined by a panel of three examiners: a first  examiner, second examiner, and chair. According to EPO officials, the  first examiner generally performs the bulk of the search and  administrative tasks, with the other two examiners actively discussing and  approving office actions. Examiners from a particular technology area are  assigned to each panel randomly. An individual examiner may serve as  first examiner on some applications and second examiner or chair on  others. EPO’s quality management system also calls for random audits,  including a minimum of two prior art search audits for each examiner per  year, and chairs of the examination committees are required to record  data on the quality of their examinations.
JPO officials also reported taking steps to ensure that the patents it  issues are of high quality. These steps include a director quality check,  consultations with other examiners, and quality audits of a sample of  examinations. According to JPO’s quality manual, JPO directors conduct  a quality check on examiners’ decisions to grant or reject applications  before the office actions are sent to the applicants. The manual also  encourages examiners to consult with their directors or other examiners  who may be able to provide guidance in examining an application. In  2014, examiners recorded approximately 83,000 consultations and  received approximately 243,000 new applications, according to JPO  officials. Experienced managers or examiners who serve as quality  management officers also perform quality audits on randomly selected  applications. According to JPO officials, the office has around 90 such  officers in particular technology areas who review entire examinations  and complete independent searches, and 4 wide-area officers who focus  on the appropriateness of examiners’ decisions but do not conduct  additional searches. According to these officials, while the number of  audits is not large enough to allow tracking of individual examiner  performance, it allows for statistical monitoring of examination quality  overall.
section_title: USPTO Is Taking Actions to Address Challenges in Identifying Prior Art, but Some Actions Have Limitations
paragraphs: USPTO has taken or begun planning various actions that may help  address challenges in identifying relevant prior art, but some of these  actions have limitations that may hinder their effectiveness. USPTO’s  actions span the following areas: (1) leveraging the work of foreign patent  offices, (2) encouraging submission of prior art from third parties, (3)  improving prior art search tools, (4) monitoring examiners’ prior art  searches, (5) evaluating the agency’s system for determining the amount  of time examiners are allotted to examine patent applications, and (6)  strengthening the technical competence of the examiner corps. These  actions may help address challenges related to prior art and applications  and to examination policies, search tools, and human capital  management. In some cases, these actions are coordinated with, similar  to, or could be informed by approaches taken by EPO or JPO.
subsections: section_title: USPTO Has Taken Actions to Leverage Related Work by Foreign Patent Offices, but Has Experienced Challenges Working with EPO to Adopt a New Patent Classification System
paragraphs: USPTO has taken actions to leverage the work of foreign patent offices  just as EPO and JPO have done, but the agency has experienced  challenges working with EPO on a new patent classification system.  USPTO is collaborating with foreign patent offices to give examiners  access to examination files from those offices. Specifically, USPTO has  (1) contributed data to information technology systems that share  published information among patent offices; (2) engaged in pilot programs  to collaborate with other offices during search and examination of related  patent applications; and (3) adopted a new, joint classification system  with EPO. These efforts may help address several of the challenges we  identified relating to prior art and applications, including the large quantity  of art available and the availability of prior art, particularly foreign patents  and art written in foreign languages.
First, USPTO contributes data to three worksharing systems that make  applications, prior art citations, and examination results from foreign  patent offices available to examiners and the public. One system, the  Common Citation Document, provides bibliographic information, such as  title and source, of patent and nonpatent prior art citations. USPTO,  EPO, and JPO launched this system in 2011, and have since expanded it  to include information from additional patent offices. Another system,  Global Dossier, provides access to the examination history of  applications, such as office actions and other correspondence. USPTO  examiners gained access to Global Dossier in 2015. USPTO officials  described Global Dossier as a system that allows examiners to quickly  and easily view office actions from participating foreign patent offices.  EPO officials described these two systems as complimentary, with each  system providing advantages based on the specific search task involved.  A third system, PATENTSCOPE, run by the World Intellectual Property  Organization, provides access to applications filed with multiple offices  under the Patent Cooperation Treaty as well as patents granted by  several regional and national offices. By allowing examiners to review  prior art that examiners from other offices found useful in examining  patent applications, these systems may help address the challenges  related to managing the large quantity of prior art and accessing prior art  that would otherwise be difficult to obtain.
Second, to enhance collaboration on related patent applications filed in  multiple countries, USPTO has entered into two pilot programs to jointly  examine certain applications: one with JPO and one with the Korean  Intellectual Property Office. Applicants who file related patent applications  with USPTO and either JPO or the Korean Intellectual Property Office  may request to enter the pilot—known as the Collaborative Search Pilot  Program—and receive expedited review of their applications. In the pilot  with JPO, USPTO examiners consider prior art from both JPO’s search  results and their own search results before responding to the applicant  with the first office action. In the pilot with the Korean Intellectual Property  Office, examiners perform independent searches and examinations, then  compare results prior to final office actions. According to USPTO officials,  these collaborative efforts could improve searches for patent and  nonpatent literature in foreign languages, as examiners from multiple  countries will perform searches in their native languages and share the  results.
Third, USPTO, in collaboration with EPO, designed and implemented a  new, joint patent classification system meant to improve examiners’ ability  to find relevant patent literature from the United States, Europe, or other  offices that adopt the system. However, USPTO and patent examiners  have experienced some challenges with this effort. As discussed earlier,  by grouping similar patents, classification systems allow examiners to  conduct prior art searches independent of an application’s language and  wording and can improve search quality by producing results that would  not be found from a specific keyword search. According to USPTO  officials, the U.S. Patent Classification system became outdated following  budget constraints in the early 2000s that undermined the agency’s  efforts to update the classification system in response to technological  change. After identifying a need to revise its classification system,  USPTO partnered with EPO to create a new system to be jointly used by  both offices. USPTO officially adopted the Cooperative Patent  Classification (CPC) system in January 2015. According to USPTO  officials, CPC gives examiners greater access to foreign patent literature  compared to the prior system and can be expanded more easily to  include new technologies.
USPTO has found that classifications for some U.S. patent applications  under the CPC system are inconsistent with EPO’s classifications, and  USPTO examiners have reported difficulty with the system. Officials from  both USPTO and EPO told us that there were disagreements between  the two offices on how to classify some applications, such as on which  categories to use or the number of categories to which applications are  assigned. Additionally, USPTO examiners in certain technology centers  reported difficulty with the CPC system in our survey. On the basis of  survey responses from our random probability sample of examiners, we  found that some examiners find that the CPC classifications make it  somewhat or much more difficult to complete a thorough prior art search  in the time allotted, as shown in table 4. For example, on the basis of our  survey, CPC classifications make it somewhat or much more difficult to  complete a thorough prior art search in the time allotted for an estimated  53 percent of the examiners in the Mechanical Engineering,  Manufacturing, and Products technology center. On the other hand, an  estimated 42 percent of examiners in the Computer Networks, Multiplex  Communication, Video Distribution, and Security technology center find  CPC classifications make it somewhat or much easier to complete a  thorough prior art search in the time allotted. In response to an open- ended survey question, one examiner described their difficulty, saying  that proper classification of prior art is essential to quality and efficiency of  examination, that classification of prior art in the CPC system is  inconsistent, and that many CPC categories still contain large amounts of  prior art despite having more detailed technology categories. USPTO  officials we interviewed also told us that the number of documents in  some CPC categories was greater than those in comparable categories  under USPTO’s previous classification system.
According to USPTO officials, adopting the CPC system helps examiners  give greater consideration to foreign patent documentation. As we note in  this report, foreign patent and foreign-language nonpatent literature is  sometimes difficult for examiners to find, according to the experts we  interviewed and our survey of examiners. However, the CPC system’s  usefulness to examiners depends, in part, on the consistency with which  the system is applied by its users. Inconsistently applying the  classification system to applications undermines its usefulness because  similar applications may not be placed in the same category by USPTO  and EPO. Also, misclassified applications may require examiners to  familiarize themselves with new technologies, which increases the  amount of time it takes for them to conduct their prior art searches and  may negatively affect their confidence in the results. At USPTO,  classification contractors apply an initial classification to applications  before they are routed to an examiner. The examiner may revise the  classification during examination. To help transition to the new system,  USPTO provided examiners and classification contractors with training on  how to classify applications and how to search for references using the  CPC system, assigned lead quality experts to each technology center to  provide training and assistance, and established a feedback tool for  examiners to report issues with the new system. According to USPTO  officials, the agency has not identified specific reasons that account for  differences between its classifications and EPO’s. However, USPTO is  updating its contract for initial classification services to require a higher  level of expertise and is drafting a plan to assess performance under the  new contract.
The agreement between USPTO and EPO that supports the CPC system  commits both offices to exchange best practices on classifications and  aim for a high level of consistency, but it does not include a specific target  or method for assessing consistency. USPTO officials told us that they  are actively working with EPO to identify ways to measure and improve  consistency, and that both offices have developed their own methods for  measuring the consistency of CPC classifications. However, as of March  2016, EPO and USPTO had not agreed upon a measure of consistency  between the offices. In the context of interagency collaborations, we have  previously reported that agencies can enhance and sustain their  collaborative efforts by, among other things, defining and articulating a  common outcome and establishing mutually reinforcing or joint  strategies. Establishing agreement with EPO on a target of consistency  in classifying patent applications and a plan for monitoring consistency  would help to implement this practice and would establish an internal  control to support the reliability and usefulness of the CPC system.
section_title: USPTO Has Taken Actions to Encourage the Submission of Prior Art by Third Parties
paragraphs: Since 2012, USPTO has had authority to receive submissions from third  parties that may help identify potentially relevant prior art for published  patent applications. This process, sometimes described as  crowdsourcing, allows experts and other interested parties to share  documents that they believe will aid USPTO’s examination of an  application. Prior art submitted by third parties can help identify prior art  that is not immediately available to examiners (e.g., because it is not  available electronically or without payment) or that may be difficult to find.  By potentially reducing the need to search for this prior art, these  submissions may address challenges related to the quantity and  availability of prior art. According to information from USPTO, third parties  submitted prior art for less than 1 percent of the approximately 600,000  patent applications received per year since third-party submissions began  in 2012.
In addition, based on our survey, we estimate that 83 percent of  examiners have rarely or never seen an application with a third-party  submission. However, USPTO evaluations of third-party submissions  found that they were generally useful to examiners and that the  documents were sometimes not available to examiners through USPTO  sources. According to USPTO officials we interviewed, an internal review  of 300 randomly selected applications for which a third party submitted  prior art found that examiners used a submission to reject a patent  application in 20 to 25 percent of such cases. Furthermore, USPTO  reviewed approximately 2,500 submissions of nonpatent literature prior  art it received since September 2012, and found 753 unique, English- language prior art documents (30 percent of the submissions received)  that were otherwise not available to examiners. These documents were  mostly journal or other scholarly articles, but also included other types of  nonpatent literature, such as marketing materials, book chapters,  conference proceedings, presentations, and manuals that could not be  found in the agency’s internal or external nonpatent literature sources.
Because potential third-party submitters have to know about an  application before they can submit potentially related art, USPTO has  taken steps to allow third parties to more easily monitor published  applications. The agency created a subscription service, whereby users  will be notified when applications are published that contain user-chosen  keywords. Beyond these efforts, USPTO also considered expanding use  of third-party submissions of prior art by allowing examiners to use  crowdsourcing to solicit such art for specific applications or topics.  According to USPTO officials, the agency decided not to pursue this  option, at least in part, because the agency is uncertain about its legal  authority to do so. Specifically, while statute allows USPTO to collect  third-party submissions, the law requires the agency to establish  procedures to ensure that no protest or opposition is initiated during  examination of a patent application. According to USPTO officials, it is  unclear if requests for third-party submissions on a specific application  could be viewed as allowing opposition.
section_title: USPTO Is Making Improvements to Prior Art Search Tools but Has Not Developed a Strategy to Assess New Sources of Art
paragraphs: USPTO is making improvements to its prior art search tools that may help  address some of the challenges examiners face in identifying relevant  prior art, but USPTO has not developed a strategy to assess  incorporating new sources of art into these tools over time.
As part of its Enhanced Patent Quality Initiative, USPTO is procuring an  automated prior art search capability that could enhance examiners’  ability to identify relevant prior art. In June 2015, USPTO requested  information on a search system that uses the claims and specification  within a patent application to search for patent and nonpatent literature  automatically, without human involvement. In its request for information,  USPTO described the intent of the system as providing a useful prior art  baseline for patent examiners to begin their own searches. Such a  system could improve the search tools available to examiners and help  address the challenge of managing the quantity of potentially relevant  prior art. USPTO anticipates that a pilot system will be available to a  limited number of examiners, and agency officials told us that they  anticipate awarding a contract in late summer 2016.
The new system could expand upon the capabilities of a currently  available system called the Patent Linguistic Utility Service. According to  USPTO officials, the Patent Linguistic Utility Service has been in use for  20 years and has limitations that prevent it from meeting the agency’s  search needs. For example, the Patent Linguistic Utility Service only  searches U.S. patent literature and cannot perform the advanced search  techniques contemplated for the new system. USPTO uses the current  system to perform searches for about one-tenth of the agency’s incoming  applications, whereas USPTO expects to use the new system for every  application. Based on our survey, an estimated 12 percent of examiners  find that the current system makes it somewhat or much easier to  complete a thorough prior art search in the time allotted, whereas 52  percent of examiners expect that an automated pre-examination search  would make it somewhat or much easier to complete a thorough prior art  search in the time allotted.
Moreover, USPTO’s 2014-2018 Strategic Plan includes an objective to  ensure optimal information technology services, including upgrading  search systems and prior art access. Toward this objective, USPTO is in  the process of a major, multiyear $405 million effort to upgrade its  information technology tools to provide examiners with a new system to  manage all aspects of patent examination, including certain aspects of  their prior art searches. The new system, called Patents End-to-End, will,  according to USPTO’s Strategic Information Technology Plan for fiscal  years 2015 through 2018, replace nearly 20 systems currently used to  search patent applications. The new system will initially replicate the prior  art search capabilities of USPTO’s current systems, such as EAST and  WEST, which focus on U.S. patent literature and include only one source  of nonpatent literature. Although searching for nonpatent literature is  required by the agency’s manual for patent examiners, under the current  and planned systems, examiners need to individually access and search  a variety of external sources to look for nonpatent literature during their  examinations. Consequently, neither the current nor planned systems  provide USPTO examiners with the capability to efficiently search for prior  art using a single, integrated search that includes both patent literature  and multiple sources of nonpatent literature.
The time it takes examiners to search the large and increasing volume of  nonpatent literature and the inefficiency of having to search many  different sources individually may lead examiners to conduct less  thorough searches of nonpatent literature, potentially missing relevant  prior art. As discussed earlier, our analysis of examiners’ survey  responses and experts’ statements suggests that examiners are less  likely to search for certain types of prior art, particularly foreign-language  patents and nonpatent literature, from which it is more difficult to find  relevant prior art. Integrating additional sources of prior art into USPTO’s  search tools is one way USPTO could increase the types and sources of  prior art that examiners consider, and would be similar to approaches that  EPO and JPO cited as helping examiners consider a wide array of  nonpatent literature sources. According to USPTO officials, the  capabilities of the new Patents End-to-End system can be expanded in  the future to include additional nonpatent literature sources. However,  USPTO officials told us that, as of March 2016, the agency does not  currently have specific plans to add additional nonpatent literature  sources to its new system because of its initial focus on developing parity  with the existing system. In addition, as of March 2016, USPTO had not  established a documented strategy to identify and assess new sources in  the future or the most optimal means of providing access to them.
According to federal standards for internal control, control activities are  the policies, procedures, techniques, and mechanisms that help ensure  that actions are taken to address risks. These activities are an integral  part of an agency’s planning to achieve effective results and efficiently  manage government resources, including the development and  maintenance of information systems. Because information technology  changes rapidly, the internal control standards note that controls must  evolve to remain effective. These standards also highlight the importance  of clearly documenting internal controls. Without a documented,  periodically updated strategy to evaluate new sources of prior art to  include in the Patents End-to-End system and a process to periodically  assess this strategy, USPTO will not have the assurance that it is taking  full advantage of its information technology investment to help examiners  more efficiently access a variety of resources for their prior art searches.
section_title: USPTO’s Monitoring of Prior Art Searches Cannot Be Used to Identify Technology-Specific Trends or Measure Changes in Search Quality over Time
paragraphs: USPTO is taking steps to strengthen monitoring of examiners’ work;  however, these efforts may not provide USPTO with adequate data to  identify and address shortcomings with examiners’ prior art searches  specific to individual technology centers and monitor search  thoroughness over time. USPTO uses two methods to review  examinations, which may help the agency monitor the effects of the  challenges described above as well as the thoroughness of searches.  First, USPTO’s Office of Patent Quality Assurance (OPQA) conducts  audits of a random sample of office actions, and some of these audits will  review examiners’ prior art searches. Prior to recent changes in their  audits, reviewers in OPQA performed about 400 audits per year that  focused on assessments of examiners’ prior art searches. Second,  supervisory patent examiners review examiners’ work products as part of  each examiner’s annual performance appraisal. USPTO supervisory  patent examiners are required to review at least four office actions of  each of their primary examiners per year—with additional reviews for  junior examiners—and to evaluate the thoroughness of examiners’ prior  art searches during these reviews.
However, in recent years, the number and consistency of OPQA staff and  supervisory patent examiners’ reviews of examiners’ prior art searches  have not been sufficient to examine trends in the thoroughness of prior art  searches at the technology center or art unit level. Using the 400 audits  OPQA performed annually, USPTO officials said that they could perform  statistically valid assessments of prior art searches only for the examiner  corps as a whole. Further, because the supervisory patent examiners’  reviews have not been conducted or documented in a consistent manner,  USPTO could not examine trends at the technology center or art unit level  by combining supervisory reviews with OPQA’s reviews. Specifically, past  OPQA audits have evaluated prior art searches by, for example,  considering if the examiner used reasonable search terms and synonyms,  and may have included an independent search to discover prior art  missed by the examiner. Auditors in OPQA documented the results of  their reviews, and the office used them to evaluate the examiner corps  from year to year. In comparison, according to a USPTO training  document, supervisory patent examiners should check that a thorough  search was conducted, but should also adjust their reviews based on an  examiner’s skills, abilities, and performance history. Supervisory  examiners do not need to perform an independent search for prior art.
Supervisory examiners may document errors found during their reviews  to inform individual employee performance assessments but are  otherwise not required to record their reviews. Because USPTO does  not require supervisory examiners to document these reviews in a  consistent form, the agency cannot analyze the data to examine issues  with prior art search quality in specific technology centers and art units.
Early in fiscal year 2016, USPTO took two steps that if finalized could  enable monitoring of prior art search trends at the technology center or art  unit level, according to agency officials. Beginning in November 2015,  OPQA made changes to its review processes that along with an increase  in staff, should allow the office to perform about 12,000 audits in 2016,  according to OPQA officials. In addition, in 2015, USPTO drafted a  master review form that could standardize OPQA and supervisory  examiner reviews with a single, consistent approach and documentation.  OPQA began using a draft version of the form in November 2015, and  USPTO began to pilot the form with some supervisory patent examiners  in 2016. As of March 2016, USPTO had not made a decision about the  final content of the review form, when supervisory patent examiners might  begin to use the new form, or how the data from OPQA and supervisory  examiner reviews would be used to assess examiners’ prior art  searches.
Furthermore, despite potential improvements in how OPQA and  supervisory patent examiners conduct and document reviews of  examiners’ prior art searches, USPTO’s ability to use these data to  monitor prior art searches may be limited because USPTO (1) does not  have a clear definition of what constitutes a thorough prior art search, (2)  may not collect sufficient information to assess examiners’ search  strategies or the sources of prior art they consider, and (3) has not  established goals and indicators for improving prior art searches.
USPTO requires examiners to perform a thorough prior art search and  record their search results or search history, but the agency has not  clearly defined what constitutes a thorough prior art search. USPTO’s  manual for patent examiners requires examiners to conduct a  thorough prior art search by identifying the field of search, the  appropriate tools, and a search strategy. It also requires examiners to  consider U.S. patents, foreign patents, and nonpatent literature,  unless they can justify with reasonable certainty that no more  pertinent references can be found. USPTO has no single method for  assessing the thoroughness of prior art searches because, as  supervisory patent examiners told us and our survey results show,  examiners’ search strategies and the sources they use will differ  based on the technology (see app. III). Examiners have access to  guidance and training on classification-based searches, but USPTO  has not documented technology-specific guidance—such as by  technology center or art unit—on what constitutes a thorough prior art  search. Specifically, in its definitions of CPC classes, USPTO  provides suggested search areas of related classes, but these  suggestions do not specify what USPTO would consider a thorough  prior art search for each class or describe the sources of prior art the  examiner should consider.
As of March 2016, USPTO’s draft of the new master review form did  not require OPQA or supervisory examiners to evaluate the  thoroughness of an examiner’s search. Instead, the form asks if the  examiner (1) searched for prior art associated with the inventor’s  name, (2) searched for prior art using classification results for the  application, and (3) recorded his or her search strategy. If a reviewer  finds that the examiner should have made a rejection but did not, the  form asks the reviewer to identify the source of prior art needed for  the missed rejection. The March 2016 draft of the form does not  include, as an October 2015 draft we reviewed did, questions  assessing if search queries were likely to result in identification of  relevant prior art. The March 2016 draft of the form also does not  include questions that address whether the examiner searched  foreign patent literature and U.S. and foreign nonpatent literature, as  is required by the agency’s manual for patent examiners.
While USPTO has taken steps to improve the quality of prior art  searches, the agency has not yet established performance goals and  indicators for improving prior art searches. One of the stated  objectives in USPTO’s 2014-2018 Strategic Plan is to enhance  accurate and consistent results in examination quality, and USPTO’s  Enhanced Patent Quality Initiative also aims to achieve excellence in  measuring patent quality. However, USPTO’s goals for patent quality  in the strategic plan and for the quality initiative do not currently  include goals or indicators assessing the thoroughness of prior art  searches. Although USPTO is not required to establish goals or  indicators for improving prior art searches, we have previously  reported that establishing program goals and associated indicators  constitutes a leading practice for planning within federal agencies.  Among other things, the Government Performance and Results Act  Modernization Act of 2010 requires that agencies (1) establish  objective, quantifiable, and measurable performance goals and (2)  establish performance indicators to measure progress toward each  performance goal.
The limitations to USPTO’s ability to monitor examiners’ prior art  searches at a technology center or art unit level may hinder the agency’s  ability to identify and address issues in the thoroughness of examiners’  prior art searches and quality of the agency’s examinations. Federal  standards for internal control state that agencies’ monitoring of their  internal controls should assess the quality of performance over time, and  that internal controls should generally be designed to assure that ongoing  monitoring occurs in the course of normal operations. Without  consistently collecting the information needed to assess the thoroughness  of prior art searches and monitoring at a technology center or art unit  level, the agency cannot identify and address issues that are more  prevalent in certain technology centers or art areas, such as variations in  the extent to which examiners in certain areas search for foreign patents  or nonpatent literature. Similarly, without greater clarity on what  constitutes a thorough prior art search in different technologies, it will be  difficult for USPTO to assess the adequacy of examiners’ searches, and  examiners may vary in their thoroughness. Moreover, without goals and  indicators for assessing the thoroughness of prior art searches, USPTO  cannot reliably assess the thoroughness of its searches or improvement  in searches over time.
section_title: USPTO Intends to Evaluate the Time Allotted for Patent Examinations but Has Not Planned to Assess the Time Needed for Thorough Prior Art Searches for Different Technologies
paragraphs: USPTO plans to evaluate changes to the agency’s system for  determining the number of applications a patent examiner is expected to  review within a specified period of time; however, the agency has not  identified plans to assess the amount of time examiners in different  technologies need to perform thorough prior art searches. According to  USPTO officials we interviewed and a document we reviewed, the time  allotted for each individual technology was determined when the  examiner performance and production system was first created in the  1970s or when subsequent art units were added. USPTO adjusted the  time allotted to examiners between fiscal years 2010 and 2012 and gave  all patent examiners a total of 2.5 additional hours per application.  However, according to USPTO officials, the agency did not evaluate art  unit or technology-specific factors prior to making this change. USPTO  also adjusted the time allotted in April 2016, when approximately 1,000  examiners received an additional 2.7 hours for examinations of certain  technologies to address concerns related to the transition to the CPC  system. According to technical comments USPTO provided on a draft of  this report, these changes were based on an initial investigation of time  needed to perform a thorough prior art search in these technologies.
The time pressures created by USPTO’s system for determining allotted  examination time have important implications for examiners’ ability to  conduct thorough prior art searches. As noted above, about two-thirds of  examiners responding to our survey reported having insufficient time to  complete a thorough prior art search given their typical workload.  Furthermore, examiners in different technologies may need different  amounts of time to conduct thorough prior art searches. For example, we  estimate based on our survey that a higher percentage of examiners in  the Mechanical Engineering technology center find that they have much  less time than needed, compared to examiners in other technology  centers (see app. III).
Supervisory patent examiners we interviewed told us that prior art  searching is the most time-consuming aspect of patent examination.  Based on our survey, an estimated 80 percent of examiners spent, on  average, from 6 to 20 hours initially examining patent applications and  completing a first office action. For the prior art search portion of these  efforts, we estimate that examiners spent an average of 8 hours per  application during the past quarter. If examiners feel pressure to  complete their examinations quickly rather than thoroughly, it could have  important implications for their prior art search efforts. In fact, as shown in  figure 7, less than half of the examiners—an estimated 46 percent—were  moderately or very confident that they found the most relevant prior art  during the time allotted. However, when they included their additional  voluntary overtime, the number of examiners who were moderately or  very confident increased to an estimated 69 percent of examiners. Even  with the inclusion of their overtime, though, an estimated 23 percent of  examiners remained not at all confident to somewhat confident.
Federal standards for internal control specify that agencies should assess  the risks the agency faces, including identifying relevant risks associated  with achieving the agency’s objectives, assessing a risk’s significance  and likelihood of its occurrence, and deciding what actions should be  taken to manage the risk. In addition, these standards note that an  agency’s operational success requires providing personnel with the right  incentives for the job. In USPTO’s 2014-2018 Strategic Plan, the agency  indicated its intent to evaluate changes to the its system for evaluating if  examiners are completing office actions in the time allotted and to make  additional modifications as needed. In November 2015, USPTO’s  Commissioner for Patents affirmed the agency’s intent to examine this  system and told us that he had committed to the examiners’ union to do  so. USPTO has historically faced a backlog of applications, and changing  the time allotted may make it more difficult for the agency to address this  backlog. However, changing the time allotted to examiners may address  the challenge of time pressures on examiners that we discuss above. As  of May 2016, USPTO had not clarified the extent to which this evaluation  will specifically consider the time needed for a thorough prior art search  for different technologies. Without evaluating the technology-specific time  pressures examiners face in their efforts to identify prior art in patent  examination, USPTO does not have assurance that the time allotted to  examiners reflects the evolving complexity of different technologies and  their associated prior art.
section_title: USPTO Has Taken Actions to Develop and Maintain Examiners’ Technical Competence but Has Not Assessed Competence Gaps in Each Technology Center
paragraphs: USPTO has taken several actions related to the challenge of ensuring  that the agency has an examiner workforce with the technical  competence—backgrounds, knowledge, or skills—needed for identifying  relevant prior art; however, USPTO does not have a process to assess  and measure each technology center’s progress toward closing any gaps  in examiners’ technical knowledge and skills. Effective management of an  organization’s workforce—its human capital—is essential; in particular,  identifying critical occupations, skills, and competencies and analyzing  workforce gaps are leading principles in workforce planning, as we and  the Office of Personnel Management have previously identified. Further,  as described in the federal standards for internal control, all personnel  need to possess and maintain a level of competence that allows them to  accomplish their assigned duties. Management needs to ensure that  skill needs are continually assessed and that the organization is able to  obtain a workforce with the skills required to achieve organizational goals.  Management also needs to provide employees the right training to  develop and retain skill levels to meet changing organizational needs.
Accordingly, USPTO developed a human capital strategy, its 2015-2018  People Plan, which replaced its 2011-2015 Strategic Human Capital Plan.  The agency also identified the role of patent examiner as a mission- critical occupation and identified the competencies needed for this  occupation. These competencies include technical work  experience/education and technical competence, which we refer to  collectively as technical competence. USPTO defines technical  competence as the ability to analyze and interpret written technical  materials, rules, regulations, instructions, and reports. According to  USPTO officials, specific technical competencies vary depending on the  technology examined by each technology center and art unit. USPTO has  further documented specific technical competencies for examiners in a  series of “job analysis worksheets,” which describe the knowledge, skills,  and abilities needed for examiner positions. Agency officials said these  were based on the hiring needs derived from a hiring model created by  senior management. For example, the job analysis worksheet for  electrical engineering patent examiners considers their education,  training, and experience in researching, analyzing, and applying scientific  principles in specified technical areas, such as image analysis, power  systems, and computer architecture.
To develop and maintain examiners’ technical competence, USPTO  offers three programs, all of which are voluntary for examiners (see table  5). In addition, USPTO officials said that technology centers provide  technology-specific training through paid vendors, periodic internal  meetings to discuss technical and quality issues, and technology fairs.  USPTO does not require examiners to complete a minimum amount of  ongoing technical training; however, based on information from USPTO  officials, 74 percent of examiners participated in at least one of the 153  technical training events held through the agency’s Patent Examiner  Technical Training Program in fiscal year 2015, with an average of 6.7  training hours per participant. The extent to which examiners participated  in this technical training program varied by technology center, from a low  of 48 percent to a high of 96 percent in fiscal year 2015, as well as by art  unit, according to information from USPTO officials.
While these are important steps that can help develop and maintain  examiners’ technical competence, USPTO officials told us that the  agency has not conducted an analysis to identify any gaps in examiners’  competence, either for the agency as a whole or for each technology  center. USPTO strategic or human capital plans since 2007 have called  for measuring the agency’s performance on closing competency/skill  gaps for mission-critical occupations. USPTO officials told us that the  technology centers use performance appraisal plans to individually  assess examiners’ skills and competency gaps. However, doing so does  not address the question of whether broader competency gaps exist at  the technology center level or how any gaps can be addressed.
Further, using USPTO’s job analysis worksheets in hiring does not  eliminate the need for ongoing workforce assessment. In particular, as  technologies change, the knowledge and skills required of examiners may  evolve accordingly, and examiners may move from one technology center  or art unit to another during their careers. According to USPTO officials,  there are also times when patent examiners are ultimately assigned to a  different technology center upon completion of their initial training than  the assignment given to them when they were hired. In addition, USPTO  has reported that examiner attrition has increased over the past 5 years.  Historically, attrition has been highest amongst examiners at USPTO for  less than 5 years. As we found in 2007, attrition of less experienced  examiners is a significant loss for the agency, in part because examiners  require 4 to 6 years of on-the-job experience before they become fully  proficient in conducting patent application reviews. When these staff  leave USPTO, the agency loses as much as 5 years of training  investment in them, and continuing turnover of many new patent  examiners makes the overall workforce less experienced.
Moreover, USPTO’s 2015-2018 People Plan notes that that the agency’s  ability to retain experienced and high-performing employees may prove  difficult with continued improvements to the economy. Because of these  factors, the technical knowledge and skills needed in a technology center  may differ from the knowledge and skills of the individual examiners  assigned to that center. Without periodically evaluating gaps between the  technical competence of examiners and the knowledge and skills needed  in each technology center, USPTO cannot ensure that it has appropriate  strategies, such as training or other efforts, to close any gaps. Moreover,  as specified in the federal standards for internal control, ongoing internal  control monitoring should assess the quality of performance over time.  The agency’s 2015-2018 People Plan, however, does not include  measures for monitoring progress in closing gaps. Without monitoring and  evaluating progress toward closing any identified gaps in technical  competence, USPTO may not have reasonable assurance that examiners  in all technology centers have the skills and knowledge to identify relevant  prior art during patent examination.
section_title: Conclusions
paragraphs: Examiners face a number of challenges in their efforts to search for prior  art, including the large volume of prior art from multiple sources to  consider, unclear patent applications, difficulties identifying or accessing  relevant nonpatent literature and prior art in foreign languages, and limits  on the time available to search for relevant prior art, among others. Some  of these challenges may reinforce one another or affect examiners in  some technology centers more than others. USPTO, recognizing the  importance of issuing quality patents, is taking a number of steps to  improve patent quality, as we discuss in our patent quality report  (GAO-16-490). USPTO is also is taking a number of steps to help  address challenges to completing a thorough prior art search.
However, while USPTO’s steps to address prior art search challenges are  promising, opportunities exist for USPTO to address limitations that may  hinder the effectiveness of some of its efforts. For example, USPTO has  experienced some challenges working with EPO to adopt and  consistently implement a new patent classification system. Because  USPTO and EPO have not identified a target level of consistency or a  plan to monitor consistency between USPTO contractor and EPO  classifications—important aspects of interagency collaboration—the  potential benefits of adopting the new classification system may be  reduced as inconsistently classified patents make it more difficult for  examiners to identify relevant U.S. and foreign patent literature.
Similarly, USPTO is undertaking a major investment in information  technology tools for examiners to manage their work and search for prior  art. The new search system USPTO plans has the potential to integrate  additional sources of nonpatent literature, an approach that EPO and JPO  have taken and cited as helping examiners consider a wide array of  nonpatent literature sources. However, USPTO has not developed and  documented a strategy to identify and assess the optimal means of  incorporating new sources of art into these tools. Such a strategy would  be consistent with federal standards for internal control. Developing and  periodically updating such a strategy could help USPTO take full  advantage of its investment in its new information technology tools to  address some of the challenges examiners face in identifying relevant  prior art.
USPTO is also taking steps to augment the number and consistency with  which reviews of examiners’ prior art searches are conducted and  documented that could help to address a weakness of its past efforts—an  insufficient number of consistently performed reviews to enable USPTO  to identify issues with examiners’ prior art searches within individual  technology centers. However, USPTO still faces limitations in its ability to  use the data to monitor the thoroughness of examiners’ prior art  searches. First, USPTO has not clearly defined what constitutes a  thorough prior art search for different technology centers, which is  important given differences in the appropriate search strategies and prior  art resources across technology areas. Second, USPTO’s draft review  form, as of March 2016, does not include questions that are important to  assess the thoroughness of examiners’ prior art searches, such as the  extent to which an examiner searched for foreign patent and nonpatent  literature. These are aspects of prior art search that are generally  required by USPTO’s manual for patent examiners but that our survey  results showed many examiners may not perform. Third, the agency has  not established goals or indicators covering prior art search performance.  Monitoring the thoroughness of examiners’ prior art searches at a level of  rigor and consistency to assess trends within individual technology  centers would better enable USPTO to identify issues that vary by  technology center and develop appropriate responses as called for by  federal internal control standards. Such monitoring would also allow  USPTO to have greater assurance that examiners are searching all  relevant sources of prior art, including foreign patents and nonpatent  literature. Furthermore, establishing goals and indicators for prior art  search would help USPTO reliably assess the thoroughness of its  searches and improvements over time.
Additionally, while monitoring the thoroughness of examiners’ prior art  searches is important, it is essential that examiners have sufficient time to  perform high-quality work. USPTO expects examiners to examine an  application within a specific amount of time. This means that examiners  may need to make trade-offs among the amount and types of prior art  sources they consider because of the limited time available to perform  their searches. However, as we also note in our patent quality report  (GAO-16-490), USPTO has not recently assessed the time needed for  examinations in different technologies. Without this information, the  agency does not know if it is allotting examiners appropriate amounts of  time to complete their work. Based on our survey, most examiners find  that they do not have sufficient time for thorough prior art searches, and  many were not confident in their ability to identify the most relevant prior  art without working voluntary overtime. USPTO’s planned effort to  evaluate the agency’s system for determining the time allotted for  examination could help the agency better understand the time pressures  examiners face; however, USPTO has not yet clarified the extent to which  the evaluation will look at any differences among art units and  technologies in the time needed for thorough prior art searches. Given the  importance of thorough prior art searches to USPTO’s patent quality  objectives, specifically assessing the time needed for a thorough prior  search for different technologies would provide USPTO with greater  assurance that it will address the risks posed by the time pressures  examiners experience, consistent with federal internal control standards.
Finally, examiners’ technical competence is an important factor in  understanding patent applications and locating relevant prior art in the  time allotted. USPTO has taken steps to identify the technical  competencies of various examiner positions and to offer voluntary  technical training opportunities to examiners. However, the agency has  not conducted an overall analysis of the technical competence of its  examiners to identify potential competency gaps for each technology  center. Periodically conducting such analyses to reflect evolving  technologies and workforce changes, as called for by federal standards  for internal control, would give USPTO greater assurance that examiners  in all technology centers have the technical skills and knowledge to  identify the most relevant prior art during patent examination.  Furthermore, analyzing the results of such analyses would help the  agency define and prioritize its strategies for closing any competency  gaps, such as through training or other efforts. Lastly, these steps, in  conjunction with developing measures to monitor progress toward closing  any gaps, would help USPTO address the effects of evolving  technologies and workforce changes over time.
section_title: Recommendations for Executive Action
paragraphs: To enhance USPTO’s ability to identify relevant prior art, we recommend  that the Secretary of Commerce direct the Director of USPTO to take the  following seven actions as the agency continues to implement its efforts  to improve prior art searches.
To ensure that USPTO’s collaborative efforts on classification help  examiners find relevant prior art, USPTO should work with EPO to identify  a target level of consistency of Cooperative Patent Classification  decisions between USPTO and EPO and develop a plan to monitor  consistency to achieve the target.
To ensure that USPTO is able to take full advantage of its investment in  new information technology tools and capabilities, USPTO should  develop and periodically update a documented strategy to identify key  sources of nonpatent literature for individual technology centers and to  assess the optimal means of providing access to these sources, such as  including them in USPTO’s search system.
To improve its monitoring of prior art searches and provide USPTO the  ability to examine and address trends in prior art search quality at the  technology center level, USPTO should take the following three actions:
Develop written guidance on what constitutes a thorough prior art  search within each technology field (i.e., mechanical, chemical,  electrical), technology center, art area, or art unit, as appropriate, and  establish goals and indicators for improving prior art searches.
Ensure that sufficient information is collected in reviews of prior art  searches to assess the quality of searches at the technology center  level, including how often examiners search for U.S. patents, foreign  patents, and nonpatent literature.
Use the audits and supervisory reviews to monitor the thoroughness  of examiners’ prior art searches and improvements over time.
To ensure that examiners have sufficient time to conduct a thorough prior  art search, USPTO should, in conjunction with implementing the  recommendation from our patent quality report to analyze the time  examiners need to perform a thorough examination, specifically assess  the time examiners need to conduct a thorough prior art search for  different technologies.
To ensure that examiners have the technical competence needed to  complete thorough prior art searches, USPTO should assess whether the  technical competencies of examiners in each technology center match  those necessary; develop strategies to address any gaps identified, such  as a technical training strategy; and establish measures to monitor  progress toward closing any gaps.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Commerce for its  review and comment. In the Department of Commerce’s written  comments, reproduced in appendix IV, the Department concurred with  our recommendations. USPTO provided additional technical comments  that we incorporated, as appropriate. We also provided a draft of this  report to EPO and JPO for their views and comment. EPO indicated it  had no comments on the draft, and JPO provided technical comments  that we incorporated, as appropriate.
In its written comments, the Department concurred with our  recommendation to work with EPO to identify a target level of consistency  for CPC decisions and develop a plan to monitor consistency. The  Department noted ongoing efforts to work with EPO and other patent  offices to develop objective metrics to measure the level of consistency. If  finalized, such metrics and a plan to monitor consistency could allow  USPTO to more fully benefit from adopting the new CPC system.
In regard to our recommendation to develop and periodically update a  strategy related to key sources of nonpatent literature, the Department  concurred and stated that USPTO’s STIC analyzes nonpatent literature  sources used by examiners and assesses them for incorporation into  USPTO’s search system. While we did not receive evidence of such  assessments during our review, they may be a useful step toward a  strategy to periodically assess the optimal means of providing examiners  access to key nonpatent literature sources and to ensure the  effectiveness of USPTO’s planned Patents End-to-End search system.
In regard to our recommendation to develop written guidance on what  constitutes a thorough prior art search and establish goals and indicators  for improving prior art searches, the Department concurred and said  USPTO would develop technology-based search training guidance and  establish enhanced goals and indicators for improving prior art searches.  Specifically in regard to developing guidance on what constitutes  thorough prior art searches, USPTO’s technical comments noted that our  report did not recognize that USPTO’s definitions of CPC classification  areas include suggestions for related classes for search. We adjusted our  report to acknowledge the information USPTO provides on these  suggested search areas. However, these suggested search areas do not  fully establish what constitutes a thorough prior art search for each CPC  class, and they do not provide guidance on examiners’ searches for  nonpatent literature.
In regard to our recommendations to ensure that reviews of examiners’  prior art searches collect sufficient information to assess search quality at  the technology center level and use these reviews to monitor examiners’  prior art searches over time, the Department concurred. The Department  said it would ensure adequate search data are collected to assess the  quality of searches at the technology center level, and would investigate  using audits and reviews to monitor the thoroughness of examiners’ prior  art searches over time. In its written comments, the Department noted  that a great deal of information on examiners’ prior art searches is  potentially available from examiners’ records and reviews by primary  examiners, supervisors, OPQA, and others. As we describe in our report,  these reviews have not been carried out or recorded in a consistent  manner in the past. Therefore, as a whole, these data do not provide  USPTO with reliable information to assess trends in prior art search  quality at the technology center level. USPTO’s recent effort to improve  the consistency of these reviews is an important step to address this  issue. However, additional steps are needed to ensure that USPTO’s  reviews of examiners’ prior art searches collect information on whether,  for example, examiners are searching patent literature, foreign patents,  and nonpatent literature, as is required by the manual for patent  examiners. Without collecting such information at the technology center  level, USPTO may be challenged to use its reviews to monitor the  thoroughness of examiners’ prior art searches and improvements over  time.
In regard to our recommendation on assessing the time examiners need  to conduct a thorough prior art search for different technologies, the  Department concurred and stated that USPTO intends to further  investigate the time needed. The Department also mentioned the April  2016 changes to the time allotted for approximately 1,000 examiners,  noting that they were based on an initial evaluation of the time examiners  need to perform a thorough prior art search. This initial evaluation  stemmed from feedback from examiners during the transition to CPC. We  did not receive details of this evaluation during our review, but recognize  that this action may have provided the agency with important information  about the time needed for prior art search in some technologies. We  continue to believe that the agency should review the time allotted for all  technologies, and not only those identified through feedback related to  CPC.
Finally, the Department concurred with our recommendation to assess  whether the technical competence of examiners in each technology  center matches those necessary and develop strategies to address gaps  if identified.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Commerce, the Director of  the USPTO, the Commissioner for Patents, and other interested parties.  In addition, the report will be available at no charge on the GAO website  at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3841 or neumannj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report (1) describes the challenges examiners face in identifying  relevant prior art, (2) describes how selected foreign patent offices have  addressed challenges in identifying relevant prior art, and (3) assesses  the extent to which the U.S. Patent and Trademark Office (USPTO) has  taken steps to address any challenges in identifying relevant prior art. Our  work for this report was coordinated with our report on patent quality.
To describe the challenges examiners face in identifying relevant prior art,  we reviewed relevant laws and USPTO documents and interviewed  USPTO officials; a group of four supervisory patent examiners; officials of  the examiners’ union, the Patent Office Professional Association; and a  group of five examiners serving as union representatives. We also  conducted semistructured interviews with subject matter experts (experts)  active in the intellectual property field and knowledgeable about the  subject of prior art. We selected and interviewed 18 experts to obtain  views of different stakeholder communities, including  four academic experts on the basis of a literature search we  conducted for articles related to USPTO and prior art;  four patent holders from the following technology fields: chemical  technologies, electrical technologies, and mechanical technologies,  based on data from USPTO that identified the companies receiving  patents in these fields from 2010 through 2014;  five attorneys based on their leadership (e.g., as chairs or co-chairs)  of patent-related committees or sections of various bar associations;  representatives of three nongovernmental organizations we identified  during background research that had activities or publications related  to patents; and  two patent data experts who were listed in a 2015 USPTO analysis  evaluating search technologies.
In addition, we conducted a web-based survey of a stratified random  sample of 3,336 eligible USPTO patent examiners from across 8 of the 11  technology-based subject matter groups (referred to as technology  centers) into which USPTO examiners are divided. Fielded between  August and November 2015, the survey was designed to collect  information on challenges USPTO faces in finding relevant prior art and  how USPTO might improve its prior art search capabilities. To identify  our survey population, we obtained from USPTO a list of patent  examiners as of May 2015. We excluded examiners from the following 3  technology centers:
We excluded the Designs technology center because these  examiners work on design patents instead of utility patents; design  patents are outside the scope of this engagement and have different  statutory and administrative requirements than utility patents.
We excluded examiners who perform “reexamination” work and not  initial patent examinations.
We excluded examiners in the patent training academy because  these examiners are recent hires who are in a 12-month training  program.
We also excluded examiners employed at USPTO for less than 1 year.  We then defined nine strata by technology center, with one technology  center separated into two strata, as described in table 6. Specifically, the  Transportation, Construction, Electronic Commerce, Agriculture, National  Security and License and Review technology center includes a diverse  set of technologies, including transportation, construction, agriculture, and  business methods. In our review, we separated the art units—subunits of  a technology center—focused on electronic commerce and business  methods (collectively referred to as business methods) in light of recent  legislation and court decisions related to business methods. This resulted  in nine strata with a target survey population totaling 7,825 eligible  examiners. From this list, we drew our stratified random sample of 3,336  eligible USPTO patent examiners. We received responses from 2,669  eligible examiners for an 80 percent response rate.
Because we used a probability procedure based on random selections,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  quantified the sampling error and express our confidence in the precision  of our particular sample’s results at a 95 percent confidence interval. This  is the interval that would contain the actual population value for 95  percent of the samples we could have drawn. Within each stratum and  overall, the confidence intervals for survey results for percentages are  generally within +/- 5 percentage points. The only estimates for which the  confidence intervals exceed 5 percentage points are certain results for  the business methods stratum. In these instances, the confidence  intervals are from 5 to 6 percentage points. In this report, our figures  containing survey results show the upper and lower bounds for estimates  at the 95 percent confidence interval. For other estimates in the report,  we have not provided the upper and lower bounds in the text or tables;  however, those details for all survey results are available in the e- supplement related to this report.
The quality of survey data can also be affected by nonsampling error,  which includes, for example, variations in how respondents interpret  questions, respondents’ willingness to offer accurate responses,  nonresponse errors, and data collection and processing errors. To  minimize nonsampling error, we took several steps in developing the  survey and in collecting and analyzing survey data. Specifically, in  developing the survey, we worked with our survey professionals, among  other things, to draft questions that were clear and unbiased. We  pretested the survey in person with five USPTO staff: three examiners  who are also representatives to the examiners’ union, a supervisory  patent examiner, and a quality assurance specialist. We used these  pretests to check that the questions were clear and unambiguous, used  correct terminology, requested information that could be feasibly  obtained, and were comprehensive and unbiased. We also obtained  comments on the survey from USPTO management and leadership from  the examiners’ union. In addition, we obtained a quality review by a  separate GAO survey methodologist. Based on these activities, we made  changes to the survey before administering it. Further, using a web-based  survey provided several advantages, including allowing examiners to  enter their responses into an electronic instrument that created an  automatic record for each respondent. This eliminated the potential for  errors that could have resulted if we had used a manual process to enter  respondents’ data from paper surveys. In addition, to account for the  complex sample design, we used survey software in our analyses to  produce appropriate estimates and confidence intervals, and the  programs we used to process and analyze the survey data were  independently verified to ensure the accuracy of this work.
To minimize nonresponse error, we made a variety of contacts with the  sample of examiners during the survey, including follow-up e-mails to  encourage responses. In addition, from October 20 through 23, 2015, we  attempted to follow up via telephone calls to all 1,102 examiners who had  neither completed the survey nor told us that they were no longer  examiners. We also analyzed nonresponse bias to (1) assess whether  any factors were associated with examiners’ propensity to respond and  (2) allow our analysis of respondents to properly reflect the sampling  universe of eligible examiners. To adjust the sampling weight for potential  nonresponse bias, we used standard weighting class adjustments based  on the sampling strata and the examiners’ years of experience at USPTO.  The weighted response rate was also 80 percent. In this report and in the  related e-supplement, we present the survey results using the  nonresponse adjusted weights, which are generalizable to the eligible  population of examiners.
We analyzed the responses to the survey for all examiners, as well as  responses by technology center and by the General Schedule (GS) level  of the examiners. We selected three categories of GS levels—less than  GS-13, GS-13, and greater than GS-13—because examiners at these  levels have different responsibilities and authorities when examining  patent applications. Specifically, examiners at the GS-14 level or above  generally may grant a patent or reject a patent application without  additional review, most examiners below the GS-14 level must have their  actions reviewed by a more senior examiner, and some GS-13 examiners  are in the process of becoming GS-14 examiners.
Question 3 in the survey asked examiners approximately how many  hours they spent per application on prior art searches in the past quarter,  when preparing the original First Action on the Merits, the means by  which examiners initially notify applicants about the patentability of their  inventions. Examiners provided a variety of open-ended responses.
Some respondents chose to provide a range, such as “5 to 10,” while  others provided a single number for hours spent per application on prior  art searches. Some examiners also provided responses that did not  clearly indicate the approximate hours per application spent on prior art  searches in the past quarter; we excluded those responses. Where  possible, we coded responses to reflect a range of numbers by assigning  a low and a high value; when a single number was provided, we coded  that number as both the low and the high value. A second analyst verified  the initial analyst’s coding. We checked the sensitivity of results overall,  within strata, and within GS level categories; results were not statistically  different when using the low value, midpoint (average between low and  high values), or high value of a range. As a result, we present results for  question 3 based on the midpoint. The 95 percent confidence intervals for  these results are within +/- 10 percent of the estimates themselves,  except for the estimate for technology center 2100 (Computer  Architecture, Software, and Information Security). The estimate for  technology center 2100 has a 95 percent confidence interval of within +/-  18 percent of the estimate itself.
For some other survey questions, we also reviewed examiners’ open- ended responses on selected topics. We selected those topics based on  our interviews with experts and USPTO officials as well as our analysis of  closed-ended survey responses. We selected the questions for which  examiners’ responses most frequently included keywords we identified for  each topic. An analyst conducted a keyword search of all responses to  the selected open-ended questions and coded responses containing the  keywords. A second analyst verified the initial analyst’s coding. Our report  provides some examples or summaries of examiners’ comments based  on this review. Examiners’ responses to open-ended questions are not  generalizable to other examiners. In addition, because we did not conduct  a systematic review of all open-ended responses to our survey, we do not  report the exact number of examiners who provided responses on the  topics we reviewed.
In addition, we conducted statistical tests of association on the results of  certain survey questions; all tests were independently verified to ensure  their accuracy. All tests of association were carried out at the 5 percent  level of significance and were Cochran-Mantel-Haenszel (CMH) Chi- square tests of general association. The testing was carried out in  SUDAAN, which is statistical software appropriate for the analysis of  survey data. The null hypothesis was that there is no association between  the two tested variables. When the association between two variables,  conditional on a third variable, is of interest, this is referred to as the  stratum-adjusted CMH test. The test statistic is Wald Chi-Square.  Specifically, among other survey questions, we performed CMH tests on  examiners’ responses to questions on how often they search for foreign  patent literature or foreign-language nonpatent literature, and how difficult  it is to obtain relevant art from these searches. We also performed CMH  tests on responses to questions on how often examiners search for  certain types of prior art and whether they have sufficient time to  complete a thorough prior art search. The types of art included were  foreign patent literature, scientific articles or presentations, foreign- language nonpatent literature, and industry-related nonpatent literature.  We performed these tests on responses overall as well as conditional on  the examiners’ technology center.
To describe how selected foreign patent offices have addressed  challenges in identifying relevant prior art, we first reviewed information  identifying the world’s five largest patent offices. From the list of the other  four offices similar in size to USPTO, we selected the European Patent  Office (EPO) and the Japan Patent Office (JPO) based on comments  observers made about the quality of their work. We conducted site visits  with EPO and JPO, during which we interviewed officials from these  offices and reviewed documents they provided. We also interviewed  stakeholders knowledgeable about the offices’ practices, such as  academics who study these offices. Because these individuals were not  necessarily knowledgeable about USPTO practices, they are not included  among the 18 experts discussed above.
To assess the extent to which USPTO has taken steps to address any  challenges in identifying relevant prior art, we reviewed documents from  the agency related to prior art search procedures and capabilities,  including ongoing and planned initiatives related to information  technology resources or capabilities, the examiner workforce and related  human capital management efforts, training practices, and international  cooperation. We also interviewed or obtained written responses to our  questions from officials from USPTO’s Office of the Commissioner for  Patents, Office of Patent Examination Policy, Office of Patent Information  Management, Office of International Patent Cooperation, Office of Human  Resources, and Office of Patent Quality Assurance, among others, and  conducted interviews with technology center directors, supervisory patent  examiners, and representatives from the USPTO examiners’ union. In  addition, we reviewed the results of the survey of examiners. In assessing  USPTO’s efforts, we identified criteria in the federal standards for internal  control; the Government Performance and Results Act of 1993, as  amended; USPTO’s Manual of Patent Examining Procedure; and  USPTO’s strategic plan.
We conducted this performance audit from November 2014 to June 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Patent Examination Process at the U.S. Patent and Trademark Office (USPTO) (Corresponds to Fig. 1)
paragraphs: Appendix II: Patent Examination Process at the  U.S. Patent and Trademark Office (USPTO)  (Corresponds to Fig. 1)
This appendix provides details on steps in the patent examination  process, including rollover information, depicted in figure 1.
section_title: Appendix III: Selected Survey Results by Technology Center or Examiners’ Level
paragraphs: Based on our survey results, U.S. Patent and Trademark Office (USPTO)  patent examiners in some technology centers or at some General  Schedule (GS) levels face certain challenges in identifying relevant prior  art more than examiners in other technology centers or at other GS  levels. Tables 8 through 15 provide information related to the effects of  the quantity of prior art, how often examiners search for certain types of  prior art and how difficult they find it to obtain relevant art from certain  types, the sufficiency of examiners’ time to complete a thorough prior art  search, average time examiners spent on prior art searching and each  First Action on the Merits, examiners’ overtime, how often examiners  encountered misclassified patent applications, and continuing education  provided by USPTO in examiners’ art areas. Because we surveyed a  generalizable stratified random sample of examiners, our results provide  estimates for each technology center included in our study. See the  related e-supplement, GAO-16-478SP, for the original survey language  and more detailed results.
section_title: Appendix IV: Comments from the U.S. Department of Commerce
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Chris Murray (Assistant Director),  Tind Shepper Ryen, Krista Breen Anderson, Hilary Benedict, Richard  Burkard, John Delicath, Alice Feldesman, Armetha Liles, Rebecca Makar,  Rob Marek, Eleni Orphanides, Kelly Rubin, Monica Savoy, Ardith Spence,  and Sonya Vartivarian made key contributions to this report.
section_title: Related GAO Products
paragraphs: Intellectual Property: Assessing Factors That Affect Patent Infringement  Litigation Could Help Improve Patent Quality. GAO-13-465. Washington,  D.C.: August 22, 2013.
U.S. Patent and Trademark Office: Performance Management Processes.  GAO-10-946R. Washington, D.C.: September 24, 2010.
Intellectual Property: Enhanced Planning by U.S. Personnel Overseas  Could Strengthen Efforts. GAO-09-863. Washington, D.C.: September 30,  2009.
U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to  Reduce the Patent Application Backlog. GAO-08-527T. Washington,  D.C.: February 27, 2008.
U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to  Reduce the Patent Application Backlog. GAO-07-1102. Washington,  D.C.: September 4, 2007.
Intellectual Property: Improvements Needed to Better Manage Patent  Office Automation and Address Workforce Challenges. GAO-05-1008T.  Washington, D.C.: September 8, 2005.
Intellectual Property: Key Processes for Managing Patent Automation  Strategy Need Strengthening. GAO-05-336. Washington, D.C.: June 17,  2005.
Intellectual Property: USPTO Has Made Progress in Hiring Examiners,  but Challenges to Retention Remain. GAO-05-720. Washington, D.C.:  June 17, 2005.