SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mirac Suzguna $^{a}$  Luke Melas-Kyriazib Suproteem K. Sarkar $^{c}$  Scott Duke Kominers $^{c,d}$  Stuart M. Shieber $^{c}$
$^{a}$ Stanford University  $^{b}$ Oxford University  $^{c}$ Harvard University  $^{d}$ a16z crypto
*Correspondence to: msuzgun@cs.stanford.edu
In this paper, we introduce the Harvard USPTO Patent Dataset (HUPD), a large-scale, well-structured, and multi-purpose corpus of English-language patent applications filed to the United States Patent and Trademark Office (USPTO) between 2004 and 2018. With more than 4.5 million patent documents, HUPD is two to three times larger than comparable corpora. Unlike previously proposed patent datasets, HUPD contains the inventor-submitted versions of patent applications—not the final versions of granted patents—thereby allowing us to study patentability at the time of filing using NLP methods for the first time. It is also novel among NLP datasets in its inclusion of rich structured metadata alongside the text of patent filings, enabling researchers to perform new NLP tasks leveraging this metadata. As a case study on the types of research HUPD makes possible, we introduce a new task to the NLP community—patent acceptance prediction. Finally, we demonstrate how our dataset can be used for three additional tasks: multi-class classification of patent subject areas, language modeling, and summarization. Overall, HUPD is one of the largest multi-purpose NLP datasets containing domain-specific textual data, along with well-structured bibliographic metadata, and aims to advance research extending language and classification models to diverse and dynamic real-world data distributions.<sup>1</sup>
Patents are key public indicators of innovation and technological advancement. They offer a simple yet powerful source for studying, measuring, and appraising innovation activity, economic growth, and emerging technology. Over the past two decades, the total number of patent applications filed to the United States Patent and Trademark Office (USPTO) per year has almost doubled. In the fiscal year 2020 alone, the USPTO received more than 650,000 patent filings, including requests for continued examinations [1]. The competitive and regulatory landscape surrounding patent-driven innovation is rapidly evolving, but despite the clear focus in textual data in the field of patent analysis, it has yet to be systematically studied by the ML and NLP communities.
The absence of large-scale, well-structured, and distilled patent data is a major hurdle preventing researchers and practitioners from applying ML tools to understand and explore innovation and technological change through patent text. In recent years, there have been efforts to produce NLP
<sup>1</sup>Our dataset, along with our code and models, is publicly available at https://patentdataset.org and distributed through HuggingFace Datasets (https://huggingface.co/datasets/HUPD/hupd). All the authors were based at Harvard University at the inception of this project.
Figure 1: Three pages of the pre-grant version of an example patent document (Method and Apparatus for Initiating a Transaction on a Mobile Device [Publication No: 2014-0207675 A1]). The highlighted sections show a subset of the 34 data fields that we include in the Harvard USPTO Patent Dataset.
datasets of patent text, including CLEF-IP 2011 [2], USPTO-2M [3], and BIGPATENT [4]. These datasets have some limitations in their scopes and features, as shown in Table 1. They may contain text and information only for granted patents, may contain subsets of patent text and metadata, and typically focus on only one particular NLP task. $^{2}$
It is thus highly desirable to have a free, publicly-available dataset that provides a wider and more encompassing repository of patent data—covering multiple sections and years—of not just granted patents but all patent applications, that allows more flexibility and control in data selection, and that can be appropriated for multiple experiments and investigations. With these desiderata in mind, we introduce a public, large-scale, consistently-structured, and multi-purpose corpus of patent data to the NLP community, called the Harvard USPTO Patent Dataset (HUPD). The dataset contains more than 4.5 million English-language utility patent applications filed to the USPTO between 2004 and 2018, and aims to advance research efforts in both patent analysis and NLP.
HUPD distinguishes itself from prior NLP patent datasets in three key aspects. First, unlike some other datasets, it does not restrict the sample to only granted patents, and instead focuses on patent applications. Patent applications contain the original set of claims and descriptions of the proposed invention written by the applicant. Because the dataset has a consistent set of patent documents at the time of filing, it avoids the dataset shift concerns that would be present in a study of accepted and rejected patent applications at different revision stages. In fact, having access to the original versions of both accepted and rejected applications allows us to introduce a completely new task to the field—binary classification of patent decisions. The goal of this task is to predict the patent acceptance from the language of a patent application at the time of submission. Second, our dataset features multiple classes of rich textual and structural information present in patent applications. Whereas many previous NLP datasets include only one or two of a patent's data fields (for example, description and abstract), HUPD contains 34 fields, including filing date, fine-grained classification codes, examiner information, and many others. The variety of information available for each patent application can enable NLP researchers to perform a wide range of tasks—such as analyzing the evolution of patent language and categories over time—that were not possible under previous NLP patent datasets. Third, HUPD uses textual and bibliographic information obtained directly from the USPTO's data products, rather than from Google's Patent search as BIGPATENT does; it is larger than previous NLP patent datasets while still being clean, comprehensive, and well-structured.
We introduce our patent dataset with several audiences in mind. For the NLP community, the universe of patent applications—with its sheer breadth and wealth of textual substance and structured format—provides an ideal domain-specific laboratory for developing and evaluating new NLP tools.<sup>3</sup> From
2We include a more detailed discussion of these NLP datasets, as well as other popular raw patent document repositories and search tools not produced primarily for the NLP community, in Section C.
$^{3}$ Patent applications follow a strict and structured framework: They generally contain claims, description, drawings, background, abstract, and summary sections. It is imperative—for both patent prosecution and any potential future litigation issues—to define the scope of the claims of the proposed invention clearly and completely, and to avoid misleading or confusing statements (for more information, see 35 US Code
Dataset	#Docs	Title	Abst	Appl	Exam	Invt	PD	Claims	Bkgd	Dsc	PCs	Years	Primary Purpose
WIPO-alpha	75,250	✓	✓	✓	✓	✓	✓	✓	✓	✓	1998-2002	Classification
CLEF-IP (2011)	1,500,000	✓	✓	✓	✓	✓	✓	✓	< 2009	Retrieval+Classification
USPTO-2M	2,000,147	✓	✓	✓	✓	2006-2015	Classification
BIGPATENT	1,341,362	✓	✓	✓	1971-2018	Summarization
Ours (HUPD)	4,518,263	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	2004-2018	Multi-Purpose
Table 1: Comparison of HUPD with other datasets whose primary goal is NLP patent analysis. The abbreviated columns mean the following. Abst: Abstract, Appl: Applicant Information, Exam: Examiner Information, Invt: Inventor Information, PD: Publication Date, Bkgd: Background, Dsc: Description, and PCs: IPC/CPC codes.
abstractive summarization of patent sections to information retrieval and named-entity recognition and extraction, one can perform a variety of both standard and novel NLP tasks on patent data. Furthermore, the well-structured nature of our dataset allows for researchers to study how concepts like acceptance criteria vary across contexts and over time. For the IP community, a key motivator of our work is that there are many rote tasks associated with patent filing and examination—including the categorization of patents into relevant technology areas and prior art search—in which machine learning methods could potentially be used to provide efficiency, value, and cost savings. Finally, for the general audience, our present work illustrates how NLP/ML tools may be efficiently deployed for advancing socially relevant objectives and studying diverse and dynamic application areas.
A patent application<sup>4</sup> typically consists of a title, abstract, set of claims, detailed description, drawings (if needed to describe the invention), and cross-references to related applications, among other written specifications. Applications are filed to the USPTO and reviewed by examiners who are expected to have knowledge and expertise associated with the subject matter of the invention.
During the examination process, an examiner determines the patentability of the invention. The examiner decides whether the proposed invention is useful, non-obvious, and statutory, and searches for already-existing patents within the technology sphere of the invention to confirm the set of claims provided is novel. Afterwards, the examiner sends an Office action to the applicant, notifying them of the USPTO's decision. If the decision is favorable, then the applicant can choose to proceed with their application and have the USPTO issue their patent. However, if the decision is unfavorable, then the applicant receives a notification of rejection; it is then up to the applicant to decide whether they wish to respond to the rejection, continue to pursue their application, and request a reexamination.
It is common to think of patent applications as being simply accepted or rejected, but the status of an application is considerably more subtle. For the task of predicting an application's outcome, it is useful to provide a formalization of what it means for a patent application to be accepted and rejected. We say that a patent application is "accepted" if it has been officially approved, granted, and published by the USPTO. There is, however, no clear-cut notion of absolute rejection in patent applications. An application might, and in fact often does (at the beginning), receive an Office action indicating a non-final or final rejection—typically on the grounds of prior art, scope of the claims, or lack of utility or novelty or obviousness, but the applicant can submit a response to the USPTO and re-open the prosecution of their application, even in the case of a final rejection. As a result, a "final rejection" does not imply a patent application has no chance of eventually being accepted after revisions. In our dataset, we label an application "rejected" if it has received an Office action of rejection—final or non-final—and was ultimately abandoned by the applicant.<sup>5</sup> We categorize all the remaining applications, which are still waiting a response from the USPTO, as "pending." These
$\S 112)$ . Conversely, patent applicants sometimes make strategically ambiguous and creative specifications in their applications towards the goal of broadening claim coverage. These characteristics, along with other idiosyncrasies, make the patent domain a uniquely challenging and powerful setting to study through ML tools.
$^{4}$ Following the terminology used by Toole et al. [5], we define a "patent application" as a non-provisional application for an invention and a "patent" as a granted patent. Prior to 2001, patent applications to the USPTO were not published: They were instead kept in secrecy until their patent issue dates (p. 174; Menell et al. [6]). Under the American Inventors Protection Act of 1999, however, most patent applications are published eighteen months after the earliest filing date, unless the applicant requests earlier publication.
Following the "Manual of Patent Examining Procedure," we treat a patent application as "abandoned" if its applicant either fails to take appropriate action and to reply to the USPTO within a specific time period or submits a written declaration of abandonment. In the case of non-final rejections, for instance, applicants are usually given six months to revise their applications [7].
distinctions are particularly useful when describing and discussing the binary classification task of patent decisions in the following sections.<sup>6</sup>
3 Related Work on Patent Analysis
Existing patent datasets for NLP focus nearly exclusively on two tasks: patent subject classification and summarization. Below, we provide overview of prior datasets and studies in these areas.[7,8]
Automated Subject Classification. Patents are classified by subject matter according to standard taxonomies, most notably the International Patent Classification (IPC) and Cooperative Patent Classification (CPC) systems. $^{9}$  These IPC/CPC codes are hierarchical—classified at a class level (e.g., G-Physics), subclass level (e.g., G06F-Electric Digital Data Processing), and so on. Previous studies attempted to predict the IPC or CPC codes of patents at the class and subclass levels using various statistical methods, including classical statistical learning tools [12–16] and neural models [17, 3, 18]. Recently, Transformers [19] have been considered for this task: Lee and Hsiang [20], for instance, fine-tuned a pre-trained BERT [21] to predict IPC/CPC codes of patents. Zaheer et al. [22] conducted similar experiments using BIGBIRD [23] and showed improvements over BERT. $^{10}$
As shown in Table 1, WIPO-alpha, CLEF-IP, and USPTO-2M have been the main gymnasia for model training for the IPC/CPC classification tasks, but these corpora are still limited in their scopes. They contain a relatively smaller set of patent text and metadata, do not allow users to choose which year ranges to focus on during training and testing, and in some cases come pre-tokenized, which may prevent users from using custom vocabularies. HUPD addresses these limitations, and also allows more flexibility in data selection and provides more comprehensive text, field, and year coverage.
Patent Text Generation and Summarization. With the growing availability and success of large language models in recent years, there has been an interest in applying language models to patents. Sharma et al. [4] initiated such explorations, introducing the first summarization dataset on patents, called BIGPATENT, and trained summarization tools on their dataset to generate the abstract section of a patent given its description section. The BIGPATENT dataset contains 1.3 million utility patents filed to the USPTO between 1971 and 2018, and was collected from the Google Patents Public Dataset via BigQuery. Our dataset differentiates itself from BIGPATENT in three aspects: (1) HUPD includes metadata and fields, including claims, background, filing date, and examiner information, that are not present in BIGPATENT; (2) in addition to accepted patents, it contains rejected and pending applications—it thus enables the study of patent acceptance/rejection<sup>11</sup>; and (3) it has approximately three times as many documents as BIGPATENT (Table 1).<sup>12</sup>
Patent Acceptance Prediction. To the best of our knowledge, our study is the first work to introduce a practical definition of rejection in patent examination to identify, analyze, and discuss the patterns in and characteristics of accepted versus rejected patent applications from a purely textual perspective. In that sense, we introduce the patent decision classification task to the NLP literature.
<sup>6</sup>We refer our readers to [8] for a discussion on data controversies around calculating patent grant rates.
<sup>7</sup>In Section C, we provide a more thorough and comprehensive comparison of our dataset against BIGPATENT and other existing large-scale NLP datasets (such as S2ORC [9] and WikiBio [10]).
For an alternative compendium of studies that make use of deep learning techniques for patent analysis, please see the study by Krestel et al. [11].
Established by the Strasbourg Agreement of 1971 and administered by the WIPO, the IPC scheme is currently being used, in various capacities, in over one hundred major patent offices worldwide. The CPC scheme is an extension of the IPC scheme and has been adopted by the USPTO since 2013. It offers a more comprehensive coverage of some new technical developments and includes an additional section "Y." In the US, every issued patent needs to be classified, based on its subject matter, according to the CPC taxonomy.
$^{10}$ In addition, Acikalin et al. [24] classify the scope of a patent in terms of exposure to the U.S. Supreme Court decision Alice Corp. v. CLS Bank International, 573 U.S. 208 (2014).
Furthermore, HUPD contains the original, inventor-submitted version of patent applications, as opposed to the granted versions. The distribution of language in the granted versions of accepted patents may change after examiner comments, making granted versions less directly comparable to pending patent applications. For those patent applications that are eventually granted and published, the published version can be retrieved using the patent number metadata in our dataset.
<sup>12</sup>We include a more in-depth comparison of our dataset with BIGPATENT in Appendix C.
Section	Brief Description	Avg # Tokens
Title	Title of the Invention	16.4
Abstract	Summary of the Background and Claims	132.0
Claims	List of Items Defining the Invention	1271.5
Background	Brief Statement of the Field of Art and Related Art of the Invention	627.11
Summary	Condensed Version of the Description	917.8
Description	Detailed Statement and Disclosure of the Invention	11855.6
Table 2: Brief description of and average number of tokens in each text-based section in HUPD, as measured by the GPT-2 tokenizer. Typically, the description section in a patent application is almost 100 times longer than the abstract section. HUPD can be used for long-sequence summarization and language modeling, inter alia.
4 The Dataset
The Harvard USPTO Patent Dataset (HUPD) contains 4,518,263 utility patent applications filed to the USPTO between January 2004 and December 2018. In this section, we elaborate on the data collection process and provide details about the data format and dataset structure. We furthermore enumerate and highlight a portion of the data's statistical properties, and acknowledge the limitations and ethical considerations of the present work. Additional information is included in Section A.
Dataset Construction. As specified by US law, all patent data is publicly accessible. Distilling this information into a NLP dataset involves obtaining the patent data and its corresponding metadata, normalizing all data to the same format, filtering missing and erroneous data, de-duplicating data, and merging all data into a single easy-to-use dataset. Here we provide an overview of this process.
Patent application texts were obtained from the USPTO Bulk Data Storage System (BDSS; Patent Application Data/XML Version) as XML files.14 As not all the original patent files follow the same XML structure, we wrote regular expressions to parse all the different formats into a normalized set of data fields, and stored these data fields as structured JSON files. Filing metadata—including acceptance decisions, filing dates, titles, and classification information—were separately obtained from the USPTO Patent Examination Research Datasets [26, 27] in February 2021. This metadata was then merged with the full-text patents from the USPTO BDSS to link patents filed before 2020 with information about examiners, office actions, and additional filing information. This merging process ensures that our dataset contains both the updated metadata structure and the patent application texts.
Finally, we assembled the continuation information for each application as follows: Applications with parent filings in the USPTO continuity data files were marked with the prefix "CONT-" in the decision status field. Hence, there are six labels in total for the decision status of applications: "Accepted," "Rejected," "Pending," "CONT-Accepted," "CONT-Rejected," and "CONT-Pending."
Statistics. Table 1 provides a quantitative comparison of our dataset with other patent data sources, while Table 2 gives information and statistics about the text-based sections in our dataset. HUPD builds on its counterparts, including BIGPATENT, because of not only its size and wider coverage but also its ability to allow users to see the evolution of patent families over time.
Limitations. From a methodological point of view, the dataset is limited to patents from the U.S. and in English, and drops image content such as drawings. From a functional point of view, some textual sections are longer than current NLP models can process, and specialized vocabulary in certain fields can create issues for existing tokenizers.
<sup>13</sup>We chose to restrict our attention to utility patent applications both because utility patent applications are treated as "inventions" and because they have constituted more than  $90\%$  of total patent applications every year in the past two decades (and beyond) [25]. Our dataset does not include any design or plant patent submissions.
14Per the USPTO's Electronic Information Products Division's Terms and Conditions, bulk data products are provided with no restrictions on use.
15Continuity data in patent applications establishes a link between an initial patent application and its subsequent continuation applications. This information is accessible on the PatEx website through two distinct files. The "continuity parents" file contains details about the original parent application, and the "continuity children" file includes information on the subsequent continuation applications derived from the parent. Collectively, these two files are referred to as the USPTO continuity files in the context of this paper. We labeled all continuations that were not continuing a provisional application or national stage entries as "CONT-."
<sup>16</sup>To avoid duplicate documents in our corpus, we excluded CONT-applications from our experiments; however, users may choose to use them to study changes in patent applications for the same invention over time.
Task Name	Setup	Metrics
Acceptance Prediction	Abstract or Claims → Patent Outcome (Decision)	Accuracy
IPC/CPC Classification	Abstract or Claims → IPC/CPC	TOP-k
Language Modeling	Abstract or Claims or Description	Perplexity
Abstractive Summarization	Claims or Description → Abstract	ROUGE/BLEU
Table 3: Summary of the four NLP tasks presented in this work, along with some evaluation metrics for them. Our dataset can be used to conduct many other NLP/IP experiments. See Section 5 for detailed information.
Potential Biases. We provide a detailed examination of HUPD for potential biases in the Appendix. We recommend that researchers who use HUPD consider these biases when interpreting their results—especially if their analyses involve inventor and/or examiner information. For now, we summarize our empirical findings in this section and note they are consistent with the previous work [28-32]. We show, among other results, that female inventors are notably underrepresented in the U.S. patenting system, that small and micro entities (e.g., independent inventors, small companies, non-profit organizations) are less likely to have positive outcomes in patent obtaining than large entities (businesses with more than 500 employees), and that patent filing and acceptance rates are not uniformly distributed across the US. Our empirical findings suggest that any study focusing on the acceptance prediction task, especially if it is using the inventor information or the small-entity indicator as part of the input, should be aware of the the potential biases present in the dataset and interpret their results in light of these considerations.