SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is natural to compare HUPD to one of the most widely-used existing patent datasets in NLP, BIGPATENT [4]. In comparison to BIGPATENT, HUPD contains not only significantly more patent documents (4.5 million vs. 1.3 million) but also much richer (bibliographic) meta-information about each patent document. Since BIGPATENT was proposed primarily for the task of abstractive summarization, it includes only four of the 34 data fields available in HUPD (publication number, application number, abstract, and description). Notably, BIGPATENT does not include the claims section, which is often considered to be the most important section for describing the contributions of an invention. Many tasks that can be performed with HUPD (analysis of text over time, fine-grained classification, etc.) are not possible to perform with the BIGPATENT data. Additionally, whereas BIGPATENT contains only granted patents, HUPD contains patent applications. As a result, prediction of patent acceptance, a new task we introduce to NLP, is not possible with BIGPATENT. Finally, the BIGPATENT is pre-tokenized using NLTK [41], which might cause issues for applications that contain chemical formulae and mathematical equations. HUPD, by contrast, provides the raw patent text and may be tokenized using a custom vocabulary for tasks in which it is important to correctly represent formulae or equations.
C.2 Comparison with Large Scientific Text Corpora
One noteworthy domain-specific NLP corpus is Allen AI's Semantic Scholar Open Research Corpus (S2ORC; Lo et al. [9]), which contains 81 million English-language academic papers spanning multiple academic disciplines. Of these papers, the authors provide full structured text from 8.1M open-access PDF files and 1.5M LATEX source files, along with their appropriate metadata. They include the title, author list, publication year, publication venue (or journal), abstract, paragraphs of the body of the text, section heading, figure and table information (along with their corresponding captions), equations, headers, footers, inline citations, and prior art as data fields in their JSONlines files. Due to the nature of academic papers, however, not all the papers have content for these data fields. With its size and comprehensive structured metadata, S2ORC is comparable to HUPD in its nature. Another concurrent work by Saier and Färber [42] introduced the unarXive dataset, a collection of over one million academic papers with links to almost 2.7 million unique publications; however, this dataset is not as comprehensive and well-structured as S2ORC or our dataset. Furthermore, Dasigi et al. [43] introduced Qasper, a dataset of information-seeking question-answering (QA) dataset over 1585 NLP papers, but their focus was limited to the evaluation of document-grounded QA models. Contract Understanding Atticus Dataset (CUAD; Hendrycks et al. [44]) is another annotated NLP dataset that contains more than 500 contracts with over 13,000 expert annotations and 41 label categories. CUAD is unique in the sense that the labels were annotated by legal experts and poses the identification of the relationship between different subtexts of a contact with different label categories as its primary task. WikiBio [10] is also a large scientific corpus that is worth mentioning: It contains almost 0.73 million unique biographies of famous people extracted from English Wikipedia, where each biography contains the first paragraph of the article and the infobox (fact table). The WikiBio dataset has been historically used for table-to-text generation [45], but it has been also adopted for question-answering [46]. We remark that our dataset provides a diverse range of language-based tasks to the community, including, but not limited to, binary classification of patent decision outcomes, multi-class IPC/CPC classification, patent clustering, long-sequence language modeling, abstractive summarization, document-level information extraction, named-entity recognition and extraction. HUPD is also notable in being one of the largest publicly-available collection of well-structured and domain-specific textual data. Finally, we remark that it has the potential to facilitate research and development in not only NLP but also IP.
C.3 Comparison with Large-Scale Web-Scraped NLP Datasets
Given the size of HUPD (350GB of raw text), it is also possible to compare it to the extremely large-scale NLP corpora currently used for language model pretraining, such as Colossal Clean Crawled Corpus (C4) [47], the Pile [48], and OpenWebText [37]. These datasets contain vast quantities of text scraped from the Internet, some of which is derived from patents. In fact, a recent analysis of the C4 dataset [49] found that "patents.google.com" is the single most-frequent source of text in the corpus,
as measured by number of tokens. Furthermore, Dodge et al. [49] found that this patent text was not clean: A significant percentage was machine-translated from non-English languages and/or extracted from images with OCR. HUPD differs markedly from these web-scraped datasets because we obtain our text directly from the USPTO BDSS; textual content in our dataset is clean, carefully-extracted, and consistently-formatted. Additionally, whereas other large-scale text datasets typically contain only a stream of text, HUPD contains numerous structured data fields for each document.[32] The main focus of HUPD is not on pre-training large language models from scratch, but due to its large size and high quality, it can be a good complement to the existing extremely-large-scale web-scraped NLP corpora.
C.4 Comparison with Popular Patent Search Tools and Large Repositories of Patent Data
In this section, we compare HUPD to popular English-language-based patent search tools and large repositories of patent data which are not constructed primarily for the ML/NLP community.
Publicly Available Patent Search and Analysis Tools. To the best of our knowledge, Google Patents is the most comprehensive and most diverse query-based patent search tool, containing over 120 million distilled patent documents (including patent applications, pre-grant publications, and granted patents), from more than one hundred patent offices all around the world. Similarly, PatentsView is a web-based patent data visualization and search tool that aims to improve the quality, impact, evaluation, and visibility of patents and patent applications filed to the USPTO. PatentsView is supported by the Office of the Chief Economist at the USPTO. The current PatentsView API allows each user to make up to 45 queries per minute. Using the API tool, users can get access to the full text of the title and abstract section of patent documents, as well the inventor, assignee, location, CPC/USPC classification details, among some other numerical and bibliographic statistics. The Lens Patent API allows search over 140 million patent meta-records worldwide. It has a free 14-day trial period (with limited API requests); afterwards, users need to pay fee for customized access, though pricing is determined on the nature and volume of the use case.
Aimed to replace the USPTO's existing search platform (Pub EAST, Pub WEST, Pat/FT, and App/FT) with a single, more comprehensive, and more modern online service, Patent Public Search is the USPTO's own recent search tool can perform and display searches over the USPTO's three big databases (namely, US-PGPUB, USPAT, and and USOCR). Patent Public Search is based on the Patents End-to-End (PE2E) search tool that the USPTO examiners use to identify prior art. There are additional online search tools developed and supported by different patent offices, such as WIPO Patentscope and Espacenet (supported by the WIPO and European Patent Office, respectively), that seek to improve public access to patent information.[33]
Managed by the EPO, the PATSTAT Global is a patent statistics database that contains bibliographic data and legal information about more than 110 million patent documents (patent applications and patents) from the patent offices of many industrial and developing countries from 1980 onwards.[34] While it has wide bibliographic coverage and is regularly updated, the PATSTAT database does require users to pay a subscription free for usage. It allows users to retrieve the title, abstract, inventor/application, classification code, technical field, other bibliographic metadata, and legal history of patent documents.[35]
Large Repositories of Raw US Patent Data. The USPTO makes several patent raw text datasets publicly available in XML formats.[36] As noted before, Patent Application Data/XML (Versions 4.0-4.4 ICE and Version 1.5) from the BDSS, together with the Patent Examination Research Dataset (PatEx) from USPTO's Public Patent Application Information Retrieval (PAIR) system, is the core
32Wikipedia contains structured fields alongside text for some documents. However, unlike patent applications and patent documents, the pages in Wikipedia contain different (non-uniform, varied) structured fields, making this information difficult to use across different tasks.
33Espanenet allows users to make queries in English, French, and German.
34The PATSTAT data is based on the EPO data and the data provided by other international patent offices on a voluntary basis. For more information: https://www.epo.org/searching-for-patents/business/patstat.html.
<sup>35</sup>We invite our readers to refer to the I<sup>3</sup> Open Innovation Dataset Index (https://iiindex.org/) to learn more about innovation-related datasets, tools, platforms, and resources.
36The USPTO Bulk Data Storage System (BDSS) website: https://bulkdata.uspto.gov/.
data we have used to construct HUPD. The patent applications in the USPTO Bulk Data system are organized by their filing application years and concatenated in big bulk XML files. Each patent application needs to be extracted and unconcatenated back to individual XML documents for clean data processing purposes. Additionally, the USPTO makes text data and rich bibliographic information for patents available via its PatentsView service.
The Google Patents Public Datasets collection makes consistently-formatted patent text and filing information available to users, though its primary purpose is not to be used as an NLP dataset. The collection contains bibliographic information on more than 90 million patent documents from 17 countries, in addition to the full textual content of millions of US patent documents, provided by IFI CLAIMS Patent Services [50]. The data from this repository can be downloaded using BigQuery.[37]
The MMatrixware REsearch Collection (MAREC) Dataset<sup>38</sup> is a static, standardized, and multilingual dataset of patent applications and granted patents. There are 19 million raw patent documents in the dataset—written in 19 languages (though the majority of them are written in English, German, and French), spanning an almost three-decade period (1976-2008), coming from the European Patent Office (EPO), Japan Patent Office (JPO), United States Patent and Trademark Office (USPTO), as well as the World Intellectual Property Organization (WIPO). The documents in the MAREC dataset are normalized to a uniform XML format. The standardized data fields are primarily the publication number, filing date, country of invention, language, citation information, inventor information, main subject classifications (e.g., IPC codes). Almost half of the documents contain full textual content of the inventions. The dataset, whose raw data size (almost 600GB) is comparable to ours, can be accessed via BigQuery and via bulk download.
We include a more directed comparison of HUPD with other patent datasets constructed for NLP patent analysis in Table 1 and Section C.1.
37The curators of the Google Patents Public Datasets also have an accompanying GitHub repository (https: //github.com/google/patents-public-data) that explains how to use the dataset on BigQuery.
38MAREC's website: http://www.ifs.tuwien.ac.at/imp/marec.shtml.
D Glossary
In this section, we provide a list of commonly used patent-related terms, concepts, and acronyms in this paper. While this list is not meant to be extensive or comprehensive, it is still inclusive and informative; it can be used by our readers as a reference guide as they read the paper. Unless otherwise marked with an asterisk * at the end, the definition for each term was taken from the USPTO's Glossary.[39]
39USPTO Glossary: https://www.uspto.gov/learning-and-resources/glossary.
of the Office takes action thereon. Express abandonment of the application may not be recognized by the USPTO before the date of issue or publication unless it is actually received by appropriate officials in time to act. Abandonment may be either of the invention or of an application. An abandoned application, in accordance with 37 CFR 1.135 and 1.138, is one which is removed from the USPTO docket of pending applications.
E Additional Tables and Figures
E.1 Binary Decision Classification
E.1.1 Complete Version of Table 4
IPC - Section	BernNB	MultiNB	Logistic	CNN	DistilBERTFT	BERTFT	RoBERTaFT
GO6F - Abstract	61.86	61.47	58.24	60.97	61.53	61.28	61.31
GO6F - Claims	63.96	62.06	58.02	63.38	63.37	62.97	63.25
H01L - Abstract	58.98	59.05	58.54	60.71	61.46	61.85	61.85
H01L - Claims	60.97	60.29	59.53	62.63	62.50	61.61	61.94
H04L - Abstract	59.35	58.75	58.75	59.89	60.54	60.52	60.05
H04L - Claims	62.13	61.04	58.04	62.34	61.42	61.47	61.74
H04W - Abstract	56.01	55.20	55.79	57.82	56.42	56.39	57.01
H04W - Claims	57.76	56.85	55.58	59.72	58.97	58.94	59.22
H04N - Abstract	60.74	60.64	58.79	60.37	62.01	61.93	61.51
H04N - Claims	62.51	61.01	57.53	63.98	62.82	61.98	62.14
A61B - Abstract	59.15	58.81	57.31	58.75	58.36	59.58	59.66
A61B - Claims	59.30	59.12	57.25	59.49	60.15	61.20	61.00
A61K - Abstract	58.14	57.82	55.46	56.85	58.47	56.45	57.08
A61K - Claims	57.31	57.93	56.20	59.06	58.72	57.91	57.84
GO1N - Abstract	59.85	59.89	57.25	59.98	59.00	60.30	61.10
GO1N - Claims	58.06	57.97	58.37	59.80	60.16	60.34	60.97
GO6Q - Abstract	61.53	61.64	58.52	60.46	61.23	61.09	61.56
GO6Q - Claims	63.96	63.31	57.17	62.90	61.88	62.19	63.25
Table 6: (Complete version of Table 4) Baseline performances of our models on the binary classification of patent decision task. All the models were trained and evaluated on the patent applications filed to the USPTO between 2011 and 2016. We note that BernNB and MultiNM denote Bernoulli and Multionomial NB classifiers trained on world-level unigrams (with minimum frequency of 3), respectively; Logistic a logistic regression model consisting of an embedding layer followed by a single linear layer trained on world-level unigrams; and CNN a Convolutional Neural Network with a 2-D convolutional layer and a max-pooling layer trained on world-level unigrams (with minimum frequency of 3). The superscript  $\mathrm{FT}$  on the Transformer models denotes that these models were fine-tuned, not trained from scratch.
E.1.2 Top IPC/CPC Codes and Their Brief Descriptions
IPC/CPC Code	Description
G06F	Electric Digital Data Processing
H01L	Semiconductor Devices
A61K	Preparations for Medical Purposes
H04L	Transmission of Digital Information
H04N	Pictorial Communication, e.g. Television
A61B	Diagnosis, Surgery, Identification
G06Q	Data Processing Systems or Methods
H04W	Wireless Communication Networks
G01N	Investigating or Analyzing Materials
Table 7: Brief descriptions of the most common nine IPC/CPC subclass codes present in our year ranges.
E.2 Multi-Class IPC/CPC Classification
E.2.1 Words with the Highest Weights in the Bernoulli NB Classifier
IPC	Words with Highest Weights
G06F	data, includes, device, semiconductor, system, substrate, one, method, second, least, structure, display, information, plurality, layer, present
H01L	user, film, second, access, data, one, semiconductor, wherein, includes, memory, unit, region, device, portion, operation, area, used, layer
A61K	device, including, computer, least, image, forming, layer, files, data, part, package, driving, conductive, includes, processing, direction
H04L	data, includes, semiconductor, device, disposed, system, one, substrate, method, layer, display, second, plurality, least, structure, content
H04N	provided, data, includes, one, device, semiconductor, method, formed, second, system, electrical, least, substrate, display, information, electrode
A61B	data, one, includes, device, method, substrate, electrode, second, semiconductor, memory, least, light, unit, layer, application, set, plurality
G06Q	data, device, substrate, includes, semiconductor, one, method, structure, system, display, present, plurality, image, non, least, n, content
H04W	semiconductor, data, device, layer, includes, disposed, high, one, display, substrate, method, second, structure, system, machine, plurality
G01N	data, device, includes, one, semiconductor, method, bus, least, layer, including, memory, substrate, image, computer, content, second, application
Table 8: Examples of words (excluding stopwords) with highest probability weights in their respective IPC subclasses in our Bernoulli naive Bayes classifier trained to predict the IPC code of a patent application based on the words in its abstract section. For a given IPC subclass label  $y$ , we looked at the probability value  $p(x_i|y)$  for each word  $x_i$  in the vocabulary, and listed the words with the highest probability values.
E.2.2 Confusion Matrix
Figure 7: Confusion matrix of IPC code classification at the subclass level. This matrix was obtained from the DistilBERT model that was fine-tuned on the abstract sections. The IPC codes were ordered by their sizes from left to right and from top to bottom, respectively. The light diagonal line present on the center figure represents high recall values. The diagonal line disappears towards the lower right corner, since patents belonging to those IPC subclasses do not appear in our test set.
E.2.3 Saliency Maps
To have a better understanding and appreciation of the behavior of our neural classifiers, we also made queries to identify which parts of the input texts our models might be attending to when making their predictions. To that end, we availed ourselves of various simple gradient-based saliency methods, such as SmoothGrad [51] and Integrated Gradient (IG; [52]).<sup>40</sup> We once again turned to the DistilBERT models for our analyses due to their relatively better performance and their potential to learn more complex representations. Figure 8 illustrates how much each input feature contributes to the prediction made by the DistilBERT model (trained on the abstract section as shown in Table 5). This analysis suggests the tokens that the model is paying attention to tend to be those that are most indicative of the technology area that each patent belongs to.<sup>41</sup>
Patent No.	Predicted Label	True Label	Saliency Map
US9272708B2	B60W	B60W	[CLS] a vehicle system includes an autonomous mode controller and an entertainment system controller . the autonomous mode controller controls a vehicle in an autonomous mode . the entertainment system controller presents media content on a first display while the vehicle is operating in the autonomous mode and on a second display when the vehicle is operating in a non- autonomous mode . a method includes determining whether a vehicle is operating in an autonomous mode , presenting media content on a first display while the vehicle is operating in the autonomous mode , and transferring presentation of the
"Autonomous Vehicle Entertainment System"
US9406017B2	G06N	G06N	[CLS] a system for training a neural network . a switch is linked to feature detectors in at least some of the layers of the neural network . for each training case , the switch randomly selective #ly di #sable #s each of the feature detectors in accordance with a pre #con #fi #gur #ed probability . the weights from each training case are then normal #-sized for applying the neural network to test data . [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
"System and Method for Addressing Overfitting in a Neural Network"
US9733811B2	G06F	G06F	[CLS] a method for profile matching includes receiving a plurality of user profiles , each user profile comprising traits of a respective user . the method includes receiving a preference indication for a first user profile of the plurality of user profiles . the method also includes determining a potential match user profile of the plurality of user profiles based on the preference indication for the first user profile . the method also includes presenting the potential match user profile to a second user . [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
"Matching Process System and Method"
Figure 8: Saliency maps generated using Integrated Gradient [52] for the DistilBERT model trained to predict the IPC subclass of a patent application based on its abstract section. All our visualizations were created using the Captum library [53]. The green color highlights tokens that contribute more heavily toward the class label, and the red color signifies tokens that contribute less strongly toward the class label. Our models predicted the class label correctly in each example. These three examples are the abstract sections of three patent applications by Ford, Google, and Kinder, respectively. The input tokens that contribute the most to our predictions appear to be associated with the primary technology areas of these patent applications. These qualitative findings confirm our belief that our models have learned domain-specific features about IPC subclasses.
40We used the PyTorch-compatible interpretability library, Captum [53], in our experiments. 41On the other hand, the saliency maps of the DistilBERT models trained on both the abstract and the claims for the binary classification of the patent decision task on a couple of examples were less insightful.
E.3 Abstractive Summarization
Model	Setting	R-1	R-2	R-L
T5 (Small)	Description→Abstract	62.87	47.20	54.36
T5 (Small)	Claims→Abstract	69.00	53.82	59.88
E.3.1 Generated Summaries
Table 10: Examples of claims summaries produced by our T5-Small model. Qualitatively, the models produce fluent patent abstracts complete with accurate details drawn from the claims section. Note also how different the structure of the above legal language is from most text used to train large language model; often, the entire abstract is a single long and complex sentence.
E.4 Cross-Category Evaluation of BERT Acceptance Prediction
Figure 9: Cross-category evaluation of BERT acceptance prediction classifiers trained on one IPC code evaluated to predict acceptance on other IPC codes. Patent categories that are conceptually similar appear to have closer shared criteria for acceptance (for example, G06F-Electric Digital Data Processing and H04N-Pictorial Communication). Most models also tend to perform well when evaluated on patent applications in H04L-Transmission of Digital Information; moreover, the model trained on H04L-Transmission of Digital Information has high predictive power across other application types. This might reflect acceptance criteria in this category involving more high-level evaluations of quality, as opposed to more domain-specific innovations in engineering or manufacturing.
F.1 Binary Decision Classification
For this first task, we looked at the patent applications that were filed to the USPTO between January 2011 and December 2016 and excluded the pending applications from our experiments. Initially, we trained individual domain-specific classifiers, ranging from NB classifiers to RoBERTa, to predict the acceptability of patent applications in the most common IPC subclasses (see Figure 2). We used both the abstract and the claims, though separately.
Figure 10: Distribution of decision status labels for patents filed between 2011 and 2016. Note that the relative share of pending to rejected applications increases over time as certain patents remain under review beyond the end of our metadata collection period. Approximately three quarters of patent applications are labeled as new filings.