SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ethical Considerations. While building HUPD, we followed the data sheets and statements introduced by Gebru et al. [33] and Bender and Friedman [34], discussing the motivations, objectives, collection process, workflow, use cases, distribution, maintenance, potential contributions, and potential misuses of our dataset and research (see: Section A).<sup>17</sup>
Impact on Underserved Communities. HUPD contains patent applications in English, a language with heavy attention from the NLP community. However, innovation is spread across many languages, cultures, and communities that are not reflected in this dataset. HUPD is thus not representative of all kinds of innovation. Furthermore, patent applications require a fixed cost to file and are not accessible to everyone. One goal of our dataset is to spur research that reduces the cost of drafting applications, potentially allowing for more people to seek intellectual property protection for their innovations.
5 Using the Dataset
Due to the versatile nature of HUPD, a wide range of NLP tasks and experiments can be constructed from the dataset by selecting the appropriate fields and metadata contained in each patent application. In this section, we highlight four tasks that we believe to be among the most valuable and relevant to the NLP and IP communities: (i) binary classification of patent decisions, (ii) multi-label classification of patent IPC/CPC categories, (iii) language modeling, and (iv) summarization. Of course, the dataset may easily be used to conduct other investigations, such as patent clustering, prior art search, and early detection of superstar inventions.[18] In what follows, we describe our four tasks of interest in detail and contextualize their importance within the world of IP. Table 3 provides a brief overview of each task and the corresponding metrics used to measure task performance.
Patent Acceptance Prediction. Given a section of an application (in particular, the abstract, claims, or description), we predict whether the application will be accepted by the USPTO. From the perspective of the NLP community, this is a standard classification task. Yet, the potential applications and benefits of this decision task, as well as its difficulty, distinguish it from prevalent binary classification benchmarks (e.g., SST, Yelp). In our experiments, we focus on applications without parent filings to make our setup simple and clear, thereby excluding all the CONT-applications. Also, we do not include any pending applications.
Automated Subject Classification. The next task is to predict the primary IPC or CPC code of a patent application given a subset of the text of the application. A coherent automated classification of patent documents into different technology fields can facilitate effective assignment of patent
<sup>17</sup>In Section A, we share a data card for our dataset, in the style of Gehrmann et al. [35].
<sup>18</sup>Since it would be challenging to simultaneously study all of these tasks made possible by our dataset, we leave some of these experiments for future work.
Figure 2: IPC distribution of accepted patent applications from 2011 to 2016 at the IPC subclass level. There are 637 IPC subclass labels in HUPD, of which the most common 20 codes make up half of the distribution. G06F-Electric Digital Data Processing is the largest IPC subclass, accounting for  $10.4\%$  of applications.
applications to examiners. In addition, it may help create a rigorous and standardized catalog of prior art for research and exploration. This task might also help the early identification of valuable inventions that bridge multiple technological domains or the emergence of new subject areas. In our experimental setups, we predict the main IPC codes of patent applications at the subclass level, as IPC codes are available for a larger set of patents than CPC codes.[19] There are 637 IPC codes at the subclass level in our dataset, but they are not uniformly distributed (see Figure 2); for instance, G06F-Electric Digital Data Processing constitutes  $10.4\%$  of accepted patents that were filed between 2011 and 2016 and the most popular 15 IPC codes make up almost  $40\%$ . Hence, it is difficult to achieve strong classification performance by only predicting the major classes.
Language Modeling. Next, we move from classification to language modeling (LM). We consider masked LM of patent claims. We concentrate on the claims section because it forms the basis of the invention described in a patent application; it has a distinctive language style; and it is considered to be the most legally potent part of a patent. We also conduct LM experiments on the abstract sections of patents, which are more similar to standard natural language. These models can be used for downstream tasks, as well as for domain-specific investigations. In Section H, we demonstrate one application of our LMs by visualizing the average embeddings of different patent categories under this fine-tuned model. The results of this exercise may reveal the textual evolution of innovation concepts and trends in patent applications across different technology areas.[20]
Abstractive Summarization. The formulation of the final task follows naturally from the structure of our patent data: Each patent contains an abstract in which the applicant summarizes the content of the patent. We use this section as the ground truth for our abstractive summarization task, and we use either the claims or the description as the source text. We perform this conditional generation task with the same motivation that inspired Sharma et al. [4]; our setup is similar to theirs apart from the size and scope of our dataset. One difference in our current setup is that our dataset allows us to explore using either the claims or description section for the source text, whereas in Sharma et al. [4] only the description section is available.[21]
In the next two sections, we establish benchmarks for the four tasks, describe the models used, and analyze our findings. In our experiments, we typically use the abstract or claims sections as the input
<sup>19</sup>Of the existing patent applications containing CPC codes (between 2011 and 2016) in our corpus, the overlap between their IPC and CPC codes at the subclass level is  $99.6\%$ .
20We perform masked LM, as in BERT [21], rather than autoregressive LM, in an effort to produce higher quality contextual representations.
21The claims sections of patent applications are typically long (on average 1272 words—almost an order of magnitude longer than the abstract sections (see Table 2)); hence, this task is inherently difficult. In contrast to most other summarization tasks (e.g., CNN/DM article summarization), it is not sufficient for the model to draw from the first or last few sentences of the input to generate the summary; rather, the information needed to generate the text may be distributed throughout the input.
IPC - Section	BernNB	MultiNB	Logistic	CNN	DistilBERTFT	BERTFT	RoBERTaFT
G06F - Abstract	61.86	61.47	58.24	60.97	61.53	61.28	61.31
G06F - Claims	63.96	62.06	58.02	63.38	63.37	62.97	63.25
H01L - Abstract	58.98	59.05	58.54	60.71	61.46	61.85	61.85
H01L - Claims	60.97	60.29	59.53	62.63	62.50	61.61	61.94
H04L - Abstract	59.35	58.75	58.75	59.89	60.54	60.52	60.05
H04L - Claims	62.13	61.04	58.04	62.34	61.42	61.47	61.74
H04N - Abstract	60.74	60.64	58.79	60.37	62.01	61.93	61.51
H04N - Claims	62.51	61.01	57.53	63.98	62.82	61.98	62.14
A61B - Abstract	59.15	58.81	57.31	58.75	58.36	59.58	59.66
A61B - Claims	59.30	59.12	57.25	59.49	60.15	61.20	61.00
G01N - Abstract	59.85	59.89	57.25	59.98	59.00	60.30	61.10
G01N - Claims	58.06	57.97	58.37	59.80	60.16	60.34	60.97
GO6Q - Abstract	61.53	61.64	58.52	60.46	61.23	61.09	61.56
GO6Q - Claims	63.96	63.31	57.17	62.90	61.88	62.19	63.25
Table 4: Baseline performances of our models on the binary classification of patent acceptance task. All the models were trained and evaluated on the patent applications filed to the USPTO between January 2011 and December 2016. All the test sets contained equal numbers of accepted and rejected applications, so the baseline accuracy to compare these models against is  $50\%$ . In all but one IPC category, the models trained on the claims sections yielded the best performance. None of the individual accuracy scores, however, went beyond  $64\%$ . In Section 7, we further report results of our conditional universal acceptance prediction classifier. The superscript  $\mathrm{FT}$  denotes that these models were fine-tuned. The common IPC categories presented in this table are G06F-Electric Digital Data Processing, H01L-Semiconductor Devices, H04L-Transmission of Digital Information, H04N-Pictorial Communication, A61B-Diagnosis, Surgery, Identification, G01N-Investigating or Analyzing Materials, and G06Q-Data Processing Systems or Methods. (See Table 6 for our full results.)
to the models; however, future studies using our dataset can easily include other sections, such as the background and description.[22]
While our dataset includes patent applications filed to the USPTO between 2004 and 2018, we primarily focused on the 2011-2016 year range in our main NLP experiments. We chose this specific time period not only because the patent applications during these years reflect more recent inventions and cover diverse industries, but also because they were suitable for training various NLP models within a day at the time of our experiments. We used the same subset across most of our experiments for consistency and coherency. (For additional details on our experimental setup, please see: Section F).
Patent Acceptance Prediction. Table 4 reports the performances of our models on the most popular IPC codes. In each IPC subclass, with the exception of G01N-Investigating or Analyzing Materials by Determining Their Chemical or Physical Properties, the best performance was achieved by the models that relied on the claims. This result is consistent with the idea that the claims define the overall scope, novelty, and usefulness of the invention. The claims, together with the prior art, provide the most useful and critical information about the patentability of an invention. It was, however, surprising to discover that there was not a significant difference between the NB classifiers and the BERT models in terms of accuracy scores across categories. We speculate that the BERT models might have mirrored the behaviors of the NB classifiers at the end, failing to go beyond word-level feature extraction. Nonetheless, we note the task is difficult, and currently uses only a limited portion of a patent text to determine the acceptability of the invention. We posit that new advances in Transformer models for longer text may improve predictive accuracy for this task.[23]
Automated Subject Classification. For the multi-class classification task, we considered only accepted patents, since they contain reliable and accredited IPC/CPC codes. Table 5 details the TOP1 and TOP5 accuracy scores for the multi-class IPC code classification results at the subclass level. First, we note that performance increases with more sophisticated models. The DistilBERT model,
22Due to space limitations, we present a subset of our results. Additional findings are in the Appendix.
23The method used here is purely on internal textual features, not on past patenting history or prior art information, which are important considerations in predicting acceptance of a patent application. We leave the incorporation of such metadata into this task for future work.
Section	BernNB	MultiNB	CNN	DistilBERTFT	RoBERTaFT
TOP1	TOP5	TOP1	TOP5	TOP1	TOP5	TOP1	TOP5	TOP1	TOP5
Abstract	40.65	63.92	47.38	75.17	53.87	81.69	61.75	89.11	61.07	88.24
Claims	39.37	65.42	48.09	77.78	56.10	83.40	63.40	90.22	62.82	89.46
Table 5: Performances of our models on the multi-class IPC classification of patent codes at the subclass level. Our DistilBERT models yielded the best performance overall under both abstract and claims input setups. TOP1 (i.e., accuracy) checks whether our prediction is the same as the actual label, whereas TOP5 measures whether the actual label of the input is amongst our five top predictions (i.e., five classes with the highest probability weights). Our high TOP5 scores indicate that our models are good at predicting the IPC codes of both popular and underrepresented classes. Figure 7 also provides evidence towards this conclusion.
trained on the claims section, achieves  $63\%$  accuracy at TOP1 and  $90\%$  accuracy at TOP5; this is notable since, as Figure 2 implies, a majority-class baseline could only yield around  $10\%$  for TOP1 and  $26.5\%$  for TOP5. In fact, the diagonal line in Figure 7 (center) also illustrates that the DistilBERT model, for instance, has learned features that enable it to successfully classify minority categories, such as F25C-Producing, Working or Handling Ice and D05B-Sewing. In general, the models trained on the abstract section performed as well as those trained on the claims section, perhaps an indication (in accord with the findings of Li et al. [3]) that the abstract alone contains useful information about the appropriate principal technology area the patent application might belong to. Model performance at the class level,[24] as opposed to the subclass level, was even stronger. The DistilBERT models achieve over  $80\%$  TOP1 accuracy at the class level. Furthermore, incorrect predictions were often from similar or related technology areas. (Our high TOP5 scores also corroborate this finding.)
Language Modeling. We trained a masked language model on patent claims in the style of BERT [21]. We trained on a subset of the full patent dataset consisting of patent applications from 2011 to 2016 (1.57M applications) and evaluated on all patent applications from 2017 (187K applications). We performed masked language modeling with DistilRoBERTa [36] (82M parameters), initializing with a model pretrained on OpenWebText [37]. We release this model publicly so that researchers may utilize it for downstream tasks.[25]
Abstractive Summarization. We find that patent descriptions and claims can both be effectively summarized into patent abstracts, with claims leading to improved performance across all metrics. Qualitatively, the models can produce fluent abstracts complete with accurate details drawn from the claims section, as shown in Table 10. Consistent with BIGPATENT, our results suggest that the summarization of patent data could serve as a new domain-specific conditional generation task for the NLP community.
7 Evolution of Innovation Criteria and Trends over Time
A key advantage of HUPD is its combination of unstructured textual content and rich, structured bibliographic metadata. Patents, like many other applied natural-language domains, exhibit concept and domain shifts—the criteria for innovation varies across categories and evolves over time. In this section, we discuss some ways that our dataset exhibits time- and state-varying concepts. We hope this feature of patent data allows for fresh studies of the nature of shifts across categories and supports the development of NLP models that accommodate concept or distributional shifts.
Universal Acceptance Classification. We explored how a universal decision classifier, trained on all the IPC categories, might perform on individual categories. To this end, we trained a conditional DistilBERT classifier, where the conditional information had the title, issue year, and IPC code, in addition to the abstract section, to predict the acceptance likelihood of a patent application.[26] We used the patent applications filed between 2011 and 2016. The model achieved an overall accuracy score of  $62\%$  on the entire class-balanced test set. When we assessed the model's performance on individual IPC subclasses, we discovered, for instance, that the model yielded almost  $64.5\%$  accuracy on G06F-Electric Digital Data Processing,  $61.9\%$  accuracy on H04N-Pictorial Communication, e.g., Television, and  $57\%$  accuracy on G06Q-Data Processing Systems or Methods.
24We note that there are only nine broad IPC classes.
25We also use this model to visualize patent categories in Section H.
[IPC Subclass] Abstract [Abstract]; the task was to predict the acceptance likelihood of a patent application.
Figure 3: Performance of a BERT decision classifier trained on applications from 2011 to 2013 and evaluated on applications produced in earlier and later years. While model performance decays over time across most categories (suggesting changing acceptance standards), acceptance criteria appear to change more quickly in faster-moving fields (e.g., H01L-Semiconductor Devices and H04W-Wireless Communication) and slower in more developed fields (e.g., A61B-Diagnosis, Surgery, Identification). Of further interest may be the rise in turnover in acceptance criteria in G06Q-Data Processing Systems or Methods starting in 2014, which may coincide with the concurrent growth in development of B2B and fintech products. In future work, it may also be illustrative to use this dataset to investigate the impact of the U.S. Supreme Court decision in Alice Corp. v. CLS Bank International, 573 U.S. 208 (2014), on the software-related patent applications issued after 2014.
Cross-Category Evaluation. In order to understand and identify relationships between different patent evaluation criteria in different IPC classes, we also took each DistilBERT model trained on one IPC subclass of patents and evaluated it across all the other popular IPC subclasses. Figure 9 provides an illustration of our empirical findings. Patent technology areas that are conceptually closer to each other, such as G06F-Electric Digital Data Processing and H04L-Transmission of Digital Information, appear to have similar standards for patent acceptance. Notably, models trained in one context do not generalize to most other contexts. This suggests the criteria for patent acceptance are sensitive to the technical demands of the specific category of each application.
Performance Over Time. We can also use our models and dataset to understand the evolving criteria for patent acceptance, as well as innovation trends over time. Figure 3 shows the performance of a decision classification model trained on patent applications from 2011 to 2013 evaluated on applications produced in earlier and later years. While model performance usually deteriorates over time, suggesting changes in the features that predict patent acceptance, the rate of decay appears to be sharper for fields anecdotally thought to be faster-moving. This property of patent data may make it useful for studying concept shift that varies by class.
Additional Tasks. Given the richness of patent text and metadata, HUPD enables research on a wide range of tasks and use cases that may be explored in future work. In Section G, we describe some of these tasks—such as long sequence modeling, patent clustering, and patent examiner assignment—as well as the potential social impact and applications of our work.
We presented the Harvard USPTO Patent Dataset (HUPD), a large, versatile, and comprehensive structured corpus of patent data constructed for the NLP community. HUPD contains 4.5 million English-language utility patent applications filed to the USPTO between 2004 and 2018. We also established benchmarks for two classification-based and two generation-based tasks on our dataset. We provided detailed qualitative analyses of the models trained for these tasks and demonstrated how our dataset presents a setting with measurable concept shift. We hope that the combination of our dataset and the models used in this paper will not only advance research in patent analysis, but eventually also help patent applicants prepare more successful patent filings and provide a domain-specific laboratory for a multitude of NLP tasks.
We thank Christopher Bavitz, Yonatan Belinkov, Tommy Bruzzese, Dallas Card, Jiafeng Chen, Kiran Dwivedi, Julia L. Englebert, Sebastian Gehrmann, Tayfun Gur, Umit Gurun, Dan Jurafsky, Peter Henderson, Ryan Kearns, Megan Ma, Daniel McFarland, Drew Pendergrass, Karen Sinclair, Kyle Swanson, Andrew A. Toole, Brandon Walton, Ian Wetherbee, Benjamin H. Wittenbrink, Michihiro Yasunaga, the members of the Lab for Economic Design at Harvard University, and attendees at our Stanford CodeX and Microsoft Research New England presentations for helpful comments and suggestions. We also would like to thank the anonymous reviewers for their valuable feedback and critiques. We are especially thankful to the USPTO for providing the bulk data and patent research datasets, and Andrew A. Toole, the Chief Economist at the USPTO, for his help navigating the USPTO's data products. We gratefully acknowledge the support of the three Microsoft Azure credit grants from the Harvard Data Science Initiative for data storage and computation. Some of the experiments presented in this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University and the Scalable Magic switch cluster. Suzgun gratefully acknowledges the support of a Harvard University Center of Mathematical Sciences and Applications (CMSA) Economic Design Fellowship and the Harvard College Research Program. Melas-Kyriazi gratefully acknowledges the support of a Rhodes Scholarship. Sarkar gratefully acknowledges the support of a National Science Foundation (NSF) Graduate Research Fellowship. Kominers gratefully acknowledges the support of the Ng Fund and the Mathematics in Economics Research Fund of the CMSA, as well as NSF grant SciSIP-1535813.
[1] USPTO. FY 2020 Performance and Accountability Report. Technical report, 2020. URL https://www.uspto.gov/sites/default/files/documents/USPTOFY20PAR.pdf.
[2] F. Piroi, M. Lupu, A. Hanbury, and V. Zenz. CLEF-IP 2011: Retrieval in the Intellectual Property Domain. In CLEF, 2011.
[3] Shaobo Li, Jie Hu, Y. Cui, and J. Hu. DeepPatent: Patent Classification with Convolutional Neural Networks and Word Embedding. Scientometrics, 117:721-744, 2018.
[4] Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1212. URL https://www.aclweb.org/anthology/P19-1212.
[5] A Toole, N Pairolero, A Giczy, J Forman, C Pulliam, M Such, K Chaki, D Orange, A Thomas Homescu, J Frumkin, et al. Inventing AI: Tracing the Diffusion of Artificial Intelligence with US Patents, 2020.
[6] P.S. Menell, M.A. Lemley, R.P. Merges, and S. Balganesh. *Intellectual Property in the New Technological Age* 2021: Vol. I Perspectives, Trade Secrets and Patents. Clause 8 Publishing, 2021. ISBN 9781945555183. URL https://books.google.com/books?id=zVJhzgEACAAJ.
[7] Bhaven Sampat and Heidi L Williams. How Do Patents Affect Follow-on Innovation? Evidence from the Human Genome. American Economic Review, 109(1):203-36, 2019.
[8] Mark A Lemley and Bhaven N Sampat. Is the Patent Office a Rubber Stamp? Emory Law Journal, 58:181, 2008.
[9] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The Semantic Scholar Open Research Corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.447. URL https://aclanthology.org/2020.acl-main.447.
[10] Rémi Lebret, David Grangier, and Michael Auli. Neural Text Generation from Structured Data with Application to the Biography Domain. In Proceedings of the 2016 Conference
9 Dataset Checklist
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(c) Did you describe the limitations of your work? [Yes] See Section 4.
(d) Did you discuss any potential negative societal impacts of your work? [Yes] See Section A (Data Card) of the Appendix.