SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To address the issue of imbalance in our decision status labels (see Figure 10), we used a weighted random sampler to select samples. We then randomly apportioned the data into training and test sets with an 85-15 split, but fixed the random seed across each model in each category to ensure fair comparability.
Our baselines for this task consisted of various subsets of Bernoulli and Multinomial naive Bayes classifiers, logistic regression (Logistic), CNN, DistilBERT [36], BERT [21], DistilROBERTa [36], RoBERTa [23], and T5-Small [54] for different tasks.42
F.2 Abstractive Summarization
We converted the summarization task into a language modeling task using the setup described by Raffel et al. [54]. The source text and summary were simply concatenated, separated by a separation phrase ("summarize:"). We used the T5-Small "Text-to-Text Transformer" architecture of Raffel et al. [54] with 60 million parameters. We initialized with a model pretrained on the C4 dataset (t5-sma11 on HuggingFace Transformers), although we found that training from scratch produced nearly the same performance due to the size of our dataset.
42We used scikit-learn [55] to train and evaluate our NB classifiers and HuggingFace's Transformers codebase [56] to implement and finetune our Transformers. We used HuggingFace's Tokenizer library for pre-processing and tokenization: For NB classifiers, Logistics, and CNNs, we adopted the "WordLevel" tokenizer; for each Transformer model, we automatically selected and applied its own associated tokenizer.
We used the same training-validation subsets used in the language modeling task: Applications from 2011-2016 for training and those from 2017 for validation. We truncated the training source texts (i.e., the claims/description) to a maximum of 1024 tokens. We trained for three epochs using the Adam [57] optimizer with batch size of 32 and a learning rate of  $5 \cdot 10^{-5}$ .
F.3 Special Focus on the 2011-2016 Year Range in Our Experiments
While the dataset consists of the patent applications filed to the USPTO between 2004 and 2018, we primarily focused on the 2011-2016 year range in our main NLP experiments. We acknowledge that the primary reason behind this specification was the limited set of compute resources we had to train and deploy our models: We had limited GPU credits and memory bandwidth at our disposal when we were training and fine-tuning our models. We ultimately decided to focus on the 2011-2016 range, because the patent applications from this time period (i) seem to reflect recent technical developments and innovations, (ii) cover a good variety of different industries and technology areas, and (iii) could be easily used to train classification, language, and summarization models under one day to study the characteristics and evolution of patent-driven innovation in the United States. Furthermore, we used the same subset throughout most of our experiments for consistency and coherency—and this also allowed us to easily cache our data and get access to it almost instantly, rather than pre-processing, tokenizing, and batching each time for a new year range. We demonstrate the utility and benefits of the dataset on an exemplary subset, but we wish to remind our readers that a major goal of this dataset is to facilitate future research that can use the comprehensive time variation and rich metadata that we make available. Therefore, we hope that these considerations justify the special focus on the patent applications in the 2011-2016 year range.
G Discussion of Additional Tasks
G.1 Long Sequence Modeling
One interesting application of HUPD that we have not yet explored is to use its patent description field as part of a benchmark for long-sequence modeling. Long-sequence modeling has recently gained significant attention due to the proliferation of efficient Transformer architectures (e.g., Longformer, Performer). Popular benchmarks for these models include synthetic tasks, byte-level text tasks, and computer vision tasks; there does not exist a large-scale long-sequence word-level NLP benchmark since it is difficult to collect a large corpus of narrowly-focused long text documents. The task of language modeling of patent descriptions might be well-suited for this purpose; HUPD contains over 4.5M descriptions with an average length of 11,855 tokens (see Table 2), which far exceeds the context length of traditional Transformer models.
G.2 Patent Clustering
A second application of the patent data is patent clustering: Given a patent application, one is tasked with finding similar patents (or patent applications). This tasks is particularly interesting to the IP community, since patent lawyers and examiners are interested in finding prior art for a given invention. To define clusters and find related patents, it would be possible to use a combination of metadata fields including fine-grained IPC/CPC classification codes, publication date, and inventor information.
G.3 Patent Examiner Assignment
A third application of the patent data makes use of the examiner field, which has not been used in the tasks discussed above. This task involves predicting the examiner to which a new patent application should be assigned. This task may be viewed as an even more-fine-grained version of patent classification, because patent examiners often focus on a very narrow sub-field in which they have expertise or knowledge. We note that this task is possible under our framework because HUPD contains both raw texts of patent applications and also rich bibliographic metadata about each patent application.[43]
G.4 Potential Social Impact and Applications & Future Work
In recent years, the USPTO has introduced a pilot program, called "Pro Se Assistance Program", to help small businesses and independent inventors file patent applications without enlisting the aid of a registered patent attorney or a legal agent. This initiative aims to enhance the quality of applications without putting any further financial burden on the shoulders of patent applicants who might have limited resources and means, as well as educating inventors about intellectual property protection and the patent filing process [59]. We hope our patent dataset and models might also provide some assistance to independent inventors and small businesses in improving the overall textual quality of their patent applications, to classify the technology areas of their inventions accurately, and to generate the abstract sections of their applications from their patent claims and/or description.
Given the scope of this paper, we could not conduct experiments on patent clustering, patent value prediction, or identification of "superstar" inventions. However, we foresee these research directions as direct valuable and impactful applications of our dataset, and suggest the research community push the boundaries on these fronts. In addition, our dataset's combination of natural language with structured metadata makes it an ideal gymnasium for evaluating and developing models that address concept shift across contexts and over time.
<sup>43</sup>It is of vital importance to assign a new patent application to an examiner who is skilled and qualified to understand the merits and shortcomings of the application in a critical and timely manner. And this is not a problem unique to patents: Shah [58], for instance, discusses the challenges and potential solutions in the current peer-review systems in scientific research, highlighting the importance of the assignment of the reviewers to papers and explaining the detrimental effects of the misutilization and mismatching of reviewer expertise. Shah [58] proposes using a two-stage assignment procedure, wherein the similarity score for every paper-reviewer pair is computed first and the maximization of the overall "utility" of paper-reviewer assignment is performed using the computer similarity scores second. Such an assignment procedure based on sum-similarity optimization methods can be studied using our dataset, for instance.
H Patent Category Visualizations
Figure 11: Visualizations of the vector representations of the data fields (abstract, summary, and title) of patent applications, embedded using four different Transformer models. Of these four models, two were fine-tuned on HUPD (Fine-tuned DistilRoBERTa and Fine-tuned T5-Small) and two were off-the-shelf, pre-trained models (RoBERTa-Base and Mini-LM v2). The Mini-LM v2 model was trained on a paraphrase dataset to produce good sentence-embeddings for text clustering. To create the figures above, we randomly sampled 500 patent applications from each of the top nine IPC categories for each year from 2008 to 2018. For each data field and each model, we computed embedding vectors using the model and reduced the dimensionality of these vectors using PCA. The above plots show the results of this dimensionality reduction, colored by IPC codes. We see that categories cluster strongly and similar categories (e.g., H04W-Wireless Communication and H04L-Transmission of Digital Information) are often close in embedding space.
Figure 12: Visualizations of the vector representations of the data fields (abstract, summary, and title) of patent applications, embedded using four different Transformer models. This figure is similar to Figure 11, but uses UMAP [60] for dimensionality reduction rather than PCA.
Figure 13: Depiction of the evolution over time of the averaged embeddings of the abstracts of patent applications from G06Q-Data Processing Systems or Methods relative to the other popular IPC codes in our dataset. To create this figure, as in the figures above, we randomly sampled 500 patent applications from each of the top nine IPC categories for each year from 2008 to 2018. Using our custom DistilRoBERTa model, we computed embedding vectors for the abstracts of each patent application and then reduced the dimensionality of these vectors using UMAP. The figure above shows the average UMAP embedding of the G06Q category for every other year, along with the average embeddings of the other categories (averaged across all samples from all years). The movement of the centroids of the G06Q category might be consistent with covariate shift over time in the distribution of language of patents in the category. G06F-Electric Digital Data Processing seems to move closer to H04L-Transmission of Digital Information/H04W-Wireless Communication. This evolution seems to be consistent with the digitization of data processing methods over the past two decades.