SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We publicly release our models, along with the data loading and tokenization code, on our website.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Section F (in the Appendix).
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] Most of our empirical results came from a single run due to computational constraints. They are intended to be illustrative of the tasks that can be performed with our dataset. We therefore do not report error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We used four RTX 8000 GPUs from an academic cluster to perform our masked language modeling and summarization experiments, which were our the most compute-intensive experiments. Each of these experiments took approximately four days, for a total of approximately 16 GPU-days each.
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the USPTO.
(b) Did you mention the license of the assets? [Yes] The dataset is released under the Creative Commons Attribution 4.0 International License.
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Our data is obtained from publicly-available sources; the patent applicants are aware that their applications will be posted publicly upon submission. We have been in contact with the Office of the Chief Economist at the USPTO on collecting and compiling the data the USPTO makes available on patent text and office actions.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The patent data includes inventor and examiner information, but as discussed above, the authors are aware that this information will be publicly distributed upon submission. We do not believe offensive content is common in patent data.
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A].
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A].
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A].
A.1 Dataset Description.
Dataset Summary. See Section 4.
Languages. The dataset contains English text only.
Domain. Patents (intellectual property).
Additional Details. The dataset contains utility patent applications filed to the USPTO between January 2004 and December 2018.
A.2 Meta Information
Dataset Curators. The dataset was created by Mirac Suzgun, Luke Melas-Kyriazi, Suproteem K. Sarkar, Scott Duke Kominers, and Stuart M. Shieber.
Licensing Information. The dataset is released under the Creative Commons Attribution 4.0 International License.
Leaderboard/Benchmarks. The dataset has no associated public leaderboard; however, it introduces four useful benchmarks for the ML/NLP and Econ/IP communities, namely binary classification of patent decisions, multi-class IPC/CPC classification of patent codes at the subclass level, language modeling, and abstractive summarization. We release the models and code used in the main experiments on our GitHub codebase so that anyone can reproduce our empirical results, build their own models, make meaningful evaluations, and compare the performances of their models to ours.
For benchmarking purposes, we decided to split the data chronologically, establishing 2011-2016 as the training range and 2017 as the test range; however, this is not an absolute split for all the experiments. The users of our dataset can easily filter the metadata and fields, specify the years that they want to focus on, choose training and test sets as they wish, and thus create their own task or experiment setups. We believe the degree of freedom our dataset offers, as well as the allowance the dataset provides for different types of experiments and investigations that NLP and IP researchers and practitioners can take using the different data fields, is of paramount importance and novelty. In particular, we do not want, in any way, to restrict the usage and flexibility of the dataset by randomly or intentionally splitting according to some criteria (e.g., split the data chronologically, based on some certain grouping of IPC/CPC codes, etc.) for all the experiments.
A.3 Dataset Structure
Data Format and Structure. Each patent application is defined by a distinct JSON file, named after its application number, and includes information about the application and publication numbers, title, decision status, filing and publication dates, primary and secondary classification codes, inventor(s), examiner, attorney, abstract, claims, background, summary, and full description of the proposed invention, among other fields. There are also supplementary variables, such as the small-entity indicator (which denotes whether the applicant is considered to be a small entity by the USPTO) and the foreign-filing indicator (which denotes whether the application was originally filed in a foreign country). In total, there are 34 data fields for each application. A full list of data fields used in the dataset is catalogued in the next section.
Data Instances. Each patent application in our patent dataset is defined by a distinct JSON file (e.g., 8914308.json), named after its unique application number. The format of the JSON files is as follows:
{
    "application_number": "...",
    "publication_number": "...",
    "title": "...",
    "decision": "...",
    "dateProduced": "...",
    "datepublished": "...",
    "main_cpc_label": "...",
    "cpc_labels": [" ..., ", "", ..."],
    "main_ipcr_label": "...",
    "ipcr_labels": [" ..., ", "", ..."],
    "patent_number": "...",
    "filing_date": "...",
    "patent-issue_date": "...",
    "abandon_date": "...",
    "uspc_class": "...",
    "uspc_subclass": "...",
    "examiner_id": "...",
    "examiner_name_last": "...",
    "examiner_name_first": "...",
    "examiner_name_middle": "...",
    "inventor_list": [
        "inventor_name_last": "...",
        "inventor_name_first": "...",
        "inventor_city": "...",
        "inventor_state": "...",
        "inventor_country": "..."
    ]
},
"abstract": "...",
"claims": "...",
"background": "...",
"summary": "...",
"full_description": "..."
}
Data Fields. In addition to distinct JSON files for each application, we consolidate patent metadata into a CSV file that links patent application numbers with both the metadata in the JSON files (e.g., dateProduced, main_cpc_label) and additional features made available by the USPTO's PatEx data associated with the patent applications. These additional features are atty_docket_number, file_location, wipo_pub_number, wipo_pub_date, patent-issue_date, small-entityindicator, foreign, appl_status_desc. We include code for querying subsets of patent applications by all the features we include for NLP experiments.
Data Statistics. The dataset contains 4,518,263 utility patent applications filed to the USPTO between January 2004 and December 2018.
A.4 Dataset Creation.
Source Data. The Harvard USPTO Patent Dataset synthesizes multiple data sources from the USPTO: While the full patent application texts were obtained from the USPTO Bulk Data Storage System (Patent Application Data/XML Versions 4.0, 4.1, 4.2, 4.3, 4.4 ICE, as well as Version 1.5) as XML files, the bibliographic filing metadata were obtained from the USPTO Patent Examination Research Dataset (in February, 2021).
Annotations. Beyond our patent decision label, for which construction details are provided in Section 2, the dataset does not contain any human-written or computer-generated annotations beyond those produced by patent applicants or the USPTO.
Personal and Sensitive Information. The dataset contains information about the inventor(s) and examiner of each patent application. These details are, however, already in the public domain and available on the USPTO's Patent Application Information Retrieval (PAIR) system, as well as on Google Patents and PatentsView.
Special Test Sets. We presented four mainstream NLP experiments on HUPD, namely binary classification of patent decisions, multi-class IPC/CPC classification of patent codes at the subclass level, language modeling, and abstractive summarization. We reported our results in Section 6.
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33   
34   
35   
36
Data Shift. A major feature of HUPD is its structure, which allows it to demonstrate the evolution of concepts over time. As we illustrate in Section 7, the criteria for patent acceptance evolve over time at different rates, depending on category. We believe this is an important feature of the dataset, not only because of the social scientific questions it raises, but also because it facilitates research on models that can accommodate concept shift in a real-world setting.
Sole Focus on the Textual Component. Patent applicants often submit accompanying (black-and-white) drawings (i.e., graphs, charts, illustrations, diagrams and figures) in their applications to help the examiner and the reader to better understand their proposed invention. We initially considered including drawings in our dataset, in combination with all textual data, as we thought that having such a multimodal dataset would enable us to introduce new, domain-specific vision-and-language tasks into the field. However, we later came to learn that the Manual of Patent Examining Procedure states that a drawing in a patent application is required only if it is totally necessary for the understanding of the proposed invention (608.02 Drawing [R-10.2019]) (italicization was added by the authors). After making some preliminary estimations, we concluded that a multimodal patent dataset that contains both textual and visual components of the patent applications between 2004 and 2018 would be over 12 TB in size; therefore, it would be very difficult not only to host and store this amount of data and have it be easy to download and by users. In light of all these considerations, we thought that we should leave the incorporation of the visual material of patent applications for future work, and focused on the textual and bibliographic material of patent applications for the time being.
A.5 Considerations for Using the Data
The dataset was created to build new and useful benchmarks for NLP and IP experiments, facilitate research on patent analysis, and eventually help small entities and businesses improve the quality of their patent applications with no cost.
Social Impact of the Dataset. We hope that our dataset will have a positive social impact on the ML/NLP and Econ/IP communities. We discuss these considerations in more detail in Section G.4.
Impact on Underserved Communities and Discussion of Biases. The dataset contains patent applications in English, a language with heavy attention from the NLP community. However, innovation is spread across many languages, cultures, and communities that are not reflected in this dataset. Our dataset is thus not representative of all kinds of innovation. Furthermore, patent applications require a fixed cost to draft and file and are not accessible to everyone. One goal of our dataset is to spur research that reduces the cost of drafting applications, potentially allowing for more people to seek intellectual property protection for their innovations.
Limitations. Please see the "Limitations" subsection of Section 4.
B Discussion of Potential Biases Embedded in the Dataset
In this section, we provide a further examination of our patent dataset for potential biases. For most of our analyses, we focus on the patent applications that were filed to the USPTO between 2011 and 2016 and investigate the correlation between inventor sex, entity size, and geographic location of patent inventors and patent outcomes. Our findings are consistent with the previous work [28-32], showing, among other things, that female inventors are notably underrepresented in the U.S. patenting system, that small and micro entities (e.g., independent inventors, small companies, non-profit organizations) are less likely to have positive outcomes in patent obtaining than large entities (e.g., companies with more than 500 employees), and that patent filing and acceptance rates are not uniformly distributed across the US. Our empirical findings suggest that any study focusing on the acceptance prediction task, especially if it is using the inventor information or the small-entity indicator as part of the input, should be aware of the the potential biases present in the dataset and interpret their results carefully in light of those biases.[27]
B.1 Differences in Patent Filing: Inventor Sex and Gender Identity
The USPTO collects and provides a limited set of data fields linked to inventors and examiners—in particular, it specifies each inventor's name, along with the city, state, country of their residence, and primary examiner's name for each patent application. As noted by Motomura [38], research efforts that examine the link between demographic information (such as race, gender identity, sex, ethnicity, and socioeconomic class of inventors) and patent acceptance prospects need to match the USPTO's publicly available records with other sources. We cannot measure inventor sex or gender identity directly. However, we can estimate inventor sex from inventor names by using imputation procedures. Following the methodology of Jensen et al. [28], we used the data from the United States Social Security Administration (SSA), together with HUPD, to estimate the distribution of female and male inventors who filed patent applications to the USPTO.
The Social Security Administration data contains names obtained from Social Security card applications for births that occurred in the United States between 1880 and 2020 (inclusive) [39]. Each year contains an ordered list of tuples—where each tuple includes a name, assigned sex, and total number of Social Security card applications associated with that name and sex in that given year—all ranked by their popularity. Using this data, we could empirically estimate the assigned sex distributions for 100,364 unique names. For instance, under this model, the name "Julia" is a female name with  $99.6\%$  probability, whereas the name "Taylor" is a female name with  $74.4\%$  probability.
We took this basic statistical model to predict the assigned sex (male or female) of the inventors for each patent application filed between 2011 and 2016. To be consistent with the framework of Jensen et al. [28], we set a strict threshold and assumed that an inventor is female if this model predicts that the first name of the inventor is a female name with at least  $95\%$  probability. Similarly, we assumed that an inventor is male if the model predicted the male class with at least  $95\%$  probability.[28] Additionally, we labeled a name as unisex if it appeared in the SSA's collection of names but had a probability below the threshold value under the female and male categories, and as foreign-sounding[29] if it did not appear in the SSA's collection of names at all.[30] Though we had 2,149,443 patent applications, in total, for 2011-2016, we excluded 781,601 of these patent applications from our analyses, since they contained inventors all of whose names were labeled as foreign-sounding under the statistical model. At the end, the final sample pool included 1,367,842 patent applications that contained at least one identifiable name under the statistical model.
27These multifaceted biases are also likely to be present in models that are not making explicit usage of applicant, inventor, or other sensitive information due to correlated features, for instance.
28We acknowledge this labeling method is incomplete and makes many simplifying assumptions. See Jensen et al. [28] for further analysis.
29 According to Masur and Ouellette [40], “[s]ince 2008, foreign inventors have filed for more U.S. patents each year than American inventors have.” (p. 12)
<sup>30</sup>It is important to note that of all the 6,020,751 inventors who filed patent applications to the USPTO between 2011 and 2016, almost half of them had foreign-sounding names that could not be labeled as male, female, or unisex under this SSA-based model; most of these inventors were, as expected, inventors from foreign countries.
We found  $87.5\%$  of these patent applicants had at least one male inventor, whereas  $17.2\%$  of the same set of patent applications had at least one female inventor.[31] The numbers were similar for patent obtaining as well: Of the 554,380 granted patent applications containing at least one identifiable name under the statistical model,  $88.0\%$  of them had at least one male inventor, whereas  $15.8\%$  of them had at least one female inventor. These findings are consistent with the empirical findings of Jensen et al. [28] and reflect a sex disparity in both patent filing and patent obtaining in the United States. These sex differences in patent filings and obtaining indicate that female inventors are underrepresented in patenting, and may also have less success getting applications granted.
One family of analyses made possible by our dataset is a more systematic analysis of the disparities in patent acceptance probabilities by sex. For example, researchers could show that among patent applications with very similar distributions of text claims, sex predicts acceptance. Causal inference studies of this nature might provide more concrete tests for examiner bias in granting patent applications. But at the same time, the presence of sex disparities in the dataset may propagate biases, for example if inventor characteristics are included in the patent acceptance prediction task. We ask users of our dataset to consider these disparities carefully as they conduct their research.
B.2 Differences in Patent Filing: Entity Size
The USPTO offers reduced application and maintenance fees for patent applicants who are qualified for the small or micro entity status, removing financial barriers to innovation for independent inventors and small businesses with limited resources. In our dataset, we include a small-entity indicator for each patent application—denoting whether the applicant is considered to be a small entity by the USPTO. Using HUPD, we explored how much the patent acceptance rates vary across different business sizes. As before, we restricted our focus to the 2011-2016 year range.
For clarity, we define three different entity statuses under the USPTO's application system. A small entity status is entitled to a patent applicant if the applicant is representing an individual (viz., independent inventor), representing a non-profit organization (e.g., university, 501(c)(3) organization, etc.), or a small business (e.g., a company that meets the standards set forth in 13 CFR 121.801 through 121.805). A micro entity is a small entity that meets additional criteria, such as not having been the inventor of a total of more than four patent applications. While small entities are eligible for a  $50\%$  discount, micro entities are eligible for a  $75\%$  discount. Applicants that pay regular application and maintenance fees (e.g., companies with more than 500 employees) are considered undiscounted (large) entities.
Figure 4 shows the distribution of patent outcomes across three entity categories—namely small, micro, and undiscounted (large)—between 2011 and 2016. Applications filed by small and micro entities constitute only a small fraction of the patent applications. Moreover, small entities have lower acceptance rates than large entities. Finally, large entities seem to not only file more CONT-applications but also have higher CONT-acceptance rates than small entities.
Additionally, we looked at the representation of female inventors in different entity categories using the assigned sex estimation model from the previous section. Of the patent applications that had at least one male, female, or unisex-sounding American name under the statistical model,  $17.8\%$  had at least one female inventor in their inventor list in the case of the small entities. This number was slightly higher  $(21.8\%)$  for the micro entities and slightly lower  $(16.9\%)$  for the large entities.
B.3 Differences in Patent Filing: Geographic Location
The USPTO limits the inventor residence information to the city and state (or foreign country) in patent applications. In our examination, we measured the geographical distribution of US-based inventors at the state-level. The top-left plot in Figure 5 shows the geographical distribution of the state-level residences of the inventors who filed patent applications to the USPTO between 2011 and 2016. According to this plot, almost a quarter of the US-based inventors had a residence in California, the most populous U.S. state, at the time of their submissions. On the other hand, 574 inventors had a residence in Alaska. The top-right plot in Figure 5 shows the geographical distribution of the residences of the US-based inventors with granted patents (for the same year-range). Finally, the plot
<sup>31</sup>Even when we included the unisex category in our analysis and looked at the percentage of the patent applications that contained at least one female or unisex inventor name, the number only increased to  $32.2\%$ .
Figure 4: Distribution of patent decision outcomes across small, micro, and large entities from 2011 to 2016. Large entities have higher acceptance rates than small and micro entities.
in the second row illustrates the average patent acceptance rate of inventors from each state in the U.S. for the 2011-2016 year range. While the success rates vary across different parts of the country, some states with large corporations and concentrated entrepreneurship areas (including California, New York, Texas, and Michigan) appear to have noticeably high patent success rates. In fact, Michigan, with its  $46.1\%$  success rate, has the highest patent acceptance rate amongst all the U.S. states, while Nevada  $(23.1\%)$  has the lowest. These results are consistent with the findings of Bell et al. [30].
Figure 5: Top-left: The geographical distribution of the state-level residences of the US-based inventors who filed patent applications to the USPTO between 2011 and 2016. Top-right: The subset of the inventors from before who had granted patents. Bottom (second row): The patent acceptance rates across different states in the US.
B.4 Differences in Patent Examination: Examiner Sex and Gender Identity
We additionally estimated the distribution of estimated assigned sex of the examiners at the USPTO. We used the same subset of the dataset as before and focused on the examiners of the patent
applications that were filed between 2011 and 2016. As shown on the left plot in Figure 6, of the 10,484 patent examiners in consideration,  $58.8\%$  of them had predicted male names,  $20.6\%$  had predicted female names,  $11.4\%$  had predicted foreign-sounding names, and the rest  $(9.2\%)$  had unisex names, according to the basic statistical model used in Section B.1.
The right plot in Figure 6 illustrates the relationship between patent decision outcomes and estimated sexes of patent examiners. According to our estimation of assigned sex, and among examiners whose names we observe in the SSA data, male patent examiners have higher accepted rates than female patent examiners: The "Accepted"/"Rejected" ratio is close to 2.0 for male patent examiners and 1.4 for female patent examiners.
Figure 6: Relationship between patent decision outcomes and patent examiners' estimated sexes.
C Further Comparisons with Existing Datasets
C.1 Comparison with BIGPATENT