Analyzing the Structure of U.S. Patents
Using Patent Families
Jun Nakamitsu
Graduate School of Science and Engineering
Chuo University
Tokyo, Japana17.8sca@g.chuo-u.ac.jp

Satoshi Fukuda
Faculty of Science and Engineering
Chuo University
Tokyo, Japan
fukuda.satoshi.3238@kc.chuo-u.ac.jp

Abstract—Researchers and developers search for patents infields related to their own research to obtain information on issues
and effective technologies in those fields for use in their research.
However, it is impossible to read through the full text of many
patents, so a method that enables patent information to be grasped
briefly is needed. In this study, we analyze the structure of U.S.
patents with the aim of extracting important information. Using
Japanese patents with structural tags such as “field”, “problem”,“solution”, and “effect”, and corresponding U.S. patents (patentfamilies), we automatically created a dataset of 81,405 U.S. patents
with structural tags. Furthermore, using this dataset, we conduct
an experiment to assign structural tags to each sentence in the U.S.
patents automatically. For the embedding layer, we use a language
representation model, Bidirectional Encoder Representations
from Transformer, pretrained on patent documents and constructa multi-label classifier that classifies a given sentence into one offour categories: “field”, “problem”, “solution”, or “effect”. Usinga loss function that considers the unbalanced amount of data foreach structural tag, we are able to classify sentences related to“field”, “problem”, “solution”, and “effect” with precision of
0.6994, recall of 0.8291, and F-measure of 0.7426.
Keywords—patent, document structure analysis, machinetranslation, machine learning

I. INTRODUCTION
When researchers and company engineers consider new
research or development, utilizing patent information isimportant for grasping the latest technical trends. On the other
hand, it is difficult to read through all the patents publishedaround the world. Under such circumstances, a method thatenables an efficient overview of technical trends is needed. To
overview technical trends, it is effective to classify patents
according to the viewpoints of technologies and problems, etc.
However, to do so, it is necessary to extract the description partof technologies and problems from each patent. Therefore, thisstudy aims to analyze the structure of U.S. patents.
Unlike U.S. patents, Japanese patents have explicit items
such as “Field of Technology” (hereinafter referred to as “field”),“Problem to Be Solved by the Invention” (hereinafter referred
to as “problem”), “Solution for Solving the Problem”(hereinafter referred to as “solution”), and “Effect of theInvention” (hereinafter referred to as “effect”). As a result,researchers and company engineers have to spend more timereading U.S. patents. Therefore, in this study, we perform astructural analysis of U.S. patents and automatically extract
sentences that provide clues for classification.

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

Hidetsugu Nanba
Faculty of Science and Engineering
Chuo University
Tokyo, Japannanba@kc.chuo-u.ac.jp

To achieve this, we analyze the structure of U.S. patents byusing patent families. We first find from U.S. patents thebilingual sentences described in the “field”, “problem”,“solution”, and “effect” sections in Japanese patents. Then, weconstruct a U.S. patent dataset with a clear structure by assigning
the same structural tags as those of the Japanese patent to thefound sentences. Finally, by applying machine learning using
the created dataset, we construct a system that can automaticallyextract sentences related to “field”, “problem”, “solution”, and“effect”, even for U.S. patents that do not have patent families.
By extracting key sentences about technology trends from
U.S. patents, we can obtain clues for clustering each patent. This
also allows researchers and company engineers to understandthe current technology of their competitors and to conduct
efficient research and development that meets global demand.
II. RELATED WORK
A. Structural Analysis of Technical Documents
Although it is important to analyze technical documents suchas patents and research papers, it is difficult to analyze the entire
text. In such cases, it is useful to clarify initially the structure of
the document to narrow down the target sentences before
conducting the analysis. In this section, we introduce a study thatused machine learning to analyze the structure of technicaldocuments.
Prabhakaran et al. [1] analyzed the structure of abstracts withthe aim of predicting the growth and decline of scientific topics.
They constructed a classifier that applies seven different labels
to sentences (“background”, “objective”, “data”, “design”,“method”, “result,” and “conclusion”) using manually labeled
abstracts, in which the authors assigned labels to sentences, as
training data. They applied Conditional Random Field to thesedata and parsed approximately 2.4 million abstracts toinvestigate the relationship between labels and topics. The
results showed that the technologies discussed in “conclusion”
sentences tended to decline, while the technologies discussed in“method” sentences were in the early stage of growth.
Li et al. [2] assumed that evidence plays an important role inbiomedical research and extracted evidential descriptions of the
figures and tables from biomedical articles. They constructed amodel consisting of embedding, attention, and tagging layers.
For embedding, they used BioGloVe [3], BioBERT [4], andSciBERT [5], which were pretrained on biomedical texts, andRecurrent Neural Network (RNN) and Long Short Term
Memory (LSTM) for attention. Experiments were conducted on

two datasets, PubMed-RCT [6] and SciDT [7], and the models
using SciBERT and LSTM performed the best.
Given this background, the purpose of this study is toanalyze the structure of U.S. patents with the aim of classifying
patents. To achieve this, we analyze the structure of U.S. patents
by assigning four types of structural tags, “field”, “problem”,“solution”, and “effect”, to each sentence in a U.S. patent. We
then construct training data with the Japanese–U.S. patentfamily and use Bidirectional Encoder Representations fromTransformer (BERT) [8] for embedding.
B. Extracting Information in Units Shorter than a SentenceTechniques for extracting important information from units
shorter than a sentence are said to be useful for document
retrieval and trend analysis. In the patent classification, which isthe purpose of this study, it is necessary to extract the names ofclassification axes from important sentences in patents, so thisresearch is very relevant for the future.
Gupta et al. [9] proposed a method for automaticallyextracting words belonging to three categories, “focus”,“technique”, and “domain”, from paper abstracts by patternmatching. In this method, for example, the object appearing
immediately after the verb “propose” is considered to belong to
the “focus” category.
Heffernan et al. [10] created a classifier to determine
whether a phrase in a paper indicates a problem or a solution to
improve the effectiveness of technical document retrieval andcompare similar papers. They first created a list of phrases
similar to “problem” and “solution” and collected sentences
containing these phrases from papers. The collected sentenceswere then parsed, and phrases that could be said to indicate aproblem or solution were selected as positive samples. The
phrases that had the same sentence structure as the positive
sample but did not indicate a problem or solution wereconsidered as the negative sample. Using the collected samples
as training data, three types of classifiers were trained: Naive
Bayes, Logistic Regression, and SVM. The experimental results
showed that the SVM-based classifier had the highest accuracy.
Weston et al. [11] constructed a system to analyze sentences
in materials science articles with the aim of assisting in obtaining
information about materials science. First, they manuallyassigned seven types of tags (“material”, “phase”, “sample
descriptor”, “property”, “application”, “synthesis method”, and“characterization”) to each word in the paper’s abstract. Then,they used those data as training data for information extractionby LSTM.
In this study, sentences with structural tags are extracted
from patents. In the future, it will be necessary to determine theaxis of technology analysis using information extraction
methods based on units shorter than sentences, such as thosepresented in this section.
C. Translation of Patent Documents
In this study, we create a dataset using Japanese and U.S.
patent families. In the process of creating the dataset, translationfrom Japanese to English is required. In this section, we
introduce efforts required for the translation of patent documents.
Identify applicable funding agency here. If none, delete this text box.

One of the tasks in NTCIR-10 [12] is the Patent MachineTranslation Task. This task provides a large test collection
containing training, development, and test data for
Chinese/English and Japanese/English patent machinetranslation. The collection contains a bilingual Japanese–
English patent translation corpus of about 3.2 million pairs. We
constructed a Japanese–English machine translation system
based on state-of-the-art Transformer architecture [13].
III. ANALYZING THE STRUCTURE OF U.S. PATENTS
A. Analyzing the Structure of U.S. Patents Using Patent
Families
As described in Section Ⅰ, U.S. patents do not explicitly
include items describing “field”, “problem”, “solution”, and“effect”. Therefore, we propose a method to construct a
structural analysis system of U.S. patents by creating a datasetwith structural tags using patent families and performing
machine learning using the dataset as training data.
Generally, patent rights are granted independently in eachcountry. To obtain patent rights in each country, the applicant
needs to apply for patents for the same invention in several
countries. Such a group of patent documents with the samecontent is called a patent family. Although the language andstructure of patents in the patent family differ, the texts thatcompose the application documents closely correspond to eachother. In this study, we analyze the structure of U.S. patents byassigning the same structural tags to sentences in U.S. patents
that share the same meaning as sentences in Japanese patents towhich the structural tags were manually assigned. Then, using
the structured U.S. patents as training data, we conduct machinelearning to construct a structural analysis system for U.S. patents.
The procedure for the structural analysis of U.S. patents isshown below.
1) Translate each sentence with a structure tag in the Japanese
patent into English.
2) Represent all the sentences in the U.S. patent and thetranslated sentences in 1) as vectors. The sentence in the U.S.
patent that has the highest cosine similarity with thetranslated sentence in 1) is assigned the same structure tag
as that of the Japanese patent.
3) Using the data in 2) as training data, perform machinelearning to extract automatically sentences related to “field”,“problem”, “solution”, and “effect” from U.S. patents.
Steps 1) and 2) are described in detail in Section Ⅲ.B, andstep 3) is described in Section Ⅲ.C.
B. Dataset Creation
First, a machine translator built using the NTCIR-10 Patent
Machine Translation Test Collection and the sequence modeling
tool FAIRSEQ [14] was used to translate sentences withstructural tags in Japanese patents into English. FAIRSEQ is atool that can be used to train text generation models such asmachine translation. The translator built in this experiment uses
Transformer and achieved a BLEU score of 44.11.
Next, we extracted sentences from the U.S. patent that sharethe same meaning as the translated sentences. First, thesentences in the translation result and the full text of the U.S.
patent are vectorized using PatentSBERTa [15], and then thecosine similarity between the sentences in the translation result
and the full text of the U.S. patent, which is a patent family of
the Japanese patent, is calculated. The sentence in the U.S.
patent with the highest score is considered to have the same
meaning as that in the Japanese patent, and is therefore assigned
the same structure tag as the Japanese patent. Using this method,a dataset of U.S. patents with structural tags was created. Since
the patent families were faithfully translated, the sentence withthe highest score was expected to be almost a bilingual sentence.
Note that a sentence may be assigned more than one structure
tag because it may have the highest score for more than one
resulting translation. In addition, not all structure tags arenecessarily present in Japanese patents. For example, someJapanese patents do not include “effect”.
Steps 1) and 2) were used to assign structure tags to 81,405
U.S. patents that had Japanese patents as their patent families.
Of the total 22,016,132 sentences, 1,366,165 sentences wereassigned at least one of the four types of structure tags. In thisexperiment, we did not use sentences that had not been assigned
any structure tags; we only classified sentences that had been
assigned one or more structure tags. If the classifier is used forthe full text of patents, there is a high possibility that structuretags will be assigned to ordinary sentences as well, but
considering that patent classification will be performed in the
future, we believe that the presence of some noise is not asignificant problem.
C. Analyzing the Structure of U.S. Patents by Machine
Learning
Machine learning is performed using the dataset created in
the previous section as training data. We used BERT to assign
structural tags to sentences. BERT is a Transformer-based
pretrained model that can be applied to any task in natural
language processing. By fine-tuning BERT to the structural tag
classification task, we built multi-label classifiers that
automatically classified a given sentence into one of four
categories: “field”, “problem”, “solution”, or “effect”.
We pretrained the BERT model using 3.5 million sentencesin the detailed description sections and claims of U.S. patents.
We then constructed a classifier using the patent-specific BERT
based on the pretrained BERT model. Due to the unbalanced
number of data for each structural tag in the dataset, we also
tested undersampling and weighted loss function methods.
IV. EXPERIMENT
To confirm the validity of the structural analysis methods for
U.S. patents proposed in Section Ⅲ, we conducted experiments
under various conditions.
A. Experimental Setup
Experimental Data
Of the 1,366,165 sentences that were automatically assigned
structural tags according to Section Ⅲ.B, 60% were used fortraining, 20% for validation, and 20% for testing. Table Ⅰ showsa breakdown of the structural tags assigned.

TABLE I.

BREAKDOWN OF STRUCTURE-TAGGED SENTENCES
Number of Sentences
Training

Validation

Field

56,486

18,969Testing

18,727

Total

94,182

Problem

243,606

81,32081,336406,262

Solution

464,662

155,017

154,825774,504

Effect106,648

35,32435,706

177,678

MethodWe examined the following three methods: patent-specific
BERT, undersampling, and weighted loss function. Furthermore,
to confirm the effectiveness of our methods, we compared themwith a classifier using BERT-base-uncased, which is a standard
BERT model for English texts. For all classifiers, the sigmoid
function and binary cross-entropy loss were used to calculate theloss. The training parameters were: maximum number of tokens= 128, batch size = 256, and number of epochs = 10. The detailsof each method are described below.
l Patent-specific BERT (our method): BERT was pretrainedusing 3.5 million sentences from U.S. patents. The classifiers
were trained using the dataset in Table Ⅰ. The learning rate
was 1e-6.
l Undersampling (our method): Due to the disproportionate
number of data for each structural tag, we undersampled thetraining data. We matched the number of sentences in eachstructural tag to the number of training sentences in the “field”
with the lowest number of sentences (56,486). The validationand testing numbers are shown in Table Ⅰ. Patent-specific
BERT was used for embedding. The learning rate was 1e-10.
l Weighted loss function (our method): The dataset had an
unbalanced number of positive and negative examples foreach structural tag. Therefore, we weighted the losses whencalculating them. We increased the weight of positive
examples by multiplying the loss by the ratio of negative topositive examples. Patent-specific BERT was used forembedding. The classifiers were trained using the dataset in
Table Ⅰ. The learning rate was 1e-6.
l BERT-base-uncased (baseline method): As a baseline
method, we used BERT-base-uncased instead of patentspecific BERT.
B. Results
We evaluated the classification of each structural tag. As
shown in Table Ⅱ, the results of the experiment showed that oneof our methods, “patent-specific BERT”, obtained an Fmeasure of 0.7426, which outperformed the others.
C. Discussion
In this experiment, we estimated bilingual sentences basedon the assumption that sentences in a patent family aretranslated in a one-to-one relationship. However, there are cases
in which a single sentence in a Japanese patent is divided into
two sentences in a U.S. patent. Conversely, two sentences in a
Japanese patent can be combined into a single sentence in a U.S.
TABLE II.

CLASSIFICATION RESULTS

Precision

Patentspecific
BERT

Undersampling

Weighted
loss
function

BERTbaseuncased(baseline)

RecallF-measure

Field

0.94570.84940.8920Problem

0.86540.75480.8052

Solution

0.82970.88770.8573

Effect0.67570.3056

0.4158Average0.82910.69940.7426

Field

0.08380.90840.1528

Problem

0.2976

0.99620.4575

Solution

0.56030.7666

0.6467

Effect0.13080.99980.2307

Average

0.26810.91780.3719Field

0.6506

0.93780.7636

Problem

0.76440.83410.7967

Solution

0.85170.83590.8433

Effect0.33870.72760.4601

Average

0.65140.83390.7159Field

0.95250.84340.8915

Problem

0.84640.77140.8061

Solution

0.83460.87580.8543

Effect0.70450.2766

0.3921

Average

0.83450.69180.7360patent. Although the method used in this experiment produced
highly accurate translations, we believe that some issues need
to be addressed in the subsequent discovery of bilingual
sentences. In the future, it will be necessary to consider the
possibility of determining multiple bilingual sentences
depending on the similarity score.
The undersampling method did not provide sufficienttraining because the number of sentences used for training was
greatly reduced. This resulted in the assignment of structural
tags to the majority of sentences, which led to a high recall but
extremely low precision.
In this experiment, sentences without any structural tags
were not used as training data. The reason for this is that
untagged sentences are far more numerous than tagged
sentences. We believe that eliminating untagged sentences
would increase the likelihood that structural tags would be
assigned to sentences that should not be assigned structural tags.
It is therefore necessary to investigate the accuracy ofclassification for sentences that should not be assigned
structural tags in the future.
V. CONCLUSION
In this study, we created a dataset containing 81,405 U.S.
patents with structural tags. Using this dataset, we conducted anexperiment to classify automatically sentences from the U.S.

patents that are important for classifying each patent onto atechnical analysis axis. The experimental results showed that
one of our methods, patent-specific BERT, obtained an Fmeasure of 0.7426, which outperformed the others.
REFERENCES[1]

Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and
Dan Jurafsky, “Predicting the rise and fall of scientific topics from trends
in their rhetorical framing,” In Proceedings of the 54th Annual Meetingof the Association for Computational Linguistics, pp.1170–1180, 2016.
[2] Xiangci Li, Gully Burns, and Nanyun Peng, “Scientific discourse tagging
for evidence extraction,” In Proceedings of the 16th Conference of theEuropean Chapter of the Association for Computational Linguistics,
pp.2550–2562, 2021.
[3] Gully A Burns, Xiangci Li, and Nanyun Peng, “Building deep learningmodels for evidence classification from the open access biomedical
literature,” Database, 2019.
[4] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu
Kim, Chan Ho So, and Jaewoo Kang, “BioBERT: a pre-trained
biomedical language representation model for biomedical text mining,”
Bioinformatics, pp.1234–1240, 2020.
[5] Iz Beltagy, Arman Cohan, and Kyle Lo, “SciBERT: pretrained
contextualized embeddings for scientific text,” In Proceedings of the 2019
Conference on Empirical Methods in Natural Language Processing andthe 9th International Joint Conference on Natural Language Processing,
pp.3615–3620, 2019.
[6] Franck Dernoncourt and Ji Young Lee, “Pubmed 200k RCT: a dataset for
sequential sentence classification in medical abstracts,” In Proceedings of
the 8th International Joint Conference on Natural Language Processing,
pp.308–313, 2017.
[7] Gully APC Burns, Pradeep Dasigi, Anita de Waard, and Eduard H Hovy,“Automated detection of discourse segment and experimental types fromthe text of cancer pathway results sections,” Database, 2016.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,“BERT: pre-training of deep bidirectional transformers for language
understanding,” arXiv preprint arXiv:1810.04805, 2017.
[9] Sonal Gupta and Christopher D. Manning, “Analyzing the dynamics ofresearch by extracting key aspects of scientific papers,” In Proceedings of
the 5th International Joint Conference on Natural Language Processing,
pp.1–9, 2011.
[10] Kevin Heffernan and Simone Teufel, “Identifying problems and solutionsin scientific text,” Scientometrics, pp.1367–1382, 2018.
[11] Leigh Weston, Vahe Tshitoyan, John M. Dagdelen, Olga Kononova,
Kristin Aslaug Persson, Gerbrand Ceder, and Anubhav Jain, “Named
entity recognition and normalization applied to large-scale information
extraction from the materials science literature,” Chemical Informationand Modeling, pp.3692–3702, 2019.
[12] Isao Goto, Ka Po Chow, Bin Lu, Eiichiro Sumita, and Benjamin K. Tsou,“Overview of the patent machine translation task at the NTCIR-10
workshop,” In Proceedings of the 10th NTCIR Conference, pp.260–286,
2013.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion
Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention
is all you need,” In Proceedings of Advances in Neural Information
Processing Systems, pp.6000–6010, 2017.
[14] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross,Nathan Ng, David Grangier, and Michael Auli, “FAIRSEQ: a fast,
extensible toolkit for sequence modeling,” In Proceedings of North
American Association for Computational Linguistics (NAACL): System
Demonstrations, pp.48–53, 2019.
[15] Hamid Bekamiri, Daniel S. Hain, and Roman Jurowetzki,“PatentSBERTa: a deep NLP based hybrid model for patent distance andclassification
using
augmented
SBERT,”
arXiv
preprint
arXiv:2103.11933, 2021.