Hunt for Buried Treasures: Extracting Unclaimed Embodiments from Patent Specifications
Year: 2023

Abstract:
Patent applicants write patent specifications that describe embodiments of inventions. Some embodiments are claimed for a patent, while others may be unclaimed due to strategic considerations. Unclaimed embodiments may be extracted by applicants later and claimed in continuing applications to gain advantages over competitors. Despite being essential for corporate intellectual property (IP) strategies, unclaimed embodiment extraction is conducted manually, and little research has been conducted on its automation. This paper presents a novel task of unclaimed embodiment extraction (UEE) and a novel dataset for the task. Our experiments with Transformer-based models demonstrated that the task was challenging as it required conducting natural language inference on patent specifications, which consisted of technical, long, syntactically and semantically involved sentences. We release the dataset and code to foster this new area of research.1

heading: 1 Introduction
text: Patents provide inventors the right to exclude others from using their inventions in exchange for disclosing how to make and use inventions by writing patent specifications. Patents have thus incentivized innovation and benefited industries. Given the increasing number of patent applications even during the COVID-19 pandemic (WIPO, 2022b), it is important to streamline patent application processes with technologies.
A patent specification describes an invention (Figure 1) by specifying one or more ways of embodying the invention, so that people skilled in the art can make and use it. A patent specification also contains claims that specify which embodiment applicants want to patent by stating the technical features necessary for the embodiment. Here, an invention refers to a mental construct inside the
1https://github.com/rakutentech/UEE_ ACL23
mind of the inventor, while the embodiment of the invention is a physical form of the invention and claims protect the embodiments (WIPO, 2022a).
A patent specification may describe a variety of embodiments, some of which may be unclaimed because claiming too diverse embodiments in a patent application may violate the unity of invention, a requirement for a patent application to relate to one invention only or to a group of inventions so linked as to form a single general inventive concept (USPTO, 2020b). Continuing application could be utilized later to claim those unclaimed embodiments in the prior patent application (the parent application). A continuing application can claim any embodiments if they are written in its parent’s description. Moreover, the filing date of continuing application is the same as its parent’s, even if it is filed years after the parent. Applicants can therefore utilize continuing applications strategically by, for instance, writing as many diverse embodiments as possible in the parent application and filing a continuing application to claim unclaimed embodiments in the parent. If the continuing application
25
does not exhaust its parent’s embodiments, applicants may have further continuing applications. In so doing, applicants can adapt the claims of continuing application to new products and services of their company, and even new products and services of their competitors, enhancing their industrial competitiveness.
Continuing application requires extracting unclaimed embodiments from a patent specification. This is tedious as it requires understanding a wide variety of embodiments that are strategically arranged in the patent specification, a legal, technical document that may consist of thousands of tokens (Tab 1). Unclaimed Embodiment Extraction (UEE) has nonetheless been conducted manually without any technological support, and little research has been conducted on UEE.
This paper introduces the novel task of UEE (Figure 1) and the first publicly available dataset for UEE. Besides its practical utility, UEE poses a new NLP challenge as it involves two decisions to make (§2), one of which, i.e. decision (ii), requires matching embodiment text in the description with claims to see if the embodiment has been claimed. Decision (ii) can be seen as a real-world natural language inference (NLI) (Bowman et al., 2015), where the hypothesis is a description paragraph and the premise is a set of claims. Although there have been studies on NLI for real-world applications (Holzenberger et al., 2020; Koreeda and Manning, 2021), decision (ii) involves a novel real-world NLI due to the following challenge: The hypothesis and the premise may consist of multiple long sentences which are written in patentese and full of technical terms in the target domain and whose syntactic and semantic structures are hard to recognize for non-IP specialists (Ferraro et al., 2014).
Although our UEE dataset has been created based on Japanese patents, extracting unclaimed embodiments from patent specifications is conducted in other countries such as the U.S. This paper gives examples in English for ease of explanation. See Appendices for Japanese examples.
Our contributions are as follows:
1. We introduce UEE, a novel, real-world NLP challenge.
2. We create and release the first dataset for UEE.
3. We conducted UEE experiments to demonstrate its difficulty.
4. We release code for reproducibility.

heading: 2 Task
text: Given a patent specification that comprises a set of claims and a set of description paragraphs about an invention, we want to determine whether each paragraph in the description describes any embodiment of the invention that has not been claimed in the claims (Figure 1). This involves two decisions:
(i) Does a given paragraph describe an embodiment of the invention?
(ii) Has the embodiment in a paragraph, if any, been claimed in the claims already?
We thus need both a paragraph and a set of claims to determine whether the paragraph contains an unclaimed embodiment.
Formally, the task is defined as follows (Figure 2). Suppose we have I patent specifications and the i-th patent specification has Ji description paragraphs. Let pi,ji be the ji-th description paragraph in the i-th specification, Ci be a set of claims in the i-th specification, and yi,ji be a binary label where yi,ji = 1 if pi,ji describes any embodiment that is not claimed in Ci and yi,ji = 0 otherwise; here, i = {1, ..., I} and ji = {1, ..., Ji}. Given N = ∑ i Ji training instances, our goal is to learn a function f(pi,ji , Ci) → yi,ji . The task involves the two decisions (i) and (ii) and a UEE model may make the two decisions separately. Our UEE baseline models in §4.1, nonetheless, make the two decisions in a single step, as it is more straightforward. We will explore different architectures for better utilization of the nature of the task (involving the two decisions) in the future.
In this study, we chose a paragraph as the unit of embodiment description, because in the patent applications, paragraphs are encouraged to be numbered to serve as the unit of work and indeed form a
meaningful unit. Other options would be a phrase, clause, or sentence. We will identify the best unit of embodiment description in the future.

heading: 3 Dataset

heading: 3.1 Data Source
text: We acquired source patent data from the Japan Patent Office (JPO) via their web form.2 The data from JPO contained Japanese patent specifications from 1993 to 2022. We obtained both parent and continuing patent specifications from this data. We created the dataset from patent specifications that had their corresponding continuing patents.

heading: 3.2 Annotation Method
text: As the task involves two decisions, (i) and (ii) in §2, our annotation method is based on the two as illustrated in Figure 3. Specifically, we label a paragraph as negative if it has no embodiment (See paragraph 0008 in Figure 3), or if the embodiment is claimed in the parent patent to which the paragraph belongs (0004 in Figure 3).
For positive annotation, we used the continuing patent generated from the patent to which the target paragraph belonged. If a paragraph describes an embodiment that is claimed in the continuing patent but not in the parent, the paragraph is labeled as positive (0034 in Figure 3). Although we can identify unclaimed embodiments from the parent patent, without relying on the continuing patent, it helps us double-check positive paragraphs.
To use continuing patents, we collected patent specifications with corresponding continuing patents from the JPO data and made pairs of parent and continuing patents as in Figure 3.
2https://www.jpo.go.jp/system/laws/ sesaku/data/download.html (Japanese)
We restricted target patents to those with International Patent Classification (IPC) codes3 that met our business needs; specifically, we mainly chose IPC codes for digital data processing (G06F), information and communication technology (G06Q), and aeroplanes (B64C). IPC is used in over 100 countries to indicate the subject of the invention.4
We conducted manual annotation on pairs of parent and continuing patents collected in this way.
On top of positive and negative labels, we leave notes that clarify reasons for annotators’ labeling decisions (e.g., CLAIM1 and NONE in Figure 3). This is because labeling decisions would be based on patent practitioners’ expertise, which may be incomprehensible to researchers, and we expect the notes to improve annotated labels’ interpretability. A negative paragraph is given the claim ID of the parent patent as the note if the paragraph’s embodiment is claimed in the parent’s claims. A negative paragraph is given the note NONE if it has no embodiment. A positive paragraph is given the claim ID of the continuing patent if its embodiment is claimed in the continuing patent’s claims.
We use the notes for experiments of decision (i) (§4.2) and decision (ii) (§4.3), too.

heading: 3.3 Annotators
text: Two experienced patent practitioners who were native speakers of Japanese were employed as annotators. We split the 11,951 instances (each consisting of a description paragraph and a set of claims) into two separate sets. Each annotator was assigned to only one set; no instance was annotated by both of them due to our budget constraints.
We nonetheless measured inter-annotator agree-
3https://ipcpub.wipo.int/ 4https://en.wikipedia.org/wiki/
International_Patent_Classification
ment by asking another experienced patent practitioner (a native speaker of Japanese) to annotate 309 instances that the above two annotators worked on after removing their labels and notes. As a result, Cohen’s kappa (Cohen, 1960) was 0.465, indicating moderate agreement (Landis and Koch, 1977). According to the kappa score, experts may disagree occasionally. The question is then how much experts’ disagreement affects the performance, as pointed out by an anonymous reviewer. We will explore this question in the future.

heading: 3.4 The Dataset
text: The resulting UEE dataset has 11,951 instances. We use 60% for training, 20% for development, and 20% for testing. Table 1 shows the statistics of the dataset. We used the tokenizer of our RoBERTa in §4.1 to count tokens. In "Label distribution" in Table 1, "POSITIVE:CLAIM"5 refers to a positive instance. The instance with paragraph 0034 in Figure 3 has this label. "NEGATIVE:CLAIM" means a negative instance with an embodiment that has already been claimed in the parent patent and hence with a note of the corresponding claim ID. "NEGATIVE:NONE" is a negative instance without embodiment. The instances with paragraphs 0004 and 0008 in Figure 3 are examples of these two types of negatives, respectively.
In Appendix A, we show an example data instance of the UEE dataset in Japanese and English
5We may omit claim IDs of the notes, e.g. "1", hereafter.
for illustration purposes.

heading: 4 Experiments
text: We conducted experiments of UEE with baseline models based on Transformer (Vaswani et al., 2017) to see the difficulty of the task (§4.1). We also conducted experiments of making the decisions (i) and (ii) in §2 as independent tasks for a better understanding of the task (§4.2 and §4.3).
These experimental results show the following:
(A) Baseline Transformer-based models deliver mediocre performances (§4.1).
(B) Despite patent specifications’ being long, UEE models do not necessarily have to deal with long documents (§4.1).
(C) The bottleneck in UEE is decision (ii) (§4.3).

heading: 4.1 UEE Baselines
text: We evaluated RoBERTa (Liu et al., 2019) and Longformer (Beltagy et al., 2020) for the UEE task, because these are ones of standard Transformer-based models, and Longformer is known to be able to deal with long documents such as patent specifications. However, we do not claim these are optimal models for the task; we will explore better models in the future. Our RoBERTa was built from a base-sized one which we call Rinna RoBERTa6 and had been pre-trained on Japanese CC-100 (Conneau et al.,
6https://huggingface.co/rinna/ japanese-roberta-base
2020) and Japanese Wikipedia. The maximum sequence length was 512. We fine-tuned Rinna RoBERTa on the UEE training set for ten epochs with the training batch size, the warm-up steps, and the learning rate being set to 128, 100, and 5e-5, respectively. We used AdamW (Loshchilov and Hutter, 2019) for optimization. We will describe the hyper-parameter settings of models we experimented with in Appendix B, hereafter.
Our Longformer was converted from Rinna RoBERTa following Beltagy et al. (2020).7 See Appendix B.2 for its hyper-parameter setting.
The first two rows in Tab 2 labeled with §4.1 show F1 of the two baseline models on the UEE test set. We fine-tuned and evaluated each model five times. The reported figures are the mean and standard deviation obtained from the five runs.
The result indicates that our baselines have nonnegligible room for improvement. Given Transformers’ successes in many tasks (e.g., a basesized RoBERTa fine-tuned and evaluated on JSNLI (Yoshikoshi et al., 2020), a Japanese NLI dataset, delivers the F1 of 0.93 (Yanaka and Mineshima, 2022)), we think that UEE is challenging.
The result also indicates that RoBERTa outperforms Longformer. Actually, we expected the opposite result, because the input to UEE models, i.e. a pair of a description paragraph and a set of claims, tends to be long; the average number of tokens in a description paragraph is 88.73 and that of tokens in a set of claims is 1210.22 (= 106.44 × 11.37), as Table 1 shows.
We suspect that this unexpected result is due to the fact that, in the UEE dataset, more than 70% of embodiments in description paragraphs with NEGATIVE:CLAIM are claimed in the first three claims. Models then do not always have to read through all the claims. This is probably because of the preferred order of claims: Claims should preferably be arranged in order of scope so that the first claim presented is the least restrictive (USPTO, 2020a); i.e. the most general claims should come first.

heading: 4.2 Decision (i)
text: We conducted experiments of making only the decisions (i), i.e. whether a paragraph described any embodiment, to see how difficult it was.
To train and test a model for decision (i), we created training, development, and test sets for de-
7See convert_model_to_long.py in the supplementary material for implementation.
cision (i) from the corresponding set of the UEE dataset as follows.8 We regarded the instances in the UEE dataset whose note is NONE as negative and the rest as positive, because note NONE indicates the corresponding paragraph has no embodiment as described in §3.2. The positive-negative ratio was then about 52:48 as Tab 1 indicates.
We built a model from Rinna RoBERTa (§4.1) again for this experiment. The experimental protocol was the same as our RoBERTa in §4.1. See Appendix B.3 for its hyper-parameter setting.
The third row in Tab 2 labeled with §4.2 shows F1 of the model on the test set. The reported figures are the mean and standard deviation obtained from five runs of fine-tuning and evaluation. This result indicates that decision (i) is a modest task.

heading: 4.3 Decision (ii)
text: We also conducted experiments to make the decision (ii), i.e. whether the embodiment of a paragraph has been claimed, as an independent task.
As discussed in §1, decision (ii) can be seen as an NLI task where the hypothesis is a paragraph and the premise is a set of claims. For training and test of models for decision (ii), in order to focus on its NLI aspect, we ignored UEE dataset instances with NEGATIVE:NONE. This is because it is obvious for a paragraph without any embodiment to be unclaimed, i.e. not entailed by a set of claims. Besides, we used not only parent patents but also their continuing patents in the UEE dataset, as it is straightforward to use them for decision (ii).
Accordingly, we created training, development, and test sets for decision (ii) from the corresponding set of the UEE dataset as follows. We generated positive instances for decision (ii) from the UEE dataset by pairing a paragraph of POSITIVE:CLAIM and a set of claims in the continuing patent; e.g. the pair of paragraph 0034 and the continuing patent’s claims in Figure 3. We also generated decision (ii) positives by pairing a paragraph of NEGATIVE:CLAIM and a set of claims in the parent patent; e.g. paragraph 0004 and the parent patent’s claims in Figure 3.
Likewise, decision (ii) negatives were generated by pairing a paragraph of POSITIVE:CLAIM and a set of claims in the parent patent and also by pairing a paragraph of NEGATIVE:CLAIM and a set of claims in the continuing patent. In Figure 3, pair of
8For dataset creation for (i) and (ii), see: Decision1/ src/dataset.py and Decision2/src/dataset. py in github.com/rakutentech/UEE_ACL23.
paragraph 0034 and the parent patent’s claims and that of paragraph 0004 and the continuing patent’s claims are decision (ii) negatives.
The positive-negative ratio was then 50:50. We fine-tuned Rinna RoBERTa with this training set. We call the resulting model RoBERTauee. The experimental protocol was the same as the previous experiments (§4.1 and §4.2).
Since decision (ii) is an NLI task, we also finetuned Rinna RoBERTa using the JSNLI dataset9 for comparison. We call it RoBERTajsnli. We used the same test set as RoBERTauee for evaluation. RoBERTajsnli’s high performance for Japanese NLI tasks has been shown in the literature (Yanaka and Mineshima, 2022).
Note that while JSNLI is a ternary classification task, i.e. entailment, contradiction, and neutral, decision (ii) is binary, i.e. positive and negative. We, therefore, need to align JSNLI’s labels with our binary labels. We experimented with two label alignment conditions: Condition A was to align entailment with positive and contradiction and neutral with negative. Condition B was the same as Condition A, except that we ignored contradiction; only neutral was aligned with negative. This is reasonable because, even if an embodiment is not claimed in a set of claims, it does not necessarily imply that the two pieces of text are contradictory.
For the hyper-parameter setting and fine-tuning of RoBERTajsnli, refer to Appendix B.5.
The last three rows in Tab 2 labeled with §4.3 show F1 of the models on the test set. Although RoBERTauee was the best among them, it has a
9We used train_w_filtering.tsv of JSNLI 1.1.
large room for improvement. This indicates that decision (ii) is difficult and is the bottleneck for UEE. Looking closely at the data revealed that a tiny fraction of text in a set of claims, which usually is a long document, tended to correspond to an embodiment (Figure 4), because each claim may consist of various technical features from more than one description paragraph. This would make decision (ii) challenging, together with the other factors discussed in §1; i.e. patent specifications consisting of technical, long, syntactically and semantically involved sentences written in patentese.
RoBERTajsnli delivered low performances under both conditions, probably because of the domain discrepancy between the NLI task in JSNLI and UEE. We think this result shows the necessity of a dedicated dataset for UEE.

heading: 4.4 Discussion
text: The baselines delivered mediocre performances for UEE. We observed that decision (ii) makes UEE difficult. Nevertheless, we believe UEE is a worthy challenge, as it would eventually contribute to the industry by streamlining patent application processes. We also believe that, to this end, utilizing the outcomes of the current study would help.

heading: 5 Related Work
text: Patent Document Processing NLP systems for real-world applications in, for instance, ecommerce (Malmasi et al., 2021, 2022), medical (Rumshisky et al., 2020; Naumann et al., 2022), and legal areas (Aletras et al., 2021; Preotiuc-Pietro et al., 2022) has gained attention, probably because it has become more feasible to serve practical needs thanks to the success of Transformer-based models and pre-training methods (Devlin et al., 2019).
Patent document processing has also been studied extensively (Aras et al., 2019; Krestel et al., 2021, 2022). Its most studied areas are machine translation (Tsujii and Yokoyama, 2007; Utsuro et al., 2019; Nakazawa et al., 2021, 2022) and information retrieval (Tait et al., 2008, 2009, 2010; Risch et al., 2020).
There have recently been studies that would directly facilitate patent applications. Sharma et al. (2019) created a dataset for summarizing patents and proposed baselines. Tonguz et al. (2021) proposed a method for claim generation formulated as text summarization. Aslanyan and Wetherbee (2022) created a dataset for phrasal matching for
better patent similarity measurement. Gao et al. (2022) proposed a method for predicting whether a given patent application would be approved.
However, to the best of our knowledge, no study has addressed UEE; we are the first to do that.
Natural Language Inference NLI has showcased the comprehension ability of NLP systems (Wang et al., 2019; Nie et al., 2020; Poliak, 2020) and provided datasets for their training (Conneau et al., 2017; Reimers and Gurevych, 2019). Introducing diverse NLI tasks would then push the boundary of NLP. Recent studies have introduced new NLI tasks targeting real-world applications (Romanov and Shivade, 2018; Holzenberger et al., 2020; Koreeda and Manning, 2021; Sadat and Caragea, 2022). Our decision (ii) is a novel realworld NLI for patents that poses a new challenge.

heading: 6 Conclusion
text: We introduced UEE, a novel NLP challenge, and created a corresponding dataset. Our experiments showed that UEE was challenging due to the difficulty of making the decision (ii). We hope that the research community will address this challenge by utilizing the UEE dataset and code that we created and released.
Future Work We have not explored better architectures for the task extensively. Although RoBERTa performed reasonably well, more capable, human-instruction-aligned architectures have been developed recently (Bahrini et al., 2023; OpenAI, 2023). We will explore the capability of these more recent large language models for the task.

heading: 7 Ethics Statement
text: The scope of this work is to introduce NLP technologies to the continuing patent application process to make it more efficient. The outcomes from our work would therefore have an industrial impact through enabling organizations to file more continuing patents with less time. There would then be a risk that, if our technologies were available to only particular organizations, fair competitions could not be ensured. We therefore decided to release the dataset and code to the public.
This work was intended to be beneficial to patentrelated processes and studies in artificial intelligence, machine learning, and NLP. The outcomes from this work should therefore be used only for these purposes.
The coverage of the UEE dataset in terms of the IPC subclass, language, and countries and regions are limited. Care must be taken when using this dataset, accordingly.
The dataset does not contain any personal information, as it has been created from publicly available patent specifications. We nonetheless took special care to check if any personal information was included in the dataset by accident when creating the dataset.
All the data we used in this work are publicly available. The pre-trained language model that we used, i.e. RoBERTa, is also publicly available. Our Longformer was converted from the RoBERTa with a method that was also known to the public. Besides, since we have released all the necessary code and dataset along with the paper, all the experimental results in the paper are reproducible.
Regarding the hiring of the annotators (the expert patent practitioners), we negotiated with their company in advance to fairly determine the charge, which was the equivalent of the cost of hiring expert patent practitioners for patent search. We explained to the human annotators about the purpose of the data annotation and how it would be used in advance of the annotation.
Regarding the compute in our experiments, we executed 30 fine-tuning processes, which took 57 hours in a single Nvidia A100 GPU in total.

heading: 8 Acknowledgment
text: We are deeply grateful to all those who played a role in the success of this project. We would like to thank Yoshimasa Utsumi, Kazutaka Ohara, Takeshi Imamura, Hsuan-Yu Kuo, Takako Yoshida, Vijay Daultani, Colin Ian Rowat, and Aztec Co., Ltd. for their invaluable input and support throughout the research process. Their insights and expertise were instrumental in shaping the direction of this project.

heading: A Example of the UEE Dataset
text: Figure 5 illustrates an entry of the dataset. Each entry is a JSON object and consists of the application number of the patent from which the target paragraph is extracted (appNum), the identifier of the target paragraph (paraNum), the paragraph text (paraTxt), a set of claims from the same application (claims), the label (label), the note (note), the application number of the continuing patent (contAppNum), and the continuing patent’s claims (contClaims). claims and contClaims consist of individual claims’ identifier (claimNum) and text (claimTxt).10 Here, paraTxt, a set of claimTxts, and label correspond to pi,ji , Ci, and yi,ji in §2, respectively. Figure 6 shows the Japanese version of Figure 5.

heading: B Hyper-parameter Setting
text: B.1 UEE Baseline Rinna RoBERTa Although we described Rinna RoBERTa’s hyperparameter setting for the baseline experiment in §4.1, we repeat it here for the sake of completeness.
The maximum sequence length was 512. We fine-tuned Rinna RoBERTa on the UEE training
10claimTxt is a list of text. This is because a claim is usually long and split into segments for readability. We keep this structure in JSON format.
{ "appNum": "2021xxxxxx", "paraNum": "0052", "paraTxt": "第１データは、第１モデルＭ１の学習に利用可能なデータ であればよく、ウェブページのタイトルに限られない。例えば、第１デ ータはウェブページから作成された要約であってもよい。"
"claims": [ {
"claimNum": "1", "claimTxt": [
"第１データと、当該第１データの分類に関する第１分類情報と、 の関係が学習された第１モデルに基づいて、第２データの分類に 関する第２分類情報を取得する第２分類情報取得部と、", "前記有効性の推定結果に基づいて処理を実行する実行部と、", "を含む処理実行システム。"
] }, {
"claimNum": "2", "claimTxt": [
"前記推定部は、前記第２モデルに基づいて有効性を推定する、", "請求項１に記載の処理実行システム。"
] }
], "label": "positive", "note": [ "1" ], "contAppNum": "2021yyyyyy", "contClaims": [
{ "claimNum": "1", "claimTxt": [
"第１データと、当該第１データの...", ]
}, }
Figure 6: Example data in Japanese
Description Paragraph ＰＯＳデータを受信した と判断した場合、ＣＰＵ １は、当該売上データを、 その注⽂時間帯毎に区分 けする。区分けされる注 ⽂時間帯は、例えば１９ 時台〜２２時台と、２２ 時台〜２４時台であるが、 これに限られない。 Claim 1 少なくとも１つの飲⾷店が提供可能な飲⾷物のメニューに関するメニュー情報を含む飲⾷店情報 と、前記飲⾷店に関する飲⾷店情報提供ページ群を構成し前記メニュー情報のうち少なくとも⼀ 部を掲載したメニュー掲載ページとを記憶する記憶⼿段と、前記飲⾷店に存在するＰＯＳシステ ムから、当該飲⾷店における所定期間毎のメニュー毎の出数、売上⾦額、利益率のうち少なくと も１つを含むＰＯＳデータを受信する通信⼿段と、前記受信されたＰＯＳデータに基づいて、前 記メニュー掲載ページ上の前記メニュー情報を更新する制御⼿段とを具備し、前記メニュー掲載 ページは、前記飲⾷店の営業時間内の複数の注⽂時間帯毎にメニュー情報表⽰欄を有し、前記制 御⼿段は、前記受信されたＰＯＳデータを前記注⽂時間帯毎に区分けし、当該注⽂時間帯毎のＰ ＯＳデータに基づいて、前記注⽂時間帯毎のメニュー表⽰欄のメニューを更新する情報処理装置。 Claim 2 請求項１に記載の情報処理装置であって、前記制御⼿段は、前記ＰＯＳデータに基づいて、前 記メニュー掲載ページ上の前記メニュー情報を、前記所定期間において出数、売上⾦額及び利益 率のうち少なくとも１つが上位の所定数のメニュー情報へと更新する情報処理装置。 Claim 3 請求項１に記載の情報処理装置であって、前記制御⼿段は、前記ＰＯＳデータに基づいて、前 記メニュー掲載ページ上の前記メニュー情報を、前記所定期間において出数及び売上のうち少な くとも１つが下位の所定数のメニュー情報へと更新する情報処理装置。 More claims follow. Only a tiny fraction of text in a set of claims tends to correspond to the embodiment in a paragraph.
Figure 7: Claimed embodiment example in Japanese
set for ten epochs with the training batch size, the warm-up steps, and the learning rate being set to 128, 100, and 5e-5, respectively. We used AdamW (Loshchilov and Hutter, 2019) for optimization.
B.2 UEE Baseline Longformer
The maximum sequence length was 4,096. We fine-tuned our Longformer on the UEE training set for ten epochs with the training batch size, the gradient accumulation steps, the warm-up steps, and the learning rate being set to 16, 2, 200, and 2e-5, respectively, based on Beltagy et al. (2020). We used the AdamW optimizer.
B.3 Decision (i) RoBERTa
The hyper-parameter setting for this model is the same as the UEE Baseline Rinna RoBERTa in B.1.
B.4 Decision (ii) RoBERTauee The hyper-parameter setting for this model is the same as the UEE Baseline Rinna RoBERTa in B.1.
B.5 Decision (ii) RoBERTajsnli We fine-tuned RoBERTajsnli for ten epochs with the training batch size, the warm-up steps, and the learning rate being 128, 500, and 3e-5, respectively. We used the AdamW optimizer. Although the instances in the JSNLI dataset have already been tokenized, we re-tokenized them with Rinna RoBERTa’s tokenizer.

heading: C Japanese Example of a Claimed Embodiment
text: Figure 7 shows the Japanese version of Figure 4.

heading: D Full Evaluation Results
text: Table 3 shows accuracy, precision, recall, and F1 of all the evaluated models.

heading: Future Work
text: Future Work We have not explored better architectures for the task extensively. Although RoBERTa performed reasonably well, more capable, human-instruction-aligned architectures have been developed recently (Bahrini et al. , 2023; OpenAI, 2023). We will explore the capability of these more recent large language models for the task