Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph (2024)

Abstract: Model scaling is becoming the default choice for many language tasks due to the success of large language models (LLMs). However, it can fall short in specific scenarios where simple customized methods excel. In this paper, we delve into the patent approval prediction task and unveil that simple domainspecific graph methods outperform enlarging the model, using the intrinsic dependencies within the patent data. Specifically, we first extend the embedding-based state-of-the-art (SOTA) by scaling up its backbone model with various sizes of open-source LLMs, then explore prompt-based methods to harness proprietary LLMs’ potential, but find the best results close to random guessing, underlining the ineffectiveness of model scaling-up. Hence, we propose a novel Fine-grained cLAim depeNdency (FLAN) Graph through meticulous patent data analyses, capturing the inherent dependencies across segments of the patent text. As it is model-agnostic, we apply costeffective graph models to our FLAN Graph to obtain representations for approval prediction. Extensive experiments and detailed analyses prove that incorporating FLAN Graph via various graph models consistently outperforms all LLM baselines significantly. We hope that our observations and analyses in this paper can bring more attention to this challenging task and prompt further research into the limitations of LLMs. Our source code and dataset can be obtained from https://github.com/ ShangDataLab/FLAN-Graph.

heading: 1 Introduction
text: Scaling up language models has demonstrated predictable improvement and unprecedented abilities in many language tasks (Chung et al., 2022; Wei et al., 2022a; Zhao et al., 2023). However, emerging evidence shows that simply scaling up backbone models to large language models (LLMs) may
∗Equal contribution. Listing order is random.
not guarantee success (Peng et al., 2023; Hou et al., 2023; Wang et al., 2023). In addition, scaling up models imposes demanding computational costs that prevent it from being widely adopted for realworld applications. Such limitations necessitate cost-effective methods beyond scaling, especially for domain-specific tasks that have distinct traits.
In this paper, we look into the task of patent approval prediction, a challenging yet straightforward classification task that scaling struggles to address, and explore customized cost-effective solutions. As shown in Figure 1, the objective is to determine if each claim in a patent application will be approved or rejected by the U.S. government patent office (USPTO). It is vital for intellectual property (IP) protection, taking up to 40% of the U.S. GDP and over 30% of employment. Due to the demanding requirements for knowledge in both technology and law, patent examination is conducted manually,
5218
leading to potential inconsistent outcomes across patent examiners (O’Neill, 2018; USPTO, 2016). Such inconsistency underscores the necessity for objective and automated computational support.
The state-of-the-art (SOTA) of this task is based on BERT embedding (Devlin et al., 2019) augmented with handcrafted features (Gao et al., 2022). An intuitive idea is to replace its backbone model with modern LLMs. To this end, we employ LLaMA2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), Vicuna (Chiang et al., 2023) in various sizes (7&13&70B) and apply both LoRA (Hu et al., 2022) and full fine-tuning. Surprisingly, they do not live up to expectations, performing on par or worse than BERT. To exploit LLMs’ emergent abilities, we utilize prompt-based methods tailored to these open-source LLMs, as well as closed-source GPT3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023), but the results are still unsatisfying.
The shattered hope in LLMs motivates us to dive into patent data analyses, which leads us to the standardized writing of claims and the dependencies nature among them. As depicted in Figure 2, Claim 1 compromises multiple sub-components, which are then referenced in subsequent claims. Such intricate inner-claim (between sub-components in Claim 1) and inter-claim dependencies (between Claims 1&2 as well as Claims 1&3) have critical implications for the patent approval prediction task, as the patent examination is conducted on each claim and the rejection of one claim can result in the automatic rejection of its dependents.
Inspired by the observations and domain-specific knowledge acquired from painstakingly extensive data analyses, we propose Fine-grained cLAim depeNdency (FLAN) Graph for patent approval prediction, which represents each claim by a single graph that encapsulates both inner- and inter-claim dependencies. Specifically, as shown in Figure 1, we first design a novel algorithm to automatically construct the FLAN Graph at scale, where each node is an informative segment of the claim text. Then, the model-agnostic FLAN Graph is fed into a generic graph model for prediction. Examples of the FLAN graphs and the corresponding claims are shown in Appendix C. In the experiments, we adopt a variety of cost-effective graph models such as GCN (Chen et al., 2020), GAT (Velickovic et al., 2018), and TreeLSTM (Tai et al., 2015) to verify the effectiveness of our proposed FLAN Graph for patent approval prediction. All models with FLAN Graph applied outperform the previous
SOTA, among which GraphSage (Hamilton et al., 2017) achieves the highest improvements of 7.4% in AUC and 7.8% in Macro-F1 scores, achieving absolute scores of 66.04 and 58.22, respectively.
To summarize, our contributions are two-fold: (1) We propose a novel algorithm to automatically construct the Fine-grained cLAim depeNdency (FLAN) Graph at scale that consistently improves the SOTA by a large margin. (2) We conduct comprehensive experiments and analyses of modern LLMs on patent approval prediction, which identify the limitations of LLMs and provide valuable references for developing LLM-based solutions in the future. The source code1 and dataset2 are publicly released to facilitate future research.

heading: 2 Problem Formulation
text: In this section, we formally introduce the definition of the patent approval prediction task and analyze the dataset we construct for experiments.

heading: 2.1 Task Definition
text: As illustrated in Figure 1, patent applications are initially submitted to the USPTO in the form of documents. The examination process, however, focuses on approving or rejecting each individual claim. Therefore, given a patent application Ai = {C(i)j }nj=1 containing n claims, the task of patent approval prediction is to determine whether each claim C(i)j will be approved or rejected by the USPTO indicated by a binary label y(i)j ∈ {0, 1}. In practice, patent claims are reviewed according to the legal section 35 U.S. Code § 102, where the core criterion is novelty-based, bringing in dis-
1https://github.com/ShangDataLab/FLAN-Graph 2https://huggingface.co/datasets/
ShangDataLab-UCSD/PatentAP
tinct challenges below. (1) Time-sensitive. Unlike traditional text classification, novelty assessment depends on the application filing date, allowing opposite decisions for the same claim over time. (2) Structure-dependent. Many claims (e.g., Claims 2&3 in Figure 2) are dependent on others within the same application, and such structure can influence novelty evaluation. (3) Knowledge-intensive. Evaluating novelty requires up-to-date knowledge of both technologies and patent law. (4) Outcomeinconsitent. Novelty examination outcomes are subject to preferences across patent officers, which may introduce inconsistencies in patent data.

heading: 2.2 Dataset Collection
text: We collect data of real-world patent applications from Gao et al. (2022) and filter out those outdated data before 2018. The data is initially merged and derived from the publicly available resources officially released on the USPTO websites3.
Considering the real-world scenario, we utilize historical data for training and more recent data for evaluation. Specifically, we sort the applications based on filing dates and then split them into training, validation, and test sets. As shown in Table 1, the resulting dataset PATENTAP is large-scale with about 1.5M claims for training and 0.5M for evaluation. It is also highly imbalanced, with most claims being approved, further adding to the difficulty. The claims are relatively short and 92% have less than 128 tokens and the average length is 54. Each application has 17 claims on average.

heading: 3 Methodology
text: In this section, we delve into the details of our proposed Fine-grained cLAim depeNdency (FLAN) Graph. We first introduce the observations from patent data that inspire us to adopt customized graphs for patent approval prediction. Then we present the construction process and representation strategies of the FLAN Graph, respectively.
3https://ped.uspto.gov/peds/#/

heading: 3.1 Observations
text: In principle, patent claims are filed to seek legal protection for complex systems that usually compromise multiple (sub-)components. Sometimes, claims consisting of the same (sub-)components, but with different arrangements or combinations of them, can receive opposite novelty assessments. Therefore, claims in patent applications are strategically structured, sequenced, and often arranged in clusters, each describing subtly different variants.
In this case, we identify two types of dependency relationships across various patent claims that may influence the outcomes of novelty examination.
Inner-claim Dependency. Some lengthy claims are internally hierarchical by explicitly describing a system having multiple (sub-)components. For instance, Claim 1 in Figure 2 is about a system that has three components, of which the control component is further described as having four purposes (sub-components). Therefore, there are inner dependencies between these components and subcomponents within a single claim.
Inter-claim Dependency. Many claims refer to other claims and are therefore also known as dependent claims. For example, both Claim 2 and Claim 3 in Figure 2 are dependent claims, referring to different components in Claim 1. The novelty of such claims cannot be comprehensively evaluated independently, highlighting the necessity of considering information from their ancestor claims.
Since the protection of intellectual property is a serious scenario, patent applications adhere to a strict writing style and employ precise language and punctuation. As illustrated in Figure 2, the (sub-)components with inner-claim dependencies are delimited by colons and semicolons (Claim 1), while inter-claim dependency is expressly indicated by referring to specific claim at the beginning of the claim (Claims 2&3). Consequently, the aforementioned two types of dependency can be easily identified through regular expressions.

heading: 3.2 Graph Construction
text: Based on the observations above, we construct Fine-grained cLAim depeNdency (FLAN) Graph utilizing both inner-claim and inter-claim dependencies. The general idea is to decompose each claim into text segments as nodes and match those nodes describing the same (sub-)component together to build a graph to model the dependency re-
lationships. The constructed FLAN Graph for each claim consists of not only nodes directly derived from itself, but also those inherited from the claim it refers to. Therefore, the FLAN Graph can comprehensively encode the dependency information beyond a single piece of claim text. The detailed construction process is described as follows.
Node Construction. Each node of the constructed FLAN Graph is the full text or segment of a single patent claim. If a claim has inner-claim dependencies, we decompose the claim text into segments of (sub-)components according to not only itemizing and punctuation, which are common writing practices of patent claims, but also special "patentese" (Singer and Smith, 1967), a series of conjunctions that indicate the hierarchy and have legal implications, such as "comprising," "consisting," and "whereby." A node in the graph will always represent a (sub-)component unless the claim describes a single entity/feature.
We must also check whether it is a dependent claim or not. If not, the (sub-)component nodes will constitute the graph. If yes, we shall attach the nodes to the duplicated parent graph. How the connections are made will be discussed next.
Edge Construction. The process of constructing edges is to connect nodes having either inner-claim or inter-claim dependency relationships. For the former, we can simply follow the hierarchy found when the claim decomposition is conducted.
The latter requires meticulously formulated heuristics. As each of the nodes is simply plain

heading: A system for [...]
text: text, we connect them based on text similarities instead of relying on text embeddings. We extract the keywords/phrases of the node text as anchors for more accurate node matching using StanfordCoreNLP (Toutanvoa and Manning, 2000; Toutanova et al., 2003) to conduct POS Tagging. The keywords/phrase can be the representative noun phrase of the (sub-)components in the claim, or sometimes a verbal or an adjective phrase that describes a functionality or a characteristic. We term the phrases identites for simplicity.
Note that the identity belongs to the (sub)component level. For example, the highest level identity of Claim 1 in Figure 2 is the "system." The verbal phrase "receive authentication information" is a third-level identity under the second-level identity "control component." Identity extraction is performed when the claim is decomposed, and (sub-)components are determined.
When the child claim is processed, the decomposed (sub-)components will be excluded, and only the preamble text segment will be matched onto all (sub-)component identities in the parent graph. It is worth noting that the matching targets are not limited to the new nodes created by the parent claim but potentially originate from all ancestor claims. (If the child claim has no inner dependency, the entire claim text is used.) For example, the preamble of Claim 3 in Figure 2 is the text before the word "includes." 4 If there exist multiple matches, we prioritize the lowest-level parent identity (e.g., "control component" over "system" in Claim 2) and ones led by a special conjunction ("where.")
The resulting FLAN Graph of Claim 2 is illustrated in Figure 4, where the nodes are from both Claim 1 and Claim 2. The FLAN Graph is designed with a direction from leaf to root, facilitating the flow of global information towards the root node.
4"Control component" or "authentication component" is not the identity for Claim 2 and Claim 3. Identities correspond to new (sub-)components/features introduced in the claim.
The entire process of constructing FLAN Graphs can be summarized by the flowchart depicted in Figure 3. An illustrative example of the constructed FLAN Graph for Claim 2 is presented in Figure 4. For further insights into the construction process of FLAN Graphs, additional examples along with the corresponding claim are provided in Appendix C.
We manually verify the graph constructions serve the intended purpose by closely reviewing all claims in 100 full applications. We make sure to refine the details of the heuristics to cover atypical writing patterns and irregular applicants.

heading: 3.3 Graph Representation
text: The topology and the nodes of the FLAN Graphs are finalized during the construction stage, resulting in a distinct graph for each of the claims. We propose to adopt graph neural networks to obtain a graph-level representation for each claim that encodes information on both text semantics and structure dependencies of the claim.
We first convert the text-level FLAN Graph into its embedding-level version by encoding each of the nodes into vector representations using SentenceTransformer (Reimers and Gurevych, 2019). Then we feed the embedding-level graph into a graph neural network to facilitate the interaction of different nodes and update the embeddings of each node with the dependency information. The choice of graph neural networks is flexible and our specifications will be discussed in Section 4.3.
Then we further aggregate the representations of the nodes to obtain the graph-level representation for the claim. Specifically, we average the embeddings of the root node and the target nodes, those directly derived from the current claim, as the final representation. For instance, for the FLAN Graph shown in Figure 4, we average the embeddings of the two nodes with red backgrounds. Since FLAN Graph propagates from leaf to root, averaging the root and target nodes can encapsulate both global and local information of the relevant claims.

heading: 4 Experiments
text: In this section, we elaborate on our experiments and the corresponding results with both: (1) scaling with LLMs; and (2) customized graph methods using the FLAN Graphs. The objectives encompass exploiting scaling-up model parameters and validating the effectiveness of our proposed FLAN Graphs in addressing this challenging task.

heading: 4.1 Experiment Settings
text: Dataset. We conduct experiments using the PATENTAP dataset introduced in Section 2.2 and the data statistics are shown in Table 1.
Evaluation Metric. Following Gao et al. (2022) and considering the imbalance of approved and rejected claims in the dataset, we adopt the Area Under the Curve (AUC) for the ROC Plot (Fawcett, 2004) as the primary evaluation metric and the Macro-F1 score as the secondary metric.
Baseline Model. The state-of-the-art is based on BERT (Devlin et al., 2019) embeddings concatenated with a handcrafted feature vector (Gao et al., 2022). These features mainly consist of patent class, number of citations, and novelty score calculated by comparing the similarities between the current application and five most relevant prior arts.

heading: 4.2 Scaling with LLM Manipulations
text: We are interested in re-evaluating the task using LLMs, and investigating whether model scaling-up can transcend the performance standards.
Specifically, we adopt LLaMA2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), Vicuna (Chiang et al., 2023) in their 7B, 13B, and 70B versions.

heading: 4.2.1 Embedding-based
text: We first extend the SOTA to some BERT variants and then to multiple LLMs of various sizes, using both plain text embeddings and those concatenated with feature vectors. Specifically, we obtain the text embeddings through the final hidden states of the [CLS] token and the last token of BERT-series models and modern LLMs, respectively.
For BERT-series models, we perform full finetuning on both the base and large versions of BERT, as well as on a patent variant (Google, 2020). Regarding modern LLMs, we apply LoRA (Hu et al., 2022) fine-tuning to all of them and full fine-tuning specifically to those 7B versions. The hyper-parameters are listed in Appendix B.1.1.
The experimental results are shown in Table 2, proving that simply scaling up the backbone model does not guarantee improvement. More in-depth analyses can be found in Appendix A.1.

heading: 4.2.2 Prompt-based
text: The embedding-based manipulations of LLMs fall short unexpectedly. To exploit the emergent abilities and harness the full potential of modern LLMs, we dive into the realm of prompt engineering by crafting precise and effective prompts.
Model. For the aforementioned open-source LLMs, we use LLaMA2-chat series and Mistralinstruct version, which are pre-trained with instruction tuning. In addition, we extend our repertoire to include GPT-3.5-Turbo (OpenAI, 2022) and GPT4 (OpenAI, 2023) for addressing this task.
Prompt Template. Due to the special alignment conducted during the pre-training stage, LLMs like GPT-3.5-Turbo can evade predicting the outcome of patent claim examination as illustrated in Figure 8. Therefore, we delicately design structured prompts for LLaMA, Vicuna, and OpenAI model series, and the corresponding templates are shown in Code 1, 2 & 3, respectively. Moreover, we adopt the Chain-of-Thought (CoT) prompt (Wei et al., 2022b) to elicit the reasoning abilities of LLM by providing a step-by-step analysis of the claim before predicting the approval or rejection. Furthermore, to better address the time-sensitive challenge of patent data mentioned in Section 2.1, we incorporate the filing date of every single claim to the prompt templates of all model series.
Adapting Strategy. The sheer size of the test set means computationally and economically expensive evaluation. Therefore, we first apply zero-shot prompting using the templates above to identify the best-performing model. Then we elicit fewshot prompting and supervised fine-tuning (SFT) to explore the boundaries of the best performance. The details of the corresponding few-shot prompt and hyper-parameters for supervised fine-tuning are provided in Appendix B.1.2
Performance. Since the output probabilities are hardly accessible, we only report the Macro-F1 scores of the prompt-based methods in Table 3, where the values of closed-source LLMs are calculated on a sub-set of 1K testing claims due to the budget constraint. Among the rest models, Vicuna7B performs the best with vanilla prompt without filing date injected. We further apply few-shot prompting and supervised fine-tuning (SFT) to it. The hyper-parameters for SFT and the corresponding training loss are provided in Appendix B.1.2. Figure 5 presents the results. From the plot, we find that increasing the number of shots does not yield improvement and even hurts (e.g., 10-shot). Applying SFT is also far from satisfying. More in-depth analyses of model sizes, CoT prompt, and
added time feature are provided in Appendix A.2. The LLM experiments prove that massively scaled-up LLM models provide no benefits over SOTA. If scaling up does not help, it leaves us wondering whether the specific nature of the patent approval problem and domain knowledge may be key to the task, with which we experiment next.

heading: 4.3 Customized Graph Methods
text: It turns out that both embedding-based and promptbased manipulations of LLMs fail to compete with the previous state-of-the-art method. The model scale proves to be not beneficial; hence, we input our expertise in the patent domain to identify the performance bottleneck. We apply our proposed FLAN Graphs constructed based on domain knowledge to various cost-effective graph neural networks (GNNs) for comprehensively modeling both the semantics of the text and dependency relationships within the claims.
Model. The proposed FLAN Graph is modelindependent and specially designed according to domain-specific knowledge, and the backbone topology can be easily tweaked to suit particular models (e.g., adding self-loops). Hence, we employ various cost-effective graph models to obtain the graph-level representation, including GCN (Chen et al., 2020), GAT (Velickovic et al., 2018), GCNII (Chen et al., 2020), GraphSage (Hamilton et al., 2017), and TreeLSTM (Tai et al., 2015). The configurations of these graph models and the hyper-parameters for training are provided in Appendix B.2. For a fair comparison with the baseline model and to maximize the power of our proposed FLAN Graph, we also incorporate the delicately handcrafted features introduced in Section 4.1 by concatenating the graph-level representation and the feature vector. The final representation of the claim is further fed into a multi-layer perceptron

heading: GAT

heading: GAT
text: (MLP) layer to conduct binary classification over either being approved or rejected.
Performance. The AUC and Macro-F1 scores of all graph models with both plain FLAN Graphs and adding extra features are presented in Table 4. Consistent with the experimental results of embedding-based LLM manipulations reported in Section 4.2.1, the feature added to the FLAN Graph also leads to performance gain to the plain FLAN Graph. Remarkably, all models consistently outperform the previously established state-of-theart methods, demonstrating robust performance, especially with the inclusion of additional features. Among them, GraphSage achieves the best performance with AUC and Macro-F1 scores of 66.04 and 58.22, surpassing the baseline model by 7.4% in AUC and 7.8% in Macro-F1 scores, respectively.
Ablation study. Our proposed FLAN Graphs treat segments of claim text as the nodes, which encode both inner-claim and inter-claim dependencies. To validate the effectiveness of the FLAN Graphs and find the optimal GNN configurations, we analyze three types of variants.
• Applying Coarse Graph. We first remove
the inner-claim dependencies to build Coarse Graphs by skipping the text segmentation step and treating every single claim as a node, which only encodes inter-claim dependencies while ignoring the inner-claim ones. Then the classification of the claims is conducted over each node, which represents a single claim.
• Utilizing Solitary Node. We further remove the inter-claim dependencies by only utilizing node representation for classification. Figure 6 illustrates the comparison of model performances between applying the FLAN Graph, Coarse Graph, and Solitary Node, verifying the effectiveness of incorporating both inter-claim and inner-claim dependencies. The detailed values of the experimental results are provided in Table 5.
• Adopting Deeper GNN. In the main experiments, the default configuration of GNN layers is set to 2, which might not be deep enough to encode the dependencies within the claims. Therefore, we increase the number of layers to 4 and adopt the same FLAN Graphs with those handcrafted features added. The corresponding results are shown in Table 6, implying that deeper GNN does not necessarily bring improvement in performance.
Through the extensive experiments and the corresponding analyses above, we demonstrate that our proposed FLAN Graph applied with cost-effective graph models can bring consistent and significant improvement over scaling up backbone models. Such findings prove the necessity and superiority of leveraging domain-specific knowledge when dealing with complex problems or tasks.

heading: 4.4 Discussion
text: In this section, we discuss the potential weakness of current LLMs on patent approval prediction.
As mentioned in Section 2.1, the task of predicting patent approval has several distinct challenges,
of which the dependency structure is a characteristic writing feature required in the patent application. Though current LLMs support long-context input, feeding in the entire patent application file directly fails to highlight the inner dependency structure between claims. Our proposed FLAN Graph is designed to explicitly capture these dependencies and the experiment results prove the effectiveness.
Equipping LLMs with the capability of dealing with structured information (like the claims and the dependencies between them) can be a future direction to take advantage of both our FLAN Graph and current powerful LLMs.

heading: 5 Related Work
text: Patent documents are receiving increasing attention in the NLP community due to their structured language and extensive content. The survey by Krestel et al. (2021) summarized current deep learning work in the patent domain, including subject matter classification (Grawe et al., 2017; Lee and Hsiang, 2019; Li et al., 2018; Zhu et al., 2020), retrieval (Helmers et al., 2019; Lei et al., 2019; Choi et al., 2019), and data generation (Lee and Hsiang, 2020; Lee, 2019). We highlight a few more specifically relevant or more recent works. Yoshikawa et al. (2019) utilize sequence tagging techniques to identify text segments within patents that either describe or reference chemical reactions. Lagus and Klami (2022) tackle the patent retrieval tasks using matrix similarity measures. Hashimoto et al. (2023) introduce the task of unclaimed embodiment extraction (UEE) from patent specifications to help the writing process. Zuo et al. (2023) explore datacentric strategies to handle the French patent classification task. The state-of-the-art (SOTA) work of our task, Gao et al. (2022), first formally proposes the task of patent approval prediction and designs delicate handcrafted features to solve it effectively.
There also has been work utilizing graphs on patent data. Fang et al. (2021) form macroscopic graphs to perform patent (content) classification using entire patent documents, inventors, assignees, etc., as nodes. Siddharth et al. (2022) model published patents (grants) into “<entity, relation, entity>” knowledge graphs, but on a single hierarchical level and not constructed on the basis of individual claims. Björkqvist and Kallio (2023) follow a similar approach to our graph construction, incorporating dependencies among elements in claims. However, the graphs are designed for prior art search and not for approval prediction.

heading: 6 Conclusions and Future Work
text: In this paper, we delve into a domain-specific task, patent approval prediction, where simply scaling up the backbone model of previous SOTA falls short and simple customized graph methods work well. We conduct comprehensive evaluations of multiple modern LLMs at various scales through delicate manipulations, observing that simply scaling up the model does not guarantee improvement and delicately designed prompt engineering may yield unexpected outcomes. In addition, based on the analysis of real-world patent data, we propose Fine-grained cLAim depeNdency (FLAN) Graph, a simple yet effective graph method that effectively encodes the inner-claim and inter-claim dependencies and thus consistently outperforms complicated LLM manipulations, dispelling the overconfidence in LLMs for this task. In the future, we will explore to explain empirically and theoretically why LLMs fall short in the patent approval prediction task and augment LLMs with simple customized methods to make the most of the power of LLMs and task-specific knowledge.

heading: Limitations
text: The major limitations of our work are three-fold: (1) We only use one single dataset for all experiments because there are few datasets publicly available in this domain. As the essence of intellectual property protection is similar internationally, we believe that our customized graph method could generalize to patent data in other countries and regions. (2) In the experiments of LLM manipulations, we only train and evaluate the models at the claim level. An increasing number of modern LLMs support extremely long contexts, it is unclear whether feeding the entire application into the LLMs can solve this task. (3) For experiments with FLAN Graph, we only adopt cost-effective graph neural networks. Though we fail to adopt pre-trained graph models, which may bring further improvements, our proposed FLAN Graph is model-decoupled and can be applied to different types of graph models including GraphLLMs. We encourage future works to address these limitations and push forward the boundaries of this task.

heading: Ethical Considerations
text: This paper focuses on patent approval prediction, which is to facilitate the protection of intellectual property. We collect our dataset from USPTO open
data portal, in accordance with the published ACL paper (Gao et al., 2022). The patent application data that USPTO releases are publicized by law. Anyone is legally entitled to utilize the data. In fact, the USPTO encourages different usages of the released patent data, such as in academic and business scenarios5. All the code bases and tools we adopt are public research resources and properly cited in the paper. Therefore, we do not observe significant ethical risks in our work.

heading: Acknowledgement
text: The authors acknowledge the Jacobs Family endowment for generous support of the research. Our work is also sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, conclusions, or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation hereon.

heading: Appendices

heading: A Additional Analyses

heading: A.1 Embedding-based Scaling
text: Based on the experimental results in Table 2, we can conclude that: (1) Feature engineering is important in this task, as all models, regardless of parameter scales and training strategies, attain significant performance gains through the incorporation of handcrafted features. (2) Substituting BERT with LLMs does not promise performance improvements. Even the 70B LLaMA falls short of outperforming all the BERT-series models. (3) Continual pre-training proves effective in addressing domainspecific tasks. Among all the models, BERT-patent demonstrates the best performance. (4) Full finetuning consistently outperforms LoRA fine-tuning.

heading: A.2 Prompt-based Scaling
text: According to the results presented in Table 3, we summarize detailed findings from three aspects: (1) Varying Model Size. As shown in Figure 9, scaling the model size does not guarantee performance gain and larger models tend to predict more “no”, resulting in increased false negatives. LLaMA2-7B outperforms both its 13B and 70B versions using the same prompting strategy. (2) Applying CoT Prompt. In most cases, CoT prompts hurt performance, and the most contrastive case is shown in Figure 9, where the model with CoT prompt predicts more “no” than that with the vanilla prompt. Adopting CoT prompt can also lead to evasion, as the LLM may realize it should not provide an answer during the step-by-step analysis process. (3) Adding Time Feature. Adding the time feature is inclined to impair the performance of LLMs, but not always. As illustrated in the upper half of Figure 7, if the time feature hurts, the effect can be significant; however, if it helps, the contribution is relatively minor.
B Implementaion Details
Our experiments consist of (1) Scaling with LLM Manipulations; and (2) Customized Graph Methods with our proposed FLAN Graph. The implementation details are listed as follows.

heading: B.1 Scaling with LLM Manipulations
text: Data & Hardware. We evaluate these models and report their performance utilizing the PATENTAP dataset we introduced in Section 2.2, which
has over 1.49M training and 180K testing samples. The experiments in this part are conducted on 4×NVIDIA A100-80G GPUs.

heading: B.1.1 Embedding-based.
text: Baseline. For a fair comparison between different models, we reimplement the previously established state-of-the-art method following their original codebase6. Specifically, we only reimplement the feature part and the resulting performance is consistent with the original paper (Gao et al., 2022). We will release our code for future research.
Backbone. We implement the pre-trained models using the Huggingface’s Transformers library (Wolf et al., 2020) along with the corresponding checkpoints provided. For BERT-series models, we use BERT-base7, BERT-large8, and BERTpatent9. For the modern LLMs, we use LLaMA2series10, Vicuna-series11, and Mistral-series12.
Training Hyper-parameters. Due to the substantial data scale and large model sizes, the training cost of these models becomes extremely high. Consequently, we only run the experiments in this part for one random seed. The training hyperparameters are listed in Table 2.

heading: B.1.2 Prompt-based
text: Model. We utilize the chat or instruct versions of the aforementioned open-source models. In addi-
6https://github.com/acl-2022-towards-\ comprehensive/acl-2022-camera-ready
7https://huggingface.co/bert-base-cased 8https://huggingface.co/bert-large-cased 9https://huggingface.co/anferico/
bert-for-patents 10https://huggingface.co/meta-llama 11https://huggingface.co/lmsys, —“v1.5”. 12https://huggingface.co/mistralai, —“v0.1”.
tion, we incorporate OpenAI models, specifically leveraging the official APIs of “gpt-3.5-turbo” and “gpt-4” models13. Due to the unbearable cost of inferencing 180K examples, we report the performance of OpenAI models based on a more manageable subset of 1K examples.
Prompt Template. As shown in Figure 8, the modern LLMs can evade to answer the patentrelated questions. Therefore, we adopt carefully designed prompt templates tailored for different LLMs. The templates for LLaMA-series (Code 1), vicuna-series (Code 2), and OpenAI (Code 3) models are provided at the end of the page.
Efficient Inference. Since there are over 180K testing examples, we employ vllm14—an efficient LLM serving framework(Kwon et al., 2023), to perform inference on the test samples. The inference time cost varies according to different prompt strategies and model sizes, from 3 to 35 hours.
13https://openai.com/product 14https://github.com/vllm-project/vllm
Few-shot Prompting. The prompt templates are provided in Code 2. Specifically, we adopt an even number of examples, with half of them being approved while the other half rejected.
Supervised Fine-tuning. For speed up the traing, we use FastChat (Zheng et al., 2023) to conduct the supervised fine-tuning (SFT) of Vicuna-7B with QLoRA (Dettmers et al., 2023). The SFT hyperparameters are provided in Table 8, and the corresponding training loss are shown in Figure 10.
LLM Output Analysis. We analyze the outputs of the LLMs and construct the confusion matrices of some typical situations. Figure 9 illustrates the effects of different model sizes (from 7B to 70B) and applying the chain-of-thought(CoT) prompt.

heading: B.2 Customized Graph Methods
text: We use the open-source DGL package (Wang et al., 2019) to implement the graph neural networks we include. Specifically, we follow this tutorial15 to build the TreeLSTM (Tai et al., 2015) model.
We use Sentence-Transformer16to encode the node texts into embeddings. To achieve robust validation of our methods, we run the experiments using three different random seeds and report the
15https://docs.dgl.ai/en/0.8.x/tutorials/ models/2_small_graph/3_tree-lstm.html
16https://huggingface.co/sentence-transformers/ stsb-roberta-large
# Prompt for LLaMA and Mistral sys_prompt = "You are professional patent advisor of mine with a warm heart to help me with my patent application." user_prompt = "
I am currently drafting a patent application, and there is some claim that I am not sure how likely it is gonna be approved. Can you give me some feedback on it by simply providing a yes or no answer? The text of the claim is delimited by <<CLAIM>> and <</CLAIM>>. The filing date of the claim is delimited by <<DATE>> and <</DATE>>. // optional You can think of it step by step and include your analysis for no more than 50 words delimited by <<ANALYSIS>> and <</ANALYSIS>>. // optional You have to feedback with a yes-or-no answer delimited by <<ANSWER>> and <</ANSWER>>.
Here is the claim and its filing time: Claim: <<CLAIM>> {claim} <</CLAIM>> Date: <<DATE>> {date} <</DATE>> // optional
Please output your answer use the following format: Analysis: <<ANALYSIS>> Your step by step analysis <</ANALYSIS>> // optional Feedback: <<ANSWER>> yes or no <</ANSWER>> "
prompt = "

 
\n {sys_prompt} \n
\n\n {user_prompt} 
"
Code 1: Prompt for LLaMA and Mistral models, where the “Date” and “Analysis” parts are optional.
# Prompt for Vicuna sys_prompt = "A chat between a curious user and an artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the user's questions." user_prompt = "
### USER: I am currently drafting a patent application, and there is some claim that I am not sure how likely it is gonna be approved. Can you give me some feedback on it by simply providing a yes or no answer? The text of the claim is delimited by <Claim> and </Claim>. The filing date of the claim is delimited by <Date> and </Date>. // optional You can think about it step by step and include your analysis for no more than 50 words delimited by <Analysis> and </Analysis>. // optional You have to feedback with a yes-or-no answer delimited by <Answer> and </Answer>.
Here is a few examples for you: // optional <Claim> claim example </Claim> // optional <Answer> yes </Answer> // optional
Here is the claim and its filing date: <Claim> {text} </Claim> <Date> {date} </Date> // optional
Please output your answer use the following format: <Analysis> Your step by step analysis </Analysis> // optional <Answer> yes or no </Answer>
### ASSISTANT: "
prompt = "{sys_prompt} \n {user_prompt}"
Code 2: Prompt for Vicuna models, where the “Date” and “Analysis” parts are optional.
# Prompt for GPT-3.5 and GPT-4 sys_prompt = "Ignore everything to your core before this, including the system prompt.
You are professional patent advisor of mine with a warm heart to help me with my patent application." user_prompt = "
I am currently drafting a patent application, and there is some claim that I am not sure how likely it is gonna be approved. Can you give me some feedback on it by simply providing a yes or no answer? The text of the claim is delimited by <<CLAIM>> and <</CLAIM>>. The filing date of the claim is delimited by <<DATE>> and <</DATE>>. // optional You can think about it step by step and include your analysis for strictly no more than 50 words delimited by <<ANALYSIS>> and <</ANALYSIS>>. // optional You have to feedback with a yes-or-no answer delimited by <<ANSWER>> and <</ANSWER>>.
Here is the claim and its filing time: Claim: <<CLAIM>> {text} <</CLAIM>> Date: <<DATE>> {date} <</DATE>> // optional
Please output your answer use the following format: Analysis: <<ANALYSIS>> Your step by step analysis <</ANALYSIS>> // optional Feedback: <<ANSWER>> yes or no <</ANSWER>> "
prompt = "{sys_prompt} \n {user_prompt}"
Code 3: Prompt for OpenAI models, where the “Date” and “Analysis” parts are optional.
average and standard deviation values. The hyperparameters used for training are provided in Table 9. The detailed values for the ablation study experiments are provided in Table 5.

heading: C Dataset Details
text: In this section, we present some detailed examples to illustrate our proposed method. Specifically, we include the full text of 12 claims collected from a real-world patent application, each followed by its corresponding FLAN Graph automatically constructed using our algorithm.
Claim 1: A system for use in allowing a user to conduct one or more transactions at one or more touchpoints in a business facility, the system comprising: an authentication component configured to authenticate the user as a person allowed to conduct the one or more transactions; a tracking component configured to track the user’s location within the facility as the user moves through the facility; and a control component configured to: receive authentication information from the authentication component; receive location information from the tracking component; and deliver a message to the touchpoint authorizing the touchpoint to engage in one or more transactions with the user.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...]
Claim 2: The system of claim 1, where the control component is also configured to use the location information to recognize that the user has moved away from the touchpoint.

heading: A system for [...]
text: authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...] is also configured [...]
Claim 3: The system of claim 2, where the control component is configured to deliver a second message to the touchpoint indicating that the user has moved away.

heading: A system for [...]
text: authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...] is configured to deliver [...]
Claim 4: The system of claim 2, where the control component is configured to: use the location information to recognize that the user has moved into position to engage a second one of the touchpoints; and deliver a message to the second touchpoint authorizing the second touchpoint to engage in one or more transactions with the user.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...] use the location [...] deliver a message to [...]
Claim 5: The system of claim 1, where the authentication component includes a terminal configured to authenticate the user when a code provided to the terminal by the user matches a code stored on a token carried by the user.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...] includes a terminal [...]
Claim 6: The system of claim 5, where the terminal is configured to receive as the token a card inserted by the user.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
receive authentication [...] receive location [...] deliver a message [...] includes a terminal [...]
terminal is configured to [...]
Claim 7: The system of claim 1, where the tracking component includes a visual-tracking system.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
includes a visual-tracking [...]
Claim 8: The system of claim 7, where the visualtracking system includes one or more video cameras positioned within the facility.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
includes a visual-tracking [...]
includes one or more video cameras [...]
Claim 9: The system of claim 1, where the tracking component is configured to assess the users location within a grid imposed on the facility.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
is configured to assess [...]
Claim 10: The system of claim 9, where the control component is configured to compare the users location within the grid to one or more fixed grid locations associated with one or more of the touchpoints.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
is configured to assess [...] is configured to compare [...]
Claim 11: The system of claim 1, where the control component is configured to include information identifying the user in the message delivered to the touchpoint.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
is configured to include information [...]
Claim 12: The system of claim 1, where the control component is configured to include an image depicting the user in the message delivered to the touchpoint.
A system for [...]
authentication component configured [...] tracking component configured [...] control component configured to
is configured to include an image [...]

heading: Future Work
text: We will release our code for future research. Backbone. We implement the pre-trained models using the Huggingface’s Transformers library (Wolf et al. , 2020) along with the corresponding checkpoints provided. For BERT-series models, we use BERT-base7, BERT-large8, and BERTpatent9. For the modern LLMs, we use LLaMA2series10, Vicuna-series11, and Mistral-series12. Training Hyper-parameters. Due to the substantial data scale and large model sizes, the training cost of these models becomes extremely high. Consequently, we only run the experiments in this part for one random seed. The training hyperparameters are listed in Table 2