Text2NKG: Fine-Grained N-ary Relation Extraction for N-ary relational Knowledge Graph Construction

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:34:53

Abstract: Beyond traditional binary relational facts, n-ary relational knowledge graphs (NKGs) are comprised of n-ary relational facts containing more than two entities, which are closer to real-world facts with broader applications. However, the construction of NKGs remains at a coarse-grained level, which is always in a single schema, ignoring the order and variable arity of entities. To address these restrictions, we propose Text2NKG, a novel fine-grained n-ary relation extraction framework for n-ary relational knowledge graph construction. We introduce a span-tuple classification approach with hetero-ordered merging and output merging to accomplish fine-grained n-ary relation extraction in different arity. Furthermore, Text2NKG supports four typical NKG schemas: hyper-relational schema, event-based schema, role-based schema, and hypergraph-based schema, with high flexibility and practicality. The experimental results demonstrate that Text2NKG achieves state-of-the-art performance in F1 scores on the fine-grained n-ary relation extraction benchmark. Our code and datasets are publicly available.

Main paper: 
 related work: 
 N-ary relational Knowledge Graph. An n-ary relational knowledge graph (NKG) consists of n-ary relational facts, which contain n entities (n ≥ 2) and several relations. The n-ary relational facts are necessary and cannot be replaced by combinations of some binary relational facts because we cannot distinguish which binary relations are combined to represent the n-ary relational fact in the whole KG. Therefore, NKG utilizes a schema in every n-ary relational fact locally and a hypergraph representation globally. Firstly, the simplest NKG schema is hypergraph-based. Wen et al. (2016) found that over 30% of Freebase Bollacker et al. (2008) entities participate facts with more than two entities, first defined n-ary relations mathematically and used star-to-clique conversion to convert triple-based facts representing n-ary relational facts into the first NKG dataset JF17K in hypergraph-based schema (r, {v i } n i=1 ). Fatemi et al. (2021) proposed FB-AUTO and M-FB15K with the same hypergraph-based schema. Secondly, Guan et al. (2019) introduced role information for n-ary relational facts and extracted Wikipeople, the first NKG dataset in role-based schema ({(k i , v i )} n i=1 ), composed of role-value pairs. Thirdly, Wikidata Vrandečić & Krötzsch (2014) , the largest knowledge base, utilizes an NKG schema based on hyper-relation (s, r, o,{(k i , v i )} n-2 i=1), which adds auxiliary key-value pairs to the main triple. Galkin et al. (2020) first proposed an NKG dataset in hyper-relational schema WD50K. Fourthly, as Guan et al. (2022) pointed out, events are also n-ary relational facts. One basic event representation has an event type, a trigger, and several key-value pairs Lu et al. (2021) . Regarding the event type as the main relation, the (trigger: value) as one of the key-value pairs, and the arguments as the rest key-value pairs, we can obtain an event-based NKG schema (r, {(k i , v i )} n i=1 ). Based on four common NKG schemas, we propose Text2NKG, the first method for extraction of structured n-ary relational facts from natural language text, which improves NKG representation and application. Problem Definition. Given an input sentence with l words s = {w 1 , w 2 , ..., w l }, an entity e is a consecutive span of words: e = {w p , w p+1 , ..., w q } ∈ E s , where p, q ∈ {1, ..., l}, andf n hr = (e 1 , r 1 , e 2 , {r i-1 , e i } n i=3 ) where {e i } n i=1 ∈ E, {r i } n-1 i=1 ∈ R.E s = {e j } m j=1is the entity set of all m entities in the sentence. The output of n-ary relation extraction, R(), is a set of n-ary relational facts F s in given NKG schema in {f n hr , f n ev , f n ro , f n hg }. Specifically, each n-ary relational fact f n ∈ F s is extracted by multi-label classification of one of the ordered span-tuple for n entities [e i ] n i=1 ∈ E s , forming an answer label-list for n r relations [r i ] nr i=1 ∈ R, where n is the arity of the extracted n-ary relational fact, and n r is the number of answer relations in the fact, which is determined by the given NKG schema: R([e i ] n i=1 ) = [r i ] n-1 i=1 , when f n = f n hr , R([e i ] n i=1 ) = [r i ] n+1 i=1 when f n = f n ev , R([e i ] n i=1 ) = [r i ] n i=1 when f n = f n ro , and R([e i ] n i=1 ) = [r 1 ] when f n = f n hg . 
methodology: 
 In this section, we first introduce the overview of the Text2NKG framework, followed by the spantuple multi-label classification, training strategy, hetero-ordered merging, and output merging.Text2NKG is a fine-grained n-ary relation extraction framework built for n-ary relational knowledge graph (NKG) construction. The input to Text2NKG is natural language text tokens labeled with entity span in sentence units. We extract 3-ary facts as an atomic unit and then merge them into n-ary facts later to realize n-ary extraction of arbitrary arity. Because if using binary facts, merging them into n-ary facts based on shared elements within these facts will lead to misunderstandings as analyzed in Section 1. On the other hand, using facts with four entities or more makes it challenging to judge which of the included 3-ary facts can be extracted as independent facts.Specifically, inspired by Ye et al. (2022) , Text2NKG first encodes the entities using BERT-based Encoder Devlin et al. (2019) with a packaged levitated marker for embedding. Then, each arrangement of ordered span-tuple with three entity embeddings will be classified with multiple labels, and the framework will be learned by the weighted cross-entropy with a null-label bias. In the decoding stage, in order to filter the n-ary relational facts whose entity compositions have isomorphic heteroordered characteristics, Text2NKG proposes a hetero-ordered merging strategy to merge the label probabilities of 3! = 6 arrangement cases of span-tuples composed of the same entities and filter out the output 3-ary relational facts existing non-conforming relations. Finally, Text2NKG combines the output 3-ary relational facts to form the final n-ary relational facts with output merging.For the given sentence token s = {w 1 , w 2 , ..., w l } and the set of entities E s , in order to perform finegrained n-ary relation extraction, we need first to encode a span-tuple (e 1 , e 2 , e 3 ) consisting of every arrangement of three ordered entities, where e 1 , e 2 , e 3 ∈ E s . Due to the high time complexity of training every span-tuple as one training item, inspired by Ye et al. (2022) , we achieve the reduction of training items by using packed levitated markers that pack one training item with each entity in E s separately. Specifically, in each packed training item, a pair of solid tokens, [S] and [/S], are added before and after the packed entity e S = {w p S , ..., w q S }, and (|E s | -1) pairs of levitated markers, [L] and [/L], according to other entities in E s , are added with the same position embeddings as the beginning and end of their corresponding entities span e Li = {w p L i , ..., w q L i } to form the input token X: X ={w 1 , ..., [S], w p S , ..., w q S , [/S], ...,EQUATIONWe encode such token by the BERT-based pre-trained model encoder Devlin et al. (2019) :EQUATIONwhere t = |X| is the imput token length, {h i } t i=1 ∈ R d , and d is embedding size. There are several span-tuples (A, B, C) in a training item. The embedding of first entity h A ∈ R 2d in the span-tuple is obtained by concat embedding of the solid markers, [S] and [/S], and the embeddings of second and third entities h B , h C ∈ R 2d are obtained by concat embeddings of levitated markers, [L] and [/L] with all A 2 m-1 arrangement of any other two entities in E s . Thus, we obtain the embedding representation of the three entities to form A 2 m-1 span-tuples in one training item. Therefore, every input sentence contains m training items with mA 2 m-1 = A 3 m span-tuples for any ordered arrangement of three entities.We then define 3 × n r linear classifiers {FNN k i } nr i=1 , k = 1, 2, 3 to classify the span-tuples for multiple-label classification. Each classifier targets the prediction of one relation r i , thus obtaining a probability lists (P i ) nr i=1 with all relations in given relation set R plus a null-label:EQUATIONwhere FNN k i ∈ R 2d×(|R|+1) , and P i ∈ R (|R|+1) .In order to train the n r classifiers for each relation prediction more accurately, we performed data augmentation strategy in terms of span-tuples. Taking the hyper-relational schema as an example, given a hyper-relational fact (A, r 1 , B, r 2 , C), we consider swapping the head and tail entities and changing the main relation to the corresponding inverse relation (B, r -1 1 , A, r 2 , C), as well as swapping the tail entities and auxiliary values, and swapping the main relation and the auxiliary key (A, r 2 , C, r 1 , B) also as labeled training span-tuple cases. Thus R hr (A, B, C) = (r 1 , r 2 ) can be augmented with 3! = 6 orders of span-tuples:EQUATIONFor other schemas, we can also obtain 6 fully-arranged cases of labeled span-tuples in a similar way, as described in Appendix A. If no n-ary relational fact exists between the three entities of span-tuples, then relation labels are set as null-label.Since most cases of span-tuple are null-label, we set a weight hyperparameter α ∈ (0, 1] between the null-labels and other labels to balance the learning of the null-label. We jointly trained the n r classifiers for each relations by cross-entropy loss L with a null-label weight bias W α :Dataset #Ent #R hr #R ev #R ro #REQUATION)where |R|+1) .W α = [α, 1.0, 1.0, ...1.0] ∈ R (In the decoding stage, since Text2NKG labels all 6 different arrangement of the same entity composition, we design a hetero-ordered merging strategy to merge the corresponding labels of these 6 hetero-ordered span-tuples into one to generate non-repetitive n-ary relational facts unsupervisedly.For hyper-relational schema (n r = 2), we combine the predicted probabilities of two labels P 1 , P 2 in 6 orders to (A, B, C) order as follows: ) + I(PEQUATIONEQUATIONwhere I() is a function for swapping the predicted probability of relations and the corresponding inverse relations. Then, we take the maximum probability to obtain labels r 1 , r 2 , forming a 3-ary relational fact (A, r 1 , B, r 2 , C) and filter it out if there are null-labels in (r 1 , r 2 ). If there are inverse relation labels in (r 1 , r 2 ), we can also transform the order of entities and relations as equation 4.For event-based schema, role-based schema, and hypergraph-based schema, all can be generated by hetero-ordered merging according to this idea, shown in Appendix B.After hetero-ordered merging, we merge the output 3-ary relational facts to form higher-arity facts, with hyper-relational schema based on the same main triple, event-based schema based on the same main relation (event type), role-based schema based on the same key-value pairs, and hypergraphbased schema based on the same hyperedge relation. This way, we can unsupervisedly obtain n-ary relational facts with dynamic number of arity numbers for NKG construction. Evaluation Metrics. We use the F 1 score with precision and recall to evaluate the dev set and the test set. For a predicted n-ary relational fact to be considered correct, the entire fact must match the ground facts completely. 
experiments: 
 Hyperparameters and Enviroment. We train 10 epochs on HyperRED using the Adam optimizer. All experiments were done on a single NVIDIA A100 GPU, and all experimental results were derived by averaging five random seed experiments. Appendix E shows Text2NKG's optimal hyperparameter settings. Appendix F shows training details. The experimental results of proposed Text2NKG and other baselines with both BERT-base and BERT-large encoders can be found in (2022), and LasUIE Fei et al. (2022) for comparison. We found that Text2NKG still works best in these schemas, which demonstrates good versatility.Data augmentation (DA), null-label weight hyperparameter (α), and hetero-ordered merging (HM) are the three main components of Text2NKG. For the different Text2NKG variants as shown in Table 2 , it can be observed that DA, α, and HM all contribute to the accurate results of our complete model. By comparing the differences, we find that HM is most effective by combining the probabilities of labels of different orders, followed by DA and α.We compared the effect for different null-label weight hyperparameters (α). As shown in Figure 4(c) , the larger the α, the greater the learning weight of null-label compared with other lables, the more relations are predicted as null-label. After filtering out the facts having null-label, fewer facts are extracted, so the precision is generally higher, and the recall is generally lower. The smaller the α, the more relations are predicted as non-null labels, thus extracting more n-ary relation facts, so the recall is generally higher, and the precision is generally lower. Comparing the results of F 1 values for different α, it is found that α = 0.01 works best. When applied in practice, the hyperparameter α can be adjusted according to specific needs to obtain the best results.We train 10 epochs on HyperRED with the optimal combination of hyperparameters. Text2NKG and all its variants have been trained on a single NVIDIA A100 GPU. Using our optimal hyperparameter settings, the time required to complete the training on HyperRED is 4h with BERT-base encoder and 10h with BERT-large encoder.G DISCUSSION OF ADVANCED LLMS SUCH AS CHATGPT IN FINE-GRAINED N-ARY RELATION EXTRACTION TASKSWe have tried to use LLM APIs such as ChatGPT and GPT to do similar n-ary relation extraction tasks, i.e., prompting model input and output formats for extraction. The advantage of ChatGPT is that it can perform similar tasks in a few-shot situation, however, for building high-quality knowledge graphs, the performance and the fineness of the n-ary relation extraction are much lower than Text2NKG. This is because ChatGPT is not good at multi-label classification tasks that contain less semantic interpretation. When the number of labels of relations in our relation collection is very large, we need to write a very long prompt to tell the LLM about our label candidate collection, which again leads to the problem of forgetting. Therefore, we have tried numerous prompt templates to enhance the extraction effect of ChatGPT, however, on fine-grained n-ary relation extraction task, the best result of ChatGPT can only reach about 10% of F 1 value on HyperRED, which is much lower than the result of 80%+ F 1 value of Text2NKG.However, advanced LLMs such as ChatGPT are a good idea for training dataset generation for Text2NKG in such tasks to save some manual labor to only verify and correct the training items generated. For future work, we will continue our research in this direction and try to combine large language models with Text2NKG-like supervised models for automated fine-grained n-ary relation extraction for n-ary relational knowledge graph construction.Anonymous Github Code: https://anonymous.4open.science/r/Text2NKG 
: 
 ; Balazevic et al. (2019) . However, real-world facts tend to contain more than two entities, which are called n-ary relational facts (n ≥ 2), which cannot be represented by merging binary relations. For example, consider the statement: "Einstein received his Bachelor degree in Mathematics and his Doctorate degree in Physics." When broken down into binary relations, the facts become: (Einstein, degree, Doctorate degree), (Einstein, major, Physics), (Einstein, degree, Bachelor), and (Einstein, major, Mathematics). However, we can't merge these binary relations effectively because we can't determine whether Einstein's doctoral major was in Physics or Mathematics. This necessitates the use of N-ary relational Knowledge Graphs (NKG) to represent such information, like (Einstein, degree, Doctorate, major, Physics). As shown in Figure 1 , an NKG consists of numerous n-ary relational facts with richer knowledge representation and a wider application capability.Each NKG has a schema to represent the structure of every n-ary relational fact in the NKG. For example, Wikidata utilizes n-ary relational facts with hyper-relational schema Rosso et al. (2020) ; Galkin et al. (2020) ; Wang et al. (2021a), i.e., (s, r, o, {(k i , v i )} n-2 i=1) adds (n -2) keyvalue pairs to the main triple to represent auxiliary information, forming an n-ary relational fact with n entities. In addition to the hyper-relational schema, the n-ary relational facts of NKGs also have event-based schema (r, To address these challenges, we propose a novel n-ary relation extraction framework, Text2NKG, which automates the generation of n-ary relational facts from natural language text for NKG construction. Text2NKG proposes a span-tuple multi-label classification method with hetero merging, which converts n-ary relation extraction into a multi-label classification problem for span-tuples consisting of all arrangements of three entities in a sentence. The number of labels is determined by the number of relations in the selected NKG schema. Text2NKG can be applied to all NKG schemas, with hyper-relational schema, event-based schema, role-based schema, and hypergraphbased schema provided as examples, which have a wide range of applications.EQUATIONIn addition, we extend the current n-ary relation extraction benchmark HyperRED Chia et al. (2022) , which is only in the hyper-relational schema, to four NKG schemas. We've done sufficient n-ary relation extraction experiments on HyperRED, and the experimental results show that Text2NKG achieves nearly 20 percentage points ahead of the existing state-of-the-art model CubeRE in F 1 scores of hyper-relational extraction. We also compared the results of Text2NKG in four schemas to verify applications. We are excited to open-source our complete code and are willing to contribute to the knowledge graph construction community. 
 
 Appendix:  


Decision: The paper focuses on the extraction of n-ary relations from text and proposes a method based on a linear multi-label classification of all possible entity encoding $3$-combinations to build $3$-ary facts. They also use data augmentation (inverse relations, reordering of hyperrelational facts) and use maximum aggregation to obtain a prediction. The $3$-ary facts are combined via a fixed rule for common $(n-1)$-ary prefixes.
The paper's strengths include (1) the motivation and relevance of the task (`rUhA`, `XsBb`), (2) the writing (`ED7F`, `XsBb`), (3) the experimental results (`rUhA`, `ED7F`, `GMbk`), and (4) the number of baselines and ablations considered (`ED7F`, `GMbk`, `XsBb`).
The following shortcomings are present: (1) the scalability of the approach (`ED7F`), (2) a lack of details regarding the use of LLM baselines (`rUhA`, `GMbk`, `XsBb`; these were added during the rebuttal), (3) a lack of detailed analysis of the individual components presented, and (4) a heuristic rule-based aggregation to higher arity.
The reviewers and authors generally participated in the discussions, and some issues were resolved during the discussions.

Reviews: review: paper_summary: paper_summary: The paper presents Text2NKG, a new framework designed for fine-grained n-ary relation extraction aimed at constructing n-ary relational knowledge graphs (NKGs). By introducing a span-tuple classification method combined with hetero-ordered merging and output merging strategies, it achieves extraction across varying entity arities while preserving their order, enhancing the granularity of NKG construction. Moreover, Text2NKG is adaptable to four prevalent NKG schemas.
strength_weakness: strength_weakness: **Lack of Implementation Detail**: Vital specifics regarding input prompts to language models like GPT-4, and whether the Generative Baseline and Pipelinge Baseline underwent supervised fine-tuning, are omitted. These details are crucial for reproducibility and assessing the thoroughness of comparative analysis.

**Inherent Limitations of BERT-style Architectures**: The reliance on BERT-based architectures may restrict the handling of long texts. This could limit the scalability and applicability of Text2NKG in domains with extensive contextual requirements.

**Limited application scope**: Although Text2NKG supports 4 typical NKG schemas, the framework seems to be specifically designed for these specific structures, which may limit its generality in practical applications.
questions: questions: Please refer to the Weaknesses
limitations: limitations: The authors have addressed the limitations.
paper_summary: paper_summary: The paper introduces a novel framework for fine-grained n-ary relation extraction aimed at constructing n-ary relational knowledge graphs (NKGs). Traditional KGs primarily focus on binary relations, but this work targets n-ary relations which involve more than two entities, aligning more closely with real-world facts.
strength_weakness: strength_weakness: 1. Although the framework shows potential for scalability, the paper does not thoroughly address the computational efficiency and scalability when applied to real-time applications.
2. While the paper includes ablation studies, more detailed analysis and discussion on the impact of each component (e.g., data augmentation, null-label weight hyperparameter) on different types of NKG schemas would provide deeper insights into their contributions.
questions: questions: 1. Can you provide an example of extracting N-ary relations in real-world information extraction?
2. How should NKG be stored and utilized, and can n-ary extraction be applied in RAG or Agent in the era of LLMs?
limitations: limitations: See the Weakness.
paper_summary: paper_summary: Text2NKG is a novel framework for fine-grained n-ary relation extraction aimed at constructing n-ary relational knowledge graphs (NKGs). Unlike traditional binary relational knowledge graphs, NKGs encompass relations involving more than two entities, making them more reflective of real-world scenarios. Text2NKG employs a span-tuple classification approach along with hetero-ordered merging and output merging techniques. It supports multiple NKG schemas—hyper-relational, event-based, role-based, and hypergraph-based—and achieves state-of-the-art performance in fine-grained n-ary relation extraction benchmarks.
strength_weakness: strength_weakness: 1) The comparison settings with the large language models need to be clearer.
2) Line 159: imput -> input.
questions: questions: What is the connection and difference between n-ary relation extraction and event extraction?
limitations: limitations: The authors have listed the limitations in the Checklist.
paper_summary: paper_summary: This work introduces Text2NKG. A proclaimed first method of extracting n-ary facts from text for construction a KG. The focus on n-ary is a more complex task than standard RE as an n-ary fact can hold more entities than the standard RDF  {subject, relation, object}.

Candidate 3-ary span tuples are formed from entities combinations in a sentence. This is embedded using BERT, and put through linear classifiers (Feed-Forward-Networks). The resulting 3-ary facts are then combined for n-ary facts using "hetero-ordered merging". The fine-tuning process makes use of a data augmentation technique, a balancing parameter to compensate for the large amount of "no relation" labels, and without merging.

The method is verified on HyperRED (the only public hyperrelational dataset currently available), and beats several SOTA methods.
strength_weakness: strength_weakness: - The provided code is complex with low readibility (>5 nested for/while statements with little/no comments and no documentation).
	- Code makes mention of ACE dataset, but this does not seem to have made it into the end-work, adding to the confusion.
	- Other than the case study in Figure 5, the error analysis is fairly weak and generally performance based. There is no inspection provided as to what explains (for example) the discrepancy between "Text2NKG" and "Text2NKG w/o HM".
	- Prompt templates for GPT that were considered (or eventually used for best result) are not included
	- BERT was trained on a copy of Wikipedia in 2018. There seems to be no consideration that the data in HyperRED may overlap with the pre-training data of BERT.
	- The work hyperfixates on a single benchmark: HyperRED, despite others being mentioned.



Comments:
Figure 2: Typo "hyper-relaional"
Line 159: typo
Line 214: Broken sentence.
Line 219: Why was it decided to name these Generative and Pipeline and not specific to the previous works? (BART and UniRE)
Table 2: The GPT results dont need 4 decimal places
Table 2: typo (pipelinge)

Equation 6: It is unclear to me how this is derived, and it is not explained why (the how is there) this specific formulation works.
Figure 5a: Figure is hard to read.
questions: questions: Questions:
	- Line 40: Several more benchmarks for NKG are mentioned (JF17K, Wikipeople, WD50K, etc.), but not considered (or no argumentation is provided). Why were these not considered?
	- From inspection, what are cases where the model fails (e.g. seemingly difficult cases where it struggles) or cases where it consistenly succeeds?
limitations: limitations: Yes

Comments: comment: Thank you for the detailed rebuttal and clarifications. Your explanation of the time complexity reduction using packed levitated markers and the additional details on the ablation studies provide a clearer understanding of the computational efficiency and the contributions of each component within the framework.

The real-world example of extracting n-ary relations in the medical domain helps illustrate the practical application of your approach. Additionally, your discussion on the storage and utilization of NKGs, particularly in relation to RAG models and intelligent agents, offers valuable insights into the potential use cases of your work.

Overall, your rebuttal effectively addresses the concerns raised, and I appreciate the additional context provided. I remain positive about the contributions of your paper.
The answer for question 2 still makes for a quantitative only-evaluation with little to no qualitative inspection (by looking at examples where the model does well or doesnt do well).

I decide to keep my previous scores.
Thank you for your clear explanations. Your responses have addressed my concerns, and I appreciate the additional insights provided. I maintain my positive decision.