Venue: NAACL 2024

# COSIGN: Contextual Facts Guided Generation for

Knowledge Graph Completion

 Jinpeng Li1, Hang Yu1\({}^{*}\), Xiangfeng Luo1, Qian Liu2

1Shanghai University, China

2School of Computer Science, The University of Auckland, New Zealand {lijinpeng,yuhang,luoxf}@shu.edu.cn, liu.qian@auckland.ac.nz

Corresponding Author is Hang Yu.

###### Abstract

Knowledge graph completion (KGC) aims to infer missing facts based on existing facts within a KG. Recently, research on generative models (GMs) has addressed the limitations of embedding methods in terms of generality and scalability. However, GM-based methods are sensitive to contextual facts on KG, so the contextual facts of poor quality can cause GMs to generate erroneous results. To improve the performance of GM-based methods for various KGC tasks, we propose a **COn**textual Fact**S** GuIded **G**eneratioN (COSIGN) model. First, to enhance the inference ability of the generative model, we designed a contextual facts collector to achieve human-like retrieval behavior. Second, a contextual facts organizer is proposed to learn the organized capabilities of LLMs through knowledge distillation. Finally, the organized contextual facts as the input of the inference generator to generate missing facts. Experimental results demonstrate that COSIGN outperforms state-of-the-art baseline techniques in terms of performance.

## 1 Introduction

A knowledge graph (KG) represents a network of real-world entities and illustrates the relationship between them Ji et al. (2021); Cambria et al. (2022). Knowledge graph completion (KGC) is a task aimed at inferring missing facts based on existing facts within KGs, including static KGC (SKGC) Bordes et al. (2013), temporal KGC (TKGC) Han et al. (2021) and few-shot KGC (FKGC) Xiong et al. (2018). TKGC involves time facts with timestamps, while FKGC predicts facts with limited or zero trained samples for relationships.

Previous approaches primarily relied on graph embedding, which involves embedding entities and relationships into high-dimensional vectors to represent their associations Gao et al. (2023). The training and inference models of SKGC Yang et al. (2014); Trouillon et al. (2016); Sun et al. (2018) depend on various transitional relationships on graph paths, while TKGC Xu et al. (2020); Han et al. (2021); Gao et al. (2023) and FKGC Xiong et al. (2018); Chen et al. (2019); Niu et al. (2021) methods further integrate with special components or learning paradigms to handle additional time information or training requirements. However, these differences in methods result in significant maintenance costs and an inability to adapt to emerging knowledge queries, ingestion, and presentation.

Recently, generative models (GMs) including large language models (LLMs) have demonstrated advanced performance in handling various natural language processing tasks. Despite having heterogeneous inputs and outputs, generative models transform these tasks into a "text-to-text" format, taking the text as input and producing another text as output Yao et al. (2019); Kim et al. (2020); Yu et al. (2022). Moreover, GMs embed a significant amount of real-world knowledge from pre-training Ye et al. (2022); Chen et al. (2022); Li et al. (2024), which holds potential benefits for KGC tasks.

However, GM-based methods are sensitive to

Figure 1: Examples of the comparative effects of _no contextual facts, scattered contextual facts, and organized contextual facts_. The timestamp only exists in TKGC.



contextual facts on KG (Kim et al., 2023). As illustrated in Figure 1, when the reasoning process lacks contextual facts, the flexibility of the model will be invoked, resulting in significant differences in the generated results. Therefore, it is necessary to gather effective contextual facts to guide the model's generation. However, scattered contextual facts can make it difficult for the generative model to understand thereby generating erroneous results. Therefore, organizing scattered contextual facts into structured and logical contextual facts is an important challenge of GM-based methods for various KGC tasks.

To address the important challenge, we propose a **CO**ntextual Fact\({}^{\textbf{S}}\)**GuIded\({}^{\textbf{G}}\)eneratioN (COSIGN) model to infer missing facts on KGs. First, to enhance the inference ability of the generative model, we designed a contextual facts collector. Given a triple \((s,r,o,m)\), the collector collects all the paths that the head entity \(s\) can infer to the tail entity \(o\) in the subgraph as the contextual facts. Second, a contextual facts organizer is proposed, which replaces inefficient manual organization works with automated information organization, to learn the organized capabilities of LLMs through knowledge distillation (Hsieh et al.,2023; Yu et al.,2023). Finally, the organized contextual facts as the input of the inference generator to generate missing facts. Additionally, we add prefix constraints to our model's inference process to ensure the rationality of the generated results. The contributions of our work can be summarized as follows:

* Inspired by human-like retrieval behavior, we have learned a contextual facts collector based on positive set confidence dominance, which greatly enhances the ability to generate contextual facts.
* We have designed a contextual facts organizer to furnish the necessary basis for inference. To the best of our knowledge, our work is the first to explore how to distillate the LLM to achieve the KGC.
* Extensive experiments verify the effectiveness of each module and its superiority over state-of-the-art baselines.

## 2 Related Works

### Embedding KGC models

Early methods such as TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), and ComplEx (Trouillon et al., 2016) achieved success by representing entities and relations as continuous vectors in a low-dimensional space. To address the limitations of these early models, neural network architectures like ConvE (Dettmers et al., 2018) and CompGCN (Sun et al., 2019) were introduced, utilizing neural networks to capture complex patterns in the graph structure. Recent research has explored integrating external information and meta-paths into embedding models (Dong et al., 2017; Zhang et al., 2020). Additionally, TTransE (Leblay and Chekol, 2018), TComplEx (Xu et al., 2020b) extended the models to incorporate temporal information into relation embeddings, and OG-NET (Zheng et al., 2022) embedded the spatial information into representations. TeLM (Xu et al., 2021) introduced a novel linear time regularization function to enhance temporal features. In contrast, LCGE (Niu and Li, 2023) established models for the timeliness and causal relationships of facts, recognizing their time sensitivity in inferring missing information. However, these methods are tailored to specific data and tasks, limiting their generality.

### Generative KGC Models

GenKGC (Xie et al., 2022) transforms the KGC task into a sequence-to-sequence generation task, utilizing generative models to enhance generalization capability. KGT5 (Saxena et al., 2022) leverages the generality of the generation model and employs the prompt methodology to concurrently address KGC tasks. SQUIRE (Bai et al., 2022) employs historical paths as context to enhance generative model performance. However, their applicability is limited to static KGC. KG-S2S (Chen et al., 2022) unifies input formats across various types of KGs to address Temporal Knowledge Graph Completion (TKGC) tasks. However, it does not incorporate context, leading to limited performance.

## 3 Preliminaries

**Knowledge Graph Completion.** Let \(\mathcal{E}\), \(\mathcal{R}\), \(\mathcal{T}\), and \(\mathcal{F}\) denote a finite set of entities, relations, timestamps, and facts, respectively. a fact list \(\mathcal{F}=\left\{\left(s,r,o,m\right)_{1},...,\left(s,r,o,m\right)_{n}\right\}\) where \(s\), \(o\in\mathcal{E}\) is subject and object entity, \(r\in\mathcal{R}\) is the tuple relation and \(m\) represents the representational form under different types of KG configurations. For instance, in the context of SKGC, \(m\) denotes null because it does not consider the time. However, in TKGC, it signifies a timestamp and \(m\in\mathcal{T}\). Knowledge Graph Completion (KGC) predicts the missing en 

tities for the queries \((s,r,?,m)\) or \(\left(?,r^{-1},o,m\right)\), where \(r^{-1}\in\mathcal{R}^{-1}\) is the reverse edge of r.

**Generative Knowledge Graph Completion.** The generative models have transformed the representation of KGC Xie et al. (2022); Saxena et al. (2022). The typical generative models of an encoder and a decoder, which can be viewed as \(P\left(y|x\right)=\prod_{i=1}^{[y]}P\left(y_{i}|y_{<i},x\right)\), where \(x\) is the input sequence and \(y\) is the generated output sequence. In generative KGC, \(x\) represents the query \(\mathcal{Q}=(s,r,?,m)\) (or \(\left(?,r^{-1},o,m\right)\)), and the predicted ground-truth object (or subject) as \(y\). It is noteworthy that both \(x\) and \(y\) are flattened textual representations of knowledge Chen et al. (2022).

## 4 Proposed Method

Figure 2 shows there are three key technical components in COSIGN: the contextual facts collector, the contextual facts organizer, and the inference generator. Please note that we take the temporal KGC task as an example when introducing the method, but our method also applies to the static KGC task and few-shot KGC.

### Contextual Facts Collector

Collecting information relevant to the query \((s_{q},r_{q},?,m_{q})\) as the context for inference has traditionally been accomplished using a neighborhood sampling strategy Ding et al. (2022); Wei et al. (2023). However, random sampling can overlook contextual facts and lack an understanding of factual details. Therefore, there is a need to design an efficient and context-aware method for collecting contextual facts to achieve human-like retrieval behavior.

**Pruning Neighborhood.** Intuitively, human memory primarily focuses on the most recent events Li et al. (2021). Thus, we extract the \(l\)-th order neighborhood subgraph \(\mathcal{N}\) of the query \((s_{q},r_{q},?,m_{q})\) to serve as the background,

\[\begin{split}&\mathcal{N}_{(s_{q},r_{q},m_{q},l)}=\left\{\left( s_{q},r_{j}^{1},n_{i}^{1},m_{q}^{1}\right),\left(n_{i}^{1},r_{j}^{2},n_{ i}^{2},m_{t}^{2}\right)\right.\\ &\qquad\quad\left.,...,\left(n_{i}^{l-1},r_{j}^{l},n_{i}^{l},m_{ t}^{l}\right)\right\},0\leq i<|\mathcal{E}|\\ &\qquad\quad,0\leq j<|\mathcal{R}|\,,0\leq t<|\mathcal{T}|\end{split} \tag{1}\]

where \(n_{i}\) represents the neighboring nodes.

**Filterting Information.** Next, we filter query-relevant facts within a subgraph using our proposed information filter module. Previous methods predominantly involved setting similarity thresholds for filtering, leading to challenges in adapting to different KGs Wang and Yu (2023); Yu et al. (2024b). Therefore, we utilize a generative model for a nuanced understanding of facts, enabling rapid adaptation to filter information across diverse KGs. Specifically, we employ a template \(X_{k}\) to textually represent both the query and the facts within the neighborhood subgraph \(\mathcal{N}\):

\[\begin{split} X_{k}&=\texttt{Query}\colon\left(s_{ q},r_{q},?,m_{q}\right)\texttt{Fact}\colon\left(s_{i},r_{j},o_{i},m_{t} \right)\\ &\quad 0\leq i<|\mathcal{E}|\,,0\leq j<|\mathcal{R}|\,,0\leq t<| \mathcal{T}|\end{split} \tag{2}\]

where \((s_{i},r_{j},o_{i},m_{t})\) represents a specific fact in the neighborhood subgraph \(\mathcal{N}\).

Figure 2: An overview of the COSIGN. The collector emulates human-like retrieval behavior, the organizer arranges contextual facts, and the inference generator generates answers based on well-organized contextual facts. Here, we use temporal KGC as an illustration.

 

Then, the template \(X_{k}\) is input into the information filter module, which outputs \(K\in\left\{\textit{yes},\textit{no}\right\}\). _yes_ indicates a positive sample related to the query, while _no_ represents a negative sample. It is crucial to emphasize that instances in which entities missing in the query are present in the facts are considered positive samples, while conversely, they constitute negative samples. Our learning objective is to maximize the likelihood of the token \(k_{i}\) given the input text \(X_{K}\) and the tokens \(k_{<i}\) in the base class \(K\), and to define the loss function as follows:

\[\mathcal{L}_{k_{0}}=-\underset{i=1}{\overset{|K|}{\sum}}\mathrm{log}P_{k_{0}} \left(k_{i}|k_{<i},X_{k}\right) \tag{3}\]

where \(P_{k_{0}}\left(k_{i}|k_{<i},X_{k}\right)\) is the log-likelihood of the \(i\)-th token of the ground class \(K\).

However, facts with high literal overlap but different semantics pose challenges for generative models. This is because, under the guidance of teacher forcing, the similarity of these facts is forcibly reduced, leading to overfitting. To tackle this issue, we propose a training approach based on a positive-set confidence advantage to diminish model sensitivity during learning. Specifically, we seek to ensure that the probability of the positive sample set outputting _yes_ surpasses that of the negative sample set,

\[\mathcal{L}_{con}=\log\left(1+\sum_{i\in\Omega_{neg},j\in\Omega_{pos}}e^{ \lambda(s_{i}-s_{j})}\right) \tag{4}\]

where \(\lambda\) is a margin value, which is detailed in the experimental section regarding its configuration. \(\Omega_{neg}\) refers to the negative sample set, and \(\Omega_{pos}\) represents the positive sample set. \(s\) represents the similarity score.

Please note that if \(\lambda(s_{i}-s_{j})\) is an extremely large negative number, it will cause the training gradient to become zero. Therefore, we have incorporated the negative log-likelihood loss \(\mathcal{L}_{k_{0}}\) to prevent such overfitting scenarios, as expressed by \(\mathcal{L}_{k_{0}}+\mathcal{L}_{con}\).

**Collecting Contextual Facts.** The logic of a single fact lacking inferential reasoning is akin to O. Henry's novels. If only a few keywords are skimmed, it is difficult to understand the twist at the end. However, the path of facts can intuitively describe the process of events. Therefore, by seeking the path between queries and facts, the completeness of information can be achieved. Here, we are seeking the shortest path, as it represents the most direct association of facts. The shortest path between query \(q\) and fact \(F_{i}\) is as follows:

\[\nu_{(q,\mathcal{F}_{i})}=\sigma\left(\mathcal{N}_{q},(s_{q},m_{q}),(s_{i},m_{ t})\right) \tag{5}\]

where \(\sigma\) is the shortest path function, and in this paper, Dijkstra's method Jurkiewicz et al. (2021); Yu et al. (2022b) is used. \(\mathcal{N}_{q}\) is the \(l\)-th order neighborhood subgraph of the query. Subsequently, all identified paths for positive sample facts will be utilized as contextual facts.

### Contextual Facts Organizer

As shown in Figure 1, only the coherent contextual fact is helpful to various KGC tasks. So, this part organizes scattered contextual facts into coherent contextual facts.

**Learning Organization.** The LLM has a good ability to organize, so we need to guide the LLM to generate organized logic. Due to the organized contextual facts is organized from the scattered contextual facts, we can obtain organized contextual facts by using reverse generation thinking. Specifically, we let the LLM generate the contextual facts \(C\) in reverse based on the query, scattered contextual facts, and answer. To facilitate the LLM's comprehension of the task's intent, the prompt should encompass three sections: task description, solution conditions, and generation constraints Yang et al. (2023), as follows:

_Your task is to summarize the relevant information related to the answer from the given context._

_I have a query (\(s_{q}\), \(r_{q}\), ?, \(m_{q}\)), an answer (o), and a series of contexts: SCFs._

_Please provide concise and coherent answers._ where _SCFs_ represents the scattered contextual facts collected in Sec 4.1. To generate reliable \(C\), we designed an evaluator to assess the recall of \(o\) in \(C\), and based on the correction approach proposed in Zhou et al. (2022), return instances with low recall to the LLM for modification.

**LLM Distillation.** During the inference stage, the answer is unknown, resulting in the inability of LLMs to generate coherent logic. However, inspired by the knowledge distillation Brown et al. (2023), it is possible to leverage a small language model (SLM) to inherit the capability of generating logical coherence from the LLM. Specifically, based on the given query and scattered contextual facts, the SLM is trained to learn the organizational 

ability of the LLM as the contextual facts organizer.

\[X_{c}= \texttt{Query:}\left(s_{q},r_{q},?,m_{q}\right) \tag{6}\] \[\texttt{Facts:~SCFs}\]

The optimization procedure will be taken as an estimate of the parameters with log-likelihood maximization as follows:

\[\mathcal{L}_{c_{\theta}}=-\sum_{i=1}^{|C|}\texttt{log}P_{c_{\theta}}\left(c_{ i}|c_{<i},X_{c}\right) \tag{7}\]

where \(P_{c_{\theta}}\left(c_{i}|c_{<i},X_{c}\right)\) is the log-likelihood of the \(i\)-th token of the contextual facts \(C\).

### Inference Generator

For the generator, the input \(X_{g}\) is formed by concatenating the query with the contextual facts \(C\). The output is required to infer the word to fill in the [MASK] token, and the words are further mapped to corresponding labels through an expression mechanism Cui et al. (2022). The input template is as follows:

\[X_{g}= \texttt{Query:}\left(s_{q},r_{q},\texttt{[MASK]},m\right) \tag{8}\] \[\texttt{Context:}~{}~{}C\]

where \(C\) signifies the contextual facts organized in Sec 4.2. The text of the object entity, denoted as \(Y=\left(y_{1},y_{2},...\right)\), can also be optimized through an estimation using log-likelihood maximization:

\[\mathcal{L}_{g_{\theta}}=-\sum_{i=1}^{|Y|}\texttt{log}P_{g_{\theta}}\left(y_{ i}|y_{<i},X_{g}\right) \tag{9}\]

where \(P_{g_{\theta}}\left(y_{i}|y_{<i},X_{g}\right)\) is the log-likelihood of the \(i\)-th token of the object entity \(Y\).

In addition, the beam search and prefix constraints techniques play a crucial role in generating coherent and plausible results.

**Beam Search.** For KGC tasks, given a query \(\left(s_{q},r_{q},?,m_{q}\right)\) or \(\left(?,r_{q}^{-1},o_{q},m_{q}\right)\), multiple valid entities can be generated. So, we adopt the standard beam search algorithm Cui et al. (2022) with a beam width of \(B\), which can generate different entity texts with high probability in each beam.

**Prefix Constraints.** Flexible auto-regressive generation can result in the generation of entities that are not present in the set \(\mathcal{E}\). To address this issue, we propose the utilization of prefix constraints Takeno et al. (2017) to govern the COSIGN decoder, ensuring the generation of valid tokens given a prefix sequence.

## 5 Experiment

We combined PyTorch Paszke et al. (2019); Yu et al. (2024) with the HuggingFace Wolf et al. (2020) Transformers library to implement COSIGN, and evaluated its performance on an AMD EPYC 7T83 CPU (64 cores) and two RTX-4090 GPUs (24GB each).

### Experimental Setup

#### 5.1.1 Datasets

We utilized three distinct types of datasets, namely SKGC, TKGC, and FKGC, to evaluate the performance of COSIGN. The SKGC dataset comprises WN18RR Toutanova and Chen (2015), FB15K-237 Schlichtkrull et al. (2018), and FB15K-237N Lv et al. (2022). Meanwhile, TKGC and FKGC datasets consist of ICEWS14 Garcia-Duran et al. (2018) and NELL-One Xiong et al. (2018), respectively. Statistics for these datasets are presented in Table 1. We employ the commonly used metrics of Mean Reciprocal Rank (MRR) and Hits@n (n \(\in\left\{1,3,10\right\}\)) to evaluate the outcomes of KGC.

#### 5.1.2 Baseline Methods

We select two types of state-of-the-art baseline models for comparison:

(1) The previous well-performed knowledge graph-embedding models (KGEs), including TransE Bordes et al. (2013), DistMult Yang et al. (2014), ComplEx Trouillon et al. (2016), GMatching Xiong et al. (2018), HyTE Dasgupta et al. (2018), ConvE Dettumers et al. (2018), RotatE Sun et al. (2018), TComplEx Lacroix et al. (2019), MetaR Chen et al. (2019), CompGCN Sun et al. (2019), DE-SimplE Goel et al. (2020), ATiSE Xu et al. (2020b), Tero Xu et al. (2020a), MTransHNiu et al. (2021), T+TransE Han et al. (2021), sToKE Gao et al. (2023), LCGE Niu and Li (2023).

(2) Existing advanced generative models (GMs): including KG-BERT Yao et al. (2019), MTL-KGC Kim et al. (2020), StatR Wang et al. (2021), KGT5 Saxena et al. (2022), GenKGC Xie et al. (2022),

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Datasets & Type & \(|\mathcal{E}|\) & \(|\mathcal{R}|\) & \(N_{train}\) & \(N_{valid}\) & \(N_{test}\) \\ \hline WN18RR & SKGC & 40,943 & 11 & 86,835 & 3,034 & 3,134 \\ FB15K–237 & SKGC & 14,541 & 237 & 27,2115 & 17,535 & 20,466 \\ FB15k-237N & SKGC & 14,541 & 93 & 87,282 & 7,041 & 8,226 \\ ICEW144 & TKGC & 6,869 & 230 & 72,826 & 8,941 & 8,963 \\ NELL-One & FKGC & 68,544 & 358 & 189,635 & 1,004 & 2,158 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the experimental datasets.

 

PKGC Lv et al. (2022), SimKGC Wang et al. (2022), KG-S2S Chen et al. (2022).

Additionally, the baseline comparisons in this paper are derived from the recommended values of the above-mentioned method.

### Experimental Results

**Static KGC.** The experimental results are shown in Table 2. On WN18RR, FB15K-237, and FB15K-237N, COSIGN achieved competitive performance. Compared to the latest generative model KG-S2S, COSIGN significantly outperforms it. Specifically, we observe a relative improvement in Hit@1 for WN18RR by 7.9% (from 0.531 to 0.610), for FB15K-237 by 5.8% (from 0.257 to 0.315), and for FB15K-237N by 7.3% (from 0.355 to 0.282). This confirms the effectiveness of incorporating organizational context. We were surprised to discover that, for the non-CPR dataset FB15K-237N, the improvement was more substantial compared to the CPR dataset FB15K-237 (non-CPR 7.3% > CPR 5.8%). This could be a result of the organizational context inadvertently enriching the semantics of non-CPR. In comparison with the graph embedding-based approach, COSIGN consistently achieves performance improvements on WN18RR, albeit maintaining moderate results on FB15K-237 and FB15K-237N.

**Temporal KGC.** To assess COSIGN's ability to handle temporal aspects in knowledge graphs, we conducted experiments on the TKGC benchmark ICEWS14. The results are presented in Table 3. Our proposed COSIGN achieved new

\begin{table}
\begin{tabular}{c|c|c c c} \hline \hline  & \multicolumn{2}{c|}{Models} & \multicolumn{2}{c}{NELL-One} \\  & MRR & H@1 & H@3 & H@10 \\ \hline \multirow{8}{*}{KGEEs} & GMatching (TransE) & One & 0.120 & 0.260 \\  & GHatching (DistMult) & One & 0.110 & 0.300 \\  & GMatching (ComplEx) & One & 0.120 & 0.310 \\  & GMatching (ComplEx) & Five & 0.140 & 0.310 \\  & MetaR & One & 0.170 & 0.400 \\  & MetaR & Five & 0.170 & 0.440 \\  & MTransH & Five & 0.210 & 0.480 \\ \hline \multirow{4}{*}{GMs} & STAR & Zero & 0.170 & 0.450 \\  & KG-S2S & Zero & 0.220 & 0.490 \\ \cline{1-1}  & Ours & Zero & **0.240** & **0.500** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The FKGC experiment results. NELL-One results are taken from Chen et al. (2022). The best-performing baseline is denoted as KG-S2S.

\begin{table}
\begin{tabular}{l|c|c c c c|c c c c|c c c c} \hline \hline \multirow{2}{*}{Models} & \multicolumn{4}{c|}{WN18RR} & \multicolumn{4}{c|}{FB15K-237} & \multicolumn{4}{c}{FB15K-237N} \\  & MRR & H@1 & H@3 & H@10 & MRR & H@1 & H@3 & H@10 & MRR & H@1 & H@3 & H@10 \\ \hline \multirow{8}{*}{KGEEs} & TranE & 0.243 & 0.043 & 0.441 & 0.532 & 0.279 & 0.198 & 0.376 & 0.441 & 0.255 & 0.152 & 0.301 & 0.459 \\  & DistMult & 0.444 & 0.412 & 0.470 & 0.504 & 0.281 & 0.199 & 0.301 & 0.446 & 0.209 & 0.143 & 0.234 & 0.330 \\  & ComplEx & 0.449 & 0.409 & 0.469 & 0.530 & 0.278 & 0.194 & 0.297 & 0.450 & 0.249 & 0.180 & 0.276 & 0.380 \\  & Rotate & 0.476 & 0.428 & 0.492 & 0.571 & 0.338 & 0.241 & 0.375 & 0.533 & 0.279 & 0.177 & 0.320 & 0.481 \\  & ConvE & 0.456 & 0.419 & 0.470 & 0.531 & 0.312 & 0.225 & 0.341 & 0.497 & 0.273 & 0.192 & 0.305 & 0.429 \\  & CompGCN & 0.479 & 0.443 & 0.494 & 0.546 & 0.355 & 0.264 & 0.390 & **0.535** & 0.316 & 0.231 & 0.349 & 0.480 \\ \hline \multirow{8}{*}{GMS} & KG-BERT & 0.216 & 0.041 & 0.302 & 0.524 & - & - & - & 0.420 & 0.203 & 0.139 & 0.201 & 0.403 \\  & MTL-KGC & 0.331 & 0.203 & 0.383 & 0.597 & 0.267 & 0.172 & 0.298 & 0.458 & 0.241 & 0.160 & 0.284 & 0.430 \\  & StatR & 0.401 & 0.243 & 0.491 & 0.709 & 0.296 & 0.205 & 0.322 & 0.482 & - & - & - \\  & PKGC & - & - & - & - & - & - & - & 0.307 & 0.232 & 0.328 & 0.471 \\  & GenKG & - & 0.287 & 0.403 & 0.535 & - & 0.192 & 0.355 & 0.439 & - & - & - \\  & KIT5 & 0.508 & 0.487 & - & 0.544 & 0.276 & 0.210 & - & 0.414 & - & - & - \\  & SimKGC & **0.671** & 0.588 & **0.731** & **0.817** & 0.336 & 0.2ately & 0.365 & 0.511 & - & - & - \\  & KG-S2S & 0.574 & 0.531 & 0.595 & 0.661 & 0.336 & 0.257 & 0.373 & 0.498 & 0.353 & 0.282 & 0.385 & 0.495 \\  & Ours & 0.641 & **0.610** & 0.654 & 0.714 & **0.368** & **0.315** & **0.434** & 0.520 & **0.394** & **0.355** & **0.457** & **0.526** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The SKGC experiment results. Building upon the work of Chen et al. (2022), we have incorporated SimKGC Wang et al. (2022). The best-performing method results are in bold and the second best results are in underline.

\begin{table}
\begin{tabular}{c|c|c c c} \hline \hline \multirow{2}{*}{Models} & \multicolumn{4}{c}{ICEWS14} \\  & MRR & H@1 & H@3 & H@10 \\ \hline \multirow{8}{*}{KGEEs} & TransE & 0.255 & 0.074 & - & 0.601 \\  & HyTE & 0.297 & 0.108 & 0.416 & 0.655 \\  & ATISE & 0.550 & 0.436 & 0.629 & 0.750 \\  & DE-SimplEx & 0.526 & 0.418 & 0.592 & 0.725 \\  & Tero & 0.526 & 0.468 & 0.621 & 0.732 \\  & TCompIIEx & 0.560 & 0.470 & 0.610 & 0.730 \\  & T+Transis & 0.553 & 0.437 & 0.627 & 0.765 \\  & sToKE & 0.659 & 0.574 & 0.693 & 0.803 \\  & LCGE & 0.667 & 0.588 & 0.714 & 0.815 \\ \hline \multirow{2}{*}{GMs} & KG-S2S & 0.595 & 0.516 & 0.642 & 0.737 \\  & Ours & **0.689** & **0.645** & **0.720** & **0.821** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The TKGC experiment results. Building upon the work of Chen et al. (2022), we have incorporated the latest stoKE Gao et al. (2023) and LCGE Niu and Li (2023) models. The best-performing baseline is denoted as LCGE.

 

state-of-the-art results in MRR, Hit@1, Hit@3, and Hit@10, confirming COSIGN's capability to learn additional temporal features from pure textual forms. We observed a relative improvement of 5.7% in Hit@1 (from 0.588 to 0.645), the most significant enhancement among recent methods. This can be attributed to the rich semantic associations in the context of ICEWS14, where COSIGN's contextual organization ability played a crucial role. We believe that as global knowledge continues to grow, semantically rich scenarios will become mainstream, providing further opportunities for COSIGN to excel.

**Few-shot KGC.** Finally, we validated the few-shot learning capability of Hit@1 on NELL-One, as shown in Table 4. We opted for the same zero-shot setting as Chen et al. (2022) (i.e., evaluation relations not occurring in the training set). Surprisingly, COSIGN outperformed all variants of previous graph embedding-based models that transferred knowledge from training data to evaluation relations (i.e., one-shot and five-shot learning). Additionally, compared to the latest generative model KG-S2S, COSIGN achieved a 2% improvement in Hit@1 (from 0.220 to 0.240) and a 1% improvement in Hit@10 (from 0.490 to 0.500). In terms of metrics, the performance improvement of COSIGN in the NELL-One scenario is limited compared to the SKGC and TKGC scenarios. This could be attributed to the presence of unseen relations in NELL-One, leading to the generation of poor-quality contextual facts that impact COSIGN's organizational ability. Nevertheless, it underscores the significance of high-quality contextual facts for reasoning.

### Ablation Study

We validate the effectiveness of each module in COSIGN across three types of datasets: WN18RR (SKGC), ICEWS14 (TKGC), and NELL-One (FKGC). "w/o CFC" or "w/o CFO" denotes the removal of the contextual facts collector (CFC) or the contextual facts organizer (CFO), respectively. The symbol \(\triangle\) represents the performance decrease when removing a specific module compared to the overall performance. Evaluations are conducted based on two variants of the generator: one utilizing small-model training (with T5-base as an example), and the other relying on non-training with a large model (with GPT3.5 as an example).

First, apply CFC and CFO to the trained generator. The experimental results are shown in Table 5. For COSIGN as a whole, both are crucial. However, the cost of losing CFC is more significant (Hit@1 decreases by 9.5% and 14.9% in WN18RR and ICEWS14, respectively). Meanwhile, we observe that CFO performs better in NELL-One compared to WN18RR and ICEWS14. This may be attributed to the rich semantic information present in WN18RR and ICEWS14, leading to the emergence of some logical implications. Additionally, the generator training process pays attention to useful features, implying that the training process itself serves as a form of data organization. This confirms why CFO's performance contribution is not as significant as CFC's.

Next, apply CFC and CFO to the non-trained generator. The experimental results are shown in Table 6. For COSIGN as a whole, both are crucial. Without CFC, the generator cannot rely on context for reasoning, resulting in a decrease of 31.4% and 27.0% in Hit@1 on WN18RR and ICEWS14, respectively. Removing CFO leads to the generator relying solely on scattered contextual facts generated by CFC, causing significant performance fluctuations with a decline of 13.9% and 13.6% in Hit@1 on WN18RR and ICEWS14, respectively. Due to the lack of samples, the non-trained generator struggles to form internal data organizational logic. Therefore, the contribution of the CFO is more pronounced on the non-trained

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline \multirow{2}{*}{Ablation} & \multicolumn{2}{c|}{WN18RR} & \multicolumn{2}{c|}{ICEWS14} & \multicolumn{2}{c}{NELL-One} \\  & H@1 & H@10 & H@1 & H@10 & H@1 & H@10 \\ \hline C-GPT & 0.576 & 0.693 & 0.624 & 0.781 & 0.180 & 0.460 \\ \hline w/o CFC & 0.262 & 0.387 & 0.354 & 0.413 & 0.100 & 0.280 \\ \(\bigtriangleup\) & **31.4\%** & **30.6\%** & **27.0\%** & **36.8\%** & **8.0\%** & **18.0\%** \\ \hline w/o CFO & 0.437 & 0.495 & 0.488 & 0.579 & 0.130 & 0.350 \\ \(\bigtriangleup\) & **13.9\%** & **19.8\%** & **13.6\%** & **20.2\%** & **5.0\%** & **11.0\%** \\ \hline \end{tabular}
\end{table}
Table 6: Application of ablation results on the non-trained generator. C-GPT refers to the generator of COSIGN utilizing the non-trained GPT3.5 model.

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline \multirow{2}{*}{Ablation} & \multicolumn{2}{c|}{WN18RR} & \multicolumn{2}{c|}{ICEWS14} & \multicolumn{2}{c}{NELL-One} \\  & H@1 & H@10 & H@1 & H@10 & H@1 & H@10 \\ \hline C-T5 & 0.610 & 0.714 & 0.645 & 0.821 & 0.240 & 0.500 \\ \hline w/o CFC & 0.515 & 0.647 & 0.496 & 0.708 & 0.200 & 0.440 \\ \(\bigtriangleup\) & **9.5\%** & **6.7\%** & **14.9\%** & **11.3\%** & **4.0\%** & **6.0\%** \\ \hline w/o CFO & 0.596 & 0.692 & 0.632 & 0.800 & 0.220 & 0.470 \\ \(\bigtriangleup\) & **1.4\%** & **2.2\%** & **1.4\%** & **2.1\%** & **2.0\%** & **3.0\%** \\ \hline \end{tabular}
\end{table}
Table 5: Application of ablation results on the trained generator. C-T5 refers to the generator of COSIGN utilizing the trained T5 model Raffel et al. (2020).

 

generator compared to the trained generator.

The overall experiments indicate that the inference of the generator not only requires contextual facts but also benefits significantly from logically organized contextual facts.

### Case Study

The real-world case studies in Figure 3 demonstrate COISGN's information organization capability through two types of KGC. Taking TKGC as an example, given a query _(Citizen (Nigeria), Make statement, ?, 2014-09-02)_, through the collector, scattered contextual facts, i.e., facts lacking strong internal logical connections, are obtained. The reason for inference failure using it is that the inference model may perceive _Make statement_ and _Express intent to settle dispute_ as more similar, leading to incorrect results, such as _Count Judge (Nigeria)_. When the organizer further refines the scattered contextual facts into logically connected contextual facts, the inference model gains a better understanding that the key supporting point for the background is _Daily Trust_, thereby enhancing the performance of the inference model.

### Model Analysis

We conducted a detailed analysis of the effectiveness of COISGN in terms of model component performance, hyperparameter settings, and runtime efficiency.

Firstly, we chose two generative models, T5-base and BART-base Lewis et al. (2019), as well as a classification model RoBERTa Liu et al. (2019), to validate the effectiveness of our proposed positive set confidence. The results are illustrated in Figure 4(a). In comparison to typical training methods, our approach led to further improvements in the performance of three models, especially achieving an average recall increase of 4% on ICEWS14. Secondly, as indicated by Figure 4(b), in the absence of answer references, the distilled performance of T5 surpasses that of GPT3.5 Wang et al. (2023). This improvement is attributed to the powerful organizational capabilities through the distillation process. Furthermore, based on Figure 4(c), it was observed that achieving performance close to optimal levels requires only about 40% of the training data volume. Thirdly, experimental results depicted in Figure 4(d) reveal that the inclusion of prefix constraints leads to an average improvement of 1.2%. Although the enhancement is modest, this simple calculation successfully constrains the rationality of the outcomes. Finally, we observed a

Figure 4: The effects of COSIGN. ”-PSA” indicates the training method using positive-set confidence advantage, while “-NLL” denotes the training method using typical negative log-likelihood. ”-PC” refers to the results generated using prefix constraints, while ”-GD” denotes the results generated directly.

Figure 3: Cases of TKGC and SKGC, inferred through organized contextual facts.

 

stable trend starting from a beam width of 40, as illustrated in Figure 4(e).

## 6 Conclusion

In this paper, we propose a generative model, called COSIGN, for various KGC tasks. Different from previous methods, we do not directly use contextual facts to guide the model to infer missing facts. In contrast, we use the organizational ability of the LLM to transform the collected scattered contextual facts into coherent contextual facts, and use coherent contextual facts to guide the model to infer missing facts. Specifically, we designed a contextual facts collector to achieve human-like retrieval behavior. Then, a contextual facts organizer is proposed to learn the organized capabilities of LLMs through knowledge distillation. Finally, the organized contextual facts as the input of the inference generator to generate missing facts. Experimental results on five datasets for three types of KGC tasks highlight its significant performance.

## 7 Limitations

While COSIGN effectively leverages contextual facts to enhance the performance of the reasoning model, it fundamentally relies on the richness of the data. When the context in the data is too sparse, it significantly impacts the model's performance. An ablation study conducted on the NELL-One dataset (Section 5.2 Few-shot experiments) also indicates that performance is greatly affected in situations of semantic sparsity in the data. in fact, it may even degrade fact verification performance. Addressing this, in the future, we intend to rely more on data that is both effective and covers a broader semantic context, thereby mitigating the performance fluctuations caused by semantic sparsity.

## 8 Ethics Statement

In this work, we aim to develop a contextual information retrieval system based on COSIGN, providing users with efficient and accurate retrieval results, including support for retrieving knowledge graphs of various types available on the internet. Our information retrieval system based on COSIGN is designed to assist users in automatically avoiding misleading information and mitigating risks associated with the propagation of misinformation. When COSIGN is misused for unconventional operations, it may lead to the widespread exploitation and dissemination of information, potentially causing social unrest. Given this concern, the responsible development of information retrieval systems is crucial to mitigate the risks of information leakage and maintain the integrity of information dissemination.

## Acknowledgements

This work was supported by National Natural Science Foundation of China Youth Fund (No.62302287), the National Key Research and Development Program of China (2021YFC3300602), and the Shanghai Committee of Science and Technology, China (Grant No.23ZR1423500).

## References

* S. Bai, X. Lv, J. Li, L. Hou, Y. Qu, Z. Dai, and F. Xiong (2022)Source: A sequence-to-sequence framework for multi-hop knowledge graph reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1649-1662. Cited by: SS1.
* H. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013)Translating embeddings for modeling multi-relational data. Advances in neural information processing systems26, pp. . External Links: Document Cited by: SS1.
* N. Brown, A. Williamson, T. Anderson, and L. Lawrence (2023)Efficient transformer knowledge distillation: a performance review. arXiv preprint arXiv:2311.13657. Cited by: SS1.
* E. Cambria, Q. Liu, S. Decherchi, F. Xing, and K. Kwok (2022)Senticnet 7: a commonsense-based neurosymbolic AI framework for explainable sentiment analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC, pp. 3829-3839. Cited by: SS1.
* C. Chen, Y. Wang, B. Li, and K. Lam (2022)Knowledge is flat: a seq2seq generative framework for various knowledge graph completion. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 4005-4017. Cited by: SS1.
* M. Chen, W. Zhang, W. Zhang, Q. Chen, and H. Chen (2019)Meta relational learning for few-shot link prediction in knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4217-4226. Cited by: SS1.
* G. Cui, S. Hu, N. Ding, L. Huang, and Z. Liu (2022)Prototypical verbalizer for prompt-based few-shot tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 7014-7024.

 

* D. Sankar Dasgupta, S. Nath Ray, and P. Talukdar (2018)Hyte: hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the conference on empirical methods in natural language processing, pp. 2001-2011. Cited by: SS4.
* T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel (2018)Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: SS5.
* Z. Ding, Y. Ma, B. He, Z. Han, and V. Tresp (2022)A simple but powerful graph encoder for temporal knowledge graph completion. In NeurIPS 2022 Temporal Graph Learning Workshop, Cited by: SS6.
* Y. Dong, N. C. Chawla, and A. Swami (2017)metapath2vec: scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 135-144. Cited by: SS1, SS2.1.
* Y. Gao, Y. He, Z. Kan, Y. Han, L. Qiao, and D. Li (2023)Learning joint structural and temporal contextualized knowledge embeddings for temporal knowledge graph completion. In Findings of the Association for Computational Linguistics (ACL), pp. 417-430. Cited by: SS1, SS2.1.
* A. Garcia-Duran, S. Dumancic, and M. Niepert (2018)Learning sequence encoders for temporal knowledge graph completion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4816-4821. Cited by: SS1, SS2.1.
* R. Goel, S. Mehran Kazemi, M. Brubaker, and P. Poupart (2020)Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 3988-3995. Cited by: SS1, SS2.1.
* A. Hagberg, P. Swart, and D. S. Chult (2008)Exploring network structure, dynamics, and function using networkx. Technical report Los Alamos National Lab.(LANL), Los Alamos, NM (United States). External Links: Link Cited by: SS1, SS2.1.
* Z. Han, G. Zhang, Y. Ma, and V. Tresp (2021)Time-dependent entity embedding is not all you need: a re-evaluation of temporal knowledge graph completion models under a unified framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8104-8118. Cited by: SS1, SS2.1.
* C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister (2023)Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv e-prints, pp. arXiv=2305. Cited by: SS1, SS2.1.
* S. Ji, S. Pan, E. Cambria, P. Martinen, and S. Yu Philip (2021)A survey on knowledge graphs: representation, acquisition, and applications. IEEE transactions on neural networks and learning systems33 (2), pp. 494-514. Cited by: SS1, SS2.1.
* P. Jurkiewicz, E. Biernacka, J. Domzat, and R. Wojcik (2021)Empirical time complexity of generic dijkstra algorithm. In 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 594-598. Cited by: SS1, SS2.1.
* B. Kim, T. Hong, Y. Ko, and J. Seo (2020)Multi-task learning for knowledge graph completion with pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 1737-1743. Cited by: SS1, SS2.1.
* J. Kim, Y. Kwon, Y. Jo, and E. Choi (2023)KG-gtP: a general framework for reasoning on knowledge graphs using large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9410-9421. Cited by: SS1, SS2.1.
* Timothee Lacroix, Guillaume Obozinski, and Nicolas Usunier (2019)Tensor decompositions for temporal knowledge base completion. In International Conference on Learning Representations, Cited by: SS1, SS2.1.
* J. Leblay and M. Dekol (2018)Deriving validity time in knowledge graph. In Companion Proceedings of the The Web Conference, pp. 1771-1776. Cited by: SS1, SS2.1.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2019)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. Cited by: SS1, SS2.1.
* J. Li, H. Yu, Z. Zhang, X. Luo, and S. Xie (2024)Concept drift adaptation by exploiting drift type. ACM Transactions on Knowledge Discovery from Data18 (4), pp. 1-22. Cited by: SS1, SS2.1.
* Z. Li, X. Jin, S. Guan, W. Li, J. Guo, Y. Wang, and X. Cheng (2021)Search from history and reason for future: two-stage reasoning on temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 4732-4743. Cited by: SS1, SS2.1.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1, SS2.1.
* Xin Lv, Y. Lin, Y. Cao, L. Hou, J. Li, Z. Liu, P. Li, and J. Zhou (2022)Do pretrained models benefit knowledge graph completion? a reliable evaluation and a reasonable approach.

 

Guanglin Niu and Bo Li. 2023. Logic and commonsense-guided temporal knowledge graph completion. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 37, pages 4569-4577.
* Niu et al. (2021) Guanglin Niu, Yang Li, Chengguang Tang, Ruiying Geng, Jian Dai, Qiao Liu, Hao Wang, Jian Sun, Fei Huang, and Luo Si. 2021. Relational learning with gated and attentive neighbor aggregator for few-shot knowledge graph completion. In _Proceedings of the 44th International ACM SIGIR conference on research and development in information retrieval_, pages 213-222.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: an imperative style, high-performance deep learning library. In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_, pages 8026-8037.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Saxena et al. (2022) Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequence knowledge graph completion and question answering. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, pages 2814-2828.
* Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In _The Semantic Web: 15th International Conference, ESWC_, pages 593-607.
* Sun et al. (2018) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2018. Rotate: Knowledge graph embedding by relational rotation in complex space. In _International Conference on Learning Representations_.
* Sun et al. (2019) Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. 2019. A re-evaluation of knowledge graph completion methods. _arXiv preprint arXiv:1911.03903_.
* Takeno et al. (2017) Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. 2017. Controlling target features in neural machine translation via prefix constraints. In _Proceedings of the 4th Workshop on Asian Translation (WAT2017)_, pages 55-63.
* Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In _Proceedings of the 3rd workshop on continuous vector space models and their compositionality_, pages 57-66.
* Trouillon et al. (2016) Theo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In _International conference on machine learning_, pages 2071-2080.
* Wang et al. (2021) Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021. Structure-augmented text representation learning for efficient knowledge graph completion. In _Proceedings of the Web Conference 2021_, pages 1737-1748.
* Wang and Yu (2023) Hongwei Wang and Dong Yu. 2023. Going beyond sentence embeddings: A token-level matching algorithm for calculating semantic textual similarity. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_, pages 563-570.
* Wang et al. (2022) Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, pages 4281-4294.
* Wang et al. (2023) Xiaofei Wang, Hayley M Sanders, Yuchen Liu, Kennarey Seang, Bach Xuan Tran, Atanas G Atanasov, Yue Qiu, Shenglan Tang, Josip Car, Ya Xing Wang, et al. 2023. Chatgpt: promise and challenges for deployment in low-and middle-income countries. _The Lancet Regional Health-Western Pacific_, 41.
* Wei et al. (2023) Xiao Wei, Jianguao Huang, Hang Yu, and Qian Liu. 2023. Ptcspell: Pre-trained corrector based on character shape and pinvin for chinese spelling correction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 6330-6343.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
* Xie et al. (2022) Xin Xie, Ningyu Zhang, Zhoubo Li, Shumin Deng, Hui Chen, Feiyu Xiong, Mosha Chen, and Huajun Chen. 2022. From discrimination to generation: Knowledge graph completion with generative transformer. In _Companion Proceedings of the Web Conference 2022_, pages 162-165.
* Xiong et al. (2018) Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018. One-shot relational learning for knowledge graphs. In _Conference on Empirical Methods in Natural Language Processing_.
* Xu et al. (2021) Chengjin Xu, Yong-Yu Chen, Mojtaba Nayyeri, and Jens Lehmann. 2021. Temporal knowledge graph completion using a linear temporal regularizer and multivector embeddings. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2569-2578 

Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. 2020a. Tero: A time-aware knowledge graph embedding via temporal rotation. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1583-1593.
* Xu et al. (2020b) Chenjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Yazdi, and Jens Lehmann. 2020b. Temporal knowledge graph completion based on time series gaussian embedding. In The Semantic Web-ISWC. 19th International Semantic Web Conference, pp. 654-671. Cited by: SS1.
* Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng (2014) Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. Cited by: SS1.
* C. Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen (2023)Large language models as optimizers. arXiv e-prints, pp. arXiv-2309. External Links: Link Cited by: SS1.
* L. Yao, C. Mao, and Y. Luo (2019)Kgbert: Bert for knowledge graph completion. arXiv e-prints, pp. arXiv-1909. External Links: Link Cited by: SS1.
* H. Ye, N. Zhang, Hui Chen, and Huajun Chen (2022)Generative knowledge graph construction: a review. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1-17. Cited by: SS1.
* H. Yu, J. Li, J. Lu, Y. Song, S. Xie, and Guangquan Zhang (2023)Type-ldd: a type-driven lite concept drift detector for data streams. IEEE Transactions on Knowledge and Data Engineering. Cited by: SS1.
* H. Yu, R. Li, S. Xie, and J. Qiu (2024)Shadow-enlightened image outpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: SS1.
* H. Yu, Z. Liu, and X. Luo (2024)Barely supervised learning for graph-based fraud detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 16548-16557. Cited by: SS1.
* H. Yu, J. Lu, Anjin Liu, Bin Wang, Ruimin Li, and Guangquan Zhang (2022)Real-time prediction system of train carriage load based on multi-stream fuzzy learning. IEEE Transactions on Intelligent Transportation Systems23 (9), pp. 15155-15165. Cited by: SS1.
* H. Yu, J. Lu, and G. Zhang (2022)Continuous support vector regression for nonstationary streaming data. IEEE transactions on cybernetics52 (5), pp. 3592-3605. Cited by: SS1.
* X. Zhang, M. Jiang, Z. Zheng, Xiao Tan, Errui Ding, and Yi Yang (2020)Understanding image retrieval re-ranking: a graph neural network perspective. arXiv preprint arXiv:2012.07620. Cited by: SS1.
* Z. Zheng, X. Wang, N. Zheng, and Y. Yang (2022)Parameter-efficient person re-identification in the 3d space. IEEE Transactions on Neural Networks and Learning Systems. Cited by: SS1.
* Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba (2022)Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. Cited by: SS1.

## Appendix A Implementation Details of the COSGIN

We analyze the details of COSGIN in terms of graph construction, hyperparameter settings, and runtime efficiency.

### Graph Construction

We use the networkx package Hagberg et al. (2008) for creating, manipulating, and storing graphs, primarily involving the design of static and temporal graph types. For static graphs, they are inherently triple structures \((s_{i},r_{j},o_{i})\in\mathcal{F}\), thus easily recognized by networks. However, for temporal graphs, being quadruple structures \((s_{i},r_{j},o_{i},m_{t})\in\mathcal{F}\), they cannot be directly loaded by networkx. Therefore, we establish entity-time pairs, combining entities with the occurrence time to form a new representation, for example: \((s_{i}-m_{t},r_{j},o_{i}-m_{t})\). Here, we introduce self-connected edges \(\hat{r}\) to enable the interconnection of information across different time points, for instance: \((s_{i}-m_{t},\hat{r},o_{i}-m_{<t})\).

For the process of reverse inference, we achieve reverse prediction of the model by constructing the reverse relation \(r^{-1}\), i.e., based on \(\big{(}?,r^{-1},o,m\big{)}\). Each \(r^{-1}\) is an added extension of \(r\) in semantic space, as detailed below:

\[r_{i}^{-1}=r_{i}+|\mathcal{R}|\,,r_{i}\in\mathcal{R} \tag{10}\]

To make it easier for the generative model to distinguish between \(r_{i}^{-1}\) and \(r_{i}\), we add a [by] tag to \(r_{i}\) as the mapping text for \(r_{i}^{-1}\), for example: \(r_{i}\)="Accuse", \(r_{i}^{-1}\) = "[by] Accuse".

### Hyperparameter setting

In our experiments, Throughout the entire training process, the model is optimized using the ADAM function. We set the learning parameters for the contextual facts collector, the contextual facts organizer, and the contextual facts generator respectively, to better adapt to the size and distribution of the dataset.

In terms of _contextual facts collector_, we set the epoch and batch size to \(\{50,64\}\) respectively, 

learning rate to \(\{1e-3,2e-4\}\), input and output length to \(\{60,20\}\) respectively. In terms of _contextual facts summarizer_, we set the epoch and batch size to \(\{30,24\}\) respectively, learning rate to \(1e-3\), input length to \(\{300,360\}\), output length to 200. In terms of _contextual facts generator_, we set the epoch and batch size to \(\{50,64\}\) respectively, learning rate to \(\{2e-4,5e-4\}\), input length to \(\{350,420\}\), output length to 50. The optimal configurations for the collector, summarizer and the generator are presented in Table 7.

We evaluated the impact of the margin value \(\lambda\) on the performance of the contextual facts collector, using recall as the evaluation metric, as shown in Figure 5. Generally, as the margin value \(\lambda\) increases, the model's performance gradually improves. However, when reaching a certain point, the performance reaches its optimal level, or even may decrease. We found that for relatively sparse datasets WN18RR and NELL-One, larger margin values are needed, set at 25 and 20 respectively. This also indicates that their internal semantic distributions are not distinct, and further relaxing the constraints of similarity can improve fault tolerance. For relatively dense datasets FB15K-237 and ICEWS14, only smaller margin values (\(\lambda\)=15) are required to achieve optimal performance for the model, as their internal semantic distinctions are relatively clear.

For the subgraph pruning threshold \(l\), it directly affects the results of the contextual facts collector. We use recall rate as the metric to evaluate the optimal \(l\), as shown in Figure 6. It's noteworthy that as \(l\) increases, the recall rate of correctly included candidates in the subgraph also increases. We observed that when \(l\) is set to 3, the recall rates for all datasets exceed 80%, which is considered sufficient for inference, and the memory resources occupied are also acceptable (with the maximum storage requirement for FB15K-237 being 782MB). Although setting \(l\) to 4 leads to a further increase in the recall rate, the corresponding storage resource exhibits exponential growth, with a maximum occupation of 1728MB for FB15K-237. In order to achieve an effective balance between recall rate and storage resources, in actual experiments, we set \(l\) to 3.

### Runtime Efficiency Analysis

To test the efficiency of COSIGN, we conducted experiments on the WN18RR, FB15K-237, ICEWS14, and NELL-One datasets, as shown in Figure 7. Due to the inherently higher computational cost of generative methods compared to conventional models, it is not feasible to directly compare their efficiency. Therefore, we only provide comparative results with recent generative model KG-S2S. While COSIGN has additional modules for collecting and summarizing contextual information compared to KG-S2S, surprisingly, its runtime efficiency is not significantly different from KG-S2S, with only a marginal increase in average runtime by 149ms. However, under the condition of sacrificing acceptable runtime efficiency, COSIGN's hit@1 performance is on average im

Figure 5: The effects of margin values.

\begin{table}
\begin{tabular}{c c c c c c} \hline  & Parameters & WN18RR & FB15K-237 & ICEWS14 & NELL-One \\ \cline{2-6}  & Epoch & 60 & 60 & 60 & 60 \\ \multirow{4}{*}{Baselines} & Batch,Size & 32 & 32 & 32 & 32 \\  & Learning Rate & Set-4 & 2e-4 & 2e-4 & 5e-4 \\  & Input\_Length & 350 & 350 & 420 & 350 \\  & Output\_Length & 50 & 50 & 50 & 50 \\ \hline \multirow{4}{*}{Baselines} & Epoch & 30 & 30 & 30 & 30 \\  & Batch, Size & 24 & 24 & 24 & 24 \\  & Learning\_Rate & 1e-3 & 1e-3 & 1e-3 & 1e-3 \\  & Input\_Length & 300 & 300 & 360 & 300 \\  & Output\_Length & 200 & 200 & 200 & 200 \\ \hline \multirow{4}{*}{Baselines} & Epoch & 50 & 50 & 50 & 50 \\  & Batch,Size & 64 & 64 & 64 & 64 \\ \cline{1-1}  & Learning\_Rate & 1e-3 & 2e-4 & 2e-4 & 1e-3 \\ \cline{1-1}  & Input\_Length & 60 & 60 & 60 & 60 \\ \cline{1-1}  & Output\_Length & 20 & 20 & 20 & 20 \\ \hline \end{tabular}
\end{table}
Table 7: The model parameters of each component in COSIGN.

Figure 6: The effects of sampling.

 

proved by 7.18% compared to KG-S2S.

## Appendix B LLM Prompt Design

During the distillation process, it is necessary to elicit the internal logic of GPT3.5 through prompts. Therefore, we designed different prompt templates and evaluated their effectiveness on the WN18RR and ICEWS14 datasets using the metric recall rate, as shown in Table 8. For each dataset, we randomly sample 500 test set examples for evaluation. We divided each type of template into two groups, one with answers and one without. Generally, adding answers gives purpose to the information summarization by GPT3.5, thus yielding better results. At the same time, we found that providing explicit tasks and constraints for GPT3.5 from the outset (as in prompt templates 3 and 4) is useful because such instructions clarify the user's intent.

For the non-trained mode of the generator, we designed a prompt to combine queries with context and let GPT3.5 complete confidence scoring for candidate entities, as shown below:

_Given an incomplete tuple (\(s\), \(r\), [MASK], \(m\)), now you will help me complete the entity [MASK] and make the tuple complete._

_I have a candidate entity set and a contextual text available for your reference. Candidate entity set: ES. Contextual text: CT. Please output the confidence scores for each candidate entity, which ranges from 0 to 1. The output format should be: entity1:score1, entity2:score2,...,entityn:screen._

where _ES_ represents the collection of all entities obtained through the contextual facts collector, while _CT_ denotes the results obtained from the contextual facts organizer. When GPT3.5 outputs confidence scores for each candidate entity, we sort the candidates based on confidence scores and evaluate the results using MRR and Hit@n.

Figure 7: The running average times of COSIGN.