Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting
Year: 2023

Abstract:
Knowledge Graph Completion (KGC) often requires both KG structural and textual information to be effective. Pre-trained Language Models (PLMs) have been used to learn the textual information, usually under the fine-tune paradigm for the KGC task. However, the finetuned PLMs often overwhelmingly focus on the textual information and overlook structural knowledge. To tackle this issue, this paper proposes CSProm-KG (Conditional Soft Prompts for KGC) which maintains a balance between structural information and textual knowledge. CSProm-KG only tunes the parameters of Conditional Soft Prompts that are generated by the entities and relations representations. We verify the effectiveness of CSProm-KG on three popular static KGC benchmarks WN18RR, FB15K-237 and Wikidata5M, and two temporal KGC benchmarks ICEWS14 and ICEWS0515. CSProm-KG outperforms competitive baseline models and sets new state-of-the-art on these benchmarks. We conduct further analysis to show (i) the effectiveness of our proposed components, (ii) the efficiency of CSProm-KG, and (iii) the flexibility of CSProm-KG 1.

heading: 1 Introduction
text: Knowledge Graphs (KGs) have both complicated graph structures and rich textual information over the facts. Despite being large, many facts are still missing. Knowledge Graph Completion (KGC) is a fundamental task to infer the missing facts from the existing KG information.
Graph-based KGC models (Bordes et al., 2013; Yang et al., 2015; Dettmers et al., 2018) represent entities and relations using trainable embeddings. These models are trained to keep the connections between entities and relations over structural paths,
∗Corresponding author 1Our source code is available at https://github.
com/chenchens190009/CSProm-KG
and tail entities are inferred via various transitional relations. Despite being effective in modelling KG structural information, these methods are unable to incorporate linguistic context. Recently, pretrained language models (PLMs) are applied to fill up this gap (Yao et al., 2019; Wang et al., 2021a; Xie et al., 2022). The proposed solutions often directly fine-tune the PLMs to choose the correct entities either relying on pure textual context or using structural add-ons as a complementary (Wang et al., 2021a). However, PLMs are normally equipped with large-scale parameters and linguistic inherence obtained from their pre-training stage. As a result, these PLM-based models remain overwhelmingly focusing on the textual information in KGs and tend to overlook the graph structure. For example, given an incompleted fact (Mona Lisa, painted by, ?), the PLM-based models may con-
11489
fuse between Leonardo DiCaprio and Leonardo da Vinci simply because they are textually similar. Thus, in this paper, we focus on the research question: Can we effectively fuse the KG structural information into the PLM-based KGC models?
To this end, we propose a novel CSProm-KG model (Conditional Soft Prompts for KGC) which is a structure-aware frozen PLMs that could effectively complete the KGC task. The core of CSProm-KG is Conditional Soft Prompt that is an structure-aware version of Soft Prompt (Li and Liang, 2021; Lester et al., 2021). Previously, Soft Prompt is a sequence of unconditional trainable vectors that are prepended to the inputs of frozen PLMs. Such design could effectively avoid the issue of over-fitting towards textual information caused by fine-tuning and allow the frozen PLMs to learn the downstream tasks (Wang et al., 2022). However, such naive Soft Prompts cannot represent any structural information in KG. To remedy this, as shown in Figure 1 (c), we propose the prompt vectors conditioned on the KG entities and relations embeddings. Specifically, we use the entity and relation embeddings to generate Conditional Soft Prompts which are then fed into the frozen PLMs to fuse the textual and structural knowledge together. The fused Conditional Soft Prompts are used as inputs to the graph-based KGC model that produces the final entity ranking results. We further propose Local Adversarial Regularization to improve CSProm-KG to distinguish textually similar entities in KG.
We evaluate CSProm-KG on various KGC tasks and conduct experiments on WN18RR, FB15K237 and Wikidata5M for Static KGC (SKGC), and on ICEWS14 and ICEWS05-15 for Temporal KGC (TKGC). CSProm-KG outperforms a number of competitive baseline models, including both graphbased and PLM-based models. We conduct ablation studies to show the strength of prompt-based methods against the fine-tuning counterparts and the effectiveness of each proposed components. We also demonstrate the flexibility of CSProm-KG with different graph-based models, and the training and inference efficiency of CSProm-KG.

heading: 2 Related Work
text: Graph-based methods Graph-based methods represent each entity and relation with a continuous vector by learning the KG spatial structures. They use these embeddings to calculate the dis-
tance between the entities and KG query to determine the correct entities. The training objective is to assign higher scores to true facts than invalid ones. In static KGC (SKGC) task, there are two types of methods: 1) Translational distance methods measure the plausibility of a fact as the distance between the two entities, (Bordes et al., 2013; Lin et al., 2015; Wang et al., 2014); 2) Semantic matching methods calculate the latent semantics of entities and relations (Nickel et al., 2011; Yang et al., 2015; Dettmers et al., 2018). In temporal KGC (TKGC) task, the systems are usually based on SKGC methods, with additional module to handle KG factual tuples timestamps (Dasgupta et al., 2018; Goel et al., 2020; Han et al., 2021).
PLM-based methods PLM-based methods represent entities and relations using their corresponding text. These methods introduce PLM to encode the text and use the PLM output to evaluate the plausibility of the given fact. On SKGC, Yao et al. (2019) encode the combined texts of a fact, then a binary classifier is employed to determine the plausibility. To reduce the inference cost in Yao et al. (2019), Wang et al. (2021a) exploit Siamese network to encode (h, r) and t separately. Unlike previous encode-only model, Xie et al. (2022); Saxena et al. (2022) explore the Seq2Seq PLM models to directly generate target entity text on KGC task.
Prompt tuning Brown et al. (2020) first find the usefulness of prompts, which are manually designed textual templates, in the GPT3 model. Wallace et al. (2019); Shin et al. (2020) extend this paradigm and propose hard prompt methods to automatically search for optimal task-specific templates. However, the selection of discrete prompts involves human efforts and difficult to be optimized together with the downstream tasks in an end-toend manner. (Li and Liang, 2021; Lester et al., 2021) relax the constraint of the discrete template with trainable continuous vectors (soft prompt) in the frozen PLM. As shown in Li and Liang (2021); Lester et al. (2021); Liu et al. (2021), frozen PLM with Soft Prompt could achieve comparative performance on various NLP tasks, despite having much less parameters than fully trainable PLM models. To the best of our knowledge, we are the first to apply Soft Prompt to PLM-based KGC model.

heading: 3 Method
text: In this section, we first formulate Knowledge Graph Completion in Sec. 3.1. We then introduce CSProm-KG in Sec. 3.2 to Sec. 3.7.

heading: 3.1 Knowledge Graph Completion
text: Knowledge graph (KG) is a directed graph with a collection of fact tuples. Let T = {V,R, L,M} be a KG instance, where V , R, L and M denote the entity, relation, edge (fact) and meta information set respectively. Each edge e ∈ L is (h, r, t,m) ∈ V × R × V × M which connects head entity h and target entity t with relation type r, and is associated with meta information m. In Static KGs (SKG), no meta information is involved (i.e. M = ∅). In Temporal KGs (TKG), each fact has a corresponding timestamp and M includes all fact timestamps. Knowledge Graph Completion (KGC) is to predict the target entity for KG queries (h, r, ?,m). The queries (?, r, t,m) are converted into (t, r−1, ?,m), where r−1 is the inverse of r. In this paper, CSProm-KG learns a score function f(h, r, t,m) : V ×R× V ×M → V that assigns a higher score for valid facts than the invalid ones.

heading: 3.2 CSProm-KG Overview
text: Motivated by the observation that Soft Prompts in a frozen PLM is effective in solving the over-fitting issue (Wang et al., 2022), we apply Soft Prompts in CSProm-KG to avoid the KGC models overly focusing on the textual information. Although several research initiatives have explored the utilization of both structural and textual information for NLP tasks (Li et al., 2022; Xiao et al., 2021), none of
them is capable of solving the over-fitting issue over textual information in the context of KGC. Figure 2 shows the architecture of CSProm-KG which includes three important components: a fully trainable Graph-based KGC model G, a frozen Pretrained language model (PLM) P , and a trainable Conditional Soft Prompt S. Firstly, the embeddings in G, which are explicitly trained to predict entities using structural knowledge, are used to generate the parameters of S. In this way, S is equipped with KG structural knowledge. We then feed the generated S, as well as the corresponding text of entities and relations, into P . Finally, the PLM outputs of S are extracted as the final inputs to G which produces final results for the KGC tasks. This allows the structural knowledge from G and the textual knowledge from P to be equally fused via S. To further improve the robustness of CSProm-KG, we propose Local Adversarial Regularization, which selects textually similar entities for training to be detailed shortly.

heading: 3.3 Graph-based KGC Model G
text: In CSProm-KG, the graph-based KGC models G represents KG entities and relations as continuous embeddings. Given a KG query (h, r, ?,m), we represent h and r as embeddings Ee and Er ∈ Rd where d is the embedding size. Ee and Er are used at both inputs and outputs. At inputs, we use these embeddings to generate Conditional Soft Prompt which further interacts with the textual inputs of the frozen PLM P . At outputs, we use these embeddings to calculate f(h, r, t,m) which produces the entity ranking for KG queries. For example, when
using ConvE as G, the corresponding f(h, r, t,m) is the dot-product between the representation of (h, r) and the tail entity embeddings. Note that, CSProm-KG is flexible enough to work well with any existing graph-based KGC models. We will show this flexibility in Sec. 4.4.

heading: 3.4 Pre-trained Language Model P
text: Let’s assume that the pre-trained language model P has l transformer layers with hidden size H . To represent a KG query (h, r, ?,m), we jointly represent h, r and m by extracting and concatenating their corresponding raw tokens, including their names and their corresponding descriptions if available. We connect the texts of h and r with a special token [SEP], and feed the joint text into the frozen PLM P . For TKGC tasks, we simply add the event timestamp after the joint text of h and r. We show the effectiveness of this design choice in Sec. 4.2.

heading: 3.5 Conditional Soft Prompt S
text: Soft Prompt prepends a sequence of trainable embeddings at the inputs to a frozen Pre-trained Language model. Li and Liang (2021) propose Layerwise Soft Prompt which inserts relatively short prompt sequences (e.g., 5 - 10 vectors) at each layer and allows frequent interaction with the entities’ and relations’ textual information in PLMs. Inspired by this, we propose a novel Conditional Soft Prompt which has k trainable vectors on each layer. Specifically, the ith input for the jth layer hji ∈ RH is defined as:
hji =   
sji i ≤ k wi (i > k) ∧ (j = 0)
Trans(hj−1: )i Otherwise (1)
where Trans(·) is the forward function of Transformer layer in P , wi is the fixed input word embedding vector and sji is the i
th prompt vector at jth layer. The Trans(·) works on the entire sequence (prompt + text). Conditional Soft Prompt is designed to connect with embeddings in G, we use the embeddings of entities and relations Ee and Er to generate Conditional Soft Prompt S. Formally,
S = [F (Ee);F (Er)] (2) F (x) = Wout · (ReLU(Win · x)) (3)
where Win ∈ Rdh×d and Wout ∈ R(l∗H∗k/2)×dh are trainable weight matrices and dh is the middle hidden size for the mapping layers. We then
re-organize F (Ee) and F (Er) into a sequence of input embeddings and evenly distribute them into each PLM layer. In this process, the input tokens for P and Conditional Soft Prompt S are fully interacted with each other, allowing the structural knowledge in G (linearly mapped to S) and textual knowledge in P to be fully fused together.

heading: 3.6 Local Adversarial Regularization
text: As PLMs are frozen, the model may lose part of flexibility in distinguishing textually similar entities via tuning of the Transformer layers. To enhance CSProm-KG’s ability to distinguish textually similar entities, inspired by (Goodfellow et al., 2015), we introduce an Adversarial Regularization term. Different from conventional adversarial regularization which generates virtual examples that do not exist, our adversarial examples are picked from the local entity set V that are of concrete meanings. Specifically, given a KG query (h, r, ?,m) and ground-truth entity t, CSProm-KG treats entities that are textually similar to t as adversarial examples. We refer these samples as Local Adversarial Regularization (LAR) entities. To allow efficient training, we define LAR samples as the ones sharing the common tokens in entity names and descriptions with t, enabling us to pre-compute these LAR samples before training. This is different from previous works (Miyato et al., 2017; Madry et al., 2018; Goodfellow et al., 2015) that generate virtual adversarial examples using training perturbation with large computational costs. Specifically, the LAR training objective is:
Ll(h, r, t,m) = max(f(h, r, t,m)
− 1 n
n∑
i=0
f(h, r, t∆i ,m) + γ, 0) (4)
where t∆i is an sampled LAR entity of t, γ is the margin hyperparameter, n is the number of sampled LAR entities.

heading: 3.7 Training and Inference
text: For training, we leverage the standard cross entropy loss with label smoothing and LAR:
Lc(h,r, t,m) = −(1− ϵ) · log p(t|h, r,m) − ϵ|V | ∑
t′∈V/t · log p(t′|h, r,m) (5)
L = ∑
(h,r,t,m)∈T Lc(h, r, t,m) + α · Ll(h, r, t,m)
(6)
where p(t|h, r,m) = exp f(h,r,t,m)∑ t′∈V exp f(h,r,t ′,m) , ϵ is the label smoothing value and α is the LAR term weight. For inference, CSProm-KG first computes the representations for KG query (h, r, ?,m), then uses the entity embeddings in G to compute the entity ranking. While other PLM-Based KGC models such as StAR (Wang et al., 2021a) requires |V | PLM forward pass computation for entity embeddings. Thus, CSProm-KG is more computationally efficient than these baselines (See Sec. 4.3).

heading: 4 Experiments
text: In this section, we first compare CSProm-KG with other competitive baselines in the SKGC and TKGC benchmarks in Sec. 4.1. We then conduct ablation studies to verify the effectiveness of our propose components in CSProm-KG in Sec. 4.2. We further show the efficiency and flexibility of CSProm-KG in Sec. 4.3 and 4.4, respectively.
Dataset WN18RR (Dettmers et al., 2018) and FB15K-237 (Toutanova and Chen, 2015) are the most popular SKGC benchmarks where all inverse relations are removed to avoid data leakage. Wikidata5M (Wang et al., 2021b) is a recently proposed large-scale SKGC benchmark. For TKGC, we use ICEWS14 (García-Durán et al., 2018) and ICEWS05-15 (García-Durán et al., 2018) which include political facts from the Integrated Crisis Early Warning System (Boschee et al., 2015). More dataset details are shown in Table 8.
Implementation Details All the experiments are conducted on a single GPU (RTX A6000). We tune the learning rate η ∈ {1e−3, 5e−4, 1e−4}, batch size B ∈ {128, 256, 384, 450}, prompt length Pl ∈ {2, 5, 10} and LAR term weight α ∈ {0.0, 0.1, 0.2}. While α > 0, we employ 8 LAR samples for each training instance and gradually increase the LAR term weight from 0 to α using a step size of αstep = 1e−5. CSProm-KG uses the BERT-Large (Devlin et al., 2019) and ConvE (Dettmers et al., 2018) model. We set the label smoothing to 0.1 and optimize CSProm-KG with AdamW (Loshchilov and Hutter, 2019). We choose the checkpoints based on the validation mean reciprocal rank (MRR). We follow the filtered setting in Bordes et al. (2013) to evaluate our model. Detailed model hyperparameters for each dataset are shown in Appendix B.

heading: 4.1 Main result
text: Table 1 and Table 2 present the main SKGC and TKGC results, respectively, which demonstrate statistical significance (t-student test, p < 0.05).
Results on SKGC As for the popular mediumsized KGC benchmarks, CSProm-KG achieves state-of-the-art or competitive performance compared with PLM-based KGC models. In particular, on FB15K-237, CSProm-KG consistently outperforms all PLM-based KGC models and achieves 6.5% (from 0.336 to 0.358) relative MRR im-
provement. These PLM-based baselines are all fully fine-tuned, indicating the importance of using parameter-effective prompts in the KGC task. Compared with graph-based methods, CSProm-KG outperforms baseline methods by a large margin on WN18RR (i.e. 0.575 v.s. 0.479 on MRR) and on FB15K-237 (i.e. 0.358 v.s. 0.355 on MRR). Noted that the improvement on FB15K-237 is barely comparable to that on WN18RR, and this discrepancy can be explained by the existence of Cartesian Product Relations (CPRs) in FB15K-237, which are noisy and semantically meaningless relations (Chen et al., 2022; Lv et al., 2022; Akrami et al., 2020). On the Wikidata5M benchmark, CSPromKG significantly outperforms previous methods, showing the advantages of CSProm-KG on the large-scale KGs. These results verify that with frozen PLM and accordingly much less trainable parameters, CSProm-KG can achieve remarkable performance on various KGs with different scales.
Results of TKGC Table 2 reports the experiment results on the ICEWS14 and ICEWS05-15 benchmarks. On ICEWS14, CSProm-KG substantially outperforms existing TKGC methods (e.g., at least 0.03 MRR higher than previous works). On ICEWS05-15, CSProm-KG is 0.028 and 0.045 higher than the best TKGC methods in terms of MRR and H@1, though being slightly worse on H@10 than Tero and ATiSE. On both benchmarks, CSProm-KG sets new state-of-the-art performance. Note that the TKGC baseline models are often specifically designed and optimized for the TKGC task, while the only modification to CSProm-KG
is to add timestamp into its input. This further shows that our proposed CSProm-KG method is a generally strong solution for various of KGC tasks.

heading: 4.2 Ablation Studies
text: We conduct ablation study to show the effectiveness of our proposed components on WN18RR. Table 3 and Figure 5 summarize the ablation study results.
KG Query Structure As we discussed in Sec. 3, for each KG Query (h, r, ?,m), we jointly concatenate their textual information and feed them into the frozen PLM (as shown in Figure 3). To demonstrate the effectiveness of this design choice, we replace it with a Separated Strategy that is similar to the Siamese network used in Wang et al. (2021a). That is, as shown in Figure 4, we separately encode the textual information of h and r using PLMs. Table 3 Line 2 shows the performance of this Sep-
arated Strategy. Compared to CSProm-KG, the performance drops by 0.055 on MRR and 0.056 on H@10. The mixture of soft prompts and text representation concatenation increase the interaction between entity and relations, allowing better representation of KG Query.
Role of Graph-based KGC Models Table 3 Line 3 shows the performance of CSProm-KG without any graph-based KGC models. For this ablation, we directly use the outputs of PLM to predict the target entity. We observe that removing this graph-based KGC model leads to a performance drop (i.e., by 0.030 MRR and 0.033 H@10). This shows that even after the complex interaction in the PLMs, an appropriate graph-based KGC model could still provide additional useful structural knowledge. This experiment verifies the necessity of combining PLM-based and graph-based KGC models together.
Soft Prompt Design Lester et al. (2021) recently propose another Soft Prompt variant which puts longer trainable vectors at the bottom input layer. We refer it as non-layer-wise Soft Prompt. Table 3 Line 4 shows the performance using this variant on WN18RR. CSProm-KG with layer-wise soft prompt model outperforms the non-layer-wise counterpart by a large margin (i.e., 0.053 MRR and 0.066 H@10), which suggests that the layerwised Soft Prompt is more effective on KGC tasks. This could be explained by the fact that, to maintain similar trainable parameters, non-layer-wised Soft Prompt requires much longer prompt vector sequences at the input, while self-attention modules are often ineffective when handling long sequences (Zaheer et al., 2020).
Local Adversarial Regularization Table 3 Lines 5 to 8 show the ablation for adversarial regularization. Line 5 shows CSProm-KG without LAR falls behind the full CSProm-KG model by 0.041 MRR, indicating the important of LAR. From Lines 6, 7, 8, we investigate the importance of LAR entity source. We observe that CSProm-KG with LAR entities that share common keywords (in name or description) outperforms the one with random LAR entities, indicating the importance of selecting appropriate adversarial examples.
PLM Training Strategy We empirically verify the effect of freezing PLM in CSProm-KG. Table 3 Lines 9 - 12 show the performance of CSProm-KG with different level of parameter frozen. In general, the more trainable parameters in CSProm-KG, the poorer CSProm-KG performs. CSProm-KG w/ fully fine-tuned drops significantly, by 0.138 MRR (Line 12). We further show the changes of performance as we increase the number of trainable parameters of the PLMs in Figure 5. We freeze the PLM parameters starting from bottom layers (orange) and starting from top layers (blue). Both experiments suggest that the performance of CSProm-KG remains nearly unchanged until the freezing operations are applied to the last few layers. As most of the layers frozen, the performance of CSProm-KG grows dramatically. Interestingly, we find freezing parameters from bottom layers performs slightly better than from top layers. This could be because lower layers in BERT could capture low-level semantics (e.g., phrase features) and this information is more beneficial to the KGC task. In summary, the frozen PLM prevents CSProm-KG from over-fitting the KG textual information, and therefore allows CSProm-KG to achieve substantial improvements in KGC tasks.
Ensemble Model CSProm-KG has successfully combined both textual and structure knowledge for KGC using Conditional Soft Prompt. To show the effectiveness of this design choice, we adopt a straightforward full-sized bagging strategy to combine the prediction from a graph-based KGC model and a PLM-based KGC model. We separately run the ConvE model and BERT model used in CSProm-KG (i.e., same configuration for fair comparsion) and use the averaged results from both models. Table 3 Line 13 shows that this ensemble model is far less effective than CSProm-KG. We believe this is because the ensemble model cannot deeply fuse structural and textual information like our proposed conditional soft-prompt.
Prompt Length As shown in Table 4, we conduct extensive studies to examine the impact of prompt length for CSProm-KG. We observe that as the prompt length increases, there is a proportional rise in both memory and computational requirements. However, the corresponding improvement in performance is marginal. Moreover, a further increase in prompt length presents considerable challenges in training the prompt model, leading to a decline in performance.
Furthermore, we conduct an investigation involving the utilization of a fully fine-tuned BERT to represent the input head entity and relation, without using prompt learning or a graph-based models. However, we find instability during the training process of this model, and consequently, the resulting model achieve very low performance compared to the results reported above.

heading: 4.3 Model Efficiency
text: Table 5 shows the model efficiency for CSPromKG and other PLM-based KGC methods on a single RTXA6000 GPU. CSProm-KG requires much less training and evaluation time. Compared with KG-BERT (Yao et al., 2019) and StAR (Wang et al., 2021a), CSProm-KG is 10x faster in training and 100x faster in evaluation. This is because both
KG-BERT and StAR require the PLM outputs to represent all KG entities, which introduces significant computational cost. In contrast, CSProm-KG only applies BERT to represent the input queries and directly uses entity embedding matrix to compute entity ranking. We also compare CSPromKG with GenKGC (Xie et al., 2022) and KGS2S (Chen et al., 2022), recently proposed PLMbased Sequence-to-Sequence KGC models. They directly generate the correct entity names and does not require to use the outputs of PLMs to represent large-scale KG entities. However, it has to maintain a huge search space for the entity names during inference and becomes much slower than CSProm-KG (e.g., 0.2m vs. 104m and 115m). In summary, CSProm-KG maintains higher-level efficiency (as well as performance) compared to other PLM-based KGC methods with similar model size.

heading: 4.4 Flexibility to Graph-based KGC models
text: As we discussed in Sec. 3.3, CSProm-KG is able to incorporate other graph-based KGC methods. To verify the flexibility of CSProm-KG, we replace the ConvE with another two popular graphbased KGC methods: TransE and DistMult. As shown in Table 6, CSProm-KG can always improve the KGC task performance after integrating with TransE, DistMult and ConvE. This indicates that CSProm-KG successfully incorporate the text information into these graph-based KGC models. In particular, CSProm-KG with TransE achieves a 2x improvement on MRR (from .243 to .499) and 10x improvement on H@1 (from .043 to .462). In short, CSProm-KG is capable of fusing its textual knowledge with the structural knowledge provided
by various of graph-based KGC models.

heading: 4.5 Case Study
text: In this section, we showcase how Conditional Soft Prompt could prevent CSProm-KG from overfitting to textual information. Table 7 lists the top two entities ranked by CSProm-KG and CSPromKG w/o Conditional Soft Prompt (i.e., CSPromKG w/ FT in Table 3). In the first case, CSPromKG produces two different occupations that are relevant to the whaler in the KG Query, whilst CSProm-KG w/o Conditional Soft Prompt ranks two sea animal names as the outputs. This could be caused by the surface keywords seaman and ship in the KG Query. In the second case, the expected entity should be an award for the band Queen. CSProm-KG successful pick up the correct answer from many award entities using the existing KG structures, while CSProm-KG w/o Conditional Soft Prompt confuses in those candidates which are textually similar and unable to rank the groundtruth entity into top-2. In summary, CSProm-KG maintains a balance between textual and structural knowledge, while CSProm-KG w/o Conditional Soft Prompt often focuses too much on the textual information in the KG Query.

heading: 5 Conclusion and Future Work
text: In this paper, we propose CSProm-KG, a PLMbased KGC model that effectively fuses the KG structural knowledge and avoids over-fitting towards textual information. The key innovation of CSProm-KG is the Conditional Soft Prompt that connects between a graph-based KGC models and a frozen PLM avoiding the textual over-fitting issue. We conduct experiments on five popular KGC benchmarks in SKGC and TKGC settings and the results show that CSProm-KG outperforms several strong graph-based and PLM-based KGC models. We also show the efficiency and flexibility of CSProm-KG. For future work, we plan to adapt our method to other relevant knowledge-intensive downstream tasks, such as fact checking and openended question answering.

heading: 6 Limitations
text: CSProm-KG successfully integrates both graphbased and textual representations in the KGC task, achieving substantial performance and efficiency improvement. However, similar to other PLMbased methods, this comes at the cost of increased computational resources (v.s. graph-based KGC models). In addition, we find that CSProm-KG may occasionally collapse on small KGC benchmarks (e.g. WN18RR) under specific random seeds. This is probably due to the nature of Soft Prompts, which involve much smaller number of trainable parameters, compared to fine-tuned models. However, we never see similar phenomena when training CSProm-KG in the large KGC benchmarks (e.g., Wikidata5M). We plan to solve these issues for CSProm-KG as future work.

heading: Acknowledgement
text: We thank the anonymous reviewers for their insightful suggestions to improve this paper. This research / project is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative and A*STAR SERC Central Research Fund (UIBR). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.

heading: A Dataset
text: We use SKGC datasets released from (Yao et al., 2019) and TKGC datasets from (García-Durán et al., 2018). We follow the original split in our experiments. Table 8 shows the statistics of the datasets. All of these datasets are open-source English-written sources without any offensive content. They are introduced only for research use.

heading: B Hyperparameters
text: Hyperparameters are selected with grid search on the validation set. The optimal hyperparameters are presented in Table 9

heading: C Baseline Methods
text: CSProm-KG is compared against a variety of stateof-the-art baseline methods on SKGC and TKGC tasks. For SKGC, we include popular graph-based methods, i.e. TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018), RotatE (Sun et al., 2019) and CompGCN (Vashishth et al., 2020). We also compare CSProm-KG against several competitive PLM-based methods, i.e. KG-BERT (Yao et al., 2019), MTL-KGC (Kim et al., 2020), StAR (Wang et al., 2021a), MLMLM (Clouâtre et al., 2021), KEPLER (Wang et al., 2021b), GenKGC (Xie et al., 2022), KGT5 (Saxena et al., 2022) and KG-S2S (Chen et al., 2022). For TKGC, we compare CSProm-KG with graph-based
TKGC baselines, including: TTransE (Leblay and Chekol, 2018), HyTE (Dasgupta et al., 2018), ATiSE (Xu et al., 2019), DE-SimplE (Goel et al., 2020), Tero (Xu et al., 2020), TComplEx (Lacroix et al., 2020), TNTComplEx (Lacroix et al., 2020), T+TransE (Han et al., 2021), T+SimplE (Han et al., 2021). PLM-based baselines for TKGC includes KG-S2S (Chen et al., 2022)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 6
7 A2. Did you discuss any potential risks of your work? The potential risk of this line of work has already been discussed in previous research and our base methods (Bert).

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: both of abstract and introduction (section 1) do
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: data citation: 4.1 Dataset, model citation: 4.1 Implementation Details

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: data: Appendix A, model: 4.1 Implementation Details.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.1 Implementation details and Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1, Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We plan to solve these issues for CSProm-KG as future work