Improving Knowledge Graph Completion with Generative Hard Negative Mining
Year: 2023

Abstract:
Contrastive learning has recently shown great potential to improve text-based knowledge graph completion (KGC). In this paper, we propose to learn a more semantically structured entity representation space in text-based KGC via hard negatives mining. Specifically, we novelly leverage a sequence-to-sequence architecture to generate high-quality hard negatives. These negatives are sampled from the same decoding distributions as the anchor (or correct entity), inherently being semantically close to the anchor and thus enjoying good hardness. A self-information-enhanced contrasting strategy is further incorporated into the Seq2Seq generator to systematically diversify the produced negatives. Extensive experiments on three KGC benchmarks demonstrate the sound hardness and diversity of our generated negatives and the resulting performance superiority on KGC.

heading: 1 Introduction
text: Knowledge Graph (KG) is an efficient method of representing global knowledge (Cui et al., 2019; Lv et al., 2022), playing a fundamental role in many Natural Language Processing (NLP) tasks like question answering (Sun et al., 2019a; Saxena et al., 2020), recommender systems (Huang et al., 2018), and web search (Ji et al., 2020), etc. The knowledge graph is composed of triples (h, r, t), where h, t, and r denote a head entity, a tail entity, and their relation, respectively. Modern public KGs such as Freebase, YAGO, and Wikidata, although covering massive knowledge with a large number of entities, are inevitably incomplete. Therefore, Knowledge graph completion (KGC) has become a popular area of research in recent years (Wang et al., 2017).
Generally, two types of methods are applied for this task. Embedding-based methods assign each entity/relation with a dense vector trained
†Corresponding authors.
with structural information of graphs. Representative efforts include TransE (Bordes et al., 2013), TransH (Wang et al., 2014), Complex (Trouillon et al., 2016), and RotatE (Sun et al., 2019b), etc. Text-based methods (Wang et al., 2019; Yao et al., 2019; Lv et al., 2022; Saxena et al., 2022) exploit textual names or descriptions of entities and relations to facilitate representation learning. Among the two categories, the performance of text-based ones generally lags behind that of embeddingbased ones, due to inadequate entity representation optimization of the former (Wang et al., 2022).
The recent robust text-based method, SimKGC (Wang et al., 2022), tackles the sub-optimal representation problem by contrasting carefully sampled in-batch, pre-batch and self negatives, inspiring us to leverage hard example mining (Kalantidis et al., 2020; Hu et al., 2020) to unlock the potential of contrastive learning for text-based KGC methods. Specifically, there
5866
are two essential properties for negative sample selection:
• Vicinity (Robinson et al., 2021; Tabassum et al., 2022a) is a metric to measure the distance between the negatives and the anchor. Lower vicinity can push the model to learn better representation boundaries.
• Uniformity (Wang and Isola, 2020; Tabassum et al., 2022a) is another one to measure the representativeness of the negatives. Higher uniformity means negative samples are uniformly distributed on the hypersphere and will make the produced representation more generalized.
To improve vicinity, we pioneeringly introduce a generative method to produce negatives in KGC. Unlike previous efforts that use in-batch or prebatch entities, we apply a sequence-to-sequence model as a generator, which is trained to directly generate the tail entity t for a given < h, r > pair. With a sampling strategy, we can get multiple tail entities from the generator, and the incorrect ones among them constitute our negatives for efficient contrastive learning. As shown in Figure 1, since the generated negative samples are decoded from the same hidden state as the correct answer, they inherently have better anchor similarity in the semantic space.
To improve uniformity, we further incorporate the generator itself with a novel self-informationenhanced contrastive learning strategy. In particular, we apply a prefix tree to obtain the conditional frequency of the tokens at each decoding step, which is converted to self-information (Shannon, 1948) and utilized as a token-wise weight for contrastive learning in the generator. Typically, the generator prefers to predict more frequent entity descriptions which consist of tokens of low selfinformation. Incorporating the weighting mechanism forces the generator to produce more distinctive representations for more informative tokens (e.g., in less frequent entity descriptions), yielding more diversified negatives for KGC.
In summary, our generative negative mining strategy innovatively balances the vicinity and uniformity of negative samples. Meanwhile, the hard negatives produced by the generator are natural candidates for inference. Compared to other textbased methods that have to enumerate all entities for inference, we can pick a small number
of generated entities as a high-quality candidate set, significantly accelerating inference speed for large-scale KGs. We evaluate our method on three popular KGC datasets (WN18RR (Dettmers et al., 2017), FB15K237 (Toutanova and Chen, 2015a), and Wikidata (Wang et al., 2019)), and the empirical results verify the superiority of our method.
Our contributions are as follows:
• We pioneeringly incorporate generative methods to produce hard negatives in text-based KGC for better contrasting effects.
• We design a novel self-information-enhanced contrastive learning method for the negative generator, providing us with high-quality negative samples.
• The negatives we generated are proven to systematically balance the hardness and diversity (or vicinity and uniformity, more formally), leading to competitive performances on three KGC benchmarks.

heading: 2 Methodology
text: The link prediction task of KGC is to infer the missing triples given an incomplete G. In this section, we mathematically describe the proposed Generative Hard Negative Mining (GHN) method in detail. The core idea of GHN is to obtain better negatives in terms of vicinity and uniformity. We first introduce the overall process of GHN in 2.1 which includes how to generate and use these negative samples to enhance KGC model training. Then, to further improve the vicinity and uniformity of generated negatives, we propose a self-informationenhanced token-level contrastive learning method which is discussed in 2.2. Finally, we introduce how to facilitate inference on KGC tasks in 2.3.

heading: 2.1 Model Architecture
text: Our GHN model mainly consists of two parts, a generator that aims to provide hard negatives to facilitate efficient contrastive learning and a predictor which is supposed to predict the tail entity given a <head entity, relation> pair. Then, we will present how to leverage generated negatives for better training efficiency.
Generator Saxena et al. (2022) has proved that a simple encoder-decoder Transformer (Vaswani et al., 2017) can perform knowledge graph completion in the form of a sequence-to-sequence task.
The performance of directly using a Transformer to predict the tail entity may not appeal. However, using a sequence-to-sequence model to generate negatives has ideal properties, such as scalability (the inference speed is independent of the scale of KG) and the better vicinity of generated negatives, considering that the generation of negatives and golden entities share the same hidden state.
First, we construct a mapping between the textual descriptions and entities/relations following Saxena et al. (2022); Wang et al. (2022). The textual mentions we used are provided by KGBERT (Yao et al., 2019). Then we formally describe how to convert link prediction queries to textual queries. Given a link prediction query (h, r, ?), where h and r represent the head entity and the relation, respectively. The textual query is the concatenation of the text mentions of h and r, separated by |. For example, given a link prediction query (St.Louis, time zones location, ?), the converted text query thr is ′St.Louis | time zones location′. The text query is the input of our generator, which is supposed to output the correct answer ′Central Time Zone′.
To obtain diverse and semantically similar negatives to the golden entity, we use the sampling strategy to get multiple predictions for the same input query. Specifically, we get a probability distribution over tokens at each step of decoding, then sample a token from the distribution and autoregressively decode until the ‘stop’ token. This procedure will be repeated multiple times to produce a negative set N q for each query. Finally, we drop the correct entity to get a set of negatives.
Predictor Following SimKGC (Wang et al., 2022), our GHN adopts a bi-encoder architecture. The first encoder BERThr takes the text query thr as input and produces the relation-aware embedding ehr, where thr is the same text query as the input of the generator. Similarly, the second encoder BERTt takes the textual mentions of the tail entity and produces its embedding et. We use mean pooling followed by L2 normalization to obtain these embeddings. Then we compute cosine similarity to these embeddings and predict the one with the largest score:
ϕ(h, r, ti) = cos (ehr, eti) ,
argmax ti
ϕ(h, r, ti), ti ∈ E , (1)
Training with Generated Negatives We first briefly introduce the objective of the generator.
The generator is optimized by minimizing a crossentropy loss with the golden entity:
LCE = − 1
K
K∑
k=1
log p (y∗k | y<k,x) , (2)
where K is the length of the target entity. The probability p (y∗k | y<k,x) is calculated by multiplying the last layer decoder hidden state of the generator sk and the softmax embedding matrix Ws together:
p (yk | y<k,x) ∝ exp (Ws · sk) , (3)
where x and y denote the input sequence and output sequence respectively.
Unlike other negative mining methods (Xiong et al., 2020; Kalantidis et al., 2020), our method does not need to calculate weights for negatives. Instead, we directly mix the generated negatives and in-batch negatives to produce a negative set N = Nib ∪Ngen, where Nib and Ngen denote the set of in-batch negatives and generated negatives respectively. In training, the generated negative samples N qgen ⊆ N q are sampled from corresponding negative set N q:
p(n | q) = T∑
t=1
log (p (nt | y<t,x)) , n ∈ N q.
(4) After constructing the negative set, we use InfoNCE (Chen et al., 2020) loss to train the predictor:
Lp = − log eϕ(h,r,t)/τ
eϕ(h,r,t)/τ + ∑|N |
i=1 e ϕ(h,r,t′i)/τ
, (5)
where τ is a learnable parameter.

heading: 2.2 Self-information enhanced Training
text: We have used a sequence-to-sequence model to provide hard negatives for the predictor. Through further investigation, we noticed that the target of generative link prediction is different from traditional generative tasks like machine translation. We call tasks like link prediction as target-constraint generation tasks. Specifically, one is able to obtain all available targets in link prediction, which is impossible for traditional generative tasks. Intuitively, more informative tokens are more critical than others. We hypothesize that informative tokens may impact the diversity of generated results since the generator prefers to predict more frequent
entity descriptions which consist of tokens of low self-information, typically. Our investigation (see section 4.3 for more details) verified this hypothesis.
Therefore, we propose a self-informationenhanced contrastive learning method to further improve the uniformity of generated negatives by producing more distinctive representations for more informative tokens. First, we will introduce how to obtain the self-information (Shannon, 1948) of tokens. Then, we will introduce how to facilitate training for the generator of GHN with selfinformation.
Self-Information is a measure of the information content associated with the outcome of a random variable (Shannon, 1948). Since all possible outputs are acquirable in KGC, we can easily calculate self-information for all tokens.
We first construct a prefix tree T where nodes are annotated with tokens from the vocabulary of entity mentions. The children of each node t ∈ T indicate all the available continuations from the prefix defined traversing the prefix tree from the root to t. Then, given the previous sequence generated by a sequence-to-sequence model w1, · · · , wn−1 and the current label token wn, the self-information of the current label token I(wn|w<n) can be calculated as:
I(wn|w<n) = − log (P (wn|w<n)) P (wn|w<n) := Count(wn|Aw1,···wn−1)∣∣Aw1,···wn−1 ∣∣ , (6)
where A is all the children nodes determined by the prefix w1, · · · , wn−1 on the prefix tree. Then we average the self-information of tokens that are present in the different positions of the vocabulary to produce the self-information for each token:
I(wn) = 1
K
K∑
k=1
I(wkn), (7)
where wkn denotes the kth token in vocabulary and K is the total times of occurrences of wn.
Self-Information-aware Contrastive Learning Token-level contrastive learning methods widen the representations of tokens that have different labels. Since we want the representation of informative tokens to be more distinctive, we assign a set of weights for each contrasting pair (higher weight for contrasting pairs with higher self-information
tokens in it) according to their self-information. As a result, the model can assign more distinctive representations for tokens with more self-information and facilitate the generation of these tokens.
Formally, given a target token ti and a negative sample tj , we assign a soft weight w(i, j) for them. The weight is determined by the self-information of both ti and tj . We use the same strategy as tokenlevel contrastive learning (Zhang et al., 2021) to build up the negative and positive samples. Given the sequence of former target tokens and current target label ti, we formulate the information-aware soft weight w(i, j) for ti and corresponding negative sample tj as:
w(i, j) = λI(ti) · I(tj), (8)
where λ is a hyperparameter. In our implementation, the mean value of information-aware weights for all negatives of each anchor is normalized to be 1. The contrastive learning object is:
LCL = − log ecos(sa,sp)
∑N i=1w(a, i)e cos(sa,si) , (9)
where sa, sp, and si denote the representation of anchor, positive token, and negative tokens, respectively. Finally, we weight the traditional generative object LCE and the self-information-aware contrastive objective LCL by a hyperparameter γ to compute the final loss LGEN for the generator:
LGEN = (1− γ)LCE + γLCL. (10)

heading: 2.3 Inference
text: Most text-based methods have to produce representation for each entity (Wang et al., 2022; Yao et al., 2019). Considering the large scale of modern KGs, the efficiency of link prediction is of concern. Therefore, we propose a simple generationclassification two-stage method to perform highefficiency inference. Specifically, the generator we used to produce hard negatives is also able to provide candidates for inference. In the first stage, the generator produces a set of candidates Nc by the same sampling strategy as we discussed in 2.1. In the second stage, the predictor computes scores for candidates and then predicts the one with the largest score:
argmax ti
ϕ(h, r, ti), ti ∈ Nc. (11)
The number of candidates |Nc| is empirically set to be 50. Be aware that the two-stage inference is not
enforced for our method, it can speed up inference only when the scale of KG is relatively large.

heading: 3 Experiments Setup

heading: 3.1 Datasets
text: We evaluate our method on three widely-used link prediction datasets, WN18RR (Dettmers et al., 2017), FB15K237 (Toutanova and Chen, 2015b), and Wikidata (Wang et al., 2019). The statistics are shown in Table 1. WN18RR consists of about 41k synsets and 11 relations from WordNet (Miller, 1995). This dataset is constructed by removing the inverse relations from FB15k (Bordes et al., 2013) which suffers from test set leakage. FB15k-237 is a subset of Freebase (Bollacker et al., 2008), it consists of about 15k entities and 237 relations. Similar to the WN18RR dataset, Dettmers et al. (2017) removed the inverse relations to address the test set leakage problem. Wikidata5M has much larger scale which consists of ∼ 4.6M entities, 822 relations, and ∼ 20 million triples. Following most of KGC methods, we use the transductive version of Wikidata5M. All the textual descriptions for WN18RR and FB15k-237 are provided by Yao et al. (2019). The Wikidata5M dataset already provides descriptions for all entities and relations.

heading: 3.2 Baselines
text: TransE (Bordes et al., 2013) constructs a relationspecific translation from the head entity to the tail entity. Trouillon et al. (2016) introduces complex number embeddings.Tucker (Balažević et al., 2019) facilitate KGC based on Tucker decomposition of the binary representation of triples. DistMult (Yang et al., 2014) models the three-way interactions in triples. RotatE (Sun et al., 2019b) models a relation as rotation in complex space. DKRL (Xie et al., 2016) leverages a CNN network to obtain text representations. KEPLER (Wang et al., 2019) uses a Transformer-based encoder trained with the typical KGE objective and the masked language modeling objective. MTL-KGC (Kim et al., 2020) proposes a multi-task learning method that can learn more relational properties. KG-BERT (Yao et al., 2019) and
StAR (Wang et al., 2021) both leverage pre-trained language models to produce the representation of entities. SimKGC (Wang et al., 2022) train the model with much more negatives by incorporating three types of negatives.

heading: 3.3 Implementation Details
text: The generator is implemented using the HuggingFace library (Wolf et al., 2019) with the pre-trained weight of BART-base (Lewis et al., 2019). The predictor is initialized with BERT (Devlin et al., 2018). And we first train the generator with Lgen as the objective until validation accuracy did not significantly increase for 5k steps. Most hyperparameters except training epochs are shared across all datasets to avoid dataset-specific tuning.
Follow SimKGC (Wang et al., 2022), the entity descriptions are truncated to a maximum of 50 tokens for a fair comparison. The size of N qgen is set to 10. We use AdamW optimizer with linear learning rate decay, the learning rate is initialized to 3× 10−5. Models are trained with batch size 1024. All experiments were performed using 4 NVIDIA A100 GPUs. For the WN18RR, FB15k-237, and Wikidata5M datasets, we train for 50, 10, and 1 epochs, respectively.

heading: 4 Experiment Results

heading: 4.1 Main Results
text: Table 2 shows the performance of the baselines and our method variants on WN18RR and FB15k237 tasks (statistically significant with p < 0.05). While Table 3 further demonstrates the performance and inference speed of baselines and our methods on Wikidata5M dataset, which has a much larger scale. We have the following observations.
First, the superiority of GHN is significant in terms of performance. Compared to previous text-based methods, GHN shows consistent performance improvement in all three tasks. Compared to embedding-based methods, GHN has a significant performance advantage in WN18RR and Wikidata5M, while marginally trailing in FB15k237. All of the text-based methods lag behind
embedding-based methods on FB15k-237 dataset for now. The possible reason is that many links in FB15k-237 dataset are not predictable based on the available information (Cao et al., 2021) and may harm the training of text-based models.
Second, though GHN already surpasses most baselines without the self-information-enhanced training method, this training method further improves the performance on link prediction tasks, especially reflected in Hits@10 metric. This improvement shows that the self-information-enhanced training method may lead to a more generalized KGC model by introducing more diverse negatives. Please refer to section 4.3 for more details.
Third, GHN also significantly accelerates the inference speed on Wikidata5M dataset. To perform link prediction on a test set, SimKGC needs to do 2× |T |+ |E| times BERT forward pass, where |T | and |E| denote the size of the test set and the number of entities in corresponding KG, respectively. While GHN needs up to 2×|T |× (|Nc|+1) times BERT forward pass and ∼ 2 × |Nc| × |T | times BART generation process. Since in large-scale datasets such as Wikidata5M, |T | << |E| (5,163 test samples and ∼ 4.6M entities for Wikidata5M), the significant acceleration is as expected. Note that the two-stage inference is not enforced, in the case that there are too many queries or the KG’s scale is relatively small, GHN has the same inference speed as SimKGC.

heading: 4.2 Overall Impacts of Generative Hard Negative mining
text: One of the key designs in GHN is leveraging a sequence-to-sequence model to generate hard negatives. To further investigate the impact of generated negatives, we construct an approximate nearest neighbor negative sampling strategy (ANN) inspired by Xiong et al. (2020); Kalantidis et al. (2020) and replace the generative negative sampling strategy in GHN with it. Specifically, we use the current predictor to find entities that are close to the golden tail entity among all entities as the hard negatives. Since the computation cost of producing representations for all entities is unaffordable, we randomly sample 500 entities for each query and pick the top 10 entities by their representations’ cosine similarity with the representation of the golden tail entity. Table 4 shows the performance comparison between GHN, ANN, and four variants of SimKGC with different negative sampling strategies.
Intuitively, since ANN picks the nearest entities to the anchor as negatives which are more “harder” than generated ones, it should further improve link prediction performance. However, it does not show the expected performance improvement. The possible reason is that the balance between vicinity and uniformity is critical since these picked negatives apparently have better vicinity than generated neg-
atives but lead to worse performance(more details in section 4.3). Besides, GHN also shows performance superiority to SimKGC with four different negative sampling strategies, which further demonstrates the effectiveness of our generative negative sampling strategy.

heading: 4.3 Effects on Representation Learning
text: To further examine what makes GHN excel and how the self-information-enhanced training method impacts the generated negatives. We conduct a further investigation from the perspective of representation learning by computing two metrics Mv and Muni to measure the vicinity and uniformity for negatives produced by different sampling strategies, inspired by Wang and Isola (2020) and Tabassum et al. (2022b). Specifically, we use the average co-
sine distance of negatives between the anchor to measure the vicinity Mv for each negative sampling strategy. To measure the uniformity Muni of negatives’ distribution on the hypersphere, we compute the logarithm of the average pairwise Gaussian potential between all negatives’ embedding. Here we choose two variants of GHN, ANN, and IB negative sampling strategies to conduct the comparison. The reason we did not add PB and SN negative sampling strategy to the comparison is as follows: PB sampling strategy shares almost the
same Mv and Muni with IB. SN sampling strategy only picks the head entity itself in the query as the only negative sample and it is impossible to calculate Muni for this sampling strategy.
Figure 2 shows that our GHN is able to produce negatives with better uniformity but worse vicinity compared to the ANN sampling strategy, while the opposite is true when compared to IB sampling strategy. Meanwhile, our generative sampling strategy has a better MRR score than both NN and IB strategies (67.8 v.s. 66.7 and 67.1 MRR score).
Leveraging self-information-aware contrastive learning (SL) to enhance the training of the generator is another key design in GHN. Figure 2 also demonstrates that SL training method significantly improves the uniformity of generated negatives while the improvement of vicinity is also visible, which empirically verified that SL method could lead to a more diverse generation.

heading: 4.4 Effects on Training Efficiency
text: An abundance of negative samples is critical for contrastive learning (Chen et al., 2020; Wang et al., 2022) while causing more demands on GPU memory. Figure 3 illustrates the MRR metrics on WN18RR dataset vary for GHN and SimKGC under different numbers of negatives for each query. By introducing generated negatives, our GHN leads to much efficiency training: GHN only needs 511 negatives to reach a similar performance as SimKGC does with 1023 negatives. Note that we do not count the pre-batch negatives used in SimKGC for a fair comparison. This observation re-emphasizes the effectiveness of the proposed generative negative mining strategy.

heading: 5 Related Work
text: Text-based KGC methods aims to leverage text information in KGs to assist KGC. DKRL (Xie et al., 2016) obtained text representations by CNN. KEPLER (Wang et al., 2019) leveraged the typical KGE training objective into the masked language model training. MTL-KGC (Kim et al., 2020) proposed to learn more relational properties in KGs with multi-task learning method. KG-BERT (Yao et al., 2019) and StAR (Wang et al., 2021) leveraged pre-trained language models to produce entity embeddings in the cross-encoder style and biencoder style, respectively. SimKGC (Wang et al., 2022) increase the number of negatives by incorporating three types of negatives and achieved notable performance improvements.
Negative Mining aims to find property negatives to assist contrastive representation learning (Mao et al., 2021). The most effective way is to use the samples within the same mini-batch for negative candidates (Wang et al., 2022; Chen et al., 2020). Another effective and widely used method is to store negative samples with an asynchronous update mechanism (Zhang et al., 2018; He et al., 2019), which allows more negative candidates to be involved during training. To make contrastive pairs difficult to discriminate, Zhang et al. (2013); Chen et al. (2017); Xiong et al. (2020) compute propensity score for each <query, sample candidate> pair. Ying et al. (2018) further utilizes PageRank score to calculate weights for negative candidates. Motivated by the generative adversarial networks (Goodfellow et al., 2014), Wang et al. (2020) proposed a sampling strategy by adaptively receiving knowledge-aware rewards. And Hu et al. (2020) proposed to adversarially generate the hard negative samples together with the representation network.

heading: 6 Conclusion
text: We have presented our method for KGC tasks, which incorporate generative methods with a novel self-information-enhanced training strategy to produce high-quality negatives. And we further reveal that the proposed method systematically balances uniformity and vicinity, two essential properties for negative sample selection. Empirical results on three widely-used datasets (WN18RR, FB15k-237, Wikidata5M) have verified the superiority of our method.

heading: 7 Limitations
text: For now, the superiority of the proposed two-stage inference speed-up method cannot adapt to inductive datasets since the generated sequences are difficult to map to unseen entities. Therefore, we will explore how to efficiently perform KGC under the inductive setting in the future.
Like the other text-based KGC methods, our GHN lag behind embedding-based methods on FB15k-237 dataset. Cao et al. (2021) claims that many links in the FB15k-237 dataset are not predictable based on the information in the KG and we hypothesize this may harm the training of textbased models. In the future, we intend to examine this more thoroughly.

heading: Acknowledgements
text: We thank anonymous reviewers for their valuable comments. This work was supported in part by the National Key R&D Program of China under Grants No.2022YFF0902703.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 1 3 4 6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 1 3 4 6
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, we will explore how to efficiently perform KGC under the inductive setting in the future. Like the other text-based KGC methods, our GHN lag behind embedding-based methods on FB15k-237 dataset. Cao et al. (2021) claims that many links in the FB15k-237 dataset are not predictable based on the information in the KG and we hypothesize this may harm the training of textbased models. In the future, we intend to examine this more thoroughly