HyperCL: A Contrastive Learning Framework for Hyper-Relational Knowledge Graph Embedding with Hierarchical Ontology (2024)

Abstract: Knowledge Graph (KG) embeddings are essential for link prediction over KGs. Compared to triplets, hyper-relational facts consisting of a base triplet and an arbitrary number of keyvalue pairs, can better characterize real-world facts and have aroused various hyper-relational embedding techniques recently. Nevertheless, existing works seldom consider the ontology of KGs, which is beneficial to link prediction tasks. A few studies attempt to incorporate the ontology information, by either utilizing the ontology as constraints on entity representations or jointly learning from hyper-relational facts and the ontology. However, existing approaches mostly overlook the ontology hierarchy and suffer from the dominance issue of facts over ontology, resulting in suboptimal performance. Against this background, we propose a universal contrastive learning framework for hyper-relational KG embeddings (HyperCL), which is flexible to integrate different hyper-relational KG embedding methods and effectively boost their link prediction performance. HyperCL designs relationaware Graph Attention Networks to capture the hierarchical ontology and a concept-aware contrastive loss to alleviate the dominance issue. We evaluate HyperCL on three real-world datasets in different link prediction tasks. Experimental results show that HyperCL consistently boosts the performance of state-of-theart baselines with an average improvement of 3.1-7.4% across the three datasets.

heading: 1 Introduction
text: Knowledge Graphs (KGs) which represent a network of real-world entities and exhibit the relationship between them, have empowered a wide range of applications, such as question answering (Yih et al., 2015) or recommender systems (Zhang et al., 2016). KGs are generally expressed as a set
*Corresponding author.
of triplets; each triplet denoted by (head, relation, tail), or (h, r, t) for short, encodes the connection from a head entity to a tail entity, such as (Apple, headquarters location, Cupertino) shown in Fig. 1. To better illustrate real-world facts, hyper-relational facts are developed in Freebase (Bollacker et al., 2008) and Wikidata (Wikidata, 2022), which consist of not only a base triplet (h, r, t), but also an arbitrary number of key-value pairs (k, v) further describing the base triplet, represented as (h, r, t, k1, v1, ...). Fig. 1 presents an example of hyperrelational facts on Wikidata (Apple, industry, software industry, of, computer program, of, operating system).
To effectively make use of hyper-relational facts, recent studies have proposed various embedding methods to solve link prediction tasks over KGs. Most of them learn to capture the structural information encoded in hyper-relational facts with Convolutional Neural Networks (CNNs) (Rosso et al., 2020), Graph Neural Networks (GNNs) (Galkin et al., 2020), or Transformer (Wang et al., 2021). However, they often neglect the importance of modeling ontology in KGs, which has shown to be significantly useful (Rosso et al., 2021). For example, the entities computer program and operating system in Fig. 1 are hard to differentiate and tend to have similar representations by current embedding methods since they are affiliated to the same hyperrelational fact and have a common key of. Nevertheless, they can be more distinguishable through the ontology. As shown in Fig. 2, computer program and operating system respectively belong to different concepts program and system. Hence, it is beneficial to incorporate the ontology information into entity representations.
In this context, existing hyper-relational KG embedding methods employ the ontology information of KGs as type constraints or joint learning. Specifically, most specific concepts for entities are considered as entity types and used to compute the
2918
similarity between entities (Liu et al., 2021). The most specific concepts for entities are identified by the ontology relation instance_of in Fig. 2. For example, enterprise and brand are two entity types for the entity Apple. However, these existing methods do not model the hierarchical structure of the ontology and thus fail to capture the semantic relations between entity types, which is a strong clue for entity representations. For example, two specific concepts program and system in Fig. 2 belong to a common abstract concept software, which indicates the semantic relatedness between entities computer program and operating system. On the other hand, joint learning methods combine hyper-relational facts with the ontology of KGs, formulating a joint model to learn the representations of both entities and concepts (Lu et al., 2023b; Luo et al., 2023a). Yet, they are either inflexible to accommodate the hierarchical ontology as the type constraint methods or lack consideration of the dominance issue of hyper-relational facts over the ontology. Specifically, due to the highly imbalanced numbers of entities and concepts (the latter is usually much less than the former), the learning process is dominated by learning from the facts rather than from ontology (using GNNs for example), resulting in the information of ontology barely encoded into entities and thus causing the suboptimal performance (as evidenced by our experiments below that addressing the dominance issue can boost the link prediction performance by 2.5-5.8%).
Against this background, we propose a universal contrastive learning framework for hyperrelational KG modeling (HyperCL), which is flexible to integrate different hyper-relational KG embedding methods. Specifically, we inherit the most
prominent encoder-decoder architecture (Galkin et al., 2020; Luo et al., 2023b) as the backbone of our framework and devise a Concept-aware Contrastive Learning (CCL) module to enhance hyperrelational KG embedding methods of this architecture (four state-of-the-art encoder-decoder models are selected to verify the effectiveness of HyperCL in the experiments). The CCL module first captures the hierarchical structure of ontology; we use relation-aware Graph Attention Networks (GATs) to encode the sophisticated concept information, which accounts for the heterogeneous relationships between concepts and incorporates the heterogeneity into entity and concept representations. Afterward, it builds two views (i.e., an instance view for hyper-relational facts and an ontology view for the hierarchical ontology) of KGs. Finally, it develops a concept-aware contrastive loss to enforce the representations of the same entities across two views to be close to each other while those of different but semantically similar entities to be apart. This design thereby alleviates the dominance issue of hyper-relational facts by first decoupling the learning process from the facts and ontology and then connecting them via our contrastive loss. HyperCL is trained using a multi-task learning strategy, being able to accelerate the optimization process. Our contributions can be summarized as follows:
• We revisit the existing approaches that employ the ontology of KGs for hyper-relational KG embeddings, and discover two key limitations: 1) the ignorance of the ontology hierarchy; and 2) the dominance issue of facts over ontology.
• We propose HyperCL framework to subtly model both hyper-relational facts and the hierarchical ontology of KGs. A CCL module is designed,
where the relation-aware GATs are used to capture the hierarchical structure of ontology while the concept-aware contrastive loss is employed to alleviate the dominance issue, both of which enhance the hyper-relational KG embeddings.
• We conduct a thorough evaluation of HyperCL to demonstrate its effectiveness in boosting the link prediction performance of four hyper-relational KG embedding methods on three real-world KGs. Results show that HyperCL can consistently boost the performance of these methods with an average improvement of 3.1-7.4% across the three datasets.

heading: 2 Related Work

heading: 2.1 Hyper-Relational Facts Modeling
text: The triple representation of a KG oversimplifies the intricate structure of information stored in the KG (Rosso et al., 2020), especially for hyper-relational facts where each fact is composed of multiple entities and relations. Some previous works employed an n-ary representation for hyper-relational facts, i.e., a set of relation-entity pairs (Wen et al., 2016; Zhang et al., 2018; Guan et al., 2019; Fatemi et al., 2021; Liu et al., 2021; Wang et al., 2023a). Upon such n-ary representations, these approaches learn either relatedness between relation-entity pairs or relatedness among all entities in a fact. However, recent studies (Rosso et al., 2020) discovered that the base triplet of a hyper-relational fact preserves the essential information, and advised directly learning from hyper-relational facts. Following this suggestion, HINGE (Rosso et al., 2020), NeuInfer (Guan et al., 2020), and ShrinkE (Xiong et al., 2023) separately model base triplets and key-value pairs. GRAN (Wang et al., 2021) proposes a heterogeneous graph to distinguish between the relation-entity connections in base triplets and those in key-value pairs. HyNT (Chung et al., 2023) devises a context Transformer to learn the representations of numeric literals in either triplets or qualifiers. HyperFormer (Hu et al., 2023) encodes the local-level sequential information in hyperrelational facts with Transformers. MSeaHKG (Di and Chen, 2021), StarE (Galkin et al., 2020), HyT (Yu and Yang, 2021), QUAD (Shomer et al., 2022), and HAHE (Luo et al., 2023b) design GNNs to represent the base triplets together with key-value pairs.
Our work focuses on a different perspective to improve current hyper-relational KG embedding
methods by subtly incorporating the ontology of KGs. To the best of our knowledge, this is the first universal framework for modeling both hyperrelational facts and ontology information.

heading: 2.2 Ontology of KGs
text: The ontology of a KG provides rich descriptions of the semantics of entities, which promotes the representation of the KG (Krompaß et al., 2015). Some recent studies utilized the concepts in ontology as entity types to constrain the representation of entities (Krompaß et al., 2015; Xie et al., 2016; Niu et al., 2020; Cui et al., 2021; Rosso et al., 2021; Yang et al., 2023; Li et al., 2023). RAM (Liu et al., 2021) extends the type-constraint mechanism to encompass hyper-relational facts, representing entity types through linear combinations of latent vectors. HELIOS (Lu et al., 2023a) investigates the problem of hyper-relational schema modeling with flat entity ontology. However, these previous works fail to accommodate the hierarchical structure of ontology. To address the above limitations, JOIE (Hao et al., 2019) and DGS (Iyer et al., 2022) develop a joint learning architecture to learn from both triplets and ontology. sHINGE (Lu et al., 2023b), tNaLP (Guan et al., 2021), and DHGE (Luo et al., 2023a) follow this fashion, using two pipelines to represent hyper-relational facts and the ontology, respectively. Nevertheless, sHINGE and tNaLP parallelly learn from multiple types for an entity, also neglecting the hierarchical nature of ontology; DHGE overlooks the dominance issue of facts over ontology, rendering the model training dominated by hyper-relational facts while the ontology information is barely encoded into entity representations.
We argue that the above concerns can be well addressed by our HyperCL framework.

heading: 3 Preliminaries
text: In this section, we introduce some important notions about the Hyper-relational Knowledge Graph (HKG) and present the latest encoder-decoder architecture for HKG embedding.

heading: 3.1 Hyper-Relational Knowledge Graphs
text: We formalize two views of the HKG and present the definition of the link prediction task on it.
Instance view of the HKG. The instance view of the HKG consists of an entity set E and an instance relation set RI . A hyper-relational fact
from the instance view can be represented as a base triplet (h, rI , t) with a set of associated keyvalue pairs {(ki, vi)}ni=1, where h, t, vi ∈ E and rI , ki ∈ RI .
Ontology view of the HKG. The ontology view of the HKG is comprised of the same entity set E , a concept set C and an ontology relation set RO. The ontology relation set can be further divided into two subsets ROe and ROc, representing entityconcept relations and concept-concept relations, respectively. Accordingly, there exist two kinds of triplets in the ontology view, (e, rOe, c) ∈ E × ROe × C and (ci, rOc, cj) ∈ C ×ROc × C.
Link prediction on the HKG. The task of link prediction on the HKG is to predict a missing element from hyper-relational facts in the instance view. For a hyper-relational fact, the missing one can be any entity in {h, t, v1, v2, . . . , vn} or any relation in {rI , k1, k2, . . . , kn}.
Since the ontology view only contains triplets, the term “hyper-relational fact” is specifically used to denote the facts in the instance view throughout this paper.

heading: 3.2 Encoder-Decoder Architecture for HKGs
text: The encoder-decoder architecture is the most prevalent and widely adopted framework for HKG embedding, which proves to be effective in link prediction (Galkin et al., 2020; Luo et al., 2023b). As shown in Fig. 3, this architecture (the grey part) is composed of an encoder (mostly GNNs) and a decoder (mostly Transformers). Specifically, the encoder captures the intricate relationship between entities E and relations RI in the instance view, encoding the structural information to obtain the updated embeddings Ê and R̂I . The decoder extends the capabilities of the architecture by capturing the semantic correlation between entities and relations within each hyper-relational fact, generating the final output for link prediction.
In this work, we inherit the encoder-decoder architecture as the backbone of our framework, and integrate multiple state-of-the-art HKG embedding methods to validate the effectiveness of our framework in the experiments.

heading: 4 Methodology
text: This section introduces our universal contrastive learning framework (HyperCL), for hyperrelational KG embeddings. As shown in Fig. 3, our Concept-aware Contrastive Learning (CCL) mod-
ule is proposed to be universally compatible with any encoder-decoder architecture. Specifically, our CCL consists of two key components: 1) relationaware graph attention networks to obtain the updated entity embeddings in the ontology view Ẽ ; 2) a concept-aware contrastive loss function that captures the shared information by both views to get the final entity representations E . In the following, we elaborate on the above two components and present a multi-task learning approach for model training.

heading: 4.1 Relation-Aware Graph Attention Layers
text: For the ontology view, we employ a graph encoder to capture the sophisticated hierarchical concept information and encode it into high-order entity embeddings. Since the vanilla Graph Neural Networks (GNNs) fail to accommodate the diverse ontology relation types, we refine the original Graph Attention Networks (GATs) (Veličković et al., 2017) with relation type embeddings to adaptively incorporate the heterogeneous relationships into node representations.
Specifically, the ontology view can be regarded as a graph GO, where each entity or concept is associated with a node in GO. Without loss of generality, we depict a single relation-aware graph attention layer in the following. Given a node i, its neighbors are denoted by Ni. The aggregation of the first-hop structural information of i can be expressed as a linear combination of its neighboring nodes’ representations:
hNi = ∑
j∈Ni αijhj (1)
where hj refers to the representation of node j and αij denotes the attention score from node i to node j, which is computed by:
αij = exp
( a [ WOhi∥WOhj∥Wbbr(i,j) ]) ∑
k∈Ni exp ( a [ WOhi∥WOhk∥Wbbr(i,k) ])
(2) where a represents the attention mechanism that applies a single layer of feed-forward neural network with the LeakyReLU activation function. WO and Wb are learnable parameters. r(i, j) denotes the relation type between node i and node j and br(i,j) denotes the embedding of r(i, j). Then the representation of node i is updated by:
h (l) i = σ
( W
(l) l ( h (l−1) i + h (l−1) Ni )) (3)
where W(l)l is the learnable parameter at the lth layer and σ refers to the activation function. Through multi-layer message passing and information aggregation, we can obtain the final embeddings of entities in the ontology view Ẽ .

heading: 4.2 Concept-Aware Contrastive Loss
text: After receiving the updated entity embeddings Ê and Ẽ from the instance and ontology views respectively, a concept-aware contrastive loss is developed to pull the representations of the same entity across two views together while separates apart those of different but semantically similar entities, thereby strengthening the distinction between entity representations and alleviating the dominance of instance view information. Furthermore, the concept-aware contrastive loss is developed to distinguish positive and hard-negative samples (entities under the same/similar concept). This design is motivated by the fact that the easy-negative samples can be effectively distinguished by minimizing the link prediction loss as existing approaches do (e.g., StarE (Galkin et al., 2020), HyT (Yu and Yang, 2021), QUAD (Shomer et al., 2022), and HAHE (Luo et al., 2023b)), while our conceptaware contrastive loss complementarily focuses on the hard-negative samples. A similar idea has also been adopted by (Yang et al., 2023), where a fastthinking process efficiently filters out easy-negative
samples, while a slow-thinking process focuses on distinguishing hard-negative samples.
Specifically, a concept-aware batch selection strategy is proposed, ensuring all entities in a batch belong to a common concept. For a concept in the ontology view, the Breadth-First Search (BFS) algorithm is used to collect entities belonging to the concept. As shown in Fig. 4, the BFS starts from the three concepts ci, cj and ck, and attains their corresponding entity sets {ei, ej}, {ek, em, ep} and {ei, ej , ek, em, ep}, respectively. Then batches are selected from these entity sets while meeting the requirement that one batch can only be randomly sampled from one entity set. To prevent overfitting, we implement a size threshold to select batches from entity sets whose sizes are larger than the threshold. Compared to the traditional random sampling-based batch selection, our strategy has three advantages: 1) enforces contrastive learning to focus on separating entities with similar semantics (common concepts), thereby facilitating the distinct representation of each entity; 2) sets up a size threshold to concentrate on those hard to be distinguished entities, thus improving the efficiency of contrastive learning; 3) maintains inherent distribution biases of concepts while implementing batch selection. For each entity ei in a batch, we hold the two views of the same entity as a positive pair (êi, ẽi). On the other hand, any other entity ej
in the same batch is deemed a negative entity and is used to construct the negative pairs (êi, ẽj) and (êj , ẽi). Finally, an extended InfoNCE loss (Wang et al., 2023b) is utilized as the contrastive loss:
LCL(i) =
− log exp
( s(êi,ẽi)
τcl
)
∑ j∈Hi∪{i} ( exp ( s(êi,ẽj) τcl ) + exp ( s(êj ,ẽi) τcl ))
(4) where s(·) is a cosine similarity metric to measure the similarity between two vectors, Hi is the set of negative entities for ei and τcl is the temperature hyperparameter controlling the strength of penalties on negative entities.

heading: 4.3 Multi-task Training
text: The CCL module subtly connects entity embeddings in both views in a self-supervised manner. To ensure the separability and flexibility of CCL, only the updated entity representations in the instance view are fed into the subsequent modules in HyperCL for link prediction, generating a link prediction loss LLP , similar to previous HKG embedding methods. Hence, the overall loss function is defined as:
L = LLP + λLCL (5)
where λ is a hyperparameter trading off the two losses.
Given the difficulty in identifying the most suitable λ, we employ a multi-task training strategy by alternating the training procedures of link prediction and contrastive learning. The corresponding parameters in the two pipelines are updated alternatively until the link prediction pipeline reaches convergence. Note that during the training phase of the contrastive learning pipeline, any entity in the same batch can be used as the negative entity for
others. The code of HyperCL is publicly available online1.

heading: 5 Experiments
text: In this section, we present the experimental setup, results, and discussion, answering the following questions. RQ1: Can HyperCL consistently boost the link prediction performance of different hyperrelational KG embedding methods? RQ2: What’s the impact of the concept-aware contrastive loss on link prediction performance? RQ3: What’s the impact of modeling the hierarchical structure of ontology on link prediction performance?

heading: 5.1 Experimental Setup

heading: 5.1.1 Datasets
text: We conduct experiments on three commonly used hyper-relational KG datasets JF17K (Wen et al., 2016), WikiPeople (Guan et al., 2019), and WD50K (Galkin et al., 2020), where the data provider already splits the training and test datasets. As these datasets do not contain ontology information, we crawl concepts from their corresponding data sources (Freebase and Wikidata). For Freebase, we extract concepts directly from the entity node depicted as "/type/object", where the hierarchical concepts for an entity are also exhibited. For Wikidata, we first collect concepts through the property "instance_of" for each entity and then extract deeper concepts through the property "subclass_of" for each concept, until no deeper concepts are found. Table 1 shows the statistics of our datasets.

heading: 5.1.2 Baselines
text: We consider a sizeable collection of state-of-the-art techniques from two categories. The first category includes model learning from hyper-relational facts only: m-TransH (Wen et al., 2016); RAE (Zhang et al., 2018); NaLP (Guan et al., 2019); NeuInfer (Guan et al., 2020); HINGE (Rosso et al., 2020); ShrinkE (Xiong et al., 2023); GRAN (Wang et al., 2021); MSeaHKG (Di and Chen, 2021); HyNT (Chung et al., 2023); HyperFormer (Hu et al., 2023); StarE (Galkin et al., 2020); HyT (Yu and Yang, 2021); QUAD (Shomer et al., 2022); HAHE (Luo et al., 2023b). The second category includes model learning from both hyper-relational facts and ontology of KGs: RAM (Liu et al., 2021); tNaLP
1https://github.com/UM-Data-Intelligence-Lab/ HyperCL_code
(Guan et al., 2021); sHINGE (Lu et al., 2023b); DHGE (Luo et al., 2023a). Detailed descriptions of baselines are in Appendix A.
Among these baselines, we instantiate our HyperCL with four state-of-the-art encoder-decoder techniques StarE, HyT, QUAD, and HAHE to validate its effectiveness.

heading: 5.1.3 Evaluation Metrics
text: Link prediction is a typical task for evaluating the performance of KG embedding. For the missing entity (or relation) in a test fact, a ranking list of entities (or relations) is predicted. In this ranking list, in addition to the ground truth from the test fact, other entities (or relations) might also be true; we thus adopt the filtered setting (Bordes et al., 2013) to remove them from the ranking list. Mean Reciprocal Rank (MRR), Hits@1, and Hits@10 are used as evaluation metrics. We report the average results in predicting all entities and all relations separately.

heading: 5.1.4 Hyperparameters and Environment
text: Our HyperCL is trained for 300 epochs with early stopping on the benchmark hardware (Intel Xeon5320@2.20GHz, 256GB RAM@3200Hz,
NVIDIA GeForce RTX 3090 24GB, Ubuntu 22.04). The hyperparameter settings for each dataset are shown in Appendix B and the training details are presented in Appendix C. Note that HyperCL only incurs a marginal computational overhead of up to 10.4% in the training time, compared to the baseline models (see Appendix C for details).

heading: 5.2 Overall Performance (RQ1)
text: Table 2 shows the overall performance on all three datasets and we highlight the best-performing result on each task and for each dataset. We observe that HyperCL+HAHE consistently outperforms all baselines, achieving 3.4% and 0.6% improvement on average over the best-performing baselines in predicting entities and relations, respectively.
Notably, we observe that our HyperCL framework consistently improves the link prediction performance of the corresponding base KG embedding methods StarE, HyT, QUAD, and HAHE in all tasks on all datasets, which demonstrates the effectiveness of HyperCL. In particular, HyperCL enhances the performance of the above four baselines with an average improvement of 3.1-7.4% across different datasets in predicting entities. An-
other interesting observation is that while tNaLP and sHINGE are extensions of NaLP and HINGE with the consideration of the ontology of KGs, they only achieve a little improvement since they neglect the hierarchical structure of the ontology. Moreover, DHGE considers the hierarchical ontology but still yields suboptimal performance compared to HyperCL-enhanced baselines. This is attributed to its joint modeling of hyper-relational facts and the hierarchical ontology, which introduces the dominance issue as illustrated in Section 1. The above observation further verifies that properly incorporating ontology is critical for performance improvement, which is the key merit of HyperCL.

heading: 5.3 Ablation Study
text: The concept-aware contrastive loss and relationaware GATs are two essential components of our HyperCL. We consider three variants to quantify their impact on link prediction performance and demonstrate their utility. Each variant is tuned individually with different optimal hyperparameters (see Appendix B for details) and the best prediction results are reported.

heading: 5.3.1 Impact of the Concept-Aware Contrastive Loss (RQ2)
text: We devise two variants of HyperCL to demonstrate the effectiveness of the concept-aware contrastive loss and its concept-aware batch selection strategy, respectively. The first variant removes the whole contrastive loss and directly integrates the ontology into the original encoder with relation-aware GATs, thus generating a joint model for knowledge embedding while maintaining the hierarchical structure of the ontology. This variant is denoted as w/o loss. The second variant replaces the concept-
aware batch selection in the contrastive loss with random sampling-based batch selection, denoted as w/o concept.
Table 3 presents the results. We observe that the concept-aware contrastive loss and the conceptaware batch selection strategy both contribute to the link prediction improvement. In particular, the concept-aware contrastive loss enhances the performance of four baselines with an average improvement of 2.5-5.8% across different datasets, accounting for 75% performance improvement of the complete HyperCL framework (compared to the average improvement of 3.1-7.4% by the complete HyperCL framework in Section 5.2). This implies that the dominance issue is a key factor resulting in the suboptimal performance of joint learning models, which can be largely mitigated by our HyperCL. In addition, the concept-aware batch selection strategy also improves the performance of baselines, implying that separating semantically similar entities indeed benefits contrastive learning.

heading: 5.3.2 Impact of the Hierarchical Ontology (RQ3)
text: We further devise a variant of HyperCL to verify the effectiveness of the hierarchical ontology, where only the most specific concepts are reserved while other deeper concepts are discarded. Thus, the ontology of KGs loses its hierarchical information, denoted by w/o hierarchy.
As shown in Table 3, the hierarchical structure of ontology facilitates the hyper-relational knowledge embedding of baselines and thus improves their link prediction performance. Moreover, we have further investigated the link prediction improvements and found that entities with deeper concepts benefit more from the hierarchical ontology. Specif-
ically, entities with 3 or more layers of concepts gain an average improvement of 5.7% across different datasets while entities with less than 3 layers of concepts only earn an average improvement of 0.8%. To intuitively demonstrate the advantage of modeling the hierarchical ontology, we conduct a case study of predicting the missing value of a hyper-relational fact in Fig. 5. The missing entity should possess similar semantics with computer program since they are affiliated with the same base triplet (Apple, industry, software industry) and have the common key of. The ground-truth entity is operating system, which is correctly predicted by HyperCL+HAHE. However, the w/o hierarchy variant answers expert system since it belongs to the same specific concept program with computer program. This implies that the loss of the hierarchical ontology narrows the view of HyperCL and makes it overlook possible candidates for link prediction, such as the entity operating system that belongs to the same high-level concept software as computer program does.

heading: 6 Conclusion
text: In this paper, we propose HyperCL, a universal contrastive learning framework for hyper-relational knowledge graph embedding, which considers the hierarchical structure of the ontology of KGs and alleviates the dominance issue of hyper-relational facts over the ontology. Experimental results show that HyperCL consistently boosts the performance of state-of-the-art baselines with an average improvement of 3.1-7.4% across three datasets, demonstrating the effectiveness of HyperCL.
In the future, we plan to further extend HyperCL to other KG embedding architectures and investigate graph augmentation techniques for conceptaware contrastive learning.

heading: 7 Limitations
text: As shown in the experiments, HyperCL is more powerful in predicting entities than relations. Besides the reason that the problem space of relations is usually much smaller than that of entities, this is also partially attributed to the current framework not taking into account the contrastive learning of relations. In the future, we will consider extending our framework to contrastive relations as a unified architecture.

heading: 8 Ethics Statement
text: This paper investigates the problem of knowledge graph link prediction, aiming at hyper-relational knowledge graph completion with ontology information to empower a wide range of web applications, such as question answering, recommender systems, and query expansion. The KG datasets used in this paper are all publicly available. Therefore, we believe it does not raise any ethical issues.

heading: A Baseline Details
text: The first category includes model learning from hyper-relational facts only: m-TransH (Wen et al., 2016) models the interaction between entities involved in each n-ary fact; RAE (Zhang et al., 2018) extends m-TransH by explicitly considering the pairwise relatedness between entities in n-ary facts; NaLP (Guan et al., 2019) learns the relatedness between relation-entity pairs under the n-ary representation of hyper-relational facts; NeuInfer (Guan et al., 2020) models both primary
triplet and its associated key-value pairs; HINGE (Rosso et al., 2020) captures both the triple-wise and quintuple-wise relatedness between elements in hyper-relational facts; ShrinkE (Xiong et al., 2023) models the primary triplets as a spatialfunctional transformation from the head into a relation-specific box; GRAN (Wang et al., 2021) represents the hyper-relational facts as a heterogeneous graph and captures the inter-vertex interactions via self-attention mechanism; MSeaHKG (Di and Chen, 2021) develops a generic messagepassing function to encode hyper-relational facts; HyNT (Chung et al., 2023) devises a context Transformer to learn the representations of numeric literals in either triplets or qualifiers; HyperFormer (Hu et al., 2023) encodes the local-level sequential information in hyper-relational facts with Transformers; StarE (Galkin et al., 2020) transforms a hyper-relational fact into a directed heterogeneous graph and extract the inter-vertex interaction using a GNN encoder; HyT (Yu and Yang, 2021) extends StarE substituting the graph encoder by a light-weight relation/entity embedding technique; QUAD (Shomer et al., 2022) also extends StarE by adopting two separate aggregators to encode the primary triplets and associated key-value pairs, respectively; HAHE (Luo et al., 2023b) adopts global and local hypergraph attention to represent hyper-relational facts. The second category includes model learning from both hyperrelational facts and ontology of KGs: RAM (Liu et al., 2021) captures the latent compatibility between the meta-relation and all involved entities by a pattern matrix; tNaLP (Guan et al., 2021) extends NaLP with the consideration of schema information; sHINGE (Lu et al., 2023b) models the hyper-relational schema information to enhance link prediction performance; DHGE (Luo et al., 2023a) jointly learns from hyper-relational facts and the hierarchical ontology of KGs.

heading: B Hyperparameter Settings
text: Three key hyperparameters of HyperCL are the number of relation-aware graph attention layers L, the threshold of concept-aware batch selection β, and the temperature of contrastive loss τcl. We use the grid search method to identify the optimal hyperparameter setting for each HyperCL-combined model. The range of candidate values for hyperparameters L, β, and τcl are {1, 2, 3, 4, 5}, {1024, 2048, 4096, 8192, 12288}, and {0.5, 0.6, 0.7, 0.8,
0.9}, respectively. Afterward, the optimal hyperparameter setting of a model is selected from exhaustive hyperparameter combinations by comparing the link prediction performance under the different combinations. The final hyperparameter settings for all models are shown in Table 4.

heading: C Training Details
text: All models in Table 4 are trained 300 epochs with early stopping on each dataset with their corresponding optimal hyperparameter settings. The average increased training time due to HyperCL on three datasets JF17K, WikiPeople, and WD50K is 1.2h, 2.9h, and 1.9h, respectively. In comparison, the training time of the most efficient baseline HAHE on JF17K, WikiPeople, and WD50K is 11.5h, 28.4h, and 22.5h, respectively. Therefore, HyperCL only incurs a marginal computational overhead (up to 10.4% in training time).

heading: Future Work
text: In the future, we will consider extending our framework to contrastive relations as a unified architecture