Message Function Search for Hyper-relational Knowledge Graph

1 INTRODUCTION . Knowledge base ( KB ) ( Auer et al. , 2007 ) is an important tool to organize and explore human knowledge , thereby promoting a series of applications , e.g. , question answering ( Lukovnikov et al. , 2017 ) and recommendation system ( Cao et al. , 2019 ) . Generally , the KB stores the n-ary fact r ( e1 , · · · , en ) ( n is arity ) that represents the relation r ∈ R between real-world entities ei ∈ E. To manipulate large scale KBs , KB embedding ( Nickel et al. , 2015 ; Wang et al. , 2017 ) proposes to encode the set of relations R and entities E into a d-dimensional vector space R ∈ R|R|×d , E ∈ R|E|×d . In last decades , the research community mainly focuses on embedding knowledge graphs ( KGs ) that only contain binary facts { r ( e1 , e2 ) } , e.g. , isCaptialOf ( Beijing , China ) . Among kinds of KG embedding models ( Rossi et al. , 2021 ) , tensor models ( Lacroix et al. , 2018 ; Balazevic et al. , 2019 ) propose to represent a KG into a 3-order tensor and decompose tensors into R and E , which achieve outstanding empirical performance and theoretical guarantees . Recent studies start to learn embeddings from n-ary facts ( n ≥ 2 ) because n-ary facts are widespread in KBs , e.g. , playCharacterIn ( LeonardNimoy , Spock , StarTrek1 ) . For example , more than 30 % of entities in Freebase ( Bollacker et al. , 2008 ) involve facts with higher arity ( Wen et al. , 2016 ) . Therefore , it is necessary to investigate the more general case of KGs , facts with mixed arities S = { r ( e1 , . . . , en ) : n ∈ { 2 , . . . , N } } named as hyper-relational KGs ( HKGs ) . Unfortunately , it is hard to extend powerful tensor models from the case of fixed arity ( e.g. , KG ) to the case of mixed arities ( i.e. , HKG ) . That is because a tensor can only model a set of facts under the same arity . Instead , some pioneer works ( Yadati , 2020 ; Galkin et al. , 2020 ) demonstrate that the multi-relational hypergraph ( MRHG ) could be a more natural way to model HKGs ( see Appx . A for more details ) . Let entities E and relations R be nodes and edge types in the MRHG G ( E , R , S ) , respectively . The length of MRHG ’ s edge ( e1 , . . . , en ) ( hyperedge ) could be variant , which can represent facts with various arities n. And hyperedges can be labeled by multiple edge types r ∈ R like r ( e1 , . . . , en ) ∈ S. Under the MRHG modeling , these works adapt powerful graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ) to embed HKGs ( Yadati , 2020 ; Galkin et al. , 2020 ) . Generally , GNNs learn node embeddings by passing messages ( e.g. , the node features ) from adjacent nodes to the center node . But in scenarios of HKGs , it is important to know the type of edge ( relation ) that connects several entities . Therefore , existing GNNs for HKG embedding design several domain-specific message functions to learn relational representationsR by capturing the interaction between entities E and relations R. Existing works manually design and fix the structures and operators of message functions . However , such rigid message function designs are not conducive to pursuing high empirical performance , as relations usually have distinct patterns in various KGs/HKGs . For example , the message function of G-MPNN ( Yadati , 2020 ) adopts the inner product way like DistMult ( Yang et al. , 2015 ) to compute the correlation between entities and relations , which has been proven to only cover symmetric relations ( Kazemi & Poole , 2018 ) . Its performance may not be good if there are many non-symmetric relations existed ( see Appx . B.1 ) . It may be a potential solution to design a universal message function to cover as many relational patterns as possible . But covering a certain pattern does not mean that the model can reach good performance on it ( Meilicke et al. , 2018 ; Rossi et al. , 2021 ) . Moreover , a pioneer work AutoSF ( Zhang et al. , 2020 ) shows that designing data-aware models can consistently achieve high performance on any given KGs . Thus , dynamically searching message functions could be an effective way to capture the data patterns of the given HKG and pursue high performance . Unfortunately , the searching method AutoSF is strictly restricted to bilinear KG models ( Yang et al. , 2015 ; Kazemi & Poole , 2018 ) , which is not applicable to message function design and the HKG scenario . Besides , although neural architecture search ( NAS ) ( Elsken et al. , 2019 ) has been introduced to search GNN architectures , current GNN search spaces ( Zhang et al. , 2021 ) are not well regularized to tasks on HKGs . Specifically , the existing search spaces of message function follow the classic way to simply aggregate node neighbors to learn node embeddings , while ignoring edge embeddings to represent relations . Thus , they can not capture the correlation between entities and relations of HKGs . In summary , rigid message function designs for HKGs are not conducive to consistently pursuing high performance on different data sets , while existing searching methods are not applicable for HKGs . To bridge this research gap , we propose the Message function SEArch for any given HKGs , named as MSeaHKG . However , it is non-trivial to design an expressive search space to enable the powerful message functions being searched , especially the space can not be too large for the sake of search efficiency . Thus , we identify the necessary computation operators that are domian-specific designs for HKGs and propose to search the interaction between these operators for capturing the relational patterns . Moreover , except for the message function search , we also incorporate other GNN components ( e.g. , aggregation function ) in the MSeaHKG search space for more performance improvements . Then , we formulate the discrete HKG models with probabilistic modelings to enable an efficient NAS algorithm working on our scenario . The main contributions are listed as : • Previous GNN searching methods generally ignore the edge representations , which fails to handle semantic meaningful relations on HKGs . Besides , their message functions can not capture complex interactions between entities with relations . In this paper , we propose a searching method to dynamically design a suitable GNN that can achieve high performance on the given HKG . • Inspired by rigid message function designs , we define a novel search space of message functions for HKGs , which enables the message function to be flexibly searched on the given HKG . Especially , the message function designs of existing GNNs for HKGs and some classic KG/HKG models can be instantiated as special cases of the proposed space . • We conduct experiments on benchmark HKGs for the link prediction and relation prediction tasks . Experimental results demonstrate that MSeaHKG can consistently achieve state-of-the-art performance by designing data-aware message functions . Besides , we also transfer MSeaHKG to other graph-based tasks and further investigate its capability . 2 RELATED WORK . 2.1 ONE-SHOT NEURAL ARCHITECTURE SEARCH IN GRAPH NEURAL NETWORK . To avoid manual efforts on neural architecture designs , NAS ( Hutter et al. , 2018 ; Yao & Wang , 2018 ) aims to automatically search suitable neural architectures for the given data and task . Generally , search Type Model Scenarios Message Function . space , search algorithm , and evaluation measurement are three important components in NAS ( Elsken et al. , 2019 ) . Search space defines what network architectures in principle should be searched . The search algorithm performs an efficient search over the search space and finds architectures that achieve good performance . Evaluation measurement decides how to evaluate the searched architectures during the search . Classical NAS methods are computationally consuming because candidate architectures are evaluated by the stand-alone way , i.e. , evaluating the performance of architecture after training it to convergence . More recently , one-shot NAS ( Pham et al. , 2018 ) proposes the weight sharing mechanism to share network weights across different candidate architectures and evaluate them on the shared weights , which can extremely reduce the search cost . Some pioneer works have explored NAS for GNNs , such as You et al . ( 2020 ) , GraphNAS ( Gao et al. , 2020 ) , AGNN ( Zhou et al. , 2019 ) , NAS-GCN ( Jiang & Balaprakash , 2020 ) . And the one-shot NAS also has been introduced to search GNN architectures recently , e.g. , SANE ( Zhao et al. , 2021 ) . As shown in the left part of Fig . 1 , most GNN searching methods follow the message passing neural networks ( MPNNs ) ( Gilmer et al. , 2017 ) to unify two steps of the GNN framework in one layer : step1 : mi ← agg ( { mgc ( ei , ej ) } ej∈N ( ei ) ) , step2 : ei ← act ( comb ( ei , mi ) ) , ( 1 ) where ei ∈ Rd represents the embedding of node ei , mi is the intermediate embeddings of ei gathered from its neighborhood N ( ei ) . The search space of operators in Eq . 1 are summarized into : • Message Function mgc ( · ) : The message function decides the way to gather information from a neighborhood ej of the center node ei . Zhang et al . ( 2021 ) summarizes the typical message in existing GNN searching methods as mgc ( ei , ej ) = aijWej , where aij denotes the attention scores between nodes ei with ej . Besides , we present more message function designs in Tab . 1 . • Aggregation Function agg ( · ) : It controls the way to aggregate message from nodes ’ neighborhood . Usually agg ∈ { sum , mean , max } , where sum ( · ) = ∑ ej∈N ( ei ) mgc ( ei , ej ) , mean ( · ) = ∑ ej∈N ( ei ) mgc ( ei , ej ) /|N ( v ) | , and max ( · ) denotes channel-wise maximum . • Combination Function comb ( · ) : It determines the way to merge messages between neighborhood and node itself . In literature , comb is selected from { concat , add , mlp } , where concat ( · ) = [ ei , mi ] , add ( · ) = ei + mi , and mlp ( · ) = MLP ( ei +mi ) ( MLP is Multi-layer Perceptron ) . • Activation Function act ( · ) : [ identity , sigmoid , tanh , relu , elu ] are some of the most commonly used activation functions ( Gao et al. , 2020 ) . Overall , mgc ( · ) in Eq . 1 only learns node embeddings , which can not encode the semantic meaningful edge types ( i.e. , relations in HKGs ) . Note that NAS-GCN ( Jiang & Balaprakash , 2020 ) takes the edge feature hij between ei and ej as input without learning edge embeddings , and recent AutoGEL ( Zhili et al. , 2021 ) simply extends the message function mgc ( · ) to learn edge embeddings without studying the interactions between entities and relations , thereby failing to handle the LP/RP tasks on HKGs . Entity Embed Relation Embed Computing Operator Role/Position Embed Positional matrix Transform matrix ... ... Cat & MLP Message Function Space

Summary: In this paper, the authors propose to conduct neural architecture search (NAS) for hyper-relational knowledge graphs (HKGs). Compared with normal graphs, HKGs can better model the complex relationships between different entities. Specifically, a novel search space is proposed inspired by recent message-passing GNNs on HKGs such as G-MPNN, CompGCN, StarE, etc., and a differentiable search algorithm is utilized. Experiments demonstrate the effectiveness of the proposed method.