Message Function Search for Hyper-relational Knowledge Graph

1 INTRODUCTION . Knowledge base ( KB ) ( Auer et al. , 2007 ) is an important tool to organize and explore human knowledge , thereby promoting a series of applications , e.g. , question answering ( Lukovnikov et al. , 2017 ) and recommendation system ( Cao et al. , 2019 ) . Generally , the KB stores the n-ary fact r ( e1 , · · · , en ) ( n is arity ) that represents the relation r ∈ R between real-world entities ei ∈ E. To manipulate large scale KBs , KB embedding ( Nickel et al. , 2015 ; Wang et al. , 2017 ) proposes to encode the set of relations R and entities E into a d-dimensional vector space R ∈ R|R|×d , E ∈ R|E|×d . In last decades , the research community mainly focuses on embedding knowledge graphs ( KGs ) that only contain binary facts { r ( e1 , e2 ) } , e.g. , isCaptialOf ( Beijing , China ) . Among kinds of KG embedding models ( Rossi et al. , 2021 ) , tensor models ( Lacroix et al. , 2018 ; Balazevic et al. , 2019 ) propose to represent a KG into a 3-order tensor and decompose tensors into R and E , which achieve outstanding empirical performance and theoretical guarantees . Recent studies start to learn embeddings from n-ary facts ( n ≥ 2 ) because n-ary facts are widespread in KBs , e.g. , playCharacterIn ( LeonardNimoy , Spock , StarTrek1 ) . For example , more than 30 % of entities in Freebase ( Bollacker et al. , 2008 ) involve facts with higher arity ( Wen et al. , 2016 ) . Therefore , it is necessary to investigate the more general case of KGs , facts with mixed arities S = { r ( e1 , . . . , en ) : n ∈ { 2 , . . . , N } } named as hyper-relational KGs ( HKGs ) . Unfortunately , it is hard to extend powerful tensor models from the case of fixed arity ( e.g. , KG ) to the case of mixed arities ( i.e. , HKG ) . That is because a tensor can only model a set of facts under the same arity . Instead , some pioneer works ( Yadati , 2020 ; Galkin et al. , 2020 ) demonstrate that the multi-relational hypergraph ( MRHG ) could be a more natural way to model HKGs ( see Appx . A for more details ) . Let entities E and relations R be nodes and edge types in the MRHG G ( E , R , S ) , respectively . The length of MRHG ’ s edge ( e1 , . . . , en ) ( hyperedge ) could be variant , which can represent facts with various arities n. And hyperedges can be labeled by multiple edge types r ∈ R like r ( e1 , . . . , en ) ∈ S. Under the MRHG modeling , these works adapt powerful graph neural networks ( GNNs ) ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ) to embed HKGs ( Yadati , 2020 ; Galkin et al. , 2020 ) . Generally , GNNs learn node embeddings by passing messages ( e.g. , the node features ) from adjacent nodes to the center node . But in scenarios of HKGs , it is important to know the type of edge ( relation ) that connects several entities . Therefore , existing GNNs for HKG embedding design several domain-specific message functions to learn relational representationsR by capturing the interaction between entities E and relations R. Existing works manually design and fix the structures and operators of message functions . However , such rigid message function designs are not conducive to pursuing high empirical performance , as relations usually have distinct patterns in various KGs/HKGs . For example , the message function of G-MPNN ( Yadati , 2020 ) adopts the inner product way like DistMult ( Yang et al. , 2015 ) to compute the correlation between entities and relations , which has been proven to only cover symmetric relations ( Kazemi & Poole , 2018 ) . Its performance may not be good if there are many non-symmetric relations existed ( see Appx . B.1 ) . It may be a potential solution to design a universal message function to cover as many relational patterns as possible . But covering a certain pattern does not mean that the model can reach good performance on it ( Meilicke et al. , 2018 ; Rossi et al. , 2021 ) . Moreover , a pioneer work AutoSF ( Zhang et al. , 2020 ) shows that designing data-aware models can consistently achieve high performance on any given KGs . Thus , dynamically searching message functions could be an effective way to capture the data patterns of the given HKG and pursue high performance . Unfortunately , the searching method AutoSF is strictly restricted to bilinear KG models ( Yang et al. , 2015 ; Kazemi & Poole , 2018 ) , which is not applicable to message function design and the HKG scenario . Besides , although neural architecture search ( NAS ) ( Elsken et al. , 2019 ) has been introduced to search GNN architectures , current GNN search spaces ( Zhang et al. , 2021 ) are not well regularized to tasks on HKGs . Specifically , the existing search spaces of message function follow the classic way to simply aggregate node neighbors to learn node embeddings , while ignoring edge embeddings to represent relations . Thus , they can not capture the correlation between entities and relations of HKGs . In summary , rigid message function designs for HKGs are not conducive to consistently pursuing high performance on different data sets , while existing searching methods are not applicable for HKGs . To bridge this research gap , we propose the Message function SEArch for any given HKGs , named as MSeaHKG . However , it is non-trivial to design an expressive search space to enable the powerful message functions being searched , especially the space can not be too large for the sake of search efficiency . Thus , we identify the necessary computation operators that are domian-specific designs for HKGs and propose to search the interaction between these operators for capturing the relational patterns . Moreover , except for the message function search , we also incorporate other GNN components ( e.g. , aggregation function ) in the MSeaHKG search space for more performance improvements . Then , we formulate the discrete HKG models with probabilistic modelings to enable an efficient NAS algorithm working on our scenario . The main contributions are listed as : • Previous GNN searching methods generally ignore the edge representations , which fails to handle semantic meaningful relations on HKGs . Besides , their message functions can not capture complex interactions between entities with relations . In this paper , we propose a searching method to dynamically design a suitable GNN that can achieve high performance on the given HKG . • Inspired by rigid message function designs , we define a novel search space of message functions for HKGs , which enables the message function to be flexibly searched on the given HKG . Especially , the message function designs of existing GNNs for HKGs and some classic KG/HKG models can be instantiated as special cases of the proposed space . • We conduct experiments on benchmark HKGs for the link prediction and relation prediction tasks . Experimental results demonstrate that MSeaHKG can consistently achieve state-of-the-art performance by designing data-aware message functions . Besides , we also transfer MSeaHKG to other graph-based tasks and further investigate its capability . 2 RELATED WORK . 2.1 ONE-SHOT NEURAL ARCHITECTURE SEARCH IN GRAPH NEURAL NETWORK . To avoid manual efforts on neural architecture designs , NAS ( Hutter et al. , 2018 ; Yao & Wang , 2018 ) aims to automatically search suitable neural architectures for the given data and task . Generally , search Type Model Scenarios Message Function . space , search algorithm , and evaluation measurement are three important components in NAS ( Elsken et al. , 2019 ) . Search space defines what network architectures in principle should be searched . The search algorithm performs an efficient search over the search space and finds architectures that achieve good performance . Evaluation measurement decides how to evaluate the searched architectures during the search . Classical NAS methods are computationally consuming because candidate architectures are evaluated by the stand-alone way , i.e. , evaluating the performance of architecture after training it to convergence . More recently , one-shot NAS ( Pham et al. , 2018 ) proposes the weight sharing mechanism to share network weights across different candidate architectures and evaluate them on the shared weights , which can extremely reduce the search cost . Some pioneer works have explored NAS for GNNs , such as You et al . ( 2020 ) , GraphNAS ( Gao et al. , 2020 ) , AGNN ( Zhou et al. , 2019 ) , NAS-GCN ( Jiang & Balaprakash , 2020 ) . And the one-shot NAS also has been introduced to search GNN architectures recently , e.g. , SANE ( Zhao et al. , 2021 ) . As shown in the left part of Fig . 1 , most GNN searching methods follow the message passing neural networks ( MPNNs ) ( Gilmer et al. , 2017 ) to unify two steps of the GNN framework in one layer : step1 : mi ← agg ( { mgc ( ei , ej ) } ej∈N ( ei ) ) , step2 : ei ← act ( comb ( ei , mi ) ) , ( 1 ) where ei ∈ Rd represents the embedding of node ei , mi is the intermediate embeddings of ei gathered from its neighborhood N ( ei ) . The search space of operators in Eq . 1 are summarized into : • Message Function mgc ( · ) : The message function decides the way to gather information from a neighborhood ej of the center node ei . Zhang et al . ( 2021 ) summarizes the typical message in existing GNN searching methods as mgc ( ei , ej ) = aijWej , where aij denotes the attention scores between nodes ei with ej . Besides , we present more message function designs in Tab . 1 . • Aggregation Function agg ( · ) : It controls the way to aggregate message from nodes ’ neighborhood . Usually agg ∈ { sum , mean , max } , where sum ( · ) = ∑ ej∈N ( ei ) mgc ( ei , ej ) , mean ( · ) = ∑ ej∈N ( ei ) mgc ( ei , ej ) /|N ( v ) | , and max ( · ) denotes channel-wise maximum . • Combination Function comb ( · ) : It determines the way to merge messages between neighborhood and node itself . In literature , comb is selected from { concat , add , mlp } , where concat ( · ) = [ ei , mi ] , add ( · ) = ei + mi , and mlp ( · ) = MLP ( ei +mi ) ( MLP is Multi-layer Perceptron ) . • Activation Function act ( · ) : [ identity , sigmoid , tanh , relu , elu ] are some of the most commonly used activation functions ( Gao et al. , 2020 ) . Overall , mgc ( · ) in Eq . 1 only learns node embeddings , which can not encode the semantic meaningful edge types ( i.e. , relations in HKGs ) . Note that NAS-GCN ( Jiang & Balaprakash , 2020 ) takes the edge feature hij between ei and ej as input without learning edge embeddings , and recent AutoGEL ( Zhili et al. , 2021 ) simply extends the message function mgc ( · ) to learn edge embeddings without studying the interactions between entities and relations , thereby failing to handle the LP/RP tasks on HKGs . Entity Embed Relation Embed Computing Operator Role/Position Embed Positional matrix Transform matrix ... ... Cat & MLP Message Function Space

Summary: The paper proposes a new neural architecture search framework over graph neural networks in hyper relational knowledge graphs (HKG). The main motivation is that the current GraphNAS system does not have an emphasis on edge representations and how message passing functions handle position-sensitive information on HKG. The paper proposes a new search space of these configurations and proposes a new HKG method that achieves state-of-the-art on several benchmarks.