Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction

1 INTRODUCTION . The ubiquity of graph-structured data and its importance in solving various real-world problems such as node and graph classification have made graph-centered machine learning an important research area ( Lü & Zhou , 2011 ; Shervashidze et al. , 2011 ; Zhu , 2005 ) . Graph neural networks ( GNNs ) offer state-of-the-art performance on many graph learning tasks and have by now become a standard methodology in the field ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Chien et al. , 2020 ) . In most such studies , GNNs take graphs with numerical node attributes as inputs and train them with task-specific labels . Recent research has shown that self-supervised learning ( SSL ) leads to performance improvements in many applications , including graph learning , natural language processing and computer vision . Several SSL approaches have also been successfully used with GNNs ( Hu et al. , 2020b ; You et al. , 2018 ; 2020 ; Hu et al. , 2020c ; Velickovic et al. , 2019 ; Kipf & Welling , 2016 ; Deng et al. , 2020 ) . The common idea behind these works is to explore the correlated information provided by the numerical node features and graph topology , which can lead to improved node representations and GNN initialization . However , one critical yet neglected issue in the current graph learning literature is how to actually obtain the numerical node features from raw data such as text , images and audio signals . As an example , when dealing with raw text features , the standard approach is to apply graph-agnostic methods such as bag-of-words , word2vec ( Mikolov et al. , 2013 ) or pre-trained BERT ( Devlin et al. , 2018 ) ( As a further example , raw texts of product descriptions are used to construct node features via the bag-of-words model for benchmarking GNNs on the ogbn-products dataset ( Hu et al. , 2020a ; Chiang et al. , 2019 ) ) . The pre-trained BERT language model , as well as convolutional neural networks ( CNNs ) ( Goyal et al. , 2019 ; Kolesnikov et al. , 2019 ) , produce numerical features that can significantly improve the performance of various downstream learners ( Devlin et al. , 2018 ) . Still , none of these works leverage graph information for actual self-supervision . Clearly , using graphagnostic methods to extract numerical features is sub-optimal , as correlations between the graph topology and raw features are ignored . Motivated by the recent success of SSL approaches for GNNs , we propose GIANT , an SSL framework that resolves the aforementioned issue of graph-agnostic feature extraction in the standard GNN learning pipeline . Our framework takes raw node attributes and generates numerical node features with graph-structured self-supervision . To integrate the graph topology information into language models such as BERT , we also propose a novel SSL task termed neighborhood prediction , which works for both homophilous and heterophilous graphs , and establish connections between neighborhood prediction and the eXtreme Multi-label Classification ( XMC ) problem ( Shen et al. , 2020 ; Yu et al. , 2020 ; Chang et al. , 2020b ) . Roughly speaking , the neighborhood of each node can be encoded using binary multi-labels ( indicating whether a node is a neighbor or not ) and the BERT model is fine-tuned by successively improving the predicted neighborhoods . This approach allows us to not only leverage the advanced solvers for the XMC problem and address the issue of graph-agnostic feature extraction , but also to perform a theoretical study of the XMC problem and determine its importance in the context of graph-guided SSL . Throughout the work , we focus on raw texts as these are the most common data used for largescale graph benchmarking . Examples include titles/abstracts in citation networks and product descriptions in co-purchase networks . To solve our proposed self-supervised XMC task , we adopt the state-of-the-art XR-Transformer method ( Zhang et al. , 2021a ) . By using the encoder from the XR-Transformer pre-trained with GIANT , we obtain informative numerical node features which consistently boost the performance of GNNs on downstream tasks . Notably , GIANT significantly improves state-of-the-art methods for node classification tasks described on the Open Graph Benchmark ( OGB ) ( Hu et al. , 2020a ) leaderboard on three large-scale graph datasets , with absolute improvements in accuracy roughly 1.5 % for the first-ranked methods , 3 % for standard GNNs and 14 % for multilayer perceptron ( MLP ) . GIANT coupled with XRTransformer is also highly scalable and can be combined with other downstream learning methods . Our contributions may be summarized as follows . 1 . We identify the issue of graph-agnostic feature extraction in standard GNN pipelines and propose a new GIANT self-supervised framework as a solution to the problem . 2 . We introduce a new approach to numerical feature extraction supervised by graph information based on the idea of neighborhood prediction . The gist of the approach is to use neighborhood pre- diction within a language model such as BERT to guide the process of fine-tuning the features . Unlike link-prediction , neighborhood prediction resolves problems associated with heterophilic graphs . 3 . We establish pertinent connections between neighborhood prediction and the XMC problem by noting that neighborhoods of individual nodes can be encoded by binary vectors which may be interpreted as multi-labels . This allows for performing neighborhood prediction via XR-Transformers , especially designed to solve XMC problems at scale . 4 . We demonstrate through extensive experiments that GIANT consistently improves the performance of tested GNNs on downstream tasks by large margins . We also report new state-of-the-art results on the OGB leaderboard , including absolute improvements in accuracy roughly 1.5 % compared to the top-ranked method , 3 % for standard GNNs and 14 % for multilayer perceptron ( MLP ) . More precisely , we improve the accuracy of the top-ranked method GAMLP ( Zhang et al. , 2021b ) from 68.25 % to 69.67 % , SGC ( Wu et al. , 2019 ) from 63.29 % to 66.10 % and MLP from 47.24 % to 61.10 % on the ogbn-papers100M dataset . 5 . We present a new theoretical analysis that verifies the benefits of key components in XRTransformers on our neighborhood prediction task . This analysis also further improves our understanding of XR-Transformers and the XMC problem . Due to the space limitation , all proofs are deferred to the Appendix . 2 BACKGROUND AND RELATED WORK . General notation . Throughout the paper , we use bold capital letters such as A to denote matrices . We use Ai for the i-th row of the matrix and Aij for its entry in row i and column j . We reserve bold lowercase letters such as a for vectors . The symbol I denotes the identity matrix while 1 denotes the all-ones vector . We use o ( · ) , O ( · ) , ω ( · ) , Θ ( · ) in the standard manner . SSL in GNNs . SSL is a topic of substantial interest due to its potential for improving the performance of GNNs on various tasks . Exploiting the correlation between node features and the graph structure is known to lead to better node representations or GNN initialization ( Hu et al. , 2020b ; You et al. , 2018 ; 2020 ; Hu et al. , 2020c ) . Several methods have been proposed for improving node representations , including ( variational ) graph autoencoders ( Kipf & Welling , 2016 ) , Deep Graph Infomax ( Velickovic et al. , 2019 ) and GraphZoom ( Deng et al. , 2020 ) . For more information , the interested reader is referred to a survey of SSL GNNs ( Xie et al. , 2021 ) . While these methods can be used as SSL modules in GNNs ( Figure 1 ) , it is clear that they do not solve the described graph agnostics issue in the standard GNN pipeline . Furthermore , as the above described SSL GNNs modules and other pre-processing and post-processing methods for GNNs such as C & S ( Huang et al. , 2021 ) and FLAG ( Kong et al. , 2020 ) in general improve graph learners , it is worth pointing out that they can be naturally be integrated into the GIANT framework . This topic is left as a future work . The XMC problem , PECOS and XR-Transformer . The XMC problem can be succinctly formulated as follows : We are given a training set { Ti , yi } ni=1 , where Ti ∈ D is the ith input text instance and yi ∈ { 0 , 1 } L is the target multi-label from an extremely large collection of labels . The goal is to learn a function f : D× [ L ] 7→ R , where f ( T , l ) captures the relevance between the input text T and the label l. The XMC problem is of importance in many real-world applications ( Jiang et al. , 2021 ; Ye et al. , 2020 ) : For example , in E-commerce dynamic search advertising , XMC arises when trying to find a “ good ” mapping from items to bid queries on the market ( Prabhu et al. , 2018 ; Prabhu & Varma , 2014 ) . In open-domain question answering , XMC problems arise when trying to map questions to “ evidence ” passages containing the answers ( Chang et al. , 2020a ; Lee et al. , 2019 ) . Many methods for the XMC problem leverage hierarchical clustering approaches for labels ( Prabhu et al. , 2018 ; You et al. , 2019 ) . This organizational structure allows one to handle potentially enormous numbers of labels , such as used by PECOS ( Yu et al. , 2020 ) . The key is to take advantage of the correlations among labels within the hierarchical clustering . In our approach , we observe that the multi-labels correspond to neighborhoods of nodes in the given graph . Neighborhoods have to be predicted using the textual information in order to best match the a priori given graph topology . We use the state-of-the-art XR-Transformer ( Zhang et al. , 2021a ) method for solving the XMC problem to achieve this goal . The high-level idea is to first cluster the output labels , and then learn the instance-to-cluster “ matchers ” ( please refer to Figure 2 ) . Note that many other methods have used PECOS ( including XR-Transformers ) for solving large-scale real-world learning problems ( Etter et al. , 2021 ; Liu et al. , 2021 ; Chang et al. , 2020b ; Baharav et al. , 2021 ; Chang et al. , 2021 ; Yadav et al. , 2021 ; Sen et al. , 2021 ) , but not in the context of self-supervised numerical feature extraction as done in our work . GNNs with raw text data . It is conceptually possible to jointly train BERT and GNNs in an end-to-end fashion , which could potentially resolve the issue of being graph agnostic in the standard pipeline . However , the excessive model complexity of BERT makes such a combination practically prohibitive due to GPU memory limitations . Furthermore , it is nontrivial to train this combination of methods with arbitrary mini-batch sizes ( Chiang et al. , 2019 ; Zeng et al. , 2020 ) . In contrast , the XRTransformer architecture naturally supports mini-batch training and scales well ( Jiang et al. , 2021 ) . Hence , our GIANT method uses XR-Transformers instead of combinations of BERT and GNNs . To the best of our knowledge , we are aware of only one prior work that uses raw text inputs for node classification problem ( Zhang et al. , 2020 ) , but it still follows the standard pipeline described in Figure 1 . Some other works apply GNNs on texts and for document classification , where the actual graphs are constructed based on the raw text . This is clearly not the focus of this work ( Yao et al. , 2019 ; Huang et al. , 2019 ; Zhang & Zhang , 2020 ; Liu et al. , 2020 ) .

Summary: This paper develops a self-supervised learning framework to extract node features with the aid of graph. Connections between neighborhood prediction and the XMC problem are also established. Experiments on large-scale data show the superiority of the proposed method.