Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction

1 INTRODUCTION . The ubiquity of graph-structured data and its importance in solving various real-world problems such as node and graph classification have made graph-centered machine learning an important research area ( Lü & Zhou , 2011 ; Shervashidze et al. , 2011 ; Zhu , 2005 ) . Graph neural networks ( GNNs ) offer state-of-the-art performance on many graph learning tasks and have by now become a standard methodology in the field ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Chien et al. , 2020 ) . In most such studies , GNNs take graphs with numerical node attributes as inputs and train them with task-specific labels . Recent research has shown that self-supervised learning ( SSL ) leads to performance improvements in many applications , including graph learning , natural language processing and computer vision . Several SSL approaches have also been successfully used with GNNs ( Hu et al. , 2020b ; You et al. , 2018 ; 2020 ; Hu et al. , 2020c ; Velickovic et al. , 2019 ; Kipf & Welling , 2016 ; Deng et al. , 2020 ) . The common idea behind these works is to explore the correlated information provided by the numerical node features and graph topology , which can lead to improved node representations and GNN initialization . However , one critical yet neglected issue in the current graph learning literature is how to actually obtain the numerical node features from raw data such as text , images and audio signals . As an example , when dealing with raw text features , the standard approach is to apply graph-agnostic methods such as bag-of-words , word2vec ( Mikolov et al. , 2013 ) or pre-trained BERT ( Devlin et al. , 2018 ) ( As a further example , raw texts of product descriptions are used to construct node features via the bag-of-words model for benchmarking GNNs on the ogbn-products dataset ( Hu et al. , 2020a ; Chiang et al. , 2019 ) ) . The pre-trained BERT language model , as well as convolutional neural networks ( CNNs ) ( Goyal et al. , 2019 ; Kolesnikov et al. , 2019 ) , produce numerical features that can significantly improve the performance of various downstream learners ( Devlin et al. , 2018 ) . Still , none of these works leverage graph information for actual self-supervision . Clearly , using graphagnostic methods to extract numerical features is sub-optimal , as correlations between the graph topology and raw features are ignored . Motivated by the recent success of SSL approaches for GNNs , we propose GIANT , an SSL framework that resolves the aforementioned issue of graph-agnostic feature extraction in the standard GNN learning pipeline . Our framework takes raw node attributes and generates numerical node features with graph-structured self-supervision . To integrate the graph topology information into language models such as BERT , we also propose a novel SSL task termed neighborhood prediction , which works for both homophilous and heterophilous graphs , and establish connections between neighborhood prediction and the eXtreme Multi-label Classification ( XMC ) problem ( Shen et al. , 2020 ; Yu et al. , 2020 ; Chang et al. , 2020b ) . Roughly speaking , the neighborhood of each node can be encoded using binary multi-labels ( indicating whether a node is a neighbor or not ) and the BERT model is fine-tuned by successively improving the predicted neighborhoods . This approach allows us to not only leverage the advanced solvers for the XMC problem and address the issue of graph-agnostic feature extraction , but also to perform a theoretical study of the XMC problem and determine its importance in the context of graph-guided SSL . Throughout the work , we focus on raw texts as these are the most common data used for largescale graph benchmarking . Examples include titles/abstracts in citation networks and product descriptions in co-purchase networks . To solve our proposed self-supervised XMC task , we adopt the state-of-the-art XR-Transformer method ( Zhang et al. , 2021a ) . By using the encoder from the XR-Transformer pre-trained with GIANT , we obtain informative numerical node features which consistently boost the performance of GNNs on downstream tasks . Notably , GIANT significantly improves state-of-the-art methods for node classification tasks described on the Open Graph Benchmark ( OGB ) ( Hu et al. , 2020a ) leaderboard on three large-scale graph datasets , with absolute improvements in accuracy roughly 1.5 % for the first-ranked methods , 3 % for standard GNNs and 14 % for multilayer perceptron ( MLP ) . GIANT coupled with XRTransformer is also highly scalable and can be combined with other downstream learning methods . Our contributions may be summarized as follows . 1 . We identify the issue of graph-agnostic feature extraction in standard GNN pipelines and propose a new GIANT self-supervised framework as a solution to the problem . 2 . We introduce a new approach to numerical feature extraction supervised by graph information based on the idea of neighborhood prediction . The gist of the approach is to use neighborhood pre- diction within a language model such as BERT to guide the process of fine-tuning the features . Unlike link-prediction , neighborhood prediction resolves problems associated with heterophilic graphs . 3 . We establish pertinent connections between neighborhood prediction and the XMC problem by noting that neighborhoods of individual nodes can be encoded by binary vectors which may be interpreted as multi-labels . This allows for performing neighborhood prediction via XR-Transformers , especially designed to solve XMC problems at scale . 4 . We demonstrate through extensive experiments that GIANT consistently improves the performance of tested GNNs on downstream tasks by large margins . We also report new state-of-the-art results on the OGB leaderboard , including absolute improvements in accuracy roughly 1.5 % compared to the top-ranked method , 3 % for standard GNNs and 14 % for multilayer perceptron ( MLP ) . More precisely , we improve the accuracy of the top-ranked method GAMLP ( Zhang et al. , 2021b ) from 68.25 % to 69.67 % , SGC ( Wu et al. , 2019 ) from 63.29 % to 66.10 % and MLP from 47.24 % to 61.10 % on the ogbn-papers100M dataset . 5 . We present a new theoretical analysis that verifies the benefits of key components in XRTransformers on our neighborhood prediction task . This analysis also further improves our understanding of XR-Transformers and the XMC problem . Due to the space limitation , all proofs are deferred to the Appendix . 2 BACKGROUND AND RELATED WORK . General notation . Throughout the paper , we use bold capital letters such as A to denote matrices . We use Ai for the i-th row of the matrix and Aij for its entry in row i and column j . We reserve bold lowercase letters such as a for vectors . The symbol I denotes the identity matrix while 1 denotes the all-ones vector . We use o ( · ) , O ( · ) , ω ( · ) , Θ ( · ) in the standard manner . SSL in GNNs . SSL is a topic of substantial interest due to its potential for improving the performance of GNNs on various tasks . Exploiting the correlation between node features and the graph structure is known to lead to better node representations or GNN initialization ( Hu et al. , 2020b ; You et al. , 2018 ; 2020 ; Hu et al. , 2020c ) . Several methods have been proposed for improving node representations , including ( variational ) graph autoencoders ( Kipf & Welling , 2016 ) , Deep Graph Infomax ( Velickovic et al. , 2019 ) and GraphZoom ( Deng et al. , 2020 ) . For more information , the interested reader is referred to a survey of SSL GNNs ( Xie et al. , 2021 ) . While these methods can be used as SSL modules in GNNs ( Figure 1 ) , it is clear that they do not solve the described graph agnostics issue in the standard GNN pipeline . Furthermore , as the above described SSL GNNs modules and other pre-processing and post-processing methods for GNNs such as C & S ( Huang et al. , 2021 ) and FLAG ( Kong et al. , 2020 ) in general improve graph learners , it is worth pointing out that they can be naturally be integrated into the GIANT framework . This topic is left as a future work . The XMC problem , PECOS and XR-Transformer . The XMC problem can be succinctly formulated as follows : We are given a training set { Ti , yi } ni=1 , where Ti ∈ D is the ith input text instance and yi ∈ { 0 , 1 } L is the target multi-label from an extremely large collection of labels . The goal is to learn a function f : D× [ L ] 7→ R , where f ( T , l ) captures the relevance between the input text T and the label l. The XMC problem is of importance in many real-world applications ( Jiang et al. , 2021 ; Ye et al. , 2020 ) : For example , in E-commerce dynamic search advertising , XMC arises when trying to find a “ good ” mapping from items to bid queries on the market ( Prabhu et al. , 2018 ; Prabhu & Varma , 2014 ) . In open-domain question answering , XMC problems arise when trying to map questions to “ evidence ” passages containing the answers ( Chang et al. , 2020a ; Lee et al. , 2019 ) . Many methods for the XMC problem leverage hierarchical clustering approaches for labels ( Prabhu et al. , 2018 ; You et al. , 2019 ) . This organizational structure allows one to handle potentially enormous numbers of labels , such as used by PECOS ( Yu et al. , 2020 ) . The key is to take advantage of the correlations among labels within the hierarchical clustering . In our approach , we observe that the multi-labels correspond to neighborhoods of nodes in the given graph . Neighborhoods have to be predicted using the textual information in order to best match the a priori given graph topology . We use the state-of-the-art XR-Transformer ( Zhang et al. , 2021a ) method for solving the XMC problem to achieve this goal . The high-level idea is to first cluster the output labels , and then learn the instance-to-cluster “ matchers ” ( please refer to Figure 2 ) . Note that many other methods have used PECOS ( including XR-Transformers ) for solving large-scale real-world learning problems ( Etter et al. , 2021 ; Liu et al. , 2021 ; Chang et al. , 2020b ; Baharav et al. , 2021 ; Chang et al. , 2021 ; Yadav et al. , 2021 ; Sen et al. , 2021 ) , but not in the context of self-supervised numerical feature extraction as done in our work . GNNs with raw text data . It is conceptually possible to jointly train BERT and GNNs in an end-to-end fashion , which could potentially resolve the issue of being graph agnostic in the standard pipeline . However , the excessive model complexity of BERT makes such a combination practically prohibitive due to GPU memory limitations . Furthermore , it is nontrivial to train this combination of methods with arbitrary mini-batch sizes ( Chiang et al. , 2019 ; Zeng et al. , 2020 ) . In contrast , the XRTransformer architecture naturally supports mini-batch training and scales well ( Jiang et al. , 2021 ) . Hence , our GIANT method uses XR-Transformers instead of combinations of BERT and GNNs . To the best of our knowledge , we are aware of only one prior work that uses raw text inputs for node classification problem ( Zhang et al. , 2020 ) , but it still follows the standard pipeline described in Figure 1 . Some other works apply GNNs on texts and for document classification , where the actual graphs are constructed based on the raw text . This is clearly not the focus of this work ( Yao et al. , 2019 ; Huang et al. , 2019 ; Zhang & Zhang , 2020 ; Liu et al. , 2020 ) .

Summary: The paper proposed a self-supervised learning framework for learning node feature by exploring the correlation between the node feature and the graph structure, which leverages the graph information based on neighborhood prediction. To be specific, the proposed GIANT approach is combined with the pre-trained language model BERT, and incorporated the XMC formalism based on XR-Transformer. Partial theoretical analysis is also presented. Experiments conducted on three large benchmark datasets show promissing improvements.