Automated Self-Supervised Learning for Graphs

Graph self-supervised learning has gained increasing attention due to its capacity to learn expressive node representations . Many pretext tasks , or loss functions have been designed from distinct perspectives . However , we observe that different pretext tasks affect downstream tasks differently across datasets , which suggests that searching over pretext tasks is crucial for graph self-supervised learning . Different from existing works focusing on designing single pretext tasks , this work aims to investigate how to automatically leverage multiple pretext tasks effectively . Nevertheless , evaluating representations derived from multiple pretext tasks without direct access to ground truth labels makes this problem challenging . To address this obstacle , we make use of a key principle of many real-world graphs , i.e. , homophily , or the principle that “ like attracts like , ” as the guidance to effectively search various self-supervised pretext tasks . We provide theoretical understanding and empirical evidence to justify the flexibility of homophily in this search task . Then we propose the AUTOSSL framework to automatically search over combinations of various self-supervised tasks . By evaluating the framework on 7 real-world datasets , our experimental results show that AUTOSSL can significantly boost the performance on downstream tasks including node clustering and node classification compared with training under individual tasks . 1 INTRODUCTION . Graphs are pivotal data structures describing the relationships between entities in various domains such as social media , biology , transportation and financial systems ( Wu et al. , 2019b ; Battaglia et al. , 2018 ) . Due to their prevalence and rich descriptive capacity , pattern mining and discovery on graph data is a prominent research area with powerful implications . As the generalization of deep neural networks on graph data , graph neural networks ( GNNs ) have proved to be powerful in learning representations for graphs and associated entities ( nodes , edges , subgraphs ) , and they have been employed in various applications such as node classification ( Kipf & Welling , 2016a ; Veličković et al. , 2018 ) , node clustering ( Pan et al. , 2018 ) , recommender systems ( Ying et al. , 2018 ) and drug discovery ( Duvenaud et al. , 2015 ) . In recent years , the explosive interest in self-supervised learning ( SSL ) has suggested its great potential in empowering stronger neural networks in an unsupervised manner ( Chen et al. , 2020 ; Kolesnikov et al. , 2019 ; Doersch et al. , 2015 ) . Many self-supervised methods have also been developed to facilitate graph representation learning ( Jin et al. , 2020 ; Xie et al. , 2021 ) such as DGI ( Veličković et al. , 2019 ) , PAR/CLU ( You et al. , 2020 ) and MVGRL ( Hassani & Khasahmadi , 2020 ) . Given graph and node attribute data , they construct pretext tasks , which are called SSL tasks , based on structural and attribute information to provide self-supervision for training graph neural networks without accessing any labeled data . For example , the pretext task of PAR is to predict the graph partitions of nodes . We examine how a variety of SSL tasks including DGI , PAR , CLU , PAIRDIS ( Peng et al. , 2020 ) and PAIRSIM ( Jin et al. , 2020 ; 2021 ) perform over 3 datasets . Their node clustering and node classification performance ranks are illustrated in Figure 1a and 1b , respectively . From these figures , we observe that different SSL tasks have distinct downstream performance cross datasets . This observation suggests that the success of SSL tasks strongly depends on the datasets and downstream tasks . Learning representations with a single task naturally leads to ignoring useful information from other tasks . As a result , searching SSL tasks is crucial , which motivates us to study on how to automatically compose a variety of graph self-supervised tasks to learn better node representations . However , combining multiple different SSL tasks for unlabeled representation learning is immensely challenging . Although promising results have been achieved in multi-task self-supervised learning for computer vision , most of them assign equal weights to SSL tasks ( Doersch & Zisserman , 2017 ; Ren & Lee , 2018 ; Zamir et al. , 2018 ) . Such combination might not always yield better performance than a single task , as different tasks have distinct importance according to specific dataset and downstream tasks . To illustrate this intuition , we combine two SSL tasks , PAIRDIS and PAIRSIM , with varied weights and illustrate the corresponding node clustering performance in Figure 1c . It clearly indicates that different choices of weights yield different performance . To circumvent this problem , we could plausibly search different weights for SSL tasks to optimize downstream tasks . However , to achieve such goal , we have two obstacles . First , the search space is huge , and thus search can be highly expensive . Hence , it is desirable to automatically learn these weights . Second , searching for optimal task weights typically requires guidance from downstream performance , which is naturally missing under the unsupervised setting . Thus , how to design an unsupervised surrogate evaluation measure that can guide the search process is necessary . It is evident that many real-world graphs such as friendship networks , citation networks , coauthorship networks and co-purchase networks ( McPherson et al. , 2001 ; Shchur et al. , 2018 ) satisfy the homophily assumption , i.e. , “ like attracts like ” , or that connected nodes tend to share the same label . This is useful prior knowledge , as it directly relates the label information of downstream tasks to the graph structure . In this work , we explicitly take advantage of this prior knowledge and assume that the predicted labels from good node embeddings should also adhere to homophily . Given the lack of ground-truth labels during SSL , we propose a pseudo-homophily measure to evaluate the quality of the node embeddings trained from specific combinations of SSL task . With pseudohomophily , we are able to design an automated framework for SSL task search , namely AUTOSSL . Our work makes three significant contributions : ( 1 ) To bridge the gap between unsupervised representation and downstream labels , we propose pseudo-homophily to measure the quality of the representation . Moreover , given graphs with high homophily , we theoretically show that pseudo-homophily maximization can help maximize the upper bound of mutual information between pseudo-labels and downstream labels . ( 2 ) Based on pseudo-homophily , we propose two strategies to efficiently search SSL tasks , one employing evolution algorithm and the other performing differentiable search via meta-gradient descent . AUTOSSL is able to adjust the task weights during search as shown in Figure 1d . ( 3 ) We evaluate the proposed AUTOSSL by composing various individual tasks on 7 real-world datasets . Extensive experiments have demonstrated that AUTOSSL can significantly improve the performance of individual tasks on node clustering and node classification ( e.g. , up to 10.0 % relative improvement on node clustering ) . 2 BACKGROUND AND RELATED WORK . Graph Neural Networks . Graph neural networks ( GNNs ) are powerful tools for extracting useful information from graph data ( Liu et al. , 2021 ; Wu et al. , 2019b ; Kipf & Welling , 2016a ; Veličković et al. , 2018 ; Hamilton et al. , 2017 ; Kipf & Welling , 2016b ; Pan et al. , 2018 ) . They aim to learn a mapping function fθ parameterized by θ to map the input graph into a low-dimensional space . Most graph neural networks follow a message passing scheme ( Gilmer et al. , 2017 ) where the node representation is obtained by aggregating the representation of its neighbors and its own representation . Self-Supervised Learning in GNNs . Graph neural networks have achieved superior performance in various applications ; but they also require costly task-dependent labels to learn rich representations . To alleviate the need for the huge amount of labeled data , recent studies have employed self-supervised learning in graph neural networks to provide additional supervision ( Jin et al. , 2020 ; Veličković et al. , 2019 ; You et al. , 2020 ; Hassani & Khasahmadi , 2020 ; Hu et al. , 2019 ; Qiu et al. , 2020 ; Zhu et al. , 2020b ) . Specifically , those SSL methods construct a pre-defined pretext task to assign pseudo-labels for unlabeled nodes/graphs and then train the model on the designed pretext task to learn representations . A recent work JOAO ( You et al. , 2021 ) on graph contrastive learning is proposed to automatically select data augmentation and focuses on graph classification task . Another related work is AUX-TS ( Han et al. , 2021 ) , which also adaptively combines different SSL tasks but the combination happens at the fine-tuning stage and thus requires label information . Multi-Task Self-Supervised Learning . Our work is also related to multi-task self-supervised learning ( Doersch & Zisserman , 2017 ; Ren & Lee , 2018 ; Zamir et al. , 2018 ) . Most of them assume the tasks with equal weights and perform training under the supervised setting . But our work learns different weights for different tasks and does not require access to labeled data . Automated Loss Function Search . Tremendous efforts have been paid to automate every aspect of machine learning applications ( Yao et al. , 2018 ; Liu et al. , 2018 ) , such as feature engineering , model architecture search and loss function search . Among them , our work is highly related to loss function search ( Zhao et al. , 2021 ; Xu et al. , 2018 ; Wang et al. , 2020 ; Li et al. , 2019 ) . However , these methods are developed under the supervised setting and not applicable in self-supervised learning . Another related work is ELo ( Piergiovanni et al. , 2020 ) , which evolves multiple self-supervised losses based on Zipf distribution matching for action recognition . However , it is designed exclusively for image data and not applicable to non-grid graph-structured data . The problem of self-supervised loss search for graphs remains rarely explored . To bridge the gap , we propose an automated framework for searching SSL losses towards graph data in an unsupervised manner . 3 AUTOMATED SELF-SUPERVISED TASK SEARCH WITH AUTOSSL . In this section , we present the proposed framework of automated self-supervised task search , namely AUTOSSL . Given a graph G , a GNN encoder fθ ( · ) and a set of n self-supervised losses ( tasks ) { ` 1 , ` 2 , . . . , ` n } , we aim at learning a set of loss weights { λ1 , λ2 , . . . , λn } such that fθ ( · ) trained with the weighted loss combination ∑n i=1 λi ` i can extract meaningful features from the given graph data . The key challenge is how to mathematically define “ meaningful features ” . If we have the access to the labels of the downstream task , we can define “ meaningful features ” as the features ( node embeddings ) that can have high performance on the given downstream task . Then we can simply adopt the downstream performance as the optimization goal and formulate the problem of automated self-supervised task search as follows : min λ1 , ··· , λn H ( fθ∗ ( G ) ) , s.t . θ∗ = argmin θ L ( fθ , { λi } , { ` i } ) = argmin θ n∑ i=1 λi ` i ( fθ ( G ) ) , ( 1 ) where H denotes the quality measure for the obtained node embeddings , and it can be any metric that evaluates the downstream performance such as cross-entropy loss for the node classification task . However , under the self-supervised setting , we do not have the access to labeled data and thus can not employ the downstream performance to measure the embedding quality . Instead , we need an unsupervised quality measure H to evaluate the quality of obtained embeddings . In a nutshell , one challenge of automated self-supervised learning is : how to construct the goal of automated task search without the access to label information of the downstream tasks .

Summary: Self-supervised learning (SSL) methods for graphs take a given graph with node attribute information and construct various SSL tasks using structural and attribution information. These tasks provide self-supervision for training graph neural networks without accessing any labeled data. Existing studies show that different SSL tasks can lead to varying downstream performance across tasks, suggesting that the success of SSL tasks strongly depends on the dataset and the downstream task. This observation is important, yet not surprising, as similar findings were made in other data modalities (text, vision). As a result, selecting the 'right' SSL tasks can be important, which serves as a motivation for this study.   This study develops an approach (called AutoSSL) for combining multiple SSL tasks for unlabeled representation learning. To this end, it defines a pseudo-homophily metric to measure the quality of learned representations. Based on the pseudo-homophily, the study describes two techniques to search & combine SSL tasks, one using an evolutionary algorithm and the other using meta-gradient descent. The AutoSSL approach is evaluated on eight datasets considering combinations of five SSL tasks (one contrastive learning task and four predictive tasks).