Automated Self-Supervised Learning for Graphs

Graph self-supervised learning has gained increasing attention due to its capacity to learn expressive node representations . Many pretext tasks , or loss functions have been designed from distinct perspectives . However , we observe that different pretext tasks affect downstream tasks differently across datasets , which suggests that searching over pretext tasks is crucial for graph self-supervised learning . Different from existing works focusing on designing single pretext tasks , this work aims to investigate how to automatically leverage multiple pretext tasks effectively . Nevertheless , evaluating representations derived from multiple pretext tasks without direct access to ground truth labels makes this problem challenging . To address this obstacle , we make use of a key principle of many real-world graphs , i.e. , homophily , or the principle that “ like attracts like , ” as the guidance to effectively search various self-supervised pretext tasks . We provide theoretical understanding and empirical evidence to justify the flexibility of homophily in this search task . Then we propose the AUTOSSL framework to automatically search over combinations of various self-supervised tasks . By evaluating the framework on 7 real-world datasets , our experimental results show that AUTOSSL can significantly boost the performance on downstream tasks including node clustering and node classification compared with training under individual tasks . 1 INTRODUCTION . Graphs are pivotal data structures describing the relationships between entities in various domains such as social media , biology , transportation and financial systems ( Wu et al. , 2019b ; Battaglia et al. , 2018 ) . Due to their prevalence and rich descriptive capacity , pattern mining and discovery on graph data is a prominent research area with powerful implications . As the generalization of deep neural networks on graph data , graph neural networks ( GNNs ) have proved to be powerful in learning representations for graphs and associated entities ( nodes , edges , subgraphs ) , and they have been employed in various applications such as node classification ( Kipf & Welling , 2016a ; Veličković et al. , 2018 ) , node clustering ( Pan et al. , 2018 ) , recommender systems ( Ying et al. , 2018 ) and drug discovery ( Duvenaud et al. , 2015 ) . In recent years , the explosive interest in self-supervised learning ( SSL ) has suggested its great potential in empowering stronger neural networks in an unsupervised manner ( Chen et al. , 2020 ; Kolesnikov et al. , 2019 ; Doersch et al. , 2015 ) . Many self-supervised methods have also been developed to facilitate graph representation learning ( Jin et al. , 2020 ; Xie et al. , 2021 ) such as DGI ( Veličković et al. , 2019 ) , PAR/CLU ( You et al. , 2020 ) and MVGRL ( Hassani & Khasahmadi , 2020 ) . Given graph and node attribute data , they construct pretext tasks , which are called SSL tasks , based on structural and attribute information to provide self-supervision for training graph neural networks without accessing any labeled data . For example , the pretext task of PAR is to predict the graph partitions of nodes . We examine how a variety of SSL tasks including DGI , PAR , CLU , PAIRDIS ( Peng et al. , 2020 ) and PAIRSIM ( Jin et al. , 2020 ; 2021 ) perform over 3 datasets . Their node clustering and node classification performance ranks are illustrated in Figure 1a and 1b , respectively . From these figures , we observe that different SSL tasks have distinct downstream performance cross datasets . This observation suggests that the success of SSL tasks strongly depends on the datasets and downstream tasks . Learning representations with a single task naturally leads to ignoring useful information from other tasks . As a result , searching SSL tasks is crucial , which motivates us to study on how to automatically compose a variety of graph self-supervised tasks to learn better node representations . However , combining multiple different SSL tasks for unlabeled representation learning is immensely challenging . Although promising results have been achieved in multi-task self-supervised learning for computer vision , most of them assign equal weights to SSL tasks ( Doersch & Zisserman , 2017 ; Ren & Lee , 2018 ; Zamir et al. , 2018 ) . Such combination might not always yield better performance than a single task , as different tasks have distinct importance according to specific dataset and downstream tasks . To illustrate this intuition , we combine two SSL tasks , PAIRDIS and PAIRSIM , with varied weights and illustrate the corresponding node clustering performance in Figure 1c . It clearly indicates that different choices of weights yield different performance . To circumvent this problem , we could plausibly search different weights for SSL tasks to optimize downstream tasks . However , to achieve such goal , we have two obstacles . First , the search space is huge , and thus search can be highly expensive . Hence , it is desirable to automatically learn these weights . Second , searching for optimal task weights typically requires guidance from downstream performance , which is naturally missing under the unsupervised setting . Thus , how to design an unsupervised surrogate evaluation measure that can guide the search process is necessary . It is evident that many real-world graphs such as friendship networks , citation networks , coauthorship networks and co-purchase networks ( McPherson et al. , 2001 ; Shchur et al. , 2018 ) satisfy the homophily assumption , i.e. , “ like attracts like ” , or that connected nodes tend to share the same label . This is useful prior knowledge , as it directly relates the label information of downstream tasks to the graph structure . In this work , we explicitly take advantage of this prior knowledge and assume that the predicted labels from good node embeddings should also adhere to homophily . Given the lack of ground-truth labels during SSL , we propose a pseudo-homophily measure to evaluate the quality of the node embeddings trained from specific combinations of SSL task . With pseudohomophily , we are able to design an automated framework for SSL task search , namely AUTOSSL . Our work makes three significant contributions : ( 1 ) To bridge the gap between unsupervised representation and downstream labels , we propose pseudo-homophily to measure the quality of the representation . Moreover , given graphs with high homophily , we theoretically show that pseudo-homophily maximization can help maximize the upper bound of mutual information between pseudo-labels and downstream labels . ( 2 ) Based on pseudo-homophily , we propose two strategies to efficiently search SSL tasks , one employing evolution algorithm and the other performing differentiable search via meta-gradient descent . AUTOSSL is able to adjust the task weights during search as shown in Figure 1d . ( 3 ) We evaluate the proposed AUTOSSL by composing various individual tasks on 7 real-world datasets . Extensive experiments have demonstrated that AUTOSSL can significantly improve the performance of individual tasks on node clustering and node classification ( e.g. , up to 10.0 % relative improvement on node clustering ) . 2 BACKGROUND AND RELATED WORK . Graph Neural Networks . Graph neural networks ( GNNs ) are powerful tools for extracting useful information from graph data ( Liu et al. , 2021 ; Wu et al. , 2019b ; Kipf & Welling , 2016a ; Veličković et al. , 2018 ; Hamilton et al. , 2017 ; Kipf & Welling , 2016b ; Pan et al. , 2018 ) . They aim to learn a mapping function fθ parameterized by θ to map the input graph into a low-dimensional space . Most graph neural networks follow a message passing scheme ( Gilmer et al. , 2017 ) where the node representation is obtained by aggregating the representation of its neighbors and its own representation . Self-Supervised Learning in GNNs . Graph neural networks have achieved superior performance in various applications ; but they also require costly task-dependent labels to learn rich representations . To alleviate the need for the huge amount of labeled data , recent studies have employed self-supervised learning in graph neural networks to provide additional supervision ( Jin et al. , 2020 ; Veličković et al. , 2019 ; You et al. , 2020 ; Hassani & Khasahmadi , 2020 ; Hu et al. , 2019 ; Qiu et al. , 2020 ; Zhu et al. , 2020b ) . Specifically , those SSL methods construct a pre-defined pretext task to assign pseudo-labels for unlabeled nodes/graphs and then train the model on the designed pretext task to learn representations . A recent work JOAO ( You et al. , 2021 ) on graph contrastive learning is proposed to automatically select data augmentation and focuses on graph classification task . Another related work is AUX-TS ( Han et al. , 2021 ) , which also adaptively combines different SSL tasks but the combination happens at the fine-tuning stage and thus requires label information . Multi-Task Self-Supervised Learning . Our work is also related to multi-task self-supervised learning ( Doersch & Zisserman , 2017 ; Ren & Lee , 2018 ; Zamir et al. , 2018 ) . Most of them assume the tasks with equal weights and perform training under the supervised setting . But our work learns different weights for different tasks and does not require access to labeled data . Automated Loss Function Search . Tremendous efforts have been paid to automate every aspect of machine learning applications ( Yao et al. , 2018 ; Liu et al. , 2018 ) , such as feature engineering , model architecture search and loss function search . Among them , our work is highly related to loss function search ( Zhao et al. , 2021 ; Xu et al. , 2018 ; Wang et al. , 2020 ; Li et al. , 2019 ) . However , these methods are developed under the supervised setting and not applicable in self-supervised learning . Another related work is ELo ( Piergiovanni et al. , 2020 ) , which evolves multiple self-supervised losses based on Zipf distribution matching for action recognition . However , it is designed exclusively for image data and not applicable to non-grid graph-structured data . The problem of self-supervised loss search for graphs remains rarely explored . To bridge the gap , we propose an automated framework for searching SSL losses towards graph data in an unsupervised manner . 3 AUTOMATED SELF-SUPERVISED TASK SEARCH WITH AUTOSSL . In this section , we present the proposed framework of automated self-supervised task search , namely AUTOSSL . Given a graph G , a GNN encoder fθ ( · ) and a set of n self-supervised losses ( tasks ) { ` 1 , ` 2 , . . . , ` n } , we aim at learning a set of loss weights { λ1 , λ2 , . . . , λn } such that fθ ( · ) trained with the weighted loss combination ∑n i=1 λi ` i can extract meaningful features from the given graph data . The key challenge is how to mathematically define “ meaningful features ” . If we have the access to the labels of the downstream task , we can define “ meaningful features ” as the features ( node embeddings ) that can have high performance on the given downstream task . Then we can simply adopt the downstream performance as the optimization goal and formulate the problem of automated self-supervised task search as follows : min λ1 , ··· , λn H ( fθ∗ ( G ) ) , s.t . θ∗ = argmin θ L ( fθ , { λi } , { ` i } ) = argmin θ n∑ i=1 λi ` i ( fθ ( G ) ) , ( 1 ) where H denotes the quality measure for the obtained node embeddings , and it can be any metric that evaluates the downstream performance such as cross-entropy loss for the node classification task . However , under the self-supervised setting , we do not have the access to labeled data and thus can not employ the downstream performance to measure the embedding quality . Instead , we need an unsupervised quality measure H to evaluate the quality of obtained embeddings . In a nutshell , one challenge of automated self-supervised learning is : how to construct the goal of automated task search without the access to label information of the downstream tasks .

Summary: The proposed work looks at the task of automated self-supervised learning (SSL) on graphs, by using pseudo-homophily as a surrogate objective combined with a search strategy for the proposed approach, AutoSSL .   Homophily is defined as the average of sameness of labels over pairs of connected vertices. Pseudo-homophily is computed by assigning labels based on k-means clustering. The work theoretically shows that maximizing pseudo-homophily is shown to maximize the upper bound of mutual information between pseudo-labels and downstream labels.   As cluster assignments are not differentiable and need search over a large space, the work looks at using evolutionary strategies (AutoSSL-ES) and differentiable search through soft cluster assignments using Gaussian mixture model (AutoSSL-DS).  Experiments are performed on 5 SSL tasks and 8 datasets. AutoSSL based approaches outperform several other unsupervised baselines and perform comparably to supervised baselines when measured using normalized mutual information, accuracy of node classification and psuedo-homophily.