Gaussian Mutual Information Maximization for Efficient Graph Self-Supervised Learning: Bridging Contrastive-based to Decorrelation-based

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 08:16:34

Abstract: Enlightened by the InfoMax principle, Graph Contrastive Learning (GCL) has achieved remarkable performance in processing large amounts of unlabeled graph data. Due to the impracticality of precisely calculating mutual information (MI), conventional contrastive methods turn to approximate its lower bound using parametric neural estimators, which inevitably introduces additional parameters and leads to increased computational complexity. Building upon a common Gaussian assumption on the distribution of node representations, a computationally tractable surrogate for the original MI can be rigorously derived, termed as Gaussian Mutual Information (GMI). Leveraging multi-view priors of GCL, we induce an efficient contrastive objective based on GMI with performance guarantees, eliminating the reliance on parameterized estimators and negative samples. The emergence of another decorrelation-based self-supervised learning branch parallels contrastive-based approaches. By positioning the proposed GMI-based objective as a pivot, we bridge the gap between these two research areas from two aspects of approximate form and consistent solution, which contributes to the advancement of a unified theoretical framework for self-supervised learning. Extensive comparison experiments and visual analysis provide compelling evidence for the effectiveness and efficiency of our method while supporting our theoretical achievements.

Main paper: 
 introduction: 
 The scarcity of task-related annotations for graph data, which usually rely on domain knowledge and specific equipment such as chemical instruments [29] , urgently calls for the emergence of advanced unsupervised learning methods without manual supervision. In this context, graph self-supervised learning (SSL) [6, 46, 55, 58, 59] arises naturally in response to the prevailing demand, approaching and even surpassing the performance of their supervised counterparts [19, 24, 48] . SSL models are trained on well-designed pretext objectives in a task-agnostic manner, whose optimization results in general, meaningful, and transferable representations for downstream applications. As a distinguished member of the SSL family, multi-view learning with Siamese networks [27] has demonstrated exceptional performance and garnered widespread interest. At the heart of such methods is to extract invariant or common information from various augmented views of the same instance (i.e., positive pairs) while adopting specific strategies to prevent model collapse. The existing multi-view learning methods can be classified into two distinct categories based on their means of addressing degradation: contrastive [20, 55, 58, 59 ] and non-contrastive [6, 46, 57] approaches. The former suppresses encoded representations from collapsing into a constant point by pushing negative pairs apart, while the latter employs special strategies such as decorrelating different representation dimensions [6, 57] or designing asymmetric network architecture [17, 46] .The concept of contrastive multi-view learning originates from information theory [2, 51] , aiming to improve the consistency between various views by maximizing their mutual information (MI). Nevertheless, the exact computation of mutual information for high-dimensional continuous variables is usually intractable. To cope with this challenge, some previous endeavors have attempted to employ parameterized neural estimators to perform an empirical evaluation of mutual information from finite samples, yielding notable achievements like MINE [4] , Jensen-Shannon estimator [31] , and InfoNCE [18] . Formally, contrastive learning methods equipped with the parameterized estimator of MI manifest as a contrastiveness between positive pairs from a joint distribution and negative pairs from two marginal ones. Despite their decent performance, these methods are accompanied by several inherent drawbacks: a) a substantial number of samples are required to obtain reliable estimation and achieve satisfactory results, which inevitably increases computational burden; b) the incorporation of parameterized MI estimators amplifies the complexity of SSL models.Deviating from the conventional graph contrastive learning methods, we delve into lightweight and efficient alternatives with no reliance on parameterized MI estimators for node-level representation learning. Assuming node representations obey a Gaussian distribution, a feasible closed-form solution can be obtained, called Gaussian Mutual Information (GMI) [36, 38] , through tractable integration operations on the native definition of MI. In its mathematical form, the estimation of GMI exclusively depends on the covariance matrices, which can be effortlessly obtained from empirical data (i.e., node representations). Independent of additional architectures, the resultant SSL objective under GMI can be directly calculated within the representation space, leading to higher computational efficiency and better resource friendliness. Most importantly, the performance of the proposed method can still hold even when actual scenarios deviate from Gaussian distributions, thereby extending its applicability beyond Gaussian constraints.As another indispensable branch of the SSL family, the decorrelationbased non-contrastive methods [3, 6, 12, 57] prevent degenerate solutions and learn diverse representations by decoupling various channels, whose objective functions exhibit an utterly distinct appearance from those of contrastive-based ones. While the distinctions between the two branches have been thoroughly discussed, their latent theoretical relationships remain enshrouded in ambiguity. Imposing a cross-view identity constraint, which enhances the perfect alignment of representations from different views of the same instance, to our proposed GMI-based objective function, we employ the newly induced objective as a pivot to elucidate the underlying connections between decorrelation-based and contrastivebased methods. On the one hand, the former is formally equivalent to a second-order Taylor series expansion of the latter. On the other hand, their objectives share consistent solutions. Overall, the decorrelation-based methods can be regarded as an instantiation of contrastive learning under the Gaussian assumption on the distribution of node representations and identity constraint.Our contributions in this paper are summarized as follows: • In light of multi-view priors and training characteristics of selfsupervised learning, we propound an extremely efficient and stable training objective based on Gaussian mutual information maximization, exhibiting unprecedented efficiency compared to previous contrastive methods. Most significantly, extensive investigations in contexts beyond normal distributions demonstrate capability of our method to generalize to non-Gaussian scenarios. • We bridge decorrelation-based self-supervised methods to our proposed contrastive objective from two aspects of approximation of form and consistency of solution, which points out a clue to demystify the relationships between various self-supervised learning methods. • Thorough empirical studies demonstrate the effectiveness and efficiency of our method compared with advanced peers. Additionally, exploratory studies and visual analysis further reveal the advantages of our method and reinforce the understanding of our theoretical achievements. 
related work 2.1 graph self-supervised learning: 
 For its remarkable performance, multi-view-based methods have been the dominant paradigm of graph self-supervised learning, which expect to explore common information from various augmented versions. A crucial aspect of these methods is to prevent degenerate solutions, where all representations are collapsed to a constant point (i.e., complete collapse) or a subspace (i.e., dimensional collapse) of the entire representation space. The current methods can be categorized into two groups, namely contrastive [20, 34, 55, 58, 59 ] and non-contrastive [6, 46, 57] approaches, based on their ways to circumvent model collapse.The contrastive-based methods usually follow the criterion of mutual information maximization [21, 28] , whose objective functions take the form of contrasting positive pairs with negative ones.As pioneer works, DGI [49] and InfoGraph [45] learn unsupervised representations by maximizing mutual information between nodelevel representations and a whole graph summary vector based on the Jenson-Shannon estimator [31] . GraphCL [55] , GRACE [58] , and GCA [59] embed the InfoNCE [18] loss into graph contrastive learning framework. From the view of information theory, InfoGCL [54] investigates how to build appropriate contrastive learning frameworks for specific tasks. GII [53] treats the structure as a separate view and realizes cross-modal information interaction between features and structure. M-ILBO [30] leverages MI estimators to maximize entropy for learning diverse representations.The non-contrastive methods discard negative samples, which require special strategies to avoid collapsed solutions. BGRL [46] utilizes asymmetric architecture and a stop-gradient strategy to prevent the two branches from merging. Graph Barlow Twins (G-BT) [6] generalizes the celebrated Barlow Twins [56] from images to graph data. CCA-SSG [57] learns augmentation-invariant information while decorrelating features in different dimensions to prevent degenerated solutions.Mutual information is a powerful and commonly used measure for general correlation between random variables, which has been applied to a range of fields, including medical image processing [37] , feature selection [1, 13] , information bottleneck [16] , and recommendation system [39] . Nevertheless, the exact computation of MI for high-dimensional variables is notoriously difficult. An alternative scheme is to estimate MI from empirical observations. The non-parametric estimators make no assumptions about the underlying distribution of data and require no specification of any parameters. The most popular class in this branch is the k-nearestneighbor-based estimators and their extensions [14, 26, 44] . Besides, the methods based on kernel density estimation (KDE) first estimate the probability density function and then compute MI by Monte-Carlo integration [40, 43] .The research on neural-network-based MI estimation [4, 18, 31] has also made significant process, which has been widely applied in representation learning. The key technical ingredient of these methods is to approximate the lower bound of MI based on dual representations of the 𝑓 -divergence [31] .Common types of collapse in multi-view self-supervised learning include complete collapse and dimensional collapse, which respectively represent the representations collapsing to a constant point and to a subspace. The most common approach to prevent collapse issues is to use negative samples to push data points apart in the representation space, including SimCLR [8] , GRACE [58] , and MVGRL [20] . Another class of methods employs asymmetric architectures with stop-gradient strategy to prevent representations of two views from colliding with each other [7, 9, 17, 47] . Besides, some methods aim to enhance representation diversity and prevent model collapse by evaluating and maximizing the entropy of representations. Representative works include CorInfoMax [32] , M-ILBO [30] , and literature [50] . Furthermore, methods like VI-CReg [3] and CCA-SSG [57] decouple correlations between different representation channels to avoid dimensional collapse. 
methodology 3.1 preliminaries and overall framework: 
 Preliminaries. Before further discussion, the preliminary conceptions presented in this paper are first provided. A graph is denoted by 𝐺 (A, X) with node set V = {𝑣 1 , ..., 𝑣 𝑁 } and edge set E, where |V | = 𝑁 indicates the number of nodes. Each node 𝑣 𝑖 ∈ V has a 𝐷-dimensional feature vector x 𝑖 ∈ R 𝐷 . Node feature matrix X = [x 1 , ..., x 𝑁 ] ⊤ ∈ R 𝑁 ×𝐷 contains feature information of all nodes and adjacency matrix A ∈ R 𝑁 ×𝑁 describes the connection relationship between different nodes. The task of node-level graph self-supervised learning is to seek good node representations H = [ h1 , ..., h𝑁 ] ⊤ ∈ R 𝑁 ×𝑑 through learning a continuous mapping 𝑓 𝜃 (A, X) : R 𝑁 ×𝑁 × R 𝑁 ×𝐷 → R 𝑁 ×𝑑 without manual labels, where 𝜃 denotes learnable model parameters and 𝑑 indicates the representation dimension.Graph View Generation. Let the transformation 𝜏 ∈ T : 𝐺 (A, X) → 𝐺 ′ (A ′ , X ′ ) map the original graph to an augmented version, where T denotes the whole function space for augmentation. Specifically, the graph augmentation 𝜏 is jointly implemented from two aspects of graph topology and feature, following previous works [58] . For topology-level augmentation, edge removal is adopted, randomly removing edges of a certain ratio 𝑝 𝑒 on the original graph. For feature-level augmentation, node feature masking randomly sets feature channels of a specific number 𝐷 • 𝑝 𝑓 in feature matrix X ∈ R 𝑁 ×𝐷 to zero, where 𝑝 𝑓 is the masking ratio.Overall Framework. In terms of basic framework, this paper inherits the common practice of prior studies. As shown in Figure 1 , two various views 𝐺 ′ 𝐴 (A ′ 𝐴 , X ′ 𝐴 ) = 𝜏 𝐴 (𝐺) and 𝐺 ′ 𝐵 (A ′ 𝐵 , X ′ 𝐵 ) = 𝜏 𝐵 (H 𝐴 = [h 𝐴 1 , ..., h 𝐴 𝑁 ] ⊤ and H 𝐵 = [h 𝐵 1 , ..., h 𝐵 𝑁 ] ⊤, each representation channel in which obey a distribution with 0-mean and 1-standard deviation. "GMIM" is the optimization objective proposed in the following sections.Contrastive learning is initially enlightened by the InfoMax principle [5] , which expects to maximize mutual information between representations from various views.Definition 3.1 (Mutual Information). Let 𝑋 and 𝑌 denote two 𝑑-dimensional continuous variables with marginal probability functions 𝑝 𝑥 (𝑋 ) and 𝑝 𝑦 (𝑌 ), respectively. Their joint probability density is indicated by 𝑝 𝑥,𝑦 (𝑋, 𝑌 ). The mutual information 𝐼 (𝑋 ; 𝑌 ) between 𝑋 and 𝑌 is defined asEQUATIONwhere X and Y denote domains corresponding to 𝑋 and 𝑌 , respectively.Nevertheless, the exact computation of mutual information for high-dimensional continuous variables is usually infeasible. First, it is challenging to estimate the probability densities from empirical observations. Second, even though they can be obtained, which may have complex forms, the integral operation in Eq. ( 1) remains difficult, even intractable. To tackle these issues, the conventional contrastive leaning methods employ parametric networks to directly estimate a lower bound of MI, which can be trained alongside the backbone via back-propagation in an end-to-end manner.Divergent from the peer works, this paper assumes a latent Gaussian distribution for node representations and drops parametric estimators, which leads to a computationally tractable surrogate. The Gaussian assumption is justifiable and extensively employed in numerous disciplines to simplify analysis and calculation, including economics, data science, and physics [33] .Proposition 3.2 (Gaussian Mutual Information). If the variables 𝑋 and 𝑌 obey two multi-dimensional Gaussian distributions, respectively, the Gaussian mutual information 𝐼 𝐺 (𝑋 ; 𝑌 ) between them isEQUATIONwhere det(•) indicates the determinant of a matrix, Σ 𝑋 and Σ 𝑌 are the covariance matrices of 𝑋 and 𝑌 , respectively, andΣ 𝑋 ,𝑌 = Σ 𝑋 Σ 𝑋𝑌 Σ ⊤ 𝑋𝑌 Σ 𝑌 is the covariance matrix of variable [𝑋 ⊤ , 𝑌 ⊤ ] ⊤ with cross-covariance matrix Σ 𝑋𝑌 .Proof. Please refer to Section 1 of supplementary materials. □The three covariance matrices Σ 𝑋 , Σ 𝑌 , and Σ 𝑋 ,𝑌 can be effortlessly estimated from the empirical data, which results in a straightforward calculation of Gaussian mutual information. The covariance matrix is a real symmetric matrix whose eigenvalues are all greater than or equal to zero. Mathematically, the determinant of a matrix is numerically equal to the product of all eigenvalues. Due to the underlying dimensional collapse issue during self-supervised pretraining, many eigenvalues of the empirical covariance matrix tend to be zero, which causes its determinant to approach zero. Therefore, a direct adoption of Eq. ( 2) for constructing a contrastive learning objective will bring about numerical instability. One feasible strategy to alleviate the numerical issue is to offset and scale the eigenvalues of the matrix performed by det(•). Considering multiview priors and training characteristics of SSL, a practical objective based on Gaussian Mutual Information Maximization (GMIM) can be formulated asEQUATIONwhereΣ 𝐴 = 1 𝑁 H ⊤ 𝐴 H 𝐴 , Σ 𝐵 = 1 𝑁 H ⊤ 𝐵 H 𝐵 , Σ 𝐴,𝐵 = 1 𝑁 H ⊤ 𝐴 H 𝐴 H ⊤ 𝐴 H 𝐵 H ⊤ 𝐵 H 𝐴 H ⊤ 𝐵 H 𝐵 ,I is an identity matrix, and 𝜂 is a scaling factor with a typical value of 0.1. The eigenvalues of I + 𝜂 • Σ 𝐴 fall into [1, +∞), and so do the other two sibling matrices.According to [10] , the following property holds:Property 1. For variables 𝑋 and 𝑌 , the relationship between entropy and mutual information isEQUATIONwhere𝐻 (𝑋 ) = - ∫ X 𝑝 𝑥 (𝑋 ) ln 𝑝 𝑥 (𝑋 )𝑑𝑋 denote information entropy of 𝑋 under 𝑝 𝑥 (𝑋 ), and 𝐻 (𝑋 |𝑌 ) = ∫ X ∫ Y 𝑝 𝑥,𝑦 (𝑋, 𝑌 ) ln 𝑝 𝑥,𝑦 (𝑋 ,𝑌 ) 𝑝 𝑦 (𝑌 ) 𝑑𝑋𝑑𝑌 is the conditional entropy of 𝑋 given 𝑌 . If 𝑋 is deterministic given 𝑌 , 𝐻 (𝑋 |𝑌 ) = 0. Symmetrically, 𝐼 (𝑋, 𝑌 ) = 𝐻 (𝑌 ) -𝐻 (𝑌 |𝑋 ) holds.From Property 1, it can be known that mutual information maximization actually involves two potential processes: increasing information entropy and reducing conditional entropy. The conditional entropy is minimized when the relationship between 𝑋 and 𝑌 can be described by a deterministic function 𝑔(•), that is, 𝑌 ′ = 𝑔(𝑋 ′ ) holds for any pair (𝑋 ′ , 𝑌 ′ ) ∼ 𝑝 𝑥,𝑦 . In our setup of overall framework, a shared graph neural network is employed, expecting that representations of different versions from the same instance can match each other perfectly. In this circumstance, 𝑔(•) is preferred to be an identity mapping. By imposing the cross-view identity constraint to mutual information maximization with the preservation of entropy maximization, we can obtain an objective under Gaussian Mutual Information Maximization with Identity Constraint (GMIM-IC):EQUATION)where 𝛽 is a coefficient balancing identity constraint term and entropy maximization term. Some analysis about Eq. ( 5) is placed in Section 2 of supplementary materials.The objective functions L GMIM-IC and L GMIM , which maximize Gaussian mutual information, can be computed directly in the representation space without relying on any additional architectures such as projection heads and estimators, demonstrating extremely high efficiency.The proposed method is developed under the Gaussian assumption for node representations. The non-Gaussian nature of realworld scenarios may lead to misleading results in calculations and analyses conducted under Gaussian assumptions. Therefore, we design constraint functions from a maximum likelihood perspective to drive the actual distribution towards the target Gaussian distribution for alignment. The 𝑗-th column data in the representation matrix H 𝐴 can be viewed as 𝑁 empirical samples of a single-dimensional random variable. Let 𝜇 where 𝐻 𝐴 𝑖 𝑗 denotes the element in the 𝑖-th row and 𝑗-th column of matrix H 𝐴 , the 𝑗-th column data can be forced to approach a Gaussian distribution. Considering all representation channels from both views, the following objective function for Gaussian constraints can be constructedEQUATION𝜅 • L 𝑔𝑎𝑢 with a weighted coefficient 𝜅 can serve as a probabilistic constraint loss, jointly supervising model training with L GMIM or L GMIM-IC . Noting that subsequent empirical studies show that our method can achieve highly competitive results even without relying on L 𝑔𝑎𝑢 . However, L 𝑔𝑎𝑢 remains worthwhile, which can act as a safeguard when our method fails in non-Gaussian scenarios. 
bridging contrastive-based to decorrelation-based: 
 Based on the symbols in this article, the decorrelation-based selfsupervised method (taking CCA-SSG [57] as an example) can be formularized asEQUATIONwhere 𝜆 denotes a balancing factor and ∥ • ∥ 𝐹 indicates the Frobenius norm of a matrix. Since the diagonal elements of Σ 𝐴 are always 1, the following equation holds:EQUATIONwhere Σ𝐴 represents the element in the 𝑖-th row and the 𝑗-th column of Σ 𝐴 . The conclusion of Eq. ( 8) still holds for 𝐵. Next, we will establish connections between the decorrelation-based methods and our objective L GMIM-IC from two perspectives. EQUATIONwhere * ∈ {𝐴, 𝐵} 1 . Applying Taylor expression to the logarithmic function in tr(ln(I + 𝜂 • Σ * )), it can be known thatEQUATIONBased on Lemma 4.1, we can obtain a second-order Taylor approximation:EQUATION)The proof of Lemma 4.1 and detailed derivations of Eq. ( 11) are placed in Section 3 of supplementary materials. Comparing Eq. ( 8) with Eq. ( 11), ∥Σ 𝐴 -I∥ 2 𝐹 is equivalent to the second-order Taylor expression ofln det(I + 𝜂 • Σ 𝐴 ) without considering the constant term. Symmetrically, the finding can be extended to view 𝐵. Besides, the invariance term in Eq. ( 7) has an identical form with the identity constraint term in Eq. ( 5). Thus, we can conclude that the objective of decorrelation-based methods such as CCA-SSG has a approximate form with that of GMIM-IC.Certainly, the objective function in Eq. ( 7) is minimized when the representations from the two views are perfectly matched and their empirical covariance matrices tend towards the identity matrix. Proposition 4.2. When ln det(I +𝜂 • Σ * ) or ln det(Σ * ) is maximized, the empirical covariance matrix Σ * will converge to an identity matrix.Proof. Refer to Section 3.2 of supplementary materials. □Obviously, the identity constraint term is minimized in Eq. ( 5) when H 𝐴 and H 𝐵 is completely aligned. Combining this observation with Proposition 4.2, it can be concluded that the decorrelationbased objective in Eq. ( 7) has the same solution as the objective based on GMIM-IC. 1 In the remaining sections of this article, * is used to represent either 𝐴 or 𝐵.Explaination 1 and 2 demonstrate the relationship between two objectives L CCA-SSG and L GMIM-IC from two aspects of approximation in form and consistency in final solutions. Consequently, the following remark emerges naturally.Remark 4.3. The decorrelation-based graph self-supervised methods, which expect to align multiple views and disentangle different representation dimensions, can actually be viewed as a special instance of mutual-information-maximization-based contrastive learning under the Gaussian assumption and identity constraint. 
theoretical analysis 5.1 preventing dimensional collapse: 
 When dimensional collapse issue exists, various representation channels are coupled to each other and present a certain correlation. Another manifestation of dimensional collapse is that data points exhibit differences in distributions along different principal directions, where some directions exhibit loose distributions with higher variance, while others present tight distributions with lower variance.Σ = 1 𝑁 H ⊤ H ∈ R 𝑑 ×𝑑 with batch-normalized representations H = [h 1 , ..., h 𝑁 ] ⊤ ∈ R 𝑁 ×𝑑 , which has 𝑑 eigenvalues [𝜆 1 , 𝜆 2 , . . . , 𝜆 𝑑 ] corresponding to 𝑑 eigen- vectors [q 1 ,q 2 , . . . , q 𝑑 ], the variance of data H along the 𝑘-th principal direction (that is, direction of q 𝑘 ) is numerically equal to 𝜆 𝑘 .Proof. Refer to Section 3.3 of supplementary materials. □ Property 2 potentially suggests that the unevenness of the eigenvalues of the covariance matrix leads to the issue of dimensional collapse. Combining with Proposition 4.2, it can be known that maximizing the logarithm of determinant can ensure entropy maximization and realize isotropic covariance, which actually guarantees the evenness of eigenvalues of the covariance matrix and thus prevents dimensional collapse issue. From the perspective of representation learning, this result will enhance the diversity, richness, and discriminability of node representations, thereby conferring advantages to downstream tasks.As the commonest indicator in contrastive learning, the InfoNCE [18] loss guides the model to learn meaningful and diverse representations by pulling together embeddings from positive pairs and pushing apart those from negative ones on the unit hypersphere.A previous work [52] decomposes the classical InfoNCE objective into two terms: alignment term and uniformity term. The alignment term expects to match two views, which shares the same purpose as our identity constraint. The uniformity term is utilized to distribute representations uniformly on the unit hypersphere S 𝑑 -1 . Proposition 5.1. When the representations scatter over the unit hypersphere S 𝑑 -1 uniformly (that is, they obey a complete uniform distribution), their entropy will reach the maximum value.Proof. Refer to Section 3.4 of supplementary materials. □ Proposition 5.1 suggests that the uniformity term implicitly realize the maximization of entropy by distributing the representations uniformly over the hypersphere. Similar to the literature [52] , some previous works [11, 50] also utilize the same entropy maximization criterion to promote uniformity and diversity of representations. In contrast, our method explicitly maximizes the entropy of representations under the assumption of Gaussian distribution. In general, the two approaches reach the similar goal by different routes. 
experiments 6.1 datasets and experimental setup: 
 Datasets. To assess our approach, six widely used benchmark datasets are adopted for experimental study, including three citation networks Cora, Citeseer, and Pubmed [41] , two co-purchase networks Amazon-Computers and Amazon-Photo [42] , and one co-authorship network Coauthor-CS [42] .Experimental Setup. The representation encoder is implemented by Graph Convolutional Network (GCN) [24] . The model parameters are initialized via Xavier initialization [15] and trained by Adam optimizer [23] . All experiments are conducted on a NVIDIA RTX 3090 GPU with 24 GB memory. The representations are first learned through our method in an unsupervised way and then evaluated by a simple linear classifier, which is the most common manner in the current self-supervised learning literature.Here, we compare our method with state-of-the-art baselines in terms of performance and efficiency.Performance Comparison. To evaluate the effectiveness of our approach, we compare our method with the state-of-the-art baselines on node classification task under the simple linear classifier. The average classification accuracy with standard deviation of 20 results is reported for each dataset. We compare our approach with unsupervised methods including DeepWalk [35] , GAE [25] , DGI [49] , GMI [34] , GRACE [58] , GCA [59] , G-BT [6] , CCA-SSG [57] InfoGCL [54] , GII [53] , GraphMAE [22] , CorInfoMax [32] , M-ILBO [30] , and MVGRL [20] . Furthermore, some supervised models including multi-layer perceptron (MLP), GCN [24] , and GAT [48] are also as baselines. We adopt the public splits on Cora, Citeseer and Pubmed, and a 1:1:8 split for training/validation/testing on the other three datasets. To make a fair comparison, for the methods without adopting the same splits as ours, we conduct experiments to get relevant results based on the officially released source code with a hyper-parameter search. Table 1 reports the classification results on six datasets. It can be observed that our method achieves high performance on all datasets and outperforms the state-of-theart peers. In particular, our method significantly outperforms neural estimator-based methods such as GRACE, GCA, and InfoGCL. These results clearly demonstrate the effectiveness of our approach. After subjecting the node representations to a rigorous statistical hypothesis testing, we discover that they do not actually conform to a Gaussian distribution. In other words, our method remains highly effective in non-Gaussian scenarios. Overall, GMIM-IC surpasses GMIM. One reason is that the identity constraint imposes stricter demands on cross-view consistency, which aligns with the practical design of the shared network architecture. Besides, GMIM-IC demonstrates comparable performance with CCA-SSG, which can serve as empirical support for our theoretical analysis.Efficiency Comparison. To illustrate the simplicity and efficiency of our model, we compare our method with other graph contrastive methods based on mutual information estimators in terms of numbers of model parameters, time consumption of training stage, and Histograms and Hypothesis Testing for Node Representations. The histograms of node representations are illustrated in Figure 2 . At first glance, the distribution of representations exhibits a Gaussian appearance. This observation served as the initial motivation of our research and sparked our curiosity about the possibility of directly performing graph self-supervised learning under Gaussian mutual information maximization. In general circumstances, mutual information cannot be directly computed and the current contrastive learning methods rely on additional neural estimators to approximate a lower bound. Without disappointment, empirical results in Table 1 demonstrate the effectiveness of our approach. Subsequently, we conduct a rigorous hypothesis testing on individual channels of representation matrices of multiple datasets based on library scipy, as shown in Algorithm 1. The outcomes indicate that node representations do not actually conform to a Gaussian distribution. This result is, in fact, promising, which implies that our approach will no longer be confined to Gaussian scenarios. In summary, visualized histograms and the Gaussian assumption provided the initial impetus for our research, while the fact that -0.9 0.9 our approach still remains its performance under the non-Gaussian conditions extends the application scenarios of our method.Effect of Gaussian Constraints. We test the effect of the Gaussian constraints L 𝑔𝑎𝑢 in Eq. ( 6) on four datasets: Cora, Citeseer, Computers, and CS, which is attached to L GMIM-IC with a weighted coefficient 𝜅. As shown in Figure 3 , we can find that L 𝑔𝑎𝑢 hardly improves the performance of our method on these existing datasets. This is because our method has already achieved highly competitive results on these datasets, unaffected by non-normality.Effect of Representation Dimension. We conduct experiments by varying the representation dimension to investigate its impacts on performance. Figure 4 summarizes the results of the three variants based on Eq. ( 2), Eq. ( 3), and Eq. ( 5) on four datasets. It can be observed that our method achieves optimal performance with an appropriately large dimension, because the representations exhibit better discriminability and linear separability in high-dimensional space. However, as the dimension becomes excessively large such as 1,024, there is a slight decrease in performance. This can be blamed on the fact that an excessively high representation dimension hinders the model from learning compact and informationdense representations. Another non-negligible underlying factor for declining performance is that higher dimensions lead to poorer estimation of the covariance matrix. Even in low-dimensional settings, our method still delivers decent performance. This finding can be attributed to the effective maximization of information entropy, which prevents dimensional collapse, enhances the diversity of representations, and ultimately improve model performance within limited dimensions. Under the objective function based on Eq. ( 2), the results in high-dimensional settings and on Computers are unavailable. In such scenarios, the covariance matrix exhibits numerous small eigenvalues, causing its determinant to approach zero. This fact introduces numerical instability and eventually disrupts training process.Impact of Balancing Coefficient. We study the impacts of the balancing coefficient 𝛽 in L GMIM-IC on performance. Figure 5 illustrates the variation of classification accuracy with varying values of the coefficient. The performance exhibits a pattern of initially increasing and later decreasing as 𝛽 goes up. When 𝛽 is small, the entropy maximization term cannot fully exploit its role in promoting diversity of representations. When 𝛽 is too large, too much emphasis on maximizing information entropy leads to informative yet meaningless representations.Synergistic Changes Between Opposite of Entropy and Decorrelation Loss. Taking L GMIM-IC as the optimization objective, we visualize the joint changes of decorrelation loss in Eq. ( 7) and opposite of entropy in Eq. (5) . For each dataset in Figure 6 , the decorrelation loss (dashed line) exhibits a nearly identical trend to the opposite of entropy (solid line). Experimental observations potentially indicate a similar effect between them, which can serve as an empirical support for Section 4.More experimental results and visual analysis are provided in the supplementary material.  
limitations, conclusion, and future work: 
 Limitations. Due to extreme limitations in computational resources, we only conducted empirical studies on graphs. Extension experiments on other types of data, such as images or multimodal data, are left for future, which relies on many GPUs. 
 
 Appendix: In this paper, we have presented a graph contrastive learning method under the common Gaussian assumption for node representations, which does not rely on any parametric mutual information estimators and negative samples. Furthermore, we provide two theoretical explanations regarding the relationship between decorrelation-based methods and contrastive-based methods. Our analysis reveals that the decorrelation-based method can be interpreted as a variant of contrastive methods when the Gaussian assumption and identity constraint are considered. Extensive comparative experiments and visual analysis have demonstrated the effectiveness, efficiency, and theoretical soundness of our method. Overall, the Gaussian assumption motivates our research, but empirical evidence demonstrates the continued effectiveness of our method in non-Gaussian scenarios, which significantly extends the practical application scope of our work.Future Work. Our research paves a new path for graph selfsupervised learning. The prospect of extending the Gaussian assumption to other distributions, such as the Cauchy distribution, stands as a viable endeavor. Furthermore, the exploration of relationships among distinct variants under different distributions represents a valuable and exciting pursuit.  


Decision: The reviewers agree that this is a well-written submission. 6uFX notes that the motivation for the method is rigorous and DneH and g4gt echo this comment. BFyx notes that self-supervised learning is a hot topic. G4gt, Dneh, and 6uFX suggest that the evaluation demonstrates good performance.

6uFX and G4Gt critique the Gaussian assumption for data. 6uFx also suggests that there could be issues with tuning parameters and computational costs on large graphs. This suggestion is picked up by BFyx who notes that the work does not really centre the term “efficient” that appears in the title. DneH has a long list of limitations, some of which are minor but calls the focus on node classification a limitation here.

The ratings here are overall on the positive side of borderline with a range from weak accept to borderline reject. Although the suitability of the work ranges in the review from “not suitable” to “definitely suitable”, the authors admit that multimodal application would be future work and relies on being “easily extended”. (Well, if it’s so easy, why didn’t the authors do it?)

Given the renewed emphasis on distinguishing this venue from others by centring clearly multimedia/multimodal research rather than research which *may* be applied to multimedia data, this limitation must be serious and hold back enthusiasm for this work.

Reviews: review: paper_summary: paper_summary: The paper introduces a method for graph self-supervised learning (SSL), focusing on Gaussian Mutual Information Maximization (GMIM) to enhance the efficiency and effectiveness of learning representations from graph data. The authors develop a theoretically motivated approach that simplifies mutual information estimation by assuming Gaussian distributions for node representations, thereby avoiding the complexities of parameterized mutual information estimators and the need for negative samples. This method also bridges the gap between contrastive and decorrelation-based SSL methods by demonstrating their theoretical and practical similarities under Gaussian assumptions.
strength_weakness: strength_weakness: *  The introduction of GMIM for graph SSL is interesting;
* The authors provide a rigorous explanation of how the method relates to and unifies different branches of SSL methodologies;
* The method is thoroughly evaluated against several benchmarks and state-of-the-art approaches on multiple datasets.
limitations: limitations: * While the Gaussian assumption simplifies the computation, it may not always hold in real-world data, which could limit the applicability of the method in scenarios where the data distribution significantly deviates from Gaussian;
* The performance of the method appears sensitive to the choice of representation dimension and balancing coefficients, which may require careful tuning in different applications;
*  The paper does not discuss the long-term computational costs associated with scaling to very large graphs, which could be a concern for practical implementations in large-scale applications.
paper_summary: paper_summary: The paper outlines the advancements in Graph Contrastive Learning , particularly focusing on the introduction of Gaussian Mutual Information (GMI) as a computationally efficient alternative to conventional methods. It highlights the bridging of GMI with decorrelation-based self-supervised learning, proposing a unified framework. Empirical evidence highlighting the superiority of the method over other methods.
strength_weakness: strength_weakness: - The paper is well-written and easy to follow. Self-supervised learning is still a hot topic in the current community.
- I noticed that the comments raised by other reviewers during the previous conference review process were well addressed. In the appendix, the paper provides detailed description about proof, experimental setting and necessary hyperparameter configuration.
limitations: limitations: - It seems the topic is out of scope of multimedia. Although this is a relatively complete work, I would recommend submitting it to other suitable machine learning conferences.
- The term 'efficient' in the title doesn't appear to be formally addressed in the paper. I observed that Table 2 includes a time comparison with other methods. This is good, but why not include the time cost for some methods, e.g. CCA-SSG, mentioned in Table 1 and compare them? This will make the experimental results more convincing.

Minor points:
- Some of the statements in the paper are a bit long-winded, such as line 369 "Mathematically, the determinant of a matrix is ​​numerically equal to the product of all eigenvalues", and readers should have this basic knowledge. Another similar situation is lines 398-400. t is recommended that the author simplify the writing appropriately.
paper_summary: paper_summary: The paper proposed a GCL method that eliminates the reliance on parameterized estimators and negative samples based on the maximization of Gaussian mutual information. The proposed method tries to bridge the contrastive-based self-supervised method to the decorrelation-based self-supervised method, and comes up with a relatively simple framework. Experiments are done to measure the performance of the method.
strength_weakness: strength_weakness: - The paper is well-written and easy to follow.
- The proposed method does not rely on structures like projection head, parameterized estimators, and negative samples, while still showing good performance.
- The theoretical part of the method is derivated rigorously.
limitations: limitations: There are some problems with the experimental part:
As for the performance comparison part:
- Experiments are only done on node classification.
- Accuracy on Computers, Photo, and Coauthor-CS is displayed with two decimals, while accuracy on the rest three datasets is not.	
- While method GMI on Coauthor-CS is marked 'OOM', InfroGCL and GII on Computers, Photo, and Coauthor-CS is marked '-' without explanation
- There are some data given without standard
As for the efficiency comparison part:
The method shows fewer costs in various respects. However, the experiments are somewhat incomplete.
- The comparison is done with 'other graph contrastive methods based on mutual information estimators', while not all methods of this kind are compared. For example, GMI, which is the basis of the proposed method, and M-ILBO, which shows relatively good performance and is discussed in section 6.3 of supplementary, are not tested in the efficiency comparison part.
- As the paper claims to bridge the gap between two research areas, the comparison may not be restricted by the kind of method.

Furthermore, there are some minor questions: 
- The theory is based on the Gaussian condition while still achieving good results on the non-Gaussian condition. Is there any explanation?  
- Or will $L_{gau}$ decrease even without adding it to loss?
paper_summary: paper_summary: The paper proposes a novel graph contrastive learning method, introducing an innovative approach in the field. The motivation behind the method is well-founded, and the approach itself is both creative and effective. The theoretical analysis conducted by the authors is accurate, although deriving it may not be particularly challenging.
strength_weakness: strength_weakness: - The article explores the information maximization approach for graph self-supervised learning effectively.
- The paper introduces a novel graph contrastive learning algorithm and offers comprehensive numerical experiments for graph self-supervised learning.
- The article explores the information maximization approach for graph self-supervised learning.
- It offers comprehensive numerical experiments for graph self-supervised learning.
- The theoretical analysis is well done.
- A novel graph contrastive learning algorithm is introduced.
- The proposed method achieves comparable results to existing approaches.
limitations: limitations: - It is reasonable to assume that the error between $h^A$ and $h^B$  follows a Gaussian distribution, but is each of them also Gaussian? In most contrastive losses, the second term (or the negative term) is typically an expectation, a mean, or a norm. However, the main concept that makes sense is the first term, which can be easily modeled as Mean Squared Error (MSE).
- The code implementation shows significant similarities to CCA-SSG.
- Title of the paper is too long.