Title: Mitigating the Popularity Bias of  Graph Collaborative Filtering: A Dimensional Collapse Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph

Collaborative Filtering

Recommendation
Abstract: Graph-based Collaborative Filtering (GCF) is widely used in personalized recommendation systems. However, GCF suffers from a fundamental problem where features tend to occupy the embedding space inefficiently (by spanning only a low-dimensional subspace). Such an effect is characterized in GCF by the embedding space being dominated by a few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popularity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches  (i.e., optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelation-enhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps  maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our  method  outperforms contrastive-based GCF models on several benchmark datasets and improves the performance for unpopular items.

Mitigating the Popularity Bias of Graph Collaborative

Abstract

Graph Collaborative Filtering (GCF) is widely used in personalized recommendation systems. However, GCF suffers from a fundamental problem where features tend to occupy the embedding space inefficiently (by spanning only a lowdimensional subspace). Such an effect is characterized in GCF by the embedding space being dominated by a few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popularity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches ( i.e ., optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelationenhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our method outperforms contrastive-based GCF models on several benchmark datasets and improves the performance for unpopular items.

1 Introduction

Collaborative Filtering (CF) has emerged as an effective technique for personalized recommendations [9, 6, 44, 8, 7]. By leveraging embedding and optimizing with a suitable loss function, CF can effectively capture complex patterns in user-item interactions [32, 16]. However, user-item interactions usually neglect the potential of the higher-hop connections. To this end, Graph Neural Networks (GNN) [36, 20, 14, 34, 50, 26] have been used to model high-hop connections in CF. Graph Collaborative Filtering (GCF) has also been proposed based on CF [15, 41].

Graph Contrastive Learning (GCL), a powerful tool for unsupervised learning, has recently been applied on a variety of different tasks [5, 47, 30, 49, 50]. Many methods also demonstrate that GCL improves the performance of GCF. The basic idea of GCL in GCF is to enhance training data with various graph augmentations and maximize the alignment between different data views [46, 43, 53]. However, the above methods ignore the disadvantages of GCL ( e.g ., dimensional collapse) and the negative impact on user and item embeddings ( e.g ., the Matthew effect) [4]. Dimensional Collapse (DC) [28] can be characterized by high similarity of features which are confined to a low-dimensional subspace or are contrated on curtain region of latent space( Figure 1b). DC limits the representational power of high-dimensional spaces by restricting the diversity of information that can be learned [28].

Although DC has been identified and tackled in a variety of domains [19], DC poses issues for GCL in the Graph Collaborative Filtering where it exacerbates the Matthew effect of popularity bias (as explained next).

A significant issue that sets GCL in GCF apart from other domains is the highly skewed data distribution over the graph where most users interact sporadically with a very small set of items within a vast interaction space comprising millions or even billions of items [4]. This results in extremely sparse user-item interactions and a long-tail item distribution following the Power Law. Propagating highly skewed data through the sparse bipartite graph via message passing enhances the Matthew effect of the popularity bias [4]. This means that several popular items may dominate the entire learning process, while other items are rarely recommended. Thus, a collapse mode may occur, where the user embedding collapses around several popular items, as shown in Figure 1a.

To migrate this issue, previous research argues that the so-called uniformity term included in the auxiliary contrastive loss can solve DC and so the uniformity term was adopted it in CF [45, 46, 38]. Particularly, Yu et al . [46] provide an empirical evidence that the features learned by the typical GCF model are non-uniformly distributed on the sphere, which causes the collapse. Wang et al . [38] have further investigated how optimizing the Bayesian Personalized Ranking (BPR) loss in CF can promote both the alignment and uniformity, and suggested to directly optimize them. However,

User Emb.

while optimizing the uniformity of both the user and item embeddings in CF can refine the em bedding space, it does not guarantee the elimination of dimensional collapse. Thus, a fundamental question arises: is there any provable remedy for DC?

To this end, we analyze the connection between dimensional collapse and GCF by demonstrating that the simplified graph convolution operation for collaborative filtering does shrink the singular space of the feature matrix. Although previous methods promote the feature uniformity, it does not prevent the dimensional collapse. Thus, we propose a new objective, Decorrelation-Enhanced GCF, that employs geometry of the LogDet divergence to relax orthogonality constraints and encourage the feature diversity. Our method maintains the range space of the feature matrix with a small condition number (well-conditioned), providing theoretical guarantees against dimensional collapse. Our experimental results demonstrate the effectiveness of the proposed approach in addressing feature collapse. We outperform contrastive-based CF models on several benchmark datasets. Notably, our approach improves the performance for unpopular items.

We summarize our contributions below:

We demonstrate that embeddings learned from existing GCF models suffer from dimensional collapse. Our theoretical analysis reveals that the dimensional collapse in GCF arises from propagation, and that the current uniformity-based objective does not guarantee to alleviate it.

To mitigate the issue of dimensional collapse, we propose a decorrelation-enhanced objective following the principle of redundancy reduction [47] and LogDet geometry.

Instead of using the Frobenius norm (the Euclidean geometry) to relax the conditional optimization problem, we propose using the LogDet divergence, a Bregman divergence with theoretical guarantees, to preserve the range space of the covariance matrix.

Our experiments show that our proposed method effectively mitigates dimensional collapse and outperforms various contrastive learning-based CF models. Notably, it achieves improved the performance for unpopular items, thus it mitigates the Matthew effect.

2 Preliminaries

Collaborative Filtering. Let U and I denote the user and item set, respectively. Given a set of observed user-item interactions R = { ( u, i ) | u interacted with i } , CF methods infer the score s ( u, i ) ∈ R for each unobserved user-item pair indicating how likely the user u is to interact with

the item i . Then, items with the highest scores for each user will be recommended based on the predictions [32, 16]. Most CF methods use an encoder network f ( · ) that maps each user and item into a low-dimensional representation f ( u ) , f ( i ) ∈ R d (where d is the dimensionality of the latent space). For example, the encoder in matrix factorization models is usually an embedding table, which directly maps each user and item to a latent vector based on their IDs. The encoder in graph-based models uses the neighborhood information. The predicted score is defined as the similarity between the user/item representations, e.g ., dot product, s ( u, i ) = f ( u ) ⊤ f ( i ) . Most studies adopt the pairwise BPR [31] loss to train the model:

where i -is a randomly sampled negative item that the user has not interacted with. Such a loss maximizes sigmoid scores: target user-item scores become higher than user-negative item scores.

LightGCN. A linear graph neural network backbone, LightGCN [15], is one of the most successful backbones in Graph Collaborative Filtering. It leverages neighborhood information at different levels of locality, and combines outputs of different layers by the sum aggregation.

To facilitate the analysis, we present the matrix from of the LightGCN. Let R ∈ R M × N be the matrix representation of set R where R ui = 1 if ( u, i ) ∈ R (else R ui = 0 ). Let M and N be the numbers of users and items, and N ′ = M + N . We then obtain the adjacency matrix of the user-item graph as:

Let the embedding matrix of input layer be E ∈ R N ′ × d , where d is the embedding size. Then the K -th layer of LightGCN in the matrix form is defined as:

where α 1 , α 2 , . . . , α K are weights, and ∑ K k =1 α k = 1 . Moreover, ˜ A = D -1 2 AD -1 2 is a normalized adjacency matrix and the degree matrix D is an N ′ × N ′ diagonal matrix in which each entry D ii denotes the number of non-zero entries in the i -th row vector of the adjacency matrix A .

Dimensional Collapse (DC). Often referred to as spectral collapse [28], dimensional collapse is a prevalent phenomenon in representation learning. It occurs when the embedding space is dominated by a small number of large singular values, while the remaining singular values decay significantly as the training progresses. This phenomenon limits the representation power of high-dimensional spaces by restricting the diversity of information that can be learned.

3 Popularity Bias from a Perspective of Dimensional Collapse

3.1 Popularity Bias vs . Dimensional Collapse in Graph Collaborative Filtering

Dimensionality collapse and Popularity Bias in CF are two sides of the same coin. The prevalence of popularity bias in GCF can be attributed to the significant imbalance within the data distribution across the recommendation bipartite graph [4]. This distribution often features numerous users engaging with a small subset of items, within a much larger interaction space encompassing millions or billions of items. Consequently, during the process of message passing within GCF, a few popular items can disproportionately influence learning, leaving out other items. This leads to the dimensional collapse in the embedding space where most user embeddings are around very few popular item embeddings.

We illustrate this by showing the 2D latent embedding space of the user and item in Figure 2:

Figure 2 (top) shows that both user and item embedding are clustered in a few regions, the gaps in the unpopular item embedding cause effective rank drop. We refer to this effect as the dimensional collapse . As the unpopular items are clustered in just one cluster, and the corresponding user cluster is non-dense so unpopular items do not get recommended often ( popularity bias ).

user embedding

item embedding

popular item emb.

unpopular item emb.

dense user clusters align well

with dense popular-item clusters

non-dense user

cluster aligns

with the unpopular-item cluster

user embedding

item embedding

popular item emb.

unpopular item emb.

this unpopular

item cluster

has no user

cluster at all

half-circle

has spread

on hyperspere

and formed 3

user clusters

two new unpopular-item clusters

have aligned with

dense user

clusters

dense popular-item clusters have not

changed much & match dense user clusters

good spread

on hyperspnere

CF with

LightGCN

CF with

LightGCN

+LogDet

Use LogDet

=

alleviated

dimensional

collapse

Mitigating

popularity

bias

+

+

+

+

+

+

+

+

ii. If only the embeddings were spread uniformly on the surface of the hypersphere, it would align with dense user clusters which would reduce the popularity bias. Figure 2 (bottom) shows that thanks to more uniformly spread unpopular items, they form now 3 clusters, and 2 of them align well with dense user clusters, which means the popularity bias has been reduced.

3.2 Dimensional Collapse in Graph Collaborative Filtering

The previous section reveals that the occurrence of dimensional collapse is closely related to the presence of popularity bias, as depicted in Figure 2. Below, we show how the typical backbone of GCL in GCF ( i.e ., LightGCN) potentially causes the DC in the learned embedding space, and why optimizing uniformity [38] (the best current solution) cannot guarantee to prevent DC.

Lemma 1 (Equivalent reformulation of LightGCN.) The propagation process of LightGCN in Equation (3) can be viewed as a result of optimization of the following objective:

where L = I -˜ A is the normalized symmetric positive semi-definite graph Laplacian matrix. □ Proof of Lemma 4 is in Appendix A.1.

Lemma 1 shows that the propagation process of LightGCN can be derived from the optimization formulation which uses the Graph Laplacian regularization. This implies that the output embedding Z is smooth due to the Graph Laplacian regularization in Equation (4). Similar derivations for the graph neural network with learnable weights were shown by Ma et al . [29] and Zhu et al . [58].

To understand the dynamics of the embedding space, we denote the state of the embedding matrix and the output of the LightGCN at time t as E ( t ) and the initial state E (0) is randomly initialized. Let { σ E n } N ′ n =1 denote eigenvalues of E (descending order)

Lemma 2 (Shrinking singular space of feature matrix.) Consider minimizing the objective in Lemma 1 with the Laplacian regularization term tr ( Z ⊤ LZ ) . The relative value of the ratio

of the smaller eigenvalues to the larger eigenvalues will decrease as t increases. Formally, σ E i ( t ) σ E j ( t ) ≤ σ E i ( t ′ ) σ E j ( t ′ ) , ∀ t < t ′ , i > j (we ignore the trivial case i = j ). Then:

□ Proof of Lemma 2 is in Appendix A.2.

Based on Lemmas 1 and 2, we conclude that due to the propagation step in Equation (3), the embedding space of feature matrix is dominated by few largest eigenvalues. This is a poof of dimensional collapse of GCF pipelines which are typically based on the LightGCN backbone.

3.3 Optimizing Uniformity in Graph Collaborative Filtering.

Typical contrastive GCF approaches adopt the BPR loss from Equation (1) and the InfoNEC loss. Wang et al . [38] argued that the so-called alignment and uniformity [38] loss terms contribute to higher recommendation performance:

where L algin captures the user-item relations. L uni encourages embeddings to be distributed evenly on the surface of ℓ 2 ball. However, L uni does not avoid the dimensional collapse shown in Figure 1.

Concluding, Contrastive GCF suffers from several problems:

DC [17] is characterized by one or more singular values in the feature matrix to be zero, e.g ., λ i = 0 . L uni may yield a finite reward (decrease of L uni ) to avoid the dimensional collapse ( λ i = 0 ). However, if the remaining loss terms yield a higher reward (decrease in their value) than L uni for occurrence of the dimensional collapse ( λ i = 0 ) then DC will occur.

As temperature 1 /τ increases, L uni degrades to a hard minimum. Take just the user part of Equation (6), include the temperature variable 1 /τ , and notice that:

which explains the empirical observation of Wang and Liu [39] that the uniformity of embeddings worsens as the temperature increases.

iii. As Lemma 2 shows, LightGCN itself is also prone to suffering from the dimensional collapse.

Thus, in what follows we consider the redundancy reduction principle to mitigate the above issues.

4 Methodology

Graph Collaborative Filtering with Decorrelation. From the perspective of redundancy reduction [47, 2], achieving high-quality self-supervised embeddings requires that: (i) positive pairs exhibit similar semantics; (ii) the embeddings follow a non-trivial constant distribution, with a trivial distribution indicating that all embeddings collapse into a single point; and (iii) there is zero correlation between different features.

In the context of collaborative filtering, we can frame this problem as follows:

where L align ( θ )= E ( i,u ) ∼R [ ∥ f θ ( u ) -f θ ( i ) ∥ 2 2 ] (from Equation (6)) captures user-item preferences. Constraints ∑ i ∈U f θ ( u ) f θ ( u ) ⊤ = ∑ i ∈I f θ ( i ) f θ ( i ) ⊤ = I help embeddings follow the isotropic Normal distribution. The isotropy prevents the dimensional collapse but its decorrelating effect may

Isotropic

& Full Rank

Isotropic

& Rank-deficient

Non-isotropic

& Rank-deficient

Non-isotropic

& Full Rank

Eigenvalue Index

Eigenvalue Index

Eigenvalue Index

Eigenvalue Index

be too strong. Equation (8) may be solved by applying Lagrangian multipliers or the Stiefel manifold (GeoTorch [25]) or by converting hard constraints into soft constraints:

However, relaxation L soft in Equation (9) suffers from the same problem as L uni in Equation (6). Specifically, these both regularization terms yield a finite penalty for any singular values λ i = 0 leading to dimensional collapse if other loss terms yield a higher reward than the penalty is. Thus, we explore the application of Bregman divergence. Our study indicates that the proposed approach preserves the range space of the covariance matrix, promoting a full-rank embedding matrix.

Decorrelation by the LogDet Divergence. Figure 3 and Table 1 summarizes properties of different decorrelation terms. While L uni and L soft encourage soft decorrelation (approximate isotropy), they may fail to impose full rank on the embedding matrix. Stiefel-based optimization enjoys hard constraints but full decorrelation (exact isotropy) may be too restrictive. In contrast, LogDet divergence based L logdet encourages isotropy in the soft manner and imposes full rank on embedding matrix as shown below. Firstly, we review the Bregman divergence and introduce our LogDet based decorrelation penalty.

Definition 1 (Bregman Divergence on Matrices.) Let X , Y ∈ S d be two symmetric matrices. Let ϕ : S n → R be a strictly convex and differentiable matrix function. The Bregman matrix divergence is defined as: D ϕ ( X , Y ) = ϕ ( X ) -ϕ ( Y ) -tr ( ( ∇ ϕ ( Y )) ⊤ ( X -Y ) ) . (10)

The Bregman divergence is a measure of the nearness between matrices X and Y . It is computed by using the first-order Taylor approximation of a convex generating function ϕ ( · ) . The Bregman divergence is always non-negative, and it becomes zero only when the two matrices are equal.

As Σ U = ∑ u ∈U f θ ( u ) f θ ( u ) ⊤ ∈ S d + and Σ I = ∑ i ∈I f θ ( i ) f θ ( i ) ⊤ ∈ S d + are symmetric, and at least positive semi-definite (PSD) matrices but we desire them to be strictly positive definite (PD) matrices ( S d ++ ) to prevent the dimensional collapse, we firstly generalize L soft in Equation (9) to its Bregman divergence based variant: L breg = D ϕ ( Σ U , I ) + D ϕ ( Σ I , I ) . (11)

Different ϕ ( · ) lead to different measures of nearness. For example, ϕ F ( X ) = ∥ X ∥ 2 F yields the Frobenius-based D ϕ F ( Σ U , I )+ D ϕ F ( Σ I , I )= ∥ Σ U -I ∥ 2 F + ∥ Σ I -I ∥ 2 F penalty L soft in Equation (9).

Let X = UΛU ⊤ be an eigenvalue decomposition of X , and Λ =diag([ λ 1 , . . . , λ d ]) . Choosing ϕ ld ( X )= -log det X = -∑ i log λ i leads to the LogDet divergence D ϕ ld ( X , Y ) = tr ( XY -1 ) -log det ( XY -1 ) -d where X , Y ∈ S d ++ . Then generalizing L soft in Equation (9) to its LogDet divergence variant by computing D ϕ ld ( Σ U , I ) + D ϕ ld ( Σ I , I ) and removing -2 d constant yields:

Our LogDet decorrelation-enhanced GCF objective becomes:

In contrast to the Frobenius-based L soft , the LogDet-based penalty L logdet maintains the range space of the covariance matrices and upper bounds the condition number of the matrices. This ensures the user/item embedding matrices are of full rank which prevents the dimensional collapse.

5 Theoretical Analysis.

To use the LogDet divergence based penalty L logdet in GCF, we discuss important to our work theoretical properties of the LogDet divergence and L logdet for rank-deficient matrices.

LogDet Divergence and L logdet for Rank-deficient Matrices.

Corollary 1 . LogDet divergence D ϕ ld ( X , Y ) is finite iff rank( X ) = rank( Y ) [23], which implies that L logdet = D ϕ ld ( Σ U , I )+ D ϕ ld ( Σ I , I ) = ∞ if rank( Σ U ) < rank( I ) = d ∨ rank( Σ I ) < rank( I ) = d .

Corollary 2 . Let Σ U , Σ I ∈ S d ++ . Note that L soft = D ϕ F ( Σ U , I )+ D ϕ F ( Σ I , I ) is finite if both ∥ Σ U ∥ 2 and ∥ Σ I ∥ 2 are finite. There exist cases where some infinitesimal change ∆ Σ U and/or ∆ Σ I will make Σ U +∆ Σ U and/or Σ I +∆ Σ I positive semi-definite (PSD). Define the gain of the Bregman divergence based decorrelation penalty as g ϕ ( L reg ) = D ϕ ( Σ U , I )+ D ϕ ( Σ I , I ) -[ D ϕ ( Σ U +∆ Σ U , I )+ D ϕ ( Σ I +∆ Σ I , I )] . Notice, in case of change from PD to SPD matrix, gain g ϕ ( L reg ) < 0 . Let the gain of the alignment loss be g ( L align ) ≪∞ . Then one may easily construct examples where g ( L align ) -g ϕ F ( L soft ) > 0 . That is to say, the net loss -g ϕ F ( L soft ) in case of covariance degradation to SPD is lesser than the net gain g ( L align ) due to increased user-item alignment. In contrast, from Corollary 1, g ( L align ) -g ϕ ld ( L logdet ) = -∞ in case of covariance degradation to SPD, which prevents the dimensional collapse.

□ Proof of Corollary 2 is in Appendix A.3.

Corollary 3 . Expression -log det ( Σ U Σ I ) from Equation (12) might fail to yield ∞ if Σ U Σ I is indefinite. However, covariance matrices are at least SPD by definition, and so is their matrix product.

Corollaries 1, 2 and 3 demonstrate that L logdet naturally maintains the full-rank constraints on covariance matrices of user and item embeddings.

̸

Minimizing D ϕ ld ( Σ , I ) Reduces the Condition Number of Σ . Feature decorrelation is closely related to the condition number. If the spectrum of a matrix is dominated by the leading eigenvalue, the matrix is ill-conditioned and has a large condition number. Let { λ i } d i =0 be the eigenvalues of the matrix Σ . The condition number is defined as cond( Σ ) = λ max λ min . The ratio between the largest eigenvalue and the smallest eigenvalue quantifies the flatness (isotropy) of the spectrum. For example, cond( Σ ) = 1 implies a fully balanced spectrum with λ i = λ j for all i = j ( e.g ., spectrum balancing [21, 22, 48, 52]). Below we show that our proposed method minimizes the upper bound of cond( Σ ) .

λ

□ Proof of Lemma 3 is in Appendix A.4.

Lemma 3 implies that the sum of condition numbers of Σ U and Σ I is upper bounded by the LogDet divergence based penalty L logdet . Minimizing that penalty implies the flattening of spectrum implies decorrelation/redundancy reduction of embedding.

6 Experiments

Datasets. To evaluate the effectiveness of our GCF with the LogDet divergence based penalty L logdet , denoted as GCF logdet , we conduct experiments on three large-scale public datasets: Yelp2018 [15], Amazon-kindle [43], and Alibaba-iFashion [43]. See statistics of these datasets in Appendix A.6. The datasets were split into the training, validation, and test set, with the ratio of 7 : 1 : 2 . As per the methodology suggested by [43, 15], we first identified the best hyperparameters on the validation set and then merged the training and validation sets to train the model. Finally, the test set was used to evaluate the model using the relevancy-based metric Recall@ 20 and the ranking-aware metric NDCG@ 20 .

Baselines. Apart from the typical GCF models, i.e ., NGCF [41] and LightGCN [15], we also compare GCF logdet with recent data CL-based recommendation models. SGL [43] and DNN+SSL [45] adopt CL as an auxiliary task and conduct feature masking for contrastive learning (CL). BUIR

[24] has a two-branch architecture with a target network and an online network, utilizing only positive examples for self-supervised recommendation. MixGCL [18] introduces the hop mixing technique for synthesizing hard negatives for GCF through embedding interpolation. NCL [27] is a recent contrastive model that designs a prototypical contrastive objective to capture the correlations between a user/item and its context. SimGCL [46] is a CL-based model for recommendation that does not rely on graph augmentation. DirectAU [38] proposes optimizing the alignment and uniformity of both user and item directly for recommendation.

Implementation. We referred to the best hyperparameter settings reported in the original papers of the baselines and then fine-tuned them with the grid search. Detailed settings are in Appendix A.5.

6.1 Main Results

Decorrelation vs . Uniformity. Within this section, we compare the performance of our proposed method, GCF logdet , with contrastive-based methods ( i.e ., SGL, DirectAU) that optimize uniformity to prevent the dimensional collapse in CF. Furthermore, we demonstrate the effectiveness of our method by varying the number of encoder layers. Table 3 shows the performance of different baseline CF methods and our GCF logdet . In the vast majority of cases, contrastive-based methods ( e.g ., SGL, DirectAU, SimGCL) significantly outperform LightGCN. The largest improvements are observed on Alibaba-iFashion, where the performance of GCF logdet surpasses that of LightGCN under 1 and 3 layers settings. This observation supports our conclusion that LightGCN is susceptible to the dimensional collapse, leading to poor results. Additionally, we observe that DirectAU outperforms SGL, indicating that graph augmentation is not the primary factor for enhancing the performance of the CL-based recommendation model. Instead, our findings demonstrate the importance of designing appropriate loss functions over sophisticated augmentation. Finally, GCF logdet achieves the best results on the all three datasets. Specifically, GCF logdet outperforms DirectAU and SGL, highlighting that decorrelation is more effective than optimizing uniformity for preventing dimensional collapse.

0

20

40

60

80

Epochs

2

4

6

8

10

Effective Rank

logdet

uni

soft

(a) Yelp2018

0

20

40

60

80

Epochs

2

4

6

8

10

Effective Rank

logdet

uni

soft

(b) Kindle

0

20

40

60

80

Epochs

2

4

6

8

10

12

14

Effective Rank

logdet

uni

soft

(c) iFashion

1000

2000

3000

4000

5000

6000

Traing Steps

12

13

14

15

16

17

18

19

Condition Num.

Yelp2018

Kindle

Alibaba-iFasion

(d) Yelp2018

Performance Comparison with the State of the Art. To further demonstrate the exceptional performance of GCF logdet , we conduct a comparative analysis with several recently proposed recommendation models that are based on augmentation or contrastive learning techniques. As illustrated in Table 3, GCF logdet outperforms the other models by a significant margin, achieving the best performances, respectively. Additionally, NCL and MixGCF, which utilize LightGCN as their backbone, exhibit competitive performance. In contrast, DNN+SSL and BUIR fall short of expectations and are not comparable to LightGCN. We attribute their suboptimal performance to the fact that DNNs have been proven to be effective only when abundant user/item features are available. In our datasets, however, such features are unavailable, and self-supervision signals are created by masking item embeddings, which makes it difficult for these models to perform well in this context. We also report performance on Recall@K and NDCG@K with K = 5 , 10 in Table 9.

6.2 Quantitative Analysis

Measuring the Dimensional Collapse. The evidence of dimensional collapse was identified in contrastive learning by observing the spectrum of representations [19]. The rank of the matrix corresponds to the number of dimensions retained by the transformation ( i.e ., the dimension of its range) but reveals nothing about the shape of the spectrum. The effective rank [33] (see the definition in the Appendix A.7) quantifies how balanced the spectrum is by means of the spectral entropy. Thus, to show the dimensional collapse tendency when training the model, instead of showing the distribution of eigenvalues, we use the effective rank. Figure 4 shows that the effective rank increases as the training progresses, indicating an increasingly balanced eigenvalue distribution. GCF logdet consistently achieves higher effective rank compared to other methods. This finding not only verifies the existence of the dimensional collapse in CF but also indicates the effectiveness of our GCF logdet . Figure 4d further shows that the condition number decreases as more batches are processed, thus balancing the spectrum of embedding matrices.

Overcoming the Popularity Bias. Below we investigate whether our GCF logdet can reduce the popularity bias by promoting more uniform representations and alleviating the problem of dimensional collapse. We partition the test set into three subsets based on item popularity. Specifically, we label 80% of the items with the fewest clicks/purchases as 'Unpopular,' 5% of the most clicked/purchased items as 'Popular,' and the remaining items as 'Normal.' We then conduct experiments to measure Recall@20 contributed by each group, with the overall Recall@20 value being the sum of the values

Unpopular

Normal

Popular

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Recall@20

LightGCN

SGL

SimGCL

DirectAU

GCF

soft

GCF

Logdet

Unpopular

Normal

Popular

Recall@20

Yelp2018

Alibaba iFashion

from all three groups. The results are presented in Figure 5 and Table 5, which show that our method significantly improves the ability to recommend unpopular items. In contrast, LightGCN tends to recommend popular items and achieves the highest recall value on Normal and Popular items.

Ablations on Different Decorrelation Penalties. Table 4 compares the performance on LightGCN and MLP equipped with different decorrelation penalties. L logdet appears to consistently outperform other penalties, thus verifying its ability to deal with the dimensional collapse and balancing the spectrum of embeddings. Appendix A.8 compares the KL Matrix Divergence and the Riemannian metric in terms of speed. Appendix A.9 compares different distance types.

7 Related Works

Graph Contrastive Learning applies CL to the graph domain. By adapting DeepInfoMax [1] to graph representation learning, DGI [37] learns embedding by maximizing the mutual information to discriminate between nodes of original and corrupted graphs. REFINE [54] uses a simple negative sampling term inspired by skip-gram models. COLES [57] and GLEN [56] link the GCL to the form of Laplacian Eigenmap with negative sampling, extended also to image domain by EASE [55]. GRACE [59]/GraphCL [13] create views via graph augmentation for node/graph-level task. COSTA [51] and SFA [52] create graph views via feature augmentations.

Graph Collaborative Filtering. Graph Neural Networks (GNNs) have proven to be powerful architectures for modeling recommendation data, replacing MLP-based models and improving the performance of neural recommender systems [20, 15]. GNNs are a foundation for several state-of-the-art recommendation models. The most common GNN variant is GCN, which has driven the development of graph neural recommendation models, such as GCMC, NGCF, and LightGCN [3, 41, 15]. These GCN-based models refine embeddings and perform graph reasoning by gathering neighborhood information in the user-item graph. Among such models, LightGCN stands out for its effectiveness and simplicity by eliminating transformation matrices and non-linear activation functions. The success of LightGCN has inspired subsequent CL-based recommendation models, such as SGL and SimGCL [43, 46].

Graph Contrastive Learning for Recommendation. Inspired by the success of CL in other fields [11, 5, 35], CL has been combined with recommendation systems. Yao et al . [45] proposed a feature dropout based two-tower architecture for large-scale item recommendation. NCL [27] designed a prototypical contrastive objective to capture the correlations between a user/item and its context. SimGCL [46] adds noise to the embedding space to perform feature augmentation to enhance the performance of recommendations. DirectAU [38] proposes optimizing the alignment and uniformity of both user and item during training. The most widely used model, SGL [43], applies edge/node dropout to augment the graph data. Although these methods have demonstrated their effectiveness, they pay little attention to why CL can enhance recommendations.

8 Conclusions

We have investigated the popularity bias in Graph Collaborative Filtering, and showed it results from the dimensional collapse of the embedding space. We have demonstrated that the commonly used backbone in GCF, LightGCN, is prone to pose the dimensional collapse. To this end, we have proposed the LogDet divergence based decorrelation penalty for the GCF. We have showed that GCF logdet promotes full-rank embedding matrices and yields small condition number with a theoretical guarantee. For these reasons, GCF logdet has outperformed other contrastive-based GCF models on several benchmark datasets and improved the performance for unpopular items.

Acknowledgments

The work described in this paper was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), RGC Research Impact Fund (RIF), R5034-18 (CUHK 2410021), and RGC General Research Funding Scheme (GRF) 14222922 (CUHK 2151185). PK is supported by CSIRO's Science Digital.

References

P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. NeurIPS , pages 15509-15519, 2019.

A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

R. v. d. Berg, T. N. Kipf, and M. Welling. Graph convolutional matrix completion. arXiv preprint arXiv:1706.02263 , 2017.

J. Chen, H. Dong, X. Wang, F. Feng, M. Wang, and X. He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems , 41(3):1-39, 2023.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607. PMLR, 2020.

Y. Chen, Y. Fang, Y. Zhang, and I. King. Bipartite graph convolutional hashing for effective and efficient top-n search in hamming space. In Proceedings of the ACM Web Conference 2023 (WebConf) . ACM, 2023.

Y. Chen, M. Yang, Y. Zhang, M. Zhao, Z. Meng, J. Hao, and I. King. Modeling scale-free graphs with hyperbolic geometry for knowledge-aware recommendation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM) , pages 94-102, 2022.

Y. Chen, Y. Yang, Y. Wang, J. Bai, X. Song, and I. King. Attentive knowledge-aware graph convolutional networks with collaborative guidance for personalized recommendation. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 299-311. IEEE, 2022.

Y. Chen, Y. Zhang, M. Yang, Z. Song, C. Ma, and I. King. Wsfe: Wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , 2023.

A. Cherian, S. Sra, A. Banerjee, and N. Papanikolopoulos. Jensen-bregman logdet divergence with application to efficient similarity search for covariance matrices. IEEE transactions on pattern analysis and machine intelligence , 35(9):2161-2174, 2012.

T. Gao, X. Yao, and D. Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 , 2021.

G. H. Golub and C. F. Van Loan. Matrix computations . JHU press, 2013.

H. Hafidi, M. Ghogho, P. Ciblat, and A. Swami. Graphcl: Contrastive self-supervised learning of graph representations. arXiv preprint arXiv:2007.08025 , 2020.

W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 639-648, 2020.

X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua. Neural collaborative filtering. In WWW , pages 173-182. International World Wide Web Conferences Steering Committee, 2017.

T. Hua, W. Wang, Z. Xue, S. Ren, Y. Wang, and H. Zhao. On feature decorrelation in selfsupervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9598-9608, 2021.

T. Huang, Y. Dong, M. Ding, Z. Yang, W. Feng, X. Wang, and J. Tang. Mixgcf: An improved training method for graph neural network-based recommender systems. In KDD , pages 665-674, 2021.

L. Jing, P. Vincent, Y . LeCun, and Y. Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 , 2021.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

P. Koniusz, F. Yan, P.-H. Gosselin, and K. Mikolajczyk. Higher-order occurrence pooling on mid-and low-level features: Visual concept detection. 2013.

P. Koniusz and H. Zhang. Power normalizations in fine-grained image, few-shot image and graph classification. IEEE Trans. Pattern Anal. Mach. Intell. , 44(2):591-609, 2022.

B. Kulis, M. A. Sustik, and I. S. Dhillon. Low-rank kernel learning with bregman matrix divergences. Journal of Machine Learning Research , 10(2), 2009.

D. Lee, S. Kang, H. Ju, C. Park, and H. Yu. Bootstrapping user and item representations for one-class collaborative filtering. In F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, and T. Sakai, editors, SIGIR , pages 1513-1522, 2021.

M. Lezcano Casado. Trivializations for gradient-based optimization on manifolds. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32, 2019.

L. Liang, Z. Xu, Z. Song, I. King, and J. Ye. Resnorm: Tackling long-tailed degree distribution issue in graph neural networks via normalization. CoRR , abs/2206.08181, 2022.

Z. Lin, C. Tian, Y. Hou, and W. X. Zhao. Improving graph collaborative filtering with neighborhood-enriched contrastive learning. In WWW , pages 2320-2329, 2022.

K. Liu, W. Tang, F. Zhou, and G. Qiu. Spectral regularization for combating mode collapse in gans. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6382-6390, 2019.

Y. Ma, X. Liu, T. Zhao, Y. Liu, J. Tang, and N. Shah. A unified view on graph neural networks as graph signal denoising. In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021 . ACM, 2021.

Y. Ma, Z. Song, X. Hu, J. Li, Y. Zhang, and I. King. Graph component contrastive learning for concept relatedness estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 13362-13370, 2023.

S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 , 2012.

F. Ricci, L. Rokach, and B. Shapira. Introduction to recommender systems handbook. In Recommender systems handbook , pages 1-35. Springer, 2010.

O. Roy and M. Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference , pages 606-610. IEEE, 2007.

Z. Song and I. King. Hierarchical heterogeneous graph attention network for syntax-aware summarization. In AAAI , pages 11340-11348. AAAI Press, 2022.

Z. Song, Z. Meng, Y. Zhang, and I. King. Semi-supervised multi-label learning for graphstructured data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1723-1733, 2021.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

P. Velickovic, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm. Deep graph infomax. In ICLR , 2019.

C. Wang, Y. Yu, W. Ma, M. Zhang, C. Chen, Y. Liu, and S. Ma. Towards representation alignment and uniformity in collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1816-1825, 2022.

F. Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2495-2504, 2021.

T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , pages 9929-9939. PMLR, 2020.

X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval , pages 165-174, 2019.

X. Wang, H. Jin, A. Zhang, X. He, T. Xu, and T.-S. Chua. Disentangled graph collaborative filtering. In SIGIR , pages 1001-1010, 2020.

J. Wu, X. Wang, F. Feng, X. He, L. Chen, J. Lian, and X. Xie. Self-supervised graph learning for recommendation. In SIGIR , pages 726-735, 2021.

M. Yang, M. Zhou, J. Liu, D. Lian, and I. King. Hrcf: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM Web Conference 2022 , pages 2462-2471, 2022.

T. Yao, X. Yi, D. Z. Cheng, F. Yu, T. Chen, A. Menon, L. Hong, E. H. Chi, S. Tjoa, J. Kang, et al. Self-supervised learning for large-scale item recommendations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 4321-4330, 2021.

J. Yu, H. Yin, X. Xia, T. Chen, L. Cui, and Q. V. H. Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In SIGIR , pages 1294-1303, 2022.

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

S. Zhang, D. Luo, L. Wang, and P. Koniusz. Few-shot object detection by second-order pooling. In ACCV , volume 12625 of Lecture Notes in Computer Science , pages 369-387. Springer, 2020.

Y. Zhang, Y. Chen, Z. Song, and I. King. Contrastive cross-scale graph knowledge synergy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD) , 2023.

Y. Zhang, H. Zhu, Z. Meng, P. Koniusz, and I. King. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022 , pages 1331-1339, 2022.

Y. Zhang, H. Zhu, Z. Song, P. Koniusz, and I. King. COSTA: Covariance-preserving feature augmentation for graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2524-2534, 2022.

Y. Zhang, H. Zhu, Z. Song, P. Koniusz, and I. King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11289-11297, 2023.

X. Zhou, A. Sun, Y. Liu, J. Zhang, and C. Miao. Selfcf: A simple framework for self-supervised collaborative filtering. arXiv preprint arXiv:2107.03019 , 2021.

H. Zhu and P. Koniusz. REFINE: Random range finder for network embedding. In ACM Conference on Information and Knowledge Management , 2021.

H. Zhu and P. Koniusz. EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9078-9088, June 2022.

H. Zhu and P. Koniusz. Generalized laplacian eigenmaps. Advances in Neural Information Processing Systems , 35:30783-30797, 2022.

H. Zhu, K. Sun, and P. Koniusz. Contrastive laplacian eigenmaps. Advances in Neural Information Processing Systems , 34:5682-5695, 2021.

M. Zhu, X. Wang, C. Shi, H. Ji, and P. Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021 , pages 1215-1226, 2021.

Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Deep graph contrastive representation learning. 2020.

Mitigating the Popularity Bias of Graph Collaborative

Filtering: A Dimensional Collapse Perspective ( Supplementary Material )

Yifei Zhang † , Hao Zhu ‡ , § , Yankai Chen † , Zixing Song † , Piotr Koniusz ∗ , § , ‡ , Irwin King † † The Chinese University of Hong Kong ‡ Australian National University, § Data61 CSIRO {yfzhang, ykchen, zxsong, king}@cse.cuhk.edu.hk allenhaozhu@gmail.com, piotr.koniusz@data61.csiro.au

A Appendices

A.1 Proof of Lemma 1

Proof 1 One can set derivative of Equation (3) with respect to Z to zero and get the optimal Z as:

Note that det( I + η L ) > 0 , thus matrix { I + η L } -1 exists. Then the corresponding closed-form solution can be written as:

Since η 1+ η < 1 for ∀ η > 0 , and matrix ˜ A has absolute eigenvalues bounded by 1, thus, all its positive powers have bounded operator norm, then the inverse matrix can be decomposed as follows with k →∞ :

Note that 1 1+ η + η (1+ η ) 2 + · · · + η K -1 (1+ η ) K + · · · = 1 and we can change the coefficient η ∈ (0 , ∞ ) to fit fusion weights α 1 , α 2 , · · · , α K . When the layer K is large enough, the propagation mechanism of LightGCN in Equation (3) approximately corresponds to the objective Equation (4).

A.2 Proof of Lemma 2

Proof 2 First, let us take the gradient of tr ( Z ⊤ LZ ) with respect to the input matrix E and denote Z = ˆ AE where ˆ A = ∑ K k =1 α k ˜ A k .

Treat the weight matrix as a function of the training step t , i.e ., E = E ( t ) , then we can derive the gradient of E ( t ) with respect to t by d E ( t ) d t = 2 QE . As both Q are fixed, we can solve the equation analytically,

As we have the non-ascending eigenvalues of Q as λ Q 1 ≥ λ Q 2 ≥ · · · ≥ λ Q D , we can define an auxiliary function f ( t ; λ Q i , λ Q j ) = exp ( λ Q i t ) / exp ( λ Q j t ) = e ( λ Q i -λ Q j ) t . It is obvious that f ( t ; λ Q i , λ Q j ) is monotonically decreasing for all i > j . As E ( t ) is a transformation of its initial state E (0) up to exp( Q t ) , we can conclude that:

Let the spectrum be following the descending order. Then we have lim t →∞ f ( t ; λ Q i , λ Q j ) = 0 , ∀ i > j if λ Q i = λ Q j .

̸

Notice the above expression analyses the decay of spectrum for matrix exp(2 Q t ) . Thus, assume E (0) is a full-rank matrix. Then

due to the well-known inequality stating that rank( XY ) ≤ min ( rank( X ) , rank( Y ) ) .

A.3 Proof of Corollary 2

Imagine that Σ U = diag([1 , 0 . 1]) . Let ∆ Σ U = diag([0 , -0 . 1]) then log det( Σ U + Σ U ) = log λ 1 + log λ 2 = log 1+log 0 = -∞ so L logdet = ∞ . In contrast, for L soft we have ( λ 1 -1) 2 +( λ 2 -1) 2 = 0 . 81 . If 10 user feature vectors f ( u ) = diag ([1 , 0 . 31]) are in relation with 10 item feature vectors f ( i ) = diag([1 , 0 . 0]) , it is easy to see that 10 · 0 . 31 2 ≈ 1 which means the alignment loss is better off with the dimensional collapse for L soft as 1 > 0 . 81 . For the LogDet penalty however we have 1 ≪∞ .

A.4 Proof of Lemma 3

We have the following:

Proof 3

Now, x -log x ≥ 1 with equality at x = 1 . Also, x -log x ≥ log x +1 -log 4 with equality at x = 2 . Letting λ 1 ≥ λ 2 · · · ≥ λ d > 0 , we have:

Thus, LogDet yields an upper bound on the condition number.

A.5 Detailed Settings

For the general settings, we create the user and item embeddings with the Xavier initialization of dimension 64; we use Adam to optimize all the models with the learning rate 0.001; the l 2 regularization coefficient 10 -4 and the batch size 2048 are used, which are common in many papers [15, 43, 42]. In SimGCL and SGL, we empirically set the temperature τ = 0 . 2 as this value is often reported the best choice in papers on CL [43, 40]. An exception is that we let τ = 0 . 15 for XSimGCL on Yelp2018, which brings a slightly better performance. Note that although the paper of SGL [43] uses Yelp2018 and Alibaba-iFashion as well, we cannot reproduce their results on Alibaba-iFashion with their given hyperparameters under the same experimental setting. So we re-search the hyperparameters of SGL and choose to present our results on this dataset in Table 3.

A.6 Dataset Statistics

A.7 Effective Rank

Definition 2 (Effective Rank.) Consider matrix X ∈ R m × n whose singular value decomposition is given by X = U Σ V T , where Σ is a diagonal matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ Q ≥ 0 with Q = min { m,n } . The distribution of singular values is defined as the ℓ 1 -norm normalized form p i = σ i / ∑ Q k =1 | σ k | . The effective rank of the matrix X , denoted as erank ( X ) , is defined as erank( X ) = exp ( H ( p 1 , p 2 , · · · , p Q )) , where H ( p 1 , p 2 , · · · , p Q ) is the Shannon entropy given by H ( p 1 , p 2 , · · · , p Q ) = -∑ Q k =1 p k log p k .

A.8 Comparison of Runtimes with the Riemannian Metric

The main advantage of D ϕ ld over the Riemannian metric D R (such as AIRM [10] and LERM [10]) is its computational speed. To compute D ϕ ld , only determinants need to be computed, which can be efficiently achieved with Cholesky factorization (for Σ U Σ I ) at a cost of 1 3 d 3 flops [12]. On the other hand, computing the Riemannian metric requires generalized eigenvalues, which takes around 4 d 3 flops for positive definite matrices. Therefore, in general, D ϕ ld is much faster to compute (see Table 7b). This speed advantage becomes even more pronounced when computing gradients. Moreover, backpropagation through the matrix determinant is generally stable whereas generalized eigenvalue decomposition suffers undetermined gradients if two eigenvalues are non-simple (equal values). As shown in Table 7a, computing ∂D ϕ ld can be over 100 times faster than ∂Dϕ R . This difference can be crucial when using gradient-based algorithms, such as neural networks, that rely on the computation of similarity measure gradients.

(b) Average times to compute function values.

A.9 Performance Comparison w.r.t. to Different Distance Types

In this section, we adopt other types of distances as in Equation (11) and compare them with proposed method. There are two different categories (1) the matrix norm, i.e ., Euclidean Norm, Nuclear Norm, and Forbenius Norm, and (2) the Bregman divergence, i.e ., von Neumann divergence (also known as Matrix Kullback-Leibler divergence) and Logdet divergence. The Forbenius norm can belong to both matrix norm and Bregman Matrix divergence. We show their formulas and experimental result in Table 8. We notice that the proposed method (Logdet Div.) achieves the best results.

A.10 Broader impact and limitations

Our method enjoys impact and limitations similar to those in graph collaborative filtering. Typical GCFmodels cannot guarantee they can utilize the feature space efficiently. The mode collapse can lead to frequent item bias resulting in a model which is biased in its recommendations. Thus, by improving recommendation of rare items we also offer a generic approach to limiting the recommendation bias which is important in many domains of life. Our method requires no special resources and just a fraction of additional computations beyond what GCF uses. Of course, our model is limited by the GCF model itself and it is applicable only to pipelines that suffer the mode collapse.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the issue of popular bias in Graph Collaborative Filtering (GCF) and demonstrates its resulting Dimensionality Collapse (DC) in the embedding space. The paper points out that LightGCN, widely used in GCF, is prone to DC. The authors introduce a decorrelated penalty based on LogDet divergence to address this problem. Through theoretical analysis and empirical evidence, they demonstrate that GCF LogDet can be extended to full-rank embedding matrices while maintaining a small condition number. On several benchmark datasets, GCF LogDet outperforms other contrastive-based GCF models and improves the recommendation effectiveness for unpopular items.\n\nSTRENGTHS:\nThis paper explores the limitations of graph collaborative filtering algorithms in the field of recommendation systems and proposes a novel graph convolutional neural network, GCF-LogDet, which effectively addresses the issues of dimensionality collapse and popularity bias in collaborative filtering. Specifically, the paper demonstrates, from the perspective of the representation space, the problem of dimensionality collapse caused by graph convolution and formulates high-quality self-supervised embeddings in recommendation systems from the perspective of reducing redundancy. Moreover, by using LogDet distance instead of the traditional SoftPlus, the paper successfully avoids the issue of drastic variance changes, providing thorough analysis and discussion, and achieving improved experimental results. Additionally, the paper exhibits high quality and clarity by providing detailed descriptions of the methodology and experimental setup, and it provides a detailed proof of the theorem in the supplementary materials.\n\nWEAKNESSES:\n1. In the paper, the title emphasizes alleviating popularity bias, while the analysis and emphasis primarily focus on the issue of dimensionality collapse. However, the direct correlation and equivalence between dimensionality collapse and popularity bias are not explicitly explained or established in the paper. To provide a clearer understanding of the relationship between dimensionality collapse and popularity bias, it would be beneficial for the authors to explicitly discuss and clarify the connection between these two issues in the introduction or related work section. Additionally, the authors should elaborate on how the proposed method specifically addresses the problem of popularity bias, providing a theoretical or empirical explanation of how mitigating dimensionality collapse can effectively alleviate popularity bias in the context of recommendation systems.\n2. This paper seems to lack a comprehensive comparison with existing methods that specifically address popularity bias in the context of graph collaborative filtering. As far as I know, [1] method introduces r-regularization based graph aggregation plugin to alleviate the popularity bias during neighbor aggregation of graph convolution. I recommend that the authors revise the experimental section to include broader comparisons, including existing methods that directly address popularity bias (e.g. [1-2]).\n[1] Zhao M, Wu L, Liang Y, et al. Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022: 50-59.\n[2] Wei T, Feng F, Chen J, et al. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1791-1800.\n3. I suggest that the authors revise the related work and comparative methods sections to include a comprehensive comparison with existing approaches that tackle dimensionality collapse. Specifically, they should discuss and compare methods such as [3-4] that have been proposed to alleviate dimensionality collapse. For example, [3] analyzes the causes of feature collapse from the perspective of graph contrastive learning and proposes a spectral data augmentation method, which is highly relevant to the work presented in this paper.\n[3] Zhang Y, Zhu H, Song Z, et al. Spectral Feature Augmentation for Graph Contrastive Learning and Beyond[J]. arXiv e-prints, 2022: arXiv: 2212.01026.(Accpet by AAAI23)\n[4] Chen X, He K. Exploring simple siamese representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 15750-15758.\n\nQUESTIONS:\n1. According to the proof of the paper, there should be a problem of feature collapse in the graph convolutional neural network. Why did the author choose the recommended scene for proof instead of more general graph data?\n2. What is the connection between dimensionality collapse and popularity bias, and can they be equated?\n3. Why are SimGCL and DirectAU not added to the comparison method in Figure 2 and Figure 4? Obviously, the author's loss function is the closest to DirectAU.\n\nLIMITATIONS:\n1. The authors did not specify the relationship between popularity bias and dimension collapse in detail, which makes readers question the rationality of the author's method.\n2. The paper does not compare with the method of alleviating the popularity bias, and it does not reflect the superiority of the method proposed by the author in alleviating the popularity bias problem.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The paper proposes a novel graph collaborative filtering (CF) approach which tackles the issue of item popularity bias. Starting from the (theoretically-proven) assumption that state-of-the-art graph CF architectures (such as LightGCN) may enhance the so-called dimensional collapse (DC) problem, the authors decide to leverage decorrelation of the learned embeddings through the LogDet divergence. Extensive experiments on three popular recommendation datasets, against multiple state-of-the-art (graph)-based recommendation approaches demonstrate the efficacy of the proposed solution. Additionally, the authors show that the model is able to outperform other graph CF contrastive solutions which optimize uniformity. Finally, a quantitative analysis proves how the approach can tackle the dimensional collapse issue and address the item popularity bias.

STRENGTHS:
+ The proposed idea is sound and well-presented.
+ The solution is well-placed in the related literature.
+ The experiments are extensive with multiple analyzed dimensions.
+ Codes and datasets are shared.

WEAKNESSES:
To the best of my knowledge, I cannot see any particular issues.

QUESTIONS:
To the best of my knowledge, I cannot think about any specific questions.

LIMITATIONS:
To the best of my knowledge, I cannot see any particular limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
In this paper, the authors study the dimension collapse problem in graph-based recommenders. The authors provide both sufficient experimental and theoretical analysis on the occurrence of dimension collapse to popular items and propose a new loss function to encourage feature diversity. Experiments are also sufficient to prove the advantages of the proposed methods. In summary, I think it is a good paper with both sound experiments and theoretical analysis. I suggest a strong acceptance.

STRENGTHS:
1. The paper provides both theoretical and experimental analysis of the dimensions collapse problem in graph collaborative filtering

2. The proposed method is simple and effective, boosting the performance of direct representation learning for collaborative filtering learning.

3. Experiments are solid, and also sufficient to validate all the advantages of the proposed method.

WEAKNESSES:
I do not think this paper has obvious weaknesses to cope with

QUESTIONS:
Do authors think the dimension collapse problem only existed in graph-based collaborative filtering? How about matrix factorization?

LIMITATIONS:
I do not think this paper has obvious limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies popularity bias in graph collaboration filtering through dimensional collapse analysis of the embeddings. The authors proposed the LogDet divergence based de-correlation penalty for GCF to improve the performance.\n\nSTRENGTHS:\nThis work has thorough analysis of dimensional collapse in GCF and its relationship towards propagation and uniformity. De-correlation enhanced objective for mitigate dimensional collapse in GCF. Proposed LogDet divergence to preserve the range4 space of the covariance matrix.\n\nWEAKNESSES:\n* While popularity bias adversely impacts the recommendation quality, the quality of the items may also play a role from the end to end recommendation quality. The paper shows the solid improvement on Recall; considering that nDCG@K does not penalize for missing items, it is a bit hard to tell the overall performance. \n\n* The model names seem inconsistent it the experimental study section, such as that in Figure 4 and Tables 2 & 3, which causes some confusion when reading the sections.\n\n* Some details of the hyperparameter setting on the validation dataset should be presented if it is not trivial for the experimental results. \n\n* While it is good to see the proposed method improves the ability to recommend unpolular items, which potentially helps the diversity, the baseline selection is a bit unclear. According to the numbers in Table 3, the baselines in Figure 4 are not the most compelling ones.\n\nQUESTIONS:\n* A minor suggestion on Figure 3, where the last subfig is used for showing the change of condition number over training steps of just one of the three datasets used in the other subfigs. Why not to plot all for completeness, if really the same trends of condition numbers were observed for all? \n\n* How is the impact of K in NDCG@K in the experiment? Is it selective in the experiment, which might imply the missing items in top-K?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper aims to study the problem that features tend to occupy the embedding space inefficiently. The authors proposes a decorrelation-enhanced GCF objective that promotes feature diversity. LogDet divergence is leveraged to form a new constraint loss.

STRENGTHS:
1.  The studied problem of embedding space degradation in graph-based collaborative filtering is interesting and important.
2.  LogDet based decorrelation penalty is developed, and its ability of maintaining the full-rank constraints is theoretically demonstrated.
3.  Experiments show improvement over some baselines.

WEAKNESSES:
1.  The dimensional collapse issue (also known as overcorrelation) has been studied in [1] but not cited.
> Jin W., Liu X., Ma Y., Aggarwal C. and Tang J. Feature over correlation in deep graph neural networks: a new perspective. KDD, 2022.
2. This issue occurs when stacking too many layers whether or not there is a popularity bias.
3. I think 'logdet' regularisation works so well because it encourages embeddings to become rich in different dimensions, rather than mitigating popularity bias.
Note that the relative proportions in the three scenarios in Figure 4 are very close.

QUESTIONS:
1. The title mentioned "Mitigating the Popularity Bias". However, it seems not to be the main aim of this paper.
2. Some details are missing. How to implement f_{\theta}.
3. Typos. E.g., migrate -> mitigate

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28