Mitigating the Popularity Bias of  Graph Collaborative Filtering: A Dimensional Collapse Perspective
Authors: Yifei Zhang
Hao Zhu
Yankai Chen
Zixing Song
Piotr Koniusz
Irwin King
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Graph
Collaborative Filtering
Recommendation

Abstract: Graph-based Collaborative Filtering (GCF) is widely used in personalized recommendation systems. However, GCF suffers from a fundamental problem where features tend to occupy the embedding space inefficiently (by spanning only a low-dimensional subspace). Such an effect is characterized in GCF by the embedding space being dominated by a few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popularity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches  (i.e., optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelation-enhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps  maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our  method  outperforms contrastive-based GCF models on several benchmark datasets and improves the performance for unpopular items.

Mitigating the Popularity Bias of Graph Collaborative Filtering: A Dimensional Collapse Perspective
Yifei Zhang', Hao Zhu Yankai Chen', Zixing Song", Piotr Koniusz *8.1, Irwin King*
The Chinese University of Hong Kong
Australian National University, 1
fyfzhang, ykchen, zxsong, king ) @cse. cuhk.edu.ht allenhaozhu @ gmail.com, koniusze @ data61 cciroo.cs
Abstract
Graph Collaborative Filtering (GCF) is widely used in personalized recommen- dation systems. However, GCF suffers from a fundamental problem where fea- tures tend to occupy the embedding space inefficiently (by spanning only low- dimensional subspace). Such an effect is characterized GCF by the emhedding space being dominated by few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popu- larity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches (i.e., optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelation- enhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our method outperforms contrastive-bast GCF models on several benchmark datasets and improves the performance for unpopular items.
Introduction
Collaborative Filtering (CF) has emerged as effective technique for personalized recommenda- tions [9, 6, 44, 8, 7]. By leveraging embedding and optimizing with a suitable loss function, CF effectively capture complex patterns in uscr-item interactions [32, 16]. However, user-item interactions usually neglect the potential of the higher-hop connections. To this end, Graph Neural Networks (GNN) [36, 20, 14, 34, 50, 26] have been used to model high-hop connections in CF. Graph Collaborative Filtering (GCF) has also been proposed based on CF [15, 41].
Graph Contrastive Learning (GCL), powerful tool for unsupervised learning, has recently been applied on variety of different tasks [5, 47, 30, 49, 50]. Many methods also demonstrate that GCL improves the performance of GCF The basic idea of GCL in GCF is to enhance training data with various graph augmentations and maximize the alignment between different data views [46, 43, 53]. However, the above methods ignore the disadvantages of GCL (e.g., dimensional collapse) and the negative impact on user and item embeddings (e.g., the Matthew effect) [4]. Dimensional Collapse (DC) [28] can be characterized by high similarity of features which confined to low- -iwdiimeniooal subspace or are contrated on curtain region of latent space( Figure 1b). DC limits the representational power of nigh-dimensional spaces by restricting the diversity of information that can be learned [28].
"The corresponding author. Code: httpps://githu...coo//ffeeeeeeee http:/ghww..eel.eeeeee////geeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Although DC has been identified and tackled in a variety of domains [19], DC poses issues for GCL in the Graph Collaborative Filtering where it exacerbates the Matthew effect of popularity bias (as explained next).
significant issue that sets GCL in GCF apart from other domains is the highly skewed data distribution over the graph where most users interact sporadically with a very small set of items within vast interaction space comprising millions or even billions of items [4]. This results in extremely sparse user-item interactions and a long-tail item distribution following the Power Law. Propagating highly skewed data through the sparse bipartite graph via message passing enhances the Matthew effect of the popularity bias [4]. This means that several popular items may dominate the entire learning process, while other items are rarely recommended. Thus, collapse mode may occur, where the user embedding collapses around several popular items, as shown in Figure la.
To migrate this issue, previous research argues that the so-called uniformity term included in the auxiliary contrastive loss can solve DC and so the uniformity term was adopted it in CF [45, 46, 38]. Particularly, Yu et al. [46] provide an empirical evidence that the fea- tures learned by the typical GCF model are non-uniformly distributed on the sphere, which causes the collapse. Wang er al. [38] have fur- ther investigated how optimizing the Bayesian Personalized Ranking (BPR) loss in CF can promote both the alignment and uniformity, and suggested to directly optimize them. However, while optimizing the uniformity of both the user and item embeddings in CF can refine the bedding space, does not guarantee the elimination of dimensional collapse. Thus, fundamental question arises: is there any provable remedy for DC?
Dar 1ack
-10
(b) Dimensional collapse
(a) User Collapse in GCF
in 3D hypersphere.
Figure
To this end, we analyze the connection between dimensional collapse and GCF by demonstrating that the simplified graph convolution operation for collaborative filtering does shrink the singular space of the feature matrix. Although previous methods promote the feature uniformity, does not prevent the dimensional collapse. Thus, we propose new objective, Decorrelation-Ein -iirelaatioo-Ennneeedd GCF, that employs geometry of the LogDet divergence to relax orthogonality constraints and encourage the feature diversity. Our method maintains the range space of the feature matrix with small condition number well-coonitionee)) providing theoretical guarantees against dimensional collapse. Our experimental results demonstrate the effectiveness of the proposed approach in addressing feature collapse. We outperform contrastive-bant CF models on several benchmark datasets. Notably, our approach improves the performance for unpopular items.
We summarize our contributions below:
i. We demonstrate that embeddings learned from existing GCF models suffer from dimensional collapse. Our theoretical analysis reveals that the dimensional collapse GCF arises from propagation, and that the current uniformity-baset objective does not guarantee alleviate it.
ii To mitigate the issue of dimensional collapse, we propose decorrelation-enhance, objective following the principle of redundancy reduction [47] and LogDet geometry.
iii. Instead of using the Frobenius norm (the Euclidean geometry) relax the conditional optimiza- tion problem, we propose using the LogDet divergence, a Bregman divergence with theoretical guarantees, preserve the range space of the covariance matrix.
Our experiments show that our proposed method effectively mitigates dimensional collapse and outperforms various contrastive learning-based CF models. Notably, it achieves improved the performance for unpopular items, thus mitigates the Matthew effect.
2 Preliminaries
Collaborative Filtering. Let and I denote the user and item set, respectively. Given set of observed user-item interactions = {(u, i) u interacted with i), CF methods infer the score s(u, i) E R for each unobserved user-item pair indicating how likely the user is to interact with

---

the item i. Then, items with the highest scores for each user will be recommended based on the predictions [32, 16]. Most CF methods use an encoder network f(-) that maps each user and item into a low-dimensional representation f(u), f(i) E IR (where d is the dimensionality of the latent space). For example, the encoder in matrix factorization models is usually an embedding table, which directly maps each user and item a latent vector based on their IDs. The encoder in graph- models uses the neighborhood information. The predicted score is defined as the similarity between the user/item representations, e.g., . dot product, s(u, i) f(u) f(). Most studies adopt the pairwise BPR [3] ]] loss to train the model:
1
Capr
log sigmoid (s(u, i) (u, :))); (M,))ER
R
where i- is randomly sampled negative item that the user has not interacted with. Such loss maximizes sigmoid scores: target user-item scores become higher than user-negative item scores.
LightGCN. A linear graph neural network backbone, LightGCN [15], is one of the most successful backbones in Graph Collaborative Filtering. It leverages neighborhood information at different levels of locality, and combines outputs of different layers by the sum aggregation.
To facilitate the analysis, we present the matrix from of the LightGCN. Let R E R MMN be the matrix representation of set R where Rusi 1 if (u, R (else Ryi 0). Let M and N be the numbers of users and items, and N M N. We then obtain the adjacency matrix of the user-item graph as:
0 R
R
DRNNNN.
A
22)
Let the embedding matrix of input layer be E R.'' where d is the embedding size. Then the K-th layer of LightGCN in the matrix form is defined as:
K
= SEE; A. K) = - AE A²A + aK A K E
aA
A
(3)
k=1
where 01,, ...,..., Ogg are weights, and Σ k= ag = 1. Moreover, À = D- is a normalized adjacency matrix and the degree matrix D is an N' N' diagonal matrix in which each entry Djs denotes the number of non-zero entries in the i-th row vector of the adjacency matrix A.
Dimensional Collapse (DC). Often referred to as spectral collapse [28], dimensional collapse is a prevalent phenomenon in representation learning. It occurs when the embedding space is dominated by a small number of large singular values, while the remaining singular values decay significantly the training progresses. This phenomenon limits the representation power of high-dimensional spaces by restricting the diversity of information that can be learned.
3 Popularity Bias from Perspective of Dimensional Collapse
3.1 Popularity Bias vs. Dimensional Collapse in Graph Collaborative Filtering
Dimensionality collapse and Popularity Bias in CF are two sides of the same coin. The prevalence of popularity bias in GCF can be attributed the significant imbalance within the data distribution across the recommendation bipartite graph [4]. This distribution often features numerous users engaging with a small subset of items, within a much larger interaction space encompassing millions billions of items. Consequently, during the process of message passing within GCF, few popular items can disproportionately influence learning, leaving out other items. This leads to the dimensional collapse in the embedding space where most user embeddings are around very few popular item embeddings.
We illustrate this by showing the 2D latent embedding space of the user and item in Figure 2:
Figure (top) shows that both user and item embedding are clustered a few regions, the gaps in the unpopular item embedding cause effective rank drop. We refer to this effect as the dimensional collapse. As the unpopular items are clustered in just one cluster, and the corresponding user cluster is non-dense unpopular items do not get recommended often (popularity bias).
3

---

user embedding
embedding
popular emb.
unpopulaar
this unpopular item cluster has no user cluster all
CF with LightGCN
X
dense clusters align well with dense popular-item clusters
non-dense cluster aligns with the unpopular-item cluster
LogDet
user embedding
Item embedding
popular item emb.
unpopular item emb.
Mitigating popularity blas
gon spress an hyppes aammer Seceersiie
CF with LightGCN +LogDet
half-circle has spread hyperspere and formed clusters
alleviated dimensional collapse
dense popular-item clusters have not changed much match dense user clusters
two new unpopular-ites clusters have aligned dense user clusters
Figure 2: Mitigating Popualrity Bias. (top) We applied 2D PCA and the & normalization embeddings of LightGCN (Yelp2018 dataset). Notice that when only a non-dense user cluster aligns with the unpopular item cluster, these items will not be recommended frequently (X). (bottom) With our LogDettbassdd loss, the unpopular-iten cluster is spread into three clusters across surface of hypersphere. Newly formed clusters align well with dense clusters (V) the unpopular items will be recommended more often. The dimensional Collapse manifests itself with gaps on the surface of hypersphere. Each plot was renormalized by its maximum density (darkest color). The 3D ball effect is to remind reader the embeddings exist on the surface hypersphere but in our simulation we use in fact 2D sphere.
ii If only the embeddings were spread uniformly on the surface the hypersphere, would align with dense user clusters which would reduce the popularity bias. Figure 2 (bottom) shows that thanks to more uniformly spread unpopular items, they form now clusters, and of them align well with dense user clusters, which means the popularity bias has been reduced.
3.2 Dimensional Collapse Graph Collaborative Filtering
The previous section reveals that the occurrence of dimensional collapse closely related the presence of popularity bias, as depicted in Figure 2. Below, we show how the typical backbone of GCL in GCF (i.e., LightGCN) potentially causes the DC in the learned embedding space, and why optimizing uniformity [38] (the best current solution) cannot guarantee to prevent DC.
Lemma (Equivalent reformulation of LightGCN.) The propagation process of LighTGCN Equation (3) can be viewed as result of optimization of the following objective:
z" arg min z
Z A +
LZ),
(4)
where L I the normalized symmetrie positive semi-definite graph Laplacian matrix. O Proof of Lemma 4 is in Appendix A.
Lemma shows that the propagation process of LightGCN can be derived from the optimization formulation which uses the Graph Laplacian regularization. This implies that output embedding
Z is smooth due to the Graph Laplacian regularization in Equation (4). Similar derivations for the graph neural network with learnable weights were shown by Ma et al. [29] and Zhu et al. [58].
To understand the dynamics of the embedding space, we denote the state of the embedding matrix and the output the LightGCN time E(t) and the initial state E(0) is randomly initialized. n= denote eigenvalues of (descending order)
Let {o E
Lemma (Shrinking singular space of feature matrix.) Consider minimizing the objective in Lemma I with the Laplacian regularization term tr (Z LZ). The relative value of the ratio

---

of the smaller eigenvalues to the larger eigenvalues will decrease as t increases. Formally, (t) =fitt)
(1) I ofie) < > j (we ignore the trivial case j). Then:
lim a1) 0, Vi > such that of < g. t-000 (()
(5)
Proof of Lemma 2 is in Appendix A.2.
Based on Lemmas I and 2, conclude that due to the propagation step in Equation (3), the embedding space of feature matrix dominated by few largest eigenvalues. This is a poof of dimensional collapse of GCF pipelines which are typically based on the LightGCN backbone.
3.3 Optimizing Uniformity in Graph Collaborative Filtering.
Typical contrastive GCF approaches adopt the BPR loss from Equation (1) and the InfoNEC loss. Wang et al. [38] argued that the so-called alignment and uniformity [38] loss terms contribute to higher recommendation performance:
Lolign B(a,i)< [ll (u)
(alignment)
(6)
funi = - log Ruu e22- (v) fo(w so(w log Ee, (2--2o01)) 25o(1) fo()
(uniformity)
where Latgin captures the user-item relations. Luni encourages embeddings to be distributed evenly on the surface lg ball. However, Luni does not avoid the dimensional collapse shown in Figure 1.
Concluding, Contrastive GCF suffers from several problems:
DC [17] is characterized by one or more singular values in the feature matrix to zero, e.g.. A; 0. Luni may yield finite reward (decrease of Luni) avoid the dimensional collapse (); = 0). However, if the remaining loss terms yield a higher reward (decrease their value)
than Luni for occurrence of the dimensional collapse (A₃ 0) then DC will occur.
ii As temperature 1/T increases, Luns degrades to hard minimum. Take just the user part of Equation (6), include the temperature variable 1/T, and notice that:
lim -T log
Eu,u ~u (22 2fo(v) fo(u")) W//
min (u.-))uuxe
2 f fo(u
(7)
which explains the empirical observation of Wang and Liu [39] that the uniformity of embeddings worsens as the temperature increases.
iii. As Lemma shows, LightGCN itself is also prone to suffering from the dimensional collapse.
Thus, in what follows we consider the redundancy reduction principle mitigate the above issues.
4 Methodology
Graph Collaborative Filtering with Decorrelation. From the perspective of redundancy reduc- tion [47, 2], achieving
self-supervised embeddings requires that: (i) positive pairs exhibit similar semantics; (ii) the embeddings follow non-trivial constant distribution, with trivial distri- bution indicating that all embeddings collapse into a single point; and (iii) there is zero correlation between different features.
In the context of collaborative filtering, we can frame this problem as follows:
Table 1: Several decorrelation terms, properties and teaser results (Yelp2018)
min Latign(()
(8)
Lunt
Lufe
Lispte
Stiefel
isorrpicc
soft
soft
soft
hard
s.t. Σ (o(w))o(uTT = [o) =I,
full-rank
soft
soft
hard
recall@:0 20 (%)
7.12
7.09
7.25
7.03
iEW
iEI
where Latign (altign()== (0)
||| (u) fo (i) 12 (from Equation (6)) captures user-item preferences. Constraints Eigu So fo(u) = Eiez fo(i) I help embeddings follow the isotropic Normal distribution. The isotropy prevents the dimensional collapse but its decorrelating effect may
5

---

Isotropic Full Rank
Isotropic Rank-deficient
Non-isotropic & Full Rank
Non-isotropic Rank-deficient
Eigenvalue Index
Eigenvalue Index
Eigenvalue Index
Eigervalue Index
Figure 3: The spectrum (eigenvalue distribution) of the covariance matrix under various loss penalties. Various penalties realize some composition of {isotropic, non-isotropic)> rank, rank-defficientt. e.g., our LogDet formulation promotes the (isotropic, full rank) case.
be strong. Equation (8) may be solved by applying Lagrangian multipliers the Stiefel manifold (Geo'Torch [25]) or by converting hard constraints into soft constraints:
min Calign (0) (0) +---Loofft where Csoft= Σ folw)fo(u)
Σ fe(i)fo(i)
However, relaxation Leaft ( in Equation (9) suffers from the same problem Cans in Equation (6). Specifically, these both regularization terms yield finite penalty for any singular values X 0 leading to dimensional collapse if other loss terms yield a higher reward than the penalty is. Thus, we explore the application of Bregman divergence. Our study indicates that the proposed approach preserves the range space of the covariance matrix, promoting a full-rank embedding matrix.
Decorrelation by the LogDet Divergence. Figure 3 and Table summarizes properties of different decorrelation terms. While Luni and Lzoft encourage soft decorrelation (approximate isotropy), they may fail to impose full rank on the embedding matrix. Stiefel-based optimization enjoys hard constraints but full decorrelation (exact isotropy) may too restrictive. In contrast, LogDet divergence based Llogdet encourages isotropy in the soft manner and imposes full rank on embedding matrix as shown below. Firstly, we review the Bregman divergence and introduce our LogDet based decorrelation penalty.
Definition (Bregman Divergence on Matrices.) Let X, E St be two symmetric matrices. Let 6:5" R. be strictly convex and differentiable matrix function. The Bregman matrix divergence is defined as:
Y
Dx Y) = ((X) +(Y) - ((V&(Y)) (X- ()))
(10)
The Bregman divergence a measure of the nearness between matrices X and Y. computed by using the first-order Taylor approximation of a convex generating function Φ(-). The Bregman divergence is always non non-negative, and becomes zero only when the two matrices are equal.
As En = Eneu (u T E S and Σz === f())) E S are symmetric, and at least positive semi-definin (PSD) matrices but desire them be strictly positive definite (PD) matrices ($#+) prevent the dimensional collapse, we firstly generalize Lzoft in Equation (9) to its Bregman divergence based variant:
Lareg D& (Eu + D DØ I).
(11)
Different d(.) lead to different measures of nearness. For example, Φr(X)= |X 13 yields the Frobenius-based D& (Eu, I) + Dar Ez 1)= En 1| I penalty Looft Equation (9).
Aa)). Let = T an eigenvalue decomposition of X, and A diag( (A,, Choosing Φia( (X) log det X E, log >; leads to the LogDet divergence Dera () t (XY-I)-
log det (XY-1) - d where X, Sd Then generalizing Leofe in Equation (9) to its LogDet
divergence variant by computing Dere (Ee+ D (Σz, I) and removing -2d constant yields:
Ltugdet tr Eu Ex) log det
(12)
Our LogDet decormelationn-enhancee GCF objective becomes:
((,))~ [Il (u) (i) [ + X- tr (Eu Σr) log det (E4Er))
(13)
In contrast the Frobenius-based Lanft, the LogDet-based penalty Lingdet maintains the range space of the covariance matrices and upper bounds the condition number of the matrices. This ensures the user/item embedding matrices are of full rank which prevents the dimensional collapse.

---

Theoretical Analysis.
To use the LogDet divergence based penalty Llogdet in GCF, we discuss important to our work theoretical properties of the LogDet divergence and Llogdet for rank-deficient matrices.
LogDet Divergence and Llogder for Rank-deficies Matrices.
Corollary 1. LogDet divergence Dora (X, Y) is finite iff rank(X) rank (Y) [23], which implies that Llogdet = Dors (Eu, 1)+ Deia Ez. 1) so if (ank(Eu) < rank(I) = d V rank( < rInk(I)= d.
Corollary 2. Let Eu, Ei S44. Note that Csaft = Der (Eu, 1) + Der (Er, 1) is finite if both [Eullz and EI 2|2 finite. There exist cases where some infinitesimal change AEu and/or AE will make Su+AEu and/or EI+AEE positive semi-definite (PSD). Define the gain of the Bregman divergence based decorrelation penalty go(Creg) D& Eu, I) Do (Do + AEu, Do+Do(E)) + AET,I)) Notice, in case of change from PD to SPD matrix, gain golCreg O. Let the gain of the alignment loss be gf (atig))) &. co. Then one may easily consiruct examples where gi ((aatign) - - (Lrsoft) > O. That is to say, the net loss - "6h g in case of covariance degradation to SPD lesser than the alignment. In contrast, from Corollary 1. g(Lutig)) - -dua (Clogdet) -00 in case of covariance degradation to SPD, which prevents the dimensional collapse.
gain g g(ating)) due to increased user-item
Proof of Corollary 2 is in Appendix A.3.
Corollary 3. Expression - log det EI) from Equation (12) might fail to yield oo if E'E Ex is indefinite. However, covariance matrices are at least SPD by definition, and is their matrix product.
Corollaries 1, and 3 demonstrate that Llogdet naturally maintains the full-rank constraints on covariance matrices of user and item embeddings.
Minimizing Deta Σ, Reduces the Condition Number of E. Feature decorrelation is closely related to the condition number. If the spectrum of a matrix is dominated by the leading eigenvalue, the matrix is ill cllcconit and has a large condition number. Let {>r] be the eigenvalues of the matrix Σ. The condition number is defined as cond = Amax
The ratio between the largest eigenvalue and the smallest eigenvalue quantifies the flatness (isotropy) of the spectrum. For example, cond(22 Σ) 1 implies fully balanced spectrum with X; X, for all i # j (e.g., spectrum balancing [21, 22, 48, 52]). Below we show that our proposed method minimizes the upper bound of cond (E).
Amii
Lemma 3. The condition number of E, cond(E) Amaa is upper bounded by D Drua (E,1) as:
Arain
cond(E) S 4exp (Dous (E,
(14)
Proof of Lemma 3 is in Appendix A.4.
Lemma implies that the sum of condition numbers of En and Σz is upper bounded by the LogDet divergence based penalty Liogdet. Minimizing that penalty implies the flattening of spectrum implies decorrelation/reduudaae reduction of embedding.
Experiments
Datasets. To evaluate the effectiveness of our GCF with the LogDet divergence based penalty Loogdet, denoted GCFiogdee conduct experiments on three large-scale public datasets: Yelp2018 8 [15], Amazon-kindle [43], and Alibaba-iFashion [43]. See statistics of these datasets in Appendix A.6. The datasets were split into the training, validation, and test set, with the ratio of 7: 1::22 2. As per the methodology suggested by [43, 15], we first identified the best hyperparameters and then merged the training and validation the test set was used to evaluate the model using the relevancy- metric Recall @ 20 and the ranking-aware metric NDCG@20.
on the validation
train the model. Finally,
Apart from the typical GCF models, i.e., NGCF [41] and LightGCN [15], we also compare GCFlogdet with recent data CL-based recommendation models. SGL [43] and DNN+SSL. [45] adopt CL auxiliary task and conduct feature masking for contrastive learning (CL). BUIR
Baselines.
7

---

Table Performance comparison for different contrastive GCF model that optimize uniformity.
Yelp2018
Amazon-Kindle
Alibaba-IFashion
Methods
@ecall 20
NDCG@20
2ecall@20
NDCG@20
Recall
NDCG@ 20
LightGCN
0.0590
0.0484
0.1871
0.1186
0.0845
0.0390
SGL-ED
0.0637
0.0526
0.1936
0.1231
0.0932
0.0442
layer
SimGCL
0.0689
0.0572
0.2087
0.1361
0.1036
0.0505
DirectAU
0.0674
0.0522
0.1971
0.1239
0.0969
0.0426
GCT togdet
0.0694
0.0577
0.2103
0.1321
0.1060
0.0510
LightGCN
0.0622
0.0504
0.2033
0.1284
0,1053
0.0505
SGL-ED
0.0668
0.0549
0.2084
0.1341
0.1062
0.0514
layers
SimGCL
0.0719
0.0601
0.2071
0.1341
0.1119
0.0548
DirectAU
0.0702
0.0584
0,2034
0.1352
0.1043
0.0549
GCFlogdet
0.0720
0.0607
0.2143
0.1381
0.1180
0.0557
LightGCN
0.0639
0.0525
0.2057
0.1315
0.0955
0.0461
SGL-ED
0.0675
0.0555
0.2090
0.1352
0,1093
0.0531
layers
SimGCL
0.0721
0.0601
0.2104
0.1374
0.1151
0.0567
DirectAU
0.0713
0.0602
0.2047
0.1385
0.1136
0.0586
GCging
0.0725
0.0612
0.2151
0.1418
0.1210
0,0601
Table Performance comparison with other models.
Yelp2018
Amazon-Kindle
Alibaba-iFashion
Method
Recall@ 20
NDCG@20 20
Recall@20
NDCG @ 20
Recall
NDCG@D
NGCF
0.0579
0.0477
0.1954
0.1263
0.1043
0.0486
LightGCN
0.0639
0.0525
0.2057
0.1315
0.1053
0.0505
NCI.
0.0670
0.0562
0.2090
0.1348
0.1088
0.0528
BUTR
0.0487
0.0404
0.0922
0.0528
0.0830
0.0384
DNN+SSL
0.0483
0,0382
0.1520
0.0989
0.0818
0.0375
MixGCF
0.0713
0.0589
0.2098
0.1355
0.1085
0.0520
SGL-ED
0.0675
0.0555
0,2090
0.1352
0.1093
0.0531
SimGCL
0.0721
0.0601
0,2104
0.1374
0.1151
0.0567
DirectAU
0.0713
0.0602
0,2047
0.1385
0.1136
0.0586
GCF logdet
0.0725
0,0612
0.2151
0.1418
0.1210
0.0601
[24] has two-branch architecture with target network and online network, utilizing only positive examples for self-supervised recommendation MixGCL [18] introduces the hop mixing technique for synthesizing hard negatives for GCF through embedding interpolation. NCL [27] is recent contrastive model that designs a prototypical contrastive objective to capture the correlations between user/item and its context. SimGCL [46] is a CL-based model for recommendation that does rely on graph augmentation. DirectAU [38] proposes optimizing the alignment and uniformity of both user and item directly for recommendation.
Implementation. We referred to the best hyperparameter settings reported in the original papers of the baselines and then fine-tunar tine-tuned them with the grid search. Detailed settings are in Appendix A.5.
6.1 Main Results
Decorrelation vs. Uniformity. Within this section, compare the performance of our proposed method, GCFlogdet, with contrastive-bee methods (i.e., SGL, DirectAU) that optimize uniformity to prevent the dimensional collapse CF. Furthermore, we demonstrate the effectiveness of our method by varying the number of encoder layers. Table 3 shows the performance of different baseline methods and our GoFiegdet- DirectAU, SimGCL) significantly outperform LightGCN. The largest improvements are observed on Alibaba-iFashion, where the performance of GCFiogdet surpasses that of LightGCN under and 3 layers settings. This observation supports our conclusion that LightGCN is susceptible to the dimensional collapse, leading to poor results. Additionally, we observe that DirectAU outperforms SGL, indicating that graph augmentation is not the primary factor for enhancing the performance of the CL-based recommendation recommendellas model. Instead, our findings demonstrate the importance of designing appropriate loss functions
the vast majority cases, contrastive-
methods (e.g.. SGL,
sophisticated augmentation. Finally, GCFlogdet achieves the best results on the three datasets. Specifically, GCFlogdet outperforms DirectAU and SGL, highlighting that decorrelation is more effective than optimizing uniformity for preventing dimensional collapse.

---

10-
Cogast
Miney
10
14
Enydr:
Conditinn
Eust
100. 2537
4005 6000
Epocts
Epocns
(a) Yclp2018
(b) Kindle
(c) iFashion
(d) Yclp2018
Figure The effective rank (higher is better) w.r.t. epoch number and condition number in Yelp2018.
Table 4: Performance comparison with other models.
Method
Yclp2018
Amazon-Kindl Kindle
iFashion
Encoder
Loss
Recall@20
NDCG@21
Recall@20
NDCG@2
Recall
NDCG@20
Luni
0.0713
0.0602
0.2107
0.1385
0.1136
0.0586
Lzoft
0.0709
0.0592
0.1982
0.1351
0.1112
0.0580
LightGCN
Stiefel
0.0703
0.0582
0.1882
0.1352
0.1071
0.0528
Lioydet
0.0725
0.0612
0.2151
0.1418
0.1210
0.0601
Luns
0.0702
0.0589
0.1960
0.1325
0.1085
0.0580
Lioft Stiefel
0.0699
0.0582
0,2060
0,0989
0.1018
0.0595
MLP
0.0691
0.0579
0.1852
0.1191
0.0958
0.0523
Llogdet
0.0721
0.0601
0.2154
0.1414
0.1151
0.0607
Performance Comparison with the State of the Art. To further demonstrate the exceptional performance of GCF Logdet. we conduct comparative analysis with several recently proposed recom- mendation models that are based on augmentation or contrastive learning techniques. As illustrated in Table 3, GCFiogdet outperforms the other models by a significant margin, achieving the best per- formances, respectively. Additionally, NCL and MixGCF, which utilize LightGCN as their backbone, exhibit competitive performance. In contrast, DNN+SSL and BUR fall short of expectations and are not comparable to LightGCN. We attribute their suboptimal performance to the fact that DNNs have been proven be effective only when abundant user/item features are available. In our datasets, however, such features are unavailable, and self-supervision signals are created by masking item embeddings, which makes it difficult for these models to perform well in this context. We also report performance on Recall@ K and NDCG@K with = 5, 10 in Table 9.
6.2 Quantitative Analysis
Mcasuring the Dimensional Collapse. The evidence of dimensional collapse was identified in contrastive learning by observing the spectrum of representations [19]. The rank of the matrix corresponds to the number of dimensions retained by the transformation (i.e., the dimension of its range) but reveals nothing about the shape of the spectrum. The effective rank [33] (see the definition in the Appendix A.7) quantifies how balanced the spectrum is by means of the spectral entropy. Thus, to show the dimensional collapse tendency when training the model, instead of showing the distribution of eigenvalues, we use the effective rank. Figure shows that the effective rank increases as the training progresses, indicating an increasingly balanced eigenvalue distribution. GCFlogdet consistently achieves higher effective rank compared to other methods. This finding not only verifies existence of the dimensional collapse in CF but also indicates the effectiveness of our GCFtogdet- Figure 4d further shows that the condition number decreases as more batches are processed, thus balancing the spectrum embedding matrices.
Overcoming the Popularity Bias. Below we investigate whether our GCFlogdet can reduce the popularity bias by promoting more uniform representations and alleviating the problem of dimensional collapse. We partition the test set into three subsets based on item popularity. Specifically, we label 80% of the items with the fewest clicks/purchases as "Unpopular," 5% of the most clicked/purchased items as "Popular," and the remaining items as "Normal." We then conduct experiments measure Recall @ @0 contributed by each group, with the overall Recall @2 value being the sum of the values

---

01.06
Yelp2018
iFasson
Model
0.35
Drackk
Unpop.
Norm. Pop.
Unpop. Norm.
Pop.
0.34
SGL
0.63
0.83
-0.11
0.33
0.05
-0.08
SimGCL
0.52
1.21
0.16
0.44
0.14
-0.03
DirectAU
0.33
0.91
0.11
0.27
0.07
-0.07
0.32
GCFsoft
0.41
1.10
0.13
0.41
0.03
-0.01
0.13
GCF fogdet
1.51
1,48
0.25
0.94
0.18 0.05
08:00
Uppa
Hormal
Fopddr
Unposde
Nortal
ABbaka
Pockies
Figure 5: DegDet vs. other baselines. LightGCN the backbone used in all methods.
Table 5: The numerical relative improvement of recall @ @0 in Figure 5 over LightGCN.
from all three groups. The results are presented in Figure 5 and Table 5, which show that our method significantly improves the ability to recommend unpopular items. In contrast, LightGCN tends to recommend popular items and achieves the highest recall value on Normal and Popular items.
Ablations on Different Decorrelation Penalties. Table 4 compares the performance on LightGCN and MLP equipped with different decorrelation penalties. Llngdet appears to consistently outperform other penalties, thus verifying its ability to deal with the dimensional collapse and balancing the spectrum of embeddings. Appendix A.8 compares the KL Matrix Divergence and the Riemannian metric in terms of speed. Appendix A.9 compares different distance types.
7 Related Works
Graph Contrastive Learning applies CL to the graph domain. By adapting DeepInfoMax [1] to graph representation learning, DGI [37] learns embedding by maximizing the mutual information to discriminate between nodes of original and corrupted graphs. REFINE [54] uses simple negative sampling term inspired by skip-gram models. COLES [57] and GLEN [56] link the GCL to the form of Laplacian Eigenmap with negative sampling, extended also to image domain by EASE [55]. GRACE [59]/GraphCL [13] create views via graph augmentation for node/graph- task.
COSTA [51] and SFA [52] create graph views via feature augmentations.
Graph Collaborative Filtering. Graph Neural Networks (GNNs) have proven to be powerful architectures for modeling recommendation data, replacing MLP-based models and improving the performance of neural recommender systems [20, 15]. GNNs are a foundation for several state-of-the-art recommendation models. The most common GNN variant GCN, which has driven the development of graph neural recommendation models, such as GCMC, NGCF, and LightGCN [3, 41, 15]. These GCN-based models refine embeddings and perform graph reasoning by gathering neighborhood information in the user-item graph. Among such models, LightGCN stands out for its effectiveness and simplicity by eliminating transformation matrices and non-linear activation functions. The success of LightGCN has inspired subsequent CL-bbased recommendation models, such as SGL and SimGCL [43, 46].
Graph Contrastive Learning for Recommendation. Inspired by the success of CL in other fields [11, 5, 35], CL has been combined with recommendation systems. Yao el al. [45] proposed feature dropout based two-tower architecture for large-scale item recommendation. NCL [27] designed a prototypical contrastive objective to capture the correlations between a user/item and its context. SimGCL [46] adds noise to the embedding space to perform feature augmentation to enhance the performance of recommendations. DirectAU [38] proposes optimizing the alignment and uniformity of both user and item during training. The most widely used model, SGL [43], applies edge/node dropout to augment the graph data. Although these methods have demonstrated their effectiveness, they pay little attention to why CL can enhance recommendations.
Conclusions
We have investigated the popularity bias in Graph Collaborative Filtering, and showed it results from the dimensional collapse of the embedding space. We have demonstrated that the commonly used backbone GCF, LightGCN, is prone to pose the dimensional collapse. To this end, we have proposed the LogDet divergence based decorrelation penalty for the GCF, We have showed that Gogie promotes full-rank embedding matrices and yields small condition number with theoretical guarantee. For these reasons, GCFiogdet has outperformed other contrastive- based GCF models on several benchmark datasets and improved the performance for unpopular items.
10

---

Acknowledgments
The work described in this paper was partially supported by the National Key Research and Devel- opment Program of China (No. 2018AAA0100204), RGC Research Impact Fund (RIF), R5034-18 18 (CUHK 2410021), and RGC General Research Funding Scheme (GRF) 14222922 (CUHK 2151 185). PK supported by CSIRO's Science Digital.
References
[1] Bachman, D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. NeurlPS, pages
2019.
[2] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variaace-invaaiinc--cvvr regularization for self-supervised learning, arXiv preprint arXiv:2105.044006 2021.
[3] R. d. Berg, T. N. Kipf, and M. Welling. Graph convolutional matrix completion. arXiv preprint IrXiv:/706.0220 2017.
[4] J. Chen, H. Dong, X. Wang. F. Feng, M. Wang, and X. He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39, 2023.
[5] Chen, S. Kornblith, M. Norouzi, and G. Hinton, simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.
[6] Y. Chen, Y. Fang, Y. Zhang, and King. Bipartite graph convolutional hashing for effective and efficient top-n search in hamming space. In Proceedings of the ACM Web Conference 2023 (WehConf). ACM, 2023.
[7] Y. Chen, M. Yang, Y. Zhang, M. Zhao, Z. Meng. J. Hao, and I. King. Modeling scale-free graphs with hyperbolic geometry for knowledge-awee recommendation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM), pages 94 94-102, 2022.
[8] Y. Chen, Y. Yang, Y. Wang, J. Bai, X. Song, and I. King. Attentive knowledge-aware graph convolutional networks with collaborative guidance for personalized recommendation. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 299-311. IEEE, 2022.
[9] Y. Chen, Y. Zhang, M. Yang, Z. Song. C. Ma, and I. King. Wsfe: Wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. The 46th International ACM SIGIR Conference Research and Development in Information Retrieval (SIGIR), 2023.
[10] A. Cherian, S. Sra, A. Banerjee, and N. Papanikolopoulos. Jensen-bregman logdet divergence with application to efficient similarity search for covariance matrices. IEEE transactions pattern analysis and machine intelligence, 35(9):2161-2117) 2012.
[11] T. Gao, X. Yao, and D. Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint Xiv:2104408822 2021.
[12] G. H. Golub and C. F. Van Loan. Matrix computations. JHU press, 2013.
[13] H. Hafidi, M. Ghogho, Ciblat, and A. Swami. Graphel: Contrastive self-supervised learning of graph representations. arXiv preprint arXiv:2007.08025 2020.
[14] W. Hamilton, 7. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[15] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages $39-648, 2020.
[16] X. He, Liao, H. Zhang, I.. Nie, Hu, and T.-S. Chua. Neural collaborative filtering. In WWW, pages 173-182. International World Wide Web Conferences Steering Committee, 2017.
11

---

[17] T. Hua, W. Wang, Z. Xue, S. Ren. Y. Wang, and H. Zhao. On feature decorrelation in self- supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598-9608, 2021.
[18] T. Huang, Y. Dong. M. Ding, Z. Yang, W. Feng, X. Wang, and J. Tang. Mixgef: An improved training method for graph neural network-based recommender systems. In KDD, pages 665-674, 2021.
[19] L. Jing, P. Vincent, Y. LeCun, and Y. Tian. Understanding dimensional collapse in contrastive self- supervuie learning. arXiv preprint arXiv:21 2021.
[20] N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[21] P. Koniusz, F. Yan, P.-H. Gosselin, and K. Mikolajezyk. Higher-order occurrence pooling on mid-and low-level features: Visual concept detection. 2013.
[22] Koniusz and Zhang. Power normalizations in fine ene-grained image, few-shot image and graph classification. IEEE Trans. Pattern Anal. Mach. Intell,, 44(2):591-609) 2022.
[23] B. Kulis, M. A. Sustik, and I. S. Dhillon. Low-rank kernel learning with bregman matrix divergences. Journal of Machine Learning Research, 10(2), 2009.
[24] D. Lee, S. Kang, H. Ju, Park, and H. Yu. Bootstrapping user and item representations for one-class collaborative filtering. In F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, and T. Sakai, editors, SIGIR, pages 1513-1522, 2021.
[25] M. Lezcano Casado. Trivializations for gradient-bast based optimization on manifolds. In Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Gamett, editors, Advances in Neural Information Prucessing Systems, volume 32, 2019.
[26] L. Liang, Z. Xu, Z. Song, King, and Ye. Resnorm: Tackling issue in graph neural networks via normalization. CoRR, fas/2206.0818)) 2022.
degree distribution
[27] 7. Lin, C. Tian, Y. Hou, and W. X. Zhao. Improving graph collaborative filtering with neighborhood enriched contrastive learning. In WWW, pages 2320-2322 2022.
[28] K. Liu, W. Tang, Zhou, and G. Qiu. Spectral regularization for combating mode collapse in gans. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1382-6390, 2019.
[29] Y. Ma, X. Liu, T. Zhao, Liu, Tang, and N. Shah. A unified view on graph neural networks as graph signal denoising. In CIKM '1: The 30th ACM International Conference Information and Knowledge Management, Virtual Event, Queensland, Australia, November 5,, 2021. ACM, 2021.
[30] Y. Ma, Z. Song, X. Hu, J. Li, Y. Zhang, and I. King. Graph component contrastive learning for concept relatedness estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13362-13370, 2023.
[31] S. Rendle, C. Freudenthaler, 7. Gantner, and Schmidt-Thiemee Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:120S.2611 2012.
[32] Ricci, L. Rokach, and B. Shapira. Introduction to recommender systems handbook. In Recommender systems handbook, pages 1-35. Springer, 2010.
[33] O. Roy and M. Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
[34] Z. Song and King. Hierarchical heterogeneous graph attention network for syntax-aware summarization. In AAAI, pages 11340-11348 AAAI Press, 2022.
[35] Z. Song, 7. Meng, Y. Zhang, and I. King, Semi-supervised multi-label learning for graph- structured data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1723-1733, 2021.
12

---

[36] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks, arXiv preprint arXiv:1710.109000 2017.
[37] Velickovic, W. Fedus, W. L. Hamilton, Liò, Y. Bengio, and R. D. Hjelm. Deep graph infomax. In ICLR, 2019.
[38] C. Wang, Y. Yu, W. Ma, M. Zhang, C. Chen, Y. Liu, and S. Ma. Towards representation alignment and uniformity in collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1816-1825, 2022.
[39] Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495-2504, 2021.
[40] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICMI., pages 9929-99399 PMI.R, 2020.
[41] X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference Research and development in Information Retrieval, pages 165-174, 2019.
[42] X. Wang, H. Jin, A. Zhang, X. He, T. Xu, and T.-S. Chua. Disentangled graph collaborative filtering. In SIGIR, pages 1001-1010, 2020.
[43] Wu, X. Wang. Feng, X. He, L. Chen, J. Lian, and X. Xie. Self-supervised graph learning for recommendation. In SIGIR, pages 726-735, 2021.
[44] M. Yang, M. Zhou, J. Liu, D. Lian, and I. King. Href: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM Web Conference 2022, pages 2472-2471, 2022.
[45] T. Yao, X. Yi, D. Z. Cheng, F. Yu, T. Chen, A. Menon, L. Hong, E. H. Chi, S. Tjoa, J. Kang, et al. Self-supervised learning for large-scale item recommendations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 4321 13300, 2021.
[46] J. Yu, H. Yin, X. Xia, T. Chen, L. Cui, and Q. V. H. Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In SIGIR, pages 1294-1303, 2022.
[47] J. Zbontar, L. Jing. I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310-12320. PMLR, 2021.
[48] S. Zhang, D. Luo, L. Wang, and P. Koniusz. Few-shot object detection by second-order pooling. In ACCV, volume 12625 of Lecture Notes in Computer Science, pages 369-387. Springer, 2020.
[49] Y. Zhang, Y. Chen, Z. Song, and I. King. Contrastive cross-scale graph knowledge synergy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD), 2023.
[50] Y. Zhang. H. Zhu, Z. Meng P. Koniusz, and I. King. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022, pages 1331-,339, 2022.
[51] Y. Zhang, H. Zhu, Z. Song, P. Koniusz, and I. King. COSTA: Covariance-preservinn feature aug- mentation for graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2524 2534, 2022.
[52] Y. Zhang, H. Zhu, Z. Song, P. Koniusz, and I. King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11289-112 11297, 2023.
[53] X. Zhou, A. Sun, Y. Liu, J. Zhang, and C. Miao. Selfef: simple framework for self-supervised collaborative filtering. arXiv preprint arXiv:2107.03019 2021.
13

---

[54] H. Zhu and P. Koniusz. REFINE: Random range finder for network embedding. In ACM Conference on Information and Knowledge Management, 2021.
[55] H. Zhu and P. Koniusz. EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9078-9088. June 2022.
[56] H. Zhu and Koniusz. Generalized laplacian eigenmaps. Advances Neural Information Processing Systems, 55:30783-307971 2022.
[57] H. Zhu, K. Sun, and P. Koniusz. Contrastive laplacian eigenmaps. Advances in Neural Information Processing Systems, 34:5682-5695, 2021.
[58] M. Zhu, X. Wang. C. Shi, H. Ji, and P. Cui. Interpreting and unifying graph neural networks with an optimization framework, In Proceedings of the Web Conference 2021, pages 1215-1226, 2021.
[59] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Deep graph contrastive representation learning. 2020.
14

---

Mitigating the Popularity Bias of Graph Collaborative Filtering: A Dimensional Collapse Perspective (Supplementary Material)
Yifei Zhang', Hao Zhu Yankai Chen', Zixing Song', Piotr Koniusz* -844 Irwin King!
The Chinese University of Hong Kong
Australian National University, #Data61$CSIR
[yfzhang, ykchen, zxsong, king (@weee.cc...eeeeeeeeeeee..ce. www..
allenhaozhuee gwwpeaawahhuugmmlll.comm gmail. com, piotr, piotr. koniusz@dataaf wew.rat..........aa.... csiro.au
A Appendices
A.1 Proof of Lemma
Proof One can set derivative of Equation (3) with respect to Z to zero and get the optimal Z as:
0 iz- AE + t (Z" LZ
Z AE L2 0.
(15)
az
Note that det + nL) > O, thus matrix {1+ n1L- exists. Then the corresponding closed-form solution can be written as:
= ((1 + - -1 ĀE
(16)
Since 9 < 1 0. and matrix À has absolute eigenvalues bounded by 1, thus, all its positive ItM powers have bounded operator norm, then the inverse matrix be decomposed as follows with k + 00:
1
Z =
I
A
ĀE
1+
1
+++1 17
1
172
I+
A + ... + (1 n)
A +... AE
(17)
1+1
1+
(1+1)²
Z
1
ÀE
1
A
(1+ AKE+...
K-1
1+1
(1 n)
Note that I+n (1+13² fit fusion weights on, .., OKK When the layer K is large enough, the propagation mechanism of LighIGCN in Equation (3) approximately corresponds to the objective Equation (4).
1 and we can change the coefficient E oo) to
(1+7)
'The corresponding author.
Code:
15

---

A.2 Proof of Lemma
Proof First, let us take the gradient of (Z LZ) with respect to the input matrix E and denote
Z = AE where A L A
OC SLsoohth
tr (Z
LZ)
OE
δE
0r ( L(ÁE)
(18)
O
LÂE 2QE.
Treat the weight matrix as a function of the training step i.e., E(I) then we can derive the gradient E(t) with respect to f. by dE(t) 2QE. As both Q are fixed, we can solve the equation analytically,
(()) = exp(2Q() E(O).
(19)
As we have the non-ascending eigenvalues of Q 2, 2 2 2- Xp. we can define an auxiliary
function / (1: exp (x91) / exp ( (49 It is obvious that L; x
is monotonically decreasing for all i > j. As E(t) is transformation of its initial state E(0) up to exp(Qt) (Qt), we can conclude that:
Vt < and > j.
E
Let the spectrum be following the descending order. Then have lim f (t:19 0, 0i
j if A + x.-
Notice the above expression analyses the decay of spectrum for matrix exp(2QI). Thus, assume E(0) is full-rank matrix. Then
rank(exp((Qt)) E(0)) S min (an)(((p(2Q))),ankkk
due to the well-known inequality stating that rank(XY) S min (rank(X), rank(())))))) (Y)).
A.3 Proof of Corollary
Imagine that En = di, 0.1]). Let AEu = diag( -0.1]) then log det Eer + Ex) log log A2 log log -o Llogdet 00. In contrast, for Lzoft we have 1 0.81. If 10 user feature vectors f(u) diag([1, 0.31]) are in relation with 10 item feature vectors f(i) diag([1, 0.0)). is easy to see that 10 0.312 2 which means the alignment loss is better off with the dimensional collapse for Leoft as 1 >> 0.81. For the LogDet penalty however we have 1 00.
A.4 Proof of Lemma
We have the following:
Proof 3
D Dona (X,I) tr(X) log deg())X))..) -d Σ ()2 - log ); - -1).
(20)
i=1
Now, - llog 2 1 with equality at 1. Also, log 2 llog
-log 4 with equality at
x 2. Letting X 2 A2 ... 2 Xa > O, we have:
D X. (log \r +1 - log 4) (log Aa + 1)
(21)
Cond(X) (X) S exp Dena IX,I)
Thus, LogDet yields upper bound the condition number.
16

---

A.5 Detailed Settings
For the general settings, we create the user and item embeddings with the Xavier initialization of dimension 64; we use Adam to optimize all the models with the learning rate 0.001; the l2 regularization coefficient 10-4 and the batch size 2048 are used, which are common in many papers [15, 43, 42]. In SimGCI. and SGI,, we empirically set the temperature = 0.2 as this value often reported the best choice in papers on CL. [43, 40]. An exception is that we T 0.15 for XSimGCL on Yelp2018. which brings slightly better performance. Note that although the paper of SGL [43] uses Yelp2018 8 and -liaaoaiiFashinn as well, we cannot reproduce their results on Alibaba-iFashion with their given hyperparameters under the same experimental setting. So we re-search the hyperparameters of SGL and choose to present our results on this dataset in Table 3.
A.6 Dataset Statisties
Table 6: Dataset statistics.
Dataset
#User
#Item
#Feedback
Density
Yelp2018
31,668
38,048
1,561,406
0.13%
Amazon-Kindle
138,333
98,572
1,909,965
0.014%
Alibaba-IFashion
300,000
81,614
1,607,813
0.007%
A.7 Effective Rank
Definition 2 (Effective Rank.) Consider matrix X E Ruxx" whose singular value decomposition is given by X UEVT, where E is a diagonal matrix with singular values σ1 Z 02 2 ... aq 2 with Q = min(m.mr n}. The distribution of singular values is defined as the εm-orrm normalized
form p; 0./ ΣR=
-==1 |o The effective rank of the matrix X, denoted erank (X), is defined as crank XX = exp [H P21.P2." PQQ),, where H (pt, P2, ... PP))) is the Shannon entropy given by
... .QQ - Σ2 k= pę: log Pk- H (PL. ...
A.8
Comparison of Runtimes with the Riemannian Metric
The main advantage of Dena over the Riemannian metric DR (such AIRM [10] and LERM [10] is its computational speed. To compute Dem.. only determinants need to be computed, which can be efficiently achieved with Cholesky factorization (for ΣxEz) at a cost of jdd flops [12]. On the other hand, computing the Riemannian metric requires generalized eigenvalues, which takes around 4d flops for positive definite matrices. Therefore, in general, Dais is much faster to compute (see Table 7b). This speed advantage becomes even more pronounced when computing gradients. Moreover, backpropagation through the matrix determinant is generally stable whereas generalized eigenvalue decomposition suffers undetermined gradients if two eigenvalues are non-simg non-simple (equal values). As shown in Table 7a, computing dDon can be over 100 times faster than ØDØR. This difference can be crucial when using gradient-based algorithms, such as neural networks, that rely on the computation of similarity measure gradients.
Table 7: Runtime computed over 1000 trials (millisecs/trial).
(a) Average times to compute gradients.
(b) Average times to compute function values.
d
ix,,
0x Dona (X, I)
d
DR(X, I)
Dena (X, I)
0.79815 1.79915:10993 1100931
0.036
5
0.025 : 0.012
0.030
10
2,38341 102094
0.058 1-0.22
10
0,036 1 0.005
0.040
20
7.4936510599
0. 0.003
20
0,085 1: 0.006
0.06 1.000
40
24.8942
0.240=0.047
40
0.270 +: 0.332
0.
80
99.482545.188
0.921 -0.028
80
1,234 + 0.05s
0.393
200
698.813 602
8.767 12.177
200
8.198 + 0.129
2.223
500
6377.22 1379.1
94.83 1.99
500
77.311
+ 0.568
28.18 1.2
1000
40443.00 24222
622.2+31.70
1000
492.743
+ 15.51
119.7 1.466
17

---

A.9
Performance Comparison w.r.. to Different Distance Types
Table 8: Main comparison on different distances. denotes the Matrix Norm and denotes the
Bregman Matrix Divergence. KL Matrix Div. is Bergman Div. associated with ((X) = E A log A
Yelp2018
iFashion
Geometries
D(Ex,1)
Recall 20
NDCG@20
Recall 20
NDCG @ 20
Euclidean Norm
Ex 11
0.0563
0.0459
0.0890
0.0404
Nuclear Norm +
Ex I
0,0632
0.0516
0.0998
0.0458
Frobenius Norm
Σx I
0.0709
0.0592
0.1112
0.0580
KL Matrix Div.
(Exx log Σx)
0.0724
0.0602
0.1110
0.0597
Logdet Div.
log det (Ex)
0.0732
0.0618
0.1270
0.0617
In this section, we adopt other types of distances as Equation (11) and compare them with proposed method. There are two different categories (1) the matrix norm, i.e. Euclidean Norm, Nuclear Norm, and Forbenius Norm, and (2) the Bregman divergence, iee,, von Neumann divergence (also known as Matrix Kullback-Leibler divergence) and Logdet divergence. The Forbenius norm can belong to both matrix norm and Bregman Matrix divergence. We show their formulas and experimental result in Table 8, We notice that the proposed method (Logdet Div.) achieves the best results.
Table 9: Result for Recall @5, Recall @ 10, and Recall 20.
Dataset
Model
Recall@5
Recall 10
Recall @ 20
GCFjoget
0.0275
0.0445
0.0732
Yelp2018
DirectAU
0.0255
0.0426
0.0720
LightGCN
0.0211
0.0336
0.0590
GCF,oget 0,0301
0.0483
0.0617
iFashion
DirectAU
0.0292
0.0493
0.0601
LightGCN
0.0284
0.0391
0.0484
A.10 Broader impact and limitations
Our method enjoys impact and limitations similar to those in graph collaborative filtering. Typical GCF models cannot guarantee they can utilize the feature space efficiently. The mode collapse can lead to frequent item bias resulting in model which is biased in its recommendations. Thus, by improving recommendation of rare items we also offer a generic approach to limiting the recommendation bias which is important in many domains of life. Our method requires special resources and just fraction of additional computations beyond what GCF uses. Of course, our model limited by the GCF model itself and it is applicable only pipelines that suffer the mode collapse.
18


Reviews:
[{"review_id": "lit7AUrQI0", "reviewer": "Reviewer_pKCn", "summary": "This paper aims to study the problem that features tend to occupy the embedding space inefficiently. The authors proposes a decorrelation-enhanced GCF objective that promotes feature diversity. LogDet divergence is leveraged to form a new constraint loss.", "questions": "1. The title mentioned \"Mitigating the Popularity Bias\". However, it seems not to be the main aim of this paper.\n2. Some details are missing. How to implement f_{\\theta}.\n3. Typos. E.g., migrate -> mitigate", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The studied problem of embedding space degradation in graph-based collaborative filtering is interesting and important.\n2.  LogDet based decorrelation penalty is developed, and its ability of maintaining the full-rank constraints is theoretically demonstrated.\n3.  Experiments show improvement over some baselines.", "weaknesses": "1.  The dimensional collapse issue (also known as overcorrelation) has been studied in [1] but not cited.\n> Jin W., Liu X., Ma Y., Aggarwal C. and Tang J. Feature over correlation in deep graph neural networks: a new perspective. KDD, 2022.\n2. This issue occurs when stacking too many layers whether or not there is a popularity bias.\n3. I think 'logdet' regularisation works so well because it encourages embeddings to become rich in different dimensions, rather than mitigating popularity bias.\nNote that the relative proportions in the three scenarios in Figure 4 are very close.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 1 (pKCn)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. The dimensional collapse issue (also known as overcorrelation) has been studied in [1] but not cited.\n- Thank you for pointing out this related work. We have now included it in the related works. **The *'Explicit Feature Dimension Decorrelation'* introduced in [1] matches closely with what we refer to as $\\mathcal{L}_{soft}$ in our paper (see Eq. 9)**. This technique involves the decorrelation of the feature matrix by encouraging a covariance matrix to approach $\\mathbf{I}$ through the utilization of the Frobenius Norm. Within our framework, this is just one solution to the case when the function $\\phi(\\cdot)$ of the Bregman Divergence is defined as $\\phi\\_F(\\mathbf{X}) = ||{\\bf X}||_F^2$ (line 187).\n- As highlighted in our discussion within Corollary 2, **it is important to note that $\\mathcal{L}_{soft}$ does not inherently ensure the preservation of the range space of the covariance matrix**. Consequently, it does not actively enforce the covariance matrix to attain full rank. Instead, this `soft' approach applies soft constraints that encourage isotropic and full-rank aspects (but cannot strictly enforce the full rank).\n- In Table 1 (main submission), we summarize the characteristics of various regularization techniques and draw comparisons between them. Our approach, denoted as $\\mathcal{L}\\_{logdet}$, encourages isotropy while simultaneously employing stringent constraint to impose full rank (see Corollary 1) unlike $\\mathcal{L}\\_{soft}$.  Importantly, **Table 4 (main submission) shows that our $\\mathcal{L}\\_{logdet}$ outperforms $\\mathcal{L}\\_{soft}$ in terms of recall@20** on Yelp2018, Amazon-Kindle and iFashion (**7.32%** *vs.* 7.09%, **22.5\\%** *vs.* 19.8% and **12.7\\%** *vs.* 11.1%).\n\n## 2. This issue occurs when stacking too many layers whether or not there is a popularity bias.\n- Stacking many layers in graph neural networks can exacerbate popularity bias, leading to an overcorrelation of features. Popularity bias persists even when employing a minimal number of graph neural network layers in GCF, e.g., the commonly used GCF backbone, LightGCN, typically incorporates no more than 3 layers.\n\n**Even with one layer (no over-smoothing)**, we get improved recall@20 with our logdet model compared to soft uniformity penalties, LightGCN, and NCF (MF-based model):\n| Yelp2018 | Amazon-Kindle | iFashion |\n|-|-|-|\n| NCF      | 0.573    | 0.1857  |  0.0831  |\n| LightGCN | 0.0591   | 0.1871  |  0.0845  |\n| GCF$_{soft}$    | 0.0672   | 0.1975  |  0.0972  |\n| GCF$_{logdet}$  | **0.0694**   | **0.2103**   | **0.1110** |\n\n- The prevalence of popularity bias in GCF can be attributed to **the significant imbalance within the data distribution across the recommendation bipartite graph**. This distribution often features numerous users engaging with a small subset of items, within a much larger interaction space encompassing millions, or even billions of items.\n\nConsequently, during the process of message passing within GCF, a few popular items can disproportionately influence the entire learning procedure, leaving out other items with infrequent recommendations. **This dynamics leads to the embedding space wherein the user embeddings are around a select few popular items, resulting in dimensional collapse**. A visual representation of this phenomenon is depicted in **Fig. 1 (the rebuttal pdf)**. Notably, the user and item embeddings are generated using a two-layer LightGCN configuration.\n\n## 3. Logdet regularisation works so well as it encourages embeddings to be rich in different dimensions, rather than mitigating popularity bias (relative proportions in Fig. 4 are very close).\n\n- The popularity bias is mainly caused by the skew distribution of users which is shown as the dimensional collapse in the latent embedding space.\n- Our method can effectively improve **the performance of recommending the unpopular item (alleviating the popularity basis)**, we add more baselines and show the relative improvement numerically in the table below:\n\n| Yelp2018 | Unpopular | Normal | Popular |\n|-|-|-|-|\n| SGL | 0.630| 0.83| -0.112|\n| SimGCL | 0.521| 1.21|  0.160|\n| DirectAU| 0.331| 0.91|  0.110| \n| GCF$_{logdet}$ | **1.51** | **1.48** | **0.252** |\n\n| Alibaba-iFasion | Unpopular | Normal | Popular |\n|-|-|-|-|\n| SGL | 0.331   | 0.055     | -0.087    |\n| SimGCL | 0.442   | 0.148     | -0.035    |\n| DirectAU| 0.272   | 0.074     | -0.070    |\n| GCF$_{logdet}$   |**0.941**| **0.185** | **0.052** |\n\n\nKindly notice, our $\\mathcal{L}\\_{logdet}$ achieves best relative improvements on  Unpopular, Normal and Popular items compared to other methods. However, considering our method alone, we get the biggest relative gain on **Unpopular** items (**0.941**), followed by **Normal** items (**0.185**) and then **Popular** items (**0.052**). Such a trend supports our observation that $\\mathcal{L}\\_{logdet}$ tackles the popularity bias.\n\nNonetheless, **we agree that having more isotropic covariance by preventing i.e. rank deficient covariance matrix** has positive impact also on normal and popular items. This is the natural consequence of such a design. \n\n**By dimensional collapse in this paper, we mean lower `effective rank' than possible** which makes embedding space less efficient, yes.\n\nKindly also **see Fig. 1 in the rebuttal PDF**. We illustrate there that indeed **unpopular items are not distributed uniformly on the surface of hypersphere (which lowers the efficient rank)** for ordinary LightGCN. **After applying the logdet penalty, the spread across the $\\ell_2$ ball surface improves**, and infrequent items are more often recommended as a result. \n\nKindly also refer to a **detailed description of Fig. 1 in Resp. 1 to Reviewer 5 (VmvT).**\n"}, {"review_id": "1fed2TMeqT", "reviewer": "Reviewer_CD6b", "summary": "This paper studies popularity bias in graph collaboration filtering through dimensional collapse analysis of the embeddings. The authors proposed the LogDet divergence based de-correlation penalty for GCF to improve the performance. ", "questions": "* A minor suggestion on Figure 3, where the last subfig is used for showing the change of condition number over training steps of just one of the three datasets used in the other subfigs. Why not to plot all for completeness, if really the same trends of condition numbers were observed for all? \n\n* How is the impact of K in NDCG@K in the experiment? Is it selective in the experiment, which might imply the missing items in top-K?", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work has thorough analysis of dimensional collapse in GCF and its relationship towards propagation and uniformity. De-correlation enhanced objective for mitigate dimensional collapse in GCF. Proposed LogDet divergence to preserve the range4 space of the covariance matrix. ", "weaknesses": "\n* While popularity bias adversely impacts the recommendation quality, the quality of the items may also play a role from the end to end recommendation quality. The paper shows the solid improvement on Recall; considering that nDCG@K does not penalize for missing items, it is a bit hard to tell the overall performance. \n\n* The model names seem inconsistent it the experimental study section, such as that in Figure 4 and Tables 2 & 3, which causes some confusion when reading the sections.\n\n* Some details of the hyperparameter setting on the validation dataset should be presented if it is not trivial for the experimental results. \n\n* While it is good to see the proposed method improves the ability to recommend unpolular items, which potentially helps the diversity, the baseline selection is a bit unclear. According to the numbers in Table 3, the baselines in Figure 4 are not the most compelling ones. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 2 (CD6b)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. The model names seem inconsistent it the experimental study (Fig. 4 and Tables 2-3).\n\nWe apologize. We have now revised our manuscript accordingly, e.g., we call now our method GCF$\\_{logdet}$ in Fig. 4 instead of DEC. We have now made it clear that our evaluations are mainly based on LightGCN and the logdet penalty. New figure is updated as Figure 2a in rebuttal pdf.\n\n\n## 2. In Fig. 3 (last sub-figure) is showing the change of condition number over training steps of just one datasets. \n\nThank you. Kindly see **Fig. 2b in the rebuttal PDF.** We have now added the figures for all three datasets. In all three cases, the condition number decreases rapidly as the number of training steps increases.\n\n\n## 3. While popularity bias adversely impacts the recommendation quality, the quality of items may also play a role in the  recommendation quality. The paper shows the solid improvement on Recall.\n\nSince we focus on the ranking ability of our model in the recommendation, Recall and NDCG are the most relevant metrics and they are standard across various works, e.g.:\n> Yu et al. Are graph augmentations necessary? simple graph contrastive learning for recommendation, SIGIR 22.\n> Wu et al.Self-supervised graph learning for recommendation. SIGIR 23\n\nTo further address the reviewer's concern about the overall performance, **we provide below experiments using AUC.** We used the Yelp2018 dataset and for the alignment loss we used a simple cross-entropy loss instead of bpr loss, as this is a standard practice for works focusing on AUC metrics:\n> Guo et al. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction\n\n DirectAU uses the alignment and uniformity loss one the LightGCN backbone.\n \\\n Our GCF uses the LightGCN (also reported in the table).\n \\\n The table below shows that GCF$\\_{logdet}$ outperforms GCF$\\_{soft}$, DirectAU, the LightGCN baseline and other methods:\n\n|  |  |  |  |  |  |  |\n|-|-|-|-|-|-|-|\n|Yelp2018: | GCF$\\_{logdet}$ | GCF$\\_{soft}$ | SimGCL| DirectAU| SGL| LightGCN |\n|AUC: | **0.853**| 0.843 | 0.850| 0.849| 0.845| 0.841|\n\n\n## 4. What is the impact of K in NDCG@K in the experiment? Is it selective in the experiment, which might imply the missing items in top-K?\n\nKindly see the results for our GCF$\\_{logdet}$  which outperforms GCF$\\_{soft}$, DirectAU (uniformity loss) and LightGCN baselines:\n\n| Yelp | Recall@5 | Recall@10 | Recall@20 |\n| - | --- | --- | --- |\n| GCF$_{loget}$   | **0.0275**   |  **0.0445** | **0.0732** |\n| DirectAU      |   0.0255     |    0.0426   |   0.0720   |\n| LightGCN      |   0.0211     |    0.0336   |   0.0590   |\n\n| Yelp | NDCG@5 | NDCG@10 | NDCG@20 |\n| --- | --- | --- | --- |\n|  GCF$_{loget}$  | **0.301** | **0.0483**|  **0.0618** |\n|  DirectAU     |   0.292   |   0.0493  |    0.0601   |\n|  LightGCN     |   0.284   |   0.0391  |    0.0484   |\n\n## 5. Hyperparameter details.\n\nWe detail the chosen hyper-parameter as follows:\n|Hyperparameter| Value|\n| --- | --- |\n|embedding size  |64    |\n|trainning epoch |50    |\n|batch size      |2048  |\n|learn rate      |0.001 |\n|$\\lambda$          |0.5   |\n|LightGCN Layers |2    |\n|L2 reg. weight  |0.0001|\n\n**We will also include all hyper-parameter details in the paper and in the code.**\n\n## 6. According to the numbers in Table 3, the baselines in Fig. 4 are not the most compelling ones.\n\nThank you. Kindly see **Fig. 2a in the rebuttal PDF**. We have now **added  SimGCL and DirectAU** accordingly. Our Logdet still performs well and our conclusions hold.\n"}, {"review_id": "VUpBbi9THZ", "reviewer": "Reviewer_Qs55", "summary": "In this paper, the authors study the dimension collapse problem in graph-based recommenders. The authors provide both sufficient experimental and theoretical analysis on the occurrence of dimension collapse to popular items and propose a new loss function to encourage feature diversity. Experiments are also sufficient to prove the advantages of the proposed methods. In summary, I think it is a good paper with both sound experiments and theoretical analysis. I suggest a strong acceptance.", "questions": "Do authors think the dimension collapse problem only existed in graph-based collaborative filtering? How about matrix factorization?", "limitations": "I do not think this paper has obvious limitations", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper provides both theoretical and experimental analysis of the dimensions collapse problem in graph collaborative filtering\n\n2. The proposed method is simple and effective, boosting the performance of direct representation learning for collaborative filtering learning.\n\n3. Experiments are solid, and also sufficient to validate all the advantages of the proposed method.\n", "weaknesses": "I do not think this paper has obvious weaknesses to cope with\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 3 (Qs55)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. Does the dimension collapse exist only in graph-based collaborative filtering? How about matrix factorization?\n\n- The phenomenon of dimensional collapse is not limited to GCF, it also extends to matrix factorization-based approaches.\n\n- The occurrence of dimensional collapse is closely related to the presence of popularity bias, as depicted in **Figure 1 of our rebuttal PDF**. The figure reveals that when the feature vectors of unpopular items are not spread on the surface of $\\ell\\_2$ ball well, the embedding may cluster in such a way that the corresponding user cluster is non-dense (so this impedes recommendation). The gaps in the embedding of unpopular items represent what we refer to as the dimensional collapse (the `effective rank' of covariance of such embedding is lesser than the well-conditioned case in the bottom Figure 1.) Figure 1 has been prepared with 2D PCA over actual embedding on Yelp2018 for LightGCN and our Logdet (also based on LightGCN).\n\n- **We further demonstrated the occurrence of dimensional collapse in matrix factorization-based techniques**. Specifically, we trained the Neural Collaborative Filtering (NCF) and LightGCN models, monitoring the effective rank of the user embedding matrix as outlined in Section A.7, Definition 2, which can be found in the appendix of the main paper. **The outcomes are depicted in Figure 2(c) of the rebuttal PDF**.\n\n- A lower effective rank signifies a reduced occurrence of dimensional collapse. As we demonstrated in the main paper, LightGCN is particularly susceptible to dimensional collapse, NCF has consistently lower effective rank, revealing that even this matrix factorization-based approaches suffers from dimensional collapse."}, {"review_id": "2kVurrj9nu", "reviewer": "Reviewer_bxqe", "summary": "The paper proposes a novel graph collaborative filtering (CF) approach which tackles the issue of item popularity bias. Starting from the (theoretically-proven) assumption that state-of-the-art graph CF architectures (such as LightGCN) may enhance the so-called dimensional collapse (DC) problem, the authors decide to leverage decorrelation of the learned embeddings through the LogDet divergence. Extensive experiments on three popular recommendation datasets, against multiple state-of-the-art (graph)-based recommendation approaches demonstrate the efficacy of the proposed solution. Additionally, the authors show that the model is able to outperform other graph CF contrastive solutions which optimize uniformity. Finally, a quantitative analysis proves how the approach can tackle the dimensional collapse issue and address the item popularity bias.", "questions": "To the best of my knowledge, I cannot think about any specific questions.", "limitations": "To the best of my knowledge, I cannot see any particular limitations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed idea is sound and well-presented.\n+ The solution is well-placed in the related literature.\n+ The experiments are extensive with multiple analyzed dimensions.\n+ Codes and datasets are shared.", "weaknesses": "To the best of my knowledge, I cannot see any particular issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 4 (bxqe)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\nTo further convince reviewer of the value of our work, **we have added Figure 1 in our rebuttal PDF.** The occurrence of dimensional collapse is closely related to the presence of popularity bias, as depicted in **Figure 1 of our rebuttal PDF**. The figure reveals that when the feature vectors of unpopular items are not spread on the surface of $\\ell\\_2$ ball well, the embedding may cluster in such a way that the corresponding user cluster is non-dense (so this impedes recommendation). The gaps in the embedding of unpopular items represent what we refer to as the dimensional collapse (the `effective rank' of covariance of such embedding is lesser than the well-conditioned case in the bottom Figure 1.) Figure 1 has been prepared with 2D PCA over actual embedding on Yelp2018 for LightGCN and our Logdet (also based on LightGCN).\n\n| Yelp2018 | Amazon-Kindle | iFashion |\n|-|-|-|\n| NCF      | 0.573    | 0.1857  |  0.0831  |\n| LightGCN | 0.0591   | 0.1871  |  0.0845  |\n| LightGCN + $\\mathcal{L}\\_{soft}$    | 0.0672   | 0.1975  |  0.0972  |\n| LightGCN + $\\mathcal{L}\\_{logdet}$  | **0.0694**   | **0.2103**   | **0.1110** |"}, {"review_id": "snFm9PQR62", "reviewer": "Reviewer_VmvT", "summary": "The paper discusses the issue of popular bias in Graph Collaborative Filtering (GCF) and demonstrates its resulting Dimensionality Collapse (DC) in the embedding space. The paper points out that LightGCN, widely used in GCF, is prone to DC. The authors introduce a decorrelated penalty based on LogDet divergence to address this problem. Through theoretical analysis and empirical evidence, they demonstrate that GCF LogDet can be extended to full-rank embedding matrices while maintaining a small condition number. On several benchmark datasets, GCF LogDet outperforms other contrastive-based GCF models and improves the recommendation effectiveness for unpopular items.", "questions": "1. According to the proof of the paper, there should be a problem of feature collapse in the graph convolutional neural network. Why did the author choose the recommended scene for proof instead of more general graph data?\n2. What is the connection between dimensionality collapse and popularity bias, and can they be equated?\n3. Why are SimGCL and DirectAU not added to the comparison method in Figure 2 and Figure 4? Obviously, the author's loss function is the closest to DirectAU.\n", "limitations": "1. The authors did not specify the relationship between popularity bias and dimension collapse in detail, which makes readers question the rationality of the author's method.\n2. The paper does not compare with the method of alleviating the popularity bias, and it does not reflect the superiority of the method proposed by the author in alleviating the popularity bias problem.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper explores the limitations of graph collaborative filtering algorithms in the field of recommendation systems and proposes a novel graph convolutional neural network, GCF-LogDet, which effectively addresses the issues of dimensionality collapse and popularity bias in collaborative filtering. Specifically, the paper demonstrates, from the perspective of the representation space, the problem of dimensionality collapse caused by graph convolution and formulates high-quality self-supervised embeddings in recommendation systems from the perspective of reducing redundancy. Moreover, by using LogDet distance instead of the traditional SoftPlus, the paper successfully avoids the issue of drastic variance changes, providing thorough analysis and discussion, and achieving improved experimental results. Additionally, the paper exhibits high quality and clarity by providing detailed descriptions of the methodology and experimental setup, and it provides a detailed proof of the theorem in the supplementary materials.", "weaknesses": "1. In the paper, the title emphasizes alleviating popularity bias, while the analysis and emphasis primarily focus on the issue of dimensionality collapse. However, the direct correlation and equivalence between dimensionality collapse and popularity bias are not explicitly explained or established in the paper. To provide a clearer understanding of the relationship between dimensionality collapse and popularity bias, it would be beneficial for the authors to explicitly discuss and clarify the connection between these two issues in the introduction or related work section. Additionally, the authors should elaborate on how the proposed method specifically addresses the problem of popularity bias, providing a theoretical or empirical explanation of how mitigating dimensionality collapse can effectively alleviate popularity bias in the context of recommendation systems.\n2. This paper seems to lack a comprehensive comparison with existing methods that specifically address popularity bias in the context of graph collaborative filtering. As far as I know, [1] method introduces r-regularization based graph aggregation plugin to alleviate the popularity bias during neighbor aggregation of graph convolution. I recommend that the authors revise the experimental section to include broader comparisons, including existing methods that directly address popularity bias (e.g. [1-2]).\n[1] Zhao M, Wu L, Liang Y, et al. Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022: 50-59.\n[2] Wei T, Feng F, Chen J, et al. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1791-1800.\n3. I suggest that the authors revise the related work and comparative methods sections to include a comprehensive comparison with existing approaches that tackle dimensionality collapse. Specifically, they should discuss and compare methods such as [3-4] that have been proposed to alleviate dimensionality collapse. For example, [3] analyzes the causes of feature collapse from the perspective of graph contrastive learning and proposes a spectral data augmentation method, which is highly relevant to the work presented in this paper.\n[3] Zhang Y, Zhu H, Song Z, et al. Spectral Feature Augmentation for Graph Contrastive Learning and Beyond[J]. arXiv e-prints, 2022: arXiv: 2212.01026.(Accpet by AAAI23)\n[4] Chen X, He K. Exploring simple siamese representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 15750-15758.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 5 (VmvT)\n\n***We thank the reviewer** for the constructive review and valuable questions that have helped us improve our work.*\n\n## 1. Title emphasizes alleviating popularity bias but the analysis focus on the dimensionality collapse. \n\n- **Dimensionality collapse (DC) and Popularity bias are two sides of the same coin**: The prevalence of popularity bias in GCF can be attributed to **the significant imbalance within the data distribution across the recommendation bipartite graph**. This distribution often features numerous users engaging with a small subset of items, within a much larger interaction space encompassing millions or billions of items.\n\\\n\\\nConsequently, during the process of message passing within GCF, a few popular items can disproportionately influence learning, leaving out other items. **This leads to the embedding space where very few user embeddings are around popular item embeddings**.\n\n- **Kindly see Figure 1 (the rebuttal pdf)**. We used Yelp2018 and (top) LightGCN *\\vs.* (bottom) our Logdet (also with LightGCN). We applied 2D PCA to embeddings and $\\ell_2$ renormalized to the unit sphere. \n \\\n \\\n The top figure shows that:\n   - the unpopular items clustered in just one cluster\n   - the corresponding user cluster is non-dense so unpopular items do not get recommended often (**popularity bias**)\n   - **the gaps in the unpopular item embedding** cause `effective rank' drop: we refer to it as **the dimensional collapse**\n   - **if only the embedding was spread uniformly on the surface of the hypersphere, it would align with dense user clusters** which would reduce **popularity bias**\n  \\\n  \\\n  The bottom figure shows that:\n   - **the Logdet managed to spread unpopular user embeddings more uniformly** on the surface of hypershpere - this means `larger effective rank' and thus reduced dimensional collapse**\n  - thanks to more uniformly spread unpopular items, now they formed 3 clusters, and 2 of them align well with dense user clusters, which means **the popularity bias has been reduced**\n\\\n\\\n**This real-data experiment explains the connection between the popularity bias and the dimensional collapse (understood as reduced `effective rank')**, and in **Fig. 3 (main paper) we showed that Logdet achieves the best effective rank**.\n\n## 2. Relation/equivalence of DC and popularity bias not explicitly explained/established.\n\n- **the above experiment explains the connection of DC to popularity bias and the effective rank** (very interesting experiment)\n- moreover, consider the alignment with uniformity loss (Eq. 6, main paper). Calculate the gradient the $\\nabla_{f_\\theta(i)} L$:\n$\\nabla_{f_\\theta(i)} L= \\underbrace{-2\\mathbb{E}\\_{(u, i)\\sim\\mathcal{R}} (f\\_\\theta(u)-f\\_\\theta(i))}\\_{\\text{term 1}}+\\underbrace{\\mathbb{E}\\_{i^-\\sim\\mathcal{I}}\\frac{-2f\\_\\theta(i^-)e^{\\left(2-2 f\\_\\theta(i)^{\\top} f\\_\\theta(i^-)\\right)}}{\\omega}}\\_{\\text{term 2}}$ where $\\omega$ is a const.\n\\\n\\\nIf $f_\\theta(i)$ is popular, **term 1 dominates** the loss and the gradient, clustering users and items together into dense cluster (as in Fig. 1). If $f_\\theta(i)$ is unpopular, **term 1** at best aligns few users and unpopular items. **Term 2** may try spread a bit item embedding but it has no dominating effect (to few iterations, finite penalties).\n\\\n\\\nIn contrast, **our logdet loss encourages covariances of users and items to be isotropic** and forces them to be **full rank**. Dense unpopular item cluster is decompacted and spread around the surface of hypersphere. User embedding is also  spread better around the surface of hypersphere. **Effective ranks are improved for user and unpopular items (Fig. 1 in the rebuttal PDF) and so the popularity bias is mitigated.**\n\n## 3. Compare with existing methods that specifically address popularity bias. \n- Thank you for pointing out following works:\n> [1] *Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering*, ACM SIGIR'22. \n\\\n> [2] *Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system*, ACM SIGKDD'21. \n\n  We have now cited them/added experiments. We chose Yelp2018 (common on both papers and us). We selected LightGCN + $r$-adj\\_norm and LightGCN+MACR:\n\n|Yelp2018|Recall@20|NDCG@20|\n|-|-|-|\n|LightGCN + Logdet (ours)|**0.0720**|**0.0607**|\n|LightGCN + $r$-adj\\_norm|0.0593|0.0485|\n|LightGCN+MACR|0.0631|0.0519|\n\n## 4. Revise the related work/compare with approaches that tackle DC.\n\nWe have now revised/added experiments on:\n- *Spectral Feature Augmentation...* (AAAI'23)\n- *Exploring simple siamese representation learning* (CVPR'21)\n\nWe have implemented them in the typical GCF setting (two-layer LightGCN+BPR loss). \n  - For SFA, we added SFA after the LightGCN layer. \n  - For Siamese, we followed SGL but changed the symmetric branch to  unsymmetric and applied the stop gradient:\n \n|Yelp2018|Recall@20|NDCG@20|\n|-|-|-|\n|Logdet|**0.0720**|**0.0607**|\n|Siamese| 0.0602|0.0494|\n|SFA|0.0651|0.0579|\n\nSFA is suboptimal for recommenders as it only partially balances the spectrum (incomplete power iter.) and adds noise (good for node classification but bad for user-item alignment).\n\n## 5. Why recommender not general graph?\n\nWe are interested in problems as in Eq. 13, where user and item embeddings must be aligned:\n* In contrast to general graph data, recommendation datasets have highly skewed data distribution over the bipartite graph (most users interact sporadically with a very small set of items within a vast interaction space of millions/billions of items)\n* Both embeddings of the user and item may collapse. **When users collapse while items are uniformly distributed**, items located in the dense region of the user will be frequently recommended while items in sparse areas will be ignored. **When an item collapses while users are uniformly distributed**, clustered Items are uniformly recommended to the random user."}]

Author rebuttal:
**Kindly check the rebuttal PDF for Figures (bottom of this panel)** requested by reviewers. Individual responses across rebuttal refer to them.

Meta-review: Overall, all reviewers agreed that the research problem addressed in the submission is of great importance and the experiments conducted are very reliable. Additionally, the authors effectively addressed most of the reviewers' concerns through their rebuttal.
Decision: Accept (spotlight)