LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation
Authors: Xuheng Cai
Chao Huang
Lianghao Xia
Xubin Ren
Venue: ICLR 2023 notable top 25% (2023)
Keywords: recommender systems
graph neural networks
contrastive learning

Abstract: Graph neural network (GNN) is a powerful learning approach for graph-based recommender systems. Recently, GNNs integrated with contrastive learning have shown superior performance in recommendation with their data augmentation schemes, aiming at dealing with highly sparse data. Despite their success, most existing graph contrastive learning methods either perform stochastic augmentation (e.g., node/edge perturbation) on the user-item interaction graph, or rely on the heuristic-based augmentation techniques (e.g., user clustering) for generating contrastive views. We argue that these methods cannot well preserve the intrinsic semantic structures and are easily biased by the noise perturbation. In this paper, we propose a simple yet effective graph contrastive learning paradigm LightGCL that mitigates these issues impairing the generality and robustness of CL-based recommenders. Our model exclusively utilizes singular value decomposition for contrastive augmentation, which enables the unconstrained structural refinement with global collaborative relation modeling. Experiments conducted on several benchmark datasets demonstrate the significant improvement in performance of our model over the state-of-the-arts. Further analyses demonstrate the superiority of LightGCL's robustness against data sparsity and popularity bias. The source code of our model is available at https://github.com/HKUDS/LightGCL.

Published as a conference paper at ICLR 2023
LIGHTGCL: SIMPLE YET EFFECTIVE GRAPH CON- TRASTIVE LEARNING FOR RECOMMENDATION
Xuheng Cai Chao Huang' Lianghao Xia
Xubin Ren
Department of Computer Science, University of Hong Kong {rickcai, thaoxiaJehku. chachuang75gmai1 ccom xubinrenca@gmai coom
ABSTRACT
Graph neural network (GNN) powerful learning approach for graph- based recommender systems. Recently, GNNs integrated with contrastive learning have shown superior performance recommendation with their data augmen- tation schemes, aiming at dealing with highly sparse data. Despite their success, most existing graph contrastive learning methods either perform stochastic aug- mentation (e.g.. node/edge perturbation) on the user-item interaction graph, rely on the heuristic-based augmentation techniques (e.g., user clustering) for generating contrastive views. We argue that these methods cannot well pre- serve the intrinsic semantic structures and are easily biased by the noise per- turbation. In this paper, we propose simple yet effective graph contrastive learning paradigm LightGCI. that mitigates these issues impairing the gener- ality and robustness of CL-based recommenders. Our model exclusively uti- lizes singular value decomposition for contrastive augmentation, which enables the unconstrained structural refinement with global collaborative relation mod- eling. Experiments conducted on several benchmark datasets demonstrate the significant improvement in performance of our model over the tteee-sf- Further analyses demonstrate the superiority of LightGCL's robustness against data sparsity and popularity bias. The source code of our model is available at attps: /ttp://ggtthh//u com www.///KKK/LLLgtSSCSCC
INTRODUCTION
Graph neural networks (GNNs) have shown effectiveness in graph-based recommender systems by extracting local collaborative signals via neighborhood representation aggregation (Wang al.) 2019 Chen et al., (2020b). In general, to learn user and item representations, GNN-based recom- menders perform embedding propagation on the user-item interaction graph by stacking multiple message passing layers for exploring high-order connectivity (He et al. 2020 Zhang et al. 2019 Liu et al., 2021a). Most GNN-based collaborative filtering models adhere the supervised learning paradigm, requiring sufficient quality labelled data for model training. However, many practical rec- ommendation scenarios struggle with the data sparsity issue learning high-quality user and item representations from limited interaction data (Liu et al.) 2021b Lin et al. 202T). To address the label scarcity issue, the benefits of contrastive learning have been brought into the recommendation for data augmentation (Wu et al. 2021). The main idea of contrastive learning in enhancing the user and item representation is to research the agreement between the generated embedding views by contrasting the defined positive pairs with negative instance counterparts (Xie al. 2022
While contrastive learning has been shown to be effective in improving the performance of graph- based recommendation methods, the view generators serve the core part of data augmentation through identifying accurate contrasting samples. Most of current graph contrastive learning (GCL) approaches employ heuristic-based contrastive view generators maximize the mutual information between the input positive pairs and push apart negative instances Wu et al. 202 Yu et al. 2022aa Xia et al., 2002b). To construct perturbed views. SGL (Wu al. 2021 has been proposed to gener- ate node pairs of positive view by corrupting the structural information of user-item interaction graph using stochastic augmentation strategies, e.g., node dropping and edge perturbation. To improve the
"Chao Huang is the corresponding author.

---

Published as a conference paper at ICLR 2023
graph contrastive learning in recommendation, SimGCL Yu et al. 2022a) offers embedding aug- mentation with random noise perturbation. To work on identifying semantic neighbors of nodes (users and items), HCCF (Xia et al. 2022b) and NCL (Lin et al.) 2022) are introduced pursue consistent representations between the structurally adjacent nodes and semantic neighbors. Despite their effectiveness, state-of-the-ar contrastive recommender systems suffer from several inherent limitations: i) Graph augmentation with random perturbation may lose useful structural informa- tion, which misleads the representation learning. ii) The success of heuristic-guider representation contrasting schemes is largely built upon the view generator, which limits the model generality and vulnerable to the noisy user behaviors. iii) Most of current GNN -bseed contrastive recommenders are limited by the over-smoothing issue which leads to indistinguishable representations.
In light of the above limitations and challenges, we revisit the graph contrastive learning paradigm for recommendation with a proposed simple yet effective augmentation method LightGCI. In our model, the graph augmentation guided by singular value decomposition (SVD) to not only dis- till the useful information of user-item interactions but also inject the global collaborative context into the representation alignment of contrastive learning. Instead of generating two handcrafted augmented views, important semantic of user-item interactions bust graph contrastive learning paradigm. This enables our self-augmented representations to be reflective of both user-specific preferences and cross-user global dependencies.
be well preserved with
Our contributions are highlighted as follows:
In this paper, we enhance the recommender systems by designing lightweight and robust graph contrastive learning framework to address the identified key challenges pertaining to this task.
We propose an effective and efficient contrastive learning paradigm LightGCL for graph aug- mentation. With the injection of glohal collaborative relations, our model can mitigate the issues brought by inaccurate contrastive signals.
Our method exhibits improved training efficiency compared existing GCL-based approaches.
Extensive experiments on several real-world datasets justify the performance superiority of our LightGCL. In-depth analyzes demonstrate the rationality and robustness of LightGCL.
2 RELATED WORK
Graph Contrastive Learning for Recommendation. A promising line of recent studies has in- corporated contrastive learning (CL) into graph-based recommenders, address the label sparsity issue with self-supervision signals. Particularly, SGL (Wu et al. 2021) and SimGCL (Yu et al., 2022a) perform data augmentation over graph structure and embeddings with random dropout oper- ations. However, such stochastic augmentation may drop important information, which may make the sparsity issue of inactive users even worse. Furthermore, some recent alternative CL-based rec- ommenders, such HCCF (Xia et al. 2022b) and NCL (Lin et al. (022). design heuristic-bart based strategies to construct view for embedding contrasting. Despite their effectiveness, their success heavily relies their incorporated heuristics (e.g.. the number of hyperedges user clusters) for contrastive view generation, which can hardly be adaptive to different recommendation tasks.
Self-Supervised Learning on Graphs. Recently, self- supervisdd learning (SSL) has advanced the graph learning paradigm by enhancing node representation from unlabeled graph data (Zhu et al. 202 b Velickovic al. 2019. Hassani & Khasahmadi] 2020 Peng et al.) 2020 Zhu al. 2020 Wu et al. 2022). For example, to improve the predictive SSL paradigm, AutoSSL (Jin et al. 2022) automatically combines multiple pretext tasks for augmentation. Towards the line of contrastive
SSL over graph structures, recent efforts focus on designing various graph contrastive learning methods (Yu al. 2022b, Yin et al.] 2022, Zhang et al. 2022, Xia al., [022a] Suresh et al., 2021). For instance, SimGRACE|XXaa et al. (2022a) proposes to generate contrastive views with the GNN encoder perturbations. In AutoGCL Yin et al. (202)), graph view generators are designed to be jointly trained with the graph encoder in an end-to-end way. Additionally, GCA (Zhu et al.) and attribute- data augmentation for contrastive view gen- eration. In this method, important edges and features will be identified for adaptive augmentation. GraphCL You et al. 2020) generates correlated graph representation views using various augmen- tation strategies, such node/edge perturbation and attribute masking.
2021b) performs both topology-
2

---

Published as a conference paper at ICLR 2023
Original Graph
GCNs
Users
1=1
Pairwise Loss for Recommendation
Lr
ApproxSVD
LonkyRLLU View
Items
Sq
VT
1=1
Reconstruct
Local-Global ((0)
1o Lo
Contrastive Learning
View
Figure
Overall structure LightGCL.
3 METHODOLOGY
In this section, we describe our proposed LightGCL framework in detail. LightGCL is lightweight graph contrastive learning paradigm as illustrated Fig. [ Complementary the GCN backbone (the upper half of the figure) extracting the local graph dependency, the SVD-guided augmentation (the lower half of the figure) empowers the graph contrastive learning with global collaborative relation analysis for learning effective user and item representations.
3.1 LOCAL GRAPH DEPENDENCY MODELING
As common practice collaborative filtering, we assign each user u, and item by with an embed- ding vector e!") ee
(v) RRd. where d is the embedding size. The collections of all user and item
embeddings are defined as E(") E TR. xxd and E(e) E Jxd where / and J are the number of users and items, respectively. Following Xia et al. (20226), we adopt a two-layer GCN to aggregate the neighboring information for each node. In layer I, the aggregation process is expressed follows:
<11) = ( E/D))
(n) j. (p(A..)) E(a)
where z/ and 2(() denote the (-th layer aggregated embedding for user u, and item Vy. (.) represents the LeakyRel.U with negative slope of 0.5. A is the normalized adjacency matrix, which we perform the edge dropout denoted as p(-), to mitigate the overfitting issue. We implement the residual connections each layer to retain the original information of the nodes follows:
(e)
(1) + 11-1 j4
11111
5.1
The final embedding for a node is the sum of its embeddings across all layers, and the inner product between the final embedding of user u, and an item v predicts u, 's preference towards uy:
(v) fd
(u)T (v)
Da.
e
3.2 EFFICIENT GLOBAL COLLABORATIVE RELATION LEARNING
To empower graph contrastive learning for recommendation with global structure learning, equip our LightGCL with the SVD scheme (Rajwade et al., 2012 Rangarajan 2001 to efficiently distill important collaborative signals from the global perspective. Specifically, the adjacency matrix A as A USVT. Here, U /V is an I I/J x J orthonormal matrix with columns being the eigenvectors of A's row-row / column-colume correlation matrix. S is an x diagonal matrix storing the singular values of A. The largest singular values usually associated
first perform SVD

---

Published as a conference paper at ICLR 2023
with the principal components of the matrix. Thus, we truncate the list of singular values to keep the largest q values, and reconstruct the adjacency matrix with the truncated matrices as À UUS.VVV V where Ug E xx and V2 E Ixxa contain the first q columns of U and V respectively. S E R is the diagonal matrix of the q largest singular values.
The reconstructed matrix À is low-rank approximation of the adjacency matrix A, for it holds that rA) q- The advantages of SVDD-based graph structure learning are two-folds. Firstly, emphasizes the principal components of the graph by identifying the user-item interactions that important and reliable to user preference representations. Secondly, the generated new graph structures preserve the global collaborative signals by considering each user-item pair. Given the A, we perform message propagation on the reconstructed user-item relation graph in each layer:
(w) 1 = cAA. E(")).
5 - (AA.
(4)
However, performing the exact SVD on large matrices is highly expensive, making it impractical for handling large-scale user-item matrix. Therefore, adopt the randomized SVD algorithm proposed by Halko et al.] (201T), whose key idea is to first approximate the range of the input matrix with a low-rank orthonormal matrix, and then perform SVD this smaller matrix.
Ug. S.y V. 9) ASVD BSVD=Ug U,,5
where q is the required rank for the decomposed matrices, and U, R2x S, E Rox4 V, E R are the approximated versions of Ug. Sy. Vg. Thus, we rewrite the message propagation rules in Eq. 4 with the approximated matrices and the collective representations of the embeddings as follows:
G ar o'Asv o (U, E 22)): G(-) o(AAv E (1) = a V, E
where G(u) and G( the collections of user and item embeddings encoded from the new gen- erated graph structure view. Note that we do not need to compute and store the large dense matrix AsvD. Instead, we can store Ug. Sq and Vq. which are of low dimensions. By pre-calculating (U,S,) and (V,S,) during the preprocessing stage with SVD, the model efficiency is improved.
3.3 SIMPLIFFED LOCAI-GI.OBAL CONTRASTIVE LEARNING
The conventional GCL methods such as SGL and SimGCL contrast node embeddings by construct- ing two extra views, while the embeddings generated from the original graph (the main-view) are not directly involved in the InfoNCE loss. The reason for adopting such cumbersome three-view paradigm may be that the random perturbation used to augment the graph may provide mislead- ing signals to the main-view embeddings. In proposed method, however, the augmented graph view is created with global collaborative relations, which can enhance the main-view representa- tions. Therefore, simplify the CL framework by directly contrasting the SVD-augmented view embeddings g with the main- view embeddings 2(()) in the InfoNCE loss (Oord et al. 2018):
(w)
exp(s(z 9.. /T))
log
El exp(s(2 (0) .g.)))/)
-=
1=
where s(.) and J stand for the cosine similarity and the temperature respectively. The InfoNCE loss c(¹) for the items are defined in the same way. To prevent overfitting, we implement a random node dropout in each batch to exclude some nodes from participating in the contrastive learning. As shown in Eq. 88 the contrastive loss is jointly optimized with our main objective function for the recommendation task (where Vi.p, and Di,n, denote the predicted scores for pair of positive and negative items of user i):
S L = L, + A1 (C(()) + -(())) + d2 |1013 -, = Σ>
I
max(00 1 Øt.p. + Di.n.) (8)
1=0 am1
4

---

Published as a conference paper at ICLR 2023
4 EVALUATION
To verify the superiority and effectiveness of the proposed LightGCL method, we perform extensive experiments to answer the following research questions:
RQ1: How does LightGCL perform on different datasets compared to various SOTA baselines?
RQ2: How does the lightweight graph contrastive learning improve the model efficiency?
RQ3: How does our model perform against data sparsity, popularity bias and over-smoothing?
RQ4: How does the local-global contrastive learning contribute to the performance of our model?
RQ5: How do different parameter settings affect our model performance?
4.1 EXPERIMENTAL SETTINGS
4.1.1 DATASETS AND EVALUATION PROTOCOLS
We evaluate our model and the baselines on five real-world datasets: Yelp (29,601 users, 24,734 items, 1,517,326 interactions): a dataset collected from the rating interactions on Yelp platform; Gowalla (50,821 users, 57,440 items, 1,172,425 interactions): a dataset containing users' check-in records collected from Gowalla platform; ML-10M (69,878 users, 10,195 items, 9,988,816 interac- tions): well-known movie-rating dataset for collaborative filtering; Amazon-book (78,578 users, 77,801 items, 2,240, 156 interactions): a dataset composed of users' ratings on books collected from Amazon; and Tmall (47,939 users, 41,390 items, 2,357,450 interactions): E-commerce dataset containing users' purchase records on different products in Tmall platform.
In accordance with He et al. (2020) and Wu et al. (021), we split the datasets into training, vali- dation and testing sets with a ratio of 7:2:1. We adopt the Recall@N and Normalized Discounted Cumulative Gain (NDCG)@M N, where N (20, 40), as the evaluation metrics.
4.1.2 BASELINE METHODS
We compare our model against 16 statc-of-the-at baselines with different learning paradigms:
MLP-enhanced Collaborative Filtering: NCF (He et al.
2017).
based Collaborative Filtering: GCCF (Chen
2020ch.
LightGCN
(He et al. 2020)
Disentangled Graph Collaborative Filtering: DGCF (Wang et al.
2020b).
Hypergraph-bane based Collaborative Filtering: HyRec (Wang et al.
2020a).
Self- Super-uuer Learning Recommender Systems: Graph CL
Yoou et al. 2020), GRACE ZZu
jet al. 2020), GCA (Zhu al. 2021b), MHCN (Yu al.
2021). SAIL Yu al.
2022b Au-
toGCL (Yin et al., (2022). SimGRACE (Xia et al.) 2022a).
SGL (Wu et al.
2021)
HCCF
Xia
ct al. 2022b), SHT (Xia al. 2022c), SimGCL (Yu et al.
2022a).
Due to space limit, the detailed descriptions of baselines are presented in Appendix A
4.1.3 HYPERPARAMETER SETTINGS
To ensure fair comparison, we tune the hyperparameters of all the baselines within the ranges sug- gested in the original papers, except the following fixed settings for all the models: the embedding size is set as 32; the batch size 256; two convolutional layers are used for GCN models.
For our LightGCL, the regularization weights A and λ2 tuned from ( (le-5, le-6, 1e-7} and (le- 4, le-5), respectively. The temperature 7 is searched from (0.3, 0.5, 1, 3 ,10). The dropout rate is chosen from {0, 0.25). The rank (i.e., q) for SVD, is set 5. We use the Adam optimizer with learning rate of 0.001 decaying at the rate of 0.98 until the rate reaches 0.0005
4.2 PERFORMANCE VALIDATION (RQ1)
We summarize the experimental result Table with the following observations and conclusions:
More detailed parameter settings can be found in our released source code.
Due space limit, results of NCF, GCCF, GraphCI., SAII., GRACE, and AutoGCL. in Appendix
5

---

Published as a conference paper at ICLR 2023
Table 1: Performance comparison with baselines on five datasets.
Data
Metric
DGCF
HyRec
LigheCCC
MHCN
SGL.
SimcCRACE
GCA
HCCF
SHT
SimGCT.
LighiGCI
p-vol.
Ingic
R@20
0.0466
0.0472
0,0482
0,0503
0,0526
0.0603
0.0621
0.0626
0.0651
0,0718
0,0793
7e-9
10%
Yelp
N@
0.0395
0.0395
0,0409
0.0424
0.0444
0.0435
0,0530
0.0527
0.0546
0.0615
0.0668
8c
8%:
RG40
0.0774
0.0791
0,0803
0.0826
0.0869
0.0989
0.1021
0.1040
0.1166
0.1292
2c-9
NG-40
0.0511
0.0522
0.0527
0.0544
0.0571
0.0656
0.0677
0.0681
0.0709
0.0778
0.0852
2e. 9
9%.
Gowalin
R
0.0944
0.0901
0,0985
0.0955
0.1030
0.0869
0.0896
0.1070
0.1232
0.1357
0.1578
1e.6
N@
0.0522
0.0498
0.0593
0.0574
0.0623
0.0528
0.0537
0.0644
0.0731
0
0.0935
26-6
14%
K440
1401
0.1356
0.1431
0.1393
0.1500
0.1276
0.1322
0.1535
0.1804
0.1956
0.2245
36-6
14%
N--0
0.0671
0.0660
0,0710
0.0689
0,0746
0.0637
0.0651
0.0767
0,0881
0.0975
0.1108
36-6
13%
M1. 10M
20
0.1763
0.1801
0.1789
0.1497
0.1833
0.2254
0.2145
0.2219
0.2173
0.2265
0.2613
1c-9
13%
N
0.2101
0.2178
0.2128
0.1814
0.2205
0.2686
0.2613
0.2629
0.2573
0.2613
0.3106
3e-9
18%
R6:40
0.2681
0.2685
0.2650
0.2250
0.2768
0.3295
0.3231
0.3265
0.3211
0.3345
0.3799
1e:10
13%
N@-40
0.2340
0.2340
0.2322
0.1962
0.2426
0.2939
0.2871
0,2880
0.3318
0,2880
0.3387
1c-9
17%
Amazon
20
0.0211
0.0302
0,0319
0.0296
0.0327
0.0381
0.0309
0.0322
0.0441
0.0474
0.0585
20-7
23%
N
0.0154
0.0225
0.0236
0.0219
0.0249
0.0238
0.0247
0.0328
0.0360
0.043
2c-6
21%
K640
0.0351
0.04322
0,04999
0.0489
0.0531
016621
0X4498
0.0325
0,0719
0.0750
0.0933
1e-7
24%
N@40
0.0201
0.0246
0.0290
0.0284
0.0312
0.0371
0.0301
0.0314
0.0420
0.0451
0.0551
9c-7
22%
R
0.0235
0.0233
0.0225
0,0203
0.0268
0.0222
0.0373
0.0314
0.0387
0,0473
0,0528
3e-5
11%
Tmall
N@
0.0163
0.0160
0.0154
0.0139
0.0183
0.0152
0,0252
0.0213
0.0262
0.0328
0.0361
1c-4
10%
RG40
0.0394
0.0350
0,0378
0.0340
0,04446
0.0367
0.0616
0.0519
0.0644
0.0766
0.0852
le:5
11%
N@040
0.0218
0.0199
0.0208
0.0188
0.0246
0.0203
0.0337
0.0284
0.0352
0.0429
0.0473
7e-5
10%
Contrastive Learning Dominates. As can be seen from the table, recent methods implementing contrastive learning (SGL, HCCF, SimGCL) exhibit consistent superiority as compared to tradi- (HyRec) models. They also perform
tional graph-based (GCCF, LightGCN) or hypergraph- better than some of other self-supervised learning approaches (MHCN). This could be attributed the effectiveness of CL to learn evenly distributed embeddings (Yu et al. 2022a).
Contrastive Learning Enhancement. Our method consistently outperforms all the contrastive learning baselines. We attribute such performance improvement to the effective augmentation of graph contrastive learning via injecting global collaborative contextual signals. Other compared contrastive tearning-base recommenders (e.g.. SGL., SimGCI, and HCCF) are easily biased by noisy interaction information and generate misleading self-supervised signals.
4.3 EFFICIENCY STUDY (RQ2)
GCL models often suffer from a high computational cost due to the construction of extra views and the convolution operations performed on them during training. However, the low-rank nature of the SVD-reconstructed graph and the simplified CL structure enable the training of our LightGCL to be highly efficient. We analyze the pre-processing and per-batch training complexity of our model in comparison to three competitive baselines, as summarized in Table 2
Table 2: Comparisons of computational complexity against baselines.
Stage
Computation
LighTGCN
SGL
SimOCL
LighTGCL
Normalization
O(E)
O(E)
O(E)
O(E)
Pre-processing
SVD
O(gE)
Augmentation
O(2pE)
Graph Convolution
((2ELd) O(2
O(2ELd + 4pELd)
O(GELd)
Q 2g(1 Ld]
Training
BPR Loss
O(2Bd)
O(2Bd)
O(2Bd)
InfoNCE Loss
O + BMd)
O(Bd+ BMd)
O((Bd + BMd)L)
Although our model requires performing the SVD in the pre-processing stage which takes (4 F), the computational cost is negligible compared to the training stage since it only needs to be per- formed once. In fact, by moving the construction of contrastive view to the pre-processing stage, we avoid the repetitive graph augmentation during training, which improves model efficiency.
Traditional GCN methods (c.g., LightGCN) only perform convolution on one graph, inducing complexity of ((ELdd) per batch. For most GaL-b methods, three contrastive views are
'In the table, E, L and d denotes the edge number, the layer number and embedding size; E (0, 1] is the edge keep rate; 9 is the required rank; 7 and J represents the number of users and items; B and M are the batch size and node number in hatch. Detailed calculations are shown in Appendix D

---

Published as a conference paper at ICLR 2023
computed per batch, leading to a complexity of roughly three times of LightGCN. In model, instead, only two contrastive views are involved. Additionally, due to the low-rank property of SVD-based graph structure learning, our graph encoder takes only O]2q(I + Jd time. For most datasets, including the five we use, 2g(7 + J) < F.. Therefore, the training complexity of our model is less than half of that of the SOTA efficient model SimGCL.
4.4 RESISTANCE AGAINST DATA SPARSITY AND POPULARITY BIAS (RQ3)
To evaluate the robustness of our model in alleviating data sparsity, we group the sparse users by their interaction degrees and calculate the Recall@ 20 of each group on Yelp and Gowalla datasets. As can be seen from the figures, the performance of HCCF and SimGCL varies across datasets, but our LightGCL consistently outperforms them in all cases. In particular, our model performs notably well on the extremely sparse user group (< 15 interactions), as the Recall@ 20 of these users is not much lower (and is even higher on Gowalla) than that of the whole dataset.
Yelp
Gowella
0.00
#HCOF
SIWOCL LiGHTCL
0.39
SemoOL
Yelp
Gowalla
Recali220
0.08
#HOCF SErGZE WLgGHSCL
HCCF #Sm50 = LightG
0.07
0.00122
0.0080
0.08
0.09
Recalie:2
0.0030
0.0100
0.05
0.0040
0.04
0.0000
0.0020
0-15
15-20
20-25
25-30 30-35
0-15
20-25
30-35
0.0000
0.0500
o. Rocalle200
- HCC7 - SemOOL -M--L0100L
Sin
0. 0.0060
JR of Senelliee0
6.0004
1224
10%
6.0002
0.0040
-30%
10%
0.55 2500
2013
znga 30yl
0.0000
0-35 10-20-30--0 25-33-2552223333330 30-35
0.0000
0.0000
0-15 15-3020-35
-9%
0:15
1520
20-25
25-30
30-35
Figure 3: LightGCL's ability to alleviate popularity bias in comparison to SOTA CL- based methods HCCF and SimGCL.
Figure 2: Performance on users of different spar- sity degrees, in terms of Recall (histograms) and relative Recall w.E. overall performances (charts).
Additionally, illustrate our model's ability to mitigate popularity bias compared to HCCF and SimGCL. Similar to Section[4.4] we group the long-tail items by their degree of interactions. Follow-
ing Wu et al. (021)). 1,, we adopt the decomposed Recall @ 20 defined as Recall's) 10
1V
where Viese refers to the set of test items for the user u, and (Vree)(s) )()) is the set of Top-K recom- mended items for u that belong to group g. The results are shown in Fig. 3 Similar to the results on sparse users, HCCF and SimGCL's performance fluctuates a lot with the influence of popularity bias. Our model performs better in most cases, which shows its resistance against popularity bias. Note that since the extremely sparse group (< 15 interactions) is significantly larger than the other groups in Gowalla, they contribute to large fraction of the Recall 20, resulting in different trend from that of Yelp in the figure.
4.5 BALANCING BETWEEN
AND EVER-UNIFORMIN (RQ3)
In this section, we illustrate the effectiveness of our model in learning moderately dispersed em- bedding distribution, by preserving user unique preference pattern and inter-user collaborative de- pendencies. We randomly sample 2,000 nodes from Yelp and Gowalla and map their embeddings to the 2-D space with t-SNE (Van der Maaten & Hinton 2008). The visualizations of these embed- dings are presented in Fig. We also calculate the Mean Average Distance (MAD) (Chen et al. 2020a) of the embeddings, summarized in Table 3
Table 3: Mean Average Distance (MAD) of the embeddings learned by different methods.
Dataset
MHCN
LightCCC
LightGCL
SimGCL 0.9657
SGL
Yelp Gewalla
0.8806
0,9469
0.9962
0.9956 0.9859
0.9247
0.9568
0.9721
0.9897
7

---

Published as a conference paper at ICLR 2023
MHCN
LightGON
LightGC
Sim6CL
Yelp
Gowalla
Figure 4: Embedding distributions on Yelp and Gowalla visualized with t-SNE.
As can be seen from Fig. the embedding distributions of non-CL methods (i.e., LightGCN, MHCN) exhibit indistinguishable clusters in the embedding space, which indicates the limitation of addressing the over-smoothing issue. On the contrary, the existing CL-based methods tend to learn over-uniform distributions, e.g., SGL Yelp learns huge cloud of evenly-distanced embed- dings with no clear community structure to well capture the collaborative relations between users; ii) highly dispersed small clusters with severe over-smoothing issue inside the clusters, e.g., the em- beddings of SimGCL on Gowalla appear to be scattered grained clusters inside which embeddings are highly similar. Compared with them, clear community structures could be identified by our method to capture collaborative effects, while the embeddings inside each community are reason- ably dispersed to be reflective user-specific preference. The MAD of our model's learned features is also in between of the two types of baselines as shown in Table 3
4.6 ABLATION STUDY (RQ4)
To investigate the effectiveness of our SVD-based graph augmentation scheme, we perform the ab- lation study answer the question of whether we could provide guidance the contrastive learning with a different approach of matrix decomposition. To this end, we implement two variants of our model, replacing the approximated SVD algorithm with other matrix decomposition methods: CL- MF adopts the view generated by a pre-trained MF (Koren et al. 2009); CL-SVD+ utilizes the SVD++ (Koren 2008) which takes implicit user feedback into consideration. As shown in Table 4 with the information distilled from MF or SVD++, the model is able to achieve satisfactory results, indicating the effectiveness of using matrix decomposition to empower CL. and the flexibility of our proposed framework. However, adopting a pre-trained CL component is not only tedious and time- consuming but also inferior to utilizing the approximate SVD algorithm in terms performance.
0.002
Recillgoo
0.002
0.000
Table 4: Ablation study on LightGCL.
0.000
Yelp
Gowalla
0.000
Variant
2ocall@
NDCG@20
2
NDCG@
0.043
CL-MF
0.0781
0.0659
0.1561
0.0929
0.000
CL-SVD++ LightGCL
0.0788
0.0666
0.1568
0.0932
0.000
0.0793
0.0668
0.1578
0.0935
Figure 5: Recall change w.E.f. g.
4.7 HYPERPARAMETER ANALYSIS (RQ5)
In this section, we investigate our model's sensitivity in relation to several key hyperparameters:: the regularization weight for InfoNCE loss '₁ the temperature 7, and the required rank of SVD q.
The impact of A1. As illustrated in Fig. 6 for the three datasets Yelp, Gowalla and ML-IOM. the model's performance reaches the peak when X₁ = 10-7 It can be noticed that X with the range of [10-6; 10-8] can often lead to performance improvement.
8

---

Published as a conference paper at ICLR 2023
bomell
MIL-UMM
Tdlp
0.000
Clrvalll
ML-SOM1
5079
1.100
0221
8350
5077
0.100
Acalbis 1000
8:25
Roull
6.40
0.20
maag
0473
9:00
0.38
088
8.000
8.20
507%
2000
6.500
0.26
100
6.000
0.01
let lnd let
Se4 3ec Se4 ter 244
244 Led
Figure 6: Impact of Å1-
Figure 7: Impact of T
The impact T. r. Fig. 7 iindicates that the model's performance is relatively stable across different selections of r from 0.1 to 10, while the best configuration of T value varies by datasets.
The selection of q- q determines the rank of SVD in our model. Experiments have shown that satisfactory results can be achieved with a small Specifically, as in Fig. [ we observe that
q = 5 sufficient to preserve important structures of the user-item interaction graph.
4.8
CASE STUDY (RQ4)
In this section, we present case study to intuitively show the effectiveness of our model to identify useful knowledge from noisy user-item interactions and make accurate recommendations accord- see that the venues visited by user #26 in Yelp mainly fall into two commu- nities: Cleveland (where the user probably lives) and Arizona (where the user may have travelled to). In the reconstructed graph, these venues assigned new weight according to their potential importance. Note that item #2583, a car rental agency in Arizona, has been assigned a negative weight, which conforms to our common sense that people generally would not visit multiple rental agencies in one trip. The SVD-augmented view also provides predictions on invisible links by assigning a large weigh to potential venues of interest, such #2647 and #658. Note that when exploiting the graph, the augmented view does not overlook the smaller Arizona community, which enables the model to predict items of minor interests that are usually overshadowed by the majority.
ingly. In Fig. 8
Ceveland
the interaeted origiaal grapt.
mude Glewaed
Purtac Crrelon
pork Clwiaal
Chedladd
Chullard
15'
lank SVD coconammtted
link origical graph
grugh with weight w
Artd
FEET
prediced
Inutiee
cateerr
2321
maseam
Chiellnd 0.1100
High' Cenlled
2559
Develed
shepping
0.294
2501
Greeked
2520
Clvillnd
p0ra
Other Ramrs..
develedd
sneepryg
Figure 8: Case study on user #26 in Yelp dataset.
5 CONCLUSION
In this paper, we propose a simple and effective augmentation method to the graph contrastive learn- ing framework for recommendation. Specifically, we explore the key idea of making the singular value decomposition powerful enough augment user-item interaction graph structures. Our key findings indicate that our graph augmentation scheme exhibits strong ability in resisting data spar- sity and popularity bias. Extensive experiments show that our model achieves new state-of-the-an art results on several public evaluation datasets. In future work, we plan to explore the potential of incorporating casual analysis into our lightweight graph contrastive learning model to enhance the recommender system with mitigating confounding effects for data augmentation.
$ Due to the fully connected nature of the seD-rrccostteeeted graph, the weights of unobserved interactions in the graph are of smaller magnitude. weight of 0.01 is already large weight in the graph.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topologiaal view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.
2020a.
Lei Chen, Le Wu, Richang Hong, Kun Zhang, and Meng Wang. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. In AAAI conference on artificial intelligence, volume 34, pp. 27-34, 2020b.
Lei Chen, Le Wu, Richang Hong. Kun Zhang, and Meng Wang. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 27-34, 2020c.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53 (2):217-288. 2011.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning (ICML), pp. 4116-4126. PMLR, 2020.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collab- orative filtering. In International conference on world wide web (WWW), pp. 173-182, 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang- Lightgen: Simplifying and powering graph convolution network for recommendation. In International con- ference on research and development Information Retrieval (SIGIR), pp. 639-648, 2020.
Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self- supervised learning for graphs. ICLR, 2022.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 126-434, 2008.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Xixun Lin, Jia Wu, Chuan Zhou, Shirui Pan, Yanan Cao, and Bin Wang. Task-adaptive neural process for user cold-start recommendation. In Proceedings of the Web Conference (WWW), pp- 1306-1316, 2021.
Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. Improving graph collaborative filter- ing with neighborhood--eniihel contrastive learning. In Proceedings of the ACM Web Conference (WWW). pp. 2320-2329, 2022.
Fan Liu, Zhiyong Cheng, Lei Zhu, Zan Gao, and Ligiang Nie. interest-aware message- paassagg-psssinggg gen for recommendation. In The Web Conference (WWW), pp. 1296- 1305, 2021a.
Weiming Liu, Jiajie Su, Chaochao Chen, and Xiaolin Zheng. Leveraging distribution alignment via stein path for cross-domain cold-start recommendation. Advances in Neural Information Process- ing Systems (NeurIPS), 12221999222994,, 2021b.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 03748, 2018
Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In The Web Conference (WWW), pp. 259-270, 2020.
Ajit Rajwade, Anand Rangarajan, and Arunava Banerjee. Image denoising using the higher order singular value decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4):849-86) 2012.
10

---

Published as a conference paper at ICLR 2023
Anand Rangarajan. Learning matrix space image representations. In International Workshop En- engy Minimization Methods in Computer Vision and Pattern Recognition, pp. 153-168. Springer, 2001.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems (NeurlPS), 34: 15920-15933, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Petar Velickovie, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR, 2(3):4, 2019.
Jianling Wang, Kaize Ding, Liangjie Hong, Huan Liu, and James Caverlee. Next-item recommenda- tion with sequential hypergraphs. In Proceedings of the 43rd international ACM SIGIR conference research and development in information retrieval, pp. 1 1101-1110, 2020a.
Xiang Wang, Xiangnan He, Meng Wang. Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In International Conference Research and Development in Information Retrieval (SIGIR), pp. 165-174, 2019.
Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. Disentangled graph collaborative filtering. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pp. 1001-1010, 2020b.
Jiancan Wu, Xiang Wang. Fuli Feng, Xiangnan He, Liang Chen, Jianxum Lian, and Xing Xie. Self- supervised graph learning for recommendation. In International conference research and development in information retrieval (SIGIR), pp. 726-735, 2021.
Zhebin Wu, Lin Shu, Ziyue Xu, Yaomin Chang, Chuan Chen, and Zibin Zheng. Robust tensor graph convolutional networks via t-svd based graph augmentation. In International Conference Knowledge Discovery and Data Mining (KDD), pp. 2090-2099, 2022.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In the ACM Web Conference (WWW), pp. 1070-1079, 2022a.
Lianghao Xia, Chao Huang, Yong Xu, Jiashu Zhao, Dawei Yin, and Jimmy Xiangji Huang. Hy- pergraph contrastive collaborative filtering. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR 2022, Madrid, Spain, July 11-15, 2022., 2022b.
Lianghao Xia, Chao Huang, and Chuxu Zhang. Self-supervised hypergraph transformer for recom- mender systems. In International Conference on Knowledge Discovery and Data Mining, KDD 2022, Washington DC, USA, August 14-18, 2022., 2022c.
Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. Contrastive learning for sequential recommendation. In International Conference Data Engineering (ICDE), pp. 1259-1273. IEEE, 2022.
Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogel: Automated graph contrastive learning via learnable view generators. In AAAI Conference on Artificial Intel- ligence (AAAI), volume 36, pp. 8892-8900, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang. and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems (NeurIPS), 33:5812-5825 2020.
Junliang Yu, Hongzhi Yin, Jundong Li. Qinyong Wang, Nguyen Quoe Viet Hung, and Xiangliang Zhang. Self-supervised chati- hypergraph convolutional network for social recommenda- tion. In Proceedings of the Web Conference 2021, pp. 413 2021.
11

---

Published as a conference paper at ICLR 2023
Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoe Viet Hung Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In Inter- national Conference Research and Development in Information Retrieval (SIGIR), pp. 1294- 1303, 2022a.
Lu Yu, Shichao Pei, Lizhong Ding, Jun Zhou, Longfei Li, Chuxu Zhang, and Xiangliang Zhang. Sail: Self-augmented graph contrastive learning. In AAAI Conference on Artificial Intelligence (AAAI), volume 36, pp. 8927-8935, 2022b.
Jiani Zhang, Xingjian Shi, Shenglin Zhao, and Irwin King. Star-gen: Stacked and reconstructed graph convolutional networks for recommender systems. International Joint Conference on Arti- ficial Intelligence (ICCAI), 2019.
Yixin Zhang, Yong Liu, Yonghui Xu, Hao Xiong, Chenyi Lei, Wei He, Lizhen Cui, and Chunyan Miao. Enhancing sequential recommendation with graph contrastive learning. International Joint Conference on Artificial Intelligence (IJCAI), 2022.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04/// 2020.
Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. arXiv preprint www.:i2219001116000 2021a.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In The Web Conference (WWW), pp. 2069-2080, 2021b.
12

---

Published as a conference paper at ICLR 2023
A
DETAILS OF THE BASELINES
MLPeehhanced Collaborative Filtering:
NCF (He et al. 2017) is collaborative filtering model that leverages neural network to exploit non-linearity. Two hidden layers are used our evaluation.
GNN-based Collaborative Filtering:
GCCF (Chen et al., 2020c) strengthens the GNN-based collaborative filtering by implementing residual network and reducing the non-linear transformation.
LightGCN (He al. 2020) adopts simplified GCN structure without embedding weight matri-
ces and non-linear projection.
Disentangled Graph Collaborative Filtering:
DGCF (Wang et al. 2020b) learns a more sophisticated representation by segmenting the embed- ding vectors to represent multiple latent intentions.
Hypergraph-baseed Collaborative Filtering:
HyRec (Wang et al. 2020a) makes of hypergraph encode multi-order information between users and items.
Self-Supervised Learning Recommender Systems:
GraphCL (You et al. 2020) utilizes random node dropping and edge masking to generate two contrastive views, which were aligned by optimizing the SSL loss function.
GRACE (Zhu
2020) proposes corrupt the graph structure by both random edge dropout
and random node feature dropping, and uses the corrupted graphs as the contrastive views.
GCA (Zhu al.
2021b) adaptively dropout the nodes and edges their importance calculated
with node centrality.
MHCN Yu et al.
2021
creates self-supervised signals for the graph representation learning by
graph infomax network.
SAIL Yu et al.
2022b
maximizes the neighborhood predicting probability between GNN-
generated high-level features and input node features.
2022) uses GNN learn to mask nodes and edges the augmented graph. It minimizes the similarity between the augmented and the original graph, while maximizing the similarity of the embeddings generated through them, so as to uncover the most important information in the graph.
AutoGCL (Yin et al.,
SimGRACE (Xia et 2022a) creates augmented view by randomly perturbing the parameters of the GNN network.
SGL Wu et al. 2021 adopts random walk sampling and probabilistic edge/node dropout to create augmented views for contrastive learning. In our experiments, we adopt the SGL-ED variant, which implements random edge dropout and exhibits the strongest performance according the original paper.
HCCF (Xia et al. 2022b) encodes global graph information with hypergraph and contrasts it against the local information encoded with GCN. In our experiments, the number hyper-edges are set 128 following the original paper.
SHT et al. 2022c) adopts hypergraph transformer framework exploit global collaborative relationships and distills the global information to generate the gross-view self-supervised signals. our experiments, the number of hyper-edges are set as 128 following the original paper.
SimGCL (Yu et al. 2022a) propose to simplify the graph augmentation process of contrastive learning by directly injecting random noises into the feature representation.
13

---

Published as a conference paper at ICLR 2023
B PERFORMANCE COMPARISON WITH BASELINES (CONTINUED)
In this appendix, we show the performance of NCF, GCCF, GraphCL, SAIL, GRACE, and Auto- GCL, which are not shown in Table due to space limit. The results are summarized in Table 5 As can be seen from the table, our model outperforms these baselines consistently.
Table 5: Performance comparison with baselines five datasets (continued).
Data
Metric
NCF
GCCF
GraphCL
SAIL
GRACE
AutoGCL
LightGC
R
0.0252
0.0462
0.0462
0.0471
0,0550
0.0593
0.0793
Yelp
0.0202
0.0398
0.0405
0.0494
R
0.0487
0.0760
0.0764
0.0713
0.0917
0.1009
0.1292
N@40
0.0289
0.0508
0.0511
0.0516
0.0605
0.0650
0.0852
R@
0.0171
0.0951
0.0997
0.0999
0.0744
0.0832
0.1578
Gowalla
N@
0.0106
0.0535
0.0603
0.0002
0.0452
0.0484
0.0938
R
0.0216
0.1392
0.1291
0.2245
No-0
0,0118
0.0684
0.0727
0.0725
0.0539
0.0605
0.1108
R
0.1097
0.1742
0.1728
0.2525
0.2613
MI.-I0M
N@
0.1297
0.2109
0.2038
0.2118
0,2476
0.2755
0.3106
KG:40
0.1634
0.2006
0.2619
0.3415
0.3799
N@40
0,1427
0.2351
0.2250
0.2332
0,2711
0.3023
0.3387
R
0.0142
0.0317
0.0360
0.0357
0.0300
0.0325
0.0585
2@
0,0085
0.0243
0.0266
0.0264
0.0271
0.0241
0.0436
R
0.0223
0.0483
0,0585
0.0581
0,0583
0.0553
0.0933
N@40
0.0133
0.0285
0.0300
0.0338
0.0345
0,0318
0.0551
R
0.0082
0.0209
0.0251
0.0254
0.0303
0.0312
0.0528
Tmall
0.0059
0.0141
Ror40
0.0140
0.0356
0.0416
0.0424
0,0S0S
0.0524
0.0852
N@:0
0.0079
0.0196
0.0236
0.0278
C THEORETICAL ANALYSIS
We conduct theoretical analyses to show that our local-global CL. (Eq. 7 augmented to maximize the similarity between embeddings of potentially related nodes, based the SVD-based global relation learning. Specifically, for node v, U, where 4 {up |A,, 0, A., / * 0}, the embeddings are not updated by s g..) in the vanilla InfoNCE loss, as vy is not adjacent to u. Instead, our local-gu occalggobbll contrastive assigns the following gradients to the embeddings of uy:
Ös
2 0
Σ (
18g
jeU
A0,
Zi,1
Oii.j
| giall
where oi,, denotes the normalization weight for node u, and uy. In this way, the embeddings of nodes in M are also pulled close to 8i,, which injects relatedness information learned by the SVD into the local-global CL optimization.
D CALCULATION OF COMPLEXITY
D.1 ADJACENCY MATRIX NORMALIZATION
For a sparse user-item matrix stored the Coordinate Format (COO), requires visiting every non- zero elements in the matrix to perform normalization. Thus, the computational complexity is in the order of the number edges OEE). Note that for the baseline SGL, it requires normalizing the two augmented graph structures during the training phase, each of which contains pE edges, induces a complexity of (()pF)) per batch.
D.2 APPROXIMATE SVD ALGORITHM
We refer the readers to Halko et al. (001T) in which the complexity of the approximate SVD algo- rithm is explained in detail.
14

---

Published as a conference paper at ICLR 2023
D.3 GRAPH CONVOLUTION
Given a sparse COO matrix A with E edges and a dense matrix E with dimensions [(J) x d, it takes ((Ed) time calculate AE. To perform graph convolution a graph, need multiply the sparse adjacency matrix with E E I and its transpose with E E R which takes (EE)) each, and ((2dd) in total, For L layers, O(2ELd) is required. For traditional CL-based methods such SGL and SimCGL, three-view structure is adopted, resulting in complexity of ((1EEdd) (for SGL it again varies a bit depending on p).
For the SVD-view of model, V. E takes O(gJd), and multiplying the result with the pre- calculated SQ, takes O(gId); U 1 takes (gId)), and multiplying the result with the pre- calculated (V, takes (g.dd). So in total it takes (2 + J)d).
D.4 BPR LOSS
In each batch with users, calculating the scores for positive and negative items both take O(B)), so in total it takes O(2Bd).
D.5
CL LOSS
In each batch with B users, calculating the numerator of InfoNCE loss takes (BB)), and calculating the denominator takes O(BMd) where M denotes the total number of nodes in the batch. Since our model adopts a per layer InfoNCE loss, a factor of L is appended.
15


Reviews:
[{"review_id": "ztwnHFitOE", "reviewer": "Reviewer_rCJG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel method for advancing recommender system with a lightweight and effective graph contrastive learning paradigm. The proposed method is technically sound from a novel perspective of efficient SVD-guided graph augmentation. To demonstrate the effectiveness of the new framework, 10 baselines are compared.", "summary_of_the_review": "This paper tackles important challenges of graph contrastive learning in recommender systems. I acknowledge the technical novelty and empirical evidence of this work.", "strength_and_weaknesses": "Strong points: \n1.The proposed method is well-motivated and technically sound, by addressing key challenges of current GCL-based recommender systems to automatically generate augmented graph view for effectively incorporating self-supervised learning signals.\n2. A lightweight graph augmentation model is built over an efficiency SVD-based graph structure learning framework.\n3.Comprehensive experiments show the superiority of the proposed method over SOTA baselines.\n4. In-depth model evaluation demonstrates the advantage of the LightGCL approach for alleviating data sparsity, over-smoothing issue, and popularity bias.\n\nWeak points:\n1. For the compared SGL baseline method, three different augmentation schemes are introduced, e.g., node/edge dropout operators, as well as random walk-based sampling approach. The details of configuring the augmentation operators in SGL method can be further described.\n2. For the proposed approach, its advantage lies in the automated graph augmentation via distilling important graph structural information using a decomposition-based method. Compared with baseline SHT, the advantage of LightGCL can be further clarified.\n3. The results need further explanation. For instance, why the methods exhibit different trends on Yelp and Gowalla (Fig. 3).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "wA3B2Cf54D7", "reviewer": "Reviewer_3y1q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new SVD-based Graph Contrastive Learning paradigm called LightGCL to learn effective representations of nodes in a user-item interaction graph from unlabeled data that are eventually used to predict users' preferences in downstream recommendation tasks. The key idea is  in the augmentation step where an alternate low-rank view of the original graph is generated by applying SVD on the original graph. The embeddings of each node from two graphs are trained to be similar by minimizing Info Noise Contrastive Loss along with the prediction loss on user-item interactions. \n\nThe proposed approach LightGCL is evaluated against SOTA methods from the various paradigms including: MLP-based Collaborative Filtering (CF), GNN-based CF, Disentangled Graph CF,  Hypergraph-based CF, and self-supervised recsys on five real-world datasets on recall@N and NDCG@N. The experiments indicate superior performance of LightGCL over all the above baselines. Additionally. authors demonstrate LightGCL outperforming a couple of baselines (HCCF and SimGCL) on less popular/interacting segments of items and users. ", "summary_of_the_review": "The paper presents an interesting approach for generating alternate view of the graph that could be a useful addition to Graph Contrastive Learning literature. The results look impressive overall, however the comparisons with some of the important papers are missing. EDIT: I am satisfied with author's follow-up edits to the paper and they reasonably address most of my comments. In light of the same, I have updated my rating for the paper. ", "strength_and_weaknesses": "Strengths:\n1. The idea of using low-rank SVD to produce augmenting view of the original graph looks neat and well-aligned with the intuition of preserving global structure in the original graph. \n\n2. The results indicate significant improvement in recall@k and ndcg@k with respect to all the baselines on all the datasets. Additionally LightGCL is able to produce good performance on sparse users with respect to the two baselines, SimGCL and HCCF. \n\n3. LightGCL is computationally more efficient to train than the above baselines. \n\n\nWeaknesses: \n1. I did not understand why related work is missing several relevant papers from the domain of Graph Contrastive Learning [1-7]. Many of these papers are well-cited and published at top-tier conference. \n2. The methods from above papers are also not included in the comparison study in the evaluation section.  \n3. The experiments on balance between over-smoothing and over-uniformity requires a bit more convincing. Ideally, if authors could demonstrate some alignment between clusters of entities learnt by the Light GCL and the domain-based clusters, that would be even better.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gXDdfAqMuc", "reviewer": "Reviewer_k7CA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a graph contrastive learning paradigm named LightGCL, the singular value decomposition is adopted for contrastive augmentation to achieve the unconstrained structure refinement with global collaborative relation modeling. Experiments on five datasets validate the effectiveness of the proposed LightGCL compared with ten baselines.", "summary_of_the_review": "Overall, the idea of using a  SVD-based graph structure learning to achieve graph augmentation in contrastive learning is interesting and efficient. However, compared with the existing work (Robust Tensor Graph Convolutional Networks via T-SVD based Graph Augmentation. KDD 2022), the highlights should be more prominent. Moreover, some important relevant literatures is missing, and more SVD-based ablation studies should be discussed. ", "strength_and_weaknesses": "Strength:\n(1) The idea of using SVD to to augment user-item interaction graph structures seems interesting to me.\n(2)  The codes and datasets of this work are provided for reproducibility.\n(3) The experimental results look promising and show relative improvements over baselines.\n\nWeakness:\n(1) The truncated T-SVD to capture the low-rankness of the multi-view augmented graph, which improves the robustness from the perspective of graph preprocessing in “Zhebin Wu, Lin Shu, Ziyue Xu, Yaomin Chang, Chuan Chen, Zibin Zheng: Robust Tensor Graph Convolutional Networks via T-SVD based Graph Augmentation. KDD 2022: 2090-2099”. Compared to this work, which also uses SVD-based graph augmentation and improved robustness, what are your highlights?\n(2) This paper focuses on a general SVD-based graph structure learning, more variants based on SVD (eg. RSVD and SVD++, ) should be designed in the ablation study, instead of only a pre-trained MF to replace it.\n(3) In Section 4.2, the authors did not provide statistical significance tests. The improvement ratio of the proposed LightGCL over the optimal baseline is also not stated.\n(4) Figure 1 shows the overall structure of LightGCL, however, lack of overall description for this figure.\n(5) As we all know, the disadvantage of SVD is that the decomposed matrix often lacks interpretation. Has the author considered this problem in recommendation?\n\nSome papers involving graph contrastive learning are missed, such as:\n-  Lu Yu, Shichao Pei, Lizhong Ding, Jun Zhou, Longfei Li, Chuxu Zhang, Xiangliang Zhang: SAIL: Self-Augmented Graph Contrastive Learning. AAAI 2022: 8927-8935.\n-  Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, Xiang Zhang: AutoGCL: Automated Graph Contrastive Learning via Learnable View Generators. AAAI 2022: 8892-8900.\n- Yixin Zhang, Yong Liu, Yonghui Xu, Hao Xiong, Chenyi Lei, Wei He, Lizhen Cui, Chunyan Miao: Enhancing Sequential Recommendation with Graph Contrastive Learning. IJCAI 2022: 2398-2405.\n- Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, Stan Z. Li: SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation. WWW 2022: 1070-1079.\n- Susheel Suresh, Pan Li, Cong Hao, Jennifer Neville: Adversarial Graph Augmentation to Improve Graph Contrastive Learning. NeurIPS 2021: 15920-15933. \n\nThere are some typos to check, such as:\n- “masked featurte reconstruction”—>“masked feature reconstruction” (page 2)\n-“Normalized Discounted Culmulative Gain”—>“ Normalize Discounted Cumulative Gain” (page 5)\n-“Addtionally”—>“ Additionally” (page 7)\n-“while the best configuration of τ value vary by datasets”—>“ while the best configuration of τ value varies by datasets” (page 9)\n-“we presents a case study”—>“ we presents a case study” (page 9) \n-“our graph augmentation scheme exhibit strong ability ”—>“our graph augmentation scheme exhibits strong ability” (page 9) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "feDuCOS0Wk", "reviewer": "Reviewer_kpFd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the emerging topic of graph contrastive learning in recommender system, by proposing a lightweight and effective method. Particularly, the augmented view is automatically generated based on singular value decomposition technique, which is interesting and novel to me. With the augmented user-item graph structures, new contrastive self-supervision signals are derived. By doing so, only one additional augmentation view is needed for contrastive learning, further improving the model efficiency as justified in this work. Extensive experiments are conducted on several benchmark datasets to validate the effectiveness and superiority of the new approach.", "summary_of_the_review": "Reasonable and well-motivated methodology. Comprehensive experiments are conducted for performance validation. The detailed setting descriptions of hypergraph-enhanced baselines can be provided, and theoretical effectiveness is also encouraged to supplement. ", "strength_and_weaknesses": "1.Reasonable and well-motivated methodology design to advance the GCL-based recommender systems from a novel perspective of decomposition-based augmentation.\n2. Sufficient and appropriate state-of-the-art methods (e.g., some recent models SGL, HCCF, SimGCL) are adopted for performance comparison.\n3. Extensive experiments to demonstrate the effectiveness of the newly proposed solution from various aspects, including model robustness against data sparsity and long-tail issue.\n4. Both the superior performance and lower model complexity are achieved by the new method compared with strong state-of-the-arts.\n\nPlease find my comments below:\n\nAlthough the proposed method is simple yet effective from the experimental perspective, the authors are encouraged to discuss the effectiveness from the theoretical perspectives.\n\nBesides, in performance comparison, I notice that different types of methods are considered as baselines, including representative GNN-based CF models (GCCF and LightGCN), and disentangled graph CF approach-DGCF. Additionally, recently proposed SSL-enhanced recommender systems SGL, SHT, HCCF are also included. For those hypergraph-enhanced approaches, the self-supervised information source mainly comes from information aggregation over hyperedges. The settings of hypergraph structures in those methods can be further described, which may be helpful to understand the effectiveness of SSL-based graph augmentation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%