How Does Message Passing Improve Collaborative Filtering?
Authors: Mingxuan Ju
William Shiao
Zhichun Guo
Yanfang Ye
Yozen Liu
Neil Shah
Tong Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Recommender Systems
Test-time Augmentation
Collaborative Filtering
Primary area: other
TL;DR: This work studies how does message passing improve collaborative filtering and proposes an efficient framework called TAG-CF to leverage the findings.

Abstract: Collaborative filtering (CF) has exhibited prominent results for recommender systems and been broadly utilized for real-world applications.
A branch of research enhances CF methods by message passing (MP) used in graph neural networks, due to its strong capabilities of extracting knowledge from graph-structured data, like user-item bipartite graphs that naturally exist in CF. They assume that MP helps CF methods in a manner akin to its benefits for graph-based learning tasks in general (e.g., node classification). However, even though MP empirically improves CF, whether or not this assumption is correct still needs verification. To address this gap, we formally investigate why MP helps CF from multiple perspectives and show that many assumptions made by previous works are not entirely accurate. With our curated ablation studies and theoretical analyses, we discover that (i) MP improves the CF performance primarily by additional representations passed from neighbors during the forward pass instead of additional gradient updates to neighbor representations during the model back-propagation and (ii) MP usually helps low-degree nodes more than high-degree nodes.}Utilizing these novel findings, we present Test-time Aggregation for Collaborative Filtering, namely TAG-CF, a test-time augmentation framework that only conducts MP once at inference time. The key novelty of TAG-CF is that it effectively utilizes graph knowledge while circumventing most of notorious computational overheads of MP. Besides, TAG-CF is extremely versatile can be used as a plug-and-play module to enhance representations trained by different CF supervision signals. Evaluated on six datasets (i.e., five academic benchmarks and one real-world industrial dataset), TAG-CF consistently improves the recommendation performance of CF methods without graph by up to 39.2% on cold users and 31.7% on all users, with little to no extra computational overheads. Furthermore, compared with trending graph-enhanced CF methods, TAG-CF delivers comparable or even better performance with less than 1% of their total training times. Our code is publicly available at https://github.com/snap-research/Test-time-Aggregation-for-CF.

How Does Message Passing Improve Collaborative Filtering?
Clark Mingxuan Jul.2- William Shiao Zhichun Guo?, Yanfang Ye²,
Yozen Liu Neil Shah', , Tong Zhao
Snaa Inc., University of Notre Dame, UUniversity of California, Riverside
1 "mmju,yliu nshah, &ong) com, {mju2, zguo5, yye7) edu, $wshia002@ucr.ee
Abstract
Collaborative filtering (CF) has exhibited prominent results for recommender sys- tems and been broadly utilized for real-world applications. A branch of research enhances CF methods by message passing (MP) used in graph neural networks, due its strong capabilities of extracting knowledge from graph-structured data, like user-item bipartite graphs that naturally exist in CF. They assume that MP helps CF methods in manner akin its benefits for graph-based learning tasks in general (e.g., node classification). However, even though MP empirically im- proves CF, whether or not this assumption is correct still needs verification. To address this gap, we formally investigate why MP helps CF from multiple per- spectives and show that many assumptions made by previous works are not en- tirely accurate. With our curated ablation studies and theoretical analyses, we discover that (i) MP improves the CF performance primarily by additional repre- sentations passed from neighbors during the forward pass instead of additional gradient updates to neighbor representations during the model back propppropnnttion and (ii) MP usually helps low-degree nodes more than high-degree nodes. Uti- lizing these novel findings, we present Test-time Aggregation for Collaborative Filtering. namely TAG-CF, test-time augmentation framework that only con- ducts MP once at inference time. The key novelty of TAG-CF that it effectively utilizes graph knowledge while circumventing most of notorious computational overheads of MP. Besides, TAG-CF is extremely versatile can be used as a plug- and-play module enhance representations trained by different CF supervision signals. Evaluated on six datasets (i.e., five academic benchmarks and one real- world industrial dataset), TAG-CF consistently improves the recommendation performance of CF methods without graph by up to 39.2% on cold users and 31.7% on all users, with little no extra computational overheads. Furthermore, compared with trending graph-enhanced CF methods, TAG-CF delivers com- parable or even better performance with less than 1% of their total training times. Our code is publicly available https://gittee com/snna--eseaarch/ -iwtttime-gggggatiooonnio
Introduction
Recommender systems essential in improving users' experiences on web services, such as product recommendations [59, 47], video recommendations [15, 55], friend suggestions [46, 72], etc. In particular, recommender systems based on collaborative filtering (CF) have shown superior performance [45, 33, 4]. CF methods use preferences for items by users to predict additional topics
"Conesponding authors. Work done during first-author's internship at Snap Inc..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

or products a user might like [51]. These methods typically learn a unique representation for each user/item and an item recommended to a user according to their representation similarities [21, 58].
One popular line of research explores Graph Neural Networks (GNNs) for CF, exhibiting improved results compared with CF frameworks without the utilization of graphs [66, 20, 60, 74, 1, 61, 80). The key mechanism behind GNNs is message passing (MP), where each node aggregates information from its neighbors in the graph, and information from neighbors that are multiple hops away can be acquired by stacked MP layers [30, 56, 18]. During the model training, traditional CF methods directly fetch user/item representations of an observed interaction (e.g.. purchase, friending, click, etc.) and enforce their pair-wise similarity [45]. Graph-enhanced CF methods extend this scheme by conducting stacked MP layers over the user-item bipartite graph, and harnessing the resulting user and item representations to calculate a pair-wise affinity.
A recent study [20] shows that removing several key components of the MP layer (e.g., learnable transformation parameters) greatly enhances GNNs' performance for CF. Its proposed method (LightGCN) achieves promising performance by linearly aggregating neighbor representations and has been used as the de facto backbone model for later works due to its simple and effective design [1. 74, 65]. However, this observation contradicts GNN architectures for classic graph learning tasks, where GNNs without these components severely under-perform [41, 62J. Additionally, existing research [20, 60] assumes that the contribution of MP for CF is similar to that for graph learning tasks in general (e.g.. node classification or link prediction) they posit that node representations are progressively refined by their neighbor information and the performance gain is positively proportional to the neighborhood density as measured in node degrees [54]. However, according to our empirical studies in Section 3.2, MP in CF improves low-degree nodes more than high-degree nodes, which also contradicts GNNs' behaviors for classic tasks [54, 23]. In light of these inconsistencies, we ask:
What role does message passing really play for collaborative filtering?
In this work, we investigate contributions brought by MP for CF from two perspectives. Firstly, we unroll the formulation of MP layer and show that its performance improvement could either come from additional representations passed from neighbors during the forward pass or accompanying gradient updates neighbor representations during the pack-propagation. With rigorously designed ablation studies, we empirically demonstrate that gains brought by the forward pass dominate those by the back- paok-propggttion Furthermore, we analyze the performance distribution w.r.t. the user degree (i.c.. the number of interactions per user) with or without message passing and discover that the message passing in CF improves low-degree users more compared to high-degree users. For the first time, we connect this phenomenon to Laplacian matrix learning [82, 10, 9], and theoretically show that popular supervision signals [45, 57] for CF inadvertently conduct message passing in the back-prope propaaattion even without treating the input data as a graph. Hence, when message passing is applied, high-degree users demonstrate limited improvement, as the benefit of message passing for high degree nodes has already been captured by the supervision signal.
With the above takeaways, we present Test-time Aggregation for Collaborative Filtering, namely TAG-CF. Specifically, unlike other graph CF methods, TAG-CF does not require any message passing during training. Instead, it is a test-time augmentation framework that only conducts a single message-passing step at inference time, and effectively enhances representations inferred from different CF supervision signals. The test-time design is inspired by our first perspective that, within total performance gains brought by message passing, gains from the forward pass dominate those brought by the backward pass. Applying message passing only at test time avoids repetitive queries (i.e., once per node and epoch) for representations of surrounding nodes, which grow exponentially as the number of layers increases. Moreover, following our second perspective that message passing helps low-degree nodes more in CF, we further offload the cost of TAG-CF by applying the one-time message passing only to low-degree nodes. We summarize our contributions as:
This is the first work that formally investigates why message passing helps collaborative filtering. We demonstrate that message passing in CF improves the performance primarily by additional representations passed from neighbors during the forward pass instead of accompanying gradient updates to neighbors during the pack-propagation and prove that message passing helps low-degree nodes more than high-degree nodes.
Given our findings, we propose TAG-CF, an efficient yet effective test-time aggregation framework to enhance representations inferred by different CF supervision signals such as BPR and DirectAU,

---

Evaluated on six datasets, TAG-CF consistently improves the performance of CF methods without graph by up to 39.2% % on cold users and 31.7% on all users, with little no extra computational overheads. Furthermore, compared with trending graph-enhanced CF methods, TAG-CF delivers comparable or even better performance with less than 1% of their total training time.
Beside promising cost-effectiveness, the recommendation performance in similar ways as the training-tine time aggregation does, further demonstrating the legitimacy of our findings.
show that test-time aggregation in TAG-CF improves
2 Preliminaries and Related Work
Collaborative Filtering. Given a set of users, set of items, and interactions between users and items, collaborative filtering (CF) methods aim at learning unique representation for each user and item, such that user and item representations can reconstruct all observable interactions [45, 57, 32]. CF methods based on matrix factorization directly utilize the inner product between pair of user and item representations to infer the existence of their interaction [32, 45]. CF methods based on neural predictors multi-layer reed-fowwrd neural networks that take user and item representations as inputs and output prediction results [21, 77J. Let LI and denote the user set and item set respectively, with user u, E UI associated with an embedding u, E Id and item i; E U associated with i, E R the similarity Sij between user u, and item ij is formulated as 8t) = =u! -1.
Graph Neural Networks. Graph neural networks (GNNs) are powerful learning frameworks to extract representative information from graphs [30, 56, 18, 70, 26], with numerous applications in large-scale ranking and forecasting tasks [53, 52, 8, 49]. They aim to map each input node into low-dimensional vectors, which can be utilized to conduct either graph-level [69] node- tasks [30]. Most GNNs explore layer- wise message passing [14], where each node iteratively extracts information from its first-order neighbors, and information from multi-hop neighbors can be captured by stacked layers. Given graph G (V,E) and node features X R[Vxxd graph convolution in GCN [30] at k-th layer is formulated as:
h (k+1) o Σ JEN(IJUU VINGIVING h W(K)),
(k)
where h.- = x,, (i) refers the set of direct neighbors of node i, and (k)) g x1 refers to parameters at the k-th layer transforming the node representation from dh (k++)) dimension.
Recent works [37, 38] have shown that GNNs make predictions based on the distribution of node neighborhoods. Moreover, GNNs' performance improvement for high-degree nodes is typically better than for low-degree nodes [54, 23, 28, 17, 63). They posit that node representations are progressively refined by their neighbor information and the performance gain is positively proportional to the neighborhood density as measured in node degrees. As we explore test-time augmentation this work, is worth noting that there also exist a group of relevant works that explore data augmentation techniques to enhance the GNN performance [28, 79, 78, 25, 24].
Message Passing for Collaborative Filtering. Recent research tends to apply the message passing scheme in GNNs to CF [20, 60, 42, 12, 50, 68, 31, 34]. In CF, they mostly conduct message passing between user-item bipartite graphs and utilize the resultant representations to calculate user-item similarities. For instance, NGCF [60] directly migrates the message passing scheme in GNNs (similar to Equation (1)) and applies it to bipartite graphs in CF. LightGCN [20] simplifies NGCF [60] by removing certain components (i.e., the self-loop, learning parameters for graph convolution, and activation functions) and further improves the recommendation performance compared with NGCE The simplified parameter-less message passing in LightGCN can be expressed as:
(k)
Σ
1
(1)
Σ
GEN(U) VIN(u))V N(1)
VIN N
NGEN(I)
where N(-)) refers to the set of items or users that the input interacts with, u (0) = u, and (0) i, With K layers, the final user/item representations and their similarities are constructed as:
K
ú K
i
Sij = uj i,.
(3)
+1 k=0
K
K=U
3

---

According to results reported in LightGCN and NGCF [20, 60, 2, 13] and empirical studies we provide in this work (i.e., Table 2 and Table 5), incorporating message passing to CF methods without graphs (i.e., matrix factorization methods [45, 21 can improve the recommendation performance by up to 20%. Utilizing LightGCN as the backbone model, later works try to further improve the performance by incorporating self-supervised learning signals [35, 74, 1, 73, 64, 27]. Graph-base CF methods assume that the contribution of message passing for CF is similar to that for graph learning tasks in general (e.g., node classification or link prediction). However, whether not this assumption is correct still needs verification, even though message passing empirically improves CF There also exists branch of research that aims accelerating or simplifying message passing in CF by adding graph-based based regularization terms during the training [48, 39, 44, 67]. While promising, they still repetitively query representations of adjacent nodes during the training.
Efficient Efforts in Matrix Factorization. branch of research specifically focuses on improving the efficiency of matrix factorization [48, 44, 22, 43, 6]. For instance, GFCF [48] and Turbo-CF [43]
explore graph signal processing to linearly convolve the interaction matrix and use the resulted matrix
directly for recommendation without training. Furthermore, SVD-GCN [44] and SVD-AE [22] utilize a low rank version of the interaction matrix to further accelerate the convolution efficiency and yet remain the promising performance. Besides, BSPM [6] studies using diffusion process to gradually reconstruct the interaction matrix and achieves promising performance with fast processing. In parallel with these existing efforts, we propose to enhance any existing matrix factorization method through test-time augmentation that harnesses graph- based heuristics.
3 How Does Message Passing Improve Collaborative Filtering?
In this section, we demonstrate why message passing (MP) helps collaborative filtering from two major perspectives: Firstly, we focus on inductive biases brought by the MP explored in LightGCN, the de facto backbonn model for graph-based based CF methods. Secondly, we consider the performance improvement on different node subgroups w.r.t. the node degree with and without MP.
3.1 Neighbor Information vs. Accompanying Gradients from Message Passing
Following the definition in Equation (2), given a one-layer LightGCN', we unroll the calculation of the similarity Sij between any user and item iz as the following:
Σ (NN)
'n''
(1++
Σ
Sij
u, +
VINII,VINNNuu. un
(.EN())
= u. ig
Σ
u. Un Σ
VIN(LI) (u,))
VIN(a.I (IN
BNEN())
ENN))
Σ Σ
iT Un
(4)
/ N
(.,)( EN())
With derived similarities between user-item pairs, their corresponding representations can be updated by objectives (e.g., BPR [45] and DirectAU [57]) that enforce the pair-wise similarity between representations of user-item pairs in the training data.
CF methods without the utilization of graphs directly calculate the similarity between user and an item with their own representations (i.c., $;j = u. i,), which aligns with the first term in Equation (4). Compared to the formulation in Equation (4), we can that three additional similarity terms are introduced as inductive biases: similarities between users who purchase the same item (i.e., u! - u. between items that share the same buyer (i.e., iT, i,), and between neighbors of an observed interaction (i.e., iT U. With these three additional terms from MP, we reason that the performance improvement brought by MP to CF methods without graph could come from (i) additional neighbor representations during the forward pass (i.e., numerical values of three extra terms in Equation (4)), or (ii) accompanying gradient updates neighbors during the back-propagation.
For the simplicity of the notation, we showcase our observation with only one layer. However, since LightGCN is fully linear, the phenomenon we show also applies variants with arbitrary layers.

---

To investigate the origin of the performance improvement brought by MP, we designed two variants of LightGCN. The first one (LighiGCN wio neigh, info) shares the same forward and back- ward procedures as LightGCN during the training but does not conduct MP during the test time. In this variant, additional gradients brought by MP are maintained as part of the result- ing model, but information from neighbors are ablated. In the second variant (LightGCNN wo grad.). ) the model shares the same forward pass but drops gradients from these three additional terms during the backward propagation. Besides these two variants, we also experiment on LightGCN without MP, denoted as LightGCNwoo both, a matrix factorization model with the same supervi- sion signal (i.e., BPR loss). Implementation details w.r.t this experiment are in Appendix C. From Table 1, we observe that the perfor- mance of all variants is downgraded compared
Table I: Performance of LightGCN variants.
with LightGCN, with the most significant degra-
Method
Yelp-2018
Gounl1a
Anazon-book
dation on LightGGNwll neigh. info- This phe-
NDCG@20
LightGCN
6.36
9.88
8.13
nomenon indicates that (i) both additional repre-
weo grad.
6.16 (3.1%) ()
9.87 (0.1%
7,80 (4.1%)
sentations passed from neighbors during the for-
neigh. info
4.71 (25.9% (%8
(29.7%))
(14.5%))
wio both
6.09 (4.2%% L)
9:83 (0.5%%
7,75 (4.7%)
ward pass and accompanying gradient updates to neighbors during the pack-propagation help the recommendation performance, and (ii) within total performance gains brought by MP, gains from the forward pass dominate those brought
Recalle120 20
LightGCN w/o grad.
11.21
18.53
12.97
10.87 (3.0%_.
18.51 (0.1%)
12.81 (1.2%
w/o neigh. info
8.44 (24.7%)
13.06 (29,5%% [)
11.25 (13.3%
w/o both
10.71 (4.5%..
18:42 (0.6%)
12.57 (3.1%)
by the propagaaog,, Comparing LightGCN with LightGCNwio grad. we notice that the incorpora- tion of gradient updates brought by MP is relatively incremental (i.e., ~2%). However, to facilitate these additional gradient updates for slightly better performance, LightGCN is required to conduct MP at each batch, which brings tremendous additional overheads.
3.2 Message Passing in CF Helps Low-degree Users More Compared with High-degrees
Both empirical and theoretical evidence have demonstrated that GNNs usually perform satisfactorily on high- nodes with rich neighbor information but not as well on low-degree nodes [54, 23]. While designing graph-based model architectures for CF, most existing methods directly borrow this line of observations [60, 20] and assume that the contribution of message passing for CF is similar to that for graph learning tasks in general. However, whether or not these observations still transfer to message passing in CF remains questionable, as there exist architectural and philosophical gaps between message passing for CF and its counterparts for GNNs, as discussed in Section 2. To validate these hypotheses, we conduct experiments over representative methods (i.e., LightGCN and matrix factorization (MF) trained with BPR) and show their performance w.r.t. the node degree in Figure 1.
We observe that, overall both MF and LightGCN perform better on high-degree users than low-degree users. According the upper two figures in Figure 1, MF behaves similarly to LightGCN, even without treating the input data as graphs, where the overall performance for high-degree user is stronger than that for low-degree users. However, the performance improvement of LightGCN from MF on low-degree users is larger than that for high-degree users (i.e., lower two figures in Figure 1). According to literature in general graph learning tasks [23, 36, 54]. the performance improvement should be positively proportional to the node degree- the gain for high-degree users should be higher than that for low-degree users. This discrepancy indicates that it might not be appropriate to accredit contributions of message passing in CF directly through ideologies designed for classic graph learning tasks (e.g.. node classification and link prediction). To bridge this gap, we connect supervision signals (i.e.. BPR and DirectAU) commonly adopted by CF methods to Laplacian matrix learning. The formulation of BPR [45] and DirectAU [57] without the incorporation of graphs can be written as:
LBPR= =
Σ
Σ
log (sij si) = Σ Σ log aug i, uf is),
(1.j)DD (1.A)ED
(i.j)ED (
DDirectAU
Σ
Ilu, 111++ + log loge-2111 e 12'
log 10ge-2111-1112 2111, (5)
(E.)ED
W.W'Eu
1.''Z
where D refers to the set of observed interactions at the training phase and i' and u' refers to any random userfitem. According to works on Laplacian matrix learning [82, 10, 38], learning node
representations over graphs can be decoupled into Laplacian quadratie form, a weighted summation of two sub-goals:
min IZ 2
x 2 tr (Z)))))
(6)
5

---

(a) Gowalla
(b) Yelp-2018
NDCG@20
0.095
MF
5000
4000
S300 Jo #
NDC VDCG@20
0405
0.06
of Users
5000
Lighee
0.005
0.045
0.08
2002
0.075
0.07
1000
0,055
Aan Relative
1000
Improvement (%)
101117 1
S (%6)
0.23
16117815 2452
#
5000
Relative Inprevement
4000
of Users
1000
1000
& 9
2000
1000
1000
1000
1011 10011223110011117119190022211110 2223 2252677277222992
# of Interactions (User Degree)
# of Interactions (User Degrec)
Figure I: Performances of LightGCN and Matrix Factorization w.r.t. the user degree across datasets.
The performance improvement brought by message passing decreases as the user degree goes up.
where Z refers to the node representation matrix after the message passing, X refers to the input feature matrix, and L refers to the Laplacian matrix. The first term regularizes the latent representation such that it does not diverge too much from the input feature; whereas the second term promotes the similarity between latent representations of adjacent nodes, which can be re-written as: tr(ZT.L (Z).L.Z) EOAED ||u - i/ in CF bipartite graphs. [82] show that K layers of linear message passing exactly optimizes the second term in Equation (6). Given this theoretical foundation, we derive the following theorem w.r.t. relations between BPR, DirectAU, and message passing in CF:
for any u, U and I; E I. objectives of BPR and DirectAU are strictly upper-bounded by the objective of message passing (i.e., LAPR S EnDED ||u - 1,11² and Liimeas S E(JjeD |lu - 1,
Theorem Assuming that || ||,
Proof of Theorem I can be found in Appendix A. According to Theorem 1, both BPR and Direc- tAU optimize the objective of message passing (i.e., E(seeD Ilu - i with some additional regularization (i.e., dissimilarity between non-existing user/item pairs for BPR, and representation uniformity for DUr,ctAU). Hence, directly optimizing these two objectives partially fulfills the effects brought by message passing during the propppropaaatonn.
Combining this theory with the aforementioned empirical observations, we show that these two supervision signals could inadvertently conduct message passing in the backward step, even without explicitly treating interaction data as graphs. Since this inadvertent message passing happens during the pack-propagation, its performance is positively correlated to the amount of training signals user/item can get. In the case of CF, the amount of training signals for user is directly proportional to the node degree. High-degree active users naturally benefit more from the inadvertent message passing from objective functions, because they acquire more training signals. Hence, when explicit message passing is applied to CF methods, the performance gain for high-degree users is less significant than that for low-degree users. Because the contribution of the message passing over high-degree nodes has been mostly fulfilled by the inadvertent message passing during the training.
To quantitatively prove this theory, we incrementally upsample low-degree training users and observe the performance improvement that TAG-CF could introduce each upsampling rate. If our line of theory is correct, then we should expect less performance improvement on low-degree users for larger upsampling rate. The results are shown in Appendix E with supporting evidence.
4 Test-time Aggregation for Collaborative Filtering
In Section 3, we demonstrate why message passing helps CF from two perspectives. Firstly, w.r.t the formulation of LightGCN, we observe that the performance gain brought by neighbor information dominates that brought by additional gradients. Secondly, w.r.t. the improvement user subgroups, we learn that message passing helps low-degree users more, compared with high-degree users.
In light of these two takeaways, we present Test-time Aggregation for Collaborative Filtering, namely TAG-CF, test-time augmentation framework that only conducts message passing once at inference time and is effective at enhancing matrix factorization methods trained by different CF supervision signals. Given a set of well-trained user/item representations, TAG-CF simply aggregates neighboring item (user) representations for given user (item) at test time. Despite its simplicity,

---

we show that our proposal can be used as a plug-and module and is effective at enhancing representations trained by different CF supervision signals.
The test-time aggregation is inspired by our first perspective that, within total performance gains brought by message passing, gains from additional neighbor representations during the forward pass dominate those brought by accompanying gradient updates to neighbors during the back-propageen Applying message passing only at test time avoids repetitive training-timi queries (i.e., once per node and epoch) of surrounding neighbors, which grow exponentially as the number of layers increases by the neighbor explosion phenomenon [16, 76, 75]. Specifically, given set of well-trained user and item representations U E Rluxxd and TR|i TAG-CF augments representations for user uu and item i as:
u = U, +
Σ IN(w)IIIN())" 1, - (EN((u))
Σ (N()) uyy MyEN(()
where and are two paperr-praaeterss that control the normalization of message passing. With
m = n = 2' Equation (7) becomes the exact formulation one-layer LightGCN (i.e., Equation (2)).
Empirically, we observe that the setup with m = = for TAG-CF does not always work for all datasets. This setup is directly migrated from message passing for homogeneous graphs [30], which might not be applicable for bipartite graphs where all neighbors are heterogeneous [7]. Unlike LightGCN which can fill this gap by adaptively tuning representations during the training. TAG-CF cannot update any parameter since it is applied at test time, and hence requires tune-able normalization hyper-parametee parameters.
Moreover, following our second perspective that message passing helps low-degree nodes more in CF, further derive TAG-CF-, which reduces the cost of TAG-CF by applying the one-time message passing only to low-degree nodes. Focusing on only low-degree nodes has two benefits: (i) it reduces the number of nodes that TAG-CF+ needs to attend to, and (ii) message passing for low-degree nodes is naturally cheaper than for high-degree nodes given the surrounding neighborhoods are sparser (mitigating neighbor explosion). The degree threshold that determines which nodes apply TAG-CF* is selected by the validation performance, with details in Appendix C.
TAG-CF can effectively enhance MF methods by conducting message passing only test time. TAG-CF effectively utilizes graphs while circumventing most of notorious computational overheads of message passing. It is extremely flexible, simple implement, and enjoys the performance benefits of graph-based CF method while paying the lowest overall scalability.
5 Experiments
We conduct extensive experiments to demonstrate the effectiveness and efficiency of TAG-CF. We aim to answer the following research questions: RQ (1): how effective is TAG-CF at improving MF methods without using graphs, RQ (2): how much overheads does TAG-CF introduce, RQ (3): can TAG-CF effectively enhance MF methods trained by different objectives, RQ (4): how effective is TAG-CF+ w.r.t. different degree cutoffs, and RQ (5): do behaviors of TAG-CF align with our findings in Section 3?
5.1
Experimental Settings
Datasets. We conduct comprehensive experiments on five commonly used academic benchmark datasets, including Amazon-bood Anime, Gowalla, Yelp2018, and MovieLens-111 Additionally, we also evaluate on a large-scale real-world industrial user-item recommendation dataset Internal. Descriptions of these datasets are provided in Appendix B.
Baselines. We compare TAG-CF with two branches of methods: (1) CF methods that do not explicitly utilize graphs, including vanilla matrix factorization (MF) methods trained by BPR and DirectAU [45, 57], Efficient Neural Matrix Factorization [3] (denoted as ENMF), and UltraGCN [39]. (2) Graph-based CF methods, including LightGCN [20] and NGCF [60]. Besides, we also compare with recent graph-based CF methods that extend LightGCN by adding additional self-superrised signals for better performance, including LightGCL [1], SimGCL [74], and SGL [65]. For the coherence of reading, we include comprehensive discussions about evaluation protocols across all methods, tuning for syper-parameters: and other implementation details in Appendix C.
7

---

Table 2: Recommendation performance (i.e., NDCG@20 and Recall @ 20) of all models across users with different numbers of interactions. The lower percentile indicates the set of nodes whose degrees ranked in the lower 30% population. Bold and underline indicate the best and second best model respectively. LightGCN and MF are trained with DirectAU [57].
Mathod
NOCF
I LightiN
ENMF
-TAG-CF
Iope <T>
MF
TAG-CF
Inpe (183
UlraCCN
-TACI-C
Inpe (1%)
NDCG@
20 LOW-DRORER USERS (LoWER PERCENTIEE
Amazon-Boook Boox
5. 7.
10
5.38anee
6.4%
8.02
3.0%
5.61201
7.7%
keim
20.13 13
27
22.25
12.58
1.6%
23.95
27
15
13.4%
28. 14a0
20.10
7.0%
Govalla
8,46
10
3872001
1.08.011
5.4%
10,00.t.ss
10.19
1.9%
8.2
8.63
5.1%
Yelp 2018
4.87
0.10
3.11
4,8%
6.0820.s
6.18
1.7%
4.89
5,44.0122
11.2%
22.13a
25.05
18.34
22.53,0:1
22.5%
20.98ac.2
29.20an:10
39.25
23.8 23.89
28.37
18.8%
Taternal
OUM
6.79
2.5
25.5%
OUM
NDCG 20
OVISAALL
Imazon Boox
6.97
8:06.
6.13
6.54,000
6.7%
8.13
1.5%
5.77anss
5.9%
Anise
22.54a
27.97
30,174003
3086.001
2,5%
24.01ac..0
27.25
9.8%
30,30a0.1
3019..011
1.9%
Govalla
8.65
11
5.23
20
1.1%
9,77
1.1%
1:33g0
9.02
5.7%
Yel 2-1p-2018
5 5.54g0
6.1.mmm
3.79 4.79
2.6%
6.25
1.8%
5 5.
5.59
10.4%
Movi Lnne-l
23 23.17
26.4
20.57atts
22.98,020
22
11.7%
22.51
20.65
31.7%
26.50 26550
29.68_021
12.0%6
Internal
6,94202
8.10
00M
8.54.m
21.3%
O0M
RECALL @ 20
Low DEORDEORSS USERS (LOWER PERCENTILE)
BrassnnBook
10.7
37
10.42
11.08
6.3%
a3.07
13.37 2003
7.92gu.15
Antme
25.74
32
38.4
3.4%
29.08.2:.9
31 21.90
9.8%
33.96an.2
38.49
7.6%
Gavalla
17.53
19.14
8.33
3.2%
18.92ac..9
19.17 a0:3
1.3%
15.57am.as
18.01
Lo
2.8%
Yelp-2015 2018
10.15
10.75m34
7.17anae
7.54,0122
5.2%
10.03at.13
10.9
3.3%
3.71an35
8.59
11.4%
MozieLens 18
22.71an0
19.58ana1
24.10g000
23.1%
23.64ac.ia
28. L0
18.9%
26.13
28.97 g0
10.9%
Internal
10.5 10.54
00M
11.15
1399 an
25.5%
00M
RECALL 8020
OIVERALE.
Amazon-Bool Book
10.30
32
10.89
11.35
4.2%
12.07
12.97 ssuce
2,4%
5.01
8.53.0
6,5%
Knlew
28 124
5282
54. 10
34 28.48
1.1%
20.15acam
31 a1
6.9%
35.8240.
37.01
3.2%
Govalla
20.9
1865
2.682
9.74.000
0.6%
18.30
18:53
1.388
15.93
16 16.36
2.7%.
Yelp 2018
10:02am0
1098
6.82
7.062000
2.3%
10.81as.00
3.7%
8.412019
0:50-022
17.8%
Nostel 11
23.93a6:4
26.30,010
26
21.31apa1
23 23.88
12.1%
36, 26.30
28 23-40
8.0%
27. 17
29.78_
9.7%
Isternal
6.91202
1339
00M
11.83r.22
14.41a00e
21.8%
00M
Table Running time (1 X 10 seconds) for MF methods and TAG-CF. Time % is the percentage of running time TAG-CF takes w.r.t. the time for corresponding MF methods. Speedt refers to the ratio of running times between training-time aggregation (i.e., LightGCN) and TAG-CF, All training steps are timed and terminated by an early stopping strategy (see Appendix C).
Methed
Sparsity
ENME
+IAG-CF
Time
UltraGCN
+TAG-CF
Time %
LighiGCN
MF
+IAG-CF
Tine
Speed!
Yelp-2011 2018
99.87%
2.15
+0.02
0.9%
5.02
+0.02
0.4%
5.81
3.17
+0.02
0.6%
1,83k
Gowalla
99.91%
4.56
+0.02
0.4%
12.55
+0.02
0.2%
13.27
7.74
40.02
0.3%
1.72K
Anozon-Rook
99.94%
11.54
+0.03
0.1%
39.25
40.03
0.15
46.62
29,21
40.03
0.1%
1.59%
Internal
00M
OOM
47.32
32.62
0.00
0.3%
1,44x
5.2 Performance Improvement to Matrix Factorization Methods
For RQ (1), Table 2 shows the performances of MF methods (MF and ENMF) as well as that of the performances of them with TAG-CF applied on their learned representations. We observe that TAG- CF unanimously improves the recommendation performance for both of them. Specifically, across all datasets, TAG-CF on average improves the low-degree NDCG@ 20 by 4.6% and 9.1% % and overall NDCG by 3.2% and 7.1% for ENMF and MF, respectively. We also observe a similar performance improvement for Recall @ 20, where TAG-CF on average improves the low-degree Recall 20 by 4.5% and 8.4% and overall Recall @ 20 by 2.1% and 7.2% for ENMF and MF, respectively. Furthermore, we notice that TAG-CF can improve the performance of UltraGCN, a method that utilizes the graph knowledge as additional supervision signals. This phenomenon demonstrates the superior effectiveness of TAG-CF, indicating that our proposed test-time aggregation can further enhance graph-enhanced MF methods.
By comparing the performance gains brought by TAG-CF on low-degree users with that on all users, we notice that gains for low-degree users are usually higher. Hence, message passing in CF helps low-degree users more than for high-degree users, which echos with our observations in Section 1. To answer RQ (5), the behavior of TAG-CB CF aligns with second perspective in Section 3.2 that the supervision signal inadvertently conducts message passing. Consequently, the room for improvement on high-degree users could be limited, part of the contributions from message passing has already been claimed by the supervision signal.
8

---

Table 4: The running time and performance of
Table 5: Performance of TAG-CF when applied
graph-based CF methods that extend LightGCN.
to models trained with BPR loss.
Method
SGL
SimGCL
LighTGL
TAG-CF
Metbed
LighTGCN
MF
+TAG-CF
Impe. (1%)
NDCG@ OVERALL
NDCG@
Low DEGREE USERS (LOWER PERCENTILE)
Anine
27.02.0.0s
30.48_0.12
28.34
27.25
30.02200r
29.36.0.33
20.56
4.1%
Yelp
5.67
1.99
4,93 0.00
6.36
Yelp
1.34
3.63
3.81
5.0%
Govalla
9.67
10.32_0..0
2.99
9, 9.88-0.0c
Govella
8.22
7.56-u.44
7.88_a..1
4.2%
Book
6.69 10.02
7.02 1o.05
3.83_n.0x 3.5
8.13 1..0
Book
5.19
4.1920.14
4.68.0.14
11.7%
Avg. Rank
NDCG@ 20 OVERALL
RECALL @ 20 OVERALL
Anime
30. 10.14
29.51
30,23 10.23
2.4%
Anime
31.29
34.93_0.14
33.64
31.95
Yelp
4..7.
3.9620.14
4.26-u.17
7.6%
Yelp
10.01 g0.0s
10.56-0.1s
8.83
11.21 2:.ce
Govalla
8.3240.05
7.51
7.99
6.4%
Gowalla
18. 18.1810221
19.22
16.99
18.53 10.ce
Book
5.07
4.15g0.13
4.32g0.18
4.1%
Book
11.15
11.51
10.06.0.00
12.97.0.00
RECALL
20 LOW-DEOREE USERS (LOWER PERCENTILL)
Avg. Rank
3.2
1,5
3.5
1.7
Anine
34.23
34.81 20.52
35.42.20.5
1.8%
RUNNING TIME (1 x 10 SECOND)
Yalp
8.1940.00
6.9320.00
7.25_0.19
4.6%
Govalla
16.1720
14.86
15.33
3.2%
Yelp
3.94
87.77
9.72
97.31
4.30
3.19
Book
10:2
7.45 j0:22
8.05 8,05_0.15
8.1%
Govalla
29.11
11.10
7.76
RECALL 1 20 OVERALL
Book
63.21
71.39
38.87
29.24
Anime
24.21
34.84.0.50
35.23
1.1%
Avg. Rank
2.2
3.8
3.0
Yelp
8.33
7.27-0037
7.62_
4.8%
Total Rank
3,6
2.8
3.9
1.9
Govalls
15.69 2our
14.47 20.21
14,9220.2
3.1%
Book
8.65
7.
7.64_0.30
3.9%
5.3 Performance Comparison Among Graph-based Methods
Comparing TAG-CF with LightGCN in Table 2, we can notice that TAG-CF mostly performs on par with and sometimes even outperforms LightGCN, without incorporating message passing during the training and only conducting test-time aggregation. This phenomenon indicates that conducting neighbor aggregation at the testing time can recover most of the contributions of training time message passing. To answer RQ (5), TAG-CF aligns with our first perspective in Section 3.1 that the performance gain from beneficial neighbor information dominates their accompanying gradients.
We further compare TAG-CF with state-of-the-arr graph- based CF methods, with their performance and efficiency shown in Table 4. Among these performant baselines, TAG-CF exhibits competitive performance, with an average rank of 1.2 on NDCG and 1.7 on Recall. Though not always the model that delivers the best performance, TAG-CF can deliver comparably promising results and introduces little computational overheads (i.e., ranked .. for running time). Considering efficiency one factor, TAG-CF achieves the best performance across all baselines with an average rank of 1.9.
While performing on par with graph-based CF methods that aggregate neighbor contents at the training time. TAG-CF enjoys the performance benefits of message passing while paying the lowest overall scalability. To answer RQ (2), according to Table 3, across all datasets, TAG-CF only introduces an average additional computational overhead of 0.05 x 103 seconds, which is less than 0.5% of the total training time for matrix factorization methods. Comparing the running time of LightGCN with that of TAG-CF, we can observe that the latter can significantly improve the computational time, and the speedup is proportional the sparsity of the dataset.
5.4 Effectiveness for Different Training Signals
To answer RQ (3), besides DirectAU, we also conduct experiments on BPR loss, as shown in Table 5. When applied to BPR, TAG-CF still consistently improves the performance by large margins (i.e., 6.3% and 5.1% average improvement on low-degree and overall NDCG respectively, and 4.4% and 3.2% on low-degree and overall Recall respectively). We notice that TAG-CF sometimes does not perform as competitively as LightGCN when both are trained with BPR. We check norms of learned representations from MF with BPR and discover that they have high variance since BPR does not explicitly enforce any regularization. This might not favor TAG-CF as a test-time augmentation method due to its simple design, which cannot adapt representations with high variance.
5.5 Performance w.r.t. User Degree
To answer RQ (4), we apply TAG-CF+ to four public datasets and the performance and the efficiency improvement are demonstrated in Figure 2. Overall, the running time improvement brought by TAG-CF+ exponentially increases as the degree decreases, since low-degree users have sparse neighborhoods and there is hence less information for TAG-CF+ to aggregation. When the degree cutoff is low (i.e., less than 100), the effectiveness of TAG-CF+ proportional increases as the degree cutoff increases.

---

When setting the cutoff to user degree of around 100, on Amazon-Boo Book, Gowalla, and Ye1p-2018, TAG-CF* can further improve TAG-CF by 125%. 17%. and 11%, respectively. with efficiency improvement of 7%, 4%, and 8%. In these cases, TAG-CF+ not only significantly improves the performance but also effectively reduces computational overheads.
However, on these three datasets, after the cutoff by- passes degree of 100, the performance improvement eventually decreases to the performance of TAG-CF (i.e., 100%), indicating that test-time aggregation jeop- ardizes the performance on high-degree nodes. On Anime, though no downgrade on high-degree users, the performance improvement of TAG-CF+ to TAG-CF is incremental. These phenomenons not only demonstrate the effec- tiveness and efficiency of TAG-CF+, but also verify our findings in Section 3.2
RT Impr. (%)
50
Amazon-Boode
250 25
Yelp-2018
150
Perf. Impr. (%)
200 20
150 15
100
100
Impe
Rngge (Ng)
Porf: Impro Kho
Porf. Impe, (1))
1p
10
10
10
10
-50
10
10"
10'
10
10
15
Gowal 1 la
150100
An Anim
150
Perf. Impr. (%)
RT Impr. (%)
100 80
100
RT Impe. (?6)
Impe- (9))
Porf. Tmige, (56)
-50
Perf. Impr. (%)
0
-100
10
10"
10
10
107
101
10P
10a
10
User Degree
User Degree
Figure 2: The performance and efficiency improvement of TAG-CF+ w.r.t. different cutoffs. TAG-CF further improves TAG-CF with less computational overheads. 100% is the original performance/effccieney of vanilla TAG-CF.
that message passing in CF helps low-degree users more than high-degree users.
6 Conclusion
In this study, investigate how message passing improves collaborative filtering. Through series of ablations, we demonstrate that the performance gain from neighbor contents dominates that from accompanying gradients brought by message passing in CF. Morcover, for the first time, we show that message passing in CF improves low-degree users more than high-degree users. We theoretically demonstrate that CF supervision signals inadvertently conduct message passing in the backward step, even without treating the data as a graph. In light of these novel takeaways, we propose TAG-CF, test-time aggregation framework effective enhancing representations trained by different CF supervision signals. Evaluated on five datasets, TAG-CF performs at par with SoTA methods with only fraction of computational overhead (i.e., less than 1.0% of the total training time).
7 Limitation and Broader Impact
One limitation of our proposal could be the utilization of graphs in large-scale machine learning pipeline. TAG-CF conducts a single-time aggregation of neighbors, which could be equivalently achieved by existing technologies such as SQL, BigQuery, etc. Furthermore, we observe no ethical concern entailed by our proposal, but we note that both ethical or unethical applications based on collaborative filtering may benefit from the effectiveness of our work. Care should be taken to ensure socially positive and beneficial results of machine learning algorithms.
8 Acknowledgments
This work was mostly conducted during the internship of Clark, William, and Zhichun at Snap Inc. We would like to thank Xin Chen and his colleagues from Snap Inc. for their help on pre- proppsccssiing the internal dataset. This work was partially supported by the NSF under grants IIS-2321504, IIS- 2334193, IIS- 2203262, IIS-2217239, CNS-2426514, CNS-2200 220-220, and CMMII-144076. Any opinions, findings, and conclusions recommendations expressed in this material those of the authors and do not necessarily reflect the views of the sponsors.
We sincerely appreciate constructive feedback from all reviewers during the paper review phase.
10

---

References
[1] X. Cai, C. Huang. I.. Xia, and Ren. Lightgel: Simple yet effective graph contrastive learning for recommendation. In Procs. of ICLR, 2023.
[2] J. Chang, C. Gao, Y. Zheng. Y. Hui, Y. Niu, Y. Song, D. Jin, and Y. Li. Sequential recommenda- tion with graph neural networks. In Procs. of SIGIR, 2021.
[3] C. Chen, M. Zhang. Y. Zhang, Liu, and Ma. Efficient neural matrix factorization without sampling for recommendation. ACM Transactions on Information Systems (TOIS), 2020.
[4] J. Chen, H. Dong, X. Wang, F. Feng, M. Wang, and X. Hc. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39 2023.
[5] E. Cho, S. A. Myers, Jnd Leskovec. Friendship and mobility: user movement in location-based social networks. In Procs of SIGKDD, 2011.
[6] Choi, S. Hong, N. Park, and S.-B. Cho. bouriig--shappnnigg process models for collaborative filtering, In Procs. of SIGIR, 2023.
[7] G. Dasoulas, J. F. Lutzeyer, and M. Vazirgiannis. Learning parametrised graph shift operators. In Procs of ICLR, 2021.
[8] A. Derrow-Pinion J. She, D. Wong, O, Lange, T. Hester, L. Perez, M. Nunkesser, S. Lee, X. Guo, B. Wiltshire, et al. Eta prediction with graph neural networks in google maps. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 3767-3776, 2021.
[9] X. Dong, D. Thanou, P. Frossard, and P. Vandergheynst. Learning laplacian matrix in smooth graph signal representations. IEEE Transactions on Signal Processing, 2016.
[10] X. Dong, D. Thanou, M. Rabbat, and P. Frossard. Learning graphs from data: A signal representation perspective. IEEE Signal Processing Magazine, 2019.
[11] Y. Fan, M. Ju, C. Zhang, and Y. Ye. Heterogeneous temporal graph neural network. In Procs, of SDM, 2022.
[12] C. Gao, X. Wang, X. He, and Y. Li. Graph neural networks for recommender system. In Procs. of WSDM, 2022.
[13] C. Gao, Y. Zheng, N. Li, Y. Li, Qin, J. Piao, Y. Quan, Chang, D. Jin, X. He, et al. A survey of graph neural networks for recommender systems: Challenges, methods, and directions. ACM Transactions on Recommender Systems, 2023.
[14] Gilmer, S. S. Schoenholz.
Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In Procs. of ICML, 2017.
[15] C. A. Gomez-Uribe and N. Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), 2015.
[16] Z. Guo, W. Shiao, S. Zhang, Y. Liu, N. V. Chawla, N. Shah, and T. Zhao. Linkless link prediction via relational distillation. In International Conference on Machine Learning, pages 12012-12033. PMLR, 2023.
[17] Z. Guo, T. Zhao, Y. Liu, K. Dong, W. Shiao, N. Shah, and N. V. Chawla. Node duplication improves cold-start link prediction. arXiv preprint arXi::2402.0971, 2024.
[18] W. L.. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Procs. of NeurIPS, 2017.
[19] F. M. Harper and J. A. Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tis), 2015.
[20] X. He, K. Deng, Wang, Y. Li, Y. Zhang, and M. Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Procs. of SIGIR, 2020.
11

---

[21] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua. Neural collaborative filtering. In Procs. of WWW, 2017.
[22] Hong, J. Choi, Y.-C. Lee, S. Kumar, and N. Park. Svd-ae: Simple autoencoders for collabora- tive filtering. In Procs. of IJCAI, 2024.
[23] W. Hu, K. Cao, K. Huang, E. W. Huang, K. Subbian, and J. Leskovec. Tuncup: A training strategy for improving generalization of graph neural networks. arXiv, 2022.
[24] W. Jin, T. Zhao, J. Ding, Y. Liu, J. Tang, and N. Shah, Empowering graph representation learning with test-time graph transformation. In Procs. of ICIR, 2023.
[25] M. Ju, Y. Fan, C. Zhang, and Y. Ye. Let graph be the go board: gradient-free node injection attack for graph neural networks via reinforcement learning. In Procs. of AAAI, 2023.
[26] M. Ju, W. Yu, T. Zhao, C. Zhang, and Y. Ye. Grape: Knowledge graph enhanced passage reader for open- open-domain question answering. In Findings of EMNLP, 2022.
[27] M. Ju, T. Zhao, Q. Wen, W. Yu, N. Shah, Y. Ye, and C, Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. In Procs. of ICLR, 2023.
[28] M. Ju, T. Zhao, W. Yu, N. Shah, and Y. Ye. Graphpatcher: Mitigating degree bias for graph neural networks via test-time augmentation. Procs. of NeurlPS, 2023.
[29] Kaggle. Anime Recommendatiosn Database. https:/ hupp://wwwe //ww://wwww. htwp//wwwggleeee..///// waaggl..oom/lee /onn/taasets//
CooprrUniicn/aiiie--eccmmmeeee aww.:aiiiieeemeeeeeee peaeecccmmendaiiiinnnaaaaaaaaaea 2023. [Online; accessed June-2023).
[30] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Procs. of ICLR, 2016.
[31] M. Kolodner, M. Ju, Z. Fan, T. Zhao, E. Ghazizadeh, Y. Wu, N. Shah, and Y. Liu. Robust training objectives improve embedding- based retrieval in industrial recommendation systems. RobustRecSys Workshop at RecSys, 2024.
[32] Y. Koren, R. Bell, and C. Volinsky. Matrix factorization techniques for recommender systems. Computer, 2009.
[33] Y. Koren, S. Rendle, and R. Bell. Advances in collaborative filtering. Recommender systems handbook, pages 91-142, 2021.
[34] P.HH. Kung, Z. Fan, T. Zhao, Y. Liu, Z. Lai, Shi, Y. Wu. Yu, N. Shah, and G. Venkataraman. Improving embedding- -base retrieval in friend recommendation with ann query expansion. In Procs. of SIRIP, 2024.
[35] Z. Lin, C. Tian, Y. Hou, and W. X. Zhao. Improving graph collaborative filtering with nighhorhood-enricher contrastive learning. In Procs of WWW, 2022.
[36] 7.. Liu, T.-K. Nguyen, and Y. Fang. Tail-gnn: Tail-node graph neural networks. In Procs. of SIGKDD, 2021.
[37] Y. Ma, X. Liu, N. Shah, and J. Tang. Is homophily a necessity for graph neural networks? In Procs. of ICLR, 2022.
[38] Y. Ma, X. Liu, T. Zhao, Y. Liu, J. Tang, and N. Shah. A unified view graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1202-1211, 2021.
[39] K. Mao, J. Zhu, X. Xiao, B. Lu, Z. Wang, and X. He. Ultragen ultra simplification of graph convolutional networks for recommendation. In Procs. of CIKM, 2021.
[40] J. McAuley and A. Yang. Addressing complex and subjective product-related queries with customer reviews. In Procs. of WWW, 2016.
[41] B. M. Oloulade, J. Gao, J. Chen, Lyu, and R. Al-Sabri. Graph neural architecture search: survey. Tsinghua Science and Technology, 2021.
12

---

[42] A. Pal, C. Eksombatchai, Y. Zhou, B. Zhao, C. Rosenberg, and J. Leskovec. Pinnersage: Multi- modal user embedding framework for recommendations at pinterest. In Procs. of SIGKDD, 2020.
[43] J.-D. Park, Y.-M. Shin, and W.-Y. Shin. Turbo-ef: Matrix decomposition-free graph filtering for fast recommendation. In Procs. of SIGIR, 2024.
[44] S. Peng, K. Sugiyama, and T. Mine. Svd-gen: simplified graph convolution paradigm for recommendation. In Procs of CIKM, 2022.
[45] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thiee Bpr: Bayesian personalized ranking from implicit feedback. In UAI, 2009.
[46] A. Sankar, Y. Liu, J. Yu, and N. Shah. Graph neural networks for friend ranking in large-scale social platforms. In Proceedings of the Web Conference 2021, pages 2535-2544 2021.
[47] J. B. Schafer, J. Konstan, and J. Riedl. Recommender systems in e-commerce. In Procs. of ACM conference on Electronic commerce, 1999.
[48] Y. Shen, Y. Wu, Y. Zhang, C. Shan, J. Zhang, B. K. Letaief, and D. Li. How powerful is graph convolution for recommendation? In Procs. of CIKM, 2021.
[49] Shi, V. Chaurasiya, Liu, S. Vij, Y. Wu, S. Kanduri, N. Shah, P. Yu, N. Srivastava, L. Shi, et al. Embedding based retrieval in friend recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3330-3334, 3034.3334222200 2023.
[50] J. Shi, H. Ji, C. Shi, X. Wang, Z. Zhang, and J. Zhou. Heterogeneous graph neural network for recommendation. arXiv preprint arXiv:2009.00799 2020.
[51] X. Su and T. M. Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.
[52] X. Tang, Y. Liu, X. He, S. Wang, and N. Shah. Friend story ranking with edge-contextual local graph convolutions. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1007-1015, 2022.
[53] X. Tang, Y. Liu, N. Shah, X. Shi, P. Mitra, and S. Wang. Knowing your fate: Friendship. action and temporal explanations for user engagement prediction on social apps. In Procs of SIGKDD, 2020.
[54] X. Tang, H. Yao, Y. Sun, Y. Wang, J. Tang, C. Aggarwal, P. Mitra, and S. Wang. Investigating and mitigating degree-relane related biases in graph convoltuional networks. In Procs. of CIKM, 2020.
[55] A. Van den Oord, S. Dieleman, and B. Schrauwen. Deep content- music recommendation. In Procs. of NeurIPS, 2013.
[56] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. In Procs. of ICLR, 2017.
[57] C. Wang, Y. Yu, W. Ma, M. Zhang, C. Chen, Y. Liu, and S. Ma. Towards representation alignment and uniformity in collaborative filtering. In Procs. of SIGKDD, 2022.
[58] H. Wang, N. Wang, and D.-Y. Yeung. Collaborative deep learning for recommender systems. In Procs. of SICKDD, 2015.
[59] R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Procs. of wwW, 2021.
[60] X. Wang, X. He, M. Wang. F. Feng, and T.-S. Chua. Neural graph collaborative filtering. In Procs. of SIGIR, 2019.
[61] X. Wang, H. Jin, A. Zhang, X. He, T. Xu, and T.-S. Chua. Disentangled graph collaborative filtering. In Procs. of SIGIR, 2020.
13

---

[62] Y. Wang, J. Jin, W. Zhang, Y. Yu, Z. Zhang, and D. Wipf. Bag of tricks for node classification with graph neural networks. arXiv preprint arXiv:2103.13355, 2021.
[63] Y. Wang, T. Zhao, Y. Zhao, Y. Liu, X. Cheng, N. Shah, and T. Derr. A topological perspective on demystifying gnn-based link prediction performance. ICLR, 2024.
[64] Y. Wei, X. Wang. Q. Li, L.. Nie, Li, X. Li, and T.-S. Chua. Contrastive learning for cold-start recommendation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5382-5390, 2021.
[65] Wu, X. Wang, Feng, X. He, L. Chen, J. Lian, and X. Xie. Self-supervised graph learning for recommendation. In Procs. SIGIR, 2021.
[66] S. Wu, F. Sun, W. Zhang, X. Xie, and B. Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1-37, 2022.
[67] L. Xia, C. Huang, Shi, and Y. Xu, Graph-less collaborative filtering. In Procs. of WwW, 2023.
[68] L. Xia, Y. Xu, C. Huang. P. Dai, and L. Bo. Graph meta network for multi-behavior recommen- dation. In Procs. of SIGIR, 2021.
[69] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In Procs. of ICLR, 2018.
[70] K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In Procs. of ICML. 2018.
[71] Yelp. Yelp Open Dataset. https: June-2023).
2023. [Online; accessed
[72] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec. Graph convolu- tional neural networks for web-scale recommender systems. In Procs. of SIGKDD, 2018.
[73] J. Yu, H. Yin, J. Li, Q. Wang, N. Q. V. Hung, and X. Zhang. Self-supervised multi-chanee hypergraph convolutional network for social recommendation. In Procs. of WWW, 2021.
[74] Yu, H. Yin, X. Xia, T. Chen, L. Cui, and Q. V. H. Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In Procs. of SIGIR, 2022.
[75] H. Zeng, M. Zhang, Y. Xia, A. Srivastava, A. Malevich, R. Kannan, V. Prasanna, L. Jin, and R. Chen. Decoupling the depth and scope of graph neural networks. Advances in Neural Information Processing Systems, 34: 19699666999779 2021.
[76] S. Zhang, Y. Liu, Y. Sun, and N. Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. arXiv preprint 10.0077, 2021.
[77] S. Zhang, L. Yao, A. Sun, and Y. Tay. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 2019.
[78] T. Zhao, W. Jin, Y. Liu, Y. Wang, G. Liu, S. Günnemann, N. Shah, and M. Jiang. Graph data augmentation for graph machine learning: A survey. arXiv preprint 2088:22202,08871 2022.
[79] T. Zhao, Y. Liu, L. Neves, O. Woodford, M. Jiang, and N. Shah. Data augmentation for graph neural networks. Procs. of AAAI, 2021.
[80] T. Zhao, N. Shah, and E. Ghazizadeh. Learning from graphs beyond message passing neural networks. In Tiny Papers Track at ICLR, 2024.
[81] W. X. Zhao, Y. Hou, X. Pan, C. Yang, Z. Zhang. Z. Lin, J. Zhang, S. Bian, Tang, W. Sun, et al. Recbole 2.0: towards a more up-to-date recommendation library. In Procs. of CIKM, 2022.
[82] M. Zhu, X. Wang. C. Shi, H. Ji, and P. Cui. Interpreting and unifying graph neural networks with an optimization framework, In Procs. of wWW, 2021.
14

---

A Proof of Theorem 1
Here we re-state Theorem before diving into its proof:
Theorem 1. Given that a K-layer GCN minimizes E(.)eDD ||u, - -,,11, During the training of MF 1 for any E l and I, E I, objectives of BPR and methods, assuming that -ulll2 || by the objective of message passing (i.c., CBPR S E(.)ED
DirectAU are strictly
l|u; 1 and ADi S (()))ED ||u, i,
One preliminary theoretical foundation for Theorem 1 to hold is that a K-layer graph convolution network (GCN) exactly optimizes the second term in Equation (6), which has been proved by [82]. For ease of reading, we re-phrase it again as the following:
Theorem 2. The message passing for GCN optimizes the following graph regularization term: O minz(tr(Z (Z)))
Proof. Set derivative of [r (r(ZZZZ) with respect to Z to zero:
(lr(Z
0 - LZ = 0 + 2 = AZ.
(8)
Öz
With K-+ oc:
( = AZ(K-1)
(9)
which indicates:
z(k) = AZ( A x- A Z = xXW.
(10)
According this theoretical foundation, it is straightforward that Theorem 2 is also applicable for the message passing of LightGCN in the setting of CF if we let A {0, 1] [Z)++ (Z),
X= = [2 and W = [U) where ||
refers to the concatenation operation. With this preliminary,
the proof to Theorem starts
Proof. DirectAU optimizes:
DirectAU
Σ (K.))D
||u - 1,
(11)
+
Σ loge 2||uu-1|1
+
Σ loge 2||i-i|||?
(12)
N,w'El
i,''
Since En
loge loge 21|
<= 0 and Σiper loge 2-1
0. we directly have
LiieaAAA < EiseeD |lu 1,1
BPR optimizes:
LBPR
Σ
Σ
log logo(sij Sik)
(13)
(D (L,))E
Σ
Σ
logo i - u. ik)
(14)
(DJ)DD (i.k)
c
Σ
Σ
log
(15)
cuiiis + u. 1s
(LEED (LA)ED
Σ
Σ
-ut is + log (cuT + +
(16)
(D (L,))D
Since ||u:|| ||, = 1 for any E l and E I. -1u--ig1== = 112 1,
-u. -i
|u 12 1. So Equation (16) can be written as:
} lu. 1 1 1+ log (04/1, + ce
CAPR
(17)
15

---

The maximum possible value of e ee ie is 2e, which is less than 10. Hence log (eu euT
1, which leads to the second part of Theorem Cbpr (t llu igll?.
B Dataset Description and Statisties
We conduct experiments on five commonly used benchmark datasets, that have been broadly utilized by the recommender system community, including Amazon book [40], Anime [29], Gowalla [5], Ye1p2018 [71], and MovieLens 1M [19]. Additionally, we also evaluate our method on large-scale industrial user-content recommendation dataset Internal, with statistics shown in Table 6.
Table 6: Statistics of datasets explored in this work. Due to privacy constrains, we only report
approximated values for Internal dataset.
Dataset
Users
# Items
Interactions
Sparsity
Amazon-book
52,643
40,981
2,984,108
99.94%
Anime
73,515
12,295
7,813,727
99.13%
Gowalla
29,858
40,981
1,027,370
99.91%
Yelp-2018
31,668
38,048
1,561,400 1,66
99.87%
MovieLens-1M
6,040
3,629
836,478
96.18%
Internal
~0.5
~0.2M
~7M
99.99%
C Additional Experimental Settings
C.1 Evaluation Protocol
We evaluate all models using metrics adopted in previous works, including NDCG@20 and Re- call@20 [20]. For the dataset split, we conduct the group-by-user splits and randomly select 80%, 10%, and 10% of observed interactions training, validation, and testing adopt an early stopping strategy, where the training will be terminated if the validation NDCG@20 stops increasing for 3 continuous epochs. We models with the best validation performance report the performance. Besides, the evaluation metrics are computed by the all-ranking protocol, where all items listed as candidates [45]. We explore this strategy since we want to evaluate the representation quality of all users. All experiments are conducted 10 times with different seeds, and we report both means and standard deviations across independent runs.
respectively. We
C.2 Hyper-parameter Tuning
We only conduct 25 searches per model for all methods to ensure the comparison fairness, so that our experiments are not biased methods with sophisticated blper--prameee we set the embedding dimensions for all models to 64 (i.e., d = 64) to ensure fair comparison, since larger dimension usually leads better performance in CF methods. For TAG-CF, we only tune m and in Equation (7) during test time from the list of [-2, -1.5, -1, -0.5, OJ. Besides, we train all models using Adam optimizer. TAG-CF's sensitivty to m and
search spaces. Furthermore,
visually plotted in Figure We can observe that m and are important for the success of TAG-CE Fortunately, across datasets, the optimal selection of m and is pretty similar (e.g.. m=n=-0.5 or m=n=0). The other solution to automatically tune and n could be initialzing and n to -0.5 (i.e., the value that generally works well across datasets) and conducting gradient descent on them using the training loss. But in this work we observe that manually tuning them on small set of candidates can already deliver promising results.
C.3 Implementation Detail
We conduct most of the baseline experiments with RecBole [81]. Besides, we use Google Cloud Platform with 12 CPU cores, 64GB RAM, and a single V100 GPU with 16GB VRAM run all experiments.
16

---

Table Improvement of TAG-CF to TAG-CF. Degree cutoffs are selected according to Figure 3.
Metric
Yelp-2018
Gowalla
Amazon-book
Anime
BPR
NDCG@20 20 Recall @20 Running Time
27.1% 31.4%
10.3%
122.4% 119.2%
0%
14.2%
0%
8%
4%
9%
0%
DIRECTAU
NDCG @ 20
34.1%
22.5%
98.3%
0%
Recall 20
29. 29.2%
30.1%
104.1%
0%
Running Time
8%
4%
9%
0%
RT Impr. (%)
Ama z.om zon-Book
250
Yelp-2018
Perf. Impr. (%)
200
150
100
10
Pert Impre (96)
Impe. (56)
50
100
10
RT Impe, (%)
Perf. Impe, (%)
500
-50
10
10P
10?
10P
10
10
107
10
Gowalla
150100
Anime time
Perf. Impr. (%)
R'T Impr. (%)
100
50
100
RT Impr. (96)
0
RT --p---pp---e (%6
50
Pert Impr. (%)
-50
Perf Impr- (%
-100
10
10'
10
101
102
10a
User Degree
User Degree
Figure 3: Improvement of TAG-CF+ w.r.t. different cutoffs. Yellow dashed lines indicate TAG-CF, and black circles refer to the optimal degree cutoff that TAG-CF# selects.
D Degree Cutoff Selection for TAG-CF+
We first sort users according to their degree and split the sorted list into 10 user buckets?, where each bucket contains non-overlapped users with similar degrees. Starting from the bucket with the lowest user degree, TAG-CF+ keeps applying -sggaimeeaggregttiion demonstrated in Equation (7) to all buckets until the validation performance starts to decrease or the performance improvement is less than 2% compared with TAG-CF. The degree cutoffs circled in Figure are the ones selected by this strategy and most of them correspond to the most performant configuration, shown in Table 7.
E Analysis of TAG-CF through Up-sampling
In Section 3, we connect CF ob- jective functions to message passing and show that they inadvertently con- duct message passing during the back- propagation. Since this inadvertent message passing happens during the back- proo--rrppggaion, its performance is positively correlated the amount of training signals a user/item can get. In the case of CF, the amount training signals for a user is directly proportional to the node degree of this user. High-degree active users naturally benefit more from the inadvertent message passing from objective functions like BPR and DirectAU, because they acquire more training signals from the objective function. Hence, when explicit message passing is applied to CF methods, the performance gain for high-degree users is less significant than that for
Table 8: Improvement (NDCG@ 20) brought by TAG-CF at different degree cutoffs and upsampling rates on ML-IM.
Up-sumpling Degree
sp-sample
Up-sample
MF
TAG-CF
Impr. (7%)
MF
TAG-CF
Impe. (1%)
40
38.8%
30.3%
80
20.10
27.13
35.9%
18.10
23.30
26.8%
160
19.39
26.63
36.6%
17.93
23.37
29.8%
'The number of buckets can be set to arbitrary numbers for finer adjustments. In this study, we pick 10 as proof of concept.
17

---

Table 9: Recommendation performance (i.e., NDCG@ 10 and Recall 10) of all models across users with different numbers of interactions. Setting explored in this table is the same as what Table has.
Method
NGCF
LighKICN
ENMP
IAGGF
Impe: (T>
MF
+TAG
Impe (193)
UliaaGNN
+TAG-C
Impe (11))
NDCG 10
LUW-DROREL USERS (LowER PERcUNTILE)
Krazsn-Busk
3,38
11
3.32
3.54
6.8%
3,705
3.49
8.7%
Acism
7.
9.77
738
8003 12
1.9%
14.25
1009
7.0%
Govalla la
5 5.542007
6.59
2:56
2682.0
5.4%
6.73amm
2.9%
5.38
6.05
Nel p-2018
2,80
2.51
180
1,80
5.2%
3.52
3.57,00e
1,68
2,82
3.16
12.
MovieLens 18
15.14
1781g030
12.55ana
15.552022
23.9%
14.44ac.1
20.1
29.2%
16.39and7
19.50g223
19.2%
NDCG 10
OVERALL
frazon-look
4.35
108
3.8
7.4%
508
1.9%
1.60
3:83 3:83,020
6.8%
Amine
1.99
9.22
10,63a001
10.929931
3.1%
8,52a0.3
9.23.m1
14.2%
10.6610.30
10.99.
3.1%
Govalla
3.65
3:49
2.1%
6.0
1.9%
1.59
5.90 20
6.1%
Telp-20st 2018
3.19
3.440m
2.18.0:
2.26,a00
3,66
3.59
2,5%
2.88
5.21
11.5%
Movielens-11
35.94
18
14.12
15
15,801g0
11.9%
15
20.5
32,675
18.25
20143
12.1%
RECALL
Low www-DEORSS
USERS (LOWER PERCENTILE
Arazon-Buok
3.75
10
365
399,..1
7.66
4.57
4.74.0.00 10
2.77
10:10
1288g0
14.0
15.15g0
3.5%
a1.41
12.38gn
14.52.030 30
3:224034
788
3.99
3.74
4.2%
7.82403
7.00.0 1s
6:40,0 is
6,66- 1
Te1p-0018
164
2.57
5.5%
3.602a032
2622n.3
2:02.a to
11.5%
Norielens 18
6.60pn:
15
5.73pms:
2007
23.4%
6.90gn.e
622
19.2%
7.66
8,54 50
11.5%
OIVERA1E
feazzn-fook
3,624035
46
381
3.09.00
4.48
of
2.8
5.00
11:12an25
12
13.45
18.7
1.9%
11.43ata4
14.19
24.44.0 30
Goralla
7.43
4.012nar
4.0
0.8%
1,57anss
7.72
2.0%
6.56
6.77
3,4%
telp-2018
3.39
3.73
2.32anss
2.41aa0o
3.7%
3.87
3.
1.83
18.7%
MovieLens-11 18
7,00
7.062020
6.20
2.02
13.2%
7.08
8:33
8,48
7.93
8.75 37
10.3%
125
27%
4.003
0.03
9.4
4/2
-159
-6,0025
6872
0.44
255
339
77
40040
4:34
85
0.00
00013
45
70
4
E7-
-0007
6:02
18
1.
3.00
3.10
AUS4
+3.24
-0.5
4329
22
-3277
+:44
3.34
37
6:22
Figure 4: The sensitivity TAG-CF to m and n in Equation (7). Numbers reported in these plots are performance improvement (%) brought by TAG-CF to MF trained by DirectAU [57] on Recall @ 20.
low-degree users. Because the contribution of the message passing over high-degree nodes has been mostly fulfilled by the inadvertent message passing during the training.
To quantitatively prove this line of theory, we incrementally up-sample low-degree training examples and observe the performance improvement that TAG-CF could introduce at each upsampling rate. our line of theory is correct, then we should expect less performance improvement on low-degree users for a larger upsampling rate. The results are shown in Table 8. From this table, though upsampling low-degree users hurts the overall performance, we can observe that the performance improvement brought by TAG-CF for low-degree users decreases, as the upsampling rate increases.
According to this experiment, we can conclude that the more supervision signals a user receives (no matter for a low-degree or high-degree user), the less performance improvement message passing can bring. This experiment quantitatively shows why the performance improvement of high-degree users could be limited more than low-degree users. Because high-degree users naturally receive more training signals during the training whereas low-degree users receive fewer training signals.
F Experiments on Ranking Metrics @ 10
This section shows the performance of all models as well as TAG-CF's improvement to them when evaluated with ranking metrics with 10 candidates. The results are shown in Table 9, with similar trends as we have observed Table
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This paper discusses how and why does message passing help collaborative filtering. We approach this question by analyzing from two perspectives (e.g., Section 3) and propose TAG-CF given findings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed limitations at the very end of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: We propose theorem in our main paper and provide proofs in the appendix (i.e., Appendix 1).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
setups to reproduce our experiments (i.c., Ap- pendix C.2). Besides, also provide source code to reproduce our experiments.
Justification: We provide
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

tuning Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We append our code others to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ https//ccppblccccuccceeeeeecc wwtpc///gude///eeeeeeeeee /CodeSumm iimmiissiionnn wwwsnnolllllooooeoeo..... onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are specified in C1 and strategies are specified in C2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run our experiments 10 times and report both mean and standard deviation of numbers report.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Resources used are specified in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https ftttpa://neurippi htt.:ppc....ppppccccc.cc.ccceeee
Answer: [Yes]
Justification: We conform the code Of Ethics and have some discussion at the very end of our paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: observe no ethical concern entailed by our proposal, but we note that both ethical or unethical applications based on collaborative filtering may benefit from the effectiveness of our work. Care should be taken to ensure socially positive and beneficial results of machine learning algorithms.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: With in the folder of our code, we include the license of all code, data, and tools we use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have doestrings for all functions in our code and we also provide readme file to help others use our code.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "BuSKPDmwKZ", "reviewer": "Reviewer_jYGx", "summary": "This paper investigates the role of message passing (MP) in collaborative filtering (CF). Unlike most GNN-based CF research, which assumes that performance gains arise from improved representation learning through GNNs, this work questions that assumption. Through empirical experiments and theoretical analyses, the paper finds that the key contributions of MP in CF are: 1) forward message passing rather than back-propagation, and 2) benefits for low-degree users instead of high-degree users.\n\nBased on these observations, this work propose a simple and efficient plug-in method called TAG-CF. TAG-CF can be easily integrated into any well-trained CF model by performing a single layer of message passing during the inference stage. To further reduce computational workload, TAG-CF+ conducts MP only on low-degree users. Experiments on four datasets verify the effectiveness of TAG-CF in improving CF models with almost no additional computational overhead.", "questions": "Besides the weakness part, I also have some questions:\n\n1. What is the setting of w/o both in Table 1. If it is a MF model using BPR, how can it achieve such good performance (Recall@20=18.42). In Table 2, MF model also shows superisingly good performance on most dataset, almost comparable to LightGCN, how does the MF set and tuned in the experiment?\n2. In Table 2, low degree users' performance even better than overall performance, which seems to be inconsistent with Figure 1, are there any explanations?\n3. How does m and n be tuned in the experiment? Are there any sensitivity experiments on these important hyper-parameters?", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on an important open problem: what is the role of MP plays in CF. This problem is worth investigating, and this work attempts to answer it from both empirical and theoretical perspectives.\n2. This article is easy to follow, with a natural transition from the analysis of MP to the proposal of TAG-CF.\n3. The proposed TAG-CF is efficient and effective. The plug-in nature of the method ensures the industrial application of the proposed method.", "weaknesses": "1. The discussion following Theorem 1 \"both BPR and DirectAU optimize the objective of message passing\" is not rigorous enough. Here are some aspects regarding to the discussion:\n   * The assumption that $\\|\\mathbf{u}_i\\|^2=\\|\\mathbf{i}_j\\|^2=1$ does not hold in LightGCN, thus the discussion on the theorem is only empirical but not strictly hold for LightGCN. In other words, the BPR loss for real-world LightGCN may not be upper-bounded by the objective of message passing.\n   * Even if the claim \"both BPR and DirectAU optimize the objective of message passing\" holds, there should be an additional experiment to show that directly train a CF model on the objective of message passing achieves comparable performance of LightGCN-BPR. \n\n2. Important baselines such as GFCF [1] and SVD-GCN [2] are missing. These works also dedicated to speed up GNN-based CF. On Gowalla dataset, both GFCF and SVD-GCN cost only around 30s for training, which is more efficient than the proposed TAG-CF. Moreover, the reported Recall@20 on Gowalla are 0.1849 and 0.1905 respectively, also shows comparable or better performance. The authors need to add these baseline in the experiment and discuss what is the advantage of TAG-CF compared to these baselines.\n\n3. According to Figure1, it is not sufficient to draw the conclusion that \"Message Passing in CF Helps Low-degree Users More Compared with High-degrees\". In Yelp dataset, the improvement of degree=16 users is only about 15%, which is less than the value of most higher-degree users (degree from 17-34). And what is the improvement for the users whose degree is less than 16? Still, this empirical experiment on only two dataset is not enough to support this conclusion.\n\n[1] Yifei Shen, et al., “How Powerful is Graph Convolution for Recommendation?” CIKM 2021\n\n[2] Shaowen Peng, et al., “SVD-GCN: A Simplified Graph Convolution Paradigm for Recommendation”. CIKM 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We really appreciate your acknowledgment of our paper’s significance, written quality, and practicality. Our detailed response to your concerns is as follows:\n## W1.1: $|u_i|^2 = |i_j|^2 = 1$ does not hold in LightGCN\n* We agree that $|u_i|^2 = |i_j|^2 = 1$ does not hold in LightGCN due to the message passing (MP) and the mean averaging across embeddings from different MP layers. All of these operations will void this assumption. However, the main purpose of this theorem to support our claim that matrix factorization (MF) methods with trending objective functions (e.g., BPR and DirectAU) partially optimize the objective of MP with some additional regularization (i.e., dissimilarity between non-existing user/item pairs for BPR, and representation uniformity for DirectAU). Hence, directly optimizing these two objectives for MF partially fulfills the effects brought by MP during the back-propagation. We will refine our theorem and restrict it to MF methods, where the assumption of $|u_i|^2 = |i_j|^2 = 1$ makes sense. We sincerely appreciate your constructive feedback and the refined theorem will be: “During the training of MF methods, objectives of BPR and DirectAU are strictly upper-bounded by the objective of message passing.”\n## W1.2: Directly training MF using the message passing objective\n* The objective of MP layer is $\\mathcal{O}=\\min_\\mathbf{Z}\\{tr(\\mathbf{Z}^\\intercal\\mathbf{L}\\mathbf{Z}))\\}$, which enforces embeddings of adjacent node to be similar. Training one MF model directly optimizing this objective is equivalent to BPR without negative sampling or DirectAU without the uniformity term, which will lead to overfitting and collapsing issues.\n* The purpose of our theorem is to connect the gain brought by the MP to node degree. Our theorem shows that BPR and DirectAU partially conduct inadvertent MP during the back-propagation. In the case of CF, the amount of training signals for a user is directly proportional to the node degree. High-degree active users naturally benefit more from the inadvertent MP from objective functions, because they acquire more training signals. Hence, when explicit MP is applied to CF methods, the performance gain for high-degree users is less significant than that for low-degree users. Because the contribution of the MP over high-degree nodes has been mostly fulfilled by the inadvertent MP during the training.\n* Retrospecting on your suggestion, we believe that studying the direct correlation between the objective of MP and matrix factorization models can provide understanding towards the role of MP for CF. Hence, we provide the value of the MP objective quantified by $\\sum_{(i,j)\\in\\mathcal{D}} ||\\mathbf{u}_i - \\mathbf{i}_j||^2$ at different training steps. The results for MF trained with DirectAU on MovieLens-1M is shown below:\n\n|Training steps (epoch)|0|2|4|6|8|10\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|$\\|\\|u_i-i_j\\|\\|^2$|0.931|0.493|0.478|0.464|0.461|0.461\n\n* We notice that the objective of MP is gradually optimized as the training progress and plateaus when the model converges, which echos with our finding that it optimizes the objective of MP with some additional regularizations.\n## W2: Missing important baselines\n* Thanks for pointing this out. Please see G1 (general response) for details. In summary, we (i) add experiments over 4 efficient baselines, and (ii) add a dedicated section about related efficient training for matrix factorization (MF). TAGCF consistently improves efficient MF methods (i.e., GC-FC, SVD-GCN, SVD-AE, and Turbo-CF). On average, TAGCF improves these methods by 5.6%, 8.9%, 3.9%, 7.8%, and 7.6% on our five datasets resp. \n## W3, Q2: Insufficient conclusion in Figure 1\n* In Figure 1, we indeed observe a local increasing trend in performance improvement from degree 16 to 20. However, when we compare the performance improvement of the whole low-degree users to that of the high-degree users, we can still observe that the improvement for low-degree users is larger than high-degree users (e.g., 4.8% on low-degree vs. 2.6% on overall for ENMF and 11.2% vs. 10.4% for UltraGCN). We do admit that there are certain cases where the low-degree improvement is slightly worse than the overall (i.e., 1.7% on low-degree vs. 1.8% on overall for MF). In fact, in these cases, the gaps in-between are very marginal (i.e., 0.1%) compared with gaps we observe for other cases (i.e., 2.2% and 0.8%). In Table 2, only 3 out of 18 cases (i.e., 16.7%) are scenarios where low-degree improvement is smaller and all remaining 15 cases (83.3%) support our findings that low-degree improvement is larger. Besides, even in these three rare cases, our proposed TAGCF still improves the performance (i.e., 6.7%, 2.3%, and 1.8% overall improvement in NDCG). So we believe that the conclusion we draw in this section is sufficient given abundant supporting evidence from experiments. \n## Q1: Setting in Table 1\nWe train both MF and LightGCN using the DirectAU loss and explore the same setting we have for Table 2. We individually tune hyper-parameters for MF and LightGCN (i.e., learning rate, batch size, and weight decay) to make sure that all models perform at their utmost, so that our findings are not based on weaker baselines. A vanilla MF model trained by DirectAU can already outperform LightGCN trained with BPR loss, which is the reason why we explore DirectAU loss for both MF and LightGCN in the first place. When we train both models using DirectAU, their performance gap is not as big as it is for gaps between models trained with BPR loss. We will include a dedicated section describing the hyper-parameters we used for each dataset. \n## Q3: Sensitivity experiments for m and n\n* Thanks for pointing this out. Please refer to G2 in our general response for details.\n\n**In light of our answers to your concerns, we hope you consider raising your score. If you have any more concerns, please do not hesitate to ask and we'll be happy to respond.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyPuv0WyGp", "reviewer": "Reviewer_yNfp", "summary": "The paper investigates on an interesting topic: how message passing is playing a role in graph-based recommender systems. Upon experiments on LightGCN model, authors posit the advantages of message passing lie in the forward pass to aggregate neighborhood information while at the same time, backward propagation has little effect on the results. Based on this assumption, authors propose test-time aggregation for collaborative filtering where the representation of users/items is aggregated at the testing time for more efficient training.", "questions": "1. What is the detailed pre-training method for the method? I am assuming the pre-training strategy is impactful for the results.\n2. An experiment on different pre-training methods is needed.\n3. Authors design m,n hyper-parameters to adjust the neighborhood impact. Are there any findings we can obtain from the two hyper-parameters? I would like to see hyper-parameter testing on them.", "limitations": "See weakness", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies roles for message passing in graph-based recommendation. \n2. It provides more insights for researchers to understand the mechanism behind graph-based recommendation.\n3. The proposed method is efficient.", "weaknesses": "1. Lack of pre-training description\n2. Lack of hyper-parameter analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We really appreciate your acknowledgment of our proposal's significance, insights, and efficiency. Please see below for our detailed response:\n\n## W1, Q1, Q2: Pre-training strategies for MF models \n* We agree with you that the pre-training strategies are impactful for the results, as TAG-CF is a test-time augmentation framework whose performance significantly depends on the starting performance of the base model that it improves on. In Table 2 of our original manuscript, we applied TAG-CF to MF methods pre-trained by ENMF, DirectAU, and UltraGCN. In Table 5, we also applied TAG-CF to MF derived by BPR. All these applications of TAG-CF to MF methods pre-trained by different methods show consistent performance improvement. Nevertheless, as suggested by other reviewers, we also apply TAG-CF to four other training-free matrix factorization methods (i.e., GFCF, SVD-GCN, SVD-AE, and Turbo-CF). As shown in the one-page additional material, TAGCF can still consistently improve the performance of these efficient matrix factorization methods (i.e., GC-FC, SVD-GCN, SVD-AE, and Turbo-CF). On average, TAGCF improves these four methods by 5.6%, 8.9%, 3.9%, 7.8%, and 7.6% on Amazon-book, Anime, Gowalla, Yelp-2018, and  MovieLens-1M respectively. \n\n## W2, Q3: Sensitivity experiments for m and n. \n* Thanks for pointing this out. Please refer to G2 in our general response for details.\n\n**In light of our answers to your concerns, we hope you consider raising your score. If you have any more concerns, please do not hesitate to ask and we'll be happy to respond.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IlViNPJF57", "reviewer": "Reviewer_3fxq", "summary": "This paper investigates the role of message passing in collaborative filtering, providing empirical analysis of its impact. Based on their findings that message passing primarily benefits through additional neighbor representations during forward passes and helps low-degree nodes more than high-degree nodes, the authors propose TAG-CF, a test-time aggregation framework for collaborative filtering. TAG-CF demonstrates competitive performance with state-of-the-art graph-based collaborative filtering methods while significantly reducing computational overhead.", "questions": "Q1. The authors have overlooked related work, such as GF-CF[1], BSPM[2], and SVD-AE[3], which are models without trainable parameters and training stages. The authors need to include these in their experimental results for comparison with test-time aggregation research or describe their commonalities and differences in the related work section. These should be mentioned as essential research before discussing test-time aggregation.\n\nQ2. When the authors refer to low-degree nodes, is this based on the entire dataset or just the test or training set? If it's based on the entire dataset, couldn't the performance improvement be biased depending on how the test set is split?\n\nQ3. The method for selecting the degree cutoff in TAG-CF+ seems somewhat heuristic. Is there a more systematic method for choosing this parameter?\n\nQ4. What problems might arise when applying this to large-scale recommendation systems in practice?\n\nQ5. Have you analyzed how TAG-CF's performance varies with the structural characteristics of the graph (e.g., node centrality, average path length, etc.)?\n\nQ6. How does the dimension of user/item representations affect the performance of TAG-CF?\n\n> [1] Shen, Yifei, et al. \"How powerful is graph convolution for recommendation?.\" Proceedings of the 30th ACM international conference on information & knowledge management. 2021.\n>\n> [2] Choi, Jeongwhan, et al. \"Blurring-sharpening process models for collaborative filtering.\" Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.\n>\n> [3] Hong, Seoyoung, et al. \"SVD-AE: Simple Autoencoders for Collaborative Filtering.\" arXiv preprint arXiv:2405.04746 (2024).", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper experimentally analyzes the impact of message passing on collaborative filtering.\n2. Based on the analysis results, an efficient TAG-CF method is proposed.\n3. Extensive experiments were conducted on various datasets.", "weaknesses": "W1. Crucially, this paper overlooks related work, which do not require training.\n\nW2.  The paper claims that TAG-CF can achieve performance similar to training-time message passing with just one aggregation at test time. More detailed analysis is needed to determine if this holds true in all situations and under what conditions these results occur.\n\nW3. While it's claimed that performance improvement is greater for low-degree nodes, the theoretical explanation for this is somewhat lacking. A deeper analysis of why this phenomenon occurs is needed. In particular, it seems highly dependent on the degree cutoff.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We really appreciate your acknowledgment of our proposal's efficiency and our experiment's extensiveness. Please see below for our detailed response:\n## W1, Q1: Missing related works\n* Thanks for pointing this out. Please see G1 (general response) for details. In summary, we (i) add experiments over 4 efficient baselines, and (ii) add a dedicated section about related efficient training for matrix factorization (MF). TAGCF consistently improves efficient MF methods (i.e., GC-FC, SVD-GCN, SVD-AE, and Turbo-CF). On average, TAGCF improves these methods by 5.6%, 8.9%, 3.9%, 7.8%, and 7.6% on our five datasets resp. \n## W2: Analysis w.r.t. behaviors of TAG-CF\n* In our paper, we claim that test-time message passing (MP) improves the performance of MF methods similarly to training-time MP. We discover that MP (both at training and testing time) helps low-degree users more than high-degree ones. In Table 2, the majority of cases (15/18 cases, or 83.3%) support this finding. In rare cases where high-degree users benefit more, the gap between groups is marginal and our proposed TAGCF still improves the performance (i.e., 6.7%, 2.3%, and 1.8% overall NDCG improvement). TAGCF consistently improves the performance of MF methods over 6 datasets (Table 2). Besides, we also apply TAGCF to training-free baselines (i.e., GFCF, SVD-GCN, SVD-AE, and Turbo-CF), as you suggest, and we observe that TAGCF consistently improves upon them. The added experiments can be found in the 1pg supplement. On average, TAGCF improves 4 training-free methods by 5.6%, 8.9%, 3.9%, 7.8%, and 7.6% on five datasets resp.\n## W3: Analysis and discussion about degree cut-off\n* We agree with you that the behaviors of TAGCF are dependent on the degree cutoff, as we show that MP (both at training and testing time) improves for low-degree users more than high-degree ones. Empirically, we observe that the majority of cases (15/18 in Table 2) support our finding. Moreover, we prove that both BPR and DirectAU optimize the objective of MP with some additional regularization (i.e., dissimilarity between non-existing user/item pairs for BPR, and representation uniformity for DirectAU).\n* Combining this theory with the prior empirical observations, we show that these two supervision signals  inadvertently conduct MP in the backward step, even without explicitly treating interaction data as graphs (see lines 219-237). To substantiate, we slowly upsample low-degree training users and observe the improvement that TAGCF introduces at each upsampling rate (see Appx. E). \n## Q2: Definition of low-degree user\n* We agree with your point: when the full set is used to calculate degree, experiments might be biased towards testing splits. We use the _training set alone_ to calculate user degrees, as our findings are closely correlated with the amount of exposure a user embedding gets _during training_.\n## Q3: Degree cut-off selection \n* There are 2 strategies to select degree cut-offs in TAG-CF+. One optimizes ranking performance, and the other focuses on budget (i.e. performance/cost ratio). \n* For the former: We treat degree cut-offs as hyper-parameter tuning and select the one that with best validation metrics. We explore this in the original draft (circled optimal points in Figure 2/3). \n* For the latter: GNN inference in industrial settings is prohibitively expensive especially for high-degree users, as their receptive field increases exponentially. As shown in Figure 2, performance gain from TAG-CF+ plateaus with a relatively small user degree. In these cases, the cost for further conducting TAG-CF+ on high-degree users grows exponentially, but doing so brings limited gains. Hence, we can also select the cut-off s.t. the performance/cost ratio starts to plateau. We will add this discussion to our manuscript. \n## Q4: Challenges in large-scale RecSys\n* The challenges of applying TAGCF to large-scale RecSys are the utilization of graphs in large-scale ML pipelines. There are often O(100 million) users/items and O(trillion) of edges connecting them. Simply loading the entire sparse interaction matrix into CPU RAM is then infeasible. TAGCF utilizes graph knowledge while avoiding most of the computational overhead of MP. It is extremely flexible, simple to implement, and enjoys benefits of graph-based CF with minimal cost.\n## Q5: Other graph characteristics\n* Thanks for pointing this out. We conduct additional experiments on Katz and Betweenness centrality. We split users equally into 3 buckets according to these characteristics and show the improvement brought by TAGCF to each bucket. Since these characteristics are expensive to compute, we only conduct this experiment on MovieLens-1M. We observe that improvement gradually increases as measurements go down, which echoes our observations on node degree. \n\n||NDCG@20 Impr.|Recall@20 Impr.\n|:-|:-:|:-:|\n||Katz \nHigh|29.2%|7.2%\nMed|33.4%|7.9%\nLow|35.2%|8.9%\n||Betweenness \nHigh|27.8%|6.9%\nMed|34.7%|8.1%\nLow|35.6%|9.0%\n## Q6: Embedding dimension\n* We conduct experiments on the improvement from TAGCF to MF with different dimensions. Due to time limitations and costs for training larger models, we only conduct experiments on Anime and MovieLens-1M. We observe that the performance of MF increases as the dimension increases. TAG-CF can consistently improve the performance of MF models with similar trends as observed in the original draft. \n\n|Dimension|64|128|256\n|:-|:-:|:-:|:-:|\n|| Anime\nNDCG@20 MF|24.01|30.14|34.73\n+TAG-CF|27.25(9.8%)|32.82(9.0%)|37.96(9.3%)\nRecall@20 MF|29.15|35.78|38.43\n+TAG-CF|31.95(6.9%)|38.10(6.5%)|41.01(6.7%)\n||MovieLens-1M\nNDCG@20 MF|22.51|24.34|25.08\n+TAG-CF|29.65(31.7%)|32.18(32.2%)|32.73(29.57)\nRecall@20 MF|26.30|28.82|29.57\n+TAG-CF|28.40(8.0%)|31.09(7.9%)|31.87(7.7%)\n\n**In light of our answers to your concerns, we hope you consider raising your score. If you have any more concerns, please do not hesitate to ask and we'll be happy to respond.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMcjjJv2nw", "reviewer": "Reviewer_vJWR", "summary": "This paper rethinks the application of message passing mechanisms in collaborative filtering methods and makes two key findings: 1) Message passing (MP) improves collaborative filtering primarily through the forward pass rather than the backward propagation process, and 2) MP is more effective for cold-start users compared to warm users. Based on these findings, this paper proposes a novel test-time aggregation method tailored for collaborative filtering, supported by theoretical derivations to demonstrate its validity. Finally, extensive experiments validate the effectiveness of the proposed method.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper rethinks the message passing mechanisms in collaborative filtering and makes two novel key findings.\n\n2. The method has theoretical guarantees to ensure its effectiveness.\n\n3. The structure of this work is logical, and the writing is well-organized.", "weaknesses": "1. Equation 4 seems unable to separate the forward pass neighbor representation from the backward pass gradient update in $s_{ij}$. Therefore, it is theoretically challenging to validate this finding, even though Table 1 empirically demonstrates that the forward pass primarily drives the performance improvement.\n\n2. Typically, evaluation metrics for recommendation models use NDCG@10 and Recall@10, as users tend to focus on items ranked higher in the list while ignoring those ranked lower. Therefore, NDCG@10 and Recall@10 better reflect the true effectiveness of recommendation models.\n\n3. It is necessary to explain some abbreviations, such as 'OOM' and '-'.\n\n4. This paper has many grammatical problems, so it is suggested to improve it. For example, \"TAG-CF is extremely versatile can be used as a plug-and-play module to enhance...\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We really appreciate your acknowledgment of our proposal's significance, theoretical soundness, and presentation. Please see below for our detailed response:\n\n## W1: Clarification on Equation 4. \n* The purpose of Equation 4 is comparing differences between a vanilla matrix factorization model and the LightGCN model in terms of how user-item similarities are computed. Once we find any differences between these two models, we can individually ablate/remove the additional inductive biases given by LightGCN and observe the performance downgrade to determine their significance to the recommendation performance. Specifically, for matrix factorization models, the similarity computation is extremely intuitive and it is as simple as the inner product between the corresponding user and item embeddings (i.e., $s_{ij}=\\mathbf{u}_i^\\intercal \\cdot \\mathbf{i}_j$). Whereas for LightGCN, the similarity computation is a combination of the same inner product and inner products between adjacent node embeddings. In this paper, we start our analysis from perspectives of forward and backward passes. \n\n* From the perspective of backward pass, the idea is that while incorporating similarities between adjacent nodes, the positive supervision signal from the target user-item pair is also back-propagated to adjacency node embeddings. This could improve the recommendation performance because of the idea behind collaborative filtering (i.e., in the case of matrix factorization, neighbors’ similarity scores are highly correlated with the similarity of the target user-item pair). To determine if this is the case and how much can such back propagation improve the recommendation performance, we remove the gradients flowed back to neighbor embeddings (i.e., in implementation, we freeze the detach gradient operation from pytorch) and accordingly observe the performance downgrade. This variant is denoted as LightGCN w/o grad., which surprisingly shows little performance downgrade. \n\n* For the perspective of forward pass, the idea is that to calculate the similarity between a pair of user and item, LightGCN also incorporates a weighted summation of similarities adjacent nodes. The underlying assumption is that neighbors’ similarity scores are highly correlated with the similarity of the target user-item pair. To determine if this operation is helpful, we need one ablation that only removes the numerical values of neighbors’ similarities and keeps everything else intact. In this case, the most straightforward way is removing the neighbor similarities from the calculation yet at the same time keeping the gradients brought by this calculation so that we do not ablate both backward and forward passes at the same time. And this is one of our variants coined LightGCN w/o neigh. Info, which proves that neighbor information drives the good performance of message passing in LightGCN. \n\n* This phenomenon indicates that (1) both additional representations passed from neighbors during the forward pass and accompanying gradient updates to neighbors during the back-propagation help the recommendation performance, and (2) within total performance gains brought by MP, gains from the forward pass dominate those brought by the back-propagation. Comparing LightGCN with LightGCN w/o neigh. Info, we notice that the incorporation of gradient updates brought by MP is relatively incremental (i.e., ~2\\%). However, to facilitate these additional gradient updates for slightly better performance, LightGCN is required to conduct MP at each batch, which brings tremendous additional overheads. This rationale motivates our further investigation and the proposal of TAG-CF. \n\n## W2: Additional resutsl for ranking metrics at 10. \n* Thanks for pointing this out. We explore this setting following existing works such as LightGCN and UltraGCN. We agree with you that Recall and NDCG at 10 can better reflect the true performance. Following your suggestion, we accordingly report Recall and NDCG at 10 in the one-page supplemental material and we observe that TAG-CF consistently brings performance improvement to baseline models when we explore more strict metrics, which further validates our proposal. \n\n## W3, W4: Presentation improvement to the manuscript. \n\n* Thanks for your suggestions. In Table 2, “OOM” refers to out-of-memory and we use “-” to represent out-of-memory as well because we want to avoid repetitive denotations. We will clarify this point in the table caption. Furthermore, we will double check spelling and grammatical errors remaining in the manuscript to ensure the readability. \n\n**In light of our answers to your concerns, we hope you consider raising your score. If you have any more concerns, please do not hesitate to ask and we'll be happy to respond.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their feedback and constructive suggestions. We are pleased that most reviewers appreciated **the promising efficiency and effectiveness of our proposal**, e.g.,: "the proposed TAG-CF is efficient and effective and ensures the industrial application of the proposed method (jYGx)", "the proposed method is efficient (yNfp)", and "efficient TAG-CF method is proposed (3fxq)". Besides, most reviewers also recognized **the promising impact of our research**, said: "focuses on an important open problem (jYGx)", "provides more insights for researchers (yNfp)", and "rethinks the message passing and makes two novel key findings (vJWR)". 

**[G1: Additional experiments and discussions w.r.t. related works]** At the same time, multiple reviewers are concerned about the coverage of our related works and experiments, especially for those efficient baselines. We appreciate the insightful suggestions and agree that our work is relevant to other efficiency-focused efforts in recommender systems. To leverage this comment, we not only conduct additional experiments over four efficient baselines (i.e., GFCF [1], SVD-GCN [2], SVD-AE [3], and Turbo-CF [4]) but also add a dedicated section discussing related works about efficient training for matrix factorization. Before we dive into the experimental results, we would like to further note that TAG-CF is a test-time augmentation framework such that TAG-CF needs to be coupled with existing matrix factorization baselines (e.g., MF-BPR, MF-DirectAU, UltraGCN, GFCF, SVD-GCN, etc). It does not make sense to compare the performance of TAGCF across baselines because the better performance might come from TAGCF itself or a more performing backbone matrix factorization model, making the comparison between baselines non-trivial. In our original manuscript, we apply TAGCF to a series of matrix factorization methods (i.e., MF-BPR, ENMF, and UltraGCN) and TAGCF consistently enhances their performance. Following this setting, we also apply TAGCF to other matrix factorization methods as suggested by reviewers. Their performance and the improvement brought by TAGCF are systematically presented in the one-page pdf. From Table 10, we can observe that TAGCF can still consistently improve the performance of these efficient matrix factorization methods (i.e., GC-FC, SVD-GCN, SVD-AE, and Turbo-CF). On average, TAGCF improves these four methods by 5.6%, 8.9%, 3.9%, 7.8%, and 7.6% on Amazon-book, Anime, Gowalla, Yelp-2018, and MovieLens-1M respectively. 

Besides additional experimental results, we also drafted a section discussing related works as follows:

Efficient efforts in marix factorization. A branch of research specifically focuses on improving the efficiency of matrix factorization [1,2,3,4,5]. For instance, GFCF [1] and Turbo-CF [4] explore graph signal processing to linearly convolve the interaction matrix and use the resulted matrix directly for recommendation without training. Furthermore, SVD-GCN [2] and SVD-AE [3] utilize a low rank version of the interaction matrix to further accelrate the convolution efficiency and yet remain the promising performance. Besides, BSPMs [5] studies using diffusion process to gradually reconsturct the interaction matrix and achieves promising performance with fast processing. In parallel with these existing efforts, we propose to enhance any existing matrix factorization method through test-time augmentation that harnesses graph-based heuristics. 

[1] Shen, et al., How Powerful is Graph Convolution for Recommendation? CIKM21

[2] Peng, et al., SVD-GCN: A Simplified Graph Convolution Paradigm for Recommendation. CIKM21

[3] Hong, et al. SVD-AE: Simple Autoencoders for Collaborative Filtering. IJCAI24

[4] Park, et al. Turbo-CF: Matrix Decomposition-Free Graph Filtering for Fast Recommendation. SIGIR24

[5] Choi, et al. Blurring-sharpening process models for collaborative filtering. SIGIR23

**[G2: Additional hyper-parameter experiments on m and n]** Furthermore, multiple reviewer also ask about TAG-CF's sensitivity to hyper-parameters (i.e., m and n in the test-time message passing). To mitigate this concern, we plotted heat maps of the performance improvement brought by TAGCF to MF with different m and n configurations and show the results in the one-page pdf. From Figure 4, we can observe that m and n are important for the success of TAG-CF. Fortunately, across datasets, the optimal selection of m and n is pretty similar (e.g., m=n=-0.5 or m=n=0). The other solution to automatically tune m and n could be initialzing m and n to -0.5 (i.e., the value that generally works well across datasets) and conducting gradient descent on them using the training loss. But in this work we observe that manually tuning them on a small set of candidates can already deliver promising results. 


**Besides this general response, we also leveraged individual feedbacks in the designated section. We hope we have satisfactorily answered your questions. If so, could you please consider increasing your rating? If you have remaining doubts or concerns, please let us know, and we will happily respond. Thank you!**

Best regards, \
TAG-CF authors

Meta-review: This paper investigates the role of message passing (MP) in collaborative filtering (CF) and proposes a novel test-time aggregation method called TAG-CF. 

Key Strengths:

- Novel insights: The paper provides new perspectives on how message passing functions in graph-based CF, challenging existing assumptions.
- Efficiency: TAG-CF demonstrates competitive performance with state-of-the-art graph-based CF methods while significantly reducing computational costs.
- Versatility: The method can be easily integrated into various existing CF models as a plug-and-play module.
- Theoretical foundation: The authors provide theoretical derivations to support the validity of their approach.
- Extensive experiments: The paper includes comprehensive evaluations across multiple datasets and baselines.

Key Weaknesses:

- Overlooked related work: Initially, the paper failed to compare against or discuss some relevant efficient CF methods that do not require training (e.g., GF-CF, BSPM, SVD-AE). However, the authors addressed this in their rebuttal by adding experiments and discussions on these methods.
- Limited analysis of conditions: More detailed analysis is needed to determine under what conditions the performance of TAG-CF is - comparable to training-time message passing.
- Theoretical explanation: The theoretical justification for why low-degree nodes benefit more from message passing could be strengthened.
- Heuristic parameter selection: The method for selecting the degree cutoff in TAG-CF+ seems somewhat ad-hoc and could benefit from a more systematic approach.
- Evaluation metrics: Initially, the paper used NDCG@20 and Recall@20, while NDCG@10 and Recall@10 might better reflect real-world recommendation scenarios. The authors addressed this in their rebuttal by providing additional results for these metrics.

The paper presents a novel and efficient approach to collaborative filtering that challenges existing assumptions about message passing in graph-based recommender systems. The proposed TAG-CF method shows promise in terms of both performance and computational efficiency. While there were some initial oversights in related work comparisons and evaluation metrics, the authors have addressed most of these concerns in their rebuttal and reviewers increased their ratings. The paper is likely to generate interest and discussion in the recommender systems community.

While the paper is very close to be rejected, it improved significantly during the review process and the question of integration of other signal in collaborative filtering is both important and difficult from an industrial perspective. I feel the idea of MP is worth the be presented and discussed at the conference.  Also one of the reviewers didn't engaged into discussion yet other reviewers with similar concerns upgraded their scores after the rebuttal.

Please include Tabla 1 better in the final version.
Decision: Accept (poster)