How Does Message Passing Improve Collaborative Filtering?

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:27:03

Abstract: Collaborative filtering (CF) has exhibited prominent results for recommender systems and been broadly utilized for real-world applications.
A branch of research enhances CF methods by message passing (MP) used in graph neural networks, due to its strong capabilities of extracting knowledge from graph-structured data, like user-item bipartite graphs that naturally exist in CF. They assume that MP helps CF methods in a manner akin to its benefits for graph-based learning tasks in general (e.g., node classification). However, even though MP empirically improves CF, whether or not this assumption is correct still needs verification. To address this gap, we formally investigate why MP helps CF from multiple perspectives and show that many assumptions made by previous works are not entirely accurate. With our curated ablation studies and theoretical analyses, we discover that (i) MP improves the CF performance primarily by additional representations passed from neighbors during the forward pass instead of additional gradient updates to neighbor representations during the model back-propagation and (ii) MP usually helps low-degree nodes more than high-degree nodes.}Utilizing these novel findings, we present Test-time Aggregation for Collaborative Filtering, namely TAG-CF, a test-time augmentation framework that only conducts MP once at inference time. The key novelty of TAG-CF is that it effectively utilizes graph knowledge while circumventing most of notorious computational overheads of MP. Besides, TAG-CF is extremely versatile can be used as a plug-and-play module to enhance representations trained by different CF supervision signals. Evaluated on six datasets (i.e., five academic benchmarks and one real-world industrial dataset), TAG-CF consistently improves the recommendation performance of CF methods without graph by up to 39.2% on cold users and 31.7% on all users, with little to no extra computational overheads. Furthermore, compared with trending graph-enhanced CF methods, TAG-CF delivers comparable or even better performance with less than 1% of their total training times. Our code is publicly available at https://github.com/snap-research/Test-time-Aggregation-for-CF.

Main paper: 
 introduction: 
 Recommender systems are essential in improving users' experiences on web services, such as product recommendations on e-commerce websites (Wang et al., 2021a; Schafer et al., 1999) , video recommendations from streaming services (Gomez-Uribe & Hunt, 2015; Van den Oord et al., 2013) , friend suggestions by social media platforms (Sankar et al., 2021; Ying et al., 2018) , etc. In particular, recommender systems based on collaborative filtering (CF) have shown superior performance (Rendle et al., 2009; Wang et al., 2022; Koren et al., 2021) . CF methods use preferences for items by users to predict additional topics or products a user might like (Su & Khoshgoftaar, 2009) . These methods typically learn a unique representation for each user/item and an item is recommended to a user according to the similarity of their representations (He et al., 2017; Wang et al., 2015) .Recently, one popular line of research explores Graph Neural Networks (GNNs) for CF, exhibiting improved results compared with CF frameworks without the utilization of graphs (He et al., 2020; Wang et al., 2019; Yu et al., 2022; Cai et al., 2023) . The key mechanism behind GNNs is message passing, where each node iteratively aggregates information from its direct neighbors in the graph, and information from neighbors that are multiple hops away can be acquired by stacked convolution layers (Kipf & Welling, 2016; Veličković et al., 2017; Hamilton et al., 2017) . During the model training, traditional CF methods directly fetch user/item representations of an observed interaction (e.g., purchase, friending, click, etc.) and enforce their pair-wise similarity (Rendle et al., 2009) . GNNenhanced CF methods extend the aforementioned scheme by conducting stacked graph convolutions over the user-item bipartite graph (Wang et al., 2019) . They harness the resultant latent user/item representations after graph convolutions to calculate pair-wise similarity or affinity.A recent study (He et al., 2020) shows that removing several significant components of the graph convolution (e.g., learnable transformation parameters and activation functions) greatly enhances GNNs' performance for CF. Its proposed method (LightGCN) achieves promising performance by linearly aggregating neighbor representations without any transformation, and it has been used as the de facto backbone model for later graph-based CF methods due to its simple and effective design (Cai et al., 2023; Yu et al., 2022; Wu et al., 2021) . However, this observation contradicts GNN architectures for classic graph learning tasks, where GNN's performance without these components could be severely jeopardized (Oloulade et al., 2021; Wang et al., 2021b) . Additionally, existing research (He et al., 2020; Wang et al., 2019) assumes that the contribution of message passing for CF is similar to that for graph learning tasks in general (e.g., node classification or link prediction -they posit that node representations are progressively refined by their neighbor information and the performance gain is positively proportional to the neighborhood density as measured in node degrees (Tang et al., 2020) . However, according to our empirical studies in Section 3.2, message passing in CF improves low-degree users more than high-degree users, which also contradicts GNNs' behaviors for classic tasks (Tang et al., 2020; Hu et al., 2022) . In light of these inconsistencies between the behaviors of message passing for CF and classic graph learning tasks, we ask:What role does message passing really play for collaborative filtering?In this work, we investigate contributions brought by message passing for CF from two perspectives. Firstly, we unroll the formulation of message passing and show that its performance improvement could either come from the beneficial neighbor information or additional gradient updates to neighbor representations. With rigorously designed ablation studies, we empirically demonstrate that performance gains brought by the beneficial neighbor information dominate those brought by additional gradient updates. Furthermore, we analyze the performance distribution w.r.t. the user degree (i.e., the number of interactions per user) with or without message passing and discover that the message passing in CF improves low-degree users more compared to high-degree users. For the first time, we connect this phenomenon to Laplacian matrix learning (Zhu et al., 2021; Dong et al., 2019; 2016) , and show that popular supervision signals (Rendle et al., 2009; Wang et al., 2022) for CF inadvertently conduct graph convolution in the backward step even without treating the input data as a graph. Hence, when message passing is applied, high-degree users demonstrate limited improvement, as the benefit of message passage for high degree nodes has already been captured by the supervision signal.With the above takeaways, we present Test-time Aggregation for Collaborative Filtering , namely TAG-CF. Specifically, TAG-CF does not require any message passing during training. Instead, it is a test-time augmentation framework that only conducts a single message-passing step at inference time, and effectively enhances representations inferred from different CF supervision signals. The test-time aggregation is inspired by our first perspective that, within total performance gains brought by message passing, gains from the beneficial neighbor information dominate those brought by additional gradient updates. Applying message passing only at test time avoids repetitive queries (i.e., once per node and epoch) for representations of surrounding neighbors, which grow exponentially as the number of layers increases. Moreover, following our second perspective that message passing helps low-degree nodes more in CF, we further offload the cost of TAG-CF by applying the one-time message passing only to low-degree nodes. In short, we summarize our contributions as:• This is the first work that formally investigates why message passing helps collaborative filtering from multiple perspectives (i.e., information passed from neighbors, additional gradients for neighbors, and individual improvement gains of subgroups w.r.t. the node degree).• With our rigorously designed ablation studies and analyses, we demonstrate that message passing in CF improves the recommendation performance primarily by information passed from neighbors instead of additional gradients, and it usually helps low-degree nodes more than high-degree nodes.  
preliminary and related work: 
 Collaborative Filtering. Given a set of users, a set of items, and interactions between users and items, collaborative filtering (CF) methods aim at learning a unique representation for each user and item, such that user and item representations can reconstruct all observable interactions (Rendle et al., 2009; Wang et al., 2022; Koren et al., 2009) . CF methods based on matrix factorization directly utilize the inner product between a pair of user and item representations to infer the existence of their interaction (Koren et al., 2009; Rendle et al., 2009) . Whereas CF methods based on neural predictors use multi-layer feed-forward neural networks that take user and item representations as inputs and output prediction results (He et al., 2017; Zhang et al., 2019) . Let U and I denote the user set and item set respectively, with user u i ∈ U associated with an embedding u i ∈ R d and item i i ∈ U associated with i i ∈ R d , the similarity s ij between user u i and item i j is formulated as s ij = û⊺ i • îj . Graph Neural Networks. Graph neural networks (GNNs) are powerful learning frameworks to extract representative information from graphs (Kipf & Welling, 2016; Veličković et al., 2017; Hamilton et al., 2017; Xu et al., 2018b) . They aim at mapping each input node into low-dimensional vectors, which can be utilized to conduct either graph-level (Xu et al., 2018a) or node-level tasks (Kipf & Welling, 2016) . Most GNNs explore layer-wise message passing (Gilmer et al., 2017) , where each node iteratively extracts information from its first-order neighbors, and information from multihop neighbors can be captured by stacked layers. Given a graph G = (V, E) and node features X ∈ R |V|×d , graph convolution (Kipf & Welling, 2016) at k-th layer is formulated as:EQUATIONwhere h 0 i = x i , N (i) refers to the set of direct neighbors of node i, andW (k) ∈ R d k ×d (k+1)refers to parameters at the k-th layer transforming the node representation from d k to d (k+1) dimension.Recent works (Ma et al., 2022; 2021) have shown that GNNs make predictions based on the distribution of node neighborhoods. And GNN's performance improvement for high-degree nodes better than that for low-degree nodes (Tang et al., 2020; Hu et al., 2022) . They posit that node representations are progressively refined by their neighbor information and the performance gain is positively proportional to the neighborhood density as measured in node degrees.Message Passing for Collaborative Filtering. Recent research tends to apply the message passing scheme in GNNs to CF (He et al., 2020; Wang et al., 2019; Pal et al., 2020) . In CF, they mostly conduct message passing between user-item bipartite graphs and utilize the resultant representations to calculate user-item similarities. For instance, NGCF (Wang et al., 2019) directly migrates the message passing scheme in GNNs (similar to Equation ( 1)) and applies it to bipartite graphs in CF. LightGCN (He et al., 2020) simplifies NGCF (Wang et al., 2019 ) by removing certain components (i.e., the self-loop, learning parameters for graph convolution, and activation functions) and further improves the recommendation performance compared with NGCF. The simplified parameter-less message passing in LightGCN can be simply formatted as:EQUATIONwhere N (•) refers to the set of items or users that the input interacts with, u (0) i = u i , and i (0) i = i i . With K layers, the final user/item representations and their similarities are constructed as:EQUATIONAccording to results reported in LightGCN and NGCF (He et al., 2020; Wang et al., 2019) and empirical studies we provide in this work (i.e., Table 2 and Table 5 ), incorporating message passing to CF methods without graphs (i.e., matrix factorization methods (Rendle et al., 2009; He et al., 2017) ) can improve the recommendation performance by up to 20%. Utilizing LightGCN as the backbone model, later works try to further improve the performance by incorporating self-supervised learning signals (Lin et al., 2022; Yu et al., 2022; Cai et al., 2023) . Graph-based CF methods assume that the contribution of message passing for CF is similar to that for graph learning tasks in general (e.g., node classification or link prediction). However, whether or not this assumption is correct still needs verification, even though message passing empirically improves CF. There also exists a branch of research that aims at accelerating or simplifying message passing in CF by adding graph-based regularization terms during the training (Shen et al., 2021; Mao et al., 2021; Peng et al., 2022) . While promising, they still repetitively query representations of adjacent nodes during the training. 
how does message passing improve collaborative filtering?: 
 In this section, we demonstrate the reason behind why message passing helps collaborative filtering from two major perspectives. The first one focuses on inductive biases brought by the message passing explored in LightGCN, the de facto backbone model for graph-based CF methods. Whereas the second perspective focuses on the performance improvement on different node subgroups w.r.t. the node degree with and without message passing.Following the definition in Equation ( 2), given a one-layer LightGCN1 , we unroll the calculation of the similarity s ij between any user u i and item i j as the following:EQUATIONWith derived similarities between user-item pairs, their corresponding representations can be updated by objectives (e.g., BPR (Rendle et al., 2009) and DirectAU (Wang et al., 2022) ) that enforce the pair-wise similarity between representations of user-item pairs in the training data.CF methods without the utilization of graphs directly calculate the similarity between user and item representations with their own representations (i.e., s ij = u ⊺ i • i j ), which aligns with the first term in Equation (4). Comparing it with the formulation in Equation (4), three additional similarity terms are introduced as inductive biases: similarities between users who purchase the same item (i.e., u ⊺ i • u n ), between items that share the same buyer (i.e., i ⊺ n • i j ), and between neighbors of an observed interaction (i.e., i ⊺ n • u n ). With these three additional terms from message passing, we reason that the performance improvement brought by message passing to CF methods without graph could come from (i) additional neighbor information (i.e., three extra terms in Equation ( 4)) positively correlated with user/item interactions, or (ii) additional gradient updates to correlated representations. The performance improvement brought by message passing decreases as the user degree goes up.From Table 1 , we observe that the performance of all variants is downgraded compared with Light-GCN, with a significant degradation on LightGCN w/o neigh. info . This phenomenon indicates that (i) both neighbor information and additional gradients brought by message passing help the recommendation performance, and (ii) within total performance gains brought by message passing, gains from the beneficial neighbor information dominate those brought by additional gradient updates. Comparing LightGCN with LightGCN w/o grad. , we notice that the incorporation of gradient updates brought by message passing is relatively incremental (i.e., ∼2%). However, to facilitate these additional gradient updates for slightly better performance, LightGCN is required to conduct message passing at each batch, which brings tremendous additional computational overhead.Recent works (Tang et al., 2020; Hu et al., 2022; Liu et al., 2021) show that the performance improvement brought by GNNs is positively correlated to the number of neighbors of the nodes in question. Both empirical and theoretical evidence have demonstrated that GNNs usually perform satisfactorily on high-degree nodes with rich neighbor information but not as well on low-degree nodes. While designing graph-based model architectures for CF, most existing methods directly borrow this line of observations (Wang et al., 2019; He et al., 2020) and assume that the contribution of message passing for CF is similar to that for graph learning tasks in general. However, whether or not these observations still transfer to message passing in CF remains questionable, as there exist architectural and philosophical gaps between message passing for CF and its counterparts for GNNs, as discussed in Section 2. For instance, training signals for node classification usually come from labels agnostic of the graph structure; whereas the training signal for CF directly utilizes links between users and items to learn their representations. To validate these hypotheses, we conduct experiments over representative methods (i.e., LightGCN and matrix factorization (MF) trained with BPR) and show their performance w.r.t. the node degree in Figure 1 .We observe that, overall both MF and LightGCN perform better on high-degree users than low-degree users. According to the upper two figures in Figure 1 , MF behaves similarly to LightGCN, even without treating the input data as graphs, where the overall performance for high-degree user is stronger than that for low-degree users. However, the performance improvement of LightGCN from MF on low-degree users is larger than that for high-degree users (i.e., lower two figures in Figure 1 ). According to literature in general graph learning tasks (Hu et al., 2022; Liu et al., 2021; Tang et al., 2020) , the performance improvement should be positively proportional to the node degree -the gain for high-degree users should be higher than that for low-degree users. This discrepancy indicates that it might not be appropriate to accredit contributions of message passing in CF directly through ideologies designed for classic graph learning tasks (e.g., node classification and link prediction).To bridge this gap, we connect supervision signals (i.e., BPR and DirectAU) commonly adopted by CF methods to Laplacian matrix learning. The formulation of BPR (Rendle et al., 2009) and DirectAU (Wang et al., 2022) without the incorporation of graphs can be written as:EQUATION)where D refers to the set of observed interactions at the training phase and i ′ and u ′ refers to any random user/item. According to works on Laplacian matrix learning (Zhu et al., 2021; Dong et al., 2019; Ma et al., 2021) , learning node representations over graphs can be decoupled into Laplacian quadratic form, a weighted summation of two sub-goals:EQUATIONwhere Z refers to the node representation matrix after the message passing, X refers to the input feature matrix, and L refers to the Laplacian matrix. The first term regularizes the latent representation such that it does not diverge too much from the input feature; whereas the second term promotes the similarity between latent representations of adjacent nodes, which can be re-written as: tr Zhu et al. (2021) show that K layers of linear message passing exactly optimizes the second term in Equation ( 7). Given this theoretical foundation, we derive the following theorem w.r.t. relations between BPR, DirectAU, and message passing in CF: Theorem 1. Assuming that ||u i || 2 = ||i j || 2 = 1 for any u i ∈ U and I j ∈ I, objectives of BPR and DirectAU are strictly upper-bounded by the objective of message passing (i.e., L BPR ≤(Z ⊺ •L•Z) = (i,j)∈D ||u i -i j || 2 in CF bipartite graphs.(i,j)∈D ||u i -i j || 2 and L DirectAU ≤ (i,j)∈D ||u i -i j || 2 ).Proof of Theorem 1 can be found in Appendix A. According to Theorem 1, both BPR and DirectAU optimize the objective of message passing (i.e., (i,j)∈D ||u i -i j || 2 ) with some additional regularization (i.e., dissimilarity between non-existing user/item pairs for BPR, and representation uniformity for DirectAU). Hence, directly optimizing these two objectives partially fulfills the effects brought by message passing during the back-propagation. Combining this theoretical relation with the aforementioned empirical observations, we show that these two supervision signals could inadvertently conduct message passing in the backward step, even without explicitly treating interaction data as graphs. Since this inadvertent message passing happens during the back-propagation, its performance is positively correlated to the amount of training signals a user/item can get. In the case of CF, the amount of training signals for a user is directly proportional to the node degree. High-degree active users naturally benefit more from the inadvertent message passing from objective functions, because they acquire more training signals from the objective function. Hence, when explicit message passing is applied to CF methods, the performance gain for high-degree users is less significant than that for low-degree users. Because the contribution of the message passing over high-degree nodes has been mostly fulfilled by the inadvertent message passing during the training.To quantitatively prove this line of theory, we incrementally upsample low-degree training examples and observe the performance improvement that TAG-CF could introduce at each upsampling rate. If our line of theory is correct, then we should expect less performance improvement on low-degree users for a larger upsampling rate. The results are shown in Appendix H.1 with supporting evidence. 
test-time aggregation for collaborative filtering: 
 In Section 3, we demonstrate why message passing helps CF from two perspectives. Firstly, w.r.t. the formulation of LightGCN, we observe that the performance gain brought by neighbor information dominates that brought by additional gradients. Secondly, w.r.t. the improvement on user subgroups, we learn that message passing helps low-degree users more, compared with high-degree users.In light of these two takeaways, we present Test-time Aggregation for Collaborative Filtering, namely TAG-CF, a test-time augmentation framework that only conducts message passing once at inference time and is effective at enhancing matrix factorization methods trained by different CF supervision signals. Given a set of well-trained user/item representations, TAG-CF simply aggregates neighboring item (user) representations for a given user (item) at test time. Despite its simplicity, we show that our proposal can be used as a plug-and-play module and is effective at enhancing representations trained by different CF supervision signals.The test-time aggregation is inspired by our first perspective that, within total performance gains brought by message passing, gains from the neighbor information dominate those brought by additional gradient updates. Applying message passing only at test time avoids repetitive training-time queries (i.e., once per node and epoch) of surrounding neighbors, which grow exponentially as the number of layers increases by the neighbor explosion phenomenon (Guo et al., 2023; Zhang et al., 2021; Zeng et al., 2021) . Specifically, given a set of well-trained user and item representations U ∈ R |U |×d and I ∈ R |I|×d , TAG-CF augments representations for user u i and item i i as:EQUATIONwhere m and n are two hyper-parameters that control the normalization of message passing. With m = n = -1 2 , Equation ( 8) becomes the exact formulation of one-layer LightGCN (i.e., Equation ( 2)). Empirically, we observe that the setup with m = n = -1 2 for TAG-CF does not always work for all datasets. This setup (i.e., m = n = -1 2 ) is directly migrated from message passing for homogeneous graphs (Kipf & Welling, 2016), which might not be applicable for bipartite graphs where all neighbors are heterogeneous (Dasoulas et al., 2021) . Unlike LightGCN which can fill this gap by adaptively tuning all representations during the training time, TAG-CF cannot update any parameters since it is applied at test time, and hence requires tune-able normalization hyper-parameters.Moreover, following our second perspective that message passing helps low-degree nodes more in CF, we further derive TAG-CF + , which reduces the cost of TAG-CF by applying the one-time message passing only to low-degree nodes with sparse neighborhoods. Focusing on only low-degree nodes has two benefits: (i) it reduces the number of nodes that TAG-CF + needs to attend to, and (ii) message passing for low-degree nodes is naturally cheaper than for high-degree nodes given the surrounding neighborhoods are sparser (mitigating neighbor explosion). The degree threshold that determines which nodes to apply TAG-CF + is selected by validation performance, with details in Appendix F.Leveraging our novel takeaways, TAG-CF can effectively enhance MF methods by conducting message passing only once at test time. It is extremely flexible, simple to implement, and enjoys the performance benefits of graph-based CF method while paying the lowest overall scalability. 
experiments: 
 In this section, we conduct extensive experiments to demonstrate the effectiveness and efficiency of TAG-CF. Specifically, we aim to answer the following research questions: RQ (1): how effective is TAG-CF at improving MF methods without using graphs, RQ (2): how much computational overhead does TAG-CF introduce, RQ (3): can TAG-CF effectively enhance MF methods trained by different objectives, RQ (4): how effective is TAG-CF + w.r.t. different degree cutoffs, and RQ (5): do behaviors of TAG-CF align with our findings in Section 3?Datasets. We conduct comprehensive experiments on four commonly used benchmark datasets, including Amazon-book, Anime, Gowalla, and Yelp2018. Additionally, we also evaluate on a large-scale internal production user-item recommendation dataset Internal. These datasets cover different domains and dimensions to fully evaluate all models.Baselines. We compare TAG-CF with two branches of methods: (1) CF methods that do not utilize graphs, including vanilla matrix factorization (MF) methods trained by BPR and DirectAU (Rendle et al., 2009; Wang et al., 2022) , and Efficient Neural Matrix Factorization (Chen et al., 2020) (denoted as ENMF). ( 2) Graph-based CF methods, including LightGCN (He et al., 2020) and NGCF (Wang et al., 2019) . Besides, we also compare with other graph-based CF methods that extend LightGCN by adding additional self-supervised signals for better performance, including LightGCL (Cai et al., 2023) , SimGCL (Yu et al., 2022) , and SGL (Wu et al., 2021) .Due to space limits, we include comprehensive discussions about dataset details, evaluation protocols, hyper-parameters, and other implementation details in Appendices B to E. For RQ (1), Table 2 shows the performances of MF methods (MF and ENMF) as well as that of the performances of them with TAG-CF applied on their learned representations. We observe that TAG-CF unanimously improves the recommendation performance for both of them. Specifically, across all datasets, TAG-CF on average improves the low-degree NDCG@20 by 4.6% and 9.1% and overall NDCG by 3.2% and 7.1% for ENMF and MF, respectively. We also observe a similar performance improvement for Recall@20, where TAG-CF on average improves the low-degree Recall@20 by 4.5% and 8.4% and overall Recall@20 by 2.1% and 7.2% for ENMF and MF, respectively.By comparing the performance gains brought by TAG-CF on low-degree users with that on all users, we notice that gains for low-degree users are usually higher. Hence, message passing in CF helps low-degree users more than for high-degree users. To answer RQ (5), the behavior of TAG-CF aligns with our second perspective in Section 3.2 that the supervision signal inadvertently conducts message passing. Consequently, the room for improvement on high-degree users could be limited, as part of the contributions from message passing has already been claimed by the supervision signal.Comparing TAG-CF with LightGCN in message passing. To answer RQ (5), TAG-CF aligns with our first perspective in Section 3.1 that the performance gain from beneficial neighbor information dominates their accompanying gradients.We further compare TAG-CF with state-of-the-art graph-based CF methods, with their performance and efficiency shown in Table 4 . Among these performant baselines, TAG-CF exhibits competitive performance, with an average rank of 1.2 on NDCG and 1.7 on Recall. Though not always the model that delivers the best performance, TAG-CF can deliver comparably promising results and introduces little computational overheads (i.e., ranked 1.0 for running time). Considering efficiency as one factor, TAG-CF achieves the best performance across all baselines and datasets with an average rank of 1.9.While performing on par with graph-based CF methods that aggregate neighbor contents at the training time, TAG-CF enjoys the performance benefits of message passing while paying the lowest overall scalability. To answer RQ (2), according to Table 3 , across all datasets, TAG-CF only introduces an average additional computational overhead of 0.05 × 10 3 seconds, which is less than 0.5% of the total training time for matrix factorization methods. Comparing the running time of LightGCN with that of TAG-CF, we can observe that the latter can significantly improve the computational time, and the speedup is proportional to the sparsity of the dataset.To answer RQ (3), besides DirectAU, we also conduct experiments on BPR loss, as shown in Table 5 . When applied to BPR, TAG-CF still consistently improves the performance by large margins (i.e., 6.3% and 5.1% average improvement on low-degree and overall NDCG respectively, and 4.4% and 3.2% on low-degree and overall Recall respectively). We notice that TAG-CF sometimes does not perform as competitively as LightGCN when both are trained with BPR. We check norms of learned representations from MF with BPR and discover that they have high variance since BPR does not explicitly enforce any regularization. This might not favor TAG-CF as a test-time augmentation method due to its simple design, which cannot adapt representations with high variance. To answer RQ (4), we apply TAG-CF + to four public datasets and the performance and the efficiency improvement are demonstrated in Figure 2. Overall, the running time improvement brought by TAG-CF + exponentially increases as the degree decreases, since low-degree users have sparse neighborhoods and there is hence less information for TAG-CF + to aggregation. When the degree cutoff is low (i.e., less than 100), the effectiveness of TAG-CF + proportional increases as the degree cutoff increases.When setting the cutoff to a user degree of around 100, on Amazon-Book, Gowalla, and Yelp-2018, TAG-CF + can further improve TAG-CF by 125%, 17%, and 11%, respectively, with efficiency improvement of 7%, 4%, and 8%. In these cases, TAG-CF + not only significantly improves the performance but also effectively reduces computational overheads. However, on these three datasets, after the cutoff bypasses a degree of 100, the performance improvement eventually decreases to the performance of TAG-CF (i.e., 100%), indicating that test-time aggregation jeopardizes the performance on high-degree nodes. On Anime, though no downgrade on high-degree users, the performance improvement of TAG-CF + to TAG-CF is incremental. These phenomenons not only demonstrate the effectiveness and efficiency of TAG-CF + , but also verify our findings in Section 3.2 that message passing in CF helps low-degree users more than high-degree users. 
conclusion: 
 In this study, we investigate how message passing improves collaborative filtering. Through a series of ablations, we demonstrate that the performance gain from neighbor contents dominates that from accompanying gradients brought by message passing in CF. Moreover, for the first time, we show that message passing in CF improves low-degree users more than high-degree users. A PROOF OF THEOREM 1Here we re-state Theorem 1 before diving into its proof:Theorem 1. Assuming that ||u i || 2 = ||i j || 2 = 1 for any u i ∈ U and I j ∈ I, objectives of BPR and DirectAU are strictly upper-bounded by the objective of message passing (i.e., L BPR ≤ (i,j)∈D ||u i -i j || 2 and L DirectAU ≤ (i,j)∈D ||u i -i j || 2 ). One preliminary theoretical foundation for Theorem 1 to hold is that a K-layer graph convolution network (GCN) exactly optimizes the second term in Equation ( 7), which has been proved by Zhu et al. (2021) . For ease of reading, we re-phrase it again as the following:Theorem 2. The message passing for GCN optimizes the following graph regularization term:O = min Z {tr(Z ⊺ LZ))}.Proof. Set derivative of tr(Z ⊺ LZ) with respect to Z to zero:EQUATION)With K→ ∞:Z (K) = AZ (K-1) (10)which indicates:EQUATIONAccording to this theoretical foundation, it is straightforward that Theorem 2 is also applicable for the message passing of LightGCN in the setting of CF if we let A = {0, 1} (|U |+|I|)×(|U |+|I|) , X = I |U |+|I| , and W = (U||I), where || refers to the concatenation operation.With these preliminaries, we can start the proof to Theorem 1 as follows:Proof. DirectAU optimizes:EQUATION) Since u,u ′ ∈U log e -2||u-u ′ || 2 <= 0 and i,i ′ ∈I log e -2||i-i ′ || 2 <= 0, we directly have L DirectAU ≤ (i,j)∈D ||u i -i j || 2 .BPR optimizes: 14) can be written as:L BPR = - (i,j)∈D (i,k) / ∈D log σ(s ij -s ik ) = - (i,j)∈D (i,k) / ∈D log σ(u ⊺ i • i j -u ⊺ i • i k ) (13) = (i,j)∈D (i,k) / ∈D -log e u ⊺ i •ij e u ⊺ i •ij + e u ⊺ i •i k = (i,j)∈D (i,k) / ∈D -u ⊺ i • i j + log e u ⊺ i •ij + e u ⊺ i •i k (14) Since ||u i || 2 = ||i j || 2 = 1 for any u i ∈ U and I j ∈ I, ||u i -i j || = 1 -2u ⊺ i • i j + 1 → -u ⊺ i •i j = 1 2 ||u i -i j || 2 -1. So Equation (EQUATION)The maximum possible value of eu ⊺ i •ij + e u ⊺ i •i k is 2e, which is less than 10. Hence log e u ⊺ i •ij + e u ⊺ i •i k < 1, which leads to the second part of Theorem 1: L BPR ≤ (i,j)∈D ||u i -i j || 2 . 
 
 Appendix: We conduct comprehensive experiments on four commonly used benchmark datasets, that have been broadly utilized by the recommender system community, including Amazon-book (McAuley & Yang, 2016) , Anime (Kaggle, 2023) , Gowalla (Cho et al., 2011 ), and Yelp2018 (Yelp, 2023) . Additionally, we also evaluate our method on a large-scale industrial user-content recommendation dataset -Internal. The detailed descriptions of these datasets are listed as the following:• Amazon-book: It is a widely used product recommendation benchmark that is sub-sampled from Amazon-review2 . In this dataset, a recommender system is asked to recommend books to users. We directly utilize the sub-sampled version created from the previous literature (Wang et al., 2022; He et al., 2020; Wang et al., 2019) to ensure a fair comparison and reproducible results. Previous works apply a 10-core setting (He & McAuley, 2016) , where each user/item at least has ten interactions.• Anime: It is an anime recommendation benchmark provided my MyAnimeList3 , where each user is able to add anime to their completed list and give it a rating. This dataset is a compilation of those ratings and a recommender system is asked to recommend anime to users.• Gowalla: This is a check-in dataset obtained from Gowalla, where each user share locations by checking-in clicks. In this dataset, a recommender system is asked to any location that a user might be interested in checking in. Similar to Amazon-book, we directly utilize the processed version created from the previous literature (Wang et al., 2022; He et al., 2020; Wang et al., 2019) .• Yelp-2018: This dataset is adopted from the 2018 edition of the yelp challenge4 In this dataset, a recommender system is asked to recommend businesses like local restaurants and bars to users. Similar to Amazon-book, we directly utilize the processed version created from the previous literature (Wang et al., 2022; He et al., 2020; Wang et al., 2019) .• Internal: It is a user-item interaction dataset collected from an anonymized social network platform. The dataset contains the user-item interactions from a sampled group of weekly active users of an anonymized country. The dataset contains roughly 7 million interactions, and it is split following standard train/val/test splittings.These datasets cover different domains and dimensions to fully evaluate all models. We download Amazon-book, Gowalla, and Yelp-2018 from the official Github repository of NGCF5 , and we acquire Anime from Kaggle6 . Their statistics are shown in Table 6 below. We evaluate all models using metrics adopted in previous works, including NDCG@20 and Recall@20 (He et al., 2020) . For the dataset split, we conduct the group-by-user splits and randomly select 80%, 10%, and 10% of a user's observed interactions as training, validation, and testing sets respectively. Besides, the evaluation metrics are computed by the all-ranking protocol, where all items are listed as candidates (Rendle et al., 2009) . We explore this strategy since we want to evaluate the representation quality of all users. All experiments are conducted 10 times with different seeds, and we report both means and standard deviations across independent runs.In this section, we describe hyper-parameters that we tune for all baselines and TAG-CF and report their optimal setups. We only conduct 25 searches per model for all methods to ensure the comparison fairness, so that our experiments are not biased to methods with sophisticated hyper-parameter search spaces. Furthermore, we set the embedding dimensions for all models to 64 (i.e., d = 64) to ensure a fair comparison, since a larger dimension usually leads to better performance in CF methods. We utilize Adam optimizer for all models. Hyper-parameter ranges and their optimal selections are reported as follows:• MF (BPR): We tune the learning rate from the list of [1e-4, 5e-4, 1e-3, 5e-3, 1e-2] and regularization weight (i.e., a term that regularizes the norm of learned representations) from the list of [1e-5, 1e-4, 1e-3, 1e-2, 0]. We fix the training batch size to 4096 across all datasets. he optimal selections for all datasets are listed as: Amazon-book: [learning rate=5e-4, regularization weight=1e-3], Gowalla: [learning rate=1e-3, regularization weight=0], Yelp-2018: [learning rate=5e-4 regularization weight=1e-5], Anime: [learning rate=5e-4, regularization weight=1e-4], and Internal: [learning rate=1e-3, regularization weight=1e-5]. • MF (DirectAU): We tune γ (i.e., the hyper-parameter that tunes the trade-off between alignment and uniformity) from the list of [0.1, 0.2, 0.5, 1, 2, 5, 10] and tune the weight decay value from the list of [0, 1e-8, 1e-6, 1e-4]. We fix the training batch size to 4096 and the learning rate to 1e-3 across all datasets. The optimal selections for all datasets are listed as: Amazon-book:[γ = 5, weight decay=1e-6], Gowalla: [γ = 1, weight decay=1e-6], Yelp-2018: [γ = 1, weight decay=1e-6],Anime: [γ = 0.5, weight decay=1e-6], and Internal: [γ = 2, weight decay=1e-8]. • LightGCN (BPR): We tune the learning rate from the list of [1e-4, 5e-4, 1e-3, 5e-3, 1e-2] and regularization weight (i.e., a term that regularizes the norm of learned representations) from the list of [1e-5, 1e-4, 1e-3, 1e-2, 0]. We fix the training batch size to 4096 and the number of message passing layers to 2 across all datasets. The optimal selections for all datasets are listed as: Amazon-book: [learning rate=1e-3, regularization weight=1e-4], Gowalla: [learning rate=1e-3, regularization weight=0], Yelp-2018: [learning rate=5e-3 regularization weight=1e-3], Anime: [learning rate=5e-4, regularization weight=1e-5], and Internal: [learning rate=5e-4, regularization weight=1e-5]. • LightGCN (DirectAU): We tune γ (i.e., the hyper-parameter that tunes the trade-off between alignment and uniformity) from the list of [0.1, 0.2, 0.5, 1, 2, 5, 10] and tune the weight decay value from the list of [0, 1e-8, 1e-6, 1e-4]. We fix the training batch size to 4096, the learning rate to 1e-3, and the number of message passing layers to 2 across all datasets. The optimal selections for all datasets are listed as: Amazon-book: Besides, we adopt an early stopping strategy to train all models, where the training for a given model will be terminated if its corresponding validation NDCG@20 stops increasing for 3 continuous epochs. We use models with the best validation performance to report the performance.We conduct most of the baseline experiments with RecBole7 (Zhao et al., 2021; 2022) (i.e., implementations of ENMF, MF, LightGCN, SimGCL, and SGL) . For DirectAU, we utilize the code from its official repository8 . We sincerely appreciate the authors of RecBole and baseline models for open-sourcing their valuable code and reliable implementations of baseline models. TAG-CF is User Degree User Degree In this section, we illustrate how TAG-CF + selects the degree cutoff to improve the recommendation efficiency and effectiveness. We first sort all users according to their degree and split the sorted list into 10 user buckets9 , where each bucket contains non-overlapped users with similar degrees (e.g., the first bucket contains user whose degrees are ranked in the 0 to 10% population). Starting from the bucket with the lowest user degree, TAG-CF + keeps applying test-time-aggregation demonstrated in Equation ( 8) to all buckets until the validation performance starts to decrease or the performance improvement is less than 2% compared with TAG-CF. The degree cutoffs circled in Figure 3 are the ones selected by this strategy and most of them correspond to the most performant configuration. The performance of TAG-CF + under the selected degree cutoff is shown in Table 7 . In our manuscript, we connect CF objective functions (i.e., BPR and DirectAU) to message passing and show that they inadvertently conduct message passing during the back-propagation. Since this inadvertent message passing happens during the back-propagation, its performance is positively correlated to the amount of training signals a user/item can get. In the case of CF, the amount of training signals for a user is directly proportional to the node degree of this user. High-degree active users naturally benefit more from the inadvertent message passing from objective functions like BPR and DirectAU, because they acquire more training signals from the objective function. Hence, when explicit message passing is applied to CF methods, the performance gain for high-degree users is less significant than that for low-degree users. Because the contribution of the message passing over high-degree nodes has been mostly fulfilled by the inadvertent message passing during the training.To quantitatively prove this line of theory, we incrementally upsample low-degree training examples and observe the performance improvement that TAG-CF could introduce at each upsampling rate. If our line of theory is correct, then we should expect less performance improvement on low-degree users for a larger upsampling rate. The results are shown in Table 9 .From this table, though upsampling low-degree users hurts the overall performance, we can observe that the performance improvement brought by TAG-CF for low-degree users decreases, as the upsampling rate increases. For instance, when we regard users with a degree less than 40 as lowdegree users, increasing the upsampling rate from 100% to 300% reduces the improvement margin by 8.1%, with similar trends on other degree cutoffs.According to this experiment, we can conclude that the more supervision signals a user receives (no matter for a low-degree or high-degree user), the less performance improvement message passing can bring. This experiment quantitatively shows why the performance improvement of high-degree users could be limited more than low-degree users. Because high-degree users naturally receive more training signals during the training whereas low-degree users receive fewer training signals. According to another recent work that uses realistic e-commerce datasets (Zheng et al., 2022) , the average user degree is usually less than 50, which aligns with the characteristics of the datasets we use in this work. Nevertheless, to verify that this phenomenon is also observable in dense datasets, we apply TAG-CF to Movielens-1M. It is a dense dataset where each user has an average number of 165 interactions, as opposed to 50 interactions for other datasets that we utilize in our manuscript. The results are shown in the table below and we can notice that our observation regarding the performance improvement brought by message passing still holds. While it might not be sensible to apply TAG-CF to graph-based CF methods that leverage message passing already, in order to fully verify our findings, we also apply TAG-CF to a graph-based method (i.e., UltraGCN (Mao et al., 2021) ) that utilizes graph structures as supervision signals only. The performance as well as efficiency of UltraGCN and UltraGCN+TAG-CF are shown in Table 10 .From Table Re1 , we observe that TAG-CF can further improve UltraGCN, even when the former utilizes graph structures during the model training. Specifically, TAG-CF can improve the performance of UltraGCN by 6%, which is significant. This observation indicates that the findings we propose in this manuscript can be applied to other algorithms. While TAG-CF can improve the performance, as shown in Table Re2 , we can also notice that MF-DirectAU + TAG-CF in total runs a lot faster than UltraGCN. This is because UltraGCN still requires repetitive querying of the graph structures while calculating its objective functions. For the simplicity of the notation, we showcase our observation with only one layer. However, since LightGCN is fully linear, the phenomenon we show also applies to variants with arbitrary layers.http://jmcauley.ucsd.edu/data/amazon.https://myanimelist.net.https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset.https://github.com/xiangwang1223/neural_graph_collaborative_filtering.https://www.kaggle.com/datasets/CooperUnion/anime-recommendations-databasehttps://recbole.io.https://github.com/THUwangcy/DirectAU.The number of buckets can be set to arbitrary numbers for finer adjustments. In this study, we pickas a proof of concept and it already effectively helps us verify the performance of TAG-CF + as well as the legitimacy of our findings. 


Decision: This paper investigates the role of message passing (MP) in collaborative filtering (CF) and proposes a novel test-time aggregation method called TAG-CF. 

Key Strengths:

- Novel insights: The paper provides new perspectives on how message passing functions in graph-based CF, challenging existing assumptions.
- Efficiency: TAG-CF demonstrates competitive performance with state-of-the-art graph-based CF methods while significantly reducing computational costs.
- Versatility: The method can be easily integrated into various existing CF models as a plug-and-play module.
- Theoretical foundation: The authors provide theoretical derivations to support the validity of their approach.
- Extensive experiments: The paper includes comprehensive evaluations across multiple datasets and baselines.

Key Weaknesses:

- Overlooked related work: Initially, the paper failed to compare against or discuss some relevant efficient CF methods that do not require training (e.g., GF-CF, BSPM, SVD-AE). However, the authors addressed this in their rebuttal by adding experiments and discussions on these methods.
- Limited analysis of conditions: More detailed analysis is needed to determine under what conditions the performance of TAG-CF is - comparable to training-time message passing.
- Theoretical explanation: The theoretical justification for why low-degree nodes benefit more from message passing could be strengthened.
- Heuristic parameter selection: The method for selecting the degree cutoff in TAG-CF+ seems somewhat ad-hoc and could benefit from a more systematic approach.
- Evaluation metrics: Initially, the paper used NDCG@20 and Recall@20, while NDCG@10 and Recall@10 might better reflect real-world recommendation scenarios. The authors addressed this in their rebuttal by providing additional results for these metrics.

The paper presents a novel and efficient approach to collaborative filtering that challenges existing assumptions about message passing in graph-based recommender systems. The proposed TAG-CF method shows promise in terms of both performance and computational efficiency. While there were some initial oversights in related work comparisons and evaluation metrics, the authors have addressed most of these concerns in their rebuttal and reviewers increased their ratings. The paper is likely to generate interest and discussion in the recommender systems community.

While the paper is very close to be rejected, it improved significantly during the review process and the question of integration of other signal in collaborative filtering is both important and difficult from an industrial perspective. I feel the idea of MP is worth the be presented and discussed at the conference.  Also one of the reviewers didn't engaged into discussion yet other reviewers with similar concerns upgraded their scores after the rebuttal.

Please include Tabla 1 better in the final version.

Reviews: review: paper_summary: paper_summary: This paper investigates the role of message passing (MP) in collaborative filtering (CF). Unlike most GNN-based CF research, which assumes that performance gains arise from improved representation learning through GNNs, this work questions that assumption. Through empirical experiments and theoretical analyses, the paper finds that the key contributions of MP in CF are: 1) forward message passing rather than back-propagation, and 2) benefits for low-degree users instead of high-degree users.

Based on these observations, this work propose a simple and efficient plug-in method called TAG-CF. TAG-CF can be easily integrated into any well-trained CF model by performing a single layer of message passing during the inference stage. To further reduce computational workload, TAG-CF+ conducts MP only on low-degree users. Experiments on four datasets verify the effectiveness of TAG-CF in improving CF models with almost no additional computational overhead.
strength_weakness: strength_weakness: 1. The discussion following Theorem 1 "both BPR and DirectAU optimize the objective of message passing" is not rigorous enough. Here are some aspects regarding to the discussion:
   * The assumption that $\|\mathbf{u}_i\|^2=\|\mathbf{i}_j\|^2=1$ does not hold in LightGCN, thus the discussion on the theorem is only empirical but not strictly hold for LightGCN. In other words, the BPR loss for real-world LightGCN may not be upper-bounded by the objective of message passing.
   * Even if the claim "both BPR and DirectAU optimize the objective of message passing" holds, there should be an additional experiment to show that directly train a CF model on the objective of message passing achieves comparable performance of LightGCN-BPR. 

2. Important baselines such as GFCF [1] and SVD-GCN [2] are missing. These works also dedicated to speed up GNN-based CF. On Gowalla dataset, both GFCF and SVD-GCN cost only around 30s for training, which is more efficient than the proposed TAG-CF. Moreover, the reported Recall@20 on Gowalla are 0.1849 and 0.1905 respectively, also shows comparable or better performance. The authors need to add these baseline in the experiment and discuss what is the advantage of TAG-CF compared to these baselines.

3. According to Figure1, it is not sufficient to draw the conclusion that "Message Passing in CF Helps Low-degree Users More Compared with High-degrees". In Yelp dataset, the improvement of degree=16 users is only about 15%, which is less than the value of most higher-degree users (degree from 17-34). And what is the improvement for the users whose degree is less than 16? Still, this empirical experiment on only two dataset is not enough to support this conclusion.

[1] Yifei Shen, et al., “How Powerful is Graph Convolution for Recommendation?” CIKM 2021

[2] Shaowen Peng, et al., “SVD-GCN: A Simplified Graph Convolution Paradigm for Recommendation”. CIKM 2022
questions: questions: Besides the weakness part, I also have some questions:

1. What is the setting of w/o both in Table 1. If it is a MF model using BPR, how can it achieve such good performance (Recall@20=18.42). In Table 2, MF model also shows superisingly good performance on most dataset, almost comparable to LightGCN, how does the MF set and tuned in the experiment?
2. In Table 2, low degree users' performance even better than overall performance, which seems to be inconsistent with Figure 1, are there any explanations?
3. How does m and n be tuned in the experiment? Are there any sensitivity experiments on these important hyper-parameters?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper investigates on an interesting topic: how message passing is playing a role in graph-based recommender systems. Upon experiments on LightGCN model, authors posit the advantages of message passing lie in the forward pass to aggregate neighborhood information while at the same time, backward propagation has little effect on the results. Based on this assumption, authors propose test-time aggregation for collaborative filtering where the representation of users/items is aggregated at the testing time for more efficient training.
strength_weakness: strength_weakness: 1. Lack of pre-training description
2. Lack of hyper-parameter analysis
questions: questions: 1. What is the detailed pre-training method for the method? I am assuming the pre-training strategy is impactful for the results.
2. An experiment on different pre-training methods is needed.
3. Authors design m,n hyper-parameters to adjust the neighborhood impact. Are there any findings we can obtain from the two hyper-parameters? I would like to see hyper-parameter testing on them.
limitations: limitations: See weakness
paper_summary: paper_summary: This paper investigates the role of message passing in collaborative filtering, providing empirical analysis of its impact. Based on their findings that message passing primarily benefits through additional neighbor representations during forward passes and helps low-degree nodes more than high-degree nodes, the authors propose TAG-CF, a test-time aggregation framework for collaborative filtering. TAG-CF demonstrates competitive performance with state-of-the-art graph-based collaborative filtering methods while significantly reducing computational overhead.
strength_weakness: strength_weakness: W1. Crucially, this paper overlooks related work, which do not require training.

W2.  The paper claims that TAG-CF can achieve performance similar to training-time message passing with just one aggregation at test time. More detailed analysis is needed to determine if this holds true in all situations and under what conditions these results occur.

W3. While it's claimed that performance improvement is greater for low-degree nodes, the theoretical explanation for this is somewhat lacking. A deeper analysis of why this phenomenon occurs is needed. In particular, it seems highly dependent on the degree cutoff.
questions: questions: Q1. The authors have overlooked related work, such as GF-CF[1], BSPM[2], and SVD-AE[3], which are models without trainable parameters and training stages. The authors need to include these in their experimental results for comparison with test-time aggregation research or describe their commonalities and differences in the related work section. These should be mentioned as essential research before discussing test-time aggregation.

Q2. When the authors refer to low-degree nodes, is this based on the entire dataset or just the test or training set? If it's based on the entire dataset, couldn't the performance improvement be biased depending on how the test set is split?

Q3. The method for selecting the degree cutoff in TAG-CF+ seems somewhat heuristic. Is there a more systematic method for choosing this parameter?

Q4. What problems might arise when applying this to large-scale recommendation systems in practice?

Q5. Have you analyzed how TAG-CF's performance varies with the structural characteristics of the graph (e.g., node centrality, average path length, etc.)?

Q6. How does the dimension of user/item representations affect the performance of TAG-CF?

> [1] Shen, Yifei, et al. "How powerful is graph convolution for recommendation?." Proceedings of the 30th ACM international conference on information & knowledge management. 2021.
>
> [2] Choi, Jeongwhan, et al. "Blurring-sharpening process models for collaborative filtering." Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.
>
> [3] Hong, Seoyoung, et al. "SVD-AE: Simple Autoencoders for Collaborative Filtering." arXiv preprint arXiv:2405.04746 (2024).
limitations: limitations: N/A
paper_summary: paper_summary: This paper rethinks the application of message passing mechanisms in collaborative filtering methods and makes two key findings: 1) Message passing (MP) improves collaborative filtering primarily through the forward pass rather than the backward propagation process, and 2) MP is more effective for cold-start users compared to warm users. Based on these findings, this paper proposes a novel test-time aggregation method tailored for collaborative filtering, supported by theoretical derivations to demonstrate its validity. Finally, extensive experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Equation 4 seems unable to separate the forward pass neighbor representation from the backward pass gradient update in $s_{ij}$. Therefore, it is theoretically challenging to validate this finding, even though Table 1 empirically demonstrates that the forward pass primarily drives the performance improvement.

2. Typically, evaluation metrics for recommendation models use NDCG@10 and Recall@10, as users tend to focus on items ranked higher in the list while ignoring those ranked lower. Therefore, NDCG@10 and Recall@10 better reflect the true effectiveness of recommendation models.

3. It is necessary to explain some abbreviations, such as 'OOM' and '-'.

4. This paper has many grammatical problems, so it is suggested to improve it. For example, "TAG-CF is extremely versatile can be used as a plug-and-play module to enhance...".
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.

Comments: title: Thanks for your reply.
Thanks for your response.
Thanks for your reply.
keep score unchanged
comment: Your valuable insights and constructive feedback on our paper are deeply appreciated. We are grateful for acknowledging the dedication we've invested in our research.
We are delighted that our rebuttal satisfactorily leverages your concerns. We appreciate your kind responses and acknowledgement of our work.
We greatly appreciate your valuable insights and constructive feedback on our paper. It's truly encouraging to see that our response effectively addresses the concerns you raised. Thank you for recognizing the efforts we've put into our work.
Thank you for addressing my concerns. I raise my score to 5.
I will keep my positive score unchanged. Thank you for author's rebuttal
Thank you for your detailed response; it has resolved my concerns. I will maintain my positive rating.