ContextGNN: Beyond Two-Tower Recommendation Systems

Authors: Yiwen Yuan, Zecheng Zhang, Xinwei He, Akihiro Nitta, Weihua Hu, Manan Shah, Blaž Stojanovič, Shenyang Huang, Jan Eric Lenssen, Jure Leskovec, Matthias Fey
Primary area: learning on graphs and other geometries & topologies
Keywords: graph neural networks, recommendation, relational deep learning

Abstract:
Recommendation systems predominantly utilize two-tower architectures, which evaluate user-item rankings through the inner product of their respective embeddings. However, one key limitation of two-tower models is that they learn a pair-agnostic representation of users and items. In contrast, pair-wise representations either scale poorly due to their quadratic complexity or are too restrictive on the candidate pairs to rank. To address these issues, we introduce Context-based Graph Neural Networks (ContextGNNs), a novel deep learning architecture for link prediction in recommendation systems. The method employs a pair-wise representation technique for familiar items situated within a user's local subgraph, while leveraging two-tower representations to facilitate the recommendation of exploratory items. A final network then predicts how to fuse both pair-wise and two-tower recommendations into a single ranking of items. We demonstrate that ContextGNN is able to adapt to different data characteristics and outperforms existing methods, both traditional and GNN-based, on a diverse set of practical recommendation tasks, improving performance by 20\% on average.

Decision: accept (poster)
Reviewer scores: 8, 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730744949779

thread_type:
official_review

<thread_object>
creator: reviewer_7mvo

<content>
summary:
This paper presents ContextGNN, a refined approach that classifies items into user-specific selections within a user-centric local graph, alongside other items. Subsequently, the GNN learns the pertinent user and item embeddings, thereby enabling the derivation of both pairwise and two-tower representations for enhanced recommendations.

strengths:
1. The paper is well-written and easy to understand.
2. The use of both pairwise and two-tower representations for recommendations is quite interesting.
3. The experiments conducted in this paper seem to demonstrate the model's effectiveness.

weaknesses:
1. The author should consider open-sourcing the code for the proposed method to enhance reproducibility.
2. In the comparison of methods, it would be beneficial to include more recent and advanced GNN-based recommender models beyond NGCF.
3. Given that the author mentions both multi-behavior and temporal graphs, it would be advantageous to illustrate how the GNN backbone is designed to accommodate such graphs.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730756869268

thread_type:
official_review

<thread_object>
creator: reviewer_u3kg

<content>
summary:
* The method uses a pair-wise representation for familiar items in a user's local subgraph and two-tower representations for recommending exploratory items. A final network combines both pair-wise and two-tower recommendations to create a unified ranking of items.

* The study utilizes CONTEXTGNNs in relational deep learning for recommendations within relational databases, which consist of various tables with rich multi-modal features and complex behavioral interactions. This environment provides an ideal setting for testing the model's effectiveness. Additionally, the research explores the challenges of modeling intricate human behavior patterns.

strengths:
* This article has a clear structure and is easy to follow.

*Tested different tasks, which is rare in previous work.

weaknesses:
* It appears that this work has not released the code or dataset, making reproducibility uncertain.

* This work lacks technical innovation and seems quite basic. Such a simple design should emphasize time complexity, runtime efficiency, and scalability, but the paper does not seem to address these aspects.

* This work does not compare with other recent baselines or commonly used models in the industry. Additionally, it does not provide specific data from the dataset. Whether in industry or research, this work does not seem to fit well. Overall, it appears too basic and simple, with relatively little effort involved.

questions:
* The paper mentions the complexity issues of pair-wise methods. Could you provide a time complexity analysis and runtime efficiency experiments of this work compared to others?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730767935330

thread_type:
official_review

<thread_object>
creator: reviewer_pueu

<content>
summary:
This paper proposes a recommendation framework that integrates pair-wise representations with two-tower representations. Specifically, it leverages pair-wise representations to capture fine-grained patterns of past user-item interactions, while utilizing two-tower representations for "distant" items. The authors have validated the effectiveness of their approach on the RELBENCH benchmark.

strengths:
1. The motivation behind the proposed method is clear and has significant practical implications for large-scale recommendation systems in the real world.
2. The authors have validated the effectiveness of the proposed method on the benchmark datasets.

weaknesses:
1. The authors emphasize in the abstract and introduction that their method is a relatively general and effective strategy in recommendation scenarios. However, the experimental section only tests the method on datasets that include rich multi-behavioral and temporal interactions. This discrepancy between the claims and the experimental validation raises concerns. Therefore, the authors should either revise their overall statements or consider incorporating a wider variety of datasets for comparative analysis.
2. The experimental section of the paper is quite limited, as it currently only reports the MAP metric and lacks detailed parameter sensitivity analysis. Moreover, the paper does not address how the proposed method's performance may be affected when user behavior information in the dataset is insufficient in type or quantity. Specifically, would we expect a significant decline in performance compared to the baseline algorithms under such conditions? Additionally, I recommend that the authors incorporate a broader range of recent comparison methods to strengthen their empirical evaluation.
3. The paper does not provide the original code, which raises concerns regarding the reproducibility of the results. Additionally, there are some errors in the equations presented.

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730789538654

thread_type:
official_review

<thread_object>
creator: reviewer_uwx2

<content>
summary:
This paper studies the two-tower architecture for recommender systems. They introduce a context-based GNNs-based framework for link prediction in recommendation by using a pair-wise representation technique for similar items in the user's local subgraph. The proposed method achieves promising performance in their experiments.

strengths:
1. The research issue is very important in the industry. 

2. Their experimental results demonstrate the effectiveness of the proposed method. 

3. The paper is easy to follow.

weaknesses:
However, there are some concerns in the paper:

1. The novelty of the proposed method is unclear.
This paper leverages local interaction graph for representation learning. The idea is very mature and has been well-studied. 
 Compared with exciting methods, what are the novelty of the proposed method? Additionally, the related work in the paper can be enhanced. 
This paper can be improved by discussing more advanced two-tower architectures. Furthermore, the related work cannot demonstrate the novelty of the proposed method.


2. The technical contributions of this work are also unclear. I was wondering what are the main challenges they are addressing. 
The proposed idea looks good. However, the proposed method doesn't have any theoretical analysis.
The model design didn't motivate well. 
This paper considers the temporal factor in modeling recommendation. However, what are the technical contributions for taking advantage of it?


3. The experimental section can be enhanced before publication. First, they can compare with some advanced recommendation baselines. There are many advanced recommendation methods for modeling user and item representations. 
Second, the datasets here are unclear to me. Whether the proposed method can be used for large-scale datasets?

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730822848831

thread_type:
official_review

<thread_object>
creator: reviewer_gubd

<content>
summary:
The paper proposes ContextGNNs, a new architecture for graph-based recommendation. Unlike previous similar approaches, such as pair-wise recommendation solutions, the authors propose a pair-wise representation only for exploring the items in the user's local subgraph, while the architecture employs the usual two-tower method for the item side. Finally, a network is designed and trained to fuse the two representations and provide the final recommendations. 

On the one hand, the ContextGNNs framework first generates a pair-wise representation for users and items, where a $k$-hop subgraph for each user is sampled, and a GNN is leveraged to obtain embedding representations and the overall item ranking. On the other hand, a two-tower representation is exploited on the item side, as it can be shown that: 1) GNNs do not learn useful information on the item side as the item interaction matrix tends to be very dense; 2) shallow item embeddings are effective; 3) GNN item representations can scale poorly with large numbers of items in the catalog. The so-learned item embeddings are injected into the GNN forward pass presented above, to allow the user representations to align with the item ones. Finally, an MLP is trained to fuse the two designed contributions. 

Results on different tasks and datasets from the RelBench platform demonstrate the efficacy of the ContextGNNs proposed baseline, also when compared to different settings of the same model.

strengths:
\+ The proposed approach is well-placed within the existing literature on (graph)-based recommendation, and the motivations behind ContextGNNs are quite solid and sound

\+ Overall, the paper is well-written and easy to follow

\+ I appreciated the locality score study, which conceptually and empirically supports the rationale behind the approach

\+ I also appreciated how the authors discussed possible extensions of the proposed approach in different and interesting new directions

\+ Even considering the page limitations, I believe the experimental setting is adequately extensive to empirically support the rationale behind the paper

weaknesses:
\- To my understanding, it seems that the list of baselines for the experiments is not completely diversified and updated to the latest advances in graph-based recommendation. I would have appreciated seeing more recent and popular approaches being tested against ContextGNNs, such as LightGCN (cited in the paper), SGL [i], UltraGCN [ii], SimGCL [iii], and LightGCL [iv].

**References**

[i] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, Xing Xie: Self-supervised Graph Learning for Recommendation. SIGIR 2021: 726-735

[ii] Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, Xiuqiang He: UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation. CIKM 2021: 1253-1262 

[iii] Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, Quoc Viet Hung Nguyen: Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation. SIGIR 2022: 1294-1303

[iv] Xuheng Cai, Chao Huang, Lianghao Xia, Xubin Ren: LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation. ICLR 2023

questions:
I have a question that does not refer to the one weakness raised. As observed by the authors in the locality score study, it is interesting to see that many validation/test links are not in the users' nearest neighborhood. Moreover, we know from the literature, that graph-based recommender systems may be affected by oversmoothing. On such a basis, my question is: how the graph-based model can learn long-range (and, evidently, ground-truth) products connected to users if the model cannot explore on more than ~3 hops (on average)? Do you have any intuitions for that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732265579128

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Answer to all Reviewers [1/3]

comment:
We thank all reviewers for their constructive feedback and helpful suggestions. We are glad that they found our work interesting (**7mVo**), well motivated (**GUBd**, **pueu**), important (**uwX2**), well-supported by experiments (**GUBd**, **pueu**, **7mVo**), easy-to-follow (**GUBd**, **7mVo**, **uwX2**, **U3kG**), and well-placed in literature (**GUBd**). In this response, we address joint comments of the reviewers while also providing individual responses to each reviewer separately. We hope that reviewers’ concerns are addressed well and would kindly ask for a score increase if that is the case or for further suggestions on how to further improve the work.

**Code Submission:** Based on reviewers’ requests, we have uploaded our code base to ensure reproducibility of the presented results: https://anonymous.4open.science/r/ContextGNN-DBCA/README.md 

**Contribution and Novelty:** Reviewers **uwX2** and **U3kG** had concerns about the novelty of our work. The novelty of our model is the fusion of both pair-wise and two-tower-based recommendations into a single architecture. Since most GNN-based RecSys models are two-tower based (e.g., NGCF [1], LightGCN [2], UltraGCN [3], SimGCL [4], SGL [5], LightGCL [6], etc), they inherit the shared fundamental weakness that the item representations are not capturing the uniqueness of a user’s view on the items since users and items are ranked based on two completely separate, pair-agnostic representations. On the other hand, pair-wise representations (which captures the user-specific view on items) can not be computed for all possible user-item pairs due to its quadratic complexity. Our contribution is a novel end-to-end mixture of these two paradigms, which is efficient, scalable, and improves performance out-of-the-box. This fundamentally changes how we think about GNN-based recommendation systems since ContextGNN integrates signals into the model that are unable to be captured by a two-tower model (e.g., repeated purchases, items that have been previously looked at, friends who just bought this particular item, co-purchases, etc), while preserving the strengths of two-tower models for exploratory purchases (e.g., popularity bias, content-based filtering, etc). Our efficiency study below also showcases that ContextGNN not only improves performance, it is also significantly faster than two-tower based approaches.

The importance of our contribution is supported further by our study on real-world datasets (Section 3), which reveals that user behavior is very diverse: Some users prefer to interact with “familiar” items local to the user, while others like to constantly explore new items distant to the user. This shows the necessity of introducing a hybrid architecture.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732719133891

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised manuscript

comment:
Dear reviewers,

we also revised our manuscript and have uploaded a new version. All changes to the initial version are marked in blue.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a novel recommendation framework that combines pair-wise representations with two-tower representations to enhance recommendation performance. The framework employs pair-wise representations to capture fine-grained patterns from past user-item interactions, while leveraging two-tower representations to handle "distant" items. The effectiveness of this approach is demonstrated through extensive validation on the RELBENCH benchmark.

Positive points:
+ The paper is well written
+ The idea proposed in the paper is sound and interesting
+ The experiments are very extensive, which can well support the authors' claims

Negative points:
- The baselines are not sufficiently up-to-date.
- The original code is not provided in the paper