SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. Two high-performing encoders - ID-based (MF [50]) and graph-based (LightGCN [51]), are selected as CF backbone models. We thoroughly compare AdvInfoNCE with two categories of the latest CL-based CF methods: augmentation-based baselines (SGL [13], NCL [52], XSimGCL [27]) and loss-based baselines (CCL [53], BC Loss [54], Adap-  $\tau$  [55]). See detailed introduction and comparison of baselines in Appendix A.
Datasets. To verify the generalization ability of AdvInfoNCE, we conduct extensive experiments on three standard unbiased datasets (KuaiRec [56], Yahoo!R3 [57], Coat [58]) and one synthetic dataset (Tencent [10]). We employ three commonly used metrics (Hit Ratio (HR@  $K$ ), Recall@  $K$ , Normalized Discounted Cumulative Gain (NDCG@  $K$ )) for evaluation, with  $K$  set by default at 20. Please refer to Appendix C and D for more experimental results over additional backbones, additional baselines, and an intuitive understanding of AdvInfoNCE.
4.1 Overall Performance Comparison (RQ1)
4.1.1 Evaluations on Unbiased Datasets
Motivation. Evaluating the efficacy and generalization ability of CF models based on partially observed interactions collected from existing recommender system is rather challenging [59]. This is
Table 1: The performance comparison on unbiased datasets over the LightGCN backbone. The improvement achieved by AdvInfoNCE is significant  $(p$ -value  $<< 0.05)$ .
KuaiRec	Yahoo!R3	Coat
Recall	NDCG	Recall	NDCG	Recall	NDCG
BPR (Rendle et al., 2012)	0.1652	0.3905	0.1487	0.0697	0.2737	0.1707
InfoNCE (van den Oord et al., 2018)	0.1800	0.4529	0.1475	0.0698	0.2689	0.1882
SGL (Wu et al., 2021)	0.1829+1.61%	0.4583+1.19%	0.1474-0.07%	0.0692-0.86%	0.2737+1.79%	0.1716-8.82%
NCL (Lin et al., 2022)	0.1764-2.00%	0.4478-1.13%	0.1407-4.61%	0.0669-4.15%	0.2778+3.31%	0.1739-7.60%
XSimGCL (Yu et al., 2022)	0.1907+5.94%	0.4531+0.04%	0.1503+1.90%	0.0708+1.43%	0.2729+1.49%	0.1655-12.06%
CCL (Mao et al., 2021)	0.1776-1.33%	0.4497-0.71%	0.1453-1.49%	0.0676-3.15%	0.2732+1.60%	0.1941+3.13%
BC Loss (Zhang et al., 2022)	0.1799-0.06%	0.4417-2.47%	0.1498+1.56%	0.0703+0.72%	0.2719+1.12%	0.1921+2.07%
Adap-τ (Chen et al., 2023)	0.1717-4.61%	0.4323-4.55%	0.1516+2.78%	0.0704+0.86%	0.2700+0.41%	0.1957+3.99%
AdvInfoNCE	0.1979*+9.94%	0.4697*+3.71%	0.1527*+3.53%	0.0718*+2.87%	0.2846*+5.84%	0.2026*+7.65%
primarily due to the exposure bias of the recommender system confounding the observed data. To address this issue, some researchers propose using unbiased evaluations [57, 58]. In these datasets, items in the test set are randomly exposed to users, while partially observed interactions are still used for training. The distribution shift between training and testing enables a fair evaluation of the generalization ability of CF methods in out-of-distribution tasks.
Results. Tables 1 and 4 present a comparative analysis of the performance on standard unbiased datasets over the LightGCN and MF backbones. The best-performing methods per test are bold and starred, while the second-best methods are underlined. The red and blue percentages respectively refer to the increase and decrease of CF methods relative to InfoNCE in each metric. We observe that:
- CL-based CF methods have driven impressive performance breakthroughs compared with BPR in most evaluation metrics. This aligns with findings from prior research [27, 54] that contrastive loss significantly boosts the generalization ability of CF methods by providing informative contrastive signals. In contrast, BPR loss, which focuses on pairwise relative ranking, often suffers from ineffective negative sampling [30] and positive-unlabeled issues [21].
- Contrastive loss, rather than graph augmentation, plays a more significant role in enhancing the generalization ability of CF models. We categorize CL-based CF baselines into two research lines: one adopts user-item bipartite graph augmentation along with augmented views as positive signals (referred to as augmentation-based), and another focuses solely on modifying contrastive loss using interacted items as positive instances (referred to as loss-based). Surprisingly, our results reveal no substantial difference between the performance of augmentation- and loss-based methods. This finding provides further evidence in support of the claim made in prior studies [27, 26] that, specifically in CF tasks, the contrastive loss itself is the predominant factor contributing to the enhancement of a CF model's generalization ability.
- AdvInfoNCE consistently outperforms the state-of-the-art CL-based CF baselines in terms of all metrics on all unbiased datasets. AdvInfoNCE shows an improvement ranging from  $3.53\%$  to  $9.94\%$  on Recall@20 compared to InfoNCE. Notably, AdvInfoNCE gains the most on the fully-exposed dataset, KuaiRec, which is considered as an ideal offline A/B testing. In contrast, most of the CL-based CF methods underperform relative to InfoNCE and behave in an unstable manner when the testing distribution shifts. This clearly indicates that AdvInfoNCE greatly enhances the robustness of the CF model in real-world scenarios. We attribute this improvement to the fine-grained ranking criterion of AdvInfoNCE that automatically differentiates hard and false negative samples. Additionally, as shown in Table 9, AdvInfoNCE only adds negligible time complexity compared to InfoNCE.
4.1.2 Evaluations on Various Out-of-distribution Settings
Motivation. In real-world scenarios, recommender systems may confront diverse, unpredictable, and unknown distribution shifts. We believe that a good recommender with powerful generalization ability should be able to handle various degrees of distribution shift. Following previous work [10], we partition the Tencent dataset into three test sets with different out-of-distribution degrees. Here, a higher  $\gamma$  value signifies a greater divergence in distribution shift (for more details on the experimental settings, see Appendix C.1). We evaluate all CF methods with identical model parameters across all test sets, without knowing any prior knowledge of the test distribution beforehand.
Table 2: The performance comparison on the Tencent dataset over the LightGCN backbone. The improvement achieved by AdvInfoNCE is significant  $(p$ -value  $<< 0.05)$ .
HR	γ = 200 Recall	NDCG	HR	γ = 10 Recall	NDCG	HR	γ = 2 Recall	NDCG	Validation NDCG
BPR (Rendle et al., 2012)	0.1141	0.0416	0.0233	0.0720	0.0262	0.0146	0.0501	0.0186	0.0109	0.0673
InfoNCE (van den Oord et al., 2018)	0.1486	0.0540	0.0320	0.0924	0.0332	0.0195	0.0646	0.0242	0.0145	0.0854
SGL (Wu et al., 2021)	0.1227	0.0455	0.0249	0.0756	0.0281	0.0157	0.0517	0.0198	0.0113	0.0729
NCL (Lin et al., 2022)	0.1132	0.0413	0.0226	0.0734	0.0274	0.0145	0.0511	0.0189	0.0123	0.0669
XSimGCL (Yu et al., 2022)	0.1392	0.0501	0.0273	0.0830	0.0297	0.0160	0.0539	0.0203	0.0110	0.0873
CCL (Mao et al., 2021)	0.1372	0.0516	0.0318	0.0925	0.0350	0.0221	0.0683	0.0266	0.0170	0.0782
BC Loss (Zhang et al., 2022)	0.1513	0.0562	0.0341	0.0984	0.0366	0.0216	0.0682	0.0260	0.0164	0.0817
Adap-τ (Chen et al., 2023)	0.1488	0.0537	0.0317	0.0940	0.0338	0.0200	0.0642	0.0239	0.0143	0.0852
AdvInfoNCE	0.1600*	0.0594*	0.0356*	0.1087*	0.0403*	0.0243*	0.0774*	0.0295*	0.0180*	0.0879
Imp.% over the strongest baselines	5.74%	5.75%	4.53%	10.50%	10.11%	9.95%	13.32%	10.90%	5.88%	-
Imp.% over InfoNCE	7.67%	10.00%	11.25%	17.64%	21.39%	24.62%	19.81%	21.90%	24.14%	-
(a) Distribution of hardness
Figure 2: Study of hardness. (2a) Illustration of hardness i.e., the probability of negative sampling  $(p(j|(u,i)))$  learned by AdvInfoNCE w.r.t. item popularity on Tencent. The dashed line represents the uniform distribution. (2b) Performance comparisons with varying hardness learning strategies on Tencent ( $\gamma = 10$ ). (2c) The trajectories of align loss and uniform loss during training progress. Lower values indicate better performance. Arrows denote the losses' changing directions.
(b) Varying hardness strategies
(c) Alignment & uniformity analysis
Results. As illustrated in Tables 2 and 5, AdvInfoNCE yields a consistent boost compared to all SOTA baselines across all levels of out-of-distribution test sets. In particular, AdvInfoNCE substantially improves Recall@20 by  $10.00\%$  to  $21.90\%$  compared to InfoNCE and by  $5.75\%$  to  $10.90\%$  compared to the second-best baseline. It's worth noting that AdvInfoNCE gains greater improvement on test sets with higher distribution shifts, while all CL-based CF baselines suffer significant performance drops due to these severe distribution shifts. We attribute this drop to the coarse relative ranking criterion which injects erroneously recognized negative instances into contrastive signals. In contrast, benefiting from the effective adversarial learning of the fine-grained ranking criterion, AdvInfoNCE can achieve a  $24.14\%$  improvement over InfoNCE w.r.t. NDCG@20 when  $\gamma = 2$ , fully stimulating the potential of contrastive loss. These results validate the strong generalization ability of AdvInfoNCE as demonstrated in Theorem 3.1, proving that optimizing AdvInfoNCE is equivalent to solving DRO problems constrained by KL divergence over high-quality negative sampling.
4.2 Study on AdvInfoNCE (RQ2)
4.2.1 Effect of Hardness
Motivation. To evaluate the effectiveness of the fine-grained hardness-aware ranking criterion learned by AdvInfoNCE, and further investigate whether it truly enhances the generalization ability of CF models, we conduct a comprehensive analysis on the Tencent dataset. This analysis contains three-fold: it studies the distribution of hardness (i.e., examining the negative sampling strategy of AdvInfoNCE); the performance trend over varying hardness learning strategies; and conducting an analysis of the alignment and uniformity properties of representations over varying hardness learning strategies. Additional experiments on the KuaiRec dataset, exhibiting similar trends and findings, can be found in Appendix D.2.
Distribution of hardness, aka. negative sampling strategy. As elucidated in Theorem 3.1, learning the hardness  $\delta_j^{(u,i)}$  in an adversarial manner essentially conducts negative sampling. Inspired by item popularity-based negative sampling strategies [60-62], we probe into the relationship between adversarial negative sampling of AdvInfoNCE and the popularity of negative items. Figure 2a
(a) End at early-stage
Figure 3: The performance on the in-distribution validation set and out-of-distribution test sets of Tencent. The total number of epochs of adversarial training is gradually increasing from (a) to (c), and the green star denotes the timepoint when we terminate adversarial training.
(b) End at mid-stage
(c) End at late-stage
depicts a skewed negative sampling distribution that focuses more on popular negative items and de-emphasizes the hardness of unpopular negative items, which are likely to be false negatives. These results highlight two findings. Firstly, in line with [61], only a minor portion of negative instances are potentially important for model learning, while a vast number of negatives are false negatives. Secondly, more frequent items constitute better hard negatives, which is consistent with the findings in previous work [60]. This popularity-correlated negative sampling validates that AdvInfoNCE automatically mitigates the influence of popularity bias during training, alleviates the overlook of false negatives in CF, and utilizes high-quality hard negative mining.
Performance over varying hardness learning strategies. Figure 2b clearly demonstrates that distilling the fine-grained hardness of negative items (AdvInfoNCE) is excelling over the uniformly random hardness (InfoNCE+Rand), equal hardness (InfoNCE), and reversed fine-grained hardness (AdvInfoNCE+Reverse) throughout the training process. This validates AdvInfoNCE's ability to simultaneously filter and up-weight more informative hard negatives while identifying and downweighting the potential false negatives. These desired properties further enable AdvInfoNCE to extract high-quality negative signals from a large amount of unobserved data.
Alignment and uniformity analysis. It is widely accepted that both alignment and uniformity are necessary for a good representation [5]. We, therefore, study the behavior of AdvInfoNCE through the lens of align loss and uniform loss, as shown in Figure 2c. AdvInfoNCE improves uniformity while maintaining a similar level of alignment as InfoNCE, whereas reversed fine-grained hardness significantly reduces representation uniformity. These findings underscore the importance of fine-grained hardness, suggesting that AdvInfoNCE learns more generalized representations.
4.2.2 Effect of the Adversarial Training Epochs
The critical hyperparameter, the number of adversarial training epochs, controls the degree of deviation from uniform negative sampling. As illustrated in Figure 3a through 3b, improved out-of-distribution performance correlates with an increased adversarial learning period. On the other hand, Figures 3b through 3c show that overly prolonged adversarial training can also degrade performance on in-distribution validation sets. Therefore, it is crucial to devise a proper adversarial training period tailored to specific datasets, which aligns with the limitation of AdvInfoNCE. Consistent observations can be found on KuaiRec in Appendix D.3.
Despite the empirical successes, the profound understanding of contrastive loss in collaborative filtering (CF) remains limited. In this work, we devised a principled AdvInfoNCE loss specially tailored for CF methods, utilizing a fine-grained hardness-aware ranking criterion to enhance the recommender's generalization ability. Grounded by theoretical proof, AdvInfoNCE empowers top- $K$  recommendations via informative negative sampling in an adversarial manner. We believe that our AdvInfoNCE, as a variant of InfoNCE, provides a valuable baseline for future contrastive learning-based CF studies and is an important stepping stone toward generalized recommenders.
This research is supported by the NExT Research Center, the National Natural Science Foundation of China (9227010114), the University Synergy Innovation Program of Anhui Province (GXXT-2022-040), and the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD110001).
[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[2] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[3] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
[4] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[5] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[6] Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Self-supervised hypergraph convolutional networks for session-based recommendation. In AAAI, 2021.
[7] Hongrui Xuan, Yi Liu, Bohan Li, and Hongzhi Yin. Knowledge enhancement for contrastive multi-behavior recommendation. In WSDM, 2023.
[8] Junliang Yu, Hongzhi Yin, Jundong Li, Qinyong Wang, Nguyen Quoc Viet Hung, and Xi-angliang Zhang. Self-supervised multi-channel hypergraph convolutional network for social recommendation. In WWW, 2021.
[9] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S^3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. CoRR, abs/2008.07873, 2020.
[10] An Zhang, Jingnan Zheng, Xiang Wang, Yancheng Yuan, and Tat seng Chua. Invariant collaborative filtering to popularity distribution shift. In WWW, 2023.
[11] An Zhang, Wenchang Ma, Jingnan Zheng, Xiang Wang, and Tat-Seng Chua. Robust collaborative filtering to popularity distribution shift. ACM Transactions on Information Systems, 2023.
[12] Jin Chen, Defu Lian, Binbin Jin, Xu Huang, Kai Zheng, and Enhong Chen. Fast variational autoencoder with inverted multi-index for collaborative filtering. In WWW, 2022.
[13] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In SIGIR, 2021.
[14] Chang Zhou, Jianxin Ma, Jianwei Zhang, Jingren Zhou, and Hongxia Yang. Contrastive learning for debiased candidate generation in large-scale recommender systems. In KDD, 2021.
[15] Lianghao Xia, Chao Huang, Chunzhen Huang, Kangyi Lin, Tao Yu, and Ben Kao. Automated self-supervised learning for recommendation. In WWW, 2023.
[16] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018.
[17] Wentao Shi, Jiawei Chen, Fuli Feng, Jizhi Zhang, Junkang Wu, Chongming Gao, and Xiangnan He. On the theories behind hard negative sampling for recommendation. In WWW, 2023.
[60] Hugo Caselles-Dupré, Florian Lesaint, and Jimena Royo-Letelier. Word2vec applied to recommendation: hyperparameters matter. In RecSys, 2018.
[61] Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. On sampling strategies for neural network-based collaborative filtering. In KDD, 2017.
[62] Steffen Rendle and Christoph Freudenthaler. Improving pairwise learning for item recommendation from implicit feedback. In WSDM, 2014.
[63] Hao Tang, Guoshuai Zhao, Yuxia Wu, and Xueming Qian. Multi-sample based contrastive loss for top-k recommendation. IEEE Transactions on Multimedia, 2021.
[64] Zhuang Liu, Yunpu Ma, Yuanxin Ouyang, and Zhang Xiong. Contrastive learning for recommender system. CoRR, abs/2101.01317, 2021.
[65] Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix X. Yu, Ting Chen, Aditya Krishna Menon, Lichan Hong, Ed H. Chi, Steve Tjoa, Jieqi (Jay) Kang, and Evan Ettinger. Self-supervised learning for large-scale item recommendations. In CIKM, 2021.
[66] Lianghao Xia, Chao Huang, Yong Xu, Jiashu Zhao, Dawei Yin, and Jimmy X. Huang. Hypergraph contrastive collaborative filtering. In SIGIR, 2022.
[67] Haoxuan Li, Yi Bin, Junrong Liao, Yang Yang, and Heng Tao Shen. Your negative may not be true negative: Boosting image-text matching with false negative elimination. In ACM MM, 2023.
[68] Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, and Xi-angnan He. On the effectiveness of sampled softmax loss for item recommendation. CoRR, abs/2201.02327, 2022.
[69] Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. Ultragcn: Ultra simplification of graph convolutional networks for recommendation. In CIKM, pages 1253-1262. ACM, 2021.
[70] Thomas N. Kipf and Max Welling. Variational graph auto-encoders. CoRR, abs/1611.07308, 2016.
[71] Hongyi Wen, Xinyang Yi, Tiansheng Yao, Jiaxi Tang, Lichan Hong, and Ed H Chi. Distributionally-robust recommendations for improving worst-case user experience. In WWW, 2022.
[72] Jin Chen, Defu Lian, Yucheng Li, Baoyun Wang, Kai Zheng, and Enhong Chen. Cacheaugmented inbatch importance resampling for training recommender retriever. In NeurIPS, 2022.
We remind important related works to understand how our AdvInfoNCE stands and its role in rich literature. Our work is related to the literature on contrastive learning-based collaborative filtering (CL-based CF) methods, and theoretical understanding of contrastive loss in collaborative filtering.
A.1 Contrastive Learning-based Collaborative Filtering
The latest CL-based CF methods can roughly fall into two research lines. The first one, which we term the "augmentation-based" approach, leverages user-item bipartite graph augmentations along with augmented views as positive signals. The second category, referred to as "loss-based" approaches, mainly focuses on the modification of contrastive loss. In loss-based CF models, interacted items serve as positive instances.
- Augmentation-based [13-15, 52, 26, 27, 63-66]. The prevailing augmentation-based paradigm in CL-based CF methods is to employ user-item bipartite graph augmentations to generate contrasting views. These contrasting views are then treated as positive instances in the application of contrastive loss, such as InfoNCE loss, to further enhance collaborative filtering signals. Recent studies have extensively explored methods for generating contrastive views. Several studies like SGL [13] and DCL [64] elaborate on data-heuristic augmentation operators such as a random node or edge dropout and random walk. NCL [52] takes a different approach and incorporates potential neighbors from both the graph structure and semantic space into contrastive views. XSimGCL [27] takes it a step further and discards ineffective graph augmentations, choosing instead to employ a simple noise-based embedding augmentation. In pursuit of high-quality augmented supervision signals instead of handcrafted strategies, AutoCF [15] designs a learnable masking function to automatically identify important centric nodes for data augmentation.
- Loss-based [53-55, 22, 67]. Recent research, such as the experiments presented in SimGCL [26] and XSimGCL [27], has empirically shown that contrastive loss can be instrumental in enhancing the performance of CF methods, often playing a more significant role than heuristic-based graph augmentation. Despite these findings, there remains a gap in the exploration of loss-based CF methods, an area ripe for further investigation. BC loss [54] incorporates bias-aware margins into the contrastive loss, enabling the learning of high-quality head and tail representations with robust discrimination and generalization abilities. Adap- $\tau$  [55] proposes an adaptive fine-grained strategy for selecting the personalized temperature  $\tau$  for each user within the contrastive loss. HDCCF [22] devises a new contrastive loss function extending the advantage of negative mining from user-item to neighbored users and items.
A.2 Theoretical Understanding of Contrastive Loss in CF
Despite the remarkable success of CL-based CF methods, there remains a lack of theoretical understanding, particularly regarding the superior generalization ability of contrastive loss. In a study conducted by [68], three model-agnostic advantages of contrastive loss are theoretically revealed, including mitigating popularity bias, mining hard negative samples, and maximizing the ranking metric. CLRec [14] sheds light on contrastive loss from a bias-reduction perspective by revealing its connection with inverse propensity weighting techniques. XSimGCL [27] suggests that contrastive learning enables the recommender to learn more evenly distributed user and item representations, thereby mitigating the prevalent popularity bias in CF.
B In-depth Analysis of AdvInfoNCE
B.1 Complete Derivation of AdvInfoNCE
Full Derivation. We first introduce the widely-used LogSumExp operator in machine learning algorithms.
$$
\max  \left(x _ {1}, x _ {2}, \dots , x _ {n}\right) \approx \log \left(\exp \left(x _ {1}\right) + \exp \left(x _ {2}\right) + \dots + \exp \left(x _ {n}\right)\right) \tag {10}
$$
The fine-grained ranking criterion for a single positive interaction  $(u,i)$  is defined as:
$$
\forall j \in \mathcal {N} _ {u}, \quad s (u, j) - s (u, i) + \delta_ {j} <   0. \tag {11}
$$
Then we probe into the Eq (11) and transform it into an optimization problem as follows:
$$
\min  _ {\theta} \max  \{0, \left\{s (u, j) - s (u, i) + \delta_ {j} \right\} _ {j \in \mathcal {N} _ {u}} \} \tag {12}
$$
We can seamlessly transform this optimization objective into the core component of our AdvInfoNCE: