A Non-Contrastive Learning Framework for Sequential Recommendation with Preference-Preserving Profile Generation
Authors: Huimin Zeng
Xiaojie Wang
Anoop Jain
Zhicheng Dou
Dong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: sequential recommendation
non-contrastive learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We present a non-contrastive learning framework for sequential recommendation

Abstract: Contrastive Learning (CL) proves to be effective for learning generalizable user representations in Sequential Recommendation (SR), but it suffers from high computational costs due to its reliance on negative samples. To overcome this limitation, we propose the first Non-Contrastive Learning (NCL) framework for SR, which eliminates computational overhead of identifying and generating negative
samples. However, without negative samples, it is challenging to learn uniform representations from only positive samples, which is prone to representation collapse. Furthermore, the alignment of the learned representations may be substantially compromised because existing ad-hoc augmentations can produce positive samples that have inconsistent user preferences. To tackle these challenges, we design a novel preference-preserving profile generation method to produce high-quality positive samples for non-contrastive training. Inspired by differential privacy, our approach creates augmented user profiles that exhibit high diversity while provably retaining consistent user preferences. With larger diversity and consistency of the positive samples, our NCL framework significantly enhances the alignment and uniformity of the learned representations, which contributes to better generalization. The experimental results on various benchmark datasets and model architectures demonstrate the effectiveness of the proposed method. Finally, our investigations reveal that both uniformity and alignment play a vital role in improving generalization for SR. Interestingly, in our data-sparse setting, alignment is usually more important than uniformity.

Published as a conference paper at ICLR 2025
A NON-CONTRASTIVE LEARNING FRAMEWORK FOR SEQUENTIAL RECOMMENDATION WITH PREFERENCE- PRESERVING PROFILE GENERATION
Huimin Zeng**, Xiaojie Wang2 Anoop Jain", Zhicheng Dou', Dong Wang* 'University of Illinois trbana-Champaign *Amazon Renmiin University China (huiminz3, [wang24]@i11inge edu. (xiojie, anoopjai @ com, cn
ABSTRACT
Contrastive Learning (CL) proves be effective for learning generalizable user representations in Sequential Recommendation (SR), but it suffers from high com- putational costs due its reliance on negative samples. To overcome this limi- tation, we propose the first Non-Contrastive Learning (NCL) framework for SR, which eliminates the computational overhead of identifying and generating neg- ative samples. However, without negative samples. it is challenging to learn uni- form representations from only positive samples, which is prone to representation collapse. Furthermore, the alignment of the learned representations may be sub- stantially compromised because existing ad-hoc augmentations can produce pos- itive samples that have inconsistent preferences. To tackle these challenges, we design novel seeference-preserviee profile generation method to produce high-quality positive samples for non-contrastive training. Inspired by differential privacy, our approach creates augmented user profiles that exhibit high diversity while provably retaining consistent user preferences. With larger diversity and consistency of the positive samples, our NCL framework significantly enhances the alignment and uniformity of the learned representations, which contributes to better generalization. The experimental results on various benchmark datasets and model architectures demonstrate the effectiveness of the proposed method. Finally, our investigations reveal that both uniformity and alignment play vital role in improving generalization for SR. Interestingly, our data-sparse setting, alignment is usually more important than uniformity.
INTRODUCTION
Contrastive Learning (CL) proves to be an effective approach to learn generalizable user representa- tions for sequential recommendation (SR) (Wang et al., 2022b; Liu al., 2021; Xie et al., 2022). A canonical CL-based recommendation framework usually employs positive samples (e.g., augmented user profiles or items) promote representation alignment and negative samples (e.g. profiles of other users or irrelevant items) for representation uniformity (Wang & Isola, 2020; Zhuo et al., 2023). With enhanced alignment and uniformity, CL-based methods achieve state-of-the-an performance.
However, Cl.bbased methods inevitably suffer from high computational costs, because they heavily rely on negative samples to avoid representation collapse during training (Grill al., 2020; Chen & He, 2021). For example, CL-based methods either require large batch sizes during training use sophisticated retrieval strategies to identify hard negative samples (Robinson al., 2020), leading to challenges hardware memories and computational efficiency (Kulatilleke et al., 2023).
In sharp comparison, Con-Contratttve Learning (NCL) emerges as promising solution that avoids using negative samples (Zhuo et al., 2023). Unlike CL using negative samples for uniformity, NCL
WWrk done while interning at Amazon.
"oorrsponniig author.

---

Published as a conference paper at ICLR 2025
is capable of learning uniform representations using only positive samples. Therefore, in this work, we aim to learn uniformity using only positive samples for SR. The rationale behind NCL is: such uniformityy can best represent the information of individual data points, because probability distri- bution that best represents the current state of knowledge about a system is the one with the largest entropy according to information theory (Jaynes, 1957; Guiasu & Shenitzer, 1985). Nevertheless, despite the success of NCL for Computer Vision (Grill et al., 2020; Zbontar et al., 2021) and Natural Language Processing (Cho et al., 2022; Zhou al., 2023), NCL for recommendation Furthermore, learning generalizable user representations from recommendation data through NCL much more complex and difficult than learning representations for images and texts.
i is unexplored.
The key challenge stems from obtaining high-quality positive user profile pairs to enable non- contrastive training. On one hand, within an NCL. framework, the absence of negative samples makes it more challenging for the recommender to learn uniform representations from the sparse recommendation data than from dense CV or NLP data. This is because the sparsity of the recom- mendation data increases the risk of representation collapse Guo et al. (2023). In recommendation, the collapsed user representations are mapped nearby in the latent space, even if they represent dras- tically different user preferences. As consequence, such collapsed representation lacks expres- siveness to capture unique preferences of different users, making it difficult for the recommender to generate meaningful recommendations. On the other hand, the representation alignment may be substantially compromised as well, due to the inconsistent user preferences encoded in the positive samples. We note that existing operations are too ad-hoc to generate consistent positive user profile pairs. For instance, data augmentations are primary methods to generate positive samples, but they are usually random (e.g., random crop, random swap, or random mask (Xie et al., 2022)) or heuris- tic (correlation- based substitution or insertion (Liu et al., 2021)). Obviously, such ad-hoc operations may fail to preserve the genuine user preferences in the positive samples. These inconsistencies confuse the model, when the model is trained to align profiles that reflect different, even conflict- ing preferences for the same user. As a consequence, the representation alignment is substantially compromised, and the model fails to capture the true user preferences, degrading the performance.
Therefore, targeting the limitations of CL-based SR methods and NCL methods, we propose the first Non-Contrastive Learning framework for Sequential Recommendation powered by preference- preserving user profile generation: NCL-SR. The proposed NCL-SR eliminates the computational overhead of identifying and generating negative samples in CL. Furthermore, we design novel profile generation to address the representation collapse issue and preference inconsistency within the training data. Inspired by Differential Privacy (DP), our approach creates augmented user profiles that retain consistent user preferences with provable guarantees. To address representation collapse, our approach generates highly diversified positive samples with controllable randomness. The diversity of positive samples helps the model explore distinct and larger portions of the representation space, which effectively reduces the risk of representation collapse. In addition, design of our approach also addresses the inconsistent preference issue
preference-preseeee
the seeference-preservvin within the positive samples by preserving the genuine user preferences within the generated positive samples. Finally, note that the DP augmentation serves the key pre-requisite to promote both alignment and uniformity under our NCL framework. With the generated augmented user profiles, the proposed framework computes non-contrastive alignment loss and con-contrastive uniformity loss from matrix information theory (Zhang et al., 2023), to improve the generalization of the learned user representations. Overall, our contribution can be summarized as follows:
We design and present the very first supf-supee non-contrastive learning framework for sequential recommendation: NCL-SR. NCL-SR learns generalizable and robust user representations for sequential recommendation.
We present novel data augmentation operation with theoretical guarantees to enable non- contrastive training. Unlike existing ad-hoc augmentation operations, our approach is guar- anteed to preserve user preferences in the augmented user profiles, addressing the represen- tation collapse issue and preference inconsistency issue within the training data.
We conduct extensive experiments encompassing diverse datasets. Our experimental re- sults suggest NCL-SR consistently exhibits superior performance over ofate-of-the-ar SR models and CL-based SR methods. Our investigations reveal that uniformity and alignment of the representations play a vital role in SR. Interestingly, we observe that alignment may be more important than uniformity.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Self-Supervised Learning for Sequential Recommendation. There are three types of self- supervised learning schemes for sequential recommendation: generative, adversarial, and con- trastive (Ren et al., 2024). Generative SR models primarily focus on generating sequence item data as recommendations (Ye al., 2023; Wu al., 2023; Wang et al., 2022d). An exemplar gen- erative SR model is BERT4Rec (Sun et al., 2019), where the recommender is trained to generate and reconstruct the masked items within user histories. In comparison, adversarial SR models (Ni et al., 2023; Lv et al., 2021) are mainly GAN(Coodfellow et al., 2020)-like in the sense that they usually train discriminators that distinguish between real and generated item sequences, while op- timizing generators to produce realistic recommendations. Compared generative and adversarial SR models, contrastive SR models comprise the majority of self-supervised SR methods and are state-of-the-art. Existing CL-based SR methods focus on designing sophisticated rules and augmen- tation operations to construct contrastive pairs. For instance, CLS4Rec (Xie et al., 2022) adopts three randomized operations to generate augmented views of user history. Based on CLS4Rec, CoSeRec (Liu et al., 2021) further proposes two informative augmentations: correlation-based item substitution and insertion. Liu al. (2021) present EC4Rec, an explanation-guided CL framework that identifies positive/negative items using training gradients. In comparison, DUORee (Qiu et al., 2022) showcases a supervised contrastive learning framework with model-level dropout augmenta- tions to further improve representation learning. In (Zheng et al., 2022), aiming at the Sequential Recommendation challenge, the authors propose novel model via dual dynamic graph modeling and hybrid metric training with contrastive learning integration. In (Su et al., 2023), novel hierarchy-aware dual clustering graph network (HADCG) model designed to explore the inherent hierarchy structures from both item popularity and collaborations. With information regularizer for intra-session clustering and contrastive training for inter- clustering, HADCG achieves substantial performance improvements in the ression-based recommendation. Finally, Shi et al. (2024) present self contrastive learning (SCL.) to enhance the uniformity of the learned item representations. However, CL-based methods usually pose challenges to hardware memories and computational efficiency, because they either require large batch sizes or sophisticated retrieval strategies to identify hard negatives (Robinson et al., 2020). Furthermore, augmentation opera- tions of CL-based methods may also introduce preference inconsistency in the positive samples, because they may fail preserve user preferences. This makes CL-based methods sensitive to data augmentation.
Non- Non-Contrastive Self-supervised Learning. Recently, there is an increasing interest in Non- Contrastive Learning (NCL) for its capability of learning robust representations without negative samples. Unlike CL which pursues both aligmment and uniformity in representation learning, NCL tends to focus more learning uniform data representations. For instance, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) introduce asymmetry in the network architecture and parameter updates avoid representation collapse. Barlow Twins (Zbontar al., 2021) enhances uniformity by making the cross-correlation matrix of two twin representations as close to the identity matrix as possible. Liu et al. (2022) present theoretical framework based on the maximum entropy en- coding principle from information theory, encompassing all previous NCL losses into a generalized uniformity loss. Finally, harmonizing alignment and uniformity, Zhang et al. (2023) propose matrix information theory, and design matrix -teoorethe self-supervised framework. Unfortunately, NCL demonstrates effectiveness only for CV and NLP tasks where the data dense and the training sig- nals are based on explicit feedback, while its effectiveness for SR is unexplored. Furthermore, the sparsity of the recommendation data leads to higher risk of representation collapse for NCL.
PRELIMINARIES
Data. We focus on sequential recommendation (SR). an SR dataset D, a data sample is a se- quence of interacted items x (sorted by timestamps) from a user's history. We denote list of interacted items as x = [z1 22, ..., 41]. For æ, its ground-truth label is the item at the next time step: y L141. Each element in and y belong to the item scope I: 2; E I.
Model. Classic sequential recommenders are ID-based models that represent items with discrete item IDs. They take sequences of item IDs as inputs, and predict new item IDs recommendations.
3

---

Published as a conference paper at ICLR 2025
Given z, ID-based recommender computes scores for all items from I, and selects items with highest scores for recommendations. In comparison, text-based recommenders represent items with textual descriptions (e.g,, title, category, reviews), and generate recommended items with free-form texts (e.g., item titles). In this work, we use text-hased model to build NCL-SR for the generality of textual features. For clarity and consistency, also texts of text-based recommender, where each element z; is the textual description of an item.
[ --.y 2i to denote the input
Contrastive Learning. Existing CL-based recommendation methods usually employ the follow- ing contrastive loss to train recommendation models (Xie et al., 2022; Zhang et al., 2024; Shi et al., 2024). Formally, for batch of training data X, the contrastive loss is computed with:
s(x(1), x((P1)
La
llog s(z(a), x(1) E s(x(a), 22 (())))
(1)
x(-)~xX
where æ(4) denotes an anchor training sample within the batch, (x()) denotes a positive sample of
(00), and (x()) denotes negative sample of x(() (assume negative samples for z(a)), s represents an arbitrary similarity scoring function (e.g., cosine similarity) in canonical form. However, CL
suffers from high computational costs due to its reliance on negative samples and is sensitive to data augmentation. The above limitations of CL motivate our NCL framework.
4 METHODOLOGY
In this work, we propose novel NCL-SR framework via preferencee-rreservnng profile generation to promote uniformity and alignment in user representations.
Uniformity ensures that that individual user preferences are preserved as much as possible in the user representations. However, within an NCL framework, the absence of negative samples makes challenging to learn uniformity from sparse recommendation data. This is because such sparsity increases the risk of representation collapse (Guo et al., 2023) where different user preferences are mapped nearby in the latent space. Consequently, it becomes difficult for the recommender to capture unique preferences of different users and generate meaningful recommendations. Alignment enforces that two users with similar preferences should be mapped to nearby representations, which makes representations robust against undesired noise factors. Nevertheless, the inconsistency within the training data hinders learning such alignment (e.g.. implicit feedback, inconsistent user prefer- ences in the augmented data). These inconsistencies confuse the model, when the model trained to align profiles that reflect different, conflicting preferences for the same user.
Therefore, we design the peeference-preserving profile generation to generate high-quality positive user profiles to facilitate non-contrastive training. To address representation collapse, our approach generates diversified positive samples. This helps the model explore distinct portions of the represen- tation space, reducing the risk of representation collapse. In addition, with the reference- reference-preservel preserved positive samples, the data inconsistency issue is effectively addressed.
4.1 PREFERENCE- PRESERVVIG PROFILE AUGMENTATION VIA DIFFERENTIAL PRIVACY
To build the NCL framework for SR, we first propose the preference-preserving user profile gen- eration. Inspired by Differential Privacy (DP) (Dwork et al., 2006), our approach creates aug- mented user profiles that retain original user preferences with provable guarantees. The preference- preserved augmentation generates diverse and consistent positive samples to overcome the represen- tation collapse and inconsistency within the training data, contributing better alignment and uni- formity of the learned user representations. For a better understanding of our seffference-peseervinn profile generation, we begin by formally defining preference-prees preseccippeeeiinggg augmentation. We provide further insights about Differential Privacy (DP) in Appendix A.
Definition (Preference-Preeeevvnng Augmentation) For a user profile x, its augmented view x is considered preference-preserving x' leads to the same expected purchase, or the expected top- 1-ranked item remains the same, when x' is fed into the same recommendation mechanism M.
With the definition of preference- seeference-preserving augmentation, we then present how DP guarantees sefference-preservaan in Theorem 1.
4

---

Published as a conference paper at ICLR 2025
Theorem (Guaranteed Preference-Preservatie with Differential Privacy) Suppose recom- mendation mechanism M satisfies differential privacy with the privacy parameter e (i.e. c-DP). For any user sequence z. if M successfully recommends the ground-truth item which is the j-th item within the item scope I: == Fj, and the predictive score for the ground-truth itm is greater than the second- largest runner-up score of another item with a small multiplicative factor e²c:
E(M(z)) > e² max B(M(z)a),
(2)
kekfs
then for its augmented view s', its predicted top-1 item remains the same if there is only limited modification from x to x::
B(((())))> max E(M(z)) )&)- k
Note that Theorem I a specific adaptation of a general DP property for our SR problem. The proof is adapted from (Wang et al., 2021), which we relegate to Appendix A. Since the recommendation data considered in this work is discrete (i.e., item texts), we use the DP exponential mechanism to build our DP augmentation pipeline, which is formally defined below.
Definition 2 (Exponential Mechanism) The exponential mechanism ME(u, Du, P) is character- ized by three components: scoring function the sensitivity Au and sampling distribution P. The scoring function (X, Xx) computes scores for each pair of inputs ( (X) and a candidate (Xe). The sensitivity Au is defined as Au == maxx, maxx (o(X.Xe) u(X', Xe)) The exponential is mechanism Mg(u, - difffrreential private if it outputs the candidate Xe with probabil- ity P proportional to 235
xm(
User Profile Augmentation via Exponential Mechanism. When implementing the exponential mechanism for NCL-SR, items will be used to perturb the original user history following the exponential mechanism. For specific item text x; E I. its synonym items other items with top-k highest cosine similarity in the representation space. Formally, the synonym set S(z,) of an item Iú with top-k similarity is computed with:
first craft set of synonym dictionary for all items. The synonym
(1,) arg max NCI EN WEE
CosSim(f(a:),
where ff represents text-based recommender model that encodes user/item texts into latent repre- sentations. After replacing randomly selected items within x = [x1, 22, .2,, 2i] with their synonyms, we obtain a set of candidate user profiles = {x' }. In terms of the exponential mechanism, the scoring function u is then defined as the cosine similarity between embedded user profile and any other candidate user profiles z':
u(I, (on)im(((e)))((a))
(5)
Note that the candidate profiles x' are perturbed compared to the original user profile as, and the perturbations can be multiple on different positions. With the definition of Equation 5, the sensitivity of the utility score u(z, x is A. = e 1/e, with being maximum of u(x, x) and 1/c being the minimum. Finally, the exponential mechanism selects and outputs x with sampling
cotca")
probability e 22g 25gg
For computational efficiency, the exponential mechanism returns the expected latent representation of X' by computing the weighted sum of x' with the importance weight for each æ being the normalized sampling probability:
eufa.a
E
P(') where pS. 8'EX'
24,
(6)
Ex"ex"
To this end, we obtain the augmented user profile for a training user. The DP user profiles effectively preserve the original user preferences and serve the foundation for the NCL framework.
5

---

Published as a conference paper at ICLR 2025
Exponential Mechanism
Forward
Backward
NCL
Augrented Profiles for KCL Loss
NCL loss
Sequential Recommander
Task loss
Batch augrented sequences
Preference-
Seeference-Prrservingg Preserving
Multi-task Optinization
Augmentation
Figure The overview of the proposed NCL-SR. NCL-SR first performs preference- pressrrving augmentation to generate high-quality positives. Then, NCL-SR optimizes combined training loss of NCL losses and the main task loss (e.g., cross-entropy or other ranking loss).
Note that critical caveat here is that the search space grows exponentially w.r.t the length of user histories if the augmentation is defined at user-level: O(k"). This is because each item T2 [1) -..5 has k replacements. As such, there are k possible candidate user profiles for the exponential mechanism to sample from, i.e., |X k Given the sparsity of recommen- dation data, computing DP augmentations for user sequences introduces prohibitive computation complexity and noises. Therefore, in our experiments, we re-define Equation and Equation 6 at item-level, successfully reducing the complexity be linear: O (). To see this, highlight that when applying the exponential mechanism to each item mechanism just needs to sample from &o candidates, and repeat for all items within the history / times, leading O(k 1). More details (guarantees, proofs, and implementation) about the efficient design are documented in Appendix B.
= [x1, ..., zi). the exponential
An illustrative example of DP profile generation is shown in Figure 1. In Figure 1, each polygon represents the synonym set for an item. The vertices within a polygon represent the embeddings of the synonym items w.r.t. the original item. For instance, for the first item (i.e., the movie "Save the Private Ryan ") of the user's history, our method first encodes this item into an embedding vector. Then, we compute the synonym set for this item with Equation 4. Therefore, all vertices in the polygon represent the embedded items. Note that there is no corresponding polygon for the third item in Figure 1, because our approach only perturbs part of the user profiles, which reduces computation costs of the exponential mechanism.
Note this efficient design feasible and remains DP,, because of the key properties of DP. First, DP has the post-processing property: any computation applied to the output of DP algorithm re- mains DP. As such, if the augmentation operation is DP, then the recommendation process based on the augmented user profile is also DP. Second, the expected output stability property of DP re- duces the computational costs of DP augmentation. This property allows us generate an expected augmented profile for each user to preserve user preferences in the augmented profiles, instead of generating massive augmented samples for Monte-Card sampling.
4.2 OVERVIEW OF NCL-SR
With the proposed
profile generation, we then introduce the overall NCL-SR. As shown in Figure 1, our proposed NCL-SR consists of 3 components.
The first component is text-based recommender that encodes user/item texts and returns rele- vance scores over candidate items. For any user sequence æ. the recommender f encodes user/item texts into representations and then returns probability distribution over the item scope I:
P = P(f. x, I) Softmax ( ([CosSim(g, P1). CosSim P2), CosSim (izs))
(7)
where q = (z) is the embedded user profile, and p, = fr(z)), xy E I is the embedded descrip- tion of an item zy. The items associated with highest probabilities are retrieved recommendations.

---

Published as a conference paper at ICLR 2025
The second component is multi-task optimization, where optimize the model w.r.t the ranking task, alignment learning task, and uniformity learning task. The multi-task optimization benefits recommendation performance, because both the recommendation task and non-contrastive training are modeling user/item relationships in the representation space. To design a general NCL. frame- work for sequential recommendation, we adopt Matrix Cross Entropy (MCE) (Zhang et al., 2023), which generalizes multiple non-contrastive losses including SimSiam (Chen & He, 2021), Barlow Twin (Zbontar et al., 2021), MEC (Liu al., 2022), etc. Formally, the matrix cross entropy between two matrices U and V is defined as
MCE(U, V) = tr( U log V + V),
(8)
where log denotes the principal matrix logarithm (Higham, 2008) and approximated via Taylor expansion. Based on the definition of matrix cross entropy. one may promote uniformity in repre- sentation learning by aligning the covariance matrix of representations with the identity matrix:
Luniform (2, z') MCD ((ZzZ)))
where 2 = f(X) and z = f(X') denote encoded, d-dimensional, 12-normalized representations of batched user profiles and their preference-preserving augmentation, respectively. C(Z, 7 rep- resents the centered covariance matrix of Z and Z Z(Z,Z') ; ZH H = IB n 1 h- Intuitively, minimizing Cunifor (Z, )') promotes the uniformity in representation learning, because the minimization makes the covariance matrix of the embedded (augmented) user representations to be close to the identity matrix, which increases the entropy of the encoded representations.
Note that Equation 9 well as other existing non-contrastive methods only promotes uniformity in representation learning. However, it has been widely shown that learning aligned user representa- tions also plays vital role in improving model generalization for sequential recommendation. This is because aligned representations are more robust against the noises caused by data inconsistency (Wang et al., 2022b; Xie al., 2022: Qiu al., 2022). Therefore, we propose integrate another alignment loss to train the model w.r.t. the alignment learning loss:
Calign (Z, z tr 2)) + MCE(C(Z, C(Z', )))
(10)
Intuitively, since Z and Z are derived from the same group of users as in Equation 9, minimizing Calign( (Z, Z') enforces the similarity of the learned representations from the same user. To this end, the multi-task optimization is a combination of losses of three tasks:
LNCL-SR
Liask + a1= Lumitorm + X2 Lalign-
(11)
The third component of NCL-SR is the preference- pereencceepeeeiiigggge profile generation introduced in the previous section. We highlight that both Equation 9 and Equation 10 require the augmented views of original user profiles. In other words, generating high-quality augmented data is the prerequisite for non-contrastive training. This component can effectively preserve the user preferences in the diverse, augmented user profiles, which overcomes the representation collapse issue and the inconsistency within the training data.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Datasets. We adopt 6 public benchmark recommendation datasets evaluate NCL-SR: Beauty, Games, Sports, Toys, Office and Auto (He & McAuley, 2016; McAuley et al., 2015)". These datasets cover different application domains, and are characterized with different sparsity and quence lengths. Following Yue et al. (2022), we use the 5-core processing to filter out infrequent items and users. To simulate the data sparsity and cold-start users, the split ratio of the training. validation, and test sets is 2:2:6. Such a split ratio is selected because we aim to explore the model's performance under a limited quantity of training data and the model's generalization on cold-start users, following the recent studies Wu et al. (2024); Qian et al. (2020); Wang et al. (2022a): Lin al. (2025). More details about dataset statistics are summarized Appendix Ct.
"The datasets are available at
*Our code is available at http://gghha..cm/hmuirreeggggCC

---

Published as a conference paper at ICLR 2025
ID-Based
Text- Based
Dataset
Metric
Improv.
BERT4Rec
SASRec
NARM
LRURec
P5
UniSRec
RecPmnr
NCL-SR
(%)
R@:01 f
0.0429
0.0599
0.0493
0.0715
0.0227
0.0437
0.0729
0.0791
8.50%
N800 f
0.0232
0.0362
0.0281
0.0439
0.0133
0.0249
0.0407
0.0440
0.23%
Beauty
R@:20 1
0.0617
0.0812
0.0719
0.0931
0.0282
0.0656
0.1035
0.1135
9.66%
N8 f
0.0279
0.0415
0.0338
0,0493
0.0147
0.0304
0.0484
0.0526
6.69%
R@10 f
0,0564
0,0934
0.0661
0.1087
0.0120
0.0691
0.0867
0.1140
4.88%
Games
N@ f
0,0294
0,0490
0.0350
0.0606
0,0063
0,0365
0.0465
0,0611
0.83%
R f
0,0932
0.1323
0.1036
0.1540
0.0196
0,1024
0.1339
0,1683
9.29%
N800 f
0.0386
0.0587
0.0444
0.0720
0.0083
0.0449
0.0583
0.0748
3.89%
R@10 f
0.0195
0.0259
0.0272
0.0327
0.0057
0.0203
0.0331
0.0441
33.2%
N@10
0.0111
0.0154
0.0147
0.0201
0.0028
0.0115
0.0172
0.0237
17.9%
Sports
R
0.0291
0.0330
0.0389
0,0464
0,0085
0,0317
0,0518
0.0660
27.4%
N880 f
0.0135
0.0172
0.0177
0.0235
0.0036
0.0143
0.0218
0.0292
24.3%
1@ f
0.0278
0,0379
0,0299
0.0655
0,0253
0,0382
0.0845
0.0941
11.4%
N 1810 f
0.0166
0.0226
0.0173
0.0420
0.0139
0.0221
0.0475
0.0537
13.1%
Toys
R f
0.0379
0.0507
0.0444
0.0858
0.0319
0.0548
0.1
0.1286
10.5%
200 f
0.0192
0.0258
0.0210
0.0471
0.0156
0.0262
0.0555
0.0623
12.3%
R0:0 f
0,0453
0.0524
0.0654
0.0926
0,0528
0.0629
0.0645
0.1047
13.1%
N800 f
0,0231
0.0325
0.0333
0.0496
0.0255
0.0300
0.0350
0.0534
7.66%
Office
R f
0.0782
0.0996
0.1155
0,1462
0.0915
0.1018
0.1025
0.1625
11.2%
N@20 f
0.0314
0.0445
0.0459
0.0631
0.0351
0.0398
0.0445
0.0681
7.92%
R810 f
0.0415
0.0439
0.0585
0.1220
0.0431
0.0646
0.1085
0.1354
11.0%
N110
0.0202
0.0216
0.0287
0.0632
0.0216
0.0355
0.0571
0.0714
13.0%
Auto
R R-0/00 t
0.0732
0.0780
0.1024
0.1829
0,0866
0. 0.1098
0.1768
0.2085
14.0%
28 f
0.0282
0.0304
0.0396
0.0785
0.0325
0,0469
0.0745
0.0899
14.5%
Table I: Main results recommendation performance of different SR models. The best results are highlighted in bold and the second best results are highlighted with underline.
Baselines and Implementation. There are two sets of baselines our experiments. The first group of baselines are state-of-the-art SR models, including ID-based models (i.e., NARM (Li al., 2017), LRURee (Yue et al., 2023), BERT4Rec (Sun et al., 2019), SASRec (Kang & McAuley, 2018)) and text-based SR model: P5 (Geng et al., 2022), RecFmr (Li et al., 2023) and UniSRT (Hou et al., 2022). The second group baselines are state-of-the- art CL-based recommendation meth- ods, including CLSS4Rc (Xie et al., 2022), CoSeRec (Liu et al., 2021), EC4Rec (Liu et al., 2021), DUORec (Qiu et al., 2022) and SCL (Shi et al., 2024). Note that some of the CL-based baselines are developed based on item IDs, and we modify such baselines into the text-based setting for (Wang et al., 2022c) to implement NCL-SR and CL-
fair comparison. We use E5 based baselines. When generating DP augmentations, we randomly perturb 3 items for each training user from datasets. For evaluation, we use normalized discounted cumulative gain (NDCG @ N) and recall @Recall@N) with N [10, 20). The predictions are ranked against all items in the dataset.
5.2 COMPARING AGAINST SR MODELS
The first set of experiments is conducted to compare NCL-SR against state-of-the-ar sequential rec- ommenders. The performance results are reported in Table 1. The last column of Table 1 is relative improvement NCL-SR compared to the best-performing baseline method (i.e., Improv.). We ob- serve: (1) NCL-SR consistently outperforms baseline methods across all metries and datasets, with an average performance improvement of 11.93% compared to the second best method. Such an ob- servation demonstrates the generality of NCL-SR, regardless of data domains. (2) The performance gains of NCL-SR are more pronounced on the sparsest dataset (i.c., Sports). where NCL-SR can go up to 33.2% on Recall@ 10. This demonstrates the substantial benefits of applying NCL-SR in sparse data domains. (3) NCL-SR generally demonstrates better retrieval performance (i.e., Recall) than ranking performance (i.e.. NDCG). For instance, there is a significant increase of 16.7% in the average Recall @ 10 scores with NCL-SR, while the relative improvement on NDCG @ 10 is slightly lower (8. (8%). This is expected, as our text-based model has a strong ability to understand and match the semantic meanings of item descriptions, which provides advantages for the retrieval task. We also implement the SR models with E5 for additional comparison, whose results are reported in Appendix F.
8

---

Published as a conference paper at ICLR 2025
CL-Based
NCL
Dataset
Metric
Improv.
CLS4Rec
CoSeRec
DUORee
EC4Rec
SCL
NCL-SR
(%)
R 7
0.0690
0.0698
0.0706
0.0628
0.0709
0.0791
11.6%
N8 1
0.0367
0.0375
0.0383
0.0344
0.0375
0.0440
14.9%
Beauty
R@20 7
0. 1008
0.1007
0.1052
0.0971
0.1026
0.1135
7.89%
N820
0.0447
0.0453
0.0470
0.0430
0.0456
0.0526
11.9%
R$10 f
0.0992
0.1057
0.1041
0.0998
0,1012
0.1140
7.85%
N810
0.0521
0.0553
0.0543
0.0524
0.0533
0.0611
10.5%
Games
R@20
0.1458
0.1560
0.1545
0.1516
0.1499
0,1683
7.89%
N820
0.0638
0.0680
0.0669
0.0654
0.0655
0.0748
10.0%
R@10 T
0.0325
0.0327
0.0332
0.0337
0.0314
0.0441
30.9%
18 1
0.0173
0.0169
0.0174
0.0180
0.0163
0.0237
31.7%
Sports
R@20 f
0.0529
0.0501
0.0504
0.0511
0.0473
0.0660
24.8%
N820 f
0.0223
0.0213
0.0217
0.0223
0.0203
0.0292
30.9%
Ra f
0. 0.088
0.0868
0.0882
0.0871
0.0895
0.0941
4.79%
N810 f
0.0511
0.0490
0.0501
0.0497
0.0511
0.0537
5.09%
Toys
R@20
0.1221
0.1199
0.1232
0.1201
0.1249
0.1286
2.96%
N820 f
0.0592
0,0573
0.0589
0.0580
0.0600
0.0623
3.83%
R810 T
0.0974
0.0967
0.0974
0.0913
0.0980
0.1047
6.84%
N800 1
0.0498
0.0487
0.0518
0.0465
0.0502
0.0534
3.09%
Office
RØ20 1
0,1471
0.1561
0.1497
0.1459
0.1500
0.1625
4.10%
N422 f
0.0623
0.0635
0.0647
0.0603
0.0634
0.0681
5.26%
0.1159
0.1171
0.1098
0.1073
0.1122
0.1354
15.6%
N8 f
0.0575
0.0576
0.0572
0.0535
0.0559
0.0714
24.0%
Auto
R@20
0.1829
0.1902
0.1927
0.1793
0.1780
0.2085
8.20%
N820
0.0742
0,0761
0,0781
0.0713
0.0724
0.0899
15.1%
Table 2: Main results on recommendation performance of different CL and NCL methods.
5.3 COMPARING AGAINST CL-BASED METHODS
We compare NCL.-SR against tate-offthheaar Cl.-based SR methods. The performance results are reported in Table 2. We observe: (1) our method demonstrates consistent performance improvements over the bascline CL methods across all metrics and datasets, with an average improvement of 12.48% compared to the second best method. (2) On the most sparse dataset (i.e., Sports), NCL- SR still achieves the most performance gain, suggesting that NCL is more efficient that CL-based methods in terms of learning generalizable representations from sparse data. Recall Due to the space limit, we provide additional experiments on exploring the relationship between CL. NCL, and data augmentation in Appendix D, and compare their memory consumption in Appendix G.
5.4 ABLATION STUDY
Alignment and Uniformity. We conduct an ablation to understand the effects of learning aligned and uniform representations for our SR task. particular, we set A = 0 in Equation 11 to remove the uniformity (Ours w/o Uni.) or set X2 0 remove the alignment (Ours w/o Align.), respec- tively. The experimental results are reported in Table 3. From Table 3, it is observed that in general, both alignment and uniformity play vital role in learning generalizable user representations for SR. Specifically, when removing the alignment loss Equation 11, the averaged performance on Recall @ 10 and NDCG@ 10 across all datasets would drop by 6.10% and 6.47%, respectively. In comparison, there is an average drop of 4.00% on Recall@ 10 and 3.92% on NDCG@ if uni- formity is removed. Furthermore, better recommendation performance can be observed when only learning the aligned user representations. For instance, on Beauty, there is noticeable increase in both Recall 20 and NDCG@20 20 without uniformity loss. To this end, we conclude that promoting alignment may be more important than uniformity in learning generalizable representations for SR applications. This further necessitates the design and merit of the proposed preference- preserrvig augmentation, which addresses the preference inconsistency caused by ad-hoc augmentations.
DP augmentation and NCL loss. We then conduct another ablation to investigate the effects of the proposed DP augmentation and the NCL losses. Specifically, we mask out either module by (1) replacing our DP augmentation with other data augmentation operations that are randomly sampled from the CL-based baselines (Ours w/o DP Aug.) (2) replacing the NCL. loss with the

---

Published as a conference paper at ICLR 2025
Variants
Metric
Beauty
Games
Sports
Toys
Office
Auto
Recal//NDCG
RecalINDDCG
Recall/NDCG
Recall/NDCG
Recall/NDCG
Recal//NDCG
NCL-SR
0.0791 0.0440
0.0441 0.0237
0.0537
0.1047 000534
0.0714
(Ours)
20
0.1135 0.0526
0.1683 0.0748
0.0660 / 0.0292
0.1286 0.0623
0.1625 0.0681
0,2085
0,0899
(1) Ours
@ 10
0.0761 / 100408
0.1133/ o. 0.0590
0.0384/ / 0.0205
0.0916/
0.0504
0.0954 / 0.0524
0.1293
0.0691
Align.
0.1124 0.0499
0.0606 0.0261
0.1270
0.1580 0.0680
0.1988
0.0865
(2) Ours
@ 10
0.0784 / 00.0437
0,1 0.0592
0,0404 / 0.0221
0.0529 0.1304
0,0936
0.0983 / 0.0051
0,1280 0.0661
w/o Uni
20
0.1149/ / 0.0529
0.1667
0.0728
0.0630 / 0.0277
0,0622
0.1519/ 0.0652
0.2171
0.0885
0.0717 0.0394
0,1029 0.0552
0.0364 0.0189
00.0517
0.1053 0.0530
0.0705
w/o DP Aug.
@ 20
0.1046 / 00.447
0,1520
0,0675
0.0543 / 0.0234
0,1
0,0606
0.1554 / 0.0656
0,2049/ / 0.0886
(2) Ours
0.0727/ / 0.0394
0.1014 0.0529
0.0369/ / 0.0198
0.0834 0.0456
0.0980 / 0.0527
0.1268
0.0678
w/o NCL
0,1077 00.088
0.1529 0.0659
0.0553 / 0.0245
0.1 0.0538
0.1465 0.0647
0,2073 0,0879
Table 3: Ablation Study on different components of NCL-SR.
(a) Uniformity sensitivity: changing A with fixed Ag in Equation 11.
Recalig10
NDCGE10
Recalig20
NDCGg20
Beauty
Games
Sports
Toys
Omcee
Auto
(b) Alignment sensitivity: changing 22 with fixed A in Equation 11.
Figure 2: Sensitivity Analysis w.r.t. Alignment and Uniformity.
contrastive loss (Ours w/o NCL). The results are reported in Table 3. It is observed that removing either module generally leads to performance degradation, indicating that both DP augmentation and NCL representation learning are necessary for improving recommendation performance.
5.5 SENSITIVITY ANALYSIS
We finally study the sensitivity of NCL-SR w.r.t the key hyperparameters A and Xg in Equation 11, where λι controls the strength of uniformity and A2 controls the strength of alignment. When performing sensitivity analysis, we select and fix the best A1. and change X2 to understand the sensitivity w.r.t. alignment. Similarly, X2 is fixed and λ₁ is changed for the sensitivity analysis w.r.t. uniformity. The performance is visualized in Figure 2. We observe that there exist different optimal configurations for different datasets. Additional sensitivity analysis can be found Appendix E.
6 CONCLUSION
In this work, we design the very first Non-Contrastive Learning framework for Sequential Recom- mendation: NCL-SR. In particular, we present a principled data augmentation operation to produce preferencee-reserviin user profiles, based which we define uniformity and alignment losses to leam user representations. Our investigations reveal that learning uniform and aligned user repre- sentations play vital role in the SR task, and alignment may be more important than uniformity. Our extensive experimental results suggest NCL-SR consistently exhibits superior performance over fate-offtheeart SR models and CL-based SR methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We thank Kiran Koshy Thekumprampil and Mukund Seshadri for their support in model training infrastructure. We also thank all the anonymous reviewers for their helpful comments.
REFERENCES
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758 15758, 2021.
Jaejin Cho, Jesús Villalba, Laureano Coro-Velazquez, and Najim Dehak. Non-contrastive self- supervised learning for utterance-level information extraction from speech. IEEE Journal of Se- lected Topics in Signal Processing, 16(6):1284-1299 2022.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-2844 Springer, 2006.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rip): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 299-315, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to superviee learning. Advances in neural information processing systems, 33:21271-21284. 2020.
Silviu Guiasu and Abe Shenitzer. The principle of maximum entropy. The mathematical intelli- gencer, 7:42-48, 1985.
Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, and Mingsheng Long. On the embedding collapse when scaling up recommendation models. arXiv preprint arXiv:2310.04400, 2023.
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517, 2016.
D Hemkumar and Pvn Prashanth. A weighted privacy mechanism under differential privacy. In 2023 4th International Conference on Intelligent Technologies (CONIT), pp. 1-6. IEEE, 2024.
Nicholas J Higham. Functions of matrices: theory and computation. SIAM, 2008
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 585-593, 2022.
Edwin Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Wang- -henn Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 197-206. IEEE, 2018.
Gayan K Kulatilleke, Marius Portmann, and Shekhar S Chandra. Efficient block contrastive learning via parameter-free meta-nosd node approximation. Neurocomputing, 561:126850, 2023.
"kkt @ amazon.com
mukunds @ @amazon.com
11

---

Published as a conference paper at ICLR 2025
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP). pp. 656-672. IEEE, 2019.
Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang. and Julian McAuley. Text is all you need: Learning language representations for sequential recommendation. arXiv preprint arXiv:2305. 13731, 2023.
Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session- based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1419, 2017.
Jianghao Lin, Xinyi Dai, Rong Shan, Bo Chen, Ruiming Tang, Yong Yu, and Weinan Zhang. Large language models make sample-efficient recommender systems. Frontiers of Computer Science, 1944)814328,, 2025.
Xin Liu, Zhongdao Wang, Ya-Li Li, and Shengjin Wang. Self-supervised learning via maximum entropy coding. Advances in Neural Information Processing Systems,35:340000 35:34091-34105, 2022.
Zhiwei Liu, Yongjun Chen, Jia Li, Philip S Yu, Julian McAuley, and Caiming Xiong. Con- trastive supervsue sequential recommendation with robust augmentation. arXiv preprint DrXiv:2108.064// 2021.
Yao Lv, Jiajie Xu, Rui Zhou, Junhua Fang, and Chengfei Liu. Ssrgan: A generative adversarial net- work for streaming sequential recommendation. In Database Systems for Advanced Application: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 1-14, 2021, Proceedings, Part III 26, pp. 36-52. Springer, 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Shuang Ni, Wei Zhou, Junhao Wen, Linfeng Hu, and Shutong Qiao. Enhancing sequential recom- mendation with contrastive generative adversarial network. Information Processing & Manage- ment, 60(6))003333)))) 2023.
Ticyun Qian, Yile Liang, Qing Li, and Hui Xiong. Attribute graph neural networks for strict cold start recommendation. IEEE Transactions on Knowledge and Data Engineering, 34(8):3597- 3610, 2020.
Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. Contrastive learning for representation degeneration problem in sequential recommendation. In Proceedings of the fifteenth ACM inter- national conference web search and data mining, pp. 813 823, 2022.
Xubin Ren, Wei Wei, Lianghao Xia, and Chao Huang. A comprehensive survey self-supervised learning for recommendation. arXiv preprint 2xXx::2404.03354 2024.
Joshua Robinson, Ching-Yao Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. 2020.
Zhengxiang Shi, Xi Wang, and Aldo Lipani. Self contrastive learning for session-based recommen- dation. In European Conference on Information Retrieval, pp. 3-20. Springer, 2024.
Jiajie Su, Chaochao Chen, Weiming Liu, Fei Wu, Xiaolin Zheng, and Haoming Lyu. Enhancing hierarchy-awer graph networks with deep dual clustering for bessidn-based recommendation. In Proceedings of the ACM Web Conference 2023, pp. 165-176, 2023.
Rishabh Subramanian. Have the cake and eat it too: Differential privacy enables privacy and precise 2023.
analytics. Journal of Big Data, 10223.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1441-1450, 2019.
12

---

Published as a conference paper at ICLR 2025
Chunyang Wang, Yanmin Zhu, Haobing Liu, Tianzi Zang, Jiadi Yu, and Feilong Tang. Deep meta- learning in recommendation systems: A survey, arXiv preprint arXiv:2206.0444 244224,,, 2022a.
Lei Wang, Ee-Peng Lim, Zhiwei Liu, and Tianxiang Zhao. Explanation guided contrastive learning for sequential recommendation. In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 2017-2027, 2022b.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- junder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. corr abs/2212.00533 (2022), 2022c.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp- 9929-9939. PMLR, 2020.
Wenjie Wang, Pengfei Tang, Jian Lou, and Li Xiong. Certified robustness to word substitution attack with differential privacy. Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1102-1112,
2021.
Yu Wang, Hengrui Zhang, Zhiwei Liu, Liangwei Yang, and Philip Yu. Contrastvae: Contrastive variational autoencoder for sequential recommendation. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Management, pp. 2056-2066, 2022d.
Xuansheng Wu, Huachi Zhou, Yucheng Shi, Wenlin Yao, Xiao Huang, and Ninghao Liu. Could small language models serve recommenders? towards data-centric cold-start recommendation. In Proceedings of the ACM on Web Conference 2024, pp. 3566-3575, 2024.
Zihao Wu, Xin Wang, Hong Chen, Kaidong Li, Yi Han, Lifeng Sun, and Wenwu Zhu, Diff4rec: Sequential recommendation with curriculum scchuuuled diffusion augmentation. In Proceedings of the s1 ACM International Conference on Multimedia, pp. 9329 9335, 2023.
Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. Contrastive learning for sequential recommendation. In 2022 IEEE 38th international con- ference data engineering (ICDE), pp. 1259-1273. IEEE, 2022.
Yaowen Ye, Lianghao Xia, and Chao Huang. Graph masked autoencoder for sequential recom- mendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 321-330, 2023.
Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, and Dong Wang. Defending substitution- based profile pollution attacks on sequential recommenders. Proceedings of the 16th ACM Conference Recommender Systems, pp. 59-70, 2022.
Zhenrui Yue, Yueqi Wang, Zhankui He, Huimin Zeng, Julian McAuley, and Dong Wang. Linear recurrent units for sequential recommendation. arXiv preprint arXiv:2310.023:2 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310- 12320. PMLR, 2021.
Yabin Zhang, Zhenlei Wang, Wenhui Yu, Lantao Hu, Peng Jiang, Kun Gai, and Xu Chen. Soft contrastive sequential recommendation. ACM Transactions on Information Systems, 2024.
Yifan Zhang, Zhiquan Tan, Jingqin Yang, Weiran Huang, and Yang Yuan. Matrix information theory for self-supervised learning. arXiv preprint arXiv:2305.17326 2023.
Xiaolin Zheng, Jiajie Su, Weiming Liu, and Chaochao Chen. Ddghm: Dual dynamic graph with hybrid metric training for cross-domain sequential recommendation. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 47 471- 2022.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets Linguage-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11028-11038. 2023.
13

---

Published as a conference paper at ICLR 2025
Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards unified theoretical understanding of non-contrastive learning via rank differential mechanism. arXiv preprint arXiv:2303.02387, 2023.
14


Reviews:
[{"review_id": "s4O9Xi6c1c", "reviewer": "Reviewer_u9dS", "summary": "Summary:\nThis paper addresses the issue of high computational cost in contrastive learning (CL)-based sequential recommendation. Innovatively, it introduces non-contrastive learning into sequential recommendation. To tackle the representation collapse and alignment problems caused by solely relying on positive samples, a novel user preference-preserving profile generation method is proposed for the Non-Contrastive Learning framework for sequential recommendation (NCL-SR). NCL-SR is capable of learning generalized and robust user representations for sequential recommendation. Experimental results demonstrate that NCL-SR outperforms traditional SR models and CL-based SR models in terms of performance.", "questions": "Limitation：\n1.What does \"s()\" represent in Eq. 1? What does \"f()\" represent in Eq. 4?\n2.What do the three polygons in the upper part of Fig. 1 represent? Why is there no corresponding polygon for the third red text?\n3.During the construction of candidate user profiles, a item-level approach is adopted. How is the number of perturbations determined? Why weren't further experiments conducted to verify the effect of perturbation quantity on recommendation performance?\n4.What is the split ratio of the training, validation, and test sets? Why wasn't it explicitly stated?\n5.The motivation of this paper is to address the high computational cost and memory consumption in negative sample extraction of CL in SR. Why weren't further experiments conducted to verify the effect of NCL-SR on computational cost and memory consumption?\n6.Further refinement of the text is needed. For instance, what is the relationship between “NCL SR” and “NCL-SR” in line 178? The term \"differential privacy\" in line 210 is in lowercase, while \"Differential Privacy\" in line 215 is in uppercase. Consistency should be maintained.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strength：\n1.This paper innovatively proposes a non-contrastive learning framework for sequential recommendation.\n2.Inspired by Differential Privacy (DP), a theoretically guaranteed user preference-preserving data augmentation method is proposed to address representation collapse and preference inconsistency issues.\n3.Two novel loss calculation methods are introduced.\n4.Extensive experiments are conducted to verify the effectiveness, and the roles of feature alignment and uniformity in sequential recommendation are analyzed.", "weaknesses": "Limitation：\n1.What does \"s()\" represent in Eq. 1? What does \"f()\" represent in Eq. 4?\n2.What do the three polygons in the upper part of Fig. 1 represent? Why is there no corresponding polygon for the third red text?\n3.During the construction of candidate user profiles, a item-level approach is adopted. How is the number of perturbations determined? Why weren't further experiments conducted to verify the effect of perturbation quantity on recommendation performance?\n4.What is the split ratio of the training, validation, and test sets? Why wasn't it explicitly stated?\n5.The motivation of this paper is to address the high computational cost and memory consumption in negative sample extraction of CL in SR. Why weren't further experiments conducted to verify the effect of NCL-SR on computational cost and memory consumption?\n6.Further refinement of the text is needed. For instance, what is the relationship between “NCL SR” and “NCL-SR” in line 178? The term \"differential privacy\" in line 210 is in lowercase, while \"Differential Privacy\" in line 215 is in uppercase. Consistency should be maintained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1U2fCWMbsw", "reviewer": "Reviewer_EEcc", "summary": "The authors propose a Non-Contrastive Learning framework for Sequential Recommendation powered by preference preserving user profile generation: NCL-SR. The NCL-SR framework eliminates the computational overhead of identifying and generating negative samples in CL. The experimental results on various benchmark datasets and model architectures demonstrate the effectiveness of the NCL-SR method.", "questions": "Please refer to the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The methodology of this paper is technically sound. The method itself is somewhat novel to me.\n\n+ To my best of knowledge, it is the first attempt to utilize matrix cross entropy in the recommendation system.\n\n+ The proposed NCL-SR achieves much better performance against other baseline models.", "weaknesses": "+ Fig.1. needs to be polished. For instance, it is unclear for me what is the plot meaning of the exponential mechanism with polygons in Fig.1? \n\n+ Why do you set the utility score as $\\Delta_u = e - 1/e$? It is unclear for me. Could you please kindly provide some insights on that? \n\n+ Could different values of $\\gamma$ affect the model performance? \n\n+ Some similar papers should be cited or discussed, e.g., [1] and [2].\n\nRef:\n\n[1] DDGHM: Dual dynamic graph with hybrid metric training for cross-domain sequential recommendation\n\n[2] Enhancing hierarchy-aware graph networks with deep dual clustering for session-based recommendation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VEJNDRAWmZ", "reviewer": "Reviewer_MpcM", "summary": "In this paper, the authors propose the first Non-Contrastive Learning (NCL) framework for Sequential Recommendation (SR). The authors design a novel preference-preserving profile generation method to produce high-quality positive samples for non-contrastive training. The paper is easy to follow and well-organized. The authors conduct extensive experiments to validate the efficacy of the proposed NCL-SR.", "questions": "Please refer to the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1: The paper is easy to follow and well-organized.\n\nS2: The authors provide theorem analysis for the proposed method which is novel and interesting to me.\n\nS3: The authors conduct extensive experiments to validate the efficacy of the proposed NCL-SR.", "weaknesses": "W1: Some illustrations should be improved. For example, the authors should motivate why you choose Matrix Cross Entropy (MCE) against some other methods (e.g., Barlow Twin, MEC and DirectAU) for contrastive learning? \n\nW2: Some method details are missing. For example, the calculation of $C(Z,Z’)$ in Eq.(9) should be provided to make the paper more readable.\n\nW3: Since you need to figure out the eigenvalue of $\\bm{V}$ in Eq.(9), how about the time complexity of your proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YZDoKbzZTa", "reviewer": "Reviewer_A3iE", "summary": "The paper introduces the NCL-SR, which is designed to reduce the computational burden found in traditional Contrastive Learning apporaches by eliminating the need for negative sampling. CL methods generally require negative samples to avoid representation collapse, but this comes with high computational costs. The main contribution of NCL-SR is its preference-preserving profile generation technique, which uses differential privacy to create diverse yet consistent positive user profile samples that retain the user’s preferences. This helps to overcome challenges such as representation collapse and preference inconsistency.", "questions": "- The authors claim that the user profile generation is inspired by differential privacy, I wonder how it inspired the proposed framework in details? Maybe some addtional literature review might help to explain.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper solves a important question in traditional SR field, that CL approaches cost quite a lot computational resources. By adopting the profile augmentation method proposed in this paper, the performance of SR can even higher than those traditional CL methods.\n\n- The authors provides theoritial guarantees for the uniformity and alignment.", "weaknesses": "- Since the main motivation of NCL is to reduce the cost of traditional CL methods, I believe the efficiency study is needed.\n- When comparing with other SR models (besides CL methods), I believe they should also be implemented with e5 for a fairer comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)