AdaptSSR: Pre-training User Model with Augmentation-Adaptive Self-Supervised Ranking
Authors: Yang Yu
Qi Liu
Kai Zhang
Yuren Zhang
Chao Song
Min Hou
Yuqing Yuan
ZHIhao Ye
ZAIXI ZHANG
Sanshi Lei Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: User Model Pre-training
Data Augmentation
Contrastive Learning
TL;DR: We propose an augmentation-adaptive self-supervised ranking task to tackle semantically inconsistent augmented behavior sequences for contrastive learning-based user model pre-training.

Abstract: User modeling, which aims to capture users' characteristics or interests, heavily relies on task-specific labeled data and suffers from the data sparsity issue. Several recent studies tackled this problem by pre-training the user model on massive user behavior sequences with a contrastive learning task. Generally, these methods assume different views of the same behavior sequence constructed via data augmentation are semantically consistent, i.e., reflecting similar characteristics or interests of the user, and thus maximizing their agreement in the feature space. However, due to the diverse interests and heavy noise in user behaviors, existing augmentation methods tend to lose certain characteristics of the user or introduce noisy behaviors. Thus, forcing the user model to directly maximize the similarity between the augmented views may result in a negative transfer. To this end, we propose to replace the contrastive learning task with a new pretext task: Augmentation-Adaptive SelfSupervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Specifically, we adopt a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employ an in-batch hard negative sampling strategy to facilitate model training. Moreover, considering the distinct impacts of data augmentation on different behavior sequences, we design an augmentation-adaptive fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the estimated similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks verify the effectiveness of AdaptSSR.

AdaptSSR: Pre-training User Model with Augmentation-Adaptiv Self-Supervised Ranking
Yang Yu! Qi Liul.2- Kai Zhang 1.2, Yuren Zhang!? Chao Song', Min Hou
Yuging Yuan", Zhihao Ye', Zaixi Zhang Sanshi Lei Yu!
Anhui Province Key Laboratory of Big Data Analysis and Application,
University Science and Technology of China
State Key Laboratory of Cognitive Intelligence
OPPO Research Institute "Hefei University of Technology (yfly1613, kkzhang0808, yr160698, zaixi) @mail ustc. edu. em {songchao12, yuanyuqing, yezhihao3 com qiliugleustc. edu. cn, (hmhoumin, meet. leiyulaggaill..oomm com
Abstract
User modeling, which aims capture users' characteristics interests, heavily relies on task-specific labeled data and suffers from the data sparsity issue. Several recent studies tackled this problem by pre-training the user model on massive user behavior sequences with a contrastive learning task. Generally, these methods assume different views of the same behavior sequence constructed via data augmen- tation are semantically consistent, i.e., reflecting similar characteristics or interests of the user, and thus maximizing their agreement in the feature space. However, due to the diverse interests and heavy noise in user behaviors, existing augmentation methods tend to lose certain characteristics of the user or introduce noisy behaviors. Thus, forcing the user model to directly maximize the similarity between the aug- mented views may result in a negative transfer. To this end, we propose to replace the contrastive learning task with Supervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the augmented views while pre- pre-training discriminative user model. Specifically, we adopt a multiple pairwise ranking loss which trains the model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employ an in-batch hard negative sampling strategy facilitate model training. Moreover, considering the distinct impacts of data augmentation on different behavior se- quences, we design an ingmentatioonadaatiii fusion mechanism automatically adjust the similarity order constraint applied to each sample based on the estimated similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks verify the effectiveness of AdaptSSR.
pretext task: Ingmentation-Aaaptii Self-
Introduction
User modeling aims to capture the user's characteristics or interests and encode them into a dense user representation for specific user-oriented task, such as user profiling, personalized recommendation. and thick-through rate prediction [4. 39
50 59] In the past decade, extensive methods [5] 9 [2]
22 53]
have been proposed model users based on their historical behaviors. Despite their great success in various tasks, these methods usually rely a large amount of task- labeled data to train accurate user models, which makes them suffer from the data sparsity problem [17 36 57
Qi Liu is corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original
Behavior
Sequence
Mask
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
Crop
Mask-and-Fill
Substitute
Space technology
Politics
Basketball
Noisy behavior
Figure 1: An illustration of the impact of different data augmentation methods on the user behavior sequence, For a more intuitive demonstration, we use a picture to represent the main topic of each news article clicked by the user. Pictures with the same color border reflect similar interests of the user. Dash borders are used to indicate behaviors replaced by the data augmentation method.
A mainstream technique tackle this challenge is the pre-training paradigm 33 46
54
56). The
user model first pre-trained on a mass of unlabeled user behavior data, which enables it to extract users' general characteristics from their historical behaviors. Then the model is transferred to benefit various downstream tasks via fine-tuning. Motivated by the recent development in CV [3] 7 [4] and NLP B 23], several works 22, & 388 explore pre-training the user model with contrastive learning task. They construct different views of the same behavior sequence via data augmentation and assume them to be semantically consistent, i.e., reflecting similar characteristies or interests of the user 22 47 Thus, these views are regarded as positive samples and a contrastive loss (e.g..
InfoNCE 30 is further applied maximize their similarity in the feature space.
Although such pre-trained user models have proven to benefit various downstream tasks, we find that their assumption about semantic consistency between the augmented views does not necessarily hold. Unlike texts and images, user behaviors usually contain not only diverse interests but heavy noise as well 4 26 43). Hence, existing data augmentation methods tend to result in augmented sequences that lose certain characteristics of the user or introduce noisy interests that the user does not have. To illustrate the impact of different augmentation methods the behavior sequence, randomly sample a user from the MSN online news platform His recent eight behaviors and the results of several data augmentation methods may infer the user's 's interests in space technology, politics, and basketball. Meanwhile, we find that various augmentation methods have distinct impacts different behavior sequences. For example, the masked behavior sequence [47 loses the user's interest in politics due to randomness, and the two sequences augmented by cropping 38 reflect the user's different interests over time. Other methods such as mask-an andd-il [2] and substitute (27] bring in behaviors related to football and tennis which the user may not be actually interested in. However, existing contrastive learning- based methods force the user model to maximize the agreement between such augmented views no matter whether they are similar or not, which may lead to a negative transfer for the downstream task.
shown in Fig. From the original behavior sequence,
To address this problem, in this paper, instead of trying to design a new data augmentation method that guarantees generate consistent augmented behavior sequences, escape the existing con- trastive learning framework and propose new pretext task: Augmentaation-Adaptive Self-Supervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the aug- mented views while pre-training a discriminative user model. Since the augmented views may reflect distinct characteristics or interests of the user, instead of directly maximizing their agreement, we adopt a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users.
MMor examples are provided in Appendix

---

Specifically, the user model is trained to project the original view closer to the implicitly augmented view than views from other users in the feature space, with the explicitly augmented view be- tween. We further utilize an in-batch hard negative sampling strategy to facilitate model training. In addition, since data augmentation has distinct impacts on different behavior sequences, we design an aigmentation-adaptiy fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the semantic similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks validate that AdaptSSR consistently outperforms existing pre-training methods while adapting to various data augmentation methods. Our code is available at htttt:://g/ii /hgbthb........ cww/ comm/ffly613///1/ AdaatsSRRR
2 Related Work
Contrastive Learning Aiming to learn high-quality representations self-supervised manner, contrastive learning has achieved great success in various domains [6 28 A1.. Its main idea is to learn a mapping function that brings positive samples closer together while pushing negative samples further apart in the feature space. One of the key issues for contrastive learning is how to construct semantically consistent positive samples. To tackle this problem, several data augmentation methods such jittering, rotation, and multi-cropping have been proposed for images [3,16] 29]. However, for user behavior sequences, due to the diverse user interests and heavy noise, it is hard to determine their semanties and even harder to provide a semantically consistent augmentation (25 32 49 which limits the effectiveness of existing contrastive learning- braneng-bbsse user model pre-training methods.
User Model Pre-training Various methods [24
53 60 have been proposed to model users based on their behaviors and task- mask-specific labeled data. However, these supervised methods usually encounter data sparsity issues [18] 33) Inspired by the success of the pre-training paradigm in CV and NLP,
33 46 have sought to empower various user-oriented tasks by pre-training the user model on unlabeled behavior sequences. Predominant methods can be categorized into two groups: generative and discriminative. Generative methods [45] 52 usually first corrupt the behavior sequence and then try to restore the masked part. However, they largely focus mining the correlation between behaviors but lack holistic user representation learning, which limits their effectiveness in user modeling
several studies [15
[8]. Recently, several discriminative methods [2] 38) have applied
contrastive learning to pre-train the user model. They assume that different views of the same behavior sequence constructed via data augmentation are semantically similar and take them as the positive sample. In contrast, we propose self-supervised ranking task that avoids directly maximizing the similarity between the augmented views and alleviates the requirement of their semantic consistency.
Data Augmentation for User Behavior Sequences In the domain of user modeling and sequential recommendation, numerous data augmentation methods (2. g 38 47 have been proposed for user behavior sequences, which can be broadly categorized into two groups: explicit and implicit. Explicit augmentation methods, such as masking and cropping (27 47 are performed at the data level, meaning that they directly modify the behavior sequence. On the other hand, implicit augmentation is performed the model level. The same sequence is input into the model twice with different inde- pendently sampled dropout masks, which adds distinct noise in the feature space R 32). Our method utilizes both implicitly and explicitly augmented views and trains the user model capture their similarity order. Moreover, owing to the augmentation-adapiie fusion mechanism, our AdaptSSR can adapt to diverse data augmentation methods and consistently improve their effectiveness.
3 Methodology
In this section, we introduce the details of our Augmentation-Adapeee Self-Supervieed Ranking (AdaptSSR) task for user model pre-training. Fig. iillustratss the framework of AdaptSSR and the pseudo-codes for the entire pre-training procedure are provided in Appendix B
3.1 Notations and Problem Statement
Suppose that we have collected large number of user behavior data from source domain, e.g., an online news platform or an e-commerce platform. Let 14 and I denote the set of users and different behaviors, where cach behavior x E I can be news article an item. It should be noted
3

---

Behavior
Explicit Data Augmentation
Implicit Data Augmentation
User Representations
Similarity Estimation
2
Sequenee S
X1
User Model
X2
View I ($)
uf
Pull Closer
i
X3
Push Away Similarity Contour Selected Pairs
Timbine
X4
View 2 (5
X5
i
14
1
X3
X3
X2 X3 X
X4
Representations of Other Users
Multiple Pairwise Ranking with In-batch Hard Negative Sampling
Mask
Crop
Reorder
Figure 2: The framework of AdaptSSR. A sequence with five user behaviors is used for illustration.
that methodology aligns with the prevalent settings in many previous works 2 real-world scenarios, where each behavior is simply represented by an ID. We refrain from using any ancillary information related user behaviors (such images text descriptions). For each user u E Ut we organize his/her behaviors in chronological order and denote the behavior sequence as S (2,,,.,,.... ...,,,..................... ,..,,, Fnn (r, E I). Our goal is pre-train user model based on these behavior sequences. The output of the user model is dense user representation u E R reflecting the general characteristics of the user, where d is the embedding dimension. For each downstream task, a simple MLP is added and the entire model is fine-tuned with the labeled data of the target task.
g
22 and
3.2
Self-Supervised Ranking
As it is hard to perform semantically consistent augmentation for the user behavior sequence, avoid directly maximizing the agreement between the augmented views and propose a self-supervised ranking task that trains the user model to capture the similarity order between the implicitly augmented view, the explicitly augmented view, and views from others. Specifically, given a behavior sequence S, we first apply the implicit data augmentation, i.e., the same sequence is input into the user model twice with different independently sampled dropout masks, and denote the generated user representations as and u ut Then, apply certain explicit data augmentation to S and denote the augmented sequence and its corresponding user representation S and é, respectively. Since differs from u+ only because of the distinct dropout masks applied by the implicit augmentation, while the difference between u and û caused by both the implicit augmentation and the explicit augmentation which modifies S' at the data level, we require them to satisfy the following similarity order: sim u. u+) b sim(u, (u, ú). Considering that the semantic consistency between S and S cannot be guaranteed, our pre-training objective to capture the following similarity order:
T: sim(u, ut) > sim u. ù)) > sim(u, 2
where u the user representation of another user. Instead of directly forcing the user model to maximize sim(u, ú) existing contrastive learning-basee methods, objective leaves room for the user model to adjust sim (u, û) properly between the upper bound sim (u, u u²) ) and the lower bound sim(u, based on the similarity between the augmented views, which alleviates the requirement of their semantic consistency.
3.3 Implementation
User Model Our method is decoupled from the structure of the user model, allowing for the integra-
tion of various existing methods, such the attention network (58) and the temporal convolutional network [5T]. In this work, we choose the widely used Transformer Encoder (TrmEnc) [40] as
the backbone of the user model due to its superior performance in sequence modeling 20 37
Given a user behavior sequence 5 = f#1. T2...., ), each behavior J; E I is projected into a dense vector x; E Rd via looking-up an embedding table E I Zd Another learnable position embedding matrix P [p, PP..P...... ... P..l nxd is used model the position information. An
attention network with learnable query vector q a is further utilized to aggregate the output

---

hidden states H of the Transformer Encoder. Thus, the user representation is calculated as follows:
hi, h2 ......... hn] = (rmEnn((22) (21 + P1:02 Pa....::
:.: En + pr.).
(2)
exp(q h,)
a, 2==1 j= exp(q hz)
(3)
u a,h
(4)
Multiple Pairwise Ranking with In-batch Hard Negative Sampling Given a batch of user behavior sequences {S.) where B is the batch size, we first apply two randomly selected explicit augmentation methods to each S; and denote the augmented sequence St and S, In this work, adopt the three random augmentation operators proposed by Xie et al. [47]: mask, crop, and reorder. Notably, our method can be integrated with various existing data augmentation methods, which will be further explored in Section 4.3 Then, for pair of Š, and S,, we input each of them into the user model twice with independently sampled dropout mask and denote the generated user representations and ü; u,"" respectively. It should be noted that we only utilize the standard dropout mask in the Transformer Encoder, which applied both the attention probabilities and the output of each sub-layer. We treat all representations generated from views of other users in the same batch as negative samples, which are denoted as U,- = {ūg, úJ &gj ; 1BB 1j=1,itti Inspired by previous works learning to rank, we adopt Multiple Pairwise Ranking (MPR) loss [48] to train the user model to capture the similarity order r, which extends the BPR loss (34] to learn two pairwise ranking orders
as ú, u."
simultaneously. For the user representation ú; and ú,"" each E {ū; u, } and w U, form quadruple for model training, and the loss function for S, is formulated follows:
c.
Σ
Σ
logo [} (s, ú.))) simm(,, v))
2
(E(ú,, wEU,
(5)
+ (1- A) (sim(ū, v) sim w))].
where (-) is the Sigmoid function and sim(-) calculates the cosine similarity between two vectors. E [0, 1] is a trade-off hyper- parameter for fusing the two pairwise ranking orders: sim ú,)) >
sim ú. v) and sim(ú, v) 2 sim (ú, w).
In Eq. (5). each quadruple is treated equally. Inspired by recent studies [ 44 55 on the impact of hard negative samples for contrastive learning, further design in-batch hard negative sampling strategy to facilitate model training. For each pairwise ranking order, we select the pair with the smallest similarity difference for training, i.c., the loss function formulated as follows:
4. log [
(sim( u,tt max sim(
v
(6)
min sim ( v) max sim(ú; w) ] vE(U.,
+ (1 A1-
wEU,"
Such pairs are currently the closest samples that the model needs to discriminate, which can provide the most significant gradient information during training. For another augmented sequence S, the loss function É, is symmetrically defined and the overall loss computed as = Σi (C:+2.)/2B.
With the loss function L, the user model is trained to capture two pairwise ranking orders: sim(ú, ) > sim(û, v) and sim (ú, v) 3> sim i.,, w) simultancously. However, we further need to combine them properly via the hyper-parameter X. As illustrated in Dig. Appendix A the effects of data augmentation vary significantly across different behavior sequences. Consequently, fixed and unified À is not enough to fuse the two pairwise ranking orders properly for different samples. Thus, we further develop automatically adjust the similarity order constraint applied to each sample based on the similarity between the augmented views. Specifically, we estimate the semantic similarity between the explicitly augmented views with the average similarity between the user representations generated from Š, and
Ingmentation-Adaplee Fusion
iigmentation-adapplee adaptive fusion mechanism
5

---

S, Then we replace the hyper-paramene parameter À in Equation (6 with dynamic coefficient X; for each training sample S, which calculated along the training procedure as follows:
A =
Σ Σ
max 8),0).
(7)
_E( )E(
To get an accurate similarity estimation at the beginning of the training, we train the user model with the MLM task [52] until convergence before applying our self-supervised ranking task. If S; and S, are semantically similar, A will be small and set the loss function L, focusing on maximizing the latter term min uē ( ".."" sim (., v) max sim(ú, w), which forces the user model to discriminate these similar explicitly augmented views from views of other users. Otherwise, A will be large and train the user model to pull the implicitly augmented view and these dissimilar explicitly augmented views apart. As a result, the user model is trained to adaptively adjust sim (ū; v) when combining the two learned pairwise ranking orders for each sample, which better deal with the distinct impacts of data augmentation on different behavior sequences.
3.4 Discussion
In this subsection, we discuss the connection between our proposed self-supervised ranking task and existing contrastive learning-basee methods.
If we set X = 0 for all training samples and input each behavior sequence into the user model once (i.e., do not apply the implicit data augmentation), our loss function L, degenerates as follows:
3 = log o sim ú, úز} max sim(úi; w)
wEV,
(8)
exp (sim ú;, ú,])
log
exp (sim (., úz)) + max wev exp (sim (6,,
where V, = (úz, 1 Most existing contrastive sauring-bbsser pre-training methods 2 32
38 adopt the InfoNCE loss 30 to train the user model, which can be formulated follows:
exp (sim (#;; ū;))
LInfONCE
log
exp (sim (ū;; úgy) Ewev; exp (sim (ū,, u))
Both loss functions aim to maximize the agreement between the augmented views. The sole distinction that C; selects the hardest in-batch negative sample, which is most similar the anchor ú, whereas LInfaNCE uses the entire negative sample set VT. Thus, with the same data augmentation method, the degenerated version of AdaptSSR is equivalent to combining contrastive learning- based methods with hard negative sampling, which has been proven to be effective by several recent studies
When A 0, the former term sim (ū;; ú,)) max simm(. v) in 2, forces the user model
capture the similarity order sim( (., a,t) sim(ú;, v) as well. In addition, our augmentation- adaptive fusion mechanism automatically adjusts the similarity order constraint applied to each sample. As result, our method alleviates the requirement of semantic consistency between the augmented views and can adapt to various data augmentation methods.
Experiments
4.1 Experimental Setup
We conduct experiments on two real-world datasets encompassing six downstream tasks. The first dataset, the Tencent Transfer Learning (TTL) dataset, was released by Yuan al. (52) and contains users' recent 100 interactions downstream labeled data of two user profiling tasks: age prediction (Ti) and life status prediction (72), and two cold-recommendatioo tasks: click recommendation (Tx) and thumb-up recommendation (Ta). The second dataset, the App dataset, consists of users' app installation behaviors collected by worldwide smartphone manufacturer, OPPO, from 2022-12 12 to 2023-03. Each user has been securely
Datasets
the QQ Browser platform. Additionally, it provides the

---

Table 1: Detailed statistics of each dataset and downstream task.
Dataset
TTI.
App
Behavior Sequences
1,470.149 645,972
1.575,837
# Different Behaviors
4,047
Avg. Sequence Length
54.84
44.13
Downstream Task
Ti
Ts 1,397.197
T4
Ts
Samples
1,470,147
1,020,277
255,646
1,178,603 2
564,940
Labels/Items
8
17,879
7,539
hashed into an anonymized to prevent privacy leakage. It also provides the downstream labeled data of gender prediction task (75), and CVR prediction task (T6). which predicts whether the user will install a target app. Detailed statistics each dataset and downstream task are listed Table
Evaluation Protocols For model pre-training, 90% user behavior sequences randomly selected for training, while the rest 10% are used for validation. For each downstream task, we randomly split the dataset by 6:2:2 for training, validation, and testing. To evaluate model performance various downstream tasks, we use classification accuracy (Ace) for multi-class classification tasks (71, T2), NDCG@ 10 for cold-recommeenaation tasks (Ta, T4), and AUC for binary classification tasks (7s, Te). To avoid the bias brought by sampled evaluation [21), we use the all-ranking protocol for the cold-recommendation task evaluation, i.e., all items not interacted by the are used candidates. We repeat each experiment 5 times and report the average results with standard deviations.
Baselines To validate the effectiveness of AdaptSSR, compare it with three sets of baselines. The first set contains several generative pre-training methods, including:
PeterRec [52] applies the MLM task [10 to the behavior sequence for user model pre-training.
PTUM 45 pre-trains the user model with two self-supervised tasks: masked behavior prediction and next behavior prediction.
The second set contains several discriminative pre-training methods, including:
CLUE 8
solely uses implicitly augmented views for contrastive pre-training.
CCL proposes mask-and-fil strategy to construct high-quality augmented views with data generator trained with the MLM task for contrastive pre-training.
IICL
38 takes different time periods of the same behavior sequence augmented views for
contrastive pre-training. An interest dictionary is used to extract multiple interests of the user.
The last set contains several data augmentation methods for user behavior sequences, including:
CL4SRec 47 augments the behavior sequence by masking, cropping, and reordering.
takes item correlation into consideration and improves CL4SRec with two more informative augmentation operators: substitute and insert.
CoSeRec 27
DuoRec 32 takes sequences with the same last behavior semantically similar positive samples.
We apply them to generate positive samples for the contrastive pre-training task, We also evaluate the performance of the model trained on the downstream task in an end-to-end manner. All these baselines share the same model structure with our AdaptSSR and only differ in the pre-training task.
Implementation and Typer-parameters Following previous works 2 dimension d as 64. the Transformer Encoder, the number of attention heads and layers are both set as 2. The dropout probability is set as 0.1. The data augmentation proportion p for each baseline method is either searched from (0.1, 0.2, ........ ... 0.9} or set as the default value in the original paper provided. The batch size and learning rate set as 128 and 2c-4 for both pre-training and fine-tuning, More implementation details are listed in Appendix
set the embedding
4.2 Overall Performance of Downstream Tasks
The experimental results on various downstream tasks are presented in Table From the results, we have several findings. First, generative pre-training methods (PeterRec and PTUM) underperform compared to most discriminative pre-training methods. This is because these methods mainly focus
7

---

Table 2: Performance (%) of various pre-training methods on downstream tasks. Impr (%) indicates the relative improvement compared with the end-to-end training. The best results are bolded.
Pre-train
T
T2
Ti
T4
Ts
To
Method
Ace
Impr
Acc
Impr NDCG@10
10
Impr NDCG@
10 Impr
AUC
Impr
AUC
Impr
None
62,87:0005
52.24=0.16
1.99=0.03
2.87=0007
78.63=0.06
75.14
PeterRee
63.6240.11
1.19 53. 03.14_0.07
1.72
2.37=0.02
19.10
3.06
6.62
19.61
1.25
76.04
1.20
PTUM
63,21+0.14
0,54 53.05=0.04
1.55
2.29::003
15.08
2.96:0.03
3.14
79,48+0.11
1,08
75,82=0013
0.90
CLUB
63.38+0.10
0.81 53.23-0.05
1.90
2.38=0002
19.60
3.05-0.21
6.27
79.90=0006
1.62
76.03=0.16
1.18
CCL
63.7610.11
1.42 53.37=0.09
2.16
2.43=0.02
22.11
3.32
15.68
80.22:+0.07
2.02
77.35=0.10
2.94
IDICL
63,88200.4
1.61 53,45=0.05
2.32
2.46=00.2
23.62
3,42=0.04
19.16
80,34:0.05 2.17 77 77.92=0.08
3.70
CL4SRec
63.7110.14
1.34 53.43_0.05
2.28
2.40
21.11
3.29
14.63
10.10
1.92
77.02=0.05 2.50
CoSeRec
63.8920.03
1.62
53,53=0.09
2.47
2.44=0.02
22.61
3.33=0.05
16.03
80.48:00066
2.35
77.71=0.09 3.42
DuoRee
63.50++..9
1.00 53.26-0006
1.95
2.39:0.01
20.10
3.11-0.16
8.36
80.03++0.09
1.78
76.85=00.9
2.28
AdaptSSR
65.53:10.04 4.23 0.02
4.15
2.61
31.16
3.73
29.97 0..30 4.67 79.92:0.05 6.36
on mining the correlation between behaviors while lacking careful design for user representation learning, which limits their performance on downstream tasks. Second, discriminative pre-training methods with explicit data augmentation (e.g., CCL, CL4SRec, CoSeRec) generally outperform the method relying solely on implicit data augmentation (CLUE). We argue that this because the implicit data augmentation caused by the dropout mask alone is too weak. The user model can easily distinguish the positive sample from others, thus providing limited knowledge for downstream tasks. Third, our AdaptSSR consistently surpasses previous SOTA contrastive learning- based pre-training methods by 2.6%, 1.7%, 6.1%. 1.% 9.1%. 2.3%, and 2.6% on each downstream task respectively. and our further t-test results show the improvements are significant at p < 0.01. This is because we train the user model to capture the similarity order r, rather than directly maximizing the similarity between the explicitly augmented views. Such ranking task alleviates the requirement of semantic consistency while maintaining the discriminability of the pre-trained user model.
4.3 Performance with Different Data Augmentation Methods
As our method alleviates the requirement of semantic consistency between the augmented views and can adapt to a variety of data augmentation methods, we further combine with several existing pre-training methods: CL4SRec, CoSeRec, and CCL, by replacing the contrastive learning (CL. task with our AdaptSSR while maintaining their data augmen- tation methods. We vary the augmentation propor-
66.0
65.5
tion p from 0.1 to 0.9 and evaluate the performance of these methods on the downstream age prediction task (Ti). The results on other downstream tasks
65.0
CL4SRee
No Pre-tmin
CoSeRee
w/ CL
64.5
CCL
w/ AdaptSSR
show similar trends and are included in Appendix D
Accuracy
64.0
From the results shown in Fig. B we find that these contrastive learning- based methods are quite sensi- tive to the data augmentation proportion. When p is close to O, the user model can easily discriminate these weakly augmented positive samples from oth- ers during pre-training and thus brings limited per- formance gain to the downstream task. However, the stronger the augmentation is, the more likely it is to generate dissimilar augmented views and may even cause a negative transfer to the downstream task. In contrast, our AdaptSSR significantly improves the performance of all these pre-training methods with similar level. This is because our self-supervised ranking task takes the potential semantic inconsistency between the augmented views into account and avoids directly maximizing their similarity. addition, our ceamentatiionadaaplee properly combine the learned pairwise ranking orders based on the estimated similarity between the explicitly augmented views constructed by various augmentation methods with different strengths, which leads to similar model performance.
63.5
63.0
62.5
62.0
0.1
0.2
0.3
0.4
0.5
0.6 0.7 0.8 0.9
Data Augmentation Proportion p
Figure 3: Effectiveness of AdaptSSR when combined with existing pre-training methods.
different augmentation proportions
fusion mechanism can
8

---

4.4 Ablation Study
In this subsection, we further validate the effectiveness of each component in our method, i.., in-batch hard negative sampling, sugmentation-adayy -adatiive fusion, and self-superrised ranking. We pre-train the user model with our AdaptSSR and its variant with one component removed. When the augmentation- adaptive fusion mechanism is removed, we search for the optimal À from {0, ..., O..,..... 1.0). When the self-supervised ranking task is removed, we use the contrastive learning task instead. The results of the downstream age prediction task (Ti) and thumb-up
65.5
3.8
65.0
3.7
64.5
3.6
Acouncy
recommendation task (Ta) are shown in Fig. 4 We first find that the model performance drops signifi- cantly by 2.8% on Ti and 11.8% on Ti when the self- supervised ranking task is removed. This is because it alleviates the requirement of semantic consistency be- tween the augmented views, which is the major issue that impedes existing contrastive learning-baid -based pre- training methods. Besides, the model performance also declines by 0.9%-1.66 on Ti and 3. 1%-5.1% 1% on T4 after we remove either in-batch hard negative fusion. This
64.0
3.5
63.5
3.4
AdsptSSR
63.0
3.3
wio in-hetch hard negarive sampling
62.5
MAN
wio an ougnentaiion adaplive fusion
3.2
wio self supervised tanking
62.0
3.1
Ta
Figure 4: Effectiveness of each component in our AdaptSSR.
sampling or ceaimeettttion-daapii because in- in-batch hard negative sampling helps the model focus on discriminating the most similar samples while using all quadruples may introduce extra noise. Our augmentatioon-adapive fusion mechanism adjusts the value of X based the similarity between the augmented views for each training sample, while a constant applies a fixed ranking order constraint to all samples and neglects the distinct impacts of data augmentation across different behavior sequences.
4.5 Efficiency Comparison
To compare the efficiency of different pre-training methods, the TTL dataset and the App dataset and list them in Table 3 pre-training time of our AdaptSSR similar that sorming-based methods, such
we record their average training time on Our observations indicate that the average
of existing contrastive as CL4SRec and CoSeRec. Although our AdaptSSR requires inputting each behavior sequence into the model twice, we implement it by duplicating the input sequences in the batch size dimension. Thus, we only need to input all the sequences into the model once, which can be well parallelized by the GPU. Besides, for each pairwise ranking order, our in-batch hard negative sampling strategy only selects the pair with the smallest similarity difference to compute the mul- tiple pairwise ranking loss, which avoids the costly softmax operation and cross-entropy loss calculation in the existing contrastive learning task. As result, our AdaptSSR will not greatly increase the overall computational cost while bringing significant performance improvement various downstream tasks.
Table 3: Average training time (hours) of dif- ferent pre-training methods each dataset.
Pre-train Method
TTL
App
PeterRec
2.927=0.022
1.537=0.010
PTUM
2.015110.009
2.05511.018
CLUE
1.453:10.016
1.633_0.015
IDICI.
1.257+0.021
1.162+0.020
CL4SRec
1.868:0.013
2.081=00011
CoSeRec
1.902.10.015
2.104:10.023
DuoRec
1.535+0.020
1.658=0.015
AdaptSSR
1.599+0.017
1.830+0.012
4.6 User Representation Similarity Distribution Analysis
To better reflect the difference between our AdaptSSR and other pre-training methods, we randomly select 10,000 users from the TTL dataset. For each user's behavior sequence, we augment it by masking and cropping with a proportion of 0.6. Then, we calculate the cosine similarity between the user representations generated from the original behavior sequence, the augmented behavior sequence, and the behavior sequence of other users. We use kernel density estimation (KDE) (31 to estimate their probability density distributions, which are visualized in Fig. 5 The figure reveals that when pre-training with existing contrastive learning-basse methods (e.g., CLASRec, CoSeRec), the similarity between the augmented views always gathers in very narrow range close to 1.0.

---

PTUM
CLUE
CCL
IDICL
25
25
25
25
20
20
20
Density
15
10-
10
10
10
5
5
5
5
0
-0.5
0.0
0.5
1.0
-0.5
0.0 CoSeRee
0.5
1.0
-1.0
-0.5 0.0
0.5
1.0
-1.0
-0,5
0.0 AdaptSSR
0.5
1.0
CL4SRec
DuoRee
25
25
25
20
20
Density
15
15
15
15
10
10
10
10
5
5
5
5
-0.5
0.5 Cosine Similarity
0.0
1.0
-0.5
0.0
0.5
1.0
0.4
0.6
0.8
1.0
-0.5
0.0
0.5
1.0
Cosine Similarity
Cosine Similarity
Cosine Similarity
Crop
Mask
Other users
Figure 5: Distributions the cosine similarity between user representations generated from the original behavior sequence, different augmented behavior sequences, and the behavior sequences of other users with various pre-training methods. The area under each curve equals to 1.
This is because these methods always maximize the similarity between the augmented views while neglecting their potential semantic inconsistency. Furthermore, when pre-trained with other methods (e.g., PTUM, CCL), the similarity between different users tends disperse over very broad range (from -0.5 to 1.0), which indicates that the user model cannot well distinguish the behavior sequence of different users. In contrast, the similarity between the augmented views distributes over relatively wider range (from 0.5 to 1.0) when pre-trained with our AdaptSSR, which well fits our findings about these semantically inconsistent augmented views. Meanwhile, the distribution of the similarity between different users is still well separated from that between the augmented views, which means the user model can also discriminate the behavior sequence of different users.
5 Conclusion
In this paper, we identified the semantic inconsistency problem faced by existing contrastive learning- based user model pre- pre-training methods. tackle this challenge, we proposed a new pretext task: Augmentation Addative Self- Self-SSuer Ranking (AdaptSSR). A multiple pairwise ranking loss is adopted train the user model capture the similarity order between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employed an in-batch hard negative sampling strategy to facilitate model training. Additionally, cope with the distinct impacts of data augmentation on different behavior sequences, we designed an ceaeeentation-daaptii fusion mechanism adjust the similarity order constraint applied each sample based on the estimated similarity between the augmented views. Our method can be further generalized to other domains where augmentation choices are not straightforward could alter semantics experiments validated that AdaptSSR consistently outperforms previous baselines by large margin on various downstream tasks while adapting to diverse data augmentation methods.
data. Extensive
Acknowledgements
This research partially supported by grants from the National Key Research and Development Program of China (No. 2021 YFF0001003), and the OPPO joint research program. We furthermore thank the anonymous reviewers for their constructive comments.
References
[1] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximizing Mutual Information Across Views. Advances in Neural Information Processing Systems, volume 32, pages 15535-15545, 2019.
10

---

[2] Shuqing Bian, Wayne Xin Zhao, Kun Zhou, Jing Cai, Yancheng He, Cunxiang Yin, and Ji- Rong Wen. Contrastive Curriculum Learning for Sequential User Behavior Modeling via Data Augmentation. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pages 3737-3746, 2021.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912-9924, 2020.
[4] Yukuo Cen. Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. Controllable Multi-Interes, Framework for Recommendation. In Proceedings of the 26th ACM SICKDD International Conference on Knowledge Discovery and Data Mining, pages 2942-2951, 2020.
[5] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum Disentangled Recommendation with Noisy Multi-feedbbckk. In Advances in Neural Information Processing Systems, volume 34, pages
2021.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[7] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[8] Mingyue Cheng, Fajie Yuan, Qi Liu, Xin Xin, and Enhong Chen. Learning Transferable User Representations with Sequential Behaviors via Contrastive Pre-training. In IEEE 2 st International Conference on Data Mining, pages 51-60, 2021.
[9] Paul Covington, Jay Adams, and Emre Sargin. Deep Neural Networks for YouTube Recommen- dations. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 191-198, 2016.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186, 2019.
[11] Bi'an Du, Xiang Gao, Wei Hu, and Xin Li. Self-Contrastive Learning with Hard Negative Sam- pling for Self-Supervised Point Cloud Learning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3133-33 233-3142,22 2021.
[12] Amir Feder, Guy Horowitz, Yoav Wald, Roi Reichart, and Nir Rosenfeld. In the Eye of the Beholder: Robust Prediction with Causal User Modeling. In Advances in Neural Information Processing Systems, volume 35, pages 14419-14433, 2022.
[13] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, 2021.
[14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap Your Own Latent New Approach to Self- Sulf-Supervieed Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284, 2020.
[15] Jie Gu, Feng Wang, Qinghui Sun, Zhiquan Ye, Xiaoxiao Xu, Jingmin Chen, and Jun Zhang. Exploiting Behavioral Consistence for Universal User Representation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4063-4071, 2021.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 9726-9735, 2020.
11

---

[17] Ruining He and Julian McAuley. Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation. In IEEE 16th International Conference on Data Mining, pages 191-200, 2016.
[18] Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards Universal Sequence Representation Learning for Recommender Systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 585-593, 2022.
[19] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard Negative Mixing for Contrastive Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21798-21809, 2020.
[20] Wang- Maang-Chengaa Kang and Julian McAuley. Self-Attentive Sequential Recommendation. In IEEE 18th International Conference on Data Mining, pages 197-206, 2018
[21] Walid Krichene and Steffen Rendle. On Sampled Metrics for Item Recommendation. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1748-1757, 2020.
[22] Shangsong Liang. Collaborative, Dynamic and Diversified User Profiling. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4269-4776 2019.
[23] Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng. Tao Qin, Wei Chen, Min Zhang, and Tie- Liu. R-Drop: Regularized Dropout for Neural Networks. In Advances in Neural Information Processing Systems, volume 34, pages 10890-10905, 2021.
[24] Hu Liu, Jing Lu, Xiwei Zhao, Sulong Xu, Hao Peng, Yutong Liu, Zehua Zhang, Jian Li, Junsheng Jin, Yongjun Bao, and Weipeng Yan. Kalman Filtering Attention for User Behavior Modeling in CTR Prediction. In Advances in Neural Information Processing Systems, volume 33, pages 9228 9238, 2020.
[25] Zheng Liu, Yu Xing, Fangzhao Wu, Mingxiao An, and Xing Xie. Hi-Fi Ark: Deep User Representation via High-Fidelity Archive Network. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pages 3059-3065, 2019.
[26] Zheng Liu, Jianxun Lian, Junhan Yang, Defu Lian, and Xing Xie. Octopus: Comprehensive and Elastic User Representation for the Generation of Recommendation Candidates. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 289-298, 2020.
(27] Zhiwei Liu, Yongjun Chen, Jia Li, Philip S. Yu, Julian McAuley, and Caiming Xiong. Con- trastive Self-supervised Sequential Recommendation with Robust Augmentation. arXiv preprint arXiv: 2108.06479, 2021.
[28] Lajanugen Logeswaran and Honglak Lee. An Efficient Framework for Learning Sentence Representations. In International Conference on Learning Representations, 2018.
[29] Ishan Misra and Laurens van der Maaten. Self-Supervised Learning of Invaraatn Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv: 1807.03748, 2018.
[31] Emanuel Parzen. On Estimation of a Probability Density Function and Mode. The Annals of Mathematical Statistics, 33(3): 1065-1066 1962.
[32] Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation. In Proceedings of the 15th ACM International Conference Web Search and Data Mining, pages 813 823, 2022.
[33] Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-BERT: Pre-training User Represen- tations for Improved Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4320-432) 2021.
12

---

[34] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Schmidt-Thieme, BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, pages 452-461, 2009.
[35] Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive Learn- ing with Hard Negative Samples. In International Conference on Learning Representations, 2021.
[36] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 161--170, 2019.
[37] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1441-1450 2019.
[38] Qinghui Sun, Jie Gu, XiaoXiao Xu, Renjun Xu, Ke Liu, Bei Yang. Hong Liu, and Huan Xu. Learning antrreet-Oriented Universal User Representation via Self-Supervision. In Proceedings of the 30th ACM International Conference Multimedia, pages 7270-7278, 2022.
[39] Jie Tang, Limin Yao, Duo Zhang, and Jing Zhang. A Combination Approach to Web User Profiling, ACM Transactions on Knowledge Discovery from Data, 5(1):1-44, 2010.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008 2017.
[41] Petar Veličković, William Fedus, William L. Hamilton, Pietro Lió, Yoshua Bengio, and Devon Hjelm. Deep Graph Infomax. In International Conference Learning Representations, 2019.
[42] Feng Wang and Huaping Liu. Understanding the Behaviour of Contrastive Loss. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 2495-2504, 2021.
[43] Qizhou Wang, Jiangchao Yao, Chen Gong, Tongliang Liu, Mingming Gong, Hongxia Yang, and Bo Han. Learning with Group Noise. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11):10192-10200 2021.
[44] Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance Wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14020-14029, 2021.
[45] Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. PTUM: Pre-training User Model from Unlabeled User Behaviors via sulervupervision. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1939-1944, 2020.
[46] Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. UserBERT: Pre- Training User Model with Contrastive Self- Super-Suerviiinn In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2087-2092, 2022.
[47] Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. Contrastive Learning for Sequential Recommendation. In IEEE 38th International Conference on Data Engineering, pages 129-11273 2022.
[48] Runlong Yu, Yunzhou Zhang, Yuyang Ye, Le Wu, Chao Wang, Qi Liu, and Enhong Chen. Mul- tiple Pairwise Ranking with Implicit Feedback. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1727-1730, 2018
[49] Wenhui Yu and Zheng Qin. Sampler Design for Implicit Feedback Data by Noisy-label Robust Learning. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 861-870, 2020.
13

---

[50] Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5489, 2022.
[51] Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose, and Xiangnan He. A Simple Convolutional Generative Network for Next Item Recommendation. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining, pages 582-590, 2019.
[52] Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. -arameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1469-1478, 2020.
[53] Kai Zhang, Hao Qian, Qing Cui, Qi Liu, Longfei Li, Jun Zhou, Jianhui Ma, and Enhong Chen. Attention Network for Arne-Grained Feature Learning CTR Prediction. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 984-992, 2021.
[54] Kai Zhang, Qi Liu, Hao Qian, Biao Xiang, Qing Cui, Jun Zhou, and Enhong Chen, EATN: An Efficient Adaptive Transfer Network for Aspect-Level Sentiment Analysis. IEEE Transactions on Knowledge and Data Engineering. 35(1):377-335 2023.
[55] Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang. Xiaokang Yang. and Pinyan Lu. M-Mix: Generating Hard Negatives via Multi-Sample Mixing for Contrastive Learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2461-2470, 2022.
[56] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based Graph Self- Self-SSuer Learning for Molecular Property Prediction. In Advances in Neural Information Processing Systems, volume 34, pages 15870- (5001, 2021.
[57] Lei Zheng, Chaozhuo Li, Chun-Ta Lu, Jiawei Zhang, and Philip S. Yu. Deep Distribution Network: Addressing the Data Sparsity Issue for Top-N Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1081-1084, 2019.
[58] Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. ATRank: An Attention Based User Behavior Modeling Framework for Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence,
4571, 2018.
[59] Guorui Zhou, Xiaoqiang Zhu, Chenru Song. Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep Interest Network for Click-Three Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1059-1066 2018.
[60] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep Interest Evolution Network for Click-through Rate Prediction. Proceedings of the AAAI Conference Artificial Intelligence, 33(1):5941-5945 2019.
14

---

A
More Examples of the Semantic Inconsistency Problem
Original
Behavior
Sequence
Mask
MASK]
[MASK]
[MASK]
[MASK]
[AAKK]
Crop
World
Organiation
Mask-and-Fill
Substitute
Coronavirus
Politics
Noisy behavior
(a) User A
Original
Behavior
Sequence
Mask
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
Crop
Mask-and-Fill
Substitute
Coronavirus
Politics
Basketball
Entertainment
Food
Noisy behavior
(b) User
Figure 6: Illustrations of the impact of various data augmentation methods on different user behavior sequences. Each picture represents news article clicked by the user. Pictures with the same color border reflect similar interests of the user. Dash borders are used to indicate behaviors replaced by the data augmentation method.
We randomly sampled impact of various data augmentation methods on different user behavior sequences and the consequent semantic inconsistency problem. The users' recent eight behaviors and the results of several data augmentation methods are shown in Fig. 6 The data augmentation proportion is set as 0.6. From the original behavior sequence user A, may infer that the user is quite concerned about coronavirus and politics. We also find that the behavior sequence augmented by masking well preserves the user's s interests. However, the two sequences augmented by cropping contain completely different interests of the user over time. Meanwhile, some behaviors replaced by maakk-nd-fill and substitute are noisy, which the user may not be interested in. Similarly, the original behavior sequence of user shows the user's potential interests in coronavirus, politics, basketball, entertainment, and food. As result, augmentation methods such as mask and crop are likely to lose certain characteristics the user, while augmentation methods such mask-and and-fil and substitute tend to bring in noisy behaviors. In
anony mous users from the MSN online platform to illustrate the
15

---

fact, it is hard to determine what are the real interests of user B from such a diverse behavior sequence since some behaviors may be the result misclicking click-baiting, thus making even harder to provide semantically consistent augmentation. From these examples, we can find that existing data augmentation methods may fail to preserve the characteristics interests the behavior sequence and cannot guarantee semantic consistency between the augmented views. Thus, directly forcing the user model to maximize the agreement between the augmented sequences may result in a negative transfer for downstream tasks.
B
The Algorithm of AdaptSSR
The pseudo-codes of the pre-training procedure with our AdaptSSR are shown Algorithm)
Algorithm Pre-training Procedure with AdaptSSR
Input: A corpus user behavior sequences S and a set of data augmentation operators A.
Output: The pre-trained user model M.
1: Train the user model .M with the MLM task on S until convergence.
2: while not converged do
3: Randomly sample batch of user behavior sequences {5:) from S.
4: for each S; do
5: Randomly select two augmentation operators f and from A.
6: ú; ut + M(S((S))) +
>> With independently sampled dropout masks.
7:
úi ut +
> With independently sampled dropout masks.
8:
U, + {(ū,, at uj ú BB
9:
A <- 1- -=EEE Eac(anat) max(sim(3,5).
10:
É, + - log A (sim(tic, ait) - max velū, sim(ie;; m))
+ (1 - X1) (min ( sim(u; v) max wau, sim (i;, w))
É, + - log a A (sim((i,, att) - max oe sim( v))
11:
+ Nr) (minoc(araa) sim(ū; v) wax sim ūi, w)
end for
13: L C- - (c.+2.)/2n.
Update the parameters of the user model M with & by backpropagation.
15: end while
C Implementation Details
In our experiments, the embedding dimension is set as 64, In the Transformer Encoder, the number of attention heads and layers both set 2. We only utilize the standard dropout mask, which is applied to both the attention probabilities and the output of each sub-layer, and the dropout probability 100 and 256 for the TTL dataset and the App
set 0. The maximum sequence length is dataset, respectively. For each downstream task, a two-layer MIP is added to the pre-trained user model, and the dimension of the intermediate layer is also set 64.
For existing pre-training methods, the data augmentation proportion p is either copied from previous works if provided or searched from {0.1, ... 0.9]. In our AdaptSSR, we adopt three random augmentation operators by default: mask, crop, and reorder. The augmentation proportion set to 0.6, 0.4, and 0.6 respectively while experimental results in Fig, have shown that our method is robust to various data augmentation methods with different strengths.
We use the Adam optimizer for model training. The batch size and learning rate are set as 128 and 2e-4 for both pre-training and fine-tuning. The test results for the models are reported at their best validation epoch. We repeat each experiment 5 times with different random seeds and report
16

---

3.6
79
NDCO@10 (%)
3.3
(%)
77
75
2.7
CL4SRec
No Pre-train
73
CL4SRec
No Pre-train
CoSeRec
CL
CuSeRee
w/ CL AdaptSSR
CCL
AdapiSSR
CCL
2.4
71
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
0.1
0.2
0.3
0.4
0.5 0.6
0.7 0.8 0.9
Data Augmentation Proportion
Data Augmentation Proportion p
Figure 7: The effectiveness of AdaptSSR when combined with existing pre-training methods on the downstream thumb-up recommendation task (left) and CVR prediction task (right).
the average results with standard deviations. We implement all experiments with Python 3.8.13 and Pytorch 1.12.1 NVIDIA Tesla V100 GPU.
D More Experimental Results
Fig. 7 shows the performance of our AdaptSSR when combined with several existing pre-training methods: CL4SRec, CoSeRec, and CCL, on the downstream thumb-up recommendation task (7a) and CVR prediction task (Ts) with the data augmentation proportion varying from 0,1 to 0.9. Similar to the results on the age prediction task (Ti), we find that these contrastive learning- based methods are highly sensitive to the data augmentation proportion. A too-small or too-large value of will lead to limited performance gain or even negative transfer for the downstream task. Contrarily, our AdaptSSR consistently boosts the effectiveness of these pre-training methods by large margin. This is because our sule-supe ranking task avoids directly maximizing the similarity between the augmented views. Moreover, our AdaptSSR also substantially improves the robustness of these pre-training methods to the augmentation proportion, which verifies the effectiveness of our geggmentationadaapiiii fusion mechanism. The dynamic coefficient A; adjusts the similarity order constraint applied each sample based on the similarity between the augmented views, thus empowering the pre-training methods to adapt diverse data augmentation methods with varying strengths.
E Limitations
In this work, after pre-training the user model with our AdaptSSR, the entire model is fine-tuned with the downstream labeled data. However, fine-tuning such large model for each downstream task can be time and space-consuming. A potential solution for this problem is Parameter--Eiin Efficient Fine- Tuning (PEFT). Several methods such as Adapter and LoRA have been demonstrated be effective for adapting the large language model to various downstream tasks by only updating small fraction of parameters. However, we empirically find that these methods perform poorly when applied the pre-trained user model. A possible reason is that the currently widely used Transformer- beasoomee--beeddddddd user model much shallower and thinner than the large language model. Most of the model parameters belong to the bottom behavior embedding table, while the upper Transformer blocks only contain very few parameters. Since existing PEFT methods mainly focus on tuning parts parameters in each Transformer block, only very small fraction of parameters in total will be updated, which may not enough to effectively adapt the user model to the downstream task. We will investigate how to transfer the are-trained user model to various downstream tasks parameter-efficienll while maintaining the performance gain brought by AdaptSSR in our future work.
17


Reviews:
[{"review_id": "bqKwSYvGON", "reviewer": "Reviewer_oeWK", "summary": "The authors tackle the problem of doing self-supervised learning for user modeling.  Inspired by the successes of contrastive learning approaches in the image setting, they adapt contrastive learning to the user modeling setting.  However, in user modeling the augmentations typically used are not very suitable for contrastive learning because they can change the semantics of the data, thus forcing similarity between augmented views can be problematic.  They instead produce three views: the anchor, a similar \"implicitly\" augmented view, and a less similar \"explicitly\" augmented view.  The implicitly augmented view is trained to be more similar to the anchor than the explicitly augmented view.  This escapes the problematic similarity training that plain contrastive learning would have in user modeling.", "questions": "Questions are listed in the weaknesses.", "limitations": "Seems sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper was well-written and the diagrams were easy to understand.  It made the paper easy to read and review.  The contributions were clearly stated and explained in the paper.\n\n2.  The method is novel and original as far as I know.  This method could be generalizable to other domains where augmentation choices are not straightforward and could alter the semantics of the data.\n\n3.  The method is well-designed: the ranking loss does help mitigate the \"make semantically different augmented views the same\" problem, and furthermore helps balance the focus of the loss between the implicit vs explicit contrast and the explicit vs other user contrast.  \n\n4.  The improvements in the empirical results are consistent and seem to be significant.  \n", "weaknesses": "1.  Going back to the example where the user behavior is represented by a sequence of images, is it possible to just do per-image augmentation (choices for these exist and are widely used) and then perform a typical InfoNCE style contrastive loss on the user embeddings?  For text one could do something similar using masking augmentations and such.  I did not see a comparison to this baseline and I wonder how well it would perform.  I think this is something that would be critical to compare against.\n\n2.  While the paper is written well, I think the paper should define what user modeling is and what the downstream tasks are earlier in the paper (or in the abstract).  For a while it was not clear to me what problem the paper was trying to solve, as someone who has not worked on user modeling.\n\n3.  Adding error bars into the results tables would help in understanding the significance of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your careful reading and constructive comments. We also very much appreciate your acknowledgment that our proposed method is novel and well-designed. Following are our detailed responses to your comments.\n\n**Weakness 1**: Going back to the example where the user behavior is represented by a sequence of images, is it possible to just do per-image augmentation and then perform a typical InfoNCE style contrastive loss on the user embeddings? For text, one could do something similar using masking augmentations and such. I did not see a comparison to this baseline and I wonder how well it would perform. I think this is something that would be critical to compare against.\n\n**Response**: Thanks for your insightful comment. Integrating multimodal information related to user behaviors into user model pre-training is an interesting and promising direction. However, it is essential to clarify that the images depicted in Figure 1 are just used as illustrations of what each news article clicked by the user is about, so the readers can understand the impact of different augmentation methods on the behavior sequence more intuitively.\n\nIn this work, our methodology aligns with the prevalent settings in many previous works [1-4] and real-world scenarios, where each behavior is simply represented by an ID. We refrain from using any ancillary information related to user behaviors (such as the image or text description), so we cannot perform per-image or per-text augmentation. Sorry for the unclear description. We will provide a more explicit explanation of the images presented in Figure 1 in the revised version of our paper.\n\n[1] Fajie Yuan et al. Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. SIGIR 2020. \\\n[2] Mingyue Cheng et al. Learning Transferable User Representations with Sequential Behaviors via Contrastive Pre-training. ICDM 2021. \\\n[3] Xu Xie et al. Contrastive Learning for Sequential Recommendation. ICDE 2022. \\\n[4] Shuqing Bian et al. Contrastive Curriculum Learning for Sequential User Behavior Modeling via Data Augmentation. CIKM 2021.  \n\n**Weakness 2**: While the paper is written well, I think the paper should define what user modeling is and what the downstream tasks are earlier in the paper (or in the abstract). For a while, it was not clear to me what problem the paper was trying to solve, as someone who has not worked on user modeling.\n\n**Response**: We appreciate your constructive suggestion. Indeed, user modeling aims to capture the user's characteristics or interests for a specific user-oriented task (e.g., personalized recommendation and click-through rate prediction) and encode them into a dense representation with a user representation model. As existing supervised user modeling methods tend to suffer from the data sparsity problem, our work aims to pre-train the user representation model on massive unlabeled user behavior data, which enables it to extract users' general characteristics or interests from their historical behaviors and can be transferred to benefit various downstream tasks. We will reorganize our paper and clarify the definition in the Abstract and Introduction section. Thank you again for your constructive comment.\n\n**Weakness 3**: Adding error bars into the results tables would help in understanding the significance of the results.\n\n**Response**: Thanks for the advice. We have added the standard deviation of each experiment to Table 2 in our paper, which is shown as follows (highlight in bold). Our further t-test results show that compared with the second-best method, the improvements of our AdaptSSR are significant at $p<0.01$ on every downstream task. We will add the results to the revised version of our paper.\n\\begin{array}{c|cc|cc|cc|cc|cc|cc}\\hline\\text{Pre-train}&\\mathcal{T}_1&&\\mathcal{T}_2&&\\mathcal{T}_3&&\\mathcal{T}_4&&\\mathcal{T}_5&&\\mathcal{T}_6&&\\\\\\\\\n\\text{Method}&\\text{Acc}&\\text{Impr\\\\%}&\\text{Acc}&\\text{Impr\\\\%}&\\text{NDCG\\@10}&\\text{Impr\\\\%}&\\text{NDCG\\@10}&\\text{Impr\\\\%}&\\text{AUC}&\\text{Impr\\\\%}&\\text{AUC}&\\text{Impr\\\\%}&\\\\\\\\\\hline\\text{None}&0.6287\\bf{\\pm0.0005}&-&0.5224\\bf{\\pm0.0016}&-&0.0199\\bf{\\pm0.0003}&-&0.0287\\bf{\\pm0.0007}&-&0.7863\\bf{\\pm0.0006}&-&0.7514\\bf{\\pm0.0014}&-&\\\\\\\\\\text{PeterRec}&0.6362\\bf{\\pm0.0011}&1.19&0.5314\\bf{\\pm0.0007}&1.72&0.0237\\bf{\\pm0.0002}&19.12&0.0306\\bf{\\pm0.0008}&6.37&0.7961\\bf{\\pm0.0013}&1.25&0.7604\\bf{\\pm0.0010}&1.20&\\\\\\\\\\text{PTUM}&0.6321\\bf{\\pm0.0014}&0.54&0.5305\\bf{\\pm0.0004}&1.55&0.0229\\bf{\\pm0.0003}&14.65&0.0296\\bf{\\pm0.0003}&3.17&0.7948\\bf{\\pm0.0011}&1.08&0.7582\\bf{\\pm0.0013}&0.90&\\\\\\\\\\text{CLUE}&0.6338\\bf{\\pm0.0010}&0.81&0.5323\\bf{\\pm0.0005}&1.90&0.0238\\bf{\\pm0.0002}&19.62&0.0305\\bf{\\pm0.0021}&6.02&0.7990\\bf{\\pm0.0006}&1.62&0.7603\\bf{\\pm0.0016}&1.18&\\\\\\\\\\text{CCL}&0.6376\\bf{\\pm0.0011}&1.42&0.5337\\bf{\\pm0.0009}&2.16&0.0243\\bf{\\pm0.0002}&21.93&0.0332\\bf{\\pm0.0013}&15.74&0.8022\\bf{\\pm0.0007}&2.02&0.7735\\bf{\\pm0.0010}&2.94&\\\\\\\\\\text{IDICL}&0.6388\\bf{\\pm0.0004}&1.61&0.5345\\bf{\\pm0.0005}&2.32&0.0246\\bf{\\pm0.0002}&23.63&0.0342\\bf{\\pm0.0004}&19.01&0.8034\\bf{\\pm0.0005}&2.17&0.7792\\bf{\\pm0.0008}&3.70&\\\\\\\\\\text{CL4SRec}&0.6371\\bf{\\pm0.0014}&1.34&0.5343\\bf{\\pm0.0005}&2.28&0.0241\\bf{\\pm0.0003}&21.12&0.0329\\bf{\\pm0.0006}&14.42&0.8014\\bf{\\pm0.0008}&1.92&0.7702\\bf{\\pm0.0005}&2.50&\\\\\\\\\\text{CoSeRec}&0.6389\\bf{\\pm0.0003}&1.62&0.5353\\bf{\\pm0.0009}&2.47&0.0244\\bf{\\pm0.0002}&22.53&0.0333\\bf{\\pm0.0005}&15.77&0.8048\\bf{\\pm0.0006}&2.35&0.7771\\bf{\\pm0.0009}&3.42&\\\\\\\\\\text{DuoRec}&0.6350\\bf{\\pm0.0009}&1.00&0.5326\\bf{\\pm0.0006}&1.95&0.0239\\bf{\\pm0.0001}&20.12&0.0311\\bf{\\pm0.0016}&8.32&0.8003\\bf{\\pm0.0009}&1.78&0.7685\\bf{\\pm0.0009}&2.28&\\\\\\\\\\text{AdaptSSR}&0.6553\\bf{\\pm0.0004}&4.23&0.5441\\bf{\\pm0.0002}&4.15&0.0261\\bf{\\pm0.0003}&30.71&0.0373\\bf{\\pm0.0003}&29.77&0.8230\\bf{\\pm0.0003}&4.67&0.7992\\bf{\\pm0.0005}&6.36&\\\\\\\\\\hline\\end{array}"}, {"review_id": "VnsQ95W6ql", "reviewer": "Reviewer_P6ts", "summary": "This paper proposes Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR), a new user model pre-training paradigm, which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Conventional methods assume that different views of the same behaviour sequence constructed via data augmentation are semantically consistent, while in practice existing augmentation methods tend to lose certain interests of the user or introduce noisy interests that the user does not have. AdaptSSR addresses this issue by adopting a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the explicitly augmented views, the implicitly augmented views, and views from other users. An explicit hard negative sampling strategy and an augmentation-adaptive fusion mechanism are also introduced to facilitate model training. Extensive experiments on both public and industrial datasets verify the effectiveness of AdaptSSR.", "questions": "Only three random augmentation operators are used in the paper. Does introducing more augmentation operators help improve the performance? Do informative augmentation operations introduced by CoSeRec help improve AdaptSSR's performance?", "limitations": "The extra computational cost introduced by AdaptSSR is not analyzed in the paper, it would be useful if the authors can demonstrate the tradeoff between training time and performance for various models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach is technically sound and the empirical results validates the effectiveness of the method.\n- The paper is well-written with very clear figures.\n- Code is available which makes it easy to reproduce the results.", "weaknesses": "An important hyperparameter sensitivity analysis is missing: how does the value of $\\lambda$ affect the model performance? Compared with existing models, AdaptSSR introduces an extra SimCSE-inspired implicit augmentation approach. It remains unclear in the paper if the performance improvement is primarily due to the introduction of implicit data augmentation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We very much appreciate your positive opinions on the contribution and presentation of this paper. We also thank you for the valuable comments and our detailed responses are as follows.\n\n**Weakness 1**: An important hyperparameter sensitivity analysis is missing: how does the value of $\\lambda$ affect the model performance?\n\n**Response**: Many thanks for this good question. $\\lambda$ is a critical hyperparameter in our method since it controls how the two learned pairwise ranking orders: $sim(u,u^+)\\geq sim(u,\\hat{u})$ and $sim(u,\\hat{u})\\geq sim(u,u^-)$ are fused. As we mentioned in Line 171 of our paper, the effects of data augmentation vary significantly across diverse behavior sequences. As a result, a fixed and unified $\\lambda$ is not enough to combine the two pairwise ranking orders properly for different samples. That's why we design an augmentation-adaptive fusion mechanism that replaces the hyperparameter $\\lambda$ with a dynamic coefficient $\\lambda_i$ for each training sample $S_i$. The value of $\\lambda_i$ is calculated based on the average similarity between the user representations generated from the augmented views $\\hat{S}_i$ and $\\tilde{S}_i$ (Equation (7)) along the training procedure. As a result, we no longer need to manually set the value of $\\lambda$, and the effectiveness of our augmentation-adaptive fusion mechanism has been verified in Section 4.4. We will highlight how we replace $\\lambda$ with $\\lambda_i$ in the revised version of our paper.\n\n**Weakness 2**: Compared with existing models, AdaptSSR introduces an extra SimCSE-inspired implicit augmentation approach. It remains unclear in the paper if the performance improvement is primarily due to the introduction of implicit data augmentation.\n\n**Response**: Many thanks for pointing it out. Actually, we have taken the impact of introducing extra implicit augmentation into account and compared our AdaptSSR with CLUE [1] in our experiments. It is worth mentioning that CLUE shares the same model structure with AdaptSSR but only uses implicitly augmented views for contrastive pre-training. From the results in Table 2, we can find that our AdaptSSR consistently outperforms CLUE by a large margin on various downstream tasks. We argue that it is because the implicit augmentation caused by the dropout mask alone is too weak. The user model can easily distinguish the positive samples from others, thus providing limited knowledge for downstream tasks. Such results illustrate that introducing extra implicit augmentation is not the primary reason for performance improvement. We will highlight the comparison between AdaptSSR and CLUE in the revised version of our paper.\n\n[1] Mingyue Cheng et al. Learning Transferable User Representations with Sequential Behaviors via Contrastive Pre-training. ICDM 2021.  \n\n**Question 1**: Only three random augmentation operators are used in the paper. Does introducing more augmentation operators help improve the performance? Do informative augmentation operations introduced by CoSeRec help improve AdaptSSR's performance?\n\n**Response**: Thanks for this insightful question. As we mentioned in Line 147 of our paper, our method can be combined with various data augmentation methods. We have evaluated the performance of AdaptSSR when combining it with several existing data augmentation methods in Section 4.3, including these informative augmentation operators introduced by CoSeRec. From the results in Figure 3 (the three dashed lines on top), we can find that our method achieves similar performance when combined with different data augmentation methods. We argue that this is because our augmentation-adaptive fusion mechanism can always properly combine the learned pairwise ranking orders based on the estimated similarity between the explicitly augmented views constructed by different augmentation methods, which leads to similar model performance. We will add this analysis to the revised version of our paper. Once again, we express our appreciation for your insightful feedback.\n\n**Limitations 1**: The extra computational cost introduced by AdaptSSR is not analyzed in the paper, it would be useful if the authors can demonstrate the tradeoff between training time and performance for various models.\n\n**Response**: We extend our gratitude for your insightful suggestions. The average pre-training time of different methods on the TTL dataset and the App dataset are listed in the following table. Our observations indicate that the average pre-training time of our AdaptSSR is similar to that of existing contrastive learning-based methods, such as CL4SRec and CoSeRec. Although AdaptSSR requires inputting each behavior sequence into the model twice, we implement it by duplicating the input sequences in the batch size dimension. Thus, we only need to input all the sequences into the model once, which can be well parallelized by the GPU. Besides, for each pairwise ranking order, our hard negative sampling strategy only selects the pair with the smallest similarity difference to compute the multiple pairwise ranking loss, which avoids the costly softmax operation and cross-entropy loss calculation in the existing contrastive learning task. As a result, our AdaptSSR will not greatly increase the overall computational cost while bringing performance improvement to various downstream tasks. We will add the result and analysis to the revised version of our paper.\n\\begin{array}{ccc}\\hline\\text{Pre-train Method}&\\text{TTL}&\\text{App}\\\\\\\\\\hline\\text{PeterRec}&2.927±0.022\\text{h}&1.537±0.010\\text{h}\\\\\\\\\n\\text{PTUM}&2.015±0.009\\text{h}&2.055±0.018\\text{h}\\\\\\\\\\text{CLUE}&1.453±0.016\\text{h}&1.633±0.015\\text{h}\\\\\\\\\n\\text{IDICL}&1.257±0.021\\text{h}&1.162±0.020\\text{h}\\\\\\\\\\text{CL4SRec}&1.868±0.013\\text{h}&2.081±0.017\\text{h}\\\\\\\\\n\\text{CoSeRec}&1.902±0.015\\text{h}&2.104±0.023\\text{h}\\\\\\\\\\text{DuoRec}&1.535±0.020\\text{h}&1.658±0.015\\text{h}\\\\\\\\\n\\text{AdaptSSR}&1.539±0.017\\text{h}&1.830±0.012\\text{h}\\\\\\\\\\hline\\end{array}"}, {"review_id": "N6jOxl45wP", "reviewer": "Reviewer_8cxi", "summary": "Recent studies have explored pre-training user models with contrastive learning tasks to address data sparsity issues in user-oriented tasks. However, existing augmentation methods may introduce noisy or irrelevant interests, leading to negative transfer. To overcome this, a new approach called Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR) is proposed, which replaces contrastive learning with a multiple pairwise ranking loss. An augmentation-adaptive fusion mechanism is also introduced to combine learned ranking orders based on the similarity between augmented views. Extensive experiments demonstrate the effectiveness of AdaptSSR across various tasks and datasets.", "questions": "1. Please explain what is implicit augmentation, how is $u^+$ generated, and why equation (1) should hold.\n2. The authors introduce a Augmentation-Adaptive Fusion coefficient $\\lambda$ needs further discussion. Is the $\\lambda$ fixed during training, i.e. stop_gradient($\\lambda$), so that the gradient calculation does not involve $\\lambda$?", "limitations": "Please address the issues highlighted in the Weaknesses section.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The paper's motivation is reasonable, directly adopting contrastive learning may lead to consistency problems in recommendations.\n2. This paper proposes a novel approach to combine implicit and explicit augmentations.", "weaknesses": "1. The main contribution of this paper is adding an order constraint in the loss function, \n   which is a rather incremental modification of existing contrastive learning framework. \n   The main idea of paper is a fusion of explicit augmentation and implicit augmentation by the loss function.\n   Thus, the novelty is limited.\n\n2. It's unclear whether the added constraint is necessary. Since \"$u$ and $u^+$ originate from exactly the same input behavior sequence\" as the authors commented in line 123, I think $sim(u, u^+)\\ge sim(u, u^-)$ and $sim(u, u^+) \\ge sim(u, \\hat{u}$ should always hold. I don't understand why we need the $sim(u, u^+)$ term here. Without the $sim(u, u^+)$ term, the proposed method reduces to common contrastive learning.\n\n3. Even if the constraint is meaningful, the authors' analysis cannot convince me why such constraint may help generalization. Why may $sim(u, \\hat{u}) > sim(u, u^+)$ harm the downstream performance? Intuitively, suppose the objective of original contrastive learning is overly strong, we should loose the constraints. For example, $sim(u, \\hat{u}) \\ge sim(u, u^-) - \\epsilon$. However, the authors make the constraints even stronger by adding another constraint term. This does not make sense to me.\n\n---\n\nEdit after rebuttal: The authors' response resolved my primary concern about technical correctness. I'd like to raise my score to a borderline reject regarding the novelty of this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time in reviewing our paper. However, we suspect that there may be certain misconceptions. In order to address your concerns, we provide detailed, point-by-point responses as follows. **Due to the character limit of each rebuttal, more responses are provided in the global response.**\n\n**Weakness 1**: The main contribution of this paper is adding an order constraint in the loss function, which is a rather incremental modification of the existing contrastive learning framework. The main idea of the paper is a fusion of explicit augmentation and implicit augmentation by the loss function. Thus, the novelty is limited.\n\n**Response**: We appreciate your feedback. Indeed, as you have pointed out, the introduction of an order constraint represents one facet of our contributions. However, it is important to clarify that this is not the sole contribution of our work and our method differs from contrastive learning in several aspects. Specifically, the main contributions of our paper are as follows:\n\n1. We identify the semantic inconsistency problem when applying contrastive learning to user behavior sequences.\n2. To tackle this problem, we escape the existing contrastive learning framework and propose a new augmentation-adaptive self-supervised ranking task. Instead of simply training the model to distinguish the positive augmented views from the negative ones as contrastive learning, we train the user model to capture a more precise and realistic similarity order between the implicitly augmented view, the explicitly augmented view, and views from other users.\n3. Different from contrastive learning which simply maximizes the similarity between the augmented views for every sample in a fixed way, we further design an augmentation-adaptive fusion mechanism that adaptively adjusts the similarity order constraint applied to each sample based on the semantic similarity between the augmented views.\n4. Extensive experiments on both public and industrial datasets verify that our AdaptSSR can be applied to different kinds of user behavior sequences and bring significant performance improvement to various downstream tasks.\n\nTherefore, our method differs from contrastive learning in terms of the constraint applied to each sample and the design of the loss function. To our best knowledge, our work is the first to tackle the semantic inconsistency problem when applying contrastive learning in the user modeling domain, and our method can be further generalized to other domains where augmentation choices are not straightforward or could alter the semantics of the data. Overall, we think the contribution and novelty of our work are non-trivial.\n\n**Weakness 2**: It's unclear whether the added constraint is necessary. Since \"$u$ and $u^+$ originate from exactly the same input behavior sequence\" as the authors commented in line 123, I think $sim(u,u^+)\\geq sim(u,u^-)$ and $sim(u,u^+)\\geq sim(u,\\hat{u})$ should always hold. I don't understand why we need the $sim(u,u^+)$ term here. Without the $sim(u,u^+)$ term, the proposed method reduces to common contrastive learning.\n\n**Response**: First, it is essential to clarify that although $u$ and $u^+$ originate from exactly the same input behavior sequence, different independently sampled dropout masks are applied in the user model, which adds distinct noise to the input sequence in the feature space. Therefore, $u$ and $u^+$ are different, and the pairwise similarity order $sim(u,u^+)\\geq sim(u,u^-)$ and $sim(u,u^+)\\geq sim(u,\\hat{u})$ do not always hold. \n\nIn addition, the $sim(u,u^+)$ term is necessary as it works as the upper bound of $sim(u,\\hat{u})$. Without this term, our method will apply the same constraint as contrastive learning and the model will directly maximize $sim(u,\\hat{u})$ for every sample while neglecting the potential semantic inconsistency between the augmented views. Besides, our augmentation-adaptive fusion mechanism can further adjust $sim(u,\\hat{u})$ between the upper bound $sim(u,u^+)$ and the lower bound $sim(u,u^-)$ properly for each sample based on the semantic similarity between the augmented views. Moreover, from the results in Table 2, we can find that our AdaptSSR consistently outperforms existing contrastive learning-based pre-training methods (e.g., CCL, CL4SRec, CoSeRec), which further verifies the effectiveness of the additional $sim(u,u^+)$ term.\n\n**Weakness 3.1**: Even if the constraint is meaningful, the authors' analysis cannot convince me why such constraint may help generalization. Why may $sim(u,\\hat{u})>sim(u,u^+)$ harm the downstream performance?\n\n**Response**: We acknowledge that we may not have made this sufficiently clear in our paper, and we appreciate the opportunity to clarify. The reason why we require $sim(u,u^+)\\geq sim(u,\\hat{u})$ is that the difference between $u$ and $u^+$ is only caused by the different dropout masks applied by the implicit augmentation, while the difference between $u$ and $\\hat{u}$ is caused by both the implicit augmentation and the explicit augmentation which directly modifies the input behavior sequence on the data level. If $sim(u,\\hat{u})>sim(u,u^+)$, it means the user model cannot well capture the characteristics and interests of the user from the behavior sequence, and the generated user embedding cannot correctly reflects the similarity between different users. Thus, it will transfer incorrect prior knowledge to the downstream task and degrade the model performance, especially when the downstream task faces a data sparsity problem."}, {"review_id": "H1KEbHSihZ", "reviewer": "Reviewer_toy8", "summary": "The paper tackles user-oriented tasks, e.g. personalized recommendation, and proposes a self-supervised method named AdaptSSR to replace the contrastive learning pre-training target task. It adopts a ranking loss that selects samples of smallest similarity differences and assigns dynamic weight coefficients to ranking parts based on the estimated similarity between the augmented views. Experiments on 6 downstream tasks from 2 datasets and several empirical analyses are conducted to verify the effectiveness of AdaptSSR.\n", "questions": "Please elaborate on the differences between the approach proposed in this paper and [49], and highlight the contribution of this paper.", "limitations": "The authors have not discussed the limitations and broader societal impacts in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The pre-training method and objective are clearly explained, and the equations in the text concisely demonstrate the proposed loss function.", "weaknesses": "The Multiple Pairwise Ranking loss, which is the core of the method, is not an original contribution of this paper, but an adaptation from Yu et al [49]. However, there is almost no mentioning of this work except for the source of the loss function, casting doubt on the novelty of the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for the time and effort you have invested in reviewing our paper. In response to your insightful comments, we have provided detailed explanations and clarifications, which are enumerated below.\n\n**Weakness and Question 1**: The Multiple Pairwise Ranking loss, which is the core of the method, is not an original contribution of this paper, but an adaptation from Yu et al [49]. However, there is almost no mention of this work except for the source of the loss function, casting doubt on the novelty of the paper. Please elaborate on the differences between the approach proposed in this paper and [49], and highlight the contribution of this paper.\n\n**Response**: First, our work differs from Yu et al [49] in the following aspects:\n\n1.  **The studied problems are completely different.** Yu et al [49] aim to train a better collaborative filtering model for item recommendation based on users' implicit feedback, while we aim to tackle the semantic inconsistency problem between the augmented views when pre-training a discriminative user model based on user behavior sequences.\n2.  **The definitions of the target ranking order are completely different.** Yu et al [49] aim to model the order of the user's preference difference between (I) an observed item and an unobserved item, (II) two unobserved items, and (III) two observed items, while we train the user model to capture the similarity orders between (I) the implicitly augmented views, (II) the explicitly augmented views, and (III) views from other users. \n3.  **How to construct the training triplet is different.** Yu et al [49] just randomly sample items from the corresponding item set, while we design an explicit hard negative sampling strategy to facilitate model training, which selects the pair with the smallest similarity difference for each pairwise ranking order.\n4.  **How to fuse the two learned pairwise ranking order is different.** Yu et al [49] use a fixed and unified hyperparameter $\\lambda$ in the original multiple pairwise ranking loss, while our augmentation-adaptive fusion mechanism takes the distinct impacts of data augmentation on different behavior sequences into account and employs a dynamic coefficient $\\lambda_i$ for each training sample $S_i$. The value of $\\lambda_i$ is calculated based on the estimated semantic similarity between the augmented views along the training procedure.\n\nThe only connection between our work and Yu et al [49] is that we both try to simultaneously learn a ranking order between three terms. That's why we adopt the Multiple Pairwise Ranking loss for model training in our work. We will highlight the difference between our work and Yu et al [49] in the revised version of our paper.\n\nSecond, the main contributions of our paper are as follows:\n\n1. We identify the semantic inconsistency problem when applying contrastive learning to user behavior sequences.\n2. To tackle this problem, we escape the existing contrastive learning framework and propose a new augmentation-adaptive self-supervised ranking task. Instead of simply training the model to distinguish the positive augmented views from the negative ones as contrastive learning, we train the user model to capture a more precise and realistic similarity order between the implicitly augmented view, the explicitly augmented view, and views from other users.\n3. Different from contrastive learning which simply maximizes the similarity between the augmented views for every sample in a fixed way, we further design an augmentation-adaptive fusion mechanism that adaptively adjusts the similarity order constraint applied to each sample based on the semantic similarity between the augmented views.\n4. Extensive experiments on both public and industrial datasets verify that our AdaptSSR can be applied to different kinds of user behavior sequences and bring significant performance improvement to various downstream tasks.\n\nTo our best knowledge, our work is the first to tackle the semantic inconsistency problem when applying contrastive learning in the user modeling domain, and our method can be further generalized to other domains where augmentation choices are not straightforward or could alter the semantics of the data. We will highlight our contributions in the revised version of our paper.\n\n**Limitation 1**: The authors have not discussed the limitations and broader societal impacts in the paper.\n\n**Response**: Thanks for pointing it out. We have discussed the limitations of our work in the Appendix due to the space limit. We will put them in the main content in the revised version of our paper."}]

Author rebuttal:
We sincerely thank all the reviewers for their appreciation and constructive comments. We have provided detailed responses to each reviewer's concerns and questions in the following rebuttals. We hope our responses will address your concerns and strengthen our paper. We are happy to respond to any new questions during the discussion period.

---

**More responses to Reviewer 8cxi**

**Weakness 3.2**: Intuitively, suppose the objective of original contrastive learning is overly strong, we should loose the constraints. For example, $sim(u,\hat{u})\geq sim(u,u^-)-\epsilon$. However, the authors make the constraints even stronger by adding another constraint term. This does not make sense to me.

**Response**: We appreciate your insightful thinking. However, the primary issue of contrastive learning does not lie in an overly strong objective. The problem is that it imposes a fixed and inaccurate constraint $sim(u,\hat{u})\geq sim(u,u^-)$ on every sample while the semantic similarity between the augmented views cannot be guaranteed. The InfoNCE-style contrastive loss will simply maximize $sim(u,\hat{u})$ no matter whether the augmented views are similar or not, which will lead to a negative transfer for the downstream task. Our method applies an adaptive similarity order constraint to each sample by adjusting $sim(u,\hat{u})$ between the upper bound $sim(u,u^+)$ and the lower bound $sim(u,u^-)$ based on the semantic similarity between the augmented views. We will refine our writing to make our motivation clearer.

**Question 1**: Please explain what is implicit augmentation, how is $u^+$ generated, and why equation (1) should hold.

**Response**: We appreciate your comment. We will respond to each of your questions in a sequential manner.

- As we mentioned in Line 99 of our paper, the implicit augmentation is performed via the dropout module in the model, which adds noise to the input sequence in the feature space.
- As we mentioned in Line 148 of our paper, given a behavior sequence $S$, we input it into the model twice with different independently sampled dropout masks and denote the generated user representations as $u$ and $u^+$. 
- Since the difference between $u$ and $u^+$ is only caused by the different dropout masks applied by the implicit augmentation, while the difference between $u$ and $\hat{u}$ is caused by both the implicit augmentation and the explicit augmentation which directly modifies the input behavior sequence on the data level, we require the model to capture the similarity order $sim(u,u^+)\geq sim(u,\hat{u})$. Similarly, the difference between $u$ and $u^-$ is caused by both the implicit augmentation and the distinct interests of different users. Therefore, we require the model to capture the similarity order $sim(u,u^+)\geq sim(u,u^-)$. Since the explicit augmentation modifies $S$ at the data level and the semantic consistency between $S$ and $\hat{S}$ cannot be guaranteed, $sim(u,\hat{u})$ should be placed between $sim(u,u^+)$ and $sim(u,u^-)$, which leads to the final similarity order $\Gamma: sim(u,u^+)\geq sim(u,\hat{u})\geq sim(u,u^-)$ (Equation (1)). Our goal is pre-training the user model to adjust $sim(u,\hat{u})$ properly between $sim(u,u^+)$ and $sim(u,u^-)$ based on the semantic similarity between the augmented views.

**Question 2**: The authors introduce an Augmentation-Adaptive Fusion coefficient $\lambda$ that needs further discussion. Is the $\lambda$ fixed during training, i.e.,$\operatorname{stop\\_gradient}(\lambda)$, so that the gradient calculation does not involve $\lambda$?

**Response**: As we mentioned in Line 175 of our paper, our augmentation-adaptive fusion mechanism replaces the fixed and unified hyperparameter $  \lambda $ with a dynamic coefficient $\lambda_i$ for each training sample $S_i$. The value of $\lambda_i$ is dynamically calculated based on the average similarity between the user representations generated from the augmented views $\hat{S}_i$ and $\tilde{S}_i$ (Equation (7)) along the training procedure. It is not a learnable parameter so it is not involved in the gradient calculation. We will highlight how $\lambda_i$ is dynamically calculated in the revised version of our paper.

Meta-review: The paper proposes and analyzes AdaptSSR, a self-supervised learning (SSL) technique for user modeling from user behavior sequences to address issues like data sparsity in real-world user platforms. Experimental results demonstrate AdaptSSR's good effectiveness across tasks and datasets, from both public domain and industry. However, some reviewers raised concerns regarding the method's clarity. For example, the notations used are not very intuitive and it is hard to understand how different implicitly augments views are generated. I highly encourage the authors to update their manuscript using reviewer suggestions.
Decision: Accept (poster)