SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {\mathcal {L}} _ {i} ^ {\prime} = - \log \sigma \left[ \operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i}, \tilde {\boldsymbol {u}} _ {j}\right) - \max  _ {\boldsymbol {w} \in \mathbf {V} _ {i} ^ {-}} \operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i}, \boldsymbol {w}\right) \right] \tag {8} \\ = - \log \frac {\exp (\sin (\hat {\boldsymbol {u}} _ {i} , \tilde {\boldsymbol {u}} _ {j}))}{\exp (\sin (\hat {\boldsymbol {u}} _ {i} , \tilde {\boldsymbol {u}} _ {j})) + \max _ {\boldsymbol {w} \in \mathbf {V} _ {i} ^ {-}} \exp (\sin (\hat {\boldsymbol {u}} _ {i} , \boldsymbol {w}))}, \\ \end{array}
$$
where  $\mathbf{V}_i^- = \{\hat{u}_j,\tilde{u}_j\}_{j = 1,j\neq i}^B$ . Most existing contrastive learning-based pre-training methods [2, 32, 38] adopt the InfoNCE loss [30] to train the user model, which can be formulated as follows:
$$
\mathcal {L} _ {\text {I n f o N C E}} = - \log \frac {\exp \left(\operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i} , \tilde {\boldsymbol {u}} _ {j}\right)\right)}{\exp \left(\operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i} , \tilde {\boldsymbol {u}} _ {j}\right)\right) + \sum_ {\boldsymbol {w} \in \mathbf {V} _ {i} ^ {-}} \exp \left(\operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i} , \boldsymbol {w}\right)\right)}. \tag {9}
$$
Both loss functions aim to maximize the agreement between the augmented views. The sole distinction is that  $\hat{\mathcal{L}}_i^{\prime}$  selects the hardest in-batch negative sample, which is most similar to the anchor  $\hat{\pmb{u}}_i$ , whereas  $\mathcal{L}_{\mathrm{InfoNCE}}$  uses the entire negative sample set  $\mathbf{V}_i^{-}$ . Thus, with the same data augmentation method, the degenerated version of AdaptSSR is equivalent to combining contrastive learning-based methods with hard negative sampling, which has been proven to be effective by several recent studies [19, 35, 42].
When  $\lambda_{i} > 0$ , the former term  $\mathrm{sim}(\hat{\pmb{u}}_i,\hat{\pmb{u}}_i^+)$  -  $\max_{\pmb {v}\in \{\tilde{\pmb{u}}_i,\tilde{\pmb{u}}_i^+\}}\mathrm{sim}(\hat{\pmb{u}}_i,\pmb {v})$  in  $\hat{\mathcal{L}}_i$  forces the user model to capture the similarity order  $\mathrm{sim}(\hat{\pmb{u}}_i,\hat{\pmb{u}}_i^+)\geqslant \mathrm{sim}(\hat{\pmb{u}}_i,\pmb {v})$  as well. In addition, our augmentation-adaptive fusion mechanism automatically adjusts the similarity order constraint applied to each sample. As a result, our method alleviates the requirement of semantic consistency between the augmented views and can adapt to various data augmentation methods.
Datasets We conduct experiments on two real-world datasets encompassing six downstream tasks. The first dataset, the Tencent Transfer Learning (TTL) dataset, was released by Yuan et al. [52] and contains users' recent 100 interactions on the QQ Browser platform. Additionally, it provides the downstream labeled data of two user profiling tasks: age prediction  $(\mathcal{T}_1)$  and life status prediction  $(\mathcal{T}_2)$ , and two cold-recommendation tasks: click recommendation  $(\mathcal{T}_3)$  and thumb-up recommendation  $(\mathcal{T}_4)$ . The second dataset, the App dataset, consists of users' app installation behaviors collected by a worldwide smartphone manufacturer, OPPO, from 2022-12 to 2023-03. Each user has been securely
Table 1: Detailed statistics of each dataset and downstream task.
Dataset	TTL	App
# Behavior Sequences	1,470,149	1,575,837
# Different Behaviors	645,972	4,047
Avg. Sequence Length	54.84	44.13
Downstream Task	\( \mathcal{T}_1 \)	\( \mathcal{T}_2 \)	\( \mathcal{T}_3 \)	\( \mathcal{T}_4 \)	\( \mathcal{T}_5 \)	\( \mathcal{T}_6 \)
# Samples	1,470,147	1,020,277	1,397,197	255,646	1,178,603	564,940
# Labels/Items	8	6	17,879	7,539	2	2
hashed into an anonymized ID to prevent privacy leakage. It also provides the downstream labeled data of a gender prediction task  $(\mathcal{T}_5)$ , and a CVR prediction task  $(\mathcal{T}_6)$ , which predicts whether the user will install a target app. Detailed statistics of each dataset and downstream task are listed in Table 1.
Evaluation Protocols For model pre-training,  $90\%$  user behavior sequences are randomly selected for training, while the rest  $10\%$  are used for validation. For each downstream task, we randomly split the dataset by 6:2:2 for training, validation, and testing. To evaluate model performance on various downstream tasks, we use classification accuracy (Acc) for multi-class classification tasks  $(\mathcal{T}_1,\mathcal{T}_2)$ , NDCG@10 for cold-recommendation tasks  $(\mathcal{T}_3,\mathcal{T}_4)$ , and AUC for binary classification tasks  $(\mathcal{T}_5,\mathcal{T}_6)$ . To avoid the bias brought by sampled evaluation [21], we use the all-ranking protocol for the cold-recommendation task evaluation, i.e., all items not interacted by the user are used as candidates. We repeat each experiment 5 times and report the average results with standard deviations.
Baselines To validate the effectiveness of AdaptSSR, we compare it with three sets of baselines. The first set contains several generative pre-training methods, including:
- PeterRec [52] applies the MLM task [10] to the behavior sequence for user model pre-training.
- PTUM [45] pre-trains the user model with two self-supervised tasks: masked behavior prediction and next  $K$  behavior prediction.
The second set contains several discriminative pre-training methods, including:
- CLUE [8] solely uses implicitly augmented views for contrastive pre-training.
- CCL [2] proposes a mask-and-fill strategy to construct high-quality augmented views with a data generator trained with the MLM task for contrastive pre-training.
- IDICL [38] takes different time periods of the same behavior sequence as augmented views for contrastive pre-training. An interest dictionary is used to extract multiple interests of the user.
The last set contains several data augmentation methods for user behavior sequences, including:
- CL4SRec [47] augments the behavior sequence by masking, cropping, and reordering.
- CoSeRec [27] takes item correlation into consideration and improves CL4SRec with two more informative augmentation operators: substitute and insert.
- DuoRec [32] takes sequences with the same last behavior as semantically similar positive samples.
We apply them to generate positive samples for the contrastive pre-training task. We also evaluate the performance of the model trained on the downstream task in an end-to-end manner. All these baselines share the same model structure with our AdaptSSR and only differ in the pre-training task.
Implementation and Hyper-parameters Following previous works [2, 8], we set the embedding dimension  $d$  as 64. In the Transformer Encoder, the number of attention heads and layers are both set as 2. The dropout probability is set as 0.1. The data augmentation proportion  $\rho$  for each baseline method is either searched from  $\{0.1, 0.2, \dots, 0.9\}$  or set as the default value in the original paper if provided. The batch size and learning rate are set as 128 and 2e-4 for both pre-training and fine-tuning. More implementation details are listed in Appendix C.
4.2 Overall Performance of Downstream Tasks
The experimental results on various downstream tasks are presented in Table 2. From the results, we have several findings. First, generative pre-training methods (PeterRec and PTUM) underperform compared to most discriminative pre-training methods. This is because these methods mainly focus
Table 2: Performance (%) of various pre-training methods on downstream tasks. Impr (%) indicates the relative improvement compared with the end-to-end training. The best results are bolded.
Pre-train Method	T1	T2	T3	T4	T5	T6
Acc	Impr	Acc	Impr	NDCG@10	Impr	NDCG@10	Impr	AUC	Impr	AUC	Impr
None	62.87±0.05	-	52.24±0.16	-	1.99±0.03	-	2.87±0.07	-	78.63±0.06	-	75.14±0.14	-
PeterRec	63.62±0.11	1.19	53.14±0.07	1.72	2.37±0.02	19.10	3.06±0.08	6.62	79.61±0.13	1.25	76.04±0.10	1.20
PTUM	63.21±0.14	0.54	53.05±0.04	1.55	2.29±0.03	15.08	2.96±0.03	3.14	79.48±0.11	1.08	75.82±0.13	0.90
CLUE	63.38±0.10	0.81	53.23±0.05	1.90	2.38±0.02	19.60	3.05±0.21	6.27	79.90±0.06	1.62	76.03±0.16	1.18
CCL	63.76±0.11	1.42	53.37±0.09	2.16	2.43±0.02	22.11	3.32±0.13	15.68	80.22±0.07	2.02	77.35±0.10	2.94
IDICL	63.88±0.04	1.61	53.45±0.05	2.32	2.46±0.02	23.62	3.42±0.04	19.16	80.34±0.05	2.17	77.92±0.08	3.70
CL4SRec	63.71±0.14	1.34	53.43±0.05	2.28	2.41±0.03	21.11	3.29±0.06	14.63	80.14±0.08	1.92	77.02±0.05	2.50
CoSeRec	63.89±0.03	1.62	53.53±0.09	2.47	2.44±0.02	22.61	3.33±0.05	16.03	80.48±0.06	2.35	77.71±0.09	3.42
DuoRec	63.50±0.09	1.00	53.26±0.06	1.95	2.39±0.01	20.10	3.11±0.16	8.36	80.03±0.09	1.78	76.85±0.09	2.28
AdaptSSR	65.53±0.04	4.23	54.41±0.02	4.15	2.61±0.03	31.16	3.73±0.03	29.97	82.30±0.03	4.67	79.92±0.05	6.36
on mining the correlation between behaviors while lacking careful design for user representation learning, which limits their performance on downstream tasks. Second, discriminative pre-training methods with explicit data augmentation (e.g., CCL, CL4SRec, CoSeRec) generally outperform the method relying solely on implicit data augmentation (CLUE). We argue that this is because the implicit data augmentation caused by the dropout mask alone is too weak. The user model can easily distinguish the positive sample from others, thus providing limited knowledge for downstream tasks. Third, our AdaptSSR consistently surpasses previous SOTA contrastive learning-based pre-training methods by  $2.6\%$ ,  $1.7\%$ ,  $6.1\%$ ,  $9.1\%$ ,  $2.3\%$ , and  $2.6\%$  on each downstream task respectively, and our further t-test results show the improvements are significant at  $p < 0.01$ . This is because we train the user model to capture the similarity order  $\Gamma$ , rather than directly maximizing the similarity between the explicitly augmented views. Such a ranking task alleviates the requirement of semantic consistency while maintaining the discriminability of the pre-trained user model.
4.3 Performance with Different Data Augmentation Methods
As our method alleviates the requirement of semantic consistency between the augmented views and can adapt to a variety of data augmentation methods, we further combine it with several existing
pre-training methods: CL4SRec, CoSeRec, and CCL, by replacing the contrastive learning (CL) task with our AdaptSSR while maintaining their data augmentation methods. We vary the augmentation proportion  $\rho$  from 0.1 to 0.9 and evaluate the performance of these methods on the downstream age prediction task  $(\mathcal{T}_1)$ . The results on other downstream tasks show similar trends and are included in Appendix D. From the results shown in Fig. 3, we find that these contrastive learning-based methods are quite sensitive to the data augmentation proportion. When  $\rho$  is close to 0, the user model can easily discriminate these weakly augmented positive samples from others during pre-training and thus brings limited performance gain to the downstream task. However, the stronger the augmentation is, the more likely it is to generate dissimilar augmented views and may even cause a negative transfer to the downstream task. In contrast, our AdaptSSR significantly improves the performance of all these pre-training methods with
different augmentation proportions to a similar level. This is because our self-supervised ranking task takes the potential semantic inconsistency between the augmented views into account and avoids directly maximizing their similarity. In addition, our augmentation-adaptive fusion mechanism can properly combine the learned pairwise ranking orders based on the estimated similarity between the explicitly augmented views constructed by various augmentation methods with different strengths, which leads to similar model performance.
Figure 3: Effectiveness of AdaptSSR when combined with existing pre-training methods.
4.4 Ablation Study
In this subsection, we further validate the effectiveness of each component in our method, i.e., in-batch hard negative sampling, augmentation-adaptive fusion, and self-supervised ranking. We pre-train the
user model with our AdaptSSR and its variant with one component removed. When the augmentation-adaptive fusion mechanism is removed, we search for the optimal  $\lambda$  from  $\{0, 0.1, \dots, 1.0\}$ . When the self-supervised ranking task is removed, we use the contrastive learning task instead. The results of the downstream age prediction task  $(\mathcal{T}_1)$  and thumb-up recommendation task  $(\mathcal{T}_4)$  are shown in Fig. 4. We first find that the model performance drops significantly by  $2.8\%$  on  $\mathcal{T}_1$  and  $11.8\%$  on  $\mathcal{T}_4$  when the self-supervised ranking task is removed. This is because it alleviates the requirement of semantic consistency between the augmented views, which is the major issue that impedes existing contrastive learning-based pretraining methods. Besides, the model performance also declines by  $0.9\% - 1.6\%$  on  $\mathcal{T}_1$  and  $3.1\% - 5.1\%$  on  $\mathcal{T}_4$  after we remove either in-batch hard negative sampling or augmentation-adaptive fusion. This is
Figure 4: Effectiveness of each component in our AdaptSSR.
because in-batch hard negative sampling helps the model focus on discriminating the most similar samples while using all quadruples may introduce extra noise. Our augmentation-adaptive fusion mechanism adjusts the value of  $\lambda_{i}$  based on the similarity between the augmented views for each training sample, while a constant  $\lambda$  applies a fixed ranking order constraint to all samples and neglects the distinct impacts of data augmentation across different behavior sequences.
4.5 Efficiency Comparison
To compare the efficiency of different pre-training methods, we record their average training time on the TTL dataset and the App dataset and list them in Table 3. Our observations indicate that the average
pre-training time of our AdaptSSR is similar to that of existing contrastive learning-based methods, such as CL4SRec and CoSeRec. Although our AdaptSSR requires inputting each behavior sequence into the model twice, we implement it by duplicating the input sequences in the batch size dimension. Thus, we only need to input all the sequences into the model once, which can be well parallelized by the GPU. Besides, for each pairwise ranking order, our in-batch hard negative sampling strategy only selects the pair with the smallest similarity difference to compute the multiple pairwise ranking loss, which avoids the costly softmax operation and cross-entropy loss calculation in the existing contrastive learning task. As a result,
Table 3: Average training time (hours) of different pre-training methods on each dataset.
Pre-train Method	TTL	App
PeterRec	2.927±0.022	1.537±0.010
PTUM	2.015±0.009	2.055±0.018
CLUE	1.453±0.016	1.633±0.015
IDICL	1.257±0.021	1.162±0.020
CL4SRec	1.868±0.013	2.081±0.017
CoSeRec	1.902±0.015	2.104±0.023
DuoRec	1.535±0.020	1.658±0.015
AdaptSSR	1.539±0.017	1.830±0.012
our AdaptSSR will not greatly increase the overall computational cost while bringing significant performance improvement to various downstream tasks.
4.6 User Representation Similarity Distribution Analysis
To better reflect the difference between our AdaptSSR and other pre-training methods, we randomly select 10,000 users from the TTL dataset. For each user's behavior sequence, we augment it by masking and cropping with a proportion of 0.6. Then, we calculate the cosine similarity between the user representations generated from the original behavior sequence, the augmented behavior sequence, and the behavior sequence of other users. We use kernel density estimation (KDE) [31] to estimate their probability density distributions, which are visualized in Fig. 5. The figure reveals that when pre-training with existing contrastive learning-based methods (e.g., CL4SRec, CoSeRec), the similarity between the augmented views always gathers in a very narrow range close to 1.0.
Figure 5: Distributions of the cosine similarity between user representations generated from the original behavior sequence, different augmented behavior sequences, and the behavior sequences of other users with various pre-training methods. The area under each curve equals to 1.
This is because these methods always maximize the similarity between the augmented views while neglecting their potential semantic inconsistency. Furthermore, when pre-trained with other methods (e.g., PTUM, CCL), the similarity between different users tends to disperse over a very broad range (from -0.5 to 1.0), which indicates that the user model cannot well distinguish the behavior sequence of different users. In contrast, the similarity between the augmented views distributes over a relatively wider range (from 0.5 to 1.0) when pre-trained with our AdaptSSR, which well fits our findings about these semantically inconsistent augmented views. Meanwhile, the distribution of the similarity between different users is still well separated from that between the augmented views, which means the user model can also discriminate the behavior sequence of different users.
In this paper, we identified the semantic inconsistency problem faced by existing contrastive learning-based user model pre-training methods. To tackle this challenge, we proposed a new pretext task: Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR). A multiple pairwise ranking loss is adopted to train the user model to capture the similarity order between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employed an in-batch hard negative sampling strategy to facilitate model training. Additionally, to cope with the distinct impacts of data augmentation on different behavior sequences, we designed an augmentation-adaptive fusion mechanism to adjust the similarity order constraint applied to each sample based on the estimated similarity between the augmented views. Our method can be further generalized to other domains where augmentation choices are not straightforward or could alter the semantics of the data. Extensive experiments validated that AdaptSSR consistently outperforms previous baselines by a large margin on various downstream tasks while adapting to diverse data augmentation methods.
This research was partially supported by grants from the National Key Research and Development Program of China (No. 2021YFF0901003), and the OPPO joint research program. We furthermore thank the anonymous reviewers for their constructive comments.
[1] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximizing Mutual Information Across Views. In Advances in Neural Information Processing Systems, volume 32, pages 15535-15545, 2019.
[50] Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5478–5489, 2022.
[51] Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose, and Xiangnan He. A Simple Convolutional Generative Network for Next Item Recommendation. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining, pages 582-590, 2019.
[52] Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1469–1478, 2020.
[53] Kai Zhang, Hao Qian, Qing Cui, Qi Liu, Longfei Li, Jun Zhou, Jianhui Ma, and Enhong Chen. Multi-Interactive Attention Network for Fine-Grained Feature Learning in CTR Prediction. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 984–992, 2021.
[54] Kai Zhang, Qi Liu, Hao Qian, Biao Xiang, Qing Cui, Jun Zhou, and Enhong Chen. EATN: An Efficient Adaptive Transfer Network for Aspect-Level Sentiment Analysis. IEEE Transactions on Knowledge and Data Engineering, 35(1):377-389, 2023.
[55] Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang, Xiaokang Yang, and Pinyin Lu. M-Mix: Generating Hard Negatives via Multi-Sample Mixing for Contrastive Learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2461–2470, 2022.
[56] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. In Advances in Neural Information Processing Systems, volume 34, pages 15870-15882, 2021.
[57] Lei Zheng, Chaozhuo Li, Chun-Ta Lu, Jiawei Zhang, and Philip S. Yu. Deep Distribution Network: Addressing the Data Sparsity Issue for Top-N Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1081-1084, 2019.
[58] Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1):4564-4571, 2018.
[59] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1059–1068, 2018.
[60] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaogiang Zhu, and Kun Gai. Deep Interest Evolution Network for Click-through Rate Prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1):5941-5948, 2019.
A More Examples of the Semantic Inconsistency Problem
Figure 6: Illustrations of the impact of various data augmentation methods on different user behavior sequences. Each picture represents a news article clicked by the user. Pictures with the same color border reflect similar interests of the user. Dash borders are used to indicate behaviors replaced by the data augmentation method.
We randomly sampled two more anonymous users from the MSN online news platform to illustrate the impact of various data augmentation methods on different user behavior sequences and the consequent semantic inconsistency problem. The users' recent eight behaviors and the results of several data augmentation methods are shown in Fig. 6. The data augmentation proportion is set as 0.6. From the original behavior sequence of user A, we may infer that the user is quite concerned about coronavirus and politics. We also find that the behavior sequence augmented by masking well preserves the user's interests. However, the two sequences augmented by cropping contain completely different interests of the user over time. Meanwhile, some behaviors replaced by mask-and-fill and substitute are noisy, which the user may not be interested in. Similarly, the original behavior sequence of user B shows the user's potential interests in coronavirus, politics, basketball, entertainment, and food. As a result, augmentation methods such as mask and crop are likely to lose certain characteristics of the user, while augmentation methods such as mask-and-fill and substitute tend to bring in noisy behaviors. In
fact, it is hard to determine what are the real interests of user B from such a diverse behavior sequence since some behaviors may be the result of misclicking or click-baiting, thus making it even harder to provide a semantically consistent augmentation. From these examples, we can find that existing data augmentation methods may fail to preserve the characteristics or interests in the behavior sequence and cannot guarantee semantic consistency between the augmented views. Thus, directly forcing the user model to maximize the agreement between the augmented sequences may result in a negative transfer for downstream tasks.
B The Algorithm of AdaptSSR
The pseudo-codes of the pre-training procedure with our AdaptSSR are shown in Algorithm 1.