SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Yu $^{1,2}$ , Qi Liu $^{1,2*}$ , Kai Zhang $^{1,2}$ , Yuren Zhang $^{1,2}$ , Chao Song $^{3}$ , Min Hou $^{4}$ , Yuqing Yuan $^{3}$ , Zhihao Ye $^{3}$ , Zaixi Zhang $^{1,2}$ , Sanshi Lei Yu $^{1,2}$ $^{1}$ Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China  
 $^{2}$ State Key Laboratory of Cognitive Intelligence  
 $^{3}$ OPPO Research Institute  $^{4}$ Hefei University of Technology  
{yflyl613, kkzhang0808, yr160698, zaixi}@mail.ustc.edu.cn  
{songchao12, yuanyuqing, yezhihao3}@oppo.com  
qiliuql@ustc.edu.cn, {hmhoumin, meet.leiyu}@gmail.com
User modeling, which aims to capture users' characteristics or interests, heavily relies on task-specific labeled data and suffers from the data sparsity issue. Several recent studies tackled this problem by pre-training the user model on massive user behavior sequences with a contrastive learning task. Generally, these methods assume different views of the same behavior sequence constructed via data augmentation are semantically consistent, i.e., reflecting similar characteristics or interests of the user, and thus maximizing their agreement in the feature space. However, due to the diverse interests and heavy noise in user behaviors, existing augmentation methods tend to lose certain characteristics of the user or introduce noisy behaviors. Thus, forcing the user model to directly maximize the similarity between the augmented views may result in a negative transfer. To this end, we propose to replace the contrastive learning task with a new pretext task: Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Specifically, we adopt a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employ an in-batch hard negative sampling strategy to facilitate model training. Moreover, considering the distinct impacts of data augmentation on different behavior sequences, we design an augmentation-adaptive fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the estimated similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks verify the effectiveness of AdaptSSR.
User modeling aims to capture the user's characteristics or interests and encode them into a dense user representation for a specific user-oriented task, such as user profiling, personalized recommendation, and click-through rate prediction [24, 39, 50, 59]. In the past decade, extensive methods [5, 9, 12, 22, 53] have been proposed to model users based on their historical behaviors. Despite their great success in various tasks, these methods usually rely on a large amount of task-specific labeled data to train accurate user models, which makes them suffer from the data sparsity problem [17, 36, 57].
*Qi Liu is the corresponding author.
Figure 1: An illustration of the impact of different data augmentation methods on the user behavior sequence. For a more intuitive demonstration, we use a picture to represent the main topic of each news article clicked by the user. Pictures with the same color border reflect similar interests of the user. Dash borders are used to indicate behaviors replaced by the data augmentation method.
A mainstream technique to tackle this challenge is the pre-training paradigm [33, 46, 54, 56]. The user model is first pre-trained on a mass of unlabeled user behavior data, which enables it to extract users' general characteristics from their historical behaviors. Then the model is transferred to benefit various downstream tasks via fine-tuning. Motivated by the recent development in CV [3, 7, 14] and NLP [13, 23], several works [2, 8, 38] explore pre-training the user model with a contrastive learning task. They construct different views of the same behavior sequence via data augmentation and assume them to be semantically consistent, i.e., reflecting similar characteristics or interests of the user [27, 47]. Thus, these views are regarded as positive samples and a contrastive loss (e.g., InfoNCE [30]) is further applied to maximize their similarity in the feature space.
Although such pre-trained user models have proven to benefit various downstream tasks, we find that their assumption about semantic consistency between the augmented views does not necessarily hold. Unlike texts and images, user behaviors usually contain not only diverse interests but heavy noise as well [4, 26, 43]. Hence, existing data augmentation methods tend to result in augmented sequences that lose certain characteristics of the user or introduce noisy interests that the user does not have. To illustrate the impact of different augmentation methods on the behavior sequence, we randomly sample a user from the MSN online news platform<sup>2</sup>. His recent eight behaviors and the results of several data augmentation methods are shown in Fig. 1. From the original behavior sequence, we may infer the user's interests in space technology, politics, and basketball. Meanwhile, we find that various augmentation methods have distinct impacts on different behavior sequences. For example, the masked behavior sequence [47] loses the user's interest in politics due to randomness, and the two sequences augmented by cropping [38] reflect the user's different interests over time. Other methods such as mask-and-fill [2] and substitute [27] bring in behaviors related to football and tennis which the user may not be actually interested in. However, existing contrastive learning-based methods force the user model to maximize the agreement between such augmented views no matter whether they are similar or not, which may lead to a negative transfer for the downstream task.
To address this problem, in this paper, instead of trying to design a new data augmentation method that guarantees to generate consistent augmented behavior sequences, we escape the existing contrastive learning framework and propose a new pretext task: Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Since the augmented views may reflect distinct characteristics or interests of the user, instead of directly maximizing their agreement, we adopt a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users.
2More examples are provided in Appendix A.
Specifically, the user model is trained to project the original view closer to the implicitly augmented view than views from other users in the feature space, with the explicitly augmented view in between. We further utilize an in-batch hard negative sampling strategy to facilitate model training. In addition, since data augmentation has distinct impacts on different behavior sequences, we design an augmentation-adaptive fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the semantic similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks validate that AdaptSSR consistently outperforms existing pre-training methods while adapting to various data augmentation methods. Our code is available at https://github.com/yfly1613/AdaptSSR.
Contrastive Learning Aiming to learn high-quality representations in a self-supervised manner, contrastive learning has achieved great success in various domains [1, 6, 28, 41]. Its main idea is to learn a mapping function that brings positive samples closer together while pushing negative samples further apart in the feature space. One of the key issues for contrastive learning is how to construct semantically consistent positive samples. To tackle this problem, several data augmentation methods such as jittering, rotation, and multi-cropping have been proposed for images [3, 16, 29]. However, for user behavior sequences, due to the diverse user interests and heavy noise, it is hard to determine their semantics and even harder to provide a semantically consistent augmentation [25, 32, 49], which limits the effectiveness of existing contrastive learning-based user model pre-training methods.
User Model Pre-training Various methods [24, 53, 60] have been proposed to model users based on their behaviors and task-specific labeled data. However, these supervised methods usually encounter data sparsity issues [18, 36]. Inspired by the success of the pre-training paradigm in CV and NLP, several studies [15, 33, 46] have sought to empower various user-oriented tasks by pre-training the user model on unlabeled behavior sequences. Predominant methods can be categorized into two groups: generative and discriminative. Generative methods [45, 52] usually first corrupt the behavior sequence and then try to restore the masked part. However, they largely focus on mining the correlation between behaviors but lack holistic user representation learning, which limits their effectiveness in user modeling [8]. Recently, several discriminative methods [2, 38] have applied contrastive learning to pre-train the user model. They assume that different views of the same behavior sequence constructed via data augmentation are semantically similar and take them as the positive sample. In contrast, we propose a self-supervised ranking task that avoids directly maximizing the similarity between the augmented views and alleviates the requirement of their semantic consistency.
Data Augmentation for User Behavior Sequences In the domain of user modeling and sequential recommendation, numerous data augmentation methods [2, 8, 38, 47] have been proposed for user behavior sequences, which can be broadly categorized into two groups: explicit and implicit. Explicit augmentation methods, such as masking and cropping [27, 47], are performed at the data level, meaning that they directly modify the behavior sequence. On the other hand, implicit augmentation is performed at the model level. The same sequence is input into the model twice with different independently sampled dropout masks, which adds distinct noise in the feature space [8, 32]. Our method utilizes both implicitly and explicitly augmented views and trains the user model to capture their similarity order. Moreover, owing to the augmentation-adaptive fusion mechanism, our AdaptSSR can adapt to diverse data augmentation methods and consistently improve their effectiveness.
In this section, we introduce the details of our Augmentation-Adaptive Self-Supervised Ranking (AdaptSSR) task for user model pre-training. Fig. 2 illustrates the framework of AdaptSSR and the pseudo-codes for the entire pre-training procedure are provided in Appendix B.
3.1 Notations and Problem Statement
Suppose that we have collected a large number of user behavior data from a source domain, e.g., an online news platform or an e-commerce platform. Let  $\mathcal{U}$  and  $\mathcal{I}$  denote the set of users and different behaviors, where each behavior  $x\in \mathcal{I}$  can be a news article or an item. It should be noted
Figure 2: The framework of AdaptSSR. A sequence with five user behaviors is used for illustration.
that our methodology aligns with the prevalent settings in many previous works [2, 8, 47, 52] and real-world scenarios, where each behavior is simply represented by an ID. We refrain from using any ancillary information related to user behaviors (such as images or text descriptions). For each user  $u \in \mathcal{U}$ , we organize his/her behaviors in chronological order and denote the behavior sequence as  $S = \{x_{1}, x_{2}, \ldots, x_{n}\}$  ( $x_{i} \in \mathcal{I}$ ). Our goal is to pre-train a user model based on these behavior sequences. The output of the user model is a dense user representation  $\boldsymbol{u} \in \mathbb{R}^{d}$  reflecting the general characteristics of the user, where  $d$  is the embedding dimension. For each downstream task, a simple MLP is added and the entire model is fine-tuned with the labeled data of the target task.
3.2 Self-Supervised Ranking
As it is hard to perform a semantically consistent augmentation for the user behavior sequence, we avoid directly maximizing the agreement between the augmented views and propose a self-supervised ranking task that trains the user model to capture the similarity order between the implicitly augmented view, the explicitly augmented view, and views from others. Specifically, given a behavior sequence  $S$ , we first apply the implicit data augmentation, i.e., the same sequence is input into the user model twice with different independently sampled dropout masks, and denote the generated user representations as  $\mathbf{u}$  and  $\mathbf{u}^{+}$ . Then, we apply certain explicit data augmentation to  $S$  and denote the augmented sequence and its corresponding user representation as  $\hat{S}$  and  $\hat{\mathbf{u}}$ , respectively. Since  $\mathbf{u}$  differs from  $\mathbf{u}^{+}$  only because of the distinct dropout masks applied by the implicit augmentation, while the difference between  $\mathbf{u}$  and  $\hat{\mathbf{u}}$  is caused by both the implicit augmentation and the explicit augmentation which modifies  $S$  at the data level, we require them to satisfy the following similarity order:  $\mathrm{sim}(\mathbf{u},\mathbf{u}^{+})\geqslant \mathrm{sim}(\mathbf{u},\hat{\mathbf{u}})$ . Considering that the semantic consistency between  $S$  and  $\hat{S}$  cannot be guaranteed, our pre-training objective is to capture the following similarity order:
$$
\boldsymbol {\Gamma}: \operatorname {s i m} (\boldsymbol {u}, \boldsymbol {u} ^ {+}) \geqslant \operatorname {s i m} (\boldsymbol {u}, \hat {\boldsymbol {u}}) \geqslant \operatorname {s i m} (\boldsymbol {u}, \boldsymbol {u} ^ {-}), \tag {1}
$$
where  $\pmb{u}^{-}$  is the user representation of another user. Instead of directly forcing the user model to maximize  $\mathrm{sim}(\pmb{u},\hat{\pmb{u}})$  as existing contrastive learning-based methods, our objective leaves room for the user model to adjust  $\mathrm{sim}(\pmb{u},\hat{\pmb{u}})$  properly between the upper bound  $\mathrm{sim}(\pmb{u},\pmb{u}^{+})$  and the lower bound  $\mathrm{sim}(\pmb{u},\pmb{u}^{-})$  based on the similarity between the augmented views, which alleviates the requirement of their semantic consistency.
3.3 Implementation
User Model Our method is decoupled from the structure of the user model, allowing for the integration of various existing methods, such as the attention network [58] and the temporal convolutional network [51]. In this work, we choose the widely used Transformer Encoder (TrmEnc) [40] as the backbone of the user model due to its superior performance in sequence modeling [20, 37]. Given a user behavior sequence  $S = \{x_{1}, x_{2}, \ldots, x_{n}\}$ , each behavior  $x_{i} \in \mathcal{I}$  is projected into a dense vector  $\boldsymbol{x}_{i} \in \mathbb{R}^{d}$  via looking-up an embedding table  $\mathbf{E} \in \mathbb{R}^{|T| \times d}$ . Another learnable position embedding matrix  $\mathbf{P} = [p_{1}, p_{2}, \ldots, p_{n}] \in \mathbb{R}^{n \times d}$  is used to model the position information. An attention network with a learnable query vector  $q \in \mathbb{R}^{d}$  is further utilized to aggregate the output
hidden states  $\mathbf{H}$  of the Transformer Encoder. Thus, the user representation is calculated as follows:
$$
\mathbf {H} = \left[ h _ {1}, h _ {2} \dots , h _ {n} \right] = \operatorname {T r m E n c} \left(\left[ \boldsymbol {x} _ {1} + \boldsymbol {p} _ {1}, \boldsymbol {x} _ {2} + \boldsymbol {p} _ {2}, \dots , \boldsymbol {x} _ {n} + \boldsymbol {p} _ {n} \right]\right), \tag {2}
$$
$$
\alpha_ {i} = \frac {\exp \left(\boldsymbol {q} ^ {\top} \boldsymbol {h} _ {i}\right)}{\sum_ {j = 1} ^ {n} \exp \left(\boldsymbol {q} ^ {\top} \boldsymbol {h} _ {j}\right)}, \tag {3}
$$
$$
\boldsymbol {u} = \sum_ {i = 1} ^ {n} \alpha_ {i} \boldsymbol {h} _ {i}. \tag {4}
$$
Multiple Pairwise Ranking with In-batch Hard Negative Sampling Given a batch of user behavior sequences  $\{S_i\}_{i=1}^B$ , where  $B$  is the batch size, we first apply two randomly selected explicit augmentation methods to each  $S_i$  and denote the augmented sequence as  $\hat{S}_i$  and  $\tilde{S}_i$ . In this work, we adopt the three random augmentation operators proposed by Xie et al. [47]: mask, crop, and reorder. Notably, our method can be integrated with various existing data augmentation methods, which will be further explored in Section 4.3. Then, for a pair of  $\hat{S}_i$  and  $\tilde{S}_i$ , we input each of them into the user model twice with independently sampled dropout mask and denote the generated user representations as  $\hat{\boldsymbol{u}}_i$ ,  $\hat{\boldsymbol{u}}_i^+$  and  $\tilde{\boldsymbol{u}}_i$ ,  $\tilde{\boldsymbol{u}}_i^+$ , respectively. It should be noted that we only utilize the standard dropout mask in the Transformer Encoder, which is applied to both the attention probabilities and the output of each sub-layer. We treat all representations generated from views of other users in the same batch as negative samples, which are denoted as  $\mathbf{U}_i^- = \{\hat{\boldsymbol{u}}_j, \hat{\boldsymbol{u}}_j^+, \tilde{\boldsymbol{u}}_j, \tilde{\boldsymbol{u}}_j^+\}_{j=1,j \neq i}$ . Inspired by previous works in learning to rank, we adopt a Multiple Pairwise Ranking (MPR) loss [48] to train the user model to capture the similarity order  $\Gamma$ , which extends the BPR loss [34] to learn two pairwise ranking orders simultaneously. For the user representation  $\hat{\boldsymbol{u}}_i$  and  $\hat{\boldsymbol{u}}_i^+$ , each  $\boldsymbol{v} \in \{\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+\}$  and  $\boldsymbol{w} \in \mathbf{U}_i^-$  form a quadruple for model training, and the loss function for  $\hat{S}_i$  is formulated as follows:
$$
\begin{array}{l} \hat {\mathcal {L}} _ {i} = - \frac {1}{2 \left| \mathbf {U} _ {i} ^ {-} \right|} \sum_ {\boldsymbol {v} \in \{\tilde {\boldsymbol {u}} _ {i}, \tilde {\boldsymbol {u}} _ {i} ^ {+} \}} \sum_ {\boldsymbol {w} \in \mathbf {U} _ {i} ^ {-}} \log \sigma \left[ \lambda \left(\sin \left(\hat {\boldsymbol {u}} _ {i}, \hat {\boldsymbol {u}} _ {i} ^ {+}\right) - \sin \left(\hat {\boldsymbol {u}} _ {i}, \boldsymbol {v}\right)\right) \right. \tag {5} \\ \left. + (1 - \lambda) (\operatorname {s i m} (\hat {\boldsymbol {u}} _ {i}, \boldsymbol {v}) - \operatorname {s i m} (\hat {\boldsymbol {u}} _ {i}, \boldsymbol {w})) \right], \\ \end{array}
$$
where  $\sigma(\cdot)$  is the Sigmoid function and  $\mathrm{sim}(\cdot)$  calculates the cosine similarity between two vectors.  $\lambda \in [0,1]$  is a trade-off hyper-parameter for fusing the two pairwise ranking orders:  $\mathrm{sim}(\hat{\pmb{u}}_i,\hat{\pmb{u}}_i^+) \geqslant \mathrm{sim}(\hat{\pmb{u}}_i,\pmb{v})$  and  $\mathrm{sim}(\hat{\pmb{u}}_i,\pmb{v}) \geqslant \mathrm{sim}(\hat{\pmb{u}}_i,\pmb{w})$ .
In Eq. (5), each quadruple is treated equally. Inspired by recent studies [11, 44, 55] on the impact of hard negative samples for contrastive learning, we further design an in-batch hard negative sampling strategy to facilitate model training. For each pairwise ranking order, we select the pair with the smallest similarity difference for training, i.e., the loss function is formulated as follows:
$$
\begin{array}{l} \hat {\mathcal {L}} _ {i} = - \log \sigma \left[ \lambda \left(\operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i}, \hat {\boldsymbol {u}} _ {i} ^ {+}\right) - \max  _ {\boldsymbol {v} \in \{\tilde {\boldsymbol {u}} _ {i}, \tilde {\boldsymbol {u}} _ {i} ^ {+} \}} \operatorname {s i m} \left(\hat {\boldsymbol {u}} _ {i}, \boldsymbol {v}\right)\right) \right. \tag {6} \\ \left. + (1 - \lambda) \left(\min  _ {\boldsymbol {v} \in \{\tilde {\boldsymbol {u}} _ {i}, \tilde {\boldsymbol {u}} _ {i} ^ {+} \}} \sin (\hat {\boldsymbol {u}} _ {i}, \boldsymbol {v}) - \max  _ {\boldsymbol {w} \in \mathbf {U} _ {i} ^ {-}} \sin (\hat {\boldsymbol {u}} _ {i}, \boldsymbol {w})\right) \right]. \\ \end{array}
$$
Such pairs are currently the closest samples that the model needs to discriminate, which can provide the most significant gradient information during training. For another augmented sequence  $\tilde{S}_i$ , the loss function  $\tilde{\mathcal{L}}_i$  is symmetrically defined and the overall loss is computed as  $\mathcal{L} = \sum_{i=1}^{B} (\hat{\mathcal{L}}_i + \tilde{\mathcal{L}}_i) / 2B$ .
Augmentation-Adaptive Fusion With the loss function  $\hat{\mathcal{L}}_i$ , the user model is trained to capture two pairwise ranking orders:  $\mathrm{sim}(\hat{\boldsymbol{u}}_i,\hat{\boldsymbol{u}}_i^+)\geqslant \mathrm{sim}(\hat{\boldsymbol{u}}_i,\boldsymbol {v})$  and  $\mathrm{sim}(\hat{\boldsymbol{u}}_i,\boldsymbol {v})\geqslant \mathrm{sim}(\hat{\boldsymbol{u}}_i,\boldsymbol {w})$  simultaneously. However, we further need to combine them properly via the hyper-parameter  $\lambda$ . As illustrated in Fig. 1 and Appendix A, the effects of data augmentation vary significantly across different behavior sequences. Consequently, a fixed and unified  $\lambda$  is not enough to fuse the two pairwise ranking orders properly for different samples. Thus, we further develop an augmentation-adaptive fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the similarity between the augmented views. Specifically, we estimate the semantic similarity between the explicitly augmented views with the average similarity between the user representations generated from  $\hat{S}_i$  and
$\tilde{S}_i$ . Then we replace the hyper-parameter  $\lambda$  in Equation (6) with a dynamic coefficient  $\lambda_{i}$  for each training sample  $S_{i}$ , which is calculated along the training procedure as follows:
$$
\lambda_ {i} = 1 - \frac {1}{4} \sum_ {\hat {\boldsymbol {s}} \in \left\{\hat {\boldsymbol {u}} _ {i}, \hat {\boldsymbol {u}} _ {i} ^ {+} \right\}} \sum_ {\tilde {\boldsymbol {s}} \in \left\{\tilde {\boldsymbol {u}} _ {i}, \tilde {\boldsymbol {u}} _ {i} ^ {+} \right\}} \max  (\sin (\hat {\boldsymbol {s}}, \tilde {\boldsymbol {s}}), 0). \tag {7}
$$
To get an accurate similarity estimation at the beginning of the training, we train the user model with the MLM task [52] until convergence before applying our self-supervised ranking task. If  $\hat{S}_i$  and  $\tilde{S}_i$  are semantically similar,  $\lambda_i$  will be small and set the loss function  $\hat{\mathcal{L}}_i$  focusing on maximizing the latter term  $\min_{\boldsymbol{v} \in \{\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+\}} \mathrm{sim}(\hat{\boldsymbol{u}}_i, \boldsymbol{v}) - \max_{\boldsymbol{w} \in \mathbf{U}_i^-} \mathrm{sim}(\hat{\boldsymbol{u}}_i, \boldsymbol{w})$ , which forces the user model to discriminate these similar explicitly augmented views from views of other users. Otherwise,  $\lambda_i$  will be large and train the user model to pull the implicitly augmented view and these dissimilar explicitly augmented views apart. As a result, the user model is trained to adaptively adjust  $\mathrm{sim}(\hat{\boldsymbol{u}}_i, \boldsymbol{v})$  when combining the two learned pairwise ranking orders for each sample, which can better deal with the distinct impacts of data augmentation on different behavior sequences.
3.4 Discussion
In this subsection, we discuss the connection between our proposed self-supervised ranking task and existing contrastive learning-based methods.
If we set  $\lambda_{i}\equiv 0$  for all training samples and input each behavior sequence into the user model once (i.e., do not apply the implicit data augmentation), our loss function  $\hat{\mathcal{L}}_i$  degenerates as follows: