SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Pre-training Procedure with AdaptSSR
Input: A corpus of user behavior sequences  $S$  and a set of data augmentation operators  $\mathcal{A}$ .  
Output: The pre-trained user model  $\mathcal{M}$ .  
1: Train the user model  $\mathcal{M}$  with the MLM task on  $S$  until convergence.  
2: while not converged do  
3: Randomly sample a batch of user behavior sequences  $\{S_i\}_{i=1}^B$  from  $S$ .  
4: for each  $S_i$  do  
5: Randomly select two augmentation operators  $f$  and  $g$  from  $\mathcal{A}$ .  
6:  $\hat{\boldsymbol{u}}_i, \hat{\boldsymbol{u}}_i^+ \gets \mathcal{M}(f(S_i)), \mathcal{M}(f(S_i))$ . ▷ With independently sampled dropout masks.  
7:  $\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+ \gets \mathcal{M}(g(S_i)), \mathcal{M}(g(S_i))$ . ▷ With independently sampled dropout masks.  
8:  $\mathbf{U}_i^- \gets \{\hat{\boldsymbol{u}}_j, \hat{\boldsymbol{u}}_j^+, \tilde{\boldsymbol{u}}_j, \tilde{\boldsymbol{u}}_j^+\}_{j=1,j\neq i}$ .  
9:  $\lambda_i \gets 1 - \frac{1}{4}\sum_{\hat{s} \in \{\hat{\boldsymbol{u}}_i, \hat{\boldsymbol{u}}_i^+\}}\sum_{\tilde{s} \in \{\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+\}}\max(\sin(\hat{s},\tilde{s}),0)$ .  
10:  $\hat{\mathcal{L}}_i \gets -\log\sigma\left[\lambda_i\left(\sin(\hat{\boldsymbol{u}}_i, \hat{\boldsymbol{u}}_i^+)\right) - \max_{\boldsymbol{v} \in \{\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+\}}\sin(\hat{\boldsymbol{u}}_i,\boldsymbol{v})\right)$   
+ (1 - λi) (minv∈{ui,ui} sim(ut,v) - maxw∈U-sim(ut,w)).  
11:  $\tilde{\mathcal{L}}_i \gets -\log\sigma\left[\lambda_i\left(\sin(\tilde{\boldsymbol{u}}_i, \tilde{\boldsymbol{u}}_i^+)\right) - \max_{\boldsymbol{v} \in \{\hat{\boldsymbol{u}}_i, \hat{\boldsymbol{u}}_i^+\}}\sin(\tilde{\boldsymbol{u}}_i,\boldsymbol{v})\right)$   
+ (1 - λi) (minv∈{ui,ui} sim(ut,v) - maxw∈U-sim(ut,w)).  
12: end for  
13: L← ∑i=1B (L̂i + L̂i)/2B.  
14: Update the parameters of the user model M with L by backpropagation.  
15: end while
In our experiments, the embedding dimension  $d$  is set as 64. In the Transformer Encoder, the number of attention heads and layers are both set as 2. We only utilize the standard dropout mask, which is applied to both the attention probabilities and the output of each sub-layer, and the dropout probability is set as 0.1. The maximum sequence length is set to 100 and 256 for the TTL dataset and the App dataset, respectively. For each downstream task, a two-layer MLP is added to the pre-trained user model, and the dimension of the intermediate layer is also set as 64.
For existing pre-training methods, the data augmentation proportion  $\rho$  is either copied from previous works if provided or searched from  $\{0.1, 0.2, \dots, 0.9\}$ . In our AdaptSSR, we adopt three random augmentation operators by default: mask, crop, and reorder. The augmentation proportion is set to 0.6, 0.4, and 0.6 respectively while our experimental results in Fig. 3 have shown that our method is robust to various data augmentation methods with different strengths.
We use the Adam optimizer for model training. The batch size and learning rate are set as 128 and 2e-4 for both pre-training and fine-tuning. The test results for all the models are reported at their best validation epoch. We repeat each experiment 5 times with different random seeds and report
Figure 7: The effectiveness of AdaptSSR when combined with existing pre-training methods on the downstream thumb-up recommendation task (left) and CVR prediction task (right).
the average results with standard deviations. We implement all experiments with Python 3.8.13 and Pytorch 1.12.1 on an NVIDIA Tesla V100 GPU.
D More Experimental Results
Fig. 7 shows the performance of our AdaptSSR when combined with several existing pre-training methods: CL4SRec, CoSeRec, and CCL, on the downstream thumb-up recommendation task  $(\mathcal{T}_4)$  and CVR prediction task  $(\mathcal{T}_6)$  with the data augmentation proportion  $\rho$  varying from 0.1 to 0.9. Similar to the results on the age prediction task  $(\mathcal{T}_1)$ , we find that these contrastive learning-based methods are highly sensitive to the data augmentation proportion. A too-small or too-large value of  $\rho$  will lead to limited performance gain or even negative transfer for the downstream task. Contrarily, our AdaptSSR consistently boosts the effectiveness of these pre-training methods by a large margin. This is because our self-supervised ranking task avoids directly maximizing the similarity between the augmented views. Moreover, our AdaptSSR also substantially improves the robustness of these pre-training methods to the augmentation proportion, which verifies the effectiveness of our augmentation-adaptive fusion mechanism. The dynamic coefficient  $\lambda_{i}$  adjusts the similarity order constraint applied to each sample based on the similarity between the augmented views, thus empowering the pre-training methods to adapt to diverse data augmentation methods with varying strengths.
E Limitations
In this work, after pre-training the user model with our AdaptSSR, the entire model is fine-tuned with the downstream labeled data. However, fine-tuning such a large model for each downstream task can be time and space-consuming. A potential solution for this problem is Parameter-Efficient Fine-Tuning (PEFT). Several methods such as Adapter and LoRA have been demonstrated to be effective for adapting the large language model to various downstream tasks by only updating a small fraction of parameters. However, we empirically find that these methods perform poorly when applied to the pre-trained user model. A possible reason is that the currently widely used Transformer-based user model is much shallower and thinner than the large language model. Most of the model parameters belong to the bottom behavior embedding table, while the upper Transformer blocks only contain very few parameters. Since existing PEFT methods mainly focus on tuning parts of parameters in each Transformer block, only a very small fraction of parameters in total will be updated, which may not be enough to effectively adapt the user model to the downstream task. We will investigate how to transfer the pre-trained user model to various downstream tasks parameter-efficiently while maintaining the performance gain brought by AdaptSSR in our future work.