SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, we introduce  $L2$ -norm into the training objective to regularize the parameters in the model to address the overfitting problem and improve generalization ability. Therefore, the overall objective function can be formulated as:
$$
\mathcal {L} = \sum_ {i \in \mathcal {U}} \mathcal {L} _ {\text {r e c}} ^ {i} + \lambda | | \Theta | | _ {2} ^ {2}. \tag {10}
$$
2.5 Summary
After the training process ends, the model can be used to conduct inference. For the inference phase, we do not conduct pre-train and prompt-tune again. What we need to do is to extract target user and item embeddings from the trained embedding tables. Then, we calculate their inner product to predict the probability that the user will interact with the item in the future.
The complete training procedure of CPTPP is illustrated by Algorithm 1. We first initialize the user and item embedding tables (line 2). Then, we apply a GCL model to conducting embedding pre-training (line  $4\sim 7$ ). Next, we step into the prompt-tuning phase and assign the pre-trained embeddings to  $\mathbf{U}_E^*$  and  $\mathbf{I}_E^*$  (line 9). Following, we input the user profile to the prompt generator to produce the personalized prompts (line 12) and combine them with  $\mathbf{U}_E^*$  (line 13). Finally, we use  $\mathbf{U}_E^*$  and  $\mathbf{I}_E^*$  to calculate the loss and update them accordingly (line  $14\sim 15$ ). The update procedure will repeat until the termination condition is achieved (line  $11\sim 17$ ).
3 Experiment
To verify the effectiveness of the proposed method, CPTPP, in this paper, we conduct extensive experiments and demonstrate the results with insightful analysis in this section.
Table 1: Dataset Statistics
Dataset	#Users	#Items	#Interactions	Density
Douban	2,848	39,586	894,887	0.794%
ML-1M	6,040	3,900	1,000,209	4.246%
Gowalla	29,858	40,981	1,027,370	0.084%
This section introduces the experimental settings, including datasets and baselines we used, performance metrics, and hyper-parameter settings for CPTPP. More details about how to get access to the datasets and implementation details are listed in Appendix A.
Datasets To verify the performance of CPTPP in the recommendation task, we select three popular datasets: Douban [34], MovieLens-1M [5], and Gowalla [13]. The detailed statistics about the three datasets are listed in Table 1. For each dataset, we randomly select  $80\%$  of historical user-item interactions as the training set, and the rest  $20\%$  records will serve as the testing set. Following the settings widely adopted by the research community [22, 6], we treat each user-item interaction record as a positive instance and conduct negative sampling to couple it with a negative instance, which is an unobserved user-item interaction in the dataset.
Baselines We select several baselines for comparison experiments: BPR-MF [9], BUIR [10], SelfCF [36], NCL [14], and SimGCL [32]. For CPTPP, we have three variations, which are CPTPP-H, CPTPP-M, and CPTPP-R, respectively. - $H$  takes historical interaction records for personalized prompt generation. - $M$  indicates that we take adjacency matrix factorization for personalized prompts generation. Furthermore, - $R$  takes high-order user relations for the personalized prompt generation.
Metrics To evaluate the quality of top- $K$  recommendation, we adopt three popular metrics, which are Hit Ratio@  $K$ , Precision@  $K$ , and NDCG@  $K$ , respectively. In our settings, the value of  $K$  is set to 5 and 20. Following the evaluation protocol in [14, 32], we take the full ranking strategy [35].
Hyper-parameter Settings To ensure reproducibility, we disclose the comprehensive hyperparameter settings for implementing our proposed CPTPP in the source codes and Appendix A.3.
3.2 Experiment Results
We conduct experiments and provide analysis in this section. More supplementary experiment results, including comparison, hyper-parameter, and ablation study, are revealed in Appendix B.
Table 2: The experiment results of comparison studies. The figures in boldface indicate the best performance achieved by one of the three versions of CPTPP, and the figures underlined indicate the best performance among all the baselines.
Datasets	Metrics	Methods
BPR-MF	BUIR	SelfCF	NCL	SimGCL	CPTPP-H	CPTPP-M	CPTPP-R
Douban	Hit Ratio@5	0.0134	0.0156	0.0161	0.0161	0.0161	0.0164	0.0165*	0.0164
Hit Ratio@20	0.0446	0.0492	0.0502	0.0507	0.0489	0.0521	0.0528*	0.0523
Precision@5	0.1812	0.2113	0.2185	0.2187	0.2182	0.2221	0.2235*	0.2224
Precision@20	0.1512	0.1667	0.1699	0.1717	0.1657	0.1766	0.1790*	0.1772
NDCG@5	0.1904	0.2209	0.2264	0.2313	0.2370	0.2359	0.2378*	0.2355
NDCG@20	0.1749	0.2019	0.2058	0.1958	0.2020	0.2065	0.2098*	0.2070
ML-1M	Hit Ratio@5	0.0469	0.0617	0.0624	0.0655	0.0631	0.0676*	0.0674	0.0672
Hit Ratio@20	0.1454	0.1519	0.1643	0.1796	0.1698	0.1851	0.1861*	0.1845
Precision@5	0.1800	0.2368	0.2396	0.2513	0.2420	0.2592*	0.2585	0.2577
Precision@20	0.1395	0.1457	0.1576	0.1723	0.1629	0.1776	0.1785*	0.1770
NDCG@5	0.1968	0.2722	0.2689	0.2818	0.2767	0.2919*	0.2895	0.2878
NDCG@20	0.2103	0.2367	0.2508	0.2683	0.2670	0.2781	0.2782*	0.2756
Gowalla	Hit Ratio@5	0.0429	0.0479	0.0497	0.0488	0.0513	0.0518	0.0512	0.0519*
Hit Ratio@20	0.1039	0.0993	0.1042	0.1040	0.1065	0.1115	0.1103	0.1120*
Precision@5	0.0624	0.0698	0.0723	0.0711	0.0746	0.0754	0.0745	0.0755*
Precision@20	0.0378	0.0361	0.0379	0.0378	0.0387	0.0406	0.0401	0.0407*
NDCG@5	0.0770	0.0911	0.0939	0.0894	0.0963	0.0963	0.0953	0.0961
NDCG@20	0.0939	0.0990	0.1036	0.1005	0.1126	0.1092	0.1083	0.1092
“*” indicates that CPTPP outperforms the best baseline significantly (i.e., two-sided t-test with  $p < 0.05$ ).
3.2.1 Overall Comparison Studies
Table 2 shows the comparison results among all the baselines and different versions of the proposed methods. (i) We can first observe that the traditional method BPR-MF is outperformed by all the other methods as they utilize contrastive learning to introduce extra unsupervised training signals. (ii) Among all the baselines, GCL-based recommendation methods, including NCL and SimGCL, significantly and consistently outperform those self-supervised recommendation methods without graph learning module equipped, BUIR and SelfCF. It is because those GCL-based methods adopt graph neural networks, leveraging the sophisticated structure semantics in user-item interaction graphs to enrich learned user embeddings and item embeddings. (iii) But we notice that SimGCL only outperforms NCL on dataset Gowalla, which has a much larger scale than the others, probably because SimGCL adopts a simplified GCL method that relieves the model overfitting problem on a large-scale dataset. It is the potential reason NCL outperforms SimGCL on smaller datasets, as the simplified GCL method may not provide sufficient self-supervised training signals. (iv) Though the proposed CPTPP solely adopts BPR loss, which is significantly different from the pre-training procedure, for the recommendation task training, we utilize the prompt learning mechanism to better adapt the embeddings pre-trained by the GCL method to the downstream task, expecting to improve the recommendation performance. According to the experiment results, all versions of our proposed method achieve competitive results. Such results reflect prompt-tuning's effectiveness in narrowing the gap between the pre-train objective and the downstream tasks.
To further evaluate the performance of the GCL-based recommendation methods, we visualize the produced user embeddings produced by t-SNE and Gaussian kernel density estimation (KDE). We can see that CPTPP has a more uniform distribution of the produced user embeddings, illustrated by the uniformity of the color maps, especially on dataset ML-1M and Gowalla. As suggested by Z. Lin et al. [14], the more uniform the embedding distribution is, the more powerful the capability to model the diverse preferences of users the produced embeddings will have, which reflects the superiority of CPTPP compared to the baselines. The visualizations and analysis are listed in Appendix B.1.
(a) CPTPP - Douban
(b) CPTPP - ML-1M
Figure 2: The visualizations of the user embeddings generated by the proposed method.
(c) CPTPP - Gowalla
3.2.2 Hyper-Parameter Studies
To investigate the properties of our proposed CPTPP method, we conduct hyper-parameter studies on an important term, the dimension size of the personalized prompt. By fixing all the other hyperparameters, we comprehensively examine the performance of three versions of the proposed CPTPP on all the datasets with different prompt sizes. Specifically, the size of the personalized prompt is selected from  $\{8,16,32,64,128,256\}$ . We choose two metrics, Precision@5 and NDCG@5, to demonstrate CPTPP's performance variations with regard to different prompt sizes. All the experiment results are shown in Figure 3 and Figure 5 in Appendix B.2. (i) The first finding we can observe is that, in most cases, CPTPP has the best performance when the prompt size is not larger than the dimensionality of user embeddings, i.e., 64. A potential reason is that the prompt is usually less informative than the pre-trained embeddings, so a sizeable prompt dimension would introduce too much noise to disturb and conceal the structural semantics contained in the pre-trained user embeddings. (ii) We also notice a significant performance improvement when prompt size is 256 in several cases, such as CPTPP-M on dataset ML-1M and CPTPP-R on dataset Gowalla. Such outlier performance could be caused by random factors during the overall training process. However, they still fail to significantly outperform the CPTPP model, which has a much smaller prompt size. Therefore, a small prompt size for prompt-tuning is a better option in practice as they achieve a relatively better recommendation quality and higher efficiency.
(a) CPTPP-H - Douban
Figure 3: The performance, demonstrated by Precision@5 and NDCG@5, of all variations of CPTPP.
(b) CPTPP-H - ML-1M
(c) CPTPP-H - Gowalla
3.2.3 Ablation Studies
As we summarize three strategies to generate personalized prompts for users, we conduct the ablation study to explore the differences among these methods. Two ablation studies are conducted in this section to illustrate the performance of three variations of the CPTPP method. The first ablation study is about the overall evaluation of recommendation quality, whose analysis is listed below. The second one is about the embedding visualizations and the related analysis, illustrated in Appendix B.3.
We notice that (i) CPTPP-M achieves the best performance on dataset Douban. Nevertheless, the performance of CPTPP-M degrades on dataset ML-1M and is the worst case on dataset Gowalla. Considering the number of users reflected in Table 1, we find that the performance of CPTPP-M drops as the dataset's number of users increases. So, CPTPP-M has good performance if the number of users in the dataset is relatively small. It may be because matrix factorization, as a naive method, cannot
fully reveal user preferences in a complex user-item interaction graph with too many user nodes. (ii) CPTPP-R utilizes high-order relationships among users to enrich the generated personalized prompts for users. In such settings, the item information would also be aggregated due to the message-passing mechanism in GNNs. Therefore, it achieves the best performance on the dataset Gowalla, having the most users and the most complex user-user relation among all the datasets. (iii) CPTPP-H has moderate performance. CPTPP-H adopts historical interaction records, formed by trainable item embeddings, to generate personalized prompts. Those trainable elements endow CPTPP-H with a more robust capability to represent user preferences than matrix factorization. It is also reasonable that CPTPP-R outperforms CPTPP-H as CPTPP-H lacks consideration of high-order user relations.
4.1 GCL in Recommendation Systems
The GCL-based recommendation system is now a trending topic in the research community. Researchers leverage the advantages of GCL to improve current graph-based recommendation methods further, achieving satisfying performance. The research scope first lies in utilizing user-item interaction records to accommodate GCL techniques. For example, HMG-CR [27] innovatively proposes a concept called hyper-meta path. Then GCL is applied to adaptively learn the behavioural patterns of users via contrasting different paths. NCL [14] improves graph-based recommendation models via neighborhood-enriched contrastive learning, including semantic neighbors and structural neighbors (r-ego graphs). To better understand the role of GCL in the recommendation systems, SGL [24] conducts a comprehensive theoretical analysis, giving insights and achieving promising performance. Based on the findings of SGL, SimGCL [32] is proposed to simplify GCL in recommendation via discarding complex augmentations, reducing the volume of GCL-based recommendation models, and performing competitive results in the experiments. However, current GCL-based recommendation methods combine the graph contrastive loss with the recommendation objectives to formulate an overall objective to train the models, suffering from various limitations summarised in the previous sections. Our method innovatively introduces the prompt mechanism to build a pre-train then prompt-tune paradigm for GCL-based recommender systems to address the limitations we discussed.
4.2 Prompt-Tuning
Prompt-tuning is a novel and trending paradigm for pre-train models in the natural language process (NLP). The core idea of prompt-tuning is to re-formulate the downstream tasks, narrowing the huge gap between them and the pre-train objective [1, 18]. There are two types of methods to achieve prompt-tuning [3]. The first is manually designing or searching for proper discrete prompts (hard prompts) [2, 8, 18]. However, such a fashion is trivial and resource-consuming as the search space is extremely large and expert knowledge is required [26] in some application scenarios. To address this limitation, another line of methods is proposed, focusing on generating continuous vector embeddings as the soft prompts [4, 15]. The application of prompt-tuning in recommendation systems has been explored. For example, P5 [3] reforms the recommendation tasks to the NLP tasks and follows the hard-prompt fashion to perform recommendations. PPR [26], instead, takes a soft-prompt and pre-fix strategy [12] to generate personalized prompts for users in the recommendation systems automatically. Nevertheless, graph learning and its applications are now out of the scope of prompt-tuning research. Besides, most current prompt learning methods require side information to produce high-quality prompts, resulting in a limited comfort zone for its applications. Our CPTPP first adopts the prompt mechanism to the GCL-based recommendation area.
In this paper, we propose a CPTPP method to adopt a prompt-tuning technique to reform and improve current GCL-based recommendation methods. To better accommodate prompt learning to graph recommendation scenarios, we summarise several graph-oriented user profiles to generate personalized user prompts to conduct prompt-tuning for downstream recommendation tasks. Comprehensive experiments have shown the effectiveness, superiority, and properties of the proposed CPTPP method. The future research directions about prompt-tuning in GCL-based recommendation may be two-fold: how to (i) generate personalized prompts and (ii) integrate prompt-tuning strategy into GCL protocols.
This research work was supported by the Research Impact Fund (No. R1015-23), the Australian Research Council (ARC) under Grant Nos. DP220103717, LE220100078, LP170100891, and DP200101374 and was partially supported by APRC - CityU New Research Initiatives (No. 9610565, Start-up Grant for New Faculty of City University of Hong Kong), CityU - HKIDS Early Career Research Grant (No. 9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No. ITS/034/22MS), Hong Kong Environmental and Conservation Fund (No. 88/2022), SIRG - CityU Strategic Interdisciplinary Research Grant (No.7020046, No. 7020074), Tencent (CCF-Tencent Open Fund, Tencent Rhino-Bird Focused Research Fund), Huawei (Huawei Innovation Research Program), Ant Group (CCF-Ant Research Fund, Ant Group Research Fund) and Kuaishou.
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[2] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online, August 2021. Association for Computational Linguistics.
[3] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 299-315, New York, NY, USA, 2022. Association for Computing Machinery.
[4] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410-8423, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[5] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4), dec 2015.
[6] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 639-648, New York, NY, USA, 2020. Association for Computing Machinery.
[7] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 173-182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.
[8] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438, 07 2020.
[9] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
A Reproducibility
This section provides supplementary details about our experimental settings for reproducibility.
A.1 Datasets
Three publicly available datasets are used in this work to examine the performance of the proposed CPTPP. Here, we provide the links for downloading these datasets for readers to retrieve:
- Douban: https://pan.baidu.com/s/1hrJP6rq#1ist/path=%2F
- ML-1M: https://grouplens.org/datasets/movielens/1m/
Gowalla: https://github.com/kuandeng/LightGCN/tree/master/Data/gowalla
A.2 Baselines
We select several baselines for comparison experiments, which are listed below:
- BPR-MF [9] adopts a matrix factorization framework to learn embeddings for users and items via optimizing the BPR loss function.
- BUIR [10] only uses positive user-item interactions to learn representations following a bootstrapped manner, consisting of an online encoder and a target encoder.
- SelfCF [36] follows a similar strategy that BUIR adopts, which drops the momentum encoder to simplify the previously proposed method.
- NCL [14] utilizes neighbor clustering to enhance GCL methods to acquire enhanced embeddings for users and items in the recommendation system.
- SimGCL [32] discusses the role of augmentations in GCL for recommendation tasks and proposes a simplified GCL method for recommendations.
A.3 Hyper-parameter Settings
Table 3: Summary of hyper-parameter settings of CPTPP.
Hyper-parameter	Notation	Dataset
Douban	ML-1M	Gowalla
Hidden dimension size	d	64	64	64
Pre-train epoch	-	10	10	10
Prompt-tune epoch	-	100	100	100
Batch size	-	512	512	2048
Learning rate	-	0.003	0.001	0.001
Regularizer weight	λ	0.0001	0.0001	0.0001
Number of GNN layers	-	2	2	2
Dropout rate	-	0.1	0.1	0.1
Temperature parameter	τ	0.2	0.2	0.2
Prompt size	p	{8, 16, 32, 64, 128, 256}	{8, 16, 32, 64, 128, 256}	{8, 16, 32, 64, 128, 256}
We list detailed hyper-parameter settings here for reproducibility. The dimensionality of the representation embeddings of users and items is set to 64, and the personalized prompt size is chosen from  $\{8,16,32,64,128\}$ . For the pre-train phase, the maximum training epoch is 10, and for the prompt-tune stage, the training epoch is set to 100. The training batch size is 512 for the relatively smaller datasets, including Douban and ML-1M. For Gowalla, it is set to 2048. The learning rate and  $\lambda$  are set to  $1e^{-3}$  and  $1e^{-4}$ , where  $\lambda$  is the weight for the  $l2$ -norm term in the overall training objective. The default number of layers of graph neural networks used in the models is set to 2. These settings are summarised in Table 3.
(a) NCL - Douban
(b) SimGCL - Douban
(c) NCL - ML-1M
(d) SimGCL - ML-1M
(e) NCL - Gowalla
(f) SimGCL - Gowalla
Figure 4: The visualization results of the user embeddings generated by baselines.
B Supplementary Experiment
In this section, several supplementary experiments are provided. Due to the page limit, the supplementary experiment results are listed and analyzed in the following instead of the main content.
(a) CPTPP-M - Douban
(b) CPTPP-M - ML-1M
(c) CPTPP-M - Gowalla
(d) CPTPP-R - Douban
Figure 5: The performance, demonstrated by the metrics Precision@5 and NDCG@5, of CPTPP-M and CPTPP-R on the selected datasets.
(e) CPTPP-R - ML-1M
(f) CPTPP-R - Gowalla
B.1 Supplementary Comparison Study
According to Figure 4, We can see that (i) embeddings learned by SimGCL fall into several hot areas on dataset ML-1M, and they are centralized in a small area on datasets Douban and Gowalla. (ii) NCL exhibits better performance as the distribution of the user embeddings expands to a relatively larger area than that of SimGCL. Compared to our proposed method CPTPP, we can observe that CPTPP has a more uniform distribution of the produced user embeddings, illustrated by the uniformity of the
color maps, especially on dataset ML-1M and Gowalla. As suggested in [14], the more uniform the embedding distribution is, the more capability to model the diverse preferences of users the method has, which reflects CPTPP's superiority.
B.2 Supplementary Hyper-Parameter Study
In Section 3.2.2, we solely illustrate the performance of CPTPP-H with different prompt sizes on the three datasets. We show the rest of the hyper-parameter study results in Figure 5. For easy reading, we list our findings here again: (i) The first thing we can observe is that, in most cases, CPTPP has the best performance when the prompt size is not larger than the dimensionality of user embeddings. A potential reason is that sizeable prompt dimensions would introduce more noise into pre-trained user embeddings, disturbing the structural semantics extracted from the user-item interaction graph by graph contrastive learning. (ii) We also notice a significant performance improvement when prompt size is 256 in several cases, such as CPTPP-M on dataset ML-1M and CPTPP-R on dataset Gowalla. However, they still fail to significantly outperform the CPTPP model, which has a much smaller prompt size. Therefore, a small prompt size for prompt-tuning is a better option in practice as they achieve a relatively good recommendation quality and higher efficiency.
(a) CPTPP-H - Douban
(b) CPTPP-M - Douban
(c) CPTPP-R - Douban
(d) CPTPP-H - ML-1M