SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoran Yang
University of Technology Sydney haoran.yang-2@student.uts.edu.au
Xiangyu Zhao*
City University of Hong Kong xianzhao@cityu.edu.hk
Yicong Li
University of Technology Sydney yicong.li@student.uts.edu.au
Hongxu Chen
University of Queensland hongxu.chen@uq.edu.au
Guandong Xu*
University of Technology Sydney
guandong.xu@uts.edu.au
Graph contrastive learning (GCL) has emerged as an effective technology for various graph learning tasks. It has been successfully applied in real-world recommender systems, where the contrastive loss and downstream recommendation objectives are combined to form the overall objective function. However, this approach deviates from the original GCL paradigm, which pre-trains graph embeddings without involving downstream training objectives. In this paper, we propose a novel framework called CPTPP, which enhances GCL-based recommender systems by leveraging prompt tuning. This framework allows us to fully exploit the advantages of the original GCL protocol. Specifically, we first summarize user profiles in graph recommender systems to automatically generate personalized user prompts. These prompts are then combined with pre-trained user embeddings for prompt tuning in downstream tasks. This helps bridge the gap between pre-training and downstream tasks. Our extensive experiments on three benchmark datasets confirm the effectiveness of CPTPP compared to state-of-the-art baselines. Additionally, a visualization experiment illustrates that user embeddings generated by CPTPP have a more uniform distribution, indicating improved modeling capability for user preferences. The implementation code is available online $^2$  for reproducibility.
Graph contrastive learning (GCL) has gained significant attention in the research community as a prominent self-supervised learning paradigm. Several recent studies have showcased the effectiveness of GCL in various general graph representation tasks [21, 16, 28, 37, 30], including node classification and link prediction. Moreover, GCL has also demonstrated its applicability in real-world domains[29], such as recommender systems [27, 32, 14]. By introducing additional self-supervision signals, GCL provides recommender systems with a means to address the challenge of improving performance.
Most recommendation methods based on GCL typically combine contrastive loss with recommendation objectives to optimize the model in an end-to-end manner. However, this training protocol does
$^{2}$ https://github.com/Haoran-Young/CPTPP
not align with the purpose of GCL, which is primarily designed for pre-training graph representations without involving downstream task objectives [21, 16]. In this approach, GCL pre-trains embeddings that are then fine-tuned on specific tasks using downstream models. Incorporating both GCL and recommendation objectives into the overall training objective can disrupt the embedding pre-training process and requires careful control of the weight placed on contrastive loss. Additionally, previous studies on GCL-based recommendation methods [27, 14] have shown that the weights of contrastive loss in the overall objective are significantly smaller compared to the weight on the recommendation objective. This is done to ensure desired performance on recommendation tasks. Therefore, based on these observations, simply combining contrastive loss with downstream recommendation objectives may not be effective for recommendation tasks.
The disparity between the pre-training objective and downstream tasks hinders the effective extraction of useful information from pre-trained embeddings by downstream models [12, 26]. Consequently, researchers often opt for combining GCL with recommendation objectives. However, it is important to note that GCL pre-training targets primarily assess the agreement of mutual information among graph elements, such as nodes, edges, and sub-graphs. This differs from conventional graph learning tasks like node classification and link prediction. Consequently, the pre-training targets of GCL also significantly diverge from downstream recommendation objectives that involve interaction (link) prediction between users and items. Consequently, the reduction of such dissimilarities is essential to enhance the performance of GCL-based recommendation approaches.
In this paper, we present the CPTPP framework as an extension of recent advancements in prompt tuning for enhancing recommendation performance [23, 33] utilizing user embeddings pre-trained by GCL. The technique of prompt tuning has emerged as a prominent method for fine-tuning pre-trained models. By constructing appropriate prompts for downstream learning modules, this approach effectively reformulates downstream tasks, thereby reducing disparities [26, 12, 15, 18]. By incorporating prompt-tuning, we can modify existing GCL-based recommendation methods to align with the original GCL protocol involving pre-training and fine-tuning. Previous endeavors have also explored the integration of prompt learning into conventional recommendation models [26, 3]. Despite their advantages, applying the prompt mechanism directly to GCL-based recommendation methods is still difficult and not straightforward, i.e., how can we generate personalized user prompts using only the user-item interaction graph without side information (e.g., age and occupation)? To address this issue, we summarise three methods to produce different user profiles, including historical interaction records, adjacency matrix factorization, and high-order user relations, based on the user-item interaction graph for the personalized user prompt generation, which is applicable in situations devoid of side information. Comprehensive experiments conducted on three publicly available datasets illustrate the effectiveness of the proposed method with different types of prompts.
In summary, the contributions of this work are three-fold: (1) We propose a reformulation of existing GCL-based recommendation methods by incorporating the prompt tuning mechanism. This allows us to fully leverage the advantages of GCL during the pre-training phase, rather than relying on the combination of contrastive loss with downstream objectives. (2) We introduce three user profiles derived from the user-item interaction graph as inputs for the prompt generator. By using these profiles, we are able to generate personalized prompts that enhance the quality of user embeddings in graph-based recommendation systems. (3) We conduct extensive experiments on three publicly available benchmark datasets to validate the effectiveness of our model. Through these experiments, we analyze the important components and hyper-parameters of our approach and also investigate the impact of different personalized prompts generated by our method.
In this section, the proposed method, graph Contrastive Pre-Training with PromPt-tuning for recommendation (CPTPP), will be introduced to reveal the intuitions and the technical details.
2.1 Framework Overview
There are three modules in the proposed CPTPP method, as shown in Figure 1: (1) graph contrastive learning module, leveraging the advantages of GCL to the pre-train user and item embeddings, (2) personalized prompts generation module, applying prompt mechanism to generate personalized
Figure 1: The overview of the proposed method CPTPP.
prompts for users, and (3) recommendation module, fusing the generated personalized prompts and pre-trained user embeddings to conduct prompt-tuning for the downstream recommendation task.
2.2 Graph Contrastive Learning Module
In order to achieve optimal performance in downstream tasks, the selection of a suitable pre-training strategy is crucial for generating high-quality inputs for downstream modules. GCL has been demonstrated as a powerful technique for graph pre-training [21, 16, 30, 37] and has emerged as an effective tool for leveraging self-supervised signals to enhance graph-based recommendation models [27, 14, 24, 32]. In the case of graph-based recommender systems, GCL represents a viable option for pre-training embeddings. Furthermore, our work specifically focuses on reforming and improving existing GCL-based recommendation methods. Therefore, it is imperative that we formulate a GCL module within our proposed method. Current GCL-based recommendation methods [14, 24, 27, 32] have explored various graph augmentation techniques on user-item interaction graphs in order to generate augmented graphs for GCL, enabling the extraction of informative semantics from the graph structures. Alternatively, some studies [31, 14] have designed context embeddings tailored for GCL in recommended settings. Although different approaches exist for constructing contrasting samples, they all share a common backbone for the GCL training protocol.
Here, we give a formal description of the GCL training protocol. Let  $u_{i}$  denote the target graph element (e.g., user node),  $u_{i}^{+}$  represent the positive sample generated from  $u_{i}$  (e.g., the neighbor node of the target node), and  $\mathcal{U}^{-} = \{u_{i,0}^{-}, u_{i,1}^{-}, \dots, u_{i,n}^{-}\}$  be the set of contrasting samples of  $u_{i}$  (e.g., non-neighbour nodes of the target node). Considering the settings of the recommendation task, we use  $G$  to represent the overall user-item graph, and all the target, positive sample, and contrasting samples are within graph  $G$ . To acquire embeddings of these graph elements, we adopt  $f(*)$  as the graph encoder to process them, and the target embedding is denoted by  $\mathbf{u}_i = f(u_i;G)$ ,  $\mathbf{u}^{+} = f(u_{i}^{+};G)$  is the embedding of the positive sample, and  $\{\mathbf{u}_{i,0}^{-},\mathbf{u}_{i,1}^{-},\dots,\mathbf{u}_{i,n}^{-}\}$  are the list of embeddings of the negative contrasting samples. Following the settings of InfoNCE [20], the self-supervised learning objective can be formulated as follows:
$$
\mathcal {L} _ {\text {c o n t r a}} = - \log \frac {\exp \left(\operatorname {s i m} \left(\mathbf {u} _ {i} , \mathbf {u} _ {i} ^ {+}\right) / \tau\right)}{\sum_ {t = 0} ^ {| \mathcal {U} ^ {-} |} \exp \left(\operatorname {s i m} \left(\mathbf {u} _ {i} , \mathbf {u} _ {i , t} ^ {-}\right) / \tau\right)}, \tag {1}
$$
where  $\tau$  is the temperature hyper-parameter and  $\mathrm{sim}(\cdot, \cdot)$  is the similarity metric. In existing research works [24, 27, 32, 31, 14], researchers usually combine the aforementioned contrastive learning loss function with the recommendation objectives to formulate an overall objective function to train the model in an end-to-end manner:
$$
\mathcal {L} _ {\text {o v e r a l l}} = \mathcal {L} _ {\text {r e c}} + \lambda \cdot \mathcal {L} _ {\text {c o n t r a}}, \tag {2}
$$
where  $\lambda$  is a hyper-parameter that controls the weight of the contrastive learning objective. However, as mentioned in Section 1, the proposed CPTPP adopts a pre-train, prompt, fine-tune manner to train the model and treats GCL as a pre-training task instead of combining the contrastive loss with recommendation objectives. To leverage recent research progress in GCL, we can adopt various GCL learning methods tailored for the recommendation task here, like NCL [14], SGL [24], and SimGCL [32], to obtain high-quality user and item embeddings. Then, the pre-trained user and item embeddings will be processed by the prompt mechanism in the following.
2.3 Prompts Generation Module
Following the pre-training phase, our method, named CPTPP, incorporates a personalized prompts generation module to utilize the pre-trained user and item embeddings effectively. The primary objective of this module is to address the limitations present in existing prompt and recommendation research. Prior studies [18, 15, 26] have highlighted the triviality and resource-intensive nature of hard prompt design, making it impractical for real-world scenarios. Additionally, we observe that most current approaches rely on side information (e.g., user descriptions) to generate prompts, and there lacks a specific paradigm for prompting in graph-based recommendation scenarios. To overcome these limitations, we propose the integration of a prompt generator [26] that generates personalized prompts tailored specifically for graph-based recommendation contexts.
2.3.1 Personalized Prompt Generator
The main scope of the generated prompts lies in narrowing the gap between the pre-training targets and the downstream objectives to utilize the pre-trained models or embeddings better. Some research works designed hard prompts tailored for recommendation tasks converting recommendation tasks into NLP tasks [3], which unifies multiple recommendation tasks in a single framework. For example, a convention recommendation task can be converted to a sentence, 'User 123 will purchase item [id_token]'. Then, NLP techniques will be applied to predict the token. However, PPR [26] argued that such an NLP-style hard prompt designing method has two major limitations: (i) It is difficult to apply NLP techniques to predict the designated tokens since these tokens could be a user ID, item ID, or ratings, which lack meaningful semantics. (ii) The designed hard prompts are universal and cannot be customized for various users or items.
To address the challenges, we adopt a method to construct personalized prompts from user profiles in a soft prompt automatic generation manner [12, 15, 26]. Let  $\mathbf{x}_i^u\in \mathbb{R}^{d\times 1}$  denote the profile of user  $i$ . We, then, concatenate all the users' profiles to form the user profile matrix  $\mathbf{X}^u = [\mathbf{x}_1^u,\mathbf{x}_2^u,\dots ,\mathbf{x}_n^u ]\in$ $\mathbb{R}^{d\times n}$ . This matrix will be fed into a two-layer perceptron  $f(\cdot)$  to acquire personalized prompts for each user  $\mathbf{P}^u = [\mathbf{p}_1^u,\mathbf{p}_2^u,\dots ,\mathbf{p}_n^u ]\in \mathbb{R}^{p\times n}$  as follows:
$$
\mathbf {P} ^ {u} = f \left(\mathbf {X} ^ {u}\right) = \mathbf {W} _ {2} \cdot \alpha \left(\mathbf {W} _ {1} \cdot \mathbf {X} ^ {u} + b _ {1}\right) + b _ {2}, \tag {3}
$$
where  $p$  is the prompt size,  $\mathbf{W}_1 \in \mathbb{R}^{h \times d}$  and  $\mathbf{W}_2 \in \mathbb{R}^{p \times h}$  are trainable weights,  $b_1 \in \mathbb{R}^{h \times n}$  and  $b_2 \in \mathbb{R}^{d \times n}$  are biases, and  $\alpha(\cdot)$  is the activation function.  $d$  and  $h$  represent the dimensions of the pre-trained embeddings and hidden dimensions, respectively. The generated prompts will be concatenated with the pre-trained user embeddings in a pre-fixed manner [12] and tuned by the downstream objectives in the recommendation module to fulfill the process of prompt-tuning. Specifically, let  $\mathbf{U}_{pre\_train} \in \mathbb{R}^{d \times n}$  denote pre-trained user embeddings. Then, we have the inputs from the user side for the recommendation module:
$$
\mathbf {U} _ {\text {c o n c a t}} = \left[ \begin{array}{l} \mathbf {P} ^ {u} \\ \mathbf {U} _ {\text {p r e} _ {\text {t r a i n}}} \end{array} \right] ^ {T} \in \mathbb {R} ^ {n \times (p + d)}. \tag {4}
$$
2.3.2 Personalized Inputs for Prompt Generation
The quality of the generated prompts depends on the personalized inputs for the generator. Current research on prompt learning for recommendation mainly focuses on utilizing existing user features
(e.g., age, gender, and occupation) and historical interaction records as the inputs to generate personalized prompts [26, 11, 25]. However, these methods are designed for conventional and sequential recommendations, which are not entirely aligned with graph recommendations. It is necessary to summarise and explore how to generate personalized prompts from the perspective of the graph recommendation system. In this section, we summarise three types of inputs for the generator to generate personalized prompts for the graph-based recommendation: historical interaction records, adjacency matrix factorization, and high-order user relations.
Historical Interaction Records. It is a common and widely-used method to illustrate users' features or preferences via aggregating his/her historical interaction records, which is feasible in various scenarios in recommendation systems. Let  $\mathcal{T}_k^u = \{i_{k,1}, i_{k,2}, \dots, i_{k,m}\}$  denote the item set which are purchased by user  $k$ . We use  $\mathbf{i}_{k,j} \in \mathbb{R}^d$  to represent the embedding of the  $j$ -th item in user  $k$ 's purchase history. Then, the profile of user  $k$  can be acquired by aggregating embeddings of those items purchased by the user  $k$ :
$$
\mathbf {x} _ {k} ^ {u} = \text {A g g r e g a t i o n} (\mathbf {i} _ {k, 1}, \mathbf {i} _ {k, 2}, \dots , \mathbf {i} _ {k, m}), \tag {5}
$$
where  $Aggregation(*)$  is the aggregation function to read out the user's profile.
The concatenation of all the user profiles can form the matrix  $\mathbf{X}^u$  to be processed by the personalized prompt generator. Let  $\mathbf{A} \in \mathbb{R}^{n \times q}$  denote the adjacency matrix for the recommendation system, which contains  $n$  users and  $q$  items. If we have the pre-trained item embeddings  $\mathbf{I}_{pre\_train} = [\mathbf{i}_1, \mathbf{i}_1, \dots, \mathbf{i}_q] \in \mathbb{R}^{d \times q}$ , then, we have:
$$
\mathbf {X} ^ {u} = \left(\mathbf {A} \cdot \mathbf {I} _ {\text {p r e} _ {\text {t r a i n}}} ^ {T}\right) ^ {T}. \tag {6}
$$
Adjacency Matrix Factorization. The adjacency matrix is an effective tool to demonstrate the user-item relations in the recommendation system. However, the adjacency matrix usually suffers from sparsity problems and thus cannot be smoothly applied in many real-world recommendation scenarios. To address this problem, researchers proposed several matrix factorizations (MF) methods [19, 7] to decompose the adjacency matrix to obtain two matrices,  $\mathbf{U}$  and  $\mathbf{V}$ , denoting the latent embeddings for users and items, which are much denser than the adjacency matrix  $\mathbf{A}$  itself. The process of MF can be formulated as follows:
$$
\underset {\mathbf {U}, \mathbf {V}} {\arg \min } \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {q} \left(\mathbf {A} _ {i, j} - \hat {\mathbf {A}} _ {i, j}\right), \tag {7}
$$
where  $\hat{\mathbf{A}}_{i,j} = \sum_k\mathbf{U}_{i,k}\cdot \mathbf{V}_{k,j}^T = \mathbf{U}_i\mathbf{V}_j^T$ . After the MF process, we have the latent matrix of users,  $\mathbf{U}\in \mathbb{R}^{n\times d}$ , serving as the user profile matrix  $\mathbf{X}^u$  after transposed  $\mathbf{X}^u = \mathbf{U}^T$  and can be fed into a personalized prompt generator to produce personalized prompts  $\mathbf{P}^u$  for each user. Specifically, we set the size of latent embeddings as  $d$ , the same as the size of pre-trained embeddings.
High-Order User Relations. Learning informative embeddings from a 1-hop user-item interaction graph is challenging when there is no side information. To address this limitation, we propose to leverage high-order user relations to enrich the learned embeddings via constructing 2-ego graphs for each user node to find the links between the other users and itself [22]. Then, we fuse the target user's purchase history and high-order neighbor embeddings to represent the target user profile.
We first construct the high-order connectivity matrix to achieve the goal. Let  $\mathbf{A}^{*} = \bar{\mathbf{A}}\cdot \bar{\mathbf{A}}\in$ $\mathbb{R}^{(n + q)\times (n + q)}$  denote the high-order connectivity matrix, where  $\bar{\mathbf{A}} = \left[ \begin{array}{cc}\mathbf{0} & \mathbf{A}\\ \mathbf{A}^T & \mathbf{0} \end{array} \right]\in \mathbb{R}^{(n + q)\times (n + q)}$  recording all the users and items to which a user or item node is connected. Then, we build the matrix  $\mathbf{E} = [\mathbf{U}_{pre\_train},\mathbf{I}_{pre\_train}]^T\in \mathbb{R}^{(n + q)\times d}$  to store pre-trained embeddings. Next, we can acquire matrix  $\mathbf{Q}\in \mathbb{R}^{n\times d}$ , which are the users' personalized profiles about high-order user relations, via:
$$
\left[ \begin{array}{l} \mathbf {Q} \\ \mathbf {M} \end{array} \right] = \mathbf {A} ^ {*} \cdot \mathbf {E} \in \mathbb {R} ^ {(n + q) \times d}, \tag {8}
$$
where  $\mathbf{M} \in \mathbb{R}^{q \times d}$ , denoting the high-order item relations and being omitted after  $\mathbf{Q}$  is extracted. Then, a matrix transpose operation is required to obtain the user profile matrix  $\mathbf{X}^u = \mathbf{Q}^T$ .
2.4 Recommendation Module
After the pre-training and the personalized prompts generation phase, a recommendation module is required so that we can verify whether the prompt-tuning module rectifies the pre-trained embeddings by GCL and makes them be adapted to the downstream recommendation tasks better. In this module, we take the inner product of user and item embeddings as the predicted score for the recommendation. Bayesian Personalized Ranking (BPR) [17] is adopted as the training objective to tune the pretrained embeddings based on the predicted scores. The motivation for formulating such a simple recommendation module is to avoid the performance gain brought by the delicate designs of those advanced recommendation models, which could affect the observations on our proposed method.
Algorithm 1: CPTPP algorithm
Input: User embedding table  $\mathbf{U}_E$ ; Item embedding table  $\mathbf{I}_E$ ; User-item interaction graph adjacency matrix  $\mathbf{A}$ ; Graph contrastive learning model  $f(*)$ ; User profile  $\mathbf{X}^u$ ; Prompt generator  $g(\cdot)$ ; Multi-layer perceptron MLP  $(\cdot)$ ; Pre-train epoch  $i$ ; Prompt-tune epoch  $j$ . Output: User and item embedding tables  $\mathbf{U}_E^*$  and  $\mathbf{I}_E^*$ .  
1 Pre-train phase:  
2 Initialize  $\mathbf{U}_E, \mathbf{I}_E; \mathbf{U}_E', \mathbf{I}_E' \leftarrow \mathbf{U}_E, \mathbf{I}_E$ ;  
3 count = 0;  
4 while count < i do  
    // Update user and item embedding tables.  
5  $\mathbf{U}_E', \mathbf{I}_E' = f(\mathbf{U}_E'; \mathbf{I}_E'; \mathbf{A})$ ;  
6 count = count + 1;  
7 end  
8 Prompt-tune phase:  
9  $\mathbf{U}_E^* \leftarrow \mathbf{U}_E'; \mathbf{I}_E^* \leftarrow \mathbf{I}_E'$ ;  
10 count = 0;  
11 while count < j do  
    // Personalized prompt generation.  
12  $\mathbf{P}^u = g(\mathbf{X}^u)$ ;  
    // Concatenate & fusion.  
13  $\mathbf{U}_E^* = \text{MLP}([\mathbf{P}^u; \mathbf{U}_E^*]^T) \in \mathbb{R}^{n \times d}$ ;  
14 Optimise  $\mathcal{L} = \sum_{i \in \mathcal{U}} \mathcal{L}_{rec}^i + \lambda ||\Theta||_2^2$ ;  
15 Update  $\mathbf{U}_E^*, \mathbf{I}_E^*$ ;  
16 count = count + 1;  
17 end  
18 return  $\mathbf{U}_E^*, \mathbf{I}_E^*$
2.4.1 Prompts and Pre-Trained Embeddings Fusion
We concatenate the generated personalized prompts and the pre-trained user embeddings in the previous step and have  $\mathbf{U}_{\text {concat }} \in \mathbb{R}^{n \times (p + d)}$ , whose dimensionality is different from the pre-trained embeddings  $\mathbf{I}_{\text {pre_train }} \in \mathbb{R}^{n \times d}$ . Hence, we need first to fuse the personalized prompts and the pre-trained user embeddings for the recommendation objective training. Specifically, we adopt a multi-layer perceptron (MLP)  $g(\cdot)$  as the mapping function that is  $g: \mathbb{R}^{n \times (p + d)} \to \mathbb{R}^{n \times d}$ . Then, we can have dimensionality-reduced user representations  $\mathbf{U}^* = g(\mathbf{U}_{\text {concat }}) \in \mathbb{R}^{n \times d}$ , enhanced by the personalized prompts. After that, we can apply the inner product to predict how likely the user  $i$  would interact with the item  $j$  by  $\hat{y}_{i,j} = \mathbf{u}_i^* \cdot \hat{\mathbf{i}}_j$ , where  $\mathbf{u}_i^*$  is the  $i$ -th row of  $\mathbf{U}^*$ .
2.4.2 Training Objective for Recommendation Task
For simplicity and fair comparison, we adopt BPR [17] loss as the training objective for the recommendation task. For each user  $i$ , we have:
$$
\mathcal {L} _ {r e c} ^ {i} = \sum_ {j ^ {+} \in \mathcal {I} _ {i} ^ {u}} \sum_ {j ^ {-} \in \mathcal {I} \backslash \mathcal {I} _ {i} ^ {u}} - \log \sigma \left(\hat {y} _ {i, j ^ {+}} - \hat {y} _ {i, j ^ {-}}\right). \tag {9}
$$
However, it is unaffordable to consider all the unobserved interactions of the user  $i$ . Therefore we sample several negative items  $\mathcal{N}_i^u$ , where  $|\mathcal{N}_i^u| < < |\mathcal{I}\backslash \mathcal{I}_i^u|$ , in practice.