SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taoran Fang $^{1}$ , Yunchao Zhang $^{1}$ , Yang Yang $^{1*}$ , Chunping Wang $^{2}$ , Lei Chen $^{2}$
<sup>1</sup>Zhejiang University, <sup>2</sup>FinVolution Group
{fangtr,3190105622,yangya}@zju.edu.cn,
{wangchunping02,chenlei04}@xinye.com
In recent years, prompt tuning has sparked a research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-based tuning methods for graph neural networks. While some pioneering work has devised specialized prompting functions for models that employ edge prediction as their pre-training tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trained GNN models under any pre-training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect to any form of prompting function. Consequently, we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF to obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality of GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than fine-tuning, with an average improvement of about  $1.4\%$  in full-shot scenarios and about  $3.2\%$  in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt-based tuning methods when applied to models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations. Our code is available at: https://github.com/zjunet/GPF.
Graph neural networks (GNNs) have garnered significant attention from researchers due to their remarkable success in graph representation learning [Kipf and Welling, 2017, Hamilton et al., 2017, Xu et al., 2019]. However, two fundamental challenges hinder the large-scale practical applications of GNNs. One is the scarcity of labeled data in the real world [Zitnik et al., 2018], and the other is the low out-of-distribution generalization ability of the trained models [Hu et al., 2020a, Knyazev et al., 2019, Yehudai et al., 2021, Morris et al., 2019]. To overcome these challenges, researchers have made substantial efforts in designing pre-trained GNN models [Xia et al., 2022b, Hu et al., 2020a,b, Lu et al., 2021] in recent years. Similar to the pre-trained models in the language field, pre-trained GNN models undergo training on extensive pre-training datasets and are subsequently adapted to downstream tasks. Most existing pre-trained GNN models obey the "pre-train, fine-tune" learning strategy [Xu et al., 2021a]. Specifically, we train a GNN model with a massive corpus of pre-training graphs, then we utilize the pre-trained GNN model as initialization and fine-tune the model parameters based on the specific downstream task.
Figure 1: Comparison of universal graph prompt tuning and existing approaches. (a) Fine-tuning updates the parameters of the pre-trained GNN model. (b) Existing specialized prompt-based tuning methods generate manual graph templates to adapt the models under certain pre-training strategies. (c) Our universal graph prompt tuning works on the feature space of the input graph. It can achieve an equivalent effect to any form of prompting function and be applied to any pre-trained GNN model.
However, the "pre-train, fine-tune" framework of pre-trained GNN models also presents several critical issues [Jin et al., 2020]. First, there is a misalignment between the objectives of pre-training tasks and downstream tasks [Liu et al., 2022a]. Most existing pre-trained models employ self-supervised tasks [Liu et al., 2021e] such as edge prediction and attribute masking as the training targets during pre-training, while the downstream tasks involve graph or node classification. This disparity in objectives leads to sub-optimal performance [Liu et al., 2023]. Additionally, ensuring that the model retains its generalization ability is challenging. Pre-trained models may suffer from catastrophic forgetting [Zhou and Cao, 2021, Liu et al., 2020] during downstream adaptation. This issue becomes particularly acute when the downstream data is small in scale, approaching the few-shot scenarios [Zhang et al., 2022]. The pre-trained model tends to over-fit the downstream data in such cases, rendering the pre-training process ineffective.
"If your question isn't getting the desired response, try rephrasing it." In recent years, a novel approach called prompt tuning has emerged as a powerful method for downstream adaptation, addressing the aforementioned challenges. This technique has achieved significant success in Natural Language Processing [Li and Liang, 2021, Lester et al., 2021a, Liu et al., 2022a,b] and Computer Vision [Bahng et al., 2022, Jia et al., 2022]. Prompt tuning provides an alternative method for adapting pre-trained models to specific downstream tasks: it freezes the parameters of the pre-trained model and modifies the input data. Unlike fine-tuning, prompt tuning diverges from tuning the parameters of the pretrained model and instead focuses on adapting the data space by transforming the input.
Despite that, applying prompt tuning on pre-trained GNN models poses significant challenges and is far from straightforward. First, the diverse pre-training strategies employed on graphs make it difficult to design suitable prompting functions. Previous research [Liu et al., 2022a] suggests that the prompting function should be closely aligned with the pre-training strategy. For pre-trained language models, the typical pre-training tasks involve masked sentence completion [Brown et al., 2020]. In order to align with this task, we may modify a sentence like "I received a gift" to "I received a gift, and I feel [Mask]" to make it closer to the task of sentence completion. However, in the case of graph pre-training, there is no unified pre-training task, making it challenging to design feasible prompting functions. Some pioneering studies [Sun et al., 2022, Liu et al., 2023] have applied prompt-based tuning methods to models pre-trained by edge prediction [Kipf and Welling, 2016b]. They introduce virtual class-prototype nodes/graphs with learnable links into the original graph, making the adaptation process more akin to edge prediction. However, these methods have limited applicability and are only compatible with specific models. When it comes to more intricate pre-training strategies, it becomes challenging to design manual prompting functions in the same manner as employed for link prediction. Consequently, no prompt-based tuning method is available for models pre-trained using alternative strategies, such as attribute masking [Hu et al., 2020a]. Furthermore, existing prompt-based tuning methods for GNN models are predominantly designed based on intuition, lacking theoretical guarantees for their effectiveness.
In this paper, we address the aforementioned issues for graph prompt tuning. To deal with the diversity of graph pre-training strategies, we propose a universal prompt-based tuning method that can be
applied to the pre-trained GNN models that employ any pre-training strategy. Figure 1 illustrates the distinction between our universal prompt-based tuning method and existing approaches. Our solution, called Graph Prompt Feature (GPF), operates on the input graph's feature space and involves adding a shared learnable vector to all node features in the graph. This approach is easily applicable to any GNN architecture. We rigorously demonstrate that GPF can achieve comparable results to any form of prompting function when applied to arbitrary pre-trained GNN models. Consequently, instead of explicitly illustrating the prompting function corresponding to each pre-training strategy, we adopt GPF to dynamically obtain the prompted graph for downstream tasks. We also introduce a theoretically stronger variant of GPF, named GPF-plus, for practical application, which incorporates different prompted features for different nodes in the graph. To guarantee the effectiveness of our proposed GPF and GPF-plus, we provide theoretical analyses to prove that GPF and GPF-plus are not weaker than full fine-tuning and can obtain better theoretical tuning results in some cases. Furthermore, we conduct extensive experiments to validate the efficacy of our methods. Despite using a significantly smaller number of tunable parameters than fine-tuning, GPF and GPF-plus achieve better results across all pre-training strategies. For models pre-trained using edge prediction, GPF and GPF-plus exhibit a substantial performance advantage over existing specialized prompt-based tuning methods. Overall, the contributions of our work can be summarized as follows:
- To the best of our knowledge, we present the first investigation of universal prompt-based tuning methods for existing pre-trained GNN models. We propose GPF and its variant, GPF-plus, as novel approaches for universal graph prompt tuning. Our methods can be applied to the pre-trained GNN models that employ any pre-training strategy.
- We provide theoretical guarantees for the effectiveness of GPF and GPF-plus. We demonstrate that GPF and GPF-plus can achieve an equivalent effect to any prompting function and can obtain better tuning results in some cases compared to fine-tuning.
- We conduct extensive experiments (both full-shot and few-shot scenarios) to validate the effectiveness of GPF and GPF-plus. The experimental results indicate that GPF and GPF-plus can perform better than fine-tuning, with an average improvement of about  $1.4\%$  in full-shot scenarios and about  $3.2\%$  in few-shot scenarios. Furthermore, GPF and GPF-plus significantly outperform existing prompt-based tuning methods when applied to models that utilize the pre-training strategy they specialize in.
Pre-trained GNN Models Inspired by the remarkable achievements of pre-trained models in Natural Language Processing [Qiu et al., 2020b] and Computer Vision [Long et al., 2022], substantial efforts have been dedicated to pre-trained GNN models (PGMs) [Xia et al., 2022b] in recent years. These methods utilize self-supervised strategies [Jin et al., 2020] to acquire meaningful representations from extensive pre-training graphs. GAE [Kipf and Welling, 2016a] first uses edge prediction as the objective task to train graph representations. Deep Graph Infomax (DGI) [Velickovic et al., 2019b] and InfoGraph [Sun et al., 2019] are proposed to garner nodes or graph representations by maximizing the mutual information between graph-level and substructure-level representations of different granularity. Hu et al. [2020a] employ attribute masking and context prediction as pre-training tasks to predict molecular properties and protein functions. Both GROVER [Rong et al., 2020] and MGSSL [Zhang et al., 2021a] propose to predict the presence of the motifs or generate them with the consideration that rich domain knowledge of molecules hides in the motifs. Graph Contrastive Learning (GCL) is another widely adopted pre-training strategy for GNN models. GraphCL [You et al., 2020] and JOAO [You et al., 2021] propose various augmentation strategies to generate different augmented views for contrastive learning. In summary, there exists a diverse range of pre-training strategies for GNN models, each characterized by unique objectives.
Prompt-based Tuning Methods Prompt-based tuning methods, originating from Natural Language Processing, have been widely used to facilitate the adaptation of pre-trained language models to various downstream tasks [Liu et al., 2021a]. Research has also explored the design of soft prompts to achieve optimal performance [Lester et al., 2021b, Liu et al., 2021c]. These methods freeze the parameters of the pre-train models and introduce additional learnable components in the input space, thereby enhancing the compatibility between inputs and pre-trained models. Aside from the success of prompts in the language field, the prompting methods are utilized in other areas. Jia et al. [2022] and Bahng et al. [2022] investigate the efficacy of adapting large-scale models in the vision field by
modifying input images at the pixel level. In the realm of graph neural networks, the exploration of prompt-based tuning methods is still limited. Some pioneering work [Sun et al., 2022, Liu et al., 2023] applies prompt-based tuning methods on the models pre-trained by edge prediction [Kipf and Welling, 2016b]. These methods introduce virtual class-prototype nodes/graphs with learnable links into the input graph, making the downstream adaptation more closely resemble edge prediction. However, these methods are specialized for models pre-trained using edge prediction and cannot be applied to models trained with other strategies. We are the first to investigate the universal prompt-based tuning methods that can be applied to the GNN models under any pre-training strategy.
We introduce graph prompt tuning for adapting pre-trained GNN models to downstream tasks. It is important to note that there are several types of downstream tasks in graph analysis, including node classification, link prediction, and graph classification. We first concentrate on the graph classification task and then extend our method to node-wise tasks. We define the notations in Section 3.1, then illustrate the process of graph prompt tuning in Section 3.2. We introduce our universal graph prompt tuning method in Section 3.3 and provide theoretical analyses in Section 3.4. Finally, we present the extension of our method to node-wise tasks (node classification and link prediction) in the appendix.
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})\in \mathbb{G}$  represents a graph, where  $\mathcal{V} = \{v_{1},v_{2},\ldots ,v_{N}\}$ ,  $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$  denote the node set and edge set respectively. The node features can be denoted as a matrix  $\mathbf{X} = \{x_1,x_2,\dots,x_N\} \in \mathbb{R}^{N\times F}$ , where  $x_{i}\in \mathbb{R}^{F}$  is the feature of the node  $v_{i}$ , and  $F$  is the dimensionality of node features.  $\mathbf{A}\in \{0,1\}^{N\times N}$  denotes the adjacency matrix, where  $\mathbf{A}_{ij} = 1$  if  $(v_{i},v_{j})\in \mathcal{E}$ .
Fine-Tuning Pre-trained Models. Given a pre-trained GNN model  $f$ , a learnable projection head  $\theta$  and a downstream task dataset  $\mathcal{D} = \{(\mathcal{G}_1, y_1), \dots, (\mathcal{G}_m, y_m)\}$ , we adjust the parameters of the pre-trained model  $f$  and the projection head  $\theta$  to maximize the likelihood of predicting the correct labels  $y$  of the downstream graph  $\mathcal{G}$ :
$$
\max  _ {f, \theta} P _ {f, \theta} (y | \mathcal {G}) \tag {1}
$$
3.2 Graph Prompt Tuning
Overall Process. Our proposed graph prompt tuning works on the input space by drawing on the design of the prompt tuning in the language field [Liu et al., 2022a]. Given a frozen pre-trained GNN model  $f$ , a learnable projection head  $\theta$ , and a downstream task dataset  $\mathcal{D} = \{(\mathcal{G}_1, y_1), \dots, (\mathcal{G}_m, y_m)\}$ , our target is to obtain a task-specific graph prompt  $g_{\phi} \colon \mathbb{G} \to \mathbb{G}$  parameterized by  $\phi$ . The graph prompt  $g_{\phi}(\cdot)$  transforms the input graph  $\mathcal{G}$  into a specific prompted graph  $g_{\phi}(\mathcal{G})$ . And then  $g_{\phi}(\mathcal{G})$  will replace  $\mathcal{G}$  as input to the pre-trained GNN model  $f$ . During the downstream task training, we select the optimal parameters of  $\phi$  and  $\theta$  that maximize the likelihood of predicting the correct labels  $y$  without tuning the pre-trained model  $f$ , which can be formulated as:
$$
\max  _ {\phi , \theta} P _ {f, \theta} (y | g _ {\phi} (\mathcal {G})) \tag {2}
$$
During the evaluation stage, the test graph  $\mathcal{G}_{\mathrm{test}}$  is first transformed by graph prompt  $g_{\phi}(\cdot)$ , and the resulting prompted graph  $g_{\phi}(\mathcal{G}_{\mathrm{test}})$  is processed through the frozen GNN model  $f$ .
Practical Usage. In this part, we provide a detailed description of the refined process of graph prompt tuning, which comprises two fundamental steps: template design and prompt optimization.
A. Template Design. Given an input graph  $\mathcal{G}$ , we first generate a graph template  $\mathcal{G}^*$ , which includes learnable components in its adjacency matrix  $\mathbf{A}^*$  and feature matrix  $\mathbf{X}^*$ . Previous research has attributed the success of prompt tuning to bridging the gap between pre-training tasks and downstream tasks [Liu et al., 2022a]. Consequently, it implies that the specific form of the graph template is influenced by the pre-training strategy employed by the model. For a specific pre-training task  $t \in \mathbb{T}$  and an input graph  $\mathcal{G}$ , the graph template  $\mathcal{G}^*$  can be expressed as:
$$
\mathcal {G} ^ {*} \colon \left(\mathbf {A} ^ {*}, \mathbf {X} ^ {*}\right) = \psi_ {t} (\mathcal {G}) \tag {3}
$$
where the graph template  $\mathcal{G}^*$  may contain learnable parameters (i.e., tunable links or node features) in its adjacency matrix or feature matrix (similar to the inclusion of learnable soft prompts in a sentence), the candidate space for  $\mathbf{A}^*$  is  $\mathbb{A}$ , and the candidate space for  $\mathbf{X}^*$  is  $\mathbb{X}$ .
B. Prompt Optimization. Once we have obtained the graph template  $\mathcal{G}^*$ , our next step is to search for the optimal  $\hat{\mathbf{A}}$  and  $\hat{\mathbf{X}}$  within their respective candidate spaces  $\mathbb{A}$  and  $\mathbb{X}$  that maximize the likelihood of correctly predicting the labels  $y$  using the pre-trained model  $f$  and a learnable projection head  $\theta$ . This process can be expressed as:
$$
\max  _ {\hat {\mathbf {A}} \in \mathbb {A}, \hat {\mathbf {X}} \in \mathbb {X}, \theta} P _ {f, \theta} (y | \mathcal {G} ^ {*}) \tag {4}
$$
The graph  $\hat{\mathcal{G}}$  composed of  $\hat{\mathbf{A}}$  and  $\hat{\mathbf{X}}$  can be considered as the the prompted graph  $g_{\phi}(\mathcal{G})$  mentioned in Formula 2.
Practical Challenges. The specific form of the graph template is closely tied to the pre-training task  $t$  employed by the model  $f$ . However, designing the prompting function  $\psi_t(\cdot)$  is challenging and varies for different pre-training tasks. Pioneering works [Sun et al., 2022, Liu et al., 2023] have proposed corresponding prompting functions  $\psi_t(\cdot)$  for a specific pre-training strategy, with a focus on models pre-trained using edge prediction. However, many other pre-training strategies [Hu et al., 2020a, Xia et al., 2022b], such as attribute masking and context prediction, are widely utilized in existing pre-trained GNN models, yet no research has been conducted on designing prompting functions for these strategies. Furthermore, existing prompting functions are all intuitively designed, and these manual prompting functions lack a guarantee of effectiveness. It raises a natural question: Can we design a universal prompting method that can be applied to any pre-trained model, regardless of the underlying pre-training strategy?
3.3 Universal Graph Prompt Design
In this section, we introduce a universal prompting method and its variant. Drawing inspiration from the success of pixel-level Visual Prompt (VP) techniques [Bahng et al., 2022, Wu et al., 2022, Xing et al., 2022] in Computer Vision, our methods introduce learnable components to the feature space of the input graph. In Section 3.4, we will demonstrate that these prompting methods can theoretically achieve an equivalent effect as any prompting function  $\psi_t(\cdot)$ .
Graph Prompt Feature (GPF). GPF focuses on incorporating additional learnable parameters into the feature space of the input graph. Specifically, the learnable component  $p$  is a vector of dimension  $F$ , where  $F$  corresponds to the dimensionality of the node features. It can be denoted as:
$$
p \in \mathbb {R} ^ {F} \tag {5}
$$
The learnable vector  $p$  is added to the graph features  $\mathbf{X}$  to generate the prompted features  $\mathbf{X}^*$ , which can be expressed as:
$$
\mathbf {X} = \left\{x _ {1}, x _ {2}, \dots , x _ {N} \right\} \quad \mathbf {X} ^ {*} = \left\{x _ {1} + p, x _ {2} + p, \dots , x _ {N} + p \right\} \tag {6}
$$
The prompted features  $\mathbf{X}^*$  replace the initial features  $\mathbf{X}$  and are processed by the pre-trained model.
Graph Prompt Feature-Plus (GPF-plus). Building upon GPF, we introduce a variant called GPF-plus, which assigns an independent learnable vector  $p_i$  to each node  $v_i$  in the graph. It can be expressed as:
$$
p _ {1}, p _ {2}, \dots p _ {N} \in \mathbb {R} ^ {F} \tag {7}
$$
$$
\mathbf {X} = \left\{x _ {1}, x _ {2}, \dots , x _ {N} \right\} \quad \mathbf {X} ^ {*} = \left\{x _ {1} + p _ {1}, x _ {2} + p _ {2}, \dots , x _ {N} + p _ {N} \right\} \tag {8}
$$
Similarly to GPF, the prompted features  $\mathbf{X}^*$  replace the initial features  $\mathbf{X}$  and are processed by the pre-trained model. However, such a design is not universally suitable for all scenarios. For instance, when training graphs have different scales (i.e., varying node numbers), it is challenging to train such a series of  $p_i$ . Additionally, when dealing with large-scale input graphs, such design requires a substantial amount of storage resources due to its  $O(N)$  learnable parameters. To address these issues, we introduce an attention mechanism in the generation of  $p_i$ , making GPF-plus more parameter-efficient and capable of handling graphs with different scales. In practice, we train only  $k$  independent basis vectors  $p^b$ , which can be expressed as:
$$
p _ {1} ^ {b}, p _ {2} ^ {b}, \dots p _ {k} ^ {b} \in \mathbb {R} ^ {F} \tag {9}
$$
where  $k$  is a hyper-parameter that can be adjusted based on the downstream dataset. To obtain  $p_i$  for node  $v_i$ , we utilize attentive aggregation of these basis vectors with the assistance of  $k$  learnable linear projections  $a$ . The calculation process can be expressed as:
$$
p _ {i} = \sum_ {j} ^ {k} \alpha_ {i, j} p _ {j} ^ {b} \quad \alpha_ {i, j} = \frac {\exp \left(a _ {j} ^ {\mathrm {T}} x _ {i}\right)}{\sum_ {l} ^ {k} \exp \left(a _ {l} ^ {\mathrm {T}} x _ {i}\right)} \tag {10}
$$
Subsequently,  $p_i$  is used to generate the prompted feature  $\mathbf{X}^*$  as described in Formula 8.
In this section, we provide theoretical analyses for our proposed GPF and GPF-plus. Our analyses are divided into two parts. First, we certify the universality of our methods. We demonstrate that our approaches can theoretically achieve results equivalent to any prompting function  $\psi_t(\cdot)$ . It confirms the versatility and applicability of our methods across different pre-training strategies. Then, we make guarantee of the effectiveness of our proposed methods. Specifically, we demonstrate that our proposed graph prompt tuning is not weaker than full fine-tuning, which means that in certain scenarios, GPF and GPF-plus can achieve superior tuning results compared to fine-tuning. It is important to note that our derivations in the following sections are based on GPF, which adds a global extra vector  $p$  to all nodes in the graph. GPF-plus, being a more powerful version, can be seen as an extension of GPF and degenerates to GPF when the hyperparameter  $k$  is set to 1. Therefore, the analyses discussed for GPF are also applicable to GPF-plus.
Before we illustrate our conclusions, we first provide some preliminaries. For a given pre-training task  $t \in \mathbb{T}$  and an input graph  $\mathcal{G} \colon (\mathbf{A}, \mathbf{X})$ , we assume the existence of a prompting function  $\psi_t(\cdot)$  that generates a graph template  $\mathcal{G}^* \colon (\mathbf{A}^*, \mathbf{X}^*) = \psi_t(\mathcal{G})$ . The candidate space for  $\mathbf{A}^*$  and  $\mathbf{X}^*$  is denoted as  $\mathbb{A}$  and  $\mathbb{X}$ , respectively.