Title: Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: few-shot molecular property prediction

molecular property prediction

parameter-efficient tuning

molecular representation learning
Abstract: Molecular property prediction (MPP) is integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Addressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs a pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, existing methods struggle with the ineffective fine-tuning of pre-trained encoders. We attribute this issue to the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, we propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC), to achieve parameter-efficient tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP-Adapters with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptability for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.

Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction

Abstract

Molecular property prediction (MPP) is integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Addressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs a pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, existing methods struggle with the ineffective fine-tuning of pre-trained encoders. We attribute this issue to the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, we propose a parameter-efficient in-context tuning method, named Pin-Tuning . Specifically, we propose a lightweight adapter for pre-trained message passing layers ( MP-Adapter ) and Bayesian weight consolidation for pretrained atom/bond embedding layers ( Emb-BWC ), to achieve parameter-efficient tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP-Adapters with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptability for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance. ‡

1 Introduction

In the field of drug discovery and material science, molecular property prediction (MPP) stands as a pivotal task [5, 9, 63]. MPP involves the prediction of molecular properties like solubility and toxicity, based on their structural and physicochemical characteristics, which is integral to the development of new pharmaceuticals and materials. However, a major challenge encountered in real-world MPP scenarios is data scarcity. Obtaining extensive molecular data with well-characterized properties can be time-consuming and expensive. To address this, few-shot molecular property prediction (FSMPP) has emerged as a crucial approach, enabling predictions with limited labeled molecules [1, 41, 4].

The methodology for general MPP typically adheres to an encoder-classifier framework [71, 23, 27, 56], as illustrated in Figure 2(a). In this streamlined framework, the encoder converts molecular

structures into vectorized representations [12, 28, 50, 67, 2], and then the classifier uses these representations to predict molecular properties. In the context of few-shot scenarios, two significant discoveries have been instrumental in advancing this task. Firstly, pre-trained molecular encoders have demonstrated consistent effectiveness in FSMPP tasks [20, 14, 58]. This indicates the utility of leveraging pre-acquired knowledge in dealing with data-limited scenarios. Secondly, unlike typical few-shot tasks such as image classification [57, 48], FSMPP tasks greatly benefits from molecular context information . This involves comprehending the seen many-to-many relationships between molecules and properties [58, 45, 73], as molecules are multi-labeled by various properties. These two discoveries have collectively led to the development of the widely used FSMPP framework that utilizes a pre-trained encoder followed by a context-aware classifier, as shown in Figure 2(b).

Despite the progress, there are observed limitations in the current approaches to FSMPP. Notably, while using a pre-trained molecular encoder generally outperforms training from scratch, fine-tuning the pre-trained encoder often leads to inferior results compared to keeping it frozen, which can be observed in Figure 1.

The observed ineffective fine-tuning can be attributed to two primary factors: (i) Imbalance between the abundance of tunable parameters and the scarcity of labeled molecules: fine-tuning all parameters of a pre-trained encoder with few labeled molecules leads to a disproportionate ratio of tunable parameters to available data. This imbalance often results in over-fitting and catastrophic forgetting [7, 6]. (ii) Limited contextual perceptiveness in the encoder: while molecular

GIN-Mol

CMPNN

Graphormer

83

84

85

86

87

88

ROC-AUC (%)

Tox21

GIN-Mol

CMPNN

Graphormer

82

83

84

85

86

87

SIDER

Train-from-Scratch

Pretrain-then-Freeze

Pretrain-then-Finetune

context is leveraged to enhance the classifier [58, 73], the encoder typically lacks the explicit capability to perceive this context, relying instead on implicit gradient-based optimization. This leads to the encoder not directly engaging with the nuanced molecular context information that is critical in FSMPP tasks. In summary, while significant strides have been made, the challenges of imbalance between the number of parameters and labeled data, along with the need for contextual perceptiveness in the encoder, necessitate more sophisticated methodologies in this domain.

Based on the aforementioned analysis, we propose the parameter-efficient in-context tuning method, named Pin-Tuning , to address the two primary challenges in FSMPP. To overcome the parameterdata imbalance, we propose a parameter-efficient chemical knowledge adaptation approach for pre-trained molecular encoders. A lightweight adapters ( MP-Adapter ) are designed to tune the pretrained message passing layers efficiently. Additionally, we impose a Bayesian weight consolidation ( Emb-BWC ) on the pre-trained embedding layers to prevent aggressive parameter updates, thereby mitigating the risk of over-fitting and catastrophic forgetting. To address the second challenge, we further endow the MP-Adapter with the capability to perceive context. This innovation allows for in-context tuning of the pre-trained molecular encoders, enabling them to adapt more effectively to specific downstream tasks. Our approach is rigorously evaluated on public datasets. The experimental results demonstrate that our method achieves superior tuning performance with fewer trainable parameters, leading to enhanced performance in few-shot molecular property prediction.

The main contributions of our work are summarized as follows:

We analyze the deficiencies of existing FSMPP approaches regarding the adaptation of pre-trained molecular encoders. The key issues include an imbalance between the number of tunable parameters and labeled molecules, as well as a lack of contextual perceptiveness in the encoders.

We propose Pin-Tuning to adapt the pre-trained molecular encoders for FSMPP tasks. This includes the MP-Adapter for message passing layers and the Emb-BWC for embedding layers, facilitating parameter-efficient tuning of pre-trained molecular encoders.

We further endow the MP-Adapter with the capability to perceive context to allows for in-context tuning, which provides more meaningful adaptation guidance during the tuning process.

We conduct extensive experiments on benchmark datasets, which show that Pin-Tuning outperforms state-of-the-art methods on FSMPP by effectively tuning pre-trained molecular encoders.

2 Related work

Few-shot molecular property prediction. Few-shot molecular property prediction aims to accurately predict the properties of new molecules with limited training data [49]. Early research applied general few-shot techniques to FSMPP. IterRefLSTM [1] is the pioneer work to leverage metric learning to solve FSMPP problem. Following this, Meta-GGNN [41] and Meta-MGNN [14] introduce meta-learning with graph neural networks, setting a foundational framework that subsequent studies have continued to build upon [39, 40, 4]. It is noteworthy that Meta-MGNN employs a pre-trained molecular encoder [20] and achieves superior results through fine-tuning in the meta-learning process compared to training from scratch. In fact, pre-trained graph neural networks [64, 36, 17, 54, 37] have shown promise in enhancing various graph-based downstream tasks [52, 13], including molecular property prediction [60, 62, 38, 72]. Recent efforts have shifted towards leveraging unique nature in FSMPP, such as the many-to-many relationships between molecules and properties arising from the multi-labeled nature of molecules, often referred to as the molecular context . PAR [58] initially employs graph structure learning [32, 55] to connect similar molecules through a homogeneous context graph. MHNfs [45] introduces a large-scale external molecular library as context to augment the limited known information. GS-Meta [73] further incorporates auxiliary task to depict the many-to-many relationships.

Parameter-efficient tuning. As pre-training techniques have advanced, tuning of pre-trained models has become increasingly crucial. Traditional full fine-tuning approaches updates all parameters, often leading to high computational costs and the risk of over-fitting, especially when available data for downstream tasks are limited [33, 15]. This challenge has led to the emergence of parameter-efficient tuning [26, 29, 34]. The philosophy of parameter-efficient tuning is to optimize a small subset of parameters, reducing the computational costs while retaining or even improving performance on downstream tasks [19, 69]. Among the various strategies, the adapters [18, 42, 59] have gained prominence. Adapters are small modules inserted between the pre-trained layers. During the tuning process, only the parameters of these adapters are updated while the rest remains frozen, which not only improves tuning efficiency but also offers an elegant solution to the generalization [70, 30, 8]. By keeping the majority of the pre-trained parameters intact, adapters preserve the rich pre-trained knowledge. This attribute is particularly valuable in many real-world applications including FSMPP.

3 Preliminaries

3.1 Problem formulation

Let {T } be a collection of tasks, where each task T involves the prediction of a property p . The training set comprising multiple tasks {T train } , is represented as D train = { ( m i , y i,t ) | t ∈ {T train }} , with m i indicating a molecule and y i,t its associated label for task t . Correspondingly, the test set D test, formed by tasks {T test } , ensures a separation of properties between training and testing phases, as the property sets { p train } and { p test } are disjoint ( { p train } ∩ { p test } = ∅ ).

The goal of FSMPP is to train a model using D train that can accurately infer new properties from a limited number of labeled molecules in D test. Episodic training has emerged as a promising strategy in meta-learning [10, 16] to deal with few-shot problem. Instead of retaining all {T train } tasks in memory, episodes { E t } B t =1 are iteratively sampled throughout the training process. For each episode E t , a particular task T t is selected from the training set, along with corresponding support set S t and query set Q t . Typically, the prediction task involves classifying molecules into two classes: positive ( y = 1 ) or negative ( y = 0 ). Then a 2-way K -shot episode E t = ( S t , Q t ) is constructed. The support set S t = { ( m s i , y s i,t ) } 2 i K =1 includes 2 K examples, each class contributing K molecules. The query set containing M molecules is denoted as Q t = { ( m q i , y q i,t ) } i M =1 .

3.2 Encoder-classifier framework for FSMPP

Encoder-classifier framework is widely adopted in FSMPP methods. As illustrated in Figure 2(a), given a molecule m whose property need to be predicted, a molecular encoder f ( · ) first learns the molecule's representation based on its structure, i.e., h m = f ( m ) ∈ R d . The molecule m is generally represented as a graph m = ( V , A , X , E ) , where V denotes the nodes (atoms), A represents the adjacent matrix defined by edges (chemical bonds), and X , E denote the original feature of atoms

and bonds, then graph neural networks (GNNs) are employed as the molecular encoders [44, 51, 21]. Subsequently, the learned molecular representation is fed into a classifier g ( · ) to obtain the prediction ˆ y = g ( h m ) . The model is trained by minimizing the discrepancy between ˆ y and the ground truth y .

Further, two key discoveries have been pivotal for FSMPP. The first is the proven effectiveness of pre-trained molecular encoders, while the second is the significant advantage gained from molecular context. Together, these discoveries have further reshaped the widely adopted FSMPP framework, which combines a pre-trained encoder followed by a context-aware classifier , as shown in Figure 2(b).

3.3 Pre-trained molecular encoders (PMEs)

Due to the scarcity of labeled data in molecular tasks, molecular pre-training has emerged as a crucial area, which involves training encoders on extensive molecular datasets to extract informative representations. Pre-GNN [20] is a classic pre-trained molecular encoder that has been widely used in addressing FSMPP tasks [14, 58, 73]. The backbone of Pre-GNN is a modified version of Graph Isomorphism Network (GIN) [65] tailored to molecules, which we call GIN-Mol, consisting of multiple atom/bond embedding layers and message passing layers.

Atom/Bond embedding layers. The raw atom features and bond features are both categorical vectors, denoted as ( i v, 1 , i v, 2 , . . . , i v, | E n | ) and ( j e, 1 , j e, 2 , . . . , j e, | E e | ) for atom v and bond e , respectively. These categorical features are embedded as:

where EmbAtom a ( · ) a ∈{ 1 ,..., | E n |} and EmbBond b ( · ) b ∈{ 1 ,..., | E e |} represent embedding operations that map integer indices to d -dimensional real vectors, i.e., h (0) v , h ( l ) e ∈ R d , l ∈ { 0 , 1 , . . . , L -1 } represents the index of encoder layers, and L is the number of encoder layers. The atom embedding layer is present only in the first encoder layer, while an bond embedding layer exists in each layer.

Message passing layers. At the l -th encoder layer, atom representations are updated by aggregating the features of neighboring atoms and chemical bonds:

where u ∈ N ( v ) ∪{ v } is the set of atoms connected to v , and h ( l ) v ∈ R d is the learned representation of atom v at the l -th layer. MLP ( · ) is implemented by 2-layer neural networks, in which the hidden dimension is d 1 . After MLP , batch normalization is applied right before the ReLU . The molecule-level representation h m ∈ R d is obtained by averaging the atom representations at the final layer.

4 The proposed Pin-Tuning method

This section delves into our motivation and proposed method. Our framework for FSMPP is depicted in Figure 2(c). The details of our principal design, Pin-Tuning for PMEs, is present in Figure 2(d).

As shown in Figure 1, pretraining then finetuning molecular encoders is a common approach. However, fully fine-tuning yields results inferior to simply freezing them. Thus, the following question arises:

How to effectively adapt pre-trained molecular encoders to downstream tasks, especially in few-shot scenarios?

We analyze the reasons of observed ineffective fine-tuning issue, and attribute it to two primary factors: (i) imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and (ii) limited contextual perceptiveness in the encoder.

4.1 Parameter-efficient tuning for PMEs

To address the first cause of observed ineffective tuning, we reform the tuning method for PMEs. Instead of conducting full fine-tuning for all parameters, we propose tuning strategies specifically tailored to the message passing layers and embedding layers in PMEs, respectively.

Pre-Trained

Molecular Encoder

Molecular Encoder

Property Classifier

Pre-Trained

Molecular Encoder

Context-Aware

Property Classifier

SR-MMP

SR-HSE

SR-p53

Property Classifier

Property Classifier

Property Classifier

ො 𝑦

ො 𝑦

ො 𝑦

ො 𝑦

ො 𝑦

ො 𝑦

ො 𝑦

Pin-Tuning for PMEs

Atom Embedding Layer

Message Passing

Layer

Bond Embedding

Layer

FeedForward

Down-Project

FeedForward

Up-Project

Nonlinearity

Atoms

Bonds

Molecular

Context

Layer Norm

= -

1

2

Φ𝑖

′ -Φ𝑖

T 𝐇(𝒟𝒫, Φ𝑖 )(Φ

𝑖

′ -Φ𝑖)

Frozen

Fine-Tuning

Pin-Tuning

(b) Existing FSMPP framework.

(c) Our FSMPP framework.

(d) Our Pin-Tuning method for PMEs.

𝒉𝐺

Molecular

Context

Molecular

Context

𝐿 ×

ℒEmb-BWC

𝑚1

𝑠

𝑚2

𝑠

𝑚 𝑞

𝑝𝑡

𝑝 𝑠𝑒𝑒𝑛

Context

Encoder

↑个

11

↑

4.1.1 MP-Adapter: message passing layer-oriented adapter

For message passing layers in PMEs, the number of parameters is disproportionately large compared to the training samples. To mitigate this imbalance, we design a lightweight adapter targeted at the message passing layers, called MP-Adapter . The pre-trained parameters in each message passing layer include parameters in the MLP and the following batch normalization. We freeze all pre-trained parameters in message passing layers and add a lightweight trainable adapter after MLP in each message passing layer. Formally, the adapter module for l -th layer can be represented as:

where FeedForward ( · ) denotes feed forward layer and LayerNorm ( · ) denotes layer normalization. To limit the number of parameters, we introduce a bottleneck architecture. The adapters downscale the original features from d dimensions to a smaller dimension d 2 , apply nonlinearity ϕ , then upscale back to d dimensions. By setting d 2 smaller than d , we can limit the number of parameters added. The adapter module has a skip-connection internally. With the skip-connection, we adopt the near-zero initialization for parameters in the adapter modules, so that the modules are initialized to approximate identity functions. Therefore, the encoder with initialized adapters is equivalent to the pre-trained encoder. Furthermore, we add a layer normalization after skip-connection for training stability.

4.1.2 Emb-BWC: embedding layer-oriented Bayesian weight consolidation

Unlike message passing layers, embedding layers contain fewer parameters. Therefore, we directly fine-tune the parameters of the embedding layers, but impose a constraint to limit the magnitude of parameter updates, preventing aggressive optimization and catastrophic forgetting.

The parameters in an embedding layer consist of an embedding matrix used for lookups based on the indices of the original features. We stack the embedding matrices of all embedding layers to form Φ ∈ R E × d , where E represents the total number of lookup entries. Further, Φ i ∈ R d denotes the i -th row's embedding vector, and Φ i,j ∈ R represents the j -th dimensional value of Φ i .

To avoid aggressive optimization of Φ , we derive a Bayesian weight consolidation framework tailored for embedding layers, called Emb-BWC , by applying Bayesian learning theory [3] to fine-tuning.

Proposition 1: ( Emb-BWC ensures an appropriate stability-plasticity trade-off for pre-trained embedding layers.) Let Φ ∈ R E × d be the pre-trained embeddings before fine-tuning, and Φ ′ ∈ R E × d be the fine-tuned embeddings. Then, the embeddings can both retain the atom and bond properties

obtained from pre-training and be appropriately updated to adapt to downstream FSMPP tasks, by introducing the following Emb-BWC loss into objective during the fine-tuning process:

where H ( D P , Φ i ) ∈ R d × d is the Hessian of the log likelihood L P of pre-training dataset D P at Φ i .

Details on the theoretical derivation of Eq. (6) are given in Appendix A. Since H ( D P , Φ i ) is intractable to compute due to the great dimensionality of Φ , we adopt the diagonal approximation of Hessian. By approximating H as a diagonal matrix, the j -th value on the diagonal of H can be considered as the importance of the parameter Φ i,j . The following three choices are considered.

Identity matrix. When using the identity matrix to approximate the negation of H , Eq. (6) is simplified to L IM Emb-BWC = 1 2 ∑ E i =1 ∑ d j =1 (Φ ′ i,j -Φ i,j ) 2 , assigning equal importance to each parameter. This loss function is also known as L2 penalty with pre-trained model as the starting point (L2-SP) [31].

Diagonal of Fisher information matrix. The Fisher information matrix (FIM) F is the negation of the expectation of the Hessian over the data distribution, i.e., F = -E D P [ H ] , and the FIM can be further simplified with a diagonal approximation. Then, the Eq. (6) is simplified to L FIM Emb-BWC = 1 2 ∑ E i =1 ˆ F i (Φ ′ i -Φ i ) 2 , where ˆ F i ∈ R d is the diagonal of F ( D P , Φ i ) ∈ R d × d and the j -th value in ˆ F i is computed as E D P ( ∂ L P /∂ Φ i,j ) 2 . This is equivalent to elastic weight consolidation (EWC) [24].

Diagonal of embedding-wise Fisher information matrix. In different property prediction tasks, the impact of the same atoms and inter-atomic interactions may be significant or negligible. Therefore, we propose this choice to assign importance to parameters based on different embeddings, rather than treating each parameter individually. By defining ˜ Φ i = ∑ j Φ i,j , the total update of the embedding Φ i can be represented as ∆Φ i = ˜ Φ ′ i -˜ Φ i = ∑ j (Φ ′ i,j -Φ i,j ) . Then, the Eq. (6) is reformulated to L EFIM Emb-EWC = 1 2 ∑ E i =1 ˜ F i ( ˜ Φ ′ i -˜ Φ i ) 2 , where ˜ F i = ∑ j E D P ( ∂ L P /∂ Φ i,j ) 2 .

Detailed derivation is given in Appendix A. Intuitively, these three approximations employ different methods to assign importance to parameters. L IM Emb-BWC assigns the same importance to each parameter, L FIM Emb-BWC assigns individual importance to each parameter, and L EFIM Emb-BWC assigns the same importance to parameters within the same embedding vector.

4.2 Enabling contextual perceptiveness in MP-Adapter

For different property prediction tasks, the decisive substructures vary. As shown in Figure 2, the ester group in the given molecule determines the property SR-HSE, while the carbon-carbon triple bond determines the property SR-MMP. If fine-tuning can be guided by molecular context, encoding context-specific molecular representations allows for dynamic representations of molecules tailored to specific tasks and enables the modeling of the context-specific significance of substructures.

Extracting molecular context information. In each episode, we consider the labels of the support molecules on the target property and seen properties, as well as the labels of the query molecules on seen properties, as the context of this episode. We adopt the form of a graph to describe the context. Figure 3 demonstrates the transformation from original context data to a context graph. In the left table, the labels of molecules m q 1 , m 2 2 for property p t are the prediction targets, and the other shaded values are the available context. The right side shows the

𝒑𝒕

𝒑𝟏

𝒔𝒆𝒆𝒏

𝒑𝟐

𝒔𝒆𝒆𝒏

𝒑𝟑

𝒔𝒆𝒆𝒏

𝑚1

𝑠

1

1

0

?

𝑚2

𝑠

1

0

0

1

𝑚3

𝑠

0

0

?

0

𝑚4

𝑠

0

1

1

0

𝑚1

q

?

0

0

1

𝑚2

q

?

1

1

0

𝑚1

𝑠

𝑚2

𝑠

𝑚3

𝑠

𝑚4

𝑠

𝑚1

𝑞

𝑚2

𝑞

𝑝𝑡

𝑝 1

𝑠𝑒𝑒𝑛

𝑝 1

𝑠𝑒𝑒𝑛

𝑝 1

𝑠𝑒𝑒𝑛

Context Information

Positive Label

Negative Label

Unknown Label

Molecule

Property

context graph constructed based on the available context. Specifically, we construct context graph G t = ( V t , A t , X t ) for episode E t . It contains M molecule nodes { m } and P property nodes { p } . Three types of edges indicate different relationships between molecules and properties.

Then we employ a GNN-based context encoder: C = ContextEncoder ( V t , A t , X t ) , where C ∈ R ( M + P ) × d 2 denotes the learned context representation matrix for E t . V t and A t denote the node set

and the adjacent matrix of the context graph, respectively, and X t denotes the initial features of nodes. The features of molecule nodes are initialized with a pre-trained molecular encoder. The property nodes are randomly initialized. When we make the prediction of molecule m 's target property p , we take the learned representations of the this molecule c m and of the target property c p as the context vectors. Details about the context encoder are provided in Appendix F.2.

In-context tuning with molecular context information. After obtaining the context vectors, we consider enabling the molecular encoder to use the context as a condition, achieving conditional molecular encoding. To achieve this, we further refine our adapter module. While neural conditional encoding has been explored in some domains, such as cross-attention [43] and ControlNet [68] for conditional image generation, these methods often come with a significant increase in the number of parameters. This contradicts our motivation of parameter-efficient tuning for few-shot tasks. In this work, we adopt a simple yet effective method. We directly concatenate the context with the output of the message passing layer, and feed them into the downscaling feed-forward layer in the MP-Adapter . Formally, the downscaling process defined in Eq. (3) is reformulated as:

where ∥ denotes concatenation. Such learned molecular representations are more easily predicted on specific properties, verified in Section 5.5 and Appendix G.

4.3 Optimization

Following MAML [10], a gradient descent strategy is adopted. Firstly, B episodes { E t } B t =1 are randomly sampled. For each episode, in the inner-loop optimization, the loss on the support set is computed as L cls t, S ( f θ ) and the parameters θ are updated by gradient descent:

where α inner is the learning rate. In the outer loop, the classification loss of query set is denoted as L cls t,Q . Together with our Emb-BWC regularizer, the meta-training loss L ( f θ ′ ) is computed and we do an outer-loop optimization with learning rate α outer across the mini-batch:

where λ is the weight of Emb-BWC regularizer. The pseudo-code is provided in Appendix B. We also provide more discussion of tunable parameter size and total model size in Appendix C.

5 Experiments

5.1 Evaluation setups

Datasets. We use five common few-shot molecular property prediction datasets from the MoleculeNet [61]: Tox21, SIDER, MUV, ToxCast, and PCBA. Standard data splits for FSMPP are adopted. Dataset statistics and more details of datasets can be found in Appendix D.

Baselines. For a comprehensive comparison, we adopt two types of baselines: (1) methods with molecular encoders trained from scratch, including Siamese Network [25], ProtoNet [46], MAML [10], TPN [35], EGNN [22], and IterRefLSTM [1]; and (2) methods which leverage pretrained molecular encoders, including Pre-GNN [20], Meta-MGNN [14], PAR [58], and GS-Meta [73]. More details about these baselines are in Appendix E.

Metrics. Following prior works [1, 58], ROC-AUC scores are calculated on the query set for each meta-testing task, to evaluate the performance of FSMPP. We run experiments 10 times with different random seeds and report the mean and standard deviations.

5.2 Performance comparison

We compare Pin-Tuning with the baselines and the results are summarized in Table 1, Table 7, and Table 8. Our method significantly outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the effectiveness and superiority of our approach.

Across all datasets, our method provides greater improvement in the 10-shot scenario than in the 5-shot scenario. This is attributed to the molecular context constructed based on support molecules. When there are more molecules in the support set, the uncertainty in the context is reduced, providing more effective adaptation guidance for our parameter-efficient tuning.

Among benchmark datasets, our method shows significant improvement on the SIDER dataset, increasing by 10.73% in the 10-shot scenario and by 8.81% in the 5-shot scenario. We consider this is related to the relatively balanced ratio of positive to negative samples, as well as the absence of missing labels in the SIDER dataset (Table 5). A balanced and low-uncertainty distribution can better benefit addressing the FSMPP task from our method.

We also observe that the standard deviations of our method's results under 10 seeds are slightly higher than that of baseline models. However, our worst-case results are still better than the best baseline model. For example, in 10-shot experiments on the Tox21 dataset, the performance of our method is 91 . 56 ± 2 . 57 . However, our 10 runs yield specific results with the worst-case ROC-AUC reaching 88.02, which is also better than the best baseline model GS-Meta's result of 86 . 67 ± 0 . 41 . Therefore, a high standard deviation does not mean our method is inferior to baseline models.

5.3 Ablation study

For MP-Adapter , the main components consist of: (i) bottleneck adapter module (Adapter), (ii) introducing molecular context to adatpers (Context), and (iii) layer normalization (LayerNorm). The results of ablation experiments are summarized in Table 2. The bottleneck adapter and the modeling of molecular context are the most critical, having the most significant impact on performance. Removing them leads to a noticeable decline, which underscores the importance of parameter-efficient tuning and context perceptiveness in FSMPP tasks. Layer normalization is used to normalize the resulting representations, which is also important for improving the optimization effect and stability.

For Emb-BWC , we verify the effectiveness of finetuning the embedding layers and regularizing them with different approximations of L Emb-BWC (Table 3). Since the embedding layers have relatively few parameters, direct fine-tuning can also enhance performance. Applying our proposed regularizers to fine-tuning can further improve the effects. Among the three regularizers, the

L IM Emb-BWC is the most effective. This indicates that keeping pre-trained parameters to some extent can better utilize pre-trained knowledge, but the parameters worth keeping in fine-tuning and the important parameters in pre-training revealed by Fisher information matrix are not completely consistent.

5.4 Sensitivity analysis







































d

2

























































×1

e

6











d

2=25





d

2=50





d

2=75





d

2 =100





d

2 =150





Effect of weight of Emb-BWC regularizer λ . Emb-BWC is applied on the embedding layers to limit the magnitude of parameter updates during fine-tuning. We vary the weight of this regularization λ from { 0 . 01 , 0 . 1 , 1 , 10 } . The first subfigure in Figure 4 shows that the performance is best when λ = 0 . 1 or 1 . When λ is too small, the parameters undergo too large updates on few-shot downstream datasets, leading to over-fitting and ineffectively utilizing the pre-trained knowledge. Too large λ causes the parameters of the embedding layers to be nearly frozen, which prevents effective adaptation.

Effect of hidden dimension of MP-Adapter d 2 . The results corresponding to different values of d 2 from { 25 , 50 , 75 , 100 , 150 } are presented in the second subfigure of Figure 4. On the Tox21 dataset, we further analyze the impact of this hyper-parameter on the number of trainable parameters. As shown in Figure 5, the number of parameters that our method needs to train is significantly less than that required by the full fine-tuning method, such as GS-Meta, while our method also performs better in terms of ROC-AUC performance due to solving over-fitting and context perceptiveness issues. When d = 50 , Pin-Tuning performs best on Tox21, and the number of parameters that need to train is only 14.2% of that required by traditional fine-tuning methods.

5.5 Case study

Dataset:Tox21,Property:SR-MMP

Dataset:Tox21,Property:SR-p53

Negative

Negative

Positive

Positive

Dataset:Tox21,Property:SR-MMP

Dataset:Tox21,Property:SR-p53

Negative

Negative

Positive

Positive

We visualized the molecular representations learned by the GS-Meta and our Pin-Tuning 's encoders in the 10-shot setting, respectively. As shown in Figure 6 and 7, Pin-Tuning can effectively adapt to different downstream tasks based on context information, generating property-specific molecular representations. Across different tasks, our method is more effective in encoding representations that

facilitate the prediction of the current property, reducing the difficulty of property prediction from the encoding representation aspect. More case studies are provided in Appendix G.

6 Conclusion

In this work, we propose a tuning method, Pin-Tuning , to address the ineffective fine-tuning of pre-trained molecular encoders in FSMPP tasks. Through the innovative parameter-efficient tuning and in-context tuning for pre-trained molecular encoders, our approach not only mitigates the issues of parameter-data imbalance but also enhances contextual perceptiveness. The promising results on public datasets underscore the potential of Pin-Tuning to advance this field, offering valuable insights for future research in drug discovery and material science.

Acknowledgments

This work is jointly supported by National Science and Technology Major Project (2023ZD0120901), National Natural Science Foundation of China (62372454, 62236010) and the Excellent Youth Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.

References

Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS Central Science , 3(4):283-293, 2017.

Dingshuo Chen, Yanqiao Zhu, Jieyu Zhang, Yuanqi Du, Zhixun Li, Q. Liu, Shu Wu, and Liang Wang. Uncovering neural scaling laws in molecular representation learning. In NeurIPS , 2023.

Haolin Chen and Philip N. Garner. Bayesian parameter-efficient fine-tuning for overcoming catastrophic forgetting. arXiv , abs/2402.12220, 2024.

Wenlin Chen, Austin Tripp, and José Miguel Hernández-Lobato. Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. In ICLR , 2023.

Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. A systematic study of key elements underlying molecular property prediction. Nature Communications , 14, 2023.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv , abs/2203.06904, 2022.

Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Y. Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , 5:220-235, 2023.

Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. 2023.

Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence , 4:127 - 134, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML , 2017.

Jonathan Godwin, Michael Schaarschmidt, Alexander L. Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Velickovic, James Kirkpatrick, and Peter W. Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In ICLR , 2022.

Renxiang Guan, Zihao Li, Wenxuan Tu, Jun Wang, Yue Liu, Xianju Li, Chang Tang, and Ruyi Feng. Contrastive multiview subspace clustering of hyperspectral images based on graph convolutional networks. IEEE Transactions on Geoscience and Remote Sensing , 2024.

Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V. Chawla. Few-shot graph learning for molecular property prediction. In WWW , 2021.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In ICLR , 2022.

Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44:5149-5169, 2020.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD , 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML , 2019.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR , 2022.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR , 2020.

Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. In NeurIPS , 2022.

Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D. Yoo. Edge-labeling graph neural network for few-shot learning. In CVPR , 2019.

Suyeon Kim, Dongha Lee, SeongKu Kang, Seonghyeon Lee, and Hwanjo Yu. Learning topology-specific experts for molecular property prediction. In AAAI , pages 8291-8299, 2023.

James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114, 117.

Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop , 2015.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP (1) , pages 3045-3059. Association for Computational Linguistics, 2021.

Han Li, Dan Zhao, and Jianyang Zeng. KPGT: knowledge-guided pre-training of graph transformer for molecular property prediction. In KDD , pages 857-867, 2022.

Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgcl: Geometric graph contrastive learning for molecular property prediction. In AAAI , pages 4541-4549, 2022.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL , 2021.

Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. In NeurIPS , 2023.

Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML , 2018.

Zhixun Li, Liang Wang, Xin Sun, Yifan Luo, Yanqiao Zhu, Dingshuo Chen, Yingtao Luo, Xiangxin Zhou, Qiang Liu, Shu Wu, Liang Wang, and Jeffrey Xu Yu. GSLB: the graph structure learning benchmark. In NeurIPS , 2023.

Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv , abs/2303.15647, 2023.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv , abs/2103.10385, 2021.

Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR , 2019.

Yue Liu, Jun Xia, Sihang Zhou, Siwei Wang, Xifeng Guo, Xihong Yang, Ke Liang, Wenxuan Tu, Stan Z. Li, and Xinwang Liu. A survey of deep graph clustering: Taxonomy, challenge, and application. arXiv , abs/2211.12875, 2022.

Yue Liu, Ke Liang, Jun Xia, Sihang Zhou, Xihong Yang, Xinwang Liu, and Stan Z. Li. Dink-net: Neural clustering on large graphs. In ICML , 2023.

Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder in masked graph modeling for molecules. In NeurIPS , 2023.

Qiujie Lv, Guanxing Chen, Ziduo Yang, Weihe Zhong, and Calvin Yu-Chian Chen. Meta learning with graph attention networks for low-data drug discovery. IEEE transactions on neural networks and learning systems , 2023.

Ziqiao Meng, Yaoman Li, Peilin Zhao, Yang Yu, and Irwin King. Meta-learning with motifbased task augmentation for few-shot molecular property prediction. In SDM , 2023.

Cuong Q. Nguyen, Constantine Kreatsoulas, and Kim Branson. Meta-learning gnn initializations for low-resource molecular property prediction. In ICML Workshop on Graph Representation Learning and Beyond , 2020.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In EACL , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR , pages 10674-10685. IEEE, 2022.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS , 2020.

Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve fewshot drug discovery. In ICLR , 2023.

Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NeurIPS , 2017.

Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang-Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJCAI , 2020.

Yisheng Song, Ting Wang, Puyu Cai, Subrota K. Mondal, and Jyoti Prakash Sahoo. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys , 2023.

Megan Stanley, John Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin H. S. Segler, Nadine Schneider, and Marc Brockschmidt. Fs-mol: A few-shot learning dataset of molecules. In NeurIPS Datasets and Benchmarks , 2021.

Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Lió. 3d infomax improves gnns for molecular property prediction. In ICML , pages 20479-20502, 2022.

Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In KDD , 2021.

Xin Sun, Liang Wang, Qiang Liu, Shu Wu, Zilei Wang, and Liang Wang. DIVE: subgraph disagreement for graph out-of-distribution generalization. In KDD , 2024.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurIPS , page 3637-3645, 2016.

Liang Wang, Xiang Tao, Qiang Liu, Shu Wu, and Liang Wang. Rethinking graph masked autoencoders through alignment and uniformity. In AAAI , 2024.

Liang Wang, Shu Wu, Q. Liu, Yanqiao Zhu, Xiang Tao, and Mengdi Zhang. Bi-level graph structure learning for next poi recommendation. IEEE Transactions on Knowledge and Data Engineering , 36:5695-5708, 2024.

Xu Wang, Huan Zhao, Wei-Wei Tu, and Quanming Yao. Automated 3d pre-training for molecular property prediction. In KDD , pages 2419-2430, 2023.

Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys , 2020.

Yaqing Wang, Abulikemu Abuduweili, Quanming Yao, and Dejing Dou. Property-aware relation networks for few-shot molecular property prediction. In NeurIPS , 2021.

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adaptations for parameter-efficient model tuning. In EMNLP , 2022.

Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence , 2022.

Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. arXiv , abs/1703.00564, 2017.

Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In ICLR , 2023.

Yi Xiao, Xiangxin Zhou, Qiang Liu, and Liang Wang. Bridging text and molecule: A survey on multimodal frameworks for molecule. arXiv , abs/2403.13830, 2024.

Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. TPAMI , 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS , 2021.

Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter W. Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. In ICLR , 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , pages 3813-3824. IEEE, 2023.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR , 2023.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In ICLR , 2024.

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. In NeurIPS , pages 15870-15882, 2021.

Yanqiao Zhu, Dingshuo Chen, Yuanqi Du, Yingze Wang, Q. Liu, and Shu Wu. Molecular contrastive pretraining with collaborative featurizations. Journal of Chemical Information and Modeling , 2024.

Xiang Zhuang, Qiang Zhang, Bin Wu, Keyan Ding, Yin Fang, and Huajun Chen. Graph sampling-based meta-learning for molecular property prediction. In IJCAI , 2023.

̸

Appendix

The organization of the appendix is as follows:

Appendix A: Derivation of Emb-BWC regularization;

Appendix B: Pseudo-code of training process;

Appendix C: Discussion of tunable parameter size and total model size;

Appendix D: Details of datasets;

Appendix E: Details of baselines;

Appendix F: Implementation details;

Appendix G: More experimental results and discussions;

Appendix H: Limitations and future directions.

A Derivation of Emb-BWC regularization

A.1 Derivation of L Emb-BWC

Let Φ ∈ R E × d be the pre-trained embeddings before fine-tuning, and Φ ′ ∈ R E × d be the fine-tuned embeddings. Further, Φ i ∈ R d denotes the i -th row's embedding vector in Φ , and Φ i,j ∈ R represents the j -th dimensional value of Φ i .

The optimization of embedding layers can be interpreted as performing a maximum a posterior (MAP) estimation of the parameters Φ ′ given the pre-training data and training data of downstream FSMPP task, which is formulated in a Bayesian framework.

In the FSMPP setting, the molecular encoder has been pre-trained on the pre-training task P using data D P , and is then fine-tuned on a downstream FSMPP task F using data D F . The overall objective is to find the optimal parameters on task F while preserving the prior knowledge obtained in pretraining on task F . Based on a prior p (Φ ′ ) of the embedding parameters, the posterior after observing the FSMPP task F can be computed with Bayes' rule:

where D F is assumed to be independent of D P . Taking a logarithm of the posterior, the MAP objective is therefore:

The first term log p ( D F | Φ ′ ) is the log likelihood of the data D F given the parameters Φ ′ , which can be expressed as the training loss function on task F = -log p ( D F | Φ ′ ) , denoted as L F (Φ ′ ) . The second term p (Φ ′ |D P ) is the posterior of the parameters given the pre-training dataset D P . Since Φ ′ = [Φ ′⊤ 1 , Φ ′⊤ 2 , . . . , Φ ′⊤ E ] ⊤ , and Φ ′ i is conditionally independent of Φ ′ j for i, j = { 1 , . . . , E } and i = j given condition D P , we have p (Φ ′ |D P ) = ∏ E i =1 p (Φ ′ i |D P ) . Thus, log p (Φ ′ |D P ) = ∑ E i =1 log p (Φ ′ i |D P )

For adapting pre-trained molecular embedding layers to downstream FMSPP tasks, this posterior must encompass the prior knowledge of the pre-trained embedding layers to reflect which parameters are important for pre-training task P . Despite the true posterior being intractable, log p (Φ ′ i |D P ) can be defined as a function f (Φ ′ i ) and approximated around the optimum point f (Φ i ) , where f (Φ i ) is the pre-trained values and ∇ f (Φ i ) = 0 . Performing a second-order Taylor expansion on f (Φ ′ i ) around Φ i gives:

where H ( D P , Φ i ) ∈ R d × d is the Hessian matrix of f (Φ ′ i ) at Φ i . The second term suggests that the posterior of the parameters on the pre-training data can be approximated by a Gaussian distribution with mean Φ ′ i and covariance H ( D P , Φ i ) -1 . Following Eq. (13), the training objective becomes:

We define the second term as our Emb-BWC regularization objective:

A.2 Derivation of L FIM Emb-BWC

Since the Fisher information matrix (FIM) F is the negation of the expectation of the Hessian over the data distribution, i.e., F = -E D P [ H ] , the objective can be reformulated as:

where F ( D P , Φ i ) ∈ R d × d is the corresponding Fisher information matrix of H ( D P , Φ i ) . Further, the Fisher information matrix can be further simplified with a diagonal approximation. Then, the objective is simplified to:

where ˆ F i ∈ R d is the diagonal of F ( D P , Φ i ) . According to the definition of the Fisher information matrix, the j -th value in ˆ F i is computed as E D P ( ∂ L P /∂ Φ i,j ) 2 . In this work, this approximated form is defined as L FIM Emb-BWC .

A.3 Derivation of L EFIM Emb-BWC

We assume that the parameters within an embedding should share the same importance. To this end, we define ˜ Φ i = ∑ j Φ i,j , then the total update of the embedding Φ i can be represented as ∆Φ i = ˜ Φ ′ i -˜ Φ i = ∑ j (Φ ′ i,j -Φ i,j ) . Then, the objective in Eq. (16) is reformulated to:

where ˜ H i = ∂ 2 L P ∂ ˜ Φ 2 i = ∂ ∂ ˜ Φ i ( ∂ L P ∂ ˜ Φ i ) . Next, we continue to derive ˜ H i . Given that ˜ Φ i = ∑ d j =1 Φ i,j , we first use the chain rule to find ∂ L P ∂ ˜ Φ i According to chain rule, the derivative of L P with respect to ∂ ˜ Φ i can be computed as:

Since ˜ Φ i = Φ i, 1 + Φ i, 2 + . . . + Φ i,d , each of ∂ Φ i,j ∂ ˜ Φ i for j = 1 , 2 , . . . , d equals 1 . Therefore, the equation simplifies to:

When taking the derivative of this with respect to Φ i,j again, using the chain rule, we have:

Given that Φ i,j (Φ i,k ) are all parameters in embedding lookup tables, they are independent of each other. Thus, when j = k , ∂ ∂ Φ i,k ( ∂ L P ∂ Φ i,j ) ∂ Φ i,k ∂ ˜ Φ i = ∂ 2 L P ∂ Φ 2 i,j , otherwise it equals 0. We finally get

We still approximate the Hessian with the FIM as in Section A.2, and combining this with the definition of FIM, we arrive at the final objective:

where ˜ F i = ∑ j E D P ( ∂ L P /∂ Φ i,j ) 2 .

B Pseudo-code of training process

To help better understand the training process, we provide the brief pseudo-code of it in Algorithm 1.

C Discussion of tunable parameter size and total model size

C.1 Tunable parameter size of molecular encoder

We compare the tunable parameter size of full fine-tuning and our Pin-Tuning . Section 3.3 describes the parameters of the PME, which include those for the embedding layers and the message passing layers. We assume there are | E n | original node features and | E e | edge features. Considering there is one node embedding layer and L edge embedding layers, the total number of parameters for the embedding part is | E n | d + L | E e | d . The parameters in the message passing layer consist of the 2-layer MLP including biases shown in Eq. (2) and its subsequent batch normalization, with each layer having L (2 dd 1 + d + d 1 +2 d ) parameters. In summary, the total number of parameters to update in full fine-tuning is

In our Pin-Tuning method, the parameters of the embedding layers are still updated. However, in each message passing layer, the original parameters are completely frozen, and the parts that require updating are the two feed-forward layers and the layer normalization in the bottleneck adapter

module, amounting to L (2 dd 2 + d + d 2 +2 d ) parameters for this part. Therefore, the total number of parameters that need to be updated in our Pin-Tuning is

The difference in the number of parameters updated between the two tuning methods is ∆ N = ( d 1 -d 2 ) L (2 d +1) .

C.2 Total model size

We provide a comparison of total model size between our Pin-Tuning and the state-of-the-art baseline method, GS-Meta. The total model size consists of both frozen parameters and trainable parameters. The results are presented in Table 4. The total size of our model is comparable to GS-Meta, but the number of parameters that need to be trained is far less than GS-Meta.

D Details of datasets

We carry out experiments in MoleculeNet benchmark [61] on five widely used few-shot molecular property prediction datasets:

Tox21 : This dataset covers qualitative toxicity measurements and was utilized in the 2014 Tox21 Data Challenge.

SIDER : The Side Effect Resource (SIDER) functions as a repository for marketed drugs and adverse drug reactions (ADR), categorized into 27 system organ classes.

MUV : The Maximum Unbiased Validation (MUV) is determined through the application of a refined nearest neighbor analysis, specifically designed for validating virtual screening techniques.

ToxCast : This dataset comprises a compilation of compounds with associated toxicity labels derived from high-throughput screening.

PCBA : PubChem BioAssay (PCBA) represents a database containing the biological activities of small molecules generated through high-throughput screening.

Dataset statistics are summarized in Table 5 and Table 6.

E Details of baselines

We compare our Pin-Tuning with two types of baseline models for few-shot molecular property prediction tasks, categorized according to the training strategy of molecular encoders: trained-fromscratch methods and pre-trained methods.

Trained-from-scratch methods:

Siamese [25]: Siamese is used to rank similarity between input molecule pairs with a dual network.

ProtoNet [46]: ProtoNet learns a metric space for few-shot classification, enabling classification by calculating the distances between each query molecule and the prototype representations of each class.

MAML [10]: MAML adapts the meta-learned parameters to achieve good generalization performance on new tasks with a small amount of training data and gradient steps.

TPN [35]: TPN classifies the entire test set at once by learning to propagate labels from labeled instances to unlabeled test instances using a graph construction module that exploits the manifold structure in the data.

EGNN [22]: EGNN predicts edge labels on a graph constructed from input samples to explicitly capture intra-cluster similarity and inter-cluster dissimilarity.

IterRefLSTM [1]: IterRefLSTM adapts Matching Networks [53] to handle molecular property prediction tasks.

Pre-trained methods:

Pre-GNN [20]: Pre-GNN is a classic pre-trained molecular model, taking the GIN as backbone and pre-training it with different self-supervised tasks.

Meta-MGNN [14]: Meta-MGNN leverages Pre-GNN for learning molecular representations and incorporates meta-learning and self-supervised learning techniques.

PAR [58]: PAR uses class prototypes to update input representations and designs label propagation for similar inputs in the relational graph, thus enabling the transformation of generic molecular embeddings into property-aware spaces.

GS-Meta [73]: GS-Meta constructs a Molecule-Property relation graph (MPG) and redefines episodes in meta-learning as subgraphs of the MPG.

Following prior work [58], for the methods we reproduced, we use GIN as the graph-based molecular encoder to extract molecular representations. Specifically, we use the GIN provided by Pre-GNN [20] which consists of 5 GIN layers with 300-dimensional hidden units. Pre-GNN, Meta-MGNN, PAR, and GS-Meta further use the pre-trained GIN which is also provided by Pre-GNN.

F Implementation details

F.1 Hardware and software

Our experiments are conducted on Linux servers equipped with an AMD CPU EPYC 7742 (256) @ 2.250GHz, 256GB RAM and NVIDIA 3090 GPUs. Our model is implemented in PyTorch version 1.12.1, PyTorch Geometric version 2.3.1 (https://pyg.org/) with CUDA version 11.3, RDKit version 2023.3.3 and Python 3.9.18. Our code is available at: https://github.com/CRIPAC-DIG/ Pin-Tuning .

F.2 Model configuration

For featurization of molecules, we use atomic number and chirality tag as original atom features, as well as bond type and bond direction as bond features, which is in line with most molecular

pre-training methods. Following previous works, we set d = 300 . For MLPs in Eq. (2), we use the ReLU activation with d 1 = 600 . Pre-trained GIN model provided by Pre-GNN [20] is adopted as the PTME in our framework. We tune the weight of update constraint (i.e., λ ) in {0.01, 0.1, 1, 10}, tune the learning rate of inner loop (i.e., α inner) in {1e-3, 5e-3, 1e-2,5e-2,1e-1, 5e-1, 1, 5}, and tune the learning rate of outer loop (i.e., α outer) in {1e-5, 1e-4, 1e-3,1e-2,1e-1}. Based on the results of hyperparameter tuning, we adopt α inner = 0 . 5 , α outer = 1 e -3 and d 2 = 50 . The ContextEncoder ( · ) described in Section 4.2 is implemented using a 2-layer message passing neural network [11]. In each MPNN layer, we employ a linear layer to aggregate messages from the neighborhoods of nodes and utilize distinct edge features to differentiate between various edge types in the context graphs. For baselines, we follow their recommended settings.

G More experimental results and discussions

More discussion of Figure 1. The results show that molecular encoders with more molecule-specific inductive biases, such as CMPNN [47] and Graphormer [66], performed slightly worse than GINMol [20] on this few-shot task. This is because more complex encoders require more parameters to provide inductive biases, which are difficult to train effectively under a few-shot setting.

More main results. The detailed comparison between Pin-Tuning and baseline models on subdatasets of ToxCast are summarized in Table 7 and Table 8. Our method outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the superiority of our method compared to existing methods.

More discussion of ablation study. Different datasets show varying sensitivity to the removal of components. On small-scale datasets like Tox21 and SIDER, removing components leads to a significant performance drop. On large-scale datasets like ToxCast and PCBA, the impact of removing components is less pronounced. This is because more episodes can be constructed on large-scale datasets, which aids in adaptation. This observation indicates that Pin-Tuning can bring considerable benefits in situations where data is extremely scarce.

More case studies. We provide more case studies in Figure 8 and 9 as a supplement to Section 5.5.

Negative

Negative

Negative

Positive

Positive

Positive

Negative

Positive

Negative

Negative

Positive

Positive

Dataset: SIDER,Property:RUD

Negative

Positive

Negative

Positive

Negative

Negative

Positive

Positive

Dataset: SIDER,Property:PPPC

Negative

Negative

Negative

Positive

Positive

Positive

Dataset: SIDER,Property:RUD

Negative

Positive

H Limitations and future directions

As we discusses in Section 5.2, although our method significantly outperforms the state-of-the-art baseline method, our method exhibits higher standard deviations in the experimental results under multiple runs with different seeds.

We further speculate that these high standard deviations might be due to the uncertainty in the context information within episodes. The explicitly introduced molecular context, on one hand, provides effective guidance for tuning pre-trained molecular encoders, but on the other hand, this information also carries a high degree of uncertainty. We aim to model the target property through the molecule-property relationships within episodes, but each episode is obtained by sampling very few samples from the large space corresponding to the target property. The uncertainty between different episodes is relatively high. How to quantify and calibrate this uncertainty is another question worth exploring, which we will investigate in our future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We present our claims and main contributions in the abstract and introduction. We propose a parameter-efficient in-context tuning method, named Pin-tuning, to effectively adapt pre-trained molecular encoders to few-shot molecular property prediction tasks.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Appendix H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the derivation of our proposed Emb-BWC regularization, with related assumptions in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The proposed method and experimental setups are clearly described. Additionally, the link of code and other details for reproducing are provided in Appendix F. The datasets are publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The link of code is provided in the abstract, and other details for environment and configurations are provided in Appendix F. The datasets are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 5.1 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars in our experimental results, such as in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Appendix F.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our method addresses the few-shot molecular property prediction problem, which is important for drug discovery and material science, as discussed in the introduction and conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly respect and credit for existing assets. We cite relevant papers and comply with the licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work introduce a new system for the FSMPP tasks, which includes: context adapter in GNN, context graph, and  weight consolidation. With these three techniques, the Pin-Tuning method achieved promising results on many FSMPP benchmarks.

STRENGTHS:
- FSMPP tasks are important and have grate research value. 
- The Pin-Tuning system is carefully designed and evaluated.

WEAKNESSES:
- The adapter based parameter efficient tunning have already been used in many tasks.

QUESTIONS:
- Why different strategies are used for embedding layers and encoder layers? What will be the results for (a) finetune encoder layers with weight constraint, and (b) freeze embedding layers and add adapters to them? 

-  According to Table 3, the simplest weight consolidation method (i.e., IM) is the best. In this case, does the BWC theory really required as it is just L2 penalty on the changed parameters?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Pin-Tuning, a parameter-efficient in-context tuning method for few-shot molecular property prediction (FSMPP), mitigating the parameter-data imbalance and enhancing the contextual perceptiveness of pre-trained molecular encoders. This method treats the embedding layers and message passing layers in the pre-trained molecular encoder separately, based on specific reasons why they could not be effectively fine-tuned before. This includes a series of Bayesian weight consolidation constraints for the embedding layers and the bottleneck adapters for the message passing layers. Additionally, molecular context is introduced into the adapter modules to enable contextual perceptiveness.  Experiments show that the proposed method effectively improves few-shot performances across a range of benchmark datasets.\n\nSTRENGTHS:\n1. The motivation is convincingly derived from observations in the pilot experiment in Figure 1, indicating the necessity to design a more effective tuning method for few-shot molecular tasks.\n2. The authors design different tuning methods for the embedding layer and the message passing layer in the pre-trained molecular encoder to accommodate their different parameter scales and parameter forms.\n3. The proposed Emb-EWC constraint is derived based on Bayesian learning theory, and the authors provide intuitive explanations for each approximation choice to make them easier to understand.\n\nWEAKNESSES:\n1. It is recommended to add details about the pilot experiment in Figure 1, such as the featurization and the specific architectures of different models, and whether the pre-training strategies are consistent.\n2. Table 4 only reports the \"total size of the molecular encoder.\" Considering that some parameters are frozen in your method, you could add a row for the \"size of the part that needs to be tuned\". This would be helpful to make the advantages of your method clearer.\n\nQUESTIONS:\n1. What are the details of the pilot experiment in Figure 1? Including the featurization, the model architecture, and the pre-training strategy of each model. More details are needed to assess the fairness of the experiment and to strengthen the persuasiveness of your claims.\n2. Does $\\mathcal{L}_{update}$ in Figure 2(d) refer to $\\mathcal{L}_{Emb-BWC}$ in Section 4.1.2? The content in the figure should be consistent with the text.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a strategy for few-shot drug discovery scenarios. The authors propose to fine-tune representations retrieved from encoder layers with adapters. In addition to the initial representations from the message passing layers, the adapters are provided with property representations and \"context-aware\" molecule representations to improve the initial representations. Both the property representations and the \"context-aware\" molecule representations stem from a context encoder. The context encoder creates learned molecule and property representations by relating the input molecule representations and the current few shot task, i.e. the property with other molecules and other already seen properties. The authors evaluate their approach in several datasets and include an ablation study.\n\nSTRENGTHS:\n**Originality**:\n- **(S-O)**:  The Context encoder contains novelty. Jointly learning task and molecule representations with GNNs and updating molecule representation based on relationships between them is novel (also see (S-Q2)).\n\n**Quality**:\n- **(S-Q1)**: The model shows strong performance for the chosen experimental setup. In the main experiment (Table 1) the authors include error bars and therefore follow best practices of comparing drug discovery models.\n- **(S-Q2)**: With their novel strategy to include in-context information, i.e. task information, the authors found a way to use label information of query and support set molecules present in the training set. This seems promising and has not been done before (S-O). E.g., compare [1] in which molecule representations are updated being aware of training molecules but not-aware of their training labels.\n\n**Clarity**:\n- **(S-C): Chapter 4 introduces very well the goal of this work. Taking into account all three textual descriptions formulas and Figure 2, the proposed method and its components become clear.\n\n**Significance**:\n- **(S-S1)**: The novel Context Encoder is relevant for the community (see (S-Q2)).\n- **(S-S2)**: For the chosen experimental setup, the author's approach outperforms chosen baseline methods. \n\n[1] Schimunek, Johannes, et al. \"Context-enriched molecule representations improve few-shot drug discovery.\" The Eleventh International Conference on Learning Representations.\n\nWEAKNESSES:\n**Originality**:\n- **(W-O1)**: Using adapters to efficiently fine-tune models is not novel. The core idea is to efficiently fine-tune a layer by adding another learnable component with less parameters. Instead of changing the model layer itself, the new component learns to modify the outcome of the model layer. This has already been done in few-shot drug discovery [2]. \n- **(W-O2)**: In-context fine-tuning in the sense of adapting molecular representations to the few-shot task is not novel and has been done e.g. here [1, 3] .\n\n**Quality**:\n- **(W-Q1)**: The authors missed to include the SOTA benchmark for few-shot drug discovery FS-Mol.\n- **(W-Q2)**: The authors missed that in-context fine-tuning (W-O2) and efficient adapter-like fine-tuning (W-O1) has already been applied to few-shot drug discovery.\n - **(W-Q3)**: Missing error bars for Figure 1, Table 2, Figure 4, and Figure 5\n\n**Significance**:\n- **(W-S1)**: The significance of the experimental section is diminished because the SOTA benchmark for few-shot drug discovery is missing (see (W-Q1))\n- **(W-S2)**: The significance of the proposed method is diminished because of (W-Q2). Being aware of the ideas which have already been applied to few-shot drug discovery would have allowed to put the context encoder into the focus of this work. The ablation study indicates that this module is important. Still in the current version of the paper, this novel piece of architecture remains underexplored.\n\n**Clarity**:\n- **(W-C1)**: Because of (W-S2, W-Q2) it remains unclear which parts of the proposed method have been tested in related work and which parts are novel. The authors might think about changing the main story line of their paper by focusing on their training label aware-fine-tuning procedure.\n- **(W-C2)**: The procedure which happens inside the context encoder is not displayed well in Figure 2. Since this is a central part, this should be improved.\n- **(W-C3)**: English editing is required\n- **(W-C4)**: The mathematical notation and formulas seem cluttered and inconsistent. E.g., in 3.2. a molecular encoder is denoted by $f(.)$ but the context encoder is denoted by ContextEncoder. Generally, the authors should stick to standard notation and avoid \"full-word notation\" in formulas.\n\n\n[2] Adler, Thomas, et al. \"Cross-domain few-shot learning by representation fusion.\" (2020).\\\n[3] Altae-Tran, Han, et al. \"Low data drug discovery with one-shot learning.\" ACS central science 3.4 (2017): 283-293.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nLimitations have been addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel Pin-Tuning method. Focusing on improving the fine-tuning process of pre-trained molecular encoders, especially for the task of Few-Shot Molecular Property Prediction (FSMPP), Pin-Tuning skillfully balances the contradiction between the number of tunable parameters and the limited labeled molecular data, while reinforcing the encoder's context-awareness. The core of the approach is the introduction of MP-Adapter, a lightweight adapter for pre-trained message-passing layers, and Emb-BWC, a Bayesian weight consolidation scheme for pre-trained atom/key embedding layers. Experimental results show that Pin-Tuning exhibits excellent performance on public datasets, significantly improves the prediction performance in few-shot scenarios with fewer trainable parameters, and proves its effectiveness in the field of molecular attribute prediction.\n\nSTRENGTHS:\n1.The paper proposes Pin-Tuning methods, including MP-Adapter and Emb-BWC, which provide new solutions to the problem of fine-tuning pre-trained models in FSMPP tasks.\n2.In the paper, it is proposed to integrate the context-aware capability into the adapter, which enhances the model's ability to perceive the molecular context and enables the model to be more effectively fine-tuned on few data samples.\n3.The Pin-Tuning method proposed in the paper is evaluated on a public dataset, showing fewer trainable parameters and greater improvement in prediction performance, which demonstrates the effectiveness of the method.\n\nWEAKNESSES:\n1.The concept of Adapter has been widely studied and applied in the field of Natural Language Processing. In this paper, does MP-Adapter just take this concept over, or is there any design and optimization specific to molecular graphical neural networks (GNNs)? Is there any comparison with existing generalized frameworks for adapter networks?\n2.The observation from the paper that the use of Identity matrix approximation works best in Emb-BWC seems to be a counterintuitive finding. This is because the Identity matrix approximation ignores possible correlations between parameters and assigns the same importance to each parameter. Is it true that correlations between parameters are less important in the molecular property prediction task than in other tasks? If so, does this reflect an intrinsic characteristic of molecular property prediction tasks?\n\nQUESTIONS:\n1.What does the a mean in Effect of weight of Emb-BWC regularizer λ in Sensitivity analysis? Is there a confusing typo here?\n2.What are the advantages of the proposed MP-Adapter over existing adapter technology?\n3.How to ensure the quality and reliability of the contextual information used in the paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}