Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction
Authors: Liang Wang
Qiang Liu
Shaozhen Liu
Xin Sun
Shu Wu
Liang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: few-shot molecular property prediction
molecular property prediction
parameter-efficient tuning
molecular representation learning
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a parameter-efficient in-context tuning method, named Pin-tuning, to effectively adapt pre-trained molecular encoders to few-shot molecular property prediction tasks.

Abstract: Molecular property prediction (MPP) is integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Addressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs a pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, existing methods struggle with the ineffective fine-tuning of pre-trained encoders. We attribute this issue to the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, we propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC), to achieve parameter-efficient tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP-Adapters with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptability for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.

Pin-Tuning: Parameter-Bfficiet In-Context Tuning for Few-Shot Molecular Property Prediction
Liang Wang 2 Qiang Liu 2 Shaozhen Liu' Xin Sun Shu Wu Liang Wang" 22
New Laboratory of Pattern Recognition (NLPR)
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)
Institute of Automation, Chinese Academy of Sciences (CASIA)
School of Artificial Intelligence, University of Chinese Academy of Sciences University of Science and Technology of China
liang. wang@cripac. ia. ac. en, qiang. liu@nlpr ia. cn, liushaozhen2025eia.. ac sunxin000@mail. ustu. edu. en, wu, wangliang) aa.ac.ccccccm........o..c.....
Abstract
Molecular property prediction (MPP) integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Ad- dressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, cxisting methods struggle with the ineffective finettning pre-trained encoders. We attribute this is- the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, propose parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers Ada-tearer) and Bayesian weight consolidation for pre- trained atom/bond embedding layers (Emb-BWC), to achieve parameter-efficien tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP Adaattee with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptabil- ity for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.*
Introduction
In the field of drug discovery and material science, molecular property prediction (MPP) stands pivotal task [5, 9, 63]. MPP involves the prediction of molecular properties like solubility and toxicity, based on their structural and physicochemical characteristics, which is integral to the development of new pharmaceuticals and materials. However, major challenge encountered in real-world MPP scenarios is data scarcity. Obtaining extensive molecular data with cell-characterized properties can be time-consuming and expensive. To address this, few-shot molecular property prediction (FSMPP) has emerged as crucial approach, enabling predictions with limited labeled molecules [1, 41, 4].
The methodology for general MPP typically adheres to an encoder-classifie framework [71, 23, 27, 56], as illustrated in Figure 2(a). In this streamlined framework, the encoder converts molecular
1 "onresponiing author
Codee is available https://gstthb..com/RPPPP AAw.AAAPAAAAAII/AAAPTTIiiia
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

structures into vectorized representations [12, 28, 50, 67, 2], and then the classifier uses these representations to predict molecular properties. In the context of few-shot scenarios, two significant discoveries have been instrumental in advancing this task. Firstly, pre-trained molecular encoders have demonstrated consistent effectiveness in FSMPP tasks [20, 14, 58]. This indicates the utility of leveraging pre-acquired knowledge in dealing with data-limited scenarios. Secondly, unlike typical few-shot tasks such as image classification 48], FSMPP tasks greatly benefits from molecular context information. This involves comprehending the seen many-to-mane relationships between molecules and properties [58, 45, 73], as molecules are multi-labeled by various properties. These two discoveries have collectively led to the development of the widely used FSMPP framework that utilizes a pre- pre-trained trained encoder followed by a context- classifier, as shown in Figure 2(b).
Despite the progress, there are observed lim- itations in the current approaches to FSMPP. Notably, while using pre-trained molecular encoder generally outperforms training from scratch, fine-tuning the pre-trained encoder of- ten leads to inferior results compared to keeping it frozen, which can be observed in Figure 1.
Train-Srom
Frocpe
Tos21
SIDER
ROC-4TI (K)
The observed ineffective fine-tuning can be at- tributed to two primary factors: (i) Imbalance be- tween the abundance of tunable parameters and the scarcity of labeled molecules: fine-tuning all parameters of pre-trained encoder with few labeled molecules leads to a disproportionate ra- tio of tunable parameters to available data. This imbalance often results in over-fitting and catas- trophic forgetting [7, 6]. (ii) Limited contextual perceptiveness in the encoder: while molecular
GIN-Mol
CMPNN
Compharmar
GIN-36el
CMENN
Grupharmur
Comparison of molecular encoders trained via different paradigms: main-froomscrrad and pretrain- www-tttheen-nnnnee The evaluation is conducted across two datasets and three encoder architectures [20, 47, 66). The results consistently demonstrate that while pretrain- ing outperforms training from scratch, the current methods do not yet effectively facilitate finetuning.
Figure 1:
cetrain-then-freee
context is leveraged to enhance the classifier [58, 73], the encoder typically lacks the explicit capabil- ity to perceive this context, relying instead on implicit gradient- gradient-based optimization. This leads to the encoder not directly engaging with the nuanced molecular context information that is critical in FSMPP tasks. In summary, while significant strides have been made, the challenges of imbalance between the number of parameters and labeled data, along with the need for contextual perceptiveness in the encoder, necessitate more sophisticated methodologies in this domain.
Based on the aforementioned analysis, we propose the parameter-efficient in-context tuning method, named Pin-Tuning, to address the two primary challenges in FSMPP. To overcome the parameter- data imbalance, we propose a parameter-efficient chemical knowledge adaptation approach for pre-trained molecular encoders. A lightweight adapters (MP Adapter) trained message passing layers efficiently. Additionally, we impose a Bayesian weight consolidation on the pre- pre-trained embedding layers to prevent aggressive parameter updates, thereby mitigating the risk of over-fitting and catastrophic forgetting. To address the second challenge, we further endow the MP- MP-Adapter with the capability perceive context. This innovation allows for in-context tuning of the pre-trained molecular encoders, enabling them to adapt more effectively to specific downstream tasks. Our approach is rigorously evaluated on public datasets. The experimental results demonstrate that our method achieves superior tuning performance with fewer trainable parameters, leading to enhanced performance in few-shot molecular property prediction.
designed to tune the pre-
The main contributions of our work are summarized as follows:
We analyze the deficiencies of existing FSMPP approaches regarding the adaptation of pre-trained molecular encoders. The key issues include an imbalance between the number of tunable parameters and labeled molecules, as well as lack of contextual perceptiveness in the encoders.
We propose Pin-Tuning to adapt the pre-trained molccular encoders for FSMPP tasks. This includes the Adapter for message passing layers and the Emb-BWC for embedding layers, facilitating parameter-efficient tuning of pre-trained molecular encoders.
We further endow the uddAtapeer with the capability to perceive context to allows for in-context tuning, which provides more meaningful adaptation guidance during the tuning process.
We conduct extensive experiments on benchmark datasets, which show that Pin-Tuning outper- forms oftt-of-the-as methods on FSMPP by effectively tuning pre-trained molecular encoders.

---

2 Related work
Few-shot molecular property prediction. Few-shot molecular property prediction aims to accurately predict the properties of new molecules with limited training data [49]. Early research applied general few-shot techniques to FSMPP. IterRefLSTM [1] is the pioneer work to leverage metric learning to solve FSMPP problem. Following this, meta-learning with graph neural networks, setting a foundational framework that subsequent studies have continued to build upon [39, 40, 4]. It is noteworthy that Meta- MGNN employs a pre- trettraneed molecular encoder [20] and achieves superior results through fine-tuning in the meta-learning process compared to training from scratch. In fact, pre-trained graph neural networks [64, 36, 17, 54, 37] have shown promise in enhancing various graph-based downstream tasks [52, 13], including molecular property prediction [60, 62, 38, 72]. Recent efforts have shifted towards leveraging unique nature in FSMPP, such as the many-to-many relationships between molecules and properties arising from the multi-labeled nature of molecules, often referred to the molecular context. PAR [58] initially employs graph structure learning [32, 55] to connect similar molecules through homogeneous context graph. MHNfs [45] introduces a large-scale external molecular library as context augment the limited known information. GS-Meta [73] further incorporates auxiliary task to depict the many-to- many-to-many relationships.
[41] and Meta-MGNNN [14] introduce
Parameter-efficient tuning. As pre-training techniques have advanced, tuning of pre-trained models has become increasingly crucial. Traditional full fine-tuning approaches updates all parameters, often leading to high computational costs and the risk of over-fitting, especially when available data for downstream tasks are limited [33, 15]. This challenge has led to the emergence of parameter-efficient tuning [26, 29, 34]. The philosophy of parameter-efficient tuning is to optimize a small subset of parameters, reducing the computational costs while retaining even improving performance on downstream tasks [19, 69]. Among the various strategies, the adapters [18, 42, 59] have gained prominence. Adapters are small modules inserted between the pre-trained layers. During the tuning process, only the parameters of these adapters are updated while the rest remains frozen, which not only improves tuning efficiency but also offers an elegant solution to the generalization [70, 30, 8]. By keeping the majority of the pre-trained parameters intact, adapters preserve the rich pre-trained knowledge. This attribute is particularly valuable in many real-world applications including FSMPP.
3 Preliminaries
3.1 Problem formulation
Let (T) be collection of tasks, where each task T involves the prediction of property p. The training set comprising multiple tasks {Tuain) ), is represented as Durain = {mm,, yi. E {Tuain) }. with m; indicating molecule and Ve.: its associated label for task t. Correspondingly, the test set Diea. formed by tasks (Tiea). ensures a separation of properties between training and testing phases, as the property sets (Purin) and (Pres) are disjoint ((Pmain) [Pess 0).
The goal of FSMPP to train a model using Dusin that can accurately infer new properties from a limited number of labeled molecules in Drest- Episodic training has emerged as a promising strategy in meta-lcarning [10, 16] to deal with few-shot problem. Instead of retaining all {Turinn tasks in memory, episodes { [E ) t== are iteratively sampled throughout the training process. For each episode E,, a particular task Te is selected from the training set, along with corresponding support set S, and query set Q. Typically, the prediction task involves classifying molecules into classes: positive (y = 1) or negative (y) = 0). Then a 2-way K-shot episode E, = (Se, Qe) is constructed. The support set S. {mn, JiK includes 2K examples, each class contributing K molecules. The query set containing M molecules denoted as Q, = {mm a 1M
3.2 Encoder-classifee framework for FSMPP
Encoder-classifier framework is widely adopted in FSMPP methods. As illustrated in Figure 2(a), given a molecule m whose property need be predicted, molecular encoder f(-) first learns the molecule's representation based on its structure, i.e., htm f(m) E Rd. The molecule m is generally represented as a graph m (V, A,X, X, E), where V denotes the nodes (atoms), A represents the adjacent matrix defined by edges (chemical bonds), and X, E denote the original feature of atoms
3

---

and bonds, then graph neural networks (GNNs) are employed as the molecular encoders [44, 51, 21]. Subsequently. the learned molecular representation fed into a classifier g(-) to obtain the prediction ý ghhm.. The model is trained by minimizing the discrepancy between y and the ground truth y-
Further, two key discoveries have been pivotal for FSMPP. The first is the proven effectiveness of pre-trained molecular encoders, while the second is the significant advantage gained from molecular context. Together, these discoveries have further reshaped the widely adopted FSMPP framework, which combines a pre-trained encoder followed by a context-aware classifier, as shown in Figure 2(b).
3.3 Pre-trained molecular encoders (PMEs)
Due the scarcity of labeled data in molecular tasks, molecular pre-training has emerged crucial area, which involves training encoders on extensive molecular datasets to extract informative representations. Pre-GNN [20] is classic pre-trained molecular encoder that has been widely used in addressing FSMPP tasks [14, 58, 73]. The backbone of Pre-GNN is a modified version of Graph Isomorphism Network (GIN) [65] tailored molecules, which we call GIN-Mol, consisting of multiple atom/bond embedding layers and message passing layers.
Atom/Bond embedding layers. The raw atom features and bond features both categorical vectors, denoted as (fv,1, fu.2..... br.)E.,)) and (J..1:Je.2,-. ........................... Je,[E,) for atom v and bond e, respectively. These categorical features embedded as:
h
EmbAtom. (Fv.a); h Σ EmbBond (Je,); (Je,t)
where EmbAtoma (1..,....) and EmbBondd (oaadd((()))) (I).e [) represent embedding operations that map integer indices to d-dimensional real vectors, i.e., h (0) ( E Rd, 1......., 1} represents the index of encoder layers, and L is the number of encoder layers. The atom embedding layer is present only in the first encoder layer, while an bond embedding layer exists in each layer.
Message passing layers. At the I-th encoder layer, atom representations are updated by aggregating the features of neighboring atoms and chemical bonds:
h 11 + (= E. h ) (E,
(1) ReLU (1P(()
(2)
where u E (w) [v} is the set of atoms connected to v, and (1) E Rd is the learned representation of atom v at the I-th layer. MLP(-) is implemented by 2-layer neural networks, in which the hidden dimension is d1. After MLP, batch normalization is applied right before the ReLU. The molccule-level representation hm E _Rd is obtained by averaging the atom representations the final layer.
4 The proposed Pin-Tuning method
This section delves into our motivation and proposed method. Our framework for FSMPP is depicted in Figure 2(c). The details of our principal design, Pin-Tuming for PMEs, present Figure 2(d).
As shown in Figure 1, pretraining then finetuning molecular encoders is common approach. However, fully fine-tuning yields results inferior simply freezing them. Thus, the following question arises:
How effectively adapt pre-trained molecular encoders to downstream tasks, especially in few-shot scenarios?
We analyze the reasons of observed ineffective fine-tuning issuc, and attribute it to two primary factors: (i) imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and (ii) limited contextual perceptiveness in the encoder.
4.1 Parameter-efffcient tuning for PMEs
To address the first cause of observed ineffective tuning, we reform the tuning method for PMEs. Instead of conducting full fine-tuning for all parameters, we propose tuning strategies specifically tailored to the message passing layers and embedding layers in PMEs, respectively.

---

Ponaa
Pin: Tuning for PMLEs
Firee Ttrting
Molecniar
Content
Molentarr
Encoder Layer
Laper Num
Progecty Cleasitier
SR-HSE
SR-p53
Awaar Propecty Classifer
Gragh
Propecty Classifier
FaxEFrwwa Ly-Prreet
SR-MNP
m
Noniioert
Malecabe
FeadFaward Boser Project
Benel Fribbesing Laysr
Message Poining
A
Contest Encoder
Pre-Trained Molecular Fncoder
Molecular Encoder
Unibodding Laper
Bends
Alorss
Vanilla MPP framework.
(b) Existing PSMPP framework.
FSMPP framework.
(d) Our Pin- Tuming method for PMEs.
Figure 2: (a) The vanilla encoder-classifer framework for MPP. (b) The framework widely adopted by existing FSMPP methods, which contains pre-trained molecular encoder and a context-aware property classifier. (c) Our proposed framework for FSMPP, in which introduce Pin-Tumine method to update the pre-trained molecular encoder followed by the property classifier. (d) The details of our proposed Pin-Tuning method for pre-trained molecular encoders. (b) and (c), we use the property names like SR-HSE to denote the molecular context in episodes.
4.1.1 MP-Adapter: message passing layer-oriented adapter
For message passing layers in PMEs, the number of parameters is disproportionately large compared to the training samples. To mitigate this imbalance, we design lightweight adapter targeted the message passing layers, called MP-Adapter. The pre-trained parameters in each message passing layer include parameters the MLP and the following batch normalization. We freeze all pre-trained parameters in message passing layers and add lightweight trainable adapter after MLP in each message passing layer. Formally, the adapter module for (-th layer can be represented as:
z FeedForward (h E RR
(3)
Ah FeedForward (6( (6(2()) R h
(4)
(1) LayerNorm(h!" Ah, ! Rd
(5)
where FeedForward denotes feed forward layer and LayerNorm(-) denotes layer normalization. To limit the number of parameters, we introduce bottleneck architecture. The adapters downscale the original features from d dimensions to a smaller dimension d2, apply nonlinearity Φ, then upscale back d dimensions. By setting d2 smaller than d, we can limit the number of parameters added. The adapter module has skip-connection internally. With the connectonn,,,, we adopt the near-zero initialization for parameters the adapter modules, so that the modules are initialized approximate identity functions. Therefore, the encoder with initialized adapters is equivalent to the pre-trained encoder. Furthermore, we add a layer normalization after skip-connection for training stability.
4.1.2 Emb-BWC: embedding layer-oriented Bayesian weight consolidation
Unlike message passing layers, embedding layers contain fewer parameters. Therefore, we directly fine-tune the parameters of the embedding layers, but impose a constraint limit the magnitude of parameter updates, preventing aggressive optimization and catastrophic forgetting.
The parameters in embedding layer consist of an embedding matrix used for lookups based on the indices of the original features. We stack the embedding matrices of all embedding layers to form IR Exd where E represents the total number of lookup entries. Further, 2; R2 denotes the i-th row's 's embedding vector, and F..j E R represents the j-th dimensional value of Φ;-
To avoid aggressive optimization of Φ derive Bayesian weight consolidation framework tailored for embedding layers, called Emb-BWC, by applying Bayesian learning theory [3] to fine-tuning.
Proposition 1: (Emb BWC ensures appropriate stability-plastiiity trade-off for pre-trained em- bedding layers.) Let & E Exd be the pre-trained embeddings before fine-tuning, and '' T Exd be the fune-tuned embeddings. Then, the embeddings can both retain the atom and bond properties
5

---

obtained from pre pre-training and be appropriately updated to adapt to downstream FSMPP tasks, by introducing the following Emb-BWC loss into objective during the fine-tuning process:
LEmb nwC
E ( -b;) HDD,,, -;;;;
(6)
where H(Dp, P;) E Bddd is the Hessian of the log likelihood Lp of pre-training dataset Dp at D;-
Details on the theoretical derivation of Eq. (6) are given in Appendix A. Since H(Dp, 2;) is intractable to compute due to the great dimensionality of Φ, we adopt the diagonal approximation of Hessian. By approximating H as a diagonal matrix, the j-th value on the diagonal of H can be considered the importance of the parameter P... The following three choices considered.
Identity matrix. When using the identity matrix to approximate the negation H, Eq. (6) is simplified BWC = - E Σi= (%', L.,))² assigning equal importance to each parameter. This loss function is also known 1.2 penalty with pre-trained model as the starting point (I.2-SP) [31].
to CIM
Diagonal of Fisher information matrix. The Fisher information matrix (FIM) F is the negation of the expectation of the Hessian over the data distribution, i.e., F ED, [H, and the FIM can be further simplified with a diagonal approximation. Then, the Eq. (6) is simplified to CFIM
Fmb ( -,,2, where F, E R. is the diagonal of F(Dp, ..) E Raxd and the j-th value in F; computed as ED, (0Cp /0 This is equivalent elastic weight consolidation (EWC) [24].
DE
Diagonal of embedding-wite wise Fisher information matrix. In different property prediction tasks, the impact of the same atoms and inter-atomic interactions may be significant negligible. Therefore, we propose this choice to assign importance to parameters based on different embeddings, rather than treating each parameter individually. By defining & Ej Pig. the total update of the embedding @ can be represented as A 1;; & $. (0,5) +..)). Then, the Eq. (6) is reformulated to Emb-EWC = E. F. 3 where F, E, ED (BCp (0Cp//0..))
CEFIM
Detailed derivation is given in Appendix A. Intuitively, these three approximations employ different methods to assign importance parameters. Fmb-aww assigns the same importance each pa- rameter, Emb-BwC assigns individual importance each parameter, and CEFIM Emb-BwC assigns the same
LFM
importance to parameters within the same embedding vector.
4.2 Enabling contextual perceptiveness in MP-Adapter
For different property prediction tasks, the decisive substructures vary. As shown in Figure 2, the ester group in the given molecule determines the property SR-HSE, while the carbon-carbon triple bond determines the property SR-MMP. If fine-tuning can be guided by molecular context, encoding context-specific molecular representations allows for dynamic representations of molecules tailored to specific tasks and enables the modeling of the context-speciftt significance of substructures.
Extracting molecular context information. In each episode, we consider the labels of the sup- port molecules on the target property and seen properties, as well as the labels of the query molecules on seen properties, as the context of this episode. We adopt the form of graph to describe the context. Figure 3 demonstrates the transformation from original context data to context graph. In the left table, the labels of molecules ml. m2 for property P: the pre- diction targets, and the other shaded values are the available context. The right side shows the context graph constructed based on the available context. Specifically, we construct context graph G. = (V,, A,, X,) for episode E,. It contains M molecule nodes {m] and P property nodes {p).
p
Contest Graph
0
1
m5
1 0
m
0
Figure 3: Convert the context information of a 2- shot episode into context graph.
Three types of edges indicate different relationships between molecules and properties.
Then we employ a GNN-based context encoder: C ContextEncoder(V), Ar, X.). where C e R PM++) denotes the learned context representation matrix for E,. V, and A, denote the node set

---

and the adjacent matrix of the context graph, respectively, and X, denotes the initial features of nodes. The features of molecule nodes are initialized with pre-trained molecular encoder. The property nodes are randomly initialized. When we make the prediction of molecule m's target property we take the learned representations of the this molecule Cm and of the target property Cp as the context vectors. Details about the context encoder are provided in Appendix F.2.
In-context tuning with molecular context information. After obtaining the context vectors, consider enabling the molecular encoder to use the context as a condition, achieving conditional molecular encoding. To achieve this, we further refine our adapter module. While neural conditional encoding has been explored in some domains, such as cross- cross-attention [43] and ControlNet [68] for conditional image generation, these methods often come with a significant increase the number of parameters. This contradicts our motivation of parameter-efficiee this work, we adopt simple yet effective method. We directly concatenate the context with the output of the message passing layer, and feed them into the downscaling feed-forward layer in the Adapter. Formally, the downscaling process defined in Eq. (3) is reformulated as:
tuning for few-shot tasks. In
(1) Fee Feed FeedForwardaem (h |m (p):
where || denotes concatenation. Such learned molecular representations are more easily predicted on specific properties, verified in Section 5.5 and Appendix G.
4.3 Optimization
Following MAML [10], a gradient descent strategy is adopted. Firstly, B episodes H are randomly sampled. For each episode, in the inner-loop optimization, the loss on the support set is computed Lela (So) and the parameters are updated by gradient descent:
Ctts (fo) = -- --E.. (y (og()( + (1- - y) log(1 - 0)).
(8)
0 + 0-amnee Oinner VoCt
where Oiinner is the learning rate. In the outer loop, the classification loss of query set is denoted as f Together with our Emb-BWC regularizer, the eeatanta loss [(for) ) computed and we do an outer-loop optimization with learning rate Oontter across the mini-batch:
1
B cols (So) + XL E
((Swr
=
B
(10)
k outer VoL(fo),
(11)
where À is the weight of Emb-BWC regularizer. The pseudo-code is provided in Appendix B. We also provide more discussion of tunable parameter size and total model size in Appendix C.
5
Experiments
5.1
Evaluation setups
Datasets. We five common few-shot molecular property prediction datasets from the Molecu- leNet [61]: Tox21, SIDER, MUV, ToxCast, and PCBA. Standard data splits for FSMPP are adopted. Dataset statistics and more details of datasets can be found in Appendix D.
Baselines. For a comprehensive comparison, we adopt two types of baselines: (1) methods with molecular encoders trained from scratch, including Siamese Network [25]. ProtoNet [46], MAML [10], TPN [35], EGNN [22], and IterRefLSTM [1]; and (2) methods which leverage pre- trained molecular encoders, including Pre-GNN [20], Meta-MGNN [14], PAR [58], and GS-Meta [73].
More details about these baselines are in Appendix E.
Metrics. Following prior works [1, 58), ROC-AUC scores are calculated on the query set for each meta-testing task, to evaluate the performance of FSMPP. We run experiments times with different random seeds and report the mean and standard deviations.

---

Table ROC-AUC scores (%) on benchmark datasets, compared with methods trained from scratch (first group) and methods that leverage pre- treitrained molecular encoder (second group). The best marked with boldface and the second best is with underline. Aimprove indicates the relative
improvements over the baseline models in percentage.
Model
Tox21
SIDER
MUV
ToxCast
PCBA
10-shol
S-shol
10-shot
5ishot
10-shot
Sishol
10-shot
5-shot
10-shot
S-shot
Siamese
80 30.40a35
30.10:
59.96ss
ProtoNet
74,98m33
72.78
64,5410am
64.09
65 65.88/4111
64.862230
68.87884)
66.26..49
64.93
62.299c.12
MAML
80.21ma24
69.1 10.17..3
70.43m020
60.92aa5
63.90223)
000000
30..59
67. 67.56..53
66,22
68.25e..5
TPN
76. 76:05mm30
95.
67. 07.84
66.5 66.52
65,
65, 130
69. 69.47m-71
66.040.7
67.61
63.66 An
EGNN
62.80.26
72.87
60.61.108
65.20k2.0
63.462250
74.02n.33
67.13
69.92
67.71em
STM
81. 10o m
69 69.63333
49.566s 170
Pre-GNN
73.96m00 82.04v.30
76. 76.7610.33
70.23 67.14k159)
15.31
74.43
75.27 76.79k..45
Meta MGNN
82. m2
75.43 26.12
66. 36.60
68.9 68.99(189)
64.07a1
76.27mm
75. 75.26mm43
72.58
12.51
PAR
84.93
78.08ma.6 13
77.70ko34
19796
68. 68.08/242)
79.4
76.89o.32
73.7
72.79kc00
GS-Meta
86.62/m41
86.42000
84, 36n,4
57/on
66.08:1115
64.500 20
82 22.65
79, 40.400.3)
77,47k.29
Pin: Tuning
91,56,259
20,95:235
93.41
92.02:3001
73.332200
70.71: 42)
84.94
83.71maa
81.26:144)
79.23
Almprove.
5.64%
5.23%
8.81%
4.82%
3,86%
1.35%
1.28%
2.27%
Table 2: Ablation analysis on the Ad-Aea,,er, in which we drop different components to form variants. We report ROC-AUC scores (%), and the best performance is highlighted in bold.
Model
Componcnt
Tox21
SIDER
MUV
ToxCast
PCBA
Adapter
Context
LayerNorm
10-shor
5-shot
1D-shot
5-chot
10-shot
5-shor
10-shot
5-shoe
1D-shor
S-shot
Pin-Tuning Adapter
91.56
90.95
93.41
92.02
73.33
70.71
84.94
83.71
81.26
79.23
79.72
78.49
72.94
66.06
62.88
80.06
78.70
73,85
72.02
wio Context
81.42
79.34
74:68
72,86
68.70
66.12
81.49
79.85
74.69
72,46
wio LayerNorm
/
86.71
84.93
91.50
90.76
70.26
67.42
83.52
82.55
80.07
78.23
5.2 Performance comparison
We compare Pin-Tuning with the baselines and the results summarized in Table 1, Table 7, and Table 8. Our method significantly outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the effectiveness and superiority of our approach.
Across all datasets, our method provides greater improvement in the 10-shot scenario than in the 5-shot scenario. This is attributed to the molecular context constructed based on support molecules. When there are more molecules in the support set, the uncertainty in the context is reduced, providing more effective adaptation guidance for our parameter- parameter-efficient tuning.
Among benchmark datasets, our method shows significant improvement on the SIDER dataset, increasing by 10.73% in the 10-shot scenario and by 8.81 % in the 5-shot scenario. We consider this related to the relatively balanced ratio of positive to negative samples, as well as the absence of missing labels in the SIDER dataset (Table 5). A balanced and low-uncertainty distribution can better benefit addressing the FSMPP task from our method.
We also observe that the standard deviations of our method's 's results under 10 seeds are slightly higher than that of baseline models. However, our worst-case results are still better than the best baseline model. For example, in 10-shot experiments on the Tox21 dataset, the performance of our method is 91.56 + 2.57. However, our 10 runs yield specific results with the worst-case ROC-AUC reaching 88.02, which is also better than the best baseline model GS-Meta's 's result of 86.67 + 0.41. Therefore, a high standard deviation does not mean our method inferior to baseline models.
5.3 Ablation study
For MP-Adapter, the main components consist of: (i) bottleneck adapter module (Adapter), (ii) introducing molecular context to adatpers (Context), and (iii) layer normalization (LayerNorm). The results of ablation experiments molecular context are the most critical, having the most significant impact on performance. Removing them leads noticeable decline, which underscores the importance of parameter-efficient tuning and context perceptiveness in FSMPP tasks. Layer normalization is used to normalize the resulting representations, which is also important for improving the optimization effect and stability.
summarized in Table 2. The bottleneck adapter and the modeling of
8

---

For Emb-BWC, we verify the effectiveness of fine- tuning the embedding layers and regularizing them with different approximations of Emm BWC (Table 3). Since the embedding layers have rela- tively few parameters, direct fine-tuning can also enhance performance. Applying our proposed regularizers to fine-tuning can further improve effects. Among the three regularizers, the Fmmh BWC. is the most effective. This indicates that keeping pre-trained parameters to extent better utilize pre- trained knowledge, but the parameters worth keeping in fine-tuning tine-tuning and the important parameters in pre-truie pre-training revealed by Fisher information matrix not completely consistent.
Table Ablation analysis on the BWC.
Fine-tune
Regularizer
Tox21
SIDER
MUV
89.70
90.12
70.76
80.24
90.17
92.06
72.37
80.74
LEmb-EWC
CM
93.41
73.22
81.26
CPM Lumb-sww
72.17
ZEPM Lib-bww
91.32
72.78
5.4 Sensitivity analysis
ROC-AUC (%)
180
ROC-AUC (%6)
Plo-Tuning =1:
Plo-Tuning (7)
92
Pla-l uning 100)
uning 150)
91
Tuning (Eo--Mara)
30-
10"
0.5
1.0
2.0
Number Traimable Parameters (clef)
Tox21
SIDER
MUV
PCBA
Figure 5: ROC-AUC (%) and number of trainable parameters of Pin- Pin-Tuning with var- ied value of d2 and full Fine-Tuning method (e.g., GS-Meta) the Tox21 dataset.
Figure 4: Effect of different hyper-paraman parameters. The y-axis represents ROC-AUC scores (%) and the x-axis the different hyper-parameters.
Effect of weight of Emb-BWC regularizer X. Emb-BWC is applied on the embedding layers to limit the magnitude of parameter updates during fine-tuning. We vary the weight of this regularization from (0.01, 0.1, 1, 10}. The first subfigure in Figure 4 shows that the performance is best when À = 0.1 or When À is too small, the parameters undergo too large updates on few-shot downstream datasets, leading to over-fitting and ineffectively utilizing the pre-trained knowledge. Too large À causes the parameters of the embedding layers be nearly frozen, which prevents effective adaptation.
Effect of hidden dimension of MP-Adapter d2. The results corresponding to different values of d2 from (25, 50, 75, 100, 150} are presented the second subfigure of Figure On the Tox21 dataset, we further analyze the impact of this hyper-parameter on the number of trainable parameters, As shown in Figure 5, the number of parameters that our method needs to train is significantly less than that required by the full fine-tuning method, such as GS-Meta, while our method also performs better in terms of ROC-AUC performance due solving wver-fittigg and context perceptiveness issues. When = 50, Pin- TinTTnnng performs best on Tox21, and the number of parameters that need to train is only 14.2% that required by traditional fine-tuning methods.
5.5
Case study
Dataset: Tox21, Property: SR-MMP
Dataset: Tox21, Property: SR-p53
Dataset: Tou21. Property: SR-MMP
Dataset: Tox21, Property: SR-553
Nagative
Negativ
Negatie
Figure 6: Molecular representations encoded by GS-Meta [73].
Figure 7: Molecular representations encoded by Pin-Tuning.
We visualized the molecular representations learned by the GS-Meta and our Pin-Tuning's encoders in the 10-shot setting, respectively. As shown in Figure and 7, Pin-Tuning can effectively adapt different downstream tasks based on context information, generating property-specine molecular representations. Across different tasks, our method is more effective in encoding representations that

---

facilitate the prediction of the current property. reducing the difficulty of property prediction from the encoding representation aspect. More case studies are provided in Appendix G.
Conclusion
In this work, we propose tuning method, Pin-Tuning, address the ineffective fine-tuning of pre-trained molecular encoders in FSMPP tasks. Through the innovative parameter-effiient tuning and in-context tuning for pre-trained molecular encoders, approach not only mitigates the issues of parameter-data imbalance but also enhances contextual perceptiveness. The promising results on public datasets underscore the potential of Pin- Sun-unnng advance this field, offering valuable insights for future research in drug discovery and material science.
Acknowledgments
This work is jointly supported by National Science and Technology Major Project (20232D0120901) National Natural Science Foundation of China (62372454, 62236010) and the Excellent Youth Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.
References
[1] Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS Central Science, 3(4):283-292) 2017.
[2] Dingshuo Chen, Yanqiao Zhu, Jieyu Zhang, Yuanqi Du, Zhixun Li, Q. Liu, Shu Wu, and Liang Wang. Uncovering neural scaling laws in molecular representation learning, In NeurlPS, 2023.
[3] Haolin Chen and Philip N. Garner. Bayesian parameter-efficient fine-tuning for overcoming catastrophic forgetting. arXiv, abs/2402. 12220, 2024.
[4] Wenlin Chen, Austin Tripp. and José Miguel Hermnndez-Lobato. Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. ICLR, 2023.
[5] Jianyuan Deng, Zhibo Yang, Hehe Wang. Iwao Ojima, Dimitris Samaras, and Fusheng Wang. A systematic study of key elements underlying molecular property prediction. Nature Commu- nications, 14, 2023.
[6] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng. Jianfci Chen, Yang Liu, Jie Tang, Juanzi Li. and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv, Abs/2203.06904 2022.
[7] Ning Ding, Yujia Qin, Guang Yang. Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Y. Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5:220-235. 2023.
[8] Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. 2023.
[9] Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. excometry-enhaneee molecular representation learning for property prediction. Nature Machine Intelligence, 4:127 134, 2021.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICML, 2017.
[11] Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017.
10

---

[12] Jonathan Godwin, Michael Schaarschmidt. Alexander L. Gaunt, Alvaro Sanchez- Gonzalez, Yulia Rubanova, Petar Velickovic, James Kirkpatrick, and Peter W. Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In ICIR, 2022.
[13] Renxiang Guan, Zihao Li, Wenxuan Tu, Jun Wang, Yue Liu, Xianju Li, Chang Tang, and Ruyi Feng. Contrastive multiview subspace clustering of hyperspectral images based on graph convolutional networks. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[14] Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V. Chawla. Few-shot graph learning for molecular property prediction. In WWW, 2021.
[15] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kirg-Kirkpatrick, and Graham Neubig. To- wards unified view of parameter-effiiien transfer learning. In ICLR, 2022.
[16] Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 14:5149-55 2020.
[17] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. KDD, 2022.
[18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. In ICML, 2019.
[19] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[20] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020.
[21] Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. In NeurIPS, 2022.
[22] Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D. Yoo. Edge-labeling graph neural network for few-shot learning. In CVPR, 2019.
[23] Suyeon Kim, Dongha Lee, Seong Kang, Seonghyeon Lee, and Hwanjo Yu. Learning topology-specific experts for molecular property prediction. In AAAI, pages 8291-8299, 2023.
[24] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catas- Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. trophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 114, 117.
[25] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, 2015.
[26] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP (1), pages 3045-3059. Association for Computational Linguistics, 2021.
[27] Han Li, Dan Zhao, and Jianyang Zeng. KPGT: knowledge-guidee pre-training of graph transformer for molecular property prediction. KDD, pages 857-867, 2022.
[28] Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgel: Geometric graph contrastive learning for molecular property prediction. In AAAI, pages 4541-4549, 2022.
[29] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[30] Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning lasiun-language models with dual knowledge graph. In NeurIPS, 2023.
11

---

[31] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018.
[32] Zhixun Li, Liang Wang. Xin Sun, Yifan Luo, Yanqiao Zhu, Dingshuo Chen, Yingtao Luo, Xiangxin Zhou, Qiang Liu, Shu Wu, Liang Wang, and Jeffrey Xu Yu. GSLB: the graph structure learning benchmark. In NeurIPS, 2023.
[33] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning, arXiv, abs/2303.15647, 2023.
[34] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv, abs/2103.10385, 2021.
[35] Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR, 2019.
[36] Yue Liu, Jun Xia, Sihang Zhou, Siwei Wang, Xifeng Guo, Xihong Yang. Ke Liang, Wenxuan Tu, Stan Z. Li, and Xinwang Liu. A survey of deep graph clustering: Taxonomy, challenge, and application. arXiv, abs/2211.12255
2022.
[37] Yue Liu, Ke Liang, Jun Xia, Sihang Zhou, Xihong Yang, Xinwang Liu, and Stan Z. Li. Dink-net: Neural clustering on large graphs. ICML, 2023.
[38] Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder in masked graph modeling for molecules. In NeurIPS, 2023.
[39] Qiujie Lv, Guanxing Chen, Ziduo Yang, Weihe Zhong, and Calvin Yu-Chian Chen. Meta learning with graph attention networks for low-data drug discovery. IEEE transactions on neural networks and learning systems, 2023.
[40] Ziqiao Meng, Yaoman Li, Peilin Zhao, Yang Yu, and Irwin King. Meta-learning with motif- based task augmentation for few-shot molecular property prediction. In SDM, 2023.
[41] Cuong Q Nguyen, Constantine Kreatsoulas, and Kim Branson. Meta-learning gnn initializations for low-resource molecular property prediction. ICML Workshop on Graph Representation Learning and Beyond, 2020.
[42] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In EACL, 2021.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10674-1068) IEEE, 2022.
[44] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.
[45] Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve few- shot drug discovery. In ICLR, 2023.
[46] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NeurIPS. 2017.
[47] Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJCAI, 2020.
[48] Yisheng Song, Ting Wang, Puyu Cai, Subrota K. Mondal, and Jyoti Prakash Sahoo. A com- prehensive survey of fhw-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 2023.
12

---

[49] Megan Stanley, John Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin H. S. Segler, Nadine Schneider, and Mare Brockschmidt. Fs-mol: A few-shot learning dataset of molecules. In NeurlPS Datasets and Benchmarks, 2021.
[50] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Lió. 3d infomax improves gnns for molecular property prediction. In ICML, pages 20479-20502, 2022.
[51] Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In KDD, 2021.
[52] Xin Sun, Liang Wang, Qiang Liu, Shu Wu, Zilei Wang, and Liang Wang. DIVE: subgraph disagreement for graph -fittr-distiiuution generalization. In KDD, 2024.
[53] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurlPS, page 3637-3645, 2016.
[54] Liang Wang, Xiang Tao, Qiang Liu, Shu Wu, and Liang Wang. Rethinking graph masked autoencoders through alignment and uniformity. In AAAI, 2024.
[55] Liang Wang, Shu Wu, Q. Liu, Yanqiao Zhu, Xiang Tao, and Mengdi Zhang. Bi-level graph structure learning for next poi recommendation. IEEE Transactions on Knowledge and Data Engineering. 36:5695-5708. 2024.
[56] Xu Wang, Huan Zhao, Wei-Wei Tu, and Quanming Yao. Automated 3d pre-training for molecular property prediction. In KDD, pages 2419-2430, 2023.
[57] Yaqing Wang. Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few examples: survey on few-shot learning. ACM Computing Surveys, 2020.
[58] Yaqing Wang, Abulikemu Abuduweili, Quanming Yao, and Dejing Dou. Property- aware relation networks for fhw-shot molecular property prediction. In NeurlPS, 2021.
[59] Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Exxtue-of-addapaation for parameter-efficien model tuning. In EMNLP, 2022.
[60] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 2022.
[61] Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning, arXiv, 0abb//703300066/ 2017.
[62] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-bert: Rethinking pre- pre-tiannin graph neural networks for molecules. In ICLR, 2023.
[63] Yi Xiao, Xiangxin Zhou, Qiang Liu, and Liang Wang. Bridging text and molecule: A survey on multimodal frameworks for molecule. arXiv,, abs/2403.13830, 2024.
[64] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. TPAMI, 2022.
[65] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[66] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tiie-an Liu. Do transformers really perform badly for graph representation? In NeurlPS, 2021.
[67] Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzaler Peter Battaglia, Razvan Pascanu, and Jonathan Godwin, Pre-training via denoising for molecular property prediction. In ICLR, 2023.
13

---

[68] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3813-3824. IEEE, 2023.
[69] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, 2023.
[70] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. Llama- addater: Efficient fine-tuning of language models with zero-init attention. In ICLR, 2024.
[71] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. In NeurIPS, pages 15870-15882, 2021.
[72] Yanqiao Zhu, Dingshuo Chen, Yuanqi Du, Yingze Wang, Q. Liu, and Shu Wu. Molecular contrastive pretraining with collaborative featurizations. Journal of Chemical Information and Modeling, 2024.
[73] Xiang Zhuang, Qiang Zhang, Bin Wu, Keyan Ding, Yin Fang, and Huajun Chen. Graph sampling-bssed meta-learning for molecular property prediction. In IJCAI, 2023.
14

---

Appendix
The organization of the appendix is as follows:
Appendix A: Derivation of Emb-BWC regularization;
Appendix B: Pseudo-code of training process;
Appendix C: Discussion of tunable parameter size and total model size;
Appendix D: Details of datasets;
Appendix E: Details of baselines;
Appendix F: Implementation details;
Appendix G: More experimental results and discussions;
Appendix H: Limitations and future directions.
A Derivation of Emb-BWC regularization
A.1 Derivation of Lmmb-BWC
Let Φ E Exd be the pre-trained embeddings before fine-tuning, and क' E TR Exd be the fine-tuned embeddings. Further, Φ Rd denotes the i-th row's embedding vector in T and P... R represents the j-th dimensional value of Φ,
The optimization of embedding layers can be interpreted as performing a maximum a posterior (MAP) estimation of the parameters &' given the pre-training data and training data of downstream FSMPP task, which is formulated in Bayesian framework.
In the FSMPP setting, the molecular encoder has been pre-trained on the pre-training task P using data Dp, and is then fine-tuned on a downstream FSMPP task F using data Dy. The overall objective to find the optimal parameters on task while preserving the prior knowledge obtained pre- training on task F. Based on prior p(T") of the embedding parameters, the posterior after observing the FSMPP task can be computed with Bayes' rule:
DD, p(Dr (p)p(('(Dp))
D(D)(Dp)
(12)
p(D> 1 (p(('(D))
p(Dr)
where D is assumed to be independent of Dp. Taking a logarithm of the posterior, the MAP objective is therefore:
T arg max log p( ( (Dp, Dy)
(13)
arg max log p (D2 + logg((0)) (()))Dp)
The first term I(D (') is the log likelihood of the data D, given the parameters s', which
can be expressed as the training loss function on task F = log pl (D) D'), denoted as (*)
The second term p(D) is the posterior of the parameters given the pre-training dataset Dp.
Since d' [ TT...... ...., 2/ and & conditionally independent of 4/ for i,j (l...... E}
and i * given condition Dp, we have p( (Dp) = II (( Thus, log D log p pD)))))
For adapting pre-trained molecular embedding layers to downstream FMSPP tasks, this posterior must encompass the prior knowledge of the pre-trained embedding layers reflect which parameters are important for pre-training task P. Despite the true posterior being intractable, log p( (Dp) can be defined function f()( and approximated around the optimum point f(2:), where f
is the pre-trained values and V f = O. Performing accond-order Taylor expansion on f(
around Φ gives:
logp(d Dp) R (T.) + (6 - +.)² V2 f (T.) (8) -
(14)
=f,
( ②)) H H(pp, (0) 2.),
15

---

where H(pp, 2.) E R is the Hessian matrix of f({)) at Φs. The second term suggests that the posterior of the parameters the pre- pre-training data can be approximated by Gaussian distribution with mean & and covariance H(pp, +;) ) Following Eq. (13), the training objective becomes:
4' = arg max log (D (') + log p @ (D)
arg max log (D= + logp(1; (D)p)
V
(15)
arg minCr((')
Σ f(0))
Σ ( 4.)" HDp,, ( +.)
F
arg f (
Σ
( ".)" H(p,, (t)p,,)))()) ( o;).
We define the second term
regularization objective:
Fmb.-RWC
1 0 43) "H(p, (:)(" 4.).
Σ
(16)
A.2 Derivation of CFI
Ewb-BBW
Since the Fisher information matrix (FIM) is the negation of the expectation of the Hessian over the data distribution, i.c., EEp [H]. the objective can be reformulated
FIM Emb-BWC
Σ / T.) FF(p, F(D,,,,)(0%) +;);
(17)
2
where F(Dp, +.)) E Raddd is the corresponding Fisher information matrix H(Dp, 4,). Further, the Fisher information matrix can be further simplified with a diagonal approximation. Then, the objective is simplified to:
FIM BWC R
F $,)²
(18)
where F, Rd is the diagonal of F(Dp, T,). According to the definition of the Fisher information matrix, the j-th value in F, is computed as Ep, (/0lp//0..... 2. In this work, this approximated form defined as LFM Emb-BWC-
A.3 Derivation of CEFM Emb-BWC
We assume that the parameters within embedding should share the same importance. To this end, we define क; = Σ, Pi.j. then the total update of the embedding + can be represented as AΦ. . Φ. E Piz). Then, the objective in Eq. (16) is reformulated to:
EFIM 1 EWC
M H c -5,²
(19)
2
where H, C
8 0Cz
Next, we continue to derive H, Given that d. = Σ Dig
04?
00;
20,
first use the chain rule to find 8Ce can be computed as:
According to chain rule, the derivative Lp with respect to 00;
84;
OCp
OCp 04..
Σ
(20)
04:
aD., 04;
Since क; T.11 + 22 +.... + Lid. each of 804, equation simplifies to:
for j = 3,2.....d equals 1. Therefore, the
ab,
dCp
OCp
(21)
0
0D.
16

---

When taking the derivative of this with respect to Pi.y again, using the chain rule, we have:
8 LL
a
dCp D
dCp 2
Σ
(22)
a P 0
j=
Given that (2r,)) are all parameters in embedding lookup tables, they are independent of each
other. Thus, when j = k, 30,,t 84...
8Lp
8 00
otherwise equals We finally get
ae.
H, Σ, Jhe = E, H(Dp, U.)is-
We still approximate the Hessian with the FIM in Section A.2, and combining this with the definition of FIM, we arrive at the final objective:
EFIM C
Emb
RC
F1(0; 22)²
(23)
1=1
where F; = BD, (8,pp//00..
B Pseudo-code of training process
To help better understand the training process, we provide the brief pseudo-code of in Algorithm 1.
Algorithm 1: Training process of Pin--unnn Fin-Tuning.
Input :raiiing set Draii
Output: Tuned few-shot molecular property prediction model with parameter while not converge do
Sample episode from training set Dirain to form a mini-batch {E for t 1 to B do
Calculate classification loss support set (fo) by Eq. (8) on E,;
0' <- 0 Oinnner V (fa);
Do inner-loop update by Eq. (9) on Eg:
Calculate classification loss on query set (ferr by Eq. (8) on E;;
Calculate update constraint Lmmb-WWC by Eq. (6);
Do outer-loop optimization by Eq. (10) and Eq. (11): t- 0 Coouter V v(f())))
Return optimized model parameter 0.
C Discussion of tunable parameter size and total model size
C.1 Tunable parameter size of molecular encoder
We compare the tunable parameter size full fine-tuning and our Pin-Tuning, Section 3.3 describes the parameters of the PME, which include those for the embedding layers and the message passing layers. We assume there are Ex original node features and EE edge features. Considering there one node embedding layer and edge embedding layers, the total number of parameters for the embedding part is E,, d + L d. The parameters in the message passing layer consist of the 2-layer MLP including biases shown in Eq. (2) and its subsequent batch normalization, with each layer having L(2dd, + d + d1 + 2d) parameters. In summary, the total number of parameters to update in full fine-tuning
NFine-Tuning 1 L(IE, (d 2dd, + 3d d1).
(24)
In our Pin-Tuming method, the parameters of the embedding layers are still updated. However, each message passing layer, the original parameters are completely frozen, and the parts that require updating are the two feed-forward layers and the layer normalization in the bottleneck adapter
17

---

Table 4: Comparison of total model size. indicates that the parameters are frozen.
GS-Meta
Ours
Size of Molecular Encnder
1.86M
1.86M
Size of Adapter
0.21M
Size of Context Encoder
0.62M
0.62M
Size of Classifier
0.18M
0.27M
Size of Total Model
2.66M
2.96M
Size of Tunable Part the Model
2.66M
1.10M
Table
Dataset statisties.
Dataset
Tox21
SIDER
MUV
ToxCast
PCBA
#Compound
7831
1427
93127
8575
437929
#Property
12
17
617
128
#Train Property
21
12
451
118
#Test Property
6
5
158
10
% Positive Label
6.24
56.76
0.31
12.60
0.84
% Negative Label
76.71
43.24
15.76
72.43
59.84
Unknowwn Label
17.05
84.21
14.97
39.32
module, amounting to L(2dd2 + d + d2 + 2d) parameters for this part. Therefore, the total number of parameters that need be updated in our Pin-Tuning is
Pin-innnng |. |d L([E.|d 2dd2 3d + d2).
(25)
The difference in the number of parameters updated between the two tuning methods is AN = (d1 d2) + 1).
C.2 Total model size
We provide comparison of total model size between our Pin- Fin-Tuning and the -tate-of-the-wee baseline method, GS-Meta The total model size consists of both frozen parameters and trainable parameters. The results are presented in Table 4. The total size of our model is comparable to GS-Meta, but the number of parameters that need to be trained is far less than GS-Meta.
D Details of datasets
We carry experiments in MoleculeNet benchmark [61] five widely used few-shot molecular property prediction datasets:
Tox21:
This dataset covers qualitative toxicity measurements and was utilized in the 2014 Tox21
Data Challenge.
SIDER: The Side Effect Resource (SIDER) functions as a repository for marketed drugs and adverse drug reactions (ADR), categorized into 27 system organ classes.
MUV: The Maximum Unbiased Validation (MUV) determined through the application of
refined nearest neighbor analysis, specifically designed for validating virtual screening techniques.
ToxCast: This dataset comprises a compilation compounds with associated toxicity labels derived from high-throughput screening.
PCBA: PubChem BioAssay (PCBA) represents a database containing the biological activities of small molecules generated through high-throughput screening.
Dataset statistics are summarized in Table 5 and Table 6.
E Details of baselines
We compare our Pin-Tunimg with two types of baseline models for few-shot molecular property prediction tasks, categorized according to the training strategy molecular encoders: mained-fromt scratch methods and pre-trained methods.
18

---

Table 6: Statistics of sub sub-datasets of ToxCast.
Assay Provider
#Compound
&Property
#Train Property
#Test Property
%e active
LLabel inactive
S:isssiigg Label
APR
1039
43
33
10
10.30
61.61
28.00
BSK
1445
115
84
31
17:71
82.29
0.00
CEETOX
CLD
305
19
14
5
30.72
68.30
0.98
OT
1782
15
11
9.78
87.78
2.41
Tanguay
1039
18
13
5
8.05
90.84
1.11
Tro-ssahfrommsssraacch methods:
Siamese [25]: Siamese used to rank similarity between input molecule pairs with a dual network.
ProtoNet [46]: ProtoNet learns metric space for few-shot classification, enabling classification by calculating the distances between each query molecule and the prototype representations of each class.
MAML [10]: MAML adapts the meta-learned parameters to achieve good generalization perfor- mance on new tasks with a small amount of training data and gradient steps.
TPN [35]: TPN classifies the entire test set once by learning propagate labels from labeled instances to unlabeled test instances using a graph construction module that exploits the manifold structure in the data.
EGNN [22]: EGNN predicts edge labels on graph constructed from input samples to explicitly capture intra-cluster similarity and inter-cluster dissimilarity.
IterRefl STM [1]: BerRefl.STM. adapts Matching Networks [53] to handle molecular property prediction tasks.
Pre-trained methods:
Pre-GNN [20]: Pre-GNN is a classic pre-trained molecular model, taking the GIN as backbone and pre-training with different self-supervised tasks.
Meta-MGNN [14]: Meta-MGNN leverages Pre-GNN for learning molecular representations and incorporates meta-leaming and self-supervised learning techniques.
PAR [58]: PAR uses class prototypes update input representations and designs label propagation for similar inputs in the relational graph, thus enabling the transformation of generic molecular embeddings into property-awan spaces.
GS-Meta [73]: GS-Meta constructs a Holecule-Property relation graph (MPG) and redefines episodes in meta-learning subgraphs of the MPG.
Following prior work [58], for the methods we reproduced, we use GIN as the graph- based molecular encoder extract molecular representations. Specifically, we use the GIN provided by Pre-GNN [20] which consists of GIN layers with 300-dimensional hidden units. Pre-GNN, Meta-MGNN, PAR,
and GS MesMeta further the pre-trained GIN which is also provided by Pre-GNN.
F
Implementation details
F1 Hardware and software
Our experiments conducted Linux servers equipped with AMD CPU EPYC 7742 (256) 2.250GHz, 256GB RAM and NVIDIA 3090 GPUs. Our model is implemented in PyTorch version 1.12.1, PyTorch Geometric version 2.3.1 (https://yy..rrg) with CUDA version 11.3, RDKit version 2023.3.3 and Python 389.18. Our code is available at: Pin-Tuning.
com/CRIPAC. DIG/
F.2 Model configuration
For featurization of molecules, we use atomic number and chirality tag as original atom features, well bond type and bond direction bond features, which is in line with most molecular
19

---

Table 10-shot performance on each sub-dataset of ToxCast.
Model
APR
ATG
BSK
CEETOX
CLD
NVS
OT
TOX21
Tanguay
ProtoNet
73.58
59.26
70.15
66.12
78,12
65.85
64,90
68.26
73.61
MAMI.
72.66
62.09
66.42
64.08
74.57
66.56
64.07
68.04
77.12
TPN
74.53
60.74
65.19
66.63
75.22
63.20
64.63
73.30
81.75
EGNN
80.33
66.17
73.43
66.51
78.85
71.05
68.21
76.40
85.23
Pre-GNN
80.61
67.59
76.65
66.52
78.88
75.09
70.52
77.92
83.05
Meta-MGNN
81.47
69.20
78.97
66.57
78.30
79.60
69.55
78.77
83.98
PAR 86.09
72.72
82.45
72.12
83.43
74.94
71.96
82.81
88.20
GS-Meta
90.15
82.54
88.21
74.19
86.34
76.29
74.47
90.63
91.47
Pin- Tuninunng
92.78
83.58
89.49
75.96
87.70
76.33
75,56
90.80
92.25
AImprove.
2.92%
1.26%
1.45%
2.39%
1.58%
0.05%
1.46%
0.19%
0.85%
Table 5-shot performance on each sub-dataset of ToxCast.
Model
APR
ATG
BSK
CEETOX
CLD
NVS
OT
TOX21
Tanguay
ProtoNet
70.38
58.11
63.96
63.41
76.70
62.27
64.52
65.99
70.98
MAML
68.88
60.01
67.05
62.42
73.32
69.18
64.56
66.73
75.88
TPN
70.76
57.92
63.41
64.73
70.44
61.36
61.99
66.49
77.27
EGNN
74.06
60.56
64.60
63.20
71.44
62.62
66.70
65.33
75.69
Pre-GNN
80.38
66.96
75.64
64.88
78.03
74.08
70.42
75.74
82.73
Meta-MGNN
81.22
69.90
79.67
65.78
77.53
73.99
69.20
76.25
83.76
PAR
83.76
70.24
80.82
69.51
81.32
70.60
71.31
79.71
84,71
GS-Meta
89,36
81.92
86.12
74.48
83.10
74.72
73.26
89.71
91.15
Pin-Tuning
89.94
82.37
87.61
75.20
85.07
75.49
74.70
90.89
92.14
Almprove. 0.65%
0.55%
1.73%
0.97%
2.37%
1.03%
1.97%
1.32%
1.09%
pre-training methods. Following previous works, we set d= 300. For MLPs in Eq. (2), we use the ReLU activation with di 600. Pre-trained GIN model provided by Pre-GNN [20] adopted the PTME in our framework. We tune the weight of update constraint (i.e., X) in (0.01, 0.1, 1, 10), tune the learning rate of inner loop (i.e., Qinner) in { (le-3, 5e-3, 1e-1,5e-2.1e-1, 5e-1, 5), and tune the learning rate of outer loop (i.e., Olouter) in [le-5, le-4, 10-3,,1e2,11--11 Based on the results of hyper- 1e and d2 50. The ContextEncoderi described in Section 4.2 is implemented using a 2- 2-layer message passing neural network [11]. In each MPNN layer, we employ linear layer to aggregate messages from the neighborhoods of nodes and utilize distinct edge features to differentiate between various edge types in the context graphs. For baselines, we follow their recommended settings.
parameter tuning, we adopt O'inner 0.5, Qoutter
G More experimental results and discussions
More discussion Figure The results show that molecular encoders with more molecule-specine inductive biases, such as CMPNN [47] and Graphormer [66], performed slightly worse than GIN- Mol [20] on this few-shot task. This is because more complex encoders require more parameters to provide inductive biases, which are difficult to train effectively under few-shot setting.
More main results. The detailed comparison between Pin-Tuning and baseline models sub- datasets of ToxCast are summarized in Table 7 and Table 8. Our method outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the superiority of our method compared to existing methods.
More discussion of ablation study. Different datasets show varying sensitivity to the removal of components. On small-scale datasets like Tox21 and SIDER, removing components leads to significant performance drop. On large-scale datasets like ToxCast and PCBA, the impact of removing components is less pronounced. This is because more episodes can be constructed on large-scale datasets, which aids in adaptation. This observation indicates that Pin-Tuning can bring considerable benefits situations where data is extremely scarce.
More studies. We provide more case studies in Figure and 9 as supplement to Section 5.5.
20

---

Dataset: Tox21, Property: SR-HSE
Dataset: SIDER, Property: CD
Dataset: SIDER, Property: ELD
Negative Positive
Negative Positive
Negative
Positive
Dataset: SIDER, Property: IPPC
Dutaset: SIDER, Property: NSD
Datuset: SIDER, Property: PPPC
Negative
Positive
Negative Positive
Negative Positive
SIDER, Property: RUD
Negative
Positive
Figure More molecular representations encoded by GS-Meta [73].
Dataset: Tox21, Property: SR-HSE
Dataset: SIDER, Property: CD
Dataset: SIDER, Property: ELD
Negative
Positive
Negatliee Positive
Negative
Positive
Dataset: SIDER, Property: IPPC
Dataset: SIDER, Property: NSD
Dataset: SIDER, Property: PPPC
Negative
Negative
Negative
Positive
Positive
Positive
Dataset: SIDER, Property: RUD
Negative
Positive
Figure 9: More molecular representations encoded by Pin- Lan--Tning
21

---

H Limitations and future directions
As we discusses in Section 5.2, although our method significantly outperforms the state-of-the-art baseline method, our method exhibits higher standard deviations in the experimental results under multiple runs with different seeds.
We further speculate that these high standard deviations might be due to the uncertainty in the context information within episodes. The explicitly introduced molecular context, on one hand, provides effective guidance for tuning pre-trained molecular encoders, but on the other hand, this information also carries high degree of uncertainty. We aim model the target property through the molecule-propee property relationships within episodes, but cach episode is obtained by sampling very few samples from the large space corresponding the target property. The uncertainty between different episodes is relatively high. How to quantify and calibrate this uncertainty is another question worth exploring, which we will investigate in our future work.
22

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present our claims and main contributions in the abstract and introduction. We propose a parameter-efficient in-context tuning method, named Pin-tuning, to effectively adapt pre-trained molecular encoders few-shot molecular property prediction tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals motivation long is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Appendix H.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tee sreech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
We provide the derivation of our proposed Emb-BWC regularization, with related assumptions in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The proposed method and experimental setups are clearly described. Additionally, the link of code and other details for reproducing are provided in Appendix F. The datasets are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
model (e.g., a large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases, in which authors welcome to describe the particular way they provide for reproducibility. In the case of closed-sourge models, it may be that access to the model is limited in some way (c.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
24

---

Justification:
The link of code is provided in the abstract, and other details for environment and configurations are provided in Appendix The datasets are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.cm ctt.ipssccppbbiic////
www.imiissoooooicyyy.. for more details.
While encourage the release of code and data, understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /niiss.p..../// cc/public/ guides/CodeSun
ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer to Section 5.1 and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report error bars in our experimental results, such as in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by eгror bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
25

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: Please refer to Appendix F.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code Ethics
Does the research conducted in the paper conform, every respect, with the NeurIPS Code of Ethics https: //tp:r://///r///e// ips. http:////eaaaap.c.ppap//aaaaicca cc/ppblicc/thiiccctui.e.......id
Question:
Answer: [Yes]
Justification:
Our research conforms with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method addresses the few-shot molecular property prediction problem, which is important for drug discovery and material science, as discussed in the introduction and conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
legitimate to point out that
26

---

The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e-g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Our research poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been seraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly respect and credit for existing assets. We cite relevant papers and comply with the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
If this information
available online, the authors are encouraged reach out the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our code is well documented.
Guidelines:
The answer NA means that the paper does not release assets.
27

---

Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
28


Reviews:
[{"review_id": "LFy6LnScvV", "reviewer": "Reviewer_1DKF", "summary": "This paper introduces a novel Pin-Tuning method. Focusing on improving the fine-tuning process of pre-trained molecular encoders, especially for the task of Few-Shot Molecular Property Prediction (FSMPP), Pin-Tuning skillfully balances the contradiction between the number of tunable parameters and the limited labeled molecular data, while reinforcing the encoder's context-awareness. The core of the approach is the introduction of MP-Adapter, a lightweight adapter for pre-trained message-passing layers, and Emb-BWC, a Bayesian weight consolidation scheme for pre-trained atom/key embedding layers. Experimental results show that Pin-Tuning exhibits excellent performance on public datasets, significantly improves the prediction performance in few-shot scenarios with fewer trainable parameters, and proves its effectiveness in the field of molecular attribute prediction.", "questions": "1.What does the a mean in Effect of weight of Emb-BWC regularizer λ in Sensitivity analysis? Is there a confusing typo here?\n2.What are the advantages of the proposed MP-Adapter over existing adapter technology?\n3.How to ensure the quality and reliability of the contextual information used in the paper?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The paper proposes Pin-Tuning methods, including MP-Adapter and Emb-BWC, which provide new solutions to the problem of fine-tuning pre-trained models in FSMPP tasks.\n2.In the paper, it is proposed to integrate the context-aware capability into the adapter, which enhances the model's ability to perceive the molecular context and enables the model to be more effectively fine-tuned on few data samples.\n3.The Pin-Tuning method proposed in the paper is evaluated on a public dataset, showing fewer trainable parameters and greater improvement in prediction performance, which demonstrates the effectiveness of the method.", "weaknesses": "1.The concept of Adapter has been widely studied and applied in the field of Natural Language Processing. In this paper, does MP-Adapter just take this concept over, or is there any design and optimization specific to molecular graphical neural networks (GNNs)? Is there any comparison with existing generalized frameworks for adapter networks?\n2.The observation from the paper that the use of Identity matrix approximation works best in Emb-BWC seems to be a counterintuitive finding. This is because the Identity matrix approximation ignores possible correlations between parameters and assigns the same importance to each parameter. Is it true that correlations between parameters are less important in the molecular property prediction task than in other tasks? If so, does this reflect an intrinsic characteristic of molecular property prediction tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer 1DKF\n\nWe thank the reviewer for your constructive feedback. Please find detailed responses below.\n\n> `W1` `Q2`  Compared to adapters in NLP, the specific design and advantages of the proposed MP-Adapter.\n\nBased on your valuable feedback, we have provided a detailed description in the `Global Response` to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to compare the performance of fine-tuning molecular encoders using NLP adapters and our MP-Adapter, empirically demonstrating the advantages of our method. **We kindly suggest referring to our `Global Response` regarding this issue.**\n\n> `W2` Discussion on the experimental observations of Emb-BWC.\n\nThank you for your insightful comments. This is a very meaningful question that deserves in-depth exploration. As we stated in Section 5.3, the results indicate that keeping pre-trained parameters to some extent can better utilize pre-trained knowledge, but the parameters worth keeping in fine-tuning and the important parameters in pre-training revealed by the Fisher information matrix are not completely consistent. Here, based on our experimental results, we provide a more in-depth discussion and our insights from the following two aspects:\n\n1. `Importance of each parameter`: This involves two questions: `(a) Is it necessary to impose constraints on the importance of parameters?` `(b) What kind of importance assignment strategy is optimal for fine-tuning molecular representations?` For the first question, since the empirical results with three types of regularizers are better than those without any regularizer, it indicates that imposing importance constraints on the parameters is beneficial for fine-tuning molecular representations. For the second question, the observation is that the more relaxed the Emb-BWC constraint, the better the fine-tuning performance. The Emb-BWC constraint measures the importance of parameters in pre-training and uses this importance to constrain the fine-tuning process. **The explanation of this observation is that the important parameters in pre-training and the parameters that need to be retained during fine-tuning do not completely overlap, and some mechanisms of message passing require considerable updates during the fine-tuning process.**\n2. `Correlation between different parameters`: Since the computation of the Hessian matrix in the original form of the Emb-BWC regularizer is intractable, we provide three diagonal approximation methods. Since all three approximation methods result in diagonal matrices, with non-zero values only on the main diagonal, they all assume that the contribution of each parameter update to the model performance is independent. In other words, **these three diagonal approximation methods imply that importance is assigned to each parameter independently, streamlining the correlations between parameters.** The off-diagonal values of the Hessian matrix can constrain the joint updates of parameters, but calculating these off-diagonal elements, whether through the original form or the approximated Fisher information matrix, is intractable due to the high-dimensional nature of the parameters. Therefore, **the better performance of the identity matrix approximation does not imply that the correlations between parameters are unimportant in molecular property prediction. Instead, it reflects that parameters which are not important during pre-training may have high importance during fine-tuning.**\n\n> `Q1` Is $a$ a typo for the weight $\\lambda$ of the Emb-BWC regularizer in Section 5.4?\n\nYes, the $a$ in the sensitivity analysis should be the regularization coefficient $\\lambda$. This is a typo, and we greatly appreciate you pointing it out.\n\n> `Q3` How to ensure the quality and reliability of the context information?\n\nSince the molecular context is encoded based on the labels of the molecules in the target task and auxiliary tasks, **the accuracy and adequacy of these labels determine the quality and reliability of the contextual information**. When the required contextual labels are complete and accurately measured, the context is sufficiently reliable. However, the real situation is often not ideal, with some missing and noisy labels.\n\n**The GNN-based context encoder we adopted has already mitigated the issues of missing and noisy labels to some extent by propagating and smoothing information on the molecule-property bipartite graph**, thereby obtaining context representations that are robust to missing and noisy labels.\n\nTo further improve the reliability of the context, **our potential solution is to rigorously measure and calibrate the uncertainty of the contextual labels**, such as measuring the entropy of the contextual label matrix. This could potentially further enhance the robustness of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJelsGzqnX", "reviewer": "Reviewer_W6tN", "summary": "The authors propose a strategy for few-shot drug discovery scenarios. The authors propose to fine-tune representations retrieved from encoder layers with adapters. In addition to the initial representations from the message passing layers, the adapters are provided with property representations and \"context-aware\" molecule representations to improve the initial representations. Both the property representations and the \"context-aware\" molecule representations stem from a context encoder. The context encoder creates learned molecule and property representations by relating the input molecule representations and the current few shot task, i.e. the property with other molecules and other already seen properties. The authors evaluate their approach in several datasets and include an ablation study.", "questions": "-", "limitations": "Limitations have been addressed in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**:\n- **(S-O)**:  The Context encoder contains novelty. Jointly learning task and molecule representations with GNNs and updating molecule representation based on relationships between them is novel (also see (S-Q2)).\n\n**Quality**:\n- **(S-Q1)**: The model shows strong performance for the chosen experimental setup. In the main experiment (Table 1) the authors include error bars and therefore follow best practices of comparing drug discovery models.\n- **(S-Q2)**: With their novel strategy to include in-context information, i.e. task information, the authors found a way to use label information of query and support set molecules present in the training set. This seems promising and has not been done before (S-O). E.g., compare [1] in which molecule representations are updated being aware of training molecules but not-aware of their training labels.\n\n**Clarity**:\n- **(S-C): Chapter 4 introduces very well the goal of this work. Taking into account all three textual descriptions formulas and Figure 2, the proposed method and its components become clear.\n\n**Significance**:\n- **(S-S1)**: The novel Context Encoder is relevant for the community (see (S-Q2)).\n- **(S-S2)**: For the chosen experimental setup, the author's approach outperforms chosen baseline methods. \n\n[1] Schimunek, Johannes, et al. \"Context-enriched molecule representations improve few-shot drug discovery.\" The Eleventh International Conference on Learning Representations.", "weaknesses": "**Originality**:\n- **(W-O1)**: Using adapters to efficiently fine-tune models is not novel. The core idea is to efficiently fine-tune a layer by adding another learnable component with less parameters. Instead of changing the model layer itself, the new component learns to modify the outcome of the model layer. This has already been done in few-shot drug discovery [2]. \n- **(W-O2)**: In-context fine-tuning in the sense of adapting molecular representations to the few-shot task is not novel and has been done e.g. here [1, 3] .\n\n**Quality**:\n- **(W-Q1)**: The authors missed to include the SOTA benchmark for few-shot drug discovery FS-Mol.\n- **(W-Q2)**: The authors missed that in-context fine-tuning (W-O2) and efficient adapter-like fine-tuning (W-O1) has already been applied to few-shot drug discovery.\n - **(W-Q3)**: Missing error bars for Figure 1, Table 2, Figure 4, and Figure 5\n\n**Significance**:\n- **(W-S1)**: The significance of the experimental section is diminished because the SOTA benchmark for few-shot drug discovery is missing (see (W-Q1))\n- **(W-S2)**: The significance of the proposed method is diminished because of (W-Q2). Being aware of the ideas which have already been applied to few-shot drug discovery would have allowed to put the context encoder into the focus of this work. The ablation study indicates that this module is important. Still in the current version of the paper, this novel piece of architecture remains underexplored.\n\n**Clarity**:\n- **(W-C1)**: Because of (W-S2, W-Q2) it remains unclear which parts of the proposed method have been tested in related work and which parts are novel. The authors might think about changing the main story line of their paper by focusing on their training label aware-fine-tuning procedure.\n- **(W-C2)**: The procedure which happens inside the context encoder is not displayed well in Figure 2. Since this is a central part, this should be improved.\n- **(W-C3)**: English editing is required\n- **(W-C4)**: The mathematical notation and formulas seem cluttered and inconsistent. E.g., in 3.2. a molecular encoder is denoted by $f(.)$ but the context encoder is denoted by ContextEncoder. Generally, the authors should stick to standard notation and avoid \"full-word notation\" in formulas.\n\n\n[2] Adler, Thomas, et al. \"Cross-domain few-shot learning by representation fusion.\" (2020).\\\n[3] Altae-Tran, Han, et al. \"Low data drug discovery with one-shot learning.\" ACS central science 3.4 (2017): 283-293.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer W6tN (1/3)\n\nWe thank the reviewer for your constructive feedback. Please find detailed responses below.\n\n> `W-O1` `W-Q2` The novelty of our adapter-based parameter-efficient tuning.\n\nWe greatly appreciate your comments. Indeed, there are several adapter-like methods used to fine-tune pre-trained models in various fields. Unlike other adapters, our MP-Adapter is specifically designed for molecular encoders based on graph neural networks (GNNs) that follow the message passing mechanism. Here, **we compare our MP-Adapter with two potentially similar methods to highlight the novelty of our approach**: (1) adapters used in natural language processing (NLP) for transformer-based language models, and (2) the representation fuser CHEF proposed in previous few-shot drug discovery work [1] designed for fully-connected deep neural networks (FCNs).\n\n1. `The comparison between our MP-Adapter and adapters used in NLP`: This comparison has been provided in our `Global Response` to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to compare the performance of fine-tuning molecular encoders using NLP adapters and our MP-Adapter, empirically demonstrating the advantages of our method. **We kindly suggest referring to our `Global Response` regarding this issue.**\n\n2. `The comparison between our MP-Adapter and the representation fuser CHEF [1]`:  CHEF fuses representations from different layers by attaching a learnable learner after each frozen pre-trained layer and ensembling the outputs of all learners. A critical difference between CHEF and our MP-Adapter is that **in CHEF, the output of each learner is fed into the final fuser for ensembling rather than being fed to the next layer of the encoder**. Therefore, in the CHEF framework, updating the output of the previous layer through the added learner does not affect the output of subsequent layers. **This approach is only suitable for global encoding models, such as the FCNs used in the experiments evaluating CHEF, which take molecular fingerprints (ECFP6) as input [1].** However, the current state-of-the-art molecular encoders are based on GNNs and take molecular graphs as input. GNN-based molecular encoders follow a message passing mechanism to achieve localized neighborhood information aggregation. In this case, the interaction between different layers of the encoder is more important, and changes in the output of the previous layer will significantly affect the message passing of the next layer. Therefore, **passing the adapter's output to the next encoding layer as input, as our MP-Adapter does, is more appropriate for GNN-based molecular encoders.**\n\nOverall, neither NLP adapters nor the representation fuser CHEF can be trivially used to fine-tune GNN-based molecular encoders. We have also summarized the comparison of them in the table below to clearly illustrate the advantages and novelty of our MP-Adapter:\n\n|                | Target Model                      | Target Component                          | Suitable for Localized Encoding Layer | Capable of Perceiving Molecular Context |\n| -------------- | --------------------------------- | ----------------------------------------- | ------------------------------------- | --------------------------------------- |\n| CHEF [1]       | FCNs                              | Fully-connected Layer                     | No                                    | No                                      |\n| NLP Adapter    | Transformer-based Language Models | Multi-head Attention / Feed Forward Layer | Yes                                   | No                                      |\n| Our MP-Adapter | GNN-based Molecular Encoders      | Message Passing Layer                     | Yes                                   | Yes                                     |\n\n> `W-O2` `W-Q2` Novelty of in-context fine-tuning.\n\nAlthough some previous works have also mentioned the concept of `molecular context`, our work differs in the definition, encoding method, and perceiving method of molecular context.\n\n- `Definition of molecular context`: In IterRefLSTM [2] and PAR [3], the context refers to the structural and property labels of support molecules concerning the target property, while in MHNfs [4], the context refers to molecules that are structurally similar to the given molecule among a large number of unlabeled molecules. Unlike these methods, **the molecular context in our approach refers to the label information of seen and unseen molecules in auxiliary properties, which are informative and more relevant to the target task**.\n- `Encoding of molecular context`: Previous works [2,3,4] typically use LSTM or attention mechanisms to learn the correlations between molecules, often neglecting the interactive relationship between molecules and properties. **We represent the molecular context as a molecule-property bipartite graph and encode it using a GNN-based context encoder, which allows for more effective and robust learning of the relationships between molecules and properties.**\n- `Perceiving of molecular context`: Previous works [2,3,4] combine non-contextual molecular representations with context representations to predict molecular properties. In this paradigm, the resulting molecular representations cannot perceive the molecular context information. **In our method, the molecular context is introduced into the fine-tuning process through the MP-Adapter, allowing the molecular encoder to be fine-tuned under the guidance of the context, thereby obtaining contextual molecular representations.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "us3f85mGCN", "reviewer": "Reviewer_u7hY", "summary": "This paper proposes Pin-Tuning, a parameter-efficient in-context tuning method for few-shot molecular property prediction (FSMPP), mitigating the parameter-data imbalance and enhancing the contextual perceptiveness of pre-trained molecular encoders. This method treats the embedding layers and message passing layers in the pre-trained molecular encoder separately, based on specific reasons why they could not be effectively fine-tuned before. This includes a series of Bayesian weight consolidation constraints for the embedding layers and the bottleneck adapters for the message passing layers. Additionally, molecular context is introduced into the adapter modules to enable contextual perceptiveness.  Experiments show that the proposed method effectively improves few-shot performances across a range of benchmark datasets.", "questions": "1. What are the details of the pilot experiment in Figure 1? Including the featurization, the model architecture, and the pre-training strategy of each model. More details are needed to assess the fairness of the experiment and to strengthen the persuasiveness of your claims.\n2. Does $\\mathcal{L}_{update}$ in Figure 2(d) refer to $\\mathcal{L}_{Emb-BWC}$ in Section 4.1.2? The content in the figure should be consistent with the text.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The motivation is convincingly derived from observations in the pilot experiment in Figure 1, indicating the necessity to design a more effective tuning method for few-shot molecular tasks.\n2. The authors design different tuning methods for the embedding layer and the message passing layer in the pre-trained molecular encoder to accommodate their different parameter scales and parameter forms.\n3. The proposed Emb-EWC constraint is derived based on Bayesian learning theory, and the authors provide intuitive explanations for each approximation choice to make them easier to understand.", "weaknesses": "1. It is recommended to add details about the pilot experiment in Figure 1, such as the featurization and the specific architectures of different models, and whether the pre-training strategies are consistent.\n2. Table 4 only reports the \"total size of the molecular encoder.\" Considering that some parameters are frozen in your method, you could add a row for the \"size of the part that needs to be tuned\". This would be helpful to make the advantages of your method clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer u7hY\n\nWe thank the reviewer for your constructive feedback. Please find detailed responses below.\n\n> `W1` `Q1` Add details on pilot experiment in Figure 1.\n\nThank you very much for your reply. We apologize for the lack of details about the pilot experiment due to the length limit of the paper. We further supplement the details of the comparison of GIN-Mol [1], CMPNN [2], and Graphormer [3] as follows.\n\n- `Featurization`: To ensure a fair comparison with existing methods for few-shot molecular property prediction (FSMPP), we employed the same featurization approach as them. Following prior works of molecular representation learning [1] and few-shot molecular property prediction [4], we used two atomic features (`atomic number` and `chiral tag`) and two bond features (`bond type` and `bond direction`). \n- `Architecture Choice`: GIN-Mol is a GIN tailored to molecules, proposed in Pre-GNN [1]. This molecular GIN has become a representative encoder in molecular pretraining, which is equipped with multiple bond embedding layers and introduces bond features into message passing. **The standard 5-layer Mol-GIN is adopted**. To facilitate interaction between node and directed edge information, CMPNN introduces a node-edge message communication module and offers multiple architectural options. We selected the **multilayer perception (MLP) to implement the message communication module**, since experiments with CMPNN have demonstrated that this architecture performs best. For Graphormer, we adopted the standard architecture based on multi-head attention, **using node degrees, shortest distances between nodes, and edge types to construct the position encoding**.\n- `Hyperparameters`: We adopted the hyperparameters provided by Mol-GIN, CMPNN and Graphormer. Primarily, the size of the hidden dimension is 300 and the number of encoding layers is 5. For Graphormer, the number of heads is set to 8, and the dropout rate for attention is 0.1.\n- `Pre-training and Adaptation Strategy`: To ensure a fair comparison, we followed prior FSMPP methods that use a pre-trained GIN as the molecular encoder. We pre-trained GIN-Mol, CMPNN, and Graphormer **with both the supervised task and the `Context Prediction` self-supervised task** [1], then adapted them to the FSMPP task through fine-tuning.\n\n> `W2` Make a comparison in terms of tunable parameter size in Table 4.\n\nThank you for your valuable feedback. We appreciate your suggestion to provide additional details in Table 4 regarding the size of the tunable part of the model. In response to your comment, we have added a new row to Table 4 that specifies the size of the parameters tuned in our method, thereby better illustrating the advantages of our approach. The revised Table 4 is shown below, and is also provided in the `Global Response` PDF.\n\n|| GS-Meta | Ours|\n| - | - | - |\n| Size of Molecular Encoder   | 1.86M | 1.86M  |\n| Size of MP-Adapter       | -  | 0.21M |\n| Size of Context Encoder   | 0.62M | 0.62M |\n| Size of Classifier    | 0.18M | 0.27M |\n| Size of Total Model   | 2.66M | 2.96M |\n| **Size of Tunable Part of the Model** | **2.66M** | **1.10M** |\n\n> `Q2` Inconsistency in the notion of the Emb--BWC regularizer between Figure 2 and Section 4.1.2.\n\nYes, $\\mathcal{L}\\_{update}$ in Figure 2 refers to $\\mathcal{L}\\_{Emb-BWC}$. Thank you for pointing out this mistake and helping us improve the clarity of the paper. We have revised this figure in the Global Response PDF.\n\n**References**\n\n[1] Strategies for Pre-training Graph Neural Networks. ICLR 2020\n\n[2] Communicative Representation Learning on Attributed Molecular Graphs. IJCAI 2020\n\n[3] Do Transformers Really Perform Bad for Graph Representation? NeurIPS 2021\n\n[4] Property-Aware Relation Networks for Few-Shot Molecular Property Prediction. NeurIPS 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5wgWh3typm", "reviewer": "Reviewer_nYSV", "summary": "This work introduce a new system for the FSMPP tasks, which includes: context adapter in GNN, context graph, and  weight consolidation. With these three techniques, the Pin-Tuning method achieved promising results on many FSMPP benchmarks.", "questions": "- Why different strategies are used for embedding layers and encoder layers? What will be the results for (a) finetune encoder layers with weight constraint, and (b) freeze embedding layers and add adapters to them? \n\n-  According to Table 3, the simplest weight consolidation method (i.e., IM) is the best. In this case, does the BWC theory really required as it is just L2 penalty on the changed parameters?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- FSMPP tasks are important and have grate research value. \n- The Pin-Tuning system is carefully designed and evaluated.", "weaknesses": "- The adapter based parameter efficient tunning have already been used in many tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer nYSV\n\nWe thank the reviewer for your constructive feedback. Please find detailed responses below.\n\n> `W1` The novelty of our MP-Adapter for parameter efficient tunning.\n\nBased on your valuable feedback, we have provided a detailed description in the `Global Response` to clarify the specific design and considerations of our MP-Adapter for molecular representation fine-tuning. Additionally, we conducted experiments to empirically demonstrate the advantage of our method. **We kindly suggest referring to our `Global Response` regarding this issue.**\n\n> `Q1` The reasons for adopting different strategies for embedding layers and message passing layers, as well as the results of the opposite strategies.\n\nOur goal is to perform parameter-efficient tuning on pre-trained molecular encoders to address the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules.\n\n- **For message passing layers, the number of parameters is disproportionately large compared to the training samples.** To mitigate this imbalance, we design a lightweight adapter targeted at the message passing layers, called MP-Adapter.\n- **Unlike message-passing layers, embedding layers have a very small number of parameters.** Therefore, we treat embedding layers in a different way than message passing layers by not using adapters. Instead, we directly fine-tune the parameters of the embedding layers, but impose a constraint called Emb-BWC to limit the magnitude of parameter updates, preventing aggressive optimization and catastrophic forgetting.\n\nBased on your constructive suggestions, we conducted experiments on the opposite design to further verify that the two strategies are suitable for their respective target components. Specifically, on the basis of our Pin-Tuning, we (Variant 1) changed the tuning strategy for the embedding layers to MP-Adapter, and (Variant 2) changed the tuning strategy for the message passing layers to Emb-BWC constraint.\n\n| Model | Tuning Strategy for Embedding Layers | Tuning Strategy for Message Passing Layers | Tox21 |  | SIDER |  | MUV | | PCBA |  |\n|-|-|-|-|-| - | - | - | - | - | - |\n| | |  | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot |\n| Pin-Tuning (Ours) | Emb-BWC | MP-Adapter| 91.56   | 90.95  | 93.41   | 92.02  | 73.33   | 70.71  | 81.26   | 79.23  |\n| Variant 1   | MP-Adapter   | MP-Adapter   | 91.05| 89.19| 90.38 | 89.81  | 72.99   | 70.50  | 80.93   | 79.79  |\n|Variant 2| Emb-BWC|Emb-BWC|88.53|87.24|88.45|87.02|69.76|68.43|79.39|78.11|\n\n**Changing the tuning strategy of any component to the opposite on the basis of Pin-Tuning results in degraded performance**, which indicates the suitability of the two tuning strategies for their respective target components. Furthermore, **selecting the appropriate tuning strategy for the message passing layers has a greater impact on performance**. Particularly on the MUV dataset, adapter-based parameter-efficient fine-tuning of the message passing layers is key to achieving success on this dataset. For the reasons behind the degraded performance of the variants, we provide the following insights:\n\n- `Reasons why the Emb-BWC is not suitable for message passing layers`: First, the parameters of message passing layers are numerous and diverse in type. Therefore, applying the same weight constraint to numerous and diverse parameters cannot account for the differences in importance between parameters, making it less flexible than adapters. Second, in our method, adapters serve as carriers for introducing molecular context during fine-tuning, which cannot be achieved by directly updating the parameters. **Overall, the Emb-BWC constraint is not flexible enough for tuning modules with a large number of parameters and cannot introduce additional molecular context information.**\n- `Reasons why the MP-Adapter is not suitable for embedding layers`: The embedding layers function as lookup tables that map raw features into vectorized features, and they inherently have a small number of parameters. The advantage of adapters lies in their ability to perform parameter-efficient tuning. However, for the embedding layers, the number of parameters to be tuned is comparable whether directly updating the embedding layers or updating them through an adapter. **Overall, adapters are more suitable for modules with a large number of parameters, rather than for modules like the embedding layers with relatively few parameters.**\n\n> `Q2` Given that the simplest weight consolidation method (i.e., IM) is the best, does the BWC theory really required as it is just L2 penalty on the changed parameters?\n\nAlthough the simplest identity matrix approximation of Emb-BWC achieved the best results, we also obtained the following important observations from this set of experiments. **These observations reveal the intrinsic nature of the importance of parameters in pre-training and fine-tuning in molecular property prediction tasks, and can provide guidance on how to effectively fine-tune these parameters.**\n\n- Since the empirical results with three types of Emb-BWC regularizers are better than those without any regularizer, it indicates that imposing importance constraints on the parameters is beneficial for fine-tuning molecular representations.\n- The principle of the BWC theory is to measure the importance of parameters during pre-training and use this importance to constrain the fine-tuning process. This approach helps in preserving the knowledge acquired during pre-training by preventing significant changes to the important parameters during fine-tuning. It is observed that the more relaxed the Emb-BWC constraint, the better the fine-tuning performance. The explanation is that the important parameters in pre-training and the parameters that need to be retained during fine-tuning do not completely overlap, and some mechanisms of message passing require considerable updates during the fine-tuning process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We sincerely appreciate all the reviewers for your valuable feedback on our paper. In this global response, we aim to address the reviewers' concerns regarding the novelty and empirical advantages of our MP-Adapter. Specifically, we intend to answer the following questions:

>`The novelty of the MP-Adapter`: Considering that adapters have been widely used in natural language processing (NLP), what is the novelty of the MP-Adapter proposed in this paper for pre-trained message passing layers? In other words, what specific designs or considerations does the MP-Adapter have for molecular tasks and models?
>
>`The empirical advantage of the MP-Adapter`: Compared to using ordinary NLP adapters, what are the advantages in empirical performance when fine-tuning pre-trained molecular encoders with MP-Adapter?

The  insertion positions and structure of our proposed MP-Adapter are tailored to molecular representation learning, taking into account the architecture of molecular encoders and the need for perceiving molecular context. Specifically, the differences between our MP-Adapter and the ordinary NLP adapters widely used in NLP are as follows:

1. **The position where the MP-Adapter is inserted into the pre-trained molecular encoder is determined by considering the overall architecture of the adapted molecular encoder and the target module.** In NLP, the adapted model are typically pre-trained transformer-based large language models. Whether it is an encoder-only, encoder-decoder, or the currently popular decoder-only language model backbone, the basic units are stacked transformer layers, which are the target for adaptation by NLP adapters. NLP adapters are usually inserted after the `multi-head attention` modules or `feed-forward` modules to fine-tune them parameter-efficiently, as these modules conduct the most critical operations in the transformer layers [1,2]. In molecular representation learning, the pre-trained molecular encoders that need to be adapted use graph neural networks (GNNs) as their model backbone. Molecular encoders follow the message passing mechanism, comprising `atom/bond embedding layers`, `message passing layers consisting of aggregation function and update function`, and the final `readout function`. The insertion positions of our MP-Adapter are specifically designed for the message passing mechanism of the GNNs. Since the update function in the message passing layer is the source of GNNs' high expressive power [3] and has the most complex parameters, the MP-Adapter is inserted after the update functions to adapt them.
2. **Our MP-Adapter takes both the output of the message passing layers and the encoded molecular context as inputs, thereby enabling fine-tuning under the guidance of the molecular context.** Unlike the adapters in NLP, our MP-Adapter not only achieves parameter-efficient fine-tuning of the pre-trained parameters but also incorporates the encoded molecular context as an additional input to the adapter, achieving in-context tuning. In Section 4.2, based on the significance of molecular context, we propose the method for encoding molecular context and incorporating it into the MP-Adapter. The initial Eq. 3 is updated to Eq. 7, reflecting the consideration of molecular context, which is also a special design specifically for fine-tuning molecular encoders.

We compared the impact of MP-Adapter and NLP Adapter on fine-tuning pre-trained molecular encoders, and the results are presented in the table below. The three models being compared are not equipped with the proposed Emb-BWC to ensure that the only difference lies in the choice of adapter. In the table, `No Adapter` represents the most state-of-the-art baseline method GS-Meta, `NLP Adapter` adds a vanilla NLP adapter after the message passing layers in GS-Meta, and `MP-Adapter` equips the message passing layers with our context-aware MP-Adapter, without our proposed Emb-EWC constraint. For each experiment, we run 10 times with different seeds and report the average ROC-AUC score.

|                                   | Tox21   |        | SIDER   |        | MUV     |        | PCBA    |        |
| --------------------------------- | ------- | ------ | ------- | ------ | ------- | ------ | ------- | ------ |
|                                   | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot |
| No Adapter                        | 86.67   | 86.43  | 84.36   | 84.57  | 66.08   | 64.50  | 79.40   | 77.47  |
| NLP Adapter                       | 87.94   | 87.80  | 85.18   | 84.79  | 71.86   | 69.58  | 79.75   | 78.35  |
| MP-Adapter                        | 90.17   | 89.59  | 92.06   | 91.43  | 72.37   | 71.65  | 80.74   | 78.51  |
| Pin-Tuning (MP-Adapter + Emb-BWC) | 91.56   | 90.95  | 93.41   | 92.02  | 73.33   | 70.71  | 81.26   | 79.23  |

From the results above, it can be observed that using the vanilla NLP Adapter can improve the fine-tuning performance in few-shot scenarios, thanks to the reduction in the number of tunable parameters and our decision on the appropriate insertion positions. Our MP-Adapter further introduces molecular context information, which is crucial for molecular property prediction, into the fine-tuning process, resulting in better performance.

**References**

[1] LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. EMNLP 2023

[2] Efficient Large Language Models: A Survey. TMLR 2024

[3] How powerful are graph neural networks? ICLR 2019

Meta-review: The paper introduces Pin-Tuning, a parameter-efficient fine-tuning approach specifically designed for GNN-based few-shot molecular property prediction tasks. This approach features a novel adapter layer (MP-Adapter) and a Bayesian weight consolidation scheme (Emb-BWC), both of which are combined to demonstrate superior performance across benchmark datasets. The authors' thorough rebuttal, which included additional experiments and clarifications, addressed the reviewers' concerns, highlighting the method's relevance and potential impact. This work offers a novel approach that will positively contribute to few-shot molecular property prediction literature and will be of significant interest to the NeurIPS audience.
Decision: Accept (poster)