Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Prediction

Authors: liang  wang
qiang  liu
shaozhen  liu
xin  sun
shu  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:55:50

Abstract: Molecular property prediction (MPP) is integral to drug discovery and material science, but often faces the challenge of data scarcity in real-world scenarios. Addressing this, few-shot molecular property prediction (FSMPP) has been developed. Unlike other few-shot tasks, FSMPP typically employs a pre-trained molecular encoder and a context-aware classifier, benefiting from molecular pre-training and molecular context information. Despite these advancements, existing methods struggle with the ineffective fine-tuning of pre-trained encoders. We attribute this issue to the imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and the lack of contextual perceptiveness in the encoders. To overcome this hurdle, we propose a parameter-efficient in-context tuning method, named Pin-Tuning. Specifically, we propose a lightweight adapter for pre-trained message passing layers (MP-Adapter) and Bayesian weight consolidation for pre-trained atom/bond embedding layers (Emb-BWC), to achieve parameter-efficient tuning while preventing over-fitting and catastrophic forgetting. Additionally, we enhance the MP-Adapters with contextual perceptiveness. This innovation allows for in-context tuning of the pre-trained encoder, thereby improving its adaptability for specific FSMPP tasks. When evaluated on public datasets, our method demonstrates superior tuning with fewer trainable parameters, improving few-shot predictive performance.

Main paper: 
 introduction: 
 In the field of drug discovery and material science, molecular property prediction (MPP) stands as a pivotal task [5, 9, 63] . MPP involves the prediction of molecular properties like solubility and toxicity, based on their structural and physicochemical characteristics, which is integral to the development of new pharmaceuticals and materials. However, a major challenge encountered in real-world MPP scenarios is data scarcity. Obtaining extensive molecular data with well-characterized properties can be time-consuming and expensive. To address this, few-shot molecular property prediction (FSMPP) has emerged as a crucial approach, enabling predictions with limited labeled molecules [1, 41, 4] .The methodology for general MPP typically adheres to an encoder-classifier framework [71, 23, 27, 56] , as illustrated in Figure 2(a) . In this streamlined framework, the encoder converts molecular Figure 1 : Comparison of molecular encoders trained via different paradigms: train-from-scratch, pretrain-then-freeze, and pretrain-then-finetune. The evaluation is conducted across two datasets and three encoder architectures [20, 47, 66] . The results consistently demonstrate that while pretraining outperforms training from scratch, the current methods do not yet effectively facilitate finetuning. Despite the progress, there are observed limitations in the current approaches to FSMPP. Notably, while using a pre-trained molecular encoder generally outperforms training from scratch, fine-tuning the pre-trained encoder often leads to inferior results compared to keeping it frozen, which can be observed in Figure 1 .The observed ineffective fine-tuning can be attributed to two primary factors: (i) Imbalance between the abundance of tunable parameters and the scarcity of labeled molecules: fine-tuning all parameters of a pre-trained encoder with few labeled molecules leads to a disproportionate ratio of tunable parameters to available data. This imbalance often results in over-fitting and catastrophic forgetting [7, 6] . (ii) Limited contextual perceptiveness in the encoder: while molecular context is leveraged to enhance the classifier [58, 73] , the encoder typically lacks the explicit capability to perceive this context, relying instead on implicit gradient-based optimization. This leads to the encoder not directly engaging with the nuanced molecular context information that is critical in FSMPP tasks. In summary, while significant strides have been made, the challenges of imbalance between the number of parameters and labeled data, along with the need for contextual perceptiveness in the encoder, necessitate more sophisticated methodologies in this domain.Based on the aforementioned analysis, we propose the parameter-efficient in-context tuning method, named Pin-Tuning, to address the two primary challenges in FSMPP. To overcome the parameterdata imbalance, we propose a parameter-efficient chemical knowledge adaptation approach for pre-trained molecular encoders. A lightweight adapters (MP-Adapter) are designed to tune the pretrained message passing layers efficiently. Additionally, we impose a Bayesian weight consolidation (Emb-BWC) on the pre-trained embedding layers to prevent aggressive parameter updates, thereby mitigating the risk of over-fitting and catastrophic forgetting. To address the second challenge, we further endow the MP-Adapter with the capability to perceive context. This innovation allows for in-context tuning of the pre-trained molecular encoders, enabling them to adapt more effectively to specific downstream tasks. Our approach is rigorously evaluated on public datasets. The experimental results demonstrate that our method achieves superior tuning performance with fewer trainable parameters, leading to enhanced performance in few-shot molecular property prediction.The main contributions of our work are summarized as follows:• We analyze the deficiencies of existing FSMPP approaches regarding the adaptation of pre-trained molecular encoders. The key issues include an imbalance between the number of tunable parameters and labeled molecules, as well as a lack of contextual perceptiveness in the encoders. • We propose Pin-Tuning to adapt the pre-trained molecular encoders for FSMPP tasks. This includes the MP-Adapter for message passing layers and the Emb-BWC for embedding layers, facilitating parameter-efficient tuning of pre-trained molecular encoders. • We further endow the MP-Adapter with the capability to perceive context to allows for in-context tuning, which provides more meaningful adaptation guidance during the tuning process. • We conduct extensive experiments on benchmark datasets, which show that Pin-Tuning outperforms state-of-the-art methods on FSMPP by effectively tuning pre-trained molecular encoders.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We present our claims and main contributions in the abstract and introduction. We propose a parameter-efficient in-context tuning method, named Pin-tuning, to effectively adapt pre-trained molecular encoders to few-shot molecular property prediction tasks.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Few-shot molecular property prediction. Few-shot molecular property prediction aims to accurately predict the properties of new molecules with limited training data [49] . Early research applied general few-shot techniques to FSMPP. IterRefLSTM [1] is the pioneer work to leverage metric learning to solve FSMPP problem. Following this, Meta-GGNN [41] and Meta-MGNN [14] introduce meta-learning with graph neural networks, setting a foundational framework that subsequent studies have continued to build upon [39, 40, 4] . It is noteworthy that Meta-MGNN employs a pre-trained molecular encoder [20] and achieves superior results through fine-tuning in the meta-learning process compared to training from scratch. In fact, pre-trained graph neural networks [64, 36, 17, 54, 37] have shown promise in enhancing various graph-based downstream tasks [52, 13] , including molecular property prediction [60, 62, 38, 72] . Recent efforts have shifted towards leveraging unique nature in FSMPP, such as the many-to-many relationships between molecules and properties arising from the multi-labeled nature of molecules, often referred to as the molecular context. PAR [58] initially employs graph structure learning [32, 55] to connect similar molecules through a homogeneous context graph. MHNfs [45] introduces a large-scale external molecular library as context to augment the limited known information. GS-Meta [73] further incorporates auxiliary task to depict the many-to-many relationships.Parameter-efficient tuning. As pre-training techniques have advanced, tuning of pre-trained models has become increasingly crucial. Traditional full fine-tuning approaches updates all parameters, often leading to high computational costs and the risk of over-fitting, especially when available data for downstream tasks are limited [33, 15] . This challenge has led to the emergence of parameter-efficient tuning [26, 29, 34] . The philosophy of parameter-efficient tuning is to optimize a small subset of parameters, reducing the computational costs while retaining or even improving performance on downstream tasks [19, 69] . Among the various strategies, the adapters [18, 42, 59 ] have gained prominence. Adapters are small modules inserted between the pre-trained layers. During the tuning process, only the parameters of these adapters are updated while the rest remains frozen, which not only improves tuning efficiency but also offers an elegant solution to the generalization [70, 30, 8] . By keeping the majority of the pre-trained parameters intact, adapters preserve the rich pre-trained knowledge. This attribute is particularly valuable in many real-world applications including FSMPP.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work in Appendix H. 
problem formulation: 
 Let {T } be a collection of tasks, where each task T involves the prediction of a property p. } = ∅).The goal of FSMPP is to train a model using D train that can accurately infer new properties from a limited number of labeled molecules in D test . Episodic training has emerged as a promising strategy in meta-learning [10, 16] to deal with few-shot problem. Instead of retaining all {T train } tasks in memory, episodes {E t } B t=1 are iteratively sampled throughout the training process. For each episode E t , a particular task T t is selected from the training set, along with corresponding support set S t and query set Q t . Typically, the prediction task involves classifying molecules into two classes: positive (y = 1) or negative (y = 0). Then a 2-way K-shot episodeE t = (S t , Q t ) is constructed. The support set S t = {(m s i , y s i,t )} 2K i=1 includes 2K examples, each class contributing K molecules. The query set containing M molecules is denoted as Q t = {(m q i , y q i,t )} M i=1 .Encoder-classifier framework is widely adopted in FSMPP methods. As illustrated in Figure 2 (a), given a molecule m whose property need to be predicted, a molecular encoder f (•) first learns the molecule's representation based on its structure, i.e., h m = f (m) ∈ R d . The molecule m is generally represented as a graph m = (V, A, X, E), where V denotes the nodes (atoms), A represents the adjacent matrix defined by edges (chemical bonds), and X, E denote the original feature of atoms and bonds, then graph neural networks (GNNs) are employed as the molecular encoders [44, 51, 21] . Subsequently, the learned molecular representation is fed into a classifier g(•) to obtain the prediction ŷ = g(h m ). The model is trained by minimizing the discrepancy between ŷ and the ground truth y.Further, two key discoveries have been pivotal for FSMPP. The first is the proven effectiveness of pre-trained molecular encoders, while the second is the significant advantage gained from molecular context. Together, these discoveries have further reshaped the widely adopted FSMPP framework, which combines a pre-trained encoder followed by a context-aware classifier, as shown in Figure 2 (b).Due to the scarcity of labeled data in molecular tasks, molecular pre-training has emerged as a crucial area, which involves training encoders on extensive molecular datasets to extract informative representations. Pre-GNN [20] is a classic pre-trained molecular encoder that has been widely used in addressing FSMPP tasks [14, 58, 73] . The backbone of Pre-GNN is a modified version of Graph Isomorphism Network (GIN) [65] tailored to molecules, which we call GIN-Mol, consisting of multiple atom/bond embedding layers and message passing layers.Atom/Bond embedding layers. The raw atom features and bond features are both categorical vectors, denoted as (i v,1 , i v,2 , . . . , i v,|En| ) and (j e,1 , j e,2 , . . . , j e,|Ee| ) for atom v and bond e, respectively. These categorical features are embedded as:EQUATIONwhere EmbAtom a (•) a∈{1,...,|En|} and EmbBond b (•) b∈{1,...,|Ee|} represent embedding operations that map integer indices to d-dimensional real vectors, i.e., h (0) v , h (l) e ∈ R d , l ∈ {0, 1, . . . , L -1} represents the index of encoder layers, and L is the number of encoder layers. The atom embedding layer is present only in the first encoder layer, while an bond embedding layer exists in each layer.Message passing layers. At the l-th encoder layer, atom representations are updated by aggregating the features of neighboring atoms and chemical bonds:EQUATIONwhere u ∈ N (v) ∪ {v} is the set of atoms connected to v, and h (l) v ∈ R d is the learned representation of atom v at the l-th layer. MLP(•) is implemented by 2-layer neural networks, in which the hidden dimension is d 1 . After MLP, batch normalization is applied right before the ReLU. The molecule-level representation h m ∈ R d is obtained by averaging the atom representations at the final layer.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We provide the derivation of our proposed Emb-BWC regularization, with related assumptions in Appendix A. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
the proposed pin-tuning method: 
 This section delves into our motivation and proposed method. Our framework for FSMPP is depicted in Figure 2(c ). The details of our principal design, Pin-Tuning for PMEs, is present in Figure 2(d) .As shown in Figure 1 , pretraining then finetuning molecular encoders is a common approach. However, fully fine-tuning yields results inferior to simply freezing them. Thus, the following question arises:How to effectively adapt pre-trained molecular encoders to downstream tasks, especially in few-shot scenarios?We analyze the reasons of observed ineffective fine-tuning issue, and attribute it to two primary factors: (i) imbalance between the abundance of tunable parameters and the scarcity of labeled molecules, and (ii) limited contextual perceptiveness in the encoder.To address the first cause of observed ineffective tuning, we reform the tuning method for PMEs. Instead of conducting full fine-tuning for all parameters, we propose tuning strategies specifically tailored to the message passing layers and embedding layers in PMEs, respectively. For message passing layers in PMEs, the number of parameters is disproportionately large compared to the training samples. To mitigate this imbalance, we design a lightweight adapter targeted at the message passing layers, called MP-Adapter. The pre-trained parameters in each message passing layer include parameters in the MLP and the following batch normalization. We freeze all pre-trained parameters in message passing layers and add a lightweight trainable adapter after MLP in each message passing layer. Formally, the adapter module for l-th layer can be represented as:EQUATIONEQUATIONEQUATIONwhere FeedForward(•) denotes feed forward layer and LayerNorm(•) denotes layer normalization.To limit the number of parameters, we introduce a bottleneck architecture. The adapters downscale the original features from d dimensions to a smaller dimension d 2 , apply nonlinearity ϕ, then upscale back to d dimensions. By setting d 2 smaller than d, we can limit the number of parameters added. The adapter module has a skip-connection internally. With the skip-connection, we adopt the near-zero initialization for parameters in the adapter modules, so that the modules are initialized to approximate identity functions. Therefore, the encoder with initialized adapters is equivalent to the pre-trained encoder. Furthermore, we add a layer normalization after skip-connection for training stability.Unlike message passing layers, embedding layers contain fewer parameters. Therefore, we directly fine-tune the parameters of the embedding layers, but impose a constraint to limit the magnitude of parameter updates, preventing aggressive optimization and catastrophic forgetting.The parameters in an embedding layer consist of an embedding matrix used for lookups based on the indices of the original features. We stack the embedding matrices of all embedding layers to form Φ ∈ R E×d , where E represents the total number of lookup entries. Further, Φ i ∈ R d denotes the i-th row's embedding vector, and Φ i,j ∈ R represents the j-th dimensional value of Φ i .To avoid aggressive optimization of Φ, we derive a Bayesian weight consolidation framework tailored for embedding layers, called Emb-BWC, by applying Bayesian learning theory [3] to fine-tuning.Proposition 1: (Emb-BWC ensures an appropriate stability-plasticity trade-off for pre-trained embedding layers.) Let Φ ∈ R E×d be the pre-trained embeddings before fine-tuning, and Φ ′ ∈ R E×d be the fine-tuned embeddings. Then, the embeddings can both retain the atom and bond properties obtained from pre-training and be appropriately updated to adapt to downstream FSMPP tasks, by introducing the following Emb-BWC loss into objective during the fine-tuning process:EQUATIONwhere H(D P , Φ i ) ∈ R d×d is the Hessian of the log likelihood L P of pre-training dataset D P at Φ i .Details on the theoretical derivation of Eq. ( 6) are given in Appendix A. Since H(D P , Φ i ) is intractable to compute due to the great dimensionality of Φ, we adopt the diagonal approximation of Hessian. By approximating H as a diagonal matrix, the j-th value on the diagonal of H can be considered as the importance of the parameter Φ i,j . The following three choices are considered.Identity matrix. When using the identity matrix to approximate the negation of H, Eq. ( 6) is simplified toL IM Emb-BWC = 1 2 E i=1 d j=1 (Φ ′ i,j -Φ i,j ) 2, assigning equal importance to each parameter. This loss function is also known as L2 penalty with pre-trained model as the starting point (L2-SP) [31] .Diagonal of Fisher information matrix. The Fisher information matrix (FIM) F is the negation of the expectation of the Hessian over the data distribution, i.e., F = -E D P [H], and the FIM can be further simplified with a diagonal approximation. Then, the Eq. ( 6) is simplified toL FIM Emb-BWC = 1 2 E i=1 Fi (Φ ′ i -Φ i ) 2, where Fi ∈ R d is the diagonal of F(D P , Φ i ) ∈ R d×d and the j-th value in Fi is computed as E D P (∂L P /∂Φ i,j ) 2 . This is equivalent to elastic weight consolidation (EWC) [24] .Diagonal of embedding-wise Fisher information matrix. In different property prediction tasks, the impact of the same atoms and inter-atomic interactions may be significant or negligible. Therefore, we propose this choice to assign importance to parameters based on different embeddings, rather than treating each parameter individually. By defining Φi = j Φ i,j , the total update of the embedding Φ i can be represented as∆Φ i = Φ′ i -Φi = j (Φ ′ i,j -Φ i,j ).Then, the Eq. ( 6) is reformulated toL EFIM Emb-EWC = 1 2 E i=1 Fi ( Φ′ i -Φi ) 2, where Fi = j E D P (∂L P /∂Φ i,j ) 2 . Detailed derivation is given in Appendix A. Intuitively, these three approximations employ different methods to assign importance to parameters. L IM Emb-BWC assigns the same importance to each parameter, L FIM Emb-BWC assigns individual importance to each parameter, and L EFIM Emb-BWC assigns the same importance to parameters within the same embedding vector.For different property prediction tasks, the decisive substructures vary. As shown in Figure 2 , the ester group in the given molecule determines the property SR-HSE, while the carbon-carbon triple bond determines the property SR-MMP. If fine-tuning can be guided by molecular context, encoding context-specific molecular representations allows for dynamic representations of molecules tailored to specific tasks and enables the modeling of the context-specific significance of substructures. Extracting molecular context information. In each episode, we consider the labels of the support molecules on the target property and seen properties, as well as the labels of the query molecules on seen properties, as the context of this episode. We adopt the form of a graph to describe the context. Figure 3 demonstrates the transformation from original context data to a context graph. In the left table, the labels of molecules m q 1 , m 2 2 for property p t are the prediction targets, and the other shaded values are the available context. The right side shows the context graph constructed based on the available context. Specifically, we construct context graph G t = (V t , A t , X t ) for episode E t . It contains M molecule nodes {m} and P property nodes {p}.𝒑 𝒕 𝒑 𝟏 𝒔𝒆𝒆𝒏 𝒑 𝟐 𝒔𝒆𝒆𝒏 𝒑 𝟑 𝒔𝒆𝒆𝒏 𝑚 1 𝑠 1 1 0 ? 𝑚 2 𝑠 1 0 0 1 𝑚 3 𝑠 0 0 ? 0 𝑚 4 𝑠 0 1 1 0 𝑚 1 q ? 0 0 1 𝑚 2 q ? 1 1 0 Context Graph 𝑚 1 𝑠 𝑚 2 𝑠 𝑚 3 𝑠 𝑚 4 𝑠 𝑚 1 𝑞 𝑚 2 𝑞 𝑝 𝑡 𝑝 1 𝑠𝑒𝑒𝑛 𝑝Then we employ a GNN-based context encoder: C = ContextEncoder(V t , A t , X t ), where C ∈ R (M +P )×d2 denotes the learned context representation matrix for E t . V t and A t denote the node set and the adjacent matrix of the context graph, respectively, and X t denotes the initial features of nodes. The features of molecule nodes are initialized with a pre-trained molecular encoder. The property nodes are randomly initialized. When we make the prediction of molecule m's target property p, we take the learned representations of the this molecule c m and of the target property c p as the context vectors. Details about the context encoder are provided in Appendix F.2.In-context tuning with molecular context information. After obtaining the context vectors, we consider enabling the molecular encoder to use the context as a condition, achieving conditional molecular encoding. To achieve this, we further refine our adapter module. While neural conditional encoding has been explored in some domains, such as cross-attention [43] and ControlNet [68] for conditional image generation, these methods often come with a significant increase in the number of parameters. This contradicts our motivation of parameter-efficient tuning for few-shot tasks. In this work, we adopt a simple yet effective method. We directly concatenate the context with the output of the message passing layer, and feed them into the downscaling feed-forward layer in the MP-Adapter. Formally, the downscaling process defined in Eq. ( 3) is reformulated as:EQUATIONwhere ∥ denotes concatenation. Such learned molecular representations are more easily predicted on specific properties, verified in Section 5.5 and Appendix G.Following MAML [10] , a gradient descent strategy is adopted. Firstly, B episodes {E t } B t=1 are randomly sampled. For each episode, in the inner-loop optimization, the loss on the support set is computed as L cls t,S (f θ ) and the parameters θ are updated by gradient descent:EQUATIONEQUATIONwhere α inner is the learning rate. In the outer loop, the classification loss of query set is denoted as L cls t,Q . Together with our Emb-BWC regularizer, the meta-training loss L(f θ ′ ) is computed and we do an outer-loop optimization with learning rate α outer across the mini-batch:EQUATIONEQUATIONwhere λ is the weight of Emb-BWC regularizer. The pseudo-code is provided in Appendix B. We also provide more discussion of tunable parameter size and total model size in Appendix C.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The proposed method and experimental setups are clearly described. Additionally, the link of code and other details for reproducing are provided in Appendix F. The datasets are publicly available. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
evaluation setups: 
 Datasets. We use five common few-shot molecular property prediction datasets from the Molecu-leNet [61] : Tox21, SIDER, MUV, ToxCast, and PCBA. Standard data splits for FSMPP are adopted. Dataset statistics and more details of datasets can be found in Appendix D.Baselines. For a comprehensive comparison, we adopt two types of baselines: (1) methods with molecular encoders trained from scratch, including Siamese Network [25] , ProtoNet [46] , MAML [10] , TPN [35] , EGNN [22] , and IterRefLSTM [1] ; and (2) methods which leverage pretrained molecular encoders, including Pre-GNN [20] , Meta-MGNN [14] , PAR [58] , and GS-Meta [73] . More details about these baselines are in Appendix E.Metrics. Following prior works [1, 58] , ROC-AUC scores are calculated on the query set for each meta-testing task, to evaluate the performance of FSMPP. We run experiments 10 times with different random seeds and report the mean and standard deviations. We compare Pin-Tuning with the baselines and the results are summarized in Table 1 , Table 7 , and Table 8 . Our method significantly outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the effectiveness and superiority of our approach.Across all datasets, our method provides greater improvement in the 10-shot scenario than in the 5-shot scenario. This is attributed to the molecular context constructed based on support molecules. When there are more molecules in the support set, the uncertainty in the context is reduced, providing more effective adaptation guidance for our parameter-efficient tuning.Among benchmark datasets, our method shows significant improvement on the SIDER dataset, increasing by 10.73% in the 10-shot scenario and by 8.81% in the 5-shot scenario. We consider this is related to the relatively balanced ratio of positive to negative samples, as well as the absence of missing labels in the SIDER dataset (Table 5 ). A balanced and low-uncertainty distribution can better benefit addressing the FSMPP task from our method.We also observe that the standard deviations of our method's results under 10 seeds are slightly higher than that of baseline models. However, our worst-case results are still better than the best baseline model. For example, in 10-shot experiments on the Tox21 dataset, the performance of our method is 91.56 ± 2.57. However, our 10 runs yield specific results with the worst-case ROC-AUC reaching 88.02, which is also better than the best baseline model GS-Meta's result of 86.67 ± 0.41. Therefore, a high standard deviation does not mean our method is inferior to baseline models.For MP-Adapter, the main components consist of: (i) bottleneck adapter module (Adapter), (ii) introducing molecular context to adatpers (Context), and (iii) layer normalization (LayerNorm). The results of ablation experiments are summarized in Table 2 . The bottleneck adapter and the modeling of molecular context are the most critical, having the most significant impact on performance. Removing them leads to a noticeable decline, which underscores the importance of parameter-efficient tuning and context perceptiveness in FSMPP tasks. Layer normalization is used to normalize the resulting representations, which is also important for improving the optimization effect and stability. For Emb-BWC, we verify the effectiveness of finetuning the embedding layers and regularizing them with different approximations of L Emb-BWC (Table 3 ). Since the embedding layers have relatively few parameters, direct fine-tuning can also enhance performance. Applying our proposed regularizers to fine-tuning can further improve the effects. Among the three regularizers, the L IM Emb-BWC is the most effective. This indicates that keeping pre-trained parameters to some extent can better utilize pre-trained knowledge, but the parameters worth keeping in fine-tuning and the important parameters in pre-training revealed by Fisher information matrix are not completely consistent. Effect of weight of Emb-BWC regularizer λ. Emb-BWC is applied on the embedding layers to limit the magnitude of parameter updates during fine-tuning. We vary the weight of this regularization λ from {0.01, 0.1, 1, 10}. The first subfigure in Figure 4 shows that the performance is best when λ = 0.1 or 1. When λ is too small, the parameters undergo too large updates on few-shot downstream datasets, leading to over-fitting and ineffectively utilizing the pre-trained knowledge. Too large λ causes the parameters of the embedding layers to be nearly frozen, which prevents effective adaptation. we further analyze the impact of this hyper-parameter on the number of trainable parameters. As shown in Figure 5 , the number of parameters that our method needs to train is significantly less than that required by the full fine-tuning method, such as GS-Meta, while our method also performs better in terms of ROC-AUC performance due to solving over-fitting and context perceptiveness issues. When d = 50, Pin-Tuning performs best on Tox21, and the number of parameters that need to train is only 14.2% of that required by traditional fine-tuning methods. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The link of code is provided in the abstract, and other details for environment and configurations are provided in Appendix F. The datasets are publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this work, we propose a tuning method, Pin-Tuning, to address the ineffective fine-tuning of pre-trained molecular encoders in FSMPP tasks. Through the innovative parameter-efficient tuning and in-context tuning for pre-trained molecular encoders, our approach not only mitigates the issues of parameter-data imbalance but also enhances contextual perceptiveness. The promising results on public datasets underscore the potential of Pin-Tuning to advance this field, offering valuable insights for future research in drug discovery and material science.In the FSMPP setting, the molecular encoder has been pre-trained on the pre-training task P using data D P , and is then fine-tuned on a downstream FSMPP task F using data D F . The overall objective is to find the optimal parameters on task F while preserving the prior knowledge obtained in pretraining on task F. Based on a prior p(Φ ′ ) of the embedding parameters, the posterior after observing the FSMPP task F can be computed with Bayes' rule:EQUATIONwhere D F is assumed to be independent of D P . Taking a logarithm of the posterior, the MAP objective is therefore:EQUATIONThe first term log p(D F |Φ ′ ) is the log likelihood of the data D F given the parameters Φ ′ , which can be expressed as the training loss function on taskF = -log p(D F |Φ ′ ), denoted as L F (Φ ′ ).The second term p(Φ ′ |D P ) is the posterior of the parameters given the pre-training dataset D P .SinceΦ ′ = [Φ ′⊤ 1 , Φ ′⊤ 2 , . . . , Φ ′⊤ E ] ⊤ ,and Φ ′ i is conditionally independent of Φ ′ j for i, j = {1, . . . , E} and i ̸ = j given condition D P , we have p(Φ′ |D P ) = E i=1 p(Φ ′ i |D P ). Thus, log p(Φ ′ |D P ) = E i=1 log p(Φ ′ i |D P )For adapting pre-trained molecular embedding layers to downstream FMSPP tasks, this posterior must encompass the prior knowledge of the pre-trained embedding layers to reflect which parameters are important for pre-training task P. Despite the true posterior being intractable, log p(Φ ′ i |D P ) can be defined as a function f (Φ ′ i ) and approximated around the optimum point f (Φ i ), where f (Φ i ) is the pre-trained values and ∇f (Φ i ) = 0. Performing a second-order Taylor expansion on f (Φ ′ i ) around Φ i gives:EQUATIONwhere H(D P , Φ i ) ∈ R d×d is the Hessian matrix of f (Φ ′ i ) at Φ i . The second term suggests that the posterior of the parameters on the pre-training data can be approximated by a Gaussian distribution with mean Φ ′ i and covariance H(D P , Φ i ) -1 . Following Eq. ( 13), the training objective becomes:EQUATIONWe define the second term as our Emb-BWC regularization objective:EQUATIONA.2 Derivation of L FIMQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to Section 5.1 and Appendix F. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report error bars in our experimental results, such as in Table 1 . 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please refer to Appendix F.1. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our method addresses the few-shot molecular property prediction problem, which is important for drug discovery and material science, as discussed in the introduction and conclusion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our research poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly respect and credit for existing assets. We cite relevant papers and comply with the licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our code is well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Since the Fisher information matrix (FIM) F is the negation of the expectation of the Hessian over the data distribution, i.e., F = -E D P [H], the objective can be reformulated as:EQUATIONwhere F(D P , Φ i ) ∈ R d×d is the corresponding Fisher information matrix of H(D P , Φ i ). Further, the Fisher information matrix can be further simplified with a diagonal approximation. Then, the objective is simplified to:EQUATIONwhere Fi ∈ R d is the diagonal of F(D P , Φ i ). According to the definition of the Fisher information matrix, the j-th value in Fi is computed as E D P (∂L P /∂Φ i,j ) 2 . In this work, this approximated form is defined as L FIM Emb-BWC .We assume that the parameters within an embedding should share the same importance. To this end, we define Φi = j Φ i,j , then the total update of the embedding Φ i can be represented as∆Φ i = Φ′ i -Φi = j (Φ ′ i,j -Φ i,j). Then, the objective in Eq. ( 16) is reformulated to:EQUATIONwhereHi = ∂ 2 L P ∂ Φ2 i = ∂ ∂ Φi ( ∂L P ∂ Φi). Next, we continue to derive Hi . Given that Φi = d j=1 Φ i,j , we first use the chain rule to find ∂L PAccording to chain rule, the derivative of L P with respect to ∂ Φi can be computed as:EQUATION)Since Φi = Φ i,1 + Φ i,2 + . . . + Φ i,d , each of ∂Φi,j ∂ Φi for j = 1, 2, . . . , d equals 1. Therefore, the equation simplifies to:EQUATIONWhen taking the derivative of this with respect to Φ i,j again, using the chain rule, we have:EQUATIONGiven that Φ i,j (Φ i,k ) are all parameters in embedding lookup tables, they are independent of each other. Thus, when j = k, ∂ ∂Φ i,k ( ∂L P ∂Φi,j )∂Φ i,k ∂ Φi = ∂ 2 L P ∂Φ 2 i,j, otherwise it equals 0. We finally getHi = j ∂ 2 L P ∂Φ 2 i,j = j H(D P , Φ i ) j,j .We still approximate the Hessian with the FIM as in Section A.2, and combining this with the definition of FIM, we arrive at the final objective:EQUATIONwhere Fi = j E D P (∂L P /∂Φ i,j ) 2 .To help better understand the training process, we provide the brief pseudo-code of it in Algorithm 1.Algorithm In our Pin-Tuning method, the parameters of the embedding layers are still updated. However, in each message passing layer, the original parameters are completely frozen, and the parts that require updating are the two feed-forward layers and the layer normalization in the bottleneck adapter The difference in the number of parameters updated between the two tuning methods is ∆N = (d 1 -d 2 )L(2d + 1).We provide a comparison of total model size between our Pin-Tuning and the state-of-the-art baseline method, GS-Meta. The total model size consists of both frozen parameters and trainable parameters. The results are presented in Table 4 . The total size of our model is comparable to GS-Meta, but the number of parameters that need to be trained is far less than GS-Meta.We carry out experiments in MoleculeNet benchmark [61] on five widely used few-shot molecular property prediction datasets:• Tox21: This dataset covers qualitative toxicity measurements and was utilized in the 2014 Tox21 Data Challenge. • SIDER: The Side Effect Resource (SIDER) functions as a repository for marketed drugs and adverse drug reactions (ADR), categorized into 27 system organ classes. • MUV: The Maximum Unbiased Validation (MUV) is determined through the application of a refined nearest neighbor analysis, specifically designed for validating virtual screening techniques. • ToxCast: This dataset comprises a compilation of compounds with associated toxicity labels derived from high-throughput screening. • PCBA: PubChem BioAssay (PCBA) represents a database containing the biological activities of small molecules generated through high-throughput screening.Dataset statistics are summarized in Table 5 and Table 6 .We compare our Pin-Tuning with two types of baseline models for few-shot molecular property prediction tasks, categorized according to the training strategy of molecular encoders: trained-fromscratch methods and pre-trained methods. Trained-from-scratch methods:• Siamese [25] : Siamese is used to rank similarity between input molecule pairs with a dual network.• ProtoNet [46] : ProtoNet learns a metric space for few-shot classification, enabling classification by calculating the distances between each query molecule and the prototype representations of each class. • MAML [10] : MAML adapts the meta-learned parameters to achieve good generalization performance on new tasks with a small amount of training data and gradient steps. • TPN [35] : TPN classifies the entire test set at once by learning to propagate labels from labeled instances to unlabeled test instances using a graph construction module that exploits the manifold structure in the data. • EGNN [22] : EGNN predicts edge labels on a graph constructed from input samples to explicitly capture intra-cluster similarity and inter-cluster dissimilarity. • IterRefLSTM [1] : IterRefLSTM adapts Matching Networks [53] to handle molecular property prediction tasks.Pre-trained methods:• Pre-GNN [20] : Pre-GNN is a classic pre-trained molecular model, taking the GIN as backbone and pre-training it with different self-supervised tasks. • Meta-MGNN [14] : Meta-MGNN leverages Pre-GNN for learning molecular representations and incorporates meta-learning and self-supervised learning techniques. • PAR [58] : PAR uses class prototypes to update input representations and designs label propagation for similar inputs in the relational graph, thus enabling the transformation of generic molecular embeddings into property-aware spaces. • GS-Meta [73] : GS-Meta constructs a Molecule-Property relation graph (MPG) and redefines episodes in meta-learning as subgraphs of the MPG.Following prior work [58] , for the methods we reproduced, we use GIN as the graph-based molecular encoder to extract molecular representations. Specifically, we use the GIN provided by Pre-GNN [20] which consists of 5 GIN layers with 300-dimensional hidden units. Pre-GNN, Meta-MGNN, PAR, and GS-Meta further use the pre-trained GIN which is also provided by Pre-GNN.F Implementation details For featurization of molecules, we use atomic number and chirality tag as original atom features, as well as bond type and bond direction as bond features, which is in line with most molecular 2), we use the ReLU activation with d 1 = 600. Pre-trained GIN model provided by Pre-GNN [20] is adopted as the PTME in our framework. We tune the weight of update constraint (i.e., λ) in {0.01, 0.1, 1, 10}, tune the learning rate of inner loop (i.e., α inner ) in {1e-3, 5e-3, 1e-2,5e-2,1e-1, 5e-1, 1, 5}, and tune the learning rate of outer loop (i.e., α outer ) in {1e-5, 1e-4, 1e-3,1e-2,1e-1}. Based on the results of hyperparameter tuning, we adopt α inner = 0.5, α outer = 1e -3 and d 2 = 50. The ContextEncoder(•) described in Section 4.2 is implemented using a 2-layer message passing neural network [11] . In each MPNN layer, we employ a linear layer to aggregate messages from the neighborhoods of nodes and utilize distinct edge features to differentiate between various edge types in the context graphs.For baselines, we follow their recommended settings.More discussion of Figure 1 . The results show that molecular encoders with more molecule-specific inductive biases, such as CMPNN [47] and Graphormer [66] , performed slightly worse than GIN-Mol [20] on this few-shot task. This is because more complex encoders require more parameters to provide inductive biases, which are difficult to train effectively under a few-shot setting.More main results. The detailed comparison between Pin-Tuning and baseline models on subdatasets of ToxCast are summarized in Table 7 and Table 8 . Our method outperforms all baseline models under both the 10-shot and 5-shot settings, demonstrating the superiority of our method compared to existing methods.More discussion of ablation study. Different datasets show varying sensitivity to the removal of components. On small-scale datasets like Tox21 and SIDER, removing components leads to a significant performance drop. On large-scale datasets like ToxCast and PCBA, the impact of removing components is less pronounced. This is because more episodes can be constructed on large-scale datasets, which aids in adaptation. This observation indicates that Pin-Tuning can bring considerable benefits in situations where data is extremely scarce.More case studies. We provide more case studies in Figure 8 and 9 as a supplement to Section 5.5. As we discusses in Section 5.2, although our method significantly outperforms the state-of-the-art baseline method, our method exhibits higher standard deviations in the experimental results under multiple runs with different seeds.We further speculate that these high standard deviations might be due to the uncertainty in the context information within episodes. The explicitly introduced molecular context, on one hand, provides effective guidance for tuning pre-trained molecular encoders, but on the other hand, this information also carries a high degree of uncertainty. We aim to model the target property through the molecule-property relationships within episodes, but each episode is obtained by sampling very few samples from the large space corresponding to the target property. The uncertainty between different episodes is relatively high. How to quantify and calibrate this uncertainty is another question worth exploring, which we will investigate in our future work.NeurIPS Paper Checklist• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces Pin-Tuning, a parameter-efficient fine-tuning approach specifically designed for GNN-based few-shot molecular property prediction tasks. This approach features a novel adapter layer (MP-Adapter) and a Bayesian weight consolidation scheme (Emb-BWC), both of which are combined to demonstrate superior performance across benchmark datasets. The authors' thorough rebuttal, which included additional experiments and clarifications, addressed the reviewers' concerns, highlighting the method's relevance and potential impact. This work offers a novel approach that will positively contribute to few-shot molecular property prediction literature and will be of significant interest to the NeurIPS audience.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel Pin-Tuning method. Focusing on improving the fine-tuning process of pre-trained molecular encoders, especially for the task of Few-Shot Molecular Property Prediction (FSMPP), Pin-Tuning skillfully balances the contradiction between the number of tunable parameters and the limited labeled molecular data, while reinforcing the encoder's context-awareness. The core of the approach is the introduction of MP-Adapter, a lightweight adapter for pre-trained message-passing layers, and Emb-BWC, a Bayesian weight consolidation scheme for pre-trained atom/key embedding layers. Experimental results show that Pin-Tuning exhibits excellent performance on public datasets, significantly improves the prediction performance in few-shot scenarios with fewer trainable parameters, and proves its effectiveness in the field of molecular attribute prediction.
strength_weakness: strength_weakness: 1.The concept of Adapter has been widely studied and applied in the field of Natural Language Processing. In this paper, does MP-Adapter just take this concept over, or is there any design and optimization specific to molecular graphical neural networks (GNNs)? Is there any comparison with existing generalized frameworks for adapter networks?
2.The observation from the paper that the use of Identity matrix approximation works best in Emb-BWC seems to be a counterintuitive finding. This is because the Identity matrix approximation ignores possible correlations between parameters and assigns the same importance to each parameter. Is it true that correlations between parameters are less important in the molecular property prediction task than in other tasks? If so, does this reflect an intrinsic characteristic of molecular property prediction tasks?
questions: questions: 1.What does the a mean in Effect of weight of Emb-BWC regularizer λ in Sensitivity analysis? Is there a confusing typo here?
2.What are the advantages of the proposed MP-Adapter over existing adapter technology?
3.How to ensure the quality and reliability of the contextual information used in the paper?
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose a strategy for few-shot drug discovery scenarios. The authors propose to fine-tune representations retrieved from encoder layers with adapters. In addition to the initial representations from the message passing layers, the adapters are provided with property representations and "context-aware" molecule representations to improve the initial representations. Both the property representations and the "context-aware" molecule representations stem from a context encoder. The context encoder creates learned molecule and property representations by relating the input molecule representations and the current few shot task, i.e. the property with other molecules and other already seen properties. The authors evaluate their approach in several datasets and include an ablation study.
strength_weakness: strength_weakness: **Originality**:
- **(W-O1)**: Using adapters to efficiently fine-tune models is not novel. The core idea is to efficiently fine-tune a layer by adding another learnable component with less parameters. Instead of changing the model layer itself, the new component learns to modify the outcome of the model layer. This has already been done in few-shot drug discovery [2]. 
- **(W-O2)**: In-context fine-tuning in the sense of adapting molecular representations to the few-shot task is not novel and has been done e.g. here [1, 3] .

**Quality**:
- **(W-Q1)**: The authors missed to include the SOTA benchmark for few-shot drug discovery FS-Mol.
- **(W-Q2)**: The authors missed that in-context fine-tuning (W-O2) and efficient adapter-like fine-tuning (W-O1) has already been applied to few-shot drug discovery.
 - **(W-Q3)**: Missing error bars for Figure 1, Table 2, Figure 4, and Figure 5

**Significance**:
- **(W-S1)**: The significance of the experimental section is diminished because the SOTA benchmark for few-shot drug discovery is missing (see (W-Q1))
- **(W-S2)**: The significance of the proposed method is diminished because of (W-Q2). Being aware of the ideas which have already been applied to few-shot drug discovery would have allowed to put the context encoder into the focus of this work. The ablation study indicates that this module is important. Still in the current version of the paper, this novel piece of architecture remains underexplored.

**Clarity**:
- **(W-C1)**: Because of (W-S2, W-Q2) it remains unclear which parts of the proposed method have been tested in related work and which parts are novel. The authors might think about changing the main story line of their paper by focusing on their training label aware-fine-tuning procedure.
- **(W-C2)**: The procedure which happens inside the context encoder is not displayed well in Figure 2. Since this is a central part, this should be improved.
- **(W-C3)**: English editing is required
- **(W-C4)**: The mathematical notation and formulas seem cluttered and inconsistent. E.g., in 3.2. a molecular encoder is denoted by $f(.)$ but the context encoder is denoted by ContextEncoder. Generally, the authors should stick to standard notation and avoid "full-word notation" in formulas.


[2] Adler, Thomas, et al. "Cross-domain few-shot learning by representation fusion." (2020).\
[3] Altae-Tran, Han, et al. "Low data drug discovery with one-shot learning." ACS central science 3.4 (2017): 283-293.
questions: questions: -
limitations: limitations: Limitations have been addressed in the appendix.
paper_summary: paper_summary: This paper proposes Pin-Tuning, a parameter-efficient in-context tuning method for few-shot molecular property prediction (FSMPP), mitigating the parameter-data imbalance and enhancing the contextual perceptiveness of pre-trained molecular encoders. This method treats the embedding layers and message passing layers in the pre-trained molecular encoder separately, based on specific reasons why they could not be effectively fine-tuned before. This includes a series of Bayesian weight consolidation constraints for the embedding layers and the bottleneck adapters for the message passing layers. Additionally, molecular context is introduced into the adapter modules to enable contextual perceptiveness.  Experiments show that the proposed method effectively improves few-shot performances across a range of benchmark datasets.
strength_weakness: strength_weakness: 1. It is recommended to add details about the pilot experiment in Figure 1, such as the featurization and the specific architectures of different models, and whether the pre-training strategies are consistent.
2. Table 4 only reports the "total size of the molecular encoder." Considering that some parameters are frozen in your method, you could add a row for the "size of the part that needs to be tuned". This would be helpful to make the advantages of your method clearer.
questions: questions: 1. What are the details of the pilot experiment in Figure 1? Including the featurization, the model architecture, and the pre-training strategy of each model. More details are needed to assess the fairness of the experiment and to strengthen the persuasiveness of your claims.
2. Does $\mathcal{L}_{update}$ in Figure 2(d) refer to $\mathcal{L}_{Emb-BWC}$ in Section 4.1.2? The content in the figure should be consistent with the text.
limitations: limitations: Yes.
paper_summary: paper_summary: This work introduce a new system for the FSMPP tasks, which includes: context adapter in GNN, context graph, and  weight consolidation. With these three techniques, the Pin-Tuning method achieved promising results on many FSMPP benchmarks.
strength_weakness: strength_weakness: - The adapter based parameter efficient tunning have already been used in many tasks.
questions: questions: - Why different strategies are used for embedding layers and encoder layers? What will be the results for (a) finetune encoder layers with weight constraint, and (b) freeze embedding layers and add adapters to them? 

-  According to Table 3, the simplest weight consolidation method (i.e., IM) is the best. In this case, does the BWC theory really required as it is just L2 penalty on the changed parameters?
limitations: limitations: yes

Comments: title: Thank you & Looking forward to your reply!
Thank you!
Thank you!
Thank you & Looking forward to your reply!
Reviewer's response to 2nd rebuttal
Thank you!
Response to the reviewer's comments (2/2)
Response to the reviewer's comments (1/2)
Answer to the authors' rebuttal
Thank you & Looking forward to your reply
Thank you & Looking forward to your reply
Thank you & Looking forward to your reply
Thank you & Looking forward to your reply
Rebuttal by Authors
Rebuttal by Authors
comment: Dear reviewer nYSV,

Thank you so much for your comments. As the discussion period is drawing to a close, we eagerly anticipate your response, which holds great significance for us. Could we kindly know if our rebuttal has addressed your concerns?

Best regards,

Authors
Thank you very much for your response! We are very glad that we have addressed your concerns. If you have any further questions, feel free to ask and we would be more than delighted to answer.

Best regards,

Authors
Thank you so much for your response! We are very pleased that we have addressed your concerns. Your constructive and insightful comments have greatly helped us improve the quality of the paper. Once again, we sincerely thank you for your comments and participation in the discussion!

Best regards,

Authors
Dear reviewer nYSV,

Thank you so much for your comments!

We have provided a detailed clarification on the novelty of our MP-Adapter and conducted additional experiments on the tuning strategies for embedding layers and message passing layers. As the discussion period is drawing to a close with only a few hours remaining, we eagerly anticipate your response, which holds great significance for us.

Best regards,

Authors
Thanks for your efforts in the rebuttal. The rebuttal addressed my concerns. The additional details regarding the pilot experiment demonstrate that the comparison is fair and reasonable, making the motivation more convincing. Moreover, the revised table and figure improve the clarity of the paper. I have also read the comments from other reviewers and all discussions, and I think this paper presents a promising approach for the community. Therefore, I have decided to raise my score and lean towards acceptance.
> What we intended to convey is that for many molecular encoders with GNN backbones, our MP-Adapter suits their message passing mechanisms

Understood!

> we have added experiments to fine-tune the GNN-based encoder using CHEF. [...] Their performance lags behind that of our MP-Adapter because they do not integrate molecular context 

This improves the quality of the manuscript. The reasoning w.r.t. the molecular context makes sense.

> Context modeling in few-shot molecular property prediction

Thank you for showing this passage. This resolves my concerns about the context discussion.

> FS-Mol experiment:

- The performance values seem reasonable now.
- The authors are able to show that their approach helps to boost PAR
- I still think there are weaknesses in the FS-Mol experiment:
    * PAR under-performs - compare performance reported in [6] with more suitable hyperparameters (differences are not that big though)
    * All presented variants are outperformed by the Frequent Hitter model [6] which is a baseline which is not aware of any support set samples and simply learns average activity across tasks. Another backbone model - perhaps a GIN-encoder based ProtoNet or Neural Similarity Search version - might have been the better choice.

Overall, I'd evaluate this paper to be a borderline paper now. Since I think the authors' manuscript has improved a lot and since the presented way of guiding fine-tuning by including molecular context is interesting, I'd adapt my score and slightly vote for acceptance.
Thank you very much for your response. If you have any further questions, feel free to ask and we would be more than delighted to answer.

Best regards,

Authors
### Context discussion:

> An extensive discussion about similarities and differences in how different approaches use context.

Thank you for recognizing our work. We understand your concerns regarding the discussion on context modeling. It is indeed necessary to further expand the manuscript's discussion on this topic, providing a detailed analysis of the similarities and differences among different approaches, and clarifying the unique aspects and contributions of our approach. We will add the following independent paragraph to the related work section:

"**Context modeling in few-shot molecular property prediction.** Recent efforts have shifted towards leveraging the unique nature of molecular property prediction, specifically the many-to-many relationships between molecules and properties that arise from the multi-labeled nature of molecules, often referred to as the *molecular context*. IterRefLSTM considers the structures and property labels of seen molecules with respect to the target property during prediction. PAR initially connects similar molecules for the target property using a homogeneous context graph. MHNfs introduces a large-scale external molecular library as context to augment the limited known information. However, the contexts constructed in these approaches are not informative enough, as they neglect the interactive relationship between molecules and properties. Unlike these methods, the molecular context in our approach includes the label information of both seen and unseen molecules in auxiliary properties. We represent the molecular context as a molecule-property bipartite graph and encode it using a GNN-based context encoder, which allows for effective and robust learning of the relationships between molecules and properties. Furthermore, the molecular context is introduced into the fine-tuning process through our MP-Adapter, enabling the molecular encoder to be fine-tuned under the guidance of the context, thereby obtaining contextual molecular representations."

We hope this addition will enhance the quality of the manuscript and clarify its value to the community.

### FS-Mol experiment:

> An experiment with support set size of 16 is not a 16-shot experiment.

Thank you for pointing out this discrepancy regarding the support set size. We apologize for the confusion caused by our mention of "16-shot" in our rebuttal. This was indeed a typo. In our experiments, we actually compared different support set sizes based on stratified sampling. Specifically, we followed the settings of MHNfs [6] and FS-Mol [7], where we conducted meta-training with a support set size of 64 and stratified sampling. For evaluation, we used support set sizes of 16, 32, 64, 128, and 256, also based on stratified sampling.

We have corrected this typo and fixed the previous mistakes in our experiments. Below, we provide the revised table with the correct experimental results.

> The reported $\Delta$AUPRC values are very high.

Thank you for pointing out this issue. Upon re-examining our previous experimental results, we realized that we reported AUPRC instead of $\Delta$AUPRC. Considering that the AUPRC of a random classifier on FS-Mol is around 0.46 (since FS-Mol is not completely class-balanced), the actual $\Delta$AUPRC of our previous results is approximately 0.03 to 0.05.

Given that these are very poor results, we carefully reviewed our experimental process. We found that the reason for the poor results was that we mistakenly used a model checkpoint saved early in the training process for evaluation. The complete training process consists of 10,000 epochs, but the evaluated model was saved around the 200th epoch. We have corrected this mistake and evaluated the fully trained model.

Using the correct model checkpoint and the correct $\Delta$AUPRC metric, we have re-evaluated our experimental results on the FS-Mol benchmark. Below, we provide the revised table with the correct experimental results:

|Support Set Size|16|32|64|128|256|
|-|-|-|-|-|-|
|PAR|0.1578±0.0336|0.1669±0.0261|0.1723±0.0305|0.1917±0.0641|0.1561±0.0266|
|PAR + MP-Adpater|**0.1728±0.0375**|0.1815±0.0253|0.1833±0.0289|0.1984±0.0638|0.1894±0.0266|
|PAR + Emb-BWC|0.1620±0.0291|0.1691±0.0228|0.1743±0.0297|0.1905±0.0644|0.1699±0.0239|
|PAR + MP-Adapter + Emb-BWC|0.1721±0.0292|**0.1827±0.0216**|**0.1893±0.0281**|**0.2049±0.0603**|**0.2041±0.0283**|

Combining the results in this table with the results from another setting in our rebuttal, our parameter-efficient tuning method demonstrates its effectiveness on the FS-Mol benchmark, regardless of whether molecular context is considered.

---

Thank you once again for your constructive comments and for helping us improve the quality and clarity of our work. We sincerely hope that our response can address your concerns.

**References**

[6] Context-Enriched Molecule Representations Improve Few-Shot Drug Discovery. ICLR 2023

[7] FS-Mol: A Few-Shot Learning Dataset of Molecules. NeurIPS 2021
Dear review W6tN,

We sincerely appreciate your thoughtful response to our rebuttal and the valuable feedback you have provided. Your insights are incredibly helpful in enhancing the quality of our paper. We hope to address each of your remaining concerns point by point below.
### Adapter / CHEF discussion:

> Both GNNs and MLPs can be considered SOTA.

Thank you very much for your insightful comments. We completely agree with your perspective. Both GNNs and MLPs are highly effective for encoding molecular data. They each have significant advantages when dealing with molecular data characterized by topological graphs and molecular fingerprints, respectively, and can both be considered SOTA molecular encoders.

Additionally, the D-MPNN [1] you mentioned is indeed a very representative molecular encoder. In our paper, we discuss GIN-Mol [2] and CMPNN [3], which are simplified and enhanced versions of D-MPNN, respectively. The difference between them is that D-MPNN designs the edge-based message passing mechanism on the basis of GIN-Mol to prevent totters, while CMPNN strengthens the interaction between atoms and bonds based on D-MPNN. The commonality among these three models is that they all use the GNN backbone based on the Graph Isomorphism Network (GIN) [4], which is known for its high expressive power (generalizes the WL test). This high expressive power comes from the aggregation scheme based on MLPs, which is considered injective on multisets (see Theorem 3 and Corollary 6 in [4]). Therefore, we fully agree with your point that MLPs are SOTA models on molecular data, and incorporating MLPs as the aggregate function in GNNs endows GNNs with stronger expressive power.

We also apologize for the inappropriate expression in our initial rebuttal. What we intended to convey is that for many molecular encoders with GNN backbones, our MP-Adapter suits their message passing mechanisms by feeding the output of the adapters into the next encoding layer.

> CHEF suits well for GNNs and can be used to fine-tune GNN-based molecular encoders.

We greatly appreciate your insightful perspective that CHEF [5] is capable of aggregating low-level features from the shallow layers and higher-level features from the deeper layers of GNNs. Considering the suitability of CHEF for fine-tuning GNNs and its distinction from adapters, we have added experiments to fine-tune the GNN-based encoder using CHEF.

|                                   | Tox21   |        | SIDER   |        | MUV     |        | PCBA    |        |
| --------------------------------- | ------- | ------ | ------- | ------ | ------- | ------ | ------- | ------ |
|                                   | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot | 10-shot | 5-shot |
| No Adapter                        | 86.67   | 86.43  | 84.36   | 84.57  | 66.08   | 64.50  | 79.40   | 77.47  |
| **CHEF**                          | 87.24   | 87.18  | 84.81   | 84.52  | 69.30   | 67.65  | 79.69   | 77.20  |
| NLP Adapter                       | 87.94   | 87.80  | 85.18   | 84.79  | 71.86   | 69.58  | 79.75   | 78.35  |
| MP-Adapter                        | 90.17   | 89.59  | 92.06   | 91.43  | 72.37   | 71.65  | 80.74   | 78.51  |
| Pin-Tuning (MP-Adapter + Emb-BWC) | 91.56   | 90.95  | 93.41   | 92.02  | 73.33   | 70.71  | 81.26   | 79.23  |

On most datasets, CHEF performs comparably to standard NLP adapters, as both use lightweight trainable components to fine-tune the output of frozen pre-trained molecular encoder. Their performance lags behind that of our MP-Adapter because they do not integrate molecular context with the message passing process. We will include these experimental results and related discussions about CHEF in the paper.

**References**

[1] Analyzing Learned Molecular Representations for Property Prediction. Journal of Chemical Information and Modeling. 2019

[2] Strategies for Pre-training Graph Neural Networks. ICLR. 2020

[3] Communicative Representation Learning on Attributed Molecular Graphs. IJCAI. 2020

[4] How powerful are graph neural networks? ICLR. 2019

[5] Cross-Domain Few-Shot Learning by Representation Fusion. 2020
Thank you for your reply. Combined with the other comments and rebuttals, I maintain my initial rating.
### Adapter / CHEF discussion:
- GNNs / MLPs:
> However, the current state-of-the-art molecular encoders are based on GNNs and take molecular graphs as input.

    This is a minor point but still worth mentioning: I'd disagree here if it was meant in the way that GNNs are SOTA and MLPs are not. As far as I know, generally and merged across multiple bioactivity datasets, both can be considered SOTA. Also see [1] which combine GNNs with MLPs to reach SOTA performance.

- CHEF for GNNs:
> This approach is only suitable for global encoding models, such as the FCNs used in the experiments evaluating CHEF, which take molecular fingerprints (ECFP6) as input

    I do think CHEF suits well for GNNs. The core idea of CHEF is to use both low-level and higher-level features for domain adaptation. This fits very well with GNNs, as representations retrieved from very early layers can be interpreted as very local features (i.e. a low-level feature), while representations from later layers capter larger parts of the molecular graph (i.e. a higher-level feature).

- This is why CHEF can be used to fine-tune GNN-based molecular encoders
> neither NLP adapters nor the representation fuser CHEF can be trivially used to fine-tune GNN-based molecular encoders. 
- Novelty:
The way label information is included is novel. Also, I appreciate the  authors' reasoning that the model might benefit from feeding the adapter's output into the next layer. However, the authors missed to include experiments which show that their claim really holds.

### Context discussion:
I appreciate the authors' discussion here and I mostly agree. (The MHNfs' representations could be considered as contextual molecular representations also because the enrichment step is analogous to a LLM which uses information from a large context-window to update token representations). The authors' main contribution is their context-guided fine-tuning, while context means both structural and label information. This is valuable to the community. However, both the quality of the manuscript as well as the value for the community heavily depend on an extensive discussion about similarities and differences in how different approaches use context. Reading answer 3/3, I am not convinced the authors addressed this enough in their current version of the manuscript.

### FS-Mol experiment:
- An experiment with support set size of 16 is not a 16-shot experiment. A support set size of 16 means that the total number of available labeled samples for tasks during inference is 16, e.g. 6 actives and 10 inactives. For the FS-Mol benchmark experiment  the support set is created using a stratified split, compare [2]. 
- The reported $\Delta$AUPRC values are very high. Recently published SOTA methods are typically reported with performance values between 0.2 and 0.3 (*) for the support set sizes 16 and 32. Considering that a random classifier achieves AUCPR values around 0.5, the reported AUCPR values indicate close-to-perfect classifiers. Given this significant performance gap, and considering that PAR has already been evaluated on FS-Mol with $\sim\Delta$AUPRC 0.17 [3], an explanation of why the evaluated models performed so well in the included experiment would be valuable.

### Summary:
I appreciate the work the authors put into the rebuttal, I see the potential of this work, and I generally would evaluate this work to be interesting for the community. However, I also think this manuscript would benefit from another review round (see adapter discussion, context discussion, and FS-Mol discussion) since some improvements still seem crucial to eventually end with a manuscript of high quality. For this reason, I believe it is too early to publish this work at this conference, and I therefore stick to my initial rating. 

[1] Yang, Kevin, et al. "Analyzing learned molecular representations for property prediction." Journal of chemical information and modeling 59.8 (2019): 3370-3388.

[2] Stanley, Megan, et al. "Fs-mol: A few-shot learning dataset of molecules." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

[3] Context-Enriched Molecule Representations Improve Few-Shot Drug Discovery. ICLR 2023
Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors
Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors
Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors
Thank you very much for your precious time and valuable comments. We hope our responses have addressed your concerns. Please let us know if you have any further questions. We are happy to discuss them further. Thank you.

Best regards,

Authors
# Responses to Reviewer W6tN (3/3)

> `W-S2` `W-C1` The contribution of parameter-efficient tuning and context encoding.

In the above response, we have clarified the novelty of our parameter-efficient tuning and in-context tuning. Here, we further clarify their contributions to performance improvement. **Both parameter-efficient tuning and context encoding can contribute to performance improvement. Specifically, on the MUV dataset, fine-tuning the message passing layers with an adapter plays a decisive role, having a greater impact than context**, as observed from the experimental results in our `Global Response`. Therefore, the influence of parameter-efficient tuning and context encoding on performance improvement depends on the data distribution, and each may significantly contribute on different datasets.

> `W-C2` `W-C3` `W-C4` Suggestions for improving the quality and clarity of the presentation.

Thank you for your suggestions to improve the presentation of the paper. We have made the following efforts:

1. We have added a description of the context encoder in Figure 2. The revised Figure 2 is provided in the Global Response PDF.
2. We have performed language editing and proofreading to enhance the clarity and readability of the paper.
3. We have standardized the use of mathematical notation and formulas. We replaced notation like "$ContextEncoder(\cdot)$" with letter-based notation to make the usage more standard and consistent.

**References**

[1] Cross-Domain Few-Shot Learning by Representation Fusion. 2020

[2] Low Data Drug Discovery with One-Shot Learning. ACS Central Science 2017

[3] Property-Aware Relation Networks for Few-Shot Molecular Property Prediction. NeurIPS 2021

[4] Context-Enriched Molecule Representations Improve Few-Shot Drug Discovery. ICLR 2023
# Responses to Reviewer W6tN (2/3)

> `W-Q1` `W-S1` Experiments on the FS-Mol benchmark.

Thank you for your constructive suggestions. We have added experimental results on the FS-Mol benchmark. Note that the data distribution of FS-Mol is significantly different from that of another import benchmark MoleculeNet:

1. FS-Mol only includes properties related to the biological activity of small molecules against protein targets, unlike MoleculeNet, which encompasses a variety of property types such as toxicity, solubility, blood-brain barrier penetration, and side effects.
2. FS-Mol contains over 5,000 tasks with 233,786 unique compounds, and 489,133 measurements have been conducted. Although FS-Mol covers a wide range of molecules and properties, the **number of molecules measured for each property is relatively small, and there is little overlap between the molecules covered by different properties**. Therefore, **it is difficult to provide effective context between different properties on the original FS-Mol dataset**.

To support our claim regarding the sparse context in FS-Mol, we conducted a statistical analysis of the distribution in the original FS-Mol dataset:

- The number of properties measured for each molecule

| Max  | Min  | Mean | 25th percentile | 50th percentile | 75th percentile | 90th percentile | 95th percentile |
| ---- | ---- | ---- | - | - | - | - | - |
| 602  | 1    | 1.94 | 1   | 1   | 2   | 3   | 4   |

- The number of molecules commonly measured between two properties

| Max  | Min  | Mean | 25th percentile | 50th percentile | 75th percentile | 90th percentile | 99th percentile |
| -- | ---- | ---- | - | - | - | - | - |
| 2230 | 0    | 0.79 | 0  | 0  | 0 | 0 | 7 |

Since the original FS-Mol is not suitable for encoding and perceiving molecule context based on labels, **we designed two evaluation settings**.

`Evaluation Setting 1`: **We evaluated our parameter-efficient tuning method under the standard FS-Mol evaluation setting, without considering molecular context.** Based on PAR [3], which is the state-of-the-art baseline model irrelevant to the context from auxiliary tasks, we tested the results of introducing MP-Adapter and Emb-BWC. We tested different support set sizes of 16, 32, and 64. We run 10 times with different seeds and report the average ΔAUPRC.

|   | FS-Mol (16-shot) | FS-Mol (32-shot) | FS-Mol (64-shot) |
| - | - | - | - |
| PAR | 47.94±0.23 | 48.18±0.21 | 48.73±0.37 |
| PAR + MP-Adpater| 49.33±0.19 | 49.43±0.26 | 49.80±0.26 |
| PAR + Emb-BWC | 48.14±0.24 | 49.80±0.33| 51.01±0.48 |
| PAR + MP-Adapter + Emb-BWC | 49.76±0.16 | 49.96±0.13 | 50.14±0.27 |

`Evaluation Setting 2`: **To better evaluate the effectiveness of our context encoding and in-context tuning on the FS-Mol dataset, we selected two subsets of data from FS-Mol with dense context information to construct two sub-datasets and evaluated them using the standard N-way K-Shot setting.** The FS-Mol-6K dataset contains 15 properties and over 6,000 molecules, while the FS-Mol-24K dataset contains 128 properties and over 24,000 molecules. Experiments were conducted under 10-shot and 5-shot settings with 10 different seeds, using the average AUC-ROC score as the evaluation metric. These two constructed sub-datasets have been uploaded to our anonymous code repository.

| | FS-Mol-6k (10-shot) | FS-Mol-6k (5-shot) | FS-Mol-24K (10-shot) | FS-Mol-24K (5-shot) |
| - | - | - | - | - |
| PAR  | 78.52±0.33  | 78.19±0.23| 63.43±0.41  | 62.25±0.28 |
| GS-Meta  | 80.36±0.73  | 79.58±0.80 | 67.28±0.66 | 66.86±0.32 |
| Pin-Tuning (Ours) | 82.28±1.99 | 81.77±1.71 | 68.94±0.69| 68.02±0.93 |

From the evaluation results of the two settings, **our parameter-efficient tuning method outperforms the baseline methods on the FS-Mol benchmark, regardless of whether molecular context is considered.**