LICO: Large Language Models for In-Context Molecular Optimization
Authors: Tung Nguyen
Aditya Grover
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
molecular optimization
black-box optimization
foundation models
in-context learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce LICO, a general-purpose model that extends arbitrary base LLMs for black-box optimization, with a particular application to the molecular domain.

Abstract: Optimizing black-box functions is a fundamental problem in science and engineering. To solve this problem, many approaches learn a surrogate function that estimates the underlying objective from limited historical evaluations. Large Language Models (LLMs), with their strong pattern-matching capabilities via pretraining on vast amounts of data, stand out as a potential candidate for surrogate modeling. However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specific data in the pretraining corpora and the challenges of articulating complex problems in natural language. In this work, we introduce LICO, a general-purpose model that extends arbitrary base LLMs for black-box optimization, with a particular application to the molecular domain. To achieve this, we equip the language model with a separate embedding layer and prediction layer, and train the model to perform in-context predictions on a diverse set of functions defined over the domain. Once trained, LICO can generalize to unseen molecule properties simply via in-context prompting. LICO performs competitively on PMO, a challenging molecular optimization benchmark comprising 23 objective functions, and achieves state-of-the-art performance on its low-budget version PMO-1K.

Published as a conference paper at ICLR 2025
LICO: LARGE LANGUAGE MODELS FOR IN-CONTEXT MOLECULAR OPTIMIZATION
Tung Nguyen & Aditya Grover Department of Computer Science University of California, Los Angeles (tungnd, adityagi @cs. acla.edu:
ABSTRACT
Optimizing black-box functions a fundamental problem in science and engineer- ing. To solve this problem, many approaches learn a surrogate function that esti- mates the underlying objective from limited historical evaluations. Large Language Models (LLMs), with their strong pattern-matching capabilities via pretraining on vast amounts of data, stand out as a potential candidate for surrogate modeling. However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specinee data in the pretraining corpora and the challenges of articulating complex problems in natural language. In this work, we introduce LICO, a general-purper purposs model that extends arbitrary base LLMs for black-box optimization, with particular application to the molecular domain. To achieve this, we equip the language model with separate embedding layer and prediction layer, and train the model to perform in-context predictions on a diverse set of functions defined over the domain. Once trained, LICO can generalize to unseen molecule properties simply via in-context prompting. LICO performs competitively on PMO, a challenging molecular optimization benchmark comprising 23 objective functions, and achieves state-of-the-an performance on its low-budget version PMO-IK.
INTRODUCTION
Black-box optimization (BBO) is the problem of optimizing an unknown, often complex objective function without direct access to its structure or derívatives. This problem is ubiquitous in many science and engineering fields, including material discovery (Hamidieh) (018)), protein engineer- ing (Brookes al.) (Brkisyaa al. [216]] Angermuelll et 20200, molecular design (Gaulton) al. 2012)), mechanical design (Berkenkamp et 2016 Liao al. (201)), and neural architecture search (Zoph & Le 2016). Typically, BBO involves an iterative process where each step constructs surrogate model to approximate the objective function. This model then guides the selection of promising candidates for subsequent evaluation. The main challenge of this approach lies in learning an effective surrogate function that can accurately estimate the objective from limited historical data.
In stark contrast, we have seen impressive generalization abilities of Large Language Models (LLMs) (Brown al., 2020 Achiam et al., 2023 Bubeck et al.) 2023 Team al.
2023 Touvron
ct al. 2023a bl Jiang et al. 2023 2024) for language-driven reasoning over many kinds of domains.
By pretraining on Internet-scale data, LLMs have demonstrated exceptional mattern-matchin
abilities
and generalization from limited observations in both natural language (Brown et al. 2020) Kojima
al.
2022
Wei et al.| (022) and other domains (Lu al.) [022;] Mirchandani et
2023 Gruver
et al.
2023). This positions LLMs as promising solution for enhancing surrogate modeling for
BBO. Some recent works have indeed shown great potential for using L.LMs for solving optimization
problems (Yang et al., [2233] Chen et al.] 2023) Zhang et al.) 2023] Liu et al.). The main idea behind these methods is to frame the optimization problem in natural language, and prompt the language model using previously collected observations to make predictions for new data points (Liu et al.) or to propose better candidates (Yang et al. 2023 Chen et al. 2023: Zhang et al., [023 Ma et al. [0033] Nie et al. 2023] Meyerson et al. 2023 Lehman et al. 2023 Bradley et al. [024 Liu al. (202aa). However, this approach has several limitations. First, performing optimization in the text space requires the problem and solution to be expressed in natural language, thus limiting this approach

---

Published as a conference paper at ICLR 2025
to selected domains. Second, the scarcity of Nomaiin-rllevann data in the text corpora used to train language models poses generalization challenges when using these models for general scientific domains such as molecular optimization. Therefore, existing works have only demonstrated the success of LLMS in neural architecture search (Liu et al., Chen et al.,
2023
Zhang et al. 2023).
prompt optimization Yang et al. 2023), and code generation (Ma et al. 2023 Lehman et al. 2023
corresponding to domains that are well-represented in the training dataset for common language
models (Brown et al., 2020; Touvron al. 2023a, Jiang et al.] 2023). Third, relying on verbose textual descriptions for the problem and its solution imposes practical constraints by inflating the
context length and reducing the number of historical observations the model can effectively utilize.
In this work, we propose Large Language Models for In- Context Optimization (LICO), general- purpose model that leverages L.LMs for black-box optimization, with a particular application to the molecular domain. To generalize a language model to new scientific domain unseen during pretraining, we equip the model with two embedding layers for embedding the previously collected molecules and their scores, and prediction head to predict the score of unseen candidates. Intuitively. the embedding layers map the molecules and their scores to the same feature space already learned by the language model, allowing the model to perform in-context learning in this space instead of the raw text space. Unlike previous methods, this approach is applicable to domains that may not be easily described in natural language such as molecular optimization. Moreover, avoiding verbose textual descriptions enables the model to condition on more historical observations, thus scaling better to harder problems that cannot be solved within few steps.
We train the new layers together with the (frozen) LLM to perform in-context predictions on a family of functions. Specifically, for each function sampled from this family, we condition the model set of inputs and their corresponding evaluations, and task the model to predict the function value of the remaining data points. This task mimics surrogate modeling in BBO, where the surrogate model has to iteratively update its estimation of the underlying objective by conditioning on historical data. An ideal function family to train the model should be close to the target objective functions we want to optimize, but also be diverse enough to encourage generalization. Therefore, we propose to combine intrinsic functions and synthetically generated functions for training LICO. Intrinsic functions are inherent properties of the imput that are easy to evaluate. In molecular optimization, for example, intrinsic functions include molecular weight, the number of rings, or heavy atom count, which are obtained via simple computation related to the actual objective functions we want to optimize such as bioactivities against a target disease. To facilitate generalization outside of the intrinsic functions, synthetic functions defined over the same target domain that are generated by Gaussian Processes. Our empirical evidence shows the importance of learning from both intrinsic and synthetic functions to the performance of the model on downstream tasks. Figure iilussrates our approach.
the molecule. These intrinsic functions closely
additionally train LICO on
After training, LICO is capable of optimizing a wide range of molecular properties purely via in- context prompting. While the methodology of LICO applies to general scientific domains, in this paper we focus on molecular optimization. This problem plays a pivotal role in advancing drug and material discovery. The complexity of molecular structures and the vastness of the chemical space present unique challenges to black-box optimization algorithms. Moreover, since molecule- relevant data is likely under-representee in the pretraining corpora of existing language models, molecular optimization is a good problem to test the performance and applicability of LICO. We demonstrate the competitive performance of LICO against the leading methods on Practical Molecular Optimization (PMO) (Gao et al.) 2022). a challenging molecular optimization benchmark with 23 objective functions. On PMO's low-budget setting, which we term PMO-1M, LICO achieves the best performance and is the highest-ranked method in the benchmark.
2 PROBLEM STATEMENT
Let f : X + R be a real-valued function that operates on a d-dimensional space X' C Rd. In black-box optimization (BBO), the goal is to find the input x* that maximizes f:
x² E arg max f(x),
(I)
xEX
where we do not have direct access to the structure or gradient information of f. In molecular optimization, X' is the space of all possible molecules, and is certain property of the molecule
2

---

Published as a conference paper at ICLR 2025
Text embedddin
51
9..
z embedding
y embedding
Prediction layer
Pretrained LLM
Semi-syathetic
F
prompt <x>
<y>
y1
<カ>
Training
Figure 1: Our proposed approach. We equip pretrained LLM with embedding layer for s, embedding layer for y, and a prediction layer. We train the model on semi-synthetic data to predict given and previous (x; y) pairs. We prepend each with a special token <x> and each with special token <y> to guide in-context reasoning.
want to optimize over, such as bioactivities against disease, While is unknown, we often have access to an unlabeled dataset D, that consists of molecules x's without the corresponding function values y's. ZINC (Sterling & Irwin 2015) is such dataset with thousands of unlabeled molecules.
To solve the optimization in equation I we can query f with a limited budget, since evaluation often involves expensive physical experiments. To overcome this challenge, common BBO approach learns a surrogate model Jo that approximates the objective ʃ from past observations Dote = {Z,,, 34 w))):. which starts empty and incrementally expands with new data points (I, fz)) query at each iteration. Formally, a surrogate model represents a predictive distribution p3 x, Dots) of the function value y conditioned prediction this surrogate guides the selection of candidates to balance exploration and exploitation. The newly selected points added to Dobs. and the process continues.
the input and the evolving observed dataset Dubs The
The success of this approach highly depends on the efficiency of the surrogate model fo in estimating from limited data in Dyhs each iteration. This resembles few-shot prediction, setting that Large Language Models (LLMs) have proven to excel in. By pretraining on vast Internet-scale data, LLMs can learn generalizable patterns from limited data, and are capable of adapting to multiple functions at test time simply via in-context prompting (Brown et al. 2020 Mirchandani et al. 2023
Krishnamoorthy et al., 2023a b). recent line of works (Yang et al., et al.] [023] Liu et al.) has exploited this ability of LLMs for optimization, but they relied on natural language as the interface, thus lacking generality to scientific domains. In this work, we propose a more general and efficient approach to leveraging LLMs for black-box optimization.
2023 Zhang et al. [023]] Chen
RELATED WORK
LLMs for Optimization Recent works have explored the use of L.MSs for optimization. The general idea behind these works is prompt the model with the textual description of the optimization problem and historical evaluations for few-shot reasoning. Yang et al. (2023); Liu et al. (2023a);
Zhang al. (2023; ; Ma al. (2023) propose prompt the language model directly suggest better candidates to eval luate given the past inputs and their corresponding scores, Meyerson et al. (2023);
Lehman al. 2023): Bradley al. (2024) integrate LLMs with evolutionary algorithms, and prompt the model to perform crossover and mutation operations based on the population at each optimization step. [iu et al. study the use of I.LMs to enhance several components in Bayesian optimization, including warmstarting, surrogate modeling, and candidate generation. Optformer (Chen al. 2022) proposed to train an LLM specialized for in-context function prediction and optimization.
LLMs for Molecular Optimization Recent works have proposed to leverage LLMs for molecular optimization via prompting (Wang et al. 2024 Liu et al. 2023b Ramos et al. 2023 Völker et al.)
2024).
leveraging LLM embeddings (Ranković & Schwaller 2023), ) or finetuning
molecular
pora
Guevorguian et al. 2024 Ye et 20.3 et al. 2023 Kristiadi et al.). MOLLEO (Wang
al.
2024) and ChatDrug (Liu 2023b) are two prominent works in the first direction. MOLLEO
3

---

Published as a conference paper at ICLR 2025
proposed to prompt pretrained LLM to perform crossover and mutation operations in standard graph genetic algorithm, but its performance largely depends on the prompt format. Similarly, ChatDrug prompts a pretrained LLM for drug editing and requires retrieval database of molecules to inject domain feedback into the L.M.. BoChemian (Ranković & Schwaller 2023) and Kristiadi et al. studied the of L.MM embeddings for Bayesian molecular optimization, and Kristiadi et al. additionally explored finetuning an LLM to serve as a surrogate for optimization. Chemlac- tica/Chemma (Guevorguian et al. 2024) and MOLGEN (ang et al. (023) proposed to pretrain and/or finetune LLMs on molecule-related corpora to generate valid molecules, which can serve as genetic algorithm in molecular optimization. DrugAssist (Ye et al., (023) creates the MolOpt-Instructie dataset that contains pairs of molecules and their property values to finetune pretrained LLM that can iteratively propose better molecules after training.
The common approach in existing works has several inherent limitations. First, for general scien- tific domains, the input x may not be easily described by natural language. Second, even when there is textual description of the input, for instance, molecules can be represented by SMILES strings WWiininee 1988), existing prompt-based works require significant prompt optimization to achieve good performance, and the optimal prompt often varies between tasks. Furthermore, from an engineering perspective, naively prompting language model with verbose textual descriptions of the input x: results in excessively long context, thus reducing the number of examples the model can condition on. For example, an L.MM with a maximum context length of 4000 can only utilize up to 100 past observations, assuming the average length of each data point is 40. This practically limits the scalability of this approach to harder problems that require more steps to solve.
LLMS for Non-language Tasks In addition to optimization, several works have studied the extension of pretrained LLMs to non-language domains with two main directions. The first direction considers problems that can be described in natural language, and prompts a pretrained LLM to solve the problem directly in the text space (Mirchandami et al. [2033] Dinh et al. 2022] Gruver et al., 2023 Luu et al., 2024; SSrruill et al., 2024). The second direction tackles more general problems by learning separate encoders for the new domain and aligning it with the embedding space of the pretrained
LLM (Lu et al., 2022] Shen et al., 2023 Tsimpoukelli et al., 2021; Li et al. 2022). Our work is closely related to the latter direction. However, as discussed in the following sections, while many of these works completely leave the word space, we find it beneficial to include language instruction while training the new modules.
4 METHOD
We introduce LICO, a methodology for extending arbitrary base LLMs for surrogate modeling in black-box optimization. While the method applies to broad scientific domains, we choose molecular optimization to demonstrate LICO in this paper. We aim to develop model capable of efficiently adapting to various objective functions after training. To achieve this, we propose simple extension to existing LLMs and an unsupervised objective using semi-synthetic data to facilitate generalization.
4.1 MODEL ARCHITECTURE
In box optimization, a surrogate model Se estimates the distribution of the function value y given the input x and past observations Dobs {(0,, ) the model has collected until the optimization iteration t:
po(y) 2, Z1, V1. T2, V2. Fn Dn).
(2)
where i; and y/2 = fzx) are drawn from an objective function f. Our goal is to explore LLMs to model Ps. As discussed earlier, we make no assumptions on the domain X' to be expressed with natural language. To extend a pretrained language model to an arbitrary new domain, we equip the model with 3 new layers- an embedding layer for the inputs x' an embedding layer for the function values !/ s,s, and a prediction layer for predicting the unknown function value y. Learning separate embedding layers offers several benefits. First, the new embedding layers encode æ and y to shared hidden space obtained by the language model via pretraining, which enables the model to escape the raw text space and perform in-context reasoning in the hidden space instead. Moreover, by embedding each input x to a single hidden vector instead of spanning it over several tokens, we effectively reduce the sequence length and thus allow the model to scale more conditioning examples.
4

---

Published as a conference paper at ICLR 2025
However, is challenging for the model to perform this prediction task without any context informa- tion about the task. This is because, from the model point of view, embeddings of z and y do not mean anything more than some high-dimensional vectors. In other words, the model does not know what task it should perform and what each token in the embedding sequence represents. To address this issue, we prepend each sequence with a task prompt and prepend each input with special token <x> and each function value y with a special token <y>. The task prompt instructs the model perform the task, while the special tokens <x> and <y> inform the model of the position of each input x and the corresponding function value y. In other words, we use a language the model has mastered (natural language) to guide the learning of a new "foreign language" (e.g., molecule). In practice, the task prompt is "Each x is a molecule and each y is the property of the corresponding molecule. Predict given x.", whereas <x> and <y> are two single characters "x" and "y". Finally, we apply the prediction layer on top of each token <yy> to predict the function value given the tokens preceding it. Each prediction consists a mcan and a standard deviation value which will be used for the selection of candidates during optimization. Figure iillustrates the architecture of LICO.
is worth noting that the combination of natural language and domain-specific embeddings is the main distinction between LICO and previous works such FPT (Lu et al. 2022) which applies pretrained LLMs to sequence classification tasks in non-language modalities. FPT also learns new embedding layers for the new domain, but relies entirely on the pretrained self-attention layers to model these embeddings without any language instructions. This distinction stems from the different nature of the tasks we aim tackle. In sequence classification, the model produces single prediction for the entire sequence, thus having a good representation of the sequence via self-attention sufficient. For in-context learning, however, the model must associate each input æ with its value y to infer the underlying function f and make predictions for unknown y. A language instruction that specifies where x is and where y is helps the model identify this association and improve its in-context reasoning. Our ablation study in 5.2.1 confirms this utility of retaining language tokens.
4.2 SEMI-SYNTIETI TRAINING
Our goal is to train LICO on the unlabeled data Dú with an unsupervised objective to facilitate efficient generalization to arbitrary objective function f in the same domain X' after training. Our key insight is that if we train the model to perform the estimation in equation 2 for a wide range of functions, it should adapt to any objective function post-training. While the true function values unknown before optimization, we can use the unlabeled data x's to generate training data from other functions. Assume we have access to a family of functions ff that operate on the same input domain X. For each function f drawn from F. we sample set of function evaluations {(ri, }:)}= and train the model to autoregressively predict y given the input x and preceding (z, y) pairs:
(()) = E log (o Ii, (e,
in which the expectation is with respect I F. Fim ~ Da and yıo - After training, the estimation in equation 2 can be done purely via in-context prompting, where we condition the model on past observations make predictions for new data points.
Ideally, the function family f should be close the downstream objective f. but also be diverse enough to encourage broad generalization across functions. To achieve this, we propose to train LICO on mix of intrinsic and synthetic functions, which term semi-synthetic training. Intrinsic functions are functions that map each input molecule x to an inherent property of z. For example, molecular weight, the number of rings, or heavy atom count are intrinsic properties of the molecule that are known from domain knowledge or can be easily computed using standard tools. These intrinsic properties are closely related to many downstream objective functions. For example, the biological activity of a drug molecule, such as its ability to inhibit a particular enzyme, is often closely related to the molecule's 's shape or conformation. Therefore, training LICO from these functions encourages the model to learn useful representations of the input and obtain good prior knowledge about the optimization domain.
However, it is important to note that are ultimately interested in optimizing other functions outside of the intrinsic function set. Training the model only on a limited set of intrinsic functions may result in overfitting and poor generalization to unseen functions. To diversify the training data,
5

---

Published as a conference paper at ICLR 2025
additionally train the model on synthetically generated functions. A synthetic function family should be easy to sample from and be capable of producing diverse functions. Many such families exist, including Gaussian Processes (GPs), randomly constructed Gaussian Mixture Models, or randomly initialized neural networks. We choose to generate synthetic functions from Gaussian Processes with a Tanimoto kernel due its simplicity and efficiency. Tanimoto kernel, also known as the Jaccard coefficient, measures the similarity between two vectors of binary values, a representation that is widely used for many scientific domains such chemistry, drug discovery, or bioinformatics. Specifically, each synthetic function f sampled as follows.
(P), K(x, = ||||| + |2 - x-
(4)
where K(x,, is the Tanimoto kernel that measures the similarity between & and x'.
The final family of functions F used to train LICO is mixture of intrinsic and synthetic functions with a certain ratio. This design choice is critical to the model's performance. Intuitively, training on both types of functions ensures proximity to the downstream objectives and good coverage of the function space for efficient generalization. The use of intrinsic functions is also the main difference between our work and ExPT (Nguyen et al., 2023), recent method that studies pure syntheric pretraining for optimization. We hypothesize that while synthetic data is sufficient for ExPT on a few simple tasks, for a more complex domain such molecular optimization, synthetic training provides too little relevant signal for the model to generalize to downstream objectives. We empirically show the importance of both intrinsic and synthetic functions in the ablation study in section 5.2.2
4.3 LICO FOR BLACK-BOX OPTIMIZATION
After training, a single LICO model can be used for optimizing various objective functions within the domain X. Optimization involves an iterative process. At each iteration we generate set of candidates {x;)]== using standard crossover and mutation operations for which the model predicts the mean μ4, and standard deviation o; conditioned on prior observations Dobs, a dataset of (æ, y) pairs collected until f. An acquisition function a then calculates a utility score based on μ4; and for each candidate, balancing between exploration (favoring high σ7) and exploitation (favoring high μt). The top h candidates determined by their utility scores are evaluated using the objective function f. These & candidates and their corresponding evaluations are incorporated into the dataset Dobs. and the cycle repeats. This process terminates once we exhaust the evaluation budget of B. Algorithm | summarizes the optimization process and Appendix A.3 outlines the optimization hyperparameters.
5 EXPERIMENTS
We evaluate LICO on molecular optimization, where the goal design new molecules with desired properties such as high chemical stability, low toxicity, or selective inhibition against target disease. This problem plays a pivotal role in advancing drug and material discovery.
5.1 PMO BENCHMARK
Benchmark We evaluate LICO on Practical Molecular Optimization (PMO) GGo et al. 2022). standard benchmark for molecular optimization with focus on sample efficiency. We exper- iment on 23 optimization objectives provided by PMO, including QED Bickerton et al. (2012). DRD2 OOiveecrona al. 2017). GSK333 JNK3 (Li et al. 2018). and 19 objective functions from Guacamol (Brown et al. 2019). QED assesses a molecule's drug-likeness by identifying certain "red flags". DRD2 is a machine learning model trained on experimental data to predict bioactivities for specific target diseases. Guacamol objectives emulate drug discovery goals through
objective (MPO) approach, considering factors like target molecule similarity, molecular weights, and CLogP. All objective values range from 0 to 1, with 1 indicating the best outcome. We consider two evaluation settings - the original PMO with a budget of 10000 oracle calls, and a budget-efficient setting with 1000 oracle calls, which we refer to PMO- 1K. We believe 1000 is more reasonable budget while still allowing optimization methods to achieve meaningful performances. To ensure a fair comparison in PMO-IK, we performed extensive hyperparameter tuning for each baseline on the

---

Published as a conference paper at ICLR 2025
Table 1: The performance of LICO and the baselines on 23 optimization tasks in PMO-IK. A higher score is better. We report the mean and stddev of scores averaged over 5 random seeds. We use blue and violet to denote the best and second-best method for each task.
Task
Graph
RUINVENT
LICO
Genetic GTN
Augmented Memory
MOLLDO
buuterrl similariny larley
0695 0.0000
0.583
0.065
(0:00-0.00
0.050 - 0.125
0.664 + 0.054
0.557 0.005
01886 + 0.025
an Lodi pinn, rpg
0.519 --.01
0.501
0.010
0:472 -0000
0.541 - -0.026
0.534 + 0.001
0.450 0.000
0.657 00077-00.22 025
ow oelweueelll ie redixcowey dop drd2 texofenadi iepo
0.411 + 00.06
0.421 = 0.04
0.570 0.020
0.447= 0.075
0.447 + 0. 028
0.855 = 0.027
0.412 = 0.005
0.503 0306::000
0.081
0.00%
0.072
0.006
0.596 = 0.010
0.604 = 0.017
0.070 0.010
0,588 = 0.007
0857 0.0000
0.853
01065
0.775
0006
0.850
0.066
0.8.0 0-005
6.795 0.0021
0.110 0.07
0.707 + 0.021
0.000 0.00
0.050 0.007
0.700
0.023
0.082 + 0.022+
0.079 0.02
0.74 - 0.02
0611 + 0.059
0.023
0.047
0.589 0.003
0.617 - 0.063
0.637 + 0.018
0.530 00:00-0::00 0.007
0.397 + 0.015
0.545 + 0.158
0.755
0.112
0.720
0.061
0.779 0 0.099
0. 755 + 0 059
0.061 = 0.059
0.737 + 0.045
0.599 4: 01,09
0.630
D.586
0.030
0
0.672 0.075
0 0.656 : 0.075
0.006 = 0.066
0.633 = 0.017
Tak3
0.346 2: 0.067
0.301
0.771
0.315
0.042
0.336
01003
0 0.409 =: 0.166
0.294 0.110
0.186 = 0,676
medilar 1
02113+-0.000
0.208
0.015
0.236
0002
0:217 0.09
212 + 008
0.218 0.01
0.236 0 021
medi 2
0 0.203-1 + 0.000
0.181
0.000
0.188
0.00
0.153 0.000
0. 2014 0.011
6111 0.000
0.19: + 0.09
nwet eanol L_ wir Lamilarty
0.427 + 0.002
0.912
0.017
0.379
0.02
0.425 0.016
0.414 0.022
0.323 0.021
0,599 = 0.020
oa izwect iiilltrre perindopel 2_.rop aod ranoll lasinn aggi foadfold_hog wit sitagliptinng intmpo thiothizene groglitazone__ee sediacovery
0.766 +00.00
0,751
0.005
0,737 = 0.007
0.758 0.008
0.763 : 0.008
0.761 0.006
0.779 = 0 000
0.458 4: 11.11
0.435
0,06
0:44 0.000
0.473 0.009
0,662 2: 0.683
0:422 = 0.013
0.65G = 0.054
0.912 1:1001
0,014
0.007
0,211 0.012
0.925
0.005
0.928 :: 0 002
09921 = 0.002
0.119 = 1100G
0.701 + 0.025
0.020
0.014
0.77-00011
0.687 0.029
0.023 + 0.022
6.01+ 0.055
0.60 0.00
0.478 + 0.009
0.461
0.008
0:47==000
0
0. 0.008
0.485 + 0.015
0::00=00000
0.173 = 0.000
0.232 + 0.083
0,229
DJGSS
0.361 0.026
0.315
0.097
0,227 : 0.0.61
0.26 0 0.080
0.183 = 0.673
0.351 # 11,03
0.822
0.225
0,311 = 0.021
0.348
0M085
0.377 :: 0.015
0.336 = 0.043
0.416 : 0.077
0,313 1 0.015
0.267
0.215
0.246 = 0.009
0.252
0.028
0,277 2: 0.015
0,202 2 0.012
0,302 = 020
valsartan_ srarts
00000 2: 11.001
0,000
0,000
0,000 0.000
0,000 2: 0.000
0,100
2: 0.000
0,000 2 0.000
00000 2 0.000
saicclon__rpo
0.392 1 0.004
0.074 2: 0.024
0.406
0.017
0,404 2 0.022
0,000 1 0.014
0.415 0.013
0.192 1 0.00
Sun of scoees (f)
11.27
10:90
10.68
11.71
11.56
10.81
11.65
first 5 tasks, and used the optimal hyperparameters for the remaining tasks. Appendix B.1 details hyperparameter search for the baselines.
Baselines We compare LICO against leading methods in PMO, namely Genetic GFN Kim et al.
2024), REINVENT (Olivecrona et al.] 2017), Augmented Memory (Guo & Schwaller 2023) Graph
GA (Jensen 2019), GP BO (Trpp et al. 202I). and MOLLEO (Wang et al.] | 2024). Genetic GFN
employs GelowNets (Bengio al. 2023) model trained to sample molecules proportional to their rewards. REINVENT is a reinforcement learning method that finetunes a pretrained RNN for generating SMILES strings, and Augmented Memory combines REINVENT with data augmentation and experience replay. Graph GA, inspired by evolutionary processes, utilizes crossover and mutation operations to explore the molecule space. GP BO is a Bayesian optimization method that augments Graph GA with a Gaussian Processes surrogate model and UCB acquisition function to guide candidate selection. MOLLEO is an LLM-based method that prompts chemistry- LLM such as BioT5 (Pei et al.) to perform mutation and crossover operations in evolutionary algorithm. Among the baselines, GP BO is the most similar to LICO, where the only difference we use an LLM for surrogate modeling instead of GP.
LICO training We use ZINC 250K as the unlabeled dataset D.. ZINC 250K contains around 250000 molecules sampled from the full ZINC database (Sterling & Irwin 2015) with moderate size and high pharmaceutical relevance and popularity. We adopt 2-radius 2048 bit Morgan molecular fingerprints as the input feature of the molecule. To generate training data, we use 47 intrinsic properties of the molecule as the intrinsic functions, which we present in detail in Appendix I We train LICO for 20000 iterations with batch size of 4, where each data point is sequence of (z, y) pairs sampled from an intrinsic or synthetic function. The ratio of synthetic data is 0.1. We use Llama-2-7b (Touvron et al.] (023b) as the base LLM, and use LoRA (Hu et al. 2021] ) for parameter-efffcient finetuning. We use the Llama-2 2-7b-aa chat checkpoint for the 1000-budget setting, and the 2-m--227B-32-- Instruct checkpoint with the Liger Kernel (Hsu et al. 2024) for training and inference with long context for the 10000-budget setting.
Evaluation details We report the area under the curve (AUC) of the top-10 average objective value against the number of function calls as the performance metric. AUC metric favors methods that obtain high values with smaller number of function calls, thus evaluating both optimization capability and sample efficiency. We min-max scale the AUC values to [0, 1]. We aggregate the performance for each method across 5 seeds for better reproducibility as suggested by PMO.
Results Table I summarizes the performance of the 7 considered methods across 23 optimization tasks in PMO- TK. Overall, LICO is the leading method in this benchmark, achieving the highest aggregated score. Specifically, LICO achieves the best MOLLEO performs competitively with LICO on this benchmark, with a sum score of 11.65. However, note that MOLLEO has significant advantages over LICO and other methods. MOLLEO used
second-best performance in 14/23 tasks.
7

---

Published as a conference paper at ICLR 2025
Table 2: The performance of LICO and the baselines on 23 optimization tasks in PMO. A higher score is better. We report the mean and stddev of scores averaged over 5 random seeds. We use blue and violet to denote the best and second-best method for each task.
Task
GP BO
Graph GA
REINVENT
LICO
Genetic GPN
Augmtented Memory
MOLLDO
"butern) wnd larity
0,886 0 0.14
0.538 0.010
0882 + 0.005
0.555 + +00019
0.9:1 00.021
0 + 0.000
0.935 + 0.016
and aniidipp ne gwlwooxi BLtooiiBrrddddeeee dleco_hop
0.583 0003-
0.061 0.020
0.655 + 0.055
0.679 + 0.027
0.700 0.002
0.091 + +0.017
0.501 + +00.28
0.723 0.73=0005
0.030 0.077
0.733 + 0.067
0.064 0.1122
0.784 = 0.082
0.796 = 0.008
0.453 +
0.629=000
0.019 0.004
0.006 : 0.044
0.619 + 0.035
0.633 0.025
0.658 00.88=0..22 0.024
0.648 4 0.000
drd2 Texofenacins
0.921 00017
0.964 2 0.012
0.945 6: 0.007
0.528 0.018
0.963 : 00.006
0.963 0.000
0.302 4: 0.003
0.722 = 0.003
0.760 6.011
0.784 2: 0.005
0.772 0,123
0.700 : 0.009
035 0.000
0.776 2: 1,019
gok-30
0.801 = 0.041
117785
0,070
0.863 4 0.003
0.876 2: 0.045
0.861 1: 0.022
0.581 2: 0.021
01563 1 1,003
Looncrs C cTh8n2c2
0.90==0.0 0.117
11562
0.055
0.832 1 0.005
0.939 2: 022
0.855 0.007
0.853 2 0.0S7
0.915 1 1,0036
Loomerss_chh wo_.ohhh10000 O22o2p
0.407 0.180
0. 0:017
0.642 1 0.54
0.819 1: 030
0.876
0.035
0.736 2 0.051
0.708 1 0.09G
3nk3
01561 0.155
0.355
8:0 130
0.783 8 010223
0.721 2 0.037
0.720
0.063
0.730 8 0.110
0.715 1.
median1
0330 = 0014
0.251 0.021
0.356 1 0.009
0.291 +: 0.000
0.355
0.000
0.320 + 0.013
0.302 + 0.031
ned an?
0.297 - 0.000
0.273 0.0.9
0.276 + 0005
0.260 + 0.019
0.289
0.007
D 291 0.008
0.206 0.015
rest rano L ind larkiy
04227 - 0.08
0 0.022
0.16 + 0.085
0.61 + 0.06-
0
0.085
0.750- 0.049
0.750 0. 102
wwiinettinib..g per perindopeil_ngg
0.787 = 0.005
01 0.006
0.877 + 0.009
0.520 + 0.002
0.848
0.008
855 = 0.004
0.31 + 01.03
0.801 0.11
01538 0.003
0.587 = 0.016
0.53T = 0.112
0,505
0.011
0.618 0.016
0.723
018
qed sanslazina soaffal altaglipcinunpo
0.987 = 0.000
11344 0.000
0.941 2: 0.000
0 1: 0.0001
0.987
0.000
0 0.942 = 0.000
1.203 2: 1.000
0.783 0013
1.7288
0.012
0.700 2: D.00
0.774 1: 0.108
0.510
0.011
0.501 2 0.000
I.7711 2: 0.022
0.565 0015
0.5AT 0.007
0,500 2: 0.019
0.54T 1: 0.1126
0.585
0..41
0.567 2 0.008
0.215 1. 0.002
0.186=0100 01055
0,435 0,075
0.021 2: 0.005
0.567 1. 034
0.577
0.033
0,254 1: 0.030
0 1. 0,020
thiothikens, rodi scovery
0.59 - 0022
0.479 022
0.544 1: 0,05
0.514 1: 0.137
0.
0.077
0.530 1: 0.041
0 6.6
024
=roglizazone, reciscovery
0410=001 0015
0
0.016
0.411 2: BX02
0.580
1: 17:0
0. 155
0.016
0.5.0 0.0888
0.552 1: 0.040
va sartar snarts
0.000 0.000
0.000 0.000
0.178
+ 0.355
0.000 0.000
0.242
0.000 0.000
0/000 0.000
zelegilen_ngg ngo
0.072 0.221
0.540 0.009
0.358 0.022
0.815 0.017
0.5:5 0.023
0.991 0.020
0:402 0.019
Sam oE (?)
18.156
13.751
14.196
15.678
15.002
14.652
Table 3: Performance of LICO on 5 tasks with different language instructions.
Task
aibuterol_ setls__snnnaieeeeeeee sinilarity
axtoddiino__ro celocoxin_ rodi scevary
dacc_hop
dzd2
Sum (1)
LICO wio Language 1100 wis Tisk prompe
0,615 2. 1114
0.491 =: 0.011
11336 0.051
0,28 1LU06
0.580l =: U.IK3
2,927
0.641 + 101
0.523 0.018
0.4ST 0.041
0.505 0.006
0.844 101
3.090
LICO
0.636 0.125
0.541 0.026
0.447 0.07
0.586 0.010
0.859
0.066
3.099
BioT5 to generate valid molecules, which has been finetuned extensively on molecules, protein, and molecule-related text data, while LICO leveraged a general LLM like Llama. Moreover, MOLLEO prompts the LLM with detailed textual description of the task such as "Your job is to generate a SELFIES molecule that looks more like a drug". ", which possibly has data contamination issues, since the finetuning data may have included similar tasks. On the other hand, we use LICO as a black-box surrogate model, where the model makes predictions based purely on in-context learning of the mapping between molecule fingerprints and their corresponding scores.
On the original PMO setting, Table 2 shows the competitive performance of LICO with the two tht methods Genetic GFN and Augmented Memory. It is important to note that other methods have significant advantages over LICO, since both Genetic GFN and Augmented Memory update their models from real data during optimization, whereas LICO performs in-context learning without being explicitly trained on data from downstream objectives. This impressive result shows the effectiveness of syntheytt training in enabling generalization to a broad range of functions via in-context prompting.
The most closely related method to LICO is GP BO, where the only difference between the two is the surrogate model. This indicates the superiority of LICO compared to GP, popular surrogate model for black-box optimization. To verify this, we compare the predictive performance of LICO and GP on several objective functions. We do this by first labeling the ZINC unlabeled dataset with the objective functions and randomly choosing a subset of the labeled data points for evaluation. For each task, we vary the number of examples given to each method from 32 to 512, and evaluate their performance on 128 held-out data points. We use negative ligelikel mean squared error, and root mean squared calibration error as the evaluation metrics. Figure 2 compares the predictive performance of LICO and GP in 3 objective functions, medianl. aenolazineenee and troglitt awtaaazooeeeereeeeeeeeeeee s scovery. The figure shows that the optimization performance of the method closely aligns with the predictive performance of the surrogate model. In medianl and ranolazine waapllaiiee__p___ _em___pp____________ where LICO outperforms GP in terms of optimization score, the model also achieves lower negative log- likelihood, mean squared error, and calibration error. Similarly, LICO has worse predictive performance in t rogl poglit azone cectooe_rrdidi rediscover where underperforms GP. This verifies our hypothesis and proves the effectiveness of LICO for surrogate modeling.
8

---

Published as a conference paper at ICLR 2025
VSE
0.10
0.08
A0004
0.8
90060
6.200
A063
0.004
00025
630
1/013
1.80
0.08
0.18
30007
00004
Figure 2: The predictive performance of LICO and GP on 3 objective functions in PMO with different metries and varying numbers of observations.
Table 4: Performance of LICO on 5 tasks with different ratios of synthetic data.
Task
ant oul negugo cell swlecox: b. redi acpvecy
decu_hop
d2
Sum (T)
LICO Intrinsic
0.508 = 0.115
0.522 + 0.029
0:412 = 0.042
0.585 + 0.006
01801
0.022
3.000
LIC'O 0.1 Symhhtic
ILG56 = 0.125
0.641 026
0.447 0.073
0.096 0.030
(1850
2: 01066
3.059
LICO 0.5 Synihetic
0.663 140
0.504 0.016
0.402 0.016
0.588 + 0.006
0.907
0.020
3.03
LICO Synthetic
0.547 0.080
0.498 0.026
0.304 0.103
0.585 0.003
0.902
0.012
0.012
2:506
5.2 ABLATION ANALYSIS
We perform various ablation studies understand the importance of different components and design choices in LICO. For the ablation experiments, we consider the first 5 tasks in Table report the aggregated performance of different models using AUC Top-10 across 5 random seeds.
[ only. We
5.2.1 LICO WITHOUT LANGUAGE INSTRUCTION
First, we examine the importance of language instructions the performance LICO. We compare 3 variants of LICO: 1) LICO without any language instruction, 2) LICO with special tokens <x> and <y> but without a task prompt, and 3) LICO with both special tokens and the task prompt. Table compares the performance of the 3 variants. LICO performs the best in 4/5 tasks, followed by LICO without the task prompt. LICO without any language instruction performs the worst, often by large margin. This result confirms the importance of guiding pretrained LLM with language instruction when applying the model to in-context reasoning a completely new domain.
5.2.2 LICO WITH DIFFERENT SYNTHETIC RATIOS
We investigate the importance of training LICO on both intrinsic and synthetic data. To this, we gradually increase the ratio of synthetic functions in the training data from 0 (intrinsic-only) to 1 (synthetic-only), and compare the performance LICO across different ratios. Table J shows that LICO with semi-synthetic training performs the best, outperforming both intrinsic-only and synthetic-only data. Training with synthetic data only performs the worst, which is expected when synthetic functions generated by GP do not include any domain knowledge that is encoded by the intrinsic functions. other words, synthetic data alone provides too little relevant signal for the model to generalize unseen downstream objectives. Training with intrinsic functions

---

Published as a conference paper at ICLR 2025
Dere2
Table 5: Performance of pretrained vs randomly initial- ized LLMs.
Task
Pretrained LLM
Scratch LLM
albuterol simi larity
0.656 = 0.125
0.575 + 0.061
amlodipine_mag
0.541 + 0.026
0.503 0.029
celecoxib reedsooerr
0.447 0.073
0.110 0.034
decc__oo hop
0.596 + 0.010
0.583 1: 0.005
drd2 0.859 0.066
0.827 0.085
Sum
3.099
2.898
Figure 3: LICO with different LLM sizes.
only, on the other hand, results in quite good performances on most tasks. However, in tasks like albuterol_sin lamity...y.... semi-synthetic training outperforms this baseline by a large margin. We hypothesize that the underlying objective in albuterol. s ammiaartyy is far from the intrinsic functions used to train LICO, leading to poor generalization. Finally, training with small (0.1) to moderate (0.5) ratios of synthetic data achieves similarly good performance.
5.2.3 RANDOMLY INITIALIZED VS PRETRAINED LLMS
To understand the importance of using a pretrained LLM, we compare LICO with an autoregressive transformer model of the same size (7B). The transformer architecture is the same as in (Garg et al., 2022), and we train this model to perform in-context learning on the semi-synthetic data from scratch. Table 5 shows the comparison. The scratch model performs much worse than LICO with a pretrained LLM on all tasks despite sharing the same number of parameters. This highlights the importance of the pattern-matching capabilities that LLMs like Llama-2 acquire via extensive language pretraining.
5.2.4 LICO WITH DIFFERENT LLM SIZES
Previous works have shown the favorable scaling laws of Large Language Models where larger models consistently perform better on downstream tasks (Kaplan et al. 2020). In this section, we investigate the scaling properties of LLMs but in the context of black-box optimization. Specifically, we compare 4 different base LLMs with different sizes Qwen--eeee 1.5 1.8B and 4B (Bai et al. (023). Phi-2 2.7B (Javaheripi al., 2023), and Llama-2 7B (Touvron al., (023b). We the same language instructions for all models. We evaluate each model on the first 8 tasks in Table T and average the results across 5 random seeds. We report the sum of performance across tasks.
The comparison in Figure shows that the optimization performance scales consistently with the model size, with Llama-2 7B being the best method. This experiment indicates that larger LLMs not only perform better in language tasks but also obtain stronger pattern-matchinn capabilities that can be transferred completely different domain. Given this scaling, we can further improve the current performance of LICO by scaling up the base LLM size.
6 CONCLUSION AND FUTURE WORK
We develop LICO, new method that leverages pretrained Large Language Models for black-box optimization. LICO extends existing LLMs non-language domains with separate embedding and prediction layers. To enable efficient generalization to various optimization tasks, we train LICO on diverse set of semi-synthetic functions for few-shot predictions. LICO achieves performance on PMO, challenging molecular optimization benchmark with over 20 objective functions. Ablation analyses highlight the importance of incorporating language instruction to guide in-context learning and semi-synthetic training for better generalization. One limitation of our method is the assumption an accessible set of intrinsic functions. While this is true for molecular optimization, it may not apply to other scientific domains. In such cases, better synthetic data generation process incorporating domain knowledge is needed to aid generalization. Future directions include evaluating LICO in other domains to test its applicability and generality, exploring other prompts that better exploit the capabilities pretrained LLMs, and using LLMs for other aspects of optimization, such candidate suggestion exploration.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08874. 2023.
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Luey Colwell. Model-based reinforcement learning for biological sequence design. 2020.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. The Journal of Machine Learning Research, 24(1):10006- 2023.
Felix Berkenkamp, Angela P Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 491-496. IEEE, 2016.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90 2012.
Herbie Bradley, Honglu Fan, Theodoros Galanos, Ryan Zhou, Daniel Scott, and Joel Lehman. The openelm library: Leveraging progress in language models for novel evolutionary algorithms. Genetic Programming Theory and Practice XX. Springer, 2024.
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp. 773-782. PMLR, 2019.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for novo molecular design. Journal of chemical information and modeling, 59(3): 1008,1108, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:23003. /27 2023.
Angelica Chen, David M Dohan, and David R So. Evoprompting: Language models for code-level neural architecture search. arXiv preprini arXiv:2302.1488. 2023.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Marc'aurelio
Kawakami, Greg Kochanski, Arnaud Doucet, Marc aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Process- ing Systems, 35:32053-32066 2022.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35: 11763-11784, 2022.
Yin Fang, Ningyu Zhang, Zhuo Chen, Lingbing Guo, Xiaohui Fan, and Huajun Chen. Domain- agnostic molecular generation with self-feedback. arXiv preprint arXiv:2301.1 12199 2023.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: benchmark for practical molecular optimization. Advances in Neural Information Processing Systems, 35: 21 21342-21357, 2022.
11

---

Published as a conference paper at ICLR 2025
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30533- 2022.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research,
2012.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
Philipp Guevorguian, Menua Bedrosian, Tigran Fahradyan, Gayane Chilingaryan, Hrant Khachatrian, and Armen Aghajanyan. Small molecule optimization with large language models. arXiv preprint arXiv:2407. 18897, 2024.
Jeff Guo and Philippe Schwaller. Augmented memory: Capitalizing on experience replay to accelerate de novo molecular design. arXiv preprint vrXiv:2305.16160, 2023.
Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a supercon- ductor. Computational Materials Science, 154:346-354, 2018.
Pin-Lan Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for Ulm training. arXiv preprint arXiv:2410.1098/ 2024. URL http::/:: 10989
org/ abs /4410
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.09685 2021.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):35673357, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot,
Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04008 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Hyeonah Kim, Minsu Kim, Sanghyeok Choi, and Jinkyoo Park. Genetic-guided gflownets: Advancing in practical molecular optimization benchmark. arXiv preprint arXiv:2402.05900 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199,22213, 2022.
Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Generative pretraining for black-box optimization. In ICML, 2023a.
Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black-box optimization. In ICML, 2023b.
Agustinus Kristiadi, Felix Strieth-Kaiin Kalthoff, Marta Skreta, Pascal Poupart, Alan Aspuru-Guzin and Geoff Pleiss. A sober look at Ilms for material discovery: Are they actually good for bayesian optimization molecules? In Forty-first International Conference on Machine Learning.
12

---

Published as a conference paper at ICLR 2025
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331-366. Springer, 2023.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek,, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199-31212 2022.
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10:1-24, 2018.
Thomas Liao, Grant Wang, Brian Yang, Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calan- dra. Data-efficient learning of morphology and controller for a microrobot. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2488-2494 IEEE, 2019.
Fei Liu, Xi Lin, Zhenkun Wang, Shunyu Yao, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. Large language model for multi-objective evolutionary optimization. arXiv preprint arXiv:2310.12541, 2023a.
Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Chatgpt-poweree conversational drug editing using retrieval and domain feedback. arXiv preprint ar arXiv:2305.18000 2023b.
Shengchao Liu, Jiongxiao Wang, Yijin Yang. Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Conversational drug editing using retrieval and domain feedback. In The Twelfth International Conference on Learning Representations, 2024.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning Representations.
Micha Livne, Zulfat Miftahutdinov, Elena Tutubalina, Maksim Kuznetsov, Daniil Polykovskiy, Annika Brundyn, Aastha Jhunjhunwala, Anthony Costa, Alex Aliper, Alán Aspuru-Guzik, Guzik, et al. nachO: Multimodal natural and chemical languages foundation model. Chemical Science, 15(22): 8380-8389, 2024.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pp. (628-7636, 2022.
Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint 1.. 2023.
Elliot Meyerson, Mark J Nelson, Herbie Bradley, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint 17X1V:2302.12170, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint 2rXiv:2307.04721, 2023.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty meta learning via sequence modeling. In ICML, 2022.
Tung Nguyen, Sudhanshu Agrawal, and Aditya Grover. Expt: Synthetic pretraining for few-shot experimental design. In NeurIPS, 2023.
Allen Nie, Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Importance of directional feedback for llm-based optimizers. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de- novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
13

---

Published as a conference paper at ICLR 2025
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingee Xia, and Rui Yan. Biot5: Enriching cross cross-modal modal integration in biology with chemical knowledge and natural language associations. In The 2023 Conference Empirical Methods in Natural Language Processing.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 21(140): 2020.
Mayk Caldas Ramos, Shane S Michtavy, Marc D Porosoff, and Andrew D White. Bayesian optimiza- tion of catalysts with in-context learning. arXiv preprint arXiv:2304.05341, 2023.
Bojana Ranković and Philippe Schwaller. Bochemian: Large language model embeddings for
bayesian optimization of chemical reactions. In NeurlPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World, 2023.
Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez. et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-401, 2016.
Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modes modal fine-tuning: Align then refine. arXiv preprint arXiv:2302.05738, 2023.
Henry W Sprueill, Carl Edwards, Khushbu Agarwal, Mariefel V Olarte, Udishnu Sanyal, Conrad Johnston, Hongbin Liu, Heng Ji, and Sutanay Choudhury. Chemreasoner: Heuristic search over large language model's knowledge space using quantum-chemical feedback. arXiv preprint arXiv:2402. 10980, 2024.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11 1):2344 2337, 2015.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arxiv:23/2.1 1880, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Austin Tripp, Gregor NC Simm, and José Miguel Heránndez-obbao. A fresh look at de novo molecular design benchmarks. In NeurIPS 2021 AI for Science Workshop, 2021.
Maria Tsimpoukelli. Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212 212, 2021.
Christoph Völker, Tehseen Rug, Kevin Maik Jablonka, and Sabine Kruschwitz. Lims can design sustainable concrete-a systematic benchmark. 2024.
Haorui Wang, Marta Skreta, Cher-Tian Ser, Wenhao Gao, Lingkai Kong, Felix Strieth-Kaltho Chenru Duan, Yuchen Zhuang, Yue Yu, Yanqiao Zhu, et al. Efficient evolutionary search over chemical space with large language models. arXiv preprint arXiv:2406.16977 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24888 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
14

---

Published as a conference paper at ICLR 2025
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.033400 2023.
Geyan Ye, Xibao Cai, Houtim Lai, Xing Wang, Junhong Huang, Longyue Wang, Wei Liu, and Xiangxiang Zeng. Drugassist: large language model for molecule optimization. arXiv preprint vrXiv:2401.10333 2023.
Michael R Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. arXiv e-prints, pp. arXiv-2312, 2023.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1..157761..0117, 2016.
15

---

Published as a conference paper at ICLR 2025
A LICO IMPLEMENTATION DETAILS
A.1 MOLECULAR INTRINSIC FUNCTIONS
We utilize 47 intrinsic properties of molecules for pretraining LICO. Table shows the intrinsic properties and their explanation.
Table 6: Inherent Properties of Molecules and their Explanations
Property
Explanation
Molecular Weight
Total mass of all atoms in the molecule.
Number Rotatable Bonds
Bonds that allow free rotation around themselves.
Number Rings
Count of ring structures the molecule.
Number of H Donors
Atoms in the molecule that can donute hydrogen atom.
Number H Acceptors
Atoms in the molecule capahle of accepting a hydragen atom.
Num Anumatic Rings
Count of rings with a puttern of alternating single and double bonds.
Num Aliphatic Rings
Count of non -nn-rrmmatic rings in the molecule.
Num Saturated Rings
Rings with single bonds only.
Num Heteroatoms
Atoms other than carbon or hydrogen.
Fraction Csp3
Fraction of carbon atoms bonded with single pair electrons. Count of all atoms except hydrogen.
Heavy Atom Count
Num Valence Electrons
Total number of electrons that can purticipate in the fornation of chemical bonds.
Num Ammatic CarboRings
Aromatic rings composed solely carhon atoms.
Num Aromatic HeteroRings
Aromatic rings containing at least one heteroatom. Saturated rings made only of carbon atoms.
Num Saturated CarboRings
Num Saturated HeteroRings
Saturated rings containing at least one heteroatom. Topological index to quantify molecular brunching. A measure of structural complexity of the molecule. Information content on the vertex degree.
BalabanJ
BerzZT
Ipc
HallK
Valence connectivity index used in molecular shape analysis. Shape descriptor based on the skeleton of the molecule. Hydrogen suppressed gruph descriptor.
Kappal
Kappu2
Kappa3
Hydrogen complete graph descriptor. Randić molecular connectivity index.
ChiO
Chil
Valence modified Randić molecular connectivity index. Randić connectivity index normalized.
ChiOn
Chiln
Valence modified Randić connectivity index normalized.
Chi2n
Second order Randić connectivity index normalized.
Chi3n
Third order Randić connectivity index normalized. Fourth order Randić connectivity index normalized, Randić connectivity index for valence electrons.
Chi4n
ChiOv
Chily
First order valence molecular connectivity index. Second order valence molccular connectivity index. Third order valence melecular counectivity index. Fourth order valence molecular connectivity index. Measure of the molecule's 's polarizubility.
Chi2v
Chi3v
Chi4y
Molar Refractivity
AMW
Average molecular weight all atoms molecule. Maxinum partial charge the molecule. Minimum partial charge in the molecule.
Max Partial Charge
Min Partial Charge
Max Abs Partial Charge
Maximum absolute value the partial charges the molecule. Minimum absolute value of the purtial churges in the molecule. I Labute's ' Approximate Surface Area, an estimate of the molecular surface area. Maximum eiectrotopological state index of the atoms in the molecule. Minimum electrotopological state index of the atoms Maximum absolute value of the electrotopological state indices in the molecule. Minimum absolute value of the electrolopological state indices in the molecule. Frequency of curbon-oxygen bonds in the molecule.
Min Abs Partial Charge
I Labute ASA
Max EState Index
Min EState Index
molecule.
Max Abs EState Index
Min Abs EState Index
C__
A.2 TRAINING DETAILS
The x embedding layer, y embedding layer, and prediction layer in LICO are MLPs with a hidden dimension of 1024. We train LICO for 20000 steps with a batch size of 4. For each data point in the batch, we randomly decide whether to sample an intrinsic or a synthetic function, with the probability of choosing synthetic functions being 0. Each data point is sequence of (x., y) pairs with length n ~ (6,44, 800). If the function is an intrinsic function, we uniformly sample a property from Table 6 otherwise sample synthetic data following Equation equation 4
We use Llama-2-7 7b (Touvron et al. 20236) as the base LLM, and use LoRA (Hu et al. 2021 for parameter-efficient finetuning. We base learning rate of 5e 4 with linear warmup for 1000
16

---

Published as a conference paper at ICLR 2025
steps and cosine decay for the remaining 19000 steps. We LoRA with rank of 16 and scale of 16.
A.3 BLACK-BOX OPTIMIZATION IYPERPARAMETERS
We Algorithm to optimize black-box function with LICO. We initialize the observed dataset Dots with population of 34 molecules sampled randomly from ZINC. At each iteration, we use the best 34 candidates in Dons to generate new candidates via crossover and mutation operations, with the mutation rate being 0.01. The candidate pool size C is 100. We predict the mean J4 and standard deviation Ø; for each candidate 2; in the pool using LICO. We employ a UCB acquisition function to compute the utility score = μ4 30,, which balances exploration and exploitation. Following (Gao et al. 2022), we set
10t, where b U[-0.5 1.5). We then pick & 15 candidates with the highest utility scores. We evaluate each selected candidate 2j using the black-box function f. and add the new data point (rz Vj) to the observed dataset Dobs The process continues with the updated observed dataset, and stops when [Dats| 1000.
When predicting 14; og = So (1r; Dots), we normalize all the y's values Dons to have mean and standard deviation 1. This is to resemble the finetuning data distribution of LICO. We then denormalize μ4 and o; to the original space.
A.4 BLACK-BOX OPTIMIZATION WITH LICO
Algorithm
D outlines the optimization algorithm using LICO the surrogate model.
Algorithm 1 Black-box optimization with LICO
Require:
objective LICO model f'a budget B, candidate pool size C, acquisition function a, batch size
Initialize Dass = {
while (Daue] < B do
Generate of candidates (#:)
for each candidate x,
Predict f4, C. = fo Dass)
Compute utility score = a(14 o.) end for
Select & candidates with the highest utility scores each selected candidate xy do
Evaluate using the actual objective U/ = (xx
Add (ır,, Mt) to the observation dataset Dohs
end for end while
B BASELINE DETAILS
TNP is a transformer-based architecture for in-context learning. We refer to Nguyen & Grover (2022) for more details about TNP. We train TNP model with 16 attention layers and 2048 hidden dimensions. Other hyperparameters are the same for LICO. After training, use TNP for black-box box optimization using Algorithm T with the same optimization hyperparameters but replace LICO with TNP.
GP BO replaces the LICO surrogate model in Algorithm I with a Gaussian Process with a Tanimoto kernel. We optimize the Gaussian Process hyperparameters via maximum likelihood estimation the initial population sampled from ZINC.
Graph GA is model-free variant of Algorithm 1
Specifically, at each iteration, Graph GA generates a set of candidates using the same crossover and mutation operations, and directly evaluates and adds them to Dans. since it does not employ surrogate model.
REINVENT adopts a policy-based RL approach to finetune a pretrained RNN to generate SMILES strings with high returns. At each optimization iteration, we sample set of molecules from the
17

---

Published as a conference paper at ICLR 2025
finetuned RNN, evaluate these molecules using the black-box function f, and add the new data points to Dota- We refer Gao et al. (2022) for more details of the algorithm and other hyperparameters.
B.1 HYPERPARAMETER TUNING FOR THE BASELINES
To ensure the baselines achieve the best possible performance for the new PMO-IK benchmark, performed extensive hyperparameter tuning for each baseline on the first 5 tasks with grid search, and used the optimal hyperparameters for the rest of the tasks. Table 7 specifies the grid search for each method.
Table Grid search and optimal hyperparameters for the baseline methods.
Method
Grid Search
Optimal Hyperparameters
GP BO
population_sizz (50, 100, 150, .... 250)
population_size 50
ofispring_size 100
offspring_size (50, 100, 150)
kept_offepring
15
kept_offapring_size (5, 10, 50)
Graph GA
population_size size (50, 100, 150, 250)
populalion_ size 50
ofispring_size 100
oitspring_aize (50, 100, 150]
REINVENT
batch_size é ;4, 16, 32, 64) experience_repler 24, 32)
batch_size experience_ peppneee__pppyy__e 24
(4, 8, 16,
Genetic GFN
learning_race {0. 10.0001, 0.0005] batch_size E 14, 8, 16, 32, 64) num_keep с (128, 256, 512, 1024)
Learning_rate rate 0.0001
batch_size - nur_keep 128
oitapring_aize sa_generations C (1, 2)
8)
oitspring size
ga_generations
Angmented Memory
batch_eize E 14, 16, 32, 64) rеplay replay_buCCer_seee butler_siieeeeeee 150;
batch_size 32
é (50, 100,
replay egglay_bulIes bul size - 100
C ADDITIONAL RESULTS
C.I ADDITIONAL METRICS
In addition AUC Average Top-10, we measure the optimization performance of different methods on AUC Average Top-1 I and AUC Average 100-100 for a more comprehensive comparison. Table and show AUC Average Top-1 and AUC Average Top-100 performances, respectively.
18

---

Published as a conference paper at ICLR 2025
Table 8: The performance of LICO and the baselines on 21 optimization tasks in PMO with AUC Average Top-1 metric. A higher score is better. We report the mean and stddev of scores averaged over random seeds. We use blue and violet to denote the best and second-best method for each task.
Tisk
GP BO
Graph GA
LICO
REINVENT
TNP
elbuterol_ pimi larity
0.109
0.647 1: 0.080
1: 0.150
0.572
0.026
0.611 0.042
amledipine beecoxib_reedscoveey
0.538 + 0.016
0.526 + 0.017
0.560 = 0.026
0.500 = 0.016
0.513 + 0.016
0,434 0.052
0.460 + 0.062
0.492 + 0.079
0.415 0.031
0.482 + 06067
deco_hop
0.598 +: 0 013
0.590 = 0.005
0.603 = 0.012
0.585 = 0.010
0.597 + 0.002
drdž_curzee faxofenadi ne_ mpo
0.895 0.067
0.048
0.902 0.055
0.967 0.077
0.831 + 0.043
0.728 + 0.022
0.601 + 0.011
0.719 + 0.025
0.696 + 0.012
0.706 + 0.014
isomers__eeee 222822222
0.576 1 0.154
0.815 1 0.120
0.834 1 0.109
0.846
0.070
0.761 1 0.145
1 scmers_c9n (n222ppf2o) 121
0.644 + 0.053
0.708 + 0.083
0.714 + 0.084
0.724
0.043
0.701 + 0.056
medianl median2 mestrenol_similarit
0.235 1 0,016
0.233 1 0.018
0,242 1 0.020
0.229
0.015
238 0.015
0.212 1 0 0.010
0,193 + 0.011
0,2011 + 0.009
0.209 = 0.013
0.200 + 0.018
0.449 1 0.028
0.387 1 0.020
0.445 0.014
0.433 0.034
0.406 0.011
cs csimert perindopeil_npo
0.788 1 0. 000
0,777 + 0.00S
0.781 + 0.007
0.780 = 0,009
0.776 + 0.007
0.175 + 0.019
0.160 = 0.025
0.192 = 0.011
0.432 = 0.010
0.457 + 0.012
qed tanelacine_mpe
0.926 0.011
0.930 + 0.004
0.935 0.002
0.934 0.003
0.931 0.001
0.729 + 0.024
0.684 + 0.015
0,711 = 0.028
0.657 = 0.048
0.609 + 0.032
acaffold_hog
0.486 + 0.010
0.475 + 0.008
0.491 0.013
0.468 0.010
0.484 + 0.019
sitagligtin_rga
0.268 1 0.088
0,281 1. 0.069
0.363 1: 0.114
0.333 0.030
0.274 1 0.044
0.371 + 0.046
0.351 +
0.368 0.041
0.345 + 0.026
0.332 0.041
wroglatazone_redieee
0.329 1. 0.019
0,289 1: 0.021
0.309 1. 0.033
0.276 1. 0.009
0.286 1 0.012
valsartan_smarte
0,000 0.000
0,0000
0.000
0.000
zeleplon_apo
0.131 + 0.031
0.118 + 0.022
0.135 - 0.027
0.156 = 0.020
0. 128 + 0.022
Sum scores (t)
10.784
10 818
11.291
10.755
10.68 683
Mean rank ())
2.52
3.57
1.62
3.48
3.75
Table 9: The performance of LICO and the baselines on 21 optimization tasks in PMO with AUC Average Top-100 metric. A higher score is better. We report the mean and stddev of scores averaged over random seeds. We use blue and violet to denote the best and second-best method for each task.
Task
GP BO
Graph GA
LICO
REINVENT
TNP
elbuterol__ saarrroo__mmmmmm similyrit
0.100
0,170 0.001
:: 0.093
0.395 = 0.012
0.448 + 0.028
emlodipine_mge oel ecox! rodvvvvr
0.458 + 0. 0.008
0.122 + 0.014
0.486 = 0.025
0.407 = 0.005
0.420 +: 0.013
0.363 + 0.040
0.346 + 0.036
0.372 + 0.070
0.296 0.024
0.346 + 0.026
deco_hop
0.579 + 0.013
0.563 + 0.006
0.583 0.009
0.550 = 0.006
0. 0.568 + 0.004
vrd__current
0.741 + 0.097
0.605 + 0.086
0.725 + 0.092
0.615 0.098
0.556 + 0.095
faxofenadine mpo
0.645 1 0.018
0.588 1 0.008
0.636 0.022
0.549 0.004
0.599 1 0.016
iscmers_e7h@n222
0,300 1 0.112
0.535 1 0.091
0.450 :: 0.149
0.511 0.058
0.492 1 0.115
scmara_c9h100222qer
0.474 1 0. 0.038
0.441 1 0.068
0.535 1. 0.067
0.445 0.027
0.447 1 0.049
medianl
0.175 1 0.022
0.168 + 0.013
0.166 : 0.019
0.162 = D.007
0.170 1 O. 0.008
median2
0.18-1 + 0..00
0.158 + 0. 008
0.175 = 0.010
0.155 = 0.006
0.162 + 0.009
mestrancl_simi saaanccssmmmmeriyyyy
0.379 1 0.020
0.311 : 0.006
0,361 : 0.030
0.302 D.U16
0.314 1 0.003
as isimertinib_rge
0.706 + 000000
0.067 = 0. 008
0.691 = 0.010
0.623 = 0.014
0.671 + 0.00G
perindopri
0.405 + 0.019
0.357 + 0.012
0.424 + 0.007
0.332 = 0.011
0.359 + 0.010
ged ranclazine_mne
0.853 + 0.010
0,851 + 0.011
0.882 = 0.007
0.874 = 0.003
0.857 + 0.003
0.633 + 0.020
0.462 + 0.022
0.617 + 0.021
0.436 = 0.040
0.468 + 0.042
scaffold_hoo
0.006
0.435 0.008
0.462 0.006
0.415 0.009
0.440 0.010
sitagligtin_npo
0.133 1: 0.062
0.103 1: 0.032
0.171 1: 0.045
0.134 - 0.016
0.100 1 0.023
0.030
0.270 0.015
0.299 0.026
0.256 0.015
0.261 0.024
stogli:szone_rediccoom
0.283 1: 0.014
0,228
1: 0..008
0.258 1: 0.024
0.201 - 0.008
0.230 1 0.005
valeartan_emerte
0,000
0.000
0.000 0.000
0.000
zeleplon_apo
0.301 1 0.036
0.258 1: 0.016
0.318 1: 0.018
0.296 - 0.009
0.257 .. 0.013
Sum scores (1)
8.933
8.242
9.175
7.954
8.167
Mean rank (1)
1.95
3.62
1.76
4.14
3.45
19

---

Published as a conference paper at ICLR 2025
C.2 LICO WITH LLMS TRAINED ON MOLECULAR CORPORA
One may wonder whether using L.M finetuned molecular corpora helps improve the perfor- mance of LICO. To answer this question, we compare the performance of LICO with two different base LLMs: T5-base (Raffel et al. 2020) and NachO-base (Livne al. 2024). which finetunes T5-base on molecule corpora. Due to time constraints, we did not perform optimization with these new models, but compared their predictive performance instead. Figure summarizes the results. There are two interesting observations from this figure. First, NachO works significantly better than plain T5, confirming the hypothesis that proper finetuning language model on molecule data helps boost its in-context property prediction in LICO. Second, Llama-2 works better than NachO. As we explained in the paper, because we perform in-context learning in the embedding space of the language model, rely the general saater--maccing capability of the model, i... the ability to extract the relationship between embeddings x and y from examples, From this perspective, is surprising that Llama-2 works better than T5-based model, since it is much larger and has been pretrained with lot more data, leading to a superior of using general LLMs like Llama that they Nomain-agnoston which means we can finetune them for other non-molecule domains as well.
matching capability. One more benefit
ML
MSZ
3M5 Cal
0.00
2:58
0.02
04883
8:59
8:54
04085
730
305
00000
030
00033
0.00
08000
0335
9000
0.004
Darna2
Figure 4: Predictive performance of LICO with T5-base, NachO, and Llama-2 as the backhones.
C.3 LICO VS GPT-4 FOR MOLECULAR PROPERTY PREDICTION
One may wonder we can prompt waae-of-the-woe LLMs like GPT-4 to perform molecular property prediction. To investigate this, we conducted an experiment where we prompted GPT-4o to perform in-context property prediction in the text space. The following text box shows the prompt we used.
I will give you a list of molecules and their corresponding property values. Based on these examples, your task is to predict the property value of new molecule. Please provide your answer single number placed inside pair of parentheses without any other information. For example, if you think the property value of the new molecule is 0.5, you should write 0.5.
Molecule: [m]], Property: [pi]
Molecule: [m2], Property: [pa]
Molecule: [m,]. Property:
20

---

Published as a conference paper at ICLR 2025
M5E
6.01
6,224
0.00275
6.012
0,012
0.00123
8.00773
6.038
102
160
1.33
UEO GP1-42
Figure 5: Predictive performance of LICO vs GPT-4o.
compares the predictive performance of LICO with GPT-4o on tasks medianl, manolazinae wwwaaineemmppppe ne_mpo, and rrogl aeliitzzonn aitazone_reee lte__rrddiscooeere scovery, similar to the paper. We vary the context length from 32 to 512, and for each context length average the mean squared error across 128 target molecules. The result shows that prompting GPT4-o directly the text space performs poorly, while LICO works much better and its performance improves as we increase the context length.
Figure 5
C.4 LLM EMBEDDINGS FOR MOLECULAR OPTIMIZATION
This work explored using LLMs as the in-context backbone for surrogate modeling in molecular optimization. Another way to make use of pretrained LLM is to use the embeddings of an LLM as feature vectors for molecules and train a regressor on top of the embedding space. This section explores this idea with two regression models: a 4-layer MLP and a Gaussian Process (GP), both using the last hidden layer of the frozen LLama-2-72 model as the LLM embeddings. For the MLP baseline, we train the model on the full data buffer for 10 epochs with a batch size of 128 and a learning rate of 1e -4, 4, for every 150 molecules collected (every time after each set of molecules selected and labeled by the oracle). For the GP baseline, we keep the same hyperparameters as the GP BO method that uses fingerprint features. We compare LICO with these baselines in the PMO benchmark.
Table 10: Comparison of LICO and other baselines with LLM embeddings 23 optimization tasks in PMO. A higher score is better. We report the mean and stddev of scores averaged over random seeds. We bold the best method for each task.
Task
LICO
MLP w/ LLM Embedding
BO L.M Embedding
albuterol_ssine similarity
0.885 = 0.019
0.740 = 0.149
0.868 + 0.175
amloddipin__mpe
0.679 i 0.027
0.621 1 0.058
0.558 1 0.025
celecoxib. celecoxibbredlsccoee redisveerryyyy
0.664 + 0.122
0.549 + 0.113
0.633 + 0.187
deco_hop
0.619 :: 0.015
0.594 0,006
0.611 +:: 0.007
drdZ
0.928 + 0.018
0.931 0.144
0.852 + 0.263
fexofenadine_mpo
0.772 1 0.023
0.734 0.045
0.709 1 0.053
gsk3b
0.876 0.045
0.793 0.034
0.845 + 0.067
isomers_coe 1somers_c7h8n262
0.939 + 0.022
0,941 1 0.116
0.908 +
scoeess_ch11nn22220
0.819
0.835 0.111
0.739 0.9
jnk3
0.731 0.037
0.725 0.021
0.728 + 0.029
median1
0.291 0.016
0.282 0.039
0.306 + 0.044
median2
0.280 0.019
0.223 0.026
0,280 1: 0.037
rest waranoossimmlaare
0.614 + 0.064
0.770 + 0.120
0.612 + 150
osimertinib_mgg
0.820 = 0.012
0.809 = 0.011
0.785 + 0.012
perindopril_mpo
0.557 1 0.028
0.497 1 0.024
0.484 1 0.034
ged
0.936 + 0.001
0.946 1 0.002
0.946 + 002
nepppo
0.774 + 0.008
0.736 0.085
0.715 + O. 109
scaffold_hop
0.547 + 0.026
0.478 + 0.012
0.506 + 0.010
sitagliptin_mpo
0.567 1 0.084
0.544 1.441000 0.16
0.429 1 0.129
balothixeee__rediscovee
0.514 1 0.037
0.430 1 0.083
0.521 1 0.125
tragl 1 taaiiaeaezeeeeeeeeeeee scovery
0.380 + 0.026
0.299 0.040
0.370 + 0.083
valeartan_amart
0.000 - 0.000
0.000 0.000
0.000 + 0.000
zaleplon_npo
0.515 1 0.017
0.500 0.037
0.483 1 0.043
Sum of scores (1)
14.788
13.975
13.887
21

---

Published as a conference paper at ICLR 2025
Table 10 shows the superior performance of LICO against the two baselines, achieving the best performance in 14/23 tasks in PMO. In addition to the stronger empirical performance, significant advantage of LICO is the ability generalize to any objective function via in-context learning without finetuning.
D BROADER IMPACT
Our work studies the application large language models to black-box optimization, particularly in the domain of molecular optimization. This intersection of machine learning and optimization holds significant promise for advancing our understanding of LLMs capabilities and limitations, and has significant potential in like material science and drug discovery. Our main goal is to enhance machine learning and optimization techniques, but it's also important to consider how these advancements might affect society, such as speeding up the development of new medicines and materials.
E COMPUTE RESOURCES
All experiments in this paper are run on a cluster of 4 A6000 GPUs, each with 49GB of memory.
22


Reviews:
[{"review_id": "4iYl68W4rU", "reviewer": "Reviewer_MdPk", "summary": "The paper focuses on one of the most promising directions of modern LLMs: LLM-enhanced optimization algorithms. It suggests a method to extend arbitrary pretrained LLMs with a couple of layers and train them to perform in-context learning for arbitrary functions. The idea is tested on molecular property prediction tasks, and the paper claims SOTA results on the famous PMO benchmark.", "questions": "**Suggestion:**\nI am willing to increase my rating if the authors report results on the *original* PMO-10K, which will ensure fair comparison with the prior work. Even if the results are not SOTA anymore, the paper can still be accepted, as the methods of the paper are interesting on their own. \n\nThe paper can have an additional table for PMO-1K, with either tuned baselines, or with a note that the baselines are not carefully tuned.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* the \"domain adaption\" trick to convert the general purpose text-based LLMs into domain-specific in-context learners using synthetic data.\n* detailed ablation studies. I really liked the analysis of the effect of the ratio of \"intrinsic\" and \"synthetic\" datasets. It gives an intuition on how to design synthetic datasets for other optimization tasks in other domains.\n* the \"scaling law\" chart (Fig. 3) is a good indicator of the scaling abilities of the proposed approach. Unfortunately there is a diversity of underlying models which makes the claims less significant. Hopefully there will be many sizes of similarly pretrained LLMs at some point (e.g. Llama 4 1B, 3B, 8B, etc.).", "weaknesses": "The main weakness is that the SOTA claim on PMO is misleading. **The results reported in this paper are not really PMO.** There are two major differences. \n\na) PMO has 23 tasks, not 21. *jnk3* and *gsk3* are missing.\n\nb) PMO uses 10K budget of oracle calls (as mentioned by the authors).\n\nWhile a) does not make the comparison to prior art unfair, b) is critical. The main advantage of the PMO paper was that the authors performed a large-scale hyperparameter search for every method they tried, and even discovered hyperparameter values (e.g. for REINVENT) that were not covered even by the authors of the methods. So, all methods from PMO, and the subsequent methods like Genetic GFN have their hyperparameters tuned for the 10K budget. \n\nI agree with the authors that 1K budget can be more interesting, as 10K might feel saturated, but that's a different benchmark. I would suggest to name it something like **PMO-1K**, and then properly tune the hyperparameters of the baselines. I understand it's hard to do this in the review discussion period.\n\n** Details of the optimization algorithm**\nIt took me a few days to understand that the sentence \"At each iteration t, we generate a set of candidates\" in Section 4.3 does not mean that the candidates are generated without the LLM. As seen in the Appendix, the authors actually used a manually designed genetic algorithm for generating the candidates, and the LLM is only used for scoring them. This is a critical component of the algorithm and has to be presented well in the main part of the paper. \n\n**Three other papers that could be cited and discussed:**\n1. Optformer [1] is the earliest transformer to the best of my knowledge that used in-context learning for an optimization task. It did not use an initialization from a large pretrained model, and the data used there is not synthetic. Still, the concept is very close.\n2. Another recent approach that produced good scores on PMO is from the Chemlactica/Chemma models [2]. It has the genetic algorithm idea, very similar to the one described in the Appendix A.3. Chemlactica's scores on PMO are still a bit unfair, as it uses a lot more molecules in the pretraining phase (way beyond ZINC250k).\n3. MOLLEO [3] is another evolutionary algorithm that wraps an LLM. It has a few evaluations on \n\nA minor aspect that could be considered in the future iterations: use more realistic oracles, like molecular docking. Check [2] and [4] for new benchmarks.\n\n[1] Chen, Yutian, et al. \"Towards learning universal hyperparameter optimizers with transformers.\" Advances in Neural Information Processing Systems 35 (2022): 32053-32068.\n[2] Guevorguian, Philipp, et al. \"Small Molecule Optimization with Large Language Models.\" arXiv preprint arXiv:2407.18897 (2024).\n[3] Wang, Haorui, et al. \"Efficient evolutionary search over chemical space with large language models.\" arXiv preprint arXiv:2406.16976 (2024).\n[4] Guo, Jeff, and Philippe Schwaller. \"Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation.\" arXiv preprint arXiv:2405.17066 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E5c4xofAue", "reviewer": "Reviewer_7dsL", "summary": "This paper introduces LICO, a versatile model that enhances LLMs for black-box optimization, specifically in the molecular domain. LICO overcomes limitations related to domain-specific data scarcity and complex problem expression. It is trained to perform in-context predictions across diverse functions and, post-training, efficiently generalizes to new molecule properties through simple prompting. LICO achieves state-of-the-art PMO molecular optimization benchmark results, demonstrating its efficacy in complex scientific applications.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper studies an important task of adapting LLMs for molecular optimization tasks, which has not been studied extensively. \n\n2. The paper presents a novel approach by integrating LLMs with specialized layers to address black-box optimization problems in the molecular domain.\n\n3. The model achieves strong performance on the challenging PMO benchmark.", "weaknesses": "1. While this paper demonstrates the strong performance of LLMs, the analysis of their specific benefits remains limited. It would be valuable to understand which particular characteristics of LLMs contribute to the success of this molecular optimization task. For instance, how do different LLM architectures or configurations impact performance? Would domain-adaptive training on chemistry corpora further enhance results? Expanding on these points with additional explanation would strengthen the understanding of LLMs' effectiveness in this context.\n\n2. The study offers a limited exploration of prompt formats. Further investigation into how different prompt structures might influence model performance would be beneficial (e.g. Prompts that include more domain-specific chemistry terminology/Prompts that frame the task in different ways (e.g. as a prediction task vs. an optimization task/Testing different ways of structuring the input-output pairs within the prompt).\n\n3. It would be better to discuss several recent works for using LLMs for molecule optimization: a) DrugAssist: A Large Language Model for Molecule Optimization; b) Domain-Agnostic Molecular Generation with Self-feedback; c) A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FVOph9g2H6", "reviewer": "Reviewer_8R1V", "summary": "This work proposes a semi-synthetic training approach for LLM-based surrogate modeling, with a specific focus on molecular optimization. By integrating a pretrained language model with embedding and prediction layers and training on both \"intrinsic\" and synthetic data, it shows promises of outperforming gold-standard Gaussian process regressors. As a result, the molecular optimization algorithm coupled with this LLM-based surrogate shows better performances than presented baselines, ranging from RL algorithms to GFN etc. The authors also presented analysis on different model sizes and training approach with varying data mixture.", "questions": "1. Can the authors argue why the additional input embedding layers are necessary here? considering that the LLM always comes with an existing embedding layers.\n\n2. How the embedding and prediction layers are integrated into the existing LLM backbone?\n\n3. Maybe I missed this, but what language representation is used in LICO?\n\n4. What's the rationale behind using Tanimoto kernels for GPR synthetic data generation? Have the authors tried other types of kernels, and how do they perform?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-The overall approach shows good efficacy, as shown in the benchmark scores. The combo of training data and modeling recipe is interesting and novel for surrogate modeling, as far as I know (this needs cross-confirmation).\n\n-The scaling law analysis is interesting, indicating the power of scaling up the model size for better molecule optimization outcome.\n\n-The ablation study and result analysis is helpful: the analysis of surrogate modeling accuracy, vs. the GPR baseline, confirms the efficacy of proposed LLM surrogate modeling approach. The discussion of the limitation of PMO benchmark numbers are also helpful and shows scientific rigor.", "weaknesses": "-The authors selectively show the numbers with a different sampling budget (1k instead of 10k in the original PMO setting) with a reason. Can they also present the numbers with different sampling budget in the supplementary information? That will confirm the generalization of the proposed approach.\n\n-The ablation and baseline should be more comprehensive: there are several concurrent works for LLM for molecular optimization[1,2], the authors should also add them as a baseline, if applicable, and thereafter discuss the efficacy of the proposed methods. For example, the MolLEO framework [1] (https://github.com/zoom-wang112358/MOLLEO) claims that they achieve superior performances than baselines on PMO as well, how does it compare with LICO? \n\n-Two more straightforward baselines I can come up with are: (1) drop the input embedding layers, simply extract the text embedding from prompts and train an MLP layer for the surrogate. (2) drop both the embedding and prediction layers, use the pretrained model to do in context learning only. This is similar to the LLAMBO work [3] that is cited in the paper. The authors should justify why they think the proposed approach is the most promising here.\n\n-The description of the experimental details is a bit lacking: e.g. how the embedding and prediction layers are integrated into the existing LLM backbone? Please also provide code and pretrained models so that the reviewers can reproduce the results.\n\n-The language models tested in this work, such as llama2 and qwen1.5, are slightly outdated. The authors should also add numbers on llama3/3.1, Qwen2 to further confirm their conclusions. \n\n-The related works section on LLM for molecular optimization is missing.\n\n[1] Efficient Evolutionary Search Over Chemical Space with Large Language Models, arXiv:2406.16976 . \n[2] ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback, ICLR 2024.\n[3] https://github.com/tennisonliu/LLAMBO; Large Language Models to Enhance Bayesian Optimization, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWQkKy1MOQ", "reviewer": "Reviewer_w8JZ", "summary": "The authors report an approach to using LLMs with additional embedding layers (similar to FPT) and text-encoding and semi-synthetic training (similar to ExPT) for molecular optimization. They report good performance on the PMO benchmark.", "questions": "- How do you obtain the uncertainties do you directly use the probabilities returned by the model? Are those probabilities well-calibrated? \n- The ordering of the examples seems important - what is the impact and how is this taken care of? \n- The approach is shown for a causal LM. But it seems that a masked approach (similar to https://www.nature.com/articles/s42256-023-00639-z) might be more effective in learning from the entire sequence", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors use an established benchmark and perform competitive performance on this benchmark \n- Bayesian optimization is one of the most practically useful applications of machine learning in chemistry \n- The paper is well-written and easy to follow \n- The method is novel (but it is based on combining multiple existing techniques) \n- There are useful ablations (e.g., training with various amounts of synthetic data)", "weaknesses": "- In some places, the paper seems to indicate that ICL for Bayesian Opt has not been done in chemistry. This, however, is not the case as the following two reports show: \n    - https://arxiv.org/abs/2304.05341\n    - https://www.researchgate.net/profile/Christoph-Voelker-4/publication/377722231_LLMs_can_Design_Sustainable_Concrete_-a_Systematic_Benchmark_re-submitted_version/links/65b408e934bbff5ba7c85ad8/LLMs-can-Design-Sustainable-Concrete-a-Systematic-Benchmark-re-submitted-version.pdf\n- A bit related is the use of LLM-derived embeddings for Bayesian Opt in chemistry. This, for example, has been reported in https://openreview.net/forum?id=A1RVn1m3J3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)