Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Authors: Rong-Xi Tan
Ming Chen
Ke Xue
Yao Wang
Yaoyuan Wang
Fu Sheng
Chao Qian
Venue: ICML 2025 poster (2025)
Keywords: Universal optimization; Offline optimization
Primary area: optimization->zeroorder_and_blackbox_optimization

Abstract: The pursuit of universal black-box optimization (BBO) algorithms is a longstanding goal. However, unlike domains such as language or vision, where scaling structured data has driven generalization, progress in offline BBO remains hindered by the lack of unified representations for heterogeneous numerical spaces. Thus, existing offline BBO approaches are constrained to single-task and fixed-dimensional settings, failing to achieve cross-domain universal optimization. Recent advances in language models (LMs) offer a promising path forward: their embeddings capture latent relationships in a unifying way, enabling universal optimization across different data types possible. In this paper, we discuss multiple potential approaches, including an end-to-end learning framework in the form of next-token prediction, as well as prioritizing the learning of latent spaces with strong representational capabilities. To validate the effectiveness of these methods, we collect offline BBO tasks and data from open-source academic works for training. Experiments demonstrate the universality and effectiveness of our proposed methods. Our findings suggest that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for general-purpose BBO algorithms. The code is provided at https://github.com/lamda-bbo/universal-offline-bbo.

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Rong-Xi Tan 12
Ming Chen
12
Ke Xue 122
Yao Wang? Yaoyuan Wang² Sheng Fu Chao Qian 122
Abstract
1. Introduction
The pursuit of universal black-box optimization (BBO) al- gorithms, i.e., capable of adapting diverse problems, has been long-standing challenge (Wolpert & Macready, 1997;
The pursuit of universal black-box optimization (BBO) algorithms is longstanding goal. How- ever, unlike domains such as language or vision, where scaling structured data has driven gener- alization, progress in offline BBO remains hin- dered by the lack of unified representations for heterogeneous numerical spaces. Thus, exist- ing offline BBO approaches are constrained to single-task and fixed-dimensional settings, failing to achieve cross-domain universal optimization. Recent advances in language models (LMs) of- fer a promising path forward: their embeddings capture latent relationships in a unifying way, en- abling universal optimization across different data types possible. In this paper, we discuss multi- ple potential approaches, including end-to-end learning framework in the form of next-token pre- diction, as well as prioritizing the learning of la- tent spaces with strong representational capabili- ties. To validate the effectiveness of these meth- ods, we collect offline BBO tasks and data from open-source academic works for training. Ex- periments demonstrate the universality and effec- tiveness of our proposed methods. Our findings suggest that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for general-purpose BBO algorithms. The code is provided at https: http::://ggtttbbtttttccoc lamda bbo/ wow.aeerallooolllleeeeeeee
Chen et al., 2022b; Lehre & Lin, 2024). Traditional BBO methods, including Bayesian optimization (Garnett, 2023)
and evolutionary algorithms (Bück, 1996; Zhou et al., 2019),
excel in many tasks (Turner et al., 2021). However, these traditional BBO methods are not only confined to single- type and fixed-dimensional settings (Chen et al., 2022b; Song et al., 2025) but also struggle to leverage large-scale offiin data (Song et al., 2024c; Wei et al., 2024), heavily relying on online evaluations, which is a severe limitation that becomes particularly challenging in many real-world expensive scenarios.
Offliee black-box optimization (Trabucco et al., 2022; Kim et al., 2025) aims to identify optimal designs for an unknown objective function using only fixed and pre-collected dataset, which attempts to mitigate the issue mentioned above. Existing offline BBO approaches have shown im- pressive performance, such as in real-world engineering design (Tanabe & Ishibuchi, 2020; Kumar et al., 2022), pro- tein design (Khan et al., 2023; Chen et al., 2023b; Kim et al., 2023), and molecule design (Gaulton al., 2012; Stanton et al., 2022; Dara et al., 2022; Xue et al., 2024). Their suc- cess relies on two assumptions: (1) the training and test tasks share identical input dimensions and variable types, and (2) sufficient historical data exists for each task.
While real-world optimization problems often exhibit in- herent correlations across domains (Bai et al., 2023; Wang et al., 2024a), existing offline BBO methods face two chal- lenges: (1) insufficient training data for model develop- ment (Nguyen et al., 2023), and (2) fundamental inability to exploit cross-task relationships. The critical barrier lies in the heterogeneity of search spaces (Fan et al., 2024). This limitation forces practitioners to collect large datasets for every new problem, due to their inability to unify parametric representations across domains, which is unsustainable in practical scenarios characterized by sparse data availability and diverse task requirements. Thus, there is an urgent need for universal offline BBO.
"Equal contribution 'National Key Laboratory for Novel Soft- ware Technology, Nanjing University, China "School of Artificial Intelligence, Nanjing University, China AAvvanced Computing and Storage Lab, Huawei Technologies Co., Ltd., China. Corre- spondence to: Ke Xue <xuck Exuck@lamda.nig waanaanjj...du.nn.. Chao Qian Silanc@nju.eduucn>>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Fortunately, recent advances have demonstrated the feasibil- ity of universal optimization in string-based search spaces using language models (LMs) (Song et al., 2024a;b; Nguyen et al., 2024; Tang et al., 2025). By tokenizing numerical parameters into string sequences, L.-based optimizers out- perform traditional regression models in cross-task gener- alization, particularly when trained on multi-task datasets spanning diverse domains. However, there are still several critical gaps in universal offline BBO. First, prior work does not discuss distinct paradigms in detail, failing to clarify their relative strengths or compatibility. Second, these works neglect the unique requirements of offline BBO, where algo- rithms must avoid overfitting to limited historical data (Tra- bucco et al., 2021; Fu Levine, 2021; Yu et al., 2021; Chen et al., 2022a; Qi al., 2022: Dao et al., 2024b; Hoang et al., 2024). Besides, the geometric properties of learned latent spaces remain under-explored, despite their direct impact on optimization stability and sample efficiency.
function / + R. i.e. x arg maxxxxxxxx (x). This optimization process relies solely on the available dataset, with no online evaluations permitted during optimization. Specifically, an algorithm operates exclusively on a fixed dataset 'D = (i)) where each instance x; represents a design (e.g.. the composition of a DNA sequence), and its corresponding scalar value yi = f(x)) denotes the objective score (e.g., a specific property score of the designed DNA sequence). Besides, for offline BBO task, there usually exists task-level metadata m. which can distinguish it from other tasks and potentially hint the information of the unknown objective function f. Thus, similar to the multi-task regression case (Song et al., 2024a), an offline BBO task can be formulated as T TXX f. m))
A prevalent approach for offline BBO is the forward ap- proach, which first trains a surrogate model, typically a deep neural network fo X - IR parameterized by 9, to learn a scoring function for the designs where the output corresponds to the predicting score; then searches for the final design by maximizing the model's output. The scoring function can be learned by regression (Trabucco et al., 2021; Chen al., 2023a) or ranking (Tan al., 2025). We provide comprehensive related work in Appendix A, including the detailed forward and backward approaches of offline BBO, and LLM for BBO.
In this paper, we propose a universal string-based offline BBO framework, UniSO. We first introduce several com- ponents for UniSO, including string-based data representa- tion, and metadata formulation. Then, we use two model architectures for universal offline BBO, including token- targeted regressor (Song et al., 2024a) and targrie regressor (Nguyen et al., 2024), formulating two Vanilla UniSO variants, i.e., UniSO-T and UniSO-N, respectively. However, several issues exist when directly applying these Vanilla UniSO variants to offline BBO tasks, as shown in Fig. 3. To address these issues, we propose two improve- ments, i.e., embedding distribution alignment via metadata guidance and local embedding smoothness enhancement. Experiments demonstrate the universality and effectiveness of our proposed methods, particularly showing that UniSO- T achieves superior cross-task generalization through multi- task training on heterogeneous search spaces.
2.2. Universal Offline BBO
Recent advances in foundational models for black-box opti- mization research (Song et al., 2024a;b; Nguyen et al., 2024; Tang et al., 2025; Song & Bahri, 2025) have shown the po- tential of universal optimization in string-based space. Thus, we first extend conventional offline BBO to the universal setting. The goal of universal offline BBO is to simulta- neously address multiple tasks through a universal foun- dational model. Consider nt optimization tasks (T.) with three key characteristics: (1) Heterogeneous design spaces: Mixed-type parameters and varying dimensionality across tasks; (2) Divergent objectives: Unique optimization functions for each task; and (3) Task-specific metadata: Distinet auxiliary information accompanying each task.
Our findings reveal three key insights: (1) a unified rep- resentation through string-based based space enables universal offline BBO possible. (2) geometric regularization of em- bedding spaces significantly enhances optimization stability, and (3) metadata-guide -udiddd learning effectively bridges the domain gap between training and unseen test tasks. These advancements collectively overcome traditional barriers in universal offline BBO, paving the way for general-purpost optimization across various types and dimensions.
While more challenging than traditional single-task BBO, the universal approach offers two key advantages. First, it enables knowledge transfer between related tasks (c.g.. morphology optimization in Ant (Brockman et al., 2016) and D Kitty (Ahn et al., 2020)), overcoming the isolation assumption of conventional methods that process tasks inde- pendently. Second, it addresses the data scarcity challenge common in real- real-world applications (Nguyen et al., 2023). Traditional methods fail with limited historical data, while the universal offline BBO model can leverage cross-task patterns to guide optimization.
2. Preliminaries
2.1. Offline BBO
Given the design space X, where X' could be be CONTINUOUS, INTEGER, CATEGORICAL, or PERMUTATION, and a fixed offline dataset D, offline BBO (Trabucco et al., 2022; Kim et al., 2025) aims to seek an optimal design x" that maximizes black-box objective
2

---

Towards Universal Offiine Black-Box Optimization via Learning Language Model Embeddings
LLM is efficient and beneficial for BBO, enabling optimiza- tion in dynamic design spaces. Following (Nguyen et al., 2024), we represent each design by a JSON dictionary- like format, e.g,, design x (0, 1) can be represented as { ("x0":O, 0, "x1 1.
Model-Inner Search
Blck box Optiniter (e.E 60)
Source Data
Suggent Conwert
Unierral Pagressr
fome Ant Morphalnny
Supeccedyytor
Obu Medime inddra af Otu yal (Pwr3s))
Descrprione 26d sgi.
Nome TE Rind
Universal Begressce
As for the score y, details of handling y encompass token- based method and numeric-based method, which are deter- mined by modeling methods of the regressor, which we will introduce in Section 3.1.3.
100 1.701.1)100000
Data Preprocassing
100 64.
20: "0.
3.1.2. FORMULATION OF METADATA
Figure 1. Framework of universal string-based offiine BBO.
While the mentioned string-based data representation ex- hibits flexibility, it alone is insufficient for effectively dis- tinguishing between tasks and performing optimization in a multi-task suite. The integration of metadata into algo- rithms is quite essential for universal offline BBO, due to the following reasons: (1) from the NFL theorem (Wolpert & Macready, 1997), an algorithm tends to treat all possi- ble tasks uniformly equal given only (x, y) data without any priors, which cannot perform well over all tasks. Thus, expert priors (Hvarfner et al., 2022; 2024) or user-defined task task-specific metadata should be considered for optimiza- tion (Song et al., 2024b; Lindauer et al., 2024); (2) meta- data enables training acceleration (Gao al., 2025) and is crucial for task distinguishment when expert priors are limited (Chen et al., 2022b).
3. Method
In this section, we introduce potential methods to solve universal offline BBO. We hegin by outlining the general framework of universal string-based offline BBO in Sec- tion 3.1, where we will present two variants based on mod- cling. Then, we discuss the issues of these variants in Sec- tion 3.2, which motivate us to improve the framework via metadala guidance and smoothness enhancement in Sec- tion 3.3 and Section 3.4, respectively. In Section 3.5, we discuss how to balance multiple losses and apply the im- provement to the variants.
3.1. UniSO: Universal String-based Offline BBO
In this subsection, we present the general framework for universal offline BBO, which is illustrated in Fig. 1. The framework comprises four main components. Firstly, we convert the design-score offline data into string represen- tations due to the heterogeneity of different search spaces from different tasks. Next, we discuss the importance and formulation for metadata, which facilitates the subsequent universal multi-task regressor training intended for down- stream optimization. We propose two modeling variants for universal regressor instantiation based on different ways of handling objective scores, which are derived from recent advancements of string-based LLM for BBO (Song al., 2024a; Nguyen et al., 2024; Tang et al., 2025). Finally, we elaborate on our string-based search strategy within the model. This pipeline aligns with the forward approach in traditional offline BBO, which typically encompasses train- ing a scoring model followed by model-inner optimization, as discussed in Section 2.1.
Thus, we extend each string x by inserting its associated metadata m in the beginning. In order to maintain concision and preserve key information for optimization, we formulate the expressions of metadata m, consisting of three text- based task specifications: (1) task name to distinguish from other tasks; (2) brief description of the task with concise natural language summary; and (3) detailed specification of the optimization objective. We provide all the metadata we used in our experiments in Appendix D. We tokenize the (m, x) data using SentencePiece tokenizer (Kudo & Richardson, 2018) by default.
3.1.3. MULTI-TASK REGRESSOR TRAINING FOR UNIVERSAL OFFLINE BBO
The scoring model for offline BBO can be modeled by regression or ranking model. Here, we initiate the scoring model by a universal multi-task regressor for simplification. As shown in Fig. 2, we consider two variants of universal end-to-end regressors. UniSO-T and UniSO-N, which differ from how deal with the objective score y.
3.1.1. STRING-BASER DATA REPRESENTATION
To solve multiple offline BBO problems with heterogeneous design spaces simultaneously. traditional methods that re- strict the algorithm inside a fixed search scope are inapplica- ble, which calls for a more flexible representation for design- score pair data. Recently, Song et al. (2024a); Nguyen et al. (2024) have shown that string representation over x using
UniSO-T: Token-targeted regressor. UniSO-T is regres- sor with the objective of predicting the tokens. Specifi- cally, it encodes the score y into a list of tokens, trains a sequence-to-sequenne ruto-regressive model to predict the objective sequence, and then maps the sequence back to
3

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
of
[o] UnISO-T: Token- targoted
[b) UniSO I:: Nameric-tagetted [_04878988"" MSE
6075902 Mumkotc takonizer
Cross Entropy
Hog roSsE
Language Model (e.g, 15)
Language Model Embedder [e.g. 15 Embedder)
Longuagg Model Tokember Tusl 9.4 Nalt
Langrage Mocel Tokenber
Figure 3. t-SNE plots comparing embedding distributions of vanilla UniSO-T and improved UniSO-T on five Design- Bench (Trabucco et al., 2022) tasks. The embedding distribu- tions of vanilla UniSO-T show mixed and overlapping embeddings with a circular pattern, lacking clear task boundaries. Our im- proved method (right) achieves three improvements: (1) separating embeddings into distinct clusters for better task discrimination, (2) maintaining proximity between similar tasks (e.g., Ant and D' Kitty) to enable knowledge sharing between related domains, and (3) generating compact and smooth intra-task embedding dis- tributions for stable representations.
Figure2 2. Model structure of UniSO-T (left) and UniSO-N (right).
numerical values. A typical modeling of UniSO-T is Om- niPred (Song et al., 2024a), which tokenizes by digits using P10 encoding from (Charton, 2022) and trains a universal regressor based on next token prediction. Inspired by OmniPred, we tokenize y using P10 encoding and im- plement a lightweight variant of the T5 model (Raffel al., 2020). The model architecture incorporates Prefix-LM train- ing (Liu et al., 2018) and is trained using cross-entropy loss on our custom dataset, as the training data of OmniPred follows a distinct organizational structure and remains un- available.
3.1.4. STRING-BASED MODEL- INNER SEARCH
Upon completion of model training. the final designs are ob- tained through output maximization via model-inner search. Given the inherent challenges of computing exact gradi- ents in discrete string space compared to continuous nu- merical space, we adopt black-box optimization strategies for this search process. Our implementation primarily em- ploys Bayesian optimization (Garnett, 2023), which have demonstrated superior performance in gradient-free opti- mization scenarios. The technical specifications of our BO implementation. including acquisition function and code dependencies, can be found in Appendix B.2.
UniSO-N: Numeric-targeted regressor. UniSO-N is an- other type of universal multi-task regressor, which first em- beds the inputs into unified latent space, and then trains a downstream regressor to map from embeddings to nu- merical objective scores. Inspired by (Nguyen et al., 2024), which uses a pre-trained T5 encoder (Raffel et al., 2020) and trains an in-context regressor for Bayesian optimization, we also employ pre-trained T5-Small embedder ?, and train a MLP regressor mapping from embeddings to y with mean squared error (MSE). We also normalize y from different tasks using the same strategy in (Nguyen et al., 2024), which incorporates three steps: 1) utilize z-score normalization to shift the score of a single task; 2) implement a normal distribution fitting procedure on the subset of observations falling below the median value, utilizing a percentile-to-2 score transformation, reduce sensitivity to bad outliers; 3) transform the scoring distribution through sequential ap- plication of min-max scaling +-
3.2. Issues of the Vanilla UniSO
Although the typical methods for UniSO-T and UniSO-N have demonstrated success in online BBO (Nguyen et al., 2024), there still exist limitations for offline BBO. Firstly, both the regressors in UniSO-T and UniSO-N cannot effec- tively distinguish different tasks without billions of data. To understand this, we visualize the t-SNE (Maaten & Hinton, 2008) plots of the embeddings of vanilla UniSO-T in the left sub-figure of Fig. 3, where the latent embeddings of vanilla UniSO-T show overlapping and circular pattern, lacking clear boundaries to distinguish tasks.
V- Umis and logarithmic transformation for re-scaling. Besides, although regressor training in offline BBO utilizes global z-score normalization, global statistics like mean value or standard variation of the embeddings are inaccessible, and thus batch normalization applied to the embeddings before objective regression.
Besides, the embedding latent spaces for both UniSO-T and UniSO-N are not particularly designed for BBO, while properties of the latent spaces, like smoothness (Lee et al., 2023) or invariance (Qi et al., 2022), are usually important for downstream optimization.
For example, y = 1.31 = 131 x 10-2 is encoded into < <+><>><>><<><<--22 hhttps: huttpss//hugginngface. cot goog//oooleeeeeoooo t5/
Aiming at addressing these limitations, we introduce our proposed improvements in the following two subsections, which can be applied to both the modelings, and regulate
t5 small
4

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
the embedding space based on two different perspectives, metadata guidance and smoothness enhancement.
However, this contrastive loss would raise non-smoothness inside task. The input embeddings from a same task (i.e., with the same metadata) can occupy arbitrary positions within a localized region of the high- oigh-dimensional space, forming either distinct clusters or non-uniform distributions, as long as they maintain adequate contrast (i.e., cosine simi- larity in the loss function) with respect to embeddings with dissimilar metadata. In Section 3.4, we further enhance the local smoothness of embeddings from single task.
3.3. Embedding Distribution Alignment via Metadata Guidance
As discussed above, the embeddings exhibit strange shape and cannot distinguish different tasks. Thus, we adopt novel approach to align the distribution of input embed- dings with their corresponding metadata embeddings via contrastive learning. The core idca is to guide the learning process by encouraging input embeddings to exhibit sim- ilarity patterns that mirror those observed in the metadata space.
3.4. Local Embedding Smoothness Enhancement
Smoothness is crucial for neural network generaliza- tion (Nakkiran et al., 2019) and robustness (Weng al., 2018), and smoothness of latent representation is also im- portant for BBO (Zhang et al., 2020; Lee et al., 2023). For given task T, to enhance the local smoothness of embed- ding from TT, inspired by (Lee et al., 2023), we regulate the embedding space via the Lipschitz loss:
Specifically, we first encode the input strings through the encoder to derive their embeddings (TS encoder outputs for UniSO-T and embedder outputs for UniSO-N), while simul- tancously employing a pre-trained expert language model embedder (T5-Small by default) to generate embeddings for metadata. After implementing mean pooling on both embeddings, we transform these intermediate representa- tions through their respective nonlinear projection heads into a shared embedding space, yielding the final latent rep- resentations z" E R2nd and zz" E Raxd, where n denotes the batch size and d represents the dimension of the shared embedding space. Such a nonlinear projection is widely used in contrastive learning (Chen al., 2020).
Σ ISiCJENT
max 0,
1// M/
1
Z2 23 1/112
where Ny is the dataset size of task and I. represents the local Lipschitz constant. We set as the median value of the Lipschitz matrix, i.e., L = mediangii<jSnr ( ( la by default following (Lee et al., 2023). The Lipschitz loss increases the correlation between the Euclidean distance of latent embeddings and the differences in their corresponding objective scores, enforcing the local embeddings to remain a local smoothness similar to the corresponding scores. Then, we compute the weighted sum of Lipschitz losses across all tasks, where the weights are designed to balance the impact of varying dataset sizes of different tasks:
These projected representations then utilized com- pute the contrastive loss to align the distribution of input embeddings with their corresponding metadata embeddings:
Ccon N(N - 1) ISiKJEN
Σ
ij log
exp(
expl
jسر N
Cup =
Lup
NT.
where:
represents the cosine similarity between input embeddings, and similarly, si is defined as the cosine similarity of the metadata embedding; 87]
1=1
where nt denotes the number of tasks, and Σ Nr. resents the weighting coefficient for task T: to compensate for the disparity in dataset sizes across tasks. In our im- plementation, due to insufficient task-specific samples in individual batches of the shuffled training dataset, we utilize an unshuffled dataset that presents tasks sequentially for computing the Lipschitz loss.
NT. rep-
4 min, : (e
is the normalized metadata similarity; is the temperature parameter; and N is the data size.
max,, (+.) ) min, , (
The contrastive loss function enforces a dual objective by minimizing the KL divergence between the input embedding similarity distribution and the metadata-derived target distri- bution. An illustrative understanding of this loss function is that embeddings of the inputs with similar metadata remain proximate while dissimilar ones maintain distinct bound- aries. As shown in the right sub-figure in Fig. 3, compared to the vanilla UniSO-T case, our proposed improvement is capable of distinguishing dissimilar tasks with the clearer clusters, while similar tasks like Ant and D' D'Kitty remain close, enabling structured information sharing for similar tasks.
3.5. Loss Balancing and Application to UniSO
With the main loss Lmain to train the universal regressor (i.e., cross-entropy for UniSO-T or MSE for UniSO-N) and the mentioned regularization losses in hand, naïve approach to handle these losses is to sum them up directly. However,
In this work, smoothness refers to continuity in the embedding space.
5

---

Towards Universal Offline Black- Box Optimization via Learning Language Model Embeddings
simply summing them up would be effected by the scales of the losses, which focuses mainly on the loss with the largest scale during gradient backpropagation (Chen et al., 2018). Thus, a crucial problem is how to balance the effects of these losses for training?
and train the model using AdamW optimizer (Loshchilov & Hutter, 2019) for 200 epochs with batch size of 1 128. After the UniSO regressor is trained, we use BO as the default optimizer to search inside the model for 200 iter- ations, maintaining a population size that corresponds to the desired number of final solutions, which is set as 128 following prior works in offline BBO (Trabucco et al., 2022; Qian et al., 2025).
To solve this, inspired by MetaBalance (He et al., 2022) which normalizes the gradients of different losses based on the gradient norm, we employ similar yet simple technique to automatically balance the gradient of different losses based on the loss values. Specifically, for the main loss Mmain and regularization losses Lcon: Cup, we calculate the gradient as:
Tasks. We consider unconstrained tasks from two bench- marks for offline BBO, the popular Design-Bench (Tra- bucco et al., 2022) and the recently proposed benchmark SOO-Bench (Qian et al., 2025). Specifically, we select Ant Morphology (Brockman et al., 2016), D'Kitty Mor- phology (Ahn et al., 2020), Superconductor (Hamidich, 2018), TF Bind 8 and TF Bind 10 (Barrera et al., 2016) from Design-Bench and GTOPX 2,3,4,6 (Schlueter et al., 1021) from SOO-Bench. These tasks cover the CONT NNOUS and CATEGORICALI search spaces, and vary from low to high dimensional cases, Detailed description and properties of these tasks can be found in Appendix C.
Ohintal 00
0Lmain 00
Lmaai
HLcon
Leon + 6 00
mmain
aLiip
(1)
Lip + 00
where 0 represents the parameters of the model and d = 10-10; is a small constant added for numerical stability. This loss propagation mechanism adjusts the relative contribu- tions of auxiliary losses by scaling them with respect to the main loss, maintaining task balance while preserving the principal optimization direction.
4.2. Experimental Results
In this section, we present our experimental findings, aiming to answer the following RQs.
The balanced loss function is incorporated into both model- ing frameworks as follows. For UniSO-T modeling, since an end-to-end universal regressor is trained from scratch, we update the parameters f with respect to Eq. (1) during training where we view the last hidden state of the encoder model as embeddings. For UniSO-N modeling, since the vanilla approach employs pre-trained embedder without parameter updates during downstream numerical regressor training, we propose a two-stage approach: first fine-tuning the embedder via &Grgual DCam + bron aLuz similar 00
Is the string-based representation for designs versatile to offline BBO in the single task cases? Pre- vious works have demonstrated the potential of utilizing string- braeg- representation for universal regression (Song et al., 2024a; Tang et al., 2025) and traditional BBO algo- rithms (Nguyen et al., 2024). However, its versatility to offline BBO is still unexplored. Thus, for a sanity check, we compare the vanilla variants of UniSO-T and UniSO-N to the expert model trained with numerical inputs within a single offline BBO task in this RQ. For a fair comparison, the experts are instantiated as the same structure of the ob- jective regressor in UniSO-N, and we normalize the input designs using batch normalization, followed by model-inner search by both EAs and gradient ascent. Training details of the expert models can be found in Appendix B.
RQI:
to "0
to Eq. (1), and then training the regressor while keeping the embedder parameters frozen.
4. Experiment
In this section, we empirically study our proposed frame- work of universal string- offline BBO on various tasks. We first introduce the experimental settings and the tasks in Section 1, and then show the performance of UniSO and answer several important research questions (RQs) in Section 4.2.
As shown in Table 1, we find that UniSO methods which utilize string representation inputs: (1) are capable of solv- ing offline BBO, with most of the final scores exceeding the best score in offline dataset D(best), except for Ant and D' Kitty in UniSO-N; (2) show competitive results against the numeric-inpus experts, where UniSO-T achieves the an average rank of 2.000 among the four methods, performing the best on 3 of the 10 tasks and being the runner-up on of 10 tasks, while the best expert, BN BO (i.e., batch nor-
4.1. Experimental Settings and Tasks
Detailed settings. Following recent works in string- based LLM for BBO (Song et al., 2024a; Nguyen et al., 2024), we use a lightweight version of encoder-decoder T5 mod- els (Raffel et al., 2020) for both the UniSO-T and UniSO-N variants. For UniSO-N series, we instantiate the downstream regressor as a MLP with two hidden layers of 2048 units
'Following recent works in offline BBO (Tan et al., 2025; Yun et al.. 2024), we exclude three tasks from Design- Bench, and provide detailed explanation in Appendix C.I.
6

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Table 1. Un-normalized scores in unconstrained tasks from Design- Beiign--eenchace and SOO-Bench, where the best and runner-up results on each task are Blue and Violet, respectively. D(best) denotes the best score in the offline dataset and BN represents batch normalization for numerical input designs. Both numeric- numeric-input experts and string-inpol UniSO methods are trained within a single task.
Numeric-input Experts
String-input UniSO
Task
D(best)
BN + BO
BN + Grad
UniSO-T
UniSO-N
Ant D'Kitty
165.326
241.350 + 288.922
229.462 # 165.869
385,945 95.402
103.669 + 263.572
199.363
102.972 + 130.192
183.263 = 62.436
243.428 20.137
-4.674 4 9.298
Superconductor
74,000
83.884 = 4,099
97.137 = 6.113
79.783 0,507
80.129 1: 3.320
TF Bind 8
0.439
0.898 + 0.088
0,959 + 0.023
0.919 # 0.033
0,502 # 0.064
TF Bind 10
0.005
0.454 # 0.091
0,888 + 0.229
0.776 = 0.101
0,390 = 0,227
GTOPX 2
195.586
-74.763 + 8.577
-128.310 + 15.616
-73.484 a: 8.081
-72.981 20.446
GTOPX 3
-151.190
40.104 + 2.748
-151,190 # 0.000
.1.952 # 8.451
-58.933 14.143
GTOPX 4
-215.716
-87.976 + 3.497
-215.710 # 0.000
-75.022 # 12.215
-113,715 # 15.528
GTOPX 6
-112.599
49.660 + 6.170
-112.599 0.000
-50.391 +: 4.048
-48.771 # 3.725
Avg. Rank
/
2.333 # 0.667
2.667 # 1.333
2.000 + 0.943
3,000 = 1.155
malization for numerical input designs and using BO as the model-inner optimization algorithm), only ranks 2.333 on average. However, UniSO-N demonstrates inferior results compared numeric- input experts. This may be because learning from LM embeddings numerical objective space is relatively challenging. We also provide results of UniSO using evolutionary algorithms as the model-inner optimizer in Table 18 in Appendix E. Under the batch normalization setting, the experimental results demonstrate that employing evolutionary algorithms for search within expert models with numerical input outperforms gradient ascent.
zero-shot and few-shot generalization ability (Nguyen et al., 2023) of improved UniSO-T and UniSO-N on tasks that are unseen in the training data. Specifically, we choose three tasks RobotPush, Rover (Wang et al., 2018), and LunarLan- der (Brockman et al., 2016). Details of these tasks can be found in Appendix C. For zero-shot setting, concatenate the metadata and the string designs suggested by model-inner optimizer and maximize the model's 's out- put. For few-shot setting, we first use the few-shot data to fine-tune the universal regressor using the main loss (i.e., cross-entropy for UniSO-T and MSE for UniSO-N) and SGD optimizer with learning rate of x 10 for 5 epochs, and then search for final designs. We use the data provided by (Wang et al., 2024a) and the poorest 100 pairs of data to construct the few-shot dataset.
directly
RQ2: How do the universal string-based offline BBO methods perform across wide range of offline BBO tasks? In Table 2, we report the results of our main ex- periments, where UniSO-T and UniSO-N are compared to D(best) and single-task experts (numeric-input MLP), which are trained after batch normalization. The term "Im- proved" represents our improvements to regularize the latent space as introduced in Section 3. Among all the considered tasks, we find that: (1) most offline BBO tasks could benefit from UniSO methods, since the results of both the vanilla and improved versions of UniSO-T and UniSO-N exceed D(best), except for three cases; (2) UniSO methods are com- petitive to single-task experts, and the improved UniSO-T achieves better empirical results than the experts; (3) the im- proved components (i.e., embedding distribution alignment and local smoothness enhancement) contribute lot both UniSO-T and UniSO-N, with generally better results and average ranks. Besides, we show the t-SNE results of im- proved UniSO-T in the right sub-figure of Fig. 3. Compared to the vanilla one, our improved method cxhibits better dis- tribution for task distinguishment while similar tasks remain close for information sharing, and shows smoother local cluster embeddings in the latent space.
As shown in Fig. 4, both improved UniSO-T and improved UniSO-N show good generalization ability. The zero-shot results significantly exceed D(best) and the few-shot results are even better than that of the single-task expert based on z-score normalization, which is stronger expert than that based on batch normalization.
Zoro shot and For shot Perfomance
Zero shet Few shot Pe Performance
2.15
mtail
Stedl
F1.00
6.30
Figure 4. Normalized performance comparison between zero-shot inference and few-shot finetuning of UniSO-T (left) and UniSO-N (right) across three tasks (RobotPush, Rover, and Lunarl.ande)) The light blue hars represent zero-shot performance, the dark blue bars show the improved performance after few-shot finetuning. The red dashed line denotes the best score D(best) in the few-shot offline dataset.
RQ3: Can UniSO generalize to unseen tasks? As string- based universal online BBO methods show impressive gen- eralization ability (Nguyen et al., 2024), we also examine the
7

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Table 2. Un-normalized scores in unconstrained tasks from Design-Benche and SO0-Bench, where the best and runner-up results on each task are Blue and Violet, respectively. D(best) denotes the best score in the offline dataset and BN represents batch normalization for numerical input designs. Single-task experts are trained within one task, while UniSO-T T and UniSO-N are done in multi-task manner.
Single-task task Numeric-inpui Experts
UniSO-T
UniSO-N
Task
D(best)
BN + BO
BN Grad
Vanilla
Improved
Vanilla
Improved
Ant
166.326
241.350 + 288.922
229.462 + 165.869
446.283 + 20.635
455.658 # 39.188
260.058 + 280.526
381.787 88,907
D' Kitty
199.363
102.972 + 130.192
183.263 = 62.436
168.263 104.068
222.007 = 33.677
4.872 13.718
42.215 # 71.095
Superconductor
74,000
83.884 4,099
97.137 = 6.113
81.047 = 6.476
82.642 + 3.467
78.791 = 2.981
82.033 # 3,412
TF Bind
0.439
0.898 0.088
0,959 0.023
0.870 0.085
0.857 0.069
0.739 0.101
0.856 0.043
TF Bind 10
0.005
0.454 + 0.091
0.888 0.229
0.929 0.802
0,944 = 0.794
0,471 = 0.150
0.528 = 0.156
GTOPX
-195.586
.74.763 8.577
-128.310 15.616
-68.526 + 14,058
-90,106 8.223
-94.656 25.60225.00 25.608
.88.063 29.409
GTOPX
-151,190
-40.104 + 2.748
-151,190 1 0,000
-47.440 6,249
-44.427 13.456
-63.007 12.559
-48,488 14.797
GTOPX
-215.716
-87,976 + 3,497
-215.710 0,000
-74.070 = 13.131
-74.779 11.032
-111,437 + 28,417
.94.579 11.591
GTOPX 6
-112.599
.49.660 * 6,170
-112.599 + 0.000
-50.314 + 6,481
-48.244 * 5,766
-46.056 * 11,547
-52.672 + 9.042
Avg. Rank
3.111 + 1,523
4,111 2.183
2.667 1.247
2.222 1.133
4.778 1.474
4.111 + 0.737
Is it necessary to train from pre-trained LM for UniSO-N? As pre-trained language models are widely used in recent string-based optimization or regression (Nguyen et al., 2024; Tang et al., 2025), we investigate whether pre- trained initialization is necessary for UniSO-N by compar- ing UniSO-N trained from scratch with that from a pre- trained checkpoint. The results in the left subtigure of Fig. 5 show that UniSO-N trained from scratch consistently out- performs the pre- pre-trained model. The OOD rank correlation denotes the rank correlation between between predicted and ground-truth objective scores in OOD regions, which is a crucial metric to evaluate the effectiveness of the surrogate model for offline BBO (Tan et al., 2025). Higher rank cor- relation indicates better surrogate model for offline BBO. The comparison in the right subfigure of Fig. 5 shows that UniSO-N trained from scratch consistently performs better than that from a pre-trained checkpoint. The pre-trained models initially demonstrate poor performance, and there is only little improvement later on. This may be due to pre-trained LMs' harmful biases for numerical optimization. We will elaborate on this in the next RQ
RQ4:
RQ5: Do LM priors indeed do harm to numeric opti- mization? To better understand LMs harmful priors for numerical optimization, we visualize the attention weight distribution (Allen-Zhu & Li, 2023) across input tokens. On the top two subfigures in Fig. 6, we visualize pre-trained T5 embedder's attention weight distribution on the TF Bind 10 task (detailed results of more tasks can be found in Ap- pendix F). We can observe that pre-trained T5 embedder exhibits strong bias towards language grammar structural tokens (especially EOS tokens), while numerical tokens that are crucial for optimization are assigned with limited at- tention. In contrast, embedder trained from scratch shows a more balanced attention distribution with stronger focus on numerical tokens. This difference in attention assign- ment provides strong evidence for LMs' harmful biases for numeric optimization. To better understand how different LMs perform, in the next RQ, we will discuss the attention mechanisms of different LMs in detail.
RQ6: How do different LMS assign attentions? We fur- ther investigate whether the attention mechanism in T5 still exists in other pre-trained LMs. Specifically, we implement Qwen2.5-1 (Qwen Team et al., 2024) and DeepSeek- RI-Distill -ertiill-wee--1.33 1.5B (DeepSeek-Al et al., 2025)* 6 on TF Bind 10 and GTOPX 6 tasks, which are shown in the bottom two subfigures of Fig. 6 and Appendix F, respectively.
Normal and Performance comparison
Comparison of DOD Rank ation
oiberi
Powe smerk
000 Runk Corciattion
62
The results reveal two key findings. Firstly, different from the T5 cases, Qwen2.5-1.55 and DeepSeek-R1-D -Distill- Qwen-1.5B demonstrate more balanced attention patterns different input components. Specifically, these two mod- els show higher attention weights on both the metadata and numerical tokens, capturing more useful information rather than mainly focusing on grammar tokens. Besides, com- pared to the Qwen base model, DeepSeek-PR R assigns more attention on numeric-related tokens, showing greater capa-
Tom portraine
23
Twining Steps
Figure 5. Left: Normalized performance comparison between UniSO-N trained from scratch (blue) and UniSO-N with pre- trained embedder (yellow) across four tasks in S0O-Bench. The red dashed line denotes the best score D(best) in the offline dataset. Right: Comparison of OOD Spearman rank correlation on the Su- perconductor task between UniSO-N from pre-trained embedder (blue) and from-scratch (red). The OOD dataset is constructed following (Chen et al., 2023a). Higher correlation indicates better surrogate model for offline BBO (Tan et al., 2025).
httiss:/hhugiingaccss /wwe////w////eeeeeeeeeeeeee 2wt.m//n.n222...55 5-1.53 co/ wttpc//peeeeeeaa...e.eeaee wwtestiiii-wween-11 1,5B
8

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Average Attention Distrbutionn of T5-Small on TF Bind 10
Average Attention Distribution T5-Small from scratch on TF Bind t0
9.10
0:00
ebertie decepesn
Componenta
Componeraa
Average Attention Distribution Owen2.5-1.58 55555 10
Average Distribution DeepSeek-Rtt-ieilee
Nomaaess Ictendion Zcre
8.00
Components
Componanta
Figure 6. Attention weight distribution comparison of different models on the TF Bind 10 (Barrera et al., 2016) task: pre-trained T5-Smal embedder (top left). T5-Smal embedder trained from scratch (top right), pre-trained Qwen2.5-1 (bottom left), and pre-trained DeepSeek- R1-k--R--Dssiii RRI--istill-ween--D Qwen-1.5B (bottom right). More visualization results on other tasks provided in Appendix F.
bility on optimization. This may come from the training data of DeepSeek-R1, which contains mathematical content aiming to enhance model's 's reasoning ability. The result suggests that LMs with stronger mathematical capabilities may be better for numerical optimization.
formance for universal offline BBO is a crucially important future work.
Additional analysis and ablation studies. We provide ad- ditional results in Appendix E. Specifically, we provide the computational cost of UniSO in Appendix E.2, and explore the integration of in-context regressors, c.g., UniSO-N with Transformer neural processes (TNPs; Nguyen Grover, 2022; Nguyen et al., 2023; 2024) in Appendix E.3. Fur- thermore, we discuss the influence of different model sizes in Appendix E.4. To further understand the contribution of each component of UniSO, in Appendix E.5, we con- duct ablation studies on metadata quality, loss components, loss balancing strategies and model-inner search algorithms. Note that recently LLM-based automatic heuristics design shows impressive performance in many fields (Romera- Paredes al., 2024; Novikov al., 2025; Zheng al., 2025). In Appendix E.7, we investigate the usage of LLM- based heuristics (Liu et al., 2024a;b;c) to further improve performance via metadata and appending auxiliary loss.
RQ7: Is the berf--erro UniSO variant comparable to the well-studied expert single-task offline BBO meth- ods? In Table and Table 2, we use batch normalization for numerical inputs single-task expert methods for fair comparison. However, more widely-used practice in the field of offline BBO is to employ a global z-score normaliza- tion for inputs. Thus, to better understand the performance of UniSO methods, in Table 8 in Appendix 1, we compare improved UniSO-T, the best-performing universal offline BBO approach Table 2, to a wide range of single-task expert offline BBO methods ods are mainly based z-score normalization for input solutions. Here the improved UniSO-T method is trained solely using Benchh tasks, and the detailed results of compared methods are referred from (Tan et al., 2025). According to the results in Table 8, UniSO-T achieves an average rank of 9.8 across 21 expert single-task offline BBO methods, outperforming some recently proposed methods, which are specifically designed for single-task offline BBO, and even achieving the best results on TF-Bind- 10. Overall, there is still improvement room for UniSO methods, com- pared to state-of-the-art single-task offline BBO methods. Thus, how to propose better techniques and improve per-
Design-Bench. These meth-
5. Conclusion
In this paper, we propose UniSO to overcome barriers in uni- versal offline BBO by unifying string-based representation, latent space regularization, and metadata-guins -uidded learning. Extensive experimental results show the universality and effectiveness of UniSO. Future works can include training on more data and exploring in-context learning for enhanced performance in diverse real-world scenarios.

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Impact Statement
Barrera, L. A., Vedenko, A., Kurland, J. V., Rogers, J. M., Gisselbrecht, S. S., Rossin, E. J., Woodard, J. C., Mari- ani, L., Kock, K. H., Inukai, S., Siggers, T., Shokri, L., Gordān, R., Sahni, N., Cotsapas, C., Hao, T., Yi, S. S., Kellis, M., Daly, M. J., Vidal, M., Hill, D. E., and Bulyk, M. L. Survey of variation in human transcription factors reveals prevalent DNA binding changes. Science, 351 (6280):4500-144, 2016.
This paper presents work whose goal is to advance the field of large language models and black-box optimization. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgement
The authors thank anonymous reviewers for their insightful and valuable comments: ICML. Reviewer 4Ajw for suggest- ing the usage of BO model-inner optimizer, mechanism interpretable analysis and LLM- heuristics, Reviewer GL3R for suggesting analysis of pre-trained checkpoint and model size, Reviewer TeSq for integration of in-context re- gressor, Reviewer F6Mo for comparison analysis between UniSO-T and UniSO-N, and ICLR FM-Wild workshop re- viewers for constructive discussion on future works. This work was supported by the National Science and Technol- ogy Major Project (2022DD01 16600), the National Science Foundation of China (62276124, 624B 1025, 624B2069), the Fundamental Research Funds for the Central Univer- sities (14380020), and Young Elite Scientists Sponsorship Program by CAST for PhD Students. The authors want to acknowledge support from the Huawei Technology cooper- ation Project.
Blank, J. and Deb, K. pymoo: Multi-objective optimization in Python. IEEE Access, 8:89497-89509, 2020.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAl Gym. arXiv: /606001560 2016.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.. Chess, B., Clark, J., Berner, C.,, McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 1877 1901, Vancouver, Canada, 2020.
Charton, F. Linear algebra with Transformers. Transactions on Machine Learning Research, 2022.
References
Ahn, M., Zhu, H., Hartikainen, K., Ponte, H., Gupta, A., Levine, S., and Kumar, V. ROBEL: Robotics benchmarks for learning with low-cost robots. In Proceedings of the 4th Conference Robot Learning (CoRL), pp. 1300- 1313, Virtual, 2020.
Chemingui, Y., Deshwal, A., Hoang, T. N., and Doppa, R. Offline model-based optimization via policy-guided gradient search. Proceedings of the 38th AAAI Confer- ence on Artificial Intelligence (AAAI), pp. 11230-11239, Vancouver, Canada, 2024.
Allen-Zhu, Z. and Li, Y. Physics of language mod- els: Part 1, learning hierarchical language structures. arXiv:2305. /0 13673, 2023.
Chen, C.,, Beckham, C.,, Liu, Z., Liu, X., and Pal, C. Ro- bust guided diffusion for offline black-box optimization. Transactions on Machine Learning Research, 2024.
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Authropic Al Technical Report, 2024.
Chen, C. S., Zhang, Y., Fu, J., Liu, X. S., and Coates, M. Bidirectional learning for offline infinite-width model- based optimization. In Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 29454 09454-29467, New Orleans, LA, 2022a.
Bäck, T. Evolutionary Algorithms in Theory and Prac- tice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, 1996.
Chen, C. S., Beckham, C., Liu, Z., Liu, X. S., and Pal, C. Parallel-mentoring for offline model-based optimization. In Advances in Neural Information Processing Systems 37 (NeurIPS), pp. 76619-76633 New Orleans, LA, 2023a.
Bai, T., Li, Y., Shen, Y., Zhang, X., Zhang, W., and Cui, B. Transfer learning for Bayesian optimization: A survey. arXiv:2302.059977 2023.
Balandat, M., Karrer, B., Jiang, D. R., Daulton, S., Letham, B., Wilson, A. G., and Bakshy, E. Botorch: A frame- work for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS), pp. 11524-21538, Virtual, 2020.
Chen, C. S., Zhang, Y., Liu, X. S., and Coates, M. Bidi- rectional learning for offline model-basee biological se- quence design. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 5351- 5366, Honolulu, HI, 2023b.
10

---

Towards Universal Offline Black-Boo Optimization via Learning Language Model Embeddings
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations. In Proceedings of the 37th International Con- ference on Machine Learning (ICML). pp. 1597-1607, Virtual, 2020.
Frazier, P. I. A tutorial on Bayesian optimization. arXiv:/8011 /800:/281,. 2018.
Fu, J. and Levine, S. Offline model-bused optimization via normalized maximum likelihood estimation. In Pro- ceedings of the 9th International Conference on Learning Representations (ICLR), Virtual, 2021.
Chen, Y., Song, X., Lee, C., Wang, Z., Zhang, Q., Dohan, D., Kawakami, K., Kochanski, G., Doucet, A., Ranzato, M., Perel, S., and de Freitas, N. Towards learning uni- versal hyperparameter optimizers with Transformers. In Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 32053-332222 New Orleans, LA, 2022b.
Gao, T., Wettig, A., He. L., Dong, Y., Malladi, S., and Chen, D. Metadata conditioning accelerates language model pre-training. arXiv:2501.019:
2025.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can Transformers learn in-context? A case study of simple function classes In Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 30583-30598, New Orleans, LA, 2022.
Chen, Z., Badrinarayanan, V., Lee, C.-Y., and Rabinovich, A. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 794-803, Stockholm, Sweden, 2018.
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T. Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. Conditional neural processes. In Proceed- ings of the 34th International Conference on Machine Stockholm, Sweden,
Dao, M. C., Nguyen, P. L., Truong, T. N., and Hoang, T. N. Incorporating surrogate gradient norm to improve offline optimization techniques. In Advances in Neural Informa- fion Processing Systems 38 (NeurIPS), pp. 8014 8046, Vancouver, Canada, 2024a.
Learning (ICML), pp. 2018.
Garnett, R. Bayesian Optimization. Cambridge University Press, 2023.
Dao, M. C., Nguyen, P. L.. Truong, N.N., and Hoang, T. N. Boosting offline optimizers with surrogate sensitivity. In Proceedings of the 41 International Conference on Ma- chine Learning (ICML), pp. 10072-10090, Vienna, Aus- tria, 2024b.
Garnett, R., Osborne, M. A., and Hennig, P. Active learning of linear embeddings for Gaussian processes. In Proceed- ings of the 30th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 230-239, Quebec, Canada, 2014.
Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. M., and Ahsan, M. J. Machine learning in drug discovery: A review. Artficial Intelligence Review, 55(3):194--1999, 2022.
Gaulton, A., Bellis, L. J.. Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., and Overington, J. P. ChEMBL: A large-scale bioactivity database for drug discovery. Nu- cleic Acids Research, 40(D1):D1 100-D1 107, 2012.
DeepSeek-AI, AI, Guo, D., Yang. D., Zhang, H., Song, J., Zhang. R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. DeepSeek-R1. Incentivizing reasoning capability in LLMS via reinforcement learning. 1rXiv:2501.12948, 2025.
Gong, Y.J.,, L., Zhou, Y., Li, Y., Chung, H. S.-H., Shi, Y.-H., and Zhang, J. Genetic learning particle swarm optimization. IEEE Transactions on Cybernetics, 46(10): 2277-2290. 2015.
Dery, L. M., Friesen, A. L.. Freitas, N. D., Ranzato, M., and Chen, Y. Multi-step planning for automated hyper- parameter optimization with OptFormer. In Foundation Models for Decision Making Workshop at NeurlPS'22. New Orleans, LA, 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. In Advances in Neu- ral Information Processing Systems 27 (NeurIPS), pp. 144, Montreal, Canada, 2014.
Fan, Z., Han, X., and Wang, Z. Transfer learning for Bayesian optimization on heterogeneous search spaces. Transactions on Machine Learning Research, 2024.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. 182.0558 2018.
Fannjiang, C. and Listgarten, J. Autofocused oracles for model- based design. In Advances in Neural Informa- tion Processing Systems 33 (NeurlPS), pp. 12945-12956 Virtual, 2020.
Hamidieh, K. A data-driven statistical model for predicting the critical temperature of a superconductor. Computa- tional Materials Science, 154:346-355 2018

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Hansen, N. The CMA evolution strategy: A tutorial. arXiv: 1et.:.104.007700 2016.
the 13th International Conference on Learning Represen- tations (ICLR), Singapore, 2025.
Hansen, N., Amold, D. V., and Auger, A. Evolution strate- gies. In Springer Handbook of Computational Intelli- gence, pp. 871-898. Springer, 2015.
Kennedy, J. and Eberbart, R. Particle swarm optimization. In Proceedings of International Conference on Neural Networks (ICNN), pp. 1942-1948, Perth, Australia, 1995.
He, Y., Feng, X., Cheng, C., Ji, G., Guo, Y., and Caverlee, MetaBalance: Improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. In Proceedings of the ACM Web Conference 2022 (WWW), pp. 1205-2215, Lyon, France, 2022.
Khan, A., Cowen- A. I., Grosnit, A., Deik, D.-G.- X., Robert, P. A., Greiff, V. Smorodina, E.,, Rawat, P., Dreczkowski, K., Akbar, R., Tutunov, R., Bou-Ammar, D., Wang, J., Storkey, A., and Bou-Ammar, H. Toward real-world automated antibody design with combinatorial Bayesian optimization. Cell Reports Methods, 3(1), 2023.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Improving neural net- works by preventing co-adaptation of feature detectors. arXiv: 207.0580, 2012.
Kim, M., Berto, F., Ahn, S., and Park, J. Bootstrapped training of sonde-cond generator for offline design of biological sequences. In Advances in Neural Informa- tion Processing Systems 36 (NeurlPS), pp. 67643-67661. New Orleans, LA, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Advances in Neural Information Pro- cessing Systems 33 (NeurIPS), pp. 6840-6851, Virtual, 2020.
Kim, M., Gu, J., Yuan, Y., Yun, T., Liu, Z., Bengio, Y., and Chen, C. Offline model-based optimization: Comprehen- sive review. arXiv:2503. 17286, 2025.
Hoang, M., Fadhel, A., Deshwal, A., Doppa, J., and Hoang, T. N. Learning surrogates for offline black-box optimiza- Proceedings of the 41st International Conference on Machine Learning (ICML),
Kong, L.., Du, Y., Mu, W., Neklyudov, K., Bortoli, V. D., Wu, D., Wang, H., Ferber, A., Ma. Y-A.,, Gomes, C. P., and Zhang, C. Diffusion models as constrained samplers for optimization with unknown constraints. In Genera- tive Models for Decision Making workshop at ICLR 24, Vienna, Austria, 2024.
tion via gradient matching.
pp. 18374-18395 Vienna, Austria, 2024.
Hollmann, N., Müller, S., Purucker, L.., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, (37(8045):31933222 2025.
Krishnamoorthy, S., Mashkaria, S. M., and Grover, A. Diffu- sion models for black-box optimization. In Proceedings of the 40th International Conference on Machine Learn- ing (ICML), pp. 17842-17857 Honolulu, HI, 2023.
Hvarfner, C., Stoll, D., Souza, A., Nardi, I.,, Lindauer, M., and Hutter, F. nBO: Augmenting acquisition functions with user beliefs for Bayesian optimization. In Proceed- ings of the 10th International Conference on Learning Representations (ICLR), Virtual, 2022.
Kuba, J. G., Abbeel, P., and Levine, S. Cliqueformer: Model-based optimization with structured Transformers. 2rXiv:2410.13//00 2024a.
Hvarfner, C., Hutter, F., and Nardi, L. A general framework for user-guided Bayesian optimization. In Proceedings of the 12th International Conference on Learning Repre- sentations (ICIR), Vienna, Austria, 2024.
Kuba, J. G., Uehara, M., Levine, S., and Abbeel, P. Func- tional graphical models: Structure enables offline data- driven optimization. In Proceedings of the 27th Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), pp.
Izzo, D. Global Optimization and Space Pruning for Space- craft Trajectory Design, pp- 178-199. Cambridge Uni- versity Press, 2010.
Valencia, Spain, 2024b.
Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pp. 66- 71, Brusscls, Belgium, 2018
Izzo, D. and Manuel López-lbäñtz, M. Optimization chal- lenges at the European Space Agency. In Proceedings of the 24th ACM Genetic and Evolutionary Computa- tion Conference (GECCO), pp. 1542-1553, Boston, MA, 2022.
Kumar, A. and Levine, S. Model inversion networks for model-based optimization. In Advances in Neural Infor- mation Processing Systems 33 (NeurIPS), pp. 5126-5137, Virtual, 2020.
Jiang, C., Shu, X., Qian, H., Lu, X., Zhou, J., Zhou, A., and Yu, Y. LLMOPT: Learning to define and solve general optimization problems from scratch, In Proceedings of
12

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Kumar, A., Yazdanbakhsh, A., Hashemi, M., Swersky, K., and Levine, S. Data-driven offline optimization for archi- tecting bardware accelerators. In Proceedings of the 10th International Conference on Learning Representations (ICLR), Virtual, 2022.
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models enhance Bayesian optimiza- tion. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 2024d.
Lange, R. T. evosax: JAX-based evolution strategies. In Pro- ceedings of the 25th Companion Conference Genetic and Evolutionary Computation (GECCO), pp. 659-662, Lisbon, Portugal, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In Proceedings of the 7th International Confer- ence on Learning Representations (ICLR), New Orleans, LA, 2019.
Lee, S., Chu, J., Kim, S., Ko, J., and Kim, H. J. Advancing Bayesian optimization via learning correlated latent space. In Advances in Neural Information Processing Systems 37 (NeurIPS), pp. 48906-48911 New Orleans, LA, 2023.
Lu, H., Qian, H., Wu, Y., Liu, Z., Zhang, Y., Zhou, A.. and Yu, Y. Degradation-resistan offline optimization via accumulative risk control. In Proceedings of the 26th European Conference on Artificial Intelligence (ECAI), pp. 1609-1616, Kraków, Poland, 2023.
Lehre, P. K. and Lin, S. No free lunch theorem and black- box complexity analysis for adversarial optimisation. In Advances in Neural Information Processing Systems 38 (NeurIPS), pp. 121570-121597, Vancouver, Canada, 2024.
Ma, Z., Guo, H., Chen, J., Peng, G., Cao, Z., Ma, Y., and Gong, Y.-J. LLaMoCo: Instruction tuning of large language models for optimization code generation. vrXiv:2403.01333 2024.
Lester, B., AI-Rfou, R., and Constant, N. The power of scale for parameter-effiient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 3045-3059, Punta Cana, Dominican Republic, 2021.
Maaten, L. V. and Hinton, G. Visualizing data using t-SNE. Journal of Machine Learning Research, 9786)22579-2655 2008.
Maraval, A., Zimmer, M., Grosnit, A., and Ammar, H. B. End-to-end meta- veta-Bayesian optimisation with Transformer neural processes. In Advances in Neural Information Pro- cessing Systems 37 (NeurIPS), pp. 1224-112600 New Orleans, LA, 2023.
Lindauer, M., Karl, F., Klier, A., Moosbauer, J., Tornede, A., Mueller, A. C.,, Hutter, F., Feurer, M., and Bischl, B. Position: A call to action for a human-centered AutoML paradigm. In Proceedings of the 41st International Con- ference on Machine Learning (ICML), pp. 30566-30584, Vienna, Austria, 2024.
Maskkaria, S. M., Krishnamoorthy, S., and Grover, A. Gen- erative pretraining for black-box optimization. In Pro- ceedings of the 40th International Conference on Ma- chine Learning (ICML), pp. 24173-24197, Honolulu, HI, 2023.
Liu, F., Xialiang, T., Yuan, M., Lin, X., Luo, F., Wang, Z Lu, Z., and Zhang, Q. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Proceedings of the 41st International Confer- ence Machine Learning (ICML), pp. Vienna, Austria, 2024a.
Müller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F. Transformers can do Bayesian inference. In Proceedings of the 10th International Conference on Learning Representations (ICLR), Virtual, 2022.
Liu, F., Yao, Y., Guo, P., Yang, Z., Zhao, Z., Lin, X., Tong, X., Yuan, M., Lu, Z., Wang, Z., al. A systematic survey on large language models for algorithm design. arXiv:2410. 14716, 2024b.
Müller, S., Feurer, M., Hollmann, N., and Hutter, F. PFNs4BO: In-context learning for Bayesian optimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 25444-25470, Honolulu, HI, 2023.
Liu, F., Zhang, R., Xie. Z. Sun, R., Li, K. Lin, X., Wang, Z., Lu, Z., and Zhang, Q. LLM4AD: A plat- form for algorithm design with large language model. arXiv:2412.17287 2024c.
Nakkiran, P., Kaplun, G., Kalimeris, D., Yang, T., Edelman, B. L., Zhang, F., and Barak, B. SGD on neural networks learns functions of increasing complexity. In Advances in Neural Information Processing Systems 33 (NeurIPS), pp. 3496-3566600 3506, Vancouver, Canada, 2019.
Liu, P. J., Saleh, M. Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating Wikipedia by summarizing long sequences. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018
Nguyen, T. and Grover, A. Transformer neural processes: Uncertainty-awe awwee aware meta learning via sequence modeling. In Proceedings of the 39th International Conference on
13

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Machine Learning (ICML), pp. 16569-16594, Baltimore, MD, 2022.
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E.,, Ruiz, F. J. R., Ellenberg. J., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Mathemati- cal discoveries from program search with large language models. Nature, 625(7995):468
Nguyen, T. and Grover, A. LICO: Large language models for in-context molecular optimization. In Proceedings of the 13th International Conference on Learning Represen- tations (ICLR), Singapore, 2025.
2024.
Schlucter, M., Neshat, M., Wahib, M., Munetomo, M., and Wagner, M. GTOPX space mission benchmarks. Soft- wareX, 14:100666 2021.
Nguyen, T., Agrawal, S., and Grover, A. EXPT: Synthetic pretraining for few-shot experimental design. In Ad- vances in Neural Information Processing Systems 36 (NeurIPS), pp. 45856-45869, New Orleans, LA, 2023.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104 (1):148-175, 2016.
Nguyen, T., Zhang. Q. Yang, B., Lee, C., Bornschein, J., Perel, S., Chen, Y., and Song, X. Predicting from strings: Language model embeddings for Bayesian optimization. arXiv:2410.10100 2024.
Shojaee, P., Meidani, K., Gupta, S., Farimani, A. B., and Reddy, C. K. LLM-SR: Scientific equation discovery via programming with large language models. In Proceed- ings of the 13th International Conference on Learning Representation (ICLR), Singapore, 2025.
Novikov, A., Vü, N., Fisenberger, M., Dupont, E.,, Huang, P.-.., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F.J. R., Mehrabian, A., Kumar, M. P., See, A., Chaudhuri, S., Holland, G., Davies, A., Nowozin. S., Kohli, P., and Balog, M. AlphaEvolve: A coding agent for scientific and algorithmic discovery. Google Deepmind Technical Report, 2025.
Song, L., Gao, C., Xue, K., Wu, C.,, Li, D., Hao, J., Zhang, Z., and Qian, C. Reinforced in-context black-box opti- mization. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, 2025.
Parberry, I. Introduction to Game Physics with Box2D. CRC Press, 2017.
Song, X. and Bahri, D. arXiv:2501.19383 2025.
Decoding-based regression.
Qi, H., Su, Y., Kumar, A., and Levine, S. Data-driven offline decision-making via invariant representation learning. In Advances in Neural Information Processing Systems 36 (NeurIPS), pp. 13226-13237, New Orleans, LA, 2022.
Song, X., Li, O., Lee, C., Yang, B., Peng, D., Perel, S., and Chen, Y. OmniPred: Language models as universal regressors. Transactions on Machine Learning Research, 2024a.
Qian, H., Zhu, Y., Shu, X., An, X., Wen, Y., Liu, S., Lu, H., Zhou, A., Tang, K., and Yu, Y. SOO-Bench: Benchmarks for evaluating the stability of offline black-box optimiza- tion. In Proceedings of the 13th International Conference on Learning Representations (ICLR), Singapore, 2025.
Song, X., Tian, Y., Lange, R. T., Lee, C., Tang, Y., and Chen, Y. Position: Leverage foundational models for black-box optimization. Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 46168- 46180, Vienna, Austria, 2024b.
Qwen Team, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., Yang. J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K. Bao, K., Yang. K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T. Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. 2xXiv2242..51112 2024.
Song, X., Zhang, Q. Lee, C., Fertig, E., Huang, T.-K., Belenki, L., Kochanski, G., Ariafar, S., Vasudevan, S., Perel, S., and Golovin, D. The Vizier Gaussian process bandit algorithm. arXiv: 2408. / /527, 2024c.
Stanton, S., Maddox, W. J., Gruver, N., Maffettone, P., De- laney, E., Greenside, P., and Wilson, A. G. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 20459-20478, Baltimore, MD, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text Transformer. Journal of Maching Learning Research, 21: 1485-5551, 2020.
Tan, R.-X., Xue, K., Lyu, S.-H., Shang, H., Wang. Y., Wang, Y., Fu, S., and Qian, C. Offline model-based optimization by learning to rank. In Proceedings of the 13th Interna- tional Conference on Learning Representations (ICLR), Singapore, 2025.
Rasmussen, C. and Williams, C. I. Gaussian Processes for Machine Learning. The MIT Press, 2006.
14

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Tanabe, R. and Ishibuchi, H. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89:106078, 2020.
Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Nado, Z., Gilmer, J., Snoek, J., and Ghahramani, Z. Pre-trained Gaussian processes for Bayesian optimization. Journal of Machine Learning Research, 25(212):1 83, 2024b.
Tang, E.,, Yang, B., and Song, X. Understanding L.M embeddings for regression. Transactions on Machine Learning Research, 2025.
Wei, Y., Zhuang, V., Soedarmadji. S., and Sui, Y. Scalable Bayesian optimization via focalized sparse Gaussian pro- cesses. In Advances in Neural Information Processing Systems 38 (NeurIPS), pp. 300443-120460 Vancouver, Canada,, 2024.
Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033, Algarve, Portugal, 2012.
Weng, T., Zhang. H., Chen, P., Yi, J., Su, D., Gao, Y., Hsieh, C., and Daniel, L. Evaluating the robustness of neural networks: An extreme value theory approach. In Pro- ceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018.
Trabucco, B., Kumar, A., Geng, X., and Levine, S. Conser- vative objective models for effective offline model-based optimization. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 10358- 10368, Virtual, 2021.
Wistuba, M. and Grabocka, J. Few-shot Bayesian optimiza- tion with deep kernel surrogates. In Proceedings of the 9th International Conference on Learning Representa- tions (ICLR), Virtual, 2021.
Trabucco, B., Geng. X., Kumar, A., and Levine, S. Design- Bench: Benchmarks for data-driven offline model-based optimization. In Proceedings of the 39th International Conference on Machine Learning (ICML), pp. 21658- 21676, Baltimore, MD, 2022.
Wolpert, D. and Macready, W. No free lunch theorems for optimization. IEEE Transactions Evolutionary Computation, 1(1):67-82, 1997.
Xue, K., Tan, R.-X., Huang, X., and Qian, C. Offline multi-objective optimization. In Proceedings of the 41st International Conference on Machine Learning (ICML., pp. -56244, Vienna, Austria, 2024.
Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laakso- nen, E., Xu, Z., and Guyon, Bayesian optimization is superior to random search for machine learning hyper- parameter tuning: Analysis of the black-box optimiza- tion challenge 2020. In NeurlPS 2020 Competition and Demonstration Track, pp. 3-26. PMLR, 2021.
Yao, M. S., Zeng. Y., Bastani, H., Gardner, J., Gee. J. C., and Bastani, O. Generative adversarial model- op- timization via source critic regularization. In Advances in Neural Information Processing Systems 38 (NeurIPS), pp. 44009 44039, Vancouver, Canada, 2024.
Van Breugel, B. and Van Der Schaar, M. Position: Why tabular foundation models should be a research priority. In Proceedings of the 41st International Conference Machine Learning (ICML), pp. 48996,48993 Vienna, Austria, 2024.
Yao, S., Liu, F., Lin, X., Lu, Z., Wang, Z., and Zhang, Q. Multi-objective evolution of heuristic using large language model. In Proceedings of the 39th AAAI Confer- ence on Artificial Intelligence (AAAI), pp. 27 27144-27152, Philadelphia, PA, 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In Advances in Neural Information Process- ing Systems 31 (NeurIPS), pp. 6000-6010, Long Beach, CA, 2017.
Yu, P., Zhang, D., He, H., Ma, X., Miao, R., Lu, Y., Zhang, Y., Kong, D., Gao, R., Xie, J., Cheng, G., and Wu, Y. N. Latent energy-based based Odyssey: Black-box optimization via expanded exploration in the energy- based latent space. arXiv:2405. 16730, 2024.
Wang, S., Xue, K., Song, L., Huang, X., and Qian, C. Monte Carlo tree search based space transfer for black box op- timization. In Advances in Neural Information Process- ing Systems 38 (NeurIPS), pp. 49591-49624, Vancouver, Canada, 2024a.
Yu, S., Ahn, S., Song, L., and Shin, ROMA: Robust model adaptation for offline madel-based optimization. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 4619-4633 Virtual, 2021.
Wang, Z., Gehring, C., Kohli, P., and Jegelka, S. Batched large-scale Bayesian optimization in high-dimensional spaces. In Proceedings of the 21st International Confer- ence on Artificial Intelligence and Statistics (AISTATS), pp. 745-754, Lanzarote, Spain, 2018
Yuan, Y., Chen, C. S., Liu, Z., Neiswanger, W., and Liu, X. S. Importance-aware aware co-teaching for offline model- based optimization. In Advances in Neural Information Processing Systems 37 (NeurIPS), pp. 55718-557777 11118-,,33,,, New Orleans, LA, 2023.
15

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Yuan, Y., Zhang, Y., Chen, C. S., Wu, H., Li, Z., Li, J., Clark, J. J., and Liu, X. S. Design editing for offline model-based optimization. Transactions Machine Learning Research, 2025.
Yun, T., Yun, S., Lee. J., and Park, J. Guided trajectory generation with diffusion models for offline model-based optimization. In Advances in Neural Information Process- ing Systems 38 (NeurlPS), pp. Canada, 2024.
Vancouver,
Zhang, F., Altas, Y., Fan, L., Vinchure, K., Nord, B., and Chen, Y. Design of physical experiments via collision- free latent space optimization. In Machine Learning and the Physical Sciences Workshop at NeurlPS'20, Virtual, 2020.
Zheng, Z., Xie, Z., Wang, Z., and Hooi, B. Monte Carlo tree search for comprehensive exploration in LLM--ased automatic heuristic design. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Vancouver, Canada, 2025.
Zhou, Z.-H., Yu, Y., and Qian, C. Evolutionary Learning: Advances in Theories and Algorithms. Springer, 2019.
16

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
A. Related Work
A.1. Online BBO
An important setup for BBO tasks is online BBO, which obtains optimal solutions via iteratively querying the unknown black-box objective. Traditional online BBO algorithms are population- -aseed search a algorithms, e.g., evolutionary algo- rithm (EA; Bäck, 1996; Zhou et al., 2019), evolution strategies (ES; Hansen et al., 2015; Hansen, 2016), and particle swarm optimization (PSO; Kennedy & Eberhart, 1995; Gong al., 2015). However, they find solutions via population-level evaluations and updates, which is unsuitable for expensive real-world BBO tasks. Bayesian optimization (BO; Garnett, 2023) is is widely used sample-efficient method for expensive BBO problems. BO first fits a surrogate model, typically Gaussian processes (GP; Rasmussen& & Williams, 2006), to approximate the objective function at each iteration, and then optimize a pre-defined acquisition function to sample the next candidate (Shahriari et al., 2016; Frazier, 2018). However, the limited efficiency of GP hinders the scalability to a large-scale scenario. Thus, recently deep learning techniques are well adopted to BO for surrogate modeling (Garnett et al., 2014: Wistuba & Grabocka, 2021: Wang et al., 2024b; Garnelo et al., 2018; Nguyen& & Grover, 2022) or acquisition instantiation (Maraval et al., 2023). However, in many scientific and industrial scenarios, online evaluation is prohibited or cannot be used anymore (Gaulton et al., 2012; Kumar et al., 2022), showing an urgent need for offlin BBO, which we will introduce in the next subsection.
A.2. Offline BBO
Offline BBO (Trabucco et al., 2022; Kim et al., 2025) methods can be generally categorized into the following two types:
Forward approach. This category usually trains a scoring surrogate and maximizes the output to obtain the final design. However, these methods suffer from OOD issue such that errors made by the surrogate in OOD region would mislead the search procedure. To mitigate it, NEMO (Fu & Levine, 2021), COMs (Trabucco et al., 2021), IOM (Qi et al., 2022), RoMA (Yu et al., 2021), BOSS (Dao et al., 2024b) and IGNITE (Dao et al., 2024a) regulate the model from different perspectives, while TriMMentorig (Chen al., 2023a) and ICT (Yuan al., 2023) ensemble surrogates to enhance robustness. Besides, BDI (Chen et al., 2022a), MATCH-OPT (Hoang al., 2024), and Cliqueformer (Kuba et al., 2024b;a) incorporate more information distilled from offline dataset, and ARCOO (Lu et al., 2023), PGS (Chemingui et al., 2024), GABO (Yao et al., 2024), and DEMO (Yuan et al., 2025) employ different techniques to guide the model-inner search. Recently, Tan et al. (2025) point out that regression is not suitable for offline BBO and call for a paradigm shift to rank the designs.
Backward approach typically fits a probabilisticc model p(xly), and samples promising designs from the model, which can be instantiated by prevalent generative models. For example, MINs (Kumar & Levine, 2020) uses GAN (Goodfellow et al., 2014), while DDOM (Krishnamoorthy et al., 2023), RGD (Chen et al., 2024), and DiffOPT (Kong et al., 2024) utilize diffusion model (Ho al., 2020). Recent works also focus on learning from trajectories (Mashkaria et al., 2023; Yun et al., 2024), synthetic priors (Nguyen et al., 2023), or sampling-fres frree latent variables (Yu et al., 2024).
A.3. LLM for BBO
Recent progress in LLM has demonstrated the potential applicability of LLM to solve BBO problems (Song al., 2024b). Works in this field can be generally categorized into two major branches: (1) leveraging general intelligence of LLM to design algorithms or heuristics via natural language (Romera-Paredee et al., 2024; Liu et al., 2024c; Novikov et al., 2025), and (2) utilizing the representation ability of LLM to improve learned BBO algorithms via tokens or embeddings.
For the first branch, works leverage LLM to design algorithm code. FunSearch (Romera-Paredes et al., 2024) proposes an iterative generation procedure for better code, which has been proven successful in many domains, e.g., symbolic regression (Shojaee et al., 2025) and scientific domains (Novikov et al., 2025). EoH (Liu et al., 2024a; Yao et al., 2025) incorporates crossover and mutation operators from evolutionary algorithms for better algorithm design, which also shows great potential in BBO (Liu et al., 2024c;b; Zheng et al., 2025). In Appendix E.7, we also discuss the potential of such methods to further improve the performance of UniSO. Besides, LLaMoCo (Ma et al., 2024) and LLMOPT (Jiang et al., 2025) train an optimization code generator via fine-tuning pre-trained LLM and training from scratch, respectively. Many of other works also employ LLM as components of BBO optimizers, e.g., LLAMBO (Liu et al., 2024d) simulates components of Bayesian optimization, such as regressor and acquisition function, with LLM. The success of this branch lies in whitening the problem nature via intelligence and prior knowledge conveyed by natural language. However, it is still limited by the training corpus (when it meets up with unseen domain) and lacks interpretability.
17

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
For the second branch, various approaches have been proposed leverage the representation power of LLM structure. The well-studied provable properties of these structure for sequential modeling enable its application for BBO, where optimization trajectory can also be viewed as sequence. Given that ireettin-bbsele Transformers (Vaswani et al., 2017) has superior scalability (Brown et al., 2020) and can learn in- in-context well (Garg et al., 2022), TNPs (Nguyen & Grover, 2022; Maraval et al., 2023) employs raw Transformers as in-context regressor for BBO. Note that Transformers trained with synthetic priors performs well on tabluar data (Hollmann et al., 2025) and Bayesian inference (Müller et al., 2022), works like PFNs4BO (Müller et al., 2023), LICO (Nguyen & Grover, 2025) and EXPT (Nguyen et al., 2023) utilize such priors for in-context optimization on both online and offline scenarios. Attention- based methods exhibit superior performance in BBO field, but they usually optimize in a fixed search space. Token-basee based methods offer another perspective by representing designs or parameters tokens and learning from historical trajectories. For example, OptFormer (Chen et al., 2022b) views parameter value as a token for hyper-paramete optimization and employs Transformer structure, combined with metadata, to represent an optimization trajectory, while Dery et al. (2022) consider multi-step improvement for OptFormer and Song et al. (2025) plug a regret-to-go token into the algorithm history to solve BBO problems. Recently, string-based approaches have emerged as a promising direction. OmniPred (Song et al., 2024a) regresses in multi-task suite based on string-based representation for arbitrary designs, metadata and scores, while Nguyen et al. (2024); Tang et al. (2025) use LLM embeddings pre-train an in-context regressor for BBO. Motivated by the superior performance of these methods, in this work, we use string-basel representation to solve universal offline BBO, discussed in Section 3.
B. Experimental Settings
We adopted the T5X architecture (Raffel et al., 2020) as our base model. For embedder in UniSO-N and metadata embedder in UniSO-T, we directly leverage the pre-trained T5-small model. Details according to other components of the model and search algorithms can be found in the following subsections.
B.1. Model Architecture Hyperparameters and Optimizer Configurations
For UniSO-T, we use the T5-based architecture. Pre-training details are listed in Table 3. We adopt T5's default tokenizer, i.e., SentenPiece (Kudo & Richardson, 2018), as the input tokenizer and P10 tokenizer (Charton, 2022) the output tokenizer. The model is trained for 200 epochs with batch size of 128. For few-shot fine-tuning we fine-tune the model for 5 epochs using SGD. During inference, we set temperature to 0.7, apply top-k sampling with k 20 and nucleus sampling with p = 0.95 to balance output quality and diversity.
For UniSO-N, we use pre-trained T5-Small embedder to embed input strings, and then apply a 1-dimensional output MLP regressor for model instantiation. Model configurations are shown in Table 4. For pretraining, the regressor is trained for 200 epochs with batch size of 128 before evaluation. For few-shot fine-tuning finettuning, we fine-tune the regressor for 5 epochs using SGD,
Table 3. Hyper-parametors and optimizer configuration for UniSO-T.
Module
Hyper- parameters
Values
Encoder
Vocab size
32128
Num layers
6
Head dimension
32
Embedding dimension
384
MLP dimension
512
Decoder
Num layers
Head dimension
32
Embedding dimension
384
MLP dimension
512
Optimizer
Name
AdamW
Learning rate Weight decay (31, 32)
x 10-4
0.01 (0.9, 0.99)
LR scheduler
Name Warmup steps
CosineAnneaaling 1000
18

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Table 4. Hyper-parameters and optimizer configuration for UniSO-N.
optimizzercomfiuuaaion
Module
Hyper-parameterd parameters
Values
Regressor
Hidden layers
2
Hidden dimension
2048
Output dimension Activation
ReLU
Optimizer
Name
Adam
Learning rate Weight decay (81,
1x 10 1 10 (0.9, 0.999
LR Scheduler
Name Warmup Steps
Cosine-nneealing 1000
B.2. Model-Inner Search Algorithms Details
For model-inner search algorithms, we consider three types of representative black-box optimizers, Bayesian optimiza- tion (BO; Garnett, 2023), evolutionary algorithms (EAs; Bäck, 1996; Zhou et al., 2019), and evolutionary strategies (ES; Hansen et al., 2015). Experimental comparison these model-inner optimizers can be found Appendix E.3, Here we present detailed hyper-paramett settings and other configurations of these optimizers.
For BO, we adopt BO-gEI for CONTINUOUS problems, which is implemented with BoTorch (Balandat et al., 2020)?. The hyper-parameters of BO- implementation are shown in Table 5. For CATEGORICAL problems, we use a transformed overlap kernel from (Khan et al., 2023) as the Gaussian process (GP) kernel function, then maximize the upper confidence bound (UCB) acquisition of GP outputs using EAs to sample candidates in each iteration.
Table 5. Hyper-parameter of BO model-inner optimizer.
Search space
Hyper-parameters
Value
GP noise variance
0.01
Initial training samples
500
Batch size
Number of restarts
10
CONTINUOUS
Initial random candidates
128
Batch limit
5
Max optimization iterations
200
BO iterations
100
MC samples
128
GP noise variance
0.01
EA population size
32
CATRGORICAL
EA generations
200
UCB ,
0.2
For EAs, we use рувоо (Blank & Deb, 2020) for implementation. We initialize the population as the top-k scoring designs in the dataset, where k 10 is the population size. For continuous tasks, simulated binary crossover (SBX) and polynomial mutation (PM). which are the default genetic operators in pymoo, and the default For categorical tasks, we use uniform crossover and random replacement mutation, which are the default operators for CATEGORICAL search spaces in (Xue et al., 2024). We iterate 100 generations to search for the final solutions.
7https://boten http:hhog/////oooooo/ooo htttps://pymoo.ctt
s osed_ cassedddooppooorrrroeee hoop_botorch_on"
19

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
For ES, we use CMA-ES (Hansen, 2016) implemented with evosax (Lange, 2023)". We set the initial a 0.., and search for 100 iterations.
C. Details of Tasks and Datasets
In this section, we introduce details of different tasks and datasets we use in our experiments. We use the datasets and tasks from Design-Bench (Trabucco et al., 2022) and SOO-Bench (Qian et al., 2025), with tasks and a dataset size of 90K in total. Detailed properties can be found in Table 6.
Table Properties of tasks and datasets Design-Bente and SOO-Bench.
Benchmark Suite
Task
Dataset size
Variable type
# dimensions
Ant D'Kitty Superconductor
10004
CONTINUOUS
56
10004
CONTINUOUE
60
Design-Bench
17010
CONTINUOUS
86
TF Bind 8
32898
CATEGORICAL
8 (3 categories)
TF Bind 10
10000
CATEGORICAL 10 (3 categories)
GTOPX
22000
CONTINUOUS
22
GTOPX
18000
CONTINUOUS
18
SOO-Bench
GTOPX 4
26000
CONTINUOUS
26
GTOPX 6
22000
CONTINUOUS
22
C.1. Design-Bench Tasks
Design-Bent Biigg-BBnnchh (Trabucco et al., 2022)!0 10 is a famous benchmark suite for offline BBO. includes various realistic tasks from real-world optimization problems, and each task corresponds to an oracle function for evaluation and a large static offline dataset. In this paper, we mainly consider tasks in Design-Benche and we directly use the open-sourced dataset of D a part of the training data. Details of these tasks are as follows.
Ant and D'Kitty Morphology. These two tasks are both robot morphology optimization problems, and the goal is to optimize the morphological structure of two simulated robots: Ant from OpenAl Gym (Brockman et al., 2016) and D' Kitty from ROBEL (Ahn et al., 2020). Specifically, the objective of Ant Morphology is to make the Ant robot run quickly and that of D' D'Kttty Morphology is to let the D' Kittty robot move exactly to the predefined location. Both these tasks use a task-specific pre-trained Soft Actor Critic (Haarnoja et al., 2018) policy as controller, and simulate in the MuJoCo (Todorov et al., 2012) simulator with a timestep of 100. The design dimension of Ant Morphology is 56 and that of D' Kitty Morphology is 60, both of which include size, orientation, and location of the limbs. The dataset sizes of these two tasks are both 10004 with CONTINUOUS design spaces.
Superconductor. The objective of the Superconductor task (Hamidieh, 2018) is to maximize the critical temperature which derived from the chemical formula for a superconducting material. Although the actual critical temperature of a decoded material is inaccessible, which needs physical experiments, the evaluation is conducted using a pre-trained random forest regressor from (Fannjiang Listgarten, 2020) as oracle function, which achieves a Spearman rank conkccorelatioon coefficient of 0.9210 held-out validation set. The design space consists of 86 CONTINUOUS variables, and the dataset size is 1 17010.
TF Bind 8 and TF Bind 10. The objective of TF Bind 8 and TF Bind 10 (Barrera et al., 2016) is to maximize the binding affinity score of the designed length 8 and length 10 DNA sequence with prevalent human transcription factor SIX6_REF_RL. Scores are obtained via direct lookup from the exhaustive evaluation database established by (Barrera et al., 2016). The design dimension is 8 for TF Bind 8 and 10 for TF Bind 10, and the design spaces are both CATEGORICAS where the number of categories of all dimensions are 4. The TF Bind dataset contains 32898 design-score pairs. Although the TF Bind 10 dataset in Design-Bench Bench includes 4161482 design-score pairs, which is too large and beyond our computation budget, we sample a subset with size of 10000 following recent work in the field of offline BBO (Tan et al., 2025).
Phttps: ///gi thgtthuub... www.c/o///oooeeeeee httm/RbbertTanggeereeeee "https:/ stttps://gtti thub. com htmm/bbandont /ecoo/cccoo/eeiccac -n-bench thttts://hugging wttp:////tattttteeeeeeeeeeeeo
des www_bench_ppgeeeeeeee
20

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
We exclude two tasks in Besign-Bench, following (Yu al., 2024; Yun al., 2024; Tan al., 2025). Specifically, exclude ChEMBL (Gaulton et al., 2012) because almost all methods produce the same oracle prediction results, as shown in (Krishnamoorthy et al., 2023; Mashkaria et al., 2023), which is not appropriate for comparison. We exclude synthetic NAS on CIFARIO (Hinton et al., 2012) due to its high computation cost for exact evaluation over multiple seeds, which exceeds our computation resources. We also excluder Hopper Controller (Brockman et al., 2016) in Design- Bench since it has a normalization and de-normalization issue due to the stochastic policy (see https: / /github. con/ brandont prabucco / des i beattnbeecheeeeeeeeeeee issues/ / 8+1 @ff.ssuecomm ssuecomment 1 108675811 15813 for details) and recent works in offline BBO (Yu et al.. 2024; Yun et al., 2024; Tan al., 2025) do not benchmark on this task as well.
C.2. SOO-Bench Tasks
SOO-Bench (Qian et al., 2025) is a recent benchmark for offline BBO, which not only provides more test problems but also proposes a novel method to evaluate the stability of the forward methods for offline BBO. We select the unconstraint tasks in S0O-Bench, GTOPX 2, GTOPX 3, GTOPX 4, and GTOPX 6 from GTOPX benchmark (Schlueter et al., 2021), except for hybrid due to environment conflicts during installation. We use the open-source code of SOO-Bench" and generate task data following the default settings in the SOO-Bens -00-Bench paper. Specifically, for given task, the dataset size is 1000 times the variable dimension and the data is uniformly drawn from the middle 50% of the overall distribution with respect to the score values. We set the random seed of the data generation procedure as by default. Both these tasks are under CONTINUOUS design spaces and their evaluations are done through simulation library modules provided by (Schlueter et al., 2021)!3. Detailed description of our selected tasks are as follows.
GTOPX encompasses comprehensive suite of real-world space trajectory optimization problems (Izzo & Manuel López- Ibáñezz 2022; Izzo, 2010). Specifically, GTOPX 2 encompasses the optimization of an intricate interplanetary trajectory for a Saturn rendezvous mission, featuring 22 decision variables with the objective of minimizing the total velocity increment (AV) required throughout the mission. GTOPX 3 and GTOPX 4 both address trajectory optimization for Mercury missions, where the primary objective is to minimize the total mission AV. These problems are distinguished by their treatment of resonant flybys: GTOPX 3 explicitly excludes such maneuvers and operates in an 18-dimensional design space, while GTOPX incorporates them within a 26-dimensional space. GTOPX 6 focuses on the optimization of multi-gravity-assistt trajectories targeting Comet 67P//huryymov- Geraroinnneer.ainooooo. employing 22 design variables to minimize the total AV requirements.
C.3. Real- Tasks
Following (Wang al., 2024a), we evaluate our method on three use the open-sourced dataset by (Wang et al., 2024a) Detailed information of these tasks are follows.
tasks, LunarLander, RobotPush, and Rover, and
LunarLander. LunarLander is a 12-dimensional CONT INNTIN task implemented in OpenAl Gym (Brockman et al., 201 15 that aims to learn the parameters of a controller for a lunar lander. The objective is to maximize the expected average return over 50 randomly generated environments.
RobotPush. RobotPush aims to minimize the distance between a designated target location and a pair of robotically- controlled objects, where there are 14 CONTINUOUS controllable variables, e.g., orientation and speed. The function implemented in (Wang et al., 2018)16 16 with physics engine Box2D (Parberry, 2017).
Rover. Rover is a 2D trajectory optimization task that simulates a rover navigation task, which is defined by (Wang et al., 2018)"7 The trajectory is optimized within a 60-dimensional CONT CONTINUOUS INUOUS unit hypercube, A cost function c(-) is defined in the hypercube to measure the trajectory quality, and the objective is to minimize the total cost.
Phttps://githne httpg/gtthhhoomooaaaa
Bhttps:/ phttps::/www.rr wttp://www................//ttt. Lander
com/
function.py
1 "nttps://githubs /
http:://www.......o/./////////// rover_functions cts.unctti.ny.......... py.
21

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
To generate diverse task trials, Wang et al. (2024a) employ a transformation from RIBBO (Song et al., 2025)18 (8 that introduces
scaling factor& and a translation vector t, where the transformed objective is computed as y = 8. f(x t). These factors are deterministically generated based on random seeds, allowing different seeds to map to distinct task. We randomly choose one seed for one task for evaluation (100 for LunarLander, 100 for RobotPush, and 150 for Rover).
D. Metadata in the Experiments
In Table 7, we deliver all the metadata we used in our experiments.
Table 7. Mctadata illustration we used in the experiments.
Task
Name
Description
Objective
Ant
Ant Morphology
quadruped rahot morphology optimization
fast possible
D' D'Kitty
D' Kitty Morphology
D' Kitty robot morphology optimization
to navigate the robot fixed location
to design the chemical formuln for superconducting
Superconductor
Superconductor
critical lemperature maximization
material that has high critical temperature
to find the length-8 DNA sequence with maximum
TF Bind 8
TF Bind 8
DNA sequence optimization
binding afrinity with SIX6_REF _RI transcription factor
to the length- 10 DNA sequence with maximum
TF Bind 10
TF Bind 10
DNA sequence optimization
binding affinity with SIX6_RF JRI transcription factor
to achieve rendezsous with Satumn aiming to minimize the total velocity change
GTOPX 2
Cassini
Complex interplanetary missions to Saturn
to minimuzze the total velocity change
GTOPX 3
Messenger (reduced)
Simulation of interplanetary missions Mercury
over the course of the mission
Interplanetary missions to Mercury, with resonant flybys of the planet
minimiize the total velocity
GTOPX 4
Messenger (full)
change incurred throughout mission
Simulation of multi- graviity-assistttte space missions to Comet wahhhyumoo--Grriinnaaae
to minimize the total velocity change
GTOPX 6
Rosetta
required throughout the mission
to maximize the mean terminal reward across
LanarLander
LunarLander
Learn parameters of controller for lunar lander
consistent batch of 50 randomly generated landscape to minimize he distance between predefined target location and two objects
RobotPush
RobotPush
Control the rubot push items designated location
Rover
Rover
trajectories oplimization for rover
design a reasonable trajectory to minimize the cost
E. Additional Experimental Results
In this section, we conduct additional experiments further discover the effectiveness of UniSO. In this section, unless explicitly specified, we use EA the default model-inner optimizer.
F. Comparison to State-of-the-Art Single-Task Offline BBO Methods
To better understand the performance of UniSO methods, in this subsection, we compare improved UniSO-T, the best- performing universal offline BBO approach in Table 2, to wide range of single-task offline BBO methods on Design- Bench (Trabucco et al., 2022). These methods are mainly based on 7-score normalization for input solutions. Here the improved UniSO-T method is trained only on Design-Bench datasets and tasks, and the detailed results of compared methods U-Yaain
are referred from (Tan et al., 2025). The objective score y are normalized via global min-max normalization y + Munas Mamrn where Ymin and Fmax denote the lowest and highest scores in the full unobserved dataset from Design-Benchh respectively. Such evaluation protocol under normalization is commonly adopted in offline BBO (Kumar Levine, 2020; Trabucco et al., 2022). In Table 8, UniSO-T achieved an average rank of 9.8 across 21 expert single-task offline BBO methods, outperforming some recently proposed methods that are specifically designed for single-task offline BBO, and achieving the best results on TF-Bind-10 10. However, there is still improvement room for UniSO methods. Thus, how to propose better techniques and improve performance (e.g., training with more data) for universal offline BBO is crucially important future work.
18 https:: / /i thub. wwt.b.coooooa.eeecc songle e 10/000/RIRR /R///IIBOObbb/ htt.///lb////dc /tttt////boeeeeeeeeeee le8c2855222222 6a 0889688299999729 problems e real reae_warrrrrreeee warld_proor em. py. 1199
22

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Table Normalized scores Design-Bench. where the runner-up results on each task are Blue and Violet. D(best) denotes the best score in the offline dataset. All methods are trained within on task, while U are done in multi-task manner. Results of all compared methods are referred from (Tan al., 2025).
Method
Venue
Ant
D D'Kitty
Superconductor
TF-Bind-8
TF-Bind-10
Avg. Rank
D(best)
0.565
0.884
0.400
0.439
0.467
BO-gHI
0.812 0.000
0.896 0.000
0.382 0.013
0.802 0.081
0.628 0.036
17.8 22
CMA-ES
1,712 + 0.754
0.725 = 0.002
0.463 4 0.042
0.944 = 0.017
0.641 = 0.036
11.4 /2
REINFORCE
0.248 1 0.039
0.541 = 0.196
0.478 + 0.017
0.935 0.049
0.673 0.074
13.8/22
Baselines
Grad. Ascent
0.273 + 0.023
0.853 = 0.018
0.969 = 0.021
0.646 0.037
11.2
Grad. Ascent Mean
0.306 + 0.053
0.875 = 0.024
0.508 # 0.019
0.985 = 0.008
0.633 = 0.030
10.6/22
Grad. Ascent Min
0.282 0.033
0.884 0.018
0.514 0.020
0.979 0.014
0.632 0.027
11,2/22
CbAS
ICMI'19
0.846 + 0.032
0.896 = 0.009
0.421 # 0.049
0.921 0.046
0.630 0.039
15.6/22
MINs
ICML'19
0.906 + 0.024
0.939 = 0.007
0.464 0.023
0,910 0.051
0.633 0.034
12.6/22
DDOM
ICML123
0.908 # 0.024
0.930 # 0.005
0.452 0.028
0.913 = 0.047
1616=00018
14,2/22
BONET
ICML'23
0.921 0.031
0.949 = 0.016
0.390 0.022
0.798 0.123
0.575 0.039
14.6/22
GTG
NeurlPS'24
0.855 + 0.044
0.942 + 0.017
0.480 + 0.055
0.910 0.040
0.619 0.029
13.6/22
COMs
ICM1'21
0.916 0.026
0.949 0.016
0.460
0.953 0.038
0.644 0.052
9.0/22
RoMA
ICML'21
0,430 + 0.048
0.767 = 0.031
0.494
0.665 = 0.000
0.553 = 0.000
18.0/22
1OM
22
0.889 + 0.034
0.928 = 0.008
0.491 # 0.034
0.925 0.054
0.628 01.36
12.6/22 22
BDI
NeurlPS'22
0.963 # 0.000
0.941 + 0.000
0.508 0.013
0.973 = 0.000
0.658 = 0.000
5,4/22
ICT
233
0.915 + 0.024
0.947 = 0.009
0.026
0.897 = 0.050
0.659
8.8/22
Tri-Mentoring
22
0.891 # 0.011
0.947 0.005
0.503
0.956 0.000
0.662 0.012
7.0/
PGS
AAAI'24
0.715 +: 0.046
0.954 = 0.022
0.024
0.889 = 0.061
0.044
13,4
FGM
AISTATS''4 24
0.923 # 0.023
0.944 = 0.014
0.481 # 0.024
0.811 0.079
=601000000
12.8 22
MATCH-OPT
ICML'24
0.933 + 0.016
0.952 + 0.008
0.504 + 0.021
0,824 0.067
0.635 0.050
7.6/
RaM-ListNet
ICLR'22 25
0.949 # 0.025
0.962 + 0.015
0.517 = 0.029
0.981 = 0.012
0.670 0.035
2.0
UniSO-T (Ours)
0.850 + 0.062
0.915 + 0.015
0.489 + 0.062
0.947 = 0.036
0.673 = 0.136
9.8 //2
E.2. Computational Cost
In this subsection, we provide the computational resource and time for individual tasks. We conduct all our experiments system with 4 GPUs (total computing power 188 TFLOPS) and a 128-core CPU. Detailed computational resources are listed in Table 9. Time distribution over all tasks is provided in Fig. 7, and detailed computational time budgets can be found Table 10.
Table Computational resources comparison between improved UniSO-N and UniSO-T. All measurements are conducted on a system with 4 GPUs (total computing power ~ 188 TFLOPS) and 128-core CPU.
Training
GPU memory
UniSO-N
22530s 71225s
21G
UniSO-T
89G
23

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Time Distribution for Different Methods
1 .4%
10.8%
98.6% 69.
89.2%
UniSO-No BO
UniSO-N+
Algarithm Time
Inference
10.2%
89.8%
98.2%
UnISO-T+BO
UniSO-T+EA
Figure 7. Time distribution comparison between algorithm time and model inference time for different UniSO models (UniSO-N, UniSO- T) and optimization methods (BO-gEI, EA). The pic charts show the percentage breakdown between algorithm time (purple) and inference time (orange) for each model. The distribution is computed over all tasks.
Table Computational budgets of improved UniSO-N UniSO-T with different black-box optimizers (BO-gEI, EA) on unconstrained Sssig--Bench
tasks from Design- Bench and 300-Bench. Each cell shows algorithm time / model inference time (in seconds). Here, BO-gEI and EA are allowed for an evaluation budget 1000 (i.c., optimizer can access model's outputs for upmost 1000 times). EA-25600 denotes the EA with extended evaluation budget of 25600, implemented with population size of 128 over 200 generations, which is default optimizer setting in the submitted version.
Model
UniSO-N
UnISD-T
Search method
BO-EEI
EA
EA-25600
BO.EEl
EA
EA MAA-5600
74.4440.26
1,8740.00
1.94=0.00
2.75+0.02
74.0640:27/
14.81+0.00
14,55+4.000
2.6240.00
202.0540.06
D' Kinty
78.22+1 /01bbi/.3///.2:t 1.72:0000
0,23:0.00
2.00000000..0000.000 2.00e0.00
2.59=0.00)
337.7550.00
17.14:0.98/
13.75a0.01
0,2610.00
13.9410.01
2.59:0.00/ / 196. 90+0.02
Superoonductor
73.4810.00 1.8910.00
0.2340.00
1.92a0.00
2.68:0.00/
4/.400000000
73.7840.00/
14.86a0.01
0.2710.00
14.7440.01
2.8010.01
2214,9410.01
TF Bind 8
59 99.67400.00
1.5810.00
0.27:0.00
00)
3.08:0.01
/35.02::0.00
60.47:0.00
13.02:0.06
0,2610.00
13.17::0.7
2,46:10.01
170.6330.00
TF Bind LO
62.57.00.02/ 1.5910.00
0.2110.00
1.7620.00
2.76g0.01
/35.28.0.00
62.08.0.02.
13.03a0.03
0.2510.00
13.0140.03
2.3310.00/ / 170.9810.15
GTOPX
331.6788.48 1.71+0.00
0.21=0.00
1.94:0.00
2.3420.00
35.200s0.01
182,4848 26 13.46640.00
0.2540.00
13.6640.00
2.49s0.02/
182.3540..6
GTOPX
179.39 799.3946.1 1.58+0.00
0.22+0.00/
1.79:0.00
2.53+0.000
34 14.8200.00
179.07+6.20
13.2340.00
0.2440.00
13.0840.00
2.45+0.02/ / 178.5440.07
GTOPX
1.6240.00
84:00.0
2.37=0.00)
35.75=00.00
88540.37
13.04.0.00
0.25#0
13.28+0
2.49:0001
184,3880000
GTOPX
180.05+17.91 7.91
1.60440.00
0.22+0.00/
1.92+0.00
2.37+0.00 35.98+0.00
176.91+18.75
13.50+0.00
0.2540.00
13.2640.00
2.56+0.00
182.3740.1
24

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
E.3. UniSO-N with In-Context Transformer Neural Processes
In this subsection, we investigate integration with in-context regressors based on Transformer neural processes (TNP; Nguyen & Grover, 2022) with UniSO-N. We treat offline BBO as a single-epoch online optimization process, where top-scoring solutions in the offline dataset serve contexts. We implement two TNP variants: (1) UniSO-N + TNP-UCB that maximizes the UCB acquisition function for one-epoch, following (Nguyen et al., 2024); (2) UniSO-N + TNP-ED that directly regresses objective scores using a single-layer MLP on TNP's hidden state outputs, following the architecture design from Section 3.2.2 of ExPT (Nguyen et al., 2023).
Table 11 demonstrates that TNP-ED outperforms TNP-UCB on out of 9 tasks from Design-Bence Bengh- and SOO-Bench. This shows that direct regression and score maximization is more effective than modeling the distribution and maximizing the acquisition function. We also conduct few-shot learning experiments unseen tasks (RobotPush, Rover, and Lunarl_ander Table 12 shows that both TNP variants successfully exceed the best scores in the offline datasets, showing great generalization potential.
However, UniSO-N with MLP consistently achieves better performance than both TNP variants. This performance gap between MLP and TNP variants can be attributed primarily to optimization constraints in our string-based setting. While previous TNP applications leverage gradient-based optimization on model outputs or acquisition functions, the non-differentiable nature of tokenization in string-based optimization necessitates usage of black-box optimizers (Nguyen et al., 2024). This limits the optimization efficiency and the reduce the effectiveness of TNP in our paper, compared to its success in other scenarios. Future work includes exploring incorporating gradient information through techniques e.g.. soft-prompt methods (Lester et al., 2021), for gradient approximation, or developing encoder-decoder architectures for legal solution string reconstruction.
Table 11. Comparison of improved UniSO-N with ML. and with in-context TNP (Nguyen & Grover, 2022) on unconstrained tasks from Design-Bench Bench and 500-Bench. where the best and runner-up results on each task are Blue and Violet. D(best) denotes the best score in the offline datasct. Here, UniSO-N + TNP-UCB maximizes the UCB acquisition function for one epoch on trained TNP to obtain final candidates, following (Nguyen et al., 2024), and UniSO-N TNP-ED directly regresses the score values and maximizes the model output, following Section 3.2.2 in (Nguyen et al., 2023).
Task Ant D' D'Kitty
D(best)
UniSO + MLP
UniSO-N + TNP-UCB
UniSO-N TNP-ED
165.326
269.691 = 77.425
110.443=22.557
292.098 229.171
199.363
173.911 = 46.662
122.3166317..2
129.005 = 41.107
Superconductor
74.000
67.333 + 10.838
58.653 + 20.171
64,544 = 2,888
TF Bind 8
0.439
0.833 + 0.005
0.638 # 0.000
0.713 = 0.000
TF Bind 10
0.005
0.959 0.115
0.674 = 0,000
0.397 = 0.000
GTOPX 2
-195,586
-124.995 56.170
-183.413 81.862
-181.144 11.685
GTOPX 3
-151.190
-62.622 + 22.261
-1800553 93.678
.99.735 # 2.637
GTOPX 4
-215.716
-110.284 17.559
-130,988 # 38.336
-178.791 = 77.6223
GTOPX 6
-112.599
-57.435 # 18.832
-108.859 + 25.493
-195.184 +: 40.626
Avg. Rank
N/A
1.111 0.314
2.667 0.471
2.222 0.629
Table 12. Few-shot experimental results of improved UniSO-N with MLP and with in-context TNP (Nguyen & Grover, 2022) on RobotPush, Rover, and I. unarl_ander where the best and runner-up results on each task are Blue and Violet. D(best) denotes the best score in the offline dataset. Here, UniSO-N + TNP-UCB maximizes the UCB acquisition function for one epoch on trained TNP to obtain final candidates, following (Nguyen et al., 2024), and UniSO-N + TNP-ED directly regresses the score values and maximizes the model output, following Section 3.2.2 in (Nguyen et al., 2023). UniSO-N + MLP utilizes few-shot data to fine-tune MLP regressor, while UniSO-N + TNP-UCB and UniSO-N+ + TNP-ED directly view the few-shot data as context points.
Task
D(best)
UniSO-N + MLP
UniSO-N + TNP-UCB
UniSO-N + TNP-ED
RobotPush
0.102
7,014 0.000
3.769 1.969
0.877 1.041
Rover
-16.148
-8.488 + 0.003
-12,593 # 1,891
-12,038 # 1.142
LunarLander
7.038
287.038 = 0.000
-113.493 + 129.223
-79.885 4 91.442
Avg. Rank
N/A
1.000 + 0.000
2.667 + 0.471
2,333 + 0.471
25

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
E.4. A Affects of Different Model Size
In this subsection, we compare UniSO-N with pre-trained embedders of different sizes. In Table 13, we find that UniSO-N with bigger pre-trate preitraneed embedder (T5-Base) performs even worse than that with T5-Small across most tasks. Such inferior results are also reflected in training loss curve in Fig. 8, where T5-Base not only starts with higher loss but also drops limitedly throughout training, while T5-Smal demonstrates consistent optimization progress.
This performance difference aligns with our observation discussed in Section 4.2 that LMs' priors may do harm to numerical optimization. Since a larger pre-trained LM embedder incorporates more prior, there may introduce more unfavourable biases for numerical optimization. Thus, how to mitigate this gap and align different modalities of natural language and numerical representation well is critical future work (Song et al., 2024b; Van Breugel & Van Der Schaar, 2024).
Table 13, Comparison of UniSO-N equipped with embedders of different scales (c.g.. T5-Small and T5- T5-Base) on unconstrained tasks from Design- Bench and SOO-Bench, where the better one is Bold. D(best) denotes the best score in the offline dataset.
Task
D(best)
UniSO-N T5-Small
UniSO-N + TS-Base
Ant
165.326
269.691 + 77.425
255.336 109.169
D' Kitty
199,363
173.911 + 46.662
199.186 = 0.000
Superconductor
74.000
67.333 # 10.838
93,816 11,763
TF Bind 8
0.439
0,833 0.005
0.597 0.081
TF Bind 10
0.005
0.959 0.115
0.394 0.140
GTOPX 2
-195.586
-124,995 56.170
-163, 866 74.375
GTOPX 3
-151.190
-62.622 + 22.261
-71.990 = 11.158
GTOPX 4
-215.716
110.2884 # 17.559
99,4000 # 39.728
GTOPX 6
-112.599
-57.435 + 18.832
-88.123 + 20.223
Avg. Rank
1,333 + 0.471
1.667 +: 0.471
Comparison of Training Loss UniSO-N
0.060
From TS-Small
from
0.055
0.050
0.045
0.035
75
100
125
1775
200
Epoch
Figure 8, Training loss curves comparing UniSO-N with T5-Small versus T5-Base embedders during training.
E,5. A Ablation Studies on Each Component
To thoroughly evaluate the effectiveness of each component our method, we conduct ablation studies on metadata quality, loss components, loss balancing strategies, and model-inner optimizers.
For metadata quality, we examine the effectiveness different metadata components (name, description, and objective) by removing each component. As shown in Table 14, experiments across in-distribution tasks and both zero-shot and few-shot scenarios unseen tasks demonstrate the contribution of each metadata component to overall performance. This shows the importance of metadata quality.
For loss components, we study them by removing each loss term individually. The results in Table 15 show that both loss components contribute significantly to UniSO-T's performance, with the complete loss achieving superior results compared to variants with individual losses removed.
26

---

Towards Universal Offline Black-Bol Box Optimization via Learning Language Model Embeddings
Table 14. Ablation studics on different metadata components (name, description, and objective) on various tasks. All experiments are conducted based on improved UniSO-T. first part includes tasks appear in training dataset, i... unconstrained tasks from Design- -Bnchh and -ench. The remaining two parts contain unseen tasks during training, where we compare them under both zero-shot and few-shot settings. The best and runner-up results on each task are Blue and Violet. (best) denotes the best score the offline dataset.
Task
D(best)
UniSO-T
w/o name
w/o desc.
w/o obj.
w/o metadata
Ant
165.326
374.665 + 56.057
345.369 40.675
362.842 # 33.008
363.812 # 59,380
358.379 64.21
D' D'Kitty
199.363
225.752 8.521
235.715 11.617
226.479 15.311
245.135 + 15.571
227.169 12.278
Superconductor
74,000
92.200 15.209
85.207 6.261
99.113 # 12.742
97,610 10.297
90.871 # 10.611
TF Bind
0,439
0.903 0.041
0,954 0.025
0,935 0,041
0,937 0.012
0,950 + 0.025
Bind
0.005
0.823 0.542
0.696 0.126
0.664 0.141
0.596
0.651 0.121
GTOPX
-195,586
-72.848 9.576
-97,806 40.646
-63.670 20,381
-80.789 8.908
-79,864 13.338
GTOPX
-151.190
-45.002 8.433
-50.788 = 8,706
-45.981 4.211
-50.660 = 10.713
48.178 12.638
GTOPX
-2157716
-84.271 8.307
-44962 + 11,300
92.163 + 9.529
-75.233 5.734
-79,887 = 14.729
GTOPX
-112.599
-47.794 # 11,943
-42.181 = 11.671
-4,551 = 12.310
-48.050 # 13.901
-45.764 # 7.685
RobotPush (zero-shot)
0.102
3.171 0.984
2.747 1.455
3,416 1,455
2.634 0.953
2.517 1.640
Rover (zero-shot)
-16.148
-8.888
2.119
-11.009 + 0.598
-10.854 0.859
-9.099 2.202
-9,089 3.070
Lnder (zero-shot)
7.038
27.971 31.186
30.105 57.577
30.892 54.657
52.108 47.941
6,251 53.042
RobutPush (few-shot)
0.102
7.067
0.169
7.026 = 0.219
7.129 + 0.486
6,310 1.677
6.155 + 1,495
Rover (few-shot)
-46.148
-8.239 + 1.270
-8.850 = 0.703
-8,084 0.569
-8.342 # 1.573
-10.511 # 2.070
LunarLander (few-shot)
7.038
248.573 + 45.386
226.726 65.244
244.252 38.329
233.169
51.037
233.919 60.467
Avg. Rank
2.333
1,350
3.600 1.451
2.467 1.310
3.067 1.340
3.533 1.087
Table 15. Ablation studies each component of the improved losses in UniSO-T few-shot performance on RohotPush, Rover, and LunarLander.
Task
Dibest)
UniSO-T
UniSO-T w/o Cxp
UniSO-T w/o Zcon
UniSO-T w/o (Crip- foo)
Ant
165.326
374.665 + 56.057
292.644 = 65.145
232,399 # 66.642
275.216 90,820
D Kitttty
199,363
225.752 # 8.521
175.796 = 64.079
224.668 + 23.942
216.070 23 209
Superconductor
74,000
92,200 15.209
90.910 + 6.538
80.910 + 14.995
86.795 + 13.466
TF Bind
0.439
0.903 0.041
0.916 0.044
0.945 = 0.049
0.940 + 0.027
TF Bind 10
0.005
0.823 0.542
0.623 + 0.062
0.823 + 0,542
0.830 + 0,539
GTOPX
195.586
-72.848 9.576
-112.357 39.361
-87.305 24.428
-132,023 + 63.084
GTOPX
-151.190
-45.602 8.433
-50.294 6.184
-660667 11,332
-60.441 17.235
GTOPX
.215.716
-84.271 8.307
96.550 + 12,745
-84.52 15.571
-100,943 + 15.044
GTOPX
-112,599
-47.794 + 11.943
-67.276 + 24.305
-43,334 : 8.402
-71.749 28.497
Avg. Rank
/
1.667
0.943
2.889 0.737
2,333 # 1.155
3.111 + 0.994
27

---

Towards Universal Offline Black-Boo Optimization via Learning Language Model Embeddings
Additionally, study the loss balancing strategy by comparing it with algorithm that remove this strategy. The comparative results in Table 16 validates the importance of loss balancing strategy static weight assignments.
Table 16. Ablation studies of loss balancing strategy in improved UniSO-T on unconstrained tasks from Besign-Bench and SOO-Bench, where the better one Bold. D(best) denotes the hest score the offline dataset.
Task
D(best)
UniSO-T w/ balance
UniSO-T w/o balance
Ant
165.326
374.665 + 56.057
185.597 165.661
D Kitty
199.363
225.752 + 8.521
203.553 + 32.507
Superconductor
74.000
92.200 # 15.209
79.635 + 5.777
TF Bind
0.439
0.903 0.041
0,929 0.049
TF Bind 10
0.005
0,823 0.542
0.696 0.126
GTOPX
-195.586
-72.848 + 9.576
-175.327 73.053
GTOPX
-151.190
-45.602 = 8.433
-56,221 = 18,342
GTOPX 4
-215.716
-84.271 8.307
-122.291 54.913
GTOPX
-112.599
-47.794 11.943
-70,352 # 25.870
Avg. Rank
1.111 + 0.314
1.889 0.314
In offline BBO, gradient-based model-inner optimizers are commonly used. However, they are challenging to apply in string spaces. Therefore, we explore several alternative black-box search algorithms, including EAs (Bäck, 1996; Zhou al., 2019). BO-qEI (Gamett, 2023), and CMA-ES (Hansen, 2016). Implementation details provided in Appendix B.2. We fix the evaluation budget 1000. Since CMA-ES cannot operate in CATEGORICAL search space, we compare these optimizers on 500-Bench tasks. As shown in Table 17, BO-qEI achieves the best performance, while EA is the runner-up.
Table 17. Comparison of different model-inner BBO optimizers (BO-qEI, CMA-ES, and EA) in improved UniSO-T. As CMA-ES cannot operate in categorical space, conduct experimental comparison continuous tasks from S0O-Bench, where the best and runner-up results on each task are Blue and Violet. D(best) denotes the best score in the offline dataset. Here all optimizers are allowed for evaluation budget of 1000 (i.c., the optimizer can access the model's outputs for upmost 1000 times) for fair comparison.
Task
D(best)
BO-gEI
CMA-ES
EA
GTOPX 2
-195.586
-80.220 # 11.852
-138.716 45.523
-99.778 = 18.512
GTOPX 3
-151.190
-48,493 = 3.745
-81.524 # 19.054
-69.278 20.009
GTOPX 4
-215.716
-80.232
13.582
-177.559
29.664
-132.543 1 38.303
GTOPX 6
-112.599
-73,306
12.892
-101.525
16.986
-62.024 23.828
Avg. Rank
1,250
0.433
3.000 0.000
1.750 # 0.433
To better understand the search behaviors of BO and EA, we visualize the optimization trajectories of both BO and EA in Fig. 9. BO demonstrates focused exploration guided by the Gaussian Process model (which captures the spectral cluster property of embeddings), while EA shows more random search behavior. The convergence curves (Fig. 10) further validates BO's superior optimization efficiency compared to EA's tendency toward local optima.
28

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
plot of UniSO embeddings trajectories
t-SNE plat UniSO-N embeddings and trajectories
GTOPX
GTOPX 2
GTOPX
GTOPX 3
GTOPX
GTOPX
GTOFX
GTOPX 6
50-
t:SWE dimension
dimension
25
-20
-75-
-40
20
40
-20
00
80
dimension
Figure 9. t-SNE plots of UniSO-N embedder comparing BO-gEl (left) and EA (right) search trajectorics. Each part represents data from single task, small scatters represent search trajectories these two optimizers. Different colors represent different tasks, with darker colors indicating later stages of the search.
UniSO-N predictions on GTOPX 2
UniSO-N predictions on GTOPX
0.40
0.3
Best value
0.2
0.25
0.1
0.0
0.15
B0
B0
EA
-0.1
200
400
600
800
1000
200
400
600
800
1000
UniSO-N predictions on GTOPX 4
UniSO-N predictions on GTOPX 6
0.4
0.4
0.3
0.2
Best value
0.1
0.0
0.0
-0.1
0.1
-02
BO
B0
EA
EA
-0.3
200
460
600
800
1000
200
400
6d0
800
1000
# evaluations
evaluations
Figure 10. The historical best change prediction value UniSO-N, with BO-gEI and EA optimizers.
29

---

Towards Universal Offline Black- Optimization via Learning Language Model Embeddings
E.6. UniSO EA Results
In this subsection, we provide detailed experimental results of UniSO EA in Tables 18 and 19.
Table 18. Un- fn-normalized scores unconstrained tasks from Design Bennc and 100-Bench, where the best and runner-up up results each task are Blue and Violet, respectively. D(best) denotes the best score in the offline dataset and BN represents batch normalization for numerical input designs. Both numeric-input experts and string-input UniSO methods trained within single task.
Numeric-input Experts
String-input UniSO
Task
D(best)
BN +
BN Grad
UniSO-T
UniSO-N
165.326
118.877 127 688
229,462-44 = 165.869
245.212 165, 165.083
254.720 + 102.151
Kiny
25.766
188.642 26.306
Superconductor
97.137
63,930
TF Bind 8
0.439
0.984 0.007
0.959 0.023
0.948 0.028
0.949 0.000
TF Bind 10
0.005
0.905 0.326
0.888 0.229
0.623
0.600 0.006
GTOPX 2
11955886
-88.054 20.878
-128.310 15.616
-76.479 4.815
-117.022 51.671
GTOPX 3
-151.190
-64.028 22.678
-151.190 0.000
-46.526 + 13.434
-71.784 16.666
GTOPX 4
-215.716
-96.432 + 10.868
-215.716
0.000
-87.714 + 8.795
-101.304 + 18.492
GTOPX 6
-112.599
-64.217
14.602
-112.599 0.000
-48.186
9.486
-80.391 12,469
Avg. Rank
2.222 1,300
3,000 1.054
1,889 1.100
2.889 0.875
Table 19. Un-normalized scores unconstrained tasks from Design-Bent Bench and S0O-Bench, where the best and runner-up up results on each task are Blue and Violet, respectively. D(best) denotes the best score in the offline datasct and BN represents batch normalization for numerical input designs. Single-task experts are trained within one task, while UniSO-T and UniSO-N are done in multi-task manner.
Single-task Experts
UniSO-T EA
UniSO-N + EA
Task
D(best)
BN + EAs
BN + Grad
Vanilla
Improved
Vanilla
Improved
Ant 165.326
118.877 + 127.688
229.462 +: 165.869
275.216 = 90.820
374.665 56.057
268.399 1 82.858
269.691 = 77.425
D'Kit D'Kitty
199.363
111.205 66.986
183.263 = 62.436
216.070 = 23.209
225.752 + 8.521
130.655 # 83.106
173.911 = 46.662
Superconductor
74.000
93.951 7.039
97.137 # 6.113
86.795 13.466
92.200 + 15,209
81,266 16.073
67,333 16.838
TF Bind
0.439
0.984 : 0.007
0,959 0.023
0,940 0.027
0,903 1 0.041
0.944 0.016
0.833 0.005
TF 10
0.005
0.905 + 0.326
0,888 0.229
0,830 0.539
0,823 1: 0.542
0.603 0.005
0.959 0.115
GTOPX
-195,586
-88055 20.878
-128.310 15.616
-132.023 63.084
-72.848 + 9,576
-117.022 $1.671
-124.995 56.170
GTOPX
-151.190
-64.128
-151.190
.45.6012 8,433
GTOPX
-215.711
-96.432 10.868
-215.716 0.000
-100.943 15,044
-84.271 8.307
-99.834
20.837
-110.284 17.559
GTOPX
-112.599
-64.217 14.602
-112.599 0.000
-71.749 # 28.497
-47.794 11,943
-71.174 12.932
-57.435 18.832
Avg. Rank
3.111 1.728
4.111 1.792
3.667 1.333
2.111 1.663
4.222 L.300
3.778 16188
30

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
E.7. Integration of LLM-based Heuristics
Note that recently LLM-based automatic heuristics design shows impressive performance in many fields (Romera-Paredee et al., 2024; Novikov et al., 2025; Zheng et al., 2025), in Appendix E.7, we investigate the integration of heuristies (Liu et al., 2024a;b;c) on UniSO-N to further improve performance. We adopt the approach in EoH (Liu et al., 2024a). Our implementation incorporates EoH from three key perspectives: (1): EoH-M: Summarizing better metadata; (2): EoH-R: Designing regularization trick to fine-tune the pre-trained LM embedder; (3): EoH-A: Implementing auxiliary loss based our improvement techniques to further enhance the model's performance.
Using Claude-3.5 -.55-etnett (Anthropic, 2024) as the backbone LLM, we design prompt templates following the EoH methodology. Results are provided in Fig. 11 and Table 20. The prompts are shown in Figures 16 to 24. The best individual solutions found by EoH-M, EoH-R, HoH-A provided in Figures 25 27, respectively.
The experimental results show that EoH-M captures more comprehensive metadata, leading to improved overall performance. Although EoH-R R underperforms, EoH-A further enhances UniSO-N's capabilities, showing the potential of LLM-base heuristics on improving the performance optimization algorithms.
Performance comparison of improved UniSO-N combined with EoH
2.00
D(best)
UniSO-N
Normalized Paaiizdd Performance
1.75
UniSO-N EOH-M
1.50
UniSO-N EoH-R
UniSO-N EoH-A
1.25
1.00
0.75
0.50
0.25
0.00
Ant
D'Kitt
Supercanductor
TF Bind
TF Bind 10
GTOPX2
GTOPX3
6TOPX4
GTOPX
Figure 11. Performance comparison improved UniSO-N combined with different EoH (Liu et al., 2024a) strategies. The bar plot shows the normalized performance of the baseline UniSO-N model and three eohanned variants (EoH-M: M: metadata summarization, EoH-R: R: regularization-basee
tieettnnng, EoH-A: auxiliary loss implementation) across nine optimization tasks. The red dashed lines indicate the best score D(best) in offline dataset for each task. Error bars represent standard deviations. The data is normalized by scaling UniSO-N's performance to for cach task.
Table 20. Un-normalized scores of improved UniSO-N and its variants with different Eoll (Liu et al., 2024a) strategies (EolI-M: metadata summarization. BoH-R: regularization-baat fine-tuning, Boll-A: auxiliary loss implementation) unconstrained tasks from Design-Bencl and S0O-Bende where the hest and runner-up results on each task are Blue and Violet, respectively. D(best) denotes the hest score in the offline dataset.
Task
D(best)
UniSO-N
UniSO-N EoH-M
UniSO-N EoH-R
UniSO-N BoH-A
Ant 165.326
269.691 77.425
316.515 + 19.141
306.577 42.614
339.697 = 52.210
D' K'tttty
199.363
173.911 = 46.662
127.644 = 13.676
197.777 0.000
218.026 4.858
Superconductor
74.000
67.333 10.838
67.894 + 5.964
84,215 4.306
85.186 + 11.518
TF Bind 8
0.439
0.833 + 0.005
0,859 = 0.167
0.739 + 0.075
0.971 = 0.000
TF Bind 10
0.005
0.959 + 0.115
0.626 0.032
1.256 + 0,000
0.778 + 0.074
GTOPX 2
-195,586
1224,995 + $6.170 170
66.611 + 32.466
-181.919 15.818
-119,284 + 14.325
GTOPX 3
-151.190
-62.622 = 22.261
-540032 = 13.158
-74.310 + 11,999
-67.640 + 2.542
GTOPX 4
-215.716
-110.284 17.559
-84.778 2.349
-136.679 + 21.707
-157.717 83,337
GTOPX
-112,599
-57.435 # 18.832
-74.435 1.096
-78.385 38.336
-63.365 1.828
Avg. Rank
/
2.667 0.943
2.333 # 1.155
3.000 + 1.054
2.000 + 1.054
31

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
F. Attention Weights Distribution Visualization
In this section, we provide the average attention weights distribution on all tasks that we use in this paper. Fig. 12 and Fig. 13 show attention distribution of T5-Small from pre-trained and from scratch, respectively, which further validate our observation of LM's harmful priors.
In Fig. 14 and Fig. 15, we compare different pre-trained LMs (T5-Small, T5-Base, Qwen2.5-1.5B, and DeepSeek-RI- Distill-Qwen-I-5 1.5B) GTOPX 6 and TF Bind 10 task, respectively. We find from the results that LMs that perform well on mathematical tasks shown great potential on numerical optimization tasks.
Jurrge Dertlettnn
InPRuce of Sipetenk tkde
NeW MENEE STEPO LOBE 20
Betrige essution ORE
Patraps CTOPA 1aden
Jesrag cosctinn DIa Folecran indi
Aarag
Figure 12. Average attention distribution of pre-trained T5-small embedder on data from different tasks. The plots present attention patterns for 12 distinct tasks: Ant, D' Kittty. Superconductor, TF Bind 8, TF Bind 10, GTOPX 2, GTOPX GTOPX 4, GTOPX 6, RobotPush, Rover, and Lnder, which consistently show that the pre-trained model exhibits strong attention bias towards the EOS token across all tasks, demonstrating the model's focus on structural elements rather than numerii-sollttion-releaaa components.
32

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
onl of Tmill
iillliigg hhmill
mastags www.
marag FEMESO ontrociv
BLEEO CTOPES LRBBEH
Clastion bistsate SPORC LIE (000-220
Cusaago STOPICK SROI
SMATgg REMESE UNETELATIO SeSCCSLL indan 144
Jestage POAYdi baitlad en howe
wimm rn mnnl mim Timil
M
Figure 13. Average attention distribution of T5-small emhedder trained from scratch different tasks. The plots show attention patterns across various components for 12 different tasks: Ant, D' D'Kitty, Superconductor, TF Bind 8, TF Bind 10, GTOPX 2, GTOPX 3, GTOPX 4. GTOPX 6, RobotPush, Rover, and LunarLander. The consistent pattern across tasks shows more balanced attention distribution on mumeric-solutioon-elevaa components, contrasting with pre-trained models' biases towards language-based structural tokens.
33

---

Towards Universal Offiine Black-Box Optimization via Learning Language Model Embeddings
Average Attention Distribttion of T5-Small on GTOPX 6
Average Attention Distribution Qwen2.5- 2.5 .. on GTOPX
Dny
Evt
Avarage Attantion Distribution TS-Base GTOPX
Average Attention Distribution DeepSeek-R1 Disill-Qwen- .55 GTOPX 6
Norma End Deban on Srrre
Comporencs
Componants
Figure 14. Different language models' average attention distribution on GTOPX 6 task. The bar plots compare the normalized attention scores across different components for four models: T5-Small (top left), Qwen2.5-1.55 (top right), T5-Base (bottom left), and DeepSeek- RI-Distil-Qwen-LL. (bottom right). Here, pre-trained T5 models exhibit strong attention bias towards the EOS token, while the DeepSeek-RI-DDee Dist--iiilll--ee---... shows more balanced attention distribution across numerii-sollttoo--elevaa components compared Qwen2.5-1.58 demonstrating the effectiveness of knowledge transfer in optimization tasks.
Average Attention Distribution of T5-Small on TF Bind 10
Average Allention Distribtiion Qwenn2 2-1 3-1.5 5B
TF Bind 10
0.30
0.25
0.15
Allention Distribulion of TS-Base Bind 10
Avarage Altention Distribution DeepSeck-Rt Dieteei---n---iii-i-n-.... 1.38 TF Bind
Norpateed Iraenton Srome
0.24
Dor
Gad
Compenenca
Componanta
Figure 15. Different language models' average attention distribution on GTOPX 6 task. The bar plots compare the normalized attention scores across different components for four models: T5-Small (top left), Qwen2.5-14 1.5B (top right). TS-Base (bottom left), DeepSeek- DDDitil-Qwen-L... (bottom right). Similar to that in GTOPX 6, pre-trained T5 models exhibit strong attention bias towards the EOS token, while DeepSeek-Re -1-eetil--- --Distil-Qwen-L-.. Qwen- shows more balanced attention distribution across cumeric-sollutinn-reeeaan components compared to Qwen2.5-1 5-5,B, demonstrating the effectiveness of knowledge transfer in optimization tasks.
34

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
G. EoH Prompts and Solutions
In this section, we provide prompts and best individual solutions for EoH (Liu et al., 2024a) integration experiments in Appendix E.7.
Prompt for Initialization
Prompt for E2
am solving universal offline black-box optimization (BBO) problem, i.., solving multiple offline BBO task simultancously. My approach is based on string- based representation of x by ,al1=**,2::*,.... I I use LM embedder to map the input string embedding space, and then apply MI.P head for regressing the objective. need help generating metadata for optimization tasks involving: robot morphology (Ant Morphology and D' Kitty Morphology). material science (Superconductor), DNA sequence optimization (TF Bind 8 and 10), and space mission trajectory optimization (GTOPX 2
am solving universal offline black-box optimization (BBO) problem, ie.. solving multiple offline BBO task simultancously. My approach is based string- based representation of by "[xI:**,x" ...."".. *,***.......
use
LM embedder to map the input string to an embedding space, and then apply an MLP head for regressing the objective. I need help generating metadata for optimization tasks involving: robot morphology (AntMorphology-v0 and DKittMoophooogy-y material science (Superconductor-vO) DNA sequence optimization (TF Bind 8 and 10), and space mission trajectory optimization (GTOPX 2 6).
tasks information:
Ant Morphology and D'Kitty Morphology: robot morphology optimization. We created these tasks optimize the morpho- logical structure of two simulated robots: Ant from OpenAl Gym and D' D'Kitty from ROBEL. For Ant Morphology, the goal is to optimize the morphology of an ant-shaped robot, to run as fast as possible, with pre-trained neural network controller. For DKitty Morphology, the goal optimize the morphology of D Kitty, a quadrupedal robot, such that a pre-trained neural network con- troller can navigate the robot to a fixed location. In this fashion, the goal for both tasks is recover a morphology with which the pre-trained controller compatible. The variable morphology parameters of both robots include size, orientation, and location the limbs, giving us 60 continuous values in total for Ant and 56 for D' Kitty. To evaluate the ground-truth value for given design,
tasks information:
have five existing metadatas follows:
No.
Code:
No.3
Code:
Please help me design a new algorithm that is different from the given ones but can be motivated by them.
robotic simulation in MuJoCo for 100 timesteps, averaging 16 independent trials. These parameters chosen reduce stochasticity and allow simulator to run minimal amount of time.
Firstly, identify the common backbone idea in the provided metadatas.
Secondly, based on the backbone idea create your new metadatas.
You need generate metadata for the 9 tasks above respectively, each metadata must be preceded by the task name, and do not generate additional explanations.
task, you to generate 3 versions of metadata, where each metadata must he preceded by the task name. Do not generate additional explanations.
Figure 16. Two examples of prompt engineering used in initialization and E2 strategy for EoH-M -.
35

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for
Prompt for M1
am solving universal offline black-box optimization (BBO) problem, i.e., solving multiple offline BBO task simultancously. My approach is based on string-based repre- sentation of by "(x1=*,,22: x2:**....)" ". use LM embedder to map the input string to an embedding space, and then apply an MLP head regressing the objective. need help generating metadata for optimization tasks involving: robot morphology (Ant Morphology and D'Kitty Morphology). material science (Superconductor), DNA sequence optimization (TF Bind and 10), and space mission trajectory optimization (GTOPX 2346).. 6).
am solving universal offline black-hox optimization (BBO) problem, i.e., solving multiple offline BBO task simultaneously. My approach is based string-based representation of "[x1:*# _2.** }". I use LM embedder to map the imput string to embedding space, and then apply MLP head for regressing the objective. need help generating metadata 9 optimization tasks involving: robot morphology (Ant Morphology and D' D'Kitty Morphology), material science (Superconductor), DNA sequence optimization (TF Bind 8 and 10), and space mission trajectory optimization (GTOPX 2 6).
tasks information:
have metadatas as follows:
No.1
Code:
tasks information:
No.3
I have existing metadatas as follows:
Code:
Please assist me in creating new metadatas that has different form but can be modificd version of the metadatas provided. You need to generate metadata for the tasks above respectively, each metadata must be preceded by the task name, and do not generate additional explanations.
Please help me create new metadatas that has totally different form from the given ones. You need to generate 9 metadata for the 9 tasks above respectively, each metadata must be preceded by the task name, and do not generate additional explanations.
Figure 17, Two examples of prompt engineering used EI and MI strategy for EoH-M.
36

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for M3
T am solving universal offline black-hox optimization (BBO) problem, i.e., solving multiple offline BBO task simultaneously. My approach is based on string-based representation of "[x1=*sx2: ,.."". I use LM embedder map input string to an embedding space, and then apply an MLP head for regressing the objective. need help generating metadata for 9 optimization tasks involving: robot morphology (Ant Morphology and D' Kitty Morphology), material science (Superconductor), DNA sequence optimization (TF Bind and 10), and space mission trajectory optimization (GTOPX 6).
tasks information:
First, you need to identify the main components in the metadatas below: No.1
Code:
No.3:
Code:
Next, analyze whether any of these components be overfit to the in-distribution instances.
Then, based on your analysis, simplify the components to enhance the generalization to potential out-of-distribution instances.
Finally, provide the revised metadata,, need to generate 9 metadata for the tasks above respectively, each metadata must be preceded by the task name, and not generate additional explanations.
Figure 18. Two examples of prompt engineering used in M3 strategy for EoH-M. Note that do not have M2 strategy, since M2 in BoH (Liu et al., 2024a) represents hyper-parameter parameter adaptation while metadata do not have any hyper ppper-paramttrs...
37

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for Initialization
Prompt for E2
I am solving universal offiine black-box optimization (BBO) problem, i.e., solving multiple offline BBO task simultaneously. My approach based on string- based representation of x by "[x1=*#xx2 .2::* use a language model embedder to map the input string embedding space, and then apply MLP head regressing objective. However. the LM embedder contains improper bias for regression. I need to design a novel loss function to optimize the embedding representations of different tasks in the language model's latent space. The goal form a clear distributional structure embeddings in latent space while maintaining distinguishabiliyy between tasks.
solving universal offline black-box optimization (BBO) problem, i.e., solving multiple offline BBO task simultaneously. My approach is based on string-based representation of x by .x2=**.. I use language model embedder to map the input string an embedding space, and then apply MLP head regressing the objective. However, the LM embedder contains improper bias for regression. need to design novel loss function to optimize the embedding representations of different tasks in the language model's latent space. The goal is to form clear distributional structure of embeddings in the latent space whilc maintaining distinguishability between tasks.
I have five existing algorithms with their codes as follows:
No.1 algorithms description: Code:
need to design novel loss function to optimize the embedding representations of different tasks in the language model's latent space. The goal is to form clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks.
No.3 algorithms description:
Code:
Please help me design a new algorithm that different from the given ones but can be motivated by them. Firstly, identify the common backbone idea in the provided algorithms.
Firstly, describe your new algorithm and main steps in one sentence. The description must be inside a brace.
Next, implement in Python as a function named latent_loss This function should accept two input(s): 'x L.embeddigg'. 'meta eebtaadinbdd.'' The function should return one output(s): loss. 'xeembedding" represents batch of embeddings be optimized with shape B*D (batch size embedding dimension). 'meta embeeddnnn represents metadata information also provided in batch embedding format with shape B*D. The output "loss" is the designed loss value. Note that both "__emhedding" and 'meta_emball eembedding are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computational stability. It important ensure the loss computation is mathematically sound and properly scaled.
Secondly, based on the backbone idea describe your new algorithm in one sentence.
Thirdly, implement it in Python as a function named 'la- tent_los'' This function should accept two inputs: 'xemmbedding', 'meta. eeeeedeembdiingg.. The function should return output: 'loss'. 'xembbedingg represents a batch embeddings to be optimized with shape B*D (batch size embedding dimension). beeet_eebeddingg represents metadata information also provided in hatch embedding format with shape B*D.The output 'loss' the designed loss value. Note that both 'x_embedding' and 'meta emhedding' are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computa- tional stability. It is important ensure the loss computation is mathematically sound and properly scaled.
All inputs and outputs are torch.Tensor. Do not give additional explanations.
All inputs and outputs are torch. Tensor. Do not give additional explanations.
Figure 19. Two examples of prompt engincering used initialization and E2 strategy for EoH-R.
38

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for E1
Prompt for M1
am solving universal offline black box optimization (BBO) problem, i..., solving multiple offline BBO task simultancously. My approach is based on string-basse based representation of by "[x1:**x22: .x:$**.....
am solving universal offline black-box optimization (BBO) problem, ie.. solving multiple offline BBO task simultancously, My approach based on string-bat based representation by "(x1=*xx2,** x2.**,... use language model embedder to map the input string to an embedding space, and then apply an MI.P head for regressing the objective. However, the LM embedder contains improper bias regression. need to design novel loss function to optimize the embedding representations of different tasks in the language model's 'ss latent space. The goal is to form clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks.
use a language model embedder to map the input string to an embedding space, and then apply MI.P head for regressing the objective. However, the LM embedder contains improper bias for regression. need to design novel Joss function optimize the embedding representations of different tasks in the language model's 's latent space. The goal is to form a clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks.
need to design an additional loss function to further fine-tune the embedder checkpoint. The goal is to obtain better performances of the final solutions over all tasks.
have one algorithm with its code as follows. algorithms description:
Code:
Please help me create a new algorithm that has a totally different form from the given ones.
Please assist me in creating new algorithm that has a different form but can be a modified version of the algorithm provided.
Firstly, describe your algorithm and main steps sentence. The description must be inside brace. Next, implement it in Python as a function named additional This function should accept two input(s): "_eembedding', 'meta_ mnett_embeddingg. The function should return one output(s): loss. 'xeembedding' represents a batch of embeddings to be optimized with shape B*D (batch size embedding dimension). 'meta eneea_eebeddingg represents metadata information also provided in batch embedding format with shape B*D. The output 'loss" the designed loss value. Note both embedding' and "meta embedding' are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computational stability. It is important to ensure the loss computation is mathematically sound and properly scaled.
Firstly, describe your algorithm and main steps in one sentence. description must be inside brace.
Next, implement it in Python as a function named additionalloss. This function should accept two imput(s): "x "__emedddngg'. "meta_ mneta_embeddinng. The function should return one output(s): loss. 'xeembedding' represents batch of embeddings to be optimized with shape B"D (batch size embedding dimension). 'meta eemeeddingg' represents metadata information also provided in batch embedding format with shape B*D. The output 'loss' is the designed loss value. Note that both "x embeddingg and 'meta embedding" are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex achieve effective embedding optimization while maintaining computational stability. is important to ensure the loss computation is mathematically sound and properly scaled.
All inputs and outputs Tensor. Do give addi- tional explanations.
All inputs and outputs torch. Tensor. Do give addi- tional explanations.
Figure 20. Two examples prompt engineering used in El and strategy for EoH-R.
39

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for M2
Prompt for M3
I solving universal offline black-box optimization (BBO) problem, i.c., solving multiple offline BBO task simultaneously. My approach is based on string-bases representation of x by "[x1:**,x2% xx.=**,...""""" I use language model embedder to map the input string an embedding space, and then apply MI.P head for regressing the objective. However, the LM embedder contains improper bias regression. need to design a novel to optimize the embedding representations of different tasks in the language model's latent space. The goal is to form clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks.
I am solving universal offline black-box opti- mization (BBO) problem, i.c. solving multiple offline BBO task simultaneously. My approach is based on string-hased representation x by "[x1: x2:=x.... ....}"
use language model embedding
embedder to map the input string space, then apply an MLP head regressing the objective. However, the LM embedder contains improper bias for regression. I need to design a novel loss function to optimize the embedding representations of different tasks in the language model's latent space.
function
goal is to form clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks.
need to design an additional loss function to further fine-tune the embedder checkpoint. The goal is to obtain better performances of the final solutions over all tasks.
Please identify the main algorithm parameters and assist me in creating a new algorithm that has different parameter settings of the score function provided.
First, you need to identify the main components in the algorithm below: algorithms description: Code:
Firstly, describe your new algorithm main steps in one sentence. The description must be inside a brace.
Next, analyze whether any of these components can be overfit to the in-distribution instances. Then, based your analysis, simplify the components enhance the generalization potential out-of- distribution instances. Finally, provide the revised code, keeping the function name, inputs, and outputs unchanged.
function named latent loss. This function should accept two input(s): 'x 'xeembedding', metta_embedding". The function should return one output(s): loss. "xeembedding" represents a batch embeddings to be optimized with shape B*D (batch size embedding dimension). 'meta_emmie eeee__embediing" represents metadata information also provided in batch embedding format with shape B*D. The output 'loss' is the designed loss value. Note that both embedding' and 'meta embedding" are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computational stahility. It is important ensure the loss computation is mathematically sound and properly scaled.
Next, implement it in Python
Note that both embeedding and 'meta embedding' are torch tensors with matching batch sizes. novel loss function should be sufficiently complex achieve effective embedding optimization while maintaining computational stability. It is important ensure the loss computation is mathematically sound and properly scaled.
All inputs and outputs are torch.Tensor Tensor. Do not give additional explanations.
All inputs and outputs are torch. additional explanations.
torch.Tensor Tensor. Do not give
Figure 21. Two examples of prompt engineering used M2 and M3 strategy for FoH-R.
40

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Prompt for Initialization
Prompt for E2
am solving universal offline black-box opti- mization (BBO) problem, i.e., solving multiple offline BBO task simultancously. My approach is based on string-baat based representation of X by "(xI:**,x2*** "..."". } use language model embedder map the input string to an embedding space, and then apply an ML.P head for regressing the objective. However, the LM embedder contains improper bias for regression. have fine-tuned T5-small embedder checkpoint that forms a clear distributional structure embeddings in the latent space while maintaining distinguishability between tasks via the following implementation:
am solving universal offline black-box optimization (BBO) problem, i.c., solving multiple offline BBO task simultancously. My approach is based on string-based representation of by "[x1:*8 xx:1= x2:
use language model embedder map the input string embedding space, then apply an MLP head for regressing the objective. However, the LM embedder contains improper bias for regression. I have fine-tuned a T5-small embedder checkpoint that forms clear distributional structure of embeddings in the latent space while maintaining distinguishahility between tasks via the following implementation:
Loss code in our implementation...
have five existing algorithms with their codes as follows:
Loss code in our implementation...
No.1 algorithms description:
Code:
I need to design additional loss function further tine-tune tune the embedder checkpoint, The goal is to obtain better performances of the final solutions over all tasks.
...
No.3 algorithms description:
Code:
Firstly, describe your new algorithm and main steps in sentence. The description must be inside brace.
Please help me design new algorithm that is different from the given but be motivated by them. Firstly, identify the common backbone idea in the provided algo- rithms.
Next, implement it in Python as function named Aaditiinnl.loss. This function should accept two input(s): x_embeddingg,, 'meta_emie mneta_ebbedding", The function should return one output(s): loss. "x emmbeddingg' represents a batch of embeddings to be optimized with shape B*D (batch size embedding dimension). represents metadata information also provided batch embedding format with shape B*D. The output 'loss' is the designed loss value. Note that both "xeembedding' and emeedeebbd torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computational stability. It is important to ensure the loss computation is mathematically sound and properly scaled.
Secondly, based on the backbone idea describe your new algorithm in one sentence.
Thirdly, implement in Python
function named 'addi- tional Joss'. This function should accept two inputs: e.mbeddding'' 'meta aneta_embedding The function should return one output: 'loss'. 'x eembedding'' represents batch of embeddings to be optimized with shape B*D (batch size embedding dimension). 'meta_emie ceettttebeddingg represents metadata information also provided in batch embedding format with shape D* The output 'loss' the designed loss value. Note that both 'x_embedding" and mneaa_mbeedding are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex achieve effective emhedding optimization while maintaining computational stability. is important to en- sure the loss computation is mathematically sound and properly scaled.
eeeeeddnne
All inputs and outputs are torch.Tensor. Do not give additional expla- nations.
All inputs and outputs are torch. Tensor. Do not give additional explanations.
Figure 22. Two examples of prompt engineering used in initialization and E2 strategy for BoH-A.

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for E1
Prompt for M1
am solving universal offline black-box optimization (BBO) problem, i.c., solving multiple offline BBO task simultancously. My approach is based representation of by "(xI:**x2:*** *2.""..."" use a language model embedder to map the input string to an embedding space, and then apply an MI.P head for regressing the objective. However, the LM embedder contains improper bias for regression. have fine-tuned T5-small embedder checkpoint that forms a clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks via the following imple- mentation:
am solving universal offline black-box optimization (BBO) problem, i.., solving multiple offline BBO task simultaneously. My approach is based on string- based representation of by "(x1:*Mx2:** x2:"*.... ... use language model embedder to map the input string to an embedding space, and then apply an ML.P head for regressing the objective. However, the LM embedder contains improper bias for regression. I have fine-tuned T5-small embedder checkpoint that forms clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks via the following imple- mentation:
string-based
Loss code our implementation...
Loss code in our implementation...
need to design an additional loss function to further fine-tune the embedder checkpoint. The goal is to obtain better performances of the final solutions over tasks.
have one algorithm with its code as follows. algorithms description:
Code:
Please help me create a new algorithm that has a totally different form from the given ones.
Please assist me in creating new algorithm that has different form but can be modified version of the algorithm provided.
Firstly, describe your new algorithm and main steps one sentence. The description must be inside a brace.
Firstly, describe your new algorithm main steps in one sentence. The description must be inside a brace. Next, implement it in Python additional_ loss. This function should accept two input(s): "x "eembedding',. "meta cenecadmbbdiing" The function should return one output(s): loss. 'xeembedding' represents a batch of embeddings to be optimized with shape B*D (batch size embedding dimension). 'meta_ene mneta_embedding represents metadata information also provided in batch embedding format with shape B*D. The output 'loss' is the designed loss value. Note that both "x embedding' and 'meta_embeddie embedding"" are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex achieve effective embedding optimization while maintaining computational stability. it important to ensure the loss computation is mathematically sound and properly scaled.
Next, implement it in Python as function named additionalloos. This function should accept two input(s): "_ebbedding'' "meta_eme eerttadmbediiig. The function should return one output(s): loss. 'xemmbediing" represents a batch of embeddings to be optimized with shape B (batch size embedding dimension). 'meta_embedee embedding' represents metadata information also provided in batch embedding format with shape B*D, The output 'loss' the designed loss value. Note that both 'x embedding" and "meta_embeddingg are torch tensors with matching batch sizes. The novel loss function should be sufficiently complex to achieve effective embedding optimization while maintaining computational stability. is important to ensure the loss computation is mathematically sound and properly scaled.
function named
All inputs and outputs are torch. tional explanations.
torch.Tensor. Tensor, Do not give addi-
inputs and outputs are tenso.Tensor. tional explanations.
Tensor. Do not give addi-
Figure 23. Two examples of prompt engineering used in El and MI strategy for Eoll-A.
42

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
Prompt for M2
Prompt for M3
am solving universal offline black-box optimization (BBO) problem, i.e. solving multiple offline BBO task simultaneously. My approach is based on string- based representation of x by map
I am solving universal offline black-box opti- mization (BBO) problem, i.c. solving multiple offline BBO task simultaneously. My approach is based on string-hased representation x by "(x1: x2:$x..... ....}"
"(xI:** _2.*** ""."". 1 use language model embedder the input string an embedding space, and then apply an MLP head for regressing contains improper hias for regression. I have fine-tuned T5-small embedder checkpoint that forms clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks via the following implementation:
use language model embedding
objective. However, the L.M embedder
embedder to map the input string space, then apply an MLP head regressing the objective. However, the LM embedder contains improper bias for regression. have fine-tuned a T5-small embedder checkpoint that forms clear distributional structure of embeddings in the latent space while maintaining distinguishability between tasks via the following implementation:
Loss code in our implementation...
need design an additional loss function to further fine-tum ine-tune the embedder checkpoint. The goal is to obtain better performances of the final solutions over all tasks.
Loss code in implementation...
First, you need to identify the main components in the algorithm below: algorithms description: Code:
Please identify the main algorithm parameters and assist in creating a new algorithm that has different parameter settings of the score function provided.
Firstly, describe your algorithm and main steps in one sentence. The description Next, implement it Python as a function named additional loss. This function should accept two input(s): 'x eembeddingg,, mneta.embeddinng.. The function should return one output(s): loss. 'xeembedding' represents batch of embeddings be optimized with shape B" (batch size * embedding dimension). mnetaeembedding" represents metadata information also provided in a batch embedding format with shape B*D. The output "loss" is the designed loss value. Note that both "xeembedding' and vneta_embedding' are torch tensors with matching hatch sizes. The novel loss function should sufficiently complex to achieve effective embedding optimization while maintaining computational stability. It is important to ensure the loss computation is mathematically sound and properly scaled.
Next, analyze whether any of these components can be overfit to the in-distribution instances. Then, based your analysis, simplify the components enhance the generalization potential out-of- distribution instances. Finally, provide the revised code, keeping the function name, inputs, and outputs unchanged.
be inside brace.
Note that both 'x emmbedding and 'meta embedding' are torch tensors with matching batch sizes. novel loss function should be sufficiently complex achieve effective embedding optimization while maintaining computational stability. It is important ensure the loss computation is mathematically sound and properly scaled.
All inputs and outputs are torch. additional explanations.
Tensor. torch.Tensor
Do not give
All inputs and outputs are Tensor. Do not give additional explanations.
Figure 24. Two examples of prompt engineering used in M2 and M3 strategy for EoH-A.
43

---

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
Ant Morphology
("type": "rebot_ serbot_ddeiggn"" "dimensions"; 60, Sinput_ranger 1-1.0, 1.0),
10, infl, "aptimization tazget:: e 16,
"simulation_engina "MuJoCo", 100)
D'kitty Morphelogy
I"type": "robet_dee" fesst__desggn,, "dimensions"; 56, "input_range";
range": [-1.0, 1.0), "output _aagte: [0, infl, "optimization target": "target searget_recching" "simulation iegiiionnnnnn "MuJoCo", evalnation_trials": 16, "simulation ienilatio__eenghhh 100)
Superconducter
("type"; "materials_science" "dimensions"; 81, "input_range"; [0.0, 1.01, "output_range" [0, infl, opptimization_targeeee "critical_ semprcaalttemerauurrr""..""" "model_type": sandon_forest", daataset_exmpless" 21263)
IF Bind 8
"type": "dna "dna_ design", Seruuence__engtiine 8, "alphabet": 1"A", "C", "G", "T"I, output output_range" 65792, exraining_exampiess 32898)
zange": 10, 1), "optimization taaiiaatioontargtt... "binding affinity", "search space";
TF Bind 10
("type": "dna den_gdesi"", "sequence "eanuennc_leegthh; 10, "alphabet": ["A", "C", "G", "T"], butput_range" [0, 1), Coptimization_targeet sandinngaffiiity" srearch_spacer 1048576, briining_xxappesse 50000)
GTOPX 2
("type": "spacecraft_ "tajjettttttt. "dimensions"; 22, "input_range"; [-1.0, 1.0), output_range": [o, infl, "optimization_targee "delta_v", missionn_type" eaturn_rendezoous", "naneuver"; "dsn")
GTOPX
I"type:: "spacecraft " "dimensions": 18, Ciaput_rangel" [-1.0, 1.01, output :atput_ranne;: [0, inf], anptimization_tareeee "delta_v"; "mission_typer ternuey,transfer" "flyby_type"; "non_ resonant"!
GTOPX
("type": "spacceraff__trejettorr""" "dimensions": 26, "input_rango"; [-1.0, 1.0),
output_range": [0, inf], eeeiimiaationttargeee "delta_v", "mission_type": mercury_transfer,, "flyby_type": "resonant")
GTOPX ("type":
"dimensions"; 22, "ingut_range"; [-1.0, 1.0), 10, infl, "optimization_tarc"" targett: "delta_v", "mission_type" comet_rendezvous" "maneuver"; "muloi_gravity__ssiittt
Figure 25, Best metadata found by EoH-M.
44

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
import torch
import torch.nn. wunctionaa
def latent_loss (x_embedding: torch.Tensor, meta_embeddingg: Tensor) torch, Tensor:
batch_size x_embedding _eeneedding.shape((O shape
Multi-scale kernel density estimation
scales terch. tensor (10.1, 0.5, 1.0], device-x embedding.devica.
dist_matri: torch. cdist (K_embedding, x x_enbedding)
10
density_loss =
for scale in scales:
12
kernel corch.exp (-dist_matriis / scale scale))
13
density kernel. sum (dim=l) / batch_size
14
density_ loss +- -torch. (density + 1e-6) .maan ()
Diffusion distance mapping transition kernel / hernel.sumddire keepdin=True) diffused_ x forch.matmul. (transition, x embedding))
17
18
19
liffusicn_lcss torch x - x_enbedding, dim=1) mean ()
21
Task-spccific repulsion
22
x_norm F.nrrmalize (x embeddiing, dim=1) meta_norm F. ..normaliz (meta_embedding. dim=l)
23
task_sim = terch. matmul 1 (x_norm, rata_norm..0 ())
task_probs F. soffftmx (taskssim / 0.1, dim=1)
26
27
task repulsiin corch.matmin (task probs,, bask_probs.t(0 repulsicn less torch.mean tttaskrrepulseee
torch. (-disttmatrixx) matrix)
30
Smooth boundary transitions
31
boundary_kernel torch.exs (-task_sim)
32
boundary_denstee = Norch.matmul becnnddar__eereeeee task task_probs)
smoothness__oce loss____________________________ = torch.meat mean beeundary_densieee task task_probs) +*2)
35
Combined loss with balanced weights
36
loss = (0.3 * density_loss +
37
0.3 diffusion_loss +
0.2 repulsion_ioss
0.2 smoothness_1oas)
40
41
return loss
Figure 26. Best code found by EoHI-.. Description by the model: the algorithm extends the local structure preservation concept by incorporating adaptive kernel-based density estimation with multi-scale diffusion distances, while utilizing task-specific repulsion forces and smooth boundary transitions through probabilistic assignment matrices.
45

---

Towards Universal Offline Black-Box Box Optimization via Learning Language Model Embeddings
import terch
def additional loss (x embedding:
terch. Tensor, meta embedding: Forch.Censon torch.
Tensor:
x_dist terch, edist ix_embedding, x_embedding)
i_dist tcrch. edist (meta_embeddingg meta_embeddinng)
rank corch.argsen (torch.argsore (x __dist, dim=1), dim=l) float ()
m rank torch. argoott (torch.argsort (m dist, dim=l), dim=l) , float ()
rank_corr torch.mean (x rank m_rank) xx2)
11
12
dist_ x_dist / dist 1e-8)
13
consistency = torch.ment mean (torch. (dist_ratio -
(dist_ratio))
15
return rank_corr 0.2 consi stency
Figure 27. Best code of EoH-A. Description by the model: balance pairwise distances in both embedding and metadata spaces while enforcing consistency through rank correlation.
46


Reviews:
[{"review_id": "ce6AdzaEzP", "reviewer": "Reviewer_TeSq", "summary": "The paper explored using LLM embeddings as the universal representation space for offline black-box optimization across different domains. The paper discussed two approaches, next-token prediction and numeric prediction, and proposed several ways to constrain the learned latent space for effective offline BBO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below please find our response.  Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/TeSq.pdf.\n\n## R1 Some typos\n\nThanks for pointing these out. We have revised them and checked them thoroughly. Thank you very much.\n\n## R2 Are single-task experts the same as numeric-input experts?\n\nYes, the single-task experts are numeric-input experts. We will revise it to make it clear. Thank you.\n\n## R3 More elaboration on why LLM cannot perform well on BBO via ICL\n\nThanks for your insightful comments. The no free lunch (NFL) theorem [1-2] states that no single algorithm can universally outperform others on average over a uniform distribution of learning problems (i.e., all problems are equally likely, with no inherent structure or bias). Crucially, LLM’s success in ICL inherently violates this uniformity assumption [3]: real-world tasks (e.g., language, vision) occupy a highly structured subspace (e.g., grammar rules, spatial locality), which LLM exploits through pre-training on non-uniform, human-aligned data distributions. In contrast, general BBO problems—when stripped of metadata—align with the NFL assumption, because the lack of prior knowledge about problem distributions (e.g., smoothness) forces algorithms to treat the space as uniform [1], rendering cross-problem generalization impossible. Thus, we introduce metadata during training, which is an attempt to explicitly break this uniformity by restricting the problem space to a structured, domain-specific subspace, thereby enabling algorithms to leverage these useful biases. We will revise it to make it clear. Thank you.\n\n## R4 Why not train an in-context regressor similar to Predicting from Strings paper?\n\nThanks for the valuable question. Unlike online BBO, traditional offline BBO only allows evaluation of solutions in **one batch** and does not utilize the in-context information. Following the common practice in offline BBO [4], we use MLP as the regressor. We agree that it is valuable to test the performance of an in-context regressor. According to your suggestion, we consider offline BBO as a one-epoch online BBO and implement the in-context regressor using TNP [5], treating the top-scoring offline solutions as the contexts. Following Predicting from Strings [6], we maximize the UCB acquisition through EA to obtain final candidates.\n\nResults in Table S1 show that UniSO-N + TNP performs worse than UniSO + MLP. This suggests a potential mismatch between ICL and the offline BBO setting. Extending these approaches to universal offline BBO remains an interesting future work. We will include this discussion in our revised paper. Thank you.\n\n## R5 What is the encoder in Section 3.3?\n\nUniSO-N: encoding input strings by a pre-trained T5-small encoder; UniSO-T: training a T5 encoder-decoder from scratch and using its encoder to encode input strings; The encoders for metadata of two approaches are the pre-trained T5-small encoder. We will revise it to make it clear. Thank you.\n\n## R6 Don’t UniSO-N train with the regressor during finetuning?\n\nThanks for pointing this out! We are sorry for not making it clear. In fact, during finetuning, we update the embedder via $\\frac{\\partial L_{total}}{\\partial\\boldsymbol{\\theta}}=\\frac{\\partial L_{con}}{\\partial\\boldsymbol{\\theta}}+\\frac{L_{con}}{L_{lip}+\\delta}\\cdot\\frac{\\partial L_{lip}}{\\partial \\boldsymbol{\\theta}}$ without incorporating the main loss. We will revise it. Thank you very much.\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**\n\n### References\n\n[1] No free lunch theorems for optimization. IEEE TEvC, 1997.\n\n[2] The supervised learning no-free-lunch theorems. Soft computing and industry: Recent applications, 2002.\n\n[3] Position: The no free lunch theorem, Kolmogorov complexity, and the role of inductive biases in machine learning. ICML, 2024.\n\n[4] Offline model-based optimization: Comprehensive review. arXiv, 2025.\n\n[5] Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. ICML, 2022.\n\n[6] Predicting from strings: Language Model embeddings for Bayesian optimization. arXiv, 2024.", "claims_and_evidence": "The paper made three claims: (1) string-based latent space enables universal offline BBO, (2) geometric regularization of the latent space improves optimization, and (3) metadata helps generalization to unseen tasks. The ablation studies empirically support claims (2) and (3). Claim (1) is true to some extent, since they showed a single model can generalize to many optimization tasks. However, the performance of the universal model is inferior to existing task-specific approaches.", "methods_and_evaluation_criteria": "Yes, they evaluated their proposed method on commonly used datasets and tasks in offline BBO.", "theoretical_claims": "The paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes, they followed the same experimental protocol as the existing works in the literature. I did not find any issues.", "relation_to_broader_scientific_literature": "The paper is strongly related to prior works in universal Bayesian optimization, which rely on LLMs to serve as a universal surrogate model across different optimization tasks. This paper extended the idea to the offline BBO setting, and proposed several modifications to make the latent space suitable for offline optimization.", "essential_references_not_discussed": "I did not find any missing references.", "other_strengths_and_weaknesses": "The main weakness is the inferior performance of the method compared to existing task-specific approaches (Table 8). The promise of having a universal model for black-box optimization is that we can train the model on massive data and exploit the relations between tasks to enhance performance on individual tasks. However, this is not sufficiently shown in the paper, as the proposed method underperforms recently published works.", "other_comments_or_suggestions": "- Typo: \n  - Line 355, The term “Improved” represents the our improvements => The term “Improved” represents our improvements\n  - Line 356: regular => regularize\n - In table 2, are single-task experts the same as numeric-input experts? If this is true, please use the same name to avoid confusion.", "questions_for_authors": "- In 3.1.2, the authors said \"(1) utilizing a single algorithm to solve multiple offline BBO problems simultaneously solely on general design-score data is impossible due to the no free lunch theorem\". Can you elaborate? LLMs can simultaneously perform many tasks via in-context learning without any metadata, why can't BBO?\n- For UniSO-N, why didn't the authors train an in-context regressor similar to the Predicting from Strings paper? Training an MLP means the model has to rely on the metadata alone to distinguish between tasks.\n- In 3.3, the authors said \"we first encode the input strings through the encoder to derive their embeddings, while simultaneously\nemploying a pre-trained expert language model embedder (t5-small by default) to generate embeddings for metadata\". What is the encoder? Is it not the same as the t5-small model used to encode metadata strings?\n- Per the last paragraph of 3.5, in the first stage, don't you have to train the regressor too, because only so you have the main loss?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xLA4XoWWde", "reviewer": "Reviewer_4Ajw", "summary": "This paper introduces UniSO, a novel framework for universal offline BBO that leverages string embedding spaces to address limitations of traditional methods in heterogeneous search spaces and cross-task generalization. Using LM embeddings, it converts numerical parameters into strings and proposes two variants: UniSO-T (seq-to-seq token predictor) and UniSO-N (embedding + MLP regressor). Enhancements via metadata-guided embedding alignment (heuristic projection tricks to encourage embedding to be well-clustered) and local smoothness (heuristic tricks enforcing local embeddings to be smoothed as their objective scores) improve performance.\n## update after rebuttal\n### *After Discussion*\n\n*I very much appreciate the author’s respect on my ideas in improving the work, as well as the respect for my research taste on offline BBO in ICML. Also, I appreciate the author’s hard work during the rebuttal.*\n\n1. ***Respect and realization of my proposed cluster-utilized BBO and replacement of EA**. I again appreciate the authors’ attempts in adopting methods beyond their original EA: BO-qEI and CMA-ES, and found BO-qEI’s effectiveness. I thank the authors’ recognition of my explanations to the reason (inherent cluster-utilized GP nature), and provide visualizations to aid the claims.*\n2. ***Respect and realization of my proposed mechanic interpretation and EoH-based improvement**. I highly value the author’s improvements in these trends. I believe the true power of LLM in assisting offline BBO is in these regards currently: (i) whitening problem-method’s black-box nature as possible by implicit spectral clustering properties; (ii) using LLM-based heuristics that could be better than human design. I especially appreciate the average attention visuals and the three approaches the author added, which show interpretability and effectiveness in certain regards.*\n3. ***Extended Experiments on other LLMs and computational budget**. I appreciate the author’s attempts in adopting other LLM, such as Deepseek and Qwen, and provide mechanic interpretative visuals to validate the evidence. I appreciate the extended time budgets studies after my replies to the rebuttal.*\n4. ***Promise in revision and code availability**. I appreciate the author’s promise in adding an algorithm block w.r.t. BO-qEI and Figure 1 in the main body, and the details provided in the appendices. I appreciate the author’s willing to incorporate so many improvements in the revised paper, and their willingness to make the code available, which would greatly contribute to the BBO community.*\n\n*Therefore, despite requiring a big revision, the revised paper warrants ICML’s attention and I would increase my score from 2 to 5 to strongly support the paper. I'm waiting to see more details in these improvements in the revised version. I would appreciate if the authors could clearly acknowledge us anonymous reviewers in the Acknowledge Section to indicate which ideas and improvements are based on the reviews of anonymous reviewers during the ICML rebuttal period. It’s my honor to have the authors’ respect and I wish good luck for the author’s future journey in offline BBO.*", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below please find our response. Corresponding experimental results and references can be found at https://anonymous.4open.science/api/repo/UniSO/file/4Ajw.pdf.\n\n## R1 EA-based BBO method doesn’t warrant ICML community’s much attention; why don’t you adopt more effective optimizers (e.g., BO) instead of EAs?\n\nWe understand your concerns, but there is a misunderstanding. Our work **is not** an EA-based BBO method. We focus on offline BBO, a recent popular topic in ML community (A paper list can be found in Table S1). However, none of these studies considers the universal settings, which is a long-standing challenge in BBO.\n\nIn offline BBO, multiple optimizers (e.g., gradient ascent, EAs) can be applied to search in the model. Since gradients are difficult to compute in string space, we use EA instead, which is a common practice in offline BBO. In fact, any BBO algorithms are applicable. According to your comments, we further add two methods: BO-$q$EI (based on BoTorch) and CMA-ES (based on evosax). The results in Table S2 show that BO-$q$EI performs best and the default EA used in our experiments is still a competitive method.\n\nWe will add these discussions to our revised paper. Thank you.\n\n## R2 The preference of ML community: Either prefer provable algorithms or using LLM-based heuristics\n\nThanks for your comments. However, we beg to differ with you on this matter. A position paper at ICML’24 [1] comprehensively discusses LLM for BBO. In our opinion, there are two major branches in LLM for BBO:\n\n- Leveraging general intelligence (e.g., reasoning, coding) of LLM to design algorithms or heuristics via **natural language** [2-3]. As LLM shows impressive general intelligence, several works utilize LLM to generate optimization code and heuristics by prompt engineering based on natural language. Representative works such as FunSearch [4] and EoH [5] iteratively search for better algorithms and heuristics, and GENIUS [6] employs LLM to find better neural architecture code.\n- Leveraging the representation ability of LLM to improve the ability of learned BBO algorithms via **tokens or embeddings** [1].  This branch mainly focuses on how to efficiently utilize the inner embeddings of LLM. For example, [7] shows that when numeric $y$ is tokenized into a sequence, LLM is capable of universal regression via pre-training at scale; [8-9] show the superiority of LLM embedders for universal regression and BBO tasks, while [10-11] represent input values as tokens and thus learn in-context via LLM to solve BBO.\n\nOur work falls into the second category, especially extending the scope to universal offline BBO for the first time. We deliver a unified representation that enables universal offline BBO possible and propose two effective advancements to overcome the barriers. The first branch you mentioned could help further enhance the performance of our approach. Thanks to the recent rapid development of frameworks [3], we can conveniently test whether this idea works in the future. \n\nAdditionally, the performance of our proposed method is not entirely heuristic; it is based on some theoretical foundations.\n\n- $L_{con}$ is used for embedding distribution alignment, which is actually isomorphic to the relaxed variant of InfoNCE, a popular loss in contrastive learning that has been well studied [12].\n- $L_{lip}$ aims to enhance smoothness. [13] has proven that it supports the lower bound on the correlation of the embedding space and the objective space. Besides, Figure 4 in [8] shows the importance of the Lipschitz continuity for regression at different model scales and problem dimensions.\n\nOverall, we believe that our work contributes substantially to the LLM for BBO community. We will revise to add these discussions. Thank you very much.\n\n## R3 Computational cost\n\nPlease refer to R1 of Reviewer F6Mo due to space limitations.\n\n## R4 Details of EA\n\nThank you for pointing this out! We are sorry for not making it clear. We initialize the population as the top-$k$ scoring designs in the dataset, where $k$ is the population size. For continuous tasks, we use simulated binary crossover and polynomial mutation; For categorical tasks, we use uniform crossover and random replacement mutation. These operators are the default operators in [14]. We will add these to our revised paper. Thank you.\n\n## R5 UniSO-T vs. UniSO-N\n\nPlease refer to R5 of Reviewer F6Mo due to space limitations.\n\n## R6 Influence of metadata components\n\nThanks to your suggestion, we have conducted experiments to verify the individual contributions of metadata components. The results in Table S3 show the effectiveness of each component in metadata. We will add more detailed discussions to our paper. Thank you.\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.** \n\nReferences are provided in Table S4.", "claims_and_evidence": "- **Claim 1**: String embeddings unify heterogeneous search spaces for universal offline BBO.\n    - **Evidence**: UniSO excels on multi-task datasets (e.g., Ant Morphology, TF Bind 8), outperforming baselines (Tables 1 & 2).\n- **Claim 2**: Metadata guidance and geometric regularization boost optimization stability.\n    - **Evidence**: t-SNE plots (Fig. 3) show improved embedding separation; enhanced UniSO-T outperforms vanilla versions (Table 2).\n- **Claim 3**: UniSO generalizes across tasks and to unseen scenarios.\n    - **Evidence**: Zero/few-shot results on unseen tasks (e.g., RobotPush) surpass dataset bests and single-task experts (Table 3).", "methods_and_evaluation_criteria": "- **Methods**:\n    - String representation (e.g., JSON format).\n    - UniSO-T (seq-to-seq token predictor) and UniSO-N (embedding + MLP regressor).\n    - Improvements: metadata-guided alignment (heuristic projection tricks to encourage embedding to be well-clustered), embedding smoothness (heuristic tricks enforcing local embeddings to be smoothed as their objective scores).\n    - Model-inner search via EAs.\n- **Evaluation**:\n    - Un-normalized scores compared to dataset bests and single-task experts.\n    - Tasks from Design-Bench, SOO-Bench, and real-world scenarios.\n    - RQs assess versatility, multi-task performance, and generalization.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- **Design**:\n    - Datasets from Design-Bench and SOO-Bench (DOUBLE, CATEGORICAL spaces).\n    - Baselines: single-task experts (BN+EAs, BN+Grad).\n    - T5-based training; zero/few-shot tests on unseen tasks.\n- **Analysis**:\n    - t-SNE validates embedding improvements.\n    - Ablation studies (Tables 4-7) assess metadata/loss contributions.\n    - Average ranks compare method performance.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Innovative string-based approach breaks single-task, fixed-dimension barriers.\n2. Comprehensive experiments span multiple scenarios with good-illustrated results.\n3. Code availability promised, aiding reproducibility.\n\n**Weaknesses**\n\n1. EA-based BOO method is a topic that doesn’t warrant ICML community’s much attention, and your methods utilize LM embeddings in a very heuristic manner. Especially, your tricks in 3.3 and 3.4 are empirical based on prior work, which is without serious provable guarantee and might be suboptimal. Nowadays, ML community either prefer provable algorithms (with assumptions over problem structure) or using LLM-based heuristics (EoH in Liu et al. (2024)) to design heuristic tricks, since human-designed heuristics are usually sub-optimal and easily defeated by LLM-based heuristics. \n2. Computational cost of training/search not detailed, potentially limiting scalability.\n3. The author didn’t show their evolutionary algorithms procedures in the main body, which serve as a crucial rule.\n4. Since t-SNE can enable clear cluster structures, why don’t you adopt more cluster-based Learning methods or corresponding Bayesian OPT for BOO insteads of EAs?", "other_comments_or_suggestions": "I suggest including detailed EAs for each tasks, hyperparameter, training time details in the appendix.", "questions_for_authors": "1. What explains UniSO-T vs. UniSO-N performance differences across task types (continuous vs. discrete)?\n2. Have you evaluated the individual contributions of metadata components (name, description, objective)?\n3. See Weakness section above.\n\n*Reference*\n\nLiu et al. (2024) Evolution of heuristics: Towards efficient automatic algorithm design using large language model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zlVLyTCm0y", "reviewer": "Reviewer_GL3R", "summary": "This paper proposes to use recent works in LLMs as regressors for offline blackbox optimization. The key contributions are:\n  * Organizing the same LLM bodies with different regression heads as UniSO-T (token output) and UniSO-N (numeric output).\n  * Proposing two regularization methods, i.e. (1) a contrastive loss to make inputs from the same task clustered and (2) smoothness alignment to make sure the embeddings form a reasonable objective landscape\n  * Comprehensive experiments on DesignBench to show:\n    * UniSO-T generally is better than UniSO-N across all scenarios.\n    * UniSO-T performs competitively in apples-to-apples comparisons in single-task settings.\n    * Adding the proposed regularization makes UniSO-T even better.\n    * If UniSO-T is allowed to pretrain on multiple tasks, it does even better, and showing task metadata also helps.\n    * Among multiple different works in offline blackbox optimization, UniSO-T performs decently, but there is room for improvement.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging and valuable comments! Below please find our response. Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/GL3R.pdf.\n\n## R1 Is \"expert\" in Table 1 a basic MLP regressor?\n\nYes. It refers to a numeric-input MLP regressor. We will revise to make it clear. Thank you.\n\n## R2  Future work like more data or more exotic search spaces\n\nThanks for your valuable comments! We wholeheartedly agree with your ideas. There are many interesting directions for future work:\n\n- Training with more data with balanced ratios from different trials;\n- Applying different modalities as metadata, instead of solely upon human-summarized text;\n- Designing efficient numeric representation or tokenization.\n\nWe will keep focusing on discovering better paradigms for universal BBO. Thanks for your encouragement!\n\n## R3  Advice for better visualization\n\nWe agree that adding more styles of presentations will be better for understanding. According to your suggestions, we additionally show the generalization capability of improved UniSO-T method on unseen tasks in Figure S1, which can better show the generalization and effectiveness of few-shot finetuning. We will change other experimental presentations in our paper in the revised version. Thank you very much!\n\n## R4 UniSO-T vs. UniSO-N\n\nPlease refer to R5 of Reviewer F6Mo due to space limitations.\n\n## R5  y-normalization\n\nWe normalize y from multiple tasks following [1], as discussed in Section 3.3:\n- Apply z-score normalization inside a task to shift the score to a standard normal distribution;\n- Perform a normal distribution fitting process on the subset of data points whose values are below the median. To decrease the influence of extreme outliers, we use a transformation from percentiles to z-scores.\n- Transform the scoring distribution through sequential application of min-max scaling $y\\leftarrow\\frac{y-y_{min}}{y_{max}-y_{min}}$ and logarithmic transformation for re-scaling.\n\nIt is quite essential for UniSO-N to normalize y, since an MLP regressor requires y normalization to prevent gradient explosion or vanishment. But objective normalization can be problematic when encountering outlier feedback values at test-time [2] (especially when few-shot evaluations are allowed). How to eliminate such issues (e.g., modeling of UniSO-T does not require y-normalization) for BBO is a promising future work.\n\n## R6  Is it necessary to start off with a pre-trained checkpoint, or does random initialization work fine too, for the T5-small model?\n\nThanks for your constructive question! We conduct a more detailed analysis and find that it is not necessary to start off with a pre-trained checkpoint. Pre-trained language model checkpoints contain priors from natural language, which, however, would introduce harmful bias for offline BBO. \n\nFrom the final performance in Table S1 and OOD rank correlation (an important metric for offline BBO [3]) curve in Figure S2, we find that UniSO-N trained from scratch consistently performs better than that from a pre-trained checkpoint. The pre-trained models initially demonstrate poor performance, and there is only little improvement later on. These biases may come from different modalities between natural language and numeric representation [4], and how to mitigate this gap and align different modalities well is critical future work in our community.\n\nWe will add this discussion to our revised version. Thank you again for the insightful suggestion!\n\n## R7  T5 model size\n\nThanks for your insightful question! Due to computational resource constraints, we can only employ models of similar size to T5-Small. As a follow-up experiment, we further train a larger model of UniSO-N, which utilizes a T5-Base embedder. However, the performance results in Table S2 and the loss curve in Figure S3 show that equipped with a larger T5-Base embedder, UniSO-N performs even worse. Since a larger pre-trained model incorporates more prior, there may introduce more unfavourable biases for offline BBO, which also aligns with our observation in our response in R6. As for UniSO-T, we fail to train with a larger model size due to limited computational resources (since UniSO-T occupies more GPU memory than UniSO-N). \n\nThanks for your great question! We will include more a detailed discussion on this point.\n\n---\n\n**Thank you again for your encouraging comments! We hope that our response has addressed your concerns, but if we missed anything please let us know.** \n\n### References\n\n[1] Predicting from strings: Language model embeddings for Bayesian optimization. arXiv, 2024.\n\n[2] OmniPred: Language models as universal regressors. TMLR, 2024.\n\n[3] Offline model-based optimization by learning to rank. ICLR, 2025.\n\n[4] Position: Why tabular foundation models should be a research priority. ICML, 2024.", "claims_and_evidence": "Yes, I could follow all of the conclusions made by the experiments - they are very clean (i.e. A/B tests via Tables to show one setting is better than another, making everything else equal).", "methods_and_evaluation_criteria": "Yes, DesignBench is a well known benchmark for offline blackbox optimization.", "theoretical_claims": "N/A - there are no theoretical claims, and that's fine.", "experimental_designs_or_analyses": "Yes, I checked all details. No issues here.", "relation_to_broader_scientific_literature": "The paper contributes well to the broader issue of \"universal\" regression methods for blackbox optimization. It demonstrates that recent works in LLM regressors can be used directly in optimization scenarios, and the addition of the general regularization methods improves optimization results significantly.\n\nThere are many interesting avenues of future work, e.g.\n  * Pretraining such regressors over a large amount of data (rather than just the data from DesignBench)\n  * Applying such techniques for more exotic search spaces (e.g. code)", "essential_references_not_discussed": "No issues - The paper's references are comprehensive.", "other_strengths_and_weaknesses": "### Strengths\n* The regularization methods make sense, but are non-obvious (i.e. the average blackbox optimization researcher is unlikely to be aware of them), and introducing them is a strong contribution for shaping LLM-embeddings better for objective functions. These regularization methods are also fairly model-agnostic, and can be applied to any LLM model and any optimization problem.\n* This paper is very clear on its experimental conclusions - I could follow every single research conclusion and agree on based on the evidence. I learned a lot from this paper as a whole.", "other_comments_or_suggestions": "### Experimental Result Presentation\nThis is a minor point, but I would suggest that the authors try to make the results more aesthetic. While I can follow the tables and their implied conclusions, there are too many tables - perhaps consider balancing between using histograms, bar plots, tables, scatter plots, etc. for better presentation - it might help the average reader unfamiliar with the field (who might get bored of too many tables).\n  * I understand for example, different tasks have different scales, which makes it hard to put all of them in the same figure. In this case, consider grouping, e.g. (TF-Binds in one subplot, GTOPX's in another...)\n\n### Additional Conclusions\nIt's interesting how UniSO-T consistently beats UniSO-N. Is there a reason why for this (e.g. UniSO-T perhaps is more end-to-end, which is better for the model)? In any case, I suggest also raising this as an important conclusion of this work.", "questions_for_authors": "### Clarification Questions\n* As mentioned - what exactly is the \"expert\" mentioned in Table 1? Is it just a MLP as a regression model?\n* When pretraining UniSO-N over multiple tasks - how was y-normalization handled? Is it possible that requiring y-normalization on every task confuses UniSO-N?\n\n### Research Questions\n* Is it necessary to start off with a pretrained checkpoint, or does random initialization work fine too, for the T5-small model?\n* Does model size matter (e.g. what happens if T5-large is used?)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YlI94VNbGq", "reviewer": "Reviewer_F6Mo", "summary": "This paper explores universal offline black-box optimization (BBO) by leveraging language model-based string embeddings. Prior LLM work has two flaws: it doesn't detail paradigms to clarify their strengths and compatibility, and neglects offline BBO's unique need to prevent overfitting to limited historical data. The paper proposes a approach, UniSO, which introduces a string-based representation to unify diverse numerical spaces and enables cross-domain optimization. UniSO employs metadata-guided learning and embedding distribution alignment to enhance task discrimination and stability.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and valuable comments! Below please find our response. Corresponding experimental results can be found at https://anonymous.4open.science/api/repo/UniSO/file/F6Mo.pdf.\n\n## R1 Computational cost\n\nThanks for your suggestions. We have compared the computational cost in Table S1. Both the training data and the computational budget are available and affordable for academic research. We will revise to include this discussion. Thank you.\n\n## R2 Instability in metadata quality\n\nThanks for your insightful question! The metadata in our experiments is a manually crafted summary of the tasks rather than generated by a model, which is concluded from Design-Bench [1] and SOO-Bench [2]. Besides, to assess model performance under varying metadata quality, we conduct experiments with ablation studies on different metadata components (i.e., name, description, and objective).\n\nTable S2 shows the results on in-distribution tasks, zero- and few-shot results on unseen tasks, revealing the effectiveness of each component in metadata. We will add the detailed discussions to the revised paper. Thank you.\n\n## R3 The approach projects input embeddings into a shared latent space, but it does not discuss how this scales with increasing dimensionality or dataset size\n\nThanks for your valuable comment. In this work, we represent inputs of varying dimensionalities using a string format `{\"x1\":val1,\"x2\":val2,...}`. These strings are tokenized and processed through an embedder to obtain embeddings of a fixed dimension. After that, we employ a nonlinear projection to map the embeddings to the shard latent space, which is a common practice [3]. Thus, our approach is capable of handling inputs with different dimensionalities.\n\nWhile the approach is flexible, it may face information compression challenges when dealing with extremely high-dimensional inputs or large datasets, as the embedded size is fixed. Investigating potential scaling laws in LLMs for universal offline BBO remains an interesting direction for future research. We will add a discussion about this in the revised paper. Thank you.\n\n## R4 Loss balancing strategy\n\nThanks for your valuable suggestion! We conduct experiments by removing this strategy. Results in Table S3 clearly demonstrate that the strategy is beneficial. We use a similar strategy after removing certain loss (e.g., $L_{lip}$): $\\frac{\\partial L_{total}}{\\partial\\boldsymbol{\\theta}}=\\frac{\\partial L_{main}}{\\partial\\boldsymbol{\\theta}}+\\frac{L_{main}}{L_{con}+\\delta}\\cdot\\frac{\\partial L_{con}}{\\partial \\boldsymbol{\\theta}}$. We will revise to make it clear. Thank you.\n\n## R5 UniSO-T vs. UniSO-N\n\nThanks for your insightful question! Their main difference is whether the initial model is pre-trained or not:\n\n- UniSO-N uses a pre-trained T5-small encoder to encode the input strings, following [4-5];\n- Since the token representation of numeric y has a different token vocabulary from pre-trained LLM [6], UniSO-T is trained from scratch.\n\nThe inferior performance of UniSO-N may stem from the inconsistent bias from different modalities between natural language and numeric representation [7]. Besides, [8] theoretically investigates the issues of different modalities.\n\nTo further investigate if training from scratch would improve the performance of UniSO-N, we compare UniSO-N both from scratch and from pre-trained in Table S4, where UniSO-N trained from scratch performs better than that trained from the pre-trained encoder.\n\nTo better understand this phenomenon, we analyze the rank correlation between the ground-truth scores and model predictions in the out-of-distribution (OOD) regions during the training, which is an important metric for offline BBO [9]. The results in Figure S1 show that pre-trained models initially show poor performance, and there is only a small improvement later on. How to align different modalities well is a critical future work.\n\nWe will add these discussions in our revised paper. Thanks again for your insightful comments!\n\n---\n\n**Thank you again for your encouraging comments! We hope that our response has addressed your concerns, but if we missed anything please let us know.**\n\n### References\n\n[1] Design-Bench: Benchmarks for data-driven offline model-based optimization. ICML, 2022.\n\n[2] SOO-Bench: Benchmarks for evaluating the stability of offline black-box optimization. ICLR, 2025.\n\n[3] A simple framework for contrastive learning of visual representations. ICML, 2020.\n\n[4] Understanding LLM embeddings for regression. TMLR, 2025.\n\n[5] Predicting from strings: Language model embeddings for Bayesian optimization. arXiv, 2024.\n\n[6] OmniPred: Language models as universal regressors. TMLR, 2024.\n\n[7] Position: Why tabular foundation models should be a research priority. ICML, 2024.\n\n[8] A theory of multimodal learning. NeurIPS, 2023.\n\n[9] Offline model-based optimization by learning to rank. ICLR, 2025.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "Related to some extent.", "essential_references_not_discussed": "yes", "other_strengths_and_weaknesses": "### *Strength:*\n\n1. It introduces a novel approach by leveraging **string-based representations** for universal offline black-box optimization.\n2. The paper proposes a framework, UniSO, which is capable of **distinguishing dissimilar tasks** with clearer clusters, demonstrating more distinct boundaries in results compared to previous methods.\n3. The proposed embedding distribution alignment via metadata and smoothness enhancement help model demonstrates **strong cross-task generalization capability**.\n4. The paper conducts extensive experiments on various tasks from different benchmarks. \n\n### *Weakness:*\n\n1. **Computational Complexity** The paper provides limited consideration of efficiency and runtime performance. The proposed method relies on large-scale language models and evolutionary algorithms, which may introduce high computational costs. The computational cost and resource requirements may increase significantly as the problem size grows. \n2. The metadata includes a brief description of the task with a concise natural language summary. Is this part manually written or generated by a model? Could this lead to **instability in metadata quality**, potentially affecting the final results?\n3. The approach projects input embeddings into a shared latent space, but it does not discuss how this scales with increasing dimensionality or dataset size.", "other_comments_or_suggestions": "None", "questions_for_authors": "### *Questions:*\n\n1. Referring to **Weakness 1**, is there any discussion on efficiency that can be provided as feedback?\n2. Referring to **Weakness 2**, is there any discussion on the details of metadata generation and its impact on performance?\n3. Is there an experimental comparison of the effectiveness of the **loss balancing strategy in Section 3.5**? In the ablation study, after removing certain losses(e.g. lip, con), what form does the balanced loss take?\n4. Is there any discussion on why the performance improvement of UniSO-N with the strategies proposed in the article is less than that of UniSO-T?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of offline black-box optimization with the goal of handling diverse tasks with heterogeneous search spaces. The key idea is to convert numerical design parameters into string sequences and utilize LLM embeddings as the representation space for surrogate modeling. Two model architectures are explored in the paper: UniSO-T, which predicts objective score tokens in a sequence-to-sequence manner, and UniSO-N, which regresses a numerical score from the LM embeddings using an MLP head. The paper received generally positive reviews from all the reviewers. Therefore, I recommend accepting the paper. 

However, I would like to point out a troubling trend  in the offline optimization literature of  not referencing any work from the online black-box optimization (primarily bayesian optimization (BO) ) literature. All the problems discussed in the paper (mixed spaces, multi-task, varying input dimensions) are widely studied in BO literature but not referenced at all in the paper. I hope the authors will add these references and properly situate the proposed paper within a large body of work in the BO literature studied over last 5-10 years.
Decision: Accept (poster)