OmniPred: Language Models as Universal Regressors
Authors: Xingyou Song
Oscar Li
Chansoo Lee
Bangding Yang
Daiyi Peng
Sagi Perel
Yutian Chen
Venue: Accepted by TMLR ()

Abstract: Regression is a powerful tool to accurately predict the outcome metric of a system given a set of parameters, but has traditionally been restricted to methods which are only applicable to a specific task. In this paper, we propose OmniPred, a framework for training language models as universal end-to-end regressors over (x,y) data from arbitrary formats. Using data sourced from Google Vizier, one of the largest proprietary blackbox optimization databases in the world, our extensive experiments demonstrate that language models are capable of very precise numerical regression using only textual representations of mathematical parameters and values, and if given the opportunity to train at scale over multiple tasks, can significantly outperform traditional regression models.

Published in Transactions on Machine Learning Research (12/2024)
OmniPred: Language Models as Universal Regressors
Xingyou Song"", Oscar Liaxt Chansoo Lee', Bangding (Jeffrey) Yang", Daiyi Peng', Sagi Perel', Yutian Chen
'Google DeepMind, *Carnegie Mellon University, 'Google
"Equal Contribution. [Work performed student researcher at Google DeepMind.
Reviewed on OpenReview: http::///ppaaeeeeeseeee///////ms
Abstract
Regression powerful tool to accurately predict the outcome metric a system given a of parameters, but has traditionally been restricted to methods which are only appli- cable to specifie lask. In this paper; propose OMNIPRED, framework for training language models as universal end-to-end regressors over (x, y) data from arbitrary formats. Using data sourced from Google Vizier, one of the largest proprietary blackbox optimization databases in the world, our extensive experiments demonstrate that language models are capable of very precise numerical regression using only textual representations of mathe- matical paramcters and values, and given the opportunity to train at scale over multiple lasks, can significantly oulperform traditional regression models.
1 Introduction
Regression is fundamental task for experimental design, in many domains such hyperparameter tuning, computer software, industrial engineering, and chemical discovery. The goal of regression is to predict a metric of general system given a set of input features z. Such regressors can later be used for various
applications, such as offline optimization (Kumar et al. 2022 Trabucco et al. 2022), online optimization (Cai
let al.
2020
low-cost benchmarking (Zela et al.
2022
Eggensperger et al. 2015 and simulation (Mendis
ct al.
2019 Hashemi et al. 2018 Kaufman et al.
2021
(leate-3, opta"Aop") Nams: convnnt cifaci0 Matrin: accurany
00
Users
Y=
X
Offline Database
X: (eileent, windower10) Nara: ypu design Netric: Letency
Automated Systems
0.00015
==?
Offline
Train
Test
Figure 1: Overview of our method. Using heterogenous offline (x,v) v) evaluation data collected from variely of sources, we train a LM-basod regressor.
In recent years, large language models (LLMs) have emerged as powerful tools for processing textual rep- resentations at scale over massive heterogeneous datasets to represent complex relationships between input features and output labels. Given that LLMs have been shown to effective for a variety of tasks beyond natural language processing, such as coding (Li et al. (022),, symbolic mathematics (Lewkowyez et al. 2022). and scientific reasoning (Singhal et al. 2022). it is reasonable wonder: Can language models be used for numeric regression?

---

Published in Transactions on Machine Learning Research (12/2024)
Answering this question is highly important not only for the traditional field of experimental design, but also for the ever-changing field of I.L.M research, especially due to recent interest in the ability to forecast outcomes of complex systems (Gruver et al. 2023) and reward modeling in reinforcement learning fine- tuning (Ziegler et al.) 2019). The textual processing abilitics of LLMs are particularly attractive, as they can polentially bypass the need lo lediously featurize inpuls (i.e. the x's) into raw numerical tensors. Prior to our work, there has been no such research specifically addressing the feasibility and utility of training a "universal" metric predictor over a large heterogenous offline dataset.
Our core contributions in summary, as follows:
We propose OMNIPRED, the first scalable yet simple metric prediction framework based free-form textual representations, applicable to general input spaces.
Through only these text and token-based representations, OMNIPRED is capable of very accurate metric predictions over experimental design data.
By leveraging multi-tase learning across vastly different input spaces and objectives, in many cases OMNIPRED can outperform traditional regression models such as MLPs and boosted trees.
These transfer learning benefits persist even on unseen tasks after online finetuning OMNIPRED on small amounts of new evaluation data.
2 Related Work and Motivation
Traditional regression methods have widely used statistical techniques such as Gaussian Processes (GPs), tree-based methods, and multilayer perceptrons (MLPs), predict a scalar objective given fixed-length feature vector, commonly seen in tabular data settings. Multitask (Bonilla et al.) 2007) and contextual (Krause & Ong] 2011) variants have been further proposed for transfer learning purposes, but still require fixed-length tensor representations of and can thus only use previous from the same input space, Additional recent works utilizing deep learning based regressors include Transformers (Hollmann et al. 2023 Huang el. al. 2020 Garg el al. 2022), recurrent neural networks (Hashemi et. al.) 2018), graph neural networks (Lukasik et al. 2020 Gao et al. 2023). and deep-hierarchical GPs (Fan et al. 2024], which allow engglh-independeece. Even 80, frequent issue is still the reliance on tensor representations of (I,
Tensor representations are commonly problem--eepndeen where each tensor element may need to be in a reasonable numerical range (o.g. in 1]) as inputs to model. Thus to represent 2, every categorical feature must be one-hot embedded against providdd choices, and scalar features may need to be rescaled against user-proviis aroviidr bounds. Dynamic yet minor input space changes such as new bounds or additional categories, are incompatible with this static representation. To represent y, a raw objective in IR may also need to be rescaled, which can be problematic at test-time when encountering outlier y-values. Dealing with this issue leads to implementing complicated nonlinear warpings (Daimon) 2011 Yeo & Johnson many of which are also data-dependent (e.g. require storing min/mas values from training data).
2000
Regressor
Dynamic Input Spaces?
Can Multitask?
Tensorize x?
Rescale y?
MLP
No
Only fixed spaces
Yes
Yes
base-
No
Ouly fixed spaces
Yus
Optional
Gaussian Process (GP)
No
Only fixed spaces
Yes
Yes
GNN Trransformrr RNN
No
Ouly fixed domains
Yes
Yes
OMNIPRED (Ours)
Yes
Yes
No
No
Table 1: Comparisons between the flexibilties of different typical regressors.
These issues are summarized in Table 1 In principle, an ideal regressor should process æ and output y, both in absolute terms, independent of changing external statistics or search constraints. For example, the underlying relationship y = z², then the regressor's prediction at should be the same regardless if the constraint [1. 5] or x [o, 100). One way to accomplish this is to represent x with an independent universal tokenizer without problem-specific numeric transformations (Zhou et. al.) 2023). This immediately
2

---

Published in Transactions on Machine Learning Research (12/2024)
unlocks a large amount of transferrability when dealing with variable-length inputs and additional contextual metadata.
Previous works in the token-based, or effectively text-to-text paradigm, have mostly been in reinforcement learning from human feedback (Ziegler et al., 2019), where regression over textual responses (the "x"), also known as reaard modelling, has been crucial to the success of recent interactive LLMs such as GPT (OpenAl 2022) and Gemini (Google 2024). While such works have shown that LLMs are able imitate human ratings in the form of ordinal variables and their softmax probabilities (Bradley & Terry 1952). J, they have not shown whether LLMs are capable of regression over precise numeric- data where y E R.
This is because the overwhelming current focus has been on subjective human-based feedback needed for determining aspects such as creativily, safely, and personality, which contain high aleatoric uncertainly and do not require high-precision measurements. Much less attention has been given towards feedback from complex and natural systems common to experimental design. Given multiple works (Hendrycks et al., 2021 Nogueira al. 2021) demonstrating their brittle and unreliable numeric abilities, is non-obvious that language models are capable of high-precision numerical prediction over token- representations. This is crucial technical challenge which our paper resolves in the quest for general-purpose predictor.
3 Methodology
3.1
Preliminaries and Problem Definition
Since regression is commonly used in blackbox optimization settings, we adopt standard terminology (Golovin et al. 2017 Liaw et al., 2018) from the field. For a given task 'T (X' J, DD m), we assume there a mapping f: + R for which we obtain trials (z. y) from ovaluating an input æ, selected from (possibly implicit) input space X. We define a sludy as an oflline collection of trials D {>r., {s:, V=>=1- To distinguish wetween different tasks, there may be observable task-level metadata m, which can additionally characterize the task and potentially even describes the behavior of the corresponding mapping f(-).
The goal of standard regression is to oblain a distribution mapping 8: X' - P(RR) such that s(x) accurately approximates f(x) over some distribution of imputs æ E X, provided that a training set Dirain is given. In our particular case, we also provide our language model with multi-task training data UfDfrain Dfrain,... ,..] from other tasks {T1, ...... While these extraneous tasks contain different objectives fi, fe.... and may even have different input spaces from each other, training such additional extraneous data may still lead to transferrability, especially for similar tasks.
A common way to measure the accuracy of predictors (deterministic or probabilistic) is to compute the gap between a final pointwise prediction against the true objective value y. For probabilistic regressors 8 X + P(R), we may aggregate by e.g. taking the median or mean of the distribution. Since different studies can have vastly different objective scales (e.g. CIFARIO accuracies are within [0, 1] while synthetic objectives are within [102, 10 we must therefore normalize the difference based on per-study statistics, i.e. for specific task, we define the study error normalized mcan absolute error (MAE):
Σ
|gggregate(s(s)) -vl
(1)
DDeest
Fmax
Drnin
(x:g)
To prevent outlier predictions from significantly swaying average errors, we further clip error maximums to 1.0, equivalent to when the regressor simply outputs boundary values from {ymin Fmax
3.2 Language Model
We use a standard language model in which the model observes a prompt and decodes response. For the regression setting, naturally these correspond to x and respectively. However, in order to allow multi-task training, the task-specific metadata m must be appended to the prompt in order to distinguish between different tasks, and thus for a given task, the prompt is actually (I, m).
For simplicity, we train relatively small 200M parameter T5 encoder-decoder (Raffel et al. 2020 from scratch to avoid any confounding effects from typical generative language pre-training. We wish to learn
3

---

Published in Transactions on Machine Learning Research (12/2024)
single set of weights 8, which can be used to form stochastic regressor 89 : + P(R) given any arbitrary task T. In contrast to settings such as (1) traditional regression requiring training a separate model 0, for each task T/ or (2) requiring completely evaluated trajectories over specialized z-tokenizations for in-context learning (Chen et al. 2022 Hollmann et al., 2023), our sctting maximizes the usage of training data, much of which may contain unfinished Irajectories or Tree=form a-formats.
Representation: As mentioned, since input spaces X' and y-scales may vary wildly across different tasks, multi-task training requires and y to be represented in absolute formats independent of the imput space and mumeric scaling of the specific study. Thus, express in key-volue format, directly mapping parameter names to values, but do not represen the input space X' allowing gencralizability to conditional parameters and dynamic constraints. We represent y with custom tokens to guarantee proper decoding via constrained decoding, using specific tokens to express sign, exponent, and significant digits, although more sophisticated alternatives (Koo et al. 2024)) are available. An example is illustrated in Table Ablations over different tokenizations are conducted Appendix
Language Model Textual Representation
aatch_sized 128, kernel: 'rbf' ,le.rningg Oaterninnnrtt.......e model: ''vm' optimizer: 'agd'
title: ( cclassiiiaation user: "some-person descriptionn 'spam detection', objective: 'acurracy (X)²
m
<5>
Table 2: Textual representations used for OMNIPRED. <+> represents single custom token. Input space and x is the same as in Figure Example y-tokenization represents a value of 725 10 72.5.
We apply standard Prefix-LM training (Raffel et al. 2020), in which for a given (prompt, response) pair, cross-entropy losses are only computed over response tokens. Pairs are sampled from training data over multiple tasks. One could additionally make the loss more metric-aware by weighting specific tokens or even reinforce with non-differentiable scores, although we maintain simplicity in this paper for now by using uniform cross-entropy. Thus the model will implicitly learn numeric distances from training data.
Training:
Sampling and Decoding: Through regular temperature decoding, can repeatedly sample y ~ se(x), to approximate the prediction distribution over R. To remain robust to strong outliers, instcad of using empirical mean, our Aggregate(-) function is defined as the empirical median, since we found it leads to lower error from ablations in Section 6.1 Since the model may need to predict over unseen regions of the input space, we can also assess the model's uncertainty by observing the concentration of sampled predictions Ū and additionally specific log probabilities across every decoded token.
Online Finetuning: To adapt to an unseen task T, common for Bayesian optimization settings (Wistuba & Grabocka 2021), the model can further be quickly finetuned online over the tasks's 's corresponding training data Diram optionally using LoRA (Hu al. 2022). Finetuning may also help to refocus over seen data, when the model is not fully optimized against specific study, e.g. if the pretraining dataset was too large.
Appendix B contains additional implementation details such as the specific architecture and vocabulary used.
4 Data
Many industries possess large collections of metric data from experiments or systems. However, such data is typically not. open-sourced as official training datasets for research. A natural dataset use may come from multiple hyperparameter optimization trajectories, which tend to have (x,y) evaluations from expensive experiments, expressed blackbox objectives y f(x).
4.1 OSS Vizier Format
The abstractions in Section 3.1 above are concretely implemented in Open Source (OSS) Vizier (Song et al. 2022). rescarch interface for blackbox and hyperparameter optimization. Every space X is Cartesian
In applications such as code search, is even infeasible express the space of all possible programs.
4

---

Published in Transactions on Machine Learning Research (12/2024)
product of parameters, each of type DOUBLE (bounded continuous interval), INTEGER (bounded integer set), DISCRETE (ordered set of real numbers), or CATEGORICAL (unordered set of strings). Every parameter may also have child parameters, only active when the corresponding parent parameter is a specific value (e.g. in Figure 2 "beta" is active only if parent categorical parameter selects "Adam", but not "SGD"). An x E X can be expressed as a labular feature, allowing baseline comparisons against traditional regression models.
Search Space
Maan: model
Name: aptimiser
CATHSGRRTES 'aamn'
'mip''
CATEGORIES: ['aget
"adax"
Nase: leazning zate
DOURER: [0, 1.0]
opt
Naze: batch sice
INE: [1. 256)
Nars: kernel
Haze: layeza I8T: (4,
Name: beta
CATROORTES: ['rter, 'paly')
DOUBLE: (0.0, 1.0]
Trial
leaxning rate:0.5
batob c1xe:128
model:
kernel:
optinizer: 'sgd'
Trial 2:
learning xate:0.2
size:14
model:
optiniser: 'adan'.
beta:0.6
Figure 2: Common example of a (possibly nested) space and suggestions x in OSS Vizier.
Task-level metadata consists of title, username, description, objective name, and optional free-form text. Since the OSS Vizier API is meant to provide optimization service for users, there can be many sources of transferrability due to user-specific settings. These include:
A single user or team regularly tuning similar experiments.
Multiple different users tuning similar experiments (e.g. training ResNets on CIFAR10).
Similar parameters used across different experiments (e.g. "learning rate").
Metadata m describing the nature of the objective function.
4.2 Datasets
BBOB (Shifted): For precise controlled experiments where we can generate synthetic datasets and per- form online evaluations, we create a multi-lask version of the BBOB benchmark (ElHara al.] (019) containing 24 different synthetic functions, by applying random domain shifts to transform a vanilla syn- thetic f(x) into f(x - c), and ranging the dimension over [2, 6]. Thus each task T is parameterized by m (function class, dimension, shift), and the corresponding objective can be seen [(,, m), allowing evaluation over unseen m. For a specific task T,, we minimize the in-study training data size Dirain but freely vary inter-study training data {Dfrain from different tasks {Tihr. Thus traditional regressors (e.g- MLPss which can only train from a single Direin will struggle to regress the corresponding Si under this limited data condition. In contrast, the LM may perform better as it will have seen trials from other tasks whose functions share similarities with fi.
Property
Statistic ( (120BB+)
Real World Data: To investigate metric predic- tion over a rich varicty of tasks, we will use data collected by Google Vizier (Golovin al., 2017). Because
# Studies
# Trials
# Distinct Users
O(14K)
not limited to training pleted trajectories over flat input spaces, we can train on more data than just the 750K studies used in the Former (Chen et al. 2022), as seen from
fully com-
Table 3: Relevant statistics on the real world database. We provide order estimates as there may be numerous ways to define e.g. "legitimate" studies trials. See Appendix D for further details and data breakdown.
Table
Since the offline dataset is collected from users' blackbox optimization trajectories, for the most part do not have online access to an actual objec-
5

---

Published in Transactions on Machine Learning Research (12/2024)
tive fxx), rather only data samples D, and thus must evaluate our predictor's accuracy via test set Dieet C DD Thus, we need to take into account how much Derain sufficiently covers the space X, which affects the difficulty of achieving high accuracy on the task. Influencing factors include:
Trial count: Users can decide when to stop tuning, and thus the size of a study can be on the order of 10 to 10a.
Diversity of trials: By default, a study's trials ((11,11;, (IT, yt)} form the trajectory of an optimization loop, and thus later trials may converge towards a single local optimumm
Space size: Approximate cardinality of a space X' exponential with respect to parameter count, and thus large imput spaces will naturally be less explored.
While we apply practical processing steps such as (1) setting a maximum initial trial limit per study and (2) randomly shuffligg the trials and then (3) deciding on a fixed train/validation,
splitting ratio (default 0.8/0.1/0.1), we cannot fully control whether each DD saturates its space X, or essentially how "easy" the task is. Instead,
baseline regressor trained only on Dirain and evaluated corresponding Diast proxy metric of the difficulty of the task.
5 Experiments
We answer the following key questions:
1. Can we simultaneously regress on multiple tasks of different input spaces and objective scales?
2. Are there benefits to multi-task training and are textual signals useful for transfer learning?
3. Does online finetuning improve accuracy over unseen studies outside of the pretraining set?
5.1
Simultaneous Regression
In Figure 3 we visually present how a BBBBB-trained model captures the overall shape of analytical functions of vastly different objective scales with high precision. Furthermore, the model is capable of expressing uncertainty estimates via iid. prediction samples.
por Shitts (-d.5,1 -9,33.99,3 38] xym-3.78 xym 5.29, k.mi.29
Senpllipsoidal-tD Stifts: 50, 50..000---1177
Sepfilpsoidal-4
Step El Eliponida--40
Shifts: 717.3.388
Shifts; [-6. 30.1..4....79...8 xym-3.16. x2 kym3.00
x-= 2.57, x.m0.30
xy 04, By 90, xxm-x.11
540
1680
320-
530
2000
1400
525
3440
100-
2
Objective Value (n)
Shitt, Waierstr ass-4D
2.77..68...78...
Shifts (3.79,2 68.3.78...62
Weierstracs-Fe 4D
Shifts: [-3.79 3. 1.66]]
Welerstrass-00 4D
Stifts: [:3 79.27.2266 29.2.68..33. 3.78.1..66
Wearstrass-DD
x--0 xymd, X.20.22
Xymd, X,m4,38, k2-2.48
Xym xy 10-00 xy=4.2
2o-22,44 qem 61.111.11
100
25-
Resenbrockootttee 4D Shifts: 2..44,,,4,,.33.3 23.57.
Resenbrocd@sttedd 4D Shitt: 3..54..74,2.33..
Rosenbreck@ottred 4D
RosenbrcciRetatedd 40
Shifts: shifts:13.54, 63.2.77 Xy- 0.74, x:=
Shitss [3.54,3.74,24 74.2.63.2.577
Xy= k0 0.40, xy- 1.61
xy- xy= 1.58, 0.79
xy xy X-
30030
Target
10000
110000
Prediction
35002
25030
25000
102000
Xg
X1
X2
X3
Figure 3: Model prediction samples over selected BBOB functions with unseen shifts. Empirical mode (bolded) and min/max max are shown from 10 samples. Over all BBOB functions, we vary the coordinate value 2, while keeping others Tzzs fixed.
In Figure for a model trained over real world data, we present an analogous visualization over hand- selected studies with drastically different input spaces, representative of objectives tuned commonly in real world settings. These include standard machine learning (e.g. image classification and language modeling).

---

Published in Transactions on Machine Learning Research (12/2024)
CIFARTO Malid. Loss)
LM1B (Valdd Loss)
Bid Simulation
Protein Design
-02
+22
-1.0
+15
Name
Space
CIFARIO
Double
Prediction
-20
LMIB LM
Double
-25
-20
da -ie -a
00
-10
<55
-3.
to 46 is
Bid Simulation
Double
LLM (XLA Lakency)
AutoML (Valid. AUC)
MobileNet (TPU Costl
Spar filter (Valid Acc.)
Protein Design
60 Categories
LLM Latency
Hybrid
AutoML
3-H, 42-T
-132
Net
Discrete
0.825
Spam Filter
13-H,
24
-155
450
055
25
83333
0.005
1830
0.50
0.2
0.54
036
6.5
Target
Figure 4: Left: Diagonal (/) is better. Model's y-prediction vs. ground truth over varying studies. Corporate- specific objective names are redacted. Right: Corresponding imput spaces. "#-H, 8-1 is shorthand for a conditional hybrid input space with # root parameters and total possible parameters.
production systems (e.g. bid simulation, LLM inference latency), and scientific research (e.g. protein and hardware design).
5.2 Multi-task Transferrability
In this subsection, we demonstrate the model's ability transfer learn, ie.. improve accuracy over a specific task using knowledge gained from other similar but non-cquivalent tasks, in contrast to "single-task" regres- sors (described in Appendix C which only observe training data from the task being evaluated. Note that single-task baselines such MLPs are incapable of multi-task training when tasks have different dimension- alities.
In Figure we clearly see that the model's accuracy improves with more lasks seen in training and eventu- ally outperforms all traditional baselines. For AutoML studies, the error is averaged from fixed subset of encountered studies. For BBOB, we can further demonstrate the model's inter-study generalization capa- bilitics over metadata m (as opposed to æ) by evaluating unseen tasks with new shifts not encountered during training.
AutoML Eval
BBOB Eval (Unseen Shifts)
0.200
0.35
0.175
Mean Study Error
Mean Study Error
0.150
0.125
SP
0.25
Random Forest
0.100
0.075
0.20
0.050
0.025
0.15
10a
10 Training Studies
10y
16a
10
16
of Training Studies
Figure 5: Lower (L) is better. Mean study prediction error of the model when varying the amount of different studies used in training (log scale). Colored horizontal lines display single-task baseline errors.

---

Published in Transactions on Machine Learning Research (12/2024)
To verify whether the model is performing trans-
Mean Study Error (L)
for learning by reading textual cues, in Table [ we compare results against the case when data is
Datasets (# Training Studies)
Original
Anonymized
BROB (50K)
0.03
0.46
"anonymized" using a study-dependent hash func- tion. For BBOB, we hash metadata m which orig- inally displayed (function class, dimension, shift). For AutoML, we hash parameter names and string values. Each study can still be uniquely identified and trained on, but the model can no longer ob- serve useful correlations from common textual clues. Interestingly, the model fails to train over the full anonymized BBOB dataset, a case when the data is too large and heterogencous.
BBOB (Full 1M)
0.01
FAIL
AutoMI. (26.3K)
0.19
0.41
AutoML (Full 540K)
0.15
0.43
Table 4: Lower (4) is better. Comparisons between models trained on original anonymized data, across BBOB-Shifted and AutoMT test trials. "FAIT." means the model failed to even train.
In Figure D further see that for the model, multi-task training consistently improves over single-task training, and in regimes with relatively lower imput space saturation (i.e. low trial to parameter count ratio) from training data, multi-task models outperform traditional basclines over several different domains. Interestingly, a single-task model trained from scratch remains a competitive choice and for certain domains such as AutoMT, can even outperform all other single-task baselines. We hypothesize this is due to language- based representations being more appropriate for the conditional structures of these domains (e.g. AutoML).
Multi-task LM vs Single-task Baselines
GP
Random Forest
Tree
Name
# Studies
Avg. TpS
Avg. SS
Moan Studty Ennor
MLP
BBOB
1M
30
4,0
Single-task
Bid Simulation
22K
698
4.6
0.3
Multi-task LM
AutuML
540K
250
(3.3, (3.3.29.9) 29.9)
Init2winit
2K
176
3.6
0.2
Protein Desiin
54K
584
125.6
Vertex (Tabulaz)
14M
88
(4.6, 42.4)
Vertex (Text)
544K
118
56.0
Data Type
Figure 6: Left: Lower (L) is better. Aggregate error across different domains. Right: Statistics on domains. Shorthand notation: "TpS" = Trials per Study, "SS" = Space Size, with brackets (#, 8) denoting conditional space with # root parameters and $ total possible parameters. Note that all baselines are single-task regressors.
5.3 Online Finetuning Analysis
Mean Study Error (L) on AutoML. Before Finetuning
We first examine the conditions in which finetun- ing may be beneficial. In Table 5 we finetune vari- ous pretrained models over AutoML studies. While there is nogligible benefit in finetuning the AutoML model on its data again, we see that a model pre- trained over the entire real world datasct is able to finetune to the same level of accuracy as pre- trained AutoML model, while BBOB-pretrainer model leads to significantly worse results than even a single-task model. This suggests that knowledge obtained from pretraining can have a large (positive or negative) influence on transferrability over specific domains such as AutoML.
Pretraining Dataset
After Finetuming
None (Single-Task)
0.98
0.20
BBOB
0.98
0.45
AutoML
0.15
0.15
All RealWorkdData
0.31
0.15
Table 5: Lower (1) is better. Mean study errors of pretrained models and their corresponding finetuned versions.
We further examine this effect by evaluating over unseen tasks, i.c. those which were newly created after the original training set was scraped, and contain studies from new users and objectives. In Figure
8

---

Published in Transactions on Machine Learning Research (12/2024)
Finetuning Comparison (Unseen Studies)
1.0
Single-task
Pretrain Finetune
Method
Mean Study Error (4)
Single-task (LM)
0.28
Study Error
0.6
Pretrsin (LM)
0.68
Pretrain + Finetune (LM)
0.21
MLP (Baseline)
0.25
0.4
Tree (Baseline)
0.32
Random Forest (Baseline)
0.32
0.2
Gaussian Process (Baseline)
0.42
0.0
20 Unique User Study (Sorted by Improvement)
40
Figure Left: Lower (L) better. Example LM study errors over unseen studies filtered over random
distinct users. Right: Aggregate comparisons across different methods over 1000 unseen studies.
we compare initialization from scratch (leading to single-task training) against pretrained model on older real world data. We that knowledge obtained from pretraining can significantly transfer and help predictions over new tasks, although seen the left with three studies, there are few cases of negative transfer.
6 Ablations
We further ablate certain important scttings and scenarios which affect the model's prediction accuracy below. Appendix A contains additional ablations, specifically (1) around our choice of y-lokenization and (2) experimental outcomes when using ranking-based regression metries.
6.1 Effect of Sampling
The LM can output extreme outliers in its y-prediction, usually due to an inaccurate prediction the exponent token or significant digits. While such issues do not occur once the model has nearly perfectly regressed on task (c.g. BBOB), they occur frequently on realistic tasks with nontrivial error (e.g. AutoML. and thus require techniques for correction. One obvious method to reduce error to increase sample count,
as demonstrated in Figure
AutoML Eval
BBOB Eval
0.25
0.08
Mean Study Eror
Mean Study Error
0.20
0.06
MLP
LM
0.04
0.10
MLP
0.02
0.05
LM
16
32
16
32
# of Samples
# of Samples
Figure 8: Lower (1) is better. Mean study error when varying the samples used during inference (log scale).
An additional method is to change the method of aggregation across samples. In Table D we see that using and mean. We hypothesize this due the median's robustness to hallucinated outlier samples which can occur with relatively high probability and also skew the mean.
the median considerably outperforms both max-likelihood
9

---

Published in Transactions on Machine Learning Research (12/2024)
Mcan Study Error (4)
Empirical Aggregation Method
AutoML (Full 540K)
BBOB (Full 1M)
Median (default)
0.15
0.01
Max-iikclihodd
0.22
0.01
Mean
0.23
0.01
Table 6: Lower (4) better. Comparisons between different aggregation methods when using 64 samples
when using full datasets for pretraining.
6.2 Uncertainty Calibration
Although the main metric used throughout our work based on pointwise prediction, an important ability for regressors is to express uncertainty when they are unable to provide an accurate prediction. This is particularly useful in applications such as Bayesian optimization, where uncertainty can be used an exploration proxy. In this section, we examine whether the model can quantify uncertainty even if we did not calibrate tune any of the models for such purposes.
StopEllipcoidal-4D x,-90.79, x--4.18
StepFllipsadal--4 [-0.30.104..1.9. Xy--1.16, x--4 xy=0.00
StenFllipsiidll-00
StepEllipsaidal-4D
(00tt.:000...0.111111 00..5.....11..9...0003 X- xy-4.22
-:050..044-1.9...3 Xp-= xy--0..-6- x.50.53
1000
200
2000
0-
-30
-1000
-200
2000
Objective Value (y)
DiffrrettPoeers-00 4D
Differentowwrs--0
DifferentPowers-D
DiferentFowers-4D
Shifts:: [1.88,0.944.174 1.74,3 274,3.888 89]
Shifts: [1.88,00.44 1.3.0.444.7........ 744.1.74
Shifts: [1 0.9,17,..4
Shifts: [1 1.9,,
1.74,-3.88) 3.89]
A1=0.15 19, m--6186 25
4= 22002.07, 1ym-1.4
4o=0.66 08, x3 1y-15
xo=1.84, x;=,10 0070
200
100
100
10
-20
-100
-100
-2
-2
0
-200
-2
2
Weierstraso-4D
Weierstrass-4D
Weierstrass-4D Shifts: (.3.79.2.68.3.72 3..8..1.66
Weierstrass-ID
Shifts: [-3.75 68.-3 3.7-16616621
Shifts: 79,2 68.3.8..116
(-3.9,2.68,.3372 2...2.88....7.1.666
4ym-4.
x= x= xs=
xy
200
200
-100
-200
-200
-200
Figure 9: Sctting similar Figure
3 but with bimodality.
We begin by demonstrating the LM's ability to nonparametrically express distributions with multiple modes in Figure Saaure@henn when we train the model against randomly sign-flipped BBÖB objectives. In contrast, traditional
methods such ensembled MLPs (Havasi et al. 2021 and Gaussian Process mixtures (Bonilla et al. 2007
must specify the mixture count priori.
Pearson, Kendall-Tars Spearman
Regressor
Uncertainty Metric
AutoML
BBOB
Gaussian Process
Predicted SD
0.254, 0.230, 0.307
0.018, 0.068, 0.048 0.360, 0.366, 0.454
LM w/ mean aggregation
Sample SD
0.560, 0.487, 0.625
LM w/ modian aggregation
Harrell-Doe Davis SE
0.525, 0.412, 0.539
0.360, 0.293, 0.380
Table 7: Higher (t) is better. Rank correlation between quantified uncertainty (SD = standard deviation,
SE = standard error) and actual error over studies with at least 10 test trials (all BBОB studies and 641 AutoML studies)
10

---

Published in Transactions on Machine Learning Research (12/2024)
Furthermore, mcasured the correlation between uncertainty and error on each study. In Table report the average correlation across studies. Interestingly, alt although Table + demonstrated that mean aggregation over LM samples is worse for prediction than median aggregation, the errors are well correlated with the standard deviation of the samples.
6.3 Study Size vs. Multi-task Gain
Intuitively, task's space becomes more saturated with trials, single-task training becomes more sufficient for accurate prediction. In Figure 10 we plot the gains from multi-task LM training over the single-task MLP baseline to validate this hypothesis. Gains are maximized al roughly =50 training trials and diminish as the number of training trials increases. Note that maximal gains do not occur with =0 trials, as presumably some training trials are still needed identify the structure of task.
AutoML (MLP vs Multi-Task LM)
(MLP Multitask LM) Error
1.00
0.75
0.50
0.25
0.00
-0.25
50- percentile
75- 15-percentile
-0.50
90-pemt percentile
-0.75
100
200
300
400
500
600
700
800
# Training Trials
Figure 10: Higher (1) is better. Study error differences between MLP and multi-task LM over individual AutoML tasks. Percentiles are computed after binning the x-axis appropriately.
7 Discussion: Limitations and Extensions
In this work, our emphasis was to demonstrate the promise of applying language modeling to general-purpose regression, and thus our design choices remained relatively simple to avoid confounding factors. We list some limitations of our specific design, which opens many more potential areas exploration.
In-Context Learning (ICL): By design, we maximized the allowed prompt length to allow flexibility representing (x, m) and thus did not use in-context regression, since single (z, m) prompt could be 10K tokens long for some applications. While online finetuning in principle allows infinite amounts of data to be absorbed at inference lime, it is worth investigating ICL. methods which allow arbitrarily long prompts well. Current methods (Chen et al. 2022) which require significant input compression are only applicable to tabular-like x' e's. Additional use of ChatGPT and other service-based chat APIs (Vacareanu et al. 2024) rely on regression as an emergent property expensive LLM--ased training, making their serious use difficult, especially as such services cannot absorb large amount of user-defined offline (æ, y) data which often contain the most relevant information for finetuning.
Hallucinations: By giving the model the freedom to sample y-values approximately all of R, wildly inaccurate outlier prodictions are now possible. This can be exacerbated by a wrong prediction over a significant float token (e.g. leading digit or exponent). Although for convenience, we used an unweighted cross-entropy loss in which all float tokens are of equal importance, prediction accuracy can be improved by weighting more significant tokens, making the training loss more aware of numerical distances over IR.
Prompt-Side Numeric Tokenization: In this work, we directly represented numeric parameter values from & into the default human readable format (e.g. 1234.5 is serialized simply to '1234.5', to be consistent
11

---

Published in Transactions on Machine Learning Research (12/2024)
with LLM literature. This may be suboptimal, as the corresponding tokens may not exactly be digit-by-digit (e.g- T5's tokenization leads to tokens ('1'', '3', $4.5')). One may instead potentially reuse the custom or in text-space, represent using other serializations
tokenization for y-values (e.g. <+>1><2><><<<>> which emphasize digits atomically, c.g. [[ 10e2 2 10e1 3 10e0 4 10e-1] ') as in (Nogucira et al. 2021
Since x includes parameter names and metadata which contain English words, warm-starting from a model pretrained on English text may improve accuracy. However, most check- points comparable to our model's size (<1B params) are not pretrained over experimental data and are unlikely to understand the numerical meaning of e.g. 'learning_ mamniinggrat'.... Furthermore, using pretrained English model introduccs numerous confounding technical choices to consider (e.g. whether to freeze the encoder, tune the learning rate, embed additional custom float tokens, and use more English-based represen- tations of x and m), but this topic is worth pursuing in the future. In this work, it is already surprising that a relatively small model trained from scratch can still achieve regression, thus suggesting our technique's broad applicability even without English understanding.
Pretrained Language Encoder:
Computational Costs: Compared to traditional baselines, language model requires accelerator usage and has relatively higher computational cost for both training and finetuning, in addition to higher inference times. In this work, we purposely designed the model to minimize costs by using =200 params which only requires at most 8 GPUs for training and 1 GPU for inference (see Appendix B
The OSS Vizier API primarily focuses on hyperparameter tuning spaces. Tradi- tionally, more complex spaces such as combinatories and graphs require sophisticated modeling techniques to form regressors, largely in part to difficulties in representing the x's as tensors. Tn addition, many appli- cations with non-expressible spaces such program synthesis impossible to traditionally regress over. We believe that text and token-based representations are highly promising and widely applicable to domains previously unexplored in the field of experimental design.
Other Input Spaces:
Other Metadata: While we performed ablations which anonymized m and parameter names, one can further investigate which lypes of metadata are particularly useful for prediction. Such meladata could contain prozy metrics introduced by previous domain specifc works, such as Jacobian Covariance for neural architecture scarch (Melor et al.] [021] and neural-network norms (Jiang et al.) 2020) for the study of generalization. The relevant code implementing machine learning or programming tasks may be especially important.
8 Impact Statement
This research addresses the ability to regress metrics against textual data. Since any textual metadata may be collected, user-specific information may be used, which raises privacy concerns. This would be particularly true on sensitive topics (c.g. predicting metrics related to personal protected characteristics).
Our research does not involve such sensitive topics for regression, as is performed over blackbox opti- mization data. We followed ethical guidelines as all users in our proprietary dataset have consented to have their tuning data saved and used for offline analysis. The proprictary real world dataset does not contain any sensitive personal information other than usernames. Furthermore, we do not release any of the trained checkpoints, as it may be possible to reverso-engineer parts of the training data, which can lead to privacy violations and data leakage.
9 Conclusion
Our OMNIPRED framework is a first step towards a universal regressor, capable of performing high-precision predictions over objectives of any scale from vastly different input spaces and applications. Its simple and scalable design allows transfer learning from large amounts of offline diverse evaluations, while its single-task variant can still perform competitively against wide variety of gold-standard basclines. Furthermore, it is capable of adapting to unseen data through finetuning, while still transferring knowledge from previous data. This research lays the groundwork for exciting new potential expansions in the field of experimental design.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Acknowledgements
We would like thank Olivier Bachem, Hado van Hasselt, John Jumper, Aviral Kumar, Yingjie Miao, Sebas- tian Nowozin, Mangpo Phothilimthnna, Zi Wang. Scott Yak, and Amir Yazdanbakhsh for useful discussions and Daniel Golovin for continuing support.
References
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction. In Advances in Neural Informalion Processing Systems, volume 20, 2007.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, ICLR, 2020.
François Charton. Linear algebra with transformers. Trans. Mach. Learn. Res., 2022, 2022.
Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 785-794. ACM, 2016.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc' Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In NeurIPS, 2022.
Takashi Daimon. Box-cox transformation. In Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, pp. 1766178. Springer, 2011.
Stéphane d' Ascoli, Pierre- Alexandre Kamienny, Guillaume Lample, and François Charton. Deep symbolic regression for recurrent sequences. CoRR, abs/ hts//2001.44000, 2022.
Katharina Eggensperger, Frank Hutter, Holger H. Hoos, and Kevin Beyton-Browir Efficient benchmarking of hyperparameter optimizers via surrogates. In Blai Bonet and Sven Koenig (ods.), Proceedings of the Nwit AAAI Conference on Arlificial Intelligence, pp. 1114 1120. AAAI Press, 2015.
Ouassim Ait ElHara, Konstantinos Varelas, Duc Manh Nguyen, Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. COCO: The large scale black-bos optimization benchmarking (bbob-largescale) test suite. ArXio, abs/1903. 0/5//9903.0333 063696, 2019.
Zhou Fan, Xinran Han, and Zi Wang. Transfer learning for bayesian optimization heterogencous scarch spaces. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Yanjie Gao, Xianyu Gu, Hongyu Zhang, Haoxiang Lin, and Mao Yang. Runtime performance prediction for deep learning models with graph neural network. 45th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 368-380. IEEE, 2023.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Newral Information Processing Systems, 2022.
Daniel Golovin, Benjamin Solnik, Subhodcop Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Dala Mining, pp. 1487-1495. ACM, 2017.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Google. Gemini: A family of highly capable multimodal models, 2024.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. CoRR, abs/2310.078800, 2023.
Milad Hashemi, Kevin Swersky, Jamie A. Smith, Grant Ayers, Heiner Litz, Jichuan Chang, Christos Kozyrakis, and Parthasarathy Ranganathan. Learning memory access patterns. In Jennifer G. Dy and An- dreas Krause (eds.), International Conference Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1924-1933. PMLR, 2018.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan; Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust pre- diction. In International Conference on Learning Representations, ICLR, 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the MATH dataset. In Joaquin Van- schoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Dalasels and Benchmarks, 2021.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. In International Conference Learaing Repre- senlations, ICLR, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference Learning Represendations, ICLR, 2022.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. CoRR, /bs/2012006678, 2020.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generaliza- tion measures and where to find them. In International Conference Learning Representations, ICLR, 2020.
Samuel Kaufman, Phitchaya Mangpo Phothilimthana, Yanqi Zhou, Charith Mendis, Sudip Roy, Amit Sabne, and Mike Burrows. A learned performance model for tensor processing units. In Alex Smola, Alex Dimakis, and lon Stoica (eds.), Proceedings of Machine Learning and Systems (MLSys). misys.org, 2021.
Terry Koo, Frederick Liu, and Luheng He. Automata-based constraints for language model decoding, 2024.
Andreas Krause and Cheng Ong. Contextual gaussian process bandit optimization. In Advances in Neural Information Processing Systems, 2011.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66 71, 2018.
Aviral Kumar, Amir Yazlanbakhsh, Milad Hashemi, Kevin Swersky, and Sergey Levine. Data-driven oflline optimization for architecting hardware accelerators. In International Conference on Learning Representa- tions, ICLR, 2022.
Ailor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur- Ari, and Vodant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advonces in Neural Information Processing Systems, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Danicl J. Mankowitz, Esme Sutherland Robson, Pushmect Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Compelition-level code generation with alphacode. CoRR, abs/2203.07811 2022.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ton Stoica. Tune: A research platform for distributed model selection and training. CoRR, abs/1807.05i00 051188, 2018.
Jovita Lukasik, David Friede, Heiner Stuckonschmidt, and Margret Keuper. Neural architecture performance prediction using graph neural networks. In Zeynep Akata, Andreas Geiger, and Torsten Sattler (eds.); Pattern Recognition 42nd DAGM German Conference, DAGM GCPR, volume 12544 of Lecture Notes in Computer Science, pp. 188-201. Springer, 2020.
Joe Mellor, Jack Turner, Amos J. Storkey, and Elliot J. Crowley. Neural architecture search without training, In Marina Mcila and Tong Zhang (ods.), International Conference on Machine Learning, ICML, volume 139, pp. 7588 7598. PMLR, 2021.
Charith Mendis, Alex Renda, Saman P. Amarasinghe, and Michael Carbin. Ithemal: Accurate, portable and fast. basic block throughput estimation using deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning, ICML, volume 97 of Proceedinggs of Machine Learning Research, pp. 4505-4515. PMLR, 2019.
Rodrigo Frassetto Nogucira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of the transformers with simple arithmetic tasks. CoRR, abs/2102 Abs/9190..3019, 2021.
OpenAL. Introducing chatgpt. 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.
Karan Singhal, Shekoofch Azizi, Tao Tu, S. Sara. Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. CoRR, abs/ 222.//22221133332 2022.
Xingyou Song, Sagi Perel, Chansoo Lee, Greg Kochanski, and Daniel Golovin. Open source vizier: Dis- tributed infrastructure and APT for reliable and flexible blackbox optimization. In Isabelle Guyon, Marius Lindauer, Mihaela van der Schaar, Frank Hutter, and Roman Garnett (eds.), International Conference on Automated Machine Learning, AutoML, volume 188 of Proceedings of Machine Learning Research, pp. 8/1-17. PMLR, 2022.
Xingyou Song, Qiuyi Zhang, Chansoo Lee, Emily Fertig, Tzu-Kuo Huang, Lior Belenki, Greg Kochanski, Setarch Ariafar, Srinivas Vasudevan, Sagi Perel, and Daniel Golovin. The vizier gaussian process bandit algorithm, 2024.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for dala-driven offine model-based optimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, 1CML, volume 162 of Proceedings of Machine Learning Research, pp. 21658-21676. PMLR, 2022.
Robert Vacarcanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdcanu. From words to numbers: Your large language model is secretly A capable regressor when given in-context examples. CoRR, abs/2404.07544, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. arXiv
preprinl or arx:22101 2021.
In-Kwon Yeo and Richard A. Johnson. new family of power transformations improve normality symmetry. Biometrika, 87()):954 959, 2000. ISSN 00063444.
Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surro- gate NAS beuchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. In Interna- tional Conference on Learning Representations, ICLR, 2022.
Qi-Le Zhou, Han-Jia Ye, Le-Ye Ye Wang, and De-Chuan Zhan. Unlocking the transferability of tokens in deep models for tabular data. CoRR, 0///2310.11114 2023.
Danicl M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano, and GeofTrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.0888 2019.
16

---

Published in Transactions on Machine Learning Research (12/2024)
APPENDIX
A
Additional Ablations
We further ablate certain settings and scenarios which affect the model's prediction accuracy below.
A.1
Tokenization
There are multiple possible ways to represent a float (e.g. 123.4) using custom tokens. Using examples from
Charton
2022:
Nogueira et al. 2021 d'Ascoli et al. 2022). the following are all possible representations:
(Default) Separate Sign and Digil-by-Digit. <>>>>=>
Merged Mantissa: <+1234><E-2>
Exponent Before Mantissa: <+><--2><>><>><>>><<>>
In T we see that these tokenization differences do not matter with large training data (e.g. multi-task), but matter very much in low data regimes (e.g. single-task). The poor accuracy using "Merged Mantissa" is especially apparent it requires large amounts of data to learn differences between 18K possible mantissa tokens.
AutoML
BBOB
Tokenization Method
Single- Task
Multi- Task
Single- Task
Multi Task
Default
0.21
0.15
0.17
0.01
Merged Mantissa
0.73
0.15
0.41
0.01
Exponent Before Mantissa
0.24
0.15
0.17
0.01
Table 8: Mean Study Error (L) comparisons between different tokenization methods.
A.2 Ranking and Correlation Metrics
Although our paper focuses on pointwise predictions which are maximally informative, we can trivially bootstrap our predictions into ranking-bard based metries, which may be downstream use for evolutionary algorithms which are agnostic y-scaling. We see that general, the multi-task LM generally maintains competitive ranking metrics.
Kendall-Tau,
Spearman Correlation
Regressor
BBOB
Bid Simulation
AutoML
Init2Winit
Protein Design
V-AI (Tab)
V-AI (Text)
Gaussian Process
0.69, 0.80
0.80, 0.91
0.04, 0.06
0.15, 0.81
0.35, 0.43
-0.03, -0.05
0.30, 0.39
Random Forest
0.59, 0.75
0.71, 0.84
0.45, 0.57
0.55, 0.67
0.40, 0.52
0.56, 0.71
0.29, 0.38
Tree
0,60, 0.74
0.82, 0.93
0.37, 0.48
0.59, 0.71
0.44, 0.57
0.55, 0.70
0.28, 0.36
MLP
0.63, 0.76
0.73, 0.85
0.37, 0.49
0.53, 0.63
0.47, 0.60
0.50, 0.64
0.25, 0.34
Single-task LM
0.01,
0.19, 0.28
0.21, 0.28
0.05, 0.08
0.15, 0.20
0.18, 0.24
0.11, 0.16
Multi-task
0.92, 0.96
0.70, 0.84
0.61, 0.73
0.65, 0.74
0.72, 0.81
0.57, 0.72
0.49, 0.58
Table
Higher (1) is better. Ranking metrics across different regressors and tasks.
17

---

Published in Transactions on Machine Learning Research (12/2024)
B
Model Details
B.1 Pretraining
We pretrained our model using T5X Raffel et al. 2020), which can be found in the open-source codebase Important hyperparameters, most of which are defaulted,
https:///iitub. wwt.tauw..oooooooo/////eooeee google rosee//oooorrrerrrrroooooooa include:
Architecture size: 12 encoder layers, decoder layers, 12 heads, 64 head dimension, 768 embedding dimension, 2048 MLP dimension.
Optimizer: Adafactor with base learning rate 0.01 and square root docay. Batch size 256.
Vocabulary and Tokenizer: SentencePiece tokenizer (Kudo & Richardson 2018 with vocabulary of 32000 subword lokens, in addition to the custom tokens for represenling y-objectives.
Early stopping: We train for maximum of 1 million steps, but carly stop based on validation loss if overfitting is detected.
The model (> 200M parameters) was pretrained using 4x4 TPU V3.
B.2 Local Training
During local training, data comes from a single study's 's limited trials (at most 1000). The training set size be lower than the batch size (256), and thus must define one epoch as seeing the training data once, ie. only one gradient step if training size < batch size, but multiple gradient steps otherwise.
We use the same settings from pretraining for consistency, but allow a maximum of 30 epochs. For early stopping, validation loss is now measured over the entire validation set instead of sampled batches. Further specific changes:
Since the model is initialized randomly, we use larger constant learning rate of 10", consistent with early learning rates encountered during pretraining.
Single-task training:
Finetuning: We reload the weights in addition the optimizer state (containing e.g. momentum parameters) from checkpoint. We use smaller fixed learning rate of 10-5, which is 10x lower than the O(10-1) learning rate normally encountered during late stages of training.
Due to the small training and relatively low finetuning steps, we used single 1x1 TPU V3.
B.3 Inference
At inference time, we perform temperature sampling with a temperature of 1.0. We restrict the logits to only decode the custom floating point tokens for representing y-values. To maximize batch size for a 1xl TPU V3, we generate samples and select the empirical median of these floating point samples as our final prediction when computing prediction error.
18

---

Published in Transactions on Machine Learning Research (12/2024)
C Baseline Details
C.1 OSS Vizier Input Space
The space is defined as a list of ParameterConfigs: each of which is one of the four primitives:
DOUBLE: Specifies the search range u..
DISCRETE: Specifies finite subset of IR.
INTEGER: Specifies an integer range [1, u).
CATEGORICAL: Specifies set of strings.
Numeric (DOUBLE, DISCRETE and INTEGER) parameters may specify optional log or reverse-log scaling. The log scaling most commonly used for tuning learning rates.
C.2 Data Processing for Flat Space
A flat space where every trial in the study specifies every parameter configured in the space. In this case, convert the parameters into the unit hypercube 1].. For numeric parameters, scale all values into the range [0, 1], by default using lincar scaling unless (reverse)-log scaling was configured. For CATEGORICAL parameters, we use one-hot encoding.
C.3 Data Processing for Conditional Space
A conditional space occurs when one paramcter may be unused, depending on its parent paramcter's value. Conditional spaces commonly appear in AutoML settings where different model classes require a different set of parameters to be tuned. Another common case is when we wish to optimize numeric hyperparameter in the log scale but include 0 in the scarch (c.g. dropout rate, regularization cocfficient), i.c. (UNUSED) Ult u) where > 0.
For a categorical parameter, we simply add an extra out-of vocabulla dimension for the one hot encoding.
For numeric parameter, we first convert parameter values NaN [0, 1], using the same scaling as in the flat space but mapping all UNUSED to NaM. We then add a custom layer (one per parameter) which is defined
Up if x NaN, 2
otherwise
where up is parameter that trained together with the rest of the model.
C.4 Regressor Baselines
Below, we list out the specific implementation details of our regressor baselines. One muanced issue is of hyperparameter tuning the regressors themselves, which could affect results. In order to be reasonably fair to all regressors (including our own OmniPred which has its own hyperparaneters), for each regressor, we used a reasonable fixed set of hyperparameters for consistency throughout all experiments.
We emphasize that paper's contributions mostly regression using flexible string representations and large-scale multi-task training, and do not claim to replace widely accepted baselines in single-task, apples-to-apples comparisons.
The GP regressor model is from the GP-Bandit implementation (Song et al. 2024 found in Open Source Vizier at https:///ithuu. htt.:/oo/ooo///viiiierooo///// and consists the following:
Gaussian Process:
TruncatedLoggorree controls the amplitude of Matern-5/2 kernel.
19

---

Published in Transactions on Machine Learning Research (12/2024)
A InmeatedllooNooraall (i.id. for each dimension i) controls the length scale for the i-th dimen- sion.
TrunateddLogNormal controls the Gaussian noise.
~ Normal(O, o) is the observation noise.
GP(A, a) is the function.
y~ f(z) + the noisy function.
The algorithm then uses L-BFGS to obtain the MAP estimate of a, A and Ø.
One caveat here is that this model requires a non-linear preprocessing on the observations and thus predicts in the preprocessed space. This preprocessing found to be critical to achieving stable regression across Vizier studies, which have a wide variety value ranges. Since the preprocessing is non-linear, we cannot obtain the predictive distribution over the raw observation in a closed form. Instead, we take 1000 samples from the GP, apply the inverse of the preprocessor, and then take the mean.
Tree and Random Forest: We use the standard API (XBBRegressor, XGBRFRegressor in https:// github. com com//mll/xxbboott/ found in XGBoost (Chen & Guestrin 2016), with the following grid-search hyperparameter sweeps for each study:
"min "min_child, cww.chhld_weee chw.hhlddigihh""" [1, 5, 10]
learning_rate": [0.001, 0.01, 0.1]
gamma": [0.0, 0.3, 0.5]
"subsample": [0.6, 0.8, 1.0]
pallsamplebbyreee [0.6, 0.8, 1.0
"max_depth". [3, 5, 7]
Although tree-based methods do not generally require rescaling, still applied consistent z-preprocessing (in particular to deal with optional log or reverse-log scaling).
Multilayer Perceptron: The base architecture consists of 2-layer ReLU dense network of hidden size 256 with a final scalar output. y-values are normalized using tf keras. Maaeerss.ommaizaaiioo.. which subtracts mean and divides by standard deviation computed empirically from the training data. Training was performed with an Adam optimizer using learning rate 10-2, full batch training over 100 epochs, and mean squared error.
20

---

Published in Transactions on Machine Learning Research (12/2024)
D Google Vizier Data
D.1 Study Preprocessing
Since Google Vizier is a service in which users control evaluations, much of the raw study data can be quite chaotic. We apply certain preprocessing techniques to make the data more conducive to training and evaluation.
Removing bad trials: Users may ignore or fail to evaluate a proposed x. Furthermore, during some trials, the y-objective may be denoted with a special "infeasible" value (e.g. a high batch size led GPU out-of-memory errors, if training encountered NaNs). We remove such trials from consideration in work, although one can extend our y-tokenization to support infeasibility in later works.
Trial count hard limit: Some raw studies can contain trials in upwards of 10 trials, which could dominate the data distribution. We therefore apply a hard limit and only consider the first 103 trials per study.
Filtering specific users: There are specific human and automated "power users" which produce orders of magnitude more studies than the average user. Some automated users in particular are simply automatic unit tests involving the use of OSS Vizier. We disregard studies from these users to prevent them from dominating the data distribution.
D.2 Real World Data Descriptions
(Overall) Entire Database: No filters were applied. All studies from the database were exported on March 31, 2023. Finetuning experiments involving unseen studies consist of studies created after this date.
Bid Simulation: Contains hyperparameters for proprietary bid simulation. The simulator estimates how advertiscments might have performed in terms of key metries such cost, impressions, clicks, and conversion volume.
AutoML: Collection of proprietary AutoML data for tuning user objectives.
Init2Winit: Data from running deterministic, scalable, and well-documented deep learning experiments, with a particular emphasis optimization and tuning experiments (e.g. ResNets CIFAR10, Transformers LMIB). Public codebase can be found in
Protein Design: Each space consists of 50+ parameters, each of which denotes a calegorical protein building block.
Vertex AI (Tabular and Text):: Vertex AI platform for automated ML model selection and training for tabular or text. data. For tabular data, Vertex AI searches over a tree of model and optimizer lypes, their hyperparameters, data transformation, and other components in the ML pipeline. For text, Vertex Al trains MI model to classify text data, extract information, understand the sentiment of the authors. For more information, see:
https: //clou//cloud.... http:///loodd.oggge..o.oo//o//// attaee/.aoootttttaaaaattttareeee
21

---

Published in Transactions on Machine Learning Research (12/2024)
D.3 Serialization Examples
For Transparency, provide examples of text representations by the model. Disclaimer: Due to privacy policies, redacted (in red) parameter names and values.
Dataset
Example x
Example
AutoML
batch_size: 128
title: 1=x407/ng4773--ggloobooieee"""""
nodel_type: REDACTED activation_fa: "tanb" batch_norm:
REDACTED
description:
objective: "val_catogoricall haacicalllrrsseoeeeeeeeeeeeeeeee "
dropout: 0.143
anc, wel.raaallvvateee veraaion: REDACTED
gabedding_combinggg "nean"
task type: "multi_clase, clttcttioii
gradient_clipp : 1.6Ser03 nun nun_hidden_ 1
hidden_
optinizer_type "Adandptimizer"
beta2: 0.999
learning_rated 0.928
alg_voeabulary_eetrreee
"adjaated_mtuunlinnee
vocabulary.stratege
"adjusted_ mutual info"
Iait2Wiit
dropout_rate:
"d_epi-Inib, -1022-2021122200
decay_tactor: 0.0379
user: REDACTED
description:
r__bparanssbess base_ bass_lrr 0.00285
objective: "valid/ce_loea"
1r sewaraaneddcay_steps_.eo
0.854
.parans.poeer: 1.94
opt bparans.one minus nonentum: 0.0567
Protein Design
"""000600:"'""
title: 10/1c/0066/1iabbbr3//aaaaa.33
p00000001:"16"
user: REDACTED
p00000002:"10
description:
"00000003:"11"
objectiva:"
p00000004:"16
"00000006:"3"
"00000608:"g"
"00000607:"14"
P00000047:"13*
Vertex Al Text
universal_ saclll_oodel_________ee
title: "20220228 "2220228-62cchhes G21c9hea-0000-258 2090
"single_dd Singlledenss__te dense_efeature
token, mad__oddl____ type: "cnn"
description: REDACTED
token, toxen_bov_coom bou_conbiner: "aqrta"
objective:"'ieree (tt.eeoooooouppp-aaeelll label"
toxen_nodel_type: "bow" rand:0
atc_study_datte "___aaset__aa:"" atc atc_study_aotes:
batch_size:
convnet: "2:3:4+100pa"
dropout, prepout_keep_probs
hidden_layen Laru__laye__de_ dima: 50
nax max_length: 1.54e103
max_cum_ caauuucllsss_____ htt.fffeee__css___eee
nax nax www_tokan_vvcak wacpa_vvabbbaaeeeeee size: 1e+05
narge, sord www.eaddennnaeeeeeeeeeee
1a405
token_ frtofcttofff::
tokenizar, spec:
word sndd_ebbeddin: REDACTED
vord exbedding 100
22


Reviews:
[{"review_id": "dJaFzPgFP2", "reviewer": "Reviewer_pUMR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n\nThe paper demonstrates a significant achievement in successfully implementing a pre-training approach without relying on self-supervised data, which is particularly noteworthy given that self-supervision is often considered crucial in large language models.\n\n\n## Weaknesses\n\nThe fine-tuning methodology employed in the study, while functional, remains relatively basic. The paper would benefit from a more comprehensive exploration of advanced post-training optimization techniques. In particular, comparing the current approach with sophisticated methods such as reinforcement learning could provide valuable insights into the model's potential for further improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper makes two primary contributions to the field of text-based regression. First, it demonstrates the successful adaptation of transformer architecture and multi-task based pre-training techniques, commonly used in language modeling, to the specific task of mapping textual data to continuous numerical values. Second, through empirical evaluation, it establishes that this approach achieves competitive performance compared to traditional regression methods like Gaussian Processes and Random Forests.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The empirical evaluation section would benefit from including a comparison with In-Context Learning approaches. Could you add this comparison to strengthen the evaluation?", "broader_impact_concerns": "The broader impact statement of the paper is fine.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zj6G5P53Oh", "reviewer": "Reviewer_Wdzp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. I really like the idea of describing the tasks and taking multiple features and using them as x and y, training in an autoregressive way. It seems novel, and works well. Making regressors easier to use and more accessible could help expand their use case and have non-ML/DS experts build models on their data\n\n\nWeaknesses:\n1. I'm having trouble fully grasping the actual effectiveness/real world applicability of the method. the authors mention reward modeling in RL, yet they do not have any experiments on that. Also, Is this really more computationally efficient than actually training regressors? It seems like the T5 model was trained from scratch, based on section 3. So how universal is this really, if we need to train from scratch? Can't we just dump all of this in the long context of an LLM? do we really need fine-tuning? \n\n2. I also got a bit confused by table 2, and the value y. the meta date m states that y is accuracy, yet it is 1.23. is it %?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a framework for training language models to perform regression over experimental data, in order to predict metrics, in a multi-task way. The overall aim is to train a \"universal regressor\" that can predict metrics across various tasks and input spaces using only textual representations, with the application being that ppl can describe tasks in natural language, and would not need to feature engineer or run actual number of trials to get an outcome. The authors test both online and offline training (as in having access to the metric function, online, or just having access to black box input output values from existing datasets of optimization trajectories) and show effectiveness. The also show that the language model-based approach outperforms traditional regression models in many cases, especially when leveraging multi-task learning across different domains, and in terms of out of domain/unseen generalization.\nThe model can adapt to unseen tasks through fine-tuning while retaining knowledge from previous data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please view the weaknesses. I also think the experimental setup could be polished to make the claims more clear and provide more background on the datasets and tasks.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HlOlFTzTJQ", "reviewer": "Reviewer_PTPJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n1. The task tackled in this paper is significant and impactful. Exploring the potential of language models as universal regressors has broad implications across various fields, including scientific research, engineering, and industrial applications.\n\n2. The use of textual representations for regression is particularly creative and demonstrates the flexibility of LLMs beyond their typical use cases. This method removes the need for complex input feature engineering, which is often required in traditional regression models.\n\n3. The extensive experimental results convincingly show that OmniPred can outperform traditional models like MLPs and boosted trees, especially when trained across multiple tasks. This inspires future exploration of LLMs in similar settings.\n\n4. The multi-task learning approach is well-executed, and the fine-tuning experiments demonstrate the adaptability of the model in unseen tasks, which is crucial in real-world applications.\n\nCons:\n.1 One area where the paper could improve is in its explanation of the OSS Vizier format. The use of this specific open-source software may not be familiar to all readers, particularly those without a background in hyperparameter optimization or blackbox optimization research. More context and explanation would help bridge this gap for the broader audience.\n\n2. While the experiments provide a comprehensive evaluation across tasks, it would be interesting to see how the model performs on high-dimensional regression problems. The complexity of high-dimensional spaces often poses challenges for standard regression models, and understanding OmniPred’s scalability in such cases would strengthen the overall findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper, titled “OmniPred: Towards Language Models as Universal Regressors,” addresses the novel task of using large language models (LLMs) for universal regression tasks across diverse real-world experimental data. The authors propose a new framework, OmniPred, that applies LLMs to perform regression based on textual representations of input-output pairs, demonstrating that LLMs can efficiently handle a wide range of tasks. The paper showcases the model’s strength in transfer learning, the ability to work across multiple domains, and a comparison with traditional regression models, highlighting significant performance improvements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It has been mentioned in the weakness part as well:\n\n1. It is suggested to provide a more detailed explanation of OSS Vizier, including its importance in blackbox optimization and how it facilitates the framework in this study. This would make the paper more accessible to a wider audience.", "broader_impact_concerns": "I don't think this work involves any ethical implications. The limitation is also well discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that this is an interesting and novel framework for training a flexible regressor using an LLM.  The experiments are compelling, and while not immediately obvious, it is a strength that this method achieves competitive performance with traditional Gaussian Process and Random Forest regressors using a relatively older/lightweight t5 model.

I also appreciate the authors willingness to incorporate the reviewers' feedback, as the paper has made improvements throughout the review process to distinguish what makes OmniPred novel compared to other recent work, e.g., the ICL work of Garg et al. (2022).

Since no reviewers raised the following point, I will accept as is (without minor revision), but I strongly encourage the authors to link to a runnable code example in the camera ready (even if it is just a notebook training the t5 model; note that a more mature package will of course go a longer way).  This will greatly decrease the barrier for readers to use this work in their own research and can significantly increase the impact of the paper.
Decision: Accept (poster)