Understanding LLM Embeddings for Regression
Authors: Eric Tang
Bangding Yang
Xingyou Song
Venue: Accepted by TMLR ()

Abstract: With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.

Published in Transactions on Machine Learning Research (2/2025)
Understanding LLM Embeddings for Regression
Eric Tang Stanford University
erickang000@@maiilccom com
Google Deep Academy Program
Bangding Yang Google
com
Xingyou Song
zingyousong@@google.coom
Google Deep
Reviewed OpenReview: https: https://openrevieuu openreview. net/ forum? id= Wt61n5XNI0
Abstract
With the rise of large language models (LLMs) for fiexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprchensive investigations into embodding- basod regression and demonstrate that LLM embeddings features can be better for nigh-dimensional regression tasks than using traditional feature engineering. This regression performance be explained in part due to LLM embeddings numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.
1 Introduction and Related Work
Regression is a fundamental statistical tool used to model the relationship between metric and se- lected set of features, playing a crucial role in var- ions fields, enabling prodictions, forecasting, and the understanding of underlying relationships within data. Traditional regression techniques often rely handcrafted features domain-specific knowl- edge to represent imput data. However, the advent of Large Language Models (LLMs) and their ability to instead process semantic representations of text has raised the question whether regression can instead be performed over free-form text.
-00
455
30
Previous works have predominantly examined the topic of LLM-based regression through decoding, i.e. generating floating point predictions using token- based sampling. For example, (Song et al., 2024 examines the case when the model is fully acces- sible and fine-tunable against data, while (Vacare- am al.) (2224] study the ability of service-based closed-source LLMs such as GPT-4 using in-context learning.
Figure 1: Rugged surface 5D Sphere function when imputs represented as Gemini embeddings of dimension 6K+, post-processed by t-SNE into 2D space.

---

Published in Transactions on Machine Learning Research (2/2025)
One understudied case however is the use of service-based LLM embeddings fixed vector representations derived from pre-trained (but frowen) language models, which are ubiquitously offered among most LLM services (OpenAl 2023; Google 2024 Authropic (024)). Although they are used frequently in recent ap- plications such as retricval (Karpukhin et al. 2020 semantic similarity (Li ct al., 2020). and a varicty of other rownstream language lasks (Liu et al. 2020), there has been very little fundamental research around their use regression, outside of specific applications such as Bayesian Optimization (Nguyen et al. 2024 Kristiadi al. 2024).
In contrast to decoding decoding-based regression techniques, embedding based regression allows the possibility of cheap data-driven training using inexpensive and customizable post-ombedding layers such as multi-layer perceptrons (MLPs). However, as shown in Figure [ when the domain of a simple function is expressed using high-dimensional embeddings, unexpected characteristics and irregularities can arise, prompting the need for a thorough analysis. Furthermore, L.L.ss by default are not explicitly trained for embedding- basod regression, rather purely for token generation, and thus it is worth analyzing the emergent behaviors of LLM embeddings when applied to regression.
This paper investigates the behavior of these LLM embeddings when used as features for standard tabular regression tasks. Most notably, our findings are:
LLM embeddings are dimensionally robust, i.e. regression performance can remain strong even over high-dimensional data, whereas traditional representations significantly suffer.
Over numeric formats, LLM embeddings preserve spsschitz-continuty and smoothness over feature space, which naturally enables regression when using downstream MLP head.
Factors which directly impact language understanding (e.g. size, pro-training, and input formatting) have more nuanced effects for regression and do not always provide siguificantly better outcomes.
2 Problem and Methodology
A regression task T = (f, consists of an underlying scalar-valued function f : X' + R over an input space X. Provided are offline training data Dtrain = ((11,1())) ..., (r)) collected from querying I and an analogous test set Prest for evaluation. Given access to training data Dirain, the goal is to obtain accurate predictions over test points (z. y) Drest, usually mcasured by an aggregate performance mcasure, c.g. mean squared error or Kendall-Tan ranking scores.
Required by nearly all learnable regression methods are features, which assume come from embedder Φ : X + Rd which takes an input x and returns a fixed-dimensional feature representation, of dimension d. Here, we use the terms 'features" and 'embedding" interchangeably, since traditional methods typically use a canonical, mamually defined fcature engincering method for tabular data, in which contimuous values are normalized and categorical selections are one-hot encoded. This feature vector Φ(x) is then sent to a downstream predictor, e.g. MLP or random forest, which trained using a loss function such as mean squared error.
Language models also provide a canonical definition of embedding, which typically consists of, in order:
1. Tokenizing a string representation x into L tokens.
2. Obtaining a "soft prompt" L via vocabulary look-up.
Applying a forward pass of a Transformer to obtain an output R/1x, Pooling down to a fixed dimension voctor in Rdd.
Afterwards, one may also attach an MLP predictor head and apply an analogous training procedure as in the traditional case. Thus we can that the only difference becomes the input representation i.e. whether we used traditional érmd or LLM--baed ÓLLM-
While it is straightforward to assume that the whole process outlined for LLMs should constitute the defini- tion of a language model embedding ΦLLM, it is not obvious how much each of these steps may contribute to the final regression result. For instance, one could simply skip applying a forward pass in step (3) and pool the soft prompt directly, or use a randomly initialized model as opposed to a pretrained one. We extensively study this case in Section 3.3
2

---

Published in Transactions on Machine Learning Research (2/2025)
2.1 Modeling Specifics
To minimize confounding factors and maintain fairness during comparisons, we use the exact same MLP prediction head (2 hidden layers, ReLU activation), loss (mean squared error), and y-normalization scheme (shifting by empirical mean and dividing by empirical deviation), regardless of using ΦLIM and Ptrad- Note however, that the embedding dimensions of the two representations may be different, and so we distinguish them using notation dilm and duad respectively, where typically dum >> derad.
To demonstrate consistent results over different familics of language models, we benchmark over both the T5 (Raffel et. al. 2020) and Gemini 1.0 (Google 2024) families, which use different. architectures (encoder-decoder and decoder-only), different vocabulary sizes (32K and 256K), and embedding dimensions, respectively, Following (Li ct al. 2020 Reimers & Gurevych 2019), we use averago-pooling as the canonical method of aggregating Transformer outputs, as empirical comparisons in Appendix A.4 found this was optimal. Thus the embedding dimension dum is equivalent to the the output feature dimension f following a forward pass, and specific sizes of dum can be found in Appendix B.3
Similar to previous work (Song al. 2024 Nguyen et al. 2024 for string representations of x from any regression task, by default we use key-value JSON format with consistent. ordering of keys, i.c. {param1 valuel, wwwamm:vaauue................... .., with specific examples shown in Appendix C.2
2.2 Regression Tasks
Appendix C.1 contains full details on our regression tasks. We first use synthetic, closed-form functions in order to produce controlled studies in which we may query any x from the input space. We use 23 functions defined from the standard Black-Box Optimization Benchmarking (BBOB) suite (Elhara et al., 2019), supporting continuous inputs of any dimension. To avoid confounding terminology between embedding 'dimension' d and the intrinsic 'dimension' of an objective f, denote the latter as 'degree-of-freedon (DOF), and thus f(-) is dependent on input coordinates z(1), ..... ,(1))...,,,(OOr, (DOF), each of which is between [-5,5] This provides comprehensive variety of both convex and non-convex objective landscapes to regress upon.
We further use real-world regression tasks representative of those encountered in the wild and in industry settings by benchmarking over offline objective evaluations over production systems, collected from hyper- parameter tuning records. These consist of four families, with each family containing least 50 individual yet similar regression tasks with varying amounts of data. The families are:
AutoML (Google Cloud 2023): : Automated Machine Learning service for Tensorflow Extended
(Google 2023) pipelines (e.g. batch size, activation, layer counts) over tabular or text data.
Wintt (Dhll et al.
2023): Learning rate scheduling parameters influencing common image
classification tasks (c.g. ResNets on CIFAR-10 and ImageNot).
XLA (Phothilimtthana
al.
2021): Tuning for the Accelerated Linear Algebra (XLA) compiler
which affects LLM serving latencies.
L2DA (Yazdanbakhsh et al.
2021): 'Learning to Design Accelorators', for improving accelerators
such as TPUs and corresponding computer architectures to improve hardware performance.
In the real world regression tasks, each parameter may be contimuous or categorical, and we define the DOF of such a task by its number of parameters. Note that for synthetic objectives, where all inputs are continuous, diadd= = DOF. However, for real-world tasks with categorical parameters, duad > DOF due to additional onc-hot encodings.
For obtaining data, we may either sample (x. v) pairs (in the case of synthetic objectives where x are uniformly sampled from X),, use the given offline data (in the case of real-world tasks, where they were actual evaluations from an optimization trajectory), using standard 8-1-1 train-validation-tost split.
Due to the inherent differing of metric scales across lasks, it would be inappropriate to aggregate results based on scale-dependent metrics such as mean squared error (MSE). Furthermore, we found that the selection of the regression metric (c.g. Kendall-Tau, Pearson, mcan squared error, mcan absolute error) did not matter for comparisons, as they all strongly correlated with each other. Thus, by default we report the Kendall- Tau ranking correlation, which is always within [0, and can also be aggregated across different tasks.
3

---

Published in Transactions on Machine Learning Research (2/2025)
3 Experimental Results
3.1 High Dimensional Regression
We begin by demonstrating cases in which LLM embeddings better represent inputs over high degree- of-freedom spaces than traditional representations. In Figure 2 we show that for a subset of functions, LLM embeddings possess surprising robustness, retaining the same performance for varying DOF' whereas traditional baselines such as XGBoost and MLPs significantly falter over higher DOFs. Appendix A.. rates this inherent to the LLM embeddings, regardless of the regression head used (e.g., XGBoost).
Sphere
RosenbrockRotated
Lunacek
NegativeSphere
0.75
0.9
0.6
0.50
0.3
0.25
0.0
75
Kendail-Tau Correlation
SharpRidge
Gallagher101Me
Gallagher2
GriewankkRosenbrock
0.8
0.75
0.75
0.75
0.50
0.25
25
100
>5
100
0.00
50
75
100
50
00
Schaffers
Rastrigin
StepEllipsoida
0.9
0.6
0.6
0.75
0.4
0.3
0.2
0.25
0.0
Degrees of Freedom (DOF)
XGBoost
MLP
Gemini Pro
T5-XXL
Figure 2: Higher (f) is better. Degrees of freedom (DOF) vs Kendall-Taa correlation for various BBOB functions. Results averaged over 12 runs of random data generation. Each task's data consists of 500 (a. y) evaluations sampled uniformly across the input space, using a 8-1-1 split for train-validation-tees.
This result is not universal however, as we show in Appendix A.1 this pattern does not apply for a few selected functions, but nonetheless it occurs in the majority the BBOB functions. We further corroborato this observation over real-world tasks Table [ We see that in general, regressions on LLM embeddings outperform traditional methods more often for tasks with higher DOFs (AutoMT, and XLA)..
Task Name
Avg. DOF
T5-Small %
T5-XXL %
Gemini Nano %
Gemini Pro %
Init2Winit
4
6.7
8.0
11.3
19.0
L2DA
10
2.7
12.0
9.3
10.7
AutoML
29
30.7
41.3
29.3
36.0
XLA
35
17.2
29.3
18.9
24.1
Table Percentage tasks in which ΦLLM outperforms dredd across various real world regression tasks. Results reported for 75 tasks per family, except for XLA, which only contains 58 tasks. Full results in Appendix A.2
4

---

Published in Transactions on Machine Learning Research (2/2025)
3.2 LLM Embedding Smoothness
Particularly due to the discrete nature of tokenization, is non-obvious whether LLM embeddings possess a notion continuity in embedding space. For example, assuming character-wise tokenization, 1.2 is not so numerically distant from 1.567, but token-wise distant, as the majority of the tokens (234 and 567) are not shared. The notion of continuity and smoothness is crucial for neural network generalization Kalimeris et al.) 2019 Neyshabur
al. 2018), robustness (Weng et al.) 2018), vulncrability to adversarial examples (Goodfellow et al. 2015), and more. We can characterize smoothness in the regression case by the dipschitz-continuuity induced by a representation & in its latent space R4.
Intuitively, similar inputs should lead to similar objective values, which can be quantifiod inversely by the Lipschitz factor [(x, = [=)()) - [(a)ll [|((x) d(x')ll with respect representation & and ||||| norm. We emphasize to the reader that the input space X does not actually have an explicit notion of distance on its own. Instead, traditionally it has always been assumed that the distance was defined canonically by Euclidean distance over the traditional embedding method, i.c. | (z) Quad (2")l22 demonstrated by common use of Fuclidean-based radial basis and Matern kernels (Genton 2002 during regression modeling. However, as seen from the results previously, it may be the case that Otrad is suboptimal for some regression tasks.
Sphere, DOF=100
RosenbrockRotated. DOF=100
Lunacek, DOF=100
80
80
60
40
20
20
20
Count
80
160
240
320
0.0
0.5
1.0
1.5
2.0
400
800
1200
1e6
Schwefel, DOF=100
LinearSlope, DOF=100
Discus, DOF=100
100
80
100
75
60
75
25
20
25
0
100
200
300
400
0.0
1.5
1e5
1e7
Normalized Lipschitz Factor
LLM
Traditional
Figure 3: Left-skewness (+) is better. NLFDs induced by ΦLLM (T5-XXL) and éuad Top: Cases where ØLLM
outperforms duad for regression. Bottom: Vice-versa where drnd outperforms ΦLLM-
In order to analyze the continuity of an embedding & with respect to oflline data D, we define a Normalized Lipschitz Factor Distribution (NLFD) as follows:
Full-batch normalize, i.c. apply shifting and scaling each o(x) so that aggregate, D has zero mean and unit variance per coordinate.
2. For each D, choose z' E D such that d(x') is the nearest & neighbor of Φ(z), and compute the Lipschitz factor 2').z))
3. To assume an average embedding norm of 1 for different embedding dimensions d, we downscale all Lipschitz factors by vd.
We see that there a high inverse relationship between the skewedness of the NLFD and regression perfor- mance. Specifically, in Figure when ΦLLM outperforms duadd for regression, ΦLLM'S 's distribution of Lipschitz factors also tends to skew relatively to zero than &trad, and vice-versa.
5

---

Published in Transactions on Machine Learning Research (2/2025)
T5-Small, 1000=100 K: 0.64, S: 0.83, P: 0.77
TS-Large, DOF-100 K: 0.71, S; 0.88, P: 0.79
TS-XL, DOF-100
T5-XXL, DOF-100
K: 0.79.. S: 0.92, 0.86
K: 0.75, 0.91, 0.83
0.6
0.6
Regression Performance Gap (Kendall)
0.0
0.0
0.3
-0.3
-0.3
-0.15
0.00
0.15
00.1
0.0
0.1
0.2
0.15
0.00
0.15
0.30
0.0
0.1
0.2
0.3
Gemini Pro, DOF=10
Gemini Pro, DOF=25
Gemini Pro. DOF=50
Gemini Pro, DOF=100
0.58, 0.79, P: 0.77
0.54, 0.67, P: 0.73
K: 0.65, S: 0.78, P: 0.82
0.7.5, 0.93, P: 0.88
0,8
0.8
0.3
0.4
0.4
-0.25
-0.3
0.4 -0.15
-0.15
0.30
-0.30
0.00
0.15
NLFD Gap (Z-Score)
Figure 4: Relationship between gaps in NLFD (via. 7-score) and regression performance for all 23 BBOB functions. Relationship is quantified using (K, S, P), which respectively are Kendall-Tau,, Spearman and Pearson correlations. Top: We vary model size within the T5 model family. Bottom: We vary the objective's DOF for Gemini Pro.
To formally quantify comparisons between NLFDs from ΦLLM and érrad, for fixed regression task, we may thus compute the Z-score using the difference the two distributions:
Mobrrdd HOLLM
(1)
002 Ptrad
+o?
&LAM
where pég and dg are respectively mean and standard deviations of the NLFD of a representation ø. We may then observe the relationship between gaps in representation smoothness vs. regression performance. that for given BBOB regression task, the Z-score
Figure E with extended results in Appendix A.3 (i.e. gap in embedding smoothness) is highly correlated with the gap in regression performance, regardless of the model used (T5 or Gemini) or the DOF of the underlying objective f.
Gemini Nano
Gemini Pro
Reference Point
20
Distance From Refornicc Point
Distance From Reference Point
t-SNE Dimension 2
t-SNE Dimension 2
5-
30
15
-10
-10
-15
-20
Reference Point
-20
-20
20
-10
t-SNE Dimension 1
t-SNE Dimension
Figure 5: 1-SNE for Gemini (Nano and Pro) embeddings of points sampled around DOF==00 reference point.
Traditional &g distance overlayed in color.
In Figure we further visualize whether ÓLLM distance awarc, i.c. whether (LLM(E) are (L)M(r)) are close embedding space (ura(()) (x) and (trad(x') are close. As mentioned before however, there is no ground truth notion of "closeness" nonetheless, we use Otrad a point of comparison. Since it is inappropriate to

---

Published in Transactions on Machine Learning Research (2/2025)
simply sample r's uniformly in high DOF space, then average distances concentrate around v VOF, we instead take reference point and sample points from fz-balls of increasing distance from the reference. We see that distances over the LLM embedding space are correlated with the traditional measure of distance, but may be non-linearly warped, which benefits LLM-based regression in certain cases as seen in Section 3.1
3.3 Model Effects
In this subsection, we comprehensively investigate the impact of many common LLM factors such as model size and pretraining on regression performance. Additionally, we also ablate the effects of different string representations and LLM subcomponents such vocabulary embeddings.
Are Larger Models Always Better? Within the rescarch community, the prevailing assumption is that there exists a direct correlation between language model size and performance improvement. However, with the rise leaderboards such as LMSYS (LMS) 2023), smaller models have been shown to outperform larger competitors, due to differences in their 'recipe", such as training data quality, pre-training and post-training tochniques, and architocture.
T5 Scaling
Gemini Scaling
0.5
Kendall-Tau Correlation
0.50
0.45
AutoML
0.40
Init2Winit
0.3
XLA
0.35
L2DA
0.2
0.30
0.25
10
10
10'00
Nano
Pro
Ultra
Model Parameters
Model Tier
Figure 6: Higher (t) is better. Model size vs regression performance on hyperparameter tuning tasks across T5 and Gemini model families. Median performance is plotted, along with 40-60 percentiles as error bars.
In Figure we see that over various real world regression tasks, T5 models exhibit clear trend of improved performance when increasing model size, when training methodology is fixed. In contrast, model tiers within the Gemini family exhibit substantial variance, and larger model sizes do not consistently translate to superior results. We hypothesize this is due to differences in Gemini 'recipes", as e.g. different model tiers may have used different pre-training datasets, architecture tweaks, and post-training configurations, whereas all T5 model sizes have only been pre-trained on the C4 web crawl corpus.
Does Language Understanding Actually Help? Recent works (Li et al. 2020 Devlin al. 2019 have claimed that logit-based embeddings mostly measure the semantic similarity between string inputs, and thus it is unconfirmed whether they may be beneficial for numeric regression tasks. To resolve this, using the T5 family, we compare against using (1) randomly initialized model for the forward pass, and (2) representing our features via vocabulary embeddings without applying a forward pass.
We first perform our study over BBOB tasks which use purely numeric tokens, where one may question the benefit of language pretraining over English corpus data. Surprisingly, Figure shows that applying forward passes of pretrained models still helps, especially with larger sizes. However, worth mentioning that even features from randomly initialized models and vocabulary embeddings alone are also dimensionally robust, as they do not suffer performance drops with higher DOFss For real-world tasks in Figure S it is more expected that applying forward passes by larger pretrained models helps. However, there is more variance in the improvement gaps, as the improvement is surprisingly quite minimal for some tasks such as AutoMI. and L2DA, while significant for Init2Winit and XLA.

---

Published in Transactions on Machine Learning Research (2/2025)
BBOB Tasks (Pre-trained vs Random Init)
Tasks (Full Vocab Table)
Kendall-Ta Correlation
Kendall-Tau Correlation
0.5
0.3
0.3
0.2
0.2
0.3
0.1
10
50
DOF
DOF
75-Small
TS-Large
T5-XL
TS-XXL
Random
TS-Small
TS-Large
TS-XL
T5-XXL
Vocab
Figure 7: Kendall-Tau regression comparisons when comparing to random initialization (left) and vocabulary beddings (right) on BBOB regression lasks. Each bar averaged across all BBOB functions a given model size and DOF.
Real-World Tasks (Pre-trained vs Random Init)
Real-World Tasks (Full Model vs Vocab Table)
Kendall- Tau Correlation
0.48-
Conrelation
0.42
0.42
THE
0.36
Kendall-Tau
0.30
0.18
0.18
0.12
0.12
AutoML
Init2Wintt
XLA
L2DA
AutoML
Init2Wnnit
XLA
L20A
Task
Task
T5-5mall
TS-Large
TS-XL
TS-XXL
T5-1 Random
TS-Small
T5-Large
T5-XL.
TS-KXL
T5.4 Vocab
Figure 8: Same Figure but applied for real world regression tasks. Each bar is averaged across 75 tasks per family.
We further ablate differences in string represen- tation, i.c whether by default to show fcature names {param1 valuee, Vvaluei,value2,... or omit them, only showing [value1, value2,...J
Real-World Tasks (Feature Names vs Omitted)
T5-Small
T5-Large
KT Corr. Difference
0.12
T5-XL
T5-XXL
In Figure g for the majority of tasks, omitting feature names does not siguificantly affect perfor-
0.04
mance, although specific tasks such as XLAA do benefit from feature names. This is surprising, presumably feature names in XLA tasks such auto cross httpccossssssi___________ as names such as setch_sizze or learning rate found in both AutoML and Init2winit.
-0.04
are not as common
InitiWink
L2DA
Task
Figure Difference in Kendall correlation when using full dictionary containing feature names, only values.
The results of Figures g and g combined lead ad- ditionally surprising conclusions, such as languago- to-mumeric transfer. For instance, inpuls æ from Init2Winit tasks only possess numeric values, and expected, removing feature names does not significantly change regression results. Yet applying forward passes by pre-trained T5 models still benefits regression, despite the fact that T5's pre-training data contains mostly web-corpus data which is unlikely to contain significant. amounts scientific, or numeric information (Dodge et al. 2021
Intuitively, as more samples are available a task, the difference inductive biases between regression methods should matter less, since predictions will be more influenced by training data. We verify this in Figure T0. where that for tasks with low numbers of
More Training Data Reduces Bascline Gaps:

---

Published in Transactions on Machine Learning Research (2/2025)
(z. y) points, there is more variability in performance between using ΦLLM and Puaad, but additional training points decreases these differences.
AutoML
XLA
(T5-XXL MLP) Kendall-Tau Correlation
0.2
-0.1
-02
-0.3
-0.3
-0.4
200
400
600
800
1000
200
400
600
800
1000
1200
1400
Number of Training Points
StdDev
Mean
Figure 10:
Performance gap belween an MLP baseline and regression over T5-XXL embeddings for individual trials within the AutoML and XLA task settings. Higher (t) is better for LLM embeddings. Error bars are plotted for {0.5, 1.0, 2.03 of the standard deviation.
4 Discussion: Limitations and Extensions
In this work, emphasis was to provide more in-depth understanding of LLM embeddings with respect to regression. While we found many different cases in which they outperform traditional representations, we cannot broadly claim that LLM embeddings should always be used in serious applications of regression. Below, we list some limitations of our work and more potential areas of exploration.
Different Modalities and Inputs: Further investigation is needed to understand how LLM embeddings perform with non-tabular data, including combinatorial objects such as graphs, trees, and other complex structures, but also diverse modalities such as images, videos, and audio data.
While investigated the effects of parameter names for tabular string representa- tions, we did not investigate the effects of different numeric representations. Since LLMs are predominantly pre-trained over human-written text, our formats also follows, e.g. 1234.5 is serialized directly into 1234. 5. However, these numbers may also be represented using scientific notation (c.g. 1. 2.23) or even customized variants, e.g. 10e2 2 10e1 10e0 10e-1 as (Nogueira et al. 2021). We suspect that the fun- damental conclusions would remain similar, although the specitic numeric results may change.
Prompt Formatting:
Different LLM Services: Our work focuses on 'depth' of understanding rather than "breadth' of results, although we did find many similar conclusions for both the T5 and Gemini model families, which are quite different in architecture, pre-training data, and post-training. While it remains to be empirically verified, we hypothesize similar conclusions may occur with other model families, such as GPT-4 (OpenAl 2023 Claude (Anthropic] 2024), and LLaMA (Touvron et al. 2023 especially as they share fundamentally similar approaches with Gemini.
Different Embedding Definitions: As mentioned in the main body, using average-pooling for LLM 'embeddings' follows previous literature for consistency, is one of the simplest methods to obtain fixed dimensional feature vector. and is also better than using max-pooling or last-token logits as shown in Appendix A.4 However, other definitions have been proposed, such as collecting intermediate outputs Chen et al. 2022 Such additional methods are worth studying in the future for understanding their effects on regression.
In- Context Learning (ICL): An alternative prompting method, particularly natural for decoder-based architectures, would be to place all previous evaluations in the context. "shots" and obtain only the logits

---

Published in Transactions on Machine Learning Research (2/2025)
from query z. However, this can severely limit the amount of training data allowed, the context window still has a finite maximum length. In this paper, we primarily focused on the zero-shot case where the context window only contains the query, which allows the downstream MLP to train over unbounded amounts of data.
Computational Costs: Compared to traditional regression techniques which can even be run on CPUs, LLM inference almost always requires accelerator usage, making them more expensive if needed in serious regression tasks. Remote procedure calls to service-based LLMs such as Gemini also adds additional layer of latency. However, compute costs for inference are orders of magnitude cheaper than for training, and typically only require few GPUs or TPUs, making embedding- regression still very feasible for most academic labs or industries.
Smoothness Computations: One limitation that our smoothness analysis is only feasible when one has online access f(-) as in the case of BBOB functions but not offline real world data, since one neods to obtain arbitrarily close (x,,²") pairs to understand local behaviors within the feature space. Our analysis however, may be extendable to any space X' (e.g. combinatorial) which admits distance metric.
5 Conclusion
We thoroughly investigated multiple important aspects around the of LLM embeddings for traditional regression. We found that LLM embeddings can be quite performant for input spaces with high degrecs of freedom, and proposed the Lipschitz factor distribution to understand the embelddngg-to-objetiive landscape and its relationship to regression performance. We further investigated the nuanced conditions for which better language understanding does improve LLM-based regression.
10

---

Published in Transactions on Machine Learning Research (2/2025)
Acknowledgments
We thank Yutian Chen, Daniel Golovin, Chansoo Lee, Tung Nguyen, and Sagi Perel for relevant discussions during experimentation and the writing of this paper. We further thank the organizers of the Google DeepMind Academy Program for providing the opportunity to do this research.
References
LMSYS: Large model systems organization, 2023. URL https: ptttps:/lmmyye org/
Anthropic. The claude model family: Opus, sonnet, haiku. 2024.
Sishuo Chen, Xiaohan Bi, Rundong Gao, and Xu Sun. Holistic sentence embeddings for better out-of- distribution detection. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6676 6686, Abu Dhabi, United Arab Emi- rates, December 2022. Association for Computational Linguistics.
George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bac, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishman, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, and Peter Mattson. Benchmarking neural network training algorithms. CoRR, (bs/2306.07179) 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pro-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- pulational Lánguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 2019. doi: 10.18653/ (118653/V1/1199 11/319-1423.
Association for Computational Linguistics,
Jesse Dodge, Maarten Sap, Ana Marasovie, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext. corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 1286-1305. Association for Computational Linguistics, 2021.
Ouassim Elhara, Konstantinos Varelas, Due Nguyen, Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. Coco: the large scale black-box optimization benchmarking (bbob-largescale) test suite. arXiv preprint (cXXi::903.06390 2019.
Mare G. Genton. Classes of kernels for machine learning: a statisties perspective. J. Mach. Learn. Res., 2: 299 312, March 2002. ISSN 1532-4435.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Google. Tix: A tonsorflow-based production-scale machine lcarning platform. https://www.tesstt
org/tfx 2023. Accessed: November 1, 2024.
Google. Gemini: A family of highly capable multimodal models, 2024.
Google Cloud. Vertex ai automl.
2023. Accessed: November 1, 2024.
11

---

Published in Transactions on Machine Learning Research (2/2025)
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity, In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett (ods.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1991-5001, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6769 0001-778.12 Association for Computational Linguistics, 2020.
Agustinus Kristiadi, Felix Strieth- Kalthof,, Marta Skreta, Pascal Poupart, Alán Aspuru- Aspuru-Guzik, and Geoff Pleiss. A sober look at Ilms for material discovery: Are they actually good for bayesian optimization over molecules? In Forty-first International Conference on Mochine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. DpenReview.net, 2024.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.); Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 9119 9130. Association for Computational Linguistics, 2020.
Qi Liu, Matt J. Kusner, and Phil Blunsom. A survey on contextual embeddings. CoRR, abs/2003.07278, 2020.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. In 6th International Conference on Learning Represen- lations, ICLR 2018, Vancowver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.ngg net., 2018.
Tung Nguyen, Qiuyi Zhang, Bangding Yang, Chansoo Lee, Jorg Bornschein, Yingjie Miao, Sagi Perel, Yutian Chen, and Xingyou Song. Predicting from strings: Language model embeddings for bayesian optimization, 2024.
Rodrigo
Frassetto Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of the transformers with simple arithmetic tasks. CoRR, 0ts/2102.13019, 2021.
OpenAL. GPT-4 technical report. CoRR, abs/2303.086
2023.
Phitchaya Mangpo Phothilimthana, Amit Sabne, Nikhil Sarda, Karthik Srinivasa Murthy, Yanqi Zhou, Christof Angermueller, Mike Burrows, Sudip Roy, Ketan Mandke, Rezsa Farahani, Yu Emma Wang, Berkin Ilbeyi, Blake A. Hechtman, Bjarke Roune, Shen Wang, Yuanzhong Xu, and Samuel J. Kaufman. A flexible approach to autotuning multi-pass machine learning compilers. In Jaejin Tee and Albert Cohen (eds.), 30th International Conference on Parallel Architectures and Compilation Techniques, PACT 2021, Allanta, GA, USA, Seplember 26-29, 2021, pp. 16. IEEE, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140011140:67 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3980 3990. Association for Computational Linguisties, 2019.
Xingyou Song, Oscar Li, Chansoo Lee, Bangding Yang, Daiyi Peng, Sagi Perel, and Yutian Chen. Omnipred: Language models as universal regressors. CoRR, abs Abs/2402..14444 2024.
12

---

Published in Transactions on Machine Learning Research (2/2025)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Lample. Llama: Open and efficient foundation language models. CoRR, 2bs/2330.13371, 2023.
Robert Vacarcanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdcanu. From words to numbers: Your large language model is secretly A capable regressor when given in-context examples. CoRR, abs/2404.07544, 2024.
Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In 6th Inter- national Conference Learning Represeniations, ICLR 2018, Vancowver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Amir Yazdanbakhsh, Christol Angermüller, Berkin Akin, Yanqi Zhou, Albin Jones, Milad Hashemi, Kevin Swersky, Satrajit Chatterjee, Ravi Narayanaswami, and James Laudon. Apollo: Transferable architecture exploration. CoRR, abs/2102.01123, 2021.
13

---

Published in Transactions on Machine Learning Research (2/2025)
Appendix
A Extended Experiments
A.1 High Dimensional Regression
For full transparency, In Figure we display BBOB functions where LLM-based regression was con- sistently dimensionally robust against. MLP and XGBoost baselines. Note that in these cases, we still certain cases where language model outperforms at least one of the baselines, e.g. in the Discus and DifferentPowers functions, Gemini and T5 outperform MLP but not XGBoost.
DifferentPowers
Ellipsoidal
LinearSlope
Discus
1.0
0.9
0.9
0.9
0.8
0.3
0.6
0.0
0.4
100
100
Kendall-Tau Correlation
BentCigar
BuecheRastrigin
AttractiveSector
Schwefel
1.00
0.9
0.75
0.75
0.6
0.50
0.50
0.3
0.25
0.25
25 Katsuura
75
100
Weierstrass
NegativeMinDiffernnce
0.9
0.4
0.5
0.0
0.2
0.3
-0.1
75
100
75
100
Degrees Freedom (DOF)
XGBoost
MLP
Gemini Pro
T5-XXL
Figure 11: Following Figure in the main body, we present. BBOB functions which LLM embeddings did not completely outperform traditional baselines.
14

---

Published in Transactions on Machine Learning Research (2/2025)
A.2 Real World Results
Despile Table 1 of the main body showing that there were numerous cases where LLM embeddings outper- form traditional ones, we remind the reader in Figure 12 that on average, LLM embeddings still slightly underperform.
Full Real World Regression Results
0.7
Kendall-Tau Correlation
0.6
MLP
T5-Large
Gemini Nano
TS-XL
0.3
Gemini Ultra
0.2
0.1
AutoTFX
Init2Winit
XLA
L2DA
Task
Figure 12: Full Results real world tasks. Displayod is the mcan Kendall- Tan Correlation all tasks within each family.
A.3 Performance Correlations
Following Figure in Table 2 we see that the relationship between the smoothness induced by the embedding and the performance in regression is consistent throughout.
Model
DOF=5
DOF=10 110
DOF-25
DOF=50
DOF=100
Gemini Nano
0.81
0.81
0.70
0.75
0.86
Gemini Pro
0.78
0.77
0.72
0.82
0.88
T5-Small
0.75
0.76
0.79
0.79
0.76
La-Large
0.78
0.73
0.79
0.85
0.79
T5-XL,
0.82
0.60
0.80
0.86
0.85
T5-XXL
0.72
0.76
0.82
0.83
0.83
Table 2: Full set of data for Pearson correlation between Kendall's regression performance and gap in NLFFD between input and embedding space for regression all 23 BBÖB functions, over DOF= 10, 25, 50, 100).
15

---

Published in Transactions on Machine Learning Research (2/2025)
A.4 Pooling Ablations
We ablate the elfects of using different pooling mechanisms T5-XXL logits, and see that average-pooling is consistently the best among the three standard pooling methods (details in Appendix B.2p. We further see that using last-token logits not only performs the worst in absolute performance, but also is generally not dimensionally robust in comparison to average-pooling or max-pooling.
Sphere
RosenbrockRotated
Lunacek
NegativeSphere
0.8
0.70
0.65
0.65
0.60
0.7
0.60
0.55
0.55
0.50
0.6
0.50
0.5
0.5
0.45
0.40
0.40
0.35
0.4
0.35
0.30
100
60
Kendall-Tal Correlation
SharpRidge
Gallagher
Gallagher21Me
Griewankkosenbrooc
0.7
0.65
0,70
0.60
0.60
0.65
0.6
0.50
0.50
0.55
0.45
0.50
0.4
0.40
0.35
0.35
0.40
0.3
0.35
40
60
100
20
40
60
100
20
60
80
100
20
60
80
100
SchaffersF7
CchaffersF7llConniitee
Rastrigin
StepEllipsoidal
0.70
0.55
0.55
0.65
0.60
0.65
0.50
0.45
0.55
0.45
0.40
0.50
0.55
0.45
0.35
0.50
0.40
0.35
0.30
0.30
0.25
0.30
0.40
20
40
60
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Degrees of Freedom (DOF)
Avg-Pool
Max-Pool
Last-Token
Figure 13: Higher (1) is better. Follows the setting Figure 2
but comparing different. embedding
pooling methods.
16

---

Published in Transactions on Machine Learning Research (2/2025)
A.5 XGBoost for LLM Embeddings
In Figure of the main body, XGBoost outperforms MLP when using Ptrad as input features. However, this pattern does not hold when using ØLLM, as shown in Figure 14 Here, XGBoost consistently underperforms against an MLP as a regression head for T5 emboddings across all BBOB functions, model sizes, and DOFs.
Despite this difference in head performance, the key observation remains: T5 embeddings exhibit dimen- sionality robustness and improve with model size regardless of whether XGBoost an MLP is used the regression head. This finding suggests that the quality of LLM embeddings fundamental property, independent of the specific regression technique employed.
Sphere
RosenbrockRotated
Lunacek
Rastrigin
0.55
0.60
0.65
0.50
T5- 55-mall
0.55
0.60
0.55
0.50
0.55
0.50
0.40
0.45
0.35
0.50
0.30
0.45
0.35
0.25
0.40
0.40
20
40
60
60
100
20
40
60
60
100
20
40
60
80
100
20
40
60
80
100
0.65
0.70
0.60
15-Large
0.65
0.55
0.55
0.55
0.50
Kendall-Tar Correlation
0.60
0.50
0.50
15
0.55
0.45
0.40
0.45
0.50
0.40
0.35
0.45
0.40
0.30
0.25
0.40
20
60
60
100
20
60
80
100
40
64
80
100
20
40
60
80
100
0.75
0.65
0.65
0.55
0.70
0.60
0.60
0.50
0.45
T5-XL
0.65
0.55
0.60
0.50
0.35
0.45
0.30
0.40
0.25
0.45
0.35
0.40
20
40
60
60
100
20
40
60
60
100
20
40
60
80
100
20
40
60
80
100
0.70
0.65
0.65
0.65
0.60
0.60
0.55
T5-XXL
0.55
0.50
0.65
0.50
0.45
0.40
0.45
0.45
0.55
0.35
0.40
0.40
0.30
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Degrees of Freedom (DOF)
XGBoost (Regression Head)
MLP (Regression Head)
Figure 14: Higher (t) is better. Follows the same setting as Figure 2 but comparing XGBoost and MLP
over ØLLM, when varying BBOB functions (horizontal axis) and T5 model sizes (vertical axis).
17

---

Published in Transactions on Machine Learning Research (2/2025)
B Exact Modeling Details
B.1
Hyperparameters Used
The full list of hyperparameters and training details for MLP-basek regression (using traditional and language model leatures):
Regression Head: MLP with ReLU hidden layers dimension 256.
Input-Normalization:: We linearly scale each coordinate in & [-1, 1], using its minimum and maximum observed values the original endpoints.
y- Normalization: We compute the empirical mean μ and standard deviation over all y-values in the task's training data, and apply + (y a as preprocessing step.
Optimizer: AdamW with sweeped learning rates across {1e-4, 5e-4, 1e-3, 5e-3, 1e-2] and weight decay across {0, 13.
Loss: Mean Squared Error.
Maximum Epochs: 300, with early stopping enabled.
Our XGBoost uses the same imput normalization method the MLP. We additionally grid-searched over the following parameters for each task:
"min_ hin__hhil_weiight' [1, 5, 10]
learning_rate" [0.001, 0.01, 0.1]
ggamma: [0.0, 0.3, 0.5]
"subsample": [0.6, 0.8, 1.0]
coosamplee_yttee bytree": [0.6, 0.8, 1.0
"max_dee max__eephh: [3, 5,
B.2 Pooling Mechanisms
Tokenized strings may be appended with additional tokens to the maximum sequence length L. Thus for example, before forward pass, a token sequence may actually appear (2,,4,....,44 L2, 1,0,0, 0,0,0,0..... 0,.... where & and 0 and 1 are padding and optional end-of-sequence (EOS) tokens, respectively. Below, specify how each pooling mechanism performs postprocessing steps over logits.
Avolagg-Pooling (default.): Zero-out padding token logits, and average across the entire length axis. Max-Pooling Same as Average-Poolingg but take the maximum across the length-axis for each coordinate.
Last-Token: Only obtain the logit of ta or the EOS token, similar the <CLS> token method but applicable to decoder-only methods as well.
Note that while Average-Pooling will have its coordinates scaled down by the maximum length L, input normalization from the regression head will re-scale them back to valid ranges.
B.3 Embedding Sizes
Table 3 displays the embedding dim for each model used in our experiments. As mentioned in the main text, note that dum is significantly larger than derad.
T5 Model
dnm
Gemini Model
dum
Small
512
Nano 1536
Large
1024
Pro 6144
XL. 2048
Ultra 14336
XXL
4096
Table 3: Embedding dimensions dum for T5 and Gemini model families.
18

---

Published in Transactions on Machine Learning Research (2/2025)
C Extended Data Details
C.1 Additional Data Details
The Black-Box Optimization Benchmarking (BBOB) suite (Elhara et al. 2019 consists of 23 synthetic objectives which supports evaluation over continuous imputs of any DOF. Such functions compre- hensively cover different landscape conditions (e.g. separability and optimality). Every function f(x) by default has a global minimum f(z) = at exactly the zero-vector input. Example of such functions include:
BBOB:
Sphere: f(x) = - i=1
DOF (zx10)2
Rastrigin: f(x) = 10 DOF Ei= DOP (2s((221))) + ||x11
BinaCr f(x) (z(1)² + 10
Real World Data: We use extensive set of hyperparameter and blackbox optimization objectives, these also tend to possess varied landscapes suitable for regression benchmarking.
AutoML: Objectives collected from the Vertex AI platform for automated ML model selection and training for tabular or text data. For tabular data, Vertex AI searches over a tree of model and optimizer types, their hyperparameters, data transformation, and other components in the MI. pipeline. For text, Vertex AI trains an ML model to classify text data, extract information, understand the sentiment of the authors. For more information, see: htttps://clou..gooon wow.ge.coooooooooooooooooooooo.. moodeees--th--mmmmmll---
vertex-ai
Wnitt:init: Data from running deterministic, scalable, and well-documented deep lcarning exper- iments, with a particular emphasis on optimization and tuning experiments (e.g. ResNets on CI- FAR10, Transformers on LMIB). Public codebase can be found init2winit
wwt.coom/oooooo com/google/
XLA: Tuning LLM latencies over variely of different settings for the XLA compiler. Such hyperparameters affect sharding, partitioning, memory consumption, and data limits when plac- ing LLM models accelerators. Such hyperparameters can be found in OpenXLA https:
//github. wtt.uu...o/oopaoeeeeeeeeeee com/openxla/xla
L2DA: Hardwwre-specific hyperparameters such as memory depth and input-output bandwidths, used to optimize computer hardware performance or reduce costs. Many such similar hyperparam- eters can be found in https: htw.://....o.etttttttaataeaaaaaa archass--rrr-eeee gyaggmmmm
C.2 String Representations
Table contains example string representations x for different regression task families.
Task Family
Example Representations
ВВоВ
[0.3221, -4.2113, 3212 1.5621]
AutoML
batch_size:122 h_l.fattur_ssoleccte 1tt.:eeecciooottrechooo0000 0.00, model_typer DORN_ESTINATORR
aetivation_fn: fs:'' batch_norm: 'False', bucketization _ttttttegy::y 'mdl',
dropout:0.071. (eidden_units:333
Init2Winit
rr.hparams.barr base_baaee_____ wtwpe_rr:.0066... opt_hparams.. 0 h.s.p..p....oo.............
b1:0 2823,
opt_hparams.O..m wps.
sat___nnn_______e (ttp__2200033.. opt_bparams... L.b. w.t.pp....g............... Sewigghtdeeayy....om
XLA
'False',
L2DA
aoput_activation_.og memtrrattiiioammmaaaaaeamaaa _tt.__moooooddddd1.____
instuuction_mmorry de.th:15.0,,
io bhanduidth_gbae... 321, narrow cartoov_mmmrrr.omm htt._mmoor__pcccciiiee capacity :taprraaaaaccttyyyeeee....... 21.00
Table 4: Example x representations from each of the regression task families. !..." denotes that there are actually more parameters, but we omit them due length.
19


Reviews:
[{"review_id": "OPwvsp23dt", "reviewer": "Reviewer_rvsG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The authors present a novel perspective by attempting to use LLM embeddings for regression tasks, which is an innovative idea.\n\n- The robustness of the proposed method to the number of data points and higher dimensional problems, as seen in Figure 9 and Figure 2, are interesting and promising observations for further exploration.\n\n\n## Weaknesses:\n\t\n**Major Concerns:**\n- The paper's motivation for using LLMs in regression is unclear. While LLMs operate on discrete tokens, regression deals with continuous spectral properties. LLMs' domain knowledge as additional prior may be one potential benefit. However, the current reported results contradict these. For example, random LLM initialization performs well (as in Figure 7), and feature names provide no meaningful advantage (as in Figure 8). This raises fundamental questions about the necessity of LLMs for this task, as neither pre-training knowledge nor domain knowledge appear to contribute significantly to performance.\n\t\n- Table 1 shows that the LLM-based method underperforms against the non-LLM baseline on real-world problems, with success rates below 50% across all datasets and mostly under 20%. Given its success on high-dimensional black-box problems (in Figure 2), testing on real-world benchmarks with similar dimensionality (≈100D) would be more appropriate.\n\t\n- Despite the motivations over LLM decoding-based regression approaches, no comparisons are made with such methods. The evaluation needs comparisons with recent LLM-based regression approaches (Song et al., 2024; Vacareanu et al., 2024) to validate the contributions. Current evaluation only includes non-LLM baselines (XGBoost, MLP). \n\t\n- Also, the selected real-world benchmarks in this paper are not that common in regression analysis. It would be helpful to provide evaluation on more common regression benchmarks such as OpenMLE or PMLB.\n\n\n**Minor Concerns:** \n- I would suggest presenting Figure 7 results also on BBOB tasks across different DOFs to clarify the role of pre-training, especially for higher-dimensional problems where the proposed method outperforms non-LLM baselines as shown in Figure 2.\n- Figure 9 is showing the proposed method's robustness to data size. Comparing this with non-LLM baselines could also highlight a potential advantage of LLMs.\n- On page 14, Section A.2, Figure 11 is mistakenly referenced as Table 11.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an LLM-based regression method that utilizes embeddings rather than decoding approaches. The authors evaluate their method on various black-box optimization benchmark problems and some real-world datasets, comparing with non-LLM regression baselines (XGBoost and MLP).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Evaluate on high-dimensional (≈100D) real-world benchmarks where the method shows promise\n\n- Add comparisons with LLM-based regression methods (Song et al., 2024; Vacareanu et al., 2024)\n\n- Present Figure 7 results on BBOB tasks across different DOFs\n\n- Compare data size robustness (Figure 9) with non-LLM baselines", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sncw6gWTpJ", "reviewer": "Reviewer_SWar", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts: I think the organization of the paper is good and the transitions between sections are smooth. The contributions are strong. Particularly, the explanation to the success of the feature representations in the regression task using the Lipschitz continuity of the representation over a dataset in Figure 4 is insightful.\n\nWeaknesses: I have the following suggestions and questions:\n\n* It would be very beneficial to explain the BBOB suite because this dataset is utilized a lot throughout the paper. It is very beneficial for a reader to know the basics of this dataset similar to other dataset AutoML, Init2Winit, etc.\n\n* In Section 2, the authors explain how they achieve a feature in $\\mathbb{R}^d$. In the forth step of this explanation, the authors mention pooling. Could the authors explain how they apply the pooling. From the next page, I understand that they utilize average pooling. However, it would be better if they explain it under these steps. I am also curious what would happen when they utilize different pooling. I think this is particularly important when the authors analyze the Lipschitz continuity of the feature space.  \n\n* The authors inform that the regression is performed using a MLP that is similar for both $\\Phi_{\\text{LLM}}$ and $\\Phi_{\\text{trad}}$. However, the authors also compare the LLM-based features with XGBoost as well. I am curious what would happen when they utilize different kind of regression such as XGBoost instead of MLP. In some cases, XGBoost outpetforms MLP. That is why I am curious about the usage of XGBoost instead of MLP. Could the authors explain these behavior in such cases?\n\n* Do the authors have any explanation of Figure 7? Specifically, why does the random initialized model is pretty succesful in AutoML and L2DA? I think this part needs more explanation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the use of embeddings from LLMs as features for regression tasks, providing a comprehensive analysis of their behavior. \n\n* It demonstrates that LLM embeddings maintain strong regression performance even in high-dimensional data scenarios, where traditional feature representations typically degrade in quality. The authors also illustrate some cases where traditional feature representations are better than LLM based features\n\n* It shows that LLM embeddings over numeric data inherently preserve Lipschitz continuity and smoothness in feature space, facilitating effective regression using downstream models such as MLPs. Additionally, the authors build a bridge between the Lipschitz continuity of the feature representation and the success of regression.\n\n* This paper quantifies the effects of LLM-related factors (model size, pre-training quality, and input formatting) on regression performance, revealing that improvements in language understanding do not always lead to better regression outcomes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I asked some questions and changes in the strengths and weaknesses section.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WOxMOA1VEQ", "reviewer": "Reviewer_YbPJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nRigorous empirical methodology with comprehensive experiments across diverse regression tasks;\n\nNovel analytical framework (NLFD) for understanding embedding properties;\n\nClear and detailed ablation studies examining various factors affecting performance.\n\nWeaknesses:\n\nIt would be good for authors to provide some insights on why LLM embeddings work well in certain high-dimensional cases but not others;\n\nThe paper could benefit from more theoretical analysis, e.g., the relationship between embedding dimension and regression performance;\n\nThe real-world tasks are all from relatively specialized domains; could include more general regression problems;", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the use of LLM embeddings in regression tasks.\n\nIt provides analysis of LLM embeddings' effectiveness in regression tasks compared to traditional feature engineering approaches, particularly for high-dimensional problems.\n\nExtensive experiments have been performed across both synthetic benchmarks (BBOB functions) and real-world regression tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Provide more theoretical justification for the dimensionality benefits observed.\n\nExpand analysis of cases where LLM embeddings underperform.\n\nInclude more diverse real-world regression tasks beyond specialized domains.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the use of LLM embeddings as features for high-dimensional regression, comparing with baselines such as XGBoost and MLP on black-box optimization benchmark problems and real-world datasets. The authors present cases in which LLM embeddings outperform traditional feature representations and also cases in which the traditional features perform better. Remarkably, the LLM embeddings are shown to preserve Lipschitz continuity in feature space. Furthermore, it is shown that an improvement in the LLM (in terms of model size and language modeling) does not necessarily lead to better regression performance. 

During the revision process, the authors have successfully addressed the concerns of the reviewers, also providing a number of additional experiments. All the three reviewers have expressed a positive recommendation. I concur and recommend acceptance.
Decision: Accept (poster)