Decoding-based Regression
Authors: Xingyou Song
Dara Bahri
Venue: Accepted by TMLR ()

Abstract: Language models have recently been shown capable of performing regression wherein numeric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence decoding models as numeric regression heads given any feature representation. We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoder-based heads are as performant as standard pointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distributions, such as in the task of density estimation.

Published in Transactions on Machine Learning Research (08/2025)
Decoding-based Regression
Xingyou Song", Dara Bahri"
Google DeepMind
"Equal Contribution.
Reviewed on OpenReview:
Abstract
Language models have recently been shown capable of performing regression wherein mu- meric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence de- coding models as mumeric regression heads given any feature representation. We find that, despite being trained in the usual way for next-token prediction via cross-cntropy loss decoder-based heads are as performant as standard wointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distribu- tions, such as in the task of density estimation.
1 Introduction
Despite being originally developed for the purpose of text generation and chat applications, large lan- guage models (LLMs) have recently been applied for new applications, particularly one of which is regression, and more broadly the prediction of mu- meric outcomes. Vacarcanu et al. (2024) have shown service-basee based LLMs such ChatGPT and Gemini capable of regression with performance comparable to that of traditional regression methods such as random forests, while Song et al. (2024): Akkauri ee al. (2025) have shown smaller cuslom language models can be trained specifically on multiple re- gression tasks for transfer learning.
Po y x
Regression Problem
0011
1001
Decoder
Ø x
x
For an input-output pair v)) where æ is feature vector and is a real number, a regression model's performance is determined by two factors: how it processes x and how its output "head" represents y. While the mentioned works (Vacarcanu et al.)
Figure 1: Given any feature representation Φ(z), we attach a decoding- head output predictive dis- tribution p)))x))
2024 Song et al.] 2024 Akhauri et al.) (2025) can be seen tert-to-tezt regression where both and y represented
tokens, this combination is not necessarily required. Tang et al. 2024 investigate the isolated case where LLM embeddings of æ are attached to feod-forward networks as regression heads, while Nguyen et al. (2024) investigate the case when these embeddings are eventually attached to Gaussian distribution heads. Both can be seen as particular instances when x is represented as text, while common regression heads are still used. However, there has not been work investigating the inverse situation, i.c. y is represented as text or structured tokens. One could do so by using decoding- basedd regression heads, where for example, tokens <i><,> <<<.><>><>> be decoded to represent 1.23, technique used in several works

---

Published in Transactions on Machine Learning Research (08/2025)
training language models for specific numeric tasks, such as arithmetic (Nogueira et al. (Charton 2022), and symbolic regression (d' (d'Ascoli et al. 2022).
2021 linear algebra
In contrast to traditional deterministic or parametric distribution (e.g. Gaussian) regression heads, decoding- based heads may be much more flexible, as they can represent any numeric distribution approximately over IR without the need for explicit normalization. However, due to their initial lack of inductive bias over numeric distances, numeric token representations and sequential dependencies may need to be learned using additional training data, and thus it is worth empirically investigating these trade-offs in isolated, controlled experiments. Our research provides valuable insights on using mumbers autoregressive decoders. Our specific contributions and findings are thus as follows:
output modality of loken-based
We formalize decoding-basee regression, i.e. explicitly define tokenization schemes for numbers, es- tablish training and inference procedures, discuss methods for pointwise estimation, and theoretically provide risk guarantoes for density estimation under common assumptions.
In experimental benchmarks, we find that properly tuned decoding-basee based regression heads are data- efficient and competitive with regular pointwise heads on tabular regression, yet are also expressive enough to perform against Gaussian mixture heads for density estimation.
2 Related Work and Motivation
The idea of text-lo-terd regression is especially relevant LLMs are currently being fine-tuned as "Generative Zhang al 2024). i.e. end-to-end scoring methods for reinforcement
Reward Models" (Mahan et al. 2024
learning feedback /Ziegler et al. 2019 Bai et al.) 2022) or LLM-as-a-Judge (Zheng et al. 2023) Such reward
modeling methods can be simpler than other forms such as Bradley-Tery (Bradley & Terry 1952 which
requires appending additional prediction heads and custom losses. However, little analysis has been done in isolation on the theoretical and modeling capabilities of using text, or more generally tokens, to represent real numbers. Understandably, one could argue that regular supervised fine-tuning over numbers represented as strings is unprincipled, considering that there is no notion of numeric distance when using cross-entropy loss.
However, we argue that token-based numeric modeling is actually natural, given observed phenomena and techniques proposed in recent works. Given a prot-processed representation d(x) E Id after x is sent through a task-specific encodor (MLP, CNN, etc.), we provide an overview of common regression heads po(v)((x)) with trainable parameters 0, which can be applied on top of Φ(x) to return numeric outputs (additional techniques in Section 5)
Pointwise Heads:
By far, the most commonly used regression head is a learnable deterministic function with weights 0, typically a simple feed-forward network (often a single linear layer) mapping Φ(x) to a single-point scalar, trained by minimizing pointwise loss like mean squared error. To allow stability during training, the y-values must be normalized space, e.g. within [].
In the case of probabilistic outputs, one may apply distributions with parametric forms. A common example a Gaussian head, e.g. po po(ylo(z)) = NNu, 0²) where μi, a are deterministic learnable functions of (xx). A more flexible variant is a finite mixture of Gaussians (Bishop) 1994), which can be extended to infinite mixtures (Rasmussen 1999). Such mixture techniques can be more
Parameteric Distribution Heads:
broadly seen within the realm of density estimation (Parzen 1962 Rosenblatt 1956) in which complex
distribution may be estimated using multiple simpler basis distributions.
Histogram (Riemann) Distribution Heads: One such basis common in deep learning applications is the piece-wise constant basis, for learning histograms over a finite support set (y1....,,,, C IR via softmax paramelrization, i.e. po (y: (x(x) = Softmax (1) (d(x)² 0) where 0 E RR", which has led to strong results in based reinforcement learning [Imani & White 2018 Bellemare et al. 2017) and tabular data (Hollmann) ct al.) 2025 Chen et al.] 2022). However, a drawback is that learning numeric distances between all of the bins requires more data as the size of the vocabulary increases. We term these as Riemonn heads, following (Hollmann et al. 2025
2

---

Published in Transactions on Machine Learning Research (08/2025)
While there have been works ordinal regression to learn rankings among these bins, such using rank- consistency (Cao et al. 2020) and soft labels / metric-awareness (Diaz & Marathe 2019). we propose a much simpler way, by simply considering the histogram distribution as a special case of decoding a sequence of length 1. By extending the sequence length instcad, there can be an exponential reduction in bin count - c.g. 1000 (=10²) bins can be expressed instead using 10 bins and 3 decoding steps. While this intuitive idea has been studied extreme classification problems (Wydmuch et al. 2018): it has not been thoroughly examined for numeric regression, which the focus of our work. Below, we introduce decoding- regression heads.
Note: To prevent confusion with the term "decoder" which is also central component of generative models like Variational Autoencoders (VAEs) (Kingma & Welling) 2014), we stress a key distinction. While both VAE decoders and distributional regression heads map a feature vector to a probability distribution, their objectives differ: a VAE decoder models p(zló(z)) to reconstruct the input ø from a latent Φ(x), whereas a regression head models p(y]o(()) to predict a separate target y. Given this difference in the output space, we avoid referring to standard distributional heads (c.g., Gaussian) as "docoders".
3 Decoding-Basee -Based Regression
In this work, we define the decoder head as an autoregressive sequence model, such as compact Transformer decoder. Given vocabulary V, the Transformer takes the feature vector 6xx) as its initial context, and generates discrete sequence of tokens (41..., E yK one token at a time. This sequence is a string repre- sentation of a number, and by modeling the probability pe (4,,....,,& ((x)) the head implicitly defines probability distribution over discrete set of representable numbers. Below, we discuss natural token representations of numbers.
II po(ts Φ(z), 4 (.....ax)).
3.1
Numeric Token Representations
Normalized Tokenization: If y is restricted to [0, 1] (via scale normalization for example), then in Section 3.3 we show any smooth densily p(ylo(x)) can be approximated with an increasing level of granularity as more tokens are used in the numeric representation, under some "universality" assumptions on po- This can be seen intuitively with a based construction, i.e. for a base B, the vocabulary contains <0>, <1>, .... <B 1>, and y is simply represented by its base-B expansion up to length K. This setting aligns with standard data-science practices of also normalizing y-values according to training data or known bounds.
Unnormalized Tokenization: However, there are cases in which we would like to use unnormalized tokenization scheme. Such cases include multi-task regression (Song et al. 2024), in which different tasks may have varying y-scales, or express very wide y-ranges for which appropriately normalizing y-values for the correct balance between numeric stability and expressiveness would be very tedious.
In this case, we may simply view normalized tokenizations as "mantissas" and then left-append sign and ex- ponent tokens to form a base-B B generalization of the common IEEE-754 floating point representation (EEE 2019). Given length parameters E and M, our tokenization is therefore <>>
><<>>.. >......... <eg><m>>... <my>> where Se, e1, , eg are the sign and base-B representation of the exponent and m1, m2, ................................ .... m. are the
most significant base-B digits of the mantissa. E.g. if (B=10, E=3, M=4), then 10 10-222 x 1.23456789 will be represented as
<D<><>>>>><>>>>>>>>>>>>>>>>>>>>>
Signs 
, <sp> can have their own dedicated <->, <+>
tokens or optionally reuse the <0>,<1> tokens from V; this made little difference in results.
Note that the vocabulary can additionally contain "spocial" tokens for representing outputs not within a supported numeric range. For example, one can use a token <NaN> to represent non-numbers, commonly used in cases where may be invalid input. We mention such cases for useful downstream applications, although the scope of this paper assumes is always within R.
Any auloregressive model can be used, so long as it supports constrained token decoding Lo enforce proper sequences which represent a valid number. By default, we use a small Transformer (Vaswani ct al. 2017) due to its strong autoregression capabilitics, with the initial token embedding as Φ(x). As we show in our experiment section, this Transformer size can be made minimal, with negligible contribution to parameter count in comparison to the encoder.
Architecture:
3

---

Published in Transactions on Machine Learning Research (08/2025)
Since the token space is finite while R is uncountable, this mapping is lossy (i.e. not invertible) and introduces notion of rounding error. However, for large enough base B and sequence lengths (both normalized and unnormalized), practically any y-value will be within the expressible range and rounding errors will be minimal. The trade-off is that the vocabulary size and sequential dependencics between tokens will also increase, and learning better numeric represenlations may thus require more training data. While it's possible to first pretrain these numeric representations as in Hollmann et al. (2025) for the histogram distribution, we show that with proper hyperparameter tuning, the Transformer decoder can be used out-of-the-box as a randomly initialized regression head.
3.2 Pointwise Estimation
In many cases, one may only be interested in a scalar quantity of interest M(ps) of the model's distribution (e.g. its mean). If po matches the true distribution p perfectly, then for given pointwise loss é RR² - R the goal is then to select (p) which minimizes Ey~p(-1a) (p), v)]. It is well established that for common error functions like L2, L1, and LO, the optimal values are the mean, median, and mode of p(-[x), respectively, Lukasik et al. 2024) also leverage this observation to enhance LLM decoding.
To estimate these (p), the mode can be approximated using e.g. beam search (Graves 2012 but efficiently estimating other common general pointwise representatives (p) from pure temperature samples is a broad topic- for example, one can efficiently approximate the true median from the Harrell-Davis estimator (Harrell
& Davis 1982 and more generally we refer the reader to Lehmann 1983 on statistical point estimators.
Especially for unnormalized tokenization, additional care needs to be taken, since in practice, the model can have a miniscule but non-zero probability of decoding an arbitrarily large outlier, even if the underlying true distribution is bounded. Such outliers can easily sway non-robust estimators such as the sample mean, as observed in Song et al. (2024). This issue fundamentally comes from the fact that some tokens are more significant than others, prompting the use of alternative tokenizations based on coding theory which are robust to corruptions, which we show can be effective in experiment section.
Alternatively, decoding techniques from the LLM literature can also be used, e.g. top-k (Fan et al. 2018) or top-p (Holzzman et al.) 2020), or even simply decreasing the temperature to increase model confidence and thereby filter out possible outliers. One can also avoid decoding altogether and use recently proposed RAFT Lukasik et al. 2025 Chiang et al. 2025) which estimates (p) using a query-based approach using a finite fixed evaluation set V, e.g. for mean, Engepulyl [y] ~ 1
N EyyyPo(v') v', although the choice of y may be non-trivial to obtain an unbiased estimate, especially over unnormalized tokenizations. This may also defeat the purpose of using decoding head, which offers several density estimation benefits, as discuss below. Overall, the choice of method for computing pointwise representations we leave hyperparameter to be tuned depending on the application.
3.3 Density Estimation and Theory
During training, to allow the full probabilistic modeling benefits of using decoding head, we apply the standard cross-entropy loss over all sequence tokens. For a model pe and target = (t1,..., ttx,, the cross- entropy loss (omitting x to simplify notation) will be:
K
Po) = Σ Σ
10 = 44) log 1k-1)
-
k=
LLEV
The expected loss over all y sampled from the true distribution is then Eysp [H(
Given our tree-based tokenization and training loss, we provide formal guarantees for estimating one- dimensional densities on 1]. Note that densities with finite support can be shifted and rescaled to have support in [0, 1]. Define d 10. + {0, 1) be the operation that returns the first k bits after the radix point in the (possibly infinite) binary representation of y. Concretely, if y 0.b, bybyb... then (v) = (D.....,by) We abuse notation and interpret A ''s output either as sequence or as the real number it represents (Etzb b,2-t) depending on the context. The analysis is presented using binary (base-2) repre-
4

---

Published in Transactions on Machine Learning Research (08/2025)
sentations (e.g. V= {0, for simplicity, but it holds for arbitrary bases. First, provide assumption on the learnability of our model and additional definitions:
Definition 1 (K-bit universality). Let (p, q) By- -oggp-logg0y q(y) denote the cross- cross-entropy between discrete distributions and q- Note that (p, p) is just the Shannon entropy of Call parametric model K-bit universal is for all discrele distributions on K-bil strings (equivalently, 2 calegories),
min H(p, H(p.)).
In other words, po is K-bit universal if it is flerible enough to fit any discrele distribution 2 colegories. Definition 2. Desine ph probability of the first k bits under po, marginalizing oul the remaining bils. Concretely,
(3((b,,...,bb))
E pol(b....... bk)).
(hx. I....,b))
Seen onother way, pe is the distribution over k-bit strings that results from auto-regressive decoding pe for exactly steps.
Definition 3. Let f: [0, 1] + be a densily function. With {Y....., Yn} as i.i.d. draws from f, define 0r the mazimum likelihood estimator the truncated sequence of K bits. Concretely,
0" (Y...... Yn) argmin
log (Ax
Define risk the mean integrated squared error between true densily f and an eslimator YN):
R(, VI
We now give our main result below, expressing the distributional fit in terms of bias and variance. The proof is deferred to Appendix B
Theorem 1. Assume
decodinggbassee regression model Ps : {0, + A is K-bit universal, and f be any twice continuously differentiable density function. If the marimam likelihood estimator at k is 1 (v) = 2 (M.....w) (2x (v)) fory E [0, 1), then the risk can be exactly computed:
-k2
S' 1'(y)
2
R R(S.S = 12
+
+
0(2 4k + 1/N), AESK
Negligtble
Bias
Variance
Note that this theorem is broad, applicable to both Riemann and decoding heads even if they perform inference at lower token length & than the maximum length K used during training. For simplicity, let us assume that the maximal length always used (i.e. à = K). Intuitively, this implics that one needs a higher resolution K to capture the curvature of f, but as the number of bins increases, more data points are required learn to scparate these 2 bins. In Figure 2 for large N=16384, we show this trend holds empirically where there is an optimal Ks55 which minimizes the error.
When N is quite small (e.g. 1024) risk (for the better) when the number of bits is large (>9), while the Riemann head still fits it tightly. Recall that we required a "universality" assumption, which says that our model can learn any discrete distribution over K-bit strings perfectly. We can decompose this assumption further into two pieces: 1) that there exists 8* our model class that achieves the minimum cross-cntropy (i.e. Pg- = p in Definition D and 2) that our SGD-based training procedure able find it. An explanation for this phenomenon is that in this regime (low sample size and large number of bits, equivalently, large number of bins), the risk profile of
that the decoder head significantly deviates from the theoretical
5

---

Published in Transactions on Machine Learning Research (08/2025)
N 1024
= 16384
0.25
2.0
0.20
1.5
0.15
Risk
1.0
0.10
0.5
0.05
0.0
0.00
2
6
8
10
2
4
6
8
10
Number of Bits (K)
Theoretical
Decoder
Riemann
Figure 2: Lower (L) is better. Risk (theoretical and empiricall) when varying K and N to fit truncated
(0.5, 0.25²) distribution using binary tokenization. Results averaged across runs each.
the classical Riemann estimator is dominated by the variance term. Few samples land in each bin and as result the histogram-based density estimate for the bins is noisy.
Tt is conceivable that a combination of the inductive bias of our model class and the implicit bias of our SGD training procedure makes the docoder less likely to fit noise; a concrete example would be that the model biased to learn smooth distributions, and so when asked to fit the highly discontinuous empirical distribution arising from dropping few samples into large number of bins, it refuses to, instead opting learn smooth approximation, and thereby driving down the variance term and hence the overall risk. This suggests the decoder head possesses implicit regularization properties which make it much more efficient with low training data.
0
1
0
Figure 3: Visualization fitting truncated Gaussian distribution. Each level & of the binary tree represents the empirical fit using & bits, and each bin gets subdivided into two.
Taking closer look at the decoding mechanism, a crucial observation is that d essentially discretizes the unit interval (and hence f well) into bins {B,}3"n where BN = [j2-*, + 1)2-*) that P(r B,) JB, f(y)dy. We can identify k-bit sequence y = O.b....b ...b with the interval [v. + 2- With a single bit (K 1) learn histogram estimator two bins [0, 1/2) and [1/2, 1) representing and 1. With

---

Published in Transactions on Machine Learning Research (08/2025)
two bits refine our prediction using four bins: [0, 10,1/4), [1/4, 1/2), (1/2,3/4), and [3/4,1) representing (0, 0), (o, 1), (1.0), (1,1) respectively (because, for example (0, 1) means 0.01, 1/4).
We can interpret binary representations in terms of binary trees on 2 leaf nodes where nodes represent intervals (the root representing [0) and left and right children represent the left and right halves of the node's interval. Reading off bits tells us how to traverse this tree, where and 1 mean traverse the left and right subtrees respectively. For example, to arrive at (0,1,1) 0.0112 3/8 our traversal is: [0,1) + [0, + (1/4,1/2) [3/8, 1/2).
When trained on K-bit sequences, our decoding head po simullancously learns K histogram estimators for f; (Ax is the k-th histogram estimator (over 2 bins). In other words, as we decode bits one-by-one aulo-regressively, we are iteratively refining our prediction. Figure shows this mechanism in detail in the case of fitting a truncated Gaussian distribution.
There are alternatives to binary representations, for example p-adic expansions, or even the ttern-Brocot free which uses the mediant to determine the child-parent relationship. An interesting research question left for future work is whether these more exotic representations of real numbers better suited for our soquence-based regression model than the standard reprosentations.
4 Experiments
Our main goals for experiments are to:
Demonstrate decoder heads can be effective swap-in replacements to common pointwisc regression
heads.
Establish the density estimation capabilities of the decoding-bars based head over any distribution over R.
Ablate the effect of decoder head size and sequenco-speciffc methods such as error correction on performance.
To maintain fairness, all neural network methods have access to the same encoder o(x), which is a large multi-layer perceptron (MLP) with ReLU activations, with hyperparameter sweeps over number of layers (up to 5) and hidden unit sizes (up to 2048). Furthermore, the decoder head uses only 1 layer and 32 units, making up for less than 10% of the total network parameter count, which minimizes its contribution to representation learning as a confounding factor.
Furthermore, for distributional heads (e.g. decoder, Riemann), sweep their specific settings (e.g. number of bins tokenization) over reasonable values additional details are found in Appendix C For the vast majorily of (abular regression problems, found that the process of training and tuning only requires at most 20 minutes on single Nvidia P100 GPU, making the decoder head relatively cheap to use. For comparisons, we use relative mean squared crror within individual tasks and scale-invariant Kendall-Tan correlation for aggregate comparisons.
4.1 Curve Fitting
We begin by visually demonstrating the fundamen- tal representation power of tokenization. In Fig- ure 1 the unnormalized decoder head is able to successfully capture the shapes of various fune- tions with which both the Riemann and pointwise head struggle. The issue with using the pointwise (1) requiring
Input Dimension
Regression Head
5
10
15
20
Unnormalized Decoder
89.56
88.71
87.49
86.11
Normalized Decoder
Pointwise 89.40
88.54
86.90
86.02
89.08
88.25
88,06
86.78
head stems from two main factors: y- n-nommalization, which leads to mumeric instabil- itics especially with functions with very high or unbounded y-ranges, and (2) struggling to model abrupt or high rates of change (i.e. large Tipschitz
Riemann
88.94
88.30
87.42
86.78
Table 1: Higher (f) is better. Mean Kendall- Tau corre-
lations over BBOB functions with (==100K) training data. Individual function results can be seen in Appendix A.2
7

---

Published in Transactions on Machine Learning Research (08/2025)
Weierstrass (Smooth)
Vertical Asymptote (Hyperbolic)
Stepwise (Alternating)
Unnormalized Decoder
Normalizedd Decoder
Riemann
Pointwise
Ground Truth
Weierstrass (Trianuular)
Vertical Asymptote (langent)
Stepwise (Increasing)
X
Figure 4: Visual fit to ground truth is better. Curve fitting plots for various 1D functions. Both models are trained over 100K (æ, y) points, where a is uniformly sampled from a bounded range. Note: These results occur regardless of xy-scales, which omitted for brevity. Riemann prediction for "Vertical Asymptote ("Tangent)" went out of range.
constants). In contrast, the unnormalized decoder head does not encounter these issues due to its ability to express very high range of y-values, with the normalized decoder also performing decently.
In Table D as sanity check over dimeensoninsii functions, synthetic continuous objectives from the Black-box Optimization Benchmarking (BBOB) suite (Elhara et al. 2019) can also be sufficiently fitted by both the unnormalized and normalized decoder heads just as well as the pointwise and Riemann heads.
4.2 Real-World Regression
In Figure over real-wor wolldworld OpenML /Vanschoren al. 2013) regression tasks from OpenML-CTR23 (Fichhe al. 2023 and AMLB (Gijsbers al.] 2024), ; using the unnormalized decoder head is competitive to using regular pointwise head given the same training data. In fact, in the majority of tasks, the decoder outperforms the pointwise head, and in few cases, the gap be quite significant (>0.3). Full results in Appendix
Figure 13 also lead to the same conclusion for normalized decoder heads.
AMLB
OpenML-CTR23
Kendail-Tau Correlation
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
Task Index
Task Index
Unnormalized Decoder
Pointwise
Figure 5: Higher (t) is better. Kendall--aau regression
AMLB and OpenML-CTR22 tasks using up to 10K maximum training points. Each bar averaged over 10 runs and bars from the same task (but different regressors) stacked top of each other and sorted by gap performance gap.
8

---

Published in Transactions on Machine Learning Research (08/2025)
both decoding heads outperform the Riemann head in the vast majority of tasks well, suggesting improved sample efficiency from minimizing vocabulary / bin sizes.
In Figure G
Decoder vs. Riemann Performance Comparison AMLB
OpenML-CTR23
1.0
1.0
Decoder Kendall-Tau Score
Decoder Kendall-Tau Score
0.8
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Riemann Kendall-Tau Score
Riemann Kendall-Tau Score
Identity (No Difference)
Unnormalized Decoder vs. Riemann
Normalized Decoder Riemann
Figure 6: Upper left (N) better for decoder heads. Paired scatter plots for comparing both decoders against the Riemann head, over real-world regression tasks.
In order to more rigorously validate the sample efficiency hypothosis, in Figure 7 we compare the use of normalized heads (normalized decoder, Riemann histogram, and pointwise), when varying the amount of training data. We omit unnormalized decoder results, aggregates samples differently. We first observe the data inefficiency of using the histogram head on selected regression tasks in certain cases, the histogram head plateaus, unable even achieve the performance of the decoder head, regardless of the amount of training data.
Task ID: 233212
Task ID: 233215
Task ID: 359930
Task ID: 359938
10-1
7x
4x10-
10-2
6
Relative Mean Squared Error
3x10
2x10
10
10
102
10a
10a
10a
10a
10*
10a
10a
10*
10a
103
10*
Task ID: 359939
Task ID: 359948
Task ID: 359951
Task 360945
8x15
2x10-
4x10
10-1
10-2
3x10m2
10
103
10a
10a
10a
Training Data Size
Normalized Decoder
Riemann
Pointwise
Figure 7: Lower (L) is better. Relative mean squared error (MSE) over selected AMLB tasks. Each method used a min-max linear scaling normalization y-values. Full results in Appendix A.1
Figure

---

Published in Transactions on Machine Learning Research (08/2025)
Furthermore, interesting observations can be made when comparing against the standard pointwise head baseline. In high data regimes (=10² data points), there are cases in which it also plateaus earlier than the decoding head. In low data regimes (=-10 data points), one would expect the decoding head to struggle more as it needs to learn numeric token reprosentations, but as it turns out, the pointwise head can perform worse due to numeric instabilities of its own. Due to undertraining, the pointwise head required appending sigmoid activation to enforce the normalized output to be within [0,1] to avoid extremely high MSE errors.
4.3 Density Estimation
In Figure we further see the decoding head's ability to perform density estimation over various shapes. Given unbounded training data it able to capture the overall distribution p(y(z) well, although there can be slight outlier noise as shown by lighter points. In Appendix A.6 we show that even baseline heads such Mixture Density Networks (MDNs) (Bishop) 1994) and Riemann dist ributions also sulfer from noisy outputs. While enforce the sampling to be tighter (e.g. lowering temperature) to remove noise, this tighter sampling can unfortunately also reduce expressivity. In general, we find that vanilla. temperature sampling with temperature =1.0 is the best way to match p(y|z).
Half Moons
Zig-Zag
Spiral
Hollow Square
X
Sample
x
Ground Truth
Figure 8: Fit to ground truth is better. Density estimation visualization various shapes using unnormalized decoder head with vanilla temperature sampling. Note that these results occur regardless of xy-scales, which are omitted for brevity.
In Table 2 we display the negative log-likelihood (NLL) on collection of representative real-world datasets from the UCI regression repository (Dua & Graff 2017) (full results over 25 datasets in Appendix A.5). We see that MDN head performance has high variability, at times able to perform the best but also extremely poorly depending on the task. Mcanwhile both decoding heads remain reliable overall (NLL<0.7 always). In comparison, the Riemann head consistently underperforms in every task.
Dataset
MDN
UD
ND
R
Airfoil
0.12 0.11
0.40 1 0.01
0.34 1 0.01
1.33 1 0.14
Bike
4.59 0.86
0.12 1 0.00
0.10 1 0.01
0.36 1 0.05
Elevators
0.30 0.43
0.15 0.00
0.13 1 0.00
1.12 1 0.02
Gas
0.68 0.25
0.02
0.02 0.00
1 1.56 + 0.21
Housing
0.22 = 0.13
0.41 + 0.03
0.38 + 0.03
Kin 40K
7.49 0.73
0.19 +119+0 0.01
0.12 + 0.01
0.39 + 0.03
Pol
1.49 0.41
0.01 0.00
0.01 0.00
0.18 + 0.02
Protein
1.07 0.44
0.34 0.00
0.41 0.01
1.55 + 0.04
Pumadyn32um
0.69 1.26
0.55 1 0.00
0.58 + 0.02
2.32 4 0.03
Wine
0.05 0.12
0.24 0.01
0.21 0.01
1.67 + 0.14
Yacht
0.21 0.10
0.39 0.02
0.23 0.05
+ 0.38
Table 2: Lower (1) is better.
Avg. NLL (1 StdDev) of test examples UCI datasets over 10 train-test, splits. Abbreviations: (UD, ND) = (unnormalized, normalized) decoder heads respectively; R Riemann.
10

---

Published in Transactions on Machine Learning Research (08/2025)
4.4 Ablation: Role of Decoding Head Size
We ablate the effect of the decoding head's size performance. We first fix the lokenization for the normalized decoding head (B=10, K=4) and then sweep the number of layers, heads, and hidden units. In Figure g we observe that larger decoding heads do sometimes help, but only up to certain point, at which overfitting can occur. This was also observed over regression over BBOB functions and with the unnormalized decoding head, but we omitted these results for brevity.
Bike
Energy
Housing
1.2
1.5
1.7
0.8
1.4
2
3
Num. Layers
1.00
1.85
1.80
1.50
NLL
0.95
1.75
1.45
0.90
1.70
1.40
2
2
Num. Heads
0.9
0.8
1.3
1.7
0.7
500
1000
500
1000
500
1000
Num. Units
Figure 9: Lower (4) is better. NLT over UCT datasets, when varying different axis (layers, heads, units) from a fixed defanlt of (3, 4, 128) respectively.
4.5 Ablation: Error Correction
One can also improve regression behavior using techniques purely by modifying sequence representations. Inspired by the field of coding theory, we can use error correction, where we may simply have the decoding head repeat its output multiple times (fI,..., tk ty, &gg th .)....) during training, and at inference perform majority voting on each location {1,...,KK
Task ID: 233213
Task ID: 359934
Task ID: 359944
Task ID: 359945
Laxage
Relative MSE
10-2
10-1
10a
10a
10-1
10-3
19x12
10-2
12:39
10-3
Lixae
10
104
103
10a
102
10
10a
10a
10
101
100
10
Sampling Size
Repeat Count
2
3
Figure 10: Lower (4) is better. Relative MSE selected AMLB tasks, when varying output repetitions.
In Figure I0 we focus on the unnormalized case when using mean aggregation, where performance can be significantly harmed from extreme outliers. We that when using regular tokenization (repeat count=1),
11

---

Published in Transactions on Machine Learning Research (08/2025)
more samples drawn, the likelihood of drawing outliers increases the error. However, the error be substantially decreased by training the decoding head to decode the same tokens repeatedly and allow better scaling with samples, although repeating too many times may make learning more difficult. Not all error correction techniques improve results however in Appendix A.4, we briefly observe negative results applying other lypes of error correction, and we leave exploring the space of such methods for future work.
5 Discussion: Limitations and Extensions
This work establishes the validity of training decoding heads with cross-entropy losses for regression. To minimize confounding factors, our designs remained very simple (e.g. using basic primitives such as softmax and vanilla attention mechanisms) yet principled (e.g. digit-by-digit tokenization and constrained sampling). We list some limitations of this work, along with more potential areas for exploration.
Many modern LLM architectures no longer use vanilla attention mech- anisms, instead opting for sparsity or low-rank approximations. In addition, MLPs have typically been replaced by mixtures of experts for memory reduction. It is worth studying further how these changes affect the performance of mumeric decoding.
Modern LLM Architectures:
Tokenization and Sampling: Our digit-by-digit tokenization and constrained decoding can be considered ideal cases, which modern LLMs only approximately implement. The main sources of differences include vocabularics which do not use individual digit tokens, and sampling procedures which are Top-P Top-K, which may accidentally choose invalid tokens. We hypothesize that these issues can decrease regression performance.
Intermediate Language and Training: In many LLM use-cases involving numeric prediction problems, the decoder not only must. return a tokenized number, but also other intermediate language tokens. Our work does not address these cases, and it unclear how fine-tuning on these intermediate language tokens may affect training dynamics and numeric prediction performance. Furthermore, modern LLM training consists of reinforcement learning (RL), and it remains to be studied how reward-based training methods affect regression results.
Multi-objective Regression: In this paper, we only studied the single-objective regression case. However, one may easily modify our paradigm to support multiple objectives , ((x)), e.g. by decoding a concatenated sequence of those objectives. This has the particular benefit of modeling objectives autore- gressively, which can be difficult to perform with classical techniques. The benefit of multi-objective density imatiioo is even more pronounced, due to the decoder's universal approximation abilities.
Other Regression Architectures: We did not compare to significantly more complex methods such stochastic networks, which use stochastic activations or weights. Early examples include Sigmoid Belief Nets Tang & Salakhutdinov 2013), which have not seen wide adoption due to their complex archi- tectures and expetattion-maxxiizattoo maximization training updates. Bayesian neural networks (Lampinen & Vehtari, 2001
Neal 1992
Titterington 2004 Goan & Fookes 2020) can be seen as more modern stochastic networks, but still possess complex inference techniques, e.g. Markov Chain Monte Carlo (MCMC) or variational inference, Similarly, Energy-based models (Teh et al. 2003) be used for image-based regression (Gustafsson et al.
2020
Liu et al. 2022) but still see limited use due MCMC required at inference time.
6 Conclusion and Future Work
We thoroughly investigated the many benefits but also drawbacks of using recoding-based regression. We described a natural tokenization scheme for both normalized and unnormalized y-values, and theoretically established its risk minimization properties. Empirically, we showed that it can be competitive as, or even outperform traditional pointwise heads for regression lasks. Furthermore, it is also capable of densily estimation over a variety of conditional distributions p(yl ((x)), and can further outperform common baseline regression heads such as Gaussian mixtures and histogram distributions. We hope this work will also be a valuable reference for the language modeling community and that it provides principled explanation for the use of supervised fine-tuning over numeric targets.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgements
We would like to thank Yutian Chen for his valuable review of the work and Bangding (Jeffrey) Yang for technical help. We further thank Yash Akhauri, Aviral Kumar, Bryan Lewandowski, Michal Lukasik, Sagi Perel, David Smalling, and Subhashini Venugopalan for useful discussions, and Daniel Golovin and Denny Zhou for continuing support.
References
Yash Akhauri, Bryan Lewandowski, Cheng- Lin, Adrian N. Reyes, Grant C. Forbes, Arissa Wongpanich, Bangding Yang, Mohamed S. Abdelfattah, Sagi Percl, and Xingyou Song. Performance prediction for large systems via lext-lo-text regression, 2025.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatffeld-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AI feedback. CoRR, abs/2212.00005 2022. doi: 10.48550/ AR..V. WTRRXVV.222..00. 2210008773.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 94th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 419.458. PMLR, 2017.
Christopher M. Bishop. Mixture density networks. Technical report, Aston University, 1994.
Ralph Allan Bradley and Milton F. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952. ISSN 00063444, 14643510.
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognit. Lett., 140:325-331, 2020. doi: 10.1016/..PATRES 2020.11.000
François Charton. Linear algebra with transformors. Trans. Mach. Learn. Res., 2022, 2022.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Mare' Aurclio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Cheng-Han Chiang, Hung-yi Lee, and Michal Lukasik. TRACT: regression-aware finc-tuning meets chain- 043881 2025. doi: 10.48350/ARXIT 250..04481.
of of-thought reasoning for juudae-a-judge CoRR,
Stéphanc d' Ascoli, Pierro-Aexandrr Kamienny, Guillaume Lample, and François Charton. Doep symbolic 2022.
regression for recurrent sequences. CoRR,
Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In IEEE Conference on Computer Vision and Pattern Recognétion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 4738 4747. Computer Vision Foundation / IEFF, 2019. doi: 10.
Dhceru Dua and Casey Graff. UCI machine learning repository, 2017. URL
ics www.iii....cii.... edu/
13

---

Published in Transactions on Machine Learning Research (08/2025)
Ouassim Elhara, Konstantinos Varelas, Due Nguyen, Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. Coco: the large scale black-box optimization benchmarking (bbob-largescale) test suite. arXiv preprint Xr::::1003.0000 06396, 2019.
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Araeual Meeling of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 889 898. Association for Computational Linguisties, 2018. doi: 10. 10.18653 /0153//V/P88/// 1082.
Scbastian Felix Fischer, Liana Harutyunyan Matthias Feurer, and Bernd Bischl. OpenMI-CTR23 curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023.
Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an automl benchmark. Mach. Learn. Res., 25:101:1-101:66, 2024.
Ethan Goan and Clinton Fookes. Bayesion Neunal Networks: An Introduction and Survey. Springer Inter- national Publishing, Cham, 2020. ISBN 978-3-030-12553-1. doi: 10.1007/978-33333-495533 3.
Alex Graves. Sequence transduction with recurrent neural networks. CoRR, Abbs/1211.37/1 2012.
Fredrik K. Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B. Schön. Energy-based models for deep probabilistic regression. In Andrea Vedaldi, Horst. Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, volume 12365 of Lecture Notes in Computer Science, pp. 323-343. Springer, 2020. doi: 10.1007/978-23 17778-3-030-5888 1180055565--). 20.
Frank E. Harrell and C. E. Davis. A new distribution-Fie quantile estimator. Biometrika, 69(3):635 640, 1982. ISSN 00063444, 14643510.
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular founda- tion model. Nat., (8044))3333 -266, 2025. doi: 10. 10.1038/541130--044-03232--
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
IEEE. Teee standard for floating-point arithmetic. TEEE Std 754-2019 (Revision of IEEE 754-2008), pp. 1 84, 2019. doi: 10.1109 10.1109/IEEESTI MIE//EEESTD..C 2019.8766229.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässon, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- chine Learning Research, pp. 2162-212111111 2171. PMLR, 2018
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (ods.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Jouko Lampinen and Aki Vehtari. Bayesian approach for neural networks networks-review and case studies. Neural Networks, 14(3):257-274, 2001. doi: 10.1016, (0000//50003-0000000000000000000
L.E. Lehmann. Theory of Point Estimation. A Wiley publication in mathematical statistics. Wiley, 1983.
Xixi Liu, Che-Tsung Lin, and Christopher Zach. Energy-basod models for deep probabilistic regression. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2693-22 2699,, 2022. doi: 10.1109/ 0PPR56361.2022.9055
14

---

Published in Transactions on Machine Learning Research (08/2025)
Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, and Sanjiv Kumar. Regression aware inference with llms. In Yaser Al-Onaizan, Mohit. Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 13667-13678. Association for Computational Linguistics, 2024.
Michal Lukasik, Zhao Meng, Harikrishua. Narasimhan, Aditya Krishna Menon, Yin Wen Chang, Felix X. Yu, and Sanjiv Kumar. Better autoregressive regression with LLMs. In The Thirteenth International Confer- ence on Learning Representations, ICLR 2025, Singapore, Singapore, April 24-28, 2025. OpenReview.ner net, 2025.
Dakota Mahan, Duy Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. CoRR, abs/2410.122 128322 2024. doi: 00.48550/ARXII 1280.
Radford M. Neal. Connectionist learning of belief networks. Artif. Intell., 56(1):71 113, 1992. doi: 10.1016/
0004-3770(022)90065-6
Tung Nguyen, Qiuyi Zhang, Bangding Yang, Chansoo Lee, Jorg Bornschein, Yingjie Miao, Sagi Perel, Yutian Chen, and Xingyou Song. Predicting from strings: Language model embeddings for bayesian optimization. CoRR, 0bbs22410.100 2024. doi: 10.48550/ARAA
1019.0
Rodrigo Frassetto Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of the transformers with simple arithmetic tasks. CoRR, abs/ abs/2102.13019, 2021.
Emanuel Parzen. On Estimation of a Probability Density Function and Mode. The Annals of Mathematical Statistics, 33(3):1065 1076, 1962. doi: 10. 111.21///0oms1177771..
Minghai Qin.
binary representation of numbers. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 2202-2205, 2018. doi: 10.1109/ISIT 8437644
Carl Edward Rasmussen. The infinite gaussian mixture model. In Sara A. Solla, Todd K. Leen, and Klaus- Robert Müller (eds.), Advances in Neural Information Processing Systems 12, (NIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999], pp. 554-560. The MIT Press, 1999.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of Density Function. The Annals of Mathematical Statistics, 27(3):832 837, 1956. doi:
Xingyou Song, Oscar Li, Chansoo Lec, Bangding Yang, Daiyi Peng, Sagi Perel, and Yutian Chen. Omnipred: Language models as universal regressors. CoRR, Abs/2402.145417 2024.
Eric Tang, Bangding Yang, and Xingyou Song. Understanding LLM embeddings for regression. CoRR, abs/2411.1470% 2024. doi: 10.48550/ 2411...0.....
Yichuan Tang and Ruslan Salakhutdinov. Learning stochastic feedforward neural networks. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Newral Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 530 538, 2013.
Yee Whye Tch, Max Welling, Simon Osindero, and Geoffrey E. Hinton. Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res., 4:1235-1260 2003.
D. M. Titterington. Bayesian Methods for Neural Networks and Related Models. Statistical Science, 19(1): 128- 139, 2004. doi: 10.221/00833223400000099
Robert Vacarcanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdcanu. From words to numbers: Your large language model is secretly A capable regressor when given in-context examples. CoRR, abs/2404.07554, 2024.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. SIGKDD Explor., 15(2):49- 60, 2013. doi: 10.1145/2644 09044199.22211990
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Mllia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (cds.), Advonces in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017.
Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysatof Dembozynski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In Samy Ben- gio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 6358 6368, 2018.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction. CoRR, abs/2408.15240, 2024. doi: 10.48550/AMA
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm- as-a-judge with mt-bench, and chatbot arena. In Advances in Neural Information Processing Systems 36; Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
Danicl M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.0855 2019.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
A Additional Experiments
A.1 Data Scaling: Extended
For completeness, we display the plots over all tasks in AMLB Gijsserrs et al., 2024 We confirm the data-efficiency of the decoder head against the Riemann distribution head nearly every regression task. Furthermore, observe numerous cases where both distributional methods outperform the pointwise head, especially in low data regimes.
Task ID; 359944
Task ID: 233212
Task ID: 359937
Task ID: 359950
Task ID: 359938
Task ID: 233213
Lavar
10
10
140304
10-2
10:3
10-2
10-2
Task ID; 359942
Task ID; 233211
Task ID: 359936
Task ID: 359952
Task ID: 359951
Task ID: 359949
6-10-1
Relative Mean Squared Error
16-4
10
10
10-2
10-2
10-1
10:
10"
10"
10
10'
10'
10
107
30'
10'
10'
10'
10'
10'
10"
10
107
10"
233215
359930
10-1
1
10-
10:4
10
10-3
10a
10
10a
10'
10'
10a
107
30"
10"
10'
10'
16'
10'
10'
10
10'
10'
10
Task
233214
Task 359948
Task ID: 359931
Task 359932
Task ID: 359933
Task ID: 359934
10
10-2
1=30
10
5x30
107
10a
10a
10'
10'
10a
30'
30'
10"
10'
3.0
10'
107
1.0
30
107
10
10
Task ID: 359939
Task ID: 359945
Task ID: 359935
Task ID: 317614
<x30>
1.30
Training Data Size
Normalized Decoder
Riemann
Pointwise
Figure 11: Lower (L) is better. Regression performance as function of training data scaling between using the normalized decoder Reimannian distribution regression heads. Each point was averaged over 10 training runs over random combinations of datapoints from the original AMLB task's training set.
17

---

Published in Transactions on Machine Learning Research (08/2025)
A.2 BBOB Curve Fitting: Extended
In Figure 12 compare the curve fitting properlies of multiple regression heads. We see overall that the decoder head is competitive and has both pros and cons for specific function landscapes from the BBOB benchmark.
Sphere
Rastrigin
BuecheRastrigin
LinearSiope
AttractiveSector
StepElllpsoidal
1.00
0.995
1.00
0.99
0.990
0.998
0.55
0.99
3.98
0.985
0.90
0.400
0.57
0.64
0.975
0.96
0.92
0.570
15
RosenbrockRotated
Ellipsoidal
Discus
BentCigar
GriewankRosenbrock
SharpRidge
1.000
Kendall-Tal Correlation
3.98
0.998
0.998
1.00
0.98
0.43
0.500
0.995
0.96
0.98
0.65
0.975
0.985
0.900
0.990
0.992
0.95
0.94
0.89
0.90
0.900
0.986
10
15
Difees
Weierstrass
SchaffersF7
Schwefel
Katsuura
1.00
0.74
0.76
0.74
3.99
0.70
0.72
0.10
0.98
0.6
0.70
0.05
0.4
0.68
0.66
0.68
0.375
0.00
0.62
0.64
0.665
-0.05
10
10
15
Lunacek
Gallagher101Me
Gallagher21Me
NegativeSphere
3.00
1.00
0.98
0.99
0.96
0.96
0.90
0.2
0.90
0.96
10
15
10
15
15
20
10
15
20
Dimension (x)
Normalized Decoder
Unnormalized Decoder
Riemann
Pointwise
Higher (t) is better. Extended results from Table 1 in the main body. Regression performance as a function of input dimension over BBOB functions using Kendall-Tau correlation. Fach point was averaged over training runs, each with 100K training points (x, where each æ is sampled uniformly from [-5, coordinato-wise. Note: Some functions such as RosenbrockRotatod or GriowankRRsenbrook are undefined when dimension 1, skip those points.
Figure 12:
18

---

Published in Transactions on Machine Learning Research (08/2025)
A.3 Individual OpenML Kendall- Taus
In Figure 13 we present full results on the AMLB and OpenMI-CTR223 regression benchmarks, over all regression heads. We see that both the unnormalized and normalized decoder heads remain competitive throughout the benchmarks.
AMLB
Kendall-Tau Corelation
0.8
0.6
0.2
0.0
16
26
Task Index
OpenML-CTR23
Kendall-Tau Correlation
0.6
0.4
0.0
11
16
21
26
Task Index
Unnomalized Decoder
Noralizzd Decoder
Riemann
Pointwise
Figure 13: Higher (t) is better. Extended results from Figure 5 Task IDs sorted by pointwise head performance.
but for all four regression heads on all tasks.
19

---

Published in Transactions on Machine Learning Research (08/202))
A.4 Alternative Tokenization Schemes: Hamming- Distance
Caaming--Distace
One possible crilicism of the default tree-based lokenization in the normalized decoding case, is the vul- nerability to small changes in the left-most significant tokens, which can cause large numeric changes in the actual number. Qin (2018) notes this and proposes an altornative "Hamming Distance-based" binary representation which is robust to bitwise edits, and upper bounds the possible distortion -/ as a function of the edit distance between the Hamming representations of y' and y. For example, if the binary length is the representation for all integers 10, 2 is { (000), (001), (010), (100), (011), (101), (111),(101), (110), (111)} which can also be used in the normalized case for {0/23, 1/2², ..... 7/22) [0, 1]. In Figure 14 we show however, such a representation may nol lead to better regression results, which we hypothesiie is due to this reprosentation being more difficult to learn.
Task [D: 233212
Task ID: 359950
Task ID: 233213
30
20-
Relative Mean Squared Error
2x
10
10
10a
102
101
10a
10a
10a
Task ID: 359951
Task ID: 359949
ID: 233215
2x
6xx00
1
10-
1x
10
3x10-
2x
122
10a
10a
10a
10a
10a
Task ID: 167210
Task ID: 359932
Task ID: 359932
10-2
10
10-2
60
10-
3 3x10-
10-1
10-3
2
102
102
10a
Sampling Size
Tree
Hamming
Figure 14: Lower (L) is better. Regression performance while vary sampling size from y po(- (x) using binary tree-based lokenization Hamming representation normalized decoder with mean aggregation. Each point was averaged over 10 training runs over random size-1000 combinations of the original AMLB task's training data points.
20

---

Published in Transactions on Machine Learning Research (08/2025)
A.5 Full UCI Density Estimation Results
We obtained the datasets from httpp::/ggithh http:///ghhaaacccccccceeeeeeeeaa cttp:/reeeaaaanccccccaaatttttta datasets which preprocessed and scraped the data from the official UCI website at lnttss:/arrciiee.io. icl.ici............ edu/
Dataset
Mixture Density Network
Unnormalized Decoder
Normalized Decoder
Riemann
Airfoil
0.12 0.11
0.40 1: 0.01
0.34 : 0.01
1.33 0.14
AutoMPG
0.21 i 0.07
0.32 1 0.03
0.41 i 0.05
1.62 i 0.17
Autos
0.32 i 0.23
0.48 1 0.05
0.47 = 0.07
2.60 1: 0.76
Bike
4.59 i 0.86
0.12 + 0.00
0.10 = 0.01
0.36 +: 0.05
Breast
0.09
0,48 +
=
2.85
Challenger
-0.29 0.66
0.14 + 0.14
0.06 = 0.08
0.87 + 0.77
Concrete
0.15 + 0.05
0.43 + 0.03
0.41 + 0.04
1.67 + 0.20
Elevators
0.30 + 0.43
0.15 + 0.00
0.13 + 0.00
1.12 + 0.02
Energy
0.40 0.14
0.17 +
0.16 0.05
0.38
Pertility
0.06 0.16
0.31 + 0.09
0.46 + 0.13
2.41 + 0.61
Gas
0.68 + 0.25
0.02 + 0.01
0.02 + 0.00
0.20 + 0.09
Housing
0.22 + 0.13
0.41 + 0.03
0.38 + 0.03
1.56 + 0.21
KeggDirected
2.41
0.05 0.00
0.05 0.00
0.22
Kin 40K
7.49 + 0.73
0.19 + 0.01
0.12 = 0.01
0.39 + 0.03
Parkinsoms
0.59 + 0.18
0.40 = 0.02
0.39 = 0.03
1.40 = 0.33
Pol
1.49 + 0.41
0.01 + 0.00
0.01 = 0.00
0.18 = 0.02
Protein
1.07 + 0.44
0.34 = 0.00
0.41 0.01
1.55 = 0.04
Pumadyn32nm
0.69 +
0.55 = 0.00
0.58 = 0.02
2.32 = 0.03
Slice
7.09 + 0.09
0.05 + 0.00
0.02 = 0.00
0.08 = 0.02
SML
1.31 + 0.59
0.21 = 0.01
0.11 = 0.01
0.35 + 0.03
Solar
-1.40 0.29
0.04 + 0.01
0.04 0.01
0.61 0.12
Stock
-0.15 0.15
0.27 + 0.04
0.32 = 0.04
1.63 + 0.46
Tlemiilcctric
0.01 + 0.00
0.46 + 0.00
0.69 = 0.00
2.70 + 0.00
Wine
0.05 + 0.12
0.24 + 0.01
0.21 = 0.01
1.67 + 0.14
Yacht
0.21 + 0.10
0.39 + 0.02
0.23 0.05
1.29 + 0.38
Table 3: Lower (L) is better. Avg. NLL (# StdDev) of Lest examples on UCI datasets over 10 train-test
splits.
21

---

Published in Transactions on Machine Learning Research (08/2025)
A.6 Density Estimation Visualization: Extended
In Figure 15 present further results density estimation with various decoder sampling techniques (top- k top-p, low temperature) alongside MDN and Riemann baselines. We see that using vanilla temperature sampling for the decoder is optimal and unbiased for capturing the shapes of all problems.
Half Moons
Zig-Zag
Spiral
Hollow Square
1007 Cennnne 2000
Decoder (Vanilla)
Decoder (Temp=0.1)
Decoder P=0.9)
Decoder
Riemann (Bins (Bins=100p 1000)
Sample
Ground Truth
Figure 15: Visualizing density estimation of p(yla:) on 1D problems. We used an unnormalized decoder with
(B 10, E M, 5). Note that these results occur regardless of xy-scales, which omitted for brevity.
22

---

Published in Transactions on Machine Learning Research (08/2025)
B
Extended Theory
Proof of Theorem Firstly, we observe that
argmin 1
Σ n=1
log po (Ax((y)) argmin H(N 2K .p).
where J& a discrete distribution that tracks the fraction of samples (Yr) J. that fall within each of the 2 unfoormly-spaced bins in [0, 1]. Formally,
(e N
bn))
Σ (()x(Tn) = (b),...,bb)).
Conditioned on the samples (Y,) f& is a distribution on k-bit strings, and so by the K-bit universality assumption, p = f It follows that pl- 8& vk - K since if two discrete distributions are equal so are any of their marginals. Then f (I) = 2* (Au(zr)) = 24 83 (Ax(z)) lines up exactly as a 2 histogram estimator for f. for all k K.
Now, we treat the problem as one of histogram estimation. Let's consider fixed k. We first observe that the risk can be written the sum of a squared bias term and a variance one. Specifically,
(ias()) + L' Variance(y)dy.
= where BI (v)) - ((y) and Variance(y) VV (v)) is the bias and variance of (x(( at fixed y respectively.
Now, label bins 1B) where B, [je, (j + 1)=) and = 2- is the bin width. Let P5 = SB, f(z)dz be the true probability mass in bin B,. With Nj as the number of samples in B,, the expected value of the estimator for y B, is E[fi' (v) = E|N,, (Ne)] (NN))/(Ne)) = Py/E.
Assume the true density f is twice continuously differentiable on [0, 1] (i.e., I E C²O 1])). This implies f. f', and f" are bounded on [0, 1]. Let M1 = su [f'(v) and M2 = suPye[o,1] (x)]
Bias Analysis: Let yu = (j + 1/2)= be the midpoint of bin By. For 2 E By, by Taylor's Theorem around Uj: f(2) f(y,) + (z (1,) + (a (ي= (E{()) for some E. between 2 and y/, Integrating over B,:
Pj = I BB, ((z)dz / [1w + (== (==)))))(( (2- n1)² ('({)) dz
2
=((1) / de + (w,) / B,
(z (2- v,)d2 /
(= y;) (S=)ddz
B,
8, 2
= =es(T;) +0+ Rg.
where the remainder term Rg = JB, (*-1,)² f" (E=))d.. Since 12 - Vyl s e/2 and [f"(())) I M2, we have
(s/2)² RRyl $ SB. 2 Madz Mas² SB, dz Mas" Thus, RN = O
The bias for E Bj is Bias(y) = Elsk' (v)) - f(y) = P: f(y) = Ef(,)) - f(y) = f(U)) - R - f(v)
Expanding f(y) around yy: f(y) = (w/) + (y - ('))) + (w-w,)² f (mw) for ny between y and y/-
+ Bias(y) = f(u)) + - [f(,;) + +
-yy-w,)f(()) + O
23

---

Published in Transactions on Machine Learning Research (08/2025)
Now, integrate the squared bias over bin By:
/ Bias(y) = / -yy-vı (-)-)))))))++()))))
B,
= -f1, B,
-
2(y
dy
(y (U- ujjdy + / O Bi
-('(w)))2
(y- -
F/2
(f'())))²
0+ du 0(e 00 0 0+(6))
(let y y- Vj)
W3/E/2 5/2
(
-(())
3
-2/2
3
(1'(w))) Oee)).
12
Summing over all bins:
L Bias(y)
-
(f()))) 53
Σ
Bias(y)
Σ
+ +
12
5=0
B,
5=0
2
2²
Σ
(f(()))
Σ 0
12
-=0
1=0
2
( ((())))++0(()))
+ 2(5)
12
2
[
12
where the third line uses known approximation error for the Riemann sum with midpoint rule applied to (S')² C
Variance Analysis: The variance for By is Variance(y) = (() = V(N5/(Ne)) = [Ney= V(N,). Since N2 Binomial(N.p,) V(N,) (NN,))>>p((111111 (1-p)).
1
Np,(1 Pí) p3(1 Pp)
Variance(y)
N2g²
N=2
(Ef(y)) +0(s)))(1 -
N
=f(ug) e² f(u))² + O
No2
f(yb)
f(y))²
+ O(e/N).
Ne
N
24

---

Published in Transactions on Machine Learning Research (08/2025)
Integrating the variance:
Variance(y)dy = Σ J
[(b))
+ 0(1/N)) dy
Ne
5= B,
2
Σ
f(u))E
+ +0(G/M)
Ne
1=0
1
Σ MWs)ee 2*e(N)))
Ne
1
(f, f(y)dy + f(a)dy+ +0(2)) O(1/N) (Riemann error for C
Ne
1
(1 +0(1/N)
NE
1
Ne
Since we typically consider asymptotics where N - oc and 0 such that Ne oo, the dominant variance term is 1/e)...
Total Risk: Combining the integrated squared bias and integrated variance:
Bias(y) + L. Variance(y)dy
R(J.N))
22
L ((()))dy++(e+) + (NE Ne
O(e/N)+
12
/ (f(())) + Ne +0(55) +(L/NN) (assuming E/N is smaller than L/N)
12
Substituting 8 = k-
2a N 2-2A 2h [ (f" 00 1/N).
R(S
12
This gives the asymptotic risk. The -2h-4h 4k 1/N) term negligible, and can be disregarded.
25

---

Published in Transactions on Machine Learning Research (08/2025)
C Exact Experimental Details
For all models, sweeped the encoder (basic MLP with ReLU activation) by varying the number of layers within [2,3,4,5] and hidden units within [256, 512, 2048).
For z-normalization, we apply a mean and std scaling, i.e. a +- (x mmeon) / where Amean: 3std are coordinate-wise mean and standard deviations over all z's the training set. The preprocessod tensor is then fed directly into the encoder.
For y-normalization, we apply min/max linear scaling, i.c. y (v (u-Byinn)(ymazz Vvmin) where Vinin; Vnna computed from the training set. This is applicable to models representing [0, 1] output range (i.e. Riemann and Normalized Decoder). For Pointwise and Mixture Density heads, we further apply a shift y + y - 0.5 to center the values within [-0.5, 0.5).
All models were trained with a maximum of 300 epochs. To prevent overfitting, we apply early stopping (patience=5) where the validation split is 0.1 on the training set. Adam learning rates were sweeped over [1-4, 5e-4).
We further describe hyperparameters and sweeps for individual heads below:
Pointwise: Uses ReLU activations on every hidden layer.
Weight decay: [0.0, 0.1, 1.0]
Unnormalized Decoder: Uses vanilla temperature sampling.
Base B: [4, 8, 10]
Exponent Digit Count E: [1, 4]
Mantissa Digit Count M: [2, 4, 8)
Transformer size: (3 layers, 128 units, 4 heads) or (1 layer, 32 units, head).
Normalized Decoder: Sampling same as unnormalized decoder.
Basc B: [2, 4, 8]
Length K: [4, 8, 6]
Transformer size: Same as unnormalized decoder.
Riemann/ HHisoggram Distribution: We specify a bin count, which uniformly partitions the range [0, 1] into equally spaced bins. Output is parameterized using softmax.
Bin Count: [16, 64, 256, 1024, 4096, 16384]
Mixture Density Network: Given a mixture count M, the distribution head consists of mixture πM E AM mean μм MR,, and standard deviation σы RM Mixtures were parameterized using softmax, while standard deviations were via ELU(x) + activation to enforce positivity.
Mixtures M: [1, 2, 5, 10, 20, 50, 1000]
26


Reviews:
[{"review_id": "8jUcRr0CL9", "reviewer": "Reviewer_tEwp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper studies an interesting problem: the application of autoregressive models to regression tasks (predicting a single real value). It provides valuable insights into a particular element of the regression pipeline: the final layer of the architecture and the tokenization scheme. Several interesting experiments on both real and synthetic data illustrate the methods considered. The theoretical contribution appears to be correct.\n\n**Weaknesses**\n\nDespite the nice contributions of the paper, its presentation is flawed. Several parts of the paper use imprecise language, making it difficult to evaluate the correctness of some claims. See examples in the Requested Changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the performance of autoregressive models on regression tasks, with a particular focus on different choices for the final layer (\"head\") of the network. It provides theoretical justification for using decoder-based heads in this setting, including a result that bounds the error of decoder-based regression. Finally, experiments demonstrate that standard (\"pointwise\") heads achieve performance comparable to that of decoder-based heads.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In my view, the paper requires moderate to substantial rewriting before it can be published. In particular, I identified the following areas where the presentation can be improved:\n\n- Abstract: The terms \"prediction heads\" and \"decoder-based heads\" are not standard. While they are defined later on page 2, their use in the abstract may confuse readers. Moreover, the word \"head\" is commonly used to refer to attention heads, so it would be helpful to clearly distinguish the two early in the text.\n- The Introduction should more clearly state the contributions of the paper. As it stands, it is unclear whether the paper is primarily theoretical or empirical, and what its main strengths are.\n- pg. 2: The sentence \"However, we argue that this approach ... works.\" is unclear.\n- Section 3.2: The statement \"If y is restricted to [0, 1] (via scale normalization for example), then we show we can...\" should specify where this result is shown.\n- Figure 5, caption: Phrases like \"Fit to ground truth is better\" and \"practically unlimited $(x, y)$ points\" should be replaced with precise, formal statements.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzFEtVtL1r", "reviewer": "Reviewer_hcDX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes simple and well-principled approaches, which are supported by some theoretical foundation, to use token-based autoregressive decoders for regression tasks.\n- Results show that such approaches are competitive (and sometimes superior) to traditional baselines. The figures also convincingly show the effectiveness of the proposed solutions.\n\n**Weaknesses**\n\n- While the paper is motivated via language modeling capabilities, the decoder heads do not really model language anymore due to the choice of tokenization schema. I would be curious to know if the approach could be extended to support regression and language modeling jointly, either by reserving a set of tokens for regression, or re-using some tokens for both tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the use of token-based autoregressive decoders for regression or density estimation tasks. The authors propose two tokenization schemes, one for normalized data and one for unnormalized values. In the first case, the model predicts a length-K sequence of base-B tokens $t_{1..k}$, approximately corresponding to a value of $\\sum_{i=K}B^{-i}t_i$. For the second case, the tokenization scheme generalizes floating point approximations, where the model predicts the sign, mantissa and exponent. Experimental results show that such token-based auto-regressive decoder heads perform similarly or better than pointwise scalar regression heads.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Would strengthen the work**\n\n- Page 3, section 3.2. You mention that \"Since the token space is finite while $\\mathbb{R}$ is uncountable, this mapping is lossy\". The mapping would also be lossy for countably infinite sets.\n- Page 3, section 3.2. You mention \"Any autoregressive model can be used, so long as it supports constrained token decoding to\nenforce proper sequences which represent a valid number.\". If you disable constrained decoding, how often does the model generate a non-valid output?\n- Page 11, section 4.5. How does predicting the output multiple times sequentially compare to predicting it in parallel? The latter wouldn't require extending the training data.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eK5nRziynq", "reviewer": "Reviewer_zAuL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n- This work provides both, theoretical and empirical, support for its claims.\n- The experiments use a wide range of datasets, which suggests that their findings are universal and not constrained to the considered data.\n- The proof is a bit hard to read, but seems to be correct. However, I have not checked it thoroughly.\n\n# Weaknesses\nThe most severe weakness is clarity. I had trouble understanding the theoretical framework, the considered neural network architecture and the setup of the experiments. In consequence, I am unable to verify the soundness of the current version of this submission. \nSpecifically, the following points need more explanation.\n\n  - Key concepts of the considered architecture remain unclear. Presumably, the model architecture consists of three parts, an autoregressive model operating on the inputs, an encoder that tokenized the response variable and the head which produces the prediction of the response variable. This is not clearly specified though, so I am unsure if my description is accurate and precise enough. \n  Specifically, the key component of the network that is studied in this work, i.e., the head is not clearly specified, and its unclear to me, what exactly is meant by a regression head, a decoder head, a pointwise head and a Riemann head.\n  - Related to the previous point is that since I do not know what the different heads are, I also do not know what exactly is evaluated in the experiments. For example, I wonder if the tokenization of the response variable is necessary to achieve a good regression or if keeping is as a float leads to similar results. Maybe this is already studied in the experiments, but I cannot tell. \n  - The theorem statement is difficult to decipher.  \n    - From my understanding, it seems, that the theorem is actually a result about the mse of the maximum likelihood estimation of the first K bits of some random variable, and that everything model specific can be neglected due to the K-bit universality assumption. Is this correct?  \n    - What is the relation between $\\hat \\theta$ and $\\hat f_N$. In which part is it relevant, that $\\hat \\theta$ is a maximum likelihood estimator?\n    - Later on, it is stated that *\"We can decompose this [K-bit universality] assumption further into two pieces: 1) that there exists θ∗ in our model class that achieves the minimum cross-entropy (i.e. pθ∗ = p in Definition 1), and 2) that our SGD-based training procedure is able to find it.\"* However, it seems that K-bit universality only concerns part 1).\n    - More generally speaking, what is the interpretation of the Theorem and why is it relevant to decoding-based regression?\n  \n  - In the preliminaries, the tokenization is described for arbitrary base B, but the theory is limited to base 2 (bits). I therefore wonder whether the choice of base 2 is for simplicity only, or if it is necessary.\n  - At the start of section 4 (Experiments) it is written that *\"This minimizes the decoding head’s contribution to representation learning (as opposed to numeric modeling), since it will be relatively negligible in size to ϕ(x), i.e. less than 10% of the total network parameter count.\"*\n  What do you mean with numeric modeling here and why do we want to minimize the decoders contribution?\n\nRegarding the experiments.\n- Comparisons with baselines are missing. This includes a comparison to standard baselines such as tree based and kernel methods, as well as neural networks other than transformers. \n- In figure 2, what is meant with 'theoretical', 'decoder', 'Riemann'. How are the risk values obtained?\n- Figure 6 is missing labels.\n- What is meant with 'vanilla temperature sampling'?\n\nIn the abstract, the following claim is made *\"We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoding-based regression is as performant as traditional approaches when benchmarked over standard regression tasks, while being flexible enough to capture arbitrary distributions, such as in the task of density estimation.\"*\nThis statement lacks sufficient support, as (i) there is no empirical comparison to *\"traditional approaches\"* and the theory only concerns specific distributions, i.e., targets are scalar and have a C2 density.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the capabilities of autoregressive models to perform regression tasks based on tokenized target variables. Specifically, this work provides an approximation bound between  distributions with twice continuously differentiable density functionsthe first K bits and their maximum likelihood K-bit approximation.\nMoreover, such models are empirically evaluated on regression and density estimation tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n**Clarity.** The formal framework and the experimental setting needs to be described clearly.\n\n**Soundness.** The empirical comparison needs to be extended such that it sufficiently supports the claim *\"decoding-based regression is as performant as traditional approaches when benchmarked over standard regression tasks.\"* Statements regarding fitting arbitrary distributions need to be made more precise.\n\n## Minor\nThe readability of Theorem 1 would improve if definitions are moved outside.\n\nA figure that illustrates the architecture and the relation between individual components (encoder, decoder, auto-regressive model / transformer, heads) would be appreciated.\n\nTypos:  \n\"**We** call **the** parametric model...  \n\"Define **the** risk R as the mean integrated squared error between **the** true ...\"  \n\"One can also improve regression behavior using techniques purely by modifying sequence representations.\"", "broader_impact_concerns": "I don't have concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)