Teaching Metric Distance to Discrete Autoregressive Language Models
Authors: Jiwan Chung
Saejin Kim
Yongrae Jo
Jaewoo Park
Dongjun Min
Youngjae Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Autoregressive Modeling
Generative Modeling
Efficient Training
Primary area: foundation or frontier models, including LLMs
TL;DR: DIST2Loss trains discrete models to respect token distances, boosting performance in diverse domains, especially with limited data.

Abstract: Large language models (LLMs) operate as autoregressive predictors over discrete token vocabularies, a formulation that has enabled their adaptation far beyond natural language to vision, robotics, and multimodal reasoning. However, training against one-hot targets disregards metric relationships between tokens and limits effectiveness on tasks where distance is meaningful, such as numerical values, spatial coordinates, or quantized embeddings. We introduce DIST2Loss, a distance-aware objective for discrete autoregressive models that replaces one-hot targets with reward-weighted distributions derived from predefined token distances. DIST2Loss can be interpreted as the closed-form solution to entropy-regularized policy optimization with known per-token rewards, retaining the core mechanism of reinforcement learning while avoiding sampling, rollouts, and instability. Our experiments show that DIST2Loss improves data efficiency and downstream performance across diverse domains. It yields tighter bounding boxes in visual grounding, accelerates robotic manipulation by improving action learning, enhances reward modeling for LLM alignment, and strengthens vector-quantized image generation. These results demonstrate that distance-aware supervision offers a simple and general alternative to one-hot supervision for discrete autoregressive models.

Published as a conference paper at ICLR 2026
TEACHING METRIC DISTANCE TO DISCRETE AUTOREGRESSIVE LANGUAGE MODELS
Jiwan Chung', Saejin Kim? Yongrae Jo Jaewoo Park', Dongjun Min', Youngjae Yu? Yonsei University 2LG Al Research "Souu National University j jiwan. gewaerrreaarcceeee.e.....
ABSTRACT
Large language models (LLMs) operate as autoregressive predictors over discrete token vocabularies, formulation that has enabled their adaptation far beyond natural language to vision, robotics, and multimodal reasoning. However, train- ing against one-hot targets disregards metric relationships between tokens and limits effectiveness on tasks where distance is meaningful, such numerical val- ues, spatial coordinates, or quantized embeddings. We introduce LosT a distance-ave objective for discrete autoregressive models that replaces one-hot targets with reward- weighted distributions derived from predefined token distances. D be interpreted as the closed-form solution to antropy-reeulariag policy optimization with known per-token rewards, retaining the core mechanism of reinforcement learning while avoiding sampling. rollouts, and instability. Our experiments show that DOST*Loss improves data efficiency and downstream perfor- mance across diverse domains. It yields tighter bounding boxes in visual grounding, accelerates robotic manipulation by improving action learning, enhances reward modeling for LLM alignment, and strengthens vector-quantized image generation. These results demonstrate that distance-awar supervision offers simple and general alternative to one-hot supervision for discrete autoregressive models.
INTRODUCTION
Large language models (LLMs) (Radford et al., 2018) have recently emerged as backbones for general-purpose foundational models across wide domains (Bommasani et al., 2021). These models rely on two probabilistic principles. First, they represent sample text sequence of tokens and train the model autoregressively. predicting each token conditioned on the previous ones. Second, each token is treated as discrete categorical variable, optimized to match one-hot target distribution.
While originally developed for natural language, LLMs are now widely adapted to tasks far beyond text. In vision, they have been coupled with discrete visual tokens for image generation and edit- ing (Esser et al., 2021; Dhariwal et al., 2020); in robotics, they are finetuned to handle control and planning tasks by treating actions trajectories as token sequences (Xiao al., 2024; Li al., 2024); and in multimodal reasoning, they are adapted to align visual, textual, and symbolic repre- sentations (Yu et al., 2024). These cases demonstrate the portability of the discrete autoregressive formulation beyond language.
key limitation of such adaptation is the inability to fully exploit numerically or metrically structured elements. These include explicit numbers, as well entities situated in broader metric spaces, such as integers in arithmetic reasoning (Yuan et al., 2023). spatial coordinates and rotation angles in object detection and manipulation (Xiao et al., 2024; Li et al., 2024), and high-dimensional quantized embeddings in image or video generation (Esser et al., 2021; Yu et al., 2024; Dhariwal et al., 2020). In conventional finetuning, the intrinsic distance structure among these elements is ignored, since tokens are reduced to one-hot categorical targets.
In this work, we introduce DIScreTized DISTance Loss Loss)"Loss), framework that integrates predefined distance relationships between tokens into the adaptation of autoregressive discrete mod- els. D requires additional data and incurs minimal computational overhead, enabling plug-and-play use across diverse setups. By encoding metric structure directly into the target distribu- tion, DosS accelerates performance improvement tasks where distances are semantically

---

Published as a conference paper at ICLR 2026
Function Regression
Reward Modeling
Robotic Manipulation
Visual Grounding
Text-to-Image
Q: What ys
@: (Instruction) (Response)
Q:
me...
Q: Detect the full zebra.
@: Draw cat.
(0.32, 0.53),
A: <IMGI0> <IMG32>
(0.85, 0.42),
A: Helpfulness: 4,
0.79
(0.79,
Safety: 1,
A: Move the arm to
Verbosity: 3,
<+ (0.2, 0.4) and rotate O 108 degree.
The full sebra is [(49, 10)(540, 465)).
A: (0.79, 0.47)
Complexity: 2
Tasks beyond language frequently involve outputs with inherent metric structure, such as quantities or coordinates, highlighting domains where distance modeling could be beneficial.
Figure I:
meaningful, including object detection (section 3.2), object manipulation (section 3.3), reward modeling (section 3.4), and image generation (section 3.5).
Conceptually, LoST can be viewed as the fonsed-form solution to entropy-regulariize policy optimization, providing a stable and efficient alternative to reinforcement learning. It constructs reward-weighted target distribution over the vocabulary and trains the model to match it through KI. divergence. This preserves the essential mechanism of reward alignment while avoiding the sampling, rollouts, and instability characteristic of traditional RL methods. Crucially, such rewards are only well defined when tokens admit meaningful metric, such as numerical values, coordinates, or quantized embeddings, so that distances can be translated into scalar quality signals. In domains without intrinsic geometry, the method reduces to one-hot supervision.
Our experiments demonstrate that DoST generalizes effectively across domains and improves downstream performance even in data-scarce settings. yields tighter bounding box predictions in visual grounding (section 3.2), accelerates the learning of robotic actions to increase success rates in manipulation tasks (section 3.3), improves reward modeling for I.LM alignment (section 3.4), and enhances the learning of rector-quantized image representations in autoregressive models (sec- tion 3.5). These results illustrate that distance-aware supervision can consistently strengthen discrete autoregressive models beyond one-hot next-token prediction.
2
METHOD
We aim to design an objective that (1) leverages the given metric to construct optimization targets, and (2) remains compatible with the categorical distributions used in LLM-based foundational models. We hypothesize that incorporating this metric prior improves data efficiency when distances meaningful. This section organized as follows: first, we review the conventional cross-entropy formulation; second, we introduce DIST" third, we interpret DIST' from reinforcement learning perspective; and finally, we extend the framework to high-dimensional metrics.
2.1 PRELIMINARIES
Notations. Let V denote the vocabulary the foundational model, and consider a subset Va C V with cardinality [al = M. Define metric space (X' d). where each element x E X represents a sequence x = (I1 ......... ..)) with x, E Va. The metric d X' X' R assigns distance d(xr, y) between any pair of sequences (x. X' x X. This distance d(x:, y) is determined by the underlying data structure, such as the Euclidean distance for integers or an embedding distance for multi-dimensional vectors.
Consider the discrete input sequence = (81, 8x), representing a sequence of tokens in an autoregressive discrete foundational model. A single forward pass through the model generates logits over the entire vocabulary V for each token in the sequence:
I fe(s<r), E
where 14 represents the logit vector at time step t and fo denotes the model parameterized by 8. These logits I are then transformed into probability distributions over the vocabulary subset Va by applying the softmax function:
po(v]8<)) softmax(l), (le),
2

---

Published as a conference paper at ICLR 2026
Instruction (Non-Targe) Tokens)
Target Tokens
4 (#1,3))
[s)
Detect location of the full zebra
The full zebra is at [( 49
10 ), ( 540
465 )}
Discrete Autoregressive Model
Model's Next-Token Predictions
2/ v
#/
LIST the SFT scenario
(Causal Language Modeling)
11
542
471
Base Cross-Entropy Loss
Discretized Distance Loss
Target Token
Ouput Token
Oupul Token Distriburion
(a)
(b)
(c)
Distribucion
Disrribation
Ve
(arts M1. Target
discretize
Token Distribution.
of
Ve
(ai s( Output
d RMSE Ganarian
p(a(
pid(E ms))
Cos (ogpofa((ss))
(d) Can
KLipula (5x) Pa(=
Figure 2: DOST trains categorical foundational models with distance target distribution instead of one-hot target. The procedure is: (a) define a token distance metric d(x; z'). ) (b) convert the metric into a continuous distribution p(z, z²), (c) discretize the distribution to obtain Pa(æ, x'), and (d) compute the KL divergence loss between the target pa and the model likelihood pe per token.
Cross-Entropy Loss. In training a discrete autoregressive model, the standard approach involves teacher-forcing where the target and model predictions are compared independently at each token. Cross-entropy loss Shannon (1948), LCE, commonly used compare two categorical distributions:
CCE Σ Σ
Ptarget (ungo(11 (;)) log pa scx)
t=1 e
where (urgge(() st) denotes the target distribution at time step t. In most cases, Prarge is one-hot distribution that corresponds to the ground truth token
2.2 DISCRETIZED DISTANCE LOSS
We define a structured element e as the unit on which the metric is defined, such as a bounding box (I1, 31, 22, 122 set of robotic joint angles. For autoregressive modeling, each element is represented as a contiguous subsequence of tokens & = [x æz} within the input sequence [..., 8, Fj 15+1,---.J where each component of the structured object becomes separate token. In practice, date value with the corresponding ground-truth component 2;, This per-position decomposition an efficient factorization of the object-level metric: evaluating all multi-token alternatives jointly would scale exponentially with the subsequence length j - i + 1. When multiple structured elements ei....,ek eK appear in single sequence, their distance-based losses are computed independently and summed.
apply DosS at each token position t independently, comparing the candi-
To incorporate the metric distance into the model's objective, we define target distribution Pa(U|zz that reflects the similarity of the tokens according to chosen distance metric d in the token space Va. This target aligns probability mass with the similarity structure, encouraging model outputs that respect the defined metric distance. Crucially, the distance metric d is derived entirely from the task's inherent structure (e.g., Euclidean distance for coordinates, angular distance for rotations, or embedding distance for VQ codes) and requires additional annotation supervision beyond the standard ground-trutt labels.
We propose formulating the target distribution pa using discretized exponential family distribution:
d(v.x.t)
exp
pa(u|z,t)
Evevy exp
d(v(...))
where the temperature hyperparameter controls the smoothness the target distribution, with lower values of 7 assigning higher probability to tokens closer to the target in the metric space. We set T for digit tokens (corresponding to a unit-variance Gaussian) and derive t via entropy matching for larger vocabularies such VQ codebooks; details provided in section C. Note that in single
3

---

Published as a conference paper at ICLR 2026
token case, where each element in the metric space consists of single token from subset of the vocabulary, we have d(v, x, t) = d(e, 24) and thus in turn Pragee ( x t) = Puarer (v/2z).
In the specific formulation is equivalent to a discretized Gaussian distribution, often referred to as a discrete Gaussian in prior work Canonne et al. (2020). Our framework generalizes this approach, offering flexible loss function applicable across range of distance metrics and training setups for foundational models.
where the root
squared enror (RMSE) is used as the distance metric, this
The discretized distance loss is defined by comparing Purger (1/22, t) distribution po ( (v) (<x)) via KI. divergence:
with the model's predicted
pa(v)ar, log pa(ulæ,
Ldist
Σ
Σ
po(v(s<c))
1=1
The final objective combines the cross-entropy loss LeE with this distance-based regularization:
L = Lce + aLdist
(6)
where a adjusts the weighting between accuracy and metric coherence. For simplicity, we fix a throughout the experiments without hyperparameter tuning.
Example. Consider a single-token case, denoted as ssingle E &single with Tsingle = (z,). To simplify, restrict the metric space to scalar Fuclidean metrics. Suppose the target token T'ingle is 5, with the Euclidean distance metric defined as d(v, z) (o x2 We construct target distribution (a(p/z) that assigns higher probabilities to tokens closer to 5 according to this distance. For example, token 4 receives higher probability than token 2, reflecting its proximity to the target within the metric space. This setup is used directly in our experiments in section 3.4.
2.3 CONNECTION TO EXTROPY-REGULARIZED POLICY OPTIMIZATION
The construction of D can be directly linked to entropy-regulariee policy optimization. In reinforcement learning, the goal is to optimize policy и over actions a E A by maximizing the expected reward. Entropy regularization augments this objective with an entropy term that penalizes peaked distributions and encourages exploration:
max Enws [R(a)] [ +H( H((T)= = Σ ((a)log=(a))
aEd
Here, R(a) the reward associated with action a, and a temperature parameter controlling the strength of regularization. The entropy term prevents the policy from collapsing too early to deterministic choice and ensures that probabilities remain distributed in proportion to their relative rewards. This objective admits closed-form optimal policy Haarnoja al. (2017):
(a) x exp
R(a)
L instantiates this result in the autoregressive modeling setting, where candidate tokens are actions and the reward R(a) is given by distance- evaluation. Rather than estimating this distribution through sampling iterative updates. DIST'Loss uses the analytical solution as the target and trains the model to minimize its KL divergence. Thus, DIST'Loss corresponds to the variance-frre, closed-form solution of entropy-regularizze reinforcement learning with known per-token rewards.
This interpretation clarifies both the efficiency and the scope of Doss: eliminates the instability associated with policy-gradiens estimators, but applies cleanly only when rewards are defined independently for each token, such as integers, coordinates, or quantized embeddings.
2.4 HIGH DIMENSIONAL DISTANCE
Our Loss is flexible and can be applied to any distance metric defined over the vocabulary Va- including the distance between high-dimensional continuous vectors. Here, we outline a practical case where the distance is defined over high-dimensional vector embeddings, which commonly used in
4

---

Published as a conference paper at ICLR 2026
MAX
JOSSE
Given input data points [(0.358, 0.589), ...], predict inputx: 0.500. The answer is 0.275.
0.12
0.18
SP:
5ft
vocab
vocab
0.26
2
10 2
10
of Training Somplies
A Trainingg Samples
05
34
Figure 3: Left: Experimental results showing MAE and RMSE across varying numbers of training samples. The y-axis inverted for visualization. Right: Overview of the task setup the meta linear regression experiment, where the model learns to perform linear regression based on the data points.
representation learning Radford et al. (2021); Caron et al. (2021) and information retrieval Karpukhin al. (2020) literature.
Consider a vector representation v(x) for each token æ E Va. where v (ax) E RR is high-dimensional embedding. Suppose that we have two singleton sequences x (x1) and y = (y1), each represented by their embedding v(z1) and v To compute the distance between these sequences, we use distance metric over their embeddings, such as cosine similarity or Euclidean distance. For instance, when using cosine similarity, the distance between v(x) and v is given by:
d d(v(x), v(v) = v(x) v(y) [ [v)I
which captures the angular separation between token embeddings. The choice of distance metric often depends on the training objective of the embedding function v. For instance, with rector-quantize representations, the distance metric typically chosen to match the quantization function used during the training of the embedder, as discussed in experiments in section 3.5.
EXPERIMENTS
We propose a general approach for leveraging metric space information to train discrete foundational models. Our method can be applied whenever model needs to generate numeric or discretized representations with regression targets. To validate its generality, we apply our approach across a toy task, (2) generative reward modeling for range of tasks: (1) synthetic function regression multimodal LLMs, (4) object manipulation in human feedback LLMs, (3) object detection within
embodied AI, and (5) image generation on vector-quantized representations, showcasing its capacity for high-ding nigh-dimensional distance modeling.
Baselines. Across our experiments, we evaluate two ablated baselines alongside the full distance- aware loss (dist): the sft bascline, which applies only the standard cross-entrope loss Lee without any distance-specific objective, and the vocab baseline, which replaces the distance loss Lais with a cross-entropy loss constrained to subset of the vocabulary Va. The vocab objective is intended to assess the impact of the distance-aware target distribution on model performance, and is defined as:
Cvecab LCE(V) + a (V)
(8)
where (Vr(V) denotes the cross-entropy loss over the entire vocabulary V and LCL(Va) is the cross-entropy loss over the numeric-constrained subset V/-
3.1 ToY: LEARNING TO REGRESS
This experiment represents a learning-to-lest task, where the model is trained to acquire the inductive bias of linear regression itself, rather than memorize specific input-output mappings. Each training sample consists of three distinct (æ, y) pairs defining a unique linear function, along with a target input for which the model must predict the corresponding output y, illustrated in fig. 3.
5

---

Published as a conference paper at ICLR 2026
Medels
RefOICO
RefE0C0+
1eSt-A
lest
et 2023)
2024)
924
Fernei22 (2hing et al., 2024g)
LIM
127K
92.8
94.7
87.4
92.8
39.4
Horence-22 2-8 1Xis at 2024:
120M
127K
92.6
923
91.5
M6.8
91.7
K2.2
39.8
82.2
Florence 2 (Kip et al, 2024)
120M
127K
93.4
95.3
92.0
88.3
92.9
83.6
91.2
91.7
Phisy lAbia et al., 2024)-57
127K
94.3
93.5
86.0
85.9
91.6
78.7
92.2
87.4
PhiSVanmuh PhiSV
127K
94.5 (10.2)
(_02))
86.0--)
K5.9 (-1
(1.6
78.2 (_0.5
92.4 (10.2)
87.66"0.3 (70.2)
127K
(10.5)
94.5 (1100
87.3(1133)
(7.3
92.2 (106)
81.4
(2.8
(70.6)
Table 1: RefCOCO (Kazemzadeh et al.. 2014; Mao et al., 2016; Yu et al., 2016) visual grounding results (accuracy, %). We fine-tune (FT) Phi3V (Abdin et al., 2024), model not trained on grounding tasks, while baselines pretrain (PT) on large-scale detection datasets.
Notably, the model is not explicitly informed that the underlying relationship is linear; must infer this structure from the input data alone.
To evaluate the data efficiency of DOSs,, we deliberately restrict the number training samples to between one and ten, where each sample corresponds to different regression function with varying slopes and intercepts. This low-data regime is principled design choice: the goal is not to optimize performance under large-scall supervision, but to assess whether our loss formulation facilitates generalization from minimal structurally meaningful supervision. This aligns with prior work in meta-learning and inductive bias evaluation (Trask al., 2018; Yu al., 2020), where models expected to extract abstract rules from very limited examples.
Setup. Each problem is defined by sampling a slope from [0.1, and an intereept from [0.0, 0.5]. For each random seed, generate ten training and 1,000 test problems, ing subsets of the training set (1-10 samples) to evaluate data efficiency. We fine-tune meta 11 allama/a //Laamma 3. -1B-Instruut (AI@ Meta, 2024) for 5,000 steps using AdamW (batch size 1, learning rate 1 x 10-5) with different loss functions, evaluating on unseen regression problems to assess structural generalization. Predictions are made at z 0.5, with performance reported as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), averaged across five random seeds. All values are reported to three decimal places. Additional details are in section D.2.
Results. The bottom panel of fig. 3 demonstrates that DIST consistently outperforms the baselines (sft and vocab) in terms of MAE, except when only single training sample is provided. This exception reflects the challenge of generalizing the linear regression property from a single example. Additionally, the vocab baseline shows high variability in regression accuracy across different training data scales due to its tendency to sharpen the target distribution on numerical outputs, leading to inconsistent performance.
3.2 MULTIMODAL: VISUAL GROUNDING
We begin by evaluating DIST the multimodal task of visual grounding, which involves generating the coordinates of the bounding box for a specified object based on the corresponding referring expression provided input.
Setup. To evaluate data efficiency, we finetune Phi3V1 (Abdin et al., 2024), which lacks pretrained grounding ability, on RefCOCO (Kazemzadeh et al., 2014; Mao et al., 2016: Yu et al., 2016) without object detection pretraining. Following (Xiao et al., 2024), combine RefCOCO, RefCOCO+, and RefCOCOg for finetuning. We focus on visual grounding, rather than object detection, to extend LLM language grounding. D is compared against strong baselines pretrained on large- scale grounding datasets, including UNINEXT (Yan et al., 2023), Ferret (You et al., 2024; Zhang et al., 2024a), and Florence-2 (Xiao et al., 2024), all finetuned on the same data. Accuracy (loU 2 0.5) the evaluation metric.
Results. Table demonstrates that incorporating DIST consistently enhances performance
over the xft baseline. In contrast, the wocab baseline results varied, underscoring the importance of
mi www.cot/phiiiiiicccc 5-visiov oin- nstruct. (4.2b)

---

Published as a conference paper at ICLR 2026
#Data
1K
10K
100K
10X
100K
RT-2 (Brohan ol., 2023)
1.9
21.9
73.1
3.8
17.7
70.4
LLaRA (Li 2024)-57
49.6
82.3
88.5
46.2
78.1
84.6
LLaRAvrocab LLaRAedior
50.8
81.0
87.0
83.5
83.4
Table 2: Object manipulation experiment results on VIMABench (Jiang et al., 2023), reported in accuracy (%). Results are presented for two test protocols (L1 and L.2) and various training data scales. For details on baseline scores, refer to section D.3.
Modelt
Type
Chor
Chat Hond
RewardBench Safery
Reasoning
Bench
U3aaRM-13B (Cui et 2024)
Soy. Chasiiee
64K (Cui al. 2024)
96.4
55.5
99.9
62.4
68.5
91.4
Tole RM- 178 (Ivison atal. 2024)
Seq. Clasoteer
64k (Chn et al, 3024)
19.4
42.3
55.5
47.4
46.1
56.2
Tolb-225-RM- 138 (Ivison et 2024)
Sog Civooteer
2M (Ivison et sl. 2014)
95.6
68.2
77.3
68.5
$1.9
91.4
Giria.5 (Brown all 3000)
(ererraiie:
02.2
44.5
65.5
59.1
65.3
63.1
Clands Setakk (Anhmpii,, 2004)
Genaraiie
73.7
92.7
$2.0
79.5
70.6
$2.0
Premellenn--78 (Kin et al., 20248)
Genaraivv
300X (Kim et 2004g)
65.3
49.1
77.1
76.5
72.0
75.8
Lama (Ale 2034)binory
Soy Closofier
(Wang
73.5
Llumsso Elunsdor
Generaties
(Wang ctal. 20046)
$3.0
Gencraiiee
(Wang 20246)
(5.00"4.))
(-196)
86.(1173)
(0.4(16.5)
353(11000)
(10.)
Table 3: Results of reward modeling experiments on RewardBench (Lambert et al., 2024) and MT- Bench (Zheng et al., 2023), reported in classification accuracy (%). Improvements D (dist) over sft are indicated with f.
target distribution for improved outcomes. With DIST Phi3V attains visual grounding performance par with state-of-the-at models trained on large-scale pretraining datasets optimized for object detection and grounding tasks. Refer to section for qualitative samples and hard-case IoU analysis showing that D predictions remain geometrically closer to the ground truth even when incorrect.
3.3 EMBODIED: ROBOTIC MANIPULATION
Robotic manipulation is another domain where foundational models frequently encounter numerical data. Here, the model must generate robotic joint actions, typically represented by position coordinates and rotation angles, based on contextual inputs and task instructions.
Setup. VIMABench (Jiang al., 2023) is benchmark for robotic manipulation, encompassing a diverse array of robot arm manipulation tasks organized into 17 distinct categories. It assesses generalization abilities across four levels (LI-L44, with this study focusing on levels L1 and L.2. Baseline models include recent multimodal LLM-based approaches, notably RT-2 (Brohan et al., 2023) and LLaRA (Li al., 2024). This work follows the experimental framework of LLaRA, which fine-tunes the multimodal LLM, LLaVA-1.52 (Liu et al., 2024), using instruction-tune -uning data. Additionally, the scalability protocol from the same study implemented. where data splits are defined according to dataset size. Consistent with A's setup, only the loss function is modified, with A-RRA serving as direct baseline. Furthermore, auxiliary tasks introduced in the study are incorporated to expand the training dataset.
Results. Table shows a consistent increase robotic manipulation accuracy with DIST Notably, its advantages pronounced in data-scarce conditions, where training data is limited to approximately IK samples, further underscoring the effectiveness of the distance metric as a meaningful prior in robotic manipulation learning. Although the performance difference between dist and sft loss narrows with the inclusion of more data, D maintains an edge in generalization. This advantage further highlighted in the more challenging L2 test protocol, where enhanced coordinate calibration by DIST significantly improves generalizability to complex tasks.
3.4 TEXTUAL: GENERATIVE REWARD MODELING
We apply D to generative reward modeling in the RLHF (Reinforcement Learning from Human Feedback) framework, where language models learn from human preference signals. Unlike
31 i uhaot i an/1lava-vl.

---

Published as a conference paper at ICLR 2026
Original
Original
Close
Random
Distant
Close
Random
Distant
Figure 4: Illustration of token distance effects on image semantics. Each row shows VQ-encoded images with four central tokens replaced by: the original, nearby token (top-10), random token, and distant token (bottom-I 10). Nearby tokens preserve semantics; random or distant ones cause distortions or semantic shifts.
Medials
Epech #Parms
FTD
50
18
FID
Full (300)
15
Ablation
MAE L
RMSE L
CligsiANN (Kang at. 2023) 2023)
S0M
3.45
225.5
mean
mean
LDM-4 (Romlacche 2022)
400M
3.601
247.7
Llama-dist
0.092
0.017
0.124
0.026
VOGAN (Esser al.. 2021)
227M
18:65
80.4
VQGAN (Esser all., 2021)
1.48
15.78
74.3
Place value weighting
0.098
0.016
0.137
0.032
LiamaGin (Sun atl. 2024) dt
H1M
10.03
116.37
157.17
Contrastive loss
0.099
0.015
0.139
0.020
LlamaGen-alr
9:41
137.44
6.27
164.32
Distance-awssn target
0.099
0.016
0.142
0.035
Llmmoin (Sun 2024)-gf
4.34
256.07
Llama-sft
0.113
0.016
0.154
0.025
Llamatien alsr
343M
4.18
209.1
3.04
25819
Table 4: Image generation results Ima- geNet (Deng et al., 2009) (section D.3)
Table 5: Ablation results meta linear regres- sion over 10 random seeds.
traditional reward models, generative reward modeling (Zheng et al., 2023; Zhang al., 2024b) uses next-token prediction within natural language templates, avoiding architectural changes required by classification approaches.
Setup. Following prior work (Wang al., 2024b) on generative reward modeling. we train lan- guage models to predict human feedback scores for instruction-respone pairs by estimating the sum of the multi-facet scores over the defined range (see section D.2). a baseline, we train standard binary classifier. Evaluation is conducted on RewardBench (Lambert et al., 2024) and MT-Bench (Zheng et al., 2023), alongside leaderboard models including UltraRM (Cui et al., 2024), Tulu-v2.5-RM (Ivison et al., 2024), GPT-3.5 (Brown et al., 2020), Claude-3- (Anthropic, 2024), and Prometheus-2 (Kim et al., 2024b). Open-source model sizes are matched to our backbone LLM (eta,,ta, 2024) for fair comparison.
Table 3 summarizes our reward modeling results. DOST shows substantial improve- ment over the standard cross-entropy loss (dist vs. sft), highlighting effectiveness in generative reward modeling. Moreover, generative reward modeling variants outperform the sequential clas- sification baseline (binary), suggesting that generative reward modeling is competitive approach, especially in data-scarce settings, it fully leverages the pretrained language modeling strengths of the LLM backbone better. The performance gain dist over binary is consistent, with notable improvements observed in the Chat Hard and Safety domains.
Results.
3.5 HIGH-DIMENSIU IMAGE GENERATION
Effects of Token Distance Image Semanties Before training the image generator, assess how token distance affects image semantics by encoding images, replacing four central tokens, and reconstructing them. Replacements use: (1) top-10 nearest tokens (excluding the original), (2) random token, and (3) bottom-10 10 distant tokens. Tokens closely aligned with the original typically
metta- ama 11. amaam--3 1-86-1 innt
8

---

Published as a conference paper at ICLR 2026
retain the semantic integrity of the image, whereas random replacements cause visual distortions, and more distant tokens introduce new, unrelated concepts, as shown in the reconstructed images in fig. 4. These findings highlight the strong influence of token distances image semantics.
Setup. Following the LlamaGen (Sun et al., 2024) pipeline, we extract discrete features using a pretrained 16x16 compression VQ model and train autoregressive transformer the resulting quantized tokens. We adopt mean squared error (MSE) as the distance metric, applied in the embedding space using the VQ model's token embeddings. Inference uses a guidance scale of 2.0, consistent with the original setup.
Results. The results in table demonstrate that LlamaGen trained with DIST consistently outperforms the standard sft baseline across various model sizes. This performance advantage is observed at both early (50 epochs) and later (300 epochs) stages of training.
3.6 ABLATION STUDY
We further investigate the contribution of each design choice in DOST using the meta linear regression experiment detailed in section 3.1. Three additional baselines are incorporated by indepen- dently ablating each component. First, we examine the impact of ablating Place value weighting or Contrastive loss for multi-token distances, as described in section A.2. We also assess the effect of substituting the Distance-aware larget with label smoothing baseline (Szegedy et al., 2016) of 0.1. Each value tokenized to three decimal places (e.g., 021)
Results As shown in table 5, each component contributes positively to DIST performance. Notably, the label smoothing baseline, lacking a distance-aware target, falls behind the dist model by wide margin. This outcome reinforces our hypothesis that modeling distance relationships is central to L 's performance gains.
4 RELATED WORK
Distance Modeling in Discrete Autoregressive Models. Extensions of LLMs are increasingly adapted for tasks that require precise spatial, temporal, and relational distance modeling. Vision- centric tasks like object detection and segmentation, which rely on generating spatial coordinates. are now addressed by multimodal LLMs (Deitke et al., 2024: You et al., 2024; Zhang et al., 2024a; Xiao et al. 2024). In LLM alignment, generative reward models mimic human feedback to guide instruction tuning (Zheng et al., 2023; Zhang et al., 2024b). LLMs have also been applied in arithmetics (Yuan et al., 2023) and timeseries forecasting (Gruver et al., 2024; Jin et al., 2023), where encoding relational and temporal proximities reduces predictive errors. Additionally, LLMs have shown potential as function regressors (Vacareanu et al., 2024; Song et al., 2024). In robotics, tasks such as manipulation and navigation represent action outputs explicitly through coordinates and joint rotations (Jiang et al., 2023; Brohan et al., 2023; Li et al., 2024) or implicitly via discrete embeddings (Metz al., 2017; Shafiullah et al., 2022). LLMS further adapted to fields like geospatial analysis (Manvi et al., 2024), RNA structure prediction (7ablocki et al., 2024), and clinical outcome forecasting (Zheng et al., 2024), where modeling distance relations is essential for understanding spatial and relational data. We introduce a simple and general training objective for distance modeling in LLM-like architectures, broadly applicable across these diverse domains.
Discretizing Continuous Distribution. The discretization of continuous distributions is well- studied area in statistics (Chakraborty, 2015). Discrete analogues continuous distributions, such as the Laplace (Ghosh et al.. 2009) and Gaussian (Canonne et al., 2020), are commonly employed in differential privacy for efficient sampling, often conjunction with federated learning (Kairouz et al., 2021). For non-analytic continuous distributions, discrete approximations using vector quanti- zation ( (Van Den Oord et al., 2017) and the Gumbel-Softmax trick (Jang et al., 2022) are common, enabling categorical representations suitable for multimodal generation tasks such as image, video, and audio synthesis (Esser et al.. 2021; Yu et al., 2024; Dhariwal et al., 2020). Recently, these quantization approaches have been adopted by general- purposse multimodal generative LLMs (Ge et al.. 2024; Wang et al., 2024a; Team, 2024). Building on these methods, we propose a training objective that embeds distance semantics into discrete autoregressive generation.

---

Published as a conference paper at ICLR 2026
Distance Modeling in Loss Functions. Metric-based objectives have shown effectiveness across applications, such as enhancing explainability in image classification (Choi et al., 2020) and boosting accuracy in few-shot learning (Gao et al., 2022). Likewise, Earth Mover Distance Optimization (EMO) better aligns distributions in language modeling compared to traditional cross-entropy (Ren al., 2024). More broadly, existing approaches incorporating distance information into training can be grouped into three categories. Loss-based methods modify the objective to account for inter-label distances: ordinal label distribution learning (Wen et al., 2023) models distances between ordinal labels explicitly, and ordinal log-loss (Castagnos et al., 2022) weights the cross- cross-entropy term by label proximity. These formulations are task- fask-specific and not readily compatible with general- purpose LLMs. Module-basee methods introduce architectural components, such as using distance or graph relationships as priors for transformer attention (Le et al., 2023). contrast, DIST operates solely in the output space and is therefore independent of model architecture. Target-based methods redistribute probability mass in the target distribution according to similarity. Class- similartil label smoothing (Chihuang & JaJa, 2021) assigns probability proportional to semantic similarity, and instance- based label smoothing (Maher Kull, 2021) uses a teacher model's 's output structure. the vocabulary of multimodal LLMs (rather than image classification labels), and it derives targets from the task's ' intrinsic metric without requiring a teacher model. Although D shares the use of soft targets and KL divergence with knowledge distillation (KD) (Hinton et al., 2015), the two differ fundamentally: KD transfers a teacher model's learned distribution to a student, whereas DOST constructs targets deterministically from the label and the task's 's intrinsic metric, requiring no teacher, no auxiliary model, and no additional supervision.
Loss differs in respects: it operates
CONCLUSION
We presented Loss, distance-aware objective for discrete autoregressive models. By replacing one-hot targets with reward- weiihted distributions derived from token metrics, DIST offers a closed-form alternative to reinforcement learning. Experiments across visual grounding, robotic manipulation, reward modeling, and image generation show improved data efficiency, demonstrating the value of distance-award asymptotic behavior under unlimited data in section B. Promising future directions include extending DiST Loss to multi-task finetuning settings, generalizing to non-metric structures such as relational or hierarchical outputs, and applying additional domains like time-series forecasting.
supervision whenever tokens admit a meaningful metric. We discuss
ETHICS STATEMENT
D is training objective and does not introduce new datasets, human subjects, or sensitive information. All experiments are performed on publicly available datasets and pretrained backbones, with additional human annotation. As the method modifies only the training loss, ethical consid- erations inherit from the original datasets and models. Potential concerns such as bias or fairesss are therefore bounded by the properties of the underlying backbones and data sources. Since the objective is designed for tasks with numerical or metric outputs, it does not introduce new risks of harmful applications beyond those already present in existing autoregressive models.
REPRODUCIBILITY STATEMENT
All experiments use publicly available backbones and datasets. Hyperparameter settings, including loss weight and temperature, are documented in Section C. Training follows standard implementations.. While training runs incur stochasticity from initialization and data order, we could not conduct extensive statistical analysis across multiple runs due computational limits. Reported results are therefore based on single or limited runs, but we verify robustness through hyperparameter ablations.
ACKNOWLEDGEMENTS
This work was partly supported by an Institute Information & communications Technology Plan- ning& & Evaluation (ITTP) grant funded by the Korean Government (MSIT) (No.RS- R1-2020-
10

---

Published as a conference paper at ICLR 2026
Artificial Intelligence Graduate School Program (Yonsei University), No. w00.RS-2022-1122000 Develop- ing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, No. R..--0011-11211343 Artificial Intelligence Graduate School Program (Scoul National University), No.RS- RS-20S--22--- 02263598, Development Self-Evolving Embodied AGI Platform Technology through Real-World Experience) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(RS-2024-003342118))
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint urXiv:2404.14211, 2024.
AI@ Meta. Llama 3 model card. 2024. URL httttt://ggithbbe www.eeteaacma//eeee wwwwo//t/mmmaaaaaaeaaaaaaaaaaaaa lllma/llamma blob/main/MoDB CARD md.
Anthropic. Claude 3 haiku, 2024. Model version: sondd---haiku22022440 Accessed: 2024- 1--01
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.072:: 2021.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski. Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision Vision-language-actinn models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 158/8, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. arXiv preprint 1411 2020.
Clément I. Canonne, Gautam Kamath, and Thomas Steinke. The discrete gaussian for differential privacy. Advances in Neural Information Processing Systems, 33:15676-15688, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
François Castagnos, Martin Wallaert, Théliau Jenn, and Amaury Moniz. A simple log-based loss function for ordinal text classification. In Proceedings of the 29th International Conference on Computational Linguistics, 2022.
Subrata Chakraborty. Generating discrete analogues of continuous probability distributions-a survey of methods and constructions. Journal of Statistical Distributions and Applications, 2:1-30, 2015.
Liu Chihuang and Joseph JaJa, Class-similarity based label smoothing for confidence calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Hongjun Choi, Anirudh Som, and Pavan Turaga. Ame-loss: Angular margin contrastive loss for improved explainability in image classification. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition workshops, pp. 838-839, 2020.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning. 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of- ofrth--rrt multimodal models. arXiv preprint arXiv:2409.17144 2024.
11

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint urXiv:2005.00344 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Farong Gao. Lijie Cai, Zhangyi Yang, Shiji Song, and Cheng Wu. Multi-distance metric network for few-shot learning. International Journal of Machine Learning and Cyhernetics, 13(9):2495-255) 2022.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer. In The Twelfth International Conference on Learning Representations, 2024.
Arpita Ghosh, Tim Roughgarden, and Mukund Sundararajan. Universally utility- mixiiiziaxiiizing privacy mechanisms. Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 351-360, 2009.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 0.5531, 2015.
Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. arXiv preprint Stl::224000927999 2024.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference Learning Representutions, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: Robot manipulation with multimodal prompts. In International Conference Machine Learning, pp. -4975-15024 PMLR, 2023.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yr Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728. 2023.
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In International Conference on Machine Learning, pp. 5201-5212. PMLR, 2021.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park. Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Sahar Kazemzadeh. Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798. 2014.
12

---

Published as a conference paper at ICLR 2026
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine- ame-grained eval- uation capability in language models. In The Twelfth International Conference Learning Representations, 2024a.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01533 2024b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster). 2015. URL http: /bblp. uni-trier de/ cbb conf/iclr: 1c1:2015 html #KingmaB14.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403.1377 2024.
Thao Minh Le, Vuong Lc, Svetha Venkatesh, and Trayen Tran. Guiding visual question answering with attention priors. In British Machine Vision Conference, 2023.
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jac Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint arXiv:2406.20095, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 26296-26306, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: /tL.ttts://oppeeee/ http:///pppeeeeieeeeeeeee// net forum?id- Bkg GRGGRR..
Mohamed Maher and Meelis Kull. Instance- based label smoothing for better calibrated classification networks. In Machine Learning and Knowledge Discovery Databases, 2021.
Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, and Stefano Ermon. Geollm: Extracting geospatial knowledge from large language models. In The Twelfth International Conference on Learning Representations, 2024.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11 11-20, 2016.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprinl trXiiv:705..50000 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Alec Radford et al. Improving language understanding by generative pre-training. OpenAI, 2018.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. (Zero-offload): Democratizing (ellion-scaae)) model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
Siyu Ren, Zhiyong Wu, and Kenny Q Zhu. Emo: Earth mover distance optimization for auto- regressive language modeling. In The Twelfth International Conference on Learning Representa- tions, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confee ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
13

---

Published as a conference paper at ICLR 2026
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Letrel Pinto. Behavior transformers: Cloning &o modes with one stone. In Advances in neural information processing systems, volume 35, pp. 22955-22966 2022.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
Xingyou Song, Oscar Li, Chansoo Lee, Bangding Yang, Daiyi Peng, Sagi Perel, and Yutian Chen. Omnipred: Language models universal regressors. arXiv preprint arXiv:2402. 14547, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06552 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Chameleon Team. Chameleon: Mixed-modas early-fusion foundation models. arXiv preprint arXiv:2405.09815. 2024.
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. Advances in neural information processing systems, 31, 2018.
Robert Vacareanu, Vlad Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly a capable regressor when given in-context examples. In First Conference on Language Modeling, 2024. URL https::/p/// /ttp///ppeeeeeeeeeeeeeeee www. forum? d= LapaUxcNFK.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in neural information processing systems, volume 30, 2017.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, 2020.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024a.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint 1xXi:22406008673 2024b.
Jiaqi Wen, Fuxian Liu, and Xin Geng. Ordinal label distribution learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-a natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/11/2220.comm http://000..mmmddmmmmssmsm URL https: / /a//accaannthtoo
org/ 2020 wmn.p-deneeee demop..ddees....m...m 6.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4818 4829, 2024.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15325-15336, 2023.
14

---

Published as a conference paper at ICLR 2026
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer. 2016.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. The Twelfth International Conference on Learning Representations, 2024.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304 02015, 2023.
LI Zablocki, LA Bugnon, M Gerard, L Di Persia, G Stegmayer, and DH Milone. Comprehensive benchmarking of large language models for rna secondary structure prediction. arXiv preprint arXiv:2410.16211 2024.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang. Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404. 07.97773, 2024a.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agar- wal. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 24, 2024b.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pp. 46595- 46623, 2023.
Wenhao Zheng, Dongsheng Peng. Hongxia Xu, Yun Li, Hongtu Zhu, Tianfan Fu, and Huaxiu Yao. Multimodal clinical trial outcome prediction with large language models. arXiv preprint arXiv:2402.06512 12, 2024.
15

---

Published as a conference paper at ICLR 2026
A MULTI-TOKEN DISTANCE
This section provides additional details on the treatment of multi-token distances, clarifying the limitations of D in this setting and outlining possible workarounds.
A.1 CREDIT ASSIGNMENT PROBLEM
Extending DOss multi-token sequences implicitly assumes that global reward can be decomposed into independent token-wise contributions, or that each token can be evaluated while holding the others fixed. In practice, this assumption fails: the model receives undifferentiated feedback across all tokens, without information about which position is responsible for the error. This the classic credit assignment problem. As result, gradients become noisy and poorly aligned with the true error source, which weakens the learning signal, slows training, and can lead to suboptimal or unstable optimization. It also undermines one of the advantages of LIST Loss, interpretability. since the induced soft targets no longer reflect a coherent metric over the token space.
For these reasons, we do not apply DIST structured multi multi-token objectives in our experiments. Instead, we focus on tasks where the reward function decomposes naturally at the token level, such as integers, coordinates, or continuous tokens. These settings preserve the benefits of DIST stable training, interpretable reward alignment, and computational efficiency. Generalizing to multi-token objectives would require explicit credit assignment mechanisms or structured training methods, which view important direction for future work.
A.2 WORKAROUNDS
Consider multi-token case where each element in the metric space consists of sequence of tokens from the vocabulary, denoted Fmulli E Xmulii with Tmulti = (x1,..... .t). For instance, this could represent multi-digit integer split into individual tokens. Applying multi-token objectives directly in autoregressive models trained with teacher-forcing is extremely inefficient, it requires training-tinme time sequence generation. To circumvent this limitation, we propose two practical alternatives.
Contrastive Target Augmentation Instead of evaluating all possible multi-token sequences, we propose sampling contrastive multi-token candidate 3 E Amuti for training. Such a candidate is selected from nearby neighbors of the target a in the metric space, without reference the training model fo. For example, in the case of integer sequences, 39 might be chosen as close neighbor to the target 40, with each digit tokenized separately.
For each token in the sequence, we extend the target distribution by incorporating the negative sample T. The contribution of each token to the overall distance is defined based on its position-wise difference from the target æ. For instance, when the target is 40, the negative sample 39 is assigned token-wise distance where the tens digit 3 has distance 0 from the target's 's 4, while the units digit 9 has distance of 1 from the target's O. We then concatenate the logits of x: and the selected logit I at each token position, forming an extended likelihood distribution. The distance loss Laisa is applied to this extended distribution.
Place Value Weighting For tasks involving multi-digit integers or sequences where token positions have different significance, we introduce place value weighting. In this approach, tokens are weighted that differences in higher place values have greater
according to their positional importance, impact on the loss. For example, in multi-digit integer setting, we directly multiply the distance loss by the place value weight for each token, assigning more weight tokens in higher positions. Let Ømulti = ( xx) represent the target sequence, with x; denoting the digit in the i-th place (e.g., thousands, hundreds, tens, units). The place-weighted loss is formulated as:
Cplace Σ wi (xis(
where wa is the place weight for position i: (thousands), (hundreds), 2 (tens), and (units).
16

---

Table 6: Hyperparameter sensitivity analysis on the loss weight coefficient Results shown for Published as a conference paper at ICLR 2026
Coefficient (a)
Accuracy (%)
1.0
77.3
0.2
77.8
0.1
85.3
0.02
80.7
0,01
79.9
0.005
76.7
0.001
75.6
SFT
75.4
reward modeling.
B DISCUSSION
B. ASYMPTOTIC BEHAVIOR
potential concern that the advantages of DIST Loss diminish with unlimited data and compute. While true in theory, this setting is representative of practical training. Realistic applications operate with limited supervision, moderate model capacity, and constrained compute, where inductive biases are critical. Under these conditions, D provides consistent improvements with negligible overhead, demonstrated in all reported experiments using standard backbones, realistic dataset sizes, and established evaluation protocols.
C ADDITIONAL EXPERIMENTS
C.1 HYPERPARAMETER SENSITIVITY
DIST introduces two tunable hyperparameters.
Loss weight We fix 0.1 unless otherwise noted. A sweep in reward modeling shows robustness across a wide range; performance drops only when a becomes too small, effectively reducing the method SFT. We additionally conduct sensitivity analysis on table which confirms that LOSS improves over the base SFT for wide range of a.
Temperature т. Controls the sharpness of the soft target distribution. We derive from the structure of the token space rather than tuning it.
For decimal digit tokens (Vi .o...., 9)), the squared Euclidean distance on the integer lattice gives d(v, =x)= (v 2² Setting - = 1 yields a likelihood kernel exp(-(v xz² )²) which corresponds discretized unit-variance Gaussian centered at 24. This natural default that assigns geometrically decaying probability to tokens farther from the target.
For VQ codebook tokens, similarity seales are not fixed priori: identical score gaps can correspond to different semantic distances depending codebook training. We therefore select using an information--heorrtiec criterion. uniform distribution over vocabulary of size K has entropy log K. We choose that the induced soft target has comparable entropy, ensuring consistent sharpness across different vocabulary sizes. For a VQ codebook with K = 16,384, this yields 9.7.
Place value weights in multi-digit numbers fixed by construction and not tunable.
C.2 ROBUSTNESS TO TASK AND METRIC VARIATIONS
To assess generalization, we conducted two experiments.
Task generalization.
We evaluated whether DOSs fine-tuning impairs unrelated tasks. A meward-modeling model tested on MMLU and visual grounding model (RefCOCO) tested on Real RealWorIdQA both show negligible degradation, reported in table 7.
17

---

Published as a conference paper at ICLR 2026
Model
Reward Aceuracy (%)
MMLU Accuracy (%)
Backbone (Llama-3,1-8)
44.5
SFT
75.3
42.8
LosT
85.3
43.9
Model
RealWorldQA Accuracy (%)
Backbone (Phi-3.5V)
54.4
Loss on RefCOCO
54.3
Table 7: Catastrophic forgetting analysis. Top: fine-tuning with DIST*LoSE for reward modeling yields minimal degradation the general-purpose MMLU benchmark. Bottom: fine-tuning with Loss for visual grounding (RefCOCO) preserves performance on the out-of-domain Real- WorldQA visual understanding benchmark.
Metric
Accuracy (%)
DIST (Euclidean)
85.3
DIST (Random)
76.0
SFT
75.3
Table 8: Sanity check with contradictory metric. Using random distance metric provides improvement over SFT, confirming that the semantic validity the metric is essential for D Results shown for reward modeling.
Contradictory metric. We trained reward models with randomly assigned distances between labels. As shown in table 8, this yielded no gains over SFT, confirming that improvements arise when distances capture meaningful structure.
D IMPLEMENTATION DETAILS
D.1 GLOBAL SETUPS
We the HuggingFace Trainer (Wolf et al., 2020) and TRL trainer (von Werra al., 2020) with DeepSpeed ZeRO-3 (Ren et al., 2021) and the Adam V optimizer (Loshchilov & Hutter, 2019). The base foundational models are detailed in table 9, with computational requirements specified in table 10.
D.2
SETUPS
Toy: Learning to Regress The learning rate is set to -5 with linear decay schedule and no warmup. Training epochs configured to ensure each model is exposed approximately 250 samples to prevent underfitting. For example, with a training dataset size of 2, the epoch count is set 125. Each experiment is repeated five times with random seeds [1 5] for statistical stability.
Textual: Generative Reward Modeling For fine-tuning, the helpsteer2 dataset (Wang et al., 2024b) was reformatted into an instraction-following structure, where scores for each of the five categories were designated as model outputs. The model was trained for two epochs with learning rate of 1 x 10-5 using the paged Adam optimizer (Kingma & Ba, 2015). The prompt used during training is illustrated fig. 5. During inference, a logit-based based score prediction function was implemented to evaluate two samples by generating score probabilities weighted averages from the softmax probabilities, assigning a final reward based on higher scores for preferred outputs.
0-20 points scale. The model calculated
18

---

Published as a conference paper at ICLR 2026
Prompt for Generative Reward Evaluation
Please act as an impartial judge and evaluate the quality of the response provided by Al assistant to the user question displayed below. Your evaluation should consider five factors helpfulness, correctness, coherence, complexity, verbosity. Here's brief explanation of each factor:
Helpfulness: Overall helpfulness the response to the prompt.
Correctness: Inclusion of all pertinent facts without errors.
Coherence: Consistency and clarity of expression.
Complexity: Intellectual depth required write response (i.e. whether the response be written by anyone with basic language competency or requires deep domain expertise). VVerbosity: Amount of detail included in the response, relative what is asked for in the prompt.
Do not allow the length of the responses influence your evaluation. Be objective as possible. Please first provide an overall score over model response. You must provide overall score as number between 0 and 20.
Then provide a set of 5 score over model response. Only provide the score number between and 4.
[User Question] luser inpur)
[Start of Model Response]
(model response to evaluate)
[End of Model Response]
Figure 5: Instruction-tuning prompt template for generative reward modeling.
Multimodal: Visual Grounding For fine-tuning, concatenate the training sets of RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Mao et al., 2016; Yu et al., 2016). All images resized to 1024 1024 constrain the range of generated digits, with coordinate values rounded to the nearest integer. During inference, outputs that cannot be parsed as bounding box coordinates considered incorrect. Training is conducted with learning rate of 2e 100 steps of linear warmup, and total of three epochs.
Embodied: Robotic Manipulation We convert the VIMA dataset (Jiang et al., 2023) into an instruction-tuning compriooiiininn-ccppiiiillllllle format using the provided script from the LLaRA (Li et al., 2024) repository... The pretrained LLaVA-1. 1.5 (Liu et al., 2024) model is then fine-tuned on the object manipulation task. Following (Li et al., 2024), we incorporate auxiliary objective augmentations from the same repository into the training set. We the oracle object detection labels for evaluation. Training is conducted with learning rate of -5, using 0.3 ratio of linear warmup and cosine decay over two epochs.
High-Dimension: Image Generation We employ the pretrained image vector quantization model from the LlamaGen (Sun et al., 2024) repository All images are resized to 384 x 384 using random center cropping. During evaluation, images are generated at 384 384 and then resized 256 x 256 for model-based metric computations. Classifier-free guidance with scale of 2.0 is applied during inference. Experimental protocols strictly adhere the repository's guidelines.
D.3 BASELINE SCORES
Embodied: Robotic Manipulation For LLaRAst. adopt results from Tables 15, 17, and 19 of the original paper (Li et al., 2024), using D-inBC + Aux with all six auxiliary tasks (epoch: 2, iteration: 14) for data sizes of 0.8k, 8k, and 80k. Notably, at the 80k scale, using all auxiliary tasks does not outperform using only subset, as reported in Table the same paper. However, we adopt the former for consistency and generalizability across different scales.
19

---

Published as a conference paper at ICLR 2026
Experiment Type
Size
Backbone
Toy (3.1)
1B
meta- metee-1Danm//ilmm:3222--------1- (AT@ Meta, 2024)
Textual (3.4)
8B
mete--laamn/ilaamm-------------- (AI@ Mcta, 2024)
Multimodal (3.2)
3.8B
nicroooft/PPni----mii---------
(Abdin et al., 2024)
Embodicd (3.3)
Litheotian/Llava-v...5 (Liu et al., 2024)
High-D gigh-Dimension (3.5)
343M
Scratch (Sun ct al., 2024)
Table 9: Base foundational models used for finetuning in each experiment type.
Experiment Type
GPU Model
VRAM (GB)
# GPUs
Toy (3.1)
RTX 3090
Textual (3.4)
A6000
Multimodal (3.2)
A6000
Embodied (3.3)
L40S
Bigh--Dmmensio (3.5)
L40S
48
8
Table 10: Computational requirements for each experiment are reported per single run; multiple runs may be needed depending on configuration or random seeds.
MAR
MAL
RASSE
MAR
KMASE
Madds
Treedee Prodhrre
Trarein Preblene
11009
1014
10215
1000
2105
11:52
0.009
Eloea 9: BBalo
0.1
0.003
0.599
0037
0.17
0009
0.1
0 006
0.64
0.01
0.098
0085
0116
0017
0.106
0.533
0.333
0.665
Liaea 32 2.BBoo BBoos
6.120
0.016
0.163
0017
0.44
0934
189
0.86
0.122
0.09
0164
5085
0122
0011
163
0,033
0111
0/99
0.151
0.014
3laea32 (Balar
81113
00008
0.833
0021
0013
0881
0.05
8.16
0081
6112
9965
163
0.993
0.80
0017
0.124
0.000
Table 11: Meta linear regression experiment results on one to ten training problems and 1,000 test problems, with scores averaged over five random seeds.
Masi Tar
Awwey
Teads
De No Anserr
CTP
Pulks
411
120m
Abe: OPtia 134
Adre: CPria
Arc. Maall
Abe Napter
Nard
Meduas
Diegoos
Stddi Swald Roped
961
400
Table 12: Fine-grained statistics on model performance on RewardBench (Lambert et al., 2024).
Image Generation We use the class-conditional ImageNet 256x256 results with CFG 2.0 from Table the LlamaGen paper (Sun et al., 2024) as baselines.
E EXTENDED QUANTITATIVE RESULTS
Toy: Learning to Regress
We provide scores corresponding to fig. 3 in the main paper in table 11.
Textual Task: Generative Reward Modeling Detailed results for each data source Reward- Bench (Lambert et al., 2024) are reported in table 12.
F ADDITIONAL QUALITATIVE SAMPLES
Textual:
Generative Reward Modeling Figure 7 shows inference results of Llama-based genera- tive reward model trained with DOST
Multimodal: Visual Grounding Figure presents qualitative results from visual grounding experiments, comparing the base cross-entropy loss with our proposed Dost.Loss. To further assess metric sensitivity, we measure the mean loU on RefCOCO testA restricted to cases where both models predict the wrong object (loU < 0.5). Correct predictions saturate IoU, so shared failures better reveal how closely each model preserves geometric structure. As shown in table 13, D produces predictions that are consistently closer to the ground truth even when incorrect.
20

---

Published as a conference paper at ICLR 2026
Model SFT DOST
IoU (ReICOCO testA, hard cases)
31.0
40.3
Table 13: Metric sensitivity on hard cases. Mean IoU on RefCOCO testA examples where both models predict the wrong bounding box (IoU < 0.5). DIST predictions remain closer to the ground truth, indicating improved geometric alignment.
Instruction-Respone Pair Sample for Object Manipulation
[Prompt]
<image0> <task>-Sweep any <p>>polk dot small block<<p>> <b><00.00, 0.617),
0.203, 0.578</b> into <p>>blue three-sided rectangle</p>> at <b><0.500, 0.625), 0.297, 0,398</b>> without exceeding <p>plastic line<//>> at <b>(00.500, 0.617), 0.328, 0.047</b>.</anr
Can you explain what needs to be done to perform the task in this scene?
Every action you take must include two locations in the format of <b>((, <)</>> and one clockwise rotation angle the format of <r>|r]</r> The first location the image coordinate where you start to sweep the object using spatula, and the second location where you stop sweeping. The image coordinate ranges from to 1. The rotation angle indicates how many degrees you rotate the spatula clockwise, and ranges from -359 to 359.
[Answer]
Step 1: Sweep the <p>polka dot small block</p> at <b>( 0.805)< rotate <>> </r>>> degrees, and stop at <b>500.000 0.609)</b>.
Figure 6: Example of an instruction-responss pair from the VIMA dataset Jiang et al. (2023), illustrating an object manipulation task. Metric-related output tokens are bolded for emphasis.
21

---

Published as a conference paper at ICLR 2026
Query: Who created the Superman cartoon character?
Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Inc Shuster. Superman first appeared in Action Comics #1,, which was published by Delective Comics, Inc. (later DC Comies) in June 1938. The churacter's 's immense popularity established him of the enduring and recognizable figures in the superhero genre.
"Chosen" Answer:
DIST helpluiness score expeciation: 2.81
"rejected" Answer: The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.
DIST belpfuiness score expectatione:
Query: Hi, I'm in the mood for Bloody Mary. Can you give me recipe for making onc?
"Chosen" Answer: Certainly! The Bloody Mary is classic cocktail known for its bold flavor and is commonly enjoyed as hrunch drink.
Here's s busic recipe for making a single serving:
#8e Ingredients:
-1 1/2 ounces (45 ml) vodka
(90 tomato juice
1/2 ounce (15 ml) lemon juice, freshly squeezed
dash Worcestershire sauce
Tahasco sauce (to taste)
Ground black pepper (to taste)
Celery salt taste)
Horscradish (optional, to taste)
celery
garnish)
1 lemon wedge or lime wedge (for gurnish)
#*# Instructions:
* *Prepare the Glass:
Rub the lemon or lime wedge along the lip of a tall glass (often a highball glass).
Dip the wetted
velery salt if you salted rim.
2. Mix the Drink:**
the glass
cubes.
Add vodka, tomato juice, and lemon juise to the glass.
Add Worcestershire sauce, a few dashes Tabasco, a pinch celery salt, ground black pepper. and horseradish (if using).
well to combine all the ingredients.
3. *Garnisb:**
a Taste and adjust the scasoning, adding more Tabasco, lemon juice, or other ingredients as necessary.
Garmish
celery
wedge
4. Adjust and Serve:*
Serve immediately.
### Variations and Tips:
**Garnish Variations: You can also garnish with olives, pickles, cocktail onions, or even strip of crispy bacon.
*"Spice Level:*> Adjust the spiciness to your liking
by adding hot your choice. **Clamato*** Some people like to use Clamato (a mixture of clam and tomato juices) instead of plain tomato juice for different twist known
Bloody Cacsar.
**Make Virgin;** make non-alcoholic version. simply omit the vodka for Virgin Mary.
Feel free to customize your Bloody Mary to your taste preferences. Enjoy your cocktail!
DIST Overall expectation: 1,33
"rejected" Answer: Sure, quick easy recipe making delicious Bloody Mary:
Ingredients:
ounces vodka
3-4 doshes Worcestershire sauce
1-2 teasporons hot sauce
lime wedge for garnish
2 ounces tomato juice
ounce hosseradish
1/2. ounce hest juice
ounce olive juice
Instructions:
Fill an Old Fashioned glass with ice cubes. In a shaker, combine the vodka, Worcestershire sauce, hot sauce, lime wedge, and tomato juice. the prepared glass top with beetjuice and olive juice. Gamish with lime wedge and Shake combine. Strain the
serve.
DIST-Los Overall expectation: 1.20
Figure 7: Qualitative examples from the generative reward modeling experiment.
22

---

Published as a conference paper at ICLR 2026
Distance Modelling Enhances Grounding Accuracy
Elephant 3:00
Dude sitting all himself
blue stripe shirt
guy behind tennis racket, baldish
the brocolli by the micky mouse the chicken
GT
Base
Ours
above roof
Error Cases: Ambiguous Ground- -ruuh Labels
NOVELLING
K9
maroon shirt black shorts bending
head locking at computer
lime with a perfect hole and line
Figure 8: Qualitative examples from the visual grounding experiment. Top: our proposed >oss loss demonstrates higher visual grounding accuracy compared to the standard cross-entrope loss. Bottom: A manual examination inference results reveals that substantial portion RefCOCO dataset (Kazemzadeh et al., 2014; Mao et al., 2016; Yu et al., 2016) contains labels that are ambiguous, even for human annotators, which may lead to underestimation of model performance.
23


Reviews:
[{"review_id": "vFdSiOvvxj", "reviewer": "Reviewer_aRs1", "summary": "This paper focuses on the field of LLM extended to non-linguistic domains (e.g., multimodal understanding, robotic manipulation, generative reward modeling) and addresses the key problem that traditional one-hot targets and cross-entropy loss ignore the metric relationships (e.g., coordinates, rotation angles, quantized embeddings) inherent in tokens. Motivated by the inefficiency of conventional fine-tuning (neglecting metric structure) and the instability of RL methods (sampling/rollout noise), the authors aim to enhance model performance in low-data regimes while maintaining compatibility with existing architectures . The core method, DIST2Loss, transforms continuous exponential family distributions derived from inherent distance metrics (e.g., Euclidean distance, RMSE) into discrete categorical targets, computes distance-aware loss via KL divergence, and fuses it with standard cross-entropy loss. Empirical results show that DIST2Loss consistently improves performance across diverse tasks with the most notable gains in data-scarce settings .", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript addresses a critical limitation of discrete autoregressive models (neglect of metric token relationships) with a principled solution—DIST2Loss directly embeds metric structure into the target distribution without relying on extra data or architectural modifications. This fills a gap in extending LLMs to non-linguistic tasks where spatial/numerical relationships matter.\n\n2. DIST2Loss is validated on five distinct tasks (meta linear regression, visual grounding, robotic manipulation, reward modeling, image generation), demonstrating its broad applicability. This cross-task consistency strengthens the credibility of the method. It also performs well in low-data regimes.", "weaknesses": "1. Insufficient details on hyperparameter $\\tau$: The temperature hyperparameter $\\tau$  controls the smoothness of the target distribution, but the manuscript only states \"small values for digits and larger values for VQ-VAE vocabularies\" without providing specific values or a systematic sensitivity analysis. \n\n2. I suggest the paper incorporates additional technical approaches for comparable distance perception methods. For instance, it could detail whether other methods achieve distance modeling through modifications to the loss function, adjustments to the target distribution, or the introduction of external modules. \n\n3. Does the default setting of $\\alpha$ = 0.1 potentially lead to a scale imbalance problem? Is it possible to incorporate an adaptive scaling mechanism to prevent this?\n\n4. The authors evaluate other capabilities solely after single-task fine-tuning (e.g., MMLU after fine-tuning for reward modeling). However, they do not validate performance in multi-task fine-tuning scenarios. \n\n5. The authors only validated general language abilities (MMLU) and did not cover non-linguistic fundamental abilities (e.g., image understanding capabilities). Including these aspects would significantly enhance the paper's quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kkc1NNxgqW", "reviewer": "Reviewer_qCXL", "summary": "The work aims to incorporate an additional regularisation term into the loss of an LLM which more explicitly forces the model to learn a representation of distance (in terms of the metric used in the regularisation term). The distance regulariser is also phrased in terms of entropy-regularisation in policy optimisation. Extensive experiments are shown, including an ablation study, which consistently support the utility of the proposed regularisation term.", "questions": "I have listed a number of questions under weaknesses regarding the limitations of presenting the mathematical details using a single subsequence. I would appreciate if these could be answered.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\nThe work is fairly original, particularly in the exact way they approach incorporating the distance metric. I'm particularly considering the discretised distance loss in this case which enables the KL divergence to work in a straight forward manner. The related work section at the end provides a nice overview of the literature and is very fair to other works in the space. I think contextualising the work in this manner should count towards originality, even if it does make the source of the ideas for the work more transparent.\n\n## Quality\nThe experimental design and extensive nature of the experimentation is a large strength of this work for me. Particularly as the work is proposing a \"framework\" it is good to see such breadth of experimentation considered. The motivation and hypothesis of the work is clear and grounded well in the literature (to my knowledge). The results which are obtained do directly test the claims of the work and are interpreted fairly overall.\n\n## Clarity\nOverall the paper is well written, with clear tables and figures. The paper is structured well to support understanding and mathematical notation is clear, consistent and mostly intuitive. I also appreciate how the sections are structured in order of task complexity (as best they can be).\n\n## Significance\nOverall I think the work has the potential to inspire future work and does provide good results on the benchmarks. Once again the extensive literature review also supports the fact that this work has broad utility across a couple domains which supports its significance.", "weaknesses": "## Clarity\nI find the manner that subsequences is introduces in Section 2.2 a bit unintuitive and requires some effort to parse. I'm puzzles that the subsequence needs to be sequential within the input sequence. I assume this is answered by the point: \"...multiple elements are present within $s$, we limit our explanation to a singe $x$-subsequence here for clarity\". I find that this is a bit too subtle of a statement to actual convey the fact that it could easily generalise (if indeed it can and so ease of explanation becomes the priority). For example, what would need to change to the formulation or equations to make it work for multiple sequences? What tasks are of this nature and why might a single sequence be a sufficient explanation? Does this require the permutation invariance of the LLM to work?\n\n## Significance and Quality\nI will ground these two sections as they share a common point. One of the primary issues for me is that the need for supervision on the distance metric is somewhat glossed over. Fundamentally the model is being given more supervision and so higher performance is expected. This limits significance to a degree, but this affect quality more for me as this should really be discussed. How easy it is to define metric spaces for a variety of problems is the determinant of the success and significance of the work and this should be more clearly acknowledged and discuss. The level of experimentation shown here does support that it is possible, but it is left to the reader to gauge and really this is where the conceptual insight of the work lies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2N2aYccBBu", "reviewer": "Reviewer_7BSi", "summary": "This paper propose to learn the connections between discrete outputs, such the correlations between them can be captured during the learning process.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The reported results are marginally improved compared with existing methods.\n\n2. The source code is produced.", "weaknesses": "1. The motivation of learning correlations among discrete concepts/semantics is not novel, which has been largely explored in the knowledge distillation works for a decade.\n\n2. The proposed DIST^2 loss is just a combination of loss used in knowledge distillation and KL-divergence, which is also proposed in 'Distilling Knowledge from Graph Convolutional Networks. CVPR 2020'.\n\n3. Overall, this work lacks of novelty and is with limited technical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y4mq0Gj8UD", "reviewer": "Reviewer_Pvxr", "summary": "The paper addresses a key limitation in adapting large language models (LLMs) to tasks involving numerical or metrically structured data. Conventional fine-tuning methods treat tokens as categorical variables, ignoring the intrinsic distance relationships among them—such as numerical proximity or spatial metrics. To overcome this, the authors propose DIST² loss, a new objective that integrates predefined distance relationships between tokens into the autoregressive loss used during training. This approach enables LLMs to better capture metric structure while remaining compatible with standard categorical token modeling. Empirical results demonstrate substantial performance improvements across tasks where distances are semantically meaningful, including object detection, object manipulation, reward modeling, and image generation.", "questions": "1. Is the same metric used for every numeral token, or are different metrics applied depending on token type (e.g., integers vs. floats vs. directions)?\n2. Are metric-based losses applied to text-only tokens, or only to numerically meaningful ones?\n3. Are such tokens, on which the metric/distance-based loss is applied, extracted manually?\n4. While the paper shows improvements in downstream tasks, could the authors demonstrate that the model achieves more accurate numerical reasoning (e.g., better alignment between predicted and true distances)? The authors could consider a design of a controlled experiment where digits/numbers are extracted from data and the trained models are asked to evaluate the distances or any related tasks. The results are then compared with true numerical distance values to directly test metric reasoning performance.\n\nThe current score reflects the questions above. I will consider updating the rating once my concerns have been resolved.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written and easy to follow.\n* The problem is important and timely, addressing a fundamental gap in how LLMs handle numerical and metric relationships.\n* Empirical results show significant improvements across diverse downstream tasks, including those with limited data\n* The proposed method integrates smoothly into existing LLM training objectives, maintaining compatibility with categorical token modeling.", "weaknesses": "The paper does not clearly discuss how different types of numerical or metric data (e.g., integers, floats, directions) are handled under the same framework. It remains unclear whether the model improves actual numerical reasoning accuracy, beyond performance metrics on downstream tasks. There is a lack of explicit evaluation of the model’s ability to reason about true numerical distances or relationships, which would more directly validate the method’s intended benefits. See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)