Teaching Metric Distance to Autoregressive Multimodal Foundational
  Models
Authors: name: Jiwan Chung; name: Saejin Kim; name: Yongrae Jo; name: Jaewoo Park; name: Dongjun Min; name: Youngjae Yu
Published: 2025-03-04 08:14:51

Abstract: As large language models expand beyond natural language to domains such as
mathematics, multimodal understanding, and embodied agents, tokens increasingly
reflect metric relationships rather than purely linguistic meaning. We
introduce DIST2Loss, a distance-aware framework designed to train
autoregressive discrete models by leveraging predefined distance relationships
among output tokens. At its core, DIST2Loss transforms continuous exponential
family distributions derived from inherent distance metrics into discrete,
categorical optimization targets compatible with the models' architectures.
This approach enables the models to learn and preserve meaningful distance
relationships during token generation while maintaining compatibility with
existing architectures. Empirical evaluations show consistent performance gains
in diverse multimodal applications, including visual grounding, robotic
manipulation, generative reward modeling, and image generation using
vector-quantized features. These improvements are pronounced in cases of
limited training data, highlighting DIST2Loss's effectiveness in
resource-constrained settings.