Orionnnn/hub-stats huggingface dataset:
When to Trust Context: Self-Reflective Debates for Context Reliability
Authors: name: Zeqi Zhou, name: Fang Wu, name: Shayan Talaei, name: Haokai Zhao, name: Cheng Meixin, name: Tinson Xu, name: Amin Saberi, name: Yejin Choi
Published: 2025-06-06 12:09:34

Large language models frequently encounter conflicts between their parametric
knowledge and contextual input, often resulting in factual inconsistencies or
hallucinations. We propose Self-Reflective Debate for Contextual Reliability
(SR-DCR), a lightweight framework that integrates token-level self-confidence
with an asymmetric multi-agent debate to adjudicate such conflicts. A critic,
deprived of context, challenges a defender who argues from the given passage; a
judge model evaluates the debate and determines the context's reliability. The
final answer is selected by combining the verdict with model confidence.
Experiments on the ClashEval benchmark demonstrate that SR-DCR consistently
enhances robustness to misleading context while maintaining accuracy on
trustworthy inputs, outperforming both classical debate and confidence-only
baselines with minimal computational overhead. The code is available at
https://github.com/smiles724/Self-Reflective-Debates.
Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible
  Reasoning
Authors: name: Jiayi Yuan, name: Hao Li, name: Xinheng Ding, name: Wenya Xie, name: Yu-Jhe Li, name: Wentian Zhao, name: Kun Wan, name: Jing Shi, name: Xia Hu, name: Zirui Liu
Published: 2025-06-11 08:23:53

Large Language Models (LLMs) are now integral across various domains and have
demonstrated impressive performance. Progress, however, rests on the premise
that benchmark scores are both accurate and reproducible. We demonstrate that
the reproducibility of LLM performance is fragile: changing system
configuration such as evaluation batch size, GPU count, and GPU version can
introduce significant difference in the generated responses. This issue is
especially pronounced in reasoning models, where minor rounding differences in
early tokens can cascade into divergent chains of thought, ultimately affecting
accuracy. For instance, under bfloat16 precision with greedy decoding, a
reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation
in accuracy and 9,000 tokens difference in response length due to differences
in GPU count, type, and evaluation batch size. We trace the root cause of this
variability to the non-associative nature of floating-point arithmetic under
limited numerical precision. This work presents the first systematic
investigation into how numerical precision affects reproducibility in LLM
inference. Through carefully controlled experiments across various hardware,
software, and precision settings, we quantify when and how model outputs
diverge. Our analysis reveals that floating-point precision -- while critical
for reproducibility -- is often neglected in evaluation practices. Inspired by
this, we develop a lightweight inference pipeline, dubbed LayerCast, that
stores weights in 16-bit precision but performs all computations in FP32,
balancing memory efficiency with numerical stability. Code is available at
https://github.com/nanomaoli/llm_reproducibility.
UFM: A Simple Path towards Unified Dense Correspondence with Flow
Authors: name: Yuchen Zhang, name: Nikhil Keetha, name: Chenwei Lyu, name: Bhuvan Jhamb, name: Yutian Chen, name: Yuheng Qiu, name: Jay Karhade, name: Shreyas Jha, name: Yaoyu Hu, name: Deva Ramanan, name: Sebastian Scherer, name: Wenshan Wang
Published: 2025-06-10 22:32:13

Dense image correspondence is central to many applications, such as visual
odometry, 3D reconstruction, object association, and re-identification.
Historically, dense correspondence has been tackled separately for
wide-baseline scenarios and optical flow estimation, despite the common goal of
matching content between two images. In this paper, we develop a Unified Flow &
Matching model (UFM), which is trained on unified data for pixels that are
co-visible in both source and target images. UFM uses a simple, generic
transformer architecture that directly regresses the (u,v) flow. It is easier
to train and more accurate for large flows compared to the typical
coarse-to-fine cost volumes in prior work. UFM is 28% more accurate than
state-of-the-art flow methods (Unimatch), while also having 62% less error and
6.7x faster than dense wide-baseline matchers (RoMa). UFM is the first to
demonstrate that unified training can outperform specialized approaches across
both domains. This result enables fast, general-purpose correspondence and
opens new directions for multi-modal, long-range, and real-time correspondence
tasks.
Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust
  MedVQA in Gastrointestinal Endoscopy
Authors: name: Sushant Gautam, name: Michael A. Riegler, name: Pål Halvorsen
Published: 2025-06-11 17:31:38

Medical Visual Question Answering (MedVQA) is a promising field for
developing clinical decision support systems, yet progress is often limited by
the available datasets, which can lack clinical complexity and visual
diversity. To address these gaps, we introduce Kvasir-VQA-x1, a new,
large-scale dataset for gastrointestinal (GI) endoscopy. Our work significantly
expands upon the original Kvasir-VQA by incorporating 159,549 new
question-answer pairs that are designed to test deeper clinical reasoning. We
developed a systematic method using large language models to generate these
questions, which are stratified by complexity to better assess a model's
inference capabilities. To ensure our dataset prepares models for real-world
clinical scenarios, we have also introduced a variety of visual augmentations
that mimic common imaging artifacts. The dataset is structured to support two
main evaluation tracks: one for standard VQA performance and another to test
model robustness against these visual perturbations. By providing a more
challenging and clinically relevant benchmark, Kvasir-VQA-x1 aims to accelerate
the development of more reliable and effective multimodal AI systems for use in
clinical settings. The dataset is fully accessible and adheres to FAIR data
principles, making it a valuable resource for the wider research community.
Code and data: https://github.com/Simula/Kvasir-VQA-x1 and
https://huggingface.co/datasets/SimulaMet/Kvasir-VQA-x1
Query-Level Uncertainty in Large Language Models
Authors: name: Lihu Chen, name: Gaël Varoquaux
Published: 2025-06-11 12:39:48

It is important for Large Language Models to be aware of the boundary of
their knowledge, the mechanism of identifying known and unknown queries. This
type of awareness can help models perform adaptive inference, such as invoking
RAG, engaging in slow and deep thinking, or adopting the abstention mechanism,
which is beneficial to the development of efficient and trustworthy AI. In this
work, we propose a method to detect knowledge boundaries via Query-Level
Uncertainty, which aims to determine if the model is able to address a given
query without generating any tokens. To this end, we introduce a novel and
training-free method called Internal Confidence, which leverages
self-evaluations across layers and tokens. Empirical results on both factual QA
and mathematical reasoning tasks demonstrate that our internal confidence can
outperform several baselines. Furthermore, we showcase that our proposed method
can be used for efficient RAG and model cascading, which is able to reduce
inference costs while maintaining performance.
Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion
  Models
Authors: name: Sungwon Hwang, name: Hyojin Jang, name: Kinam Kim, name: Minho Park, name: Jaegul choo
Published: 2025-06-10 20:34:47

Fine-tuning Video Diffusion Models (VDMs) at the user level to generate
videos that reflect specific attributes of training data presents notable
challenges, yet remains underexplored despite its practical importance.
Meanwhile, recent work such as Representation Alignment (REPA) has shown
promise in improving the convergence and quality of DiT-based image diffusion
models by aligning, or assimilating, its internal hidden states with external
pretrained visual features, suggesting its potential for VDM fine-tuning. In
this work, we first propose a straightforward adaptation of REPA for VDMs and
empirically show that, while effective for convergence, it is suboptimal in
preserving semantic consistency across frames. To address this limitation, we
introduce Cross-frame Representation Alignment (CREPA), a novel regularization
technique that aligns hidden states of a frame with external features from
neighboring frames. Empirical evaluations on large-scale VDMs, including
CogVideoX-5B and Hunyuan Video, demonstrate that CREPA improves both visual
fidelity and cross-frame semantic coherence when fine-tuned with
parameter-efficient methods such as LoRA. We further validate CREPA across
diverse datasets with varying attributes, confirming its broad applicability.
Project page: https://crepavideo.github.io
Vision Matters: Simple Visual Perturbations Can Boost Multimodal Math
  Reasoning
Authors: name: Yuting Li, name: Lai Wei, name: Kaipeng Zheng, name: Jingyuan Huang, name: Linghe Kong, name: Lichao Sun, name: Weiran Huang
Published: 2025-06-11 13:39:46

Despite the rapid progress of multimodal large language models (MLLMs), they
have largely overlooked the importance of visual processing. In a simple yet
revealing experiment, we interestingly find that language-only models, when
provided with image captions, can achieve comparable or even better performance
than MLLMs that consume raw visual inputs. This suggests that current MLLMs may
generate accurate visual descriptions but fail to effectively integrate them
during reasoning. Motivated by this, we propose a simple visual perturbation
framework that enhances perceptual robustness without requiring algorithmic
modifications or additional training data. Our approach introduces three
targeted perturbations: distractor concatenation, dominance-preserving mixup,
and random rotation, that can be easily integrated into existing post-training
pipelines including SFT, DPO, and GRPO. Through extensive experiments across
multiple datasets, we demonstrate consistent improvements in mathematical
reasoning performance, with gains comparable to those achieved through
algorithmic changes. Additionally, we achieve competitive performance among
open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual
perturbation. Through comprehensive ablation studies, we analyze the
effectiveness of different perturbation strategies, revealing that each
perturbation type contributes uniquely to different aspects of visual
reasoning. Our findings highlight the critical role of visual perturbation in
multimodal mathematical reasoning: better reasoning begins with better seeing.
Our code is available at https://github.com/YutingLi0606/Vision-Matters.
Hidden in plain sight: VLMs overlook their visual representations
Authors: name: Stephanie Fu, name: Tyler Bonnen, name: Devin Guillory, name: Trevor Darrell
Published: 2025-06-09 17:59:54

Language provides a natural interface to specify and evaluate performance on
visual tasks. To realize this possibility, vision language models (VLMs) must
successfully integrate visual and linguistic information. Our work compares
VLMs to a direct readout of their visual encoders to understand their ability
to integrate across these modalities. Across a series of vision-centric
benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform
substantially worse than their visual encoders, dropping to near-chance
performance. We investigate these results through a series of analyses across
the entire VLM: namely 1) the degradation of vision representations, 2)
brittleness to task prompt, and 3) the language model's role in solving the
task. We find that the bottleneck in performing these vision-centric tasks lies
in this third category; VLMs are not effectively using visual information
easily accessible throughout the entire model, and they inherit the language
priors present in the LLM. Our work helps diagnose the failure modes of
open-source VLMs, and presents a series of evaluations useful for future
investigations into visual understanding within VLMs.
Multiverse: Your Language Models Secretly Decide How to Parallelize and
  Merge Generation
Authors: name: Xinyu Yang, name: Yuwei An, name: Hongyi Liu, name: Tianqi Chen, name: Beidi Chen
Published: 2025-06-11 17:59:23

Autoregressive Large Language Models (AR-LLMs) frequently exhibit implicit
parallelism in sequential generation. Inspired by this, we introduce
Multiverse, a new generative model that enables natively parallel generation.
Multiverse internalizes a MapReduce paradigm, generating automatically through
three stages: (i) a Map stage for adaptive task decomposition, (ii) a Process
stage for parallel subtask execution, and (iii) a Reduce stage for lossless
result synthesis. Next, we build a real-world Multiverse reasoning model with
co-design of data, algorithm, and system, enabling rapid and seamless transfer
from frontier AR-LLMs. Starting from sequential reasoning chains, we create
Multiverse 1K by converting them into structured training data using an
automated LLM-assisted pipeline, avoiding costly human annotations.
Algorithmically, we design Multiverse Attention to separate parallel reasoning
steps while keeping compatibility with causal attention for efficient training.
Systematically, we implement Multiverse Engine to enable parallel inference. It
features a dedicated scheduler that dynamically switches between sequential and
parallel generation, triggered directly by the model. After a 3-hour
fine-tuning with 1K examples, our Multiverse-32B stands as the only
open-sourced non-AR model achieving performance on par with leading AR-LLMs of
the same scale, evidenced by AIME24 & 25 scores of 54% and 46%, respectively.
Moreover, our budget control experiments show that Multiverse-32B exhibits
superior scaling, outperforming AR-LLMs by 1.87% on average using the same
context length. Such scaling further leads to practical efficiency gain,
achieving up to 2x speedup across varying batch sizes. We have open-sourced the
entire Multiverse ecosystem, including data, model weights, engine, supporting
tools, as well as complete data curation prompts and detailed training and
evaluation recipes.
Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia
  Games
Authors: name: Niv Eckhaus, name: Uri Berger, name: Gabriel Stanovsky
Published: 2025-06-05 17:53:44

LLMs are used predominantly in synchronous communication, where a human user
and a model communicate in alternating turns. In contrast, many real-world
settings are inherently asynchronous. For example, in group chats, online team
meetings, or social games, there is no inherent notion of turns; therefore, the
decision of when to speak forms a crucial part of the participant's decision
making. In this work, we develop an adaptive asynchronous LLM-agent which, in
addition to determining what to say, also decides when to say it. To evaluate
our agent, we collect a unique dataset of online Mafia games, including both
human participants, as well as our asynchronous agent. Overall, our agent
performs on par with human players, both in game performance, as well as in its
ability to blend in with the other human players. Our analysis shows that the
agent's behavior in deciding when to speak closely mirrors human patterns,
although differences emerge in message content. We release all our data and
code to support and encourage further research for more realistic asynchronous
communication between LLM agents. This work paves the way for integration of
LLMs into realistic human group settings, from assistance in team discussions
to educational and professional environments where complex social dynamics must
be navigated.
Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling
  Paradigms for Text-to-Music Generation
Authors: name: Or Tal, name: Felix Kreuk, name: Yossi Adi
Published: 2025-06-10 08:37:45

Recent progress in text-to-music generation has enabled models to synthesize
high-quality musical segments, full compositions, and even respond to
fine-grained control signals, e.g. chord progressions. State-of-the-art (SOTA)
systems differ significantly across many dimensions, such as training datasets,
modeling paradigms, and architectural choices. This diversity complicates
efforts to evaluate models fairly and pinpoint which design choices most
influence performance. While factors like data and architecture are important,
in this study we focus exclusively on the modeling paradigm. We conduct a
systematic empirical analysis to isolate its effects, offering insights into
associated trade-offs and emergent behaviors that can guide future
text-to-music generation systems. Specifically, we compare the two arguably
most common modeling paradigms: Auto-Regressive decoding and Conditional
Flow-Matching. We conduct a controlled comparison by training all models from
scratch using identical datasets, training configurations, and similar backbone
architectures. Performance is evaluated across multiple axes, including
generation quality, robustness to inference configurations, scalability,
adherence to both textual and temporally aligned conditioning, and editing
capabilities in the form of audio inpainting. This comparative study sheds
light on distinct strengths and limitations of each paradigm, providing
actionable insights that can inform future architectural and training decisions
in the evolving landscape of text-to-music generation. Audio sampled examples
are available at: https://huggingface.co/spaces/ortal1602/ARvsFM
MIRAGE: Multimodal foundation model and benchmark for comprehensive
  retinal OCT image analysis
Authors: name: José Morano, name: Botond Fazekas, name: Emese Sükei, name: Ronald Fecso, name: Taha Emre, name: Markus Gumpinger, name: Georg Faustmann, name: Marzieh Oghbaie, name: Ursula Schmidt-Erfurth, name: Hrvoje Bogunović
Published: 2025-06-10 15:25:55

Artificial intelligence (AI) has become a fundamental tool for assisting
clinicians in analyzing ophthalmic images, such as optical coherence tomography
(OCT). However, developing AI models often requires extensive annotation, and
existing models tend to underperform on independent, unseen data. Foundation
models (FMs), large AI models trained on vast unlabeled datasets, have shown
promise in overcoming these challenges. Nonetheless, available FMs for
ophthalmology lack extensive validation, especially for segmentation tasks, and
focus on a single imaging modality. In this context, we propose MIRAGE, a novel
multimodal FM for the analysis of OCT and scanning laser ophthalmoscopy (SLO)
images. Additionally, we propose a new evaluation benchmark with OCT/SLO
classification and segmentation tasks. The comparison with general and
specialized FMs and segmentation methods shows the superiority of MIRAGE in
both types of tasks, highlighting its suitability as a basis for the
development of robust AI systems for retinal OCT image analysis. Both MIRAGE
and the evaluation benchmark are publicly available:
https://github.com/j-morano/MIRAGE.
Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
Authors: name: Pengyi Li, name: Matvey Skripkin, name: Alexander Zubrey, name: Andrey Kuznetsov, name: Ivan Oseledets
Published: 2025-06-05 19:55:15

Large language models (LLMs) excel at reasoning, yet post-training remains
critical for aligning their behavior with task goals. Existing reinforcement
learning (RL) methods often depend on costly human annotations or external
reward models. We propose Reinforcement Learning via Self-Confidence (RLSC),
which uses the model's own confidence as reward signals-eliminating the need
for labels, preference models, or reward engineering. Applied to
Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps,
RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on
Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a
simple, scalable post-training method for inference models, requiring only a
small number of samples and unlabelled supervision.
Autoregressive Adversarial Post-Training for Real-Time Interactive Video
  Generation
Authors: name: Shanchuan Lin, name: Ceyuan Yang, name: Hao He, name: Jianwen Jiang, name: Yuxi Ren, name: Xin Xia, name: Yang Zhao, name: Xuefeng Xiao, name: Lu Jiang
Published: 2025-06-11 03:04:23

Existing large-scale video generation models are computationally intensive,
preventing adoption in real-time and interactive applications. In this work, we
propose autoregressive adversarial post-training (AAPT) to transform a
pre-trained latent video diffusion model into a real-time, interactive video
generator. Our model autoregressively generates a latent frame at a time using
a single neural function evaluation (1NFE). The model can stream the result to
the user in real time and receive interactive responses as controls to generate
the next latent frame. Unlike existing approaches, our method explores
adversarial training as an effective paradigm for autoregressive generation.
This not only allows us to design an architecture that is more efficient for
one-step generation while fully utilizing the KV cache, but also enables
training the model in a student-forcing manner that proves to be effective in
reducing error accumulation during long video generation. Our experiments
demonstrate that our 8B model achieves real-time, 24fps, streaming video
generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to
a minute long (1440 frames). Visit our research website at
https://seaweed-apt.com/2
Reparameterized LLM Training via Orthogonal Equivalence Transformation
Authors: name: Zeju Qiu, name: Simon Buchholz, name: Tim Z. Xiao, name: Maximilian Dax, name: Bernhard Schölkopf, name: Weiyang Liu
Published: 2025-06-09 17:59:34

While large language models (LLMs) are driving the rapid advancement of
artificial intelligence, effectively and reliably training these large models
remains one of the field's most significant challenges. To address this
challenge, we propose POET, a novel reParameterized training algorithm that
uses Orthogonal Equivalence Transformation to optimize neurons. Specifically,
POET reparameterizes each neuron with two learnable orthogonal matrices and a
fixed random weight matrix. Because of its provable preservation of spectral
properties of weight matrices, POET can stably optimize the objective function
with improved generalization. We further develop efficient approximations that
make POET flexible and scalable for training large-scale neural networks.
Extensive experiments validate the effectiveness and scalability of POET in
training LLMs.