Orionnnn/hub-stats huggingface dataset:
NOSA: Native and Offloadable Sparse Attention
Authors: name: Yuxiang Huang, name: Chaojun Xiao, name: Xu Han, name: Zhiyuan Liu
Published: 2025-10-15 14:33:16

Trainable sparse attention has emerged as a promising solution to address the
decoding efficiency bottleneck of LLMs in long-context processing,
significantly saving memory accesses while minimally impacting task
performance. However, existing sparse attention methods leave a crucial
limitation unresolved: the size of the key-value (KV) cache remains unreduced,
which constrains on-GPU batch sizes and throttles decoding throughput,
especially in large-scale batched inference. In this paper, we show that
trainable sparse attention naturally exhibits strong locality in token
selection across adjacent decoding steps, thereby enabling KV cache offloading
without altering the underlying attention computation. However, the inherent
locality remains insufficient to achieve efficient offloading, as the transfer
of selected KV pairs between the CPU and GPU continues to dominate the overall
decoding cost. Building on this insight, we present NOSA, a trainable sparse
attention framework designed to natively support KV cache offloading. NOSA
introduces explicit locality constraints by decomposing token selection into
query-aware and query-agnostic components, thereby reducing KV transfers while
preserving the same attention computation as used during training. We pretrain
a 1B-parameter model with NOSA and conduct extensive benchmarks, showing that
it preserves near-lossless performance while achieving up to a 2.3x improvement
in decoding throughput compared with the vanilla trainable sparse attention
baseline (InfLLM-V2).
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn
  Dialogue
Authors: name: Wenwen Tong, name: Hewei Guo, name: Dongchuan Ran, name: Jiangnan Chen, name: Jiefan Lu, name: Kaibin Wang, name: Keqiang Li, name: Xiaoxu Zhu, name: Jiakui Li, name: Kehan Li, name: Xueheng Li, name: Lumin Li, name: Chenxu Guo, name: Jiasheng Zhou, name: Jiandong Chen, name: Xianye Wu, name: Jiahao Wang, name: Silei Wu, name: Lei Chen, name: Hanming Deng, name: Yuxuan Song, name: Dinghao Zhou, name: Guiping Zhong, name: Ken Zheng, name: Shiyin Kang, name: Lewei Lu
Published: 2025-10-15 16:52:48

We introduce InteractiveOmni, a unified and open-source omni-modal large
language model for audio-visual multi-turn interaction, ranging from 4B to 8B
parameters, designed to lead the field of lightweight models by offering
comprehensive omni-modal understanding and speech generation capabilities. To
achieve this, we integrate the vision encoder, audio encoder, large language
model, and speech decoder into a unified model for understanding and generation
tasks. We design a multi-stage training strategy to ensure robust cross-modal
capabilities, including pre-training for omni-modal understanding, followed by
post-training with speech conversation and audio-visual interaction. To enable
human-like long-term conversational ability, we meticulously curate a
multi-turn training dataset that enhances the model's ability to handle complex
and multi-turn interactions. To effectively evaluate the multi-turn memory and
speech interaction capabilities, we construct the multi-modal multi-turn memory
benchmark and the multi-turn speech interaction benchmark. Experiments
demonstrate that InteractiveOmni significantly outperforms leading open-source
models and provides a more intelligent multi-turn audio-visual experience,
particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B
is comparable to the much larger model like Qwen2.5-Omni-7B on general
benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B
while utilizing only 50% of the model size. Achieving state-of-the-art results
against similarly sized models across image, audio, video understanding, and
speech generation tasks, InteractiveOmni is an accessible, open-source
foundation for next-generation intelligent interactive systems.
UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity
  MoE
Authors: name: Zhenyu Liu, name: Yunxin Li, name: Xuanyu Zhang, name: Qixun Teng, name: Shenyuan Jiang, name: Xinyu Chen, name: Haoyuan Shi, name: Jinchao Li, name: Qi Wang, name: Haolan Chen, name: Fanbo Meng, name: Mingjun Zhao, name: Yu Xu, name: Yancheng He, name: Baotian Hu, name: Min Zhang
Published: 2025-10-15 09:30:25

Recent advances in unified multimodal models indicate a clear trend towards
comprehensive content generation. However, the auditory domain remains a
significant challenge, with music and speech often developed in isolation,
hindering progress towards universal audio synthesis. This separation stems
from inherent task conflicts and severe data imbalances, which impede the
development of a truly unified audio generation model. To address this
challenge, we propose UniMoE-Audio, a unified speech and music generation model
within a novel Dynamic-Capacity Mixture-of-Experts (MoE) framework.
Architecturally, UniMoE-Audio introduces a Top-P routing strategy for dynamic
expert number allocation, and a hybrid expert design comprising routed experts
for domain-specific knowledge, shared experts for domain-agnostic features, and
null experts for adaptive computation skipping. To tackle data imbalance, we
introduce a three-stage training curriculum: 1) Independent Specialist Training
leverages original datasets to instill domain-specific knowledge into each
"proto-expert" without interference; 2) MoE Integration and Warmup incorporates
these specialists into the UniMoE-Audio architecture, warming up the gate
module and shared expert using a subset of balanced dataset; and 3) Synergistic
Joint Training trains the entire model end-to-end on the fully balanced
dataset, fostering enhanced cross-domain synergy. Extensive experiments show
that UniMoE-Audio not only achieves state-of-the-art performance on major
speech and music generation benchmarks, but also demonstrates superior
synergistic learning, mitigating the performance degradation typically seen in
naive joint training. Our findings highlight the substantial potential of
specialized MoE architecture and curated training strategies in advancing the
field of universal audio generation. Homepage:
https://mukioxun.github.io/Uni-MoE-site/home.html
ParallelBench: Understanding the Trade-offs of Parallel Decoding in
  Diffusion LLMs
Authors: name: Wonjun Kang, name: Kevin Galim, name: Seunghyuk Oh, name: Minjae Lee, name: Yuchen Zeng, name: Shuibai Zhang, name: Coleman Hooper, name: Yuezhou Hu, name: Hyung Il Koo, name: Nam Ik Cho, name: Kangwook Lee
Published: 2025-10-06 12:41:31

While most autoregressive LLMs are constrained to one-by-one decoding,
diffusion LLMs (dLLMs) have attracted growing interest for their potential to
dramatically accelerate inference through parallel decoding. Despite this
promise, the conditional independence assumption in dLLMs causes parallel
decoding to ignore token dependencies, inevitably degrading generation quality
when these dependencies are strong. However, existing works largely overlook
these inherent challenges, and evaluations on standard benchmarks (e.g., math
and coding) are not sufficient to capture the quality degradation caused by
parallel decoding. To address this gap, we first provide an
information-theoretic analysis of parallel decoding. We then conduct case
studies on analytically tractable synthetic list operations from both data
distribution and decoding strategy perspectives, offering quantitative insights
that highlight the fundamental limitations of parallel decoding. Building on
these insights, we propose ParallelBench, the first benchmark specifically
designed for dLLMs, featuring realistic tasks that are trivial for humans and
autoregressive LLMs yet exceptionally challenging for dLLMs under parallel
decoding. Using ParallelBench, we systematically analyze both dLLMs and
autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can
suffer dramatic quality degradation in real-world scenarios, and (ii) current
parallel decoding strategies struggle to adapt their degree of parallelism
based on task difficulty, thus failing to achieve meaningful speedup without
compromising quality. Our findings underscore the pressing need for innovative
decoding methods that can overcome the current speed-quality trade-off. We
release our benchmark to help accelerate the development of truly efficient
dLLMs.
CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in
  Latent World Models for Autonomous Driving
Authors: name: Xiaoji Zheng, name: Ziyuan Yang, name: Yanhao Chen, name: Yuhang Peng, name: Yuanrong Tang, name: Gengyuan Liu, name: Bokui Chen, name: Jiangtao Gong
Published: 2025-10-14 14:21:52

End-to-end autonomous driving models trained solely with imitation learning
(IL) often suffer from poor generalization. In contrast, reinforcement learning
(RL) promotes exploration through reward maximization but faces challenges such
as sample inefficiency and unstable convergence. A natural solution is to
combine IL and RL. Moving beyond the conventional two-stage paradigm (IL
pretraining followed by RL fine-tuning), we propose CoIRL-AD, a competitive
dual-policy framework that enables IL and RL agents to interact during
training. CoIRL-AD introduces a competition-based mechanism that facilitates
knowledge exchange while preventing gradient conflicts. Experiments on the
nuScenes dataset show an 18% reduction in collision rate compared to baselines,
along with stronger generalization and improved performance on long-tail
scenarios. Code is available at: https://github.com/SEU-zxj/CoIRL-AD.
MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic
  Training
Authors: name: Taicheng Guo, name: Hai Wang, name: ChaoChun Liu, name: Mohsen Golalikhani, name: Xin Chen, name: Xiangliang Zhang, name: Chandan K. Reddy
Published: 2025-10-12 16:12:05

Multi-turn Text-to-SQL aims to translate a user's conversational utterances
into executable SQL while preserving dialogue coherence and grounding to the
target schema. However, most existing systems only regard this task as a simple
text translation task and follow a short-horizon paradigm, generating a query
per turn without execution, explicit verification, and refinement, which leads
to non-executable or incoherent outputs. We present MTSQL-R1, an agentic
training framework for long-horizon multi-turn Text-to-SQL. We cast the task as
a Markov Decision Process (MDP) in which an agent interacts with (i) a database
for execution feedback and (ii) a persistent dialogue memory for coherence
verification, performing an iterative propose to execute -> verify -> refine
cycle until all checks pass. Experiments on COSQL and SPARC demonstrate that
MTSQL-R1 consistently outperforms strong baselines, highlighting the importance
of environment-driven verification and memory-guided refinement for
conversational semantic parsing. Full recipes (including code, trained models,
logs, reasoning trajectories, etc.) will be released after the internal review
to contribute to community research.
GraphTracer: Graph-Guided Failure Tracing in LLM Agents for Robust
  Multi-Turn Deep Search
Authors: name: Heng Zhang, name: Yuling Shi, name: Xiaodong Gu, name: Haochen You, name: Zijian Zhang, name: Lubin Gan, name: Yilei Yuan, name: Jin Huang
Published: 2025-10-12 12:55:42

Multi-agent systems powered by Large Language Models excel at complex tasks
through coordinated collaboration, yet they face high failure rates in
multi-turn deep search scenarios. Existing temporal attribution methods
struggle to accurately diagnose root causes, particularly when errors propagate
across multiple agents. Attempts to automate failure attribution by analyzing
action sequences remain ineffective due to their inability to account for
information dependencies that span agents. This paper identifies two core
challenges: (i) distinguishing symptoms from root causes in multi-agent
error propagation, and (ii) tracing information dependencies beyond
temporal order. To address these issues, we introduce GraphTracer, a
framework that redefines failure attribution through information flow analysis.
GraphTracer constructs Information Dependency Graphs (IDGs) to explicitly
capture how agents reference and build on prior outputs. It localizes root
causes by tracing through these dependency structures instead of relying on
temporal sequences. GraphTracer also uses graph-aware synthetic data generation
to target critical nodes, creating realistic failure scenarios. Evaluations on
the Who\&When benchmark and integration into production systems demonstrate
that GraphTracer-8B achieves up to 18.18\% higher attribution accuracy compared
to state-of-the-art models and enables 4.8\% to 14.2\% performance improvements
in deployed multi-agent frameworks, establishing a robust solution for
multi-agent system debugging.
HyperAgent: Leveraging Hypergraphs for Topology Optimization in
  Multi-Agent Communication
Authors: name: Heng Zhang, name: Yuling Shi, name: Xiaodong Gu, name: Zijian Zhang, name: Haochen You, name: Lubin Gan, name: Yilei Yuan, name: Jin Huang
Published: 2025-10-12 13:47:42

Recent advances in large language model-powered multi-agent systems have
demonstrated remarkable collective intelligence through effective
communication. However, existing approaches face two primary challenges: (i)
Ineffective group collaboration modeling, as they rely on pairwise
edge representations in graph structures, limiting their ability to capture
relationships among multiple agents; and (ii) Limited task-adaptiveness
in communication topology design, leading to excessive communication cost for
simple tasks and insufficient coordination for complex scenarios. These issues
restrict the scalability and practical deployment of adaptive collaboration
frameworks. To address these challenges, we propose HyperAgent, a
hypergraph-based framework that optimizes communication topologies and
effectively captures group collaboration patterns using direct hyperedge
representations. Unlike edge-based approaches, HyperAgent uses hyperedges to
link multiple agents within the same subtask and employs hypergraph
convolutional layers to achieve one-step information aggregation in
collaboration groups. Additionally, it incorporates a variational autoencoder
framework with sparsity regularization to dynamically adjust hypergraph
topologies based on task complexity. Experiments highlight the superiority of
HyperAgent in both performance and efficiency. For instance, on GSM8K,
HyperAgent achieves 95.07\% accuracy while reducing token consumption by
25.33\%, demonstrating the potential of hypergraph-based optimization for
multi-agent communication.
Revisiting Model Interpolation for Efficient Reasoning
Authors: name: Taiqiang Wu, name: Runming Yang, name: Tao Liu, name: Jiahao Wang, name: Ngai Wong
Published: 2025-10-13 03:30:01

Model merging, typically on Instruct and Thinking models, has shown
remarkable performance for efficient reasoning. In this paper, we
systematically revisit the simplest merging method that interpolates two
weights directly. Particularly, we observe that model interpolation follows a
three-stage evolutionary paradigm with distinct behaviors on the reasoning
trajectory. These dynamics provide a principled guide for navigating the
performance-cost trade-off. Empirical results demonstrate that a strategically
interpolated model surprisingly surpasses sophisticated model merging baselines
on both efficiency and effectiveness. We further validate our findings with
extensive ablation studies on model layers, modules, and decoding strategies.
Ultimately, this work demystifies model interpolation and offers a practical
framework for crafting models with precisely targeted reasoning capabilities.
Code is available at https://github.com/wutaiqiang/MI{Github}.
InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for
  Generalist Robot Policy
Authors: name: Xinyi Chen, name: Yilun Chen, name: Yanwei Fu, name: Ning Gao, name: Jiaya Jia, name: Weiyang Jin, name: Hao Li, name: Yao Mu, name: Jiangmiao Pang, name: Yu Qiao, name: Yang Tian, name: Bin Wang, name: Bolun Wang, name: Fangjing Wang, name: Hanqing Wang, name: Tai Wang, name: Ziqin Wang, name: Xueyuan Wei, name: Chao Wu, name: Shuai Yang, name: Jinhui Ye, name: Junqiu Yu, name: Jia Zeng, name: Jingjing Zhang, name: Jinyu Zhang, name: Shi Zhang, name: Feng Zheng, name: Bowen Zhou, name: Yangkun Zhu
Published: 2025-10-15 17:30:05

We introduce InternVLA-M1, a unified framework for spatial grounding and
robot control that advances instruction-following robots toward scalable,
general-purpose intelligence. Its core idea is spatially guided
vision-language-action training, where spatial grounding serves as the critical
link between instructions and robot actions. InternVLA-M1 employs a two-stage
pipeline: (i) spatial grounding pre-training on over 2.3M spatial reasoning
data to determine ``where to act'' by aligning instructions with visual,
embodiment-agnostic positions, and (ii) spatially guided action post-training
to decide ``how to act'' by generating embodiment-aware actions through
plug-and-play spatial prompting. This spatially guided training recipe yields
consistent gains: InternVLA-M1 outperforms its variant without spatial guidance
by +14.6% on SimplerEnv Google Robot, +17% on WidowX, and +4.3% on LIBERO
Franka, while demonstrating stronger spatial reasoning capability in box,
point, and trace prediction. To further scale instruction following, we built a
simulation engine to collect 244K generalizable pick-and-place episodes,
enabling a 6.2% average improvement across 200 tasks and 3K+ objects. In
real-world clustered pick-and-place, InternVLA-M1 improved by 7.3%, and with
synthetic co-training, achieved +20.6% on unseen objects and novel
configurations. Moreover, in long-horizon reasoning-intensive scenarios, it
surpassed existing works by over 10%. These results highlight spatially guided
training as a unifying principle for scalable and resilient generalist robots.
Code and models are available at
https://github.com/InternRobotics/InternVLA-M1.
Direct Multi-Token Decoding
Authors: name: Xuan Luo, name: Weizhi Wang, name: Xifeng Yan
Published: 2025-10-13 21:42:37

Decoder-only transformers have become the standard architecture for large
language models (LLMs) due to their strong performance. Recent studies suggest
that, in pre-trained LLMs, early, middle, and late layers may serve distinct
roles: Early layers focus on understanding the input context, middle layers
handle task-specific processing, and late layers convert abstract
representations into output tokens. We hypothesize that once representations
have been processed by the early and middle layers, the resulting hidden states
may encapsulate sufficient information to support the generation of multiple
tokens using only the late layers, eliminating the need to repeatedly traverse
the early and middle layers. We refer to this inference paradigm as Direct
Multi-Token Decoding (DMTD). Unlike speculative decoding, our method introduces
no additional parameters, auxiliary routines, or post-generation verification.
Despite being trained on a limited dataset, a fine-tuned DMTD Qwen3-4B model
has already demonstrated promising results, achieving up to a 2x speedup with
only minor performance loss. Moreover, as shown in our scaling analysis, its
performance is expected to further improve with larger training datasets.
UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
Authors: name: Tiancheng Gu, name: Kaicheng Yang, name: Kaichen Zhang, name: Xiang An, name: Ziyong Feng, name: Yueyi Zhang, name: Weidong Cai, name: Jiankang Deng, name: Lidong Bing
Published: 2025-10-15 13:07:00

Universal multimodal embedding models are foundational to various tasks.
Existing approaches typically employ in-batch negative mining by measuring the
similarity of query-candidate pairs. However, these methods often struggle to
capture subtle semantic differences among candidates and lack diversity in
negative samples. Moreover, the embeddings exhibit limited discriminative
ability in distinguishing false and hard negatives. In this paper, we leverage
the advanced understanding capabilities of MLLMs to enhance representation
learning and present a novel Universal Multimodal Embedding (UniME-V2) model.
Our approach first constructs a potential hard negative set through global
retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes
MLLMs to assess the semantic alignment of query-candidate pairs and generate
soft semantic matching scores. These scores serve as a foundation for hard
negative mining, mitigating the impact of false negatives and enabling the
identification of diverse, high-quality hard negatives. Furthermore, the
semantic matching scores are used as soft labels to mitigate the rigid
one-to-one mapping constraint. By aligning the similarity matrix with the soft
semantic matching score matrix, the model learns semantic distinctions among
candidates, significantly enhancing its discriminative capacity. To further
improve performance, we propose UniME-V2-Reranker, a reranking model trained on
our mined hard negatives through a joint pairwise and listwise optimization
approach. We conduct comprehensive experiments on the MMEB benchmark and
multiple retrieval tasks, demonstrating that our method achieves
state-of-the-art performance on average across all tasks.
FlashWorld: High-quality 3D Scene Generation within Seconds
Authors: name: Xinyang Li, name: Tengfei Wang, name: Zixiao Gu, name: Shengchuan Zhang, name: Chunchao Guo, name: Liujuan Cao
Published: 2025-10-15 15:35:48

We propose FlashWorld, a generative model that produces 3D scenes from a
single image or text prompt in seconds, 10~100times faster than previous
works while possessing superior rendering quality. Our approach shifts from the
conventional multi-view-oriented (MV-oriented) paradigm, which generates
multi-view images for subsequent 3D reconstruction, to a 3D-oriented approach
where the model directly produces 3D Gaussian representations during multi-view
generation. While ensuring 3D consistency, 3D-oriented method typically suffers
poor visual quality. FlashWorld includes a dual-mode pre-training phase
followed by a cross-mode post-training phase, effectively integrating the
strengths of both paradigms. Specifically, leveraging the prior from a video
diffusion model, we first pre-train a dual-mode multi-view diffusion model,
which jointly supports MV-oriented and 3D-oriented generation modes. To bridge
the quality gap in 3D-oriented generation, we further propose a cross-mode
post-training distillation by matching distribution from consistent 3D-oriented
mode to high-quality MV-oriented mode. This not only enhances visual quality
while maintaining 3D consistency, but also reduces the required denoising steps
for inference. Also, we propose a strategy to leverage massive single-view
images and text prompts during this process to enhance the model's
generalization to out-of-distribution inputs. Extensive experiments demonstrate
the superiority and efficiency of our method.