Orionnnn/hub-stats huggingface dataset:
Native Hybrid Attention for Efficient Sequence Modeling
Authors: name: Jusen Du, name: Jiaxi Hu, name: Tao Zhang, name: Weigao Sun, name: Yu Cheng
Published: 2025-10-08 13:44:57

Transformers excel at sequence modeling but face quadratic complexity, while
linear attention offers improved efficiency but often compromises recall
accuracy over long contexts. In this work, we introduce Native Hybrid Attention
(NHA), a novel hybrid architecture of linear and full attention that integrates
both intra \& inter-layer hybridization into a unified layer design. NHA
maintains long-term context in key-value slots updated by a linear RNN, and
augments them with short-term tokens from a sliding window. A single
softmax attention operation is then applied over all keys and values,
enabling per-token and per-head context-dependent weighting without requiring
additional fusion parameters. The inter-layer behavior is controlled through a
single hyperparameter, the sliding window size, which allows smooth adjustment
between purely linear and full attention while keeping all layers structurally
uniform. Experimental results show that NHA surpasses Transformers and other
hybrid baselines on recall-intensive and commonsense reasoning tasks.
Furthermore, pretrained LLMs can be structurally hybridized with NHA, achieving
competitive accuracy while delivering significant efficiency gains. Code is
available at https://github.com/JusenD/NHA.
Ming-UniVision: Joint Image Understanding and Generation with a Unified
  Continuous Tokenizer
Authors: name: Ziyuan Huang, name: DanDan Zheng, name: Cheng Zou, name: Rui Liu, name: Xiaolong Wang, name: Kaixiang Ji, name: Weilong Chai, name: Jianxin Sun, name: Libin Wang, name: Yongjie Lv, name: Taozhi Huang, name: Jiajia Liu, name: Qingpei Guo, name: Ming Yang, name: Jingdong Chen, name: Jun Zhou
Published: 2025-10-08 02:50:14

Visual tokenization remains a core challenge in unifying visual understanding
and generation within the autoregressive paradigm. Existing methods typically
employ tokenizers in discrete latent spaces to align with the tokens from large
language models, where the quantization errors can limit semantic
expressiveness and degrade the capability of vision-language understanding. To
address this, we introduce MingTok, a new family of visual tokenizers with a
continuous latent space, for unified autoregressive generation and
understanding. While understanding tasks favor discriminative high-dimensional
features, generation tasks prefer compact low-level codes. Thus, to reconcile
these competing demands, MingTok adopts a three-stage sequential architecture
involving low-level encoding, semantic expansion, and visual reconstruction.
Built on top of it, Ming-UniVision eliminates the need for task-specific visual
representations, and unifies diverse vision-language tasks under a single
autoregrsssive prediction paradigm. By formulating both understanding and
generation as next-token prediction in a shared continuous space, it seamlessly
supports multi-round, in-context tasks such as iterative understanding,
generation and editing. Empirically, we find that using a unified continuous
visual representation reconciles the competing requirements on the tokenizers
by the understanding and generation tasks, thereby leading to state-of-the-art
level performance across both domains. We hope our findings will facilitate
unified visual tokenization in the continuous domain. Inference code and model
weights are released to benefit community.
RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
Authors: name: Hongzhi Zang, name: Mingjie Wei, name: Si Xu, name: Yongji Wu, name: Zhen Guo, name: Yuanqing Wang, name: Hao Lin, name: Liangzhi Shi, name: Yuqing Xie, name: Zhexuan Xu, name: Zhihao Liu, name: Kang Chen, name: Wenhao Tang, name: Quanlu Zhang, name: Weinan Zhang, name: Chao Yu, name: Yu Wang
Published: 2025-10-08 07:05:13

Recent progress in vision and language foundation models has significantly
advanced multimodal understanding, reasoning, and generation, inspiring a surge
of interest in extending such capabilities to embodied settings through
vision-language-action (VLA) models. Yet, most VLA models are still trained
with supervised fine-tuning (SFT), which struggles to generalize under
distribution shifts due to error accumulation. Reinforcement learning (RL)
offers a promising alternative by directly optimizing task performance through
interaction, but existing attempts remain fragmented and lack a unified
platform for fair and systematic comparison across model architectures and
algorithmic designs. To address this gap, we introduce RLinf-VLA, a unified and
efficient framework for scalable RL training of VLA models. The system adopts a
highly flexible resource allocation design that addresses the challenge of
integrating rendering, training, and inference in RL+VLA training. In
particular, for GPU-parallelized simulators, RLinf-VLA implements a novel
hybrid fine-grained pipeline allocation mode, achieving a 1.61x-1.88x speedup
in training. Through a unified interface, RLinf-VLA seamlessly supports diverse
VLA architectures (e.g., OpenVLA, OpenVLA-OFT), multiple RL algorithms (e.g.,
PPO, GRPO), and various simulators (e.g., ManiSkill, LIBERO). In simulation, a
unified model achieves 98.11\% across 130 LIBERO tasks and 97.66\% across 25
ManiSkill tasks. Beyond empirical performance, our study distills a set of best
practices for applying RL to VLA training and sheds light on emerging patterns
in this integration. Furthermore, we present preliminary deployment on a
real-world Franka robot, where RL-trained policies exhibit stronger
generalization than those trained with SFT. We envision RLinf-VLA as a
foundation to accelerate and standardize research on embodied intelligence.
Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in
  MLLMs
Authors: name: Yongyi Su, name: Haojie Zhang, name: Shijie Li, name: Nanqing Liu, name: Jingyi Liao, name: Junyi Pan, name: Yuan Liu, name: Xiaofen Xing, name: Chong Sun, name: Chen Li, name: Nancy F. Chen, name: Shuicheng Yan, name: Xulei Yang, name: Xun Xu
Published: 2025-10-02 12:23:57

Multimodal large language models (MLLMs) have advanced rapidly in recent
years. However, existing approaches for vision tasks often rely on indirect
representations, such as generating coordinates as text for detection, which
limits performance and prevents dense prediction tasks like segmentation. To
overcome these challenges, we introduce Patch-as-Decodable Token (PaDT), a
unified paradigm that enables MLLMs to directly generate both textual and
diverse visual outputs. Central to PaDT are Visual Reference Tokens (VRTs),
derived from visual patch embeddings of query images and interleaved seamlessly
with LLM's output textual tokens. A lightweight decoder then transforms LLM's
outputs into detection, segmentation, and grounding predictions. Unlike prior
methods, PaDT processes VRTs independently at each forward pass and dynamically
expands the embedding table, thus improving localization and differentiation
among similar objects. We further tailor a training strategy for PaDT by
randomly selecting VRTs for supervised fine-tuning and introducing a robust
per-token cross-entropy loss. Our empirical studies across four visual
perception and understanding tasks suggest PaDT consistently achieving
state-of-the-art performance, even compared with significantly larger MLLM
models. The code is available at https://github.com/Gorilla-Lab-SCUT/PaDT.
The African Languages Lab: A Collaborative Approach to Advancing
  Low-Resource African NLP
Authors: name: Sheriff Issaka, name: Keyi Wang, name: Yinka Ajibola, name: Oluwatumininu Samuel-Ipaye, name: Zhaoyi Zhang, name: Nicte Aguillon Jimenez, name: Evans Kofi Agyei, name: Abraham Lin, name: Rohan Ramachandran, name: Sadick Abdul Mumin, name: Faith Nchifor, name: Mohammed Shuraim, name: Lieqi Liu, name: Erick Rosas Gonzalez, name: Sylvester Kpei, name: Jemimah Osei, name: Carlene Ajeneza, name: Persis Boateng, name: Prisca Adwoa Dufie Yeboah, name: Saadia Gabriel
Published: 2025-10-07 07:42:52

Despite representing nearly one-third of the world's languages, African
languages remain critically underserved by modern NLP technologies, with 88\%
classified as severely underrepresented or completely ignored in computational
linguistics. We present the African Languages Lab (All Lab), a comprehensive
research initiative that addresses this technological gap through systematic
data collection, model development, and capacity building. Our contributions
include: (1) a quality-controlled data collection pipeline, yielding the
largest validated African multi-modal speech and text dataset spanning 40
languages with 19 billion tokens of monolingual text and 12,628 hours of
aligned speech data; (2) extensive experimental validation demonstrating that
our dataset, combined with fine-tuning, achieves substantial improvements over
baseline models, averaging +23.69 ChrF++, +0.33 COMET, and +15.34 BLEU points
across 31 evaluated languages; and (3) a structured research program that has
successfully mentored fifteen early-career researchers, establishing
sustainable local capacity. Our comparative evaluation against Google Translate
reveals competitive performance in several languages while identifying areas
that require continued development.
NorMuon: Making Muon more efficient and scalable
Authors: name: Zichong Li, name: Liming Liu, name: Chen Liang, name: Weizhu Chen, name: Tuo Zhao
Published: 2025-10-07 01:13:41

The choice of optimizer significantly impacts the training efficiency and
computational costs of large language models (LLMs). Recently, the Muon
optimizer has demonstrated promising results by orthogonalizing parameter
updates, improving optimization geometry through better conditioning. Despite
Muon's emergence as a candidate successor to Adam, the potential for jointly
leveraging their strengths has not been systematically explored. In this work,
we bridge this gap by proposing NorMuon (Neuron-wise Normalized Muon), an
optimizer that synergistically combines orthogonalization with neuron-level
adaptive learning rates. Our analysis reveals that while Muon effectively
reduces condition numbers, the resulting updates exhibit highly non-uniform
neuron norms, causing certain neurons to dominate the optimization process.
NorMuon addresses this imbalance by maintaining second-order momentum
statistics for each neuron and applying row-wise normalization after
orthogonalization, ensuring balanced parameter utilization while preserving
Muon's conditioning benefits. To enable practical deployment at scale, we
develop an efficient distributed implementation under the FSDP2 framework that
strategically distributes orthogonalization computations across devices.
Experiments across multiple model scales demonstrate that NorMuon consistently
outperforms both Adam and Muon, achieving 21.74% better training efficiency
than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting, while
maintaining a comparable memory footprint to Muon. Our findings suggest that
orthogonalization and adaptive learning rates are complementary rather than
competing approaches, opening new avenues for optimizer design in large-scale
deep learning.
Heptapod: Language Modeling on Visual Signals
Authors: name: Yongxin Zhu, name: Jiawei Chen, name: Yuanzhe Chen, name: Zhuo Chen, name: Dongya Jia, name: Jian Cong, name: Xiaobin Zhuang, name: Yuping Wang, name: Yuxuan Wang
Published: 2025-10-08 05:54:46

We introduce Heptapod, an image autoregressive model that adheres to the
foundational principles of language modeling. Heptapod employs causal
attention, eliminates reliance on CFG, and eschews the trend
of semantic tokenizers. Our key innovation is next 2D distribution
prediction: a causal Transformer with reconstruction-focused visual tokenizer,
learns to predict the distribution over the entire 2D spatial grid of images at
each timestep. This learning objective unifies the sequential modeling of
autoregressive framework with the holistic self-supervised learning of masked
autoencoding, enabling the model to capture comprehensive image semantics via
generative training. On the ImageNet generation benchmark, Heptapod achieves an
FID of 2.70, significantly outperforming previous causal autoregressive
approaches. We hope our work inspires a principled rethinking of language
modeling on visual signals and beyond.
StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact
  State Representation
Authors: name: Mingyu Liu, name: Jiuhe Shu, name: Hui Chen, name: Zeju Li, name: Canyu Zhao, name: Jiange Yang, name: Shenyuan Gao, name: Hao Chen, name: Chunhua Shen
Published: 2025-10-06 17:37:24

A fundamental challenge in embodied intelligence is developing expressive and
compact state representations for efficient world modeling and decision making.
However, existing methods often fail to achieve this balance, yielding
representations that are either overly redundant or lacking in task-critical
information. We propose an unsupervised approach that learns a highly
compressed two-token state representation using a lightweight encoder and a
pre-trained Diffusion Transformer (DiT) decoder, capitalizing on its strong
generative prior. Our representation is efficient, interpretable, and
integrates seamlessly into existing VLA-based models, improving performance by
14.3% on LIBERO and 30% in real-world task success with minimal inference
overhead. More importantly, we find that the difference between these tokens,
obtained via latent interpolation, naturally serves as a highly effective
latent action, which can be further decoded into executable robot actions. This
emergent capability reveals that our representation captures structured
dynamics without explicit supervision. We name our method StaMo for its ability
to learn generalizable robotic Motion from compact State representation, which
is encoded from static images, challenging the prevalent dependence to learning
latent action on complex architectures and video data. The resulting latent
actions also enhance policy co-training, outperforming prior methods by 10.4%
with improved interpretability. Moreover, our approach scales effectively
across diverse data sources, including real-world robot data, simulation, and
human egocentric video.
Online Generic Event Boundary Detection
Authors: name: Hyungrok Jung, name: Daneul Kim, name: Seunggyun Lim, name: Jeany Son, name: Jonghyun Choi
Published: 2025-10-08 10:23:45

Generic Event Boundary Detection (GEBD) aims to interpret long-form videos
through the lens of human perception. However, current GEBD methods require
processing complete video frames to make predictions, unlike humans processing
data online and in real-time. To bridge this gap, we introduce a new task,
Online Generic Event Boundary Detection (On-GEBD), aiming to detect boundaries
of generic events immediately in streaming videos. This task faces unique
challenges of identifying subtle, taxonomy-free event changes in real-time,
without the access to future frames. To tackle these challenges, we propose a
novel On-GEBD framework, Estimator, inspired by Event Segmentation Theory (EST)
which explains how humans segment ongoing activity into events by leveraging
the discrepancies between predicted and actual information. Our framework
consists of two key components: the Consistent Event Anticipator (CEA), and the
Online Boundary Discriminator (OBD). Specifically, the CEA generates a
prediction of the future frame reflecting current event dynamics based solely
on prior frames. Then, the OBD measures the prediction error and adaptively
adjusts the threshold using statistical tests on past errors to capture
diverse, subtle event transitions. Experimental results demonstrate that
Estimator outperforms all baselines adapted from recent online video
understanding models and achieves performance comparable to prior offline-GEBD
methods on the Kinetics-GEBD and TAPOS datasets.
Are We Using the Right Benchmark: An Evaluation Framework for Visual
  Token Compression Methods
Authors: name: Chenfei Liao, name: Wensong Wang, name: Zichen Wen, name: Xu Zheng, name: Yiyu Wang, name: Haocong He, name: Yuanhuiyi Lyu, name: Lutao Jiang, name: Xin Zou, name: Yuqian Fu, name: Bin Ren, name: Linfeng Zhang, name: Xuming Hu
Published: 2025-10-08 15:44:28

Recent endeavors to accelerate inference in Multimodal Large Language Models
(MLLMs) have primarily focused on visual token compression. The effectiveness
of these methods is typically assessed by measuring the accuracy drop on
established benchmarks, comparing model performance before and after
compression. However, these benchmarks are originally designed to assess the
perception and reasoning capabilities of MLLMs, rather than to evaluate
compression techniques. As a result, directly applying them to visual token
compression introduces a task mismatch. Strikingly, our investigation reveals
that simple image downsampling consistently outperforms many advanced
compression methods across multiple widely used benchmarks. Through extensive
experiments, we make the following observations: (i) Current benchmarks are
noisy for the visual token compression task. (ii) Down-sampling is able to
serve as a data filter to evaluate the difficulty of samples in the visual
token compression task. Motivated by these findings, we introduce VTC-Bench, an
evaluation framework that incorporates a data filtering mechanism to denoise
existing benchmarks, thereby enabling fairer and more accurate assessment of
visual token compression methods. All data and code are available at
https://github.com/Chenfei-Liao/VTC-Bench.
Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era
  of Large Language Models
Authors: name: Rajvee Sheth, name: Samridhi Raj Sinha, name: Mahavir Patil, name: Himanshu Beniwal, name: Mayank Singh
Published: 2025-10-08 14:04:14

Code-switching (CSW), the alternation of languages and scripts within a
single utterance, remains a fundamental challenge for multiling ual NLP, even
amidst the rapid advances of large language models (LLMs). Most LLMs still
struggle with mixed-language inputs, limited CSW datasets, and evaluation
biases, hindering deployment in multilingual societies. This survey provides
the first comprehensive analysis of CSW-aware LLM research, reviewing
unique_references studies spanning five research areas, 12 NLP tasks,
30+ datasets, and 80+ languages. We classify recent advances by architecture,
training strategy, and evaluation methodology, outlining how LLMs have reshaped
CSW modeling and what challenges persist. The paper concludes with a roadmap
emphasizing the need for inclusive datasets, fair evaluation, and
linguistically grounded models to achieve truly multilingual intelligence. A
curated collection of all resources is maintained at
https://github.com/lingo-iitgn/awesome-code-mixing/.
G^2RPO: Granular GRPO for Precise Reward in Flow Models
Authors: name: Yujie Zhou, name: Pengyang Ling, name: Jiazi Bu, name: Yibin Wang, name: Yuhang Zang, name: Jiaqi Wang, name: Li Niu, name: Guangtao Zhai
Published: 2025-10-02 12:57:12

The integration of online reinforcement learning (RL) into diffusion and flow
models has recently emerged as a promising approach for aligning generative
models with human preferences. Stochastic sampling via Stochastic Differential
Equations (SDE) is employed during the denoising process to generate diverse
denoising directions for RL exploration. While existing methods effectively
explore potential high-value samples, they suffer from sub-optimal preference
alignment due to sparse and narrow reward signals. To address these challenges,
we propose a novel Granular-GRPO (G^2RPO ) framework that achieves
precise and comprehensive reward assessments of sampling directions in
reinforcement learning of flow models. Specifically, a Singular Stochastic
Sampling strategy is introduced to support step-wise stochastic exploration
while enforcing a high correlation between the reward and the injected noise,
thereby facilitating a faithful reward for each SDE perturbation. Concurrently,
to eliminate the bias inherent in fixed-granularity denoising, we introduce a
Multi-Granularity Advantage Integration module that aggregates advantages
computed at multiple diffusion scales, producing a more comprehensive and
robust evaluation of the sampling directions. Experiments conducted on various
reward models, including both in-domain and out-of-domain evaluations,
demonstrate that our G^2RPO significantly outperforms existing
flow-based GRPO baselines,highlighting its effectiveness and robustness.
When Benchmarks Age: Temporal Misalignment through Large Language Model
  Factuality Evaluation
Authors: name: Xunyi Jiang, name: Dingyi Chang, name: Julian McAuley, name: Xin Xu
Published: 2025-10-08 17:06:07

The rapid evolution of large language models (LLMs) and the real world has
outpaced the static nature of widely used evaluation benchmarks, raising
concerns about their reliability for evaluating LLM factuality. While
substantial works continue to rely on the popular but old benchmarks, their
temporal misalignment with real-world facts and modern LLMs, and their effects
on LLM factuality evaluation remain underexplored. Therefore, in this work, we
present a systematic investigation of this issue by examining five popular
factuality benchmarks and eight LLMs released across different years. An
up-to-date fact retrieval pipeline and three metrics are tailored to quantify
benchmark aging and its impact on LLM factuality evaluation. Experimental
results and analysis illustrate that a considerable portion of samples in the
widely used factuality benchmarks are outdated, leading to unreliable
assessments of LLM factuality. We hope our work can provide a testbed to assess
the reliability of a benchmark for LLM factuality evaluation and inspire more
research on the benchmark aging issue. Codes are available in
https://github.com/JiangXunyi/BenchAge.
OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot
Authors: name: Junhan Zhu, name: Hesong Wang, name: Mingluo Su, name: Zefang Wang, name: Huan Wang
Published: 2025-10-08 08:19:15

Large-scale text-to-image diffusion models, while powerful, suffer from
prohibitive computational cost. Existing one-shot network pruning methods can
hardly be directly applied to them due to the iterative denoising nature of
diffusion models. To bridge the gap, this paper presents OBS-Diff, a novel
one-shot pruning framework that enables accurate and training-free compression
of large-scale text-to-image diffusion models. Specifically, (i) OBS-Diff
revitalizes the classic Optimal Brain Surgeon (OBS), adapting it to the complex
architectures of modern diffusion models and supporting diverse pruning
granularity, including unstructured, N:M semi-structured, and structured (MHA
heads and FFN neurons) sparsity; (ii) To align the pruning criteria with the
iterative dynamics of the diffusion process, by examining the problem from an
error-accumulation perspective, we propose a novel timestep-aware Hessian
construction that incorporates a logarithmic-decrease weighting scheme,
assigning greater importance to earlier timesteps to mitigate potential error
accumulation; (iii) Furthermore, a computationally efficient group-wise
sequential pruning strategy is proposed to amortize the expensive calibration
process. Extensive experiments show that OBS-Diff achieves state-of-the-art
one-shot pruning for diffusion models, delivering inference acceleration with
minimal degradation in visual quality.