Orionnnn/hub-stats huggingface dataset:
LayerCake: Token-Aware Contrastive Decoding within Large Language Model
  Layers
Authors: name: Jingze Zhu, name: Yongliang Wu, name: Wenbo Zhu, name: Jiawang Cao, name: Yanqiang Zheng, name: Jiawei Chen, name: Xu Yang, name: Bernt Schiele, name: Jonas Fischer, name: Xinting Hu
Published: 2025-07-06 14:35:43

Large language models (LLMs) excel at natural language understanding and
generation but remain vulnerable to factual errors, limiting their reliability
in knowledge-intensive tasks. While decoding-time strategies provide a
promising efficient solution without training, existing methods typically treat
token-level and layer-level signals in isolation, overlooking the joint
dynamics between them. In this work, we introduce a token-aware,
layer-localized contrastive decoding method that aligns specific token types
with their most influential transformer layers to improve factual generation.
Through empirical attention analysis, we identify two key patterns: punctuation
tokens receive dominant attention in early layers, while conceptual tokens
govern semantic reasoning in intermediate layers. By selectively suppressing
attention to these token types at their respective depths, we achieve the
induction of controlled factual degradation and derive contrastive signals to
guide the final factual decoding. Our method requires no additional training or
model modification, and experiments demonstrate that our method consistently
improves factuality across multiple LLMs and various benchmarks.
SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual
  Dyadic Interactive Human Generation
Authors: name: Youliang Zhang, name: Zhaoyang Li, name: Duomin Wang, name: Jiahe Zhang, name: Deyu Zhou, name: Zixin Yin, name: Xili Dai, name: Gang Yu, name: Xiu Li
Published: 2025-07-14 02:22:47

The rapid development of large-scale models has catalyzed significant
breakthroughs in the digital human domain. These advanced methodologies offer
high-fidelity solutions for avatar driving and rendering, leading academia to
focus on the next major challenge: audio-visual dyadic interactive virtual
human. To facilitate research in this emerging area, we present SpeakerVid-5M
dataset, the first large-scale, high-quality dataset designed for audio-visual
dyadic interactive virtual human generation. Totaling over 8,743 hours,
SpeakerVid-5M contains more than 5.2 million video clips of human portraits. It
covers diverse scales and interaction types, including monadic talking,
listening, and dyadic conversations. Crucially, the dataset is structured along
two key dimensions: interaction type and data quality. First, it is categorized
into four types (dialogue branch, single branch, listening branch and
multi-turn branch) based on the interaction scenario. Second, it is stratified
into a large-scale pre-training subset and a curated, high-quality subset for
Supervised Fine-Tuning (SFT). This dual structure accommodates a wide array of
2D virtual human tasks. In addition, we provide an autoregressive (AR)-based
video chat baseline trained on this data, accompanied by a dedicated set of
metrics and test data to serve as a benchmark VidChatBench for future work.
Both the dataset and the corresponding data processing code will be publicly
released. Project page: https://dorniwang.github.io/SpeakerVid-5M/
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large
  Audio Language Models
Authors: name: Arushi Goel, name: Sreyan Ghosh, name: Jaehyeon Kim, name: Sonal Kumar, name: Zhifeng Kong, name: Sang-gil Lee, name: Chao-Han Huck Yang, name: Ramani Duraiswami, name: Dinesh Manocha, name: Rafael Valle, name: Bryan Catanzaro
Published: 2025-07-10 19:40:21

We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large
audio-language model that advances reasoning and understanding across speech,
sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder
trained using a novel strategy for joint representation learning across all 3
modalities of speech, sound, and music; (ii) flexible, on-demand thinking,
allowing the model to do chain-of-thought-type reasoning before answering;
(iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning
(including speech) up to 10 minutes; and (v) voice-to-voice interaction. To
enable these capabilities, we propose several large-scale training datasets
curated using novel strategies, including AudioSkills-XL, LongAudio-XL,
AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based
training strategy. Trained on only open-source audio data, AF3 achieves new
SOTA results on over 20+ (long) audio understanding and reasoning benchmarks,
surpassing both open-weight and closed-source models trained on much larger
datasets.
Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection
Authors: name: Subhajit Maity, name: Ayan Kumar Bhunia, name: Subhadeep Koley, name: Pinaki Nath Chowdhury, name: Aneeshan Sain, name: Yi-Zhe Song
Published: 2025-07-10 17:59:49

Keypoint detection, integral to modern machine perception, faces challenges
in few-shot learning, particularly when source data from the same distribution
as the query is unavailable. This gap is addressed by leveraging sketches, a
popular form of human expression, providing a source-free alternative. However,
challenges arise in mastering cross-modal embeddings and handling user-specific
sketch styles. Our proposed framework overcomes these hurdles with a
prototypical setup, combined with a grid-based locator and prototypical domain
adaptation. We also demonstrate success in few-shot convergence across novel
keypoints and classes through extensive experiments.
DOTResize: Reducing LLM Width via Discrete Optimal Transport-based
  Neuron Merging
Authors: name: Neha Verma, name: Kenton Murray, name: Kevin Duh
Published: 2025-07-06 19:49:46

Model compression offers a promising path to reducing the cost and
inaccessibility of large pre-trained models, without significantly compromising
their impressive performance. Large Transformer models, including large
language models (LLMs), often contain computational redundancy, which can serve
as a target for new model compression methods. In this work, we specifically
target neuron-level redundancies in model layers by combining groups of similar
neurons into fewer neurons. We frame this width reduction as a Discrete Optimal
Transport problem, and propose DOTResize, a novel Transformer compression
method that uses optimal transport theory to transform and compress model
weights. To ensure applicability within the Transformer architecture, we
motivate and incorporate entropic regularization and matrix factorization into
the transportation maps produced by our method. Unlike pruning-based approaches
which discard neurons based on importance measures, DOTResize re-projects the
entire neuron width, allowing the retention and redistribution of useful signal
across the reduced layer. Empirical results show that compared to simple or
state-of-the-art neuron width-pruning techniques, DOTResize can outperform
these methods across multiple LLM families and sizes, while achieving
measurable reductions in real-world computational cost.
Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for
  Visual Reasoning
Authors: name: Yana Wei, name: Liang Zhao, name: Jianjian Sun, name: Kangheng Lin, name: Jisheng Yin, name: Jingcheng Hu, name: Yinmin Zhang, name: En Yu, name: Haoran Lv, name: Zejia Weng, name: Jia Wang, name: Chunrui Han, name: Yuang Peng, name: Qi Han, name: Zheng Ge, name: Xiangyu Zhang, name: Daxin Jiang, name: Vishal M. Patel
Published: 2025-07-07 17:59:03

The remarkable reasoning capability of large language models (LLMs) stems
from cognitive behaviors that emerge through reinforcement with verifiable
rewards. This work investigates how to transfer this principle to Multimodal
LLMs (MLLMs) to unlock advanced visual reasoning. We introduce a two-stage
paradigm built on Qwen2.5-VL-7B: a massive linguistic cold-start fine-tuning,
followed by multimodal reinforcement learning (RL) spanning nearly 1,000 steps,
surpassing all previous open-source efforts in scale. This pioneering work
reveals three fundamental insights: 1) Behavior transfer emerges surprisingly
early in cold start due to linguistic mental imagery. 2) Cold start broadly
memorizes visual behaviors, while RL critically discerns and scales up
effective patterns. 3) Transfer strategically favors high-utility behaviors
such as visual reflection. Our resulting model, Open-Vision-Reasoner (OVR),
achieves state-of-the-art performance on a suite of reasoning benchmarks,
including 95.3% on MATH500, 51.8% on MathVision and 54.6% on MathVerse. We
release our model, data, and training dynamics to catalyze the development of
more capable, behavior-aligned multimodal reasoners.
Vision Foundation Models as Effective Visual Tokenizers for
  Autoregressive Image Generation
Authors: name: Anlin Zheng, name: Xin Wen, name: Xuanyang Zhang, name: Chuofan Ma, name: Tiancai Wang, name: Gang Yu, name: Xiangyu Zhang, name: Xiaojuan Qi
Published: 2025-07-11 09:32:45

Leveraging the powerful representations of pre-trained vision foundation
models -- traditionally used for visual comprehension -- we explore a novel
direction: building an image tokenizer directly atop such models, a largely
underexplored area. Specifically, we employ a frozen vision foundation model as
the encoder of our tokenizer. To enhance its effectiveness, we introduce two
key components: (1) a region-adaptive quantization framework that reduces
redundancy in the pre-trained features on regular 2D grids, and (2) a semantic
reconstruction objective that aligns the tokenizer's outputs with the
foundation model's representations to preserve semantic fidelity. Based on
these designs, our proposed image tokenizer, VFMTok, achieves substantial
improvements in image reconstruction and generation quality, while also
enhancing token efficiency. It further boosts autoregressive (AR) generation --
achieving a gFID of 2.07 on ImageNet benchmarks, while accelerating model
convergence by three times, and enabling high-fidelity class-conditional
synthesis without the need for classifier-free guidance (CFG). The code will be
released publicly to benefit the community.
KV Cache Steering for Inducing Reasoning in Small Language Models
Authors: name: Max Belitsky, name: Dawid J. Kopiczko, name: Michael Dorkenwald, name: M. Jehanzeb Mirza, name: Cees G. M. Snoek, name: Yuki M. Asano
Published: 2025-07-11 17:59:36

We propose cache steering, a lightweight method for implicit steering of
language models via a one-shot intervention applied directly to the key-value
cache. To validate its effectiveness, we apply cache steering to induce
chain-of-thought reasoning in small language models. Our approach leverages
GPT-4o-generated reasoning traces to construct steering vectors that shift
model behavior toward more explicit, multi-step reasoning without fine-tuning
or prompt modifications. Experimental evaluations on diverse reasoning
benchmarks demonstrate that cache steering improves both the qualitative
structure of model reasoning and quantitative task performance. Compared to
prior activation steering techniques that require continuous interventions, our
one-shot cache steering offers substantial advantages in terms of
hyperparameter stability, inference-time efficiency, and ease of integration,
making it a more robust and practical solution for controlled generation.
Neural-Driven Image Editing
Authors: name: Pengfei Zhou, name: Jie Xia, name: Xiaopeng Peng, name: Wangbo Zhao, name: Zilong Ye, name: Zekai Li, name: Suorong Yang, name: Jiadong Pan, name: Yuanxiang Chen, name: Ziqiao Wang, name: Kai Wang, name: Qian Zheng, name: Xiaojun Chang, name: Gang Pan, name: Shurong Dong, name: Kaipeng Zhang, name: Yang You
Published: 2025-07-07 18:31:50

Traditional image editing typically relies on manual prompting, making it
labor-intensive and inaccessible to individuals with limited motor control or
language abilities. Leveraging recent advances in brain-computer interfaces
(BCIs) and generative models, we propose LoongX, a hands-free image editing
approach driven by multimodal neurophysiological signals. LoongX utilizes
state-of-the-art diffusion models trained on a comprehensive dataset of 23,928
image editing pairs, each paired with synchronized electroencephalography
(EEG), functional near-infrared spectroscopy (fNIRS), photoplethysmography
(PPG), and head motion signals that capture user intent. To effectively address
the heterogeneity of these signals, LoongX integrates two key modules. The
cross-scale state space (CS3) module encodes informative modality-specific
features. The dynamic gated fusion (DGF) module further aggregates these
features into a unified latent space, which is then aligned with edit semantics
via fine-tuning on a diffusion transformer (DiT). Additionally, we pre-train
the encoders using contrastive learning to align cognitive states with semantic
intentions from embedded natural language. Extensive experiments demonstrate
that LoongX achieves performance comparable to text-driven methods (CLIP-I:
0.6605 vs. 0.6558; DINO: 0.4812 vs. 0.4636) and outperforms them when neural
signals are combined with speech (CLIP-T: 0.2588 vs. 0.2549). These results
highlight the promise of neural-driven generative models in enabling
accessible, intuitive image editing and open new directions for
cognitive-driven creative technologies. Datasets and code will be released to
support future work and foster progress in this emerging area.
What Has a Foundation Model Found? Using Inductive Bias to Probe for
  World Models
Authors: name: Keyon Vafa, name: Peter G. Chang, name: Ashesh Rambachan, name: Sendhil Mullainathan
Published: 2025-07-09 15:36:15

Foundation models are premised on the idea that sequence prediction can
uncover deeper domain understanding, much like how Kepler's predictions of
planetary motion later led to the discovery of Newtonian mechanics. However,
evaluating whether these models truly capture deeper structure remains a
challenge. We develop a technique for evaluating foundation models that
examines how they adapt to synthetic datasets generated from some postulated
world model. Our technique measures whether the foundation model's inductive
bias aligns with the world model, and so we refer to it as an inductive bias
probe. Across multiple domains, we find that foundation models can excel at
their training tasks yet fail to develop inductive biases towards the
underlying world model when adapted to new tasks. We particularly find that
foundation models trained on orbital trajectories consistently fail to apply
Newtonian mechanics when adapted to new physics tasks. Further analysis reveals
that these models behave as if they develop task-specific heuristics that fail
to generalize.
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with
  Chunk-Level Activation Sparsity
Authors: name: Chenyang Song, name: Weilin Zhao, name: Xu Han, name: Chaojun Xiao, name: Yingfa Chen, name: Yuxuan Li, name: Zhiyuan Liu, name: Maosong Sun
Published: 2025-07-11 17:28:56

To alleviate the computational burden of large language models (LLMs),
architectures with activation sparsity, represented by mixture-of-experts
(MoE), have attracted increasing attention. However, the non-differentiable and
inflexible routing of vanilla MoE hurts model performance. Moreover, while each
token activates only a few parameters, these sparsely-activated architectures
exhibit low chunk-level sparsity, indicating that the union of multiple
consecutive tokens activates a large ratio of parameters. Such a sparsity
pattern is unfriendly for acceleration under low-resource conditions (e.g.,
end-side devices) and incompatible with mainstream acceleration techniques
(e.g., speculative decoding). To address these challenges, we introduce a novel
MoE architecture, BlockFFN, as well as its efficient training and deployment
techniques. Specifically, we use a router integrating ReLU activation and
RMSNorm for differentiable and flexible routing. Next, to promote both
token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training
objectives are designed, making BlockFFN more acceleration-friendly. Finally,
we implement efficient acceleration kernels, combining activation sparsity and
speculative decoding for the first time. The experimental results demonstrate
the superior performance of BlockFFN over other MoE baselines, achieving over
80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67times speedup on
real end-side devices than dense models. All codes and checkpoints are
available publicly (https://github.com/thunlp/BlockFFN).
One Token to Fool LLM-as-a-Judge
Authors: name: Yulai Zhao, name: Haolin Liu, name: Dian Yu, name: S. Y. Kung, name: Haitao Mi, name: Dong Yu
Published: 2025-07-11 17:55:22

Generative reward models (also known as LLMs-as-judges), which use large
language models (LLMs) to evaluate answer quality, are increasingly adopted in
reinforcement learning with verifiable rewards (RLVR). They are often preferred
over rigid rule-based metrics, especially for complex reasoning tasks involving
free-form outputs. In this paradigm, an LLM is typically prompted to compare a
candidate answer against a ground-truth reference and assign a binary reward
indicating correctness. Despite the seeming simplicity of this comparison task,
we find that generative reward models exhibit surprising vulnerabilities to
superficial manipulations: non-word symbols (e.g., ":" or ".") or reasoning
openers like "Thought process:" and "Let's solve this problem step by step."
can often lead to false positive rewards. We demonstrate that this weakness is
widespread across LLMs, datasets, and prompt formats, posing a serious threat
for core algorithmic paradigms that rely on generative reward models, such as
rejection sampling, preference optimization, and RLVR. To mitigate this issue,
we introduce a simple yet effective data augmentation strategy and train a new
generative reward model with substantially improved robustness. Our findings
highlight the urgent need for more reliable LLM-based evaluation methods. We
release our robust, general-domain reward model and its synthetic training data
at https://huggingface.co/sarosavo/Master-RM and
https://huggingface.co/datasets/sarosavo/Master-RM.
Robust Multimodal Large Language Models Against Modality Conflict
Authors: name: Zongmeng Zhang, name: Wengang Zhou, name: Jie Zhao, name: Houqiang Li
Published: 2025-07-09 11:18:38

Despite the impressive capabilities of multimodal large language models
(MLLMs) in vision-language tasks, they are prone to hallucinations in
real-world scenarios. This paper investigates the hallucination phenomenon in
MLLMs from the perspective of modality conflict. Unlike existing works focusing
on the conflicts between model responses and inputs, we study the inherent
conflicts in inputs from different modalities that place MLLMs in a dilemma and
directly lead to hallucinations. We formally define the modality conflict and
construct a dataset named Multimodal Modality Conflict (MMMC) to simulate this
phenomenon in vision-language tasks. Three methods based on prompt engineering,
supervised fine-tuning, and reinforcement learning are proposed to alleviate
the hallucination caused by modality conflict. Extensive experiments are
conducted on the MMMC dataset to analyze the merits and demerits of these
methods. Our results show that the reinforcement learning method achieves the
best performance in mitigating the hallucination under modality conflict, while
the supervised fine-tuning method shows promising and stable performance. Our
work sheds light on the unnoticed modality conflict that leads to
hallucinations and provides more insights into the robustness of MLLMs.
Lumos-1: On Autoregressive Video Generation from a Unified Model
  Perspective
Authors: name: Hangjie Yuan, name: Weihua Chen, name: Jun Cen, name: Hu Yu, name: Jingyun Liang, name: Shuning Chang, name: Zhihui Lin, name: Tao Feng, name: Pengwei Liu, name: Jiazheng Xing, name: Hao Luo, name: Jiasheng Tang, name: Fan Wang, name: Yi Yang
Published: 2025-07-11 17:59:42

Autoregressive large language models (LLMs) have unified a vast range of
language tasks, inspiring preliminary efforts in autoregressive video
generation. Existing autoregressive video generators either diverge from
standard LLM architectures, depend on bulky external text encoders, or incur
prohibitive latency due to next-token decoding. In this paper, we introduce
Lumos-1, an autoregressive video generator that retains the LLM architecture
with minimal architectural modifications. To inject spatiotemporal correlations
in LLMs, we identify the efficacy of incorporating 3D RoPE and diagnose its
imbalanced frequency spectrum ranges. Therefore, we propose MM-RoPE, a RoPE
scheme that preserves the original textual RoPE while providing comprehensive
frequency spectra and scaled 3D positions for modeling multimodal
spatiotemporal data. Moreover, Lumos-1 resorts to a token dependency strategy
that obeys intra-frame bidirectionality and inter-frame temporal causality.
Based on this dependency strategy, we identify the issue of frame-wise loss
imbalance caused by spatial information redundancy and solve it by proposing
Autoregressive Discrete Diffusion Forcing (AR-DF). AR-DF introduces temporal
tube masking during training with a compatible inference-time masking policy to
avoid quality degradation. By using memory-efficient training techniques, we
pre-train Lumos-1 on only 48 GPUs, achieving performance comparable to EMU3 on
GenEval, COSMOS-Video2World on VBench-I2V, and OpenSoraPlan on VBench-T2V. Code
and models are available at https://github.com/alibaba-damo-academy/Lumos.
NeuralOS: Towards Simulating Operating Systems via Neural Generative
  Models
Authors: name: Luke Rivard, name: Sun Sun, name: Hongyu Guo, name: Wenhu Chen, name: Yuntian Deng
Published: 2025-07-11 17:59:40

We introduce NeuralOS, a neural framework that simulates graphical user
interfaces (GUIs) of operating systems by directly predicting screen frames in
response to user inputs such as mouse movements, clicks, and keyboard events.
NeuralOS combines a recurrent neural network (RNN), which tracks computer
state, with a diffusion-based neural renderer that generates screen images. The
model is trained on a large-scale dataset of Ubuntu XFCE recordings, which
include both randomly generated interactions and realistic interactions
produced by AI agents. Experiments show that NeuralOS successfully renders
realistic GUI sequences, accurately captures mouse interactions, and reliably
predicts state transitions like application launches. Although modeling
fine-grained keyboard interactions precisely remains challenging, NeuralOS
offers a step toward creating fully adaptive, generative neural interfaces for
future human-computer interaction systems.