Orionnnn/hub-stats huggingface dataset:
FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios
Authors: name: Shiyi Zhang, name: Junhao Zhuang, name: Zhaoyang Zhang, name: Ying Shan, name: Yansong Tang
Published: 2025-05-06 17:58:02

Action customization involves generating videos where the subject performs
actions dictated by input control signals. Current methods use pose-guided or
global motion customization but are limited by strict constraints on spatial
structure, such as layout, skeleton, and viewpoint consistency, reducing
adaptability across diverse subjects and scenarios. To overcome these
limitations, we propose FlexiAct, which transfers actions from a reference
video to an arbitrary target image. Unlike existing methods, FlexiAct allows
for variations in layout, viewpoint, and skeletal structure between the subject
of the reference video and the target image, while maintaining identity
consistency. Achieving this requires precise action control, spatial structure
adaptation, and consistency preservation. To this end, we introduce RefAdapter,
a lightweight image-conditioned adapter that excels in spatial adaptation and
consistency preservation, surpassing existing methods in balancing appearance
consistency and structural flexibility. Additionally, based on our
observations, the denoising process exhibits varying levels of attention to
motion (low frequency) and appearance details (high frequency) at different
timesteps. So we propose FAE (Frequency-aware Action Extraction), which, unlike
existing methods that rely on separate spatial-temporal architectures, directly
achieves action extraction during the denoising process. Experiments
demonstrate that our method effectively transfers actions to subjects with
diverse layouts, skeletons, and viewpoints. We release our code and model
weights to support further research at
https://shiyi-zh0408.github.io/projectpages/FlexiAct/
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
  Inference
Authors: name: Yaoqi Chen, name: Jinkai Zhang, name: Baotong Lu, name: Qianxi Zhang, name: Chengruidong Zhang, name: Jingjia Luo, name: Di Liu, name: Huiqiang Jiang, name: Qi Chen, name: Jing Liu, name: Bailu Ding, name: Xiao Yan, name: Jiawei Jiang, name: Chen Chen, name: Mingxing Zhang, name: Yuqing Yang, name: Fan Yang, name: Mao Yang
Published: 2025-05-05 18:01:17

The growing context lengths of large language models (LLMs) pose significant
challenges for efficient inference, primarily due to GPU memory and bandwidth
constraints. We present RetroInfer, a novel system that reconceptualizes the
key-value (KV) cache as a vector storage system which exploits the inherent
attention sparsity to accelerate long-context LLM inference. At its core is the
wave index, an Attention-aWare VEctor index that enables efficient and accurate
retrieval of critical tokens through techniques such as tripartite attention
approximation, accuracy-bounded attention estimation, and segmented clustering.
Complementing this is the wave buffer, which coordinates KV cache placement and
overlaps computation and data transfer across GPU and CPU to sustain high
throughput. Unlike prior sparsity-based methods that struggle with token
selection and hardware coordination, RetroInfer delivers robust performance
without compromising model accuracy. Experiments on long-context benchmarks
show up to 4.5X speedup over full attention within GPU memory limits and up to
10.5X over sparse attention baselines when KV cache is extended to CPU memory,
all while preserving full-attention-level accuracy.
Unified Multimodal Chain-of-Thought Reward Model through Reinforcement
  Fine-Tuning
Authors: name: Yibin Wang, name: Zhimin Li, name: Yuhang Zang, name: Chunyu Wang, name: Qinglin Lu, name: Cheng Jin, name: Jiaqi Wang
Published: 2025-05-06 08:46:41

Recent advances in multimodal Reward Models (RMs) have shown significant
promise in delivering reward signals to align vision models with human
preferences. However, current RMs are generally restricted to providing direct
responses or engaging in shallow reasoning processes with limited depth, often
leading to inaccurate reward signals. We posit that incorporating explicit long
chains of thought (CoT) into the reward reasoning process can significantly
strengthen their reliability and robustness. Furthermore, we believe that once
RMs internalize CoT reasoning, their direct response accuracy can also be
improved through implicit reasoning capabilities. To this end, this paper
proposes UnifiedReward-Think, the first unified multimodal CoT-based reward
model, capable of multi-dimensional, step-by-step long-chain reasoning for both
visual understanding and generation reward tasks. Specifically, we adopt an
exploration-driven reinforcement fine-tuning approach to elicit and incentivize
the model's latent complex reasoning ability: (1) We first use a small amount
of image generation preference data to distill the reasoning process of GPT-4o,
which is then used for the model's cold start to learn the format and structure
of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge
and generalization capabilities, we prepare large-scale unified multimodal
preference data to elicit the model's reasoning process across various vision
tasks. During this phase, correct reasoning outputs are retained for rejection
sampling to refine the model (3) while incorrect predicted samples are finally
used for Group Relative Policy Optimization (GRPO) based reinforcement
fine-tuning, enabling the model to explore diverse reasoning paths and optimize
for correct and robust solutions. Extensive experiments across various vision
reward tasks demonstrate the superiority of our model.
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Authors: name: Andrew Zhao, name: Yiran Wu, name: Yang Yue, name: Tong Wu, name: Quentin Xu, name: Yang Yue, name: Matthieu Lin, name: Shenzhi Wang, name: Qingyun Wu, name: Zilong Zheng, name: Gao Huang
Published: 2025-05-06 09:08:00

Reinforcement learning with verifiable rewards (RLVR) has shown promise in
enhancing the reasoning capabilities of large language models by learning
directly from outcome-based rewards. Recent RLVR works that operate under the
zero setting avoid supervision in labeling the reasoning process, but still
depend on manually curated collections of questions and answers for training.
The scarcity of high-quality, human-produced examples raises concerns about the
long-term scalability of relying on human supervision, a challenge already
evident in the domain of language model pretraining. Furthermore, in a
hypothetical future where AI surpasses human intelligence, tasks provided by
humans may offer limited learning potential for a superintelligent system. To
address these concerns, we propose a new RLVR paradigm called Absolute Zero, in
which a single model learns to propose tasks that maximize its own learning
progress and improves reasoning by solving them, without relying on any
external data. Under this paradigm, we introduce the Absolute Zero Reasoner
(AZR), a system that self-evolves its training curriculum and reasoning ability
by using a code executor to both validate proposed code reasoning tasks and
verify answers, serving as an unified source of verifiable reward to guide
open-ended yet grounded learning. Despite being trained entirely without
external data, AZR achieves overall SOTA performance on coding and mathematical
reasoning tasks, outperforming existing zero-setting models that rely on tens
of thousands of in-domain human-curated examples. Furthermore, we demonstrate
that AZR can be effectively applied across different model scales and is
compatible with various model classes.
Invoke Interfaces Only When Needed: Adaptive Invocation for Large
  Language Models in Question Answering
Authors: name: Jihao Zhao, name: Chunlai Zhou, name: Biao Qin
Published: 2025-05-05 01:45:56

The collaborative paradigm of large and small language models (LMs)
effectively balances performance and cost, yet its pivotal challenge lies in
precisely pinpointing the moment of invocation when hallucinations arise in
small LMs. Previous optimization efforts primarily focused on post-processing
techniques, which were separate from the reasoning process of LMs, resulting in
high computational costs and limited effectiveness. In this paper, we propose a
practical invocation evaluation metric called AttenHScore, which calculates the
accumulation and propagation of hallucinations during the generation process of
small LMs, continuously amplifying potential reasoning errors. By dynamically
adjusting the detection threshold, we achieve more accurate real-time
invocation of large LMs. Additionally, considering the limited reasoning
capacity of small LMs, we leverage uncertainty-aware knowledge reorganization
to assist them better capture critical information from different text chunks.
Extensive experiments reveal that our AttenHScore outperforms most baseline in
enhancing real-time hallucination detection capabilities across multiple QA
datasets, especially when addressing complex queries. Moreover, our strategies
eliminate the need for additional model training and display flexibility in
adapting to various transformer-based LMs.
Rethinking RGB-Event Semantic Segmentation with a Novel Bidirectional
  Motion-enhanced Event Representation
Authors: name: Zhen Yao, name: Xiaowen Ying, name: Mooi Choo Chuah
Published: 2025-05-02 19:19:58

Event cameras capture motion dynamics, offering a unique modality with great
potential in various computer vision tasks. However, RGB-Event fusion faces
three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal
misalignment. Existing voxel grid representations neglect temporal correlations
between consecutive event windows, and their formulation with simple
accumulation of asynchronous and sparse events is incompatible with the
synchronous and dense nature of RGB modality. To tackle these challenges, we
propose a novel event representation, Motion-enhanced Event Tensor (MET), which
transforms sparse event voxels into a dense and temporally coherent form by
leveraging dense optical flows and event temporal features. In addition, we
introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a
Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to
mitigate modal misalignment, while bidirectional flow aggregation and temporal
fusion mechanisms resolve spatiotemporal misalignment. Experimental results on
two large-scale datasets demonstrate that our framework significantly
outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our
code is available at: https://github.com/zyaocoder/BRENet.
Attention Mechanisms Perspective: Exploring LLM Processing of
  Graph-Structured Data
Authors: name: Zhong Guan, name: Likang Wu, name: Hongke Zhao, name: Ming He, name: Jianpin Fan
Published: 2025-05-04 14:40:31

Attention mechanisms are critical to the success of large language models
(LLMs), driving significant advancements in multiple fields. However, for
graph-structured data, which requires emphasis on topological connections, they
fall short compared to message-passing mechanisms on fixed links, such as those
employed by Graph Neural Networks (GNNs). This raises a question: ``Does
attention fail for graphs in natural language settings?'' Motivated by these
observations, we embarked on an empirical study from the perspective of
attention mechanisms to explore how LLMs process graph-structured data. The
goal is to gain deeper insights into the attention behavior of LLMs over graph
structures. We uncovered unique phenomena regarding how LLMs apply attention to
graph-structured data and analyzed these findings to improve the modeling of
such data by LLMs. The primary findings of our research are: 1) While LLMs can
recognize graph data and capture text-node interactions, they struggle to model
inter-node relationships within graph structures due to inherent architectural
constraints. 2) The attention distribution of LLMs across graph nodes does not
align with ideal structural patterns, indicating a failure to adapt to graph
topology nuances. 3) Neither fully connected attention nor fixed connectivity
is optimal; each has specific limitations in its application scenarios.
Instead, intermediate-state attention windows improve LLM training performance
and seamlessly transition to fully connected windows during inference. Source
code: https://github.com/millioniron/LLM_exploration{LLM4Exploration}
Learning Heterogeneous Mixture of Scene Experts for Large-scale Neural
  Radiance Fields
Authors: name: Zhenxing Mi, name: Ping Yin, name: Xue Xiao, name: Dan Xu
Published: 2025-05-04 06:25:14

Recent NeRF methods on large-scale scenes have underlined the importance of
scene decomposition for scalable NeRFs. Although achieving reasonable
scalability, there are several critical problems remaining unexplored, i.e.,
learnable decomposition, modeling scene heterogeneity, and modeling efficiency.
In this paper, we introduce Switch-NeRF++, a Heterogeneous Mixture of Hash
Experts (HMoHE) network that addresses these challenges within a unified
framework. It is a highly scalable NeRF that learns heterogeneous decomposition
and heterogeneous NeRFs efficiently for large-scale scenes in an end-to-end
manner. In our framework, a gating network learns to decomposes scenes and
allocates 3D points to specialized NeRF experts. This gating network is
co-optimized with the experts, by our proposed Sparsely Gated Mixture of
Experts (MoE) NeRF framework. We incorporate a hash-based gating network and
distinct heterogeneous hash experts. The hash-based gating efficiently learns
the decomposition of the large-scale scene. The distinct heterogeneous hash
experts consist of hash grids of different resolution ranges, enabling
effective learning of the heterogeneous representation of different scene
parts. These design choices make our framework an end-to-end and highly
scalable NeRF solution for real-world large-scale scene modeling to achieve
both quality and efficiency. We evaluate our accuracy and scalability on
existing large-scale NeRF datasets and a new dataset with very large-scale
scenes (>6.5km^2) from UrbanBIS. Extensive experiments demonstrate that our
approach can be easily scaled to various large-scale scenes and achieve
state-of-the-art scene rendering accuracy. Furthermore, our method exhibits
significant efficiency, with an 8x acceleration in training and a 16x
acceleration in rendering compared to Switch-NeRF. Codes will be released in
https://github.com/MiZhenxing/Switch-NeRF.
Practical Efficiency of Muon for Pretraining
Authors: name: Essential AI, name: Ishaan Shah, name: Anthony M. Polloreno, name: Karl Stratos, name: Philip Monk, name: Adarsh Chaluvaraju, name: Andrew Hojel, name: Andrew Ma, name: Anil Thomas, name: Ashish Tanwer, name: Darsh J Shah, name: Khoi Nguyen, name: Kurt Smith, name: Michael Callahan, name: Michael Pust, name: Mohit Parmar, name: Peter Rushton, name: Platon Mazarakis, name: Ritvik Kapila, name: Saurabh Srivastava, name: Somanshu Singla, name: Tim Romanski, name: Yash Vanjani, name: Ashish Vaswani
Published: 2025-05-04 19:14:43

We demonstrate that Muon, the simplest instantiation of a second-order
optimizer, explicitly expands the Pareto frontier over AdamW on the
compute-time tradeoff. We find that Muon is more effective than AdamW in
retaining data efficiency at large batch sizes, far beyond the so-called
critical batch size, while remaining computationally efficient, thus enabling
more economical training. We study the combination of Muon and the maximal
update parameterization (muP) for efficient hyperparameter transfer and present
a simple telescoping algorithm that accounts for all sources of error in muP
while introducing only a modest overhead in resources. We validate our findings
through extensive experiments with model sizes up to four billion parameters
and ablations on the data distribution and architecture.
ReplaceMe: Network Simplification via Layer Pruning and Linear
  Transformations
Authors: name: Dmitriy Shopkhoev, name: Ammar Ali, name: Magauiya Zhussip, name: Valentin Malykh, name: Stamatios Lefkimmiatis, name: Nikos Komodakis, name: Sergey Zagoruyko
Published: 2025-05-05 17:47:42

We introduce ReplaceMe, a generalized training-free depth pruning method that
effectively replaces transformer blocks with a linear operation, while
maintaining high performance for low compression ratios. In contrast to
conventional pruning approaches that require additional training or
fine-tuning, our approach requires only a small calibration dataset that is
used to estimate a linear transformation to approximate the pruned blocks. This
estimated linear mapping can be seamlessly merged with the remaining
transformer blocks, eliminating the need for any additional network parameters.
Our experiments show that ReplaceMe consistently outperforms other
training-free approaches and remains highly competitive with state-of-the-art
pruning methods that involve extensive retraining/fine-tuning and architectural
modifications. Applied to several large language models (LLMs), ReplaceMe
achieves up to 25% pruning while retaining approximately 90% of the original
model's performance on open benchmarks - without any training or healing steps,
resulting in minimal computational overhead (see Fig.1). We provide an
open-source library implementing ReplaceMe alongside several state-of-the-art
depth pruning techniques, available at this repository.
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and
  Attack-Defense Evaluation
Authors: name: Vaidehi Patil, name: Yi-Lin Sung, name: Peter Hase, name: Jie Peng, name: Tianlong Chen, name: Mohit Bansal
Published: 2025-05-01 01:54:00

LLMs trained on massive datasets may inadvertently acquire sensitive
information such as personal details and potentially harmful content. This risk
is further heightened in multimodal LLMs as they integrate information from
multiple modalities (image and text). Adversaries can exploit this knowledge
through multimodal prompts to extract sensitive details. Evaluating how
effectively MLLMs can forget such information (targeted unlearning)
necessitates the creation of high-quality, well-annotated image-text pairs.
While prior work on unlearning has focused on text, multimodal unlearning
remains underexplored. To address this gap, we first introduce a multimodal
unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
an attack-and-defense framework to evaluate methods for deleting specific
multimodal knowledge from MLLMs. We extend a visual question-answering dataset
using an automated pipeline that generates varying-proximity samples for
testing generalization and specificity, followed by manual filtering for
maintaining high quality. We then evaluate six defense objectives against seven
attacks (four whitebox, three blackbox), including a novel whitebox method
leveraging interpretability of hidden states. Our results show multimodal
attacks outperform text- or image-only ones, and that the most effective
defense removes answer information from internal model states. Additionally,
larger models exhibit greater post-editing robustness, suggesting that scale
enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
unlearning in MLLMs.
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization
  in Rejection Sampling and RL
Authors: name: Jiarui Yao, name: Yifan Hao, name: Hanning Zhang, name: Hanze Dong, name: Wei Xiong, name: Nan Jiang, name: Tong Zhang
Published: 2025-05-05 06:26:00

Chain-of-thought (CoT) reasoning in large language models (LLMs) can be
formalized as a latent variable problem, where the model needs to generate
intermediate reasoning steps. While prior approaches such as iterative
reward-ranked fine-tuning (RAFT) have relied on such formulations, they
typically apply uniform inference budgets across prompts, which fails to
account for variability in difficulty and convergence behavior. This work
identifies the main bottleneck in CoT training as inefficient stochastic
gradient estimation due to static sampling strategies. We propose GVM-RAFT, a
prompt-specific Dynamic Sample Allocation Strategy designed to minimize
stochastic gradient variance under a computational budget constraint. The
method dynamically allocates computational resources by monitoring prompt
acceptance rates and stochastic gradient norms, ensuring that the resulting
gradient variance is minimized. Our theoretical analysis shows that the
proposed dynamic sampling strategy leads to accelerated convergence guarantees
under suitable conditions. Experiments on mathematical reasoning show that
GVM-RAFT achieves a 2-4x speedup and considerable accuracy improvements over
vanilla RAFT. The proposed dynamic sampling strategy is general and can be
incorporated into other reinforcement learning algorithms, such as GRPO,
leading to similar improvements in convergence and test accuracy. Our code is
available at https://github.com/RLHFlow/GVM.
Grokking in the Wild: Data Augmentation for Real-World Multi-Hop
  Reasoning with Transformers
Authors: name: Roman Abramov, name: Felix Steinbauer, name: Gjergji Kasneci
Published: 2025-04-29 13:33:29

Transformers have achieved great success in numerous NLP tasks but continue
to exhibit notable gaps in multi-step factual reasoning, especially when
real-world knowledge is sparse. Recent advances in grokking have demonstrated
that neural networks can transition from memorizing to perfectly generalizing
once they detect underlying logical patterns - yet these studies have primarily
used small, synthetic tasks. In this paper, for the first time, we extend
grokking to real-world factual data and address the challenge of dataset
sparsity by augmenting existing knowledge graphs with carefully designed
synthetic data to raise the ratio phi_r of inferred facts to atomic facts
above the threshold required for grokking. Surprisingly, we find that even
factually incorrect synthetic data can strengthen emergent reasoning circuits
rather than degrade accuracy, as it forces the model to rely on relational
structure rather than memorization. When evaluated on multi-hop reasoning
benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA -
substantially improving over strong baselines and matching or exceeding current
state-of-the-art results. We further provide an in-depth analysis of how
increasing phi_r drives the formation of generalizing circuits inside
Transformers. Our findings suggest that grokking-based data augmentation can
unlock implicit multi-hop reasoning capabilities, opening the door to more
robust and interpretable factual reasoning in large-scale language models.