Orionnnn/hub-stats huggingface dataset:
StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence
  Training of LLMs
Authors: name: Qijun Luo, name: Mengqi Li, name: Lei Zhao, name: Xiao Li
Published: 2025-06-03 16:54:15

Training language models on long sequence data is a demanding requirement for
enhancing the model's capability on complex tasks, e.g., long-chain reasoning.
However, as the sequence length scales up, the memory cost for storing
activation values becomes huge during the Backpropagation (BP) process, even
with the application of gradient checkpointing technique. To tackle this
challenge, we propose a memory-efficient and exact BP method called StreamBP,
which performs a linear decomposition of the chain rule along the sequence
dimension in a layer-wise manner, significantly reducing the memory cost of
activation values and logits. The proposed method is applicable to common
objectives such as SFT, GRPO, and DPO. From an implementation perspective,
StreamBP achieves less computational FLOPs and faster BP speed by leveraging
the causal structure of the language model. Compared to gradient checkpointing,
StreamBP scales up the maximum sequence length of BP by 2.8-5.5 times larger,
while using comparable or even less BP time. Note that StreamBP's sequence
length scaling ability can be directly transferred to batch size scaling for
accelerating training. We further develop a communication-efficient distributed
StreamBP to effectively support multi-GPU training and broaden its
applicability. Our code can be easily integrated into the training pipeline of
any transformer models and is available at https://github.com/Ledzy/StreamBP.
Rectified Point Flow: Generic Point Cloud Pose Estimation
Authors: name: Tao Sun, name: Liyuan Zhu, name: Shengyu Huang, name: Shuran Song, name: Iro Armeni
Published: 2025-06-05 17:36:03

We introduce Rectified Point Flow, a unified parameterization that formulates
pairwise point cloud registration and multi-part shape assembly as a single
conditional generative problem. Given unposed point clouds, our method learns a
continuous point-wise velocity field that transports noisy points toward their
target positions, from which part poses are recovered. In contrast to prior
work that regresses part-wise poses with ad-hoc symmetry handling, our method
intrinsically learns assembly symmetries without symmetry labels. Together with
a self-supervised encoder focused on overlapping points, our method achieves a
new state-of-the-art performance on six benchmarks spanning pairwise
registration and shape assembly. Notably, our unified formulation enables
effective joint training on diverse datasets, facilitating the learning of
shared geometric priors and consequently boosting accuracy. Project page:
https://rectified-pointflow.github.io/.
Geometry-Editable and Appearance-Preserving Object Compositon
Authors: name: Jianman Lin, name: Haojie Li, name: Chunmei Qing, name: Zhijing Yang, name: Liang Lin, name: Tianshui Chen
Published: 2025-05-27 09:05:28

General object composition (GOC) aims to seamlessly integrate a target object
into a background scene with desired geometric properties, while simultaneously
preserving its fine-grained appearance details. Recent approaches derive
semantic embeddings and integrate them into advanced diffusion models to enable
geometry-editable generation. However, these highly compact embeddings encode
only high-level semantic cues and inevitably discard fine-grained appearance
details. We introduce a Disentangled Geometry-editable and
Appearance-preserving Diffusion (DGAD) model that first leverages semantic
embeddings to implicitly capture the desired geometric transformations and then
employs a cross-attention retrieval mechanism to align fine-grained appearance
features with the geometry-edited representation, facilitating both precise
geometry editing and faithful appearance preservation in object composition.
Specifically, DGAD builds on CLIP/DINO-derived and reference networks to
extract semantic embeddings and appearance-preserving representations, which
are then seamlessly integrated into the encoding and decoding pipelines in a
disentangled manner. We first integrate the semantic embeddings into
pre-trained diffusion models that exhibit strong spatial reasoning capabilities
to implicitly capture object geometry, thereby facilitating flexible object
manipulation and ensuring effective editability. Then, we design a dense
cross-attention mechanism that leverages the implicitly learned object geometry
to retrieve and spatially align appearance features with their corresponding
regions, ensuring faithful appearance consistency. Extensive experiments on
public benchmarks demonstrate the effectiveness of the proposed DGAD framework.
Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting
Authors: name: Duochao Shi, name: Weijie Wang, name: Donny Y. Chen, name: Zeyu Zhang, name: Jia-Wang Bian, name: Bohan Zhuang, name: Chunhua Shen
Published: 2025-06-05 17:58:23

Depth maps are widely used in feed-forward 3D Gaussian Splatting (3DGS)
pipelines by unprojecting them into 3D point clouds for novel view synthesis.
This approach offers advantages such as efficient training, the use of known
camera poses, and accurate geometry estimation. However, depth discontinuities
at object boundaries often lead to fragmented or sparse point clouds, degrading
rendering quality -- a well-known limitation of depth-based representations. To
tackle this issue, we introduce PM-Loss, a novel regularization loss based on a
pointmap predicted by a pre-trained transformer. Although the pointmap itself
may be less accurate than the depth map, it effectively enforces geometric
smoothness, especially around object boundaries. With the improved depth map,
our method significantly improves the feed-forward 3DGS across various
architectures and scenes, delivering consistently better rendering results. Our
project page: https://aim-uofa.github.io/PMLoss
EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an
  Egocentric World?
Authors: name: Yuqian Yuan, name: Ronghao Dang, name: Long Li, name: Wentong Li, name: Dian Jiao, name: Xin Li, name: Deli Zhao, name: Fan Wang, name: Wenqiao Zhang, name: Jun Xiao, name: Yueting Zhuang
Published: 2025-06-05 17:44:12

The emergence of multimodal large language models (MLLMs) has driven
breakthroughs in egocentric vision applications. These applications necessitate
persistent, context-aware understanding of objects, as users interact with
tools in dynamic and cluttered environments. However, existing embodied
benchmarks primarily focus on static scene exploration, emphasizing object's
appearance and spatial attributes while neglecting the assessment of dynamic
changes arising from users' interactions. To address this gap, we introduce
EOC-Bench, an innovative benchmark designed to systematically evaluate
object-centric embodied cognition in dynamic egocentric scenarios. Specially,
EOC-Bench features 3,277 meticulously annotated QA pairs categorized into three
temporal categories: Past, Present, and Future, covering 11 fine-grained
evaluation dimensions and 3 visual object referencing types. To ensure thorough
assessment, we develop a mixed-format human-in-the-loop annotation framework
with four types of questions and design a novel multi-scale temporal accuracy
metric for open-ended temporal evaluation. Based on EOC-Bench, we conduct
comprehensive evaluations of various proprietary, open-source, and object-level
MLLMs. EOC-Bench serves as a crucial tool for advancing the embodied object
cognitive capabilities of MLLMs, establishing a robust foundation for
developing reliable core models for embodied systems.
Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
  Capabilities Through Evaluation Design
Authors: name: Lin Sun, name: Weihong Lin, name: Jinzhu Wu, name: Yongfu Zhu, name: Xiaoqi Jian, name: Guangxiang Zhao, name: Change Jia, name: Linglin Zhang, name: Sai-er Hu, name: Yuhan Wu, name: Xiangzheng Zhang
Published: 2025-06-05 08:09:11

Reasoning models represented by the Deepseek-R1-Distill series have been
widely adopted by the open-source community due to their strong performance in
mathematics, science, programming, and other domains. However, our study
reveals that their benchmark evaluation results are subject to significant
fluctuations caused by various factors. Subtle differences in evaluation
conditions can lead to substantial variations in results. Similar phenomena are
observed in other open-source inference models fine-tuned based on the
Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
claimed performance improvements difficult to reproduce reliably. Therefore, we
advocate for the establishment of a more rigorous paradigm for model
performance evaluation and present our empirical assessments of the
Deepseek-R1-Distill series models.
Qwen3 Embedding: Advancing Text Embedding and Reranking Through
  Foundation Models
Authors: name: Yanzhao Zhang, name: Mingxin Li, name: Dingkun Long, name: Xin Zhang, name: Huan Lin, name: Baosong Yang, name: Pengjun Xie, name: An Yang, name: Dayiheng Liu, name: Junyang Lin, name: Fei Huang, name: Jingren Zhou
Published: 2025-06-05 15:49:48

In this work, we introduce the Qwen3 Embedding series, a significant
advancement over its predecessor, the GTE-Qwen series, in text embedding and
reranking capabilities, built upon the Qwen3 foundation models. Leveraging the
Qwen3 LLMs' robust capabilities in multilingual text understanding and
generation, our innovative multi-stage training pipeline combines large-scale
unsupervised pre-training with supervised fine-tuning on high-quality datasets.
Effective model merging strategies further ensure the robustness and
adaptability of the Qwen3 Embedding series. During the training process, the
Qwen3 LLMs serve not only as backbone models but also play a crucial role in
synthesizing high-quality, rich, and diverse training data across multiple
domains and languages, thus enhancing the training pipeline. The Qwen3
Embedding series offers a spectrum of model sizes (0.6B, 4B, 8B) for both
embedding and reranking tasks, addressing diverse deployment scenarios where
users can optimize for either efficiency or effectiveness. Empirical
evaluations demonstrate that the Qwen3 Embedding series achieves
state-of-the-art results across diverse benchmarks. Notably, it excels on the
multilingual evaluation benchmark MTEB for text embedding, as well as in
various retrieval tasks, including code retrieval, cross-lingual retrieval and
multilingual retrieval. To facilitate reproducibility and promote
community-driven research and development, the Qwen3 Embedding models are
publicly available under the Apache 2.0 license.
VideoREPA: Learning Physics for Video Generation through Relational
  Alignment with Foundation Models
Authors: name: Xiangdong Zhang, name: Jiaqi Liao, name: Shaofeng Zhang, name: Fanqing Meng, name: Xiangpeng Wan, name: Junchi Yan, name: Yu Cheng
Published: 2025-05-29 17:06:44

Recent advancements in text-to-video (T2V) diffusion models have enabled
high-fidelity and realistic video synthesis. However, current T2V models often
struggle to generate physically plausible content due to their limited inherent
ability to accurately understand physics. We found that while the
representations within T2V models possess some capacity for physics
understanding, they lag significantly behind those from recent video
self-supervised learning methods. To this end, we propose a novel framework
called VideoREPA, which distills physics understanding capability from video
understanding foundation models into T2V models by aligning token-level
relations. This closes the physics understanding gap and enable more
physics-plausible generation. Specifically, we introduce the Token Relation
Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft
guidance suitable for finetuning powerful pre-trained T2V models, a critical
departure from prior representation alignment (REPA) methods. To our knowledge,
VideoREPA is the first REPA method designed for finetuning T2V models and
specifically for injecting physical knowledge. Empirical evaluations show that
VideoREPA substantially enhances the physics commonsense of baseline method,
CogVideoX, achieving significant improvement on relevant benchmarks and
demonstrating a strong capacity for generating videos consistent with intuitive
physics. More video results are available at https://videorepa.github.io/.
Contextual Integrity in LLMs via Reasoning and Reinforcement Learning
Authors: name: Guangchen Lan, name: Huseyin A. Inan, name: Sahar Abdelnabi, name: Janardhan Kulkarni, name: Lukas Wutschitz, name: Reza Shokri, name: Christopher G. Brinton, name: Robert Sim
Published: 2025-05-29 21:26:21

As the era of autonomous agents making decisions on behalf of users unfolds,
ensuring contextual integrity (CI) -- what is the appropriate information to
share while carrying out a certain task -- becomes a central question to the
field. We posit that CI demands a form of reasoning where the agent needs to
reason about the context in which it is operating. To test this, we first
prompt LLMs to reason explicitly about CI when deciding what information to
disclose. We then extend this approach by developing a reinforcement learning
(RL) framework that further instills in models the reasoning necessary to
achieve CI. Using a synthetic, automatically created, dataset of only sim700
examples but with diverse contexts and information disclosure norms, we show
that our method substantially reduces inappropriate information disclosure
while maintaining task performance across multiple model sizes and families.
Importantly, improvements transfer from this synthetic dataset to established
CI benchmarks such as PrivacyLens that has human annotations and evaluates
privacy leakage of AI assistants in actions and tool calls.
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Authors: name: Jiahui Wang, name: Zuyan Liu, name: Yongming Rao, name: Jiwen Lu
Published: 2025-06-05 17:59:55

Multimodal Large Language Models (MLLMs) are commonly derived by extending
pre-trained Large Language Models (LLMs) with visual capabilities. In this
work, we investigate how MLLMs process visual inputs by analyzing their
attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
subset (approximately less than 5%) of attention heads in LLMs actively
contribute to visual understanding, termed visual heads. To identify these
heads efficiently, we design a training-free framework that quantifies
head-level visual relevance through targeted response analysis. Building on
this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
allocates asymmetric computation budgets to heads in LLMs based on their visual
scores, leveraging the sparity of visual heads for accelerating the inference
of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
particularity of visual, SparseMM prioritizes stress and retaining visual
semantics during decoding. Extensive evaluations across mainstream multimodal
benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
memory reduction during generation while maintaining performance parity on
efficiency test. Our project is open sourced at
https://github.com/CR400AF-A/SparseMM.
Language-Image Alignment with Fixed Text Encoders
Authors: name: Jingfeng Yang, name: Ziyang Wu, name: Yue Zhao, name: Yi Ma
Published: 2025-06-04 17:51:56

Currently, the most dominant approach to establishing language-image
alignment is to pre-train text and image encoders jointly through contrastive
learning, such as CLIP and its variants. In this work, we question whether such
a costly joint training is necessary. In particular, we investigate if a
pre-trained fixed large language model (LLM) offers a good enough text encoder
to guide visual representation learning. That is, we propose to learn
Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by
training only the image encoder. Somewhat surprisingly, through comprehensive
benchmarking and ablation studies, we find that this much simplified framework
LIFT is highly effective and it outperforms CLIP in most scenarios that involve
compositional understanding and long captions, while achieving considerable
gains in computational efficiency. Our work takes a first step towards
systematically exploring how text embeddings from LLMs can guide visual
learning and suggests an alternative design choice for learning
language-aligned visual representations.
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language
  Models for Robotics
Authors: name: Enshen Zhou, name: Jingkun An, name: Cheng Chi, name: Yi Han, name: Shanyu Rong, name: Chi Zhang, name: Pengwei Wang, name: Zhongyuan Wang, name: Tiejun Huang, name: Lu Sheng, name: Shanghang Zhang
Published: 2025-06-04 17:59:27

Spatial referring is a fundamental capability of embodied robots to interact
with the 3D physical world. However, even with the powerful pretrained vision
language models (VLMs), recent approaches are still not qualified to accurately
understand the complex 3D scenes and dynamically reason about the
instruction-indicated locations for interaction. To this end, we propose
RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding
by integrating a disentangled but dedicated depth encoder via supervised
fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial
reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process
reward functions tailored for spatial referring tasks. To support SFT and RFT
training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x
prior), covering 31 spatial relations (vs. 15 prior) and supporting complex
reasoning processes (up to 5 steps). In addition, we introduce
RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial
referring with multi-step reasoning. Experiments show that SFT-trained
RoboRefer achieves state-of-the-art spatial understanding, with an average
success rate of 89.6%. RFT-trained RoboRefer further outperforms all other
baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average
accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various
control policies to execute long-horizon, dynamic tasks across diverse robots
(e,g., UR5, G1 humanoid) in cluttered real-world scenes.
Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric
  Approach
Authors: name: Ziheng Zhao, name: Lisong Dai, name: Ya Zhang, name: Yanfeng Wang, name: Weidi Xie
Published: 2025-06-03 17:57:34

Automated interpretation of CT images-particularly localizing and describing
abnormal findings across multi-plane and whole-body scans-remains a significant
challenge in clinical radiology. This work aims to address this challenge
through four key contributions: (i) On taxonomy, we collaborate with senior
radiologists to propose a comprehensive hierarchical classification system,
with 404 representative abnormal findings across all body regions; (ii) On
data, we contribute a dataset containing over 14.5K CT images from multiple
planes and all human body regions, and meticulously provide grounding
annotations for over 19K abnormalities, each linked to the detailed description
and cast into the taxonomy; (iii) On model development, we propose
OminiAbnorm-CT, which can automatically ground and describe abnormal findings
on multi-plane and whole-body CT images based on text queries, while also
allowing flexible interaction through visual prompts; (iv) On benchmarks, we
establish three representative evaluation tasks based on real clinical
scenarios. Through extensive experiments, we show that OminiAbnorm-CT can
significantly outperform existing methods on all the tasks and metrics.
Aligning Latent Spaces with Flow Priors
Authors: name: Yizhuo Li, name: Yuying Ge, name: Yixiao Ge, name: Ying Shan, name: Ping Luo
Published: 2025-06-05 16:59:53

This paper presents a novel framework for aligning learnable latent spaces to
arbitrary target distributions by leveraging flow-based generative models as
priors. Our method first pretrains a flow model on the target features to
capture the underlying distribution. This fixed flow model subsequently
regularizes the latent space via an alignment loss, which reformulates the flow
matching objective to treat the latents as optimization targets. We formally
prove that minimizing this alignment loss establishes a computationally
tractable surrogate objective for maximizing a variational lower bound on the
log-likelihood of latents under the target distribution. Notably, the proposed
method eliminates computationally expensive likelihood evaluations and avoids
ODE solving during optimization. As a proof of concept, we demonstrate in a
controlled setting that the alignment loss landscape closely approximates the
negative log-likelihood of the target distribution. We further validate the
effectiveness of our approach through large-scale image generation experiments
on ImageNet with diverse target distributions, accompanied by detailed
discussions and ablation studies. With both theoretical and empirical
validation, our framework paves a new way for latent space alignment.
MedAgentGym: Training LLM Agents for Code-Based Medical Reasoning at
  Scale
Authors: name: Ran Xu, name: Yuchen Zhuang, name: Yishan Zhong, name: Yue Yu, name: Xiangru Tang, name: Hang Wu, name: May D. Wang, name: Peifeng Ruan, name: Donghan Yang, name: Tao Wang, name: Guanghua Xiao, name: Carl Yang, name: Yang Xie, name: Wenqi Shi
Published: 2025-06-04 19:38:55

We introduce MedAgentGYM, the first publicly available training environment
designed to enhance coding-based medical reasoning capabilities in large
language model (LLM) agents. MedAgentGYM comprises 72,413 task instances across
129 categories derived from authentic real-world biomedical scenarios. Tasks
are encapsulated within executable coding environments, each featuring detailed
task descriptions, interactive feedback mechanisms, verifiable ground-truth
annotations, and scalable training trajectory generation. Extensive
benchmarking of over 30 LLMs reveals a notable performance disparity between
commercial API-based models and open-source counterparts. Leveraging
MedAgentGYM, Med-Copilot-7B achieves substantial performance gains through
supervised fine-tuning (+36.44%) and continued reinforcement learning
(+42.47%), emerging as an affordable and privacy-preserving alternative
competitive with gpt-4o. By offering both a comprehensive benchmark and
accessible, expandable training resources within unified execution
environments, MedAgentGYM delivers an integrated platform to develop LLM-based
coding assistants for advanced biomedical research and practice.
AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual
  Counting for MLLMs
Authors: name: Lidong Lu, name: Guo Chen, name: Zhiqi Li, name: Yicheng Liu, name: Tong Lu
Published: 2025-06-05 17:58:33

Despite progress in video understanding, current MLLMs struggle with counting
tasks. Existing benchmarks are limited by short videos, close-set queries, lack
of clue annotations, and weak multimodal coverage. In this paper, we introduce
CG-AV-Counting, a manually-annotated clue-grounded counting benchmark with
1,027 multimodal questions and 5,845 annotated clues over 497 long videos. It
supports both black-box and white-box evaluation, serving as a comprehensive
testbed for both end-to-end and reasoning-based counting. To explore ways to
improve model's counting capability, we propose AV-Reasoner, a model trained
with GRPO and curriculum learning to generalize counting ability from related
tasks. AV-Reasoner achieves state-of-the-art results across multiple
benchmarks, demonstrating the effectiveness of reinforcement learning. However,
experiments show that on out-of-domain benchmarks, reasoning in the language
space fails to bring performance gains. The code and benchmark have been
realeased on https://av-reasoner.github.io.