Orionnnn/hub-stats huggingface dataset:
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Authors: name: Shijie Zhou, name: Alexander Vilesov, name: Xuehai He, name: Ziyu Wan, name: Shuwang Zhang, name: Aditya Nagachandra, name: Di Chang, name: Dongdong Chen, name: Xin Eric Wang, name: Achuta Kadambi
Published: 2025-08-04 06:06:06

Vision language models (VLMs) have shown remarkable capabilities in
integrating linguistic and visual reasoning but remain fundamentally limited in
understanding dynamic spatiotemporal interactions. Humans effortlessly track
and reason about object movements, rotations, and perspective shifts-abilities
essential for robust dynamic real-world understanding yet notably lacking in
current VLMs. In this paper, we introduce VLM4D, the first benchmark
specifically designed to evaluate the spatiotemporal reasoning capabilities of
VLMs. Our benchmark comprises diverse real-world and synthetic videos
accompanied by carefully curated question-answer pairs emphasizing
translational and rotational motions, perspective awareness, and motion
continuity. Through comprehensive evaluations of state-of-the-art open and
closed-source VLMs, we identify significant performance gaps compared to human
baselines, highlighting fundamental deficiencies in existing models. Extensive
analysis reveals that VLMs struggle particularly with integrating multiple
visual cues and maintaining temporal coherence. We further explore promising
directions, such as leveraging 4D feature field reconstruction and targeted
spatiotemporal supervised fine-tuning, demonstrating their effectiveness in
enhancing spatiotemporal comprehension. Our work aims to encourage deeper
exploration into improving VLMs' spatial and temporal grounding, paving the way
towards more capable and reliable visual intelligence for dynamic environments.
Hidden Dynamics of Massive Activations in Transformer Training
Authors: name: Jorge Gallego-Feliciano, name: S. Aaron McClendon, name: Juan Morinelli, name: Stavros Zervoudakis, name: Antonios Saravanos
Published: 2025-08-05 16:29:51

Massive activations are scalar values in transformer hidden states that
achieve values orders of magnitude larger than typical activations and have
been shown to be critical for model functionality. While prior work has
characterized these phenomena in fully trained models, the temporal dynamics of
their emergence during training remain poorly understood. We present the first
comprehensive analysis of massive activation development throughout transformer
training, using the Pythia model family as our testbed. Through systematic
analysis of various model sizes across multiple training checkpoints, we
demonstrate that massive activation emergence follows predictable mathematical
patterns that can be accurately modeled using an exponentially-modulated
logarithmic function with five key parameters. We develop a machine learning
framework to predict these mathematical parameters from architectural
specifications alone, achieving high accuracy for steady-state behavior and
moderate accuracy for emergence timing and magnitude. These findings enable
architects to predict and potentially control key aspects of massive activation
emergence through design choices, with significant implications for model
stability, training cycle length, interpretability, and optimization. Our
findings demonstrate that the emergence of massive activations is governed by
model design and can be anticipated, and potentially controlled, before
training begins.
OS Agents: A Survey on MLLM-based Agents for General Computing Devices
  Use
Authors: name: Xueyu Hu, name: Tao Xiong, name: Biao Yi, name: Zishu Wei, name: Ruixuan Xiao, name: Yurun Chen, name: Jiasheng Ye, name: Meiling Tao, name: Xiangxin Zhou, name: Ziyu Zhao, name: Yuhuai Li, name: Shengze Xu, name: Shenzhi Wang, name: Xinchen Xu, name: Shuofei Qiao, name: Zhaokai Wang, name: Kun Kuang, name: Tieyong Zeng, name: Liang Wang, name: Jiwei Li, name: Yuchen Eleanor Jiang, name: Wangchunshu Zhou, name: Guoyin Wang, name: Keting Yin, name: Zhou Zhao, name: Hongxia Yang, name: Fan Wu, name: Shengyu Zhang, name: Fei Wu
Published: 2025-08-06 14:33:45

The dream to create AI assistants as capable and versatile as the fictional
J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution
of (multi-modal) large language models ((M)LLMs), this dream is closer to
reality, as (M)LLM-based Agents using computing devices (e.g., computers and
mobile phones) by operating within the environments and interfaces (e.g.,
Graphical User Interface (GUI)) provided by operating systems (OS) to automate
tasks have significantly advanced. This paper presents a comprehensive survey
of these advanced agents, designated as OS Agents. We begin by elucidating the
fundamentals of OS Agents, exploring their key components including the
environment, observation space, and action space, and outlining essential
capabilities such as understanding, planning, and grounding. We then examine
methodologies for constructing OS Agents, focusing on domain-specific
foundation models and agent frameworks. A detailed review of evaluation
protocols and benchmarks highlights how OS Agents are assessed across diverse
tasks. Finally, we discuss current challenges and identify promising directions
for future research, including safety and privacy, personalization and
self-evolution. This survey aims to consolidate the state of OS Agents
research, providing insights to guide both academic inquiry and industrial
development. An open-source GitHub repository is maintained as a dynamic
resource to foster further innovation in this field. We present a 9-page
version of our work, accepted by ACL 2025, to provide a concise overview to the
domain.
LightSwitch: Multi-view Relighting with Material-guided Diffusion
Authors: name: Yehonathan Litman, name: Fernando De la Torre, name: Shubham Tulsiani
Published: 2025-08-08 17:59:52

Recent approaches for 3D relighting have shown promise in integrating 2D
image relighting generative priors to alter the appearance of a 3D
representation while preserving the underlying structure. Nevertheless,
generative priors used for 2D relighting that directly relight from an input
image do not take advantage of intrinsic properties of the subject that can be
inferred or cannot consider multi-view data at scale, leading to subpar
relighting. In this paper, we propose Lightswitch, a novel finetuned
material-relighting diffusion framework that efficiently relights an arbitrary
number of input images to a target lighting condition while incorporating cues
from inferred intrinsic properties. By using multi-view and material
information cues together with a scalable denoising scheme, our method
consistently and efficiently relights dense multi-view data of objects with
diverse material compositions. We show that our 2D relighting prediction
quality exceeds previous state-of-the-art relighting priors that directly
relight from images. We further demonstrate that LightSwitch matches or
outperforms state-of-the-art diffusion inverse rendering methods in relighting
synthetic and real objects in as little as 2 minutes.
Memp: Exploring Agent Procedural Memory
Authors: name: Runnan Fang, name: Yuan Liang, name: Xiaobin Wang, name: Jialong Wu, name: Shuofei Qiao, name: Pengjun Xie, name: Fei Huang, name: Huajun Chen, name: Ningyu Zhang
Published: 2025-08-08 16:20:56

Large Language Models (LLMs) based agents excel at diverse tasks, yet they
suffer from brittle procedural memory that is manually engineered or entangled
in static parameters. In this work, we investigate strategies to endow agents
with a learnable, updatable, and lifelong procedural memory. We propose Memp
that distills past agent trajectories into both fine-grained, step-by-step
instructions and higher-level, script-like abstractions, and explore the impact
of different strategies for Build, Retrieval, and Update of procedural memory.
Coupled with a dynamic regimen that continuously updates, corrects, and
deprecates its contents, this repository evolves in lockstep with new
experience. Empirical evaluation on TravelPlanner and ALFWorld shows that as
the memory repository is refined, agents achieve steadily higher success rates
and greater efficiency on analogous tasks. Moreover, procedural memory built
from a stronger model retains its value: migrating the procedural memory to a
weaker model yields substantial performance gains.
GENIE: Gaussian Encoding for Neural Radiance Fields Interactive Editing
Authors: name: Mikołaj Zieliński, name: Krzysztof Byrski, name: Tomasz Szczepanik, name: Przemysław Spurek
Published: 2025-08-04 18:59:23

Neural Radiance Fields (NeRF) and Gaussian Splatting (GS) have recently
transformed 3D scene representation and rendering. NeRF achieves high-fidelity
novel view synthesis by learning volumetric representations through neural
networks, but its implicit encoding makes editing and physical interaction
challenging. In contrast, GS represents scenes as explicit collections of
Gaussian primitives, enabling real-time rendering, faster training, and more
intuitive manipulation. This explicit structure has made GS particularly
well-suited for interactive editing and integration with physics-based
simulation. In this paper, we introduce GENIE (Gaussian Encoding for Neural
Radiance Fields Interactive Editing), a hybrid model that combines the
photorealistic rendering quality of NeRF with the editable and structured
representation of GS. Instead of using spherical harmonics for appearance
modeling, we assign each Gaussian a trainable feature embedding. These
embeddings are used to condition a NeRF network based on the k nearest
Gaussians to each query point. To make this conditioning efficient, we
introduce Ray-Traced Gaussian Proximity Search (RT-GPS), a fast nearest
Gaussian search based on a modified ray-tracing pipeline. We also integrate a
multi-resolution hash grid to initialize and update Gaussian features.
Together, these components enable real-time, locality-aware editing: as
Gaussian primitives are repositioned or modified, their interpolated influence
is immediately reflected in the rendered output. By combining the strengths of
implicit and explicit representations, GENIE supports intuitive scene
manipulation, dynamic interaction, and compatibility with physical simulation,
bridging the gap between geometry-based editing and neural rendering. The code
can be found under (https://github.com/MikolajZielinski/genie)
MeshLLM: Empowering Large Language Models to Progressively Understand
  and Generate 3D Mesh
Authors: name: Shuangkang Fang, name: I-Chao Shen, name: Yufeng Wang, name: Yi-Hsuan Tsai, name: Yi Yang, name: Shuchang Zhou, name: Wenrui Ding, name: Takeo Igarashi, name: Ming-Hsuan Yang
Published: 2025-08-02 07:37:37

We present MeshLLM, a novel framework that leverages large language models
(LLMs) to understand and generate text-serialized 3D meshes. Our approach
addresses key limitations in existing methods, including the limited dataset
scale when catering to LLMs' token length and the loss of 3D structural
information during mesh serialization. We introduce a Primitive-Mesh
decomposition strategy, which divides 3D meshes into structurally meaningful
subunits. This enables the creation of a large-scale dataset with 1500k+
samples, almost 50 times larger than previous methods, which aligns better with
the LLM scaling law principles. Furthermore, we propose inferring face
connectivity from vertices and local mesh assembly training strategies,
significantly enhancing the LLMs' ability to capture mesh topology and spatial
structures. Experiments show that MeshLLM outperforms the state-of-the-art
LLaMA-Mesh in both mesh generation quality and shape understanding,
highlighting its great potential in processing text-serialized 3D meshes.
UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and
  Precise Inference-Time Grounding
Authors: name: Shuquan Lian, name: Yuhang Wu, name: Jia Ma, name: Zihan Song, name: Bingqi Chen, name: Xiawu Zheng, name: Hui Li
Published: 2025-07-29 17:22:07

The emergence of Multimodal Large Language Models (MLLMs) has driven
significant advances in Graphical User Interface (GUI) agent capabilities.
Nevertheless, existing GUI agent training and inference techniques still suffer
from a dilemma for reasoning designs, ineffective reward, and visual noise. To
address these issues, we introduce UI-AGILE, a comprehensive framework
enhancing GUI agents at both the training and inference stages. For training,
we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process:
1) a Continuous Reward function to incentivize high-precision grounding; 2) a
"Simple Thinking" reward to balance planning with speed and grounding accuracy;
and 3) a Cropping-based Resampling strategy to mitigate the sparse reward
problem and improve learning on complex tasks. For inference, we present
Decomposed Grounding with Selection, a novel method that dramatically improves
grounding accuracy on high-resolution displays by breaking the image into
smaller, manageable parts. Experiments show that UI-AGILE achieves the
state-of-the-art performance on two benchmarks ScreenSpot-Pro and
ScreenSpot-v2. For instance, using both our proposed training and inference
enhancement methods brings 23% grounding accuracy improvement over the best
baseline on ScreenSpot-Pro.
Adapting Vision-Language Models Without Labels: A Comprehensive Survey
Authors: name: Hao Dong, name: Lijun Sheng, name: Jian Liang, name: Ran He, name: Eleni Chatzi, name: Olga Fink
Published: 2025-08-07 16:27:37

Vision-Language Models (VLMs) have demonstrated remarkable generalization
capabilities across a wide range of tasks. However, their performance often
remains suboptimal when directly applied to specific downstream scenarios
without task-specific adaptation. To enhance their utility while preserving
data efficiency, recent research has increasingly focused on unsupervised
adaptation methods that do not rely on labeled data. Despite the growing
interest in this area, there remains a lack of a unified, task-oriented survey
dedicated to unsupervised VLM adaptation. To bridge this gap, we present a
comprehensive and structured overview of the field. We propose a taxonomy based
on the availability and nature of unlabeled visual data, categorizing existing
approaches into four key paradigms: Data-Free Transfer (no data), Unsupervised
Domain Transfer (abundant data), Episodic Test-Time Adaptation (batch data),
and Online Test-Time Adaptation (streaming data). Within this framework, we
analyze core methodologies and adaptation strategies associated with each
paradigm, aiming to establish a systematic understanding of the field.
Additionally, we review representative benchmarks across diverse applications
and highlight open challenges and promising directions for future research. An
actively maintained repository of relevant literature is available at
https://github.com/tim-learn/Awesome-LabelFree-VLMs.
Voost: A Unified and Scalable Diffusion Transformer for Bidirectional
  Virtual Try-On and Try-Off
Authors: name: Seungyong Lee, name: Jeong-gi Kwak
Published: 2025-08-06 19:10:58

Virtual try-on aims to synthesize a realistic image of a person wearing a
target garment, but accurately modeling garment-body correspondence remains a
persistent challenge, especially under pose and appearance variation. In this
paper, we propose Voost - a unified and scalable framework that jointly learns
virtual try-on and try-off with a single diffusion transformer. By modeling
both tasks jointly, Voost enables each garment-person pair to supervise both
directions and supports flexible conditioning over generation direction and
garment category, enhancing garment-body relational reasoning without
task-specific networks, auxiliary losses, or additional labels. In addition, we
introduce two inference-time techniques: attention temperature scaling for
robustness to resolution or mask variation, and self-corrective sampling that
leverages bidirectional consistency between tasks. Extensive experiments
demonstrate that Voost achieves state-of-the-art results on both try-on and
try-off benchmarks, consistently outperforming strong baselines in alignment
accuracy, visual fidelity, and generalization.
Pruning the Unsurprising: Efficient Code Reasoning via First-Token
  Surprisal
Authors: name: Wenhao Zeng, name: Yaoning Wang, name: Chao Hu, name: Yuling Shi, name: Chengcheng Wan, name: Hongyu Zhang, name: Xiaodong Gu
Published: 2025-08-08 03:46:21

Recently, Large Reasoning Models (LRMs) have demonstrated remarkable
capabilities in code reasoning by scaling up the length of Chain-of-Thought
(CoT). However, excessively long reasoning traces introduce substantial
challenges in terms of training cost, inference latency, and deployment
feasibility. While various CoT compression approaches have emerged to address
this challenge, they face inherent trade-offs: token-level methods often
disrupt syntactic and logical coherence, while step-level methods based on
perplexity fail to reliably capture the logically critical reasoning steps. In
this paper, we propose ASAP (Anchor-guided, Surprisal-based Pruning), a novel
coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided
pruning to preserve the core reasoning structure, which efficiently reduces the
search space for subsequent processing. It then enables a logic-aware pruning
by selecting logically essential reasoning steps based on a novel first-token
surprisal metric. Finally, ASAP teaches models to autonomously generate and
leverage these concise CoTs at inference time, enabling efficient reasoning in
coding tasks. Experiments show that ASAP achieves state-of-the-art accuracy
across multiple code generation benchmarks while substantially reducing
training and inference costs. On the challenging LiveCodeBench v4_v5 benchmark,
our approach reduces token generation by 23.5% and inference latency by 43.5%
compared to the strongest baseline, while achieving a competitive accuracy of
36.19% in Pass@1. Our results highlight a promising direction for building
powerful and efficient LRMs.
InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy
  Optimization
Authors: name: Yuhang Liu, name: Zeyu Liu, name: Shuanghe Zhu, name: Pengxiang Li, name: Congkai Xie, name: Jiasheng Wang, name: Xueyu Hu, name: Xiaotian Han, name: Jianbo Yuan, name: Xinyao Wang, name: Shengyu Zhang, name: Hongxia Yang, name: Fei Wu
Published: 2025-08-07 17:49:56

The emergence of Multimodal Large Language Models (MLLMs) has propelled the
development of autonomous agents that operate on Graphical User Interfaces
(GUIs) using pure visual input. A fundamental challenge is robustly grounding
natural language instructions. This requires a precise spatial alignment, which
accurately locates the coordinates of each element, and, more critically, a
correct semantic alignment, which matches the instructions to the functionally
appropriate UI element. Although Reinforcement Learning with Verifiable Rewards
(RLVR) has proven to be effective at improving spatial alignment for these
MLLMs, we find that inefficient exploration bottlenecks semantic alignment,
which prevent models from learning difficult semantic associations. To address
this exploration problem, we present Adaptive Exploration Policy Optimization
(AEPO), a new policy optimization framework. AEPO employs a multi-answer
generation strategy to enforce broader exploration, which is then guided by a
theoretically grounded Adaptive Exploration Reward (AER) function derived from
first principles of efficiency eta=U/C. Our AEPO-trained models, InfiGUI-G1-3B
and InfiGUI-G1-7B, establish new state-of-the-art results across multiple
challenging GUI grounding benchmarks, achieving significant relative
improvements of up to 9.0% against the naive RLVR baseline on benchmarks
designed to test generalization and semantic understanding. Resources are
available at https://github.com/InfiXAI/InfiGUI-G1.
MELLA: Bridging Linguistic Capability and Cultural Groundedness for
  Low-Resource Language MLLMs
Authors: name: Yufei Gao, name: Jiaying Fei, name: Nuo Chen, name: Ruirui Chen, name: Guohang Yan, name: Yunshi Lan, name: Botian Shi
Published: 2025-08-07 15:36:24

Multimodal Large Language Models (MLLMs) have shown remarkable performance in
high-resource languages. However, their effectiveness diminishes significantly
in the contexts of low-resource languages. Current multilingual enhancement
methods are often limited to text modality or rely solely on machine
translation. While such approaches help models acquire basic linguistic
capabilities and produce "thin descriptions", they neglect the importance of
multimodal informativeness and cultural groundedness, both of which are crucial
for serving low-resource language users effectively. To bridge this gap, in
this study, we identify two significant objectives for a truly effective MLLM
in low-resource language settings, namely 1) linguistic capability and 2)
cultural groundedness, placing special emphasis on cultural awareness. To
achieve these dual objectives, we propose a dual-source strategy that guides
the collection of data tailored to each goal, sourcing native web alt-text for
culture and MLLM-generated captions for linguistics. As a concrete
implementation, we introduce MELLA, a multimodal, multilingual dataset.
Experiment results show that after fine-tuning on MELLA, there is a general
performance improvement for the eight languages on various MLLM backbones, with
models producing "thick descriptions". We verify that the performance gains are
from both cultural knowledge enhancement and linguistic capability enhancement.
Our dataset can be found at https://opendatalab.com/applyMultilingualCorpus.
Learning to Reason for Factuality
Authors: name: Xilun Chen, name: Ilia Kulikov, name: Vincent-Pierre Berges, name: Barlas Oğuz, name: Rulin Shao, name: Gargi Ghosh, name: Jason Weston, name: Wen-tau Yih
Published: 2025-08-07 17:57:09

Reasoning Large Language Models (R-LLMs) have significantly advanced complex
reasoning tasks but often struggle with factuality, generating substantially
more hallucinations than their non-reasoning counterparts on long-form
factuality benchmarks. However, extending online Reinforcement Learning (RL), a
key component in recent R-LLM advancements, to the long-form factuality setting
poses several unique challenges due to the lack of reliable verification
methods. Previous work has utilized automatic factuality evaluation frameworks
such as FActScore to curate preference data in the offline RL setting, yet we
find that directly leveraging such methods as the reward in online RL leads to
reward hacking in multiple ways, such as producing less detailed or relevant
responses. We propose a novel reward function that simultaneously considers the
factual precision, response detail level, and answer relevance, and applies
online RL to learn high quality factual reasoning. Evaluated on six long-form
factuality benchmarks, our factual reasoning model achieves an average
reduction of 23.1 percentage points in hallucination rate, a 23% increase in
answer detail level, and no degradation in the overall response helpfulness.