Orionnnn/hub-stats huggingface dataset:
MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame
  Optical Flow Estimation
Authors: name: Vladislav Bargatin, name: Egor Chistov, name: Alexander Yakovenko, name: Dmitriy Vatolin
Published: 2025-06-29 09:01:42

Recent advances in optical flow estimation have prioritized accuracy at the
cost of growing GPU memory consumption, particularly for high-resolution
(FullHD) inputs. We introduce MEMFOF, a memory-efficient multi-frame optical
flow method that identifies a favorable trade-off between multi-frame
estimation and GPU memory usage. Notably, MEMFOF requires only 2.09 GB of GPU
memory at runtime for 1080p inputs, and 28.5 GB during training, which uniquely
positions our method to be trained at native 1080p without the need for
cropping or downsampling. We systematically revisit design choices from
RAFT-like architectures, integrating reduced correlation volumes and
high-resolution training protocols alongside multi-frame estimation, to achieve
state-of-the-art performance across multiple benchmarks while substantially
reducing memory overhead. Our method outperforms more resource-intensive
alternatives in both accuracy and runtime efficiency, validating its robustness
for flow estimation at high resolutions. At the time of submission, our method
ranks first on the Spring benchmark with a 1-pixel (1px) outlier rate of 3.289,
leads Sintel (clean) with an endpoint error (EPE) of 0.963, and achieves the
best Fl-all error on KITTI-2015 at 2.94%. The code is available at
https://github.com/msu-video-group/memfof.
Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
Authors: name: Jianing Zhang, name: Jiayi Zhu, name: Feiyu Ji, name: Xiaokang Yang, name: Xiaoyun Yuan
Published: 2025-06-28 04:48:37

Metalenses offer significant potential for ultra-compact computational
imaging but face challenges from complex optical degradation and computational
restoration difficulties. Existing methods typically rely on precise optical
calibration or massive paired datasets, which are non-trivial for real-world
imaging systems. Furthermore, a lack of control over the inference process
often results in undesirable hallucinated artifacts. We introduce
Degradation-Modeled Multipath Diffusion for tunable metalens photography,
leveraging powerful natural image priors from pretrained models instead of
large datasets. Our framework uses positive, neutral, and negative-prompt paths
to balance high-frequency detail generation, structural fidelity, and
suppression of metalens-specific degradation, alongside pseudo data
augmentation. A tunable decoder enables controlled trade-offs between fidelity
and perceptual quality. Additionally, a spatially varying degradation-aware
attention (SVDA) module adaptively models complex optical and sensor-induced
degradation. Finally, we design and build a millimeter-scale MetaCamera for
real-world validation. Extensive results show that our approach outperforms
state-of-the-art methods, achieving high-fidelity and sharp image
reconstruction. More materials: https://dmdiff.github.io/.
Listener-Rewarded Thinking in VLMs for Image Preferences
Authors: name: Alexander Gambashidze, name: Li Pengyi, name: Matvey Skripkin, name: Andrey Galichin, name: Anton Gusarov, name: Konstantin Sobolev, name: Andrey Kuznetsov, name: Ivan Oseledets
Published: 2025-06-28 09:53:17

Training robust and generalizable reward models for human visual preferences
is essential for aligning text-to-image and text-to-video generative models
with human intent. However, current reward models often fail to generalize, and
supervised fine-tuning leads to memorization, demanding complex annotation
pipelines. While reinforcement learning (RL), specifically Group Relative
Policy Optimization (GRPO), improves generalization, we uncover a key failure
mode: a significant drop in reasoning accuracy occurs when a model's reasoning
trace contradicts that of an independent, frozen vision-language model
("listener") evaluating the same output. To address this, we introduce a
listener-augmented GRPO framework. Here, the listener re-evaluates the
reasoner's chain-of-thought to provide a dense, calibrated confidence score,
shaping the RL reward signal. This encourages the reasoner not only to answer
correctly, but to produce explanations that are persuasive to an independent
model. Our listener-shaped reward scheme achieves best accuracy on the
ImageReward benchmark (67.4%), significantly improves out-of-distribution (OOD)
performance on a large-scale human preference dataset (1.2M votes, up to +6%
over naive reasoner), and reduces reasoning contradictions compared to strong
GRPO and SFT baselines. These results demonstrate that listener-based rewards
provide a scalable, data-efficient path to aligning vision-language models with
nuanced human preferences. We will release our reasoning model here:
https://huggingface.co/alexgambashidze/qwen2.5vl_image_preference_reasoner.
RoboScape: Physics-informed Embodied World Model
Authors: name: Yu Shang, name: Xin Zhang, name: Yinzhou Tang, name: Lei Jin, name: Chen Gao, name: Wei Wu, name: Yong Li
Published: 2025-06-29 08:19:45

World models have become indispensable tools for embodied intelligence,
serving as powerful simulators capable of generating realistic robotic videos
while addressing critical data scarcity challenges. However, current embodied
world models exhibit limited physical awareness, particularly in modeling 3D
geometry and motion dynamics, resulting in unrealistic video generation for
contact-rich robotic scenarios. In this paper, we present RoboScape, a unified
physics-informed world model that jointly learns RGB video generation and
physics knowledge within an integrated framework. We introduce two key
physics-informed joint training tasks: temporal depth prediction that enhances
3D geometric consistency in video rendering, and keypoint dynamics learning
that implicitly encodes physical properties (e.g., object shape and material
characteristics) while improving complex motion modeling. Extensive experiments
demonstrate that RoboScape generates videos with superior visual fidelity and
physical plausibility across diverse robotic scenarios. We further validate its
practical utility through downstream applications including robotic policy
training with generated data and policy evaluation. Our work provides new
insights for building efficient physics-informed world models to advance
embodied intelligence research. The code is available at:
https://github.com/tsinghua-fib-lab/RoboScape.
MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning
Authors: name: Yulun Jiang, name: Yekun Chai, name: Maria Brbić, name: Michael Moor
Published: 2025-06-28 19:44:32

The ability to process information from multiple modalities and to reason
through it step-by-step remains a critical challenge in advancing artificial
intelligence. However, existing reasoning benchmarks focus on text-only
reasoning, or employ multimodal questions that can be answered by directly
retrieving information from a non-text modality. Thus, complex reasoning
remains poorly understood in multimodal domains. Here, we present MARBLE, a
challenging multimodal reasoning benchmark that is designed to scrutinize
multimodal language models (MLLMs) in their ability to carefully reason
step-by-step through complex multimodal problems and environments. MARBLE is
composed of two highly challenging tasks, M-Portal and M-Cube, that require the
crafting and understanding of multistep plans under spatial, visual, and
physical constraints. We find that current MLLMs perform poorly on MARBLE --
all the 12 advanced models obtain near-random performance on M-Portal and 0%
accuracy on M-Cube. Only in simplified subtasks some models outperform the
random baseline, indicating that complex reasoning is still a challenge for
existing MLLMs. Moreover, we show that perception remains a bottleneck, where
MLLMs occasionally fail to extract information from the visual inputs. By
shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the
development of the next generation of models with the ability to reason and
plan across many, multimodal reasoning steps.
Ovis-U1 Technical Report
Authors: name: Guo-Hua Wang, name: Shanshan Zhao, name: Xinjie Zhang, name: Liangfu Cao, name: Pengxin Zhan, name: Lunhao Duan, name: Shiyin Lu, name: Minghao Fu, name: Xiaohao Chen, name: Jianshan Zhao, name: Yang Li, name: Qing-Guo Chen
Published: 2025-06-29 00:40:17

In this report, we introduce Ovis-U1, a 3-billion-parameter unified model
that integrates multimodal understanding, text-to-image generation, and image
editing capabilities. Building on the foundation of the Ovis series, Ovis-U1
incorporates a diffusion-based visual decoder paired with a bidirectional token
refiner, enabling image generation tasks comparable to leading models like
GPT-4o. Unlike some previous models that use a frozen MLLM for generation
tasks, Ovis-U1 utilizes a new unified training approach starting from a
language model. Compared to training solely on understanding or generation
tasks, unified training yields better performance, demonstrating the
enhancement achieved by integrating these two tasks. Ovis-U1 achieves a score
of 69.6 on the OpenCompass Multi-modal Academic Benchmark, surpassing recent
state-of-the-art models such as Ristretto-3B and SAIL-VL-1.5-2B. In
text-to-image generation, it excels with scores of 83.72 and 0.89 on the
DPG-Bench and GenEval benchmarks, respectively. For image editing, it achieves
4.00 and 6.42 on the ImgEdit-Bench and GEdit-Bench-EN, respectively. As the
initial version of the Ovis unified model series, Ovis-U1 pushes the boundaries
of multimodal understanding, generation, and editing.
UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence
  with Spatial Reasoning and Understanding
Authors: name: Jie Feng, name: Shengyuan Wang, name: Tianhui Liu, name: Yanxin Xi, name: Yong Li
Published: 2025-06-29 13:04:27

Urban research involves a wide range of scenarios and tasks that require the
understanding of multi-modal data. Current methods often focus on specific data
types and lack a unified framework in urban field for processing them
comprehensively. The recent success of multi-modal large language models
(MLLMs) presents a promising opportunity to overcome this limitation. In this
paper, we introduce UrbanLLaVA, a multi-modal large language model
designed to process these four types of data simultaneously and achieve strong
performance across diverse urban tasks compared with general MLLMs. In
UrbanLLaVA, we first curate a diverse urban instruction dataset
encompassing both single-modal and cross-modal urban data, spanning from
location view to global view of urban environment. Additionally, we propose a
multi-stage training framework that decouples spatial reasoning enhancement
from domain knowledge learning, thereby improving the compatibility and
downstream performance of UrbanLLaVA across diverse urban tasks.
Finally, we also extend existing benchmark for urban research to assess the
performance of MLLMs across a wide range of urban tasks. Experimental results
from three cities demonstrate that UrbanLLaVA outperforms
open-source and proprietary MLLMs in both single-modal tasks and complex
cross-modal tasks and shows robust generalization abilities across cities.
Source codes and data are openly accessible to the research community via
https://github.com/tsinghua-fib-lab/UrbanLLaVA.
VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs
Authors: name: Raghavv Goel, name: Sudhanshu Agrawal, name: Mukul Gagrani, name: Junyoung Park, name: Yifan Zao, name: He Zhang, name: Tian Liu, name: Yiping Yang, name: Xin Yuan, name: Jiuyan Lu, name: Chris Lott, name: Mingu Lee
Published: 2025-06-28 00:26:40

In this paper, we introduce a simple training-free technique to improve the
performance of drafter-based speculative decoding (SpD) methods that
incorporates language modeling head (LM head) during drafting process. A
drafter-based speculative decoding leverages one or more smaller language
models, a.k.a. drafters or draft models, to sample a draft sequence or tree
consisting of multiple tokens, followed by verification by a base LLM, a target
model, accepting a subset as its valid generation. As it is usually considered
that the speculative decoding requires one-to-one mapping between vocabularies
of the target model and the draft model, it has been natural to share the
vocabulary between them, or even share the LM head as in EAGLE or Medusa. We
first identify that this draft token sampling scheme inherently contains an
unnecessary inference overhead in drafting, especially for some target LLMs
with very large vocabularies. Then, we propose a simple technique, VocabTrim,
to mitigate the drafting overhead to improve the generation speed in
memory-bound environment. VocabTrim reconstructs the drafter LM head to contain
only a limited set of tokens, selected by the most frequently sampled from the
vocabulary of the target model. While limiting the vocabulary in drafting
slightly degrades the acceptance rate, it significantly reduces the drafting
latency in memory-bound process which is often the case on edge devices,
resulting in higher memory-bound speed up (MBSU). We show that our method can
boost the memory-bound speed-up for Llama-3 models on Spec-Bench, specifically
by 16% for Llama-3.2-3B-Instruct.
ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language
  Models for Audio Generation and Editing
Authors: name: Huadai Liu, name: Jialei Wang, name: Kaicheng Luo, name: Wen Wang, name: Qian Chen, name: Zhou Zhao, name: Wei Xue
Published: 2025-06-26 16:32:06

While end-to-end video-to-audio generation has greatly improved, producing
high-fidelity audio that authentically captures the nuances of visual content
remains challenging. Like professionals in the creative industries, such
generation requires sophisticated reasoning about items such as visual
dynamics, acoustic environments, and temporal relationships. We present
ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning
to enable stepwise, interactive audio generation and editing for videos. Our
approach decomposes the process into three complementary stages: foundational
foley generation that creates semantically coherent soundscapes, interactive
object-centric refinement through precise user interactions, and targeted
editing guided by natural language instructions. At each stage, a multimodal
large language model generates contextually aligned CoT reasoning that guides a
unified audio foundation model. Furthermore, we introduce AudioCoT, a
comprehensive dataset with structured reasoning annotations that establishes
connections between visual content, textual descriptions, and sound synthesis.
Experiments demonstrate that ThinkSound achieves state-of-the-art performance
in video-to-audio generation across both audio metrics and CoT metrics and
excels in out-of-distribution Movie Gen Audio benchmark. The demo page is
available at https://ThinkSound-Project.github.io.
SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
Authors: name: Samir Khaki, name: Xiuyu Li, name: Junxian Guo, name: Ligeng Zhu, name: Chenfeng Xu, name: Konstantinos N. Plataniotis, name: Amir Yazdanbakhsh, name: Kurt Keutzer, name: Song Han, name: Zhijian Liu
Published: 2025-06-19 17:53:34

Fine-tuning LLMs is both computationally and memory-intensive. While
parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the
number of trainable parameters and lower memory usage, they do not decrease
computational cost. In some cases, they may even slow down fine-tuning. In this
paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning
through contextual sparsity. We propose a lightweight, training-free SVD
sparsity estimator that dynamically selects a sparse subset of weights for loss
and gradient computation. Also, we systematically analyze and address
sensitivity across layers, tokens, and training steps. Our experimental results
show that SparseLoRA reduces computational cost by up to 2.2 times and a
measured speedup of up to 1.6 times while maintaining accuracy across various
downstream tasks, including commonsense and arithmetic reasoning, code
generation, and instruction following.
Teaching a Language Model to Speak the Language of Tools
Authors: name: Simeon Emanuilov
Published: 2025-06-29 20:47:27

External tool integration through function-calling is essential for practical
language model applications, yet most multilingual models lack reliable
tool-use capabilities in non-English languages. Even state-of-the-art
multilingual models struggle with determining when to use tools and generating
the structured outputs required for function calls, often exhibiting language
confusion when prompted in lower-resource languages. This work presents a
methodology for adapting existing language models to enable robust tool use in
any target language, using Bulgarian as a case study. The approach involves
continued training of the BgGPT model series (2.6B, 9B, 27B parameters) on a
novel bilingual dataset of 10,035 function-calling examples designed to support
standardized protocols like MCP (Model Context Protocol). The research
introduces TUCAN (Tool-Using Capable Assistant Navigator), which achieves up to
28.75% improvement in function-calling accuracy over base models while
preserving core language understanding, as verified on established Bulgarian
benchmarks. Beyond accuracy gains, TUCAN models demonstrate production-ready
response formatting with clean, parsable function calls, contrasting with the
verbose and inconsistent outputs of base models. The models, evaluation
framework, and dataset are released to enable replication for other languages.
This work demonstrates a practical approach for extending tool-augmented
capabilities beyond English-centric systems.
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via
  Multi-Agent Multi-Turn Reinforcement Learning
Authors: name: Bo Liu, name: Leon Guertler, name: Simon Yu, name: Zichen Liu, name: Penghui Qi, name: Daniel Balcells, name: Mickel Liu, name: Cheston Tan, name: Weiyan Shi, name: Min Lin, name: Wee Sun Lee, name: Natasha Jaques
Published: 2025-06-30 17:58:13

Recent advances in reinforcement learning have shown that language models can
develop sophisticated reasoning through training on tasks with verifiable
rewards, but these approaches depend on human-curated problem-answer pairs and
domain-specific reward engineering. We introduce SPIRAL, a self-play framework
where models learn by playing multi-turn, zero-sum games against continuously
improving versions of themselves, eliminating the need for human supervision.
Through self-play, SPIRAL generates an infinite curriculum of progressively
challenging problems as models must constantly adapt to stronger opponents. To
enable this self-play training at scale, We implement a fully online,
multi-turn, multi-agent reinforcement learning system for LLMs and propose
role-conditioned advantage estimation (RAE) to stabilize multi-agent training.
Using SPIRAL, self-play on zero-sum games produces reasoning capabilities that
transfer broadly. Training Qwen3-4B-Base on Kuhn Poker alone achieves 8.6%
improvement on math and 8.4% on general reasoning, outperforming SFT on 25,000
expert game trajectories. Analysis reveals that this transfer occurs through
three cognitive patterns: systematic decomposition, expected value calculation,
and case-by-case analysis. Multi-game training (TicTacToe, Kuhn Poker, Simple
Negotiation) further enhances performance as each game develops distinct
reasoning strengths. Applying SPIRAL to a strong reasoning model
(DeepSeek-R1-Distill-Qwen-7B) can still lead to 2.0% average improvement. These
results demonstrate that zero-sum games naturally develop transferable
reasoning capabilities, highlighting a promising direction for autonomous
reasoning development.
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Authors: name: Jianyu Wang, name: Zhiqiang Hu, name: Lidong Bing
Published: 2025-06-22 07:53:07

We propose a novel prompt design paradigm that challenges conventional wisdom
in large language model (LLM) prompting. While conventional wisdom prioritizes
well-crafted instructions and demonstrations for in-context learning (ICL), we
show that pruning random demonstrations into seemingly incoherent "gibberish"
can remarkably improve performance across diverse tasks. Notably, the
"gibberish" always matches or surpasses state-of-the-art automatic prompt
optimization techniques, achieving substantial gains regardless of LLM
alignment. Nevertheless, discovering an effective pruning strategy is
non-trivial, as existing attribution methods and prompt compression algorithms
fail to deliver robust results, let alone human intuition. In terms of this, we
propose a self-discover prompt optimization framework, PromptQuine, an
evolutionary search framework that automatically searches for the pruning
strategy by itself using only low-data regimes. Much like the emergent
complexity in nature--such as symbiosis and self-organization--arising in
response to resource constraints, our framework evolves and refines
unconventional yet highly effective prompts by leveraging only the tokens
present within the context. We demonstrate its effectiveness across
classification, multi-choice question answering, generation and math reasoning
tasks across LLMs, while achieving decent runtime efficiency. We hope our
findings can guide mechanistic studies on in-context learning, and provide a
call to action, to pave the way for more open-ended search algorithms for more
effective LLM prompting.
Consistent Time-of-Flight Depth Denoising via Graph-Informed Geometric
  Attention
Authors: name: Weida Wang, name: Changyong He, name: Jin Zeng, name: Di Qiu
Published: 2025-06-30 06:29:24

Depth images captured by Time-of-Flight (ToF) sensors are prone to noise,
requiring denoising for reliable downstream applications. Previous works either
focus on single-frame processing, or perform multi-frame processing without
considering depth variations at corresponding pixels across frames, leading to
undesirable temporal inconsistency and spatial ambiguity. In this paper, we
propose a novel ToF depth denoising network leveraging motion-invariant graph
fusion to simultaneously enhance temporal stability and spatial sharpness.
Specifically, despite depth shifts across frames, graph structures exhibit
temporal self-similarity, enabling cross-frame geometric attention for graph
fusion. Then, by incorporating an image smoothness prior on the fused graph and
data fidelity term derived from ToF noise distribution, we formulate a maximum
a posterior problem for ToF denoising. Finally, the solution is unrolled into
iterative filters whose weights are adaptively learned from the graph-informed
geometric attention, producing a high-performance yet interpretable network.
Experimental results demonstrate that the proposed scheme achieves
state-of-the-art performance in terms of accuracy and consistency on synthetic
DVToF dataset and exhibits robust generalization on the real Kinectv2 dataset.
Source code will be released at
https://github.com/davidweidawang/GIGA-ToF{https://github.com/davidweidawang/GIGA-ToF}.