Orionnnn/hub-stats huggingface dataset:
DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal
  Consistent Video Generation
Authors: name: Runze Zhang, name: Guoguang Du, name: Xiaochuan Li, name: Qi Jia, name: Liang Jin, name: Lu Liu, name: Jingjing Wang, name: Cong Xu, name: Zhenhua Guo, name: Yaqian Zhao, name: Xiaoli Gong, name: Rengang Li, name: Baoyu Fan
Published: 2025-03-08 04:37:38

Spatio-temporal consistency is a critical research topic in video generation.
A qualified generated video segment must ensure plot plausibility and coherence
while maintaining visual consistency of objects and scenes across varying
viewpoints. Prior research, especially in open-source projects, primarily
focuses on either temporal or spatial consistency, or their basic combination,
such as appending a description of a camera movement after a prompt without
constraining the outcomes of this movement. However, camera movement may
introduce new objects to the scene or eliminate existing ones, thereby
overlaying and affecting the preceding narrative. Especially in videos with
numerous camera movements, the interplay between multiple plots becomes
increasingly complex. This paper introduces and examines integral
spatio-temporal consistency, considering the synergy between plot progression
and camera techniques, and the long-term impact of prior content on subsequent
generation. Our research encompasses dataset construction through to the
development of the model. Initially, we constructed a DropletVideo-10M dataset,
which comprises 10 million videos featuring dynamic camera motion and object
actions. Each video is annotated with an average caption of 206 words,
detailing various camera movements and plot developments. Following this, we
developed and trained the DropletVideo model, which excels in preserving
spatio-temporal coherence during video generation. The DropletVideo dataset and
model are accessible at https://dropletx.github.io.
BlobCtrl: A Unified and Flexible Framework for Element-level Image
  Generation and Editing
Authors: name: Yaowei Li, name: Lingen Li, name: Zhaoyang Zhang, name: Xiaoyu Li, name: Guangzhi Wang, name: Hongxiang Li, name: Xiaodong Cun, name: Ying Shan, name: Yuexian Zou
Published: 2025-03-17 17:58:05

Element-level visual manipulation is essential in digital content creation,
but current diffusion-based methods lack the precision and flexibility of
traditional tools. In this work, we introduce BlobCtrl, a framework that
unifies element-level generation and editing using a probabilistic blob-based
representation. By employing blobs as visual primitives, our approach
effectively decouples and represents spatial location, semantic content, and
identity information, enabling precise element-level manipulation. Our key
contributions include: 1) a dual-branch diffusion architecture with
hierarchical feature fusion for seamless foreground-background integration; 2)
a self-supervised training paradigm with tailored data augmentation and score
functions; and 3) controllable dropout strategies to balance fidelity and
diversity. To support further research, we introduce BlobData for large-scale
training and BlobBench for systematic evaluation. Experiments show that
BlobCtrl excels in various element-level manipulation tasks while maintaining
computational efficiency, offering a practical solution for precise and
flexible visual content creation. Project page:
https://liyaowei-stu.github.io/project/BlobCtrl/
Rewards Are Enough for Fast Photo-Realistic Text-to-image Generation
Authors: name: Yihong Luo, name: Tianyang Hu, name: Weijian Luo, name: Kenji Kawaguchi, name: Jing Tang
Published: 2025-03-17 11:21:43

Aligning generated images to complicated text prompts and human preferences
is a central challenge in Artificial Intelligence-Generated Content (AIGC).
With reward-enhanced diffusion distillation emerging as a promising approach
that boosts controllability and fidelity of text-to-image models, we identify a
fundamental paradigm shift: as conditions become more specific and reward
signals stronger, the rewards themselves become the dominant force in
generation. In contrast, the diffusion losses serve as an overly expensive form
of regularization. To thoroughly validate our hypothesis, we introduce R0, a
novel conditional generation approach via regularized reward maximization.
Instead of relying on tricky diffusion distillation losses, R0 proposes a new
perspective that treats image generations as an optimization problem in data
space which aims to search for valid images that have high compositional
rewards. By innovative designs of the generator parameterization and proper
regularization techniques, we train state-of-the-art few-step text-to-image
generative models with R0 at scales. Our results challenge the conventional
wisdom of diffusion post-training and conditional generation by demonstrating
that rewards play a dominant role in scenarios with complex conditions. We hope
our findings can contribute to further research into human-centric and
reward-centric generation paradigms across the broader field of AIGC. Code is
available at https://github.com/Luo-Yihong/R0.
MTV-Inpaint: Multi-Task Long Video Inpainting
Authors: name: Shiyuan Yang, name: Zheng Gu, name: Liang Hou, name: Xin Tao, name: Pengfei Wan, name: Xiaodong Chen, name: Jing Liao
Published: 2025-03-14 13:54:10

Video inpainting involves modifying local regions within a video, ensuring
spatial and temporal consistency. Most existing methods focus primarily on
scene completion (i.e., filling missing regions) and lack the capability to
insert new objects into a scene in a controllable manner. Fortunately, recent
advancements in text-to-video (T2V) diffusion models pave the way for
text-guided video inpainting. However, directly adapting T2V models for
inpainting remains limited in unifying completion and insertion tasks, lacks
input controllability, and struggles with long videos, thereby restricting
their applicability and flexibility. To address these challenges, we propose
MTV-Inpaint, a unified multi-task video inpainting framework capable of
handling both traditional scene completion and novel object insertion tasks. To
unify these distinct tasks, we design a dual-branch spatial attention mechanism
in the T2V diffusion U-Net, enabling seamless integration of scene completion
and object insertion within a single framework. In addition to textual
guidance, MTV-Inpaint supports multimodal control by integrating various image
inpainting models through our proposed image-to-video (I2V) inpainting mode.
Additionally, we propose a two-stage pipeline that combines keyframe inpainting
with in-between frame propagation, enabling MTV-Inpaint to effectively handle
long videos with hundreds of frames. Extensive experiments demonstrate that
MTV-Inpaint achieves state-of-the-art performance in both scene completion and
object insertion tasks. Furthermore, it demonstrates versatility in derived
applications such as multi-modal inpainting, object editing, removal, image
object brush, and the ability to handle long videos. Project page:
https://mtv-inpaint.github.io/.
Error Analyses of Auto-Regressive Video Diffusion Models: A Unified
  Framework
Authors: name: Jing Wang, name: Fengzhuo Zhang, name: Xiaoli Li, name: Vincent Y. F. Tan, name: Tianyu Pang, name: Chao Du, name: Aixin Sun, name: Zhuoran Yang
Published: 2025-03-12 15:32:44

A variety of Auto-Regressive Video Diffusion Models (ARVDM) have achieved
remarkable successes in generating realistic long-form videos. However,
theoretical analyses of these models remain scant. In this work, we develop
theoretical underpinnings for these models and use our insights to improve the
performance of existing models. We first develop Meta-ARVDM, a unified
framework of ARVDMs that subsumes most existing methods. Using Meta-ARVDM, we
analyze the KL-divergence between the videos generated by Meta-ARVDM and the
true videos. Our analysis uncovers two important phenomena inherent to ARVDM --
error accumulation and memory bottleneck. By deriving an information-theoretic
impossibility result, we show that the memory bottleneck phenomenon cannot be
avoided. To mitigate the memory bottleneck, we design various network
structures to explicitly use more past frames. We also achieve a significantly
improved trade-off between the mitigation of the memory bottleneck and the
inference efficiency by compressing the frames. Experimental results on DMLab
and Minecraft validate the efficacy of our methods. Our experiments also
demonstrate a Pareto-frontier between the error accumulation and memory
bottleneck across different methods.
Basic Category Usage in Vision Language Models
Authors: name: Hunter Sawyer, name: Jesse Roberts, name: Kyle Moore
Published: 2025-03-16 14:50:54

The field of psychology has long recognized a basic level of categorization
that humans use when labeling visual stimuli, a term coined by Rosch in 1976.
This level of categorization has been found to be used most frequently, to have
higher information density, and to aid in visual language tasks with priming in
humans. Here, we investigate basic level categorization in two recently
released, open-source vision-language models (VLMs). This paper demonstrates
that Llama 3.2 Vision Instruct (11B) and Molmo 7B-D both prefer basic level
categorization consistent with human behavior. Moreover, the models'
preferences are consistent with nuanced human behaviors like the biological
versus non-biological basic level effects and the well established expert basic
level shift, further suggesting that VLMs acquire cognitive categorization
behaviors from the human data on which they are trained.
Investigating Human-Aligned Large Language Model Uncertainty
Authors: name: Kyle Moore, name: Jesse Roberts, name: Daryl Watson, name: Pamela Wisniewski
Published: 2025-03-16 14:45:43

Recent work has sought to quantify large language model uncertainty to
facilitate model control and modulate user trust. Previous works focus on
measures of uncertainty that are theoretically grounded or reflect the average
overt behavior of the model. In this work, we investigate a variety of
uncertainty measures, in order to identify measures that correlate with human
group-level uncertainty. We find that Bayesian measures and a variation on
entropy measures, top-k entropy, tend to agree with human behavior as a
function of model size. We find that some strong measures decrease in
human-similarity with model size, but, by multiple linear regression, we find
that combining multiple uncertainty measures provide comparable human-alignment
with reduced size-dependency.
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning
Authors: name: Ye Liu, name: Kevin Qinghong Lin, name: Chang Wen Chen, name: Mike Zheng Shou
Published: 2025-03-17 17:59:33

Videos, with their unique temporal dimension, demand precise grounded
understanding, where answers are directly linked to visual, interpretable
evidence. Despite significant breakthroughs in reasoning capabilities within
Large Language Models, multi-modal reasoning - especially for videos - remains
unexplored. In this work, we introduce VideoMind, a novel video-language agent
designed for temporal-grounded video understanding. VideoMind incorporates two
key innovations: (i) We identify essential capabilities for video temporal
reasoning and develop a role-based agentic workflow, including a planner for
coordinating different roles, a grounder for temporal localization, a verifier
to assess temporal interval accuracy, and an answerer for question-answering.
(ii) To efficiently integrate these diverse roles, we propose a novel
Chain-of-LoRA strategy, enabling seamless role-switching via lightweight LoRA
adaptors while avoiding the overhead of multiple models, thus balancing
efficiency and flexibility. Extensive experiments on 14 public benchmarks
demonstrate that our agent achieves state-of-the-art performance on diverse
video understanding tasks, including 3 on grounded video question-answering, 6
on video temporal grounding, and 5 on general video question-answering,
underscoring its effectiveness in advancing video agent and long-form temporal
reasoning.
Being-0: A Humanoid Robotic Agent with Vision-Language Models and
  Modular Skills
Authors: name: Haoqi Yuan, name: Yu Bai, name: Yuhui Fu, name: Bohan Zhou, name: Yicheng Feng, name: Xinrun Xu, name: Yi Zhan, name: Börje F. Karlsson, name: Zongqing Lu
Published: 2025-03-16 14:53:53

Building autonomous robotic agents capable of achieving human-level
performance in real-world embodied tasks is an ultimate goal in humanoid robot
research. Recent advances have made significant progress in high-level
cognition with Foundation Models (FMs) and low-level skill development for
humanoid robots. However, directly combining these components often results in
poor robustness and efficiency due to compounding errors in long-horizon tasks
and the varied latency of different modules. We introduce Being-0, a
hierarchical agent framework that integrates an FM with a modular skill
library. The FM handles high-level cognitive tasks such as instruction
understanding, task planning, and reasoning, while the skill library provides
stable locomotion and dexterous manipulation for low-level control. To bridge
the gap between these levels, we propose a novel Connector module, powered by a
lightweight vision-language model (VLM). The Connector enhances the FM's
embodied capabilities by translating language-based plans into actionable skill
commands and dynamically coordinating locomotion and manipulation to improve
task success. With all components, except the FM, deployable on low-cost
onboard computation devices, Being-0 achieves efficient, real-time performance
on a full-sized humanoid robot equipped with dexterous hands and active vision.
Extensive experiments in large indoor environments demonstrate Being-0's
effectiveness in solving complex, long-horizon tasks that require challenging
navigation and manipulation subtasks. For further details and videos, visit
https://beingbeyond.github.io/being-0.
R1-VL: Learning to Reason with Multimodal Large Language Models via
  Step-wise Group Relative Policy Optimization
Authors: name: Jingyi Zhang, name: Jiaxing Huang, name: Huanjin Yao, name: Shunyu Liu, name: Xikun Zhang, name: Shijian Lu, name: Dacheng Tao
Published: 2025-03-17 08:51:44

Recent studies generally enhance MLLMs' reasoning capabilities via supervised
fine-tuning on high-quality chain-of-thought reasoning data, which often leads
models to merely imitate successful reasoning paths without understanding what
the wrong reasoning paths are. In this work, we aim to enhance the MLLMs'
reasoning ability beyond passively imitating positive reasoning paths. To this
end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new
online reinforcement learning framework that enables MLLMs to self-improve
reasoning ability via simple, effective and dense step-wise rewarding.
Specifically, StepGRPO introduces two novel rule-based reasoning rewards:
Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity
Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary
intermediate reasoning steps via a soft key-step matching technique, while
StepRAR rewards reasoning paths that follow a well-structured and logically
consistent reasoning process through a reasoning completeness and logic
evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series
of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive
experiments over 8 benchmarks demonstrate the superiority of our methods.
reWordBench: Benchmarking and Improving the Robustness of Reward Models
  with Transformed Inputs
Authors: name: Zhaofeng Wu, name: Michihiro Yasunaga, name: Andrew Cohen, name: Yoon Kim, name: Asli Celikyilmaz, name: Marjan Ghazvininejad
Published: 2025-03-14 17:59:41

Reward models have become a staple in modern NLP, serving as not only a
scalable text evaluator, but also an indispensable component in many alignment
recipes and inference-time algorithms. However, while recent reward models
increase performance on standard benchmarks, this may partly be due to
overfitting effects, which would confound an understanding of their true
capability. In this work, we scrutinize the robustness of reward models and the
extent of such overfitting. We build **reWordBench**, which systematically
transforms reward model inputs in meaning- or ranking-preserving ways. We show
that state-of-the-art reward models suffer from substantial performance
degradation even with minor input transformations, sometimes dropping to
significantly below-random accuracy, suggesting brittleness. To improve reward
model robustness, we propose to explicitly train them to assign similar scores
to paraphrases, and find that this approach also improves robustness to other
distinct kinds of transformations. For example, our robust reward model reduces
such degradation by roughly half for the Chat Hard subset in RewardBench.
Furthermore, when used in alignment, our robust reward models demonstrate
better utility and lead to higher-quality outputs, winning in up to 59% of
instances against a standardly trained RM.
WideRange4D: Enabling High-Quality 4D Reconstruction with Wide-Range
  Movements and Scenes
Authors: name: Ling Yang, name: Kaixin Zhu, name: Juanxi Tian, name: Bohan Zeng, name: Mingbao Lin, name: Hongjuan Pei, name: Wentao Zhang, name: Shuicheng Yan
Published: 2025-03-17 17:58:18

With the rapid development of 3D reconstruction technology, research in 4D
reconstruction is also advancing, existing 4D reconstruction methods can
generate high-quality 4D scenes. However, due to the challenges in acquiring
multi-view video data, the current 4D reconstruction benchmarks mainly display
actions performed in place, such as dancing, within limited scenarios. In
practical scenarios, many scenes involve wide-range spatial movements,
highlighting the limitations of existing 4D reconstruction datasets.
Additionally, existing 4D reconstruction methods rely on deformation fields to
estimate the dynamics of 3D objects, but deformation fields struggle with
wide-range spatial movements, which limits the ability to achieve high-quality
4D scene reconstruction with wide-range spatial movements. In this paper, we
focus on 4D scene reconstruction with significant object spatial movements and
propose a novel 4D reconstruction benchmark, WideRange4D. This benchmark
includes rich 4D scene data with large spatial variations, allowing for a more
comprehensive evaluation of the generation capabilities of 4D generation
methods. Furthermore, we introduce a new 4D reconstruction method, Progress4D,
which generates stable and high-quality 4D results across various complex 4D
scene reconstruction tasks. We conduct both quantitative and qualitative
comparison experiments on WideRange4D, showing that our Progress4D outperforms
existing state-of-the-art 4D reconstruction methods. Project:
https://github.com/Gen-Verse/WideRange4D
DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale
  Text-to-Image Models
Authors: name: Dewei Zhou, name: Mingwei Li, name: Zongxin Yang, name: Yi Yang
Published: 2025-03-17 07:30:16

Image-conditioned generation methods, such as depth- and canny-conditioned
approaches, have demonstrated remarkable abilities for precise image synthesis.
However, existing models still struggle to accurately control the content of
multiple instances (or regions). Even state-of-the-art models like FLUX and
3DIS face challenges, such as attribute leakage between instances, which limits
user control. To address these issues, we introduce DreamRenderer, a
training-free approach built upon the FLUX model. DreamRenderer enables users
to control the content of each instance via bounding boxes or masks, while
ensuring overall visual harmony. We propose two key innovations: 1) Bridge
Image Tokens for Hard Text Attribute Binding, which uses replicated image
tokens as bridge tokens to ensure that T5 text embeddings, pre-trained solely
on text data, bind the correct visual attributes for each instance during Joint
Attention; 2) Hard Image Attribute Binding applied only to vital layers.
Through our analysis of FLUX, we identify the critical layers responsible for
instance attribute rendering and apply Hard Image Attribute Binding only in
these layers, using soft binding in the others. This approach ensures precise
control while preserving image quality. Evaluations on the COCO-POS and
COCO-MIG benchmarks demonstrate that DreamRenderer improves the Image Success
Ratio by 17.7% over FLUX and enhances the performance of layout-to-image models
like GLIGEN and 3DIS by up to 26.8%. Project Page:
https://limuloo.github.io/DreamRenderer/.
Edit Transfer: Learning Image Editing via Vision In-Context Relations
Authors: name: Lan Chen, name: Qi Mao, name: Yuchao Gu, name: Mike Zheng Shou
Published: 2025-03-17 16:04:44

We introduce a new setting, Edit Transfer, where a model learns a
transformation from just a single source-target example and applies it to a new
query image. While text-based methods excel at semantic manipulations through
textual prompts, they often struggle with precise geometric details (e.g.,
poses and viewpoint changes). Reference-based editing, on the other hand,
typically focuses on style or appearance and fails at non-rigid
transformations. By explicitly learning the editing transformation from a
source-target pair, Edit Transfer mitigates the limitations of both text-only
and appearance-centric references. Drawing inspiration from in-context learning
in large language models, we propose a visual relation in-context learning
paradigm, building upon a DiT-based text-to-image model. We arrange the edited
example and the query image into a unified four-panel composite, then apply
lightweight LoRA fine-tuning to capture complex spatial transformations from
minimal examples. Despite using only 42 training samples, Edit Transfer
substantially outperforms state-of-the-art TIE and RIE methods on diverse
non-rigid scenarios, demonstrating the effectiveness of few-shot visual
relation learning.
Personalize Anything for Free with Diffusion Transformer
Authors: name: Haoran Feng, name: Zehuan Huang, name: Lin Li, name: Hairong Lv, name: Lu Sheng
Published: 2025-03-16 17:51:16

Personalized image generation aims to produce images of user-specified
concepts while enabling flexible editing. Recent training-free approaches,
while exhibit higher computational efficiency than training-based methods,
struggle with identity preservation, applicability, and compatibility with
diffusion transformers (DiTs). In this paper, we uncover the untapped potential
of DiT, where simply replacing denoising tokens with those of a reference
subject achieves zero-shot subject reconstruction. This simple yet effective
feature injection technique unlocks diverse scenarios, from personalization to
image editing. Building upon this observation, we propose Personalize
Anything, a training-free framework that achieves personalized image
generation in DiT through: 1) timestep-adaptive token replacement that enforces
subject consistency via early-stage injection and enhances flexibility through
late-stage regularization, and 2) patch perturbation strategies to boost
structural diversity. Our method seamlessly supports layout-guided generation,
multi-subject personalization, and mask-controlled editing. Evaluations
demonstrate state-of-the-art performance in identity preservation and
versatility. Our work establishes new insights into DiTs while delivering a
practical paradigm for efficient personalization.