Orionnnn/hub-stats huggingface dataset:
Unconditional Priors Matter! Improving Conditional Generation of
  Fine-Tuned Diffusion Models
Authors: name: Prin Phunyaphibarn, name: Phillip Y. Lee, name: Jaihoon Kim, name: Minhyuk Sung
Published: 2025-03-26 05:11:38

Classifier-Free Guidance (CFG) is a fundamental technique in training
conditional diffusion models. The common practice for CFG-based training is to
use a single network to learn both conditional and unconditional noise
prediction, with a small dropout rate for conditioning. However, we observe
that the joint learning of unconditional noise with limited bandwidth in
training results in poor priors for the unconditional case. More importantly,
these poor unconditional noise predictions become a serious reason for
degrading the quality of conditional generation. Inspired by the fact that most
CFG-based conditional models are trained by fine-tuning a base model with
better unconditional generation, we first show that simply replacing the
unconditional noise in CFG with that predicted by the base model can
significantly improve conditional generation. Furthermore, we show that a
diffusion model other than the one the fine-tuned model was trained on can be
used for unconditional noise replacement. We experimentally verify our claim
with a range of CFG-based conditional models for both image and video
generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and
InstructPix2Pix.
Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long
  Video Generation
Authors: name: Tianhao Qi, name: Jianlong Yuan, name: Wanquan Feng, name: Shancheng Fang, name: Jiawei Liu, name: SiYu Zhou, name: Qian He, name: Hongtao Xie, name: Yongdong Zhang
Published: 2025-03-25 17:46:50

Sora has unveiled the immense potential of the Diffusion Transformer (DiT)
architecture in single-scene video generation. However, the more challenging
task of multi-scene video generation, which offers broader applications,
remains relatively underexplored. To bridge this gap, we propose Mask^2DiT, a
novel approach that establishes fine-grained, one-to-one alignment between
video segments and their corresponding text annotations. Specifically, we
introduce a symmetric binary mask at each attention layer within the DiT
architecture, ensuring that each text annotation applies exclusively to its
respective video segment while preserving temporal coherence across visual
tokens. This attention mechanism enables precise segment-level
textual-to-visual alignment, allowing the DiT architecture to effectively
handle video generation tasks with a fixed number of scenes. To further equip
the DiT architecture with the ability to generate additional scenes based on
existing ones, we incorporate a segment-level conditional mask, which
conditions each newly generated segment on the preceding video segments,
thereby enabling auto-regressive scene extension. Both qualitative and
quantitative experiments confirm that Mask^2DiT excels in maintaining visual
consistency across segments while ensuring semantic alignment between each
segment and its corresponding text description. Our project page is
https://tianhao-qi.github.io/Mask2DiTProject.
Efficient Model Development through Fine-tuning Transfer
Authors: name: Pin-Jie Lin, name: Rishab Balasubramanian, name: Fengyuan Liu, name: Nikhil Kandpal, name: Tu Vu
Published: 2025-03-25 23:24:43

Modern LLMs struggle with efficient updates, as each new pretrained model
version requires repeating expensive alignment processes. This challenge also
applies to domain- or language-specific models, where fine-tuning on
specialized data must be redone for every new base model release. In this
paper, we explore the transfer of fine-tuning updates between model versions.
Specifically, we derive the diff vector from one source model version, which
represents the weight changes from fine-tuning, and apply it to the base model
of a different target version. Through empirical evaluations on various
open-weight model versions, we show that transferring diff vectors can
significantly improve the target base model, often achieving performance
comparable to its fine-tuned counterpart. For example, reusing the fine-tuning
updates from Llama 3.0 8B leads to an absolute accuracy improvement of 10.7% on
GPQA over the base Llama 3.1 8B without additional training, surpassing Llama
3.1 8B Instruct. In a multilingual model development setting, we show that this
approach can significantly increase performance on target-language tasks
without retraining, achieving an absolute improvement of 4.7% and 15.5% on
Global MMLU for Malagasy and Turkish, respectively, compared to Llama 3.1 8B
Instruct. Our controlled experiments reveal that fine-tuning transfer is most
effective when the source and target models are linearly connected in the
parameter space. Additionally, we demonstrate that fine-tuning transfer offers
a stronger and more computationally efficient starting point for further
fine-tuning. Finally, we propose an iterative recycling-then-finetuning
approach for continuous model development, which improves both efficiency and
effectiveness. Our findings suggest that fine-tuning transfer is a viable
strategy to reduce training costs while maintaining model performance.
Any6D: Model-free 6D Pose Estimation of Novel Objects
Authors: name: Taeyeop Lee, name: Bowen Wen, name: Minjun Kang, name: Gyuree Kang, name: In So Kweon, name: Kuk-Jin Yoon
Published: 2025-03-24 13:46:21

We introduce Any6D, a model-free framework for 6D object pose estimation that
requires only a single RGB-D anchor image to estimate both the 6D pose and size
of unknown objects in novel scenes. Unlike existing methods that rely on
textured 3D models or multiple viewpoints, Any6D leverages a joint object
alignment process to enhance 2D-3D alignment and metric scale estimation for
improved pose accuracy. Our approach integrates a render-and-compare strategy
to generate and refine pose hypotheses, enabling robust performance in
scenarios with occlusions, non-overlapping views, diverse lighting conditions,
and large cross-environment variations. We evaluate our method on five
challenging datasets: REAL275, Toyota-Light, HO3D, YCBINEOAT, and LM-O,
demonstrating its effectiveness in significantly outperforming state-of-the-art
methods for novel object pose estimation. Project page:
https://taeyeop.com/any6d
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D
  Object Placement
Authors: name: Ian Huang, name: Yanan Bao, name: Karen Truong, name: Howard Zhou, name: Cordelia Schmid, name: Leonidas Guibas, name: Alireza Fathi
Published: 2025-03-06 19:34:15

Scene generation with 3D assets presents a complex challenge, requiring both
high-level semantic understanding and low-level geometric reasoning. While
Multimodal Large Language Models (MLLMs) excel at semantic tasks, their
application to 3D scene generation is hindered by their limited grounding on 3D
geometry. In this paper, we investigate how to best work with MLLMs in an
object placement task. Towards this goal, we introduce a novel framework,
FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the
extraction of relevant geometric details from the 3D scene, (2) constructing
and solving geometric constraints on the extracted low-level geometry, and (3)
pruning for final placements that conform to common sense. By combining
geometric reasoning with real-world understanding of MLLMs, our method can
propose object placements that satisfy both geometric constraints as well as
high-level semantic common-sense considerations. Our experiments show that
these capabilities allow our method to place objects more effectively in
complex scenes with intricate geometry, surpassing the quality of prior work.
LLaVAction: evaluating and training multi-modal large language models
  for action recognition
Authors: name: Shaokai Ye, name: Haozhe Qi, name: Alexander Mathis, name: Mackenzie W. Mathis
Published: 2025-03-24 14:24:17

Understanding human behavior requires measuring behavioral actions. Due to
its complexity, behavior is best mapped onto a rich, semantic structure such as
language. The recent development of multi-modal large language models (MLLMs)
is a promising candidate for a wide range of action understanding tasks. In
this work, we focus on evaluating and then improving MLLMs to perform action
recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most
challenging egocentric action datasets, to the form of video multiple question
answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult
incorrect answers as distractors, leading MLLMs struggle to recognize the
correct actions. We propose a series of methods that greatly improve the MLLMs'
ability to perform action recognition, achieving state-of-the-art on both the
EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points
in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other
action-related video benchmarks such as EgoSchema, PerceptionTest,
LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising
path forward for complex action tasks. Code and models are available at:
https://github.com/AdaptiveMotorControlLab/LLaVAction.
DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis
Authors: name: Yuming Gu, name: Phong Tran, name: Yujian Zheng, name: Hongyi Xu, name: Heyuan Li, name: Adilbek Karmanov, name: Hao Li
Published: 2025-03-19 19:47:04

Generating high-quality 360-degree views of human heads from single-view
images is essential for enabling accessible immersive telepresence applications
and scalable personalized content creation. While cutting-edge methods for full
head generation are limited to modeling realistic human heads, the latest
diffusion-based approaches for style-omniscient head synthesis can produce only
frontal views and struggle with view consistency, preventing their conversion
into true 3D models for rendering from arbitrary angles. We introduce a novel
approach that generates fully consistent 360-degree head views, accommodating
human, stylized, and anthropomorphic forms, including accessories like glasses
and hats. Our method builds on the DiffPortrait3D framework, incorporating a
custom ControlNet for back-of-head detail generation and a dual appearance
module to ensure global front-back consistency. By training on continuous view
sequences and integrating a back reference image, our approach achieves robust,
locally continuous view synthesis. Our model can be used to produce
high-quality neural radiance fields (NeRFs) for real-time, free-viewpoint
rendering, outperforming state-of-the-art methods in object synthesis and
360-degree head generation for very challenging input portraits.
OpenCity3D: What do Vision-Language Models know about Urban
  Environments?
Authors: name: Valentin Bieri, name: Marco Zamboni, name: Nicolas S. Blumer, name: Qingxuan Chen, name: Francis Engelmann
Published: 2025-03-21 01:11:21

Vision-language models (VLMs) show great promise for 3D scene understanding
but are mainly applied to indoor spaces or autonomous driving, focusing on
low-level tasks like segmentation. This work expands their use to urban-scale
environments by leveraging 3D reconstructions from multi-view aerial imagery.
We propose OpenCity3D, an approach that addresses high-level tasks, such as
population density estimation, building age classification, property price
prediction, crime rate assessment, and noise pollution evaluation. Our findings
highlight OpenCity3D's impressive zero-shot and few-shot capabilities,
showcasing adaptability to new contexts. This research establishes a new
paradigm for language-driven urban analytics, enabling applications in
planning, policy, and environmental monitoring. See our project page:
opencity3d.github.io
ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in
  Vision-Language Models
Authors: name: Dohwan Ko, name: Sihyeon Kim, name: Yumin Suh, name: Vijay Kumar B. G, name: Minseo Yoon, name: Manmohan Chandraker, name: Hyunwoo J. Kim
Published: 2025-03-25 05:08:06

Spatio-temporal reasoning is essential in understanding real-world
environments in various fields, eg, autonomous driving and sports analytics.
Recent advances have improved the spatial reasoning ability of Vision-Language
Models (VLMs) by introducing large-scale data, but these models still struggle
to analyze kinematic elements like traveled distance and speed of moving
objects. To bridge this gap, we construct a spatio-temporal reasoning dataset
and benchmark involving kinematic instruction tuning, referred to as STKit and
STKit-Bench. They consist of real-world videos with 3D annotations, detailing
object motion dynamics: traveled distance, speed, movement direction,
inter-object distance comparisons, and relative movement direction. To further
scale such data construction to videos without 3D labels, we propose an
automatic pipeline to generate pseudo-labels using 4D reconstruction in
real-world scale. With our kinematic instruction tuning data for
spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for
spatio-temporal reasoning, which exhibits outstanding performance on
STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across
diverse domains and tasks, outperforming baselines on other spatio-temporal
benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned
spatio-temporal reasoning with existing abilities, ST-VLM enables complex
multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on
  Aerial Images
Authors: name: Yara AlaaEldin, name: Francesca Odone
Published: 2025-03-23 08:25:07

Understanding the geometric and semantic properties of the scene is crucial
in autonomous navigation and particularly challenging in the case of Unmanned
Aerial Vehicle (UAV) navigation. Such information may be by obtained by
estimating depth and semantic segmentation maps of the surrounding environment
and for their practical use in autonomous navigation, the procedure must be
performed as close to real-time as possible. In this paper, we leverage
monocular cameras on aerial robots to predict depth and semantic maps in
low-altitude unstructured environments. We propose a joint deep-learning
architecture that can perform the two tasks accurately and rapidly, and
validate its effectiveness on MidAir and Aeroscapes benchmark datasets. Our
joint-architecture proves to be competitive or superior to the other single and
joint architecture methods while performing its task fast predicting 20.2 FPS
on a single NVIDIA quadro p5000 GPU and it has a low memory footprint. All
codes for training and prediction can be found on this link:
https://github.com/Malga-Vision/Co-SemDepth
PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable
  Objects from Videos
Authors: name: Hanxiao Jiang, name: Hao-Yu Hsu, name: Kaifeng Zhang, name: Hsin-Ni Yu, name: Shenlong Wang, name: Yunzhu Li
Published: 2025-03-23 07:49:19

Creating a physical digital twin of a real-world object has immense potential
in robotics, content creation, and XR. In this paper, we present PhysTwin, a
novel framework that uses sparse videos of dynamic objects under interaction to
produce a photo- and physically realistic, real-time interactive virtual
replica. Our approach centers on two key components: (1) a physics-informed
representation that combines spring-mass models for realistic physical
simulation, generative shape models for geometry, and Gaussian splats for
rendering; and (2) a novel multi-stage, optimization-based inverse modeling
framework that reconstructs complete geometry, infers dense physical
properties, and replicates realistic appearance from videos. Our method
integrates an inverse physics framework with visual perception cues, enabling
high-fidelity reconstruction even from partial, occluded, and limited
viewpoints. PhysTwin supports modeling various deformable objects, including
ropes, stuffed animals, cloth, and delivery packages. Experiments show that
PhysTwin outperforms competing methods in reconstruction, rendering, future
prediction, and simulation under novel interactions. We further demonstrate its
applications in interactive real-time simulation and model-based robotic motion
planning.
FullDiT: Multi-Task Video Generative Foundation Model with Full
  Attention
Authors: name: Xuan Ju, name: Weicai Ye, name: Quande Liu, name: Qiulin Wang, name: Xintao Wang, name: Pengfei Wan, name: Di Zhang, name: Kun Gai, name: Qiang Xu
Published: 2025-03-25 17:59:06

Current video generative foundation models primarily focus on text-to-video
tasks, providing limited control for fine-grained video content creation.
Although adapter-based approaches (e.g., ControlNet) enable additional controls
with minimal fine-tuning, they encounter challenges when integrating multiple
conditions, including: branch conflicts between independently trained adapters,
parameter redundancy leading to increased computational cost, and suboptimal
performance compared to full fine-tuning. To address these challenges, we
introduce FullDiT, a unified foundation model for video generation that
seamlessly integrates multiple conditions via unified full-attention
mechanisms. By fusing multi-task conditions into a unified sequence
representation and leveraging the long-context learning ability of full
self-attention to capture condition dynamics, FullDiT reduces parameter
overhead, avoids conditions conflict, and shows scalability and emergent
ability. We further introduce FullBench for multi-task video generation
evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art
results, highlighting the efficacy of full-attention in complex multi-task
video generation.
xKV: Cross-Layer SVD for KV-Cache Compression
Authors: name: Chi-Chih Chang, name: Chien-Yu Lin, name: Yash Akhauri, name: Wei-Cheng Lin, name: Kai-Chiang Wu, name: Luis Ceze, name: Mohamed S. Abdelfattah
Published: 2025-03-24 17:06:37

Large Language Models (LLMs) with long context windows enable powerful
applications but come at the cost of high memory consumption to store the Key
and Value states (KV-Cache). Recent studies attempted to merge KV-cache from
multiple layers into shared representations, yet these approaches either
require expensive pretraining or rely on assumptions of high per-token cosine
similarity across layers which generally does not hold in practice. We find
that the dominant singular vectors are remarkably well-aligned across multiple
layers of the KV-Cache. Exploiting this insight, we propose xKV, a simple
post-training method that applies Singular Value Decomposition (SVD) on the
KV-Cache of grouped layers. xKV consolidates the KV-Cache of multiple layers
into a shared low-rank subspace, significantly reducing KV-Cache sizes. Through
extensive evaluations on the RULER long-context benchmark with widely-used LLMs
(e.g., Llama-3.1 and Qwen2.5), xKV achieves up to 6.8x higher compression rates
than state-of-the-art inter-layer technique while improving accuracy by 2.7%.
Moreover, xKV is compatible with the emerging Multi-Head Latent Attention (MLA)
(e.g., DeepSeek-Coder-V2), yielding a notable 3x compression rates on coding
tasks without performance degradation. These results highlight xKV's strong
capability and versatility in addressing memory bottlenecks for long-context
LLM inference. Our code is publicly available at:
https://github.com/abdelfattah-lab/xKV.
Can Vision-Language Models Answer Face to Face Questions in the
  Real-World?
Authors: name: Reza Pourreza, name: Rishit Dagli, name: Apratim Bhattacharyya, name: Sunny Panchal, name: Guillaume Berger, name: Roland Memisevic
Published: 2025-03-25 05:13:12

AI models have made significant strides in recent years in their ability to
describe and answer questions about real-world images. They have also made
progress in the ability to converse with users in real-time using audio input.
This raises the question: have we reached the point where AI models, connected
to a camera and microphone, can converse with users in real-time about scenes
and events that are unfolding live in front of the camera? This has been a
long-standing goal in AI and is a prerequisite for real-world AI assistants and
humanoid robots to interact with humans in everyday situations. In this work,
we introduce a new dataset and benchmark, the Qualcomm Interactive Video
Dataset (IVD), which allows us to assess the extent to which existing models
can support these abilities, and to what degree these capabilities can be
instilled through fine-tuning. The dataset is based on a simple
question-answering setup, where users ask questions that the system has to
answer, in real-time, based on the camera and audio input. We show that
existing models fall far behind human performance on this task, and we identify
the main sources for the performance gap. However, we also show that for many
of the required perceptual skills, fine-tuning on this form of data can
significantly reduce this gap.
WikiAutoGen: Towards Multi-Modal Wikipedia-Style Article Generation
Authors: name: Zhongyu Yang, name: Jun Chen, name: Dannong Xu, name: Junjie Fei, name: Xiaoqian Shen, name: Liangbing Zhao, name: Chun-Mei Feng, name: Mohamed Elhoseiny
Published: 2025-03-24 18:51:55

Knowledge discovery and collection are intelligence-intensive tasks that
traditionally require significant human effort to ensure high-quality outputs.
Recent research has explored multi-agent frameworks for automating
Wikipedia-style article generation by retrieving and synthesizing information
from the internet. However, these methods primarily focus on text-only
generation, overlooking the importance of multimodal content in enhancing
informativeness and engagement. In this work, we introduce WikiAutoGen, a novel
system for automated multimodal Wikipedia-style article generation. Unlike
prior approaches, WikiAutoGen retrieves and integrates relevant images
alongside text, enriching both the depth and visual appeal of generated
content. To further improve factual accuracy and comprehensiveness, we propose
a multi-perspective self-reflection mechanism, which critically assesses
retrieved content from diverse viewpoints to enhance reliability, breadth, and
coherence, etc. Additionally, we introduce WikiSeek, a benchmark comprising
Wikipedia articles with topics paired with both textual and image-based
representations, designed to evaluate multimodal knowledge generation on more
challenging topics. Experimental results show that WikiAutoGen outperforms
previous methods by 8%-29% on our WikiSeek benchmark, producing more accurate,
coherent, and visually enriched Wikipedia-style articles. We show some of our
generated examples in https://wikiautogen.github.io/ .
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
  Language Modeling
Authors: name: Haebin Shin, name: Lei Ji, name: Xiao Liu, name: Yeyun Gong
Published: 2025-03-24 20:19:31

Using large teacher models to guide the training of smaller student models
has become the prevailing paradigm for efficient and effective learning.
However, vocabulary mismatches between teacher and student language models pose
significant challenges in language modeling, resulting in divergent token
sequences and output distributions. To overcome these limitations, we propose
Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
approach that bridges the gap caused by vocabulary mismatch through two key
methods: (1) Token-level Lexical Alignment, which aligns token sequences across
mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
of teacher model to guide effective student training. We demonstrate its
effectiveness in language modeling with 1B student model using various 7B
teacher models with different vocabularies. Notably, with
Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
consistently benefits from stronger teacher models, providing a robust solution
to vocabulary mismatches in language modeling.