Orionnnn/hub-stats huggingface dataset:
Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video
  Generation Control
Authors: name: Zekai Gu, name: Rui Yan, name: Jiahao Lu, name: Peng Li, name: Zhiyang Dou, name: Chenyang Si, name: Zhen Dong, name: Qifeng Liu, name: Cheng Lin, name: Ziwei Liu, name: Wenping Wang, name: Yuan Liu
Published: 2025-01-07 15:01:58

Diffusion models have demonstrated impressive performance in generating
high-quality videos from text prompts or images. However, precise control over
the video generation process, such as camera manipulation or content editing,
remains a significant challenge. Existing methods for controlled video
generation are typically limited to a single control type, lacking the
flexibility to handle diverse control demands. In this paper, we introduce
Diffusion as Shader (DaS), a novel approach that supports multiple video
control tasks within a unified architecture. Our key insight is that achieving
versatile video control necessitates leveraging 3D control signals, as videos
are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods
limited to 2D control signals, DaS leverages 3D tracking videos as control
inputs, making the video diffusion process inherently 3D-aware. This innovation
allows DaS to achieve a wide range of video controls by simply manipulating the
3D tracking videos. A further advantage of using 3D tracking videos is their
ability to effectively link frames, significantly enhancing the temporal
consistency of the generated videos. With just 3 days of fine-tuning on 8 H800
GPUs using less than 10k videos, DaS demonstrates strong control capabilities
across diverse tasks, including mesh-to-video generation, camera control,
motion transfer, and object manipulation.
Cosmos World Foundation Model Platform for Physical AI
Authors: name: NVIDIA, name: Niket Agarwal, name: Arslan Ali, name: Maciej Bala, name: Yogesh Balaji, name: Erik Barker, name: Tiffany Cai, name: Prithvijit Chattopadhyay, name: Yongxin Chen, name: Yin Cui, name: Yifan Ding, name: Daniel Dworakowski, name: Jiaojiao Fan, name: Michele Fenzi, name: Francesco Ferroni, name: Sanja Fidler, name: Dieter Fox, name: Songwei Ge, name: Yunhao Ge, name: Jinwei Gu, name: Siddharth Gururani, name: Ethan He, name: Jiahui Huang, name: Jacob Huffman, name: Pooya Jannaty, name: Jingyi Jin, name: Seung Wook Kim, name: Gergely Klár, name: Grace Lam, name: Shiyi Lan, name: Laura Leal-Taixe, name: Anqi Li, name: Zhaoshuo Li, name: Chen-Hsuan Lin, name: Tsung-Yi Lin, name: Huan Ling, name: Ming-Yu Liu, name: Xian Liu, name: Alice Luo, name: Qianli Ma, name: Hanzi Mao, name: Kaichun Mo, name: Arsalan Mousavian, name: Seungjun Nah, name: Sriharsha Niverty, name: David Page, name: Despoina Paschalidou, name: Zeeshan Patel, name: Lindsey Pavao, name: Morteza Ramezanali, name: Fitsum Reda, name: Xiaowei Ren, name: Vasanth Rao Naik Sabavat, name: Ed Schmerling, name: Stella Shi, name: Bartosz Stefaniak, name: Shitao Tang, name: Lyne Tchapmi, name: Przemek Tredak, name: Wei-Cheng Tseng, name: Jibin Varghese, name: Hao Wang, name: Haoxiang Wang, name: Heng Wang, name: Ting-Chun Wang, name: Fangyin Wei, name: Xinyue Wei, name: Jay Zhangjie Wu, name: Jiashu Xu, name: Wei Yang, name: Lin Yen-Chen, name: Xiaohui Zeng, name: Yu Zeng, name: Jing Zhang, name: Qinsheng Zhang, name: Yuxuan Zhang, name: Qingqing Zhao, name: Artur Zolkowski
Published: 2025-01-07 06:55:50

Physical AI needs to be trained digitally first. It needs a digital twin of
itself, the policy model, and a digital twin of the world, the world model. In
this paper, we present the Cosmos World Foundation Model Platform to help
developers build customized world models for their Physical AI setups. We
position a world foundation model as a general-purpose world model that can be
fine-tuned into customized world models for downstream applications. Our
platform covers a video curation pipeline, pre-trained world foundation models,
examples of post-training of pre-trained world foundation models, and video
tokenizers. To help Physical AI builders solve the most critical problems of
our society, we make our platform open-source and our models open-weight with
permissive licenses available via https://github.com/NVIDIA/Cosmos.
MotionBench: Benchmarking and Improving Fine-grained Video Motion
  Understanding for Vision Language Models
Authors: name: Wenyi Hong, name: Yean Cheng, name: Zhuoyi Yang, name: Weihan Wang, name: Lefan Wang, name: Xiaotao Gu, name: Shiyu Huang, name: Yuxiao Dong, name: Jie Tang
Published: 2025-01-06 11:57:38

In recent years, vision language models (VLMs) have made significant
advancements in video understanding. However, a crucial capability -
fine-grained motion comprehension - remains under-explored in current
benchmarks. To address this gap, we propose MotionBench, a comprehensive
evaluation benchmark designed to assess the fine-grained motion comprehension
of video understanding models. MotionBench evaluates models' motion-level
perception through six primary categories of motion-oriented question types and
includes data collected from diverse sources, ensuring a broad representation
of real-world video content. Experimental results reveal that existing VLMs
perform poorly in understanding fine-grained motions. To enhance VLM's ability
to perceive fine-grained motion within a limited sequence length of LLM, we
conduct extensive experiments reviewing VLM architectures optimized for video
feature compression and propose a novel and efficient Through-Encoder (TE)
Fusion method. Experiments show that higher frame rate inputs and TE Fusion
yield improvements in motion understanding, yet there is still substantial room
for enhancement. Our benchmark aims to guide and motivate the development of
more capable video understanding models, emphasizing the importance of
fine-grained motion comprehension. Project page: https://motion-bench.github.io .
MagicFace: High-Fidelity Facial Expression Editing with Action-Unit
  Control
Authors: name: Mengting Wei, name: Tuomas Varanka, name: Xingxun Jiang, name: Huai-Qian Khor, name: Guoying Zhao
Published: 2025-01-04 11:28:49

We address the problem of facial expression editing by controling the
relative variation of facial action-unit (AU) from the same person. This
enables us to edit this specific person's expression in a fine-grained,
continuous and interpretable manner, while preserving their identity, pose,
background and detailed facial attributes. Key to our model, which we dub
MagicFace, is a diffusion model conditioned on AU variations and an ID encoder
to preserve facial details of high consistency. Specifically, to preserve the
facial details with the input identity, we leverage the power of pretrained
Stable-Diffusion models and design an ID encoder to merge appearance features
through self-attention. To keep background and pose consistency, we introduce
an efficient Attribute Controller by explicitly informing the model of current
background and pose of the target. By injecting AU variations into a denoising
UNet, our model can animate arbitrary identities with various AU combinations,
yielding superior results in high-fidelity expression editing compared to other
facial expression editing works. Code is publicly available at
https://github.com/weimengting/MagicFace.
Magic Mirror: ID-Preserved Video Generation in Video Diffusion
  Transformers
Authors: name: Yuechen Zhang, name: Yaoyang Liu, name: Bin Xia, name: Bohao Peng, name: Zexin Yan, name: Eric Lo, name: Jiaya Jia
Published: 2025-01-07 16:48:31

We present Magic Mirror, a framework for generating identity-preserved videos
with cinematic-level quality and dynamic motion. While recent advances in video
diffusion models have shown impressive capabilities in text-to-video
generation, maintaining consistent identity while producing natural motion
remains challenging. Previous methods either require person-specific
fine-tuning or struggle to balance identity preservation with motion diversity.
Built upon Video Diffusion Transformers, our method introduces three key
components: (1) a dual-branch facial feature extractor that captures both
identity and structural features, (2) a lightweight cross-modal adapter with
Conditioned Adaptive Normalization for efficient identity integration, and (3)
a two-stage training strategy combining synthetic identity pairs with video
data. Extensive experiments demonstrate that Magic Mirror effectively balances
identity consistency with natural motion, outperforming existing methods across
multiple metrics while requiring minimal parameters added. The code and model
will be made publicly available at:
https://github.com/dvlab-research/MagicMirror/
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One
  Vision Token
Authors: name: Shaolei Zhang, name: Qingkai Fang, name: Zhe Yang, name: Yang Feng
Published: 2025-01-07 16:03:14

The advent of real-time large multimodal models (LMMs) like GPT-4o has
sparked considerable interest in efficient LMMs. LMM frameworks typically
encode visual inputs into vision tokens (continuous representations) and
integrate them and textual instructions into the context of large language
models (LLMs), where large-scale parameters and numerous context tokens
(predominantly vision tokens) result in substantial computational overhead.
Previous efforts towards efficient LMMs always focus on replacing the LLM
backbone with smaller models, while neglecting the crucial issue of token
quantity. In this paper, we introduce LLaVA-Mini, an efficient LMM with minimal
vision tokens. To achieve a high compression ratio of vision tokens while
preserving visual information, we first analyze how LMMs understand vision
tokens and find that most vision tokens only play a crucial role in the early
layers of LLM backbone, where they mainly fuse visual information into text
tokens. Building on this finding, LLaVA-Mini introduces modality pre-fusion to
fuse visual information into text tokens in advance, thereby facilitating the
extreme compression of vision tokens fed to LLM backbone into one token.
LLaVA-Mini is a unified large multimodal model that can support the
understanding of images, high-resolution images, and videos in an efficient
manner. Experiments across 11 image-based and 7 video-based benchmarks
demonstrate that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token
instead of 576. Efficiency analyses reveal that LLaVA-Mini can reduce FLOPs by
77%, deliver low-latency responses within 40 milliseconds, and process over
10,000 frames of video on the GPU hardware with 24GB of memory.
ProTracker: Probabilistic Integration for Robust and Accurate Point
  Tracking
Authors: name: Tingyang Zhang, name: Chen Wang, name: Zhiyang Dou, name: Qingzhe Gao, name: Jiahui Lei, name: Baoquan Chen, name: Lingjie Liu
Published: 2025-01-06 18:55:52

In this paper, we propose ProTracker, a novel framework for robust and
accurate long-term dense tracking of arbitrary points in videos. The key idea
of our method is incorporating probabilistic integration to refine multiple
predictions from both optical flow and semantic features for robust short-term
and long-term tracking. Specifically, we integrate optical flow estimations in
a probabilistic manner, producing smooth and accurate trajectories by
maximizing the likelihood of each prediction. To effectively re-localize
challenging points that disappear and reappear due to occlusion, we further
incorporate long-term feature correspondence into our flow predictions for
continuous trajectory generation. Extensive experiments show that ProTracker
achieves the state-of-the-art performance among unsupervised and
self-supervised approaches, and even outperforms supervised methods on several
benchmarks. Our code and model will be publicly available upon publication.
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level
  Reward Models
Authors: name: Mingyang Song, name: Zhaochen Su, name: Xiaoye Qu, name: Jiawei Zhou, name: Yu Cheng
Published: 2025-01-06 16:31:45

Process-level Reward Models (PRMs) are crucial for complex reasoning and
decision-making tasks, where each intermediate step plays an important role in
the reasoning process. Since language models are prone to various types of
errors during the reasoning process, PRMs are required to possess nuanced
capabilities for detecting various implicit error types in real-world
scenarios. However, current benchmarks primarily focus on step correctness,
failing to evaluate PRMs' performance systematically. To address this gap, we
introduce PRMBench, a process-level benchmark specifically designed to assess
the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216
carefully designed problems and 83,456 step-level labels, evaluating models
across multiple dimensions, including simplicity, soundness, and sensitivity.
In our experiments on 15 models, spanning both open-source PRMs and
closed-source large language models prompted as critic models, we uncover
significant weaknesses in current PRMs. These findings underscore the
challenges inherent in process-level evaluation and highlight key directions
for future research. We hope PRMBench can be a robust bench for advancing
research on PRM evaluation and development.
Automated Generation of Challenging Multiple-Choice Questions for Vision
  Language Model Evaluation
Authors: name: Yuhui Zhang, name: Yuchang Su, name: Yiming Liu, name: Xiaohan Wang, name: James Burgess, name: Elaine Sui, name: Chenyu Wang, name: Josiah Aklilu, name: Alejandro Lozano, name: Anjiang Wei, name: Ludwig Schmidt, name: Serena Yeung-Levy
Published: 2025-01-06 18:57:31

The rapid development of vision language models (VLMs) demands rigorous and
reliable evaluation. However, current visual question answering (VQA)
benchmarks often depend on open-ended questions, making accurate evaluation
difficult due to the variability in natural language responses. To address
this, we introduce AutoConverter, an agentic framework that automatically
converts these open-ended questions into multiple-choice format, enabling
objective evaluation while reducing the costly question creation process. Our
experiments demonstrate that AutoConverter can generate correct and challenging
multiple-choice questions, with VLMs demonstrating consistently similar or
lower accuracy on these questions compared to human-created ones. Using
AutoConverter, we construct VMCBench, a benchmark created by transforming 20
existing VQA datasets into a unified multiple-choice format, totaling 9,018
questions. We comprehensively evaluate 33 state-of-the-art VLMs on VMCBench,
setting a new standard for scalable, consistent, and reproducible VLM
evaluation.
AutoPresent: Designing Structured Visuals from Scratch
Authors: name: Jiaxin Ge, name: Zora Zhiruo Wang, name: Xuhui Zhou, name: Yi-Hao Peng, name: Sanjay Subramanian, name: Qinyue Tan, name: Maarten Sap, name: Alane Suhr, name: Daniel Fried, name: Graham Neubig, name: Trevor Darrell
Published: 2025-01-01 18:09:32

Designing structured visuals such as presentation slides is essential for
communicative needs, necessitating both content creation and visual planning
skills. In this work, we tackle the challenge of automated slide generation,
where models produce slide presentations from natural language (NL)
instructions. We first introduce the SlidesBench benchmark, the first benchmark
for slide generation with 7k training and 585 testing examples derived from 310
slide decks across 10 domains. SlidesBench supports evaluations that are
(i)reference-based to measure similarity to a target slide, and
(ii)reference-free to measure the design quality of generated slides alone. We
benchmark end-to-end image generation and program generation methods with a
variety of models, and find that programmatic methods produce higher-quality
slides in user-interactable formats. Built on the success of program
generation, we create AutoPresent, an 8B Llama-based model trained on 7k pairs
of instructions paired with code for slide generation, and achieve results
comparable to the closed-source model GPT-4o. We further explore iterative
design refinement where the model is tasked to self-refine its own output, and
we found that this process improves the slide's quality. We hope that our work
will provide a basis for future work on generating structured visuals.
Scaling Laws for Floating Point Quantization Training
Authors: name: Xingwu Sun, name: Shuaipeng Li, name: Ruobing Xie, name: Weidong Han, name: Kan Wu, name: Zhen Yang, name: Yixing Li, name: An Wang, name: Shuai Li, name: Jinbao Xue, name: Yu Cheng, name: Yangyu Tao, name: Zhanhui Kang, name: Chengzhong Xu, name: Di Wang, name: Jie Jiang
Published: 2025-01-05 02:30:41

Low-precision training is considered an effective strategy for reducing both
training and downstream inference costs. Previous scaling laws for precision
mainly focus on integer quantization, which pay less attention to the
constituents in floating-point quantization and thus cannot well fit the LLM
losses in this scenario. In contrast, while floating-point quantization
training is more commonly implemented in production, the research on it has
been relatively superficial. In this paper, we thoroughly explore the effects
of floating-point quantization targets, exponent bits, mantissa bits, and the
calculation granularity of the scaling factor in floating-point quantization
training performance of LLM models. While presenting an accurate floating-point
quantization unified scaling law, we also provide valuable suggestions for the
community: (1) Exponent bits contribute slightly more to the model performance
than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
different bit numbers, which is available for future reference by hardware
manufacturers; (2) We discover the formation of the critical data size in
low-precision LLM training. Too much training data exceeding the critical data
size will inversely bring in degradation of LLM performance; (3) The optimal
floating-point quantization precision is directly proportional to the
computational power, but within a wide computational power range, we estimate
that the best cost-performance precision lies between 4-8 bits.
DepthMaster: Taming Diffusion Models for Monocular Depth Estimation
Authors: name: Ziyang Song, name: Zerong Wang, name: Bo Li, name: Hao Zhang, name: Ruijie Zhu, name: Li Liu, name: Peng-Tao Jiang, name: Tianzhu Zhang
Published: 2025-01-05 15:18:32

Monocular depth estimation within the diffusion-denoising paradigm
demonstrates impressive generalization ability but suffers from low inference
speed. Recent methods adopt a single-step deterministic paradigm to improve
inference efficiency while maintaining comparable performance. However, they
overlook the gap between generative and discriminative features, leading to
suboptimal results. In this work, we propose DepthMaster, a single-step
diffusion model designed to adapt generative features for the discriminative
depth estimation task. First, to mitigate overfitting to texture details
introduced by generative features, we propose a Feature Alignment module, which
incorporates high-quality semantic features to enhance the denoising network's
representation capability. Second, to address the lack of fine-grained details
in the single-step deterministic framework, we propose a Fourier Enhancement
module to adaptively balance low-frequency structure and high-frequency
details. We adopt a two-stage training strategy to fully leverage the potential
of the two modules. In the first stage, we focus on learning the global scene
structure with the Feature Alignment module, while in the second stage, we
exploit the Fourier Enhancement module to improve the visual quality. Through
these efforts, our model achieves state-of-the-art performance in terms of
generalization and detail preservation, outperforming other diffusion-based
methods across various datasets. Our project page can be found at
https://indu1ge.github.io/DepthMaster_page.
BoostStep: Boosting mathematical capability of Large Language Models via
  improved single-step reasoning
Authors: name: Beichen Zhang, name: Yuhong Liu, name: Xiaoyi Dong, name: Yuhang Zang, name: Pan Zhang, name: Haodong Duan, name: Yuhang Cao, name: Dahua Lin, name: Jiaqi Wang
Published: 2025-01-06 18:59:13

Cutting-edge large language models (LLMs) demonstrate promising performance
in solving complex math problems with a divide-and-conquer pipeline and the
assistance of in-context learning (ICL) examples. However, their potential for
improvement is limited by two critical problems within their ICL examples:
granularity-mismatch and the ensuing negative-effect noise problem.
Specifically, the LLMs are capable of the dividing process yet mostly failed by
inaccurate reasoning within a few conquer steps, while the ICL examples
retrieved in question-grained sometimes lack relevant steps for a specific
challenging reasoning step. Further, this disconnect may hinder the correct
reasoning due to its irrelevance. To this end, we focus on improving the
reasoning quality within each step and present BoostStep. BoostStep aligns the
granularity between the retrieving and reasoning on step grained, and provides
highly related ICL examples for each reasoning step with a novel `first-try'
strategy. BoostStep provides more relevant examples than the coarse
question-grained strategy, enhancing the model reasoning quality within each
step steadily. BoostStep is a general and robust reasoning-enhancing method
that not only improves standalone reasoning performance but also integrates
seamlessly with Monte Carlo Tree Search methods (MCTS) to refine both candidate
generation and decision-making. Quantitatively, it improves GPT-4o and
Qwen2.5-Math-72B by 3.6\% and 2.0\% respectively on various mathematical
benchmarks, and 7.5\% gain combined with MCTS.
Dispider: Enabling Video LLMs with Active Real-Time Interaction via
  Disentangled Perception, Decision, and Reaction
Authors: name: Rui Qian, name: Shuangrui Ding, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Dahua Lin, name: Jiaqi Wang
Published: 2025-01-06 18:55:10

Active Real-time interaction with video LLMs introduces a new paradigm for
human-computer interaction, where the model not only understands user intent
but also responds while continuously processing streaming video on the fly.
Unlike offline video LLMs, which analyze the entire video before answering
questions, active real-time interaction requires three capabilities: 1)
Perception: real-time video monitoring and interaction capturing. 2) Decision:
raising proactive interaction in proper situations, 3) Reaction: continuous
interaction with users. However, inherent conflicts exist among the desired
capabilities. The Decision and Reaction require a contrary Perception scale and
grain, and the autoregressive decoding blocks the real-time Perception and
Decision during the Reaction. To unify the conflicted capabilities within a
harmonious system, we present Dispider, a system that disentangles Perception,
Decision, and Reaction. Dispider features a lightweight proactive streaming
video processing module that tracks the video stream and identifies optimal
moments for interaction. Once the interaction is triggered, an asynchronous
interaction module provides detailed responses, while the processing module
continues to monitor the video in the meantime. Our disentangled and
asynchronous design ensures timely, contextually accurate, and computationally
efficient responses, making Dispider ideal for active real-time interaction for
long-duration video streams. Experiments show that Dispider not only maintains
strong performance in conventional video QA tasks, but also significantly
surpasses previous online models in streaming scenario responses, thereby
validating the effectiveness of our architecture. The code and model are
released at https://github.com/Mark12Ding/Dispider.