Orionnnn/hub-stats huggingface dataset:
SkyReels-A2: Compose Anything in Video Diffusion Transformers
Authors: name: Zhengcong Fei, name: Debang Li, name: Di Qiu, name: Jiahua Wang, name: Yikun Dou, name: Rui Wang, name: Jingtao Xu, name: Mingyuan Fan, name: Guibin Chen, name: Yang Li, name: Yahui Zhou
Published: 2025-04-03 09:50:50

This paper presents SkyReels-A2, a controllable video generation framework
capable of assembling arbitrary visual elements (e.g., characters, objects,
backgrounds) into synthesized videos based on textual prompts while maintaining
strict consistency with reference images for each element. We term this task
elements-to-video (E2V), whose primary challenges lie in preserving the
fidelity of each reference element, ensuring coherent composition of the scene,
and achieving natural outputs. To address these, we first design a
comprehensive data pipeline to construct prompt-reference-video triplets for
model training. Next, we propose a novel image-text joint embedding model to
inject multi-element representations into the generative process, balancing
element-specific consistency with global coherence and text alignment. We also
optimize the inference pipeline for both speed and output stability. Moreover,
we introduce a carefully curated benchmark for systematic evaluation, i.e, A2
Bench. Experiments demonstrate that our framework can generate diverse,
high-quality videos with precise element control. SkyReels-A2 is the first
open-source commercial grade model for the generation of E2V, performing
favorably against advanced closed-source commercial models. We anticipate
SkyReels-A2 will advance creative applications such as drama and virtual
e-commerce, pushing the boundaries of controllable video generation.
Adaptive Layer-skipping in Pre-trained LLMs
Authors: name: Xuan Luo, name: Weizhi Wang, name: Xifeng Yan
Published: 2025-03-31 07:20:58

Various layer-skipping methods have been proposed to accelerate token
generation in large language models (LLMs). However, they have overlooked a
fundamental question: How do computational demands vary across the generation
of different tokens? In this work, we introduce FlexiDepth, a method that
dynamically adjusts the number of Transformer layers used in text generation.
By incorporating a plug-in router and adapter, FlexiDepth enables adaptive
layer-skipping in LLMs without modifying their original parameters. Introducing
FlexiDepth to Llama-3-8B model achieves layer skipping of 8 layers out of 32,
and meanwhile maintains the full 100\% benchmark performance. Experimental
results with FlexiDepth demonstrate that computational demands in LLMs
significantly vary based on token type. Specifically, generating repetitive
tokens or fixed phrases requires fewer layers, whereas producing tokens
involving computation or high uncertainty requires more layers. Interestingly,
this adaptive allocation pattern aligns with human intuition. To advance
research in this area, we open sourced FlexiDepth and a dataset documenting
FlexiDepth's layer allocation patterns for future exploration.
Quamba2: A Robust and Scalable Post-training Quantization Framework for
  Selective State Space Models
Authors: name: Hung-Yueh Chiang, name: Chi-Chih Chang, name: Natalia Frumkin, name: Kai-Chiang Wu, name: Mohamed S. Abdelfattah, name: Diana Marculescu
Published: 2025-03-28 21:10:39

State Space Models (SSMs) are emerging as a compelling alternative to
Transformers because of their consistent memory usage and high performance.
Despite this, scaling up SSMs on cloud services or limited-resource devices is
challenging due to their storage requirements and computational power. To
overcome this, quantizing SSMs with low bit-width data formats can reduce model
size and benefit from hardware acceleration. As SSMs are prone to
quantization-induced errors, recent efforts have focused on optimizing a
particular model or bit-width for efficiency without sacrificing performance.
However, distinct bit-width configurations are essential for different
scenarios, like W4A8 for boosting large-batch decoding speed, and W4A16 for
enhancing generation speed in short prompt applications for a single user. To
this end, we present Quamba2, compatible with W8A8, W4A8, and W4A16 for both
Mamba1 and Mamba2 backbones, addressing the growing demand for SSM deployment
on various platforms. Based on the channel order preserving and activation
persistence of SSMs, we propose an offline approach to quantize inputs of a
linear recurrence in 8-bit by sorting and clustering for input x, combined
with a per-state-group quantization for input-dependent parameters B and C.
To ensure compute-invariance in the SSM output, we rearrange weights offline
according to the clustering sequence. The experiments show that Quamba2-8B
outperforms several state-of-the-art SSM quantization methods and delivers
1.3times and 3times speed-ups in the pre-filling and generation stages,
respectively, while offering 4times memory reduction with only a 1.6%
average accuracy drop. The evaluation on MMLU shows the generalizability and
robustness of our framework. The code and quantized models will be released at:
https://github.com/enyac-group/Quamba.
Target-Aware Video Diffusion Models
Authors: name: Taeksoo Kim, name: Hanbyul Joo
Published: 2025-03-24 17:59:59

We present a target-aware video diffusion model that generates videos from an
input image in which an actor interacts with a specified target while
performing a desired action. The target is defined by a segmentation mask and
the desired action is described via a text prompt. Unlike existing controllable
image-to-video diffusion models that often rely on dense structural or motion
cues to guide the actor's movements toward the target, our target-aware model
requires only a simple mask to indicate the target, leveraging the
generalization capabilities of pretrained models to produce plausible actions.
This makes our method particularly effective for human-object interaction (HOI)
scenarios, where providing precise action guidance is challenging, and further
enables the use of video diffusion models for high-level action planning in
applications such as robotics. We build our target-aware model by extending a
baseline model to incorporate the target mask as an additional input. To
enforce target awareness, we introduce a special token that encodes the
target's spatial information within the text prompt. We then fine-tune the
model with our curated dataset using a novel cross-attention loss that aligns
the cross-attention maps associated with this token with the input target mask.
To further improve performance, we selectively apply this loss to the most
semantically relevant transformer blocks and attention regions. Experimental
results show that our target-aware model outperforms existing solutions in
generating videos where actors interact accurately with the specified targets.
We further demonstrate its efficacy in two downstream applications: video
content creation and zero-shot 3D HOI motion synthesis.
MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for
  Zero-Shot Speech Synthesis
Authors: name: Ziyue Jiang, name: Yi Ren, name: Ruiqi Li, name: Shengpeng Ji, name: Boyang Zhang, name: Zhenhui Ye, name: Chen Zhang, name: Bai Jionghao, name: Xiaoda Yang, name: Jialong Zuo, name: Yu Zhang, name: Rui Liu, name: Xiang Yin, name: Zhou Zhao
Published: 2025-02-26 08:22:00

While recent zero-shot text-to-speech (TTS) models have significantly
improved speech quality and expressiveness, mainstream systems still suffer
from issues related to speech-text alignment modeling: 1) models without
explicit speech-text alignment modeling exhibit less robustness, especially for
hard sentences in practical applications; 2) predefined alignment-based models
suffer from naturalness constraints of forced alignments. This paper introduces
MegaTTS 3, a TTS system featuring an innovative sparse alignment
algorithm that guides the latent diffusion transformer (DiT). Specifically, we
provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of
alignment without limiting the search space, thereby achieving high
naturalness. Moreover, we employ a multi-condition classifier-free guidance
strategy for accent intensity adjustment and adopt the piecewise rectified flow
technique to accelerate the generation process. Experiments demonstrate that
MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports
highly flexible control over accent intensity. Notably, our system can generate
high-quality one-minute speech with only 8 sampling steps. Audio samples are
available at https://sditdemo.github.io/sditdemo/.
Medical large language models are easily distracted
Authors: name: Krithik Vishwanath, name: Anton Alyakin, name: Daniel Alexander Alber, name: Jin Vivian Lee, name: Douglas Kondziolka, name: Eric Karl Oermann
Published: 2025-04-01 21:34:01

Large language models (LLMs) have the potential to transform medicine, but
real-world clinical scenarios contain extraneous information that can hinder
performance. The rise of assistive technologies like ambient dictation, which
automatically generates draft notes from live patient encounters, has the
potential to introduce additional noise making it crucial to assess the ability
of LLM's to filter relevant data. To investigate this, we developed
MedDistractQA, a benchmark using USMLE-style questions embedded with simulated
real-world distractions. Our findings show that distracting statements
(polysemous words with clinical meanings used in a non-clinical context or
references to unrelated health conditions) can reduce LLM accuracy by up to
17.9%. Commonly proposed solutions to improve model performance such as
retrieval-augmented generation (RAG) and medical fine-tuning did not change
this effect and in some cases introduced their own confounders and further
degraded performance. Our findings suggest that LLMs natively lack the logical
mechanisms necessary to distinguish relevant from irrelevant clinical
information, posing challenges for real-world applications. MedDistractQA and
our results highlights the need for robust mitigation strategies to enhance LLM
resilience to extraneous information.
Towards Physically Plausible Video Generation via VLM Planning
Authors: name: Xindi Yang, name: Baolu Li, name: Yiming Zhang, name: Zhenfei Yin, name: Lei Bai, name: Liqian Ma, name: Zhiyong Wang, name: Jianfei Cai, name: Tien-Tsin Wong, name: Huchuan Lu, name: Xu Jia
Published: 2025-03-30 09:03:09

Video diffusion models (VDMs) have advanced significantly in recent years,
enabling the generation of highly realistic videos and drawing the attention of
the community in their potential as world simulators. However, despite their
capabilities, VDMs often fail to produce physically plausible videos due to an
inherent lack of understanding of physics, resulting in incorrect dynamics and
event sequences. To address this limitation, we propose a novel two-stage
image-to-video generation framework that explicitly incorporates physics. In
the first stage, we employ a Vision Language Model (VLM) as a coarse-grained
motion planner, integrating chain-of-thought and physics-aware reasoning to
predict a rough motion trajectories/changes that approximate real-world
physical dynamics while ensuring the inter-frame consistency. In the second
stage, we use the predicted motion trajectories/changes to guide the video
generation of a VDM. As the predicted motion trajectories/changes are rough,
noise is added during inference to provide freedom to the VDM in generating
motion with more fine details. Extensive experimental results demonstrate that
our framework can produce physically plausible motion, and comparative
evaluations highlight the notable superiority of our approach over existing
methods. More video results are available on our Project Page:
https://madaoer.github.io/projects/physically_plausible_video_generation.
DASH: Detection and Assessment of Systematic Hallucinations of VLMs
Authors: name: Maximilian Augustin, name: Yannic Neuhaus, name: Matthias Hein
Published: 2025-03-30 19:45:09

Vision-language models (VLMs) are prone to object hallucinations, where they
erroneously indicate the presenceof certain objects in an image. Existing
benchmarks quantify hallucinations using relatively small, labeled datasets.
However, this approach is i) insufficient to assess hallucinations that arise
in open-world settings, where VLMs are widely used, and ii) inadequate for
detecting systematic errors in VLMs. We propose DASH (Detection and Assessment
of Systematic Hallucinations), an automatic, large-scale pipeline designed to
identify systematic hallucinations of VLMs on real-world images in an
open-world setting. A key component is DASH-OPT for image-based retrieval,
where we optimize over the ''natural image manifold'' to generate images that
mislead the VLM. The output of DASH consists of clusters of real and
semantically similar images for which the VLM hallucinates an object. We apply
DASH to PaliGemma and two LLaVA-NeXT models across 380 object classes and, in
total, find more than 19k clusters with 950k images. We study the transfer of
the identified systematic hallucinations to other VLMs and show that
fine-tuning PaliGemma with the model-specific images obtained with DASH
mitigates object hallucinations. Code and data are available at
https://YanNeu.github.io/DASH.
Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal
  Representations
Authors: name: Jeonghyeon Kim, name: Sangheum Hwang
Published: 2025-03-24 16:00:21

Prior research on out-of-distribution detection (OoDD) has primarily focused
on single-modality models. Recently, with the advent of large-scale pretrained
vision-language models such as CLIP, OoDD methods utilizing such multi-modal
representations through zero-shot and prompt learning strategies have emerged.
However, these methods typically involve either freezing the pretrained weights
or only partially tuning them, which can be suboptimal for downstream datasets.
In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve
notable OoDD performance. Despite some recent works demonstrating the impact of
fine-tuning methods for OoDD, there remains significant potential for
performance improvement. We investigate the limitation of na\"ive fine-tuning
methods, examining why they fail to fully leverage the pretrained knowledge.
Our empirical analysis suggests that this issue could stem from the modality
gap within in-distribution (ID) embeddings. To address this, we propose a
training objective that enhances cross-modal alignment by regularizing the
distances between image and text embeddings of ID data. This adjustment helps
in better utilizing pretrained textual information by aligning similar
semantics from different modalities (i.e., text and image) more closely in the
hyperspherical representation space. We theoretically demonstrate that the
proposed regularization corresponds to the maximum likelihood estimation of an
energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark
datasets, we show that our method, combined with post-hoc OoDD approaches
leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms
existing methods, achieving state-of-the-art OoDD performance and leading ID
accuracy.
VerifiAgent: a Unified Verification Agent in Language Model Reasoning
Authors: name: Jiuzhou Han, name: Wray Buntine, name: Ehsan Shareghi
Published: 2025-04-01 04:05:03

Large language models demonstrate remarkable reasoning capabilities but often
produce unreliable or incorrect responses. Existing verification methods are
typically model-specific or domain-restricted, requiring significant
computational resources and lacking scalability across diverse reasoning tasks.
To address these limitations, we propose VerifiAgent, a unified verification
agent that integrates two levels of verification: meta-verification, which
assesses completeness and consistency in model responses, and tool-based
adaptive verification, where VerifiAgent autonomously selects appropriate
verification tools based on the reasoning type, including mathematical,
logical, or commonsense reasoning. This adaptive approach ensures both
efficiency and robustness across different verification scenarios. Experimental
results show that VerifiAgent outperforms baseline verification methods (e.g.,
deductive verifier, backward verifier) among all reasoning tasks. Additionally,
it can further enhance reasoning accuracy by leveraging feedback from
verification results. VerifiAgent can also be effectively applied to inference
scaling, achieving better results with fewer generated samples and costs
compared to existing process reward models in the mathematical reasoning
domain. Code is available at https://github.com/Jiuzhouh/VerifiAgent
Boost Your Own Human Image Generation Model via Direct Preference
  Optimization with AI Feedback
Authors: name: Sanghyeon Na, name: Yonggyu Kim, name: Hyunjoon Lee
Published: 2024-05-30 16:18:05

The generation of high-quality human images through text-to-image (T2I)
methods is a significant yet challenging task. Distinct from general image
generation, human image synthesis must satisfy stringent criteria related to
human pose, anatomy, and alignment with textual prompts, making it particularly
difficult to achieve realistic results. Recent advancements in T2I generation
based on diffusion models have shown promise, yet challenges remain in meeting
human-specific preferences. In this paper, we introduce a novel approach
tailored specifically for human image generation utilizing Direct Preference
Optimization (DPO). Specifically, we introduce an efficient method for
constructing a specialized DPO dataset for training human image generation
models without the need for costly human feedback. We also propose a modified
loss function that enhances the DPO training process by minimizing artifacts
and improving image fidelity. Our method demonstrates its versatility and
effectiveness in generating human images, including personalized text-to-image
generation. Through comprehensive evaluations, we show that our approach
significantly advances the state of human image generation, achieving superior
results in terms of natural anatomies, poses, and text-image alignment.
Understanding R1-Zero-Like Training: A Critical Perspective
Authors: name: Zichen Liu, name: Changyu Chen, name: Wenjun Li, name: Penghui Qi, name: Tianyu Pang, name: Chao Du, name: Wee Sun Lee, name: Min Lin
Published: 2025-03-26 17:59:14

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can
directly enhance the reasoning capabilities of LLMs without supervised
fine-tuning. In this work, we critically examine R1-Zero-like training by
analyzing its two core components: base models and RL. We investigate a wide
range of base models, including DeepSeek-V3-Base, to understand how pretraining
characteristics influence RL performance. Our analysis reveals that
DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models
demonstrate strong reasoning capabilities even without prompt templates,
suggesting potential pretraining biases. Additionally, we identify an
optimization bias in Group Relative Policy Optimization (GRPO), which
artificially increases response length (especially for incorrect outputs)
during training. To address this, we introduce Dr. GRPO, an unbiased
optimization method that improves token efficiency while maintaining reasoning
performance. Leveraging these insights, we present a minimalist R1-Zero recipe
that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a
new state-of-the-art. Our code is available at
https://github.com/sail-sg/understand-r1-zero.
VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in
  One Step
Authors: name: Hanyang Wang, name: Fangfu Liu, name: Jiawei Chi, name: Yueqi Duan
Published: 2025-04-02 17:59:21

Recovering 3D scenes from sparse views is a challenging task due to its
inherent ill-posed problem. Conventional methods have developed specialized
solutions (e.g., geometry regularization or feed-forward deterministic model)
to mitigate the issue. However, they still suffer from performance degradation
by minimal overlap across input views with insufficient visual information.
Fortunately, recent video generative models show promise in addressing this
challenge as they are capable of generating video clips with plausible 3D
structures. Powered by large pretrained video diffusion models, some pioneering
research start to explore the potential of video generative prior and create 3D
scenes from sparse views. Despite impressive improvements, they are limited by
slow inference time and the lack of 3D constraint, leading to inefficiencies
and reconstruction artifacts that do not align with real-world geometry
structure. In this paper, we propose VideoScene to distill the video diffusion
model to generate 3D scenes in one step, aiming to build an efficient and
effective tool to bridge the gap from video to 3D. Specifically, we design a
3D-aware leap flow distillation strategy to leap over time-consuming redundant
information and train a dynamic denoising policy network to adaptively
determine the optimal leap timestep during inference. Extensive experiments
demonstrate that our VideoScene achieves faster and superior 3D scene
generation results than previous video diffusion models, highlighting its
potential as an efficient tool for future video to 3D applications. Project
Page: https://hanyang-21.github.io/VideoScene
ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and
  Diffusion Refinement
Authors: name: Runhui Huang, name: Chunwei Wang, name: Junwei Yang, name: Guansong Lu, name: Yunlong Yuan, name: Jianhua Han, name: Lu Hou, name: Wei Zhang, name: Lanqing Hong, name: Hengshuang Zhao, name: Hang Xu
Published: 2025-04-02 17:45:00

We present ILLUME+ that leverages dual visual tokenization and a diffusion
decoder to improve both deep semantic understanding and high-fidelity image
generation. Existing unified models have struggled to simultaneously handle the
three fundamental capabilities in a unified model: understanding, generation,
and editing. Models like Chameleon and EMU3 utilize VQGAN for image
discretization, due to the lack of deep semantic interaction, they lag behind
specialist models like LLaVA in visual understanding tasks. To mitigate this,
LaViT and ILLUME employ semantic encoders for tokenization, but they struggle
with image editing due to poor texture preservation. Meanwhile, Janus series
decouples the input and output image representation, limiting their abilities
to seamlessly handle interleaved image-text understanding and generation. In
contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which
preserves both fine-grained textures and text-aligned semantics while enabling
a coarse-to-fine image representation strategy for multimodal understanding and
generation. Additionally, we employ a diffusion model as the image detokenizer
for enhanced generation quality and efficient super-resolution. ILLUME+ follows
a continuous-input, discrete-output scheme within the unified MLLM and adopts a
progressive training procedure that supports dynamic resolution across the
vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible
and efficient context-aware image editing and generation across diverse tasks.
ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs
and specialized models across multimodal understanding, generation, and editing
benchmarks. With its strong performance, ILLUME+ provides a scalable and
versatile foundation for future multimodal applications. Project Page:
https://illume-unified-mllm.github.io/.