Orionnnn/hub-stats huggingface dataset:
FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow
  Models
Authors: name: Vladimir Kulikov, name: Matan Kleiner, name: Inbar Huberman-Spiegelglas, name: Tomer Michaeli
Published: 2024-12-11 18:50:29

Editing real images using a pre-trained text-to-image (T2I) diffusion/flow
model often involves inverting the image into its corresponding noise map.
However, inversion by itself is typically insufficient for obtaining
satisfactory results, and therefore many methods additionally intervene in the
sampling process. Such methods achieve improved results but are not seamlessly
transferable between model architectures. Here, we introduce FlowEdit, a
text-based editing method for pre-trained T2I flow models, which is
inversion-free, optimization-free and model agnostic. Our method constructs an
ODE that directly maps between the source and target distributions
(corresponding to the source and target text prompts) and achieves a lower
transport cost than the inversion approach. This leads to state-of-the-art
results, as we illustrate with Stable Diffusion 3 and FLUX. Code and examples
are available on the project's webpage.
MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation
Authors: name: Bo Li, name: Shaolin Zhu, name: Lijie Wen
Published: 2024-12-10 03:12:35

Image Translation (IT) holds immense potential across diverse domains,
enabling the translation of textual content within images into various
languages. However, existing datasets often suffer from limitations in scale,
diversity, and quality, hindering the development and evaluation of IT models.
To address this issue, we introduce MIT-10M, a large-scale parallel corpus of
multilingual image translation with over 10M image-text pairs derived from
real-world data, which has undergone extensive data cleaning and multilingual
translation validation. It contains 840K images in three sizes, 28 categories,
tasks with three levels of difficulty and 14 languages image-text pairs, which
is a considerable improvement on existing datasets. We conduct extensive
experiments to evaluate and train models on MIT-10M. The experimental results
clearly indicate that our dataset has higher adaptability when it comes to
evaluating the performance of the models in tackling challenging and complex
image translation tasks in the real world. Moreover, the performance of the
model fine-tuned with MIT-10M has tripled compared to the baseline model,
further confirming its superiority.
Learning Flow Fields in Attention for Controllable Person Image
  Generation
Authors: name: Zijian Zhou, name: Shikun Liu, name: Xiao Han, name: Haozhe Liu, name: Kam Woh Ng, name: Tian Xie, name: Yuren Cong, name: Hang Li, name: Mengmeng Xu, name: Juan-Manuel Pérez-Rúa, name: Aditya Patel, name: Tao Xiang, name: Miaojing Shi, name: Sen He
Published: 2024-12-11 15:51:14

Controllable person image generation aims to generate a person image
conditioned on reference images, allowing precise control over the person's
appearance or pose. However, prior methods often distort fine-grained textural
details from the reference image, despite achieving high overall image quality.
We attribute these distortions to inadequate attention to corresponding regions
in the reference image. To address this, we thereby propose learning flow
fields in attention (Leffa), which explicitly guides the target query to attend
to the correct reference key in the attention layer during training.
Specifically, it is realized via a regularization loss on top of the attention
map within a diffusion-based baseline. Our extensive experiments show that
Leffa achieves state-of-the-art performance in controlling appearance (virtual
try-on) and pose (pose transfer), significantly reducing fine-grained detail
distortion while maintaining high image quality. Additionally, we show that our
loss is model-agnostic and can be used to improve the performance of other
diffusion models.
StreamChat: Chatting with Streaming Video
Authors: name: Jihao Liu, name: Zhiding Yu, name: Shiyi Lan, name: Shihao Wang, name: Rongyao Fang, name: Jan Kautz, name: Hongsheng Li, name: Jose M. Alvare
Published: 2024-12-11 18:59:54

This paper presents StreamChat, a novel approach that enhances the
interaction capabilities of Large Multimodal Models (LMMs) with streaming video
content. In streaming interaction scenarios, existing methods rely solely on
visual information available at the moment a question is posed, resulting in
significant delays as the model remains unaware of subsequent changes in the
streaming video. StreamChat addresses this limitation by innovatively updating
the visual context at each decoding step, ensuring that the model utilizes
up-to-date video content throughout the decoding process. Additionally, we
introduce a flexible and efficient crossattention-based architecture to process
dynamic streaming inputs while maintaining inference efficiency for streaming
interactions. Furthermore, we construct a new dense instruction dataset to
facilitate the training of streaming interaction models, complemented by a
parallel 3D-RoPE mechanism that encodes the relative temporal information of
visual and text tokens. Experimental results demonstrate that StreamChat
achieves competitive performance on established image and video benchmarks and
exhibits superior capabilities in streaming interaction scenarios compared to
state-of-the-art video LMM.
StyleMaster: Stylize Your Video with Artistic Generation and Translation
Authors: name: Zixuan Ye, name: Huijuan Huang, name: Xintao Wang, name: Pengfei Wan, name: Di Zhang, name: Wenhan Luo
Published: 2024-12-10 18:44:08

Style control has been popular in video generation models. Existing methods
often generate videos far from the given style, cause content leakage, and
struggle to transfer one video to the desired style. Our first observation is
that the style extraction stage matters, whereas existing methods emphasize
global style but ignore local textures. In order to bring texture features
while preventing content leakage, we filter content-related patches while
retaining style ones based on prompt-patch similarity; for global style
extraction, we generate a paired style dataset through model illusion to
facilitate contrastive learning, which greatly enhances the absolute style
consistency. Moreover, to fill in the image-to-video gap, we train a
lightweight motion adapter on still videos, which implicitly enhances
stylization extent, and enables our image-trained model to be seamlessly
applied to videos. Benefited from these efforts, our approach, StyleMaster, not
only achieves significant improvement in both style resemblance and temporal
coherence, but also can easily generalize to video style transfer with a gray
tile ControlNet. Extensive experiments and visualizations demonstrate that
StyleMaster significantly outperforms competitors, effectively generating
high-quality stylized videos that align with textual content and closely
resemble the style of reference images. Our project page is at
https://zixuan-ye.github.io/stylemaster
POINTS1.5: Building a Vision-Language Model towards Real World
  Applications
Authors: name: Yuan Liu, name: Le Tian, name: Xiao Zhou, name: Xinyu Gao, name: Kavio Yu, name: Yang Yu, name: Jie Zhou
Published: 2024-12-11 15:08:25

Vision-language models have made significant strides recently, demonstrating
superior performance across a range of tasks, e.g. optical character
recognition and complex diagram analysis. Building on this trend, we introduce
a new vision-language model, POINTS1.5, designed to excel in various real-world
applications. POINTS1.5 is an enhancement of POINTS1.0 and incorporates several
key innovations: i) We replace the original CLIP vision encoder, which had a
fixed image resolution, with a NaViT-style vision encoder that supports native
dynamic high resolution. This allows POINTS1.5 to process images of any
resolution without needing to split them into tiles. ii) We add bilingual
support to POINTS1.5, significantly enhancing its capability in Chinese. Due to
the scarcity of open-source Chinese datasets for vision-language models, we
collect numerous images from the Internet and annotate them using a combination
of manual and automatic methods. iii) We propose a set of rigorous filtering
methods for visual instruction tuning datasets. We comprehensively evaluate all
these filtering methods, and choose the most effective ones to obtain the final
visual instruction tuning set. Thanks to these innovations, POINTS1.5
significantly outperforms POINTS1.0 and demonstrates strong performance across
a range of real-world applications. Notably, POINTS1.5-7B is trained on fewer
than 4 billion tokens and ranks first on the OpenCompass leaderboard among
models with fewer than 10 billion parameters
Generative Densification: Learning to Densify Gaussians for
  High-Fidelity Generalizable 3D Reconstruction
Authors: name: Seungtae Nam, name: Xiangyu Sun, name: Gyeongjin Kang, name: Younggeun Lee, name: Seungjun Oh, name: Eunbyung Park
Published: 2024-12-09 06:20:51

Generalized feed-forward Gaussian models have achieved significant progress
in sparse-view 3D reconstruction by leveraging prior knowledge from large
multi-view datasets. However, these models often struggle to represent
high-frequency details due to the limited number of Gaussians. While the
densification strategy used in per-scene 3D Gaussian splatting (3D-GS)
optimization can be adapted to the feed-forward models, it may not be ideally
suited for generalized scenarios. In this paper, we propose Generative
Densification, an efficient and generalizable method to densify Gaussians
generated by feed-forward models. Unlike the 3D-GS densification strategy,
which iteratively splits and clones raw Gaussian parameters, our method
up-samples feature representations from the feed-forward models and generates
their corresponding fine Gaussians in a single forward pass, leveraging the
embedded prior knowledge for enhanced generalization. Experimental results on
both object-level and scene-level reconstruction tasks demonstrate that our
method outperforms state-of-the-art approaches with comparable or smaller model
sizes, achieving notable improvements in representing fine details.
3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
Authors: name: Wufei Ma, name: Haoyu Chen, name: Guofeng Zhang, name: Celso M de Melo, name: Alan Yuille, name: Jieneng Chen
Published: 2024-12-10 18:55:23

3D spatial reasoning is the ability to analyze and interpret the positions,
orientations, and spatial relationships of objects within the 3D space. This
allows models to develop a comprehensive understanding of the 3D scene,
enabling their applicability to a broader range of areas, such as autonomous
navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have
achieved remarkable progress in a wide range of image and video understanding
tasks, their capabilities to perform 3D spatial reasoning on diverse natural
images are less studied. In this work we present the first comprehensive 3D
spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual
question-answer pairs across 12 question types. We conduct robust and thorough
evaluation of 3D spatial reasoning capabilities by balancing the data
distribution and adopting a novel FlipEval strategy. To further study the
robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench
includes two subsets with 3D spatial reasoning questions on paired images with
common and uncommon viewpoints. We benchmark a wide range of open-sourced and
proprietary LMMs, uncovering their limitations in various aspects of 3D
awareness, such as height, orientation, location, and multi-object reasoning,
as well as their degraded performance on images with uncommon camera
viewpoints. Our 3DSRBench provide valuable findings and insights about the
future development of LMMs with strong 3D reasoning capabilities. Our project
page and dataset is available https://3dsrbench.github.io.
LAION-SG: An Enhanced Large-Scale Dataset for Training Complex
  Image-Text Models with Structural Annotations
Authors: name: Zejian Li, name: Chenye Meng, name: Yize Li, name: Ling Yang, name: Shengyuan Zhang, name: Jiarui Ma, name: Jiayi Li, name: Guang Yang, name: Changyuan Yang, name: Zhiyuan Yang, name: Jinxiong Chang, name: Lingyun Sun
Published: 2024-12-11 17:57:10

Recent advances in text-to-image (T2I) generation have shown remarkable
success in producing high-quality images from text. However, existing T2I
models show decayed performance in compositional image generation involving
multiple objects and intricate relationships. We attribute this problem to
limitations in existing datasets of image-text pairs, which lack precise
inter-object relationship annotations with prompts only. To address this
problem, we construct LAION-SG, a large-scale dataset with high-quality
structural annotations of scene graphs (SG), which precisely describe
attributes and relationships of multiple objects, effectively representing the
semantic structure in complex scenes. Based on LAION-SG, we train a new
foundation model SDXL-SG to incorporate structural annotation information into
the generation process. Extensive experiments show advanced models trained on
our LAION-SG boast significant performance improvements in complex scene
generation over models on existing datasets. We also introduce CompSG-Bench, a
benchmark that evaluates models on compositional image generation, establishing
a new standard for this domain.
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse
  Viewpoints
Authors: name: Jianhong Bai, name: Menghan Xia, name: Xintao Wang, name: Ziyang Yuan, name: Xiao Fu, name: Zuozhu Liu, name: Haoji Hu, name: Pengfei Wan, name: Di Zhang
Published: 2024-12-10 18:55:17

Recent advancements in video diffusion models have shown exceptional
abilities in simulating real-world dynamics and maintaining 3D consistency.
This progress inspires us to investigate the potential of these models to
ensure dynamic consistency across various viewpoints, a highly desirable
feature for applications such as virtual filming. Unlike existing methods
focused on multi-view generation of single objects for 4D reconstruction, our
interest lies in generating open-world videos from arbitrary viewpoints,
incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play
module that enhances a pre-trained text-to-video model for multi-camera video
generation, ensuring consistent content across different viewpoints.
Specifically, we introduce a multi-view synchronization module to maintain
appearance and geometry consistency across these viewpoints. Given the scarcity
of high-quality training data, we design a hybrid training scheme that
leverages multi-camera images and monocular videos to supplement Unreal
Engine-rendered multi-camera videos. Furthermore, our method enables intriguing
extensions, such as re-rendering a video from novel viewpoints. We also release
a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project
page: https://jianhongbai.github.io/SynCamMaster/.
Mogo: RQ Hierarchical Causal Transformer for High-Quality 3D Human
  Motion Generation
Authors: name: Dongjie Fu
Published: 2024-12-05 08:30:43

In the field of text-to-motion generation, Bert-type Masked Models (MoMask,
MMM) currently produce higher-quality outputs compared to GPT-type
autoregressive models (T2M-GPT). However, these Bert-type models often lack the
streaming output capability required for applications in video game and
multimedia environments, a feature inherent to GPT-type models. Additionally,
they demonstrate weaker performance in out-of-distribution generation. To
surpass the quality of BERT-type models while leveraging a GPT-type structure,
without adding extra refinement models that complicate scaling data, we propose
a novel architecture, Mogo (Motion Only Generate Once), which generates
high-quality lifelike 3D human motions by training a single transformer model.
Mogo consists of only two main components: 1) RVQ-VAE, a hierarchical residual
vector quantization variational autoencoder, which discretizes continuous
motion sequences with high precision; 2) Hierarchical Causal Transformer,
responsible for generating the base motion sequences in an autoregressive
manner while simultaneously inferring residuals across different layers.
Experimental results demonstrate that Mogo can generate continuous and cyclic
motion sequences up to 260 frames (13 seconds), surpassing the 196 frames (10
seconds) length limitation of existing datasets like HumanML3D. On the
HumanML3D test set, Mogo achieves a FID score of 0.079, outperforming both the
GPT-type model T2M-GPT (FID = 0.116), AttT2M (FID = 0.112) and the BERT-type
model MMM (FID = 0.080). Furthermore, our model achieves the best quantitative
performance in out-of-distribution generation.
ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion
  Transformer
Authors: name: Jinyi Hu, name: Shengding Hu, name: Yuxuan Song, name: Yufei Huang, name: Mingxuan Wang, name: Hao Zhou, name: Zhiyuan Liu, name: Wei-Ying Ma, name: Maosong Sun
Published: 2024-12-10 18:13:20

The recent surge of interest in comprehensive multimodal models has
necessitated the unification of diverse modalities. However, the unification
suffers from disparate methodologies. Continuous visual generation necessitates
the full-sequence diffusion-based approach, despite its divergence from the
autoregressive modeling in the text domain. We posit that autoregressive
modeling, i.e., predicting the future based on past deterministic experience,
remains crucial in developing both a visual generation model and a potential
unified multimodal model. In this paper, we explore an interpolation between
the autoregressive modeling and full-parameters diffusion to model visual
information. At its core, we present ACDiT, an Autoregressive blockwise
Conditional Diffusion Transformer, where the block size of diffusion, i.e., the
size of autoregressive units, can be flexibly adjusted to interpolate between
token-wise autoregression and full-sequence diffusion. ACDiT is easy to
implement, as simple as creating a Skip-Causal Attention Mask (SCAM) during
training. During inference, the process iterates between diffusion denoising
and autoregressive decoding that can make full use of KV-Cache. We verify the
effectiveness of ACDiT on image and video generation tasks. We also demonstrate
that benefitted from autoregressive modeling, ACDiT can be seamlessly used in
visual understanding tasks despite being trained on the diffusion objective.
The analysis of the trade-off between autoregressive modeling and diffusion
demonstrates the potential of ACDiT to be used in long-horizon visual
generation tasks. These strengths make it promising as the backbone of future
unified models.
GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
Authors: name: Ashish Goswami, name: Satyam Kumar Modi, name: Santhosh Rishi Deshineni, name: Harman Singh, name: Prathosh A. P, name: Parag Singla
Published: 2024-12-08 22:29:56

Text-to-image (T2I) generation has seen significant progress with diffusion
models, enabling generation of photo-realistic images from text prompts.
Despite this progress, existing methods still face challenges in following
complex text prompts, especially those requiring compositional and multi-step
reasoning. Given such complex instructions, SOTA models often make mistakes in
faithfully modeling object attributes, and relationships among them. In this
work, we present an alternate paradigm for T2I synthesis, decomposing the task
of complex multi-step generation into three steps, (a) Generate: we first
generate an image using existing diffusion models (b) Plan: we make use of
Multi-Modal LLMs (MLLMs) to identify the mistakes in the generated image
expressed in terms of individual objects and their properties, and produce a
sequence of corrective steps required in the form of an edit-plan. (c) Edit: we
make use of an existing text-guided image editing models to sequentially
execute our edit-plan over the generated image to get the desired image which
is faithful to the original instruction. Our approach derives its strength from
the fact that it is modular in nature, is training free, and can be applied
over any combination of image generation and editing models. As an added
contribution, we also develop a model capable of compositional editing, which
further helps improve the overall accuracy of our proposed approach. Our method
flexibly trades inference time compute with performance on compositional text
prompts. We perform extensive experimental evaluation across 3 benchmarks and
10 T2I models including DALLE-3 and the latest -- SD-3.5-Large. Our approach
not only improves the performance of the SOTA models, by upto 3 points, it also
reduces the performance gap between weaker and stronger models.
https://dair-iitd.github.io/GraPE/{https://dair-iitd.github.io/GraPE/}
HARP: Hesitation-Aware Reframing in Transformer Inference Pass
Authors: name: Romain Storaï, name: Seung-won Hwang
Published: 2024-12-10 08:12:22

This paper aims to improve the performance of large language models by
addressing the variable computational demands in inference steps, where some
tokens require more computational resources than others. We present HARP, a
simple modification to "off-the-shelf" Transformer forward pass. Drawing from
hesitation and the framing effect in decision-making, HARP selectively applies
additional computation when the model encounters uncertainty during token
generation. Our method mimics human cognitive processes by pausing at difficult
decision points and reframing inputs for a different perspective. Unlike other
approaches, HARP is model-agnostic, training-free, and easy to implement. We
thoroughly evaluate our method across various downstream tasks and model sizes,
demonstrating performance improvements up to +5.16%. Notably, HARP achieves
these gains while maintaining inference times twice faster than beam search.
Simple and yet with significant gains, HARP offers a practical solution for
enhancing the performance of Transformer-based language models with minimal
computational impact.
Mobile Video Diffusion
Authors: name: Haitam Ben Yahia, name: Denis Korzhenkov, name: Ioannis Lelekas, name: Amir Ghodrati, name: Amirhossein Habibian
Published: 2024-12-10 15:19:10

Video diffusion models have achieved impressive realism and controllability
but are limited by high computational demands, restricting their use on mobile
devices. This paper introduces the first mobile-optimized video diffusion
model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD),
we reduce memory and computational cost by reducing the frame resolution,
incorporating multi-scale temporal representations, and introducing two novel
pruning schema to reduce the number of channels and temporal blocks.
Furthermore, we employ adversarial finetuning to reduce the denoising to a
single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs.
4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents
for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are
available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/