Orionnnn/hub-stats huggingface dataset:
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and
  Generating with Multimodal LLMs
Authors: name: Ling Yang, name: Zhaochen Yu, name: Chenlin Meng, name: Minkai Xu, name: Stefano Ermon, name: Bin Cui
Published: 2024-01-22 06:16:29

Diffusion models have exhibit exceptional performance in text-to-image
generation and editing. However, existing methods often face challenges when
handling complex text prompts that involve multiple objects with multiple
attributes and relationships. In this paper, we propose a brand new
training-free text-to-image generation/editing framework, namely Recaption,
Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning
ability of multimodal LLMs to enhance the compositionality of text-to-image
diffusion models. Our approach employs the MLLM as a global planner to
decompose the process of generating complex images into multiple simpler
generation tasks within subregions. We propose complementary regional diffusion
to enable region-wise compositional generation. Furthermore, we integrate
text-guided image generation and editing within the proposed RPG in a
closed-loop fashion, thereby enhancing generalization ability. Extensive
experiments demonstrate our RPG outperforms state-of-the-art text-to-image
diffusion models, including DALL-E 3 and SDXL, particularly in multi-category
object composition and text-image semantic alignment. Notably, our RPG
framework exhibits wide compatibility with various MLLM architectures (e.g.,
MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available
at: https://github.com/YangLing0818/RPG-DiffusionMaster
Rambler: Supporting Writing With Speech via LLM-Assisted Gist
  Manipulation
Authors: name: Susan Lin, name: Jeremy Warner, name: J. D. Zamfirescu-Pereira, name: Matthew G. Lee, name: Sauhard Jain, name: Michael Xuelin Huang, name: Piyawat Lertvittayakumjorn, name: Shanqing Cai, name: Shumin Zhai, name: Björn Hartmann, name: Can Liu
Published: 2024-01-19 17:39:56

Dictation enables efficient text input on mobile devices. However, writing
with speech can produce disfluent, wordy, and incoherent text and thus requires
heavy post-processing. This paper presents Rambler, an LLM-powered graphical
user interface that supports gist-level manipulation of dictated text with two
main sets of functions: gist extraction and macro revision. Gist extraction
generates keywords and summaries as anchors to support the review and
interaction with spoken text. LLM-assisted macro revisions allow users to
respeak, split, merge and transform dictated text without specifying precise
editing locations. Together they pave the way for interactive dictation and
revision that help close gaps between spontaneous spoken words and
well-structured writing. In a comparative study with 12 participants performing
verbal composition tasks, Rambler outperformed the baseline of a speech-to-text
editor + ChatGPT, as it better facilitates iterative revisions with enhanced
user control over the content while supporting surprisingly diverse user
strategies.
Zero Bubble Pipeline Parallelism
Authors: name: Penghui Qi, name: Xinyi Wan, name: Guangxing Huang, name: Min Lin
Published: 2023-11-30 10:40:34

Pipeline parallelism is one of the key components for large-scale distributed
training, yet its efficiency suffers from pipeline bubbles which were deemed
inevitable. In this work, we introduce a scheduling strategy that, to our
knowledge, is the first to successfully achieve zero pipeline bubbles under
synchronous training semantics. The key idea behind this improvement is to
split the backward computation into two parts, one that computes gradient for
the input and another that computes for the parameters. Based on this idea, we
handcraft novel pipeline schedules that significantly outperform the baseline
methods. We further develop an algorithm that automatically finds an optimal
schedule based on specific model configuration and memory limit. Additionally,
to truly achieve zero bubble, we introduce a novel technique to bypass
synchronizations during the optimizer step. Experimental evaluations show that
our method outperforms the 1F1B schedule up to 23% in throughput under a
similar memory limit. This number can be further pushed to 31% when the memory
constraint is relaxed. We believe our results mark a major step forward in
harnessing the true potential of pipeline parallelism. We open sourced our
implementation based on the popular Megatron-LM repository on
https://github.com/sail-sg/zero-bubble-pipeline-parallelism.
Understanding Video Transformers via Universal Concept Discovery
Authors: name: Matthew Kowal, name: Achal Dave, name: Rares Ambrus, name: Adrien Gaidon, name: Konstantinos G. Derpanis, name: Pavel Tokmakov
Published: 2024-01-19 17:27:21

This paper studies the problem of concept-based interpretability of
transformer representations for videos. Concretely, we seek to explain the
decision-making process of video transformers based on high-level,
spatiotemporal concepts that are automatically discovered. Prior research on
concept-based interpretability has concentrated solely on image-level tasks.
Comparatively, video models deal with the added temporal dimension, increasing
complexity and posing challenges in identifying dynamic concepts over time. In
this work, we systematically address these challenges by introducing the first
Video Transformer Concept Discovery (VTCD) algorithm. To this end, we propose
an efficient approach for unsupervised identification of units of video
transformer representations - concepts, and ranking their importance to the
output of a model. The resulting concepts are highly interpretable, revealing
spatio-temporal reasoning mechanisms and object-centric representations in
unstructured video models. Performing this analysis jointly over a diverse set
of supervised and self-supervised representations, we discover that some of
these mechanism are universal in video transformers. Finally, we demonstrate
that VTCDcan be used to improve model performance for fine-grained tasks.
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
Authors: name: Lihe Yang, name: Bingyi Kang, name: Zilong Huang, name: Xiaogang Xu, name: Jiashi Feng, name: Hengshuang Zhao
Published: 2024-01-19 18:59:52

This work presents Depth Anything, a highly practical solution for robust
monocular depth estimation. Without pursuing novel technical modules, we aim to
build a simple yet powerful foundation model dealing with any images under any
circumstances. To this end, we scale up the dataset by designing a data engine
to collect and automatically annotate large-scale unlabeled data (~62M), which
significantly enlarges the data coverage and thus is able to reduce the
generalization error. We investigate two simple yet effective strategies that
make data scaling-up promising. First, a more challenging optimization target
is created by leveraging data augmentation tools. It compels the model to
actively seek extra visual knowledge and acquire robust representations.
Second, an auxiliary supervision is developed to enforce the model to inherit
rich semantic priors from pre-trained encoders. We evaluate its zero-shot
capabilities extensively, including six public datasets and randomly captured
photos. It demonstrates impressive generalization ability. Further, through
fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs
are set. Our better depth model also results in a better depth-conditioned
ControlNet. Our models are released at
https://github.com/LiheYoung/Depth-Anything.
Synthesizing Moving People with 3D Control
Authors: name: Boyi Li, name: Jathushan Rajasegaran, name: Yossi Gandelsman, name: Alexei A. Efros, name: Jitendra Malik
Published: 2024-01-19 18:59:11

In this paper, we present a diffusion model-based framework for animating
people from a single image for a given target 3D motion sequence. Our approach
has two core components: a) learning priors about invisible parts of the human
body and clothing, and b) rendering novel body poses with proper clothing and
texture. For the first part, we learn an in-filling diffusion model to
hallucinate unseen parts of a person given a single image. We train this model
on texture map space, which makes it more sample-efficient since it is
invariant to pose and viewpoint. Second, we develop a diffusion-based rendering
pipeline, which is controlled by 3D human poses. This produces realistic
renderings of novel poses of the person, including clothing, hair, and
plausible in-filling of unseen regions. This disentangled approach allows our
method to generate a sequence of images that are faithful to the target motion
in the 3D pose and, to the input image in terms of visual similarity. In
addition to that, the 3D control allows various synthetic camera trajectories
to render a person. Our experiments show that our method is resilient in
generating prolonged motions and varied challenging and complex poses compared
to prior methods. Please check our website for more details:
https://boyiliee.github.io/3DHM.github.io/.
ActAnywhere: Subject-Aware Video Background Generation
Authors: name: Boxiao Pan, name: Zhan Xu, name: Chun-Hao Paul Huang, name: Krishna Kumar Singh, name: Yang Zhou, name: Leonidas J. Guibas, name: Jimei Yang
Published: 2024-01-19 17:16:16

Generating video background that tailors to foreground subject motion is an
important problem for the movie industry and visual effects community. This
task involves synthesizing background that aligns with the motion and
appearance of the foreground subject, while also complies with the artist's
creative intention. We introduce ActAnywhere, a generative model that automates
this process which traditionally requires tedious manual efforts. Our model
leverages the power of large-scale video diffusion models, and is specifically
tailored for this task. ActAnywhere takes a sequence of foreground subject
segmentation as input and an image that describes the desired scene as
condition, to produce a coherent video with realistic foreground-background
interactions while adhering to the condition frame. We train our model on a
large-scale dataset of human-scene interaction videos. Extensive evaluations
demonstrate the superior performance of our model, significantly outperforming
baselines. Moreover, we show that ActAnywhere generalizes to diverse
out-of-distribution samples, including non-human subjects. Please visit our
project webpage at https://actanywhere.github.io.
Medusa: Simple LLM Inference Acceleration Framework with Multiple
  Decoding Heads
Authors: name: Tianle Cai, name: Yuhong Li, name: Zhengyang Geng, name: Hongwu Peng, name: Jason D. Lee, name: Deming Chen, name: Tri Dao
Published: 2024-01-19 15:48:40

The inference process in Large Language Models (LLMs) is often limited due to
the absence of parallelism in the auto-regressive decoding process, resulting
in most operations being restricted by the memory bandwidth of accelerators.
While methods such as speculative decoding have been suggested to address this
issue, their implementation is impeded by the challenges associated with
acquiring and maintaining a separate draft model. In this paper, we present
Medusa, an efficient method that augments LLM inference by adding extra
decoding heads to predict multiple subsequent tokens in parallel. Using a
tree-based attention mechanism, Medusa constructs multiple candidate
continuations and verifies them simultaneously in each decoding step. By
leveraging parallel processing, Medusa introduces only minimal overhead in
terms of single-step latency while substantially reducing the number of
decoding steps required.
  We present two levels of fine-tuning procedures for Medusa to meet the needs
of different use cases: Medusa-1: Medusa is directly fine-tuned on top of a
frozen backbone LLM, enabling lossless inference acceleration. Medusa-2: Medusa
is fine-tuned together with the backbone LLM, enabling better prediction
accuracy of Medusa heads and higher speedup but needing a special training
recipe that preserves the backbone model's capabilities.
  Moreover, we propose several extensions that improve or expand the utility of
Medusa, including a self-distillation to handle situations where no training
data is available and a typical acceptance scheme to boost the acceptance rate
while maintaining generation quality. We evaluate Medusa on models of various
sizes and training procedures. Our experiments demonstrate that Medusa-1 can
achieve over 2.2x speedup without compromising generation quality, while
Medusa-2 further improves the speedup to 2.3-3.6x.
Inflation with Diffusion: Efficient Temporal Adaptation for
  Text-to-Video Super-Resolution
Authors: name: Xin Yuan, name: Jinoo Baek, name: Keyang Xu, name: Omer Tov, name: Hongliang Fei
Published: 2024-01-18 22:25:16

We propose an efficient diffusion-based text-to-video super-resolution (SR)
tuning approach that leverages the readily learned capacity of pixel level
image diffusion model to capture spatial information for video generation. To
accomplish this goal, we design an efficient architecture by inflating the
weightings of the text-to-image SR model into our video generation framework.
Additionally, we incorporate a temporal adapter to ensure temporal coherence
across video frames. We investigate different tuning approaches based on our
inflated architecture and report trade-offs between computational costs and
super-resolution quality. Empirical evaluation, both quantitative and
qualitative, on the Shutterstock video dataset, demonstrates that our approach
is able to perform text-to-video SR generation with good visual quality and
temporal consistency. To evaluate temporal coherence, we also present
visualizations in video format in
https://drive.google.com/drive/folders/1YVc-KMSJqOrEUdQWVaI-Yfu8Vsfu_1aO?usp=sharing .
SHINOBI: Shape and Illumination using Neural Object Decomposition via
  BRDF Optimization In-the-wild
Authors: name: Andreas Engelhardt, name: Amit Raj, name: Mark Boss, name: Yunzhi Zhang, name: Abhishek Kar, name: Yuanzhen Li, name: Deqing Sun, name: Ricardo Martin Brualla, name: Jonathan T. Barron, name: Hendrik P. A. Lensch, name: Varun Jampani
Published: 2024-01-18 18:01:19

We present SHINOBI, an end-to-end framework for the reconstruction of shape,
material, and illumination from object images captured with varying lighting,
pose, and background. Inverse rendering of an object based on unconstrained
image collections is a long-standing challenge in computer vision and graphics
and requires a joint optimization over shape, radiance, and pose. We show that
an implicit shape representation based on a multi-resolution hash encoding
enables faster and robust shape reconstruction with joint camera alignment
optimization that outperforms prior work. Further, to enable the editing of
illumination and object reflectance (i.e. material) we jointly optimize BRDF
and illumination together with the object's shape. Our method is class-agnostic
and works on in-the-wild image collections of objects to produce relightable 3D
assets for several use cases such as AR/VR, movies, games, etc. Project page:
https://shinobi.aengelhardt.com Video:
https://www.youtube.com/watch?v=iFENQ6AcYd8&feature=youtu.be
VMamba: Visual State Space Model
Authors: name: Yue Liu, name: Yunjie Tian, name: Yuzhong Zhao, name: Hongtian Yu, name: Lingxi Xie, name: Yaowei Wang, name: Qixiang Ye, name: Yunfan Liu
Published: 2024-01-18 17:55:39

Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as
the two most popular foundation models for visual representation learning.
While CNNs exhibit remarkable scalability with linear complexity w.r.t. image
resolution, ViTs surpass them in fitting capabilities despite contending with
quadratic complexity. A closer inspection reveals that ViTs achieve superior
visual modeling performance through the incorporation of global receptive
fields and dynamic weights. This observation motivates us to propose a novel
architecture that inherits these components while enhancing computational
efficiency. To this end, we draw inspiration from the recently introduced state
space model and propose the Visual State Space Model (VMamba), which achieves
linear complexity without sacrificing global receptive fields. To address the
encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM)
to traverse the spatial domain and convert any non-causal visual image into
order patch sequences. Extensive experimental results substantiate that VMamba
not only demonstrates promising capabilities across various visual perception
tasks, but also exhibits more pronounced advantages over established benchmarks
as the image resolution increases. Source code has been available at
https://github.com/MzeroMiko/VMamba.
FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder
Authors: name: Tan Dat Nguyen, name: Ji-Hoon Kim, name: Youngjoon Jang, name: Jaehun Kim, name: Joon Son Chung
Published: 2024-01-18 14:57:25

The goal of this paper is to generate realistic audio with a lightweight and
fast diffusion-based vocoder, named FreGrad. Our framework consists of the
following three key components: (1) We employ discrete wavelet transform that
decomposes a complicated waveform into sub-band wavelets, which helps FreGrad
to operate on a simple and concise feature space, (2) We design a
frequency-aware dilated convolution that elevates frequency awareness,
resulting in generating speech with accurate frequency information, and (3) We
introduce a bag of tricks that boosts the generation quality of the proposed
model. In our experiments, FreGrad achieves 3.7 times faster training time and
2.2 times faster inference speed compared to our baseline while reducing the
model size by 0.6 times (only 1.78M parameters) without sacrificing the output
quality. Audio samples are available at:
https://mm.kaist.ac.kr/projects/FreGrad.
CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
Authors: name: Zhao Wang, name: Aoxue Li, name: Enze Xie, name: Lingting Zhu, name: Yong Guo, name: Qi Dou, name: Zhenguo Li
Published: 2024-01-18 13:23:51

Customized text-to-video generation aims to generate high-quality videos
guided by text prompts and subject references. Current approaches designed for
single subjects suffer from tackling multiple subjects, which is a more
challenging and practical scenario. In this work, we aim to promote
multi-subject guided text-to-video customization. We propose CustomVideo, a
novel framework that can generate identity-preserving videos with the guidance
of multiple subjects. To be specific, firstly, we encourage the co-occurrence
of multiple subjects via composing them in a single image. Further, upon a
basic text-to-video diffusion model, we design a simple yet effective attention
control strategy to disentangle different subjects in the latent space of
diffusion model. Moreover, to help the model focus on the specific object area,
we segment the object from given reference images and provide a corresponding
object mask for attention learning. Also, we collect a multi-subject
text-to-video generation dataset as a comprehensive benchmark, with 69
individual subjects and 57 meaningful pairs. Extensive qualitative,
quantitative, and user study results demonstrate the superiority of our method,
compared with the previous state-of-the-art approaches.
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Authors: name: Sadeep Jayasumana, name: Srikumar Ramalingam, name: Andreas Veit, name: Daniel Glasner, name: Ayan Chakrabarti, name: Sanjiv Kumar
Published: 2023-11-30 19:11:01

As with many machine learning problems, the progress of image generation
methods hinges on good evaluation metrics. One of the most popular is the
Frechet Inception Distance (FID). FID estimates the distance between a
distribution of Inception-v3 features of real images, and those of images
generated by the algorithm. We highlight important drawbacks of FID:
Inception's poor representation of the rich and varied content generated by
modern text-to-image models, incorrect normality assumptions, and poor sample
complexity. We call for a reevaluation of FID's use as the primary quality
metric for generated images. We empirically demonstrate that FID contradicts
human raters, it does not reflect gradual improvement of iterative
text-to-image models, it does not capture distortion levels, and that it
produces inconsistent results when varying the sample size. We also propose an
alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
estimator that does not make any assumptions on the probability distribution of
the embeddings and is sample efficient. Through extensive experiments and
analysis, we demonstrate that FID-based evaluations of text-to-image models may
be unreliable, and that CMMD offers a more robust and reliable assessment of
image quality.
DiffusionGPT: LLM-Driven Text-to-Image Generation System
Authors: name: Jie Qin, name: Jie Wu, name: Weifeng Chen, name: Yuxi Ren, name: Huixia Li, name: Hefeng Wu, name: Xuefeng Xiao, name: Rui Wang, name: Shilei Wen
Published: 2024-01-18 15:30:58

Diffusion models have opened up new avenues for the field of image
generation, resulting in the proliferation of high-quality models shared on
open-source platforms. However, a major challenge persists in current
text-to-image systems are often unable to handle diverse inputs, or are limited
to single model results. Current unified attempts often fall into two
orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
expert model to output. To combine the best of both worlds, we propose
DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
generation system capable of seamlessly accommodating various types of prompts
and integrating domain-expert models. DiffusionGPT constructs domain-specific
Trees for various generative models based on prior knowledge. When provided
with an input, the LLM parses the prompt and employs the Trees-of-Thought to
guide the selection of an appropriate model, thereby relaxing input constraints
and ensuring exceptional performance across diverse domains. Moreover, we
introduce Advantage Databases, where the Tree-of-Thought is enriched with human
feedback, aligning the model selection process with human preferences. Through
extensive experiments and comparisons, we demonstrate the effectiveness of
DiffusionGPT, showcasing its potential for pushing the boundaries of image
synthesis in diverse domains.