Orionnnn/hub-stats huggingface dataset:
Multimodal Data and Resource Efficient Device-Directed Speech Detection
  with Large Foundation Models
Authors: name: Dominik Wagner, name: Alexander Churchill, name: Siddharth Sigtia, name: Panayiotis Georgiou, name: Matt Mirsamadi, name: Aarshee Mishra, name: Erik Marchi
Published: 2023-12-06 17:29:03

Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.
Language-Informed Visual Concept Learning
Authors: name: Sharon Lee, name: Yunzhi Zhang, name: Shangzhe Wu, name: Jiajun Wu
Published: 2023-12-06 16:24:47

Our understanding of the visual world is centered around various concept
axes, characterizing different aspects of visual entities. While different
concept axes can be easily specified by language, e.g. color, the exact visual
nuances along each axis often exceed the limitations of linguistic
articulations, e.g. a particular style of painting. In this work, our goal is
to learn a language-informed visual concept representation, by simply
distilling large pre-trained vision-language models. Specifically, we train a
set of concept encoders to encode the information pertinent to a set of
language-informed concept axes, with an objective of reproducing the input
image through a pre-trained Text-to-Image (T2I) model. To encourage better
disentanglement of different concept encoders, we anchor the concept embeddings
to a set of text embeddings obtained from a pre-trained Visual Question
Answering (VQA) model. At inference time, the model extracts concept embeddings
along various axes from new test images, which can be remixed to generate
images with novel compositions of visual concepts. With a lightweight test-time
finetuning procedure, it can also generalize to novel concepts unseen at
training.
Analyzing and Improving the Training Dynamics of Diffusion Models
Authors: name: Tero Karras, name: Miika Aittala, name: Jaakko Lehtinen, name: Janne Hellsten, name: Timo Aila, name: Samuli Laine
Published: 2023-12-05 11:55:47

Diffusion models currently dominate the field of data-driven image synthesis
with their unparalleled scaling to large datasets. In this paper, we identify
and rectify several causes for uneven and ineffective training in the popular
ADM diffusion model architecture, without altering its high-level structure.
Observing uncontrolled magnitude changes and imbalances in both the network
activations and weights over the course of training, we redesign the network
layers to preserve activation, weight, and update magnitudes on expectation. We
find that systematic application of this philosophy eliminates the observed
drifts and imbalances, resulting in considerably better networks at equal
computational complexity. Our modifications improve the previous record FID of
2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
sampling.
  As an independent contribution, we present a method for setting the
exponential moving average (EMA) parameters post-hoc, i.e., after completing
the training run. This allows precise tuning of EMA length without the cost of
performing several training runs, and reveals its surprising interactions with
network architecture, training time, and guidance.
Orthogonal Adaptation for Modular Customization of Diffusion Models
Authors: name: Ryan Po, name: Guandao Yang, name: Kfir Aberman, name: Gordon Wetzstein
Published: 2023-12-05 02:17:48

Customization techniques for text-to-image models have paved the way for a
wide range of previously unattainable applications, enabling the generation of
specific concepts across diverse contexts and styles. While existing methods
facilitate high-fidelity customization for individual concepts or a limited,
pre-defined set of them, they fall short of achieving scalability, where a
single model can seamlessly render countless concepts. In this paper, we
address a new problem called Modular Customization, with the goal of
efficiently merging customized models that were fine-tuned independently for
individual concepts. This allows the merged model to jointly synthesize
concepts in one image without compromising fidelity or incurring any additional
computational costs.
  To address this problem, we introduce Orthogonal Adaptation, a method
designed to encourage the customized models, which do not have access to each
other during fine-tuning, to have orthogonal residual weights. This ensures
that during inference time, the customized models can be summed with minimal
interference.
  Our proposed method is both simple and versatile, applicable to nearly all
optimizable weights in the model architecture. Through an extensive set of
quantitative and qualitative evaluations, our method consistently outperforms
relevant baselines in terms of efficiency and identity preservation,
demonstrating a significant leap toward scalable customization of diffusion
models.
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded
  Diffusion Model
Authors: name: Lingmin Ran, name: Xiaodong Cun, name: JiaWei Liu, name: Rui Zhao, name: Song Zijie, name: Xintao Wang, name: Jussi Keppo, name: Mike Zheng Shou
Published: 2023-12-04 09:19:38

We introduce X-Adapter, a universal upgrader to enable the pretrained
plug-and-play modules (e.g., ControlNet, LoRA) to work directly with the
upgraded text-to-image diffusion model (e.g., SDXL) without further retraining.
We achieve this goal by training an additional network to control the frozen
upgraded model with the new text-image data pairs. In detail, X-Adapter keeps a
frozen copy of the old model to preserve the connectors of different plugins.
Additionally, X-Adapter adds trainable mapping layers that bridge the decoders
from models of different versions for feature remapping. The remapped features
will be used as guidance for the upgraded model. To enhance the guidance
ability of X-Adapter, we employ a null-text training strategy for the upgraded
model. After training, we also introduce a two-stage denoising strategy to
align the initial latents of X-Adapter and the upgraded model. Thanks to our
strategies, X-Adapter demonstrates universal compatibility with various plugins
and also enables plugins of different versions to work together, thereby
expanding the functionalities of diffusion community. To verify the
effectiveness of the proposed method, we conduct extensive experiments and the
results show that X-Adapter may facilitate wider application in the upgraded
foundational diffusion model.
DragVideo: Interactive Drag-style Video Editing
Authors: name: Yufan Deng, name: Ruida Wang, name: Yuhao Zhang, name: Yu-Wing Tai, name: Chi-Keung Tang
Published: 2023-12-03 10:41:06

Editing visual content on videos remains a formidable challenge with two main
issues: 1) direct and easy user control to produce 2) natural editing results
without unsightly distortion and artifacts after changing shape, expression and
layout. Inspired by DragGAN, a recent image-based drag-style editing technique,
we address above issues by proposing DragVideo, where a similar drag-style user
interaction is adopted to edit video content while maintaining temporal
consistency. Empowered by recent diffusion models as in DragDiffusion,
DragVideo contains the novel Drag-on-Video U-Net (DoVe) editing method, which
optimizes diffused video latents generated by video U-Net to achieve the
desired control. Specifically, we use Sample-specific LoRA fine-tuning and
Mutual Self-Attention control to ensure faithful reconstruction of video from
the DoVe method. We also present a series of testing examples for drag-style
video editing and conduct extensive experiments across a wide array of
challenging editing tasks, such as motion editing, skeleton editing, etc,
underscoring DragVideo's versatility and generality. Our codes including the
DragVideo web user interface will be released.
Generating Fine-Grained Human Motions Using ChatGPT-Refined Descriptions
Authors: name: Xu Shi, name: Chuanchen Luo, name: Junran Peng, name: Hongwen Zhang, name: Yunlian Sun
Published: 2023-12-05 14:01:43

Recently, significant progress has been made in text-based motion generation,
enabling the generation of diverse and high-quality human motions that conform
to textual descriptions. However, it remains challenging to generate
fine-grained or stylized motions due to the lack of datasets annotated with
detailed textual descriptions. By adopting a divide-and-conquer strategy, we
propose a new framework named Fine-Grained Human Motion Diffusion Model
(FG-MDM) for human motion generation. Specifically, we first parse previous
vague textual annotation into fine-grained description of different body parts
by leveraging a large language model (GPT-3.5). We then use these fine-grained
descriptions to guide a transformer-based diffusion model. FG-MDM can generate
fine-grained and stylized motions even outside of the distribution of the
training data. Our experimental results demonstrate the superiority of FG-MDM
over previous methods, especially the strong generalization capability. We will
release our fine-grained textual annotations for HumanML3D and KIT.
ReconFusion: 3D Reconstruction with Diffusion Priors
Authors: name: Rundi Wu, name: Ben Mildenhall, name: Philipp Henzler, name: Keunhong Park, name: Ruiqi Gao, name: Daniel Watson, name: Pratul P. Srinivasan, name: Dor Verbin, name: Jonathan T. Barron, name: Ben Poole, name: Aleksander Holynski
Published: 2023-12-05 18:59:58

3D reconstruction methods such as Neural Radiance Fields (NeRFs) excel at
rendering photorealistic novel views of complex scenes. However, recovering a
high-quality NeRF typically requires tens to hundreds of input images,
resulting in a time-consuming capture process. We present ReconFusion to
reconstruct real-world scenes using only a few photos. Our approach leverages a
diffusion prior for novel view synthesis, trained on synthetic and multiview
datasets, which regularizes a NeRF-based 3D reconstruction pipeline at novel
camera poses beyond those captured by the set of input images. Our method
synthesizes realistic geometry and texture in underconstrained regions while
preserving the appearance of observed regions. We perform an extensive
evaluation across various real-world datasets, including forward-facing and
360-degree scenes, demonstrating significant performance improvements over
previous few-view NeRF reconstruction approaches.
Alchemist: Parametric Control of Material Properties with Diffusion
  Models
Authors: name: Prafull Sharma, name: Varun Jampani, name: Yuanzhen Li, name: Xuhui Jia, name: Dmitry Lagun, name: Fredo Durand, name: William T. Freeman, name: Mark Matthews
Published: 2023-12-05 18:58:26

We propose a method to control material attributes of objects like roughness,
metallic, albedo, and transparency in real images. Our method capitalizes on
the generative prior of text-to-image models known for photorealism, employing
a scalar value and instructions to alter low-level material properties.
Addressing the lack of datasets with controlled material attributes, we
generated an object-centric synthetic dataset with physically-based materials.
Fine-tuning a modified pre-trained text-to-image model on this synthetic
dataset enables us to edit material properties in real-world images while
preserving all other attributes. We show the potential application of our model
to material edited NeRFs.
MVHumanNet: A Large-scale Dataset of Multi-view Daily Dressing Human
  Captures
Authors: name: Zhangyang Xiong, name: Chenghong Li, name: Kenkun Liu, name: Hongjie Liao, name: Jianqiao Hu, name: Junyi Zhu, name: Shuliang Ning, name: Lingteng Qiu, name: Chongjie Wang, name: Shijie Wang, name: Shuguang Cui, name: Xiaoguang Han
Published: 2023-12-05 18:50:12

In this era, the success of large language models and text-to-image models
can be attributed to the driving force of large-scale datasets. However, in the
realm of 3D vision, while remarkable progress has been made with models trained
on large-scale synthetic and real-captured object data like Objaverse and
MVImgNet, a similar level of progress has not been observed in the domain of
human-centric tasks partially due to the lack of a large-scale human dataset.
Existing datasets of high-fidelity 3D human capture continue to be mid-sized
due to the significant challenges in acquiring large-scale high-quality 3D
human data. To bridge this gap, we present MVHumanNet, a dataset that comprises
multi-view human action sequences of 4,500 human identities. The primary focus
of our work is on collecting human data that features a large number of diverse
identities and everyday clothing using a multi-view human capture system, which
facilitates easily scalable data collection. Our dataset contains 9,000 daily
outfits, 60,000 motion sequences and 645 million frames with extensive
annotations, including human masks, camera parameters, 2D and 3D keypoints,
SMPL/SMPLX parameters, and corresponding textual descriptions. To explore the
potential of MVHumanNet in various 2D and 3D visual tasks, we conducted pilot
studies on view-consistent action recognition, human NeRF reconstruction,
text-driven view-unconstrained human image generation, as well as 2D
view-unconstrained human image and 3D avatar generation. Extensive experiments
demonstrate the performance improvements and effective applications enabled by
the scale provided by MVHumanNet. As the current largest-scale 3D human
dataset, we hope that the release of MVHumanNet data with annotations will
foster further innovations in the domain of 3D human-centric tasks at scale.
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation
Authors: name: Peng Wang, name: Yichun Shi
Published: 2023-12-02 20:41:27

We introduce "ImageDream," an innovative image-prompt, multi-view diffusion
model for 3D object generation. ImageDream stands out for its ability to
produce 3D models of higher quality compared to existing state-of-the-art,
image-conditioned methods. Our approach utilizes a canonical camera
coordination for the objects in images, improving visual geometry accuracy. The
model is designed with various levels of control at each block inside the
diffusion model based on the input image, where global control shapes the
overall object layout and local control fine-tunes the image details. The
effectiveness of ImageDream is demonstrated through extensive evaluations using
a standard prompt list. For more information, visit our project page at
https://Image-Dream.github.io.
StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D
Authors: name: Pengsheng Guo, name: Hans Hao, name: Adam Caccavale, name: Zhongzheng Ren, name: Edward Zhang, name: Qi Shan, name: Aditya Sankar, name: Alexander G. Schwing, name: Alex Colburn, name: Fangchang Ma
Published: 2023-12-02 02:27:58

In the realm of text-to-3D generation, utilizing 2D diffusion models through
score distillation sampling (SDS) frequently leads to issues such as blurred
appearances and multi-faced geometry, primarily due to the intrinsically noisy
nature of the SDS loss. Our analysis identifies the core of these challenges as
the interaction among noise levels in the 2D diffusion process, the
architecture of the diffusion network, and the 3D model representation. To
overcome these limitations, we present StableDreamer, a methodology
incorporating three advances. First, inspired by InstructNeRF2NeRF, we
formalize the equivalence of the SDS generative prior and a simple supervised
L2 reconstruction loss. This finding provides a novel tool to debug SDS, which
we use to show the impact of time-annealing noise levels on reducing
multi-faced geometries. Second, our analysis shows that while image-space
diffusion contributes to geometric precision, latent-space diffusion is crucial
for vivid color rendition. Based on this observation, StableDreamer introduces
a two-stage training strategy that effectively combines these aspects,
resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D
Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance
the overall quality, reduce memory usage during training, and accelerate
rendering speeds, and better capture semi-transparent objects. StableDreamer
reduces multi-face geometries, generates fine details, and converges stably.
LivePhoto: Real Image Animation with Text-guided Motion Control
Authors: name: Xi Chen, name: Zhiheng Liu, name: Mengting Chen, name: Yutong Feng, name: Yu Liu, name: Yujun Shen, name: Hengshuang Zhao
Published: 2023-12-05 17:59:52

Despite the recent progress in text-to-video generation, existing studies
usually overlook the issue that only spatial contents but not temporal motions
in synthesized videos are under the control of text. Towards such a challenge,
this work presents a practical system, named LivePhoto, which allows users to
animate an image of their interest with text descriptions. We first establish a
strong baseline that helps a well-learned text-to-image generator (i.e., Stable
Diffusion) take an image as a further input. We then equip the improved
generator with a motion module for temporal modeling and propose a carefully
designed training pipeline to better link texts and motions. In particular,
considering the facts that (1) text can only describe motions roughly (e.g.,
regardless of the moving speed) and (2) text may include both content and
motion descriptions, we introduce a motion intensity estimation module as well
as a text re-weighting module to reduce the ambiguity of text-to-motion
mapping. Empirical evidence suggests that our approach is capable of well
decoding motion-related textual instructions into videos, such as actions,
camera movements, or even conjuring new contents from thin air (e.g., pouring
water into an empty glass). Interestingly, thanks to the proposed intensity
learning mechanism, our system offers users an additional control signal (i.e.,
the motion intensity) besides text for video customization.
Fine-grained Controllable Video Generation via Object Appearance and
  Context
Authors: name: Hsin-Ping Huang, name: Yu-Chuan Su, name: Deqing Sun, name: Lu Jiang, name: Xuhui Jia, name: Yukun Zhu, name: Ming-Hsuan Yang
Published: 2023-12-05 17:47:33

Text-to-video generation has shown promising results. However, by taking only
natural languages as input, users often face difficulties in providing detailed
information to precisely control the model's output. In this work, we propose
fine-grained controllable video generation (FACTOR) to achieve detailed
control. Specifically, FACTOR aims to control objects' appearances and context,
including their location and category, in conjunction with the text prompt. To
achieve detailed control, we propose a unified framework to jointly inject
control signals into the existing text-to-video model. Our model consists of a
joint encoder and adaptive cross-attention layers. By optimizing the encoder
and the inserted layer, we adapt the model to generate videos that are aligned
with both text prompts and fine-grained control. Compared to existing methods
relying on dense control signals such as edge maps, we provide a more intuitive
and user-friendly interface to allow object-level fine-grained control. Our
method achieves controllability of object appearances without finetuning, which
reduces the per-subject optimization efforts for the users. Extensive
experiments on standard benchmark datasets and user-provided inputs validate
that our model obtains a 70% improvement in controllability metrics over
competitive baselines.
FaceStudio: Put Your Face Everywhere in Seconds
Authors: name: Yuxuan Yan, name: Chi Zhang, name: Rui Wang, name: Pei Cheng, name: Gang Yu, name: Bin Fu
Published: 2023-12-05 11:02:45

This study investigates identity-preserving image synthesis, an intriguing
task in image generation that seeks to maintain a subject's identity while
adding a personalized, stylistic touch. Traditional methods, such as Textual
Inversion and DreamBooth, have made strides in custom image creation, but they
come with significant drawbacks. These include the need for extensive resources
and time for fine-tuning, as well as the requirement for multiple reference
images. To overcome these challenges, our research introduces a novel approach
to identity-preserving synthesis, with a particular focus on human images. Our
model leverages a direct feed-forward mechanism, circumventing the need for
intensive fine-tuning, thereby facilitating quick and efficient image
generation. Central to our innovation is a hybrid guidance framework, which
combines stylized images, facial images, and textual prompts to guide the image
generation process. This unique combination enables our model to produce a
variety of applications, such as artistic portraits and identity-blended
images. Our experimental results, including both qualitative and quantitative
evaluations, demonstrate the superiority of our method over existing baseline
models and previous works, particularly in its remarkable efficiency and
ability to preserve the subject's identity with high fidelity.
GPT4Point: A Unified Framework for Point-Language Understanding and
  Generation
Authors: name: Zhangyang Qi, name: Ye Fang, name: Zeyi Sun, name: Xiaoyang Wu, name: Tong Wu, name: Jiaqi Wang, name: Dahua Lin, name: Hengshuang Zhao
Published: 2023-12-05 18:59:55

Multimodal Large Language Models (MLLMs) have excelled in 2D image-text
comprehension and image generation, but their understanding of the 3D world is
notably deficient, limiting progress in 3D language understanding and
generation. To solve this problem, we introduce GPT4Point, an innovative
groundbreaking point-language multimodal model designed specifically for
unified 3D object understanding and generation within the MLLM framework.
GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text
reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point
is equipped with advanced capabilities for controllable 3D generation, it can
get high-quality results through a low-quality point-text feature maintaining
the geometric shapes and colors. To support the expansive needs of 3D
object-text pairs, we develop Pyramid-XL, a point-language dataset annotation
engine. It constructs a large-scale database over 1M objects of varied text
granularity levels from the Objaverse-XL dataset, essential for training
GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D
point-language understanding capabilities. In extensive evaluations, GPT4Point
has demonstrated superior performance in understanding and generation.