Orionnnn/hub-stats huggingface dataset:
Inserting Anybody in Diffusion Models via Celeb Basis
Authors: name: Ge Yuan, name: Xiaodong Cun, name: Yong Zhang, name: Maomao Li, name: Chenyang Qi, name: Xintao Wang, name: Ying Shan, name: Huicheng Zheng
Published: 2023-06-01 17:30:24

Exquisite demand exists for customizing the pretrained large text-to-image
model, e.g., Stable Diffusion, to generate innovative concepts, such
as the users themselves. However, the newly-added concept from previous
customization methods often shows weaker combination abilities than the
original ones even given several images during training. We thus propose a new
personalization method that allows for the seamless integration of a unique
individual into the pre-trained diffusion model using just one facial
photograph and only 1024 learnable parameters under 3
minutes. So as we can effortlessly generate stunning images of this person in
any pose or position, interacting with anyone and doing anything imaginable
from text prompts. To achieve this, we first analyze and build a well-defined
celeb basis from the embedding space of the pre-trained large text encoder.
Then, given one facial photo as the target identity, we generate its own
embedding by optimizing the weight of this basis and locking all other
parameters. Empowered by the proposed celeb basis, the new identity in our
customized model showcases a better concept combination ability than previous
personalization methods. Besides, our model can also learn several new
identities at once and interact with each other where the previous
customization model fails to. The code will be released.
Wuerstchen: Efficient Pretraining of Text-to-Image Models
Authors: name: Pablo Pernias, name: Dominic Rampas, name: Marc Aubreville
Published: 2023-06-01 13:00:53

We introduce Wuerstchen, a novel technique for text-to-image synthesis that
unites competitive performance with unprecedented cost-effectiveness and ease
of training on constrained hardware. Building on recent advancements in machine
learning, our approach, which utilizes latent diffusion strategies at strong
latent image compression rates, significantly reduces the computational burden,
typically associated with state-of-the-art models, while preserving, if not
enhancing, the quality of generated images. Wuerstchen achieves notable speed
improvements at inference time, thereby rendering real-time applications more
viable. One of the key advantages of our method lies in its modest training
requirements of only 9,200 GPU hours, slashing the usual costs significantly
without compromising the end performance. In a comparison against the
state-of-the-art, we found the approach to yield strong competitiveness. This
paper opens the door to a new line of research that prioritizes both
performance and computational accessibility, hence democratizing the use of
sophisticated AI technologies. Through Wuerstchen, we demonstrate a compelling
stride forward in the realm of text-to-image synthesis, offering an innovative
path to explore in future research.
SafeDiffuser: Safe Planning with Diffusion Probabilistic Models
Authors: name: Wei Xiao, name: Tsun-Hsuan Wang, name: Chuang Gan, name: Daniela Rus
Published: 2023-05-31 19:38:12

Diffusion model-based approaches have shown promise in data-driven planning,
but there are no safety guarantees, thus making it hard to be applied for
safety-critical applications. To address these challenges, we propose a new
method, called SafeDiffuser, to ensure diffusion probabilistic models satisfy
specifications by using a class of control barrier functions. The key idea of
our approach is to embed the proposed finite-time diffusion invariance into the
denoising diffusion procedure, which enables trustworthy diffusion data
generation. Moreover, we demonstrate that our finite-time diffusion invariance
method through generative models not only maintains generalization performance
but also creates robustness in safe data generation. We test our method on a
series of safe planning tasks, including maze path generation, legged robot
locomotion, and 3D space manipulation, with results showing the advantages of
robustness and guarantees over vanilla diffusion models.
Bytes Are All You Need: Transformers Operating Directly On File Bytes
Authors: name: Maxwell Horton, name: Sachin Mehta, name: Ali Farhadi, name: Mohammad Rastegari
Published: 2023-05-31 23:18:21

Modern deep learning approaches usually transform inputs into a
modality-specific form. For example, the most common deep learning approach to
image classification involves decoding image file bytes into an RGB tensor
which is passed into a neural network. Instead, we investigate performing
classification directly on file bytes, without the need for decoding files at
inference time. Using file bytes as model inputs enables the development of
models which can operate on multiple input modalities. Our model,
ByteFormer, achieves an ImageNet Top-1 classification accuracy of
77.33% when training and testing directly on TIFF file bytes using a
transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
when operating on RGB images). Without modifications or hyperparameter tuning,
ByteFormer achieves 95.42% classification accuracy when operating on WAV
files from the Speech Commands v2 dataset (compared to state-of-the-art
accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
applications in privacy-preserving inference. ByteFormer is capable of
performing inference on particular obfuscated input representations with no
loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
with a hypothetical privacy-preserving camera which avoids forming full images
by consistently masking 90% of pixel channels, while still achieving
71.35% accuracy on ImageNet. Our code will be made available at
https://github.com/apple/ml-cvnets/tree/main/examples/byteformer.
Brainformers: Trading Simplicity for Efficiency
Authors: name: Yanqi Zhou, name: Nan Du, name: Yanping Huang, name: Daiyi Peng, name: Chang Lan, name: Da Huang, name: Siamak Shakeri, name: David So, name: Andrew Dai, name: Yifeng Lu, name: Zhifeng Chen, name: Quoc Le, name: Claire Cui, name: James Laundon, name: Jeff Dean
Published: 2023-05-29 18:42:01

Transformers are central to recent successes in natural language processing
and computer vision. Transformers have a mostly uniform backbone where layers
alternate between feed-forward and self-attention in order to build a deep
network. Here we investigate this design choice and find that more complex
blocks that have different permutations of layer primitives can be more
efficient. Using this insight, we develop a complex block, named Brainformer,
that consists of a diverse sets of layers such as sparsely gated feed-forward
layers, dense feed-forward layers, attention layers, and various forms of layer
normalization and activation functions. Brainformer consistently outperforms
the state-of-the-art dense and sparse Transformers, in terms of both quality
and efficiency. A Brainformer model with 8 billion activated parameters per
token demonstrates 2x faster training convergence and 5x faster step time
compared to its GLaM counterpart. In downstream task evaluation, Brainformer
also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM
with a similar number of activated parameters. Finally, Brainformer largely
outperforms a Primer dense model derived with NAS with similar computation per
token on fewshot evaluations.
Birth of a Transformer: A Memory Viewpoint
Authors: name: Alberto Bietti, name: Vivien Cabannes, name: Diane Bouchacourt, name: Herve Jegou, name: Leon Bottou
Published: 2023-06-01 15:30:33

Large language models based on transformers have achieved great empirical
successes. However, as they are deployed more widely, there is a growing need
to better understand their internal mechanisms in order to make them more
reliable. These models appear to store vast amounts of knowledge from their
training data, and to adapt quickly to new information provided in their
context or prompt. We study how transformers balance these two types of
knowledge by considering a synthetic setup where tokens are generated from
either global or context-specific bigram distributions. By a careful empirical
analysis of the training process on a simplified two-layer transformer, we
illustrate the fast learning of global bigrams and the slower development of an
"induction head" mechanism for the in-context bigrams. We highlight the role of
weight matrices as associative memories, provide theoretical insights on how
gradients enable their learning during training, and study the role of
data-distributional properties.
The Hidden Language of Diffusion Models
Authors: name: Hila Chefer, name: Oran Lang, name: Mor Geva, name: Volodymyr Polosukhin, name: Assaf Shocher, name: Michal Irani, name: Inbar Mosseri, name: Lior Wolf
Published: 2023-06-01 17:57:08

Text-to-image diffusion models have demonstrated an unparalleled ability to
generate high-quality, diverse images from a textual concept (e.g., "a doctor",
"love"). However, the internal process of mapping text to a rich visual
representation remains an enigma. In this work, we tackle the challenge of
understanding concept representations in text-to-image models by decomposing an
input text prompt into a small set of interpretable elements. This is achieved
by learning a pseudo-token that is a sparse weighted combination of tokens from
the model's vocabulary, with the objective of reconstructing the images
generated for the given concept. Applied over the state-of-the-art Stable
Diffusion model, this decomposition reveals non-trivial and surprising
structures in the representations of concepts. For example, we find that some
concepts such as "a president" or "a composer" are dominated by specific
instances (e.g., "Obama", "Biden") and their interpolations. Other concepts,
such as "happiness" combine associated terms that can be concrete ("family",
"laughter") or abstract ("friendship", "emotion"). In addition to peering into
the inner workings of Stable Diffusion, our method also enables applications
such as single-image decomposition to tokens, bias detection and mitigation,
and semantic image manipulation. Our code will be available at:
https://hila-chefer.github.io/Conceptor/
SQL-PaLM: Improved Large Language ModelAdaptation for Text-to-SQL
Authors: name: Ruoxi Sun, name: Sercan O Arik, name: Hootan Nakhost, name: Hanjun Dai, name: Rajarishi Sinha, name: Pengcheng Yin, name: Tomas Pfister
Published: 2023-05-26 21:39:05

One impressive emergent capability of large language models (LLMs) is
generation of code, including Structured Query Language (SQL) for databases.
For the task of converting natural language text to SQL queries, Text-to-SQL,
adaptation of LLMs is of paramount importance, both in in-context learning and
fine-tuning settings, depending on the amount of adaptation data used. In this
paper, we propose an LLM-based Text-to-SQL model SQL-PaLM, leveraging on
PaLM-2, that pushes the state-of-the-art in both settings. Few-shot SQL-PaLM is
based on an execution-based self-consistency prompting approach designed for
Text-to-SQL, and achieves 77.3% in test-suite accuracy on Spider, which to our
best knowledge is the first to outperform previous state-of-the-art with
fine-tuning by a significant margin, 4%. Furthermore, we demonstrate that the
fine-tuned SQL-PALM outperforms it further by another 1%. Towards applying
SQL-PaLM to real-world scenarios we further evaluate its robustness on other
challenging variants of Spider and demonstrate the superior generalization
capability of SQL-PaLM. In addition, via extensive case studies, we demonstrate
the impressive intelligent capabilities and various success enablers of
LLM-based Text-to-SQL.
MERT: Acoustic Music Understanding Model with Large-Scale
  Self-supervised Training
Authors: name: Yizhi Li, name: Ruibin Yuan, name: Ge Zhang, name: Yinghao Ma, name: Xingran Chen, name: Hanzhi Yin, name: Chenghua Lin, name: Anton Ragni, name: Emmanouil Benetos, name: Norbert Gyenge, name: Roger Dannenberg, name: Ruibo Liu, name: Wenhu Chen, name: Gus Xia, name: Yemin Shi, name: Wenhao Huang, name: Yike Guo, name: Jie Fu
Published: 2023-05-31 18:27:43

Self-supervised learning (SSL) has recently emerged as a promising paradigm
for training generalisable models on large-scale data in the fields of vision,
text, and speech. Although SSL has been proven effective in speech and audio,
its application to music audio has yet to be thoroughly explored. This is
primarily due to the distinctive challenges associated with modelling musical
knowledge, particularly its tonal and pitched characteristics of music. To
address this research gap, we propose an acoustic Music undERstanding model
with large-scale self-supervised Training (MERT), which incorporates teacher
models to provide pseudo labels in the masked language modelling (MLM) style
acoustic pre-training. In our exploration, we identified a superior combination
of teacher models, which outperforms conventional speech and audio approaches
in terms of performance. This combination includes an acoustic teacher based on
Residual Vector Quantization - Variational AutoEncoder (RVQ-VAE) and a musical
teacher based on the Constant-Q Transform (CQT). These teachers effectively
guide our student model, a BERT-style transformer encoder, to better model
music audio. In addition, we introduce an in-batch noise mixture augmentation
to enhance the representation robustness. Furthermore, we explore a wide range
of settings to overcome the instability in acoustic language model
pre-training, which allows our designed paradigm to scale from 95M to 330M
parameters. Experimental results indicate that our model can generalise and
perform well on 14 music understanding tasks and attains state-of-the-art
(SOTA) overall scores. The code and models are online:
https://github.com/yizhilll/MERT.
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM
Authors: name: Nghi D. Q. Bui, name: Hung Le, name: Yue Wang, name: Junnan Li, name: Akhilesh Deepak Gotmare, name: Steven C. H. Hoi
Published: 2023-05-31 05:24:48

Code intelligence plays a key role in transforming modern software
engineering. Recently, deep learning-based models, especially Transformer-based
large language models (LLMs), have demonstrated remarkable potential in
tackling these tasks by leveraging massive open-source code data and
programming language features. However, the development and deployment of such
models often require expertise in both machine learning and software
engineering, creating a barrier for the model adoption. In this paper, we
present CodeTF, an open-source Transformer-based library for state-of-the-art
Code LLMs and code intelligence. Following the principles of modular design and
extensible framework, we design CodeTF with a unified interface to enable rapid
access and development across different types of models, datasets and tasks.
Our library supports a collection of pretrained Code LLM models and popular
code benchmarks, including a standardized interface to train and serve code
LLMs efficiently, and data features such as language-specific parsers and
utility functions for extracting code attributes. In this paper, we describe
the design principles, the architecture, key modules and components, and
compare with other related library tools. Finally, we hope CodeTF is able to
bridge the gap between machine learning/generative AI and software engineering,
providing a comprehensive open-source solution for developers, researchers, and
practitioners.
MuseCoco: Generating Symbolic Music from Text
Authors: name: Peiling Lu, name: Xin Xu, name: Chenfei Kang, name: Botao Yu, name: Chengyi Xing, name: Xu Tan, name: Jiang Bian
Published: 2023-05-31 18:34:16

Generating music from text descriptions is a user-friendly mode since the
text is a relatively easy interface for user engagement. While some approaches
utilize texts to control music audio generation, editing musical elements in
generated audio is challenging for users. In contrast, symbolic music offers
ease of editing, making it more accessible for users to manipulate specific
musical elements. In this paper, we propose MuseCoco, which generates symbolic
music from text descriptions with musical attributes as the bridge to break
down the task into text-to-attribute understanding and attribute-to-music
generation stages. MuseCoCo stands for Music Composition Copilot that empowers
musicians to generate music directly from given text descriptions, offering a
significant improvement in efficiency compared to creating music entirely from
scratch. The system has two main advantages: Firstly, it is data efficient. In
the attribute-to-music generation stage, the attributes can be directly
extracted from music sequences, making the model training self-supervised. In
the text-to-attribute understanding stage, the text is synthesized and refined
by ChatGPT based on the defined attribute templates. Secondly, the system can
achieve precise control with specific attributes in text descriptions and
offers multiple control options through attribute-conditioned or
text-conditioned approaches. MuseCoco outperforms baseline systems in terms of
musicality, controllability, and overall score by at least 1.27, 1.08, and 1.32
respectively. Besides, there is a notable enhancement of about 20% in objective
control accuracy. In addition, we have developed a robust large-scale model
with 1.2 billion parameters, showcasing exceptional controllability and
musicality.
Make-Your-Video: Customized Video Generation Using Textual and
  Structural Guidance
Authors: name: Jinbo Xing, name: Menghan Xia, name: Yuxin Liu, name: Yuechen Zhang, name: Yong Zhang, name: Yingqing He, name: Hanyuan Liu, name: Haoxin Chen, name: Xiaodong Cun, name: Xintao Wang, name: Ying Shan, name: Tien-Tsin Wong
Published: 2023-06-01 17:43:27

Creating a vivid video from the event or scenario in our imagination is a
truly fascinating experience. Recent advancements in text-to-video synthesis
have unveiled the potential to achieve this with prompts only. While text is
convenient in conveying the overall scene context, it may be insufficient to
control precisely. In this paper, we explore customized video generation by
utilizing text as context description and motion structure (e.g. frame-wise
depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
joint-conditional video generation using a Latent Diffusion Model that is
pre-trained for still image synthesis and then promoted for video generation
with the introduction of temporal modules. This two-stage learning scheme not
only reduces the computing resources required, but also improves the
performance by transferring the rich concepts available in image datasets
solely into video generation. Moreover, we use a simple yet effective causal
attention mask strategy to enable longer video synthesis, which mitigates the
potential quality degradation effectively. Experimental results show the
superiority of our method over existing baselines, particularly in terms of
temporal coherence and fidelity to users' guidance. In addition, our model
enables several intriguing applications that demonstrate potential for
practical usage.
Diffusion Self-Guidance for Controllable Image Generation
Authors: name: Dave Epstein, name: Allan Jabri, name: Ben Poole, name: Alexei A. Efros, name: Aleksander Holynski
Published: 2023-06-01 17:59:56

Large-scale generative models are capable of producing high-quality images
from detailed text descriptions. However, many aspects of an image are
difficult or impossible to convey through text. We introduce self-guidance, a
method that provides greater control over generated images by guiding the
internal representations of diffusion models. We demonstrate that properties
such as the shape, location, and appearance of objects can be extracted from
these representations and used to steer sampling. Self-guidance works similarly
to classifier guidance, but uses signals present in the pretrained model
itself, requiring no additional models or training. We show how a simple set of
properties can be composed to perform challenging image manipulations, such as
modifying the position or size of objects, merging the appearance of objects in
one image with the layout of another, composing objects from many images into
one, and more. We also show that self-guidance can be used to edit real images.
For results and an interactive demo, see our project page at
https://dave.ml/selfguidance/
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image
  Generation
Authors: name: Minghui Hu, name: Jianbin Zheng, name: Daqing Liu, name: Chuanxia Zheng, name: Chaoyue Wang, name: Dacheng Tao, name: Tat-Jen Cham
Published: 2023-06-01 17:55:32

Text-conditional diffusion models are able to generate high-fidelity images
with diverse contents. However, linguistic representations frequently exhibit
ambiguous descriptions of the envisioned objective imagery, requiring the
incorporation of additional control signals to bolster the efficacy of
text-guided diffusion models. In this work, we propose Cocktail, a pipeline to
mix various modalities into one embedding, amalgamated with a generalized
ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a
spatial guidance sampling method, to actualize multi-modal and
spatially-refined control for text-conditional diffusion models. Specifically,
we introduce a hyper-network gControlNet, dedicated to the alignment and
infusion of the control signals from disparate modalities into the pre-trained
diffusion model. gControlNet is capable of accepting flexible modality signals,
encompassing the simultaneous reception of any combination of modality signals,
or the supplementary fusion of multiple modality signals. The control signals
are then fused and injected into the backbone model according to our proposed
ControlNorm. Furthermore, our advanced spatial guidance sampling methodology
proficiently incorporates the control signal into the designated region,
thereby circumventing the manifestation of undesired objects within the
generated image. We demonstrate the results of our method in controlling
various modalities, proving high-quality synthesis and fidelity to multiple
external signals.