Orionnnn/hub-stats huggingface dataset:
Diffusion Models Beat GANs on Image Classification
Authors: name: Soumik Mukhopadhyay, name: Matthew Gwilliam, name: Vatsal Agarwal, name: Namitha Padmanabhan, name: Archana Swaminathan, name: Srinidhi Hegde, name: Tianyi Zhou, name: Abhinav Shrivastava
Published: 2023-07-17 17:59:40

While many unsupervised learning models focus on one family of tasks, either
generative or discriminative, we explore the possibility of a unified
representation learner: a model which uses a single pre-training stage to
address both families of tasks simultaneously. We identify diffusion models as
a prime candidate. Diffusion models have risen to prominence as a
state-of-the-art method for image generation, denoising, inpainting,
super-resolution, manipulation, etc. Such models involve training a U-Net to
iteratively predict and remove noise, and the resulting model can synthesize
high fidelity, diverse, novel images. The U-Net architecture, as a
convolution-based architecture, generates a diverse set of feature
representations in the form of intermediate feature maps. We present our
findings that these embeddings are useful beyond the noise prediction task, as
they contain discriminative information and can also be leveraged for
classification. We explore optimal methods for extracting and using these
embeddings for classification tasks, demonstrating promising results on the
ImageNet classification task. We find that with careful feature selection and
pooling, diffusion models outperform comparable generative-discriminative
methods such as BigBiGAN for classification tasks. We investigate diffusion
models in the transfer learning regime, examining their performance on several
fine-grained visual classification datasets. We compare these embeddings to
those generated by competing architectures and pre-trainings for classification
tasks.
Retentive Network: A Successor to Transformer for Large Language Models
Authors: name: Yutao Sun, name: Li Dong, name: Shaohan Huang, name: Shuming Ma, name: Yuqing Xia, name: Jilong Xue, name: Jianyong Wang, name: Furu Wei
Published: 2023-07-17 16:40:01

In this work, we propose Retentive Network (RetNet) as a foundation
architecture for large language models, simultaneously achieving training
parallelism, low-cost inference, and good performance. We theoretically derive
the connection between recurrence and attention. Then we propose the retention
mechanism for sequence modeling, which supports three computation paradigms,
i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
representation allows for training parallelism. The recurrent representation
enables low-cost O(1) inference, which improves decoding throughput, latency,
and GPU memory without sacrificing performance. The chunkwise recurrent
representation facilitates efficient long-sequence modeling with linear
complexity, where each chunk is encoded parallelly while recurrently
summarizing the chunks. Experimental results on language modeling show that
RetNet achieves favorable scaling results, parallel training, low-cost
deployment, and efficient inference. The intriguing properties make RetNet a
strong successor to Transformer for large language models. Code will be
available at https://aka.ms/retnet.
Planting a SEED of Vision in Large Language Model
Authors: name: Yuying Ge, name: Yixiao Ge, name: Ziyun Zeng, name: Xintao Wang, name: Ying Shan
Published: 2023-07-16 13:41:39

We present SEED, an elaborate image tokenizer that empowers Large Language
Models (LLMs) with the emergent ability to SEE and Draw at the same time.
Research on image tokenizers has previously reached an impasse, as frameworks
employing quantized visual tokens have lost prominence due to subpar
performance and convergence in multimodal comprehension (compared to BLIP-2,
etc.) or generation (compared to Stable Diffusion, etc.). Despite the
limitations, we remain confident in its natural capacity to unify visual and
textual representations, facilitating scalable multimodal training with LLM's
original recipe. In this study, we identify two crucial principles for the
architecture and training of SEED that effectively ease subsequent alignment
with LLMs. (1) Image tokens should be independent of 2D physical patch
positions and instead be produced with a 1D causal dependency, exhibiting
intrinsic interdependence that aligns with the left-to-right autoregressive
prediction mechanism in LLMs. (2) Image tokens should capture high-level
semantics consistent with the degree of semantic abstraction in words, and be
optimized for both discriminativeness and reconstruction during the tokenizer
training phase. As a result, the off-the-shelf LLM is able to perform both
image-to-text and text-to-image generation by incorporating our SEED through
efficient LoRA tuning. Comprehensive multimodal pretraining and instruction
tuning, which may yield improved results, are reserved for future
investigation. This version of SEED was trained in 5.7 days using only 64 V100
GPUs and 5M publicly available image-text pairs. Our preliminary study
emphasizes the great potential of discrete visual tokens in versatile
multimodal LLMs and the importance of proper image tokenizers in broader
research.
AlpaGasus: Training A Better Alpaca with Fewer Data
Authors: name: Lichang Chen, name: Shiyang Li, name: Jun Yan, name: Hai Wang, name: Kalpa Gunaratna, name: Vikas Yadav, name: Zheng Tang, name: Vijay Srinivasan, name: Tianyi Zhou, name: Heng Huang, name: Hongxia Jin
Published: 2023-07-17 17:59:40

Large language models~(LLMs) obtain instruction-following capability through
instruction-finetuning (IFT) on supervised instruction/response data. However,
widely used IFT datasets (e.g., Alpaca's 52k data) surprisingly contain many
low-quality instances with incorrect or irrelevant responses, which are
misleading and detrimental to IFT. In this paper, we propose a simple and
effective data selection strategy that automatically identifies and removes
low-quality data using a strong LLM (e.g., ChatGPT). To this end, we introduce
AlpaGasus, which is finetuned on only 9k high-quality data filtered from the
52k Alpaca data. AlpaGasus significantly outperforms the original Alpaca as
evaluated by GPT-4 on multiple test sets and its 13B variant matches >90%
performance of its teacher LLM (i.e., Text-Davinci-003) on test tasks. It also
provides 5.7x faster training, reducing the training time for a 7B variant from
80 minutes (for Alpaca) to 14 minutes We apply IFT for the same
number of epochs as Alpaca(7B) but on fewer data, using 4timesNVIDIA A100
(80GB) GPUs and following the original Alpaca setting and hyperparameters..
Overall, AlpaGasus demonstrates a novel data-centric IFT paradigm that can be
generally applied to instruction-tuning data, leading to faster training and
better instruction-following models. Our project page is available at:
https://lichang-chen.github.io/AlpaGasus/.
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
Authors: name: Yang Zhao, name: Zhijie Lin, name: Daquan Zhou, name: Zilong Huang, name: Jiashi Feng, name: Bingyi Kang
Published: 2023-07-17 15:51:47

LLMs have demonstrated remarkable abilities at interacting with humans
through language, especially with the usage of instruction-following data.
Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further
enlarge their abilities by incorporating multi-modal inputs, including image,
video, and speech. Despite their effectiveness at generating precise and
detailed language understanding of the given modality signal, these LLMs give
up the ability to ground specific parts of inputs, thus only constructing a
coarse-grained mapping. However, explicit and informative correspondence
between text and other modalities will not only improve the user experience but
also help to expand the application scenario of multi-modal LLMs. Therefore, we
propose BuboGPT, a multi-modal LLM with visual grounding that can perform
cross-modal interaction between vision, audio and language, providing
fine-grained understanding of visual objects and other given modalities. As a
result, BuboGPT is able to point out the specific location of an object in the
image, when it is generating response or description for that object. Our
contributions are two-fold: 1) An off-the-shelf visual grounding module based
on SAM that extracts entities in a sentence and find corresponding masks in the
image. 2) A two-stage training scheme and instruction dataset to endow joint
text-image-audio understanding. Our experiments show that BuboGPT achieves
impressive multi-modality understanding and visual grounding abilities during
the interaction with human. It performs consistently well when provided by
arbitrary modality combinations (either aligned or unaligned). Our code, model
and dataset are available at https://bubo-gpt.github.io .
NIFTY: Neural Object Interaction Fields for Guided Human Motion
  Synthesis
Authors: name: Nilesh Kulkarni, name: Davis Rempe, name: Kyle Genova, name: Abhijit Kundu, name: Justin Johnson, name: David Fouhey, name: Leonidas Guibas
Published: 2023-07-14 17:59:38

We address the problem of generating realistic 3D motions of humans
interacting with objects in a scene. Our key idea is to create a neural
interaction field attached to a specific object, which outputs the distance to
the valid interaction manifold given a human pose as input. This interaction
field guides the sampling of an object-conditioned human motion diffusion
model, so as to encourage plausible contacts and affordance semantics. To
support interactions with scarcely available data, we propose an automated
synthetic data pipeline. For this, we seed a pre-trained motion model, which
has priors for the basics of human movement, with interaction-specific anchor
poses extracted from limited motion capture data. Using our guided diffusion
model trained on generated synthetic data, we synthesize realistic motions for
sitting and lifting with several objects, outperforming alternative approaches
in terms of motion quality and successful action completion. We call our
framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.
DreamTeacher: Pretraining Image Backbones with Deep Generative Models
Authors: name: Daiqing Li, name: Huan Ling, name: Amlan Kar, name: David Acuna, name: Seung Wook Kim, name: Karsten Kreis, name: Antonio Torralba, name: Sanja Fidler
Published: 2023-07-14 17:17:17

In this work, we introduce a self-supervised feature representation learning
framework DreamTeacher that utilizes generative networks for pre-training
downstream image backbones. We propose to distill knowledge from a trained
generative model into standard image backbones that have been well engineered
for specific perception tasks. We investigate two types of knowledge
distillation: 1) distilling learned generative features onto target image
backbones as an alternative to pretraining these backbones on large labeled
datasets such as ImageNet, and 2) distilling labels obtained from generative
networks with task heads onto logits of target backbones. We perform extensive
analyses on multiple generative models, dense prediction benchmarks, and
several pre-training regimes. We empirically find that our DreamTeacher
significantly outperforms existing self-supervised representation learning
approaches across the board. Unsupervised ImageNet pre-training with
DreamTeacher leads to significant improvements over ImageNet classification
pre-training on downstream datasets, showcasing generative models, and
diffusion generative models specifically, as a promising approach to
representation learning on large, diverse datasets without requiring manual
annotation.
Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech
  Prompts
Authors: name: Ziyue Jiang, name: Jinglin Liu, name: Yi Ren, name: Jinzheng He, name: Chen Zhang, name: Zhenhui Ye, name: Pengfei Wei, name: Chunfeng Wang, name: Xiang Yin, name: Zejun Ma, name: Zhou Zhao
Published: 2023-07-14 08:21:25

Zero-shot text-to-speech aims at synthesizing voices with unseen speech
prompts. Previous large-scale multispeaker TTS models have successfully
achieved this goal with an enrolled recording within 10 seconds. However, most
of them are designed to utilize only short speech prompts. The limited
information in short speech prompts significantly hinders the performance of
fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a
generic zero-shot multispeaker TTS model that is capable of synthesizing speech
for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a
multi-reference timbre encoder to extract timbre information from multiple
reference speeches; 2) and train a prosody language model with arbitrary-length
speech prompts; With these designs, our model is suitable for prompts of
different lengths, which extends the upper bound of speech quality for
zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce
arbitrary-source prompts, which leverages the probabilities derived from
multiple P-LLM outputs to produce expressive and controlled prosody.
Furthermore, we propose a phoneme-level auto-regressive duration model to
introduce in-context learning capabilities to duration modeling. Experiments
demonstrate that our method could not only synthesize identity-preserving
speech with a short prompt of an unseen speaker but also achieve improved
performance with longer speech prompts. Audio samples can be found in
https://mega-tts.github.io/mega2_demo/.
Learning to Retrieve In-Context Examples for Large Language Models
Authors: name: Liang Wang, name: Nan Yang, name: Furu Wei
Published: 2023-07-14 05:23:08

Large language models (LLMs) have demonstrated their ability to learn
in-context, allowing them to perform various tasks based on a few input-output
examples. However, the effectiveness of in-context learning is heavily reliant
on the quality of the selected examples. In this paper, we propose a novel
framework to iteratively train dense retrievers that can identify high-quality
in-context examples for LLMs. Our framework initially trains a reward model
based on LLM feedback to evaluate the quality of candidate examples, followed
by knowledge distillation to train a bi-encoder based dense retriever. Our
experiments on a suite of 30 tasks demonstrate that our framework significantly
enhances in-context learning performance. Furthermore, we show the
generalization ability of our framework to unseen tasks during training. An
in-depth analysis reveals that our model improves performance by retrieving
examples with similar patterns, and the gains are consistent across LLMs of
varying sizes.
Copy Is All You Need
Authors: name: Tian Lan, name: Deng Cai, name: Yan Wang, name: Heyan Huang, name: Xian-Ling Mao
Published: 2023-07-13 05:03:26

The dominant text generation models compose the output by sequentially
selecting words from a fixed vocabulary. In this paper, we formulate text
generation as progressively copying text segments (e.g., words or phrases) from
an existing text collection. We compute the contextualized representations of
meaningful text segments and index them using efficient vector search toolkits.
The task of text generation is then decomposed into a series of copy-and-paste
operations: at each time step, we seek suitable text spans from the text
collection rather than selecting from a standalone vocabulary. Experiments on
the standard language modeling benchmark (WikiText-103) show that our approach
achieves better generation quality according to both automatic and human
evaluations. Besides, its inference efficiency is comparable to token-level
autoregressive models thanks to the reduction of decoding steps. We also show
that our approach allows for effective domain adaptation by simply switching to
domain-specific text collection without extra training. Finally, we observe
that our approach attains additional performance gains by simply scaling up to
larger text collections, again without further training.Our source
codes are publicly available at
\url{https://github.com/gmftbyGMFTBY/Copyisallyouneed.}
DIALGEN: Collaborative Human-LM Generated Dialogues for Improved
  Understanding of Human-Human Conversations
Authors: name: Bo-Ru Lu, name: Nikita Haduong, name: Chia-Hsuan Lee, name: Zeqiu Wu, name: Hao Cheng, name: Paul Koester, name: Jean Utke, name: Tao Yu, name: Noah A. Smith, name: Mari Ostendorf
Published: 2023-07-13 20:02:50

Applications that could benefit from automatic understanding of human-human
conversations often come with challenges associated with private information in
real-world data such as call center or clinical conversations. Working with
protected data also increases costs of annotation, which limits technology
development. To address these challenges, we propose DIALGEN, a
human-in-the-loop semi-automated dialogue generation framework. DIALGEN uses a
language model (ChatGPT) that can follow schema and style specifications to
produce fluent conversational text, generating a complex conversation through
iteratively generating subdialogues and using human feedback to correct
inconsistencies or redirect the flow. In experiments on structured
summarization of agent-client information gathering calls, framed as dialogue
state tracking, we show that DIALGEN data enables significant improvement in
model performance.
Self-consistency for open-ended generations
Authors: name: Siddhartha Jain, name: Xiaofei Ma, name: Anoop Deoras, name: Bing Xiang
Published: 2023-07-11 17:51:48

In this paper, we present a novel approach for improving the quality and
consistency of generated outputs from large-scale pre-trained language models
(LLMs). Self-consistency has emerged as an effective approach for prompts with
fixed answers, selecting the answer with the highest number of votes. In this
paper, we introduce a generalized framework for self-consistency that extends
its applicability beyond problems that have fixed-answer answers. Through
extensive simulations, we demonstrate that our approach consistently recovers
the optimal or near-optimal generation from a set of candidates. We also
propose lightweight parameter-free similarity functions that show significant
and consistent improvements across code generation, autoformalization, and
summarization tasks, even without access to token log probabilities. Our method
incurs minimal computational overhead, requiring no auxiliary reranker models
or modifications to the existing model.
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image
  Models
Authors: name: Nataniel Ruiz, name: Yuanzhen Li, name: Varun Jampani, name: Wei Wei, name: Tingbo Hou, name: Yael Pritch, name: Neal Wadhwa, name: Michael Rubinstein, name: Kfir Aberman
Published: 2023-07-13 17:59:47

Personalization has emerged as a prominent aspect within the field of
generative AI, enabling the synthesis of individuals in diverse contexts and
styles, while retaining high-fidelity to their identities. However, the process
of personalization presents inherent challenges in terms of time and memory
requirements. Fine-tuning each personalized model needs considerable GPU time
investment, and storing a personalized model per subject can be demanding in
terms of storage capacity. To overcome these challenges, we propose
HyperDreamBooth-a hypernetwork capable of efficiently generating a small set of
personalized weights from a single image of a person. By composing these
weights into the diffusion model, coupled with fast finetuning, HyperDreamBooth
can generate a person's face in various contexts and styles, with high subject
details while also preserving the model's crucial knowledge of diverse styles
and semantic modifications. Our method achieves personalization on faces in
roughly 20 seconds, 25x faster than DreamBooth and 125x faster than Textual
Inversion, using as few as one reference image, with the same quality and style
diversity as DreamBooth. Also our method yields a model that is 10000x smaller
than a normal DreamBooth model. Project page: https://hyperdreambooth.github.io
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
Authors: name: Yingqing He, name: Menghan Xia, name: Haoxin Chen, name: Xiaodong Cun, name: Yuan Gong, name: Jinbo Xing, name: Yong Zhang, name: Xintao Wang, name: Chao Weng, name: Ying Shan, name: Qifeng Chen
Published: 2023-07-13 17:57:13

Generating videos for visual storytelling can be a tedious and complex
process that typically requires either live-action filming or graphics
animation rendering. To bypass these challenges, our key idea is to utilize the
abundance of existing video clips and synthesize a coherent storytelling video
by customizing their appearances. We achieve this by developing a framework
comprised of two functional modules: (i) Motion Structure Retrieval, which
provides video candidates with desired scene or motion context described by
query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates
plot-aligned videos under the guidance of motion structure and text prompts.
For the first module, we leverage an off-the-shelf video retrieval system and
extract video depths as motion structure. For the second module, we propose a
controllable video generation model that offers flexible controls over
structure and characters. The videos are synthesized by following the
structural guidance and appearance instruction. To ensure visual consistency
across clips, we propose an effective concept personalization approach, which
allows the specification of the desired character identities through text
prompts. Extensive experiments demonstrate that our approach exhibits
significant advantages over various existing baselines.
Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image
  Models
Authors: name: Moab Arar, name: Rinon Gal, name: Yuval Atzmon, name: Gal Chechik, name: Daniel Cohen-Or, name: Ariel Shamir, name: Amit H. Bermano
Published: 2023-07-13 17:46:42

Text-to-image (T2I) personalization allows users to guide the creative image
generation process by combining their own visual concepts in natural language
prompts. Recently, encoder-based techniques have emerged as a new effective
approach for T2I personalization, reducing the need for multiple images and
long training times. However, most existing encoders are limited to a
single-class domain, which hinders their ability to handle diverse concepts. In
this work, we propose a domain-agnostic method that does not require any
specialized dataset or prior information about the personalized concepts. We
introduce a novel contrastive-based regularization technique to maintain high
fidelity to the target concept characteristics while keeping the predicted
embeddings close to editable regions of the latent space, by pushing the
predicted tokens toward their nearest existing CLIP tokens. Our experimental
results demonstrate the effectiveness of our approach and show how the learned
tokens are more semantic than tokens predicted by unregularized models. This
leads to a better representation that achieves state-of-the-art performance
while being more flexible than previous methods.