Orionnnn/hub-stats huggingface dataset:
FreeU: Free Lunch in Diffusion U-Net
Authors: name: Chenyang Si, name: Ziqi Huang, name: Yuming Jiang, name: Ziwei Liu
Published: 2023-09-20 17:56:18

In this paper, we uncover the untapped potential of diffusion U-Net, which
serves as a "free lunch" that substantially improves the generation quality on
the fly. We initially investigate the key contributions of the U-Net
architecture to the denoising process and identify that its main backbone
primarily contributes to denoising, whereas its skip connections mainly
introduce high-frequency features into the decoder module, causing the network
to overlook the backbone semantics. Capitalizing on this discovery, we propose
a simple yet effective method-termed "FreeU" - that enhances generation quality
without additional training or finetuning. Our key insight is to strategically
re-weight the contributions sourced from the U-Net's skip connections and
backbone feature maps, to leverage the strengths of both components of the
U-Net architecture. Promising results on image and video generation tasks
demonstrate that our FreeU can be readily integrated to existing diffusion
models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion,
to improve the generation quality with only a few lines of code. All you need
is to adjust two scaling factors during inference. Project page:
https://chenyangsi.top/FreeU/.
End-to-End Speech Recognition Contextualization with Large Language
  Models
Authors: name: Egor Lakomkin, name: Chunyang Wu, name: Yassir Fathullah, name: Ozlem Kalinli, name: Michael L. Seltzer, name: Christian Fuegen
Published: 2023-09-19 20:28:57

In recent years, Large Language Models (LLMs) have garnered significant
attention from the research community due to their exceptional performance and
generalization capabilities. In this paper, we introduce a novel method for
contextualizing speech recognition models incorporating LLMs. Our approach
casts speech recognition as a mixed-modal language modeling task based on a
pretrained LLM. We provide audio features, along with optional text tokens for
context, to train the system to complete transcriptions in a decoder-only
fashion. As a result, the system is implicitly incentivized to learn how to
leverage unstructured contextual information during training. Our empirical
results demonstrate a significant improvement in performance, with a 6% WER
reduction when additional textual context is provided. Moreover, we find that
our method performs competitively and improve by 7.5% WER overall and 17% WER
on rare words against a baseline contextualized RNN-T system that has been
trained on more than twenty five times larger speech dataset. Overall, we
demonstrate that by only adding a handful number of trainable parameters via
adapters, we can unlock contextualized speech recognition capability for the
pretrained LLM while keeping the same text-only input functionality.
DreamLLM: Synergistic Multimodal Comprehension and Creation
Authors: name: Runpei Dong, name: Chunrui Han, name: Yuang Peng, name: Zekun Qi, name: Zheng Ge, name: Jinrong Yang, name: Liang Zhao, name: Jianjian Sun, name: Hongyu Zhou, name: Haoran Wei, name: Xiangwen Kong, name: Xiangyu Zhang, name: Kaisheng Ma, name: Li Yi
Published: 2023-09-20 17:58:05

This paper presents DreamLLM, a learning framework that first achieves
versatile Multimodal Large Language Models (MLLMs) empowered with frequently
overlooked synergy between multimodal comprehension and creation. DreamLLM
operates on two fundamental principles. The first focuses on the generative
modeling of both language and image posteriors by direct sampling in the raw
multimodal space. This approach circumvents the limitations and information
loss inherent to external feature extractors like CLIP, and a more thorough
multimodal understanding is obtained. Second, DreamLLM fosters the generation
of raw, interleaved documents, modeling both text and image contents, along
with unstructured layouts. This allows DreamLLM to learn all conditional,
marginal, and joint multimodal distributions effectively. As a result, DreamLLM
is the first MLLM capable of generating free-form interleaved content.
Comprehensive experiments highlight DreamLLM's superior performance as a
zero-shot multimodal generalist, reaping from the enhanced learning synergy.
Chain-of-Verification Reduces Hallucination in Large Language Models
Authors: name: Shehzaad Dhuliawala, name: Mojtaba Komeili, name: Jing Xu, name: Roberta Raileanu, name: Xian Li, name: Asli Celikyilmaz, name: Jason Weston
Published: 2023-09-20 17:50:55

Generation of plausible yet incorrect factual information, termed
hallucination, is an unsolved issue in large language models. We study the
ability of language models to deliberate on the responses they give in order to
correct their mistakes. We develop the Chain-of-Verification (CoVe) method
whereby the model first (i) drafts an initial response; then (ii) plans
verification questions to fact-check its draft; (iii) answers those questions
independently so the answers are not biased by other responses; and (iv)
generates its final verified response. In experiments, we show CoVe decreases
hallucinations across a variety of tasks, from list-based questions from
Wikidata, closed book MultiSpanQA and longform text generation.
Kosmos-2.5: A Multimodal Literate Model
Authors: name: Tengchao Lv, name: Yupan Huang, name: Jingye Chen, name: Lei Cui, name: Shuming Ma, name: Yaoyao Chang, name: Shaohan Huang, name: Wenhui Wang, name: Li Dong, name: Weiyao Luo, name: Shaoxiang Wu, name: Guoxin Wang, name: Cha Zhang, name: Furu Wei
Published: 2023-09-20 15:50:08

We present Kosmos-2.5, a multimodal literate model for machine reading of
text-intensive images. Pre-trained on large-scale text-intensive images,
Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
generating spatially-aware text blocks, where each block of text is assigned
its spatial coordinates within the image, and (2) producing structured text
output that captures styles and structures into the markdown format. This
unified multimodal literate capability is achieved through a shared Transformer
architecture, task-specific prompts, and flexible text representations. We
evaluate Kosmos-2.5 on end-to-end document-level text recognition and
image-to-markdown text generation. Furthermore, the model can be readily
adapted for any text-intensive image understanding task with different prompts
through supervised fine-tuning, making it a general-purpose tool for real-world
applications involving text-rich images. This work also paves the way for the
future scaling of multimodal large language models.
LMDX: Language Model-based Document Information Extraction and
  Localization
Authors: name: Vincent Perot, name: Kai Kang, name: Florian Luisier, name: Guolong Su, name: Xiaoyu Sun, name: Ramya Sree Boppana, name: Zilong Wang, name: Jiaqi Mu, name: Hao Zhang, name: Nan Hua
Published: 2023-09-19 22:32:56

Large Language Models (LLM) have revolutionized Natural Language Processing
(NLP), improving state-of-the-art on many existing tasks and exhibiting
emergent capabilities. However, LLMs have not yet been successfully applied on
semi-structured document information extraction, which is at the core of many
document processing workflows and consists of extracting key entities from a
visually rich document (VRD) given a predefined target schema. The main
obstacles to LLM adoption in that task have been the absence of layout encoding
within LLMs, critical for a high quality extraction, and the lack of a
grounding mechanism ensuring the answer is not hallucinated. In this paper, we
introduce Language Model-based Document Information Extraction and Localization
(LMDX), a methodology to adapt arbitrary LLMs for document information
extraction. LMDX can do extraction of singular, repeated, and hierarchical
entities, both with and without training data, while providing grounding
guarantees and localizing the entities within the document. In particular, we
apply LMDX to the PaLM 2-S LLM and evaluate it on VRDU and CORD benchmarks,
setting a new state-of-the-art and showing how LMDX enables the creation of
high quality, data-efficient parsers.
360^circ Reconstruction From a Single Image Using Space Carved
  Outpainting
Authors: name: Nuri Ryu, name: Minsu Gong, name: Geonung Kim, name: Joo-Haeng Lee, name: Sunghyun Cho
Published: 2023-09-19 03:07:26

We introduce POP3D, a novel framework that creates a full 360^circ-view 3D
model from a single image. POP3D resolves two prominent issues that limit the
single-view reconstruction. Firstly, POP3D offers substantial generalizability
to arbitrary categories, a trait that previous methods struggle to achieve.
Secondly, POP3D further improves reconstruction fidelity and naturalness, a
crucial aspect that concurrent works fall short of. Our approach marries the
strengths of four primary components: (1) a monocular depth and normal
predictor that serves to predict crucial geometric cues, (2) a space carving
method capable of demarcating the potentially unseen portions of the target
object, (3) a generative model pre-trained on a large-scale image dataset that
can complete unseen regions of the target, and (4) a neural implicit surface
reconstruction method tailored in reconstructing objects using RGB images along
with monocular geometric cues. The combination of these components enables
POP3D to readily generalize across various in-the-wild images and generate
state-of-the-art reconstructions, outperforming similar works by a significant
margin. Project page: http://cg.postech.ac.kr/research/POP3D
Q-Transformer: Scalable Offline Reinforcement Learning via
  Autoregressive Q-Functions
Authors: name: Yevgen Chebotar, name: Quan Vuong, name: Alex Irpan, name: Karol Hausman, name: Fei Xia, name: Yao Lu, name: Aviral Kumar, name: Tianhe Yu, name: Alexander Herzog, name: Karl Pertsch, name: Keerthana Gopalakrishnan, name: Julian Ibarz, name: Ofir Nachum, name: Sumedh Sontakke, name: Grecia Salazar, name: Huong T Tran, name: Jodilyn Peralta, name: Clayton Tan, name: Deeksha Manjunath, name: Jaspiar Singht, name: Brianna Zitkovich, name: Tomas Jackson, name: Kanishka Rao, name: Chelsea Finn, name: Sergey Levine
Published: 2023-09-18 21:00:38

In this work, we present a scalable reinforcement learning method for
training multi-task policies from large offline datasets that can leverage both
human demonstrations and autonomously collected data. Our method uses a
Transformer to provide a scalable representation for Q-functions trained via
offline temporal difference backups. We therefore refer to the method as
Q-Transformer. By discretizing each action dimension and representing the
Q-value of each action dimension as separate tokens, we can apply effective
high-capacity sequence modeling techniques for Q-learning. We present several
design decisions that enable good performance with offline RL training, and
show that Q-Transformer outperforms prior offline RL algorithms and imitation
learning techniques on a large diverse real-world robotic manipulation task
suite. The project's website and videos can be found at
https://q-transformer.github.io
FoleyGen: Visually-Guided Audio Generation
Authors: name: Xinhao Mei, name: Varun Nagaraja, name: Gael Le Lan, name: Zhaoheng Ni, name: Ernie Chang, name: Yangyang Shi, name: Vikas Chandra
Published: 2023-09-19 11:33:43

Recent advancements in audio generation have been spurred by the evolution of
large-scale deep learning models and expansive datasets. However, the task of
video-to-audio (V2A) generation continues to be a challenge, principally
because of the intricate relationship between the high-dimensional visual and
auditory data, and the challenges associated with temporal synchronization. In
this study, we introduce FoleyGen, an open-domain V2A generation system built
on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural
audio codec for bidirectional conversion between waveforms and discrete tokens.
The generation of audio tokens is facilitated by a single Transformer model,
which is conditioned on visual features extracted from a visual encoder. A
prevalent problem in V2A generation is the misalignment of generated audio with
the visible actions in the video. To address this, we explore three novel
visual attention mechanisms. We further undertake an exhaustive evaluation of
multiple visual encoders, each pretrained on either single-modal or multi-modal
tasks. The experimental results on VGGSound dataset show that our proposed
FoleyGen outperforms previous systems across all objective metrics and human
evaluations.
SlimPajama-DC: Understanding Data Combinations for LLM Training
Authors: name: Zhiqiang Shen, name: Tianhua Tao, name: Liqun Ma, name: Willie Neiswanger, name: Joel Hestness, name: Natalia Vassilieva, name: Daria Soboleva, name: Eric Xing
Published: 2023-09-19 17:59:54

This paper aims to understand the impacts of various data combinations (e.g.,
web text, wikipedia, github, books) on the training of large language models
using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
dataset, which has been refined and further deduplicated to 627B tokens from
the extensive 1.2T tokens RedPajama dataset contributed by Together. We've
termed our research as SlimPajama-DC, an empirical analysis designed to uncover
fundamental characteristics and best practices associated with employing
SlimPajama in the training of large language models. During our research with
SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
local deduplication. We analyze and discuss how global (across different
sources of datasets) and local (within the single source of dataset)
deduplications affect the performance of trained models. (2) Proportions of
high-quality/highly-deduplicated multi-source datasets in the combination. To
study this, we construct six configurations of SlimPajama dataset and train
individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best
configuration outperforms the 1.3B model trained on RedPajama using the same
number of training tokens by a significant margin. All our 1.3B models are
trained on Cerebras 16times CS-2 cluster with a total of 80 PFLOP/s in bf16
mixed precision. We further extend our discoveries (such as increasing data
diversity is crucial after global deduplication) on a 7B model with large
batch-size training. Our models and the separate SlimPajama-DC datasets are
available at: https://huggingface.co/MBZUAI-LLM and
https://huggingface.co/datasets/cerebras/SlimPajama-627B.
OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model
  Pre-trained from Scratch
Authors: name: Juntao Li, name: Zecheng Tang, name: Yuyang Ding, name: Pinzheng Wang, name: Pei Guo, name: Wangjie You, name: Dan Qiao, name: Wenliang Chen, name: Guohong Fu, name: Qiaoming Zhu, name: Guodong Zhou, name: Min Zhang
Published: 2023-09-19 15:46:40

Large language models (LLMs) with billions of parameters have demonstrated
outstanding performance on various natural language processing tasks. This
report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
to contribute an LLM variant to the Chinese-oriented open-source model
community. We enhance OpenBA with effective and efficient techniques as well as
adopt a three-stage training strategy to train the model from scratch. Our
solution can also achieve very competitive performance with only 380B tokens,
which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
the main details to pre-train an analogous model, including pre-training data
processing, Bilingual Flan data collection, the empirical observations that
inspire our model architecture design, training objectives of different stages,
and other enhancement techniques. We have refactored our code to follow the
design principles of the Huggingface Transformers Library, making it more
convenient for developers to use, and released checkpoints of different
training stages at https://huggingface.co/openBA. More details of our project
are available at https://github.com/OpenNLG/openBA.git.
Language Modeling Is Compression
Authors: name: Grégoire Delétang, name: Anian Ruoss, name: Paul-Ambroise Duquenne, name: Elliot Catt, name: Tim Genewein, name: Christopher Mattern, name: Jordi Grau-Moya, name: Li Kevin Wenliang, name: Matthew Aitchison, name: Laurent Orseau, name: Marcus Hutter, name: Joel Veness
Published: 2023-09-19 14:50:38

It has long been established that predictive models can be transformed into
lossless compressors and vice versa. Incidentally, in recent years, the machine
learning community has focused on training increasingly large and powerful
self-supervised (language) models. Since these large language models exhibit
impressive predictive capabilities, they are well-positioned to be strong
compressors. In this work, we advocate for viewing the prediction problem
through the lens of compression and evaluate the compression capabilities of
large (foundation) models. We show that large language models are powerful
general-purpose predictors and that the compression viewpoint provides novel
insights into scaling laws, tokenization, and in-context learning. For example,
Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to
43.4% and LibriSpeech samples to 16.4% of their raw size, beating
domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively.
Finally, we show that the prediction-compression equivalence allows us to use
any compressor (like gzip) to build a conditional generative model.
Baichuan 2: Open Large-scale Language Models
Authors: name: Aiyuan Yang, name: Bin Xiao, name: Bingning Wang, name: Borong Zhang, name: Chao Yin, name: Chenxu Lv, name: Da Pan, name: Dian Wang, name: Dong Yan, name: Fan Yang, name: Fei Deng, name: Feng Wang, name: Feng Liu, name: Guangwei Ai, name: Guosheng Dong Haizhou Zhao, name: Hang Xu, name: Haoze Sun, name: Hongda Zhang, name: Hui Liu, name: Jiaming Ji, name: Jian Xie, name: Juntao Dai, name: Kun Fang, name: Lei Su Liang Song, name: Lifeng Liu, name: Liyun Ru, name: Luyao Ma, name: Mang Wang, name: Mickel Liu, name: MingAn Lin, name: Nuolan Nie, name: Peidong Guo, name: Ruiyang Sun, name: Tao Zhang, name: Tianpeng Li, name: Tianyu Li, name: Wei Cheng, name: Weipeng Chen, name: Xiangrong Zeng, name: Xiaochuan Wang, name: Xiaoxi Chen, name: Xin Men, name: Xin Yu, name: Xuehai Pan, name: Yanjun Shen, name: Yiding Wang, name: Yiyu Li, name: Youxin Jiang, name: Yuchen Gao, name: Yupeng Zhang, name: Zenan Zhou, name: Zhiying Wu
Published: 2023-09-19 04:13:22

Large language models (LLMs) have demonstrated remarkable performance on a
variety of natural language tasks based on just a few examples of natural
language instructions, reducing the need for extensive feature engineering.
However, most powerful LLMs are closed-source or limited in their capability
for languages other than English. In this technical report, we present Baichuan
2, a series of large-scale multilingual language models containing 7 billion
and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.
Baichuan 2 matches or outperforms other open-source models of similar size on
public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan
2 excels in vertical domains such as medicine and law. We will release all
pre-training model checkpoints to benefit the research community in better
understanding the training dynamics of Baichuan 2.
Stabilizing RLHF through Advantage Model and Selective Rehearsal
Authors: name: Baolin Peng, name: Linfeng Song, name: Ye Tian, name: Lifeng Jin, name: Haitao Mi, name: Dong Yu
Published: 2023-09-18 23:06:32

Large Language Models (LLMs) have revolutionized natural language processing,
yet aligning these models with human values and preferences using RLHF remains
a significant challenge. This challenge is characterized by various
instabilities, such as reward hacking and catastrophic forgetting. In this
technical report, we propose two innovations to stabilize RLHF training: 1)
Advantage Model, which directly models advantage score i.e., extra reward
compared to the expected rewards and regulates score distributions across tasks
to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic
forgetting by strategically selecting data for PPO training and knowledge
rehearsing. Our experimental analysis on public and proprietary datasets
reveals that the proposed methods not only increase stability in RLHF training
but also achieve higher reward scores and win rates.
Multimodal Foundation Models: From Specialists to General-Purpose
  Assistants
Authors: name: Chunyuan Li, name: Zhe Gan, name: Zhengyuan Yang, name: Jianwei Yang, name: Linjie Li, name: Lijuan Wang, name: Jianfeng Gao
Published: 2023-09-18 17:56:28

This paper presents a comprehensive survey of the taxonomy and evolution of
multimodal foundation models that demonstrate vision and vision-language
capabilities, focusing on the transition from specialist models to
general-purpose assistants. The research landscape encompasses five core
topics, categorized into two classes. (i) We start with a survey of
well-established research areas: multimodal foundation models pre-trained for
specific purposes, including two topics -- methods of learning vision backbones
for visual understanding and text-to-image generation. (ii) Then, we present
recent advances in exploratory, open research areas: multimodal foundation
models that aim to play the role of general-purpose assistants, including three
topics -- unified vision models inspired by large language models (LLMs),
end-to-end training of multimodal LLMs, and chaining multimodal tools with
LLMs. The target audiences of the paper are researchers, graduate students, and
professionals in computer vision and vision-language multimodal communities who
are eager to learn the basics and recent advances in multimodal foundation
models.
A Distributed Data-Parallel PyTorch Implementation of the Distributed
  Shampoo Optimizer for Training Neural Networks At-Scale
Authors: name: Hao-Jun Michael Shi, name: Tsung-Hsien Lee, name: Shintaro Iwasaki, name: Jose Gallego-Posada, name: Zhijing Li, name: Kaushik Rangadurai, name: Dheevatsa Mudigere, name: Michael Rabbat
Published: 2023-09-12 18:11:10

Shampoo is an online and stochastic optimization algorithm belonging to the
AdaGrad family of methods for training neural networks. It constructs a
block-diagonal preconditioner where each block consists of a coarse Kronecker
product approximation to full-matrix AdaGrad for each parameter of the neural
network. In this work, we provide a complete description of the algorithm as
well as the performance optimizations that our implementation leverages to
train deep networks at-scale in PyTorch. Our implementation enables fast
multi-GPU distributed data-parallel training by distributing the memory and
computation associated with blocks of each parameter via PyTorch's DTensor data
structure and performing an AllGather primitive on the computed search
directions at each iteration. This major performance enhancement enables us to
achieve at most a 10% performance reduction in per-step wall-clock time
compared against standard diagonal-scaling-based adaptive gradient methods. We
validate our implementation by performing an ablation study on training
ImageNet ResNet50, demonstrating Shampoo's superiority over standard training
recipes with minimal hyperparameter tuning.