Orionnnn/hub-stats huggingface dataset:
Teaching Arithmetic to Small Transformers
Authors: name: Nayoung Lee, name: Kartik Sreenivasan, name: Jason D. Lee, name: Kangwook Lee, name: Dimitris Papailiopoulos
Published: 2023-07-07 04:33:31

Large language models like GPT-4 exhibit emergent capabilities across
general-purpose tasks, such as basic arithmetic, when trained on extensive text
data, even though these tasks are not explicitly encoded by the unsupervised,
next-token prediction objective. This study investigates how small
transformers, trained from random initialization, can efficiently learn
arithmetic operations such as addition, multiplication, and elementary
functions like square root, using the next-token prediction objective. We first
demonstrate that conventional training data is not the most effective for
arithmetic learning, and simple formatting changes can significantly improve
accuracy. This leads to sharp phase transitions as a function of training data
scale, which, in some cases, can be explained through connections to low-rank
matrix completion. Building on prior work, we then train on chain-of-thought
style data that includes intermediate step results. Even in the complete
absence of pretraining, this approach significantly and simultaneously improves
accuracy, sample complexity, and convergence speed. We also study the interplay
between arithmetic and text data during training and examine the effects of
few-shot prompting, pretraining, and model scale. Additionally, we discuss
length generalization challenges. Our work highlights the importance of
high-quality, instructive data that considers the particular characteristics of
the next-word prediction objective for rapidly eliciting arithmetic
capabilities.
A Survey on Evaluation of Large Language Models
Authors: name: Yupeng Chang, name: Xu Wang, name: Jindong Wang, name: Yuan Wu, name: Kaijie Zhu, name: Hao Chen, name: Linyi Yang, name: Xiaoyuan Yi, name: Cunxiang Wang, name: Yidong Wang, name: Wei Ye, name: Yue Zhang, name: Yi Chang, name: Philip S. Yu, name: Qiang Yang, name: Xing Xie
Published: 2023-07-06 16:28:35

Large language models (LLMs) are gaining increasing popularity in both
academia and industry, owing to their unprecedented performance in various
applications. As LLMs continue to play a vital role in both research and daily
use, their evaluation becomes increasingly critical, not only at the task
level, but also at the society level for better understanding of their
potential risks. Over the past years, significant efforts have been made to
examine LLMs from various perspectives. This paper presents a comprehensive
review of these evaluation methods for LLMs, focusing on three key dimensions:
what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide
an overview from the perspective of evaluation tasks, encompassing general
natural language processing tasks, reasoning, medical usage, ethics,
educations, natural and social sciences, agent applications, and other areas.
Secondly, we answer the `where' and `how' questions by diving into the
evaluation methods and benchmarks, which serve as crucial components in
assessing performance of LLMs. Then, we summarize the success and failure cases
of LLMs in different tasks. Finally, we shed light on several future challenges
that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to
researchers in the realm of LLMs evaluation, thereby aiding the development of
more proficient LLMs. Our key point is that evaluation should be treated as an
essential discipline to better assist the development of LLMs. We consistently
maintain the related open-source materials at:
https://github.com/MLGroupJLU/LLM-eval-survey.
Lost in the Middle: How Language Models Use Long Contexts
Authors: name: Nelson F. Liu, name: Kevin Lin, name: John Hewitt, name: Ashwin Paranjape, name: Michele Bevilacqua, name: Fabio Petroni, name: Percy Liang
Published: 2023-07-06 17:54:11

While recent language models have the ability to take long contexts as input,
relatively little is known about how well the language models use longer
context. We analyze language model performance on two tasks that require
identifying relevant information within their input contexts: multi-document
question answering and key-value retrieval. We find that performance is often
highest when relevant information occurs at the beginning or end of the input
context, and significantly degrades when models must access relevant
information in the middle of long contexts. Furthermore, performance
substantially decreases as the input context grows longer, even for explicitly
long-context models. Our analysis provides a better understanding of how
language models use their input context and provides new evaluation protocols
for future long-context models.
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong
  General Audio Event Taggers
Authors: name: Yuan Gong, name: Sameer Khurana, name: Leonid Karlinsky, name: James Glass
Published: 2023-07-06 17:58:28

In this paper, we focus on Whisper, a recent automatic speech recognition
model trained with a massive 680k hour labeled speech corpus recorded in
diverse conditions. We first show an interesting finding that while Whisper is
very robust against real-world background sounds (e.g., music), its audio
representation is actually not noise-invariant, but is instead highly
correlated to non-speech sounds, indicating that Whisper recognizes speech
conditioned on the noise type. With this finding, we build a unified audio
tagging and speech recognition model Whisper-AT by freezing the backbone of
Whisper, and training a lightweight audio tagging model on top of it. With <1%
extra computational cost, Whisper-AT can recognize audio events, in addition to
spoken text, in a single forward pass.
VideoGLUE: Video General Understanding Evaluation of Foundation Models
Authors: name: Liangzhe Yuan, name: Nitesh Bharadwaj Gundavarapu, name: Long Zhao, name: Hao Zhou, name: Yin Cui, name: Lu Jiang, name: Xuan Yang, name: Menglin Jia, name: Tobias Weyand, name: Luke Friedman, name: Mikhail Sirotenko, name: Huisheng Wang, name: Florian Schroff, name: Hartwig Adam, name: Ming-Hsuan Yang, name: Ting Liu, name: Boqing Gong
Published: 2023-07-06 17:47:52

We evaluate existing foundation models video understanding capabilities using
a carefully designed experiment protocol consisting of three hallmark tasks
(action recognition, temporal localization, and spatiotemporal localization),
eight datasets well received by the community, and four adaptation methods
tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
adapting to general video understanding tasks. Our main findings are as
follows. First, task-specialized models significantly outperform the six FMs
studied in this work, in sharp contrast to what FMs have achieved in natural
language and image understanding. Second,video-native FMs, whose pretraining
data contains the video modality, are generally better than image-native FMs in
classifying motion-rich videos, localizing actions in time, and understanding a
video of more than one action. Third, the video-native FMs can perform well on
video tasks under light adaptations to downstream tasks(e.g., freezing the FM
backbones), while image-native FMs win in full end-to-end finetuning. The first
two observations reveal the need and tremendous opportunities to conduct
research on video-focused FMs, and the last confirms that both tasks and
adaptation methods matter when it comes to the evaluation of FMs.
Focused Transformer: Contrastive Training for Context Scaling
Authors: name: Szymon Tworkowski, name: Konrad Staniszewski, name: Mikołaj Pacek, name: Yuhuai Wu, name: Henryk Michalewski, name: Piotr Miłoś
Published: 2023-07-06 17:52:10

Large language models have an exceptional capability to incorporate new
information in a contextual manner. However, the full potential of such an
approach is often restrained due to a limitation in the effective context
length. One solution to this issue is to endow an attention layer with access
to an external memory, which comprises of (key, value) pairs. Yet, as the
number of documents increases, the proportion of relevant keys to irrelevant
ones decreases, leading the model to focus more on the irrelevant keys. We
identify a significant challenge, dubbed the distraction issue, where keys
linked to different semantic values might overlap, making them hard to
distinguish. To tackle this problem, we introduce the Focused Transformer
(FoT), a technique that employs a training process inspired by contrastive
learning. This novel approach enhances the structure of the (key, value) space,
enabling an extension of the context length. Our method allows for fine-tuning
pre-existing, large-scale models to lengthen their effective context. This is
demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The
resulting models, which we name LongLLaMA, exhibit advancements in tasks
requiring a long context. We further illustrate that our LongLLaMA models
adeptly manage a 256 k context length for passkey retrieval.
Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts
Authors: name: Mounica Maddela, name: Megan Ung, name: Jing Xu, name: Andrea Madotto, name: Heather Foran, name: Y-Lan Boureau
Published: 2023-07-06 04:40:52

Many cognitive approaches to well-being, such as recognizing and reframing
unhelpful thoughts, have received considerable empirical support over the past
decades, yet still lack truly widespread adoption in self-help format. A
barrier to that adoption is a lack of adequately specific and diverse dedicated
practice material. This work examines whether current language models can be
leveraged to both produce a virtually unlimited quantity of practice material
illustrating standard unhelpful thought patterns matching specific given
contexts, and generate suitable positive reframing proposals. We propose
PATTERNREFRAME, a novel dataset of about 10k examples of thoughts containing
unhelpful thought patterns conditioned on a given persona, accompanied by about
27k positive reframes. By using this dataset to train and/or evaluate current
models, we show that existing models can already be powerful tools to help
generate an abundance of tailored practice material and hypotheses, with no or
minimal additional model training required.
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for
  Efficient LLM Inference
Authors: name: Luciano Del Corro, name: Allie Del Giorno, name: Sahaj Agarwal, name: Bin Yu, name: Ahmed Awadallah, name: Subhabrata Mukherjee
Published: 2023-07-05 19:59:09

Autoregressive large language models (LLMs) have made remarkable progress in
various natural language generation tasks. However, they incur high computation
cost and latency resulting from the autoregressive token-by-token generation.
To address this issue, several approaches have been proposed to reduce
computational cost using early-exit strategies. These strategies enable faster
text generation using reduced computation without applying the full computation
graph to each token. While existing token-level early exit methods show
promising results for online inference, they cannot be readily applied for
batch inferencing and Key-Value caching. This is because they have to wait
until the last token in a batch exits before they can stop computing. This
severely limits the practical application of such techniques. In this paper, we
propose a simple and effective token-level early exit method, SkipDecode,
designed to work seamlessly with batch inferencing and KV caching. It overcomes
prior constraints by setting up a singular exit point for every token in a
batch at each sequence position. It also guarantees a monotonic decrease in
exit points, thereby eliminating the need to recompute KV Caches for preceding
tokens. Rather than terminating computation prematurely as in prior works, our
approach bypasses lower to middle layers, devoting most of the computational
resources to upper layers, allowing later tokens to benefit from the compute
expenditure by earlier tokens. Our experimental results show that SkipDecode
can obtain 2x to 5x inference speedups with negligible regression across a
variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7
billion parameters, all the while being directly compatible with batching and
KV caching optimization techniques.
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
  Understanding
Authors: name: Jiabo Ye, name: Anwen Hu, name: Haiyang Xu, name: Qinghao Ye, name: Ming Yan, name: Yuhao Dan, name: Chenlin Zhao, name: Guohai Xu, name: Chenliang Li, name: Junfeng Tian, name: Qian Qi, name: Ji Zhang, name: Fei Huang
Published: 2023-07-04 11:28:07

Document understanding refers to automatically extract, analyze and
comprehend information from various types of digital documents, such as a web
page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl,
have demonstrated promising zero-shot capabilities in shallow OCR-free text
recognition, indicating their potential for OCR-free document understanding.
Nevertheless, without in-domain training, these models tend to ignore
fine-grained OCR features, such as sophisticated tables or large blocks of
text, which are essential for OCR-free document understanding. In this paper,
we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding.
Specifically, we first construct a instruction tuning dataset featuring a wide
range of visual-text understanding tasks. Then, we strengthen the OCR-free
document understanding ability by jointly train the model on language-only,
general vision-and-language, and document instruction tuning dataset with our
unified instruction tuning strategy. We also build an OCR-free document
instruction understanding evaluation set LLMDoc to better compare models'
capabilities on instruct compliance and document understanding. Experimental
results show that our model outperforms existing multi-modal models,
demonstrating its strong ability of document understanding. Besides, without
specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream
tasks. Our code, models, training data and evaluation set are available at
https://github.com/X-PLUG/mPLUG-DocOwl.
Elastic Decision Transformer
Authors: name: Yueh-Hua Wu, name: Xiaolong Wang, name: Masashi Hamaya
Published: 2023-07-05 17:58:21

This paper introduces Elastic Decision Transformer (EDT), a significant
advancement over the existing Decision Transformer (DT) and its variants.
Although DT purports to generate an optimal trajectory, empirical evidence
suggests it struggles with trajectory stitching, a process involving the
generation of an optimal or near-optimal trajectory from the best parts of a
set of sub-optimal trajectories. The proposed EDT differentiates itself by
facilitating trajectory stitching during action inference at test time,
achieved by adjusting the history length maintained in DT. Further, the EDT
optimizes the trajectory by retaining a longer history when the previous
trajectory is optimal and a shorter one when it is sub-optimal, enabling it to
"stitch" with a more optimal trajectory. Extensive experimentation demonstrates
EDT's ability to bridge the performance gap between DT-based and Q
Learning-based approaches. In particular, the EDT outperforms Q Learning-based
methods in a multi-task regime on the D4RL locomotion benchmark and Atari
games. Videos are available at: https://kristery.github.io/edt/
Physics-based Motion Retargeting from Sparse Inputs
Authors: name: Daniele Reda, name: Jungdam Won, name: Yuting Ye, name: Michiel van de Panne, name: Alexander Winkler
Published: 2023-07-04 21:57:05

Avatars are important to create interactive and immersive experiences in
virtual worlds. One challenge in animating these characters to mimic a user's
motion is that commercial AR/VR products consist only of a headset and
controllers, providing very limited sensor data of the user's pose. Another
challenge is that an avatar might have a different skeleton structure than a
human and the mapping between them is unclear. In this work we address both of
these challenges. We introduce a method to retarget motions in real-time from
sparse human sensor data to characters of various morphologies. Our method uses
reinforcement learning to train a policy to control characters in a physics
simulator. We only require human motion capture data for training, without
relying on artist-generated animations for each avatar. This allows us to use
large motion capture datasets to train general policies that can track unseen
users from real and sparse data in real-time. We demonstrate the feasibility of
our approach on three characters with different skeleton structure: a dinosaur,
a mouse-like creature and a human. We show that the avatar poses often match
the user surprisingly well, despite having no sensor information of the lower
body available. We discuss and ablate the important components in our
framework, specifically the kinematic retargeting step, the imitation, contact
and action reward as well as our asymmetric actor-critic observations. We
further explore the robustness of our method in a variety of settings including
unbalancing, dancing and sports motions.
EmoGen: Eliminating Subjective Bias in Emotional Music Generation
Authors: name: Chenfei Kang, name: Peiling Lu, name: Botao Yu, name: Xu Tan, name: Wei Ye, name: Shikun Zhang, name: Jiang Bian
Published: 2023-07-03 05:54:29

Music is used to convey emotions, and thus generating emotional music is
important in automatic music generation. Previous work on emotional music
generation directly uses annotated emotion labels as control signals, which
suffers from subjective bias: different people may annotate different emotions
on the same music, and one person may feel different emotions under different
situations. Therefore, directly mapping emotion labels to music sequences in an
end-to-end way would confuse the learning process and hinder the model from
generating music with general emotions. In this paper, we propose EmoGen, an
emotional music generation system that leverages a set of emotion-related music
attributes as the bridge between emotion and music, and divides the generation
into two stages: emotion-to-attribute mapping with supervised clustering, and
attribute-to-music generation with self-supervised learning. Both stages are
beneficial: in the first stage, the attribute values around the clustering
center represent the general emotions of these samples, which help eliminate
the impacts of the subjective bias of emotion labels; in the second stage, the
generation is completely disentangled from emotion labels and thus free from
the subjective bias. Both subjective and objective evaluations show that EmoGen
outperforms previous methods on emotion control accuracy and music quality
respectively, which demonstrate our superiority in generating emotional music.
Music samples generated by EmoGen are available via this
link:https://ai-muzic.github.io/emogen/, and the code is available at this
link:https://github.com/microsoft/muzic/.
MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
Authors: name: Jakob Drachmann Havtorn, name: Amelie Royer, name: Tijmen Blankevoort, name: Babak Ehteshami Bejnordi
Published: 2023-07-05 14:22:31

The input tokens to Vision Transformers carry little semantic meaning as they
are defined as regular equal-sized patches of the input image, regardless of
its content. However, processing uniform background areas of an image should
not necessitate as much compute as dense, cluttered areas. To address this
issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
method introduces a conditional gating mechanism that selects the optimal token
scale for every image region, such that the number of tokens is dynamically
determined per input. The proposed gating module is lightweight, agnostic to
the choice of transformer backbone, and trained within a few epochs (e.g., 20
epochs on ImageNet) with little training overhead. In addition, to enhance the
conditional behavior of the gate during training, we introduce a novel
generalization of the batch-shaping loss. We show that our gating module is
able to learn meaningful semantics despite operating locally at the coarse
patch-level. We validate MSViT on the tasks of classification and segmentation
where it leads to improved accuracy-complexity trade-off.
DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models
Authors: name: Chong Mou, name: Xintao Wang, name: Jiechong Song, name: Ying Shan, name: Jian Zhang
Published: 2023-07-05 16:43:56

Despite the ability of existing large-scale text-to-image (T2I) models to
generate high-quality images from detailed textual descriptions, they often
lack the ability to precisely edit the generated or real images. In this paper,
we propose a novel image editing method, DragonDiffusion, enabling Drag-style
manipulation on Diffusion models. Specifically, we construct classifier
guidance based on the strong correspondence of intermediate features in the
diffusion model. It can transform the editing signals into gradients via
feature correspondence loss to modify the intermediate representation of the
diffusion model. Based on this guidance strategy, we also build a multi-scale
guidance to consider both semantic and geometric alignment. Moreover, a
cross-branch self-attention is added to maintain the consistency between the
original image and the editing result. Our method, through an efficient design,
achieves various editing modes for the generated or real images, such as object
moving, object resizing, object appearance replacement, and content dragging.
It is worth noting that all editing and content preservation signals come from
the image itself, and the model does not require fine-tuning or additional
modules. Our source code will be available at
https://github.com/MC-E/DragonDiffusion.
SDXL: Improving Latent Diffusion Models for High-Resolution Image
  Synthesis
Authors: name: Dustin Podell, name: Zion English, name: Kyle Lacey, name: Andreas Blattmann, name: Tim Dockhorn, name: Jonas Müller, name: Joe Penna, name: Robin Rombach
Published: 2023-07-04 23:04:57

We present SDXL, a latent diffusion model for text-to-image synthesis.
Compared to previous versions of Stable Diffusion, SDXL leverages a three times
larger UNet backbone: The increase of model parameters is mainly due to more
attention blocks and a larger cross-attention context as SDXL uses a second
text encoder. We design multiple novel conditioning schemes and train SDXL on
multiple aspect ratios. We also introduce a refinement model which is used to
improve the visual fidelity of samples generated by SDXL using a post-hoc
image-to-image technique. We demonstrate that SDXL shows drastically improved
performance compared the previous versions of Stable Diffusion and achieves
results competitive with those of black-box state-of-the-art image generators.
In the spirit of promoting open research and fostering transparency in large
model training and evaluation, we provide access to code and model weights at
https://github.com/Stability-AI/generative-models