Orionnnn/hub-stats huggingface dataset:
PEARL: Prompting Large Language Models to Plan and Execute Actions Over
  Long Documents
Authors: name: Simeng Sun, name: Yang Liu, name: Shuohang Wang, name: Chenguang Zhu, name: Mohit Iyyer
Published: 2023-05-23 23:06:04

Strategies such as chain-of-thought prompting improve the performance of
large language models (LLMs) on complex reasoning tasks by decomposing input
examples into intermediate steps. However, it remains unclear how to apply such
methods to reason over long input documents, in which both the decomposition
and the output of each intermediate step are non-trivial to obtain. In this
work, we propose PEARL, a prompting framework to improve reasoning over long
documents, which consists of three stages: action mining, plan formulation, and
plan execution. More specifically, given a question about a long document,
PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE,
FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain
the answer. Each stage of PEARL is implemented via zero-shot or few-shot
prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate
PEARL on a challenging subset of the QuALITY dataset, which contains questions
that require complex reasoning over long narrative texts. PEARL outperforms
zero-shot and chain-of-thought prompting on this dataset, and ablation
experiments show that each stage of PEARL is critical to its performance.
Overall, PEARL is a first step towards leveraging LLMs to reason over long
documents.
LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
Authors: name: Philippe Laban, name: Wojciech Kryściński, name: Divyansh Agarwal, name: Alexander R. Fabbri, name: Caiming Xiong, name: Shafiq Joty, name: Chien-Sheng Wu
Published: 2023-05-23 21:50:06

With the recent appearance of LLMs in practical settings, having methods that
can effectively detect factual inconsistencies is crucial to reduce the
propagation of misinformation and improve trust in model outputs. When testing
on existing factual consistency benchmarks, we find that a few large language
models (LLMs) perform competitively on classification benchmarks for factual
inconsistency detection compared to traditional non-LLM methods. However, a
closer analysis reveals that most LLMs fail on more complex formulations of the
task and exposes issues with existing evaluation benchmarks, affecting
evaluation precision. To address this, we propose a new protocol for
inconsistency detection benchmark creation and implement it in a 10-domain
benchmark called SummEdits. This new benchmark is 20 times more cost-effective
per sample than previous benchmarks and highly reproducible, as we estimate
inter-annotator agreement at about 0.9. Most LLMs struggle on SummEdits, with
performance close to random chance. The best-performing model, GPT-4, is still
8\% below estimated human performance, highlighting the gaps in LLMs' ability
to reason about facts and detect inconsistencies when they occur.
Control-A-Video: Controllable Text-to-Video Generation with Diffusion
  Models
Authors: name: Weifeng Chen, name: Jie Wu, name: Pan Xie, name: Hefeng Wu, name: Jiashi Li, name: Xin Xia, name: Xuefeng Xiao, name: Liang Lin
Published: 2023-05-23 09:03:19

This paper presents a controllable text-to-video (T2V) diffusion model, named
Video-ControlNet, that generates videos conditioned on a sequence of control
signals, such as edge or depth maps. Video-ControlNet is built on a pre-trained
conditional text-to-image (T2I) diffusion model by incorporating a
spatial-temporal self-attention mechanism and trainable temporal layers for
efficient cross-frame modeling. A first-frame conditioning strategy is proposed
to facilitate the model to generate videos transferred from the image domain as
well as arbitrary-length videos in an auto-regressive manner. Moreover,
Video-ControlNet employs a novel residual-based noise initialization strategy
to introduce motion prior from an input video, producing more coherent videos.
With the proposed architecture and strategies, Video-ControlNet can achieve
resource-efficient convergence and generate superior quality and consistent
videos with fine-grained control. Extensive experiments demonstrate its success
in various video generative tasks such as video editing and video style
transfer, outperforming previous methods in terms of consistency and quality.
Project Page: https://controlavideo.github.io/
Enhancing Detail Preservation for Customized Text-to-Image Generation: A
  Regularization-Free Approach
Authors: name: Yufan Zhou, name: Ruiyi Zhang, name: Tong Sun, name: Jinhui Xu
Published: 2023-05-23 01:14:53

Recent text-to-image generation models have demonstrated impressive
capability of generating text-aligned images with high fidelity. However,
generating images of novel concept provided by the user input image is still a
challenging task. To address this problem, researchers have been exploring
various methods for customizing pre-trained text-to-image generation models.
Currently, most existing methods for customizing pre-trained text-to-image
generation models involve the use of regularization techniques to prevent
over-fitting. While regularization will ease the challenge of customization and
leads to successful content creation with respect to text guidance, it may
restrict the model capability, resulting in the loss of detailed information
and inferior performance. In this work, we propose a novel framework for
customized text-to-image generation without the use of regularization.
Specifically, our proposed framework consists of an encoder network and a novel
sampling method which can tackle the over-fitting problem without the use of
regularization. With the proposed framework, we are able to customize a
large-scale text-to-image generation model within half a minute on single GPU,
with only one image provided by the user. We demonstrate in experiments that
our proposed framework outperforms existing methods, and preserves more
fine-grained details.
Enhancing Chat Language Models by Scaling High-quality Instructional
  Conversations
Authors: name: Ning Ding, name: Yulin Chen, name: Bokai Xu, name: Yujia Qin, name: Zhi Zheng, name: Shengding Hu, name: Zhiyuan Liu, name: Maosong Sun, name: Bowen Zhou
Published: 2023-05-23 16:49:14

Fine-tuning on instruction data has been widely validated as an effective
practice for implementing chat language models like ChatGPT. Scaling the
diversity and quality of such data, although straightforward, stands a great
chance of leading to improved performance. This paper aims to improve the upper
bound of open-source models further. We first provide a systematically
designed, diverse, informative, large-scale dataset of instructional
conversations, UltraChat, which does not involve human queries. Our objective
is to capture the breadth of interactions that a human might have with an AI
assistant and employs a comprehensive framework to generate multi-turn
conversation iteratively. UltraChat contains 1.5 million high-quality
multi-turn dialogues and covers a wide range of topics and instructions. Our
statistical analysis of UltraChat reveals its superiority in various key
metrics, including scale, average length, diversity, coherence, etc.,
solidifying its position as a leading open-source dataset. Building upon
UltraChat, we fine-tune a LLaMA model to create a powerful conversational
model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently
outperforms other open-source models, including Vicuna, the previously
recognized state-of-the-art open-source model. The dataset and the model will
be publicly released\url{https://github.com/thunlp/UltraChat}.
QLoRA: Efficient Finetuning of Quantized LLMs
Authors: name: Tim Dettmers, name: Artidoro Pagnoni, name: Ari Holtzman, name: Luke Zettlemoyer
Published: 2023-05-23 17:50:33

We present QLoRA, an efficient finetuning approach that reduces memory usage
enough to finetune a 65B parameter model on a single 48GB GPU while preserving
full 16-bit finetuning task performance. QLoRA backpropagates gradients through
a frozen, 4-bit quantized pretrained language model into Low Rank
Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all
previous openly released models on the Vicuna benchmark, reaching 99.3% of the
performance level of ChatGPT while only requiring 24 hours of finetuning on a
single GPU. QLoRA introduces a number of innovations to save memory without
sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is
information theoretically optimal for normally distributed weights (b) double
quantization to reduce the average memory footprint by quantizing the
quantization constants, and (c) paged optimziers to manage memory spikes. We
use QLoRA to finetune more than 1,000 models, providing a detailed analysis of
instruction following and chatbot performance across 8 instruction datasets,
multiple model types (LLaMA, T5), and model scales that would be infeasible to
run with regular finetuning (e.g. 33B and 65B parameter models). Our results
show that QLoRA finetuning on a small high-quality dataset leads to
state-of-the-art results, even when using smaller models than the previous
SoTA. We provide a detailed analysis of chatbot performance based on both human
and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable
alternative to human evaluation. Furthermore, we find that current chatbot
benchmarks are not trustworthy to accurately evaluate the performance levels of
chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to
ChatGPT. We release all of our models and code, including CUDA kernels for
4-bit training.
Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks
Authors: name: Tiedong Liu, name: Bryan Kian Hsiang Low
Published: 2023-05-23 16:20:30

We introduce Goat, a fine-tuned LLaMA model that significantly outperforms
GPT-4 on a range of arithmetic tasks. Fine-tuned on a synthetically generated
dataset, Goat achieves state-of-the-art performance on BIG-bench arithmetic
sub-task. In particular, the zero-shot Goat-7B matches or even surpasses the
accuracy achieved by the few-shot PaLM-540B. Surprisingly, Goat can achieve
near-perfect accuracy on large-number addition and subtraction through
supervised fine-tuning only, which is almost impossible with previous
pretrained language models, such as Bloom, OPT, GPT-NeoX, etc. We attribute
Goat's exceptional performance to LLaMA's consistent tokenization of numbers.
To tackle more challenging tasks like large-number multiplication and division,
we propose an approach that classifies tasks based on their learnability, and
subsequently decomposes unlearnable tasks, such as multi-digit multiplication
and division, into a series of learnable tasks by leveraging basic arithmetic
principles. We thoroughly examine the performance of our model, offering a
comprehensive evaluation of the effectiveness of our proposed decomposition
steps. Additionally, Goat-7B can be easily trained using LoRA on a 24GB VRAM
GPU, facilitating reproducibility for other researchers. We release our model,
dataset, and the Python script for dataset generation.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Authors: name: Viorica Pătrăucean, name: Lucas Smaira, name: Ankush Gupta, name: Adrià Recasens Continente, name: Larisa Markeeva, name: Dylan Banarse, name: Skanda Koppula, name: Joseph Heyward, name: Mateusz Malinowski, name: Yi Yang, name: Carl Doersch, name: Tatiana Matejovicova, name: Yury Sulsky, name: Antoine Miech, name: Alex Frechette, name: Hanna Klimczak, name: Raphael Koster, name: Junlin Zhang, name: Stephanie Winkler, name: Yusuf Aytar, name: Simon Osindero, name: Dima Damen, name: Andrew Zisserman, name: João Carreira
Published: 2023-05-23 07:54:37

We propose a novel multimodal video benchmark - the Perception Test - to
evaluate the perception and reasoning skills of pre-trained multimodal models
(e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus
on computational tasks (e.g. classification, detection or tracking), the
Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and
types of reasoning (descriptive, explanatory, predictive, counterfactual)
across video, audio, and text modalities, to provide a comprehensive and
efficient evaluation tool. The benchmark probes pre-trained models for their
transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
For these purposes, the Perception Test introduces 11.6k real-world videos, 23s
average length, designed to show perceptually interesting situations, filmed by
around 100 participants worldwide. The videos are densely annotated with six
types of labels (multiple-choice and grounded video question-answers, object
and point tracks, temporal action and sound segments), enabling both language
and non-language evaluations. The fine-tuning and validation splits of the
benchmark are publicly available (CC-BY license), in addition to a challenge
server with a held-out test split. Human baseline results compared to
state-of-the-art video QA models show a significant gap in performance (91.4%
vs 43.6%), suggesting that there is significant room for improvement in
multimodal video understanding.
  Dataset, baselines code, and challenge server are available at
https://github.com/deepmind/perception_test
Aligning Large Language Models through Synthetic Feedback
Authors: name: Sungdong Kim, name: Sanghwan Bae, name: Jamin Shin, name: Soyoung Kang, name: Donghyun Kwak, name: Kang Min Yoo, name: Minjoon Seo
Published: 2023-05-23 06:41:16

Aligning large language models (LLMs) to human values has become increasingly
important as it enables sophisticated steering of LLMs, e.g., making them
follow given instructions while keeping them less toxic. However, it requires a
significant amount of human demonstrations and feedback. Recently, open-sourced
models have attempted to replicate the alignment learning process by distilling
data from already aligned LLMs like InstructGPT or ChatGPT. While this process
reduces human efforts, constructing these datasets has a heavy dependency on
the teacher models. In this work, we propose a novel framework for alignment
learning with almost no human labor and no dependency on pre-aligned LLMs.
First, we perform reward modeling (RM) with synthetic feedback by contrasting
responses from vanilla LLMs with various sizes and prompts. Then, we use the RM
for simulating high-quality demonstrations to train a supervised policy and for
further optimizing the model with reinforcement learning. Our resulting model,
Aligned Language Model with Synthetic Training dataset (ALMoST), outperforms
open-sourced models, including Alpaca, Dolly, and OpenAssistant, which are
trained on the outputs of InstructGPT or human-annotated instructions. Our
7B-sized model outperforms the 12-13B models in the A/B tests using GPT-4 as
the judge with about 75% winning rate on average.
How Language Model Hallucinations Can Snowball
Authors: name: Muru Zhang, name: Ofir Press, name: William Merrill, name: Alisa Liu, name: Noah A. Smith
Published: 2023-05-22 23:14:28

A major risk of using language models in practical applications is their
tendency to hallucinate incorrect statements. Hallucinations are often
attributed to knowledge gaps in LMs, but we hypothesize that in some cases,
when justifying previously generated hallucinations, LMs output false claims
that they can separately recognize as incorrect. We construct three
question-answering datasets where ChatGPT and GPT-4 often state an incorrect
answer and offer an explanation with at least one incorrect claim. Crucially,
we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes,
respectively. We refer to this phenomenon as hallucination snowballing: an LM
over-commits to early mistakes, leading to more mistakes that it otherwise
would not make.
Textually Pretrained Speech Language Models
Authors: name: Michael Hassid, name: Tal Remez, name: Tu Anh Nguyen, name: Itai Gat, name: Alexis Conneau, name: Felix Kreuk, name: Jade Copet, name: Alexandre Defossez, name: Gabriel Synnaeve, name: Emmanuel Dupoux, name: Roy Schwartz, name: Yossi Adi
Published: 2023-05-22 13:12:16

Speech language models (SpeechLMs) process and generate acoustic data only,
without textual supervision. In this work, we propose TWIST, a method for
training SpeechLMs using a warm-start from a pretrained textual language
models. We show using both automatic and human evaluations that TWIST
outperforms a cold-start SpeechLM across the board. We empirically analyze the
effect of different model design choices such as the speech tokenizer, the
pretrained textual model, and the dataset size. We find that model and dataset
scale both play an important role in constructing better-performing SpeechLMs.
Based on our observations, we present the largest (to the best of our
knowledge) SpeechLM both in terms of number of parameters and training data. We
additionally introduce two spoken versions of the StoryCloze textual benchmark
to further improve model evaluation and advance future research in the field.
Speech samples can be found on our website:
https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
Augmenting Autotelic Agents with Large Language Models
Authors: name: Cédric Colas, name: Laetitia Teodorescu, name: Pierre-Yves Oudeyer, name: Xingdi Yuan, name: Marc-Alexandre Côté
Published: 2023-05-21 15:42:41

Humans learn to master open-ended repertoires of skills by imagining and
practicing their own goals. This autotelic learning process, literally the
pursuit of self-generated (auto) goals (telos), becomes more and more
open-ended as the goals become more diverse, abstract and creative. The
resulting exploration of the space of possible skills is supported by an
inter-individual exploration: goal representations are culturally evolved and
transmitted across individuals, in particular using language. Current
artificial agents mostly rely on predefined goal representations corresponding
to goal spaces that are either bounded (e.g. list of instructions), or
unbounded (e.g. the space of possible visual inputs) but are rarely endowed
with the ability to reshape their goal representations, to form new
abstractions or to imagine creative goals. In this paper, we introduce a
language model augmented autotelic agent (LMA3) that leverages a pretrained
language model (LM) to support the representation, generation and learning of
diverse, abstract, human-relevant goals. The LM is used as an imperfect model
of human cultural transmission; an attempt to capture aspects of humans'
common-sense, intuitive physics and overall interests. Specifically, it
supports three key components of the autotelic architecture: 1)~a relabeler
that describes the goals achieved in the agent's trajectories, 2)~a goal
generator that suggests new high-level goals along with their decomposition
into subgoals the agent already masters, and 3)~reward functions for each of
these goals. Without relying on any hand-coded goal representations, reward
functions or curriculum, we show that LMA3 agents learn to master a large
diversity of skills in a task-agnostic text-based environment.
CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code
  Authoring
Authors: name: Vijayaraghavan Murali, name: Chandra Maddila, name: Imad Ahmad, name: Michael Bolin, name: Daniel Cheng, name: Negar Ghorbani, name: Renuka Fernandez, name: Nachiappan Nagappan
Published: 2023-05-20 00:45:15

The rise of large language models (LLMs) has unlocked various applications of
this technology in software development. In particular, generative LLMs have
been shown to effectively power AI-based code authoring tools that can suggest
entire statements or blocks of code during code authoring. In this paper we
present CodeCompose, an AI-assisted code authoring tool developed and deployed
at Meta internally. CodeCompose is based on the InCoder LLM that merges
generative capabilities with bi-directionality. We have scaled up CodeCompose
to serve tens of thousands of developers at Meta, across 10+ programming
languages and several coding surfaces.
  We discuss unique challenges in terms of user experience and metrics that
arise when deploying such tools in large-scale industrial settings. We present
our experience in making design decisions about the model and system
architecture for CodeCompose that addresses these challenges. Finally, we
present metrics from our large-scale deployment of CodeCompose that shows its
impact on Meta's internal code authoring experience over a 15-day time window,
where 4.5 million suggestions were made by CodeCompose. Quantitative metrics
reveal that (i) CodeCompose has an acceptance rate of 22% across several
languages, and (ii) 8% of the code typed by users of CodeCompose is through
accepting code suggestions from CodeCompose. Qualitative feedback indicates an
overwhelming 91.5% positive reception for CodeCompose. In addition to assisting
with code authoring, CodeCompose is also introducing other positive side
effects such as encouraging developers to generate more in-code documentation,
helping them with the discovery of new APIs, etc.
OPT-R: Exploring the Role of Explanations in Finetuning and Prompting
  for Reasoning Skills of Large Language Models
Authors: name: Badr AlKhamissi, name: Siddharth Verma, name: Ping Yu, name: Zhijing Jin, name: Asli Celikyilmaz, name: Mona Diab
Published: 2023-05-19 20:58:22

In this paper, we conduct a thorough investigation into the reasoning
capabilities of Large Language Models (LLMs), focusing specifically on the Open
Pretrained Transformers (OPT) models as a representative of such models. Our
study entails finetuning three different sizes of OPT on a carefully curated
reasoning corpus, resulting in two sets of finetuned models: OPT-R, finetuned
without explanations, and OPT-RE, finetuned with explanations. We then evaluate
all models on 57 out-of-domain tasks drawn from the SUPER-NATURALINSTRUCTIONS
benchmark, covering 26 distinct reasoning skills, utilizing three prompting
techniques. Through a comprehensive grid of 27 configurations and 6,156 test
evaluations, we investigate the dimensions of finetuning, prompting, and scale
to understand the role of explanations on different reasoning skills. Our
findings reveal that having explanations in the fewshot exemplar has no
significant impact on the model's performance when the model is finetuned,
while positively affecting the non-finetuned counterpart. Moreover, we observe
a slight yet consistent increase in classification accuracy as we incorporate
explanations during prompting and finetuning, respectively. Finally, we offer
insights on which skills benefit the most from incorporating explanations
during finetuning and prompting, such as Numerical (+20.4%) and Analogical
(+13.9%) reasoning, as well as skills that exhibit negligible or negative
effects.