Orionnnn/hub-stats huggingface dataset:
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented
  Languages
Authors: name: Sebastian Ruder, name: Jonathan H. Clark, name: Alexander Gutkin, name: Mihir Kale, name: Min Ma, name: Massimo Nicosia, name: Shruti Rijhwani, name: Parker Riley, name: Jean-Michel A. Sarr, name: Xinyi Wang, name: John Wieting, name: Nitish Gupta, name: Anna Katanova, name: Christo Kirov, name: Dana L. Dickinson, name: Brian Roark, name: Bidisha Samanta, name: Connie Tao, name: David I. Adelani, name: Vera Axelrod, name: Isaac Caswell, name: Colin Cherry, name: Dan Garrette, name: Reeve Ingle, name: Melvin Johnson, name: Dmitry Panteleev, name: Partha Talukdar
Published: 2023-05-19 18:00:03

Data scarcity is a crucial issue for the development of highly multilingual
NLP systems. Yet for many under-represented languages (ULs) -- languages for
which NLP re-search is particularly far behind in meeting user needs -- it is
feasible to annotate small amounts of data. Motivated by this, we propose
XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather
than zero-shot; its focus on user-centric tasks -- tasks with broad adoption by
speakers of high-resource languages; and its focus on under-represented
languages where this scarce-data scenario tends to be most realistic. XTREME-UP
evaluates the capabilities of language models across 88 under-represented
languages over 9 key user-centric technologies including ASR, OCR, MT, and
information access tasks that are of general utility. We create new datasets
for OCR, autocomplete, semantic parsing, and transliteration, and build on and
refine existing datasets for other tasks. XTREME-UP provides methodology for
evaluating many modeling scenarios including text-only, multi-modal (vision,
audio, and text),supervised parameter tuning, and in-context learning. We
evaluate commonly used models on the benchmark. We release all code and scripts
to train and evaluate models
RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text
Authors: name: Wangchunshu Zhou, name: Yuchen Eleanor Jiang, name: Peng Cui, name: Tiannan Wang, name: Zhenxin Xiao, name: Yifan Hou, name: Ryan Cotterell, name: Mrinmaya Sachan
Published: 2023-05-22 17:58:10

The fixed-size context of Transformer makes GPT models incapable of
generating arbitrarily long text. In this paper, we introduce RecurrentGPT, a
language-based simulacrum of the recurrence mechanism in RNNs. RecurrentGPT is
built upon a large language model (LLM) such as ChatGPT and uses natural
language to simulate the Long Short-Term Memory mechanism in an LSTM. At each
timestep, RecurrentGPT generates a paragraph of text and updates its
language-based long-short term memory stored on the hard drive and the prompt,
respectively. This recurrence mechanism enables RecurrentGPT to generate texts
of arbitrary length without forgetting. Since human users can easily observe
and edit the natural language memories, RecurrentGPT is interpretable and
enables interactive generation of long text. RecurrentGPT is an initial step
towards next-generation computer-assisted writing systems beyond local editing
suggestions. In addition to producing AI-generated content (AIGC), we also
demonstrate the possibility of using RecurrentGPT as an interactive fiction
that directly interacts with consumers. We call this usage of generative models
by ``AI As Contents'' (AIAC), which we believe is the next form of conventional
AIGC. We further demonstrate the possibility of using RecurrentGPT to create
personalized interactive fiction that directly interacts with readers instead
of interacting with writers. More broadly, RecurrentGPT demonstrates the
utility of borrowing ideas from popular model designs in cognitive science and
deep learning for prompting LLMs. Our code is available at
https://github.com/aiwaves-cn/RecurrentGPT and an online demo is available at
https://www.aiwaves.org/recurrentgpt.
Training Diffusion Models with Reinforcement Learning
Authors: name: Kevin Black, name: Michael Janner, name: Yilun Du, name: Ilya Kostrikov, name: Sergey Levine
Published: 2023-05-22 17:57:41

Diffusion models are a class of flexible generative models trained with an
approximation to the log-likelihood objective. However, most use cases of
diffusion models are not concerned with likelihoods, but instead with
downstream objectives such as human-perceived image quality or drug
effectiveness. In this paper, we investigate reinforcement learning methods for
directly optimizing diffusion models for such objectives. We describe how
posing denoising as a multi-step decision-making problem enables a class of
policy gradient algorithms, which we refer to as denoising diffusion policy
optimization (DDPO), that are more effective than alternative reward-weighted
likelihood approaches. Empirically, DDPO is able to adapt text-to-image
diffusion models to objectives that are difficult to express via prompting,
such as image compressibility, and those derived from human feedback, such as
aesthetic quality. Finally, we show that DDPO can improve prompt-image
alignment using feedback from a vision-language model without the need for
additional data collection or human annotation.
ControlVideo: Training-free Controllable Text-to-Video Generation
Authors: name: Yabo Zhang, name: Yuxiang Wei, name: Dongsheng Jiang, name: Xiaopeng Zhang, name: Wangmeng Zuo, name: Qi Tian
Published: 2023-05-22 14:48:53

Text-driven diffusion models have unlocked unprecedented abilities in image
generation, whereas their video counterpart still lags behind due to the
excessive training cost of temporal modeling. Besides the training burden, the
generated videos also suffer from appearance inconsistency and structural
flickers, especially in long video synthesis. To address these challenges, we
design a training-free framework called ControlVideo to enable
natural and efficient text-to-video generation. ControlVideo, adapted from
ControlNet, leverages coarsely structural consistency from input motion
sequences, and introduces three modules to improve video generation. Firstly,
to ensure appearance coherence between frames, ControlVideo adds fully
cross-frame interaction in self-attention modules. Secondly, to mitigate the
flicker effect, it introduces an interleaved-frame smoother that employs frame
interpolation on alternated frames. Finally, to produce long videos
efficiently, it utilizes a hierarchical sampler that separately synthesizes
each short clip with holistic coherency. Empowered with these modules,
ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs
quantitatively and qualitatively. Notably, thanks to the efficient designs, it
generates both short and long videos within several minutes using one NVIDIA
2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
RWKV: Reinventing RNNs for the Transformer Era
Authors: name: Bo Peng, name: Eric Alcaide, name: Quentin Anthony, name: Alon Albalak, name: Samuel Arcadinho, name: Huanqi Cao, name: Xin Cheng, name: Michael Chung, name: Matteo Grella, name: Kranthi Kiran GV, name: Xuzheng He, name: Haowen Hou, name: Przemyslaw Kazienko, name: Jan Kocon, name: Jiaming Kong, name: Bartlomiej Koptyra, name: Hayden Lau, name: Krishna Sri Ipsit Mantri, name: Ferdinand Mom, name: Atsushi Saito, name: Xiangru Tang, name: Bolun Wang, name: Johan S. Wind, name: Stansilaw Wozniak, name: Ruichong Zhang, name: Zhenyuan Zhang, name: Qihang Zhao, name: Peng Zhou, name: Jian Zhu, name: Rui-Jie Zhu
Published: 2023-05-22 13:57:41

Transformers have revolutionized almost all natural language processing (NLP)
tasks but suffer from memory and computational complexity that scales
quadratically with sequence length. In contrast, recurrent neural networks
(RNNs) exhibit linear scaling in memory and computational requirements but
struggle to match the same performance as Transformers due to limitations in
parallelization and scalability. We propose a novel model architecture,
Receptance Weighted Key Value (RWKV), that combines the efficient
parallelizable training of Transformers with the efficient inference of RNNs.
Our approach leverages a linear attention mechanism and allows us to formulate
the model as either a Transformer or an RNN, which parallelizes computations
during training and maintains constant computational and memory complexity
during inference, leading to the first non-transformer architecture to be
scaled to tens of billions of parameters. Our experiments reveal that RWKV
performs on par with similarly sized Transformers, suggesting that future work
can leverage this architecture to create more efficient models. This work
presents a significant step towards reconciling the trade-offs between
computational efficiency and model performance in sequence processing tasks.
AudioToken: Adaptation of Text-Conditioned Diffusion Models for
  Audio-to-Image Generation
Authors: name: Guy Yariv, name: Itai Gat, name: Lior Wolf, name: Yossi Adi, name: Idan Schwartz
Published: 2023-05-22 14:02:44

In recent years, image generation has shown a great leap in performance,
where diffusion models play a central role. Although generating high-quality
images, such models are mainly conditioned on textual descriptions. This begs
the question: "how can we adopt such models to be conditioned on other
modalities?". In this paper, we propose a novel method utilizing latent
diffusion models trained for text-to-image-generation to generate images
conditioned on audio recordings. Using a pre-trained audio encoding model, the
proposed method encodes audio into a new token, which can be considered as an
adaptation layer between the audio and text representations. Such a modeling
paradigm requires a small number of trainable parameters, making the proposed
approach appealing for lightweight optimization. Results suggest the proposed
method is superior to the evaluated baseline methods, considering objective and
subjective metrics. Code and samples are available at:
https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set
  Operations
Authors: name: Chaitanya Malaviya, name: Peter Shaw, name: Ming-Wei Chang, name: Kenton Lee, name: Kristina Toutanova
Published: 2023-05-19 14:19:32

Formulating selective information needs results in queries that implicitly
specify set operations, such as intersection, union, and difference. For
instance, one might search for "shorebirds that are not sandpipers" or
"science-fiction films shot in England". To study the ability of retrieval
systems to meet such information needs, we construct QUEST, a dataset of 3357
natural language queries with implicit set operations, that map to a set of
entities corresponding to Wikipedia documents. The dataset challenges models to
match multiple constraints mentioned in queries with corresponding evidence in
documents and correctly perform various set operations. The dataset is
constructed semi-automatically using Wikipedia category names. Queries are
automatically composed from individual categories, then paraphrased and further
validated for naturalness and fluency by crowdworkers. Crowdworkers also assess
the relevance of entities based on their documents and highlight attribution of
query constraints to spans of document text. We analyze several modern
retrieval systems, finding that they often struggle on such queries. Queries
involving negation and conjunction are particularly challenging and systems are
further challenged with combinations of these operations.
How Does Generative Retrieval Scale to Millions of Passages?
Authors: name: Ronak Pradeep, name: Kai Hui, name: Jai Gupta, name: Adam D. Lelkes, name: Honglei Zhuang, name: Jimmy Lin, name: Donald Metzler, name: Vinh Q. Tran
Published: 2023-05-19 17:33:38

Popularized by the Differentiable Search Index, the emerging paradigm of
generative retrieval re-frames the classic information retrieval problem into a
sequence-to-sequence modeling task, forgoing external indices and encoding an
entire document corpus within a single Transformer. Although many different
approaches have been proposed to improve the effectiveness of generative
retrieval, they have only been evaluated on document corpora on the order of
100k in size. We conduct the first empirical study of generative retrieval
techniques across various corpus scales, ultimately scaling up to the entire MS
MARCO passage ranking task with a corpus of 8.8M passages and evaluating model
sizes up to 11B parameters. We uncover several findings about scaling
generative retrieval to millions of passages; notably, the central importance
of using synthetic queries as document representations during indexing, the
ineffectiveness of existing proposed architecture modifications when accounting
for compute cost, and the limits of naively scaling model parameters with
respect to retrieval performance. While we find that generative retrieval is
competitive with state-of-the-art dual encoders on small corpora, scaling to
millions of passages remains an important and unsolved challenge. We believe
these findings will be valuable for the community to clarify the current state
of generative retrieval, highlight the unique challenges, and inspire new
research directions.
Pengi: An Audio Language Model for Audio Tasks
Authors: name: Soham Deshmukh, name: Benjamin Elizalde, name: Rita Singh, name: Huaming Wang
Published: 2023-05-19 17:20:56

In the domain of audio processing, Transfer Learning has facilitated the rise
of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches
have led to the development of versatile models capable of tackling a wide
array of tasks, while delivering state-of-the-art performance. However, current
models inherently lack the capacity to produce the requisite language for
open-ended tasks, such as Audio Captioning or Audio Question & Answering. We
introduce Pengi, a novel Audio Language Model that leverages Transfer Learning
by framing all audio tasks as text-generation tasks. It takes as input, an
audio recording, and text, and generates free-form text as output. The input
audio is represented as a sequence of continuous embeddings by an audio
encoder. A text encoder does the same for the corresponding text input. Both
sequences are combined as a prefix to prompt a pre-trained frozen language
model. The unified architecture of Pengi enables open-ended tasks and
close-ended tasks without any additional fine-tuning or task-specific
extensions. When evaluated on 22 downstream tasks, our approach yields
state-of-the-art performance in several of them. Our results show that
connecting language models with audio models is a major step towards
general-purpose audio understanding
Scaling laws for language encoding models in fMRI
Authors: name: Richard Antonello, name: Aditya Vaidya, name: Alexander G. Huth
Published: 2023-05-19 17:53:03

Representations from transformer-based unidirectional language models are
known to be effective at predicting brain responses to natural language.
However, most studies comparing language models to brains have used GPT-2 or
similarly sized language models. Here we tested whether larger open-source
models such as those from the OPT and LLaMA families are better at predicting
brain responses recorded using fMRI. Mirroring scaling results from other
contexts, we found that brain prediction performance scales log-linearly with
model size from 125M to 30B parameter models, with ~15% increased encoding
performance as measured by correlation with a held-out test set across 3
subjects. Similar log-linear behavior was observed when scaling the size of the
fMRI training set. We also characterized scaling for acoustic encoding models
that use HuBERT, WavLM, and Whisper, and we found comparable improvements with
model size. A noise ceiling analysis of these large, high-performance encoding
models showed that performance is nearing the theoretical maximum for brain
areas such as the precuneus and higher auditory cortex. These results suggest
that increasing scale in both models and data will yield incredibly effective
models of language processing in the brain, enabling better scientific
understanding as well as applications such as decoding.
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Authors: name: Hiroki Furuta, name: Ofir Nachum, name: Kuang-Huei Lee, name: Yutaka Matsuo, name: Shixiang Shane Gu, name: Izzeddin Gur
Published: 2023-05-19 17:44:34

The progress of autonomous web navigation has been hindered by the dependence
on billions of exploratory interactions via online reinforcement learning, and
domain-specific model designs that make it difficult to leverage generalization
from rich out-of-domain data. In this work, we study data-driven offline
training for web agents with vision-language foundation models. We propose an
instruction-following multimodal agent, WebGUM, that observes both webpage
screenshots and HTML pages and outputs web navigation actions, such as click
and type. WebGUM is trained by jointly finetuning an instruction-finetuned
language model and a vision transformer on a large corpus of demonstrations. We
empirically demonstrate this recipe improves the agent's ability of grounded
visual perception, HTML comprehension and multi-step reasoning, outperforming
prior works by a significant margin. On the MiniWoB benchmark, we improve over
the previous best offline methods by more than 31.9%, being close to reaching
online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
achieves superior performance to the existing SoTA, PaLM-540B. We also collect
347K high-quality demonstrations using our trained models, 38 times larger than
prior work, and make them available to promote future research in this
direction.
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage
  Leveraging Generative Models
Authors: name: Akshita Jha, name: Aida Davani, name: Chandan K. Reddy, name: Shachi Dave, name: Vinodkumar Prabhakaran, name: Sunipa Dev
Published: 2023-05-19 17:30:19

Stereotype benchmark datasets are crucial to detect and mitigate social
stereotypes about groups of people in NLP models. However, existing datasets
are limited in size and coverage, and are largely restricted to stereotypes
prevalent in the Western society. This is especially problematic as language
technologies gain hold across the globe. To address this gap, we present
SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative
capabilities of large language models such as PaLM, and GPT-3, and leveraging a
globally diverse rater pool to validate the prevalence of those stereotypes in
society. SeeGULL is in English, and contains stereotypes about identity groups
spanning 178 countries across 8 different geo-political regions across 6
continents, as well as state-level identities within the US and India. We also
include fine-grained offensiveness scores for different stereotypes and
demonstrate their global disparities. Furthermore, we include comparative
annotations about the same groups by annotators living in the region vs. those
that are based in North America, and demonstrate that within-region stereotypes
about groups differ from those prevalent in North America. CONTENT WARNING:
This paper contains stereotype examples that may be offensive.
Cross-Lingual Supervision improves Large Language Models Pre-training
Authors: name: Andrea Schioppa, name: Xavier Garcia, name: Orhan Firat
Published: 2023-05-19 16:14:07

The recent rapid progress in pre-training Large Language Models has relied on
using self-supervised language modeling objectives like next token prediction
or span corruption. On the other hand, Machine Translation Systems are mostly
trained using cross-lingual supervision that requires aligned data between
source and target languages. We demonstrate that pre-training Large Language
Models on a mixture of a self-supervised Language Modeling objective and the
supervised Machine Translation objective, therefore including cross-lingual
parallel data during pre-training, yields models with better in-context
learning abilities. As pre-training is a very resource-intensive process and a
grid search on the best mixing ratio between the two objectives is
prohibitively expensive, we propose a simple yet effective strategy to learn it
during pre-training.
Controlling the Extraction of Memorized Data from Large Language Models
  via Prompt-Tuning
Authors: name: Mustafa Safa Ozdayi, name: Charith Peris, name: Jack FitzGerald, name: Christophe Dupuy, name: Jimit Majmudar, name: Haidar Khan, name: Rahil Parikh, name: Rahul Gupta
Published: 2023-05-19 15:45:29

Large Language Models (LLMs) are known to memorize significant portions of
their training data. Parts of this memorized content have been shown to be
extractable by simply querying the model, which poses a privacy risk. We
present a novel approach which uses prompt-tuning to control the extraction
rates of memorized content in LLMs. We present two prompt training strategies
to increase and decrease extraction rates, which correspond to an attack and a
defense, respectively. We demonstrate the effectiveness of our techniques by
using models from the GPT-Neo family on a public benchmark. For the 1.3B
parameter GPT-Neo model, our attack yields a 9.3 percentage point increase in
extraction rate compared to our baseline. Our defense can be tuned to achieve
different privacy-utility trade-offs by a user-specified hyperparameter. We
achieve an extraction rate reduction of up to 97.7% relative to our baseline,
with a perplexity increase of 16.9%.
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive
  Critiquing
Authors: name: Zhibin Gou, name: Zhihong Shao, name: Yeyun Gong, name: Yelong Shen, name: Yujiu Yang, name: Nan Duan, name: Weizhu Chen
Published: 2023-05-19 15:19:44

Recent developments in large language models (LLMs) have been impressive.
However, these models sometimes show inconsistencies and problematic behavior,
such as hallucinating facts, generating flawed code, or creating offensive and
toxic content. Unlike these models, humans typically utilize external tools to
cross-check and refine their initial content, like using a search engine for
fact-checking, or a code interpreter for debugging. Inspired by this
observation, we introduce a framework called CRITIC that allows LLMs, which are
essentially "black boxes" to validate and progressively amend their own outputs
in a manner similar to human interaction with tools. More specifically,
starting with an initial output, CRITIC interacts with appropriate tools to
evaluate certain aspects of the text, and then revises the output based on the
feedback obtained during this validation process. Comprehensive evaluations
involving free-form question answering, mathematical program synthesis, and
toxicity reduction demonstrate that CRITIC consistently enhances the
performance of LLMs. Meanwhile, our research highlights the crucial importance
of external feedback in promoting the ongoing self-improvement of LLMs.