Orionnnn/hub-stats huggingface dataset:
OlaGPT: Empowering LLMs With Human-like Problem-Solving Abilities
Authors: name: Yuanzhen Xie, name: Tao Xie, name: Mingxiong Lin, name: WenTao Wei, name: Chenglin Li, name: Beibei Kong, name: Lei Chen, name: Chengxiang Zhuo, name: Bo Hu, name: Zang Li
Published: 2023-05-23 09:36:51

In most current research, large language models (LLMs) are able to perform
reasoning tasks by generating chains of thought through the guidance of
specific prompts. However, there still exists a significant discrepancy between
their capability in solving complex reasoning problems and that of humans. At
present, most approaches focus on chains of thought (COT) and tool use, without
considering the adoption and application of human cognitive frameworks. It is
well-known that when confronting complex reasoning challenges, humans typically
employ various cognitive abilities, and necessitate interaction with all
aspects of tools, knowledge, and the external environment information to
accomplish intricate tasks. This paper introduces a novel intelligent
framework, referred to as OlaGPT. OlaGPT carefully studied a cognitive
architecture framework, and propose to simulate certain aspects of human
cognition. The framework involves approximating different cognitive modules,
including attention, memory, reasoning, learning, and corresponding scheduling
and decision-making mechanisms. Inspired by the active learning mechanism of
human beings, it proposes a learning unit to record previous mistakes and
expert opinions, and dynamically refer to them to strengthen their ability to
solve similar problems. The paper also outlines common effective reasoning
frameworks for human problem-solving and designs Chain-of-Thought (COT)
templates accordingly. A comprehensive decision-making mechanism is also
proposed to maximize model accuracy. The efficacy of OlaGPT has been
stringently evaluated on multiple reasoning datasets, and the experimental
outcomes reveal that OlaGPT surpasses state-of-the-art benchmarks,
demonstrating its superior performance. Our implementation of OlaGPT is
available on GitHub: https://github.com/oladata-team/OlaGPT.
Break-A-Scene: Extracting Multiple Concepts from a Single Image
Authors: name: Omri Avrahami, name: Kfir Aberman, name: Ohad Fried, name: Daniel Cohen-Or, name: Dani Lischinski
Published: 2023-05-25 17:59:04

Text-to-image model personalization aims to introduce a user-provided concept
to the model, allowing its synthesis in diverse contexts. However, current
methods primarily focus on the case of learning a single concept from multiple
images with variations in backgrounds and poses, and struggle when adapted to a
different scenario. In this work, we introduce the task of textual scene
decomposition: given a single image of a scene that may contain several
concepts, we aim to extract a distinct text token for each concept, enabling
fine-grained control over the generated scenes. To this end, we propose
augmenting the input image with masks that indicate the presence of target
concepts. These masks can be provided by the user or generated automatically by
a pre-trained segmentation model. We then present a novel two-phase
customization process that optimizes a set of dedicated textual embeddings
(handles), as well as the model weights, striking a delicate balance between
accurately capturing the concepts and avoiding overfitting. We employ a masked
diffusion loss to enable handles to generate their assigned concepts,
complemented by a novel loss on cross-attention maps to prevent entanglement.
We also introduce union-sampling, a training strategy aimed to improve the
ability of combining multiple concepts in generated images. We use several
automatic metrics to quantitatively compare our method against several
baselines, and further affirm the results using a user study. Finally, we
showcase several applications of our method. Project page is available at:
https://omriavrahami.com/break-a-scene/
Randomized Positional Encodings Boost Length Generalization of
  Transformers
Authors: name: Anian Ruoss, name: Grégoire Delétang, name: Tim Genewein, name: Jordi Grau-Moya, name: Róbert Csordás, name: Mehdi Bennani, name: Shane Legg, name: Joel Veness
Published: 2023-05-26 11:47:52

Transformers have impressive generalization capabilities on tasks with a
fixed context length. However, they fail to generalize to sequences of
arbitrary length, even for seemingly simple tasks such as duplicating a string.
Moreover, simply training on longer sequences is inefficient due to the
quadratic computation complexity of the global attention mechanism. In this
work, we demonstrate that this failure mode is linked to positional encodings
being out-of-distribution for longer sequences (even for relative encodings)
and introduce a novel family of positional encodings that can overcome this
problem. Concretely, our randomized positional encoding scheme simulates the
positions of longer sequences and randomly selects an ordered subset to fit the
sequence's length. Our large-scale empirical evaluation of 6000 models across
15 algorithmic reasoning tasks shows that our method allows Transformers to
generalize to sequences of unseen length (increasing test accuracy by 12.0% on
average).
A Closer Look at In-Context Learning under Distribution Shifts
Authors: name: Kartik Ahuja, name: David Lopez-Paz
Published: 2023-05-26 07:47:21

In-context learning, a capability that enables a model to learn from input
examples on the fly without necessitating weight updates, is a defining
characteristic of large language models. In this work, we follow the setting
proposed in (Garg et al., 2022) to better understand the generality and
limitations of in-context learning from the lens of the simple yet fundamental
task of linear regression. The key question we aim to address is: Are
transformers more adept than some natural and simpler architectures at
performing in-context learning under varying distribution shifts? To compare
transformers, we propose to use a simple architecture based on set-based
Multi-Layer Perceptrons (MLPs). We find that both transformers and set-based
MLPs exhibit in-context learning under in-distribution evaluations, but
transformers more closely emulate the performance of ordinary least squares
(OLS). Transformers also display better resilience to mild distribution shifts,
where set-based MLPs falter. However, under severe distribution shifts, both
models' in-context learning abilities diminish.
Scan and Snap: Understanding Training Dynamics and Token Composition in
  1-layer Transformer
Authors: name: Yuandong Tian, name: Yiping Wang, name: Beidi Chen, name: Simon Du
Published: 2023-05-25 15:59:13

Transformer architecture has shown impressive performance in multiple
research domains and has become the backbone of many neural network models.
However, there is limited understanding on how it works. In particular, with a
simple predictive loss, how the representation emerges from the gradient
training dynamics remains a mystery. In this paper, for 1-layer
transformer with one self-attention layer plus one decoder layer, we analyze
its SGD training dynamics for the task of next token prediction in a
mathematically rigorous manner. We open the black box of the dynamic process of
how the self-attention layer combines input tokens, and reveal the nature of
underlying inductive bias. More specifically, with the assumption (a) no
positional encoding, (b) long input sequence, and (c) the decoder layer learns
faster than the self-attention layer, we prove that self-attention acts as a
discriminative scanning algorithm: starting from uniform attention, it
gradually attends more to distinct key tokens for a specific next token to be
predicted, and pays less attention to common key tokens that occur across
different next tokens. Among distinct tokens, it progressively drops attention
weights, following the order of low to high co-occurrence between the key and
the query token in the training set. Interestingly, this procedure does not
lead to winner-takes-all, but decelerates due to a phase transition that
is controllable by the learning rates of the two layers, leaving (almost) fixed
token combination. We verify this \emph{scan and snap} dynamics on
synthetic and real-world data (WikiText).
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
Authors: name: Jannik Kossen, name: Mark Collier, name: Basil Mustafa, name: Xiao Wang, name: Xiaohua Zhai, name: Lucas Beyer, name: Andreas Steiner, name: Jesse Berent, name: Rodolphe Jenatton, name: Efi Kokiopoulou
Published: 2023-05-26 14:59:16

We introduce Three Towers (3T), a flexible method to improve the contrastive
learning of vision-language models by incorporating pretrained image
classifiers. While contrastive models are usually trained from scratch, LiT
(Zhai et al., 2022) has recently shown performance gains from using pretrained
classifier embeddings. However, LiT directly replaces the image tower with the
frozen embeddings, excluding any potential benefits of contrastively training
the image tower. With 3T, we propose a more flexible strategy that allows the
image tower to benefit from both pretrained embeddings and contrastive
training. To achieve this, we introduce a third tower that contains the frozen
pretrained embeddings, and we encourage alignment between this third tower and
the main image-text towers. Empirically, 3T consistently improves over LiT and
the CLIP-style from-scratch baseline for retrieval tasks. For classification,
3T reliably improves over the from-scratch baseline, and while it underperforms
relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
and Places365 pretraining.
Training Socially Aligned Language Models in Simulated Human Society
Authors: name: Ruibo Liu, name: Ruixin Yang, name: Chenyan Jia, name: Ge Zhang, name: Denny Zhou, name: Andrew M. Dai, name: Diyi Yang, name: Soroush Vosoughi
Published: 2023-05-26 14:17:36

Social alignment in AI systems aims to ensure that these models behave
according to established societal values. However, unlike humans, who derive
consensus on value judgments through social interaction, current language
models (LMs) are trained to rigidly replicate their training corpus in
isolation, leading to subpar generalization in unfamiliar scenarios and
vulnerability to adversarial attacks. This work presents a novel training
paradigm that permits LMs to learn from simulated social interactions. In
comparison to existing methodologies, our approach is considerably more
scalable and efficient, demonstrating superior performance in alignment
benchmarks and human evaluations. This paradigm shift in the training of LMs
brings us a step closer to developing AI systems that can robustly and
accurately reflect societal norms and values.
MixCE: Training Autoregressive Language Models by Mixing Forward and
  Reverse Cross-Entropies
Authors: name: Shiyue Zhang, name: Shijie Wu, name: Ozan Irsoy, name: Steven Lu, name: Mohit Bansal, name: Mark Dredze, name: David Rosenberg
Published: 2023-05-26 14:14:51

Autoregressive language models are trained by minimizing the cross-entropy of
the model distribution Q relative to the data distribution P -- that is,
minimizing the forward cross-entropy, which is equivalent to maximum likelihood
estimation (MLE). We have observed that models trained in this way may
"over-generalize", in the sense that they produce non-human-like text.
Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P
relative to Q, is a better reflection of how a human would evaluate text
generated by a model. Hence, we propose learning with MixCE, an objective that
mixes the forward and reverse cross-entropies. We evaluate models trained with
this objective on synthetic data settings (where P is known) and real data, and
show that the resulting models yield better generated text without complex
decoding strategies. Our code and models are publicly available at
https://github.com/bloomberg/mixce-acl2023
Do GPTs Produce Less Literal Translations?
Authors: name: Vikas Raunak, name: Arul Menezes, name: Matt Post, name: Hany Hassan Awadallah
Published: 2023-05-26 10:38:31

Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose
language models capable of addressing many natural language generation or
understanding tasks. On the task of Machine Translation (MT), multiple works
have investigated few-shot prompting mechanisms to elicit better translations
from LLMs. However, there has been relatively little investigation on how such
translations differ qualitatively from the translations generated by standard
Neural Machine Translation (NMT) models. In this work, we investigate these
differences in terms of the literalness of translations produced by the two
systems. Using literalness measures involving word alignment and monotonicity,
we find that translations out of English (E-X) from GPTs tend to be less
literal, while exhibiting similar or better scores on MT quality metrics. We
demonstrate that this finding is borne out in human evaluations as well. We
then show that these differences are especially pronounced when translating
sentences that contain idiomatic expressions.
Backpack Language Models
Authors: name: John Hewitt, name: John Thickstun, name: Christopher D. Manning, name: Percy Liang
Published: 2023-05-26 09:26:23

We present Backpacks: a new neural architecture that marries strong modeling
performance with an interface for interpretability and control. Backpacks learn
multiple non-contextual sense vectors for each word in a vocabulary, and
represent a word in a sequence as a context-dependent, non-negative linear
combination of sense vectors in this sequence. We find that, after training,
sense vectors specialize, each encoding a different aspect of a word. We can
interpret a sense vector by inspecting its (non-contextual, linear) projection
onto the output space, and intervene on these interpretable hooks to change the
model's behavior in predictable ways. We train a 170M-parameter Backpack
language model on OpenWebText, matching the loss of a GPT-2 small
(124Mparameter) Transformer. On lexical similarity evaluations, we find that
Backpack sense vectors outperform even a 6B-parameter Transformer LM's word
embeddings. Finally, we present simple algorithms that intervene on sense
vectors to perform controllable text generation and debiasing. For example, we
can edit the sense vocabulary to tend more towards a topic, or localize a
source of gender bias to a sense vector and globally suppress that sense.
Impossible Distillation: from Low-Quality Model to High-Quality Dataset
  & Model for Summarization and Paraphrasing
Authors: name: Jaehun Jung, name: Peter West, name: Liwei Jiang, name: Faeze Brahman, name: Ximing Lu, name: Jillian Fisher, name: Taylor Sorensen, name: Yejin Choi
Published: 2023-05-26 05:19:24

It is commonly perceived that the strongest language models (LMs) rely on a
combination of massive scale, instruction data, and human feedback to perform
specialized tasks -- e.g. summarization and paraphrasing, without supervision.
In this paper, we propose that language models can learn to summarize and
paraphrase sentences, with none of these 3 factors. We present Impossible
Distillation, a framework that distills a task-specific dataset directly from
an off-the-shelf LM, even when it is impossible for the LM itself to reliably
solve the task. By training a student model on the generated dataset and
amplifying its capability through self-distillation, our method yields a
high-quality model and dataset from a low-quality teacher model, without the
need for scale or supervision. Using Impossible Distillation, we are able to
distill an order of magnitude smaller model (with only 770M parameters) that
outperforms 175B parameter GPT-3, in both quality and controllability, as
confirmed by automatic and human evaluations. Furthermore, as a useful
byproduct of our approach, we obtain DIMSUM+, a high-quality dataset with 3.4M
sentence summaries and paraphrases. Our analyses show that this dataset, as a
purely LM-generated corpus, is more diverse and more effective for
generalization to unseen domains than all human-authored datasets -- including
Gigaword with 4M samples.
Role-Play with Large Language Models
Authors: name: Murray Shanahan, name: Kyle McDonell, name: Laria Reynolds
Published: 2023-05-25 11:36:52

As dialogue agents become increasingly human-like in their performance, it is
imperative that we develop effective ways to describe their behaviour in
high-level terms without falling into the trap of anthropomorphism. In this
paper, we foreground the concept of role-play. Casting dialogue agent behaviour
in terms of role-play allows us to draw on familiar folk psychological terms,
without ascribing human characteristics to language models they in fact lack.
Two important cases of dialogue agent behaviour are addressed this way, namely
(apparent) deception and (apparent) self-awareness.
PandaGPT: One Model To Instruction-Follow Them All
Authors: name: Yixuan Su, name: Tian Lan, name: Huayang Li, name: Jialu Xu, name: Yan Wang, name: Deng Cai
Published: 2023-05-25 04:16:07

We present PandaGPT, an approach to emPower large lANguage moDels with visual
and Auditory instruction-following capabilities. Our pilot experiments show
that PandaGPT can perform complex tasks such as detailed image description
generation, writing stories inspired by videos, and answering questions about
audios. More interestingly, PandaGPT can take multimodal inputs simultaneously
and compose their semantics naturally. For example, PandaGPT can connect how
objects look in an image/video and how they sound in an audio. To do so,
PandaGPT combines the multimodal encoders from ImageBind and the large language
models from Vicuna. Notably, only aligned image-text pairs are required for the
training of PandaGPT. Thanks to the strong capability of ImageBind in embedding
data from different modalities into the same space, PandaGPT displays emergent,
i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,
video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an
initial step toward building AGI that can perceive and understand inputs in
different modalities holistically, as we humans do. Our project page is at
https://panda-gpt.github.io/.
Lexinvariant Language Models
Authors: name: Qian Huang, name: Eric Zelikman, name: Sarah Li Chen, name: Yuhuai Wu, name: Gregory Valiant, name: Percy Liang
Published: 2023-05-24 19:10:46

Token embeddings, a mapping from discrete lexical symbols to continuous
vectors, are at the heart of any language model (LM). However, lexical symbol
meanings can also be determined and even redefined by their structural role in
a long context. In this paper, we ask: is it possible for a language model to
be performant without any fixed token embeddings? Such a language model
would have to rely entirely on the co-occurence and repetition of tokens in the
context rather than the a priori identity of any token. To answer
this, we study lexinvariantlanguage models that are invariant to
lexical symbols and therefore do not need fixed token embeddings in practice.
First, we prove that we can construct a lexinvariant LM to converge to the true
language model at a uniform rate that is polynomial in terms of the context
length, with a constant factor that is sublinear in the vocabulary size.
Second, to build a lexinvariant LM, we simply encode tokens using random
Gaussian vectors, such that each token maps to the same representation within
each sequence but different representations across sequences. Empirically, we
demonstrate that it can indeed attain perplexity comparable to that of a
standard language model, given a sufficiently long context. We further explore
two properties of the lexinvariant language models: First, given text generated
from a substitution cipher of English, it implicitly implements Bayesian
in-context deciphering and infers the mapping to the underlying real tokens
with high accuracy. Second, it has on average 4X better accuracy over synthetic
in-context reasoning tasks. Finally, we discuss regularizing standard language
models towards lexinvariance and potential practical applications.
Think Before You Act: Decision Transformers with Internal Working Memory
Authors: name: Jikun Kang, name: Romain Laroche, name: Xindi Yuan, name: Adam Trischler, name: Xue Liu, name: Jie Fu
Published: 2023-05-24 01:20:22

Large language model (LLM)-based decision-making agents have shown the
ability to generalize across multiple tasks. However, their performance relies
on massive data and compute. We argue that this inefficiency stems from the
forgetting phenomenon, in which a model memorizes its behaviors in parameters
throughout training. As a result, training on a new task may deteriorate the
model's performance on previous tasks. In contrast to LLMs' implicit memory
mechanism, the human brain utilizes distributed memory storage, which helps
manage and organize multiple skills efficiently, mitigating the forgetting
phenomenon. Thus inspired, we propose an internal working memory module to
store, blend, and retrieve information for different downstream tasks.
Evaluation results show that the proposed method improves training efficiency
and generalization in both Atari games and meta-world object manipulation
tasks. Moreover, we demonstrate that memory fine-tuning further enhances the
adaptability of the proposed architecture.
Mindstorms in Natural Language-Based Societies of Mind
Authors: name: Mingchen Zhuge, name: Haozhe Liu, name: Francesco Faccio, name: Dylan R. Ashley, name: Róbert Csordás, name: Anand Gopalakrishnan, name: Abdullah Hamdi, name: Hasan Abed Al Kader Hammoud, name: Vincent Herrmann, name: Kazuki Irie, name: Louis Kirsch, name: Bing Li, name: Guohao Li, name: Shuming Liu, name: Jinjie Mai, name: Piotr Piękos, name: Aditya Ramesh, name: Imanol Schlag, name: Weimin Shi, name: Aleksandar Stanić, name: Wenyi Wang, name: Yuhui Wang, name: Mengmeng Xu, name: Deng-Ping Fan, name: Bernard Ghanem, name: Jürgen Schmidhuber
Published: 2023-05-26 16:21:25

Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire
diverse societies of large multimodal neural networks (NNs) that solve problems
by interviewing each other in a "mindstorm." Recent implementations of NN-based
societies of minds consist of large language models (LLMs) and other NN-based
experts communicating through a natural language interface. In doing so, they
overcome the limitations of single LLMs, improving multimodal zero-shot
reasoning. In these natural language-based societies of mind (NLSOMs), new
agents -- all communicating through the same universal symbolic language -- are
easily added in a modular fashion. To demonstrate the power of NLSOMs, we
assemble and experiment with several of them (having up to 129 members),
leveraging mindstorms in them to solve some practical AI tasks: visual question
answering, image captioning, text-to-image synthesis, 3D generation, egocentric
retrieval, embodied AI, and general language-based task solving. We view this
as a starting point towards much larger NLSOMs with billions of agents-some of
which may be humans. And with this emergence of great societies of
heterogeneous minds, many new research questions have suddenly become paramount
to the future of artificial intelligence. What should be the social structure
of an NLSOM? What would be the (dis)advantages of having a monarchical rather
than a democratic structure? How can principles of NN economies be used to
maximize the total reward of a reinforcement learning NLSOM? In this work, we
identify, discuss, and try to answer some of these questions.