Orionnnn/hub-stats huggingface dataset:
Frontier Language Models are not Robust to Adversarial Arithmetic, or
  "What do I need to say so you agree 2+2=5?
Authors: name: C. Daniel Freeman, name: Laura Culp, name: Aaron Parisi, name: Maxwell L Bileschi, name: Gamaleldin F Elsayed, name: Alex Rizkowsky, name: Isabelle Simpson, name: Alex Alemi, name: Azade Nova, name: Ben Adlam, name: Bernd Bohnet, name: Gaurav Mishra, name: Hanie Sedghi, name: Igor Mordatch, name: Izzeddin Gur, name: Jaehoon Lee, name: JD Co-Reyes, name: Jeffrey Pennington, name: Kelvin Xu, name: Kevin Swersky, name: Kshiteej Mahajan, name: Lechao Xiao, name: Rosanne Liu, name: Simon Kornblith, name: Noah Constant, name: Peter J. Liu, name: Roman Novak, name: Sharad Vikram, name: Yundi Qian, name: Noah Fiedel, name: Jascha Sohl-Dickstein
Published: 2023-11-08 19:07:10

We introduce and study the problem of adversarial arithmetic, which provides
a simple yet challenging testbed for language model alignment. This problem is
comprised of arithmetic questions posed in natural language, with an arbitrary
adversarial string inserted before the question is complete. Even in the simple
setting of 1-digit addition problems, it is easy to find adversarial prompts
that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
even to steer models to a particular wrong answer. We additionally provide a
simple algorithm for finding successful attacks by querying those same models,
which we name "prompt inversion rejection sampling" (PIRS). We finally show
that models can be partially hardened against these attacks via reinforcement
learning and via agentic constitutional loops. However, we were not able to
make a language model fully robust against adversarial arithmetic attacks.
To See is to Believe: Prompting GPT-4V for Better Visual Instruction
  Tuning
Authors: name: Junke Wang, name: Lingchen Meng, name: Zejia Weng, name: Bo He, name: Zuxuan Wu, name: Yu-Gang Jiang
Published: 2023-11-13 18:59:31

Existing visual instruction tuning methods typically prompt large language
models with textual descriptions to generate instruction-following data.
Despite the promising performance achieved, these descriptions are derived from
image annotations, which are oftentimes coarse-grained. Furthermore, the
instructions might even contradict the visual content without observing the
entire visual context. To address this challenge, we introduce a fine-grained
visual instruction dataset, LVIS-Instruct4V, which contains 220K visually
aligned and context-aware instructions produced by prompting the powerful
GPT-4V with images from LVIS. Through experimental validation and case studies,
we demonstrate that high-quality visual instructional data could improve the
performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a
wide spectrum of benchmarks by clear margins. Notably, by simply replacing the
LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA
on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet
(40.2 vs. 35.4). We release our data and model at
https://github.com/X2FD/LVIS-INSTRUCT4V.
ChatAnything: Facetime Chat with LLM-Enhanced Personas
Authors: name: Yilin Zhao, name: Xinbin Yuan, name: Shanghua Gao, name: Zhijie Lin, name: Qibin Hou, name: Jiashi Feng, name: Daquan Zhou
Published: 2023-11-12 08:29:41

In this technical report, we target generating anthropomorphized personas for
LLM-based characters in an online manner, including visual appearance,
personality and tones, with only text descriptions. To achieve this, we first
leverage the in-context learning capability of LLMs for personality generation
by carefully designing a set of system prompts. We then propose two novel
concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for
diverse voice and appearance generation. For MoV, we utilize the text-to-speech
(TTS) algorithms with a variety of pre-defined tones and select the most
matching one based on the user-provided text description automatically. For
MoD, we combine the recent popular text-to-image generation techniques and
talking head algorithms to streamline the process of generating talking
objects. We termed the whole framework as ChatAnything. With it, users could be
able to animate anything with any personas that are anthropomorphic using just
a few text inputs. However, we have observed that the anthropomorphic objects
produced by current generative models are often undetectable by pre-trained
face landmark detectors, leading to failure of the face motion generation, even
if these faces possess human-like appearances because those images are nearly
seen during the training (e.g., OOD samples). To address this issue, we
incorporate pixel-level guidance to infuse human face landmarks during the
image generation phase. To benchmark these metrics, we have built an evaluation
dataset. Based on it, we verify that the detection rate of the face landmark is
significantly increased from 57.0% to 92.5% thus allowing automatic face
animation based on generated speech content. The code and more results can be
found at https://chatanything.github.io/.
GOAT: GO to Any Thing
Authors: name: Matthew Chang, name: Theophile Gervet, name: Mukul Khanna, name: Sriram Yenamandra, name: Dhruv Shah, name: So Yeon Min, name: Kavit Shah, name: Chris Paxton, name: Saurabh Gupta, name: Dhruv Batra, name: Roozbeh Mottaghi, name: Jitendra Malik, name: Devendra Singh Chaplot
Published: 2023-11-10 23:31:51

In deployment scenarios such as homes and warehouses, mobile robots are
expected to autonomously navigate for extended periods, seamlessly executing
tasks articulated in terms that are intuitively understandable by human
operators. We present GO To Any Thing (GOAT), a universal navigation system
capable of tackling these requirements with three key features: a) Multimodal:
it can tackle goals specified via category labels, target images, and language
descriptions, b) Lifelong: it benefits from its past experience in the same
environment, and c) Platform Agnostic: it can be quickly deployed on robots
with different embodiments. GOAT is made possible through a modular system
design and a continually augmented instance-aware semantic memory that keeps
track of the appearance of objects from different viewpoints in addition to
category-level semantics. This enables GOAT to distinguish between different
instances of the same category to enable navigation to targets specified by
images and language descriptions. In experimental comparisons spanning over 90
hours in 9 different homes consisting of 675 goals selected across 200+
different object instances, we find GOAT achieves an overall success rate of
83%, surpassing previous methods and ablations by 32% (absolute improvement).
GOAT improves with experience in the environment, from a 60% success rate at
the first goal to a 90% success after exploration. In addition, we demonstrate
that GOAT can readily be applied to downstream tasks such as pick and place and
social navigation.
Music ControlNet: Multiple Time-varying Controls for Music Generation
Authors: name: Shih-Lun Wu, name: Chris Donahue, name: Shinji Watanabe, name: Nicholas J. Bryan
Published: 2023-11-13 04:24:14

Text-to-music generation models are now capable of generating high-quality
music audio in broad styles. However, text control is primarily suitable for
the manipulation of global musical attributes like genre, mood, and tempo, and
is less suitable for precise control over time-varying attributes such as the
positions of beats in time or the changing dynamics of the music. We propose
Music ControlNet, a diffusion-based music generation model that offers multiple
precise, time-varying controls over generated audio. To imbue text-to-music
models with time-varying control, we propose an approach analogous to
pixel-wise control of the image-domain ControlNet method. Specifically, we
extract controls from training audio yielding paired data, and fine-tune a
diffusion-based conditional generative model over audio spectrograms given
melody, dynamics, and rhythm controls. While the image-domain Uni-ControlNet
method already allows generation with any subset of controls, we devise a new
strategy to allow creators to input controls that are only partially specified
in time. We evaluate both on controls extracted from audio and controls we
expect creators to provide, demonstrating that we can generate realistic music
that corresponds to control inputs in both settings. While few comparable music
generation models exist, we benchmark against MusicGen, a recent model that
accepts text and melody input, and show that our model generates music that is
49% more faithful to input melodies despite having 35x fewer parameters,
training on 11x less data, and enabling two additional forms of time-varying
control. Sound examples can be found at https://MusicControlNet.github.io/web/.
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for
  Multi-modal Large Language Models
Authors: name: Ziyi Lin, name: Chris Liu, name: Renrui Zhang, name: Peng Gao, name: Longtian Qiu, name: Han Xiao, name: Han Qiu, name: Chen Lin, name: Wenqi Shao, name: Keqin Chen, name: Jiaming Han, name: Siyuan Huang, name: Yichi Zhang, name: Xuming He, name: Hongsheng Li, name: Yu Qiao
Published: 2023-11-13 18:59:47

We present SPHINX, a versatile multi-modal large language model (MLLM) with a
joint mixing of model weights, tuning tasks, and visual embeddings. First, for
stronger vision-language alignment, we unfreeze the large language model (LLM)
during pre-training, and introduce a weight mix strategy between LLMs trained
by real-world and synthetic data. By directly integrating the weights from two
domains, the mixed LLM can efficiently incorporate diverse semantics with
favorable robustness. Then, to enable multi-purpose capabilities, we mix a
variety of tasks for joint visual instruction tuning, and design task-specific
instructions to avoid inter-task conflict. In addition to the basic visual
question answering, we include more challenging tasks such as region-level
understanding, caption grounding, document layout detection, and human pose
estimation, contributing to mutual enhancement over different scenarios.
Additionally, we propose to extract comprehensive visual embeddings from
various network architectures, pre-training paradigms, and information
granularity, providing language models with more robust image representations.
Based on our proposed joint mixing, SPHINX exhibits superior multi-modal
understanding capabilities on a wide range of applications. On top of this, we
further propose an efficient strategy aiming to better capture fine-grained
appearances of high-resolution images. With a mixing of different scales and
high-resolution sub-images, SPHINX attains exceptional visual parsing and
reasoning performance on existing evaluation benchmarks. We hope our work may
cast a light on the exploration of joint mixing in future MLLM research. Code
is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
MEGAVERSE: Benchmarking Large Language Models Across Languages,
  Modalities, Models and Tasks
Authors: name: Sanchit Ahuja, name: Divyanshu Aggarwal, name: Varun Gumma, name: Ishaan Watts, name: Ashutosh Sathe, name: Millicent Ochieng, name: Rishav Hada, name: Prachi Jain, name: Maxamed Axmed, name: Kalika Bali, name: Sunayana Sitaram
Published: 2023-11-13 16:45:37

Recently, there has been a rapid advancement in research on Large Language
Models (LLMs), resulting in significant progress in several Natural Language
Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
research to comprehend the models' capabilities and limitations. However, much
of this research has been confined to the English language, leaving LLM
building and evaluation for non-English languages relatively unexplored. There
has been an introduction of several new LLMs, necessitating their evaluation on
non-English languages. This study aims to expand our MEGA benchmarking suite by
including six new datasets to form the MEGAVERSE benchmark. The benchmark
comprises 22 datasets covering 81 languages, including low-resource African
languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4,
PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two
multimodal datasets in the benchmark and assess the performance of the
LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the
Llama models on various tasks, notably on low-resource languages, with GPT4
outperforming PaLM2 on more datasets than vice versa. However, issues such as
data contamination must be addressed to obtain an accurate assessment of LLM
performance on non-English languages.
Story-to-Motion: Synthesizing Infinite and Controllable Character
  Animation from Long Text
Authors: name: Zhongfei Qing, name: Zhongang Cai, name: Zhitao Yang, name: Lei Yang
Published: 2023-11-13 16:22:38

Generating natural human motion from a story has the potential to transform
the landscape of animation, gaming, and film industries. A new and challenging
task, Story-to-Motion, arises when characters are required to move to various
locations and perform specific motions based on a long text description. This
task demands a fusion of low-level control (trajectories) and high-level
control (motion semantics). Previous works in character control and
text-to-motion have addressed related aspects, yet a comprehensive solution
remains elusive: character control methods do not handle text description,
whereas text-to-motion methods lack position constraints and often produce
unstable motions. In light of these limitations, we propose a novel system that
generates controllable, infinitely long motions and trajectories aligned with
the input text. (1) We leverage contemporary Large Language Models to act as a
text-driven motion scheduler to extract a series of (text, position, duration)
pairs from long text. (2) We develop a text-driven motion retrieval scheme that
incorporates motion matching with motion semantic and trajectory constraints.
(3) We design a progressive mask transformer that addresses common artifacts in
the transition motion such as unnatural pose and foot sliding. Beyond its
pioneering role as the first comprehensive solution for Story-to-Motion, our
system undergoes evaluation across three distinct sub-tasks: trajectory
following, temporal action composition, and motion blending, where it
outperforms previous state-of-the-art motion synthesis methods across the
board. Homepage: https://story2motion.github.io/.
Towards General-Purpose Speech Abilities for Large Language Models Using
  Unpaired Data
Authors: name: Yassir Fathullah, name: Chunyang Wu, name: Egor Lakomkin, name: Junteng Jia, name: Yuan Shangguan, name: Jay Mahadeokar, name: Ozlem Kalinli, name: Christian Fuegen, name: Mike Seltzer
Published: 2023-11-12 06:56:14

In this work, we extend the instruction-tuned Llama-2 model with end-to-end
general-purpose speech processing and reasoning abilities while maintaining the
wide range of LLM capabilities, without using any carefully curated paired
data. The proposed model can utilize audio prompts as a replacement for text
and sustain a conversation. Such a model also has extended cross-modal
capabilities such as being able to perform speech question answering, speech
translation, and audio summarization amongst many other closed and open-domain
tasks. This is unlike prior approaches in speech, in which LLMs are extended to
handle audio for a limited number of pre-designated tasks. Experiments show
that our end-to-end approach is on par with or outperforms a cascaded system
(speech recognizer + LLM) in terms of modeling the response to a prompt.
Furthermore, unlike a cascade, our approach shows the ability to interchange
text and audio modalities and utilize the prior context in a conversation to
provide better results.
Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small
  Scorer
Authors: name: Bowen Tan, name: Yun Zhu, name: Lijuan Liu, name: Eric Xing, name: Zhiting Hu, name: Jindong Chen
Published: 2023-11-12 03:25:34

Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in
multi-tasking under a unified instruction-following paradigm, where they also
exhibit remarkable generalization abilities to unseen tasks. Despite their
impressive performance, these LLMs, with sizes ranging from several billion to
hundreds of billions of parameters, demand substantial computational resources,
making their training and inference expensive and inefficient. Furthermore,
adapting these models to downstream applications, particularly complex tasks,
is often unfeasible due to the extensive hardware requirements for finetuning,
even when utilizing parameter-efficient approaches such as prompt tuning.
Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and
FLAN-PaLM-540B, are not publicly accessible, severely limiting their
customization potential. To address these challenges, we introduce a pretrained
small scorer, Cappy, designed to enhance the performance and efficiency of
multi-task LLMs. With merely 360 million parameters, Cappy functions either
independently on classification tasks or serve as an auxiliary component for
LLMs, boosting their performance. Moreover, Cappy enables efficiently
integrating downstream supervision without requiring LLM finetuning nor the
access to their parameters. Our experiments demonstrate that, when working
independently on 11 language understanding tasks from PromptSource, Cappy
outperforms LLMs that are several orders of magnitude larger. Besides, on 45
complex tasks from BIG-Bench, Cappy boosts the performance of the advanced
multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to
cooperate with other LLM adaptations, including finetuning and in-context
learning, offering additional performance enhancement.
Trusted Source Alignment in Large Language Models
Authors: name: Vasilisa Bashlovkina, name: Zhaobin Kuang, name: Riley Matthews, name: Edward Clifford, name: Yennie Jun, name: William W. Cohen, name: Simon Baumgartner
Published: 2023-11-12 00:25:25

Large language models (LLMs) are trained on web-scale corpora that inevitably
include contradictory factual information from sources of varying reliability.
In this paper, we propose measuring an LLM property called trusted source
alignment (TSA): the model's propensity to align with content produced by
trusted publishers in the face of uncertainty or controversy. We present
FactCheckQA, a TSA evaluation dataset based on a corpus of fact checking
articles. We describe a simple protocol for evaluating TSA and offer a detailed
analysis of design considerations including response extraction, claim
contextualization, and bias in prompt formulation. Applying the protocol to
PaLM-2, we find that as we scale up the model size, the model performance on
FactCheckQA improves from near-random to up to 80% balanced accuracy in
aligning with trusted sources.
LayoutPrompter: Awaken the Design Ability of Large Language Models
Authors: name: Jiawei Lin, name: Jiaqi Guo, name: Shizhao Sun, name: Zijiang James Yang, name: Jian-Guang Lou, name: Dongmei Zhang
Published: 2023-11-11 07:15:14

Conditional graphic layout generation, which automatically maps user
constraints to high-quality layouts, has attracted widespread attention today.
Although recent works have achieved promising performance, the lack of
versatility and data efficiency hinders their practical applications. In this
work, we propose LayoutPrompter, which leverages large language models (LLMs)
to address the above problems through in-context learning. LayoutPrompter is
made up of three key components, namely input-output serialization, dynamic
exemplar selection and layout ranking. Specifically, the input-output
serialization component meticulously designs the input and output formats for
each layout generation task. Dynamic exemplar selection is responsible for
selecting the most helpful prompting exemplars for a given input. And a layout
ranker is used to pick the highest quality layout from multiple outputs of
LLMs. We conduct experiments on all existing layout generation tasks using four
public datasets. Despite the simplicity of our approach, experimental results
show that LayoutPrompter can compete with or even outperform state-of-the-art
approaches on these tasks without any model training or fine-tuning. This
demonstrates the effectiveness of this versatile and training-free approach. In
addition, the ablation studies show that LayoutPrompter is significantly
superior to the training-based baseline in a low-data regime, further
indicating the data efficiency of LayoutPrompter. Our project is available at
https://github.com/microsoft/LayoutGeneration/tree/main/LayoutPrompter.
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone
  GUI Navigation
Authors: name: An Yan, name: Zhengyuan Yang, name: Wanrong Zhu, name: Kevin Lin, name: Linjie Li, name: Jianfeng Wang, name: Jianwei Yang, name: Yiwu Zhong, name: Julian McAuley, name: Jianfeng Gao, name: Zicheng Liu, name: Lijuan Wang
Published: 2023-11-13 18:53:37

We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical
user interface (GUI) navigation task. MM-Navigator can interact with a
smartphone screen as human users, and determine subsequent actions to fulfill
given instructions. Our findings demonstrate that large multimodal models
(LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its
advanced screen interpretation, action reasoning, and precise action
localization capabilities. We first benchmark MM-Navigator on our collected iOS
screen dataset. According to human assessments, the system exhibited a 91\%
accuracy rate in generating reasonable action descriptions and a 75\% accuracy
rate in executing the correct actions for single-step instructions on iOS.
Additionally, we evaluate the model on a subset of an Android screen navigation
dataset, where the model outperforms previous GUI navigators in a zero-shot
fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for
future research into the GUI navigation task. The project page is at
https://github.com/zzxslp/MM-Navigator.
The Impact of Large Language Models on Scientific Discovery: a
  Preliminary Study using GPT-4
Authors: name: Microsoft Research AI4Science, name: Microsoft Azure Quantum
Published: 2023-11-13 14:26:12

In recent years, groundbreaking advancements in natural language processing
have culminated in the emergence of powerful large language models (LLMs),
which have showcased remarkable capabilities across a vast array of domains,
including the understanding, generation, and translation of natural language,
and even tasks that extend beyond language processing. In this report, we delve
into the performance of LLMs within the context of scientific discovery,
focusing on GPT-4, the state-of-the-art language model. Our investigation spans
a diverse range of scientific areas encompassing drug discovery, biology,
computational chemistry (density functional theory (DFT) and molecular dynamics
(MD)), materials design, and partial differential equations (PDE). Evaluating
GPT-4 on scientific tasks is crucial for uncovering its potential across
various research domains, validating its domain-specific expertise,
accelerating scientific progress, optimizing resource allocation, guiding
future model development, and fostering interdisciplinary research. Our
exploration methodology primarily consists of expert-driven case assessments,
which offer qualitative insights into the model's comprehension of intricate
scientific concepts and relationships, and occasionally benchmark testing,
which quantitatively evaluates the model's capacity to solve well-defined
domain-specific problems. Our preliminary exploration indicates that GPT-4
exhibits promising potential for a variety of scientific applications,
demonstrating its aptitude for handling complex problem-solving and knowledge
integration tasks. Broadly speaking, we evaluate GPT-4's knowledge base,
scientific understanding, scientific numerical calculation abilities, and
various scientific prediction capabilities.