Orionnnn/hub-stats huggingface dataset:
An Interactive Agent Foundation Model
Authors: name: Zane Durante, name: Bidipta Sarkar, name: Ran Gong, name: Rohan Taori, name: Yusuke Noda, name: Paul Tang, name: Ehsan Adeli, name: Shrinidhi Kowshika Lakshmikanth, name: Kevin Schulman, name: Arnold Milstein, name: Demetri Terzopoulos, name: Ade Famoti, name: Noboru Kuno, name: Ashley Llorens, name: Hoi Vo, name: Katsu Ikeuchi, name: Li Fei-Fei, name: Jianfeng Gao, name: Naoki Wake, name: Qiuyuan Huang
Published: 2024-02-08 18:58:02

The development of artificial intelligence systems is transitioning from
creating static, task-specific models to dynamic, agent-based systems capable
of performing well in a wide range of applications. We propose an Interactive
Agent Foundation Model that uses a novel multi-task agent training paradigm for
training AI agents across a wide range of domains, datasets, and tasks. Our
training paradigm unifies diverse pre-training strategies, including visual
masked auto-encoders, language modeling, and next-action prediction, enabling a
versatile and adaptable AI framework. We demonstrate the performance of our
framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
Our model demonstrates its ability to generate meaningful and contextually
relevant outputs in each area. The strength of our approach lies in its
generality, leveraging a variety of data sources such as robotics sequences,
gameplay data, large-scale video datasets, and textual information for
effective multimodal and multi-task learning. Our approach provides a promising
avenue for developing generalist, action-taking, multimodal systems.
SpiRit-LM: Interleaved Spoken and Written Language Model
Authors: name: Tu Anh Nguyen, name: Benjamin Muller, name: Bokai Yu, name: Marta R. Costa-jussa, name: Maha Elbayad, name: Sravya Popuri, name: Paul-Ambroise Duquenne, name: Robin Algayres, name: Ruslan Mavlyutov, name: Itai Gat, name: Gabriel Synnaeve, name: Juan Pino, name: Benoit Sagot, name: Emmanuel Dupoux
Published: 2024-02-08 15:39:32

We introduce SPIRIT-LM, a foundation multimodal language model that freely
mixes text and speech. Our model is based on a pretrained text language model
that we extend to the speech modality by continuously training it on text and
speech units. Speech and text sequences are concatenated as a single set of
tokens, and trained with a word-level interleaving method using a small
automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
versions: a BASE version that uses speech semantic units and an EXPRESSIVE
version that models expressivity using pitch and style units in addition to the
semantic units. For both versions, the text is encoded with subword BPE tokens.
The resulting model displays both the semantic abilities of text models and the
expressive abilities of speech models. Additionally, we demonstrate that
SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
(i.e. ASR, TTS, Speech Classification).
Question Aware Vision Transformer for Multimodal Reasoning
Authors: name: Roy Ganz, name: Yair Kittenplon, name: Aviad Aberdam, name: Elad Ben Avraham, name: Oren Nuriel, name: Shai Mazor, name: Ron Litman
Published: 2024-02-08 08:03:39

Vision-Language (VL) models have gained significant research focus, enabling
remarkable advances in multimodal reasoning. These architectures typically
comprise a vision encoder, a Large Language Model (LLM), and a projection
module that aligns visual features with the LLM's representation space. Despite
their success, a critical limitation persists: the vision encoding process
remains decoupled from user queries, often in the form of image-related
questions. Consequently, the resulting visual features may not be optimally
attuned to the query-specific elements of the image. To address this, we
introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
reasoning, which embeds question awareness directly within the vision encoder.
This integration results in dynamic visual features focusing on relevant image
aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
efficiently into any VL architecture. Extensive experiments demonstrate the
effectiveness of applying our method to various multimodal architectures,
leading to consistent improvement across diverse tasks and showcasing its
potential for enhancing visual and scene-text understanding.
Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains
Authors: name: Junhong Shen, name: Neil Tenenholtz, name: James Brian Hall, name: David Alvarez-Melis, name: Nicolo Fusi
Published: 2024-02-06 20:11:54

Large Language Models (LLMs) have demonstrated remarkable proficiency in
understanding and generating natural language. However, their capabilities wane
in highly specialized domains underrepresented in the pretraining corpus, such
as physical and biomedical sciences. This work explores how to repurpose
general LLMs into effective task solvers for specialized domains. We introduce
a novel, model-agnostic framework for learning custom input tags, which are
parameterized as continuous vectors appended to the LLM's embedding layer, to
condition the LLM. We design two types of input tags: domain tags are used to
delimit specialized representations (e.g., chemical formulas) and provide
domain-relevant context; function tags are used to represent specific functions
(e.g., predicting molecular properties) and compress function-solving
instructions. We develop a three-stage protocol to learn these tags using
auxiliary data and domain knowledge. By explicitly disentangling task domains
from task functions, our method enables zero-shot generalization to unseen
problems through diverse combinations of the input tags. It also boosts LLM's
performance in various specialized domains, such as predicting protein or
chemical properties and modeling drug-target interactions, outperforming expert
models tailored to these tasks.
More Agents Is All You Need
Authors: name: Junyou Li, name: Qin Zhang, name: Yangbin Yu, name: Qiang Fu, name: Deheng Ye
Published: 2024-02-03 05:55:24

We find that, simply via a sampling-and-voting method, the performance of
large language models (LLMs) scales with the number of agents instantiated.
Also, this method is orthogonal to existing complicated methods to further
enhance LLMs, while the degree of enhancement is correlated to the task
difficulty. We conduct comprehensive experiments on a wide range of LLM
benchmarks to verify the presence of our finding, and to study the properties
that can facilitate its occurrence. Our code is publicly available at:
https://anonymous.4open.science/r/more_agent_is_all_you_need.
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Authors: name: Xing Han Lù, name: Zdeněk Kasner, name: Siva Reddy
Published: 2024-02-08 18:58:02

We propose the problem of conversational web navigation, where a digital
agent controls a web browser and follows user instructions to solve real-world
tasks in a multi-turn dialogue fashion. To support this problem, we introduce
WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
demonstrations of conversational web navigation. Our benchmark covers a broad
range of patterns on over 150 real-world websites and can be used to train and
evaluate agents in diverse scenarios. Due to the magnitude of information
present, Large Language Models (LLMs) cannot process entire web pages in
real-time. To solve this bottleneck, we design a retrieval-inspired model that
efficiently prunes HTML pages by ranking relevant elements. We use the selected
elements, along with screenshots and action history, to assess a variety of
models for their ability to replicate human behavior when navigating the web.
Our experiments span from small text-only to proprietary multimodal LLMs. We
find that smaller finetuned decoders surpass the best zero-shot LLMs (including
GPT-4V), but also larger finetuned multimodal models which were explicitly
pretrained on screenshots. However, all finetuned models struggle to generalize
to unseen websites. Our findings highlight the need for large multimodal models
that can generalize to novel settings. Our code, data and models are available
for research: https://mcgill-nlp.github.io/weblinx
In-Context Principle Learning from Mistakes
Authors: name: Tianjun Zhang, name: Aman Madaan, name: Luyu Gao, name: Steven Zheng, name: Swaroop Mishra, name: Yiming Yang, name: Niket Tandon, name: Uri Alon
Published: 2024-02-08 04:42:29

In-context learning (ICL, also known as few-shot prompting) has been the
standard method of adapting LLMs to downstream tasks, by learning from a few
input-output examples. Nonetheless, all ICL-based approaches only learn from
correct input-output pairs. In this paper, we revisit this paradigm, by
learning more from the few given input-output examples. We introduce Learning
Principles (LEAP): First, we intentionally induce the model to make mistakes on
these few examples; then we reflect on these mistakes, and learn explicit
task-specific "principles" from them, which help solve similar problems and
avoid common mistakes; finally, we prompt the model to answer unseen test
questions using the original few-shot examples and these learned general
principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop
question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning,
and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the
strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and
Claude-2.1. For example, LEAP improves over the standard few-shot prompting
using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does
not require any more input or examples than the standard few-shot prompting
settings.
EfficientViT-SAM: Accelerated Segment Anything Model Without Performance
  Loss
Authors: name: Zhuoyang Zhang, name: Han Cai, name: Song Han
Published: 2024-02-07 16:28:36

We present EfficientViT-SAM, a new family of accelerated segment anything
models. We retain SAM's lightweight prompt encoder and mask decoder while
replacing the heavy image encoder with EfficientViT. For the training, we begin
with the knowledge distillation from the SAM-ViT-H image encoder to
EfficientViT. Subsequently, we conduct end-to-end training on the SA-1B
dataset. Benefiting from EfficientViT's efficiency and capacity,
EfficientViT-SAM delivers 48.9x measured TensorRT speedup on A100 GPU over
SAM-ViT-H without sacrificing performance. Our code and pre-trained models are
released at https://github.com/mit-han-lab/efficientvit.
Grandmaster-Level Chess Without Search
Authors: name: Anian Ruoss, name: Grégoire Delétang, name: Sourabh Medapati, name: Jordi Grau-Moya, name: Li Kevin Wenliang, name: Elliot Catt, name: John Reid, name: Tim Genewein
Published: 2024-02-07 00:36:24

The recent breakthrough successes in machine learning are mainly attributed
to scale: namely large-scale attention-based architectures and datasets of
unprecedented scale. This paper investigates the impact of training at scale
for chess. Unlike traditional chess engines that rely on complex heuristics,
explicit search, or a combination of both, we train a 270M parameter
transformer model with supervised learning on a dataset of 10 million chess
games. We annotate each board in the dataset with action-values provided by the
powerful Stockfish 16 engine, leading to roughly 15 billion data points. Our
largest model reaches a Lichess blitz Elo of 2895 against humans, and
successfully solves a series of challenging chess puzzles, without any
domain-specific tweaks or explicit search algorithms. We also show that our
model outperforms AlphaZero's policy and value networks (without MCTS) and
GPT-3.5-turbo-instruct. A systematic investigation of model and dataset size
shows that strong chess performance only arises at sufficient scale. To
validate our results, we perform an extensive series of ablations of design
choices and hyperparameters.
The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax
  Mimicry
Authors: name: Michael Zhang, name: Kush Bhatia, name: Hermann Kumbong, name: Christopher Ré
Published: 2024-02-06 19:31:26

Linear attentions have shown potential for improving Transformer efficiency,
reducing attention's quadratic complexity to linear in sequence length. This
holds exciting promise for (1) training linear Transformers from scratch, (2)
"finetuned-conversion" of task-specific Transformers into linear versions that
recover task performance, and (3) "pretrained-conversion" of Transformers such
as large language models into linear versions finetunable on downstream tasks.
However, linear attentions often underperform standard softmax attention in
quality. To close this performance gap, we find prior linear attentions lack
key properties of softmax attention tied to good performance: low-entropy (or
"spiky") weights and dot-product monotonicity. We further observe surprisingly
simple feature maps that retain these properties and match softmax performance,
but are inefficient to compute in linear attention. We thus propose Hedgehog, a
learnable linear attention that retains the spiky and monotonic properties of
softmax attention while maintaining linear complexity. Hedgehog uses simple
trainable MLPs to produce attention weights mimicking softmax attention.
Experiments show Hedgehog recovers over 99% of standard Transformer quality in
train-from-scratch and finetuned-conversion settings, outperforming prior
linear attentions up to 6 perplexity points on WikiText-103 with causal GPTs,
and up to 8.7 GLUE score points on finetuned bidirectional BERTs. Hedgehog also
enables pretrained-conversion. Converting a pretrained GPT-2 into a linear
attention variant achieves state-of-the-art 16.7 perplexity on WikiText-103 for
125M subquadratic decoder models. We finally turn a pretrained Llama-2 7B into
a viable linear attention Llama. With low-rank adaptation, Hedgehog-Llama2 7B
achieves 28.1 higher ROUGE-1 points over the base standard attention model,
where prior linear attentions lead to 16.5 point drops.
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content
  Creation
Authors: name: Jiaxiang Tang, name: Zhaoxi Chen, name: Xiaokang Chen, name: Tengfei Wang, name: Gang Zeng, name: Ziwei Liu
Published: 2024-02-07 17:57:03

3D content creation has achieved significant progress in terms of both
quality and speed. Although current feed-forward models can produce 3D objects
in seconds, their resolution is constrained by the intensive computation
required during training. In this paper, we introduce Large Multi-View Gaussian
Model (LGM), a novel framework designed to generate high-resolution 3D models
from text prompts or single-view images. Our key insights are two-fold: 1) 3D
Representation: We propose multi-view Gaussian features as an efficient yet
powerful representation, which can then be fused together for differentiable
rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput
backbone operating on multi-view images, which can be produced from text or
single-view image input by leveraging multi-view diffusion models. Extensive
experiments demonstrate the high fidelity and efficiency of our approach.
Notably, we maintain the fast speed to generate 3D objects within 5 seconds
while boosting the training resolution to 512, thereby achieving
high-resolution 3D content generation.
Fast Timing-Conditioned Latent Audio Diffusion
Authors: name: Zach Evans, name: CJ Carr, name: Josiah Taylor, name: Scott H. Hawley, name: Jordi Pons
Published: 2024-02-07 13:23:25

Generating long-form 44.1kHz stereo audio from text prompts can be
computationally demanding. Further, most previous works do not tackle that
music and sound effects naturally vary in their duration. Our research focuses
on the efficient generation of long-form, variable-length stereo music and
sounds at 44.1kHz using text prompts with a generative model. Stable Audio is
based on latent diffusion, with its latent defined by a fully-convolutional
variational autoencoder. It is conditioned on text prompts as well as timing
embeddings, allowing for fine control over both the content and length of the
generated music and sounds. Stable Audio is capable of rendering stereo signals
of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. Despite its compute
efficiency and fast inference, it is one of the best in two public
text-to-music and -audio benchmarks and, differently from state-of-the-art
models, can generate music with structure and stereo sounds.
Hydragen: High-Throughput LLM Inference with Shared Prefixes
Authors: name: Jordan Juravsky, name: Bradley Brown, name: Ryan Ehrlich, name: Daniel Y. Fu, name: Christopher Ré, name: Azalia Mirhoseini
Published: 2024-02-07 18:53:01

Transformer-based large language models (LLMs) are now deployed to hundreds
of millions of users. LLM inference is commonly performed on batches of
sequences that share a prefix, such as few-shot examples or a chatbot system
prompt. Decoding in this large-batch setting can be bottlenecked by the
attention operation, which reads large key-value (KV) caches from memory and
computes inefficient matrix-vector products for every sequence in the batch. In
this work, we introduce Hydragen, a hardware-aware exact implementation of
attention with shared prefixes. Hydragen computes attention over the shared
prefix and unique suffixes separately. This decomposition enables efficient
prefix attention by batching queries together across sequences, reducing
redundant memory reads and enabling the use of hardware-friendly matrix
multiplications. Our method can improve end-to-end LLM throughput by up to 32x
against competitive baselines, with speedup growing with the batch size and
shared prefix length. Hydragen also enables the use of very long shared
contexts: with a high batch size, increasing the prefix length from 1K to 16K
tokens decreases Hydragen throughput by less than 15%, while the throughput of
baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix
decomposition and can be applied to tree-based prompt sharing patterns,
allowing us to further reduce inference time on competitive programming
problems by 55%.
TP-Aware Dequantization
Authors: name: Adnan Hoque, name: Mudhakar Srivatsa, name: Chih-Chieh Yang, name: Raghu Ganti
Published: 2024-01-15 08:01:40

In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay
Authors: name: Natasha Butt, name: Blazej Manczak, name: Auke Wiggers, name: Corrado Rainone, name: David Zhang, name: Michaël Defferrard, name: Taco Cohen
Published: 2024-02-07 13:55:27

Large language models are increasingly solving tasks that are commonly
believed to require human-level reasoning ability. However, these models still
perform very poorly on benchmarks of general intelligence such as the
Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a
programming-by-examples problem, and introduce a novel and scalable method for
language model self-improvement called Code Iteration (CodeIt). Our method
iterates between 1) program sampling and hindsight relabeling, and 2) learning
from prioritized experience replay. By relabeling the goal of an episode (i.e.,
the target program output given input) to the realized output produced by the
sampled program, our method effectively deals with the extreme sparsity of
rewards in program synthesis. Applying CodeIt to the ARC dataset, we
demonstrate that prioritized hindsight replay, along with pre-training and
data-augmentation, leads to successful inter-task generalization. CodeIt is the
first neuro-symbolic approach that scales to the full ARC evaluation dataset.
Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art
performance and outperforming existing neural and symbolic baselines.
Direct Language Model Alignment from Online AI Feedback
Authors: name: Shangmin Guo, name: Biao Zhang, name: Tianlin Liu, name: Tianqi Liu, name: Misha Khalman, name: Felipe Llinares, name: Alexandre Rame, name: Thomas Mesnard, name: Yao Zhao, name: Bilal Piot, name: Johan Ferret, name: Mathieu Blondel
Published: 2024-02-07 12:31:13

Direct alignment from preferences (DAP) methods, such as DPO, have recently
emerged as efficient alternatives to reinforcement learning from human feedback
(RLHF), that do not require a separate reward model. However, the preference
datasets used in DAP methods are usually collected ahead of training and never
updated, thus the feedback is purely offline. Moreover, responses in these
datasets are often sampled from a language model distinct from the one being
aligned, and since the model evolves over training, the alignment phase is
inevitably off-policy. In this study, we posit that online feedback is key and
improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as
annotator: on each training iteration, we sample two responses from the current
model and prompt the LLM annotator to choose which one is preferred, thus
providing online feedback. Despite its simplicity, we demonstrate via human
evaluation in several tasks that OAIF outperforms both offline DAP and RLHF
methods. We further show that the feedback leveraged in OAIF is easily
controllable, via instruction prompts to the LLM annotator.
Progressive Gradient Flow for Robust N:M Sparsity Training in
  Transformers
Authors: name: Abhimanyu Rajeshkumar Bambhaniya, name: Amir Yazdanbakhsh, name: Suvinay Subramanian, name: Sheng-Chun Kao, name: Shivani Agrawal, name: Utku Evci, name: Tushar Krishna
Published: 2024-02-07 10:55:59

N:M Structured sparsity has garnered significant interest as a result of
relatively modest overhead and improved efficiency. Additionally, this form of
sparsity holds considerable appeal for reducing the memory footprint owing to
their modest representation overhead. There have been efforts to develop
training recipes for N:M structured sparsity, they primarily focus on
low-sparsity regions (sim50\%). Nonetheless, performance of models trained
using these approaches tends to decline when confronted with high-sparsity
regions (>80\%). In this work, we study the effectiveness of existing sparse
training recipes at high-sparsity regions and argue that these methods
fail to sustain the model quality on par with low-sparsity regions. We
demonstrate that the significant factor contributing to this disparity is the
presence of elevated levels of induced noise in the gradient magnitudes. To
mitigate this undesirable effect, we employ decay mechanisms to progressively
restrict the flow of gradients towards pruned elements. Our approach improves
the model quality by up to 2% and 5% in vision and language models at
high sparsity regime, respectively. We also evaluate the trade-off between
model accuracy and training compute cost in terms of FLOPs. At iso-training
FLOPs, our method yields better performance compared to conventional sparse
training recipes, exhibiting an accuracy improvement of up to 2%. The source
code is available at
https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity.