Orionnnn/hub-stats huggingface dataset:
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Authors: name: Yadong Lu, name: Chunyuan Li, name: Haotian Liu, name: Jianwei Yang, name: Jianfeng Gao, name: Yelong Shen
Published: 2023-09-18 17:30:46

Visual instruction tuning has recently shown encouraging progress with
open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However,
most existing studies of open-source LMM are performed using models with 13B
parameters or smaller. In this paper we present an empirical study of scaling
LLaVA up to 33B and 65B/70B, and share our findings from our explorations in
image resolution, data mixing and parameter-efficient training methods such as
LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language
capabilities when completing real-world tasks in the wild.
  We find that scaling LMM consistently enhances model performance and improves
language capabilities, and performance of LoRA/QLoRA tuning of LMM are
comparable to the performance of full-model fine-tuning. Additionally, the
study highlights the importance of higher image resolutions and mixing
multimodal-language data to improve LMM performance, and visual instruction
tuning can sometimes improve LMM's pure language capability. We hope that this
study makes state-of-the-art LMM research at a larger scale more accessible,
thus helping establish stronger baselines for future research. Code and
checkpoints will be made public.
Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
  Pretrained 2D Diffusion Models
Authors: name: James Burgess, name: Kuan-Chieh Wang, name: Serena Yeung
Published: 2023-09-14 18:52:16

Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.
  ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts.
Retrieval-Augmented Text-to-Audio Generation
Authors: name: Yi Yuan, name: Haohe Liu, name: Xubo Liu, name: Qiushi Huang, name: Mark D. Plumbley, name: Wenwu Wang
Published: 2023-09-14 22:35:39

Despite recent progress in text-to-audio (TTA) generation, we show that the
state-of-the-art models, such as AudioLDM, trained on datasets with an
imbalanced class distribution, such as AudioCaps, are biased in their
generation performance. Specifically, they excel in generating common audio
classes while underperforming in the rare ones, thus degrading the overall
generation performance. We refer to this problem as long-tailed text-to-audio
generation. To address this issue, we propose a simple retrieval-augmented
approach for TTA models. Specifically, given an input text prompt, we first
leverage a Contrastive Language Audio Pretraining (CLAP) model to retrieve
relevant text-audio pairs. The features of the retrieved audio-text data are
then used as additional conditions to guide the learning of TTA models. We
enhance AudioLDM with our proposed approach and denote the resulting augmented
system as Re-AudioLDM. On the AudioCaps dataset, Re-AudioLDM achieves a
state-of-the-art Frechet Audio Distance (FAD) of 1.37, outperforming the
existing approaches by a large margin. Furthermore, we show that Re-AudioLDM
can generate realistic audio for complex scenes, rare audio classes, and even
unseen audio types, indicating its potential in TTA tasks.
Scaling Laws for Sparsely-Connected Foundation Models
Authors: name: Elias Frantar, name: Carlos Riquelme, name: Neil Houlsby, name: Dan Alistarh, name: Utku Evci
Published: 2023-09-15 16:29:27

We explore the impact of parameter sparsity on the scaling behavior of
Transformers trained on massive datasets (i.e., "foundation models"), in both
vision and language domains. In this setting, we identify the first scaling law
describing the relationship between weight sparsity, number of non-zero
parameters, and amount of training data, which we validate empirically across
model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
characterize the "optimal sparsity", the sparsity level which yields the best
performance for a given effective model size and training budget. For a fixed
number of non-zero parameters, we identify that the optimal sparsity increases
with the amount of data used for training. We also extend our study to
different sparsity structures (such as the hardware-friendly n:m pattern) and
strategies (such as starting from a pretrained dense model). Our findings shed
light on the power and limitations of weight sparsity across various parameter
and computational settings, offering both theoretical understanding and
practical implications for leveraging sparsity towards computational efficiency
improvements.
Replacing softmax with ReLU in Vision Transformers
Authors: name: Mitchell Wortsman, name: Jaehoon Lee, name: Justin Gilmer, name: Simon Kornblith
Published: 2023-09-15 17:43:40

Previous research observed accuracy degradation when replacing the attention
softmax with a point-wise activation such as ReLU. In the context of vision
transformers, we find that this degradation is mitigated when dividing by
sequence length. Our experiments training small to large vision transformers on
ImageNet-21k indicate that ReLU-attention can approach or match the performance
of softmax-attention in terms of scaling behavior as a function of compute.
Sparse Autoencoders Find Highly Interpretable Features in Language
  Models
Authors: name: Hoagy Cunningham, name: Aidan Ewart, name: Logan Riggs, name: Robert Huben, name: Lee Sharkey
Published: 2023-09-15 17:56:55

One of the roadblocks to a better understanding of neural networks' internals
is polysemanticity, where neurons appear to activate in multiple,
semantically distinct contexts. Polysemanticity prevents us from identifying
concise, human-understandable explanations for what neural networks are doing
internally. One hypothesised cause of polysemanticity is
superposition, where neural networks represent more features than they
have neurons by assigning features to an overcomplete set of directions in
activation space, rather than to individual neurons. Here, we attempt to
identify those directions, using sparse autoencoders to reconstruct the
internal activations of a language model. These autoencoders learn sets of
sparsely activating features that are more interpretable and monosemantic than
directions identified by alternative approaches, where interpretability is
measured by automated methods. Ablating these features enables precise model
editing, for example, by removing capabilities such as pronoun prediction,
while disrupting model behaviour less than prior techniques. This work
indicates that it is possible to resolve superposition in language models using
a scalable, unsupervised method. Our method may serve as a foundation for
future mechanistic interpretability work, which we hope will enable greater
model transparency and steerability.
Compositional Foundation Models for Hierarchical Planning
Authors: name: Anurag Ajay, name: Seungwook Han, name: Yilun Du, name: Shaung Li, name: Abhi Gupta, name: Tommi Jaakkola, name: Josh Tenenbaum, name: Leslie Kaelbling, name: Akash Srivastava, name: Pulkit Agrawal
Published: 2023-09-15 17:44:05

To make effective decisions in novel environments with long-horizon goals, it
is crucial to engage in hierarchical reasoning across spatial and temporal
scales. This entails planning abstract subgoal sequences, visually reasoning
about the underlying plans, and executing actions in accordance with the
devised plan through visual-motor control. We propose Compositional Foundation
Models for Hierarchical Planning (HiP), a foundation model which leverages
multiple expert foundation model trained on language, vision and action data
individually jointly together to solve long-horizon tasks. We use a large
language model to construct symbolic plans that are grounded in the environment
through a large video diffusion model. Generated video plans are then grounded
to visual-motor control, through an inverse dynamics model that infers actions
from generated videos. To enable effective reasoning within this hierarchy, we
enforce consistency between the models via iterative refinement. We illustrate
the efficacy and adaptability of our approach in three different long-horizon
table-top manipulation tasks.
Connecting Large Language Models with Evolutionary Algorithms Yields
  Powerful Prompt Optimizers
Authors: name: Qingyan Guo, name: Rui Wang, name: Junliang Guo, name: Bei Li, name: Kaitao Song, name: Xu Tan, name: Guoqing Liu, name: Jiang Bian, name: Yujiu Yang
Published: 2023-09-15 16:50:09

Large Language Models (LLMs) excel in various tasks, but they rely on
carefully crafted prompts that often demand substantial human effort. To
automate this process, in this paper, we propose a novel framework for discrete
prompt optimization, called EvoPrompt, which borrows the idea of evolutionary
algorithms (EAs) as they exhibit good performance and fast convergence. To
enable EAs to work on discrete prompts, which are natural language expressions
that need to be coherent and human-readable, we connect LLMs with EAs. This
approach allows us to simultaneously leverage the powerful language processing
capabilities of LLMs and the efficient optimization performance of EAs.
Specifically, abstaining from any gradients or parameters, EvoPrompt starts
from a population of prompts and iteratively generates new prompts with LLMs
based on the evolutionary operators, improving the population based on the
development set. We optimize prompts for both closed- and open-source LLMs
including GPT-3.5 and Alpaca, on 9 datasets spanning language understanding and
generation tasks. EvoPrompt significantly outperforms human-engineered prompts
and existing methods for automatic prompt generation by up to 25% and 14%
respectively. Furthermore, EvoPrompt demonstrates that connecting LLMs with EAs
creates synergies, which could inspire further research on the combination of
LLMs and conventional algorithms.
LASER: LLM Agent with State-Space Exploration for Web Navigation
Authors: name: Kaixin Ma, name: Hongming Zhang, name: Hongwei Wang, name: Xiaoman Pan, name: Dong Yu
Published: 2023-09-15 05:44:08

Large language models (LLMs) have been successfully adapted for interactive
decision-making tasks like web navigation. While achieving decent performance,
previous methods implicitly assume a forward-only execution mode for the model,
where they only provide oracle trajectories as in-context examples to teach the
model how to reason in the interactive environment. Consequently, the model
could not handle more challenging scenarios not covered in the in-context
examples, e.g., mistakes, leading to sub-optimal performance. To address this
issue, we propose to model the interactive task as state space exploration,
where the LLM agent transitions among a pre-defined set of states by performing
actions to complete the task. This formulation enables flexible back-tracking,
allowing the model to easily recover from errors. We evaluate our proposed LLM
Agent with State-Space ExploRation (LASER) on the WebShop task. Experimental
results show that our LASER agent significantly outperforms previous methods
and closes the gap with human performance on the web navigation task.
Leveraging Contextual Information for Effective Entity Salience
  Detection
Authors: name: Rajarshi Bhowmik, name: Marco Ponza, name: Atharva Tendle, name: Anant Gupta, name: Rebecca Jiang, name: Xingyu Lu, name: Qian Zhao, name: Daniel Preotiuc-Pietro
Published: 2023-09-14 19:04:40

In text documents such as news articles, the content and key events usually
revolve around a subset of all the entities mentioned in a document. These
entities, often deemed as salient entities, provide useful cues of the
aboutness of a document to a reader. Identifying the salience of entities was
found helpful in several downstream applications such as search, ranking, and
entity-centric summarization, among others. Prior work on salient entity
detection mainly focused on machine learning models that require heavy feature
engineering. We show that fine-tuning medium-sized language models with a
cross-encoder style architecture yields substantial performance gains over
feature engineering approaches. To this end, we conduct a comprehensive
benchmarking of four publicly available datasets using models representative of
the medium-sized pre-trained language model family. Additionally, we show that
zero-shot prompting of instruction-tuned language models yields inferior
results, indicating the task's uniqueness and complexity.
A Data Source for Reasoning Embodied Agents
Authors: name: Jack Lanchantin, name: Sainbayar Sukhbaatar, name: Gabriel Synnaeve, name: Yuxuan Sun, name: Kavya Srinet, name: Arthur Szlam
Published: 2023-09-14 18:17:16

Recent progress in using machine learning models for reasoning tasks has been
driven by novel model architectures, large-scale pre-training protocols, and
dedicated reasoning datasets for fine-tuning. In this work, to further pursue
these advances, we introduce a new data generator for machine reasoning that
integrates with an embodied agent. The generated data consists of templated
text queries and answers, matched with world-states encoded into a database.
The world-states are a result of both world dynamics and the actions of the
agent. We show the results of several baseline models on instantiations of
train sets. These include pre-trained language models fine-tuned on a
text-formatted representation of the database, and graph-structured
Transformers operating on a knowledge-graph representation of the database. We
find that these models can answer some questions about the world-state, but
struggle with others. These results hint at new research directions in
designing neural reasoning models and database representations. Code to
generate the data will be released at github.com/facebookresearch/neuralmemory
Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping
Authors: name: Adam Rashid, name: Satvik Sharma, name: Chung Min Kim, name: Justin Kerr, name: Lawrence Chen, name: Angjoo Kanazawa, name: Ken Goldberg
Published: 2023-09-14 18:10:29

Grasping objects by a specific part is often crucial for safety and for
executing downstream tasks. Yet, learning-based grasp planners lack this
behavior unless they are trained on specific object part data, making it a
significant challenge to scale object diversity. Instead, we propose LERF-TOGO,
Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which
uses vision-language models zero-shot to output a grasp distribution over an
object given a natural language query. To accomplish this, we first reconstruct
a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D
language field queryable with text. However, LERF has no sense of objectness,
meaning its relevancy outputs often return incomplete activations over an
object which are insufficient for subsequent part queries. LERF-TOGO mitigates
this lack of spatial grouping by extracting a 3D object mask via DINO features
and then conditionally querying LERF on this mask to obtain a semantic
distribution over the object with which to rank grasps from an off-the-shelf
grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object
parts on 31 different physical objects, and find it selects grasps on the
correct part in 81% of all trials and grasps successfully in 69%. See the
project website at: lerftogo.github.io
Investigating Answerability of LLMs for Long-Form Question Answering
Authors: name: Meghana Moorthy Bhat, name: Rui Meng, name: Ye Liu, name: Yingbo Zhou, name: Semih Yavuz
Published: 2023-09-15 07:22:56

As we embark on a new era of LLMs, it becomes increasingly crucial to
understand their capabilities, limitations, and differences. Toward making
further progress in this direction, we strive to build a deeper understanding
of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective
open-source LLMs and their distilled counterparts. To this end, we specifically
focus on long-form question answering (LFQA) because it has several practical
and impactful applications (e.g., troubleshooting, customer service, etc.) yet
is still understudied and challenging for LLMs. We propose a
question-generation method from abstractive summaries and show that generating
follow-up questions from summaries of long documents can create a challenging
setting for LLMs to reason and infer from long contexts. Our experimental
results confirm that: (1) our proposed method of generating questions from
abstractive summaries pose a challenging setup for LLMs and shows performance
gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2)
open-source LLMs exhibit decreased reliance on context for generated questions
from the original document, but their generation capabilities drop
significantly on generated questions from summaries -- especially for longer
contexts (>1024 tokens)
OmnimatteRF: Robust Omnimatte with 3D Background Modeling
Authors: name: Geng Lin, name: Chen Gao, name: Jia-Bin Huang, name: Changil Kim, name: Yipeng Wang, name: Matthias Zwicker, name: Ayush Saraf
Published: 2023-09-14 14:36:22

Video matting has broad applications, from adding interesting effects to
casually captured movies to assisting video production professionals. Matting
with associated effects such as shadows and reflections has also attracted
increasing research activity, and methods like Omnimatte have been proposed to
separate dynamic foreground objects of interest into their own layers. However,
prior works represent video backgrounds as 2D image layers, limiting their
capacity to express more complicated scenes, thus hindering application to
real-world videos. In this paper, we propose a novel video matting method,
OmnimatteRF, that combines dynamic 2D foreground layers and a 3D background
model. The 2D layers preserve the details of the subjects, while the 3D
background robustly reconstructs scenes in real-world videos. Extensive
experiments demonstrate that our method reconstructs scenes with better quality
on various videos.
Generative Image Dynamics
Authors: name: Zhengqi Li, name: Richard Tucker, name: Noah Snavely, name: Aleksander Holynski
Published: 2023-09-14 17:54:01

We present an approach to modeling an image-space prior on scene dynamics.
Our prior is learned from a collection of motion trajectories extracted from
real video sequences containing natural, oscillating motion such as trees,
flowers, candles, and clothes blowing in the wind. Given a single image, our
trained model uses a frequency-coordinated diffusion sampling process to
predict a per-pixel long-term motion representation in the Fourier domain,
which we call a neural stochastic motion texture. This representation can be
converted into dense motion trajectories that span an entire video. Along with
an image-based rendering module, these trajectories can be used for a number of
downstream applications, such as turning still images into seamlessly looping
dynamic videos, or allowing users to realistically interact with objects in
real pictures.
AudioSR: Versatile Audio Super-resolution at Scale
Authors: name: Haohe Liu, name: Ke Chen, name: Qiao Tian, name: Wenwu Wang, name: Mark D. Plumbley
Published: 2023-09-13 21:00:09

Audio super-resolution is a fundamental task that predicts high-frequency
components for low-resolution audio, enhancing audio quality in digital
applications. Previous methods have limitations such as the limited scope of
audio types (e.g., music, speech) and specific bandwidth settings they can
handle (e.g., 4kHz to 8kHz). In this paper, we introduce a diffusion-based
generative model, AudioSR, that is capable of performing robust audio
super-resolution on versatile audio types, including sound effects, music, and
speech. Specifically, AudioSR can upsample any input audio signal within the
bandwidth range of 2kHz to 16kHz to a high-resolution audio signal at 24kHz
bandwidth with a sampling rate of 48kHz. Extensive objective evaluation on
various audio super-resolution benchmarks demonstrates the strong result
achieved by the proposed model. In addition, our subjective evaluation shows
that AudioSR can acts as a plug-and-play module to enhance the generation
quality of a wide range of audio generative models, including AudioLDM,
Fastspeech2, and MusicGen. Our code and demo are available at
https://audioldm.github.io/audiosr.
Ambiguity-Aware In-Context Learning with Large Language Models
Authors: name: Lingyu Gao, name: Aditi Chaudhary, name: Krishna Srinivasan, name: Kazuma Hashimoto, name: Karthik Raman, name: Michael Bendersky
Published: 2023-09-14 17:48:34

In-context learning (ICL) i.e. showing LLMs only a few task-specific
demonstrations has led to downstream gains with no task-specific fine-tuning
required. However, LLMs are sensitive to the choice of prompts, and therefore a
crucial research question is how to select good demonstrations for ICL. One
effective strategy is leveraging semantic similarity between the ICL
demonstrations and test inputs by using a text retriever, which however is
sub-optimal as that does not consider the LLM's existing knowledge about that
task. From prior work (Min et al., 2022), we already know that labels paired
with the demonstrations bias the model predictions. This leads us to our
hypothesis whether considering LLM's existing knowledge about the task,
especially with respect to the output label space can help in a better
demonstration selection strategy. Through extensive experimentation on three
text classification tasks, we find that it is beneficial to not only choose
semantically similar ICL demonstrations but also to choose those demonstrations
that help resolve the inherent label ambiguity surrounding the test example.
Interestingly, we find that including demonstrations that the LLM previously
mis-classified and also fall on the test example's decision boundary, brings
the most performance gain.