Orionnnn/hub-stats huggingface dataset:
Grammar Prompting for Domain-Specific Language Generation with Large
  Language Models
Authors: name: Bailin Wang, name: Zi Wang, name: Xuezhi Wang, name: Yuan Cao, name: Rif A. Saurous, name: Yoon Kim
Published: 2023-05-30 17:26:01

Large language models (LLMs) can learn to perform a wide range of natural
language tasks from just a handful of in-context examples. However, for
generating strings from highly structured languages (e.g., semantic parsing to
complex domain-specific languages), it is challenging for the LLM to generalize
from just a few exemplars. We explore grammar prompting as a simple
approach for enabling LLMs to use external knowledge and domain-specific
constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
during in-context learning. Grammar prompting augments each demonstration
example with a specialized grammar that is minimally sufficient for generating
the particular output example, where the specialized grammar is a subset of the
full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
test input, and then generates the output according to the rules of the
grammar. Experiments demonstrate that grammar prompting can enable LLMs to
perform competitively on a diverse set of DSL generation tasks, including
semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
molecule generation (SMILES).
LANCE: Stress-testing Visual Models by Generating Language-guided
  Counterfactual Images
Authors: name: Viraj Prabhu, name: Sriram Yenamandra, name: Prithvijit Chattopadhyay, name: Judy Hoffman
Published: 2023-05-30 16:09:16

We propose an automated algorithm to stress-test a trained visual model by
generating language-guided counterfactual test images (LANCE). Our method
leverages recent progress in large language modeling and text-based image
editing to augment an IID test set with a suite of diverse, realistic, and
challenging test images without altering model weights. We benchmark the
performance of a diverse set of pretrained models on our generated data and
observe significant and consistent performance drops. We further analyze model
sensitivity across different types of edits, and demonstrate its applicability
at surfacing previously unknown class-level model biases in ImageNet.
Faith and Fate: Limits of Transformers on Compositionality
Authors: name: Nouha Dziri, name: Ximing Lu, name: Melanie Sclar, name: Xiang Lorraine Li, name: Liwei Jian, name: Bill Yuchen Lin, name: Peter West, name: Chandra Bhagavatula, name: Ronan Le Bras, name: Jena D. Hwang, name: Soumya Sanyal, name: Sean Welleck, name: Xiang Ren, name: Allyson Ettinger, name: Zaid Harchaoui, name: Yejin Choi
Published: 2023-05-29 23:24:14

Transformer large language models (LLMs) have sparked admiration for their
exceptional performance on tasks that demand intricate multi-step reasoning.
Yet, these models simultaneously show failures on surprisingly trivial
problems. This begs the question: Are these errors incidental, or do they
signal more substantial limitations? In an attempt to demystify Transformers,
we investigate the limits of these models across three representative
compositional tasks -- multi-digit multiplication, logic grid puzzles, and a
classic dynamic programming problem. These tasks require breaking problems down
into sub-steps and synthesizing these steps into a precise answer. We formulate
compositional tasks as computation graphs to systematically quantify the level
of complexity, and break down reasoning steps into intermediate sub-procedures.
Our empirical findings suggest that Transformers solve compositional tasks by
reducing multi-step compositional reasoning into linearized subgraph matching,
without necessarily developing systematic problem-solving skills. To round off
our empirical study, we provide theoretical arguments on abstract multi-step
reasoning problems that highlight how Transformers' performance will rapidly
decay with increased task complexity.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors: name: Xi Chen, name: Josip Djolonga, name: Piotr Padlewski, name: Basil Mustafa, name: Soravit Changpinyo, name: Jialin Wu, name: Carlos Riquelme Ruiz, name: Sebastian Goodman, name: Xiao Wang, name: Yi Tay, name: Siamak Shakeri, name: Mostafa Dehghani, name: Daniel Salz, name: Mario Lucic, name: Michael Tschannen, name: Arsha Nagrani, name: Hexiang Hu, name: Mandar Joshi, name: Bo Pang, name: Ceslee Montgomery, name: Paulina Pietrzyk, name: Marvin Ritter, name: AJ Piergiovanni, name: Matthias Minderer, name: Filip Pavetic, name: Austin Waters, name: Gang Li, name: Ibrahim Alabdulmohsin, name: Lucas Beyer, name: Julien Amelot, name: Kenton Lee, name: Andreas Peter Steiner, name: Yang Li, name: Daniel Keysers, name: Anurag Arnab, name: Yuanzhong Xu, name: Keran Rong, name: Alexander Kolesnikov, name: Mojtaba Seyedhosseini, name: Anelia Angelova, name: Xiaohua Zhai, name: Neil Houlsby, name: Radu Soricut
Published: 2023-05-29 18:58:38

We present the training recipe and results of scaling up PaLI-X, a
multilingual vision and language model, both in terms of size of the components
and the breadth of its training task mixture. Our model achieves new levels of
performance on a wide-range of varied and complex tasks, including multiple
image-based captioning and question-answering tasks, image-based document
understanding and few-shot (in-context) learning, as well as object detection,
video question answering, and video captioning. PaLI-X advances the
state-of-the-art on most vision-and-language benchmarks considered (25+ of
them). Finally, we observe emerging capabilities, such as complex counting and
multilingual object detection, tasks that are not explicitly in the training
mix.
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature
  Adaptation of Vision-Language Models
Authors: name: Zhiwei Jia, name: Pradyumna Narayana, name: Arjun R. Akula, name: Garima Pruthi, name: Hao Su, name: Sugato Basu, name: Varun Jampani
Published: 2023-05-28 04:49:01

Image ad understanding is a crucial task with wide real-world applications.
Although highly challenging with the involvement of diverse atypical scenes,
real-world entities, and reasoning over scene-texts, how to interpret image ads
is relatively under-explored, especially in the era of foundational
vision-language models (VLMs) featuring impressive generalizability and
adaptability. In this paper, we perform the first empirical study of image ad
understanding through the lens of pre-trained VLMs. We benchmark and reveal
practical challenges in adapting these VLMs to image ad understanding. We
propose a simple feature adaptation strategy to effectively fuse multimodal
information for image ads and further empower it with knowledge of real-world
entities. We hope our study draws more attention to image ad understanding
which is broadly relevant to the advertising industry.
HiFA: High-fidelity Text-to-3D with Advanced Diffusion Guidance
Authors: name: Joseph Zhu, name: Peiye Zhuang
Published: 2023-05-30 05:56:58

Automatic text-to-3D synthesis has achieved remarkable advancements through
the optimization of 3D models. Existing methods commonly rely on pre-trained
text-to-image generative models, such as diffusion models, providing scores for
2D renderings of Neural Radiance Fields (NeRFs) and being utilized for
optimizing NeRFs. However, these methods often encounter artifacts and
inconsistencies across multiple views due to their limited understanding of 3D
geometry. To address these limitations, we propose a reformulation of the
optimization loss using the diffusion prior. Furthermore, we introduce a novel
training approach that unlocks the potential of the diffusion prior. To improve
3D geometry representation, we apply auxiliary depth supervision for
NeRF-rendered images and regularize the density field of NeRFs. Extensive
experiments demonstrate the superiority of our method over prior works,
resulting in advanced photo-realism and improved multi-view consistency.
StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity
  3D Avatar Generation
Authors: name: Chi Zhang, name: Yiwen Chen, name: Yijun Fu, name: Zhenglin Zhou, name: Gang YU, name: Billzb Wang, name: Bin Fu, name: Tao Chen, name: Guosheng Lin, name: Chunhua Shen
Published: 2023-05-30 13:09:21

The recent advancements in image-text diffusion models have stimulated
research interest in large-scale 3D generative models. Nevertheless, the
limited availability of diverse 3D resources presents significant challenges to
learning. In this paper, we present a novel method for generating high-quality,
stylized 3D avatars that utilizes pre-trained image-text diffusion models for
data generation and a Generative Adversarial Network (GAN)-based 3D generation
network for training. Our method leverages the comprehensive priors of
appearance and geometry offered by image-text diffusion models to generate
multi-view images of avatars in various styles. During data generation, we
employ poses extracted from existing 3D models to guide the generation of
multi-view images. To address the misalignment between poses and images in
data, we investigate view-specific prompts and develop a coarse-to-fine
discriminator for GAN training. We also delve into attribute-related prompts to
increase the diversity of the generated avatars. Additionally, we develop a
latent diffusion model within the style space of StyleGAN to enable the
generation of avatars based on image inputs. Our approach demonstrates superior
performance over current state-of-the-art methods in terms of visual quality
and diversity of the produced avatars.
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Authors: name: Jiawei Huang, name: Yi Ren, name: Rongjie Huang, name: Dongchao Yang, name: Zhenhui Ye, name: Chen Zhang, name: Jinglin Liu, name: Xiang Yin, name: Zejun Ma, name: Zhou Zhao
Published: 2023-05-29 10:41:28

Large diffusion models have been successful in text-to-audio (T2A) synthesis
tasks, but they often suffer from common issues such as semantic misalignment
and poor temporal consistency due to limited natural language understanding and
data scarcity. Additionally, 2D spatial structures widely used in T2A works
lead to unsatisfactory audio quality when generating variable-length audio
samples since they do not adequately prioritize temporal information. To
address these challenges, we propose Make-an-Audio 2, a latent diffusion-based
T2A method that builds on the success of Make-an-Audio. Our approach includes
several techniques to improve semantic alignment and temporal consistency:
Firstly, we use pre-trained large language models (LLMs) to parse the text into
structured <event & order> pairs for better temporal information capture. We
also introduce another structured-text encoder to aid in learning semantic
alignment during the diffusion denoising process. To improve the performance of
variable length generation and enhance the temporal information extraction, we
design a feed-forward Transformer-based diffusion denoiser. Finally, we use
LLMs to augment and transform a large amount of audio-label data into
audio-text datasets to alleviate the problem of scarcity of temporal data.
Extensive experiments show that our method outperforms baseline models in both
objective and subjective metrics, and achieves significant gains in temporal
information understanding, semantic consistency, and sound quality.
Nested Diffusion Processes for Anytime Image Generation
Authors: name: Noam Elata, name: Bahjat Kawar, name: Tomer Michaeli, name: Michael Elad
Published: 2023-05-30 14:28:43

Diffusion models are the current state-of-the-art in image generation,
synthesizing high-quality images by breaking down the generation process into
many fine-grained denoising steps. Despite their good performance, diffusion
models are computationally expensive, requiring many neural function
evaluations (NFEs). In this work, we propose an anytime diffusion-based method
that can generate viable images when stopped at arbitrary times before
completion. Using existing pretrained diffusion models, we show that the
generation scheme can be recomposed as two nested diffusion processes, enabling
fast iterative refinement of a generated image. We use this Nested Diffusion
approach to peek into the generation process and enable flexible scheduling
based on the instantaneous preference of the user. In experiments on ImageNet
and Stable Diffusion-based text-to-image generation, we show, both
qualitatively and quantitatively, that our method's intermediate generation
quality greatly exceeds that of the original diffusion model, while the final
slow generation result remains comparable.
Real-World Image Variation by Aligning Diffusion Inversion Chain
Authors: name: Yuechen Zhang, name: Jinbo Xing, name: Eric Lo, name: Jiaya Jia
Published: 2023-05-30 04:09:47

Recent diffusion model advancements have enabled high-fidelity images to be
generated using text prompts. However, a domain gap exists between generated
images and real-world images, which poses a challenge in generating
high-quality variations of real-world images. Our investigation uncovers that
this domain gap originates from a latents' distribution gap in different
diffusion processes. To address this issue, we propose a novel inference
pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes
diffusion models to generate image variations from a single image exemplar. Our
pipeline enhances the generation quality of image variations by aligning the
image generation process to the source image's inversion chain. Specifically,
we demonstrate that step-wise latent distribution alignment is essential for
generating high-quality variations. To attain this, we design a cross-image
self-attention injection for feature interaction and a step-wise distribution
normalization to align the latent features. Incorporating these alignment
processes into a diffusion model allows RIVAL to generate high-quality image
variations without further parameter optimization. Our experimental results
demonstrate that our proposed approach outperforms existing methods with
respect to semantic-condition similarity and perceptual quality. Furthermore,
this generalized inference pipeline can be easily applied to other
diffusion-based generation tasks, such as image-conditioned text-to-image
generation and example-based image inpainting.
GPT4Tools: Teaching Large Language Model to Use Tools via
  Self-instruction
Authors: name: Rui Yang, name: Lin Song, name: Yanwei Li, name: Sijie Zhao, name: Yixiao Ge, name: Xiu Li, name: Ying Shan
Published: 2023-05-30 05:27:21

This paper aims to efficiently enable Large Language Models (LLMs) to use
multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have
shown great potential for tool usage through sophisticated prompt engineering.
Nevertheless, these models typically rely on prohibitive computational costs
and publicly inaccessible data. To address these challenges, we propose the
GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and
OPT, to use tools. It generates an instruction-following dataset by prompting
an advanced teacher with various multi-modal contexts. By using the Low-Rank
Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs
to solve a range of visual problems, including visual comprehension and image
generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to
use tools, which is performed in both zero-shot and fine-tuning ways. Extensive
experiments demonstrate the effectiveness of our method on various language
models, which not only significantly improves the accuracy of invoking seen
tools, but also enables the zero-shot capacity for unseen tools. The code and
demo are available at https://github.com/StevenGrove/GPT4Tools.
Controllable Text-to-Image Generation with GPT-4
Authors: name: Tianjun Zhang, name: Yi Zhang, name: Vibhav Vineet, name: Neel Joshi, name: Xin Wang
Published: 2023-05-29 19:56:47

Current text-to-image generation models often struggle to follow textual
instructions, especially the ones requiring spatial reasoning. On the other
hand, Large Language Models (LLMs), such as GPT-4, have shown remarkable
precision in generating code snippets for sketching out text inputs
graphically, e.g., via TikZ. In this work, we introduce Control-GPT to guide
the diffusion-based text-to-image pipelines with programmatic sketches
generated by GPT-4, enhancing their abilities for instruction following.
Control-GPT works by querying GPT-4 to write TikZ code, and the generated
sketches are used as references alongside the text instructions for diffusion
models (e.g., ControlNet) to generate photo-realistic images. One major
challenge to training our pipeline is the lack of a dataset containing aligned
text, images, and sketches. We address the issue by converting instance masks
in existing datasets into polygons to mimic the sketches used at test time. As
a result, Control-GPT greatly boosts the controllability of image generation.
It establishes a new state-of-art on the spatial arrangement and object
positioning generation and enhances users' control of object positions, sizes,
etc., nearly doubling the accuracy of prior models. Our work, as a first
attempt, shows the potential for employing LLMs to enhance the performance in
computer vision tasks.
What indeed can GPT models do in chemistry? A comprehensive benchmark on
  eight tasks
Authors: name: Taicheng Guo, name: Kehan Guo, name: Bozhao nan, name: Zhengwen Liang, name: Zhichun Guo, name: Nitesh V. Chawla, name: Olaf Wiest, name: Xiangliang Zhang
Published: 2023-05-27 14:17:33

Large Language Models (LLMs) with strong abilities in natural language
processing tasks have emerged and have been rapidly applied in various kinds of
areas such as science, finance and software engineering. However, the
capability of LLMs to advance the field of chemistry remains unclear. In this
paper,we establish a comprehensive benchmark containing 8 practical chemistry
tasks, including 1) name prediction, 2) property prediction, 3) yield
prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
from products), 6)text-based molecule design, 7) molecule captioning, and 8)
reagent selection. Our analysis draws on widely recognized datasets including
BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
capacities of LLMs within the context of practical chemistry. Three GPT models
(GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
zero-shot and few-shot in-context learning settings with carefully selected
demonstration examples and specially crafted prompts. The key results of our
investigation are 1) GPT-4 outperforms the other two models among the three
evaluated; 2) GPT models exhibit less competitive performance in tasks
demanding precise understanding of molecular SMILES representation, such as
reaction prediction and retrosynthesis;3) GPT models demonstrate strong
capabilities in text-related explanation tasks such as molecule captioning; and
4) GPT models exhibit comparable or better performance to classical machine
learning models when applied to chemical problems that can be transformed into
classification or ranking tasks, such as property prediction, and yield
prediction.
Fine-Tuning Language Models with Just Forward Passes
Authors: name: Sadhika Malladi, name: Tianyu Gao, name: Eshaan Nichani, name: Alex Damian, name: Jason D. Lee, name: Danqi Chen, name: Sanjeev Arora
Published: 2023-05-27 02:28:10

Fine-tuning language models (LMs) has yielded success on diverse downstream
tasks, but as LMs grow in size, backpropagation requires a prohibitively large
amount of memory. Zeroth-order (ZO) methods can in principle estimate gradients
using only two forward passes but are theorized to be catastrophically slow for
optimizing large models. In this work, we propose a memory-efficient
zerothorder optimizer (MeZO), adapting the classical ZO-SGD method to operate
in-place, thereby fine-tuning LMs with the same memory footprint as inference.
For example, with a single A100 80GB GPU, MeZO can train a 30-billion parameter
model, whereas fine-tuning with backpropagation can train only a 2.7B LM with
the same budget. We conduct comprehensive experiments across model types
(masked and autoregressive LMs), model scales (up to 66B), and downstream tasks
(classification, multiple-choice, and generation). Our results demonstrate that
(1) MeZO significantly outperforms in-context learning and linear probing; (2)
MeZO achieves comparable performance to fine-tuning with backpropagation across
multiple tasks, with up to 12x memory reduction; (3) MeZO is compatible with
both full-parameter and parameter-efficient tuning techniques such as LoRA and
prefix tuning; (4) MeZO can effectively optimize non-differentiable objectives
(e.g., maximizing accuracy or F1). We support our empirical findings with
theoretical insights, highlighting how adequate pre-training and task prompts
enable MeZO to fine-tune huge models, despite classical ZO analyses suggesting
otherwise.
Ghost in the Minecraft: Generally Capable Agents for Open-World
  Enviroments via Large Language Models with Text-based Knowledge and Memory
Authors: name: Xizhou Zhu, name: Yuntao Chen, name: Hao Tian, name: Chenxin Tao, name: Weijie Su, name: Chenyu Yang, name: Gao Huang, name: Bin Li, name: Lewei Lu, name: Xiaogang Wang, name: Yu Qiao, name: Zhaoxiang Zhang, name: Jifeng Dai
Published: 2023-05-25 17:59:49

The captivating realm of Minecraft has attracted substantial research
interest in recent years, serving as a rich platform for developing intelligent
agents capable of functioning in open-world environments. However, the current
research landscape predominantly focuses on specific objectives, such as the
popular "ObtainDiamond" task, and has not yet shown effective generalization to
a broader spectrum of tasks. Furthermore, the current leading success rate for
the "ObtainDiamond" task stands at around 20%, highlighting the limitations of
Reinforcement Learning (RL) based controllers used in existing methods. To
tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel
framework integrates Large Language Models (LLMs) with text-based knowledge and
memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These
agents, equipped with the logic and common sense capabilities of LLMs, can
skillfully navigate complex, sparse-reward environments with text-based
interactions. We develop a set of structured actions and leverage LLMs to
generate action plans for the agents to execute. The resulting LLM-based agent
markedly surpasses previous methods, achieving a remarkable improvement of
+47.5% in success rate on the "ObtainDiamond" task, demonstrating superior
robustness compared to traditional RL-based controllers. Notably, our agent is
the first to procure all items in the Minecraft Overworld technology tree,
demonstrating its extensive capabilities. GITM does not need any GPU for
training, but a single CPU node with 32 CPU cores is enough. This research
shows the potential of LLMs in developing capable agents for handling
long-horizon, complex tasks and adapting to uncertainties in open-world
environments. See the project website at https://github.com/OpenGVLab/GITM.