Orionnnn/hub-stats huggingface dataset:
LLaVA-Med: Training a Large Language-and-Vision Assistant for
  Biomedicine in One Day
Authors: name: Chunyuan Li, name: Cliff Wong, name: Sheng Zhang, name: Naoto Usuyama, name: Haotian Liu, name: Jianwei Yang, name: Tristan Naumann, name: Hoifung Poon, name: Jianfeng Gao
Published: 2023-06-01 16:50:07

Conversational generative AI has demonstrated remarkable promise for
empowering biomedical practitioners, but current investigations focus on
unimodal text. Multimodal conversational AI has seen rapid progress by
leveraging billions of image-text pairs from the public web, but such
general-domain vision-language models still lack sophistication in
understanding and conversing about biomedical images. In this paper, we propose
a cost-efficient approach for training a vision-language conversational
assistant that can answer open-ended research questions of biomedical images.
The key idea is to leverage a large-scale, broad-coverage biomedical
figure-caption dataset extracted from PubMed Central, use GPT-4 to
self-instruct open-ended instruction-following data from the captions, and then
fine-tune a large general-domain vision-language model using a novel curriculum
learning method. Specifically, the model first learns to align biomedical
vocabulary using the figure-caption pairs as is, then learns to master
open-ended conversational semantics using GPT-4 generated instruction-following
data, broadly mimicking how a layperson gradually acquires biomedical
knowledge. This enables us to train a Large Language and Vision Assistant for
BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med
exhibits excellent multimodal conversational capability and can follow
open-ended instruction to assist with inquiries about a biomedical image. On
three standard biomedical visual question answering datasets, LLaVA-Med
outperforms previous supervised state-of-the-art on certain metrics. To
facilitate biomedical multimodal research, we will release our
instruction-following data and the LLaVA-Med model.
ReviewerGPT? An Exploratory Study on Using Large Language Models for
  Paper Reviewing
Authors: name: Ryan Liu, name: Nihar B. Shah
Published: 2023-06-01 12:45:53

Given the rapid ascent of large language models (LLMs), we study the
question: (How) can large language models help in reviewing of scientific
papers or proposals? We first conduct some pilot studies where we find that (i)
GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly,
OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to
identify errors) outperforms prompting to simply write a review. With these
insights, we study the use of LLMs (specifically, GPT-4) for three tasks:
  1. Identifying errors: We construct 13 short computer science papers each
with a deliberately inserted error, and ask the LLM to check for the
correctness of these papers. We observe that the LLM finds errors in 7 of them,
spanning both mathematical and conceptual errors.
  2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist
questions in the respective sections of 15 NeurIPS 2022 papers. We find that
across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy.
  3. Choosing the "better" paper: We generate 10 pairs of abstracts,
deliberately designing each pair in such a way that one abstract was clearly
superior than the other. The LLM, however, struggled to discern these
relatively straightforward distinctions accurately, committing errors in its
evaluations for 6 out of the 10 pairs.
  Based on these experiments, we think that LLMs have a promising use as
reviewing assistants for specific reviewing tasks, but not (yet) for complete
evaluations of papers or proposals.
StyleDrop: Text-to-Image Generation in Any Style
Authors: name: Kihyuk Sohn, name: Nataniel Ruiz, name: Kimin Lee, name: Daniel Castro Chin, name: Irina Blok, name: Huiwen Chang, name: Jarred Barber, name: Lu Jiang, name: Glenn Entis, name: Yuanzhen Li, name: Yuan Hao, name: Irfan Essa, name: Michael Rubinstein, name: Dilip Krishnan
Published: 2023-06-01 17:59:51

Pre-trained large text-to-image models synthesize impressive images with an
appropriate use of text prompts. However, ambiguities inherent in natural
language and out-of-distribution effects make it hard to synthesize image
styles, that leverage a specific design pattern, texture or material. In this
paper, we introduce StyleDrop, a method that enables the synthesis of images
that faithfully follow a specific style using a text-to-image model. The
proposed method is extremely versatile and captures nuances and details of a
user-provided style, such as color schemes, shading, design patterns, and local
and global effects. It efficiently learns a new style by fine-tuning very few
trainable parameters (less than 1% of total model parameters) and improving
the quality via iterative training with either human or automated feedback.
Better yet, StyleDrop is able to deliver impressive results even when the user
supplies only a single image that specifies the desired style. An extensive
study shows that, for the task of style tuning text-to-image models, StyleDrop
implemented on Muse convincingly outperforms other methods, including
DreamBooth and textual inversion on Imagen or Stable Diffusion. More results
are available at our project website: https://styledrop.github.io
Humans in 4D: Reconstructing and Tracking Humans with Transformers
Authors: name: Shubham Goel, name: Georgios Pavlakos, name: Jathushan Rajasegaran, name: Angjoo Kanazawa, name: Jitendra Malik
Published: 2023-05-31 17:59:52

We present an approach to reconstruct humans and track them over time. At the
core of our approach, we propose a fully "transformerized" version of a network
for human mesh recovery. This network, HMR 2.0, advances the state of the art
and shows the capability to analyze unusual poses that have in the past been
difficult to reconstruct from single images. To analyze video, we use 3D
reconstructions from HMR 2.0 as input to a tracking system that operates in 3D.
This enables us to deal with multiple people and maintain identities through
occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art
results for tracking people from monocular video. Furthermore, we demonstrate
the effectiveness of HMR 2.0 on the downstream task of action recognition,
achieving significant improvements over previous pose-based action recognition
approaches. Our code and models are available on the project website:
https://shubham-goel.github.io/4dhumans/.
Bigger, Better, Faster: Human-level Atari with human-level efficiency
Authors: name: Max Schwarzer, name: Johan Obando-Ceron, name: Aaron Courville, name: Marc Bellemare, name: Rishabh Agarwal, name: Pablo Samuel Castro
Published: 2023-05-30 23:23:25

We introduce a value-based RL agent, which we call BBF, that achieves
super-human performance in the Atari 100K benchmark. BBF relies on scaling the
neural networks used for value estimation, as well as a number of other design
choices that enable this scaling in a sample-efficient manner. We conduct
extensive analyses of these design choices and provide insights for future
work. We end with a discussion about updating the goalposts for
sample-efficient RL research on the ALE. We make our code and data publicly
available at
https://github.com/google-research/google-research/tree/master/bigger_better_faster.
Improving CLIP Training with Language Rewrites
Authors: name: Lijie Fan, name: Dilip Krishnan, name: Phillip Isola, name: Dina Katabi, name: Yonglong Tian
Published: 2023-05-31 17:59:04

Contrastive Language-Image Pre-training (CLIP) stands as one of the most
effective and scalable methods for training transferable vision models using
paired image and text data. CLIP models are trained using contrastive loss,
which typically relies on data augmentations to prevent overfitting and
shortcuts. However, in the CLIP training paradigm, data augmentations are
exclusively applied to image inputs, while language inputs remain unchanged
throughout the entire training process, limiting the exposure of diverse texts
to the same image. In this paper, we introduce Language augmented CLIP
(LaCLIP), a simple yet highly effective approach to enhance CLIP training
through language rewrites. Leveraging the in-context learning capability of
large language models, we rewrite the text descriptions associated with each
image. These rewritten texts exhibit diversity in sentence structure and
vocabulary while preserving the original key concepts and meanings. During
training, LaCLIP randomly selects either the original texts or the rewritten
versions as text augmentations for each image. Extensive experiments on CC3M,
CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with
language rewrites significantly improves the transfer performance without
computation or memory overhead during training. Specifically for ImageNet
zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on
LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.
Human or Not? A Gamified Approach to the Turing Test
Authors: name: Daniel Jannai, name: Amos Meron, name: Barak Lenz, name: Yoav Levine, name: Yoav Shoham
Published: 2023-05-31 16:32:22

We present "Human or Not?", an online game inspired by the Turing test, that
measures the capability of AI chatbots to mimic humans in dialog, and of humans
to tell bots from other humans. Over the course of a month, the game was played
by over 1.5 million users who engaged in anonymous two-minute chat sessions
with either another human or an AI language model which was prompted to behave
like humans. The task of the players was to correctly guess whether they spoke
to a person or to an AI. This largest scale Turing-style test conducted to date
revealed some interesting facts. For example, overall users guessed the
identity of their partners correctly in only 68% of the games. In the subset of
the games in which users faced an AI bot, users had even lower correct guess
rates of 60% (that is, not much higher than chance). This white paper details
the development, deployment, and results of this unique experiment. While this
experiment calls for many extensions and refinements, these findings already
begin to shed light on the inevitable near future which will commingle humans
and AI.
Deliberate then Generate: Enhanced Prompting Framework for Text
  Generation
Authors: name: Bei Li, name: Rui Wang, name: Junliang Guo, name: Kaitao Song, name: Xu Tan, name: Hany Hassan, name: Arul Menezes, name: Tong Xiao, name: Jiang Bian, name: JingBo Zhu
Published: 2023-05-31 13:23:04

Large language models (LLMs) have shown remarkable success across a wide
range of natural language generation tasks, where proper prompt designs make
great impacts. While existing prompting methods are normally restricted to
providing correct information, in this paper, we encourage the model to
deliberate by proposing a novel Deliberate then Generate (DTG) prompting
framework, which consists of error detection instructions and candidates that
may contain errors. DTG is a simple yet effective technique that can be applied
to various text generation tasks with minimal modifications. We conduct
extensive experiments on 20+ datasets across 7 text generation tasks, including
summarization, translation, dialogue, and more. We show that DTG consistently
outperforms existing prompting methods and achieves state-of-the-art
performance on multiple text generation tasks. We also provide in-depth
analyses to reveal the underlying mechanisms of DTG, which may inspire future
research on prompting for LLMs.
PlaSma: Making Small Language Models Better Procedural Knowledge Models
  for (Counterfactual) Planning
Authors: name: Faeze Brahman, name: Chandra Bhagavatula, name: Valentina Pyatkin, name: Jena D. Hwang, name: Xiang Lorraine Li, name: Hirona J. Arai, name: Soumya Sanyal, name: Keisuke Sakaguchi, name: Xiang Ren, name: Yejin Choi
Published: 2023-05-31 00:55:40

Procedural planning, which entails decomposing a high-level goal into a
sequence of temporally ordered steps, is an important yet intricate task for
machines. It involves integrating common-sense knowledge to reason about
complex contextualized situations that are often counterfactual, e.g.
"scheduling a doctor's appointment without a phone". While current approaches
show encouraging results using large language models (LLMs), they are hindered
by drawbacks such as costly API calls and reproducibility issues. In this
paper, we advocate planning using smaller language models. We present PlaSma, a
novel two-pronged approach to endow small language models with procedural
knowledge and (counterfactual) planning capabilities. More concretely, we
develop symbolic procedural knowledge distillation to enhance the implicit
knowledge in small language models and an inference-time algorithm to
facilitate more structured and accurate reasoning. In addition, we introduce a
novel task, Counterfactual Planning, that requires a revision of a plan to cope
with a counterfactual situation. In both the original and counterfactual
setting, we show that orders-of-magnitude smaller models (770M-11B parameters)
can compete and often surpass their larger teacher models' capabilities.
Blockwise Parallel Transformer for Long Context Large Models
Authors: name: Hao Liu, name: Pieter Abbeel
Published: 2023-05-30 19:25:51

Transformers have emerged as the cornerstone of state-of-the-art natural
language processing models, showcasing exceptional performance across a wide
range of AI applications. However, the memory demands posed by the
self-attention mechanism and the large feedforward network in Transformers
limit their ability to handle long sequences, thereby creating challenges for
tasks involving multiple long sequences or long-term dependencies. We present a
distinct approach, Blockwise Parallel Transformer (BPT), that leverages
blockwise computation of self-attention and feedforward network fusion to
minimize memory costs. By processing longer input sequences while maintaining
memory efficiency, BPT enables training sequences up to 32 times longer than
vanilla Transformers and 2 to 4 times longer than previous memory-efficient
methods. Extensive experiments on language modeling and reinforcement learning
tasks demonstrate the effectiveness of BPT in reducing memory requirements and
improving performance.
Understanding and Mitigating Copying in Diffusion Models
Authors: name: Gowthami Somepalli, name: Vasu Singla, name: Micah Goldblum, name: Jonas Geiping, name: Tom Goldstein
Published: 2023-05-31 17:58:02

Images generated by diffusion models like Stable Diffusion are increasingly
widespread. Recent works and even lawsuits have shown that these models are
prone to replicating their training data, unbeknownst to the user. In this
paper, we first analyze this memorization problem in text-to-image diffusion
models. While it is widely believed that duplicated images in the training set
are responsible for content replication at inference time, we observe that the
text conditioning of the model plays a similarly important role. In fact, we
see in our experiments that data replication often does not happen for
unconditional models, while it is common in the text-conditional case.
Motivated by our findings, we then propose several techniques for reducing data
replication at both training and inference time by randomizing and augmenting
image captions in the training set.
Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D
  Diffusion-based Editor
Authors: name: Ruizhi Shao, name: Jingxiang Sun, name: Cheng Peng, name: Zerong Zheng, name: Boyao Zhou, name: Hongwen Zhang, name: Yebin Liu
Published: 2023-05-31 17:55:28

Recent years have witnessed considerable achievements in editing images with
text instructions. When applying these editors to dynamic scene editing, the
new-style scene tends to be temporally inconsistent due to the frame-by-frame
nature of these 2D editors. To tackle this issue, we propose Control4D, a novel
approach for high-fidelity and temporally consistent 4D portrait editing.
Control4D is built upon an efficient 4D representation with a 2D
diffusion-based editor. Instead of using direct supervisions from the editor,
our method learns a 4D GAN from it and avoids the inconsistent supervision
signals. Specifically, we employ a discriminator to learn the generation
distribution based on the edited images and then update the generator with the
discrimination signals. For more stable training, multi-level information is
extracted from the edited images and used to facilitate the learning of the
generator. Experimental results show that Control4D surpasses previous
approaches and achieves more photo-realistic and consistent 4D editing
performances. The link to our project website is
https://control4darxiv.github.io.
Efficient Diffusion Policies for Offline Reinforcement Learning
Authors: name: Bingyi Kang, name: Xiao Ma, name: Chao Du, name: Tianyu Pang, name: Shuicheng Yan
Published: 2023-05-31 17:55:21

Offline reinforcement learning (RL) aims to learn optimal policies from
offline datasets, where the parameterization of policies is crucial but often
overlooked. Recently, Diffsuion-QL significantly boosts the performance of
offline RL by representing a policy with a diffusion model, whose success
relies on a parametrized Markov Chain with hundreds of steps for sampling.
However, Diffusion-QL suffers from two critical limitations. 1) It is
computationally inefficient to forward and backward through the whole Markov
chain during training. 2) It is incompatible with maximum likelihood-based RL
algorithms (e.g., policy gradient methods) as the likelihood of diffusion
models is intractable. Therefore, we propose efficient diffusion policy (EDP)
to overcome these two challenges. EDP approximately constructs actions from
corrupted ones at training to avoid running the sampling chain. We conduct
extensive experiments on the D4RL benchmark. The results show that EDP can
reduce the diffusion policy training time from 5 days to 5 hours on
gym-locomotion tasks. Moreover, we show that EDP is compatible with various
offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on
D4RL by large margins over previous methods. Our code is available at
https://github.com/sail-sg/edp.
Tree-Ring Watermarks: Fingerprints for Diffusion Images that are
  Invisible and Robust
Authors: name: Yuxin Wen, name: John Kirchenbauer, name: Jonas Geiping, name: Tom Goldstein
Published: 2023-05-31 17:00:31

Watermarking the outputs of generative models is a crucial technique for
tracing copyright and preventing potential harm from AI-generated content. In
this paper, we introduce a novel technique called Tree-Ring Watermarking that
robustly fingerprints diffusion model outputs. Unlike existing methods that
perform post-hoc modifications to images after sampling, Tree-Ring Watermarking
subtly influences the entire sampling process, resulting in a model fingerprint
that is invisible to humans. The watermark embeds a pattern into the initial
noise vector used for sampling. These patterns are structured in Fourier space
so that they are invariant to convolutions, crops, dilations, flips, and
rotations. After image generation, the watermark signal is detected by
inverting the diffusion process to retrieve the noise vector, which is then
checked for the embedded signal. We demonstrate that this technique can be
easily applied to arbitrary diffusion models, including text-conditioned Stable
Diffusion, as a plug-in with negligible loss in FID. Our watermark is
semantically hidden in the image space and is far more robust than watermarking
alternatives that are currently deployed. Code is available at
github.com/YuxinWenRick/tree-ring-watermark.
LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
Authors: name: Yuma Koizumi, name: Heiga Zen, name: Shigeki Karita, name: Yifan Ding, name: Kohei Yatabe, name: Nobuyuki Morioka, name: Michiel Bacchiani, name: Yu Zhang, name: Wei Han, name: Ankur Bapna
Published: 2023-05-30 07:30:21

This paper introduces a new speech dataset called ``LibriTTS-R'' designed for
text-to-speech (TTS) use. It is derived by applying speech restoration to the
LibriTTS corpus, which consists of 585 hours of speech data at 24 kHz sampling
rate from 2,456 speakers and the corresponding texts. The constituent samples
of LibriTTS-R are identical to those of LibriTTS, with only the sound quality
improved. Experimental results show that the LibriTTS-R ground-truth samples
showed significantly improved sound quality compared to those in LibriTTS. In
addition, neural end-to-end TTS trained with LibriTTS-R achieved speech
naturalness on par with that of the ground-truth samples. The corpus is freely
available for download from http://www.openslr.org/141/.
AlteredAvatar: Stylizing Dynamic 3D Avatars with Fast Style Adaptation
Authors: name: Thu Nguyen-Phuoc, name: Gabriel Schwartz, name: Yuting Ye, name: Stephen Lombardi, name: Lei Xiao
Published: 2023-05-30 17:32:12

This paper presents a method that can quickly adapt dynamic 3D avatars to
arbitrary text descriptions of novel styles. Among existing approaches for
avatar stylization, direct optimization methods can produce excellent results
for arbitrary styles but they are unpleasantly slow. Furthermore, they require
redoing the optimization process from scratch for every new input. Fast
approximation methods using feed-forward networks trained on a large dataset of
style images can generate results for new inputs quickly, but tend not to
generalize well to novel styles and fall short in quality. We therefore
investigate a new approach, AlteredAvatar, that combines those two approaches
using the meta-learning framework. In the inner loop, the model learns to
optimize to match a single target style well; while in the outer loop, the
model learns to stylize efficiently across many styles. After training,
AlteredAvatar learns an initialization that can quickly adapt within a small
number of update steps to a novel style, which can be given using texts, a
reference image, or a combination of both. We show that AlteredAvatar can
achieve a good balance between speed, flexibility and quality, while
maintaining consistency across a wide range of novel views and facial
expressions.
Geometric Algebra Transformers
Authors: name: Johann Brehmer, name: Pim de Haan, name: Sönke Behrends, name: Taco Cohen
Published: 2023-05-28 18:48:50

Problems involving geometric data arise in a variety of fields, including
computer vision, robotics, chemistry, and physics. Such data can take numerous
forms, such as points, direction vectors, planes, or transformations, but to
date there is no single architecture that can be applied to such a wide variety
of geometric types while respecting their symmetries. In this paper we
introduce the Geometric Algebra Transformer (GATr), a general-purpose
architecture for geometric data. GATr represents inputs, outputs, and hidden
states in the projective geometric algebra, which offers an efficient
16-dimensional vector space representation of common geometric objects as well
as operators acting on them. GATr is equivariant with respect to E(3), the
symmetry group of 3D Euclidean space. As a transformer, GATr is scalable,
expressive, and versatile. In experiments with n-body modeling and robotic
planning, GATr shows strong improvements over non-geometric baselines.