Orionnnn/hub-stats huggingface dataset:
ControlVideo: Adding Conditional Control for One Shot Text-to-Video
  Editing
Authors: name: Min Zhao, name: Rongzhen Wang, name: Fan Bao, name: Chongxuan Li, name: Jun Zhu
Published: 2023-05-26 17:13:55

In this paper, we present ControlVideo, a novel method for text-driven video
editing. Leveraging the capabilities of text-to-image diffusion models and
ControlNet, ControlVideo aims to enhance the fidelity and temporal consistency
of videos that align with a given text while preserving the structure of the
source video. This is achieved by incorporating additional conditions such as
edge maps, fine-tuning the key-frame and temporal attention on the source
video-text pair with carefully designed strategies. An in-depth exploration of
ControlVideo's design is conducted to inform future research on one-shot tuning
video diffusion models. Quantitatively, ControlVideo outperforms a range of
competitive baselines in terms of faithfulness and consistency while still
aligning with the textual prompt. Additionally, it delivers videos with high
visual realism and fidelity w.r.t. the source content, demonstrating
flexibility in utilizing controls containing varying degrees of source video
information, and the potential for multiple control combinations. The project
page is available at
https://ml.cs.tsinghua.edu.cn/controlvideo/{https://ml.cs.tsinghua.edu.cn/controlvideo/}.
ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image
Authors: name: Zhenzhen Weng, name: Zeyu Wang, name: Serena Yeung
Published: 2023-05-25 18:23:20

Recent advancements in text-to-image generation have enabled significant
progress in zero-shot 3D shape generation. This is achieved by score
distillation, a methodology that uses pre-trained text-to-image diffusion
models to optimize the parameters of a 3D neural presentation, e.g. Neural
Radiance Field (NeRF). While showing promising results, existing methods are
often not able to preserve the geometry of complex shapes, such as human
bodies. To address this challenge, we present ZeroAvatar, a method that
introduces the explicit 3D human body prior to the optimization process.
Specifically, we first estimate and refine the parameters of a parametric human
body from a single image. Then during optimization, we use the posed parametric
body as additional geometry constraint to regularize the diffusion model as
well as the underlying density field. Lastly, we propose a UV-guided texture
regularization term to further guide the completion of texture on invisible
body parts. We show that ZeroAvatar significantly enhances the robustness and
3D consistency of optimization-based image-to-3D avatar generation,
outperforming existing zero-shot image-to-3D methods.
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion
  Models
Authors: name: Ying Fan, name: Olivia Watkins, name: Yuqing Du, name: Hao Liu, name: Moonkyung Ryu, name: Craig Boutilier, name: Pieter Abbeel, name: Mohammad Ghavamzadeh, name: Kangwook Lee, name: Kimin Lee
Published: 2023-05-25 17:35:38

Learning from human feedback has been shown to improve text-to-image models.
These techniques first learn a reward function that captures what humans care
about in the task and then improve the models based on the learned reward
function. Even though relatively simple approaches (e.g., rejection sampling
based on reward scores) have been investigated, fine-tuning text-to-image
models with the reward function remains challenging. In this work, we propose
using online reinforcement learning (RL) to fine-tune text-to-image models. We
focus on diffusion models, defining the fine-tuning task as an RL problem, and
updating the pre-trained text-to-image diffusion models using policy gradient
to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
policy optimization with KL regularization. We conduct an analysis of KL
regularization for both RL fine-tuning and supervised fine-tuning. In our
experiments, we show that DPOK is generally superior to supervised fine-tuning
with respect to both image-text alignment and image quality.
Large Language Models as Tool Makers
Authors: name: Tianle Cai, name: Xuezhi Wang, name: Tengyu Ma, name: Xinyun Chen, name: Denny Zhou
Published: 2023-05-26 17:50:11

Recent research shows the potential of enhancing the problem-solving ability
of large language models (LLMs) through the use of external tools. However,
prior work along this line depends on the availability of existing tools. In
this work, we take an initial step towards removing this dependency by
proposing a closed-loop framework, referred to as LLMs As Tool Makers (LATM),
where LLMs create their own reusable tools for problem-solving. Our approach
consists of two key phases: 1) tool making: an LLM acts as the tool maker that
crafts tools for given tasks, where a tool is implemented as a Python utility
function. 2) tool using: an LLM acts as the tool user, which applies the tool
built by the tool maker for problem-solving. The tool user can be either the
same or a different LLM from the tool maker. Tool-making enables an LLM to
continually generate tools that can be applied to different requests so that
future requests can call the corresponding APIs when beneficial for solving the
tasks. Furthermore, the division of labor among LLMs for tool-making and
tool-using phases introduces the opportunity to achieve cost effectiveness
without degrading the quality of generated tools and problem solutions. For
example, recognizing that tool-making demands more sophisticated capabilities
than tool-using, we can apply a powerful yet resource-intensive model as the
tool maker, and a lightweight while cost-effective model as the tool user. We
validate the effectiveness of our approach across a variety of complex
reasoning tasks, including Big-Bench tasks. With GPT-4 as the tool maker and
GPT-3.5 as the tool user, LATM can achieve performance that is on par with
using GPT-4 for both tool making and tool using, while the inference cost is
significantly reduced.
Playing repeated games with Large Language Models
Authors: name: Elif Akata, name: Lion Schulz, name: Julian Coda-Forno, name: Seong Joon Oh, name: Matthias Bethge, name: Eric Schulz
Published: 2023-05-26 12:17:59

Large Language Models (LLMs) are transforming society and permeating into
diverse applications. As a result, LLMs will frequently interact with us and
other agents. It is, therefore, of great societal value to understand how LLMs
behave in interactive social settings. Here, we propose to use behavioral game
theory to study LLM's cooperation and coordination behavior. To do so, we let
different LLMs (GPT-3, GPT-3.5, and GPT-4) play finitely repeated games with
each other and with other, human-like strategies. Our results show that LLMs
generally perform well in such tasks and also uncover persistent behavioral
signatures. In a large set of two players-two strategies games, we find that
LLMs are particularly good at games where valuing their own self-interest pays
off, like the iterated Prisoner's Dilemma family. However, they behave
sub-optimally in games that require coordination. We, therefore, further focus
on two games from these distinct families. In the canonical iterated Prisoner's
Dilemma, we find that GPT-4 acts particularly unforgivingly, always defecting
after another agent has defected only once. In the Battle of the Sexes, we find
that GPT-4 cannot match the behavior of the simple convention to alternate
between options. We verify that these behavioral signatures are stable across
robustness checks. Finally, we show how GPT-4's behavior can be modified by
providing further information about the other player as well as by asking it to
predict the other player's actions before making a choice. These results enrich
our understanding of LLM's social behavior and pave the way for a behavioral
game theory for machines.
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with
  Variational Score Distillation
Authors: name: Zhengyi Wang, name: Cheng Lu, name: Yikai Wang, name: Fan Bao, name: Chongxuan Li, name: Hang Su, name: Jun Zhu
Published: 2023-05-25 16:19:18

Score distillation sampling (SDS) has shown great promise in text-to-3D
generation by distilling pretrained large-scale text-to-image diffusion models,
but suffers from over-saturation, over-smoothing, and low-diversity problems.
In this work, we propose to model the 3D parameter as a random variable instead
of a constant as in SDS and present variational score distillation (VSD), a
principled particle-based variational framework to explain and address the
aforementioned issues in text-to-3D generation. We show that SDS is a special
case of VSD and leads to poor samples with both small and large CFG weights. In
comparison, VSD works well with various CFG weights as ancestral sampling from
diffusion models and simultaneously improves the diversity and sample quality
with a common CFG weight (i.e., 7.5). We further present various improvements
in the design space for text-to-3D such as distillation time schedule and
density initialization, which are orthogonal to the distillation algorithm yet
not well explored. Our overall approach, dubbed ProlificDreamer, can generate
high rendering resolution (i.e., 512times512) and high-fidelity NeRF with
rich structure and complex effects (e.g., smoke and drops). Further,
initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and
photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/
On Architectural Compression of Text-to-Image Diffusion Models
Authors: name: Bo-Kyeong Kim, name: Hyoung-Kyu Song, name: Thibault Castells, name: Shinkook Choi
Published: 2023-05-25 07:28:28

Exceptional text-to-image (T2I) generation results of Stable Diffusion models
(SDMs) come with substantial computational demands. To resolve this issue,
recent research on efficient SDMs has prioritized reducing the number of
sampling steps and utilizing network quantization. Orthogonal to these
directions, this study highlights the power of classical architectural
compression for general-purpose T2I synthesis by introducing block-removed
knowledge-distilled SDMs (BK-SDMs). We eliminate several residual and attention
blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of
parameters, MACs per sampling step, and latency. We conduct distillation-based
pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training
pairs) on a single A100 GPU. Despite being trained with limited resources, our
compact models can imitate the original SDM by benefiting from transferred
knowledge and achieve competitive results against larger multi-billion
parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate
the applicability of our lightweight pretrained models in personalized
generation with DreamBooth finetuning.
Custom-Edit: Text-Guided Image Editing with Customized Diffusion Models
Authors: name: Jooyoung Choi, name: Yunjey Choi, name: Yunji Kim, name: Junho Kim, name: Sungroh Yoon
Published: 2023-05-25 06:46:28

Text-to-image diffusion models can generate diverse, high-fidelity images
based on user-provided text prompts. Recent research has extended these models
to support text-guided image editing. While text guidance is an intuitive
editing interface for users, it often fails to ensure the precise concept
conveyed by users. To address this issue, we propose Custom-Edit, in which we
(i) customize a diffusion model with a few reference images and then (ii)
perform text-guided editing. Our key discovery is that customizing only
language-relevant parameters with augmented prompts improves reference
similarity significantly while maintaining source similarity. Moreover, we
provide our recipe for each customization and editing process. We compare
popular customization methods and validate our findings on two editing methods
using various datasets.
Efficient Neural Music Generation
Authors: name: Max W. Y. Lam, name: Qiao Tian, name: Tang Li, name: Zongyu Yin, name: Siyuan Feng, name: Ming Tu, name: Yuliang Ji, name: Rui Xia, name: Mingbo Ma, name: Xuchen Song, name: Jitong Chen, name: Yuping Wang, name: Yuxuan Wang
Published: 2023-05-25 05:02:35

Recent progress in music generation has been remarkably advanced by the
state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
sampling with the MusicLM requires processing through these LMs one by one to
obtain the fine-grained acoustic tokens, making it computationally expensive
and prohibitive for a real-time generation. Efficient music generation with a
quality on par with MusicLM remains a significant challenge. In this paper, we
present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
model that generates music audios of state-of-the-art quality meanwhile
reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
audio VAE-GAN to efficiently decode the conditioning semantic tokens into
waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
by incorporating the semantic information into segments of latents effectively
via cross-attention at each denoising step. Our experimental results suggest
the superiority of MeLoDy, not only in its practical advantages on sampling
speed and infinitely continuable generation, but also in its state-of-the-art
musicality, audio quality, and text correlation.
  Our samples are available at https://Efficient-MeLoDy.github.io/.
Manifold Diffusion Fields
Authors: name: Ahmed A. Elhag, name: Joshua M. Susskind, name: Miguel Angel Bautista
Published: 2023-05-24 21:42:45

We present Manifold Diffusion Fields (MDF), an approach to learn generative
models of continuous functions defined over Riemannian manifolds. Leveraging
insights from spectral geometry analysis, we define an intrinsic coordinate
system on the manifold via the eigen-functions of the Laplace-Beltrami
Operator. MDF represents functions using an explicit parametrization formed by
a set of multiple input-output pairs. Our approach allows to sample continuous
functions on manifolds and is invariant with respect to rigid and isometric
transformations of the manifold. Empirical results on several datasets and
manifolds show that MDF can capture distributions of such functions with better
diversity and fidelity than previous approaches.
Unsupervised Semantic Correspondence Using Stable Diffusion
Authors: name: Eric Hedlin, name: Gopal Sharma, name: Shweta Mahajan, name: Hossam Isack, name: Abhishek Kar, name: Andrea Tagliasacchi, name: Kwang Moo Yi
Published: 2023-05-24 21:34:34

Text-to-image diffusion models are now capable of generating images that are
often indistinguishable from real images. To generate such images, these models
must understand the semantics of the objects they are asked to generate. In
this work we show that, without any training, one can leverage this semantic
knowledge within diffusion models to find semantic correspondences -- locations
in multiple images that have the same semantic meaning. Specifically, given an
image, we optimize the prompt embeddings of these models for maximum attention
on the regions of interest. These optimized embeddings capture semantic
information about the location, which can then be transferred to another image.
By doing so we obtain results on par with the strongly supervised state of the
art on the PF-Willow dataset and significantly outperform (20.9% relative for
the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
CUB-200 and SPair-71k datasets.
The False Promise of Imitating Proprietary LLMs
Authors: name: Arnav Gudibande, name: Eric Wallace, name: Charlie Snell, name: Xinyang Geng, name: Hao Liu, name: Pieter Abbeel, name: Sergey Levine, name: Dawn Song
Published: 2023-05-25 05:00:12

An emerging method to cheaply improve a weaker language model is to finetune
it on outputs from a stronger model, such as a proprietary system like ChatGPT
(e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply
imitate the proprietary model's capabilities using a weaker open-source model.
In this work, we critically analyze this approach. We first finetune a series
of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data
sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the
models using crowd raters and canonical NLP benchmarks. Initially, we were
surprised by the output quality of our imitation models -- they appear far
better at following instructions, and crowd workers rate their outputs as
competitive with ChatGPT. However, when conducting more targeted automatic
evaluations, we find that imitation models close little to none of the gap from
the base LM to ChatGPT on tasks that are not heavily supported in the imitation
data. We show that these performance discrepancies may slip past human raters
because imitation models are adept at mimicking ChatGPT's style but not its
factuality. Overall, we conclude that model imitation is a false promise: there
exists a substantial capabilities gap between open and closed LMs that, with
current methods, can only be bridged using an unwieldy amount of imitation data
or by using more capable base LMs. In turn, we argue that the highest leverage
action for improving open-source models is to tackle the difficult challenge of
developing better base LMs, rather than taking the shortcut of imitating
proprietary systems.
Voyager: An Open-Ended Embodied Agent with Large Language Models
Authors: name: Guanzhi Wang, name: Yuqi Xie, name: Yunfan Jiang, name: Ajay Mandlekar, name: Chaowei Xiao, name: Yuke Zhu, name: Linxi Fan, name: Anima Anandkumar
Published: 2023-05-25 17:46:38

We introduce Voyager, the first LLM-powered embodied lifelong learning agent
in Minecraft that continuously explores the world, acquires diverse skills, and
makes novel discoveries without human intervention. Voyager consists of three
key components: 1) an automatic curriculum that maximizes exploration, 2) an
ever-growing skill library of executable code for storing and retrieving
complex behaviors, and 3) a new iterative prompting mechanism that incorporates
environment feedback, execution errors, and self-verification for program
improvement. Voyager interacts with GPT-4 via blackbox queries, which bypasses
the need for model parameter fine-tuning. The skills developed by Voyager are
temporally extended, interpretable, and compositional, which compounds the
agent's abilities rapidly and alleviates catastrophic forgetting. Empirically,
Voyager shows strong in-context lifelong learning capability and exhibits
exceptional proficiency in playing Minecraft. It obtains 3.3x more unique
items, travels 2.3x longer distances, and unlocks key tech tree milestones up
to 15.3x faster than prior SOTA. Voyager is able to utilize the learned skill
library in a new Minecraft world to solve novel tasks from scratch, while other
techniques struggle to generalize. We open-source our full codebase and prompts
at https://voyager.minedojo.org/.
SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and
  Reasoning
Authors: name: Yue Wu, name: So Yeon Min, name: Shrimai Prabhumoye, name: Yonatan Bisk, name: Ruslan Salakhutdinov, name: Amos Azaria, name: Tom Mitchell, name: Yuanzhi Li
Published: 2023-05-24 18:14:35

Open-world survival games pose significant challenges for AI algorithms due
to their multi-tasking, deep exploration, and goal prioritization requirements.
Despite reinforcement learning (RL) being popular for solving games, its high
sample complexity limits its effectiveness in complex open-world games like
Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's
original academic paper and use the knowledge learned to reason and play the
game through a large language model (LLM). Prompted with the LaTeX source as
game context and a description of the agent's current observation, our SPRING
framework employs a directed acyclic graph (DAG) with game-related questions as
nodes and dependencies as edges. We identify the optimal action to take in the
environment by traversing the DAG and calculating LLM responses for each node
in topological order, with the LLM's answer to final node directly translating
to environment actions. In our experiments, we study the quality of in-context
"reasoning" induced by different forms of prompts under the setting of the
Crafter open-world environment. Our experiments suggest that LLMs, when
prompted with consistent chain-of-thought, have great potential in completing
sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4
outperforms all state-of-the-art RL baselines, trained for 1M steps, without
any training. Finally, we show the potential of games as a test bed for LLMs.
Is GPT-4 a Good Data Analyst?
Authors: name: Liying Cheng, name: Xingxuan Li, name: Lidong Bing
Published: 2023-05-24 11:26:59

As large language models (LLMs) have demonstrated their powerful capabilities
in plenty of domains and tasks, including context understanding, code
generation, language generation, data storytelling, etc., many data analysts
may raise concerns if their jobs will be replaced by AI. This controversial
topic has drawn a lot of attention in public. However, we are still at a stage
of divergent opinions without any definitive conclusion. Motivated by this, we
raise the research question of "is GPT-4 a good data analyst?" in this work and
aim to answer it by conducting head-to-head comparative studies. In detail, we
regard GPT-4 as a data analyst to perform end-to-end data analysis with
databases from a wide range of domains. We propose a framework to tackle the
problems by carefully designing the prompts for GPT-4 to conduct experiments.
We also design several task-specific evaluation metrics to systematically
compare the performance between several professional human data analysts and
GPT-4. Experimental results show that GPT-4 can achieve comparable performance
to humans. We also provide in-depth discussions about our results to shed light
on further studies before we reach the conclusion that GPT-4 can replace data
analysts.
Leveraging GPT-4 for Automatic Translation Post-Editing
Authors: name: Vikas Raunak, name: Amr Sharaf, name: Hany Hassan Awadallah, name: Arul Menezes
Published: 2023-05-24 08:30:05

While Neural Machine Translation (NMT) represents the leading approach to
Machine Translation (MT), the outputs of NMT models still require translation
post-editing to rectify errors and enhance quality, particularly under critical
settings. In this work, we formalize the task of translation post-editing with
Large Language Models (LLMs) and explore the use of GPT-4 to automatically
post-edit NMT outputs across several language pairs. Our results demonstrate
that GPT-4 is adept at translation post-editing and produces meaningful edits
even when the target language is not English. Notably, we achieve
state-of-the-art performance on WMT-22 English-Chinese, English-German,
Chinese-English and German-English language pairs using GPT-4 based
post-editing, as evaluated by state-of-the-art MT quality metrics.