Orionnnn/hub-stats huggingface dataset:
MagicDance: Realistic Human Dance Video Generation with Motions & Facial
  Expressions Transfer
Authors: name: Di Chang, name: Yichun Shi, name: Quankai Gao, name: Jessica Fu, name: Hongyi Xu, name: Guoxian Song, name: Qing Yan, name: Xiao Yang, name: Mohammad Soleymani
Published: 2023-11-18 10:22:44

In this work, we propose MagicDance, a diffusion-based model for 2D human
motion and facial expression transfer on challenging human dance videos.
Specifically, we aim to generate human dance videos of any target identity
driven by novel pose sequences while keeping the identity unchanged. To this
end, we propose a two-stage training strategy to disentangle human motions and
appearance (e.g., facial expressions, skin tone and dressing), consisting of
the pretraining of an appearance-control block and fine-tuning of an
appearance-pose-joint-control block over human dance poses of the same dataset.
Our novel design enables robust appearance control with temporally consistent
upper body, facial attributes, and even background. The model also generalizes
well on unseen human identities and complex motion sequences without the need
for any fine-tuning with additional data with diverse human attributes by
leveraging the prior knowledge of image diffusion models. Moreover, the
proposed model is easy to use and can be considered as a plug-in
module/extension to Stable Diffusion. We also demonstrate the model's ability
for zero-shot 2D animation generation, enabling not only the appearance
transfer from one identity to another but also allowing for cartoon-like
stylization given only pose inputs. Extensive experiments demonstrate our
superior performance on the TikTok dataset.
NeuroPrompts: An Adaptive Framework to Optimize Prompts for
  Text-to-Image Generation
Authors: name: Shachar Rosenman, name: Vasudev Lal, name: Phillip Howard
Published: 2023-11-20 22:57:47

Despite impressive recent advances in text-to-image diffusion models,
obtaining high-quality images often requires prompt engineering by humans who
have developed expertise in using them. In this work, we present NeuroPrompts,
an adaptive framework that automatically enhances a user's prompt to improve
the quality of generations produced by text-to-image models. Our framework
utilizes constrained text decoding with a pre-trained language model that has
been adapted to generate prompts similar to those produced by human prompt
engineers. This approach enables higher-quality text-to-image generations and
provides user control over stylistic features via constraint set specification.
We demonstrate the utility of our framework by creating an interactive
application for prompt enhancement and image generation using Stable Diffusion.
Additionally, we conduct experiments utilizing a large dataset of
human-engineered prompts for text-to-image generation and show that our
approach automatically produces enhanced prompts that result in superior image
quality. We make our code, a screencast video demo and a live demo instance of
NeuroPrompts publicly available.
Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models
Authors: name: Rohit Gandikota, name: Joanna Materzynska, name: Tingrui Zhou, name: Antonio Torralba, name: David Bau
Published: 2023-11-20 18:59:01

We present a method to create interpretable concept sliders that enable
precise control over attributes in image generations from diffusion models. Our
approach identifies a low-rank parameter direction corresponding to one concept
while minimizing interference with other attributes. A slider is created using
a small set of prompts or sample images; thus slider directions can be created
for either textual or visual concepts. Concept Sliders are plug-and-play: they
can be composed efficiently and continuously modulated, enabling precise
control over image generation. In quantitative experiments comparing to
previous editing techniques, our sliders exhibit stronger targeted edits with
lower interference. We showcase sliders for weather, age, styles, and
expressions, as well as slider compositions. We show how sliders can transfer
latents from StyleGAN for intuitive editing of visual concepts for which
textual description is difficult. We also find that our method can help address
persistent quality issues in Stable Diffusion XL including repair of object
deformations and fixing distorted hands. Our code, data, and trained sliders
are available at https://sliders.baulab.info/
GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via
  Blender-Oriented GPT Planning
Authors: name: Jiaxi Lv, name: Yi Huang, name: Mingfu Yan, name: Jiancheng Huang, name: Jianzhuang Liu, name: Yifan Liu, name: Yafei Wen, name: Xiaoxin Chen, name: Shifeng Chen
Published: 2023-11-21 14:24:37

Recent advances in text-to-video generation have harnessed the power of
diffusion models to create visually compelling content conditioned on text
prompts. However, they usually encounter high computational costs and often
struggle to produce videos with coherent physical motions. To tackle these
issues, we propose GPT4Motion, a training-free framework that leverages the
planning capability of large language models such as GPT, the physical
simulation strength of Blender, and the excellent image generation ability of
text-to-image diffusion models to enhance the quality of video synthesis.
Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a
user textual prompt, which commands Blender's built-in physics engine to craft
fundamental scene components that encapsulate coherent physical motions across
frames. Then these components are inputted into Stable Diffusion to generate a
video aligned with the textual prompt. Experimental results on three basic
physical motion scenarios, including rigid object drop and collision, cloth
draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate
high-quality videos efficiently in maintaining motion coherency and entity
consistency. GPT4Motion offers new insights in text-to-video research,
enhancing its quality and broadening its horizon for future explorations.
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval
  Score Matching
Authors: name: Yixun Liang, name: Xin Yang, name: Jiantao Lin, name: Haodong Li, name: Xiaogang Xu, name: Yingcong Chen
Published: 2023-11-19 09:59:09

The recent advancements in text-to-3D generation mark a significant milestone
in generative models, unlocking new possibilities for creating imaginative 3D
assets across various real-world scenarios. While recent advancements in
text-to-3D generation have shown promise, they often fall short in rendering
detailed and high-quality 3D models. This problem is especially prevalent as
many methods base themselves on Score Distillation Sampling (SDS). This paper
identifies a notable deficiency in SDS, that it brings inconsistent and
low-quality updating direction for the 3D model, causing the over-smoothing
effect. To address this, we propose a novel approach called Interval Score
Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes
interval-based score matching to counteract over-smoothing. Furthermore, we
incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline.
Extensive experiments show that our model largely outperforms the
state-of-the-art in quality and training efficiency.
Make Pixels Dance: High-Dynamic Video Generation
Authors: name: Yan Zeng, name: Guoqiang Wei, name: Jiani Zheng, name: Jiaxin Zou, name: Yang Wei, name: Yuchen Zhang, name: Hang Li
Published: 2023-11-18 06:25:58

Creating high-dynamic videos such as motion-rich actions and sophisticated
visual effects poses a significant challenge in the field of artificial
intelligence. Unfortunately, current state-of-the-art video generation methods,
primarily focusing on text-to-video generation, tend to produce video clips
with minimal motions despite maintaining high fidelity. We argue that relying
solely on text instructions is insufficient and suboptimal for video
generation. In this paper, we introduce PixelDance, a novel approach based on
diffusion models that incorporates image instructions for both the first and
last frames in conjunction with text instructions for video generation.
Comprehensive experimental results demonstrate that PixelDance trained with
public data exhibits significantly better proficiency in synthesizing videos
with complex scenes and intricate motions, setting a new standard for video
generation.
Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
  Expression
Authors: name: Animesh Sinha, name: Bo Sun, name: Anmol Kalia, name: Arantxa Casanova, name: Elliot Blanchard, name: David Yan, name: Winnie Zhang, name: Tony Nelli, name: Jiahui Chen, name: Hardik Shah, name: Licheng Yu, name: Mitesh Kumar Singh, name: Ankit Ramchandani, name: Maziar Sanjabi, name: Sonal Gupta, name: Amy Bearman, name: Dhruv Mahajan
Published: 2023-11-17 03:00:29

We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
(LDMs) in a distinct domain with high visual quality, prompt alignment and
scene diversity. We choose sticker image generation as the target domain, as
the images significantly differ from photorealistic samples typically generated
by large-scale LDMs. We start with a competent text-to-image model, like Emu,
and show that relying on prompt engineering with a photorealistic model to
generate stickers leads to poor prompt alignment and scene diversity. To
overcome these drawbacks, we first finetune Emu on millions of sticker-like
images collected using weak supervision to elicit diversity. Next, we curate
human-in-the-loop (HITL) Alignment and Style datasets from model generations,
and finetune to improve prompt alignment and style alignment respectively.
Sequential finetuning on these datasets poses a tradeoff between better style
alignment and prompt alignment gains. To address this tradeoff, we propose a
novel fine-tuning method called Style Tailoring, which jointly fits the content
and style distribution and achieves best tradeoff. Evaluation results show our
method improves visual quality by 14%, prompt alignment by 16.2% and scene
diversity by 15.3%, compared to prompt engineering the base Emu model for
stickers generation.
System 2 Attention (is something you might need too)
Authors: name: Jason Weston, name: Sainbayar Sukhbaatar
Published: 2023-11-20 15:04:50

Soft attention in Transformer-based Large Language Models (LLMs) is
susceptible to incorporating irrelevant information from the context into its
latent representations, which adversely affects next token generations. To help
rectify these issues, we introduce System 2 Attention (S2A), which leverages
the ability of LLMs to reason in natural language and follow instructions in
order to decide what to attend to. S2A regenerates the input context to only
include the relevant portions, before attending to the regenerated context to
elicit the final response. In experiments, S2A outperforms standard
attention-based LLMs on three tasks containing opinion or irrelevant
information, QA, math word problems and longform generation, where S2A
increases factuality and objectivity, and decreases sycophancy.
TPTU-v2: Boosting Task Planning and Tool Usage of Large Language
  Model-based Agents in Real-world Systems
Authors: name: Yilun Kong, name: Jingqing Ruan, name: Yihong Chen, name: Bin Zhang, name: Tianpeng Bao, name: Shiwei Shi, name: Guoqing Du, name: Xiaoru Hu, name: Hangyu Mao, name: Ziyue Li, name: Xingyu Zeng, name: Rui Zhao
Published: 2023-11-19 12:37:30

Large Language Models (LLMs) have demonstrated proficiency in addressing
tasks that necessitate a combination of task planning and the usage of external
tools that require a blend of task planning and the utilization of external
tools, such as APIs. However, real-world complex systems present three
prevalent challenges concerning task planning and tool usage: (1) The real
system usually has a vast array of APIs, so it is impossible to feed the
descriptions of all APIs to the prompt of LLMs as the token length is limited;
(2) the real system is designed for handling complex tasks, and the base LLMs
can hardly plan a correct sub-task order and API-calling order for such tasks;
(3) Similar semantics and functionalities among APIs in real systems create
challenges for both LLMs and even humans in distinguishing between them. In
response, this paper introduces a comprehensive framework aimed at enhancing
the Task Planning and Tool Usage (TPTU) abilities of LLM-based agents operating
within real-world systems. Our framework comprises three key components
designed to address these challenges: (1) the API Retriever selects the most
pertinent APIs for the user task among the extensive array available; (2) LLM
Finetuner tunes a base LLM so that the finetuned LLM can be more capable for
task planning and API calling; (3) the Demo Selector adaptively retrieves
different demonstrations related to hard-to-distinguish APIs, which is further
used for in-context learning to boost the final performance. We validate our
methods using a real-world commercial system as well as an open-sourced
academic dataset, and the outcomes clearly showcase the efficacy of each
individual component as well as the integrated framework.
M^{2}UGen: Multi-modal Music Understanding and Generation with the
  Power of Large Language Models
Authors: name: Atin Sakkeer Hussain, name: Shansong Liu, name: Chenshuo Sun, name: Ying Shan
Published: 2023-11-19 06:50:52

The current landscape of research leveraging large language models (LLMs) is
experiencing a surge. Many works harness the powerful reasoning capabilities of
these models to comprehend various modalities, such as text, speech, images,
videos, etc. They also utilize LLMs to understand human intention and generate
desired outputs like images, videos, and music. However, research that combines
both understanding and generation using LLMs is still limited and in its
nascent stage. To address this gap, we introduce a Multi-modal Music
Understanding and Generation (M^{2}UGen) framework that integrates LLM's
abilities to comprehend and generate music for different modalities. The
M^{2}UGen framework is purpose-built to unlock creative potential from
diverse sources of inspiration, encompassing music, image, and video through
the use of pretrained MERT, ViT, and ViViT models, respectively. To enable
music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging
multi-modal understanding and music generation is accomplished through the
integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
model to generate extensive datasets that support text/image/video-to-music
generation, facilitating the training of our M^{2}UGen framework. We conduct
a thorough evaluation of our proposed framework. The experimental results
demonstrate that our model achieves or surpasses the performance of the current
state-of-the-art models.
AutoStory: Generating Diverse Storytelling Images with Minimal Human
  Effort
Authors: name: Wen Wang, name: Canyu Zhao, name: Hao Chen, name: Zhekai Chen, name: Kecheng Zheng, name: Chunhua Shen
Published: 2023-11-19 06:07:37

Story visualization aims to generate a series of images that match the story
described in texts, and it requires the generated images to satisfy high
quality, alignment with the text description, and consistency in character
identities. Given the complexity of story visualization, existing methods
drastically simplify the problem by considering only a few specific characters
and scenarios, or requiring the users to provide per-image control conditions
such as sketches. However, these simplifications render these methods
incompetent for real applications. To this end, we propose an automated story
visualization system that can effectively generate diverse, high-quality, and
consistent sets of story images, with minimal human interactions. Specifically,
we utilize the comprehension and planning capabilities of large language models
for layout planning, and then leverage large-scale text-to-image models to
generate sophisticated story images based on the layout. We empirically find
that sparse control conditions, such as bounding boxes, are suitable for layout
planning, while dense control conditions, e.g., sketches and keypoints, are
suitable for generating high-quality image content. To obtain the best of both
worlds, we devise a dense condition generation module to transform simple
bounding box layouts into sketch or keypoint control conditions for final image
generation, which not only improves the image quality but also allows easy and
intuitive user interactions. In addition, we propose a simple yet effective
method to generate multi-view consistent character images, eliminating the
reliance on human labor to collect or draw character images.
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer
  Learning
Authors: name: Clifton Poth, name: Hannah Sterz, name: Indraneil Paul, name: Sukannya Purkayastha, name: Leon Engländer, name: Timo Imhof, name: Ivan Vulić, name: Sebastian Ruder, name: Iryna Gurevych, name: Jonas Pfeiffer
Published: 2023-11-18 13:53:26

We introduce Adapters, an open-source library that unifies
parameter-efficient and modular transfer learning in large language models. By
integrating 10 diverse adapter methods into a unified interface, Adapters
offers ease of use and flexible configuration. Our library allows researchers
and practitioners to leverage adapter modularity through composition blocks,
enabling the design of complex adapter setups. We demonstrate the library's
efficacy by evaluating its performance against full fine-tuning on various NLP
tasks. Adapters provides a powerful tool for addressing the challenges of
conventional fine-tuning paradigms and promoting more efficient and modular
transfer learning. The library is available via https://adapterhub.ml/adapters.
Orca 2: Teaching Small Language Models How to Reason
Authors: name: Arindam Mitra, name: Luciano Del Corro, name: Shweti Mahajan, name: Andres Codas, name: Clarisse Simoes, name: Sahaj Agrawal, name: Xuxi Chen, name: Anastasia Razdaibiedina, name: Erik Jones, name: Kriti Aggarwal, name: Hamid Palangi, name: Guoqing Zheng, name: Corby Rosset, name: Hamed Khanpour, name: Ahmed Awadallah
Published: 2023-11-18 11:44:52

Orca 1 learns from rich signals, such as explanation traces, allowing it to
outperform conventional instruction-tuned models on benchmarks like BigBench
Hard and AGIEval. In Orca 2, we continue exploring how improved training
signals can enhance smaller LMs' reasoning abilities. Research on training
small LMs has often relied on imitation learning to replicate the output of
more capable models. We contend that excessive emphasis on imitation may
restrict the potential of smaller models. We seek to teach small LMs to employ
different solution strategies for different tasks, potentially different from
the one used by the larger model. For example, while larger models might
provide a direct answer to a complex task, smaller models may not have the same
capacity. In Orca 2, we teach the model various reasoning techniques
(step-by-step, recall then generate, recall-reason-generate, direct answer,
etc.). More crucially, we aim to help the model learn to determine the most
effective solution strategy for each task. We evaluate Orca 2 using a
comprehensive set of 15 diverse benchmarks (corresponding to approximately 100
tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of
similar size and attains performance levels similar or better to those of
models 5-10x larger, as assessed on complex tasks that test advanced reasoning
abilities in zero-shot settings. We open-source Orca 2 to encourage further
research on the development, evaluation, and alignment of smaller LMs.
Exponentially Faster Language Modelling
Authors: name: Peter Belcak, name: Roger Wattenhofer
Published: 2023-11-15 18:42:50

Language models only really need to use an exponential fraction of their
neurons for individual inferences. As proof, we present FastBERT, a BERT
variant that uses 0.3\% of its neurons during inference while performing on par
with similar BERT models. FastBERT selectively engages just 12 out of 4095
neurons for each layer inference. This is achieved by replacing feedforward
networks with fast feedforward networks (FFFs). While no truly efficient
implementation currently exists to unlock the full acceleration potential of
conditional neural execution, we provide high-level CPU code achieving 78x
speedup over the optimized baseline feedforward implementation, and a PyTorch
implementation delivering 40x speedup over the equivalent batched feedforward
inference. We publish our training code, benchmarking setup, and model weights.
Memory Augmented Language Models through Mixture of Word Experts
Authors: name: Cicero Nogueira dos Santos, name: James Lee-Thorp, name: Isaac Noble, name: Chung-Ching Chang, name: David Uthus
Published: 2023-11-15 18:19:56

Scaling up the number of parameters of language models has proven to be an
effective approach to improve performance. For dense models, increasing model
size proportionally increases the model's computation footprint. In this work,
we seek to aggressively decouple learning capacity and FLOPs through
Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
based routing functions and experts. Our proposed approach, dubbed Mixture of
Word Experts (MoWE), can be seen as a memory augmented model, where a large set
of word-specific experts play the role of a sparse memory. We demonstrate that
MoWE performs significantly better than the T5 family of models with similar
number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
regular MoE models on knowledge intensive tasks and has similar performance to
more complex memory augmented approaches that often require to invoke custom
mechanisms to search the sparse memory.
ProAgent: From Robotic Process Automation to Agentic Process Automation
Authors: name: Yining Ye, name: Xin Cong, name: Shizuo Tian, name: Jiannan Cao, name: Hao Wang, name: Yujia Qin, name: Yaxi Lu, name: Heyang Yu, name: Huadong Wang, name: Yankai Lin, name: Zhiyuan Liu, name: Maosong Sun
Published: 2023-11-02 14:32:16

From ancient water wheels to robotic process automation (RPA), automation
technology has evolved throughout history to liberate human beings from arduous
tasks. Yet, RPA struggles with tasks needing human-like intelligence,
especially in elaborate design of workflow construction and dynamic
decision-making in workflow execution. As Large Language Models (LLMs) have
emerged human-like intelligence, this paper introduces Agentic Process
Automation (APA), a groundbreaking automation paradigm using LLM-based agents
for advanced automation by offloading the human labor to agents associated with
construction and execution. We then instantiate ProAgent, an LLM-based agent
designed to craft workflows from human instructions and make intricate
decisions by coordinating specialized agents. Empirical experiments are
conducted to detail its construction and execution procedure of workflow,
showcasing the feasibility of APA, unveiling the possibility of a new paradigm
of automation driven by agents. Our code is public at
https://github.com/OpenBMB/ProAgent.