Orionnnn/hub-stats huggingface dataset:
HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a
  Single Image
Authors: name: Tong Wu, name: Zhibing Li, name: Shuai Yang, name: Pan Zhang, name: Xinggang Pan, name: Jiaqi Wang, name: Dahua Lin, name: Ziwei Liu
Published: 2023-12-07 18:58:09

3D content creation from a single image is a long-standing yet highly
desirable task. Recent advances introduce 2D diffusion priors, yielding
reasonable results. However, existing methods are not hyper-realistic enough
for post-generation usage, as users cannot view, render and edit the resulting
3D content from a full range. To address these challenges, we introduce
HyperDreamer with several key designs and appealing properties: 1) Viewable:
360 degree mesh modeling with high-resolution textures enables the creation of
visually compelling 3D models from a full range of observation points. 2)
Renderable: Fine-grained semantic segmentation and data-driven priors are
incorporated as guidance to learn reasonable albedo, roughness, and specular
properties of the materials, enabling semantic-aware arbitrary material
estimation. 3) Editable: For a generated model or their own data, users can
interactively select any region via a few clicks and efficiently edit the
texture with text-based guidance. Extensive experiments demonstrate the
effectiveness of HyperDreamer in modeling region-aware materials with
high-resolution textures and enabling user-friendly editing. We believe that
HyperDreamer holds promise for advancing 3D content creation and finding
applications in various domains.
Controllable Human-Object Interaction Synthesis
Authors: name: Jiaman Li, name: Alexander Clegg, name: Roozbeh Mottaghi, name: Jiajun Wu, name: Xavier Puig, name: C. Karen Liu
Published: 2023-12-06 21:14:20

Synthesizing semantic-aware, long-horizon, human-object interaction is
critical to simulate realistic human behaviors. In this work, we address the
challenging problem of generating synchronized object motion and human motion
guided by language descriptions in 3D scenes. We propose Controllable
Human-Object Interaction Synthesis (CHOIS), an approach that generates object
motion and human motion simultaneously using a conditional diffusion model
given a language description, initial object and human states, and sparse
object waypoints. While language descriptions inform style and intent,
waypoints ground the motion in the scene and can be effectively extracted using
high-level planning methods. Naively applying a diffusion model fails to
predict object motion aligned with the input waypoints and cannot ensure the
realism of interactions that require precise hand-object contact and
appropriate contact grounded by the floor. To overcome these problems, we
introduce an object geometry loss as additional supervision to improve the
matching between generated object motion and input object waypoints. In
addition, we design guidance terms to enforce contact constraints during the
sampling process of the trained diffusion model.
Pearl: A Production-ready Reinforcement Learning Agent
Authors: name: Zheqing Zhu, name: Rodrigo de Salvo Braz, name: Jalaj Bhandari, name: Daniel Jiang, name: Yi Wan, name: Yonathan Efroni, name: Liyuan Wang, name: Ruiyang Xu, name: Hongbo Guo, name: Alex Nikulkov, name: Dmytro Korenkevych, name: Urun Dogan, name: Frank Cheng, name: Zheng Wu, name: Wanqiao Xu
Published: 2023-12-06 18:29:23

Reinforcement Learning (RL) offers a versatile framework for achieving
long-term goals. Its generality allows us to formalize a wide range of problems
that real-world intelligent systems encounter, such as dealing with delayed
rewards, handling partial observability, addressing the exploration and
exploitation dilemma, utilizing offline data to improve online performance, and
ensuring safety constraints are met. Despite considerable progress made by the
RL research community in addressing these issues, existing open-source RL
libraries tend to focus on a narrow portion of the RL solution pipeline,
leaving other aspects largely unattended. This paper introduces Pearl, a
Production-ready RL agent software package explicitly designed to embrace these
challenges in a modular fashion. In addition to presenting preliminary
benchmark results, this paper highlights Pearl's industry adoptions to
demonstrate its readiness for production usage. Pearl is open sourced on Github
at github.com/facebookresearch/pearl and its official website is located at
pearlagent.github.io.
Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Authors: name: Jiayi Guo, name: Xingqian Xu, name: Yifan Pu, name: Zanlin Ni, name: Chaofei Wang, name: Manushree Vasu, name: Shiji Song, name: Gao Huang, name: Humphrey Shi
Published: 2023-12-07 16:26:23

Recently, diffusion models have made remarkable progress in text-to-image
(T2I) generation, synthesizing images with high fidelity and diverse contents.
Despite this advancement, latent space smoothness within diffusion models
remains largely unexplored. Smooth latent spaces ensure that a perturbation on
an input latent corresponds to a steady change in the output image. This
property proves beneficial in downstream tasks, including image interpolation,
inversion, and editing. In this work, we expose the non-smoothness of diffusion
latent spaces by observing noticeable visual fluctuations resulting from minor
latent variations. To tackle this issue, we propose Smooth Diffusion, a new
category of diffusion models that can be simultaneously high-performing and
smooth. Specifically, we introduce Step-wise Variation Regularization to
enforce the proportion between the variations of an arbitrary input latent and
that of the output image is a constant at any diffusion training step. In
addition, we devise an interpolation standard deviation (ISTD) metric to
effectively assess the latent space smoothness of a diffusion model. Extensive
quantitative and qualitative experiments demonstrate that Smooth Diffusion
stands out as a more desirable solution not only in T2I generation but also
across various downstream tasks. Smooth Diffusion is implemented as a
plug-and-play Smooth-LoRA to work with various community models. Code is
available at https://github.com/SHI-Labs/Smooth-Diffusion.
GenTron: Delving Deep into Diffusion Transformers for Image and Video
  Generation
Authors: name: Shoufa Chen, name: Mengmeng Xu, name: Jiawei Ren, name: Yuren Cong, name: Sen He, name: Yanping Xie, name: Animesh Sinha, name: Ping Luo, name: Tao Xiang, name: Juan-Manuel Perez-Rua
Published: 2023-12-07 18:59:30

In this study, we explore Transformer-based diffusion models for image and
video generation. Despite the dominance of Transformer architectures in various
fields due to their flexibility and scalability, the visual generative domain
primarily utilizes CNN-based U-Net architectures, particularly in
diffusion-based models. We introduce GenTron, a family of Generative models
employing Transformer-based diffusion, to address this gap. Our initial step
was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
process involving thorough empirical exploration of the conditioning mechanism.
We then scale GenTron from approximately 900M to over 3B parameters, observing
significant improvements in visual quality. Furthermore, we extend GenTron to
text-to-video generation, incorporating novel motion-free guidance to enhance
video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
underscoring its strengths in compositional generation. We believe this work
will provide meaningful insights and serve as a valuable reference for future
research.
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
Authors: name: Zhiwu Qing, name: Shiwei Zhang, name: Jiayu Wang, name: Xiang Wang, name: Yujie Wei, name: Yingya Zhang, name: Changxin Gao, name: Nong Sang
Published: 2023-12-07 17:59:07

Despite diffusion models having shown powerful abilities to generate
photorealistic images, generating videos that are realistic and diverse still
remains in its infancy. One of the key reasons is that current methods
intertwine spatial content and temporal dynamics together, leading to a notably
increased complexity of text-to-video generation (T2V). In this work, we
propose HiGen, a diffusion model-based method that improves performance by
decoupling the spatial and temporal factors of videos from two perspectives,
i.e., structure level and content level. At the structure level, we decompose
the T2V task into two steps, including spatial reasoning and temporal
reasoning, using a unified denoiser. Specifically, we generate spatially
coherent priors using text during spatial reasoning and then generate
temporally coherent motions from these priors during temporal reasoning. At the
content level, we extract two subtle cues from the content of the input video
that can express motion and appearance changes, respectively. These two cues
then guide the model's training for generating videos, enabling flexible
content variations and enhancing temporal stability. Through the decoupled
paradigm, HiGen can effectively reduce the complexity of this task and generate
realistic videos with semantics accuracy and motion stability. Extensive
experiments demonstrate the superior performance of HiGen over the
state-of-the-art T2V methods.
AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
Authors: name: Jiwen Yu, name: Xiaodong Cun, name: Chenyang Qi, name: Yong Zhang, name: Xintao Wang, name: Ying Shan, name: Jian Zhang
Published: 2023-12-06 13:39:35

Large-scale text-to-video (T2V) diffusion models have great progress in
recent years in terms of visual quality, motion and temporal consistency.
However, the generation process is still a black box, where all attributes
(e.g., appearance, motion) are learned and generated jointly without precise
control ability other than rough text descriptions. Inspired by image animation
which decouples the video as one specific appearance with the corresponding
motion, we propose AnimateZero to unveil the pre-trained text-to-video
diffusion model, i.e., AnimateDiff, and provide more precise appearance and
motion control abilities for it. For appearance control, we borrow intermediate
latents and their features from the text-to-image (T2I) generation for ensuring
the generated first frame is equal to the given generated image. For temporal
control, we replace the global temporal attention of the original T2V model
with our proposed positional-corrected window attention to ensure other frames
align with the first frame well. Empowered by the proposed methods, AnimateZero
can successfully control the generating progress without further training. As a
zero-shot image animator for given images, AnimateZero also enables multiple
new applications, including interactive video generation and real image
animation. The detailed experiments demonstrate the effectiveness of the
proposed method in both T2V and related applications.
Gen2Det: Generate to Detect
Authors: name: Saksham Suri, name: Fanyi Xiao, name: Animesh Sinha, name: Sean Chang Culatana, name: Raghuraman Krishnamoorthi, name: Chenchen Zhu, name: Abhinav Shrivastava
Published: 2023-12-07 18:59:58

Recently diffusion models have shown improvement in synthetic image quality
as well as better control in generation. We motivate and present Gen2Det, a
simple modular pipeline to create synthetic training data for object detection
for free by leveraging state-of-the-art grounded image generation methods.
Unlike existing works which generate individual object instances, require
identifying foreground followed by pasting on other images, we simplify to
directly generating scene-centric images. In addition to the synthetic data,
Gen2Det also proposes a suite of techniques to best utilize the generated data,
including image-level filtering, instance-level filtering, and better training
recipe to account for imperfections in the generation. Using Gen2Det, we show
healthy improvements on object detection and segmentation tasks under various
settings and agnostic to detection methods. In the long-tailed detection
setting on LVIS, Gen2Det improves the performance on rare categories by a large
margin while also significantly improving the performance on other categories,
e.g. we see an improvement of 2.13 Box AP and 1.84 Mask AP over just training
on real data on LVIS with Mask R-CNN. In the low-data regime setting on COCO,
Gen2Det consistently improves both Box and Mask AP by 2.27 and 1.85 points. In
the most general detection setting, Gen2Det still demonstrates robust
performance gains, e.g. it improves the Box and Mask AP on COCO by 0.45 and
0.32 points.
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Authors: name: Zhen Li, name: Mingdeng Cao, name: Xintao Wang, name: Zhongang Qi, name: Ming-Ming Cheng, name: Ying Shan
Published: 2023-12-07 17:32:29

Recent advances in text-to-image generation have made remarkable progress in
synthesizing realistic human photos conditioned on given text prompts. However,
existing personalized generation methods cannot simultaneously satisfy the
requirements of high efficiency, promising identity (ID) fidelity, and flexible
text controllability. In this work, we introduce PhotoMaker, an efficient
personalized text-to-image generation method, which mainly encodes an arbitrary
number of input ID images into a stack ID embedding for preserving ID
information. Such an embedding, serving as a unified ID representation, can not
only encapsulate the characteristics of the same input ID comprehensively, but
also accommodate the characteristics of different IDs for subsequent
integration. This paves the way for more intriguing and practically valuable
applications. Besides, to drive the training of our PhotoMaker, we propose an
ID-oriented data construction pipeline to assemble the training data. Under the
nourishment of the dataset constructed through the proposed pipeline, our
PhotoMaker demonstrates better ID preservation ability than test-time
fine-tuning based methods, yet provides significant speed improvements,
high-quality generation results, strong generalization capabilities, and a wide
range of applications. Our project page is available at
https://photo-maker.github.io/
DreamVideo: Composing Your Dream Videos with Customized Subject and
  Motion
Authors: name: Yujie Wei, name: Shiwei Zhang, name: Zhiwu Qing, name: Hangjie Yuan, name: Zhiheng Liu, name: Yu Liu, name: Yingya Zhang, name: Jingren Zhou, name: Hongming Shan
Published: 2023-12-07 16:57:26

Customized generation using diffusion models has made impressive progress in
image generation, but remains unsatisfactory in the challenging video
generation task, as it requires the controllability of both subjects and
motions. To that end, we present DreamVideo, a novel approach to generating
personalized videos from a few static images of the desired subject and a few
videos of target motion. DreamVideo decouples this task into two stages,
subject learning and motion learning, by leveraging a pre-trained video
diffusion model. The subject learning aims to accurately capture the fine
appearance of the subject from provided images, which is achieved by combining
textual inversion and fine-tuning of our carefully designed identity adapter.
In motion learning, we architect a motion adapter and fine-tune it on the given
videos to effectively model the target motion pattern. Combining these two
lightweight and efficient adapters allows for flexible customization of any
subject with any motion. Extensive experimental results demonstrate the
superior performance of our DreamVideo over the state-of-the-art methods for
customized video generation. Our project page is at
https://dreamvideo-t2v.github.io.
Efficient Monotonic Multihead Attention
Authors: name: Xutai Ma, name: Anna Sun, name: Siqi Ouyang, name: Hirofumi Inaguma, name: Paden Tomasello
Published: 2023-12-07 18:34:57

We introduce the Efficient Monotonic Multihead Attention (EMMA), a
state-of-the-art simultaneous translation model with numerically-stable and
unbiased monotonic alignment estimation. In addition, we present improved
training and inference strategies, including simultaneous fine-tuning from an
offline translation model and reduction of monotonic alignment variance. The
experimental results demonstrate that the proposed model attains
state-of-the-art performance in simultaneous speech-to-text translation on the
Spanish and English translation task.
Generating Illustrated Instructions
Authors: name: Sachit Menon, name: Ishan Misra, name: Rohit Girdhar
Published: 2023-12-07 18:59:20

We introduce the new task of generating Illustrated Instructions, i.e.,
visual instructions customized to a user's needs. We identify desiderata unique
to this task, and formalize it through a suite of automatic and human
evaluation metrics, designed to measure the validity, consistency, and efficacy
of the generations. We combine the power of large language models (LLMs)
together with strong text-to-image generation diffusion models to propose a
simple approach called StackedDiffusion, which generates such illustrated
instructions given text as input. The resulting model strongly outperforms
baseline approaches and state-of-the-art multimodal LLMs; and in 30% of cases,
users even prefer it to human-generated articles. Most notably, it enables
various new and exciting applications far beyond what static articles on the
web can provide, such as personalized instructions complete with intermediate
steps and pictures in response to a user's individual situation.
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
Authors: name: Chengshu Li, name: Jacky Liang, name: Andy Zeng, name: Xinyun Chen, name: Karol Hausman, name: Dorsa Sadigh, name: Sergey Levine, name: Li Fei-Fei, name: Fei Xia, name: Brian Ichter
Published: 2023-12-07 17:51:43

Code provides a general syntactic structure to build complex programs and
perform precise computations when paired with a code interpreter - we
hypothesize that language models (LMs) can leverage code-writing to improve
Chain of Thought reasoning not only for logic and arithmetic tasks, but also
for semantic ones (and in particular, those that are a mix of both). For
example, consider prompting an LM to write code that counts the number of times
it detects sarcasm in an essay: the LM may struggle to write an implementation
for "detect_sarcasm(string)" that can be executed by the interpreter (handling
the edge cases would be insurmountable). However, LMs may still produce a valid
solution if they not only write code, but also selectively "emulate" the
interpreter by generating the expected output of "detect_sarcasm(string)". In
this work, we propose Chain of Code (CoC), a simple yet surprisingly effective
extension that improves LM code-driven reasoning. The key idea is to encourage
LMs to format semantic sub-tasks in a program as flexible pseudocode that the
interpreter can explicitly catch undefined behaviors and hand off to simulate
with an LM (as an "LMulator"). Experiments demonstrate that Chain of Code
outperforms Chain of Thought and other baselines across a variety of
benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of 12% over
Chain of Thought. In a nutshell, CoC broadens the scope of reasoning questions
that LMs can answer by "thinking in code".
Beyond Surface: Probing LLaMA Across Scales and Layers
Authors: name: Nuo Chen, name: Ning Wu, name: Shining Liang, name: Ming Gong, name: Linjun Shou, name: Dongmei Zhang, name: Jia Li
Published: 2023-12-07 14:50:41

This paper presents an in-depth analysis of Large Language Models (LLMs),
focusing on LLaMA, a prominent open-source foundational model in natural
language processing. Instead of assessing LLaMA through its generative output,
we design multiple-choice tasks to probe its intrinsic understanding in
high-order tasks such as reasoning and computation. We examine the model
horizontally, comparing different sizes, and vertically, assessing different
layers. We unveil several key and uncommon findings based on the designed
probing tasks: (1) Horizontally, enlarging model sizes almost could not
automatically impart additional knowledge or computational prowess. Instead, it
can enhance reasoning abilities, especially in math problem solving, and helps
reduce hallucinations, but only beyond certain size thresholds; (2) In vertical
analysis, the lower layers of LLaMA lack substantial arithmetic and factual
knowledge, showcasing logical thinking, multilingual and recognitive abilities,
with top layers housing most computational power and real-world knowledge.
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
  Tuning
Authors: name: Bolin Lai, name: Xiaoliang Dai, name: Lawrence Chen, name: Guan Pang, name: James M. Rehg, name: Miao Liu
Published: 2023-12-06 19:02:40

Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper, we
introduce a novel problem -- egocentric action frame generation. The goal is to
synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user's environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail to
control the state change of an action within the corresponding egocentric image
pixel space. To this end, we finetune a visual large language model (VLLM) via
visual instruction tuning for curating the enriched action descriptions to
address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
action frame generation using image and text embeddings from VLLM as additional
conditioning. We validate our proposed model on two egocentric datasets --
Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
image manipulation models in both quantitative and qualitative evaluation. We
also conduct detailed ablation studies and analysis to provide insights on our
method.
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Authors: name: Zeyi Sun, name: Ye Fang, name: Tong Wu, name: Pan Zhang, name: Yuhang Zang, name: Shu Kong, name: Yuanjun Xiong, name: Dahua Lin, name: Jiaqi Wang
Published: 2023-12-06 18:59:30

Contrastive Language-Image Pre-training (CLIP) plays an essential role in
extracting valuable content information from images across diverse tasks. It
aligns textual and visual modalities to comprehend the entire image, including
all the details, even those irrelevant to specific tasks. However, for a finer
understanding and controlled editing of images, it becomes crucial to focus on
specific regions of interest, which can be indicated as points, masks, or boxes
by humans or perception models. To fulfill the requirements, we introduce
Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to
suggest attentive regions and fine-tuned with constructed millions of RGBA
region-text pairs. Alpha-CLIP not only preserves the visual recognition ability
of CLIP but also enables precise control over the emphasis of image contents.
It demonstrates effectiveness in various tasks, including but not limited to
open-world recognition, multimodal large language models, and conditional 2D /
3D generation. It has a strong potential to serve as a versatile tool for
image-related tasks.