Orionnnn/hub-stats huggingface dataset:
VideoCrafter2: Overcoming Data Limitations for High-Quality Video
  Diffusion Models
Authors: name: Haoxin Chen, name: Yong Zhang, name: Xiaodong Cun, name: Menghan Xia, name: Xintao Wang, name: Chao Weng, name: Ying Shan
Published: 2024-01-17 08:30:32

Text-to-video generation aims to produce a video based on a given prompt.
Recently, several commercial video models have been able to generate plausible
videos with minimal noise, excellent details, and high aesthetic scores.
However, these models rely on large-scale, well-filtered, high-quality videos
that are not accessible to the community. Many existing research works, which
train models using the low-quality WebVid-10M dataset, struggle to generate
high-quality videos because the models are optimized to fit WebVid-10M. In this
work, we explore the training scheme of video models extended from Stable
Diffusion and investigate the feasibility of leveraging low-quality videos and
synthesized high-quality images to obtain a high-quality video model. We first
analyze the connection between the spatial and temporal modules of video models
and the distribution shift to low-quality videos. We observe that full training
of all modules results in a stronger coupling between spatial and temporal
modules than only training temporal modules. Based on this stronger coupling,
we shift the distribution to higher quality without motion degradation by
finetuning spatial modules with high-quality images, resulting in a generic
high-quality video model. Evaluations are conducted to demonstrate the
superiority of the proposed method, particularly in picture quality, motion,
and concept composition.
Extending LLMs' Context Window with 100 Samples
Authors: name: Yikai Zhang, name: Junlong Li, name: Pengfei Liu
Published: 2024-01-13 07:57:01

Large Language Models (LLMs) are known to have limited extrapolation ability
beyond their pre-trained context window, constraining their application in
downstream tasks with lengthy inputs. Recent studies have sought to extend
LLMs' context window by modifying rotary position embedding (RoPE), a popular
position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
resource-intensive and lack comparative experiments to assess their
applicability. In this work, we identify the inherent need for LLMs' attention
entropy (i.e. the information entropy of attention scores) to maintain
stability and introduce a novel extension to RoPE which combines adjusting
RoPE's base frequency and scaling the attention logits to help LLMs efficiently
adapt to a larger context window. We validate the superiority of our method in
both fine-tuning performance and robustness across different context window
sizes on various context-demanding tasks. Notably, our method extends the
context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6
training steps, showcasing extraordinary efficiency. Finally, we also explore
how data compositions and training curricula affect context window extension
for specific downstream tasks, suggesting fine-tuning LLMs with lengthy
conversations as a good starting point. We release our code and SFT data at
https://github.com/GAIR-NLP/Entropy-ABF.
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
Authors: name: Jiaheng Liu, name: Zhiqi Bai, name: Yuanxing Zhang, name: Chenchen Zhang, name: Yu Zhang, name: Ge Zhang, name: Jiakai Wang, name: Haoran Que, name: Yukang Chen, name: Wenbo Su, name: Tiezheng Ge, name: Jie Fu, name: Wenhu Chen, name: Bo Zheng
Published: 2024-01-13 02:11:20

Typically, training LLMs with long context sizes is computationally
expensive, requiring extensive training hours and GPU resources. Existing
long-context extension methods usually need additional training procedures to
support corresponding long-context windows, where the long-context training
data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
the aforementioned issues, we propose an Efficient and Extreme length extension
method for Large Language Models, called E 2 -LLM, with only one training
procedure and dramatically reduced computation cost, which also removes the
need to collect long-context data. Concretely, first, the training data of our
E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
greatly. Second, the training procedure on the short training context window is
performed only once time, and we can support different evaluation context
windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
we introduce two different augmentation methods on the scale and position index
parameters for different samples in training. It aims to make the model more
robust to the different relative differences when directly interpolating the
arbitrary context length at inference. Comprehensive experimental results on
multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
challenging long-context tasks.
Contrastive Preference Optimization: Pushing the Boundaries of LLM
  Performance in Machine Translation
Authors: name: Haoran Xu, name: Amr Sharaf, name: Yunmo Chen, name: Weiting Tan, name: Lingfeng Shen, name: Benjamin Van Durme, name: Kenton Murray, name: Young Jin Kim
Published: 2024-01-16 15:04:51

Moderate-sized large language models (LLMs) -- those with 7B or 13B
parameters -- exhibit promising machine translation (MT) performance. However,
even the top-performing 13B LLM-based translation models, like ALMA, does not
match the performance of state-of-the-art conventional encoder-decoder
translation models or larger-scale LLMs such as GPT-4. In this study, we bridge
this performance gap. We first assess the shortcomings of supervised
fine-tuning for LLMs in the MT task, emphasizing the quality issues present in
the reference data, despite being human-generated. Then, in contrast to SFT
which mimics reference translations, we introduce Contrastive Preference
Optimization (CPO), a novel approach that trains models to avoid generating
adequate but not perfect translations. Applying CPO to ALMA models with only
22K parallel sentences and 12M parameters yields significant improvements. The
resulting model, called ALMA-R, can match or exceed the performance of the WMT
competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.
Scalable Pre-training of Large Autoregressive Image Models
Authors: name: Alaaeldin El-Nouby, name: Michal Klein, name: Shuangfei Zhai, name: Miguel Angel Bautista, name: Alexander Toshev, name: Vaishaal Shankar, name: Joshua M Susskind, name: Armand Joulin
Published: 2024-01-16 18:03:37

This paper introduces AIM, a collection of vision models pre-trained with an
autoregressive objective. These models are inspired by their textual
counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
properties. Specifically, we highlight two key findings: (1) the performance of
the visual features scale with both the model capacity and the quantity of
data, (2) the value of the objective function correlates with the performance
of the model on downstream tasks. We illustrate the practical implication of
these findings by pre-training a 7 billion parameter AIM on 2 billion images,
that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
this scale, we observe no sign of saturation in performance, suggesting that
AIM potentially represents a new frontier for training large-scale vision
models. The pre-training of AIM is similar to the pre-training of LLMs, and
does not require any image-specific strategy to stabilize the training at
scale.
Tuning Language Models by Proxy
Authors: name: Alisa Liu, name: Xiaochuang Han, name: Yizhong Wang, name: Yulia Tsvetkov, name: Yejin Choi, name: Noah A. Smith
Published: 2024-01-16 18:49:55

Despite the general capabilities of large pretrained language models, they
consistently benefit from further adaptation to better achieve desired
behaviors. However, tuning these models has become increasingly
resource-intensive, or impossible when model weights are private. We introduce
proxy-tuning, a lightweight decoding-time algorithm that operates on top of
black-box LMs to achieve the result of directly tuning the model, but by
accessing only its prediction over the output vocabulary. Our method instead
tunes a smaller LM, then applies the difference between the predictions of the
small tuned and untuned LMs to shift the original predictions of the base model
in the direction of tuning, while retaining the benefits of larger scale
pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
truly-tuned chat version, when evaluated across knowledge, reasoning, and
safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
are actually more truthful than directly tuned models, possibly because
decoding-time guidance better retains the model's factual knowledge. We then
demonstrate the generality of proxy-tuning by applying it for domain adaptation
on code, and task-specific finetuning on question-answering and math problems.
Our work demonstrates the promise of using small tuned LMs to efficiently
customize large, potentially proprietary LMs through decoding-time guidance.
HexaGen3D: StableDiffusion is just one step away from Fast and Diverse
  Text-to-3D Generation
Authors: name: Antoine Mercier, name: Ramin Nakhli, name: Mahesh Reddy, name: Rajeev Yasarla, name: Hong Cai, name: Fatih Porikli, name: Guillaume Berger
Published: 2024-01-15 14:41:15

Despite the latest remarkable advances in generative modeling, efficient
generation of high-quality 3D assets from textual prompts remains a difficult
task. A key challenge lies in data scarcity: the most extensive 3D datasets
encompass merely millions of assets, while their 2D counterparts contain
billions of text-image pairs. To address this, we propose a novel approach
which harnesses the power of large, pretrained 2D diffusion models. More
specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image
model to jointly predict 6 orthographic projections and the corresponding
latent triplane. We then decode these latents to generate a textured mesh.
HexaGen3D does not require per-sample optimization, and can infer high-quality
and diverse objects from textual prompts in 7 seconds, offering significantly
better quality-to-latency trade-offs when comparing to existing approaches.
Furthermore, HexaGen3D demonstrates strong generalization to new objects or
compositions.
Quantum Denoising Diffusion Models
Authors: name: Michael Kölle, name: Gerhard Stenzel, name: Jonas Stein, name: Sebastian Zielinski, name: Björn Ommer, name: Claudia Linnhoff-Popien
Published: 2024-01-13 11:38:08

In recent years, machine learning models like DALL-E, Craiyon, and Stable
Diffusion have gained significant attention for their ability to generate
high-resolution images from concise descriptions. Concurrently, quantum
computing is showing promising advances, especially with quantum machine
learning which capitalizes on quantum mechanics to meet the increasing
computational requirements of traditional machine learning algorithms. This
paper explores the integration of quantum machine learning and variational
quantum circuits to augment the efficacy of diffusion-based image generation
models. Specifically, we address two challenges of classical diffusion models:
their low sampling speed and the extensive parameter requirements. We introduce
two quantum diffusion models and benchmark their capabilities against their
classical counterparts using MNIST digits, Fashion MNIST, and CIFAR-10. Our
models surpass the classical models with similar parameter counts in terms of
performance metrics FID, SSIM, and PSNR. Moreover, we introduce a consistency
model unitary single sampling architecture that combines the diffusion
procedure into a single step, enabling a fast one-step image generation.
InstantID: Zero-shot Identity-Preserving Generation in Seconds
Authors: name: Qixun Wang, name: Xu Bai, name: Haofan Wang, name: Zekui Qin, name: Anthony Chen
Published: 2024-01-15 07:50:18

There has been significant progress in personalized image synthesis with
methods such as Textual Inversion, DreamBooth, and LoRA. Yet, their real-world
applicability is hindered by high storage demands, lengthy fine-tuning
processes, and the need for multiple reference images. Conversely, existing ID
embedding-based methods, while requiring only a single forward inference, face
challenges: they either necessitate extensive fine-tuning across numerous model
parameters, lack compatibility with community pre-trained models, or fail to
maintain high face fidelity. Addressing these limitations, we introduce
InstantID, a powerful diffusion model-based solution. Our plug-and-play module
adeptly handles image personalization in various styles using just a single
facial image, while ensuring high fidelity. To achieve this, we design a novel
IdentityNet by imposing strong semantic and weak spatial conditions,
integrating facial and landmark images with textual prompts to steer the image
generation. InstantID demonstrates exceptional performance and efficiency,
proving highly beneficial in real-world applications where identity
preservation is paramount. Moreover, our work seamlessly integrates with
popular pre-trained text-to-image diffusion models like SD1.5 and SDXL, serving
as an adaptable plugin. Our codes and pre-trained checkpoints will be available
at https://github.com/InstantID/InstantID.
Towards A Better Metric for Text-to-Video Generation
Authors: name: Jay Zhangjie Wu, name: Guian Fang, name: Haoning Wu, name: Xintao Wang, name: Yixiao Ge, name: Xiaodong Cun, name: David Junhao Zhang, name: Jia-Wei Liu, name: Yuchao Gu, name: Rui Zhao, name: Weisi Lin, name: Wynne Hsu, name: Ying Shan, name: Mike Zheng Shou
Published: 2024-01-15 15:42:39

Generative models have demonstrated remarkable capability in synthesizing
high-quality text, images, and videos. For video generation, contemporary
text-to-video models exhibit impressive capabilities, crafting visually
stunning videos. Nonetheless, evaluating such videos poses significant
challenges. Current research predominantly employs automated metrics such as
FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis,
particularly in the temporal assessment of video content, thus rendering them
unreliable indicators of true video quality. Furthermore, while user studies
have the potential to reflect human perception accurately, they are hampered by
their time-intensive and laborious nature, with outcomes that are often tainted
by subjective bias. In this paper, we investigate the limitations inherent in
existing metrics and introduce a novel evaluation pipeline, the Text-to-Video
Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video
Alignment, which scrutinizes the fidelity of the video in representing the
given text description, and (2) Video Quality, which evaluates the video's
overall production caliber with a mixture of experts. Moreover, to evaluate the
proposed metrics and facilitate future improvements on them, we present the
TVGE dataset, collecting human judgements of 2,543 text-to-video generated
videos on the two criteria. Experiments on the TVGE dataset demonstrate the
superiority of the proposed T2VScore on offering a better metric for
text-to-video generation.
PALP: Prompt Aligned Personalization of Text-to-Image Models
Authors: name: Moab Arar, name: Andrey Voynov, name: Amir Hertz, name: Omri Avrahami, name: Shlomi Fruchter, name: Yael Pritch, name: Daniel Cohen-Or, name: Ariel Shamir
Published: 2024-01-11 18:35:33

Content creators often aim to create personalized images using personal
subjects that go beyond the capabilities of conventional text-to-image models.
Additionally, they may want the resulting image to encompass a specific
location, style, ambiance, and more. Existing personalization methods may
compromise personalization ability or the alignment to complex textual prompts.
This trade-off can impede the fulfillment of user prompts and subject fidelity.
We propose a new approach focusing on personalization methods for a
single prompt to address this issue. We term our approach prompt-aligned
personalization. While this may seem restrictive, our method excels in
improving text alignment, enabling the creation of images with complex and
intricate prompts, which may pose a challenge for current techniques. In
particular, our method keeps the personalized model aligned with a target
prompt using an additional score distillation sampling term. We demonstrate the
versatility of our method in multi- and single-shot settings and further show
that it can compose multiple subjects or use inspiration from reference images,
such as artworks. We compare our approach quantitatively and qualitatively with
existing baselines and state-of-the-art techniques.
Object-Centric Diffusion for Efficient Video Editing
Authors: name: Kumara Kahatapitiya, name: Adil Karjauv, name: Davide Abati, name: Fatih Porikli, name: Yuki M. Asano, name: Amirhossein Habibian
Published: 2024-01-11 08:36:15

Diffusion-based video editing have reached impressive quality and can
transform either the global style, local structure, and attributes of given
video inputs, following textual edit prompts. However, such solutions typically
incur heavy memory and computational costs to generate temporally-coherent
frames, either in the form of diffusion inversion and/or cross-frame attention.
In this paper, we conduct an analysis of such inefficiencies, and suggest
simple yet effective modifications that allow significant speed-ups whilst
maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as
OCD, to further reduce latency by allocating computations more towards
foreground edited regions that are arguably more important for perceptual
quality. We achieve this by two novel proposals: i) Object-Centric Sampling,
decoupling the diffusion steps spent on salient regions or background,
allocating most of the model capacity to the former, and ii) Object-Centric 3D
Token Merging, which reduces cost of cross-frame attention by fusing redundant
tokens in unimportant background regions. Both techniques are readily
applicable to a given video editing model without retraining, and can
drastically reduce its memory and computational cost. We evaluate our proposals
on inversion-based and control-signal-based editing pipelines, and show a
latency reduction up to 10x for a comparable synthesis quality.
TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering
Authors: name: Linus Franke, name: Darius Rückert, name: Laura Fink, name: Marc Stamminger
Published: 2024-01-11 16:06:36

Point-based radiance field rendering has demonstrated impressive results for
novel view synthesis, offering a compelling blend of rendering quality and
computational efficiency. However, also latest approaches in this domain are
not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al.
2023] struggles when tasked with rendering highly detailed scenes, due to
blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022]
can accommodate crisper images, but the neural reconstruction network decreases
performance, it grapples with temporal instability and it is unable to
effectively address large gaps in the point cloud.
  In this paper, we present TRIPS (Trilinear Point Splatting), an approach that
combines ideas from both Gaussian Splatting and ADOP. The fundamental concept
behind our novel technique involves rasterizing points into a screen-space
image pyramid, with the selection of the pyramid layer determined by the
projected point size. This approach allows rendering arbitrarily large points
using a single trilinear write. A lightweight neural network is then used to
reconstruct a hole-free image including detail beyond splat resolution.
Importantly, our render pipeline is entirely differentiable, allowing for
automatic optimization of both point sizes and positions.
  Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art
methods in terms of rendering quality while maintaining a real-time frame rate
of 60 frames per second on readily available hardware. This performance extends
to challenging scenarios, such as scenes featuring intricate geometry,
expansive landscapes, and auto-exposed footage.
Diffusion Priors for Dynamic View Synthesis from Monocular Videos
Authors: name: Chaoyang Wang, name: Peiye Zhuang, name: Aliaksandr Siarohin, name: Junli Cao, name: Guocheng Qian, name: Hsin-Ying Lee, name: Sergey Tulyakov
Published: 2024-01-10 23:26:41

Dynamic novel view synthesis aims to capture the temporal evolution of visual
content within videos. Existing methods struggle to distinguishing between
motion and structure, particularly in scenarios where camera poses are either
unknown or constrained compared to object motion. Furthermore, with information
solely from reference images, it is extremely challenging to hallucinate unseen
regions that are occluded or partially observed in the given videos. To address
these issues, we first finetune a pretrained RGB-D diffusion model on the video
frames using a customization technique. Subsequently, we distill the knowledge
from the finetuned model to a 4D representations encompassing both dynamic and
static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves
geometric consistency while preserving the scene identity. We perform thorough
experiments to evaluate the efficacy of the proposed method qualitatively and
quantitatively. Our results demonstrate the robustness and utility of our
approach in challenging cases, further advancing dynamic novel view synthesis.
Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for
  Text-to-Image Generation
Authors: name: Seung Hyun Lee, name: Yinxiao Li, name: Junjie Ke, name: Innfarn Yoo, name: Han Zhang, name: Jiahui Yu, name: Qifei Wang, name: Fei Deng, name: Glenn Entis, name: Junfeng He, name: Gang Li, name: Sangpil Kim, name: Irfan Essa, name: Feng Yang
Published: 2024-01-11 05:36:36

Recent works demonstrate that using reinforcement learning (RL) with quality
rewards can enhance the quality of generated images in text-to-image (T2I)
generation. However, a simple aggregation of multiple rewards may cause
over-optimization in certain metrics and degradation in others, and it is
challenging to manually find the optimal weights. An effective strategy to
jointly optimize multiple rewards in RL for T2I generation is highly desirable.
This paper introduces Parrot, a novel multi-reward RL framework for T2I
generation. Through the use of the batch-wise Pareto optimal selection, Parrot
automatically identifies the optimal trade-off among different rewards during
the RL optimization of the T2I generation. Additionally, Parrot employs a joint
optimization approach for the T2I model and the prompt expansion network,
facilitating the generation of quality-aware text prompts, thus further
enhancing the final image quality. To counteract the potential catastrophic
forgetting of the original user prompt due to prompt expansion, we introduce
original prompt centered guidance at inference time, ensuring that the
generated image remains faithful to the user input. Extensive experiments and a
user study demonstrate that Parrot outperforms several baseline methods across
various quality criteria, including aesthetics, human preference, image
sentiment, and text-image alignment.