Orionnnn/hub-stats huggingface dataset:
Efficient Video Diffusion Models via Content-Frame Motion-Latent
  Decomposition
Authors: name: Sihyun Yu, name: Weili Nie, name: De-An Huang, name: Boyi Li, name: Jinwoo Shin, name: Anima Anandkumar
Published: 2024-03-21 05:48:48

Video diffusion models have recently made great progress in generation
quality, but are still limited by the high memory and computational
requirements. This is because current video diffusion models often attempt to
process high-dimensional videos directly. To tackle this issue, we propose
content-motion latent diffusion model (CMD), a novel efficient extension of
pretrained image diffusion models for video generation. Specifically, we
propose an autoencoder that succinctly encodes a video as a combination of a
content frame (like an image) and a low-dimensional motion latent
representation. The former represents the common content, and the latter
represents the underlying motion in the video, respectively. We generate the
content frame by fine-tuning a pretrained image diffusion model, and we
generate the motion latent representation by training a new lightweight
diffusion model. A key innovation here is the design of a compact latent space
that can directly utilizes a pretrained image diffusion model, which has not
been done in previous latent video diffusion models. This leads to considerably
better quality generation and reduced computational costs. For instance, CMD
can sample a video 7.7times faster than prior approaches by generating a
video of 512times1024 resolution and length 16 in 3.1 seconds. Moreover, CMD
achieves an FVD score of 212.7 on WebVid-10M, 27.3% better than the previous
state-of-the-art of 292.4.
Evolutionary Optimization of Model Merging Recipes
Authors: name: Takuya Akiba, name: Makoto Shing, name: Yujin Tang, name: Qi Sun, name: David Ha
Published: 2024-03-19 22:56:53

We present a novel application of evolutionary algorithms to automate the
creation of powerful foundation models. While model merging has emerged as a
promising approach for LLM development due to its cost-effectiveness, it
currently relies on human intuition and domain knowledge, limiting its
potential. Here, we propose an evolutionary approach that overcomes this
limitation by automatically discovering effective combinations of diverse
open-source models, harnessing their collective intelligence without requiring
extensive additional training data or compute. Our approach operates in both
parameter space and data flow space, allowing for optimization beyond just the
weights of the individual models. This approach even facilitates cross-domain
merging, generating models like a Japanese LLM with Math reasoning
capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art
performance on a variety of established Japanese LLM benchmarks, even
surpassing models with significantly more parameters, despite not being
explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM
generated through our approach demonstrates its effectiveness in describing
Japanese culture-specific content, outperforming previous Japanese VLMs. This
work not only contributes new state-of-the-art models back to the open-source
community, but also introduces a new paradigm for automated model composition,
paving the way for exploring alternative, efficient approaches to foundation
model development.
DepthFM: Fast Monocular Depth Estimation with Flow Matching
Authors: name: Ming Gui, name: Johannes S. Fischer, name: Ulrich Prestel, name: Pingchuan Ma, name: Dmytro Kotovenko, name: Olga Grebenkova, name: Stefan Andreas Baumann, name: Vincent Tao Hu, name: Björn Ommer
Published: 2024-03-20 17:51:53

Monocular depth estimation is crucial for numerous downstream vision tasks
and applications. Current discriminative approaches to this problem are limited
due to blurry artifacts, while state-of-the-art generative methods suffer from
slow sampling due to their SDE nature. Rather than starting from noise, we seek
a direct mapping from input image to depth map. We observe that this can be
effectively framed using flow matching, since its straight trajectories through
solution space offer efficiency and high quality. Our study demonstrates that a
pre-trained image diffusion model can serve as an adequate prior for a flow
matching depth model, allowing efficient training on only synthetic data to
generalize to real images. We find that an auxiliary surface normals loss
further improves the depth estimates. Due to the generative nature of our
approach, our model reliably predicts the confidence of its depth estimates. On
standard benchmarks of complex natural scenes, our lightweight approach
exhibits state-of-the-art performance at favorable low computational cost
despite only being trained on little synthetic data.
Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific
  Adaptation
Authors: name: Fu-Yun Wang, name: Xiaoshi Wu, name: Zhaoyang Huang, name: Xiaoyu Shi, name: Dazhong Shen, name: Guanglu Song, name: Yu Liu, name: Hongsheng Li
Published: 2024-03-20 16:53:45

Video outpainting is a challenging task, aiming at generating video content
outside the viewport of the input video while maintaining inter-frame and
intra-frame consistency. Existing methods fall short in either generation
quality or flexibility. We introduce MOTIA Mastering Video Outpainting Through
Input-Specific Adaptation, a diffusion-based pipeline that leverages both the
intrinsic data-specific patterns of the source video and the image/video
generative prior for effective outpainting. MOTIA comprises two main phases:
input-specific adaptation and pattern-aware outpainting. The input-specific
adaptation phase involves conducting efficient and effective pseudo outpainting
learning on the single-shot source video. This process encourages the model to
identify and learn patterns within the source video, as well as bridging the
gap between standard generative processes and outpainting. The subsequent
phase, pattern-aware outpainting, is dedicated to the generalization of these
learned patterns to generate outpainting outcomes. Additional strategies
including spatial-aware insertion and noise travel are proposed to better
leverage the diffusion model's generative prior and the acquired video patterns
from source videos. Extensive evaluations underscore MOTIA's superiority,
outperforming existing state-of-the-art methods in widely recognized
benchmarks. Notably, these advancements are achieved without necessitating
extensive, task-specific tuning.
RadSplat: Radiance Field-Informed Gaussian Splatting for Robust
  Real-Time Rendering with 900+ FPS
Authors: name: Michael Niemeyer, name: Fabian Manhardt, name: Marie-Julie Rakotosaona, name: Michael Oechsle, name: Daniel Duckworth, name: Rama Gosula, name: Keisuke Tateno, name: John Bates, name: Dominik Kaeser, name: Federico Tombari
Published: 2024-03-20 17:59:55

Recent advances in view synthesis and real-time rendering have achieved
photorealistic quality at impressive rendering speeds. While Radiance
Field-based methods achieve state-of-the-art quality in challenging scenarios
such as in-the-wild captures and large-scale scenes, they often suffer from
excessively high compute requirements linked to volumetric rendering. Gaussian
Splatting-based methods, on the other hand, rely on rasterization and naturally
achieve real-time rendering but suffer from brittle optimization heuristics
that underperform on more challenging scenes. In this work, we present
RadSplat, a lightweight method for robust real-time rendering of complex
scenes. Our main contributions are threefold. First, we use radiance fields as
a prior and supervision signal for optimizing point-based scene
representations, leading to improved quality and more robust optimization.
Next, we develop a novel pruning technique reducing the overall point count
while maintaining high quality, leading to smaller and more compact scene
representations with faster inference speeds. Finally, we propose a novel
test-time filtering approach that further accelerates rendering and allows to
scale to larger, house-sized scenes. We find that our method enables
state-of-the-art synthesis of complex captures at 900+ FPS.
ZigMa: Zigzag Mamba Diffusion Model
Authors: name: Vincent Tao Hu, name: Stefan Andreas Baumann, name: Ming Gui, name: Olga Grebenkova, name: Pingchuan Ma, name: Johannes Fischer, name: Bjorn Ommer
Published: 2024-03-20 17:59:14

The diffusion model has long been plagued by scalability and quadratic
complexity issues, especially within transformer-based structures. In this
study, we aim to leverage the long sequence modeling capability of a
State-Space Model called Mamba to extend its applicability to visual data
generation. Firstly, we identify a critical oversight in most current
Mamba-based vision methods, namely the lack of consideration for spatial
continuity in the scan scheme of Mamba. Secondly, building upon this insight,
we introduce a simple, plug-and-play, zero-parameter method named Zigzag Mamba,
which outperforms Mamba-based baselines and demonstrates improved speed and
memory utilization compared to transformer-based baselines. Lastly, we
integrate Zigzag Mamba with the Stochastic Interpolant framework to investigate
the scalability of the model on large-resolution visual datasets, such as
FacesHQ 1024times 1024 and UCF101, MultiModal-CelebA-HQ, and MS COCO
256times 256. Code will be released at https://taohu.me/zigma/
VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis
Authors: name: Yumeng Li, name: William Beluch, name: Margret Keuper, name: Dan Zhang, name: Anna Khoreva
Published: 2024-03-20 10:58:58

Despite tremendous progress in the field of text-to-video (T2V) synthesis,
open-sourced T2V diffusion models struggle to generate longer videos with
dynamically varying and evolving content. They tend to synthesize quasi-static
videos, ignoring the necessary visual change-over-time implied in the text
prompt. At the same time, scaling these models to enable longer, more dynamic
video synthesis often remains computationally intractable. To address this
challenge, we introduce the concept of Generative Temporal Nursing (GTN), where
we aim to alter the generative process on the fly during inference to improve
control over the temporal dynamics and enable generation of longer videos. We
propose a method for GTN, dubbed VSTAR, which consists of two key ingredients:
1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis
based on the original single prompt leveraging LLMs, which gives accurate
textual guidance to different visual states of longer videos, and 2) Temporal
Attention Regularization (TAR) - a regularization technique to refine the
temporal attention units of the pre-trained T2V diffusion models, which enables
control over the video dynamics. We experimentally showcase the superiority of
the proposed approach in generating longer, visually appealing videos over
existing open-sourced T2V models. We additionally analyze the temporal
attention maps realized with and without VSTAR, demonstrating the importance of
applying our method to mitigate neglect of the desired visual change over time.
HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
  Large Language Models
Authors: name: Wenqiao Zhang, name: Tianwei Lin, name: Jiang Liu, name: Fangxun Shu, name: Haoyuan Li, name: Lei Zhang, name: He Wanggui, name: Hao Zhou, name: Zheqi Lv, name: Hao Jiang, name: Juncheng Li, name: Siliang Tang, name: Yueting Zhuang
Published: 2024-03-20 09:42:43

Recent advancements indicate that scaling up Multimodal Large Language Models
(MLLMs) effectively enhances performance on downstream multimodal tasks. The
prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
text-like tokens using a static vision-language mapper, thereby enabling
static LLMs to develop the capability to comprehend visual information
through visual instruction tuning. Although promising, the static tuning
strategy~The static tuning refers to the trained model with static
parameters. that shares the same parameters may constrain performance across
different downstream multimodal tasks. In light of this, we introduce
HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
in conjunction with a dynamic visual expert and language expert, respectively.
These experts are derived from HyperNetworks, which generates adaptive
parameter shifts through visual and language guidance, enabling dynamic
projector and LLM modeling in two-stage training.
  Our experiments demonstrate that our solution significantly surpasses LLaVA
on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
LLaVA-Bench. ~Our project is available on the link
https://github.com/DCDmllm/HyperLLaVA.
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Authors: name: Yaowei Zheng, name: Richong Zhang, name: Junhao Zhang, name: Yanhan Ye, name: Zheyan Luo
Published: 2024-03-20 08:08:54

Efficient fine-tuning is vital for adapting large language models (LLMs) to
downstream tasks. However, it requires non-trivial efforts to implement these
methods on different models. We present LlamaFactory, a unified framework that
integrates a suite of cutting-edge efficient training methods. It allows users
to flexibly customize the fine-tuning of 100+ LLMs without the need for coding
through the built-in web UI LlamaBoard. We empirically validate the efficiency
and effectiveness of our framework on language modeling and text generation
tasks. It has been released at https://github.com/hiyouga/LLaMA-Factory and
already received over 13,000 stars and 1,600 forks.
Compress3D: a Compressed Latent Space for 3D Generation from a Single
  Image
Authors: name: Bowen Zhang, name: Tianyu Yang, name: Yu Li, name: Lei Zhang, name: Xi Zhao
Published: 2024-03-20 11:51:04

3D generation has witnessed significant advancements, yet efficiently
producing high-quality 3D assets from a single image remains challenging. In
this paper, we present a triplane autoencoder, which encodes 3D models into a
compact triplane latent space to effectively compress both the 3D geometry and
texture information. Within the autoencoder framework, we introduce a 3D-aware
cross-attention mechanism, which utilizes low-resolution latent representations
to query features from a high-resolution 3D feature volume, thereby enhancing
the representation capacity of the latent space. Subsequently, we train a
diffusion model on this refined latent space. In contrast to solely relying on
image embedding for 3D generation, our proposed method advocates for the
simultaneous utilization of both image embedding and shape embedding as
conditions. Specifically, the shape embedding is estimated via a diffusion
prior model conditioned on the image embedding. Through comprehensive
experiments, we demonstrate that our method outperforms state-of-the-art
algorithms, achieving superior performance while requiring less training data
and time. Our approach enables the generation of high-quality 3D assets in
merely 7 seconds on a single A100 GPU.
Towards 3D Molecule-Text Interpretation in Language Models
Authors: name: Sihang Li, name: Zhiyuan Liu, name: Yanchen Luo, name: Xiang Wang, name: Xiangnan He, name: Kenji Kawaguchi, name: Tat-Seng Chua, name: Qi Tian
Published: 2024-01-25 03:42:00

Language Models (LMs) have greatly influenced diverse domains. However, their
inherent limitation in comprehending 3D molecular structures has considerably
constrained their potential in the biomolecular domain. To bridge this gap, we
focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
3D molecules by equipping the LM with a 3D molecular encoder. This integration
is achieved by a 3D molecule-text projector, bridging the 3D molecular
encoder's representation space and the LM's input space. Moreover, to enhance
3D-MoLM's ability of cross-modal molecular understanding and instruction
following, we meticulously curated a 3D molecule-centric instruction tuning
dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
and LM. It significantly surpasses existing baselines on downstream tasks,
including molecule-text retrieval, molecule captioning, and more challenging
open-text molecular QA tasks, especially focusing on 3D-dependent properties.
Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos
Authors: name: Hadi Alzayer, name: Zhihao Xia, name: Xuaner Zhang, name: Eli Shechtman, name: Jia-Bin Huang, name: Michael Gharbi
Published: 2024-03-19 17:59:58

We propose a generative model that, given a coarsely edited image,
synthesizes a photorealistic output that follows the prescribed layout. Our
method transfers fine details from the original image and preserves the
identity of its parts. Yet, it adapts it to the lighting and context defined by
the new layout. Our key insight is that videos are a powerful source of
supervision for this task: objects and camera motions provide many observations
of how the world changes with viewpoint, lighting, and physical interactions.
We construct an image dataset in which each sample is a pair of source and
target frames extracted from the same video at randomly chosen time intervals.
We warp the source frame toward the target using two motion models that mimic
the expected test-time user edits. We supervise our model to translate the
warped image into the ground truth, starting from a pretrained diffusion model.
Our model design explicitly enables fine detail transfer from the source frame
to the generated image, while closely following the user-specified layout. We
show that by using simple segmentations and coarse 2D manipulations, we can
synthesize a photorealistic edit faithful to the user's input while addressing
second-order effects like harmonizing the lighting and physical interactions
between edited objects.
When Do We Not Need Larger Vision Models?
Authors: name: Baifeng Shi, name: Ziyang Wu, name: Maolin Mao, name: Xin Wang, name: Trevor Darrell
Published: 2024-03-19 17:58:39

Scaling up the size of vision models has been the de facto standard to obtain
more powerful visual representations. In this work, we discuss the point beyond
which larger vision models are not necessary. First, we demonstrate the power
of Scaling on Scales (S^2), whereby a pre-trained and frozen smaller vision
model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform
larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth
estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation.
Notably, S^2 achieves state-of-the-art performance in detailed understanding
of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the
conditions under which S^2 is a preferred scaling approach compared to
scaling on model size. While larger models have the advantage of better
generalization on hard examples, we show that features of larger vision models
can be well approximated by those of multi-scale smaller models. This suggests
most, if not all, of the representations learned by current large pre-trained
models can also be obtained from multi-scale smaller models. Our results show
that a multi-scale smaller model has comparable learning capacity to a larger
model, and pre-training smaller models with S^2 can match or even exceed the
advantage of larger models. We release a Python package that can apply S^2 on
any vision model with one line of code:
https://github.com/bfshi/scaling_on_scales.
SceneScript: Reconstructing Scenes With An Autoregressive Structured
  Language Model
Authors: name: Armen Avetisyan, name: Christopher Xie, name: Henry Howard-Jenkins, name: Tsun-Yi Yang, name: Samir Aroudj, name: Suvam Patra, name: Fuyang Zhang, name: Duncan Frost, name: Luke Holland, name: Campbell Orme, name: Jakob Engel, name: Edward Miller, name: Richard Newcombe, name: Vasileios Balntas
Published: 2024-03-19 18:01:29

We introduce SceneScript, a method that directly produces full scene models
as a sequence of structured language commands using an autoregressive,
token-based approach. Our proposed scene representation is inspired by recent
successes in transformers & LLMs, and departs from more traditional methods
which commonly describe scenes as meshes, voxel grids, point clouds or radiance
fields. Our method infers the set of structured language commands directly from
encoded visual data using a scene language encoder-decoder architecture. To
train SceneScript, we generate and release a large-scale synthetic dataset
called Aria Synthetic Environments consisting of 100k high-quality in-door
scenes, with photorealistic and ground-truth annotated renders of egocentric
scene walkthroughs. Our method gives state-of-the art results in architectural
layout estimation, and competitive results in 3D object detection. Lastly, we
explore an advantage for SceneScript, which is the ability to readily adapt to
new commands via simple additions to the structured language, which we
illustrate for tasks such as coarse 3D object part reconstruction.
Evaluating Frontier Models for Dangerous Capabilities
Authors: name: Mary Phuong, name: Matthew Aitchison, name: Elliot Catt, name: Sarah Cogan, name: Alexandre Kaskasoli, name: Victoria Krakovna, name: David Lindner, name: Matthew Rahtz, name: Yannis Assael, name: Sarah Hodkinson, name: Heidi Howard, name: Tom Lieberum, name: Ramana Kumar, name: Maria Abi Raad, name: Albert Webson, name: Lewis Ho, name: Sharon Lin, name: Sebastian Farquhar, name: Marcus Hutter, name: Gregoire Deletang, name: Anian Ruoss, name: Seliem El-Sayed, name: Sasha Brown, name: Anca Dragan, name: Rohin Shah, name: Allan Dafoe, name: Toby Shevlane
Published: 2024-03-20 17:54:26

To understand the risks posed by a new AI system, we must understand what it
can and cannot do. Building on prior work, we introduce a programme of new
"dangerous capability" evaluations and pilot them on Gemini 1.0 models. Our
evaluations cover four areas: (1) persuasion and deception; (2) cyber-security;
(3) self-proliferation; and (4) self-reasoning. We do not find evidence of
strong dangerous capabilities in the models we evaluated, but we flag early
warning signs. Our goal is to help advance a rigorous science of dangerous
capability evaluation, in preparation for future models.
RewardBench: Evaluating Reward Models for Language Modeling
Authors: name: Nathan Lambert, name: Valentina Pyatkin, name: Jacob Morrison, name: LJ Miranda, name: Bill Yuchen Lin, name: Khyathi Chandu, name: Nouha Dziri, name: Sachin Kumar, name: Tom Zick, name: Yejin Choi, name: Noah A. Smith, name: Hannaneh Hajishirzi
Published: 2024-03-20 17:49:54

Reward models (RMs) are at the crux of successful RLHF to align pretrained
models to human preferences, yet there has been relatively little study that
focuses on evaluation of those reward models. Evaluating reward models presents
an opportunity to understand the opaque technologies used for alignment of
language models and which values are embedded in them. To date, very few
descriptors of capabilities, training methods, or open-source reward models
exist. In this paper, we present RewardBench, a benchmark dataset and code-base
for evaluation, to enhance scientific understanding of reward models. The
RewardBench dataset is a collection of prompt-win-lose trios spanning chat,
reasoning, and safety, to benchmark how reward models perform on challenging,
structured and out-of-distribution queries. We created specific comparison
datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect
facts) why one answer should be preferred to another. On the RewardBench
leaderboard, we evaluate reward models trained with a variety of methods, such
as the direct MLE training of classifiers and the implicit reward modeling of
Direct Preference Optimization (DPO), and on a spectrum of datasets. We present
many findings on propensity for refusals, reasoning limitations, and
instruction following shortcomings of various reward models towards a better
understanding of the RLHF process.