Orionnnn/hub-stats huggingface dataset:
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Authors: name: Dejia Xu, name: Weili Nie, name: Chao Liu, name: Sifei Liu, name: Jan Kautz, name: Zhangyang Wang, name: Arash Vahdat
Published: 2024-06-04 17:27:19

Recently video diffusion models have emerged as expressive generative tools
for high-quality video content creation readily available to general users.
However, these models often do not offer precise control over camera poses for
video generation, limiting the expression of cinematic language and user
control. To address this issue, we introduce CamCo, which allows fine-grained
Camera pose Control for image-to-video generation. We equip a pre-trained
image-to-video generator with accurately parameterized camera pose input using
Pl\"ucker coordinates. To enhance 3D consistency in the videos produced, we
integrate an epipolar attention module in each attention block that enforces
epipolar constraints to the feature maps. Additionally, we fine-tune CamCo on
real-world videos with camera poses estimated through structure-from-motion
algorithms to better synthesize object motion. Our experiments show that CamCo
significantly improves 3D consistency and camera control capabilities compared
to previous models while effectively generating plausible object motion.
Project page: https://ir1d.github.io/CamCo/
I4VGen: Image as Stepping Stone for Text-to-Video Generation
Authors: name: Xiefan Guo, name: Jinlin Liu, name: Miaomiao Cui, name: Di Huang
Published: 2024-06-04 11:48:44

Text-to-video generation has lagged behind text-to-image synthesis in quality
and diversity due to the complexity of spatio-temporal modeling and limited
video-text datasets. This paper presents I4VGen, a training-free and
plug-and-play video diffusion inference framework, which enhances text-to-video
generation by leveraging robust image techniques. Specifically, following
text-to-image-to-video, I4VGen decomposes the text-to-video generation into two
stages: anchor image synthesis and anchor image-guided video synthesis.
Correspondingly, a well-designed generation-selection pipeline is employed to
achieve visually-realistic and semantically-faithful anchor image, and an
innovative Noise-Invariant Video Score Distillation Sampling is incorporated to
animate the image to a dynamic video, followed by a video regeneration process
to refine the video. This inference strategy effectively mitigates the
prevalent issue of non-zero terminal signal-to-noise ratio. Extensive
evaluations show that I4VGen not only produces videos with higher visual
realism and textual fidelity but also integrates seamlessly into existing
image-to-video diffusion models, thereby improving overall video quality.
Learning Temporally Consistent Video Depth from Video Diffusion Priors
Authors: name: Jiahao Shao, name: Yuanbo Yang, name: Hongyu Zhou, name: Youmin Zhang, name: Yujun Shen, name: Matteo Poggi, name: Yiyi Liao
Published: 2024-06-03 16:20:24

This work addresses the challenge of video depth estimation, which expects
not only per-frame accuracy but, more importantly, cross-frame consistency.
Instead of directly developing a depth estimator from scratch, we reformulate
the prediction task into a conditional generation problem. This allows us to
leverage the prior knowledge embedded in existing video generation models,
thereby reducing learn- ing difficulty and enhancing generalizability.
Concretely, we study how to tame the public Stable Video Diffusion (SVD) to
predict reliable depth from input videos using a mixture of image depth and
video depth datasets. We empirically confirm that a procedural training
strategy - first optimizing the spatial layers of SVD and then optimizing the
temporal layers while keeping the spatial layers frozen - yields the best
results in terms of both spatial accuracy and temporal consistency. We further
examine the sliding window strategy for inference on arbitrarily long videos.
Our observations indicate a trade-off between efficiency and performance, with
a one-frame overlap already producing favorable results. Extensive experimental
results demonstrate the superiority of our approach, termed ChronoDepth, over
existing alternatives, particularly in terms of the temporal consistency of the
estimated depth. Additionally, we highlight the benefits of more consistent
video depth in two practical applications: depth-conditioned video generation
and novel view synthesis. Our project page is available at
https://jhaoshao.github.io/ChronoDepth/{this http URL}.
Show, Don't Tell: Aligning Language Models with Demonstrated Feedback
Authors: name: Omar Shaikh, name: Michelle Lam, name: Joey Hejna, name: Yijia Shao, name: Michael Bernstein, name: Diyi Yang
Published: 2024-06-02 23:13:56

Language models are aligned to emulate the collective voice of many,
resulting in outputs that align with no one in particular. Steering LLMs away
from generic output is possible through supervised finetuning or RLHF, but
requires prohibitively large datasets for new ad-hoc tasks. We argue that it is
instead possible to align an LLM to a specific setting by leveraging a very
small number (<10) of demonstrations as feedback. Our method, Demonstration
ITerated Task Optimization (DITTO), directly aligns language model outputs to a
user's demonstrated behaviors. Derived using ideas from online imitation
learning, DITTO cheaply generates online comparison data by treating users'
demonstrations as preferred over output from the LLM and its intermediate
checkpoints. We evaluate DITTO's ability to learn fine-grained style and task
alignment across domains such as news articles, emails, and blog posts.
Additionally, we conduct a user study soliciting a range of demonstrations from
participants (N=16). Across our benchmarks and user study, we find that
win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and
other self-play methods by an average of 19% points. By using demonstrations as
feedback directly, DITTO offers a novel method for effective customization of
LLMs.
Artificial Generational Intelligence: Cultural Accumulation in
  Reinforcement Learning
Authors: name: Jonathan Cook, name: Chris Lu, name: Edward Hughes, name: Joel Z. Leibo, name: Jakob Foerster
Published: 2024-06-01 10:33:32

Cultural accumulation drives the open-ended and diverse progress in
capabilities spanning human history. It builds an expanding body of knowledge
and skills by combining individual exploration with inter-generational
information transmission. Despite its widespread success among humans, the
capacity for artificial learning agents to accumulate culture remains
under-explored. In particular, approaches to reinforcement learning typically
strive for improvements over only a single lifetime. Generational algorithms
that do exist fail to capture the open-ended, emergent nature of cultural
accumulation, which allows individuals to trade-off innovation and imitation.
Building on the previously demonstrated ability for reinforcement learning
agents to perform social learning, we find that training setups which balance
this with independent learning give rise to cultural accumulation. These
accumulating agents outperform those trained for a single lifetime with the
same cumulative experience. We explore this accumulation by constructing two
models under two distinct notions of a generation: episodic generations, in
which accumulation occurs via in-context learning and train-time generations,
in which accumulation occurs via in-weights learning. In-context and in-weights
cultural accumulation can be interpreted as analogous to knowledge and skill
accumulation, respectively. To the best of our knowledge, this work is the
first to present general models that achieve emergent cultural accumulation in
reinforcement learning, opening up new avenues towards more open-ended learning
systems, as well as presenting new opportunities for modelling human culture.
MMLU-Pro: A More Robust and Challenging Multi-Task Language
  Understanding Benchmark
Authors: name: Yubo Wang, name: Xueguang Ma, name: Ge Zhang, name: Yuansheng Ni, name: Abhranil Chandra, name: Shiguang Guo, name: Weiming Ren, name: Aaran Arulraj, name: Xuan He, name: Ziyan Jiang, name: Tianle Li, name: Max Ku, name: Kai Wang, name: Alex Zhuang, name: Rongqi Fan, name: Xiang Yue, name: Wenhu Chen
Published: 2024-06-03 17:53:00

In the age of large-scale language models, benchmarks like the Massive
Multitask Language Understanding (MMLU) have been pivotal in pushing the
boundaries of what AI can achieve in language comprehension and reasoning
across diverse domains. However, as models continue to improve, their
performance on these benchmarks has begun to plateau, making it increasingly
difficult to discern differences in model capabilities. This paper introduces
MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
MMLU benchmark by integrating more challenging, reasoning-focused questions and
expanding the choice set from four to ten options. Additionally, MMLU-Pro
eliminates the trivial and noisy questions in MMLU. Our experimental results
show that MMLU-Pro not only raises the challenge, causing a significant drop in
accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
under varying prompts. With 24 different prompt styles tested, the sensitivity
of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
reasoning achieved better performance on MMLU-Pro compared to direct answering,
which is in stark contrast to the findings on the original MMLU, indicating
that MMLU-Pro includes more complex reasoning questions. Our assessments
confirm that MMLU-Pro is a more discriminative benchmark to better track
progress in the field.
μLO: Compute-Efficient Meta-Generalization of Learned Optimizers
Authors: name: Benjamin Thérien, name: Charles-Étienne Joseph, name: Boris Knyazev, name: Edouard Oyallon, name: Irina Rish, name: Eugene Belilovsky
Published: 2024-05-31 19:28:47

Learned optimizers (LOs) can significantly reduce the wall-clock training
time of neural networks, substantially reducing training costs. However, they
often suffer from poor meta-generalization, especially when training networks
larger than those seen during meta-training. To address this, we use the
recently proposed Maximal Update Parametrization (muP), which allows
zero-shot generalization of optimizer hyperparameters from smaller to larger
models. We extend muP theory to learned optimizers, treating the
meta-training problem as finding the learned optimizer under muP. Our
evaluation shows that LOs meta-trained with muP substantially improve
meta-generalization as compared to LOs trained under standard parametrization
(SP). Notably, when applied to large-width models, our best muLO, trained
for 103 GPU-hours, matches or exceeds the performance of VeLO, the largest
publicly available learned optimizer, meta-trained with 4000 TPU-months of
compute. Moreover, muLOs demonstrate better generalization than their SP
counterparts to deeper networks and to much longer training horizons (25 times
longer) than those seen during meta-training.
ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
  Generation
Authors: name: Shaoshu Yang, name: Yong Zhang, name: Xiaodong Cun, name: Ying Shan, name: Ran He
Published: 2024-06-03 00:31:13

Video generation has made remarkable progress in recent years, especially
since the advent of the video diffusion models. Many video generation models
can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
However, most video models can only generate low frame rate videos due to the
limited GPU memory as well as the difficulty of modeling a large set of frames.
The training videos are always uniformly sampled at a specified interval for
temporal compression. Previous methods promote the frame rate by either
training a video interpolation model in pixel space as a postprocessing stage
or training an interpolation model in latent space for a specific base video
model. In this paper, we propose a training-free video interpolation method for
generative video diffusion models, which is generalizable to different models
in a plug-and-play manner. We investigate the non-linearity in the feature
space of video diffusion models and transform a video model into a
self-cascaded video diffusion model with incorporating the designed hidden
state correction modules. The self-cascaded architecture and the correction
module are proposed to retain the temporal consistency between key frames and
the interpolated frames. Extensive evaluations are preformed on multiple
popular video models to demonstrate the effectiveness of the propose method,
especially that our training-free method is even comparable to trained
interpolation models supported by huge compute resources and large-scale
datasets.
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small
  Reference Models
Authors: name: Zachary Ankner, name: Cody Blakeney, name: Kartik Sreenivasan, name: Max Marion, name: Matthew L. Leavitt, name: Mansheej Paul
Published: 2024-05-30 23:50:20

In this work, we investigate whether small language models can determine
high-quality subsets of large-scale text datasets that improve the performance
of larger language models. While existing work has shown that pruning based on
the perplexity of a larger model can yield high-quality data, we investigate
whether smaller models can be used for perplexity-based pruning and how pruning
is affected by the domain composition of the data being pruned. We demonstrate
that for multiple dataset compositions, perplexity-based pruning of pretraining
data can significantly improve downstream task performance: pruning
based on perplexities computed with a 125 million parameter model improves the
average performance on downstream tasks of a 3 billion parameter model by up to
2.04 and achieves up to a 1.45times reduction in pretraining steps to reach
commensurate baseline performance. Furthermore, we demonstrate that such
perplexity-based data pruning also yields downstream performance gains in the
over-trained and data-constrained regimes.
Transformers are SSMs: Generalized Models and Efficient Algorithms
  Through Structured State Space Duality
Authors: name: Tri Dao, name: Albert Gu
Published: 2024-05-31 17:50:01

While Transformers have been the main architecture behind deep learning's
success in language modeling, state-space models (SSMs) such as Mamba have
recently been shown to match or outperform Transformers at small to medium
scale. We show that these families of models are actually quite closely
related, and develop a rich framework of theoretical connections between SSMs
and variants of attention, connected through various decompositions of a
well-studied class of structured semiseparable matrices. Our state space
duality (SSD) framework allows us to design a new architecture (Mamba-2) whose
core layer is an a refinement of Mamba's selective SSM that is 2-8X faster,
while continuing to be competitive with Transformers on language modeling.
4-bit Shampoo for Memory-Efficient Network Training
Authors: name: Sike Wang, name: Jia Li, name: Pan Zhou, name: Hua Huang
Published: 2024-05-28 13:02:56

Second-order optimizers, maintaining a matrix termed a preconditioner, are
superior to first-order optimizers in both theory and practice. The states
forming the preconditioner and its inverse root restrict the maximum size of
models trained by second-order optimizers. To address this, compressing 32-bit
optimizer states to lower bitwidths has shown promise in reducing memory usage.
However, current approaches only pertain to first-order optimizers. In this
paper, we propose the first 4-bit second-order optimizers, exemplified by 4-bit
Shampoo, maintaining performance similar to that of 32-bit ones. We show that
quantizing the eigenvector matrix of the preconditioner in 4-bit Shampoo is
remarkably better than quantizing the preconditioner itself both theoretically
and experimentally. By rectifying the orthogonality of the quantized
eigenvector matrix, we enhance the approximation of the preconditioner's
eigenvector matrix, which also benefits the computation of its inverse 4-th
root. Besides, we find that linear square quantization slightly outperforms
dynamic tree quantization when quantizing second-order optimizer states.
Evaluation on various networks for image classification demonstrates that our
4-bit Shampoo achieves comparable test accuracy to its 32-bit counterpart while
being more memory-efficient. The source code will be made available.
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of
  Multi-modal LLMs in Video Analysis
Authors: name: Chaoyou Fu, name: Yuhan Dai, name: Yondong Luo, name: Lei Li, name: Shuhuai Ren, name: Renrui Zhang, name: Zihan Wang, name: Chenyu Zhou, name: Yunhang Shen, name: Mengdan Zhang, name: Peixian Chen, name: Yanwei Li, name: Shaohui Lin, name: Sirui Zhao, name: Ke Li, name: Tong Xu, name: Xiawu Zheng, name: Enhong Chen, name: Rongrong Ji, name: Xing Sun
Published: 2024-05-31 17:59:47

In the quest for artificial general intelligence, Multi-modal Large Language
Models (MLLMs) have emerged as a focal point in recent advancements. However,
the predominant focus remains on developing their capabilities in static image
understanding. The potential of MLLMs in processing sequential visual data is
still insufficiently explored, highlighting the absence of a comprehensive,
high-quality assessment of their performance. In this paper, we introduce
Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of
MLLMs in Video analysis. Our work distinguishes from existing benchmarks
through four key features: 1) Diversity in video types, spanning 6 primary
visual domains with 30 subfields to ensure broad scenario generalizability; 2)
Duration in temporal dimension, encompassing both short-, medium-, and
long-term videos, ranging from 11 seconds to 1 hour, for robust contextual
dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides
video frames, including subtitles and audios, to unveil the all-round
capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual
labeling by expert annotators to facilitate precise and reliable model
assessment. 900 videos with a total of 256 hours are manually selected and
annotated by repeatedly viewing all the video content, resulting in 2,700
question-answer pairs. With Video-MME, we extensively evaluate various
state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as
open-source image models like InternVL-Chat-V1.5 and video models like
LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the
best-performing commercial model, significantly outperforming the open-source
models. Our dataset along with these findings underscores the need for further
improvements in handling longer sequences and multi-modal data. Project Page:
https://video-mme.github.io
Kaleido Diffusion: Improving Conditional Diffusion Models with
  Autoregressive Latent Modeling
Authors: name: Jiatao Gu, name: Ying Shen, name: Shuangfei Zhai, name: Yizhe Zhang, name: Navdeep Jaitly, name: Joshua M. Susskind
Published: 2024-05-31 17:41:11

Diffusion models have emerged as a powerful tool for generating high-quality
images from textual descriptions. Despite their successes, these models often
exhibit limited diversity in the sampled images, particularly when sampling
with a high classifier-free guidance weight. To address this issue, we present
Kaleido, a novel approach that enhances the diversity of samples by
incorporating autoregressive latent priors. Kaleido integrates an
autoregressive language model that encodes the original caption and generates
latent variables, serving as abstract and intermediary representations for
guiding and facilitating the image generation process. In this paper, we
explore a variety of discrete latent representations, including textual
descriptions, detection bounding boxes, object blobs, and visual tokens. These
representations diversify and enrich the input conditions to the diffusion
models, enabling more diverse outputs. Our experimental results demonstrate
that Kaleido effectively broadens the diversity of the generated image samples
from a given textual description while maintaining high image quality.
Furthermore, we show that Kaleido adheres closely to the guidance provided by
the generated latent variables, demonstrating its capability to effectively
control and direct the image generation process.
4Diffusion: Multi-view Video Diffusion Model for 4D Generation
Authors: name: Haiyu Zhang, name: Xinyuan Chen, name: Yaohui Wang, name: Xihui Liu, name: Yunhong Wang, name: Yu Qiao
Published: 2024-05-31 08:18:39

Current 4D generation methods have achieved noteworthy efficacy with the aid
of advanced diffusion generative models. However, these methods lack multi-view
spatial-temporal modeling and encounter challenges in integrating diverse prior
knowledge from multiple diffusion models, resulting in inconsistent temporal
appearance and flickers. In this paper, we propose a novel 4D generation
pipeline, namely 4Diffusion aimed at generating spatial-temporally consistent
4D content from a monocular video. We first design a unified diffusion model
tailored for multi-view video generation by incorporating a learnable motion
module into a frozen 3D-aware diffusion model to capture multi-view
spatial-temporal correlations. After training on a curated dataset, our
diffusion model acquires reasonable temporal consistency and inherently
preserves the generalizability and spatial consistency of the 3D-aware
diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling
loss, which is based on our multi-view video diffusion model, to optimize 4D
representation parameterized by dynamic NeRF. This aims to eliminate
discrepancies arising from multiple diffusion models, allowing for generating
spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to
enhance the appearance details and facilitate the learning of dynamic NeRF.
Extensive qualitative and quantitative experiments demonstrate that our method
achieves superior performance compared to previous methods.
MotionLLM: Understanding Human Behaviors from Human Motions and Videos
Authors: name: Ling-Hao Chen, name: Shunlin Lu, name: Ailing Zeng, name: Hao Zhang, name: Benyou Wang, name: Ruimao Zhang, name: Lei Zhang
Published: 2024-05-30 17:59:50

This study delves into the realm of multi-modality (i.e., video and motion
modalities) human behavior understanding by leveraging the powerful
capabilities of Large Language Models (LLMs). Diverging from recent LLMs
designed for video-only or motion-only understanding, we argue that
understanding human behavior necessitates joint modeling from both videos and
motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
and semantics effectively. In light of this, we present MotionLLM, a
straightforward yet effective framework for human motion understanding,
captioning, and reasoning. Specifically, MotionLLM adopts a unified
video-motion training strategy that leverages the complementary advantages of
existing coarse video-text data and fine-grained motion-text data to glean rich
spatial-temporal insights. Furthermore, we collect a substantial dataset,
MoVid, comprising diverse videos, motions, captions, and instructions.
Additionally, we propose the MoVid-Bench, with carefully manual annotations,
for better evaluation of human behavior understanding on video and motion.
Extensive experiments show the superiority of MotionLLM in the caption,
spatial-temporal comprehension, and reasoning ability.