Orionnnn/hub-stats huggingface dataset:
GaussianImage: 1000 FPS Image Representation and Compression by 2D
  Gaussian Splatting
Authors: name: Xinjie Zhang, name: Xingtong Ge, name: Tongda Xu, name: Dailan He, name: Yan Wang, name: Hongwei Qin, name: Guo Lu, name: Jing Geng, name: Jun Zhang
Published: 2024-03-13 14:02:54

Implicit neural representations (INRs) recently achieved great success in
image representation and compression, offering high visual quality and fast
rendering speeds with 10-1000 FPS, assuming sufficient GPU resources are
available. However, this requirement often hinders their use on low-end devices
with limited memory. In response, we propose a groundbreaking paradigm of image
representation and compression by 2D Gaussian Splatting, named GaussianImage.
We first introduce 2D Gaussian to represent the image, where each Gaussian has
8 parameters including position, covariance and color. Subsequently, we unveil
a novel rendering algorithm based on accumulated summation. Remarkably, our
method with a minimum of 3times lower GPU memory usage and 5times faster
fitting time not only rivals INRs (e.g., WIRE, I-NGP) in representation
performance, but also delivers a faster rendering speed of 1500-2000 FPS
regardless of parameter size. Furthermore, we integrate existing vector
quantization technique to build an image codec. Experimental results
demonstrate that our codec attains rate-distortion performance comparable to
compression-based INRs such as COIN and COIN++, while facilitating decoding
speeds of approximately 1000 FPS. Additionally, preliminary proof of concept
shows that our codec surpasses COIN and COIN++ in performance when using
partial bits-back coding.
VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
Authors: name: Enric Corona, name: Andrei Zanfir, name: Eduard Gabriel Bazavan, name: Nikos Kolotouros, name: Thiemo Alldieck, name: Cristian Sminchisescu
Published: 2024-03-13 17:59:02

We propose VLOGGER, a method for audio-driven human video generation from a
single input image of a person, which builds on the success of recent
generative diffusion models. Our method consists of 1) a stochastic
human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture
that augments text-to-image models with both spatial and temporal controls.
This supports the generation of high quality video of variable length, easily
controllable through high-level representations of human faces and bodies. In
contrast to previous work, our method does not require training for each
person, does not rely on face detection and cropping, generates the complete
image (not just the face or the lips), and considers a broad spectrum of
scenarios (e.g. visible torso or diverse subject identities) that are critical
to correctly synthesize humans who communicate. We also curate MENTOR, a new
and diverse dataset with 3d pose and expression annotations, one order of
magnitude larger than previous ones (800,000 identities) and with dynamic
gestures, on which we train and ablate our main technical contributions.
  VLOGGER outperforms state-of-the-art methods in three public benchmarks,
considering image quality, identity preservation and temporal consistency while
also generating upper-body gestures. We analyze the performance of VLOGGER with
respect to multiple diversity metrics, showing that our architectural choices
and the use of MENTOR benefit training a fair and unbiased model at scale.
Finally we show applications in video editing and personalization.
Follow-Your-Click: Open-domain Regional Image Animation via Short
  Prompts
Authors: name: Yue Ma, name: Yingqing He, name: Hongfa Wang, name: Andong Wang, name: Chenyang Qi, name: Chengfei Cai, name: Xiu Li, name: Zhifeng Li, name: Heung-Yeung Shum, name: Wei Liu, name: Qifeng Chen
Published: 2024-03-13 05:44:37

Despite recent advances in image-to-video generation, better controllability
and local animation are less explored. Most existing image-to-video methods are
not locally aware and tend to move the entire scene. However, human artists may
need to control the movement of different objects or regions. Additionally,
current I2V methods require users not only to describe the target motion but
also to provide redundant detailed descriptions of frame contents. These two
issues hinder the practical utilization of current I2V tools. In this paper, we
propose a practical framework, named Follow-Your-Click, to achieve image
animation with a simple user click (for specifying what to move) and a short
motion prompt (for specifying how to move). Technically, we propose the
first-frame masking strategy, which significantly improves the video generation
quality, and a motion-augmented module equipped with a short motion prompt
dataset to improve the short prompt following abilities of our model. To
further control the motion speed, we propose flow-based motion magnitude
control to control the speed of target movement more precisely. Our framework
has simpler yet precise user control and better generation performance than
previous methods. Extensive experiments compared with 7 baselines, including
both commercial tools and research methods on 8 metrics, suggest the
superiority of our approach. Project Page: https://follow-your-click.github.io/
Chronos: Learning the Language of Time Series
Authors: name: Abdul Fatir Ansari, name: Lorenzo Stella, name: Caner Turkmen, name: Xiyuan Zhang, name: Pedro Mercado, name: Huibin Shen, name: Oleksandr Shchur, name: Syama Sundar Rangapuram, name: Sebastian Pineda Arango, name: Shubham Kapoor, name: Jasper Zschiegner, name: Danielle C. Maddix, name: Michael W. Mahoney, name: Kari Torkkola, name: Andrew Gordon Wilson, name: Michael Bohlke-Schneider, name: Yuyang Wang
Published: 2024-03-12 16:53:54

We introduce Chronos, a simple yet effective framework for pretrained
probabilistic time series models. Chronos tokenizes time series values using
scaling and quantization into a fixed vocabulary and trains existing
transformer-based language model architectures on these tokenized time series
via the cross-entropy loss. We pretrained Chronos models based on the T5 family
(ranging from 20M to 710M parameters) on a large collection of publicly
available datasets, complemented by a synthetic dataset that we generated via
Gaussian processes to improve generalization. In a comprehensive benchmark
consisting of 42 datasets, and comprising both classical local models and deep
learning methods, we show that Chronos models: (a) significantly outperform
other methods on datasets that were part of the training corpus; and (b) have
comparable and occasionally superior zero-shot performance on new datasets,
relative to methods that were trained specifically on them. Our results
demonstrate that Chronos models can leverage time series data from diverse
domains to improve zero-shot accuracy on unseen forecasting tasks, positioning
pretrained models as a viable tool to greatly simplify forecasting pipelines.
Learning Generalizable Feature Fields for Mobile Manipulation
Authors: name: Ri-Zhao Qiu, name: Yafei Hu, name: Ge Yang, name: Yuchen Song, name: Yang Fu, name: Jianglong Ye, name: Jiteng Mu, name: Ruihan Yang, name: Nikolay Atanasov, name: Sebastian Scherer, name: Xiaolong Wang
Published: 2024-03-12 11:51:55

An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner, so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherit to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we treat
generative novel view synthesis as a pre-training task, and then align the
resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
ability to generalize to open-set objects as well as running time, when
performing open-vocabulary mobile manipulation in dynamic scenes.
MoAI: Mixture of All Intelligence for Large Language and Vision Models
Authors: name: Byung-Kwan Lee, name: Beomchan Park, name: Chae Won Kim, name: Yong Man Ro
Published: 2024-03-12 10:44:13

The rise of large language models (LLMs) and instruction tuning has led to
the current trend of instruction-tuned large language and vision models
(LLVMs). This trend involves either meticulously curating numerous instruction
tuning datasets tailored to specific objectives or enlarging LLVMs to manage
vast amounts of vision language (VL) data. However, current LLVMs have
disregarded the detailed and comprehensive real-world scene understanding
available from specialized computer vision (CV) models in visual perception
tasks such as segmentation, detection, scene graph generation (SGG), and
optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
the large capacity and emergent capabilities of their LLM backbones. Therefore,
we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
auxiliary visual information obtained from the outputs of external
segmentation, detection, SGG, and OCR models. MoAI operates through two newly
introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
outputs of the external CV models, the MoAI-Compressor aligns and condenses
them to efficiently use relevant auxiliary visual information for VL tasks.
MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
auxiliary features from the external CV models, and (3) language features by
utilizing the concept of Mixture of Experts. Through this integration, MoAI
significantly outperforms both open-source and closed-source LLVMs in numerous
zero-shot VL tasks, particularly those related to real-world scene
understanding such as object existence, positions, relations, and OCR without
enlarging the model size or curating extra visual instruction tuning datasets.
Motion Mamba: Efficient and Long Sequence Motion Generation with
  Hierarchical and Bidirectional Selective SSM
Authors: name: Zeyu Zhang, name: Akide Liu, name: Ian Reid, name: Richard Hartley, name: Bohan Zhuang, name: Hao Tang
Published: 2024-03-12 10:25:29

Human motion generation stands as a significant pursuit in generative
computer vision, while achieving long-sequence and efficient motion generation
remains challenging. Recent advancements in state space models (SSMs), notably
Mamba, have showcased considerable promise in long sequence modeling with an
efficient hardware-aware design, which appears to be a promising direction to
build motion generation model upon it. Nevertheless, adapting SSMs to motion
generation faces hurdles since the lack of a specialized design architecture to
model motion sequence. To address these challenges, we propose Motion Mamba, a
simple and efficient approach that presents the pioneering motion generation
model utilized SSMs. Specifically, we design a Hierarchical Temporal Mamba
(HTM) block to process temporal data by ensemble varying numbers of isolated
SSM modules across a symmetric U-Net architecture aimed at preserving motion
consistency between frames. We also design a Bidirectional Spatial Mamba (BSM)
block to bidirectionally process latent poses, to enhance accurate motion
generation within a temporal frame. Our proposed method achieves up to 50% FID
improvement and up to 4 times faster on the HumanML3D and KIT-ML datasets
compared to the previous best diffusion-based method, which demonstrates strong
capabilities of high-quality long sequence motion modeling and real-time human
motion generation. See project website
https://steve-zeyu-zhang.github.io/MotionMamba/
FAX: Scalable and Differentiable Federated Primitives in JAX
Authors: name: Keith Rush, name: Zachary Charles, name: Zachary Garrett
Published: 2024-03-11 19:51:01

We present FAX, a JAX-based library designed to support large-scale
distributed and federated computations in both data center and cross-device
applications. FAX leverages JAX's sharding mechanisms to enable native
targeting of TPUs and state-of-the-art JAX runtimes, including Pathways. FAX
embeds building blocks for federated computations as primitives in JAX. This
enables three key benefits. First, FAX computations can be translated to XLA
HLO. Second, FAX provides a full implementation of federated automatic
differentiation, greatly simplifying the expression of federated computations.
Last, FAX computations can be interpreted out to existing production
cross-device federated compute systems. We show that FAX provides an easily
programmable, performant, and scalable framework for federated computations in
the data center. FAX is available at
https://github.com/google-research/google-research/tree/master/fax .
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
Authors: name: Sainbayar Sukhbaatar, name: Olga Golovneva, name: Vasu Sharma, name: Hu Xu, name: Xi Victoria Lin, name: Baptiste Rozière, name: Jacob Kahn, name: Daniel Li, name: Wen-tau Yih, name: Jason Weston, name: Xian Li
Published: 2024-03-12 16:54:58

We investigate efficient methods for training Large Language Models (LLMs) to
possess capabilities in multiple specialized domains, such as coding, math
reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts
from a seed model, which is branched to train experts in embarrassingly
parallel fashion with high throughput and reduced communication cost. After
individual experts are asynchronously trained, BTX brings together their
feedforward parameters as experts in Mixture-of-Expert (MoE) layers and
averages the remaining parameters, followed by an MoE-finetuning stage to learn
token-level routing. BTX generalizes two special cases, the Branch-Train-Merge
method, which does not have the MoE finetuning stage to learn routing, and
sparse upcycling, which omits the stage of training experts asynchronously.
Compared to alternative approaches, BTX achieves the best accuracy-efficiency
tradeoff.
Synth^2: Boosting Visual-Language Models with Synthetic Captions and
  Image Embeddings
Authors: name: Sahand Sharifzadeh, name: Christos Kaplanis, name: Shreya Pathak, name: Dharshan Kumaran, name: Anastasija Ilic, name: Jovana Mitrovic, name: Charles Blundell, name: Andrea Banino
Published: 2024-03-12 15:36:42

The creation of high-quality human-labeled image-caption datasets presents a
significant bottleneck in the development of Visual-Language Models (VLMs). We
propose a novel approach that leverages the strengths of Large Language Models
(LLMs) and image generation models to create synthetic image-text pairs for
efficient and effective VLM training. Our method employs pretraining a
text-to-image model to synthesize image embeddings starting from captions
generated by an LLM. These synthetic pairs are then used to train a VLM.
Extensive experiments demonstrate that the VLM trained with synthetic data
exhibits comparable performance on image captioning, while requiring a fraction
of the data used by models trained solely on human-annotated data. In
particular, we outperform the baseline by 17% through augmentation with a
synthetic dataset. Furthermore, we show that synthesizing in the image
embedding space is 25% faster than in the pixel space. This research introduces
a promising technique for generating large-scale, customizable image datasets,
leading to enhanced VLM performance and wider applicability across various
domains, all with improved data efficiency and resource utilization.
DragAnything: Motion Control for Anything using Entity Representation
Authors: name: Wejia Wu, name: Zhuang Li, name: Yuchao Gu, name: Rui Zhao, name: Yefei He, name: David Junhao Zhang, name: Mike Zheng Shou, name: Yan Li, name: Tingting Gao, name: Di Zhang
Published: 2024-03-12 08:57:29

We introduce DragAnything, which utilizes a entity representation to achieve
motion control for any object in controllable video generation. Comparison to
existing motion control methods, DragAnything offers several advantages.
Firstly, trajectory-based is more userfriendly for interaction, when acquiring
other guidance signals (e.g., masks, depth maps) is labor-intensive. Users only
need to draw a line (trajectory) during interaction. Secondly, our entity
representation serves as an open-domain embedding capable of representing any
object, enabling the control of motion for diverse entities, including
background. Lastly, our entity representation allows simultaneous and distinct
motion control for multiple objects. Extensive experiments demonstrate that our
DragAnything achieves state-of-the-art performance for FVD, FID, and User
Study, particularly in terms of object motion control, where our method
surpasses the previous methods (e.g., DragNUWA) by 26% in human voting.
Multistep Consistency Models
Authors: name: Jonathan Heek, name: Emiel Hoogeboom, name: Tim Salimans
Published: 2024-03-11 15:26:34

Diffusion models are relatively easy to train but require many steps to
generate samples. Consistency models are far more difficult to train, but
generate samples in a single step.
  In this paper we propose Multistep Consistency Models: A unification between
Consistency Models (Song et al., 2023) and TRACT (Berthelot et al., 2023) that
can interpolate between a consistency model and a diffusion model: a trade-off
between sampling speed and sampling quality. Specifically, a 1-step consistency
model is a conventional consistency model whereas we show that a infty-step
consistency model is a diffusion model.
  Multistep Consistency Models work really well in practice. By increasing the
sample budget from a single step to 2-8 steps, we can train models more easily
that generate higher quality samples, while retaining much of the sampling
speed benefits. Notable results are 1.4 FID on Imagenet 64 in 8 step and 2.1
FID on Imagenet128 in 8 steps with consistency distillation. We also show that
our method scales to a text-to-image diffusion model, generating samples that
are very close to the quality of the original model.
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference
  Acceleration for Large Vision-Language Models
Authors: name: Liang Chen, name: Haozhe Zhao, name: Tianyu Liu, name: Shuai Bai, name: Junyang Lin, name: Chang Zhou, name: Baobao Chang
Published: 2024-03-11 14:35:32

In this study, we identify the inefficient attention phenomena in Large
Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5,
QwenVL-Chat and Video-LLaVA. We find out that the attention computation over
visual tokens is of extreme inefficiency in the deep layers of popular LVLMs,
suggesting a need for a sparser approach compared to textual data handling. To
this end, we introduce FastV, a versatile plug-and-play method designed to
optimize computational efficiency by learning adaptive attention patterns in
early layers and pruning visual tokens in subsequent ones. Our evaluations
demonstrate FastV's ability to dramatically reduce computational costs (e.g., a
45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a
wide range of image and video understanding tasks. The computational efficiency
and performance trade-off of FastV are highly customizable and
pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve
a lower budget than that of a 7B-parameter model, while still maintaining
superior performance. We believe FastV has practical values for deployment of
LVLMs in edge devices and commercial models. Code is released at
https://github.com/pkunlp-icler/FastV.
Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
  Single GPU
Authors: name: Changyue Liao, name: Mo Sun, name: Zihan Yang, name: Kaiqi Chen, name: Binhang Yuan, name: Fei Wu, name: Zeke Wang
Published: 2024-03-11 08:25:53

Recent advances in large language models have brought immense value to the
world, with their superior capabilities stemming from the massive number of
parameters they utilize. However, even the GPUs with the highest memory
capacities, currently peaking at 80GB, are far from sufficient to accommodate
these vast parameters and their associated optimizer states when conducting
stochastic gradient descent-based optimization. One approach to hosting such
huge models is to aggregate device memory from many GPUs. However, this
approach introduces prohibitive costs for most academic researchers, who always
have a limited budget for many high-end GPU servers. In this paper, we focus on
huge model fine-tuning on a single, even low-end, GPU in a commodity server,
which is accessible to most AI researchers. In such a scenario, the
state-of-the-art work ZeRO-Infinity suffers from two severe issues when running
in a commodity server: 1) low GPU utilization due to inefficient swapping, and
2) limited trainable model size due to CPU memory capacity. The underlying
reason is that ZeRO-Infinity is optimized for running on high-end GPU servers.
To this end, we present Fuyou, a low-cost training framework that enables
efficient 100B huge model fine-tuning on a low-end server with a low-end GPU
and limited CPU memory capacity. The key idea is to add the SSD-CPU
communication as an optimization dimension and thus carefully co-optimize
computation and data swapping from a systematic approach to maximize GPU
utilization. The experimental results show that 1) Fuyou is able to fine-tune
175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while
ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model,
Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves
45 TFLOPS.
Algorithmic progress in language models
Authors: name: Anson Ho, name: Tamay Besiroglu, name: Ege Erdil, name: David Owen, name: Robi Rahman, name: Zifan Carl Guo, name: David Atkinson, name: Neil Thompson, name: Jaime Sevilla
Published: 2024-03-09 06:26:21

We investigate the rate at which algorithms for pre-training language models
have improved since the advent of deep learning. Using a dataset of over 200
language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
find that the compute required to reach a set performance threshold has halved
approximately every 8 months, with a 95% confidence interval of around 5 to 14
months, substantially faster than hardware gains per Moore's Law. We estimate
augmented scaling laws, which enable us to quantify algorithmic progress and
determine the relative contributions of scaling models versus innovations in
training algorithms. Despite the rapid pace of algorithmic progress and the
development of new architectures such as the transformer, our analysis reveals
that the increase in compute made an even larger contribution to overall
performance improvements over this time period. Though limited by noisy
benchmark data, our analysis quantifies the rapid progress in language
modeling, shedding light on the relative contributions from compute and
algorithms.