Orionnnn/hub-stats huggingface dataset:
Clockwork Diffusion: Efficient Generation With Model-Step Distillation
Authors: name: Amirhossein Habibian, name: Amir Ghodrati, name: Noor Fathima, name: Guillaume Sautiere, name: Risheek Garrepalli, name: Fatih Porikli, name: Jens Petersen
Published: 2023-12-13 13:30:27

This work aims to improve the efficiency of text-to-image diffusion models.
While diffusion models use computationally expensive UNet-based denoising
operations in every generation step, we identify that not all operations are
equally relevant for the final output quality. In particular, we observe that
UNet layers operating on high-res feature maps are relatively sensitive to
small perturbations. In contrast, low-res feature maps influence the semantic
layout of the final image and can often be perturbed with no noticeable change
in the output. Based on this observation, we propose Clockwork Diffusion, a
method that periodically reuses computation from preceding denoising steps to
approximate low-res feature maps at one or more subsequent steps. For multiple
baselines, and for both text-to-image generation and image editing, we
demonstrate that Clockwork leads to comparable or improved perceptual scores
with drastically reduced computational complexity. As an example, for Stable
Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
CLIP change.
Distributed Inference and Fine-tuning of Large Language Models Over The
  Internet
Authors: name: Alexander Borzunov, name: Max Ryabinin, name: Artem Chumachenko, name: Dmitry Baranchuk, name: Tim Dettmers, name: Younes Belkada, name: Pavel Samygin, name: Colin Raffel
Published: 2023-12-13 18:52:49

Large language models (LLMs) are useful in many NLP tasks and become more
capable with size, with the best open-source models having over 50 billion
parameters. However, using these 50B+ models requires high-end hardware, making
them inaccessible to most researchers. In this work, we investigate methods for
cost-efficient inference and fine-tuning of LLMs, comparing local and
distributed strategies. We observe that a large enough model (50B+) can run
efficiently even on geodistributed devices in a consumer-grade network. This
could allow running LLM efficiently by pooling together idle compute resources
of multiple research groups and volunteers. We address two open problems: (1)
how to perform inference and fine-tuning reliably if any device can disconnect
abruptly and (2) how to partition LLMs between devices with uneven hardware,
joining and leaving at will. In order to do that, we develop special
fault-tolerant inference algorithms and load-balancing protocols that
automatically assign devices to maximize the total system throughput. We
showcase these algorithms in Petals - a decentralized system that runs Llama 2
(70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
interactive generation. We evaluate the performance of our system in simulated
conditions and a real-world setup spanning two continents.
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Authors: name: Bowen Wen, name: Wei Yang, name: Jan Kautz, name: Stan Birchfield
Published: 2023-12-13 18:28:09

We present FoundationPose, a unified foundation model for 6D object pose
estimation and tracking, supporting both model-based and model-free setups. Our
approach can be instantly applied at test-time to a novel object without
fine-tuning, as long as its CAD model is given, or a small number of reference
images are captured. We bridge the gap between these two setups with a neural
implicit representation that allows for effective novel view synthesis, keeping
the downstream pose estimation modules invariant under the same unified
framework. Strong generalizability is achieved via large-scale synthetic
training, aided by a large language model (LLM), a novel transformer-based
architecture, and contrastive learning formulation. Extensive evaluation on
multiple public datasets involving challenging scenarios and objects indicate
our unified approach outperforms existing methods specialized for each task by
a large margin. In addition, it even achieves comparable results to
instance-level methods despite the reduced assumptions. Project page:
https://nvlabs.github.io/FoundationPose/
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
Authors: name: Róbert Csordás, name: Piotr Piękos, name: Kazuki Irie, name: Jürgen Schmidhuber
Published: 2023-12-13 09:00:21

The costly self-attention layers in modern Transformers require memory and
compute quadratic in sequence length. Existing approximation methods usually
underperform and fail to obtain significant speedups in practice. Here we
present SwitchHead - a novel method that reduces both compute and memory
requirements and achieves wall-clock speedup, while matching the language
modeling performance of baseline Transformers with the same parameter budget.
SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output
projections and requires 4 to 8 times fewer attention matrices than standard
Transformers. Our novel attention can also be combined with MoE MLP layers,
resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is
public.
PromptBench: A Unified Library for Evaluation of Large Language Models
Authors: name: Kaijie Zhu, name: Qinlin Zhao, name: Hao Chen, name: Jindong Wang, name: Xing Xie
Published: 2023-12-13 05:58:34

The evaluation of large language models (LLMs) is crucial to assess their
performance and mitigate potential security risks. In this paper, we introduce
PromptBench, a unified library to evaluate LLMs. It consists of several key
components that are easily used and extended by researchers: prompt
construction, prompt engineering, dataset and model loading, adversarial prompt
attack, dynamic evaluation protocols, and analysis tools. PromptBench is
designed to be an open, general, and flexible codebase for research purposes
that can facilitate original study in creating new benchmarks, deploying
downstream applications, and designing new evaluation protocols. The code is
available at: https://github.com/microsoft/promptbench and will be continuously
supported.
Foundation Models in Robotics: Applications, Challenges, and the Future
Authors: name: Roya Firoozi, name: Johnathan Tucker, name: Stephen Tian, name: Anirudha Majumdar, name: Jiankai Sun, name: Weiyu Liu, name: Yuke Zhu, name: Shuran Song, name: Ashish Kapoor, name: Karol Hausman, name: Brian Ichter, name: Danny Driess, name: Jiajun Wu, name: Cewu Lu, name: Mac Schwager
Published: 2023-12-13 02:20:42

We survey applications of pretrained foundation models in robotics.
Traditional deep learning models in robotics are trained on small datasets
tailored for specific tasks, which limits their adaptability across diverse
applications. In contrast, foundation models pretrained on internet-scale data
appear to have superior generalization capabilities, and in some instances
display an emergent ability to find zero-shot solutions to problems that are
not present in the training data. Foundation models may hold the potential to
enhance various components of the robot autonomy stack, from perception to
decision-making and control. For example, large language models can generate
code or provide common sense reasoning, while vision-language models enable
open-vocabulary visual recognition. However, significant open research
challenges remain, particularly around the scarcity of robot-relevant training
data, safety guarantees and uncertainty quantification, and real-time
execution. In this survey, we study recent papers that have used or built
foundation models to solve robotics problems. We explore how foundation models
contribute to improving robot capabilities in the domains of perception,
decision-making, and control. We discuss the challenges hindering the adoption
of foundation models in robot autonomy and provide opportunities and potential
pathways for future advancements. The GitHub project corresponding to this
paper (Preliminary release. We are committed to further enhancing and updating
this work to ensure its quality and relevance) can be found here:
https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models
Interfacing Foundation Models' Embeddings
Authors: name: Xueyan Zou, name: Linjie Li, name: Jianfeng Wang, name: Jianwei Yang, name: Mingyu Ding, name: Zhengyuan Yang, name: Feng Li, name: Hao Zhang, name: Shilong Liu, name: Arul Aravinthan, name: Yong Jae Lee, name: Lijuan Wang
Published: 2023-12-12 18:58:02

We present FIND, a generalized interface for aligning foundation models'
embeddings. As shown in teaser figure, a lightweight transformer interface
without tuning any foundation model weights is enough for a unified image
(segmentation) and dataset-level (retrieval) understanding. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, etc., under the
same architecture and weights. (2) Prototypable. Different tasks are able to be
implemented through prototyping attention masks and embedding types. (3)
Extendable. The proposed interface is adaptive to new tasks, and new models.
(4) Interleavable. With the benefit of multi-task multi-modal training, the
proposed interface creates an interleaved shared embedding space. In light of
the interleaved embedding space, we introduce the FIND-Bench, which introduces
new training and evaluation annotations to the COCO dataset for interleave
segmentation and retrieval. Our approach achieves state-of-the-art performance
on FIND-Bench and competitive performance on standard retrieval and
segmentation settings. The training, evaluation, and demo code as well as the
dataset have been released at https://github.com/UX-Decoder/FIND.
FreeControl: Training-Free Spatial Control of Any Text-to-Image
  Diffusion Model with Any Condition
Authors: name: Sicheng Mo, name: Fangzhou Mu, name: Kuan Heng Lin, name: Yanli Liu, name: Bochen Guan, name: Yin Li, name: Bolei Zhou
Published: 2023-12-12 18:59:14

Recent approaches such as ControlNet offer users fine-grained spatial control
over text-to-image (T2I) diffusion models. However, auxiliary modules have to
be trained for each type of spatial condition, model architecture, and
checkpoint, putting them at odds with the diverse intents and preferences a
human designer would like to convey to the AI models during the content
creation process. In this work, we present FreeControl, a training-free
approach for controllable T2I generation that supports multiple conditions,
architectures, and checkpoints simultaneously. FreeControl designs structure
guidance to facilitate the structure alignment with a guidance image, and
appearance guidance to enable the appearance sharing between images generated
using the same seed. Extensive qualitative and quantitative experiments
demonstrate the superior performance of FreeControl across a variety of
pre-trained T2I models. In particular, FreeControl facilitates convenient
training-free control over many different architectures and checkpoints, allows
the challenging input conditions on which most of the existing training-free
methods fail, and achieves competitive synthesis quality with training-based
approaches.
CCM: Adding Conditional Controls to Text-to-Image Consistency Models
Authors: name: Jie Xiao, name: Kai Zhu, name: Han Zhang, name: Zhiheng Liu, name: Yujun Shen, name: Yu Liu, name: Xueyang Fu, name: Zheng-Jun Zha
Published: 2023-12-12 04:16:03

Consistency Models (CMs) have showed a promise in creating visual content
efficiently and with high quality. However, the way to add new conditional
controls to the pretrained CMs has not been explored. In this technical report,
we consider alternative strategies for adding ControlNet-like conditional
control to CMs and present three significant findings. 1) ControlNet trained
for diffusion models (DMs) can be directly applied to CMs for high-level
semantic controls but struggles with low-level detail and realism control. 2)
CMs serve as an independent class of generative models, based on which
ControlNet can be trained from scratch using Consistency Training proposed by
Song et al. 3) A lightweight adapter can be jointly optimized under multiple
conditions through Consistency Training, allowing for the swift transfer of
DMs-based ControlNet to CMs. We study these three solutions across various
conditional controls, including edge, depth, human pose, low-resolution image
and masked image with text-to-image latent consistency models.
COLMAP-Free 3D Gaussian Splatting
Authors: name: Yang Fu, name: Sifei Liu, name: Amey Kulkarni, name: Jan Kautz, name: Alexei A. Efros, name: Xiaolong Wang
Published: 2023-12-12 18:39:52

While neural rendering has led to impressive advances in scene reconstruction
and novel view synthesis, it relies heavily on accurately pre-computed camera
poses. To relax this constraint, multiple efforts have been made to train
Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the
implicit representations of NeRFs provide extra challenges to optimize the 3D
structure and camera poses at the same time. On the other hand, the recently
proposed 3D Gaussian Splatting provides new opportunities given its explicit
point cloud representations. This paper leverages both the explicit geometric
representation and the continuity of the input video stream to perform novel
view synthesis without any SfM preprocessing. We process the input frames in a
sequential manner and progressively grow the 3D Gaussians set by taking one
input frame at a time, without the need to pre-compute the camera poses. Our
method significantly improves over previous approaches in view synthesis and
camera pose estimation under large motion changes. Our project page is
https://oasisyang.github.io/colmap-free-3dgs
Fast Training of Diffusion Transformer with Extreme Masking for 3D Point
  Clouds Generation
Authors: name: Shentong Mo, name: Enze Xie, name: Yue Wu, name: Junsong Chen, name: Matthias Nießner, name: Zhenguo Li
Published: 2023-12-12 12:50:33

Diffusion Transformers have recently shown remarkable effectiveness in
generating high-quality 3D point clouds. However, training voxel-based
diffusion models for high-resolution 3D voxels remains prohibitively expensive
due to the cubic complexity of attention operators, which arises from the
additional dimension of voxels. Motivated by the inherent redundancy of 3D
compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
tailored for efficient 3D point cloud generation, which greatly reduces
training costs. Specifically, we draw inspiration from masked autoencoders to
dynamically operate the denoising process on masked voxelized point clouds. We
also propose a novel voxel-aware masking strategy to adaptively aggregate
background/foreground information from voxelized point clouds. Our method
achieves state-of-the-art performance with an extreme masking ratio of nearly
99%. Moreover, to improve multi-category 3D generation, we introduce
Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
distinct diffusion path with different experts, relieving gradient conflict.
Experimental results on the ShapeNet dataset demonstrate that our method
achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
metrics when generating 128-resolution voxel point clouds, using only 6.5% of
the original training cost.
How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary
  Investigation
Authors: name: Zhongyi Han, name: Guanglin Zhou, name: Rundong He, name: Jindong Wang, name: Xing Xie, name: Tailin Wu, name: Yilong Yin, name: Salman Khan, name: Lina Yao, name: Tongliang Liu, name: Kun Zhang
Published: 2023-12-12 16:48:07

In machine learning, generalization against distribution shifts -- where
deployment conditions diverge from the training scenarios -- is crucial,
particularly in fields like climate modeling, biomedicine, and autonomous
driving. The emergence of foundation models, distinguished by their extensive
pretraining and task versatility, has led to an increased interest in their
adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced
publicly accessible multimodal foundation model, with extensive applications
across various domains, including anomaly detection, video understanding, image
generation, and medical diagnosis. However, its robustness against data
distributions remains largely underexplored. Addressing this gap, this study
rigorously evaluates GPT-4V's adaptability and generalization capabilities in
dynamic environments, benchmarking against prominent models like CLIP and
LLaVA. We delve into GPT-4V's zero-shot generalization across 13 diverse
datasets spanning natural, medical, and molecular domains. We further
investigate its adaptability to controlled data perturbations and examine the
efficacy of in-context learning as a tool to enhance its adaptation. Our
findings delineate GPT-4V's capability boundaries in distribution shifts,
shedding light on its strengths and limitations across various scenarios.
Importantly, this investigation contributes to our understanding of how AI
foundation models generalize to distribution shifts, offering pivotal insights
into their adaptability and robustness. Code is publicly available at
https://github.com/jameszhou-gl/gpt-4v-distribution-shift.
DiffMorpher: Unleashing the Capability of Diffusion Models for Image
  Morphing
Authors: name: Kaiwen Zhang, name: Yifan Zhou, name: Xudong Xu, name: Xingang Pan, name: Bo Dai
Published: 2023-12-12 16:28:08

Diffusion models have achieved remarkable image generation quality surpassing
previous generative models. However, a notable limitation of diffusion models,
in comparison to GANs, is their difficulty in smoothly interpolating between
two image samples, due to their highly unstructured latent space. Such a smooth
interpolation is intriguing as it naturally serves as a solution for the image
morphing task with many applications. In this work, we present DiffMorpher, the
first approach enabling smooth and natural image interpolation using diffusion
models. Our key idea is to capture the semantics of the two images by fitting
two LoRAs to them respectively, and interpolate between both the LoRA
parameters and the latent noises to ensure a smooth semantic transition, where
correspondence automatically emerges without the need for annotation. In
addition, we propose an attention interpolation and injection technique and a
new sampling schedule to further enhance the smoothness between consecutive
images. Extensive experiments demonstrate that DiffMorpher achieves starkly
better image morphing effects than previous methods across a variety of object
categories, bridging a critical functional gap that distinguished diffusion
models from GANs.
FreeInit: Bridging Initialization Gap in Video Diffusion Models
Authors: name: Tianxing Wu, name: Chenyang Si, name: Yuming Jiang, name: Ziqi Huang, name: Ziwei Liu
Published: 2023-12-12 18:59:16

Though diffusion-based video generation has witnessed rapid progress, the
inference results of existing models still exhibit unsatisfactory temporal
consistency and unnatural dynamics. In this paper, we delve deep into the noise
initialization of video diffusion models, and discover an implicit
training-inference gap that attributes to the unsatisfactory inference quality.
Our key findings are: 1) the spatial-temporal frequency distribution of the
initial latent at inference is intrinsically different from that for training,
and 2) the denoising process is significantly influenced by the low-frequency
components of the initial noise. Motivated by these observations, we propose a
concise yet effective inference sampling strategy, FreeInit, which
significantly improves temporal consistency of videos generated by diffusion
models. Through iteratively refining the spatial-temporal low-frequency
components of the initial latent during inference, FreeInit is able to
compensate the initialization gap between training and inference, thus
effectively improving the subject appearance and temporal consistency of
generation results. Extensive experiments demonstrate that FreeInit
consistently enhances the generation results of various text-to-video
generation models without additional training.
PEEKABOO: Interactive Video Generation via Masked-Diffusion
Authors: name: Yash Jain, name: Anshul Nasery, name: Vibhav Vineet, name: Harkirat Behl
Published: 2023-12-12 18:43:05

Recently there has been a lot of progress in text-to-video generation, with
state-of-the-art models being capable of generating high quality, realistic
videos. However, these models lack the capability for users to interactively
control and generate videos, which can potentially unlock new areas of
application. As a first step towards this goal, we tackle the problem of
endowing diffusion-based video generation models with interactive
spatio-temporal control over their output. To this end, we take inspiration
from the recent advances in segmentation literature to propose a novel
spatio-temporal masked attention module - Peekaboo. This module is a
training-free, no-inference-overhead addition to off-the-shelf video generation
models which enables spatio-temporal control. We also propose an evaluation
benchmark for the interactive video generation task. Through extensive
qualitative and quantitative evaluation, we establish that Peekaboo enables
control video generation and even obtains a gain of upto 3.8x in mIoU over
baseline models.
VILA: On Pre-training for Visual Language Models
Authors: name: Ji Lin, name: Hongxu Yin, name: Wei Ping, name: Yao Lu, name: Pavlo Molchanov, name: Andrew Tao, name: Huizi Mao, name: Jan Kautz, name: Mohammad Shoeybi, name: Song Han
Published: 2023-12-12 18:58:18

Visual language models (VLMs) rapidly progressed with the recent success of
large language models. There have been growing efforts on visual instruction
tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
visual language pre-training process, where the model learns to perform joint
modeling on both modalities. In this work, we examine the design options for
VLM pre-training by augmenting LLM towards VLM through step-by-step
controllable comparisons. We introduce three main findings: (1) freezing LLMs
during pre-training can achieve decent zero-shot performance, but lack
in-context learning capability, which requires unfreezing the LLM; (2)
interleaved pre-training data is beneficial whereas image-text pairs alone are
not optimal; (3) re-blending text-only instruction data to image-text data
during instruction fine-tuning not only remedies the degradation of text-only
tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
we build VILA, a Visual Language model family that consistently outperforms the
state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
and whistles. Multi-modal pre-training also helps unveil appealing properties
of VILA, including multi-image reasoning, enhanced in-context learning, and
better world knowledge.