Orionnnn/hub-stats huggingface dataset:
Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
Authors: name: Bichen Wu, name: Ching-Yao Chuang, name: Xiaoyan Wang, name: Yichen Jia, name: Kapil Krishnakumar, name: Tong Xiao, name: Feng Liang, name: Licheng Yu, name: Peter Vajda
Published: 2023-12-20 01:49:47

In this paper, we introduce Fairy, a minimalist yet robust adaptation of
image-editing diffusion models, enhancing them for video editing applications.
Our approach centers on the concept of anchor-based cross-frame attention, a
mechanism that implicitly propagates diffusion features across frames, ensuring
superior temporal coherence and high-fidelity synthesis. Fairy not only
addresses limitations of previous models, including memory and processing
speed. It also improves temporal consistency through a unique data augmentation
strategy. This strategy renders the model equivariant to affine transformations
in both source and target images. Remarkably efficient, Fairy generates
120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds,
outpacing prior works by at least 44x. A comprehensive user study, involving
1000 generated samples, confirms that our approach delivers superior quality,
decisively outperforming established methods.
TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Authors: name: Han Shu, name: Wenshuo Li, name: Yehui Tang, name: Yiman Zhang, name: Yihao Chen, name: Houqiang Li, name: Yunhe Wang, name: Xinghao Chen
Published: 2023-12-21 12:26:11

Recently segment anything model (SAM) has shown powerful segmentation
capability and has drawn great attention in computer vision fields. Massive
following works have developed various applications based on the pretrained SAM
and achieved impressive performance on downstream vision tasks. However, SAM
consists of heavy architectures and requires massive computational capacity,
which hinders the further application of SAM on computation constrained edge
devices. To this end, in this paper we propose a framework to obtain a tiny
segment anything model (TinySAM) while maintaining the strong zero-shot
performance. We first propose a full-stage knowledge distillation method with
online hard prompt sampling strategy to distill a lightweight student model. We
also adapt the post-training quantization to the promptable segmentation task
and further reduce the computational cost. Moreover, a hierarchical segmenting
everything strategy is proposed to accelerate the everything inference by
2times with almost no performance degradation. With all these proposed
methods, our TinySAM leads to orders of magnitude computational reduction and
pushes the envelope for efficient segment anything task. Extensive experiments
on various zero-shot transfer tasks demonstrate the significantly advantageous
performance of our TinySAM against counterpart methods. Pre-trained models and
codes will be available at https://github.com/xinghaochen/TinySAM and
https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM.
DreamTuner: Single Image is Enough for Subject-Driven Generation
Authors: name: Miao Hua, name: Jiawei Liu, name: Fei Ding, name: Wei Liu, name: Jie Wu, name: Qian He
Published: 2023-12-21 09:37:14

Diffusion-based models have demonstrated impressive capabilities for
text-to-image generation and are expected for personalized applications of
subject-driven generation, which require the generation of customized concepts
with one or a few reference images. However, existing methods based on
fine-tuning fail to balance the trade-off between subject learning and the
maintenance of the generation capabilities of pretrained models. Moreover,
other methods that utilize additional image encoders tend to lose important
details of the subject due to encoding compression. To address these
challenges, we propose DreamTurner, a novel method that injects reference
information from coarse to fine to achieve subject-driven image generation more
effectively. DreamTurner introduces a subject-encoder for coarse subject
identity preservation, where the compressed general subject features are
introduced through an attention layer before visual-text cross-attention. We
then modify the self-attention layers within pretrained text-to-image models to
self-subject-attention layers to refine the details of the target subject. The
generated image queries detailed features from both the reference image and
itself in self-subject-attention. It is worth emphasizing that
self-subject-attention is an effective, elegant, and training-free method for
maintaining the detailed features of customized subjects and can serve as a
plug-and-play solution during inference. Finally, with additional
subject-driven fine-tuning, DreamTurner achieves remarkable performance in
subject-driven image generation, which can be controlled by a text or other
conditions such as pose. For further details, please visit the project page at
https://dreamtuner-diffusion.github.io/.
DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for
  Single Image Talking Face Generation
Authors: name: Chenxu Zhang, name: Chao Wang, name: Jianfeng Zhang, name: Hongyi Xu, name: Guoxian Song, name: You Xie, name: Linjie Luo, name: Yapeng Tian, name: Xiaohu Guo, name: Jiashi Feng
Published: 2023-12-21 05:03:18

The generation of emotional talking faces from a single portrait image
remains a significant challenge. The simultaneous achievement of expressive
emotional talking and accurate lip-sync is particularly difficult, as
expressiveness is often compromised for the accuracy of lip-sync. As widely
adopted by many prior works, the LSTM network often fails to capture the
subtleties and variations of emotional expressions. To address these
challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
framework, tailored for generating diverse expressions and accurate lip-sync
concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
that generates diverse highly dynamic emotional expressions and head poses in
accordance with the audio and the referenced emotion style. Given the strong
correlation between lip motion and audio, we then refine the dynamics with
enhanced lip-sync accuracy using audio features and emotion style. To this end,
we deploy a video-to-video rendering module to transfer the expressions and lip
motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
expressiveness, lip-sync accuracy and perceptual quality.
Neural feels with neural fields: Visuo-tactile perception for in-hand
  manipulation
Authors: name: Sudharshan Suresh, name: Haozhi Qi, name: Tingfan Wu, name: Taosha Fan, name: Luis Pineda, name: Mike Lambeta, name: Jitendra Malik, name: Mrinal Kalakrishnan, name: Roberto Calandra, name: Michael Kaess, name: Joseph Ortiz, name: Mustafa Mukadam
Published: 2023-12-20 22:36:37

To achieve human-level dexterity, robots must infer spatial awareness from
multimodal sensing to reason over contact interactions. During in-hand
manipulation of novel objects, such spatial awareness involves estimating the
object's pose and shape. The status quo for in-hand perception primarily
employs vision, and restricts to tracking a priori known objects. Moreover,
visual occlusion of objects in-hand is imminent during manipulation, preventing
current systems to push beyond tasks without occlusion. We combine vision and
touch sensing on a multi-fingered hand to estimate an object's pose and shape
during in-hand manipulation. Our method, NeuralFeels, encodes object geometry
by learning a neural field online and jointly tracks it by optimizing a pose
graph problem. We study multimodal in-hand perception in simulation and the
real-world, interacting with different objects via a proprioception-driven
policy. Our experiments show final reconstruction F-scores of 81% and average
pose drifts of 4.7,mm, further reduced to 2.3,mm with known
CAD models. Additionally, we observe that under heavy visual occlusion we can
achieve up to 94% improvements in tracking compared to vision-only methods.
Our results demonstrate that touch, at the very least, refines and, at the very
best, disambiguates visual estimates during in-hand manipulation. We release
our evaluation dataset of 70 experiments, FeelSight, as a step towards
benchmarking in this domain. Our neural representation driven by multimodal
sensing can serve as a perception backbone towards advancing robot dexterity.
Videos can be found on our project website
https://suddhu.github.io/neural-feels/
VideoPoet: A Large Language Model for Zero-Shot Video Generation
Authors: name: Dan Kondratyuk, name: Lijun Yu, name: Xiuye Gu, name: José Lezama, name: Jonathan Huang, name: Rachel Hornung, name: Hartwig Adam, name: Hassan Akbari, name: Yair Alon, name: Vighnesh Birodkar, name: Yong Cheng, name: Ming-Chang Chiu, name: Josh Dillon, name: Irfan Essa, name: Agrim Gupta, name: Meera Hahn, name: Anja Hauth, name: David Hendon, name: Alonso Martinez, name: David Minnen, name: David Ross, name: Grant Schindler, name: Mikhail Sirotenko, name: Kihyuk Sohn, name: Krishna Somandepalli, name: Huisheng Wang, name: Jimmy Yan, name: Ming-Hsuan Yang, name: Xuan Yang, name: Bryan Seybold, name: Lu Jiang
Published: 2023-12-21 18:46:41

We present VideoPoet, a language model capable of synthesizing high-quality
video, with matching audio, from a large variety of conditioning signals.
VideoPoet employs a decoder-only transformer architecture that processes
multimodal inputs -- including images, videos, text, and audio. The training
protocol follows that of Large Language Models (LLMs), consisting of two
stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
incorporates a mixture of multimodal generative objectives within an
autoregressive Transformer framework. The pretrained LLM serves as a foundation
that can be adapted for a range of video generation tasks. We present empirical
results demonstrating the model's state-of-the-art capabilities in zero-shot
video generation, specifically highlighting VideoPoet's ability to generate
high-fidelity motions. Project page: http://sites.research.google/videopoet/
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed
  Diffusion Models
Authors: name: Huan Ling, name: Seung Wook Kim, name: Antonio Torralba, name: Sanja Fidler, name: Karsten Kreis
Published: 2023-12-21 11:41:02

Text-guided diffusion models have revolutionized image and video generation
and have also been successfully used for optimization-based 3D object
synthesis. Here, we instead focus on the underexplored text-to-4D setting and
synthesize dynamic, animated 3D objects using score distillation methods with
an additional temporal dimension. Compared to previous work, we pursue a novel
compositional generation-based approach, and combine text-to-image,
text-to-video, and 3D-aware multiview diffusion models to provide feedback
during 4D object optimization, thereby simultaneously enforcing temporal
consistency, high-quality visual appearance and realistic geometry. Our method,
called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with
deformation fields as 4D representation. Crucial to AYG is a novel method to
regularize the distribution of the moving 3D Gaussians and thereby stabilize
the optimization and induce motion. We also propose a motion amplification
mechanism as well as a new autoregressive synthesis scheme to generate and
combine multiple 4D sequences for longer generation. These techniques allow us
to synthesize vivid dynamic scenes, outperform previous work qualitatively and
quantitatively and achieve state-of-the-art text-to-4D performance. Due to the
Gaussian 4D representation, different 4D animations can be seamlessly combined,
as we demonstrate. AYG opens up promising avenues for animation, simulation and
digital content creation as well as synthetic data generation.
Model-Based Control with Sparse Neural Dynamics
Authors: name: Ziang Liu, name: Genggeng Zhou, name: Jeff He, name: Tobia Marcucci, name: Li Fei-Fei, name: Jiajun Wu, name: Yunzhu Li
Published: 2023-12-20 06:25:02

Learning predictive models from observations using deep neural networks
(DNNs) is a promising new approach to many real-world planning and control
problems. However, common DNNs are too unstructured for effective planning, and
current control methods typically rely on extensive sampling or local gradient
descent. In this paper, we propose a new framework for integrated model
learning and predictive control that is amenable to efficient optimization
algorithms. Specifically, we start with a ReLU neural model of the system
dynamics and, with minimal losses in prediction accuracy, we gradually sparsify
it by removing redundant neurons. This discrete sparsification process is
approximated as a continuous problem, enabling an end-to-end optimization of
both the model architecture and the weight parameters. The sparsified model is
subsequently used by a mixed-integer predictive controller, which represents
the neuron activations as binary variables and employs efficient
branch-and-bound algorithms. Our framework is applicable to a wide variety of
DNNs, from simple multilayer perceptrons to complex graph neural dynamics. It
can efficiently handle tasks involving complicated contact dynamics, such as
object pushing, compositional object sorting, and manipulation of deformable
objects. Numerical and hardware experiments show that, despite the aggressive
sparsification, our framework can deliver better closed-loop performance than
existing state-of-the-art methods.
Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
Authors: name: Saurabh Saxena, name: Junhwa Hur, name: Charles Herrmann, name: Deqing Sun, name: David J. Fleet
Published: 2023-12-20 18:27:47

While methods for monocular depth estimation have made significant strides on
standard benchmarks, zero-shot metric depth estimation remains unsolved.
Challenges include the joint modeling of indoor and outdoor scenes, which often
exhibit significantly different distributions of RGB and depth, and the
depth-scale ambiguity due to unknown camera intrinsics. Recent work has
proposed specialized multi-head architectures for jointly modeling indoor and
outdoor scenes. In contrast, we advocate a generic, task-agnostic diffusion
model, with several advancements such as log-scale depth parameterization to
enable joint modeling of indoor and outdoor scenes, conditioning on the
field-of-view (FOV) to handle scale ambiguity and synthetically augmenting FOV
during training to generalize beyond the limited camera intrinsics in training
datasets. Furthermore, by employing a more diverse training mixture than is
common, and an efficient diffusion parameterization, our method, DMD (Diffusion
for Metric Depth) achieves a 25\% reduction in relative error (REL) on
zero-shot indoor and 33\% reduction on zero-shot outdoor datasets over the
current SOTA using only a small number of denoising steps. For an overview see
https://diffusion-vision.github.io/dmd
RadEdit: stress-testing biomedical vision models via diffusion image
  editing
Authors: name: Fernando Pérez-García, name: Sam Bond-Taylor, name: Pedro P. Sanchez, name: Boris van Breugel, name: Daniel C. Castro, name: Harshita Sharma, name: Valentina Salvatelli, name: Maria T. A. Wetscherek, name: Hannah Richardson, name: Matthew P. Lungren, name: Aditya Nori, name: Javier Alvarez-Valle, name: Ozan Oktay, name: Maximilian Ilse
Published: 2023-12-20 09:27:41

Biomedical imaging datasets are often small and biased, meaning that
real-world performance of predictive models can be substantially lower than
expected from internal testing. This work proposes using generative image
editing to simulate dataset shifts and diagnose failure modes of biomedical
vision models; this can be used in advance of deployment to assess readiness,
potentially reducing cost and patient harm. Existing editing methods can
produce undesirable changes, with spurious correlations learned due to the
co-occurrence of disease and treatment interventions, limiting practical
applicability. To address this, we train a text-to-image diffusion model on
multiple chest X-ray datasets and introduce a new editing method RadEdit that
uses multiple masks, if present, to constrain changes and ensure consistency in
the edited images. We consider three types of dataset shifts: acquisition
shift, manifestation shift, and population shift, and demonstrate that our
approach can diagnose failures and quantify model robustness without additional
data collection, complementing more qualitative tools for explainable AI.
Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
  Models
Authors: name: Angela Castillo, name: Jonas Kohler, name: Juan C. Pérez, name: Juan Pablo Pérez, name: Albert Pumarola, name: Bernard Ghanem, name: Pablo Arbeláez, name: Ali Thabet
Published: 2023-12-19 17:08:48

This paper presents a comprehensive study on the role of Classifier-Free
Guidance (CFG) in text-conditioned diffusion models from the perspective of
inference efficiency. In particular, we relax the default choice of applying
CFG in all diffusion steps and instead search for efficient guidance policies.
We formulate the discovery of such policies in the differentiable Neural
Architecture Search framework. Our findings suggest that the denoising steps
proposed by CFG become increasingly aligned with simple conditional steps,
which renders the extra neural network evaluation of CFG redundant, especially
in the second half of the denoising process. Building upon this insight, we
propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
omits network evaluations when the denoising process displays convergence. Our
experiments demonstrate that AG preserves CFG's image quality while reducing
computation by 25%. Thus, AG constitutes a plug-and-play alternative to
Guidance Distillation, achieving 50% of the speed-ups of the latter while being
training-free and retaining the capacity to handle negative prompts. Finally,
we uncover further redundancies of CFG in the first half of the diffusion
process, showing that entire neural function evaluations can be replaced by
simple affine transformations of past score estimates. This method, termed
LinearAG, offers even cheaper inference at the cost of deviating from the
baseline model. Our findings provide insights into the efficiency of the
conditional denoising process that contribute to more practical and swift
deployment of text-conditioned diffusion models.
MaskINT: Video Editing via Interpolative Non-autoregressive Masked
  Transformers
Authors: name: Haoyu Ma, name: Shahin Mahdizadehaghdam, name: Bichen Wu, name: Zhipeng Fan, name: Yuchao Gu, name: Wenliang Zhao, name: Lior Shapira, name: Xiaohui Xie
Published: 2023-12-19 07:05:39

Recent advances in generative AI have significantly enhanced image and video
editing, particularly in the context of text prompt control. State-of-the-art
approaches predominantly rely on diffusion models to accomplish these tasks.
However, the computational demands of diffusion-based methods are substantial,
often necessitating large-scale paired datasets for training, and therefore
challenging the deployment in practical applications. This study addresses this
challenge by breaking down the text-based video editing process into two
separate stages. In the first stage, we leverage an existing text-to-image
diffusion model to simultaneously edit a few keyframes without additional
fine-tuning. In the second stage, we introduce an efficient model called
MaskINT, which is built on non-autoregressive masked generative transformers
and specializes in frame interpolation between the keyframes, benefiting from
structural guidance provided by intermediate frames. Our comprehensive set of
experiments illustrates the efficacy and efficiency of MaskINT when compared to
other diffusion-based methodologies. This research offers a practical solution
for text-based video editing and showcases the potential of non-autoregressive
masked generative transformers in this domain.
Splatter Image: Ultra-Fast Single-View 3D Reconstruction
Authors: name: Stanislaw Szymanowicz, name: Christian Rupprecht, name: Andrea Vedaldi
Published: 2023-12-20 16:14:58

We introduce the Splatter Image, an ultra-fast approach for monocular 3D
object reconstruction which operates at 38 FPS. Splatter Image is based on
Gaussian Splatting, which has recently brought real-time rendering, fast
training, and excellent scaling to multi-view reconstruction. For the first
time, we apply Gaussian Splatting in a monocular reconstruction setting. Our
approach is learning-based, and, at test time, reconstruction only requires the
feed-forward evaluation of a neural network. The main innovation of Splatter
Image is the surprisingly straightforward design: it uses a 2D image-to-image
network to map the input image to one 3D Gaussian per pixel. The resulting
Gaussians thus have the form of an image, the Splatter Image. We further extend
the method to incorporate more than one image as input, which we do by adding
cross-view attention. Owning to the speed of the renderer (588 FPS), we can use
a single GPU for training while generating entire images at each iteration in
order to optimize perceptual metrics like LPIPS. On standard benchmarks, we
demonstrate not only fast reconstruction but also better results than recent
and much more expensive baselines in terms of PSNR, LPIPS, and other metrics.
UniSDF: Unifying Neural Representations for High-Fidelity 3D
  Reconstruction of Complex Scenes with Reflections
Authors: name: Fangjinhua Wang, name: Marie-Julie Rakotosaona, name: Michael Niemeyer, name: Richard Szeliski, name: Marc Pollefeys, name: Federico Tombari
Published: 2023-12-20 18:59:42

Neural 3D scene representations have shown great potential for 3D
reconstruction from 2D images. However, reconstructing real-world captures of
complex scenes still remains a challenge. Existing generic 3D reconstruction
methods often struggle to represent fine geometric details and do not
adequately model reflective surfaces of large-scale scenes. Techniques that
explicitly focus on reflective surfaces can model complex and detailed
reflections by exploiting better reflection parameterizations. However, we
observe that these methods are often not robust in real unbounded scenarios
where non-reflective as well as reflective components are present. In this
work, we propose UniSDF, a general purpose 3D reconstruction method that can
reconstruct large complex scenes with reflections. We investigate both
view-based as well as reflection-based color prediction parameterization
techniques and find that explicitly blending these representations in 3D space
enables reconstruction of surfaces that are more geometrically accurate,
especially for reflective surfaces. We further combine this representation with
a multi-resolution grid backbone that is trained in a coarse-to-fine manner,
enabling faster reconstructions than prior methods. Extensive experiments on
object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF
360 and Ref-NeRF real demonstrate that our method is able to robustly
reconstruct complex large-scale scenes with fine details and reflective
surfaces. Please see our project page at
https://fangjinhuawang.github.io/UniSDF.