Orionnnn/hub-stats huggingface dataset:
Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields
Authors: name: Lily Goli, name: Cody Reading, name: Silvia Selllán, name: Alec Jacobson, name: Andrea Tagliasacchi
Published: 2023-09-06 17:44:34

Neural Radiance Fields (NeRFs) have shown promise in applications like view
synthesis and depth estimation, but learning from multiview images faces
inherent uncertainties. Current methods to quantify them are either heuristic
or computationally demanding. We introduce BayesRays, a post-hoc framework to
evaluate uncertainty in any pre-trained NeRF without modifying the training
process. Our method establishes a volumetric uncertainty field using spatial
perturbations and a Bayesian Laplace approximation. We derive our algorithm
statistically and show its superior performance in key metrics and
applications. Additional results available at: https://bayesrays.github.io.
ResFields: Residual Neural Fields for Spatiotemporal Signals
Authors: name: Marko Mihajlovic, name: Sergey Prokudin, name: Marc Pollefeys, name: Siyu Tang
Published: 2023-09-06 16:59:36

Neural fields, a category of neural networks trained to represent
high-frequency signals, have gained significant attention in recent years due
to their impressive performance in modeling complex 3D data, especially large
neural signed distance (SDFs) or radiance fields (NeRFs) via a single
multi-layer perceptron (MLP). However, despite the power and simplicity of
representing signals with an MLP, these methods still face challenges when
modeling large and complex temporal signals due to the limited capacity of
MLPs. In this paper, we propose an effective approach to address this
limitation by incorporating temporal residual layers into neural fields, dubbed
ResFields, a novel class of networks specifically designed to effectively
represent complex temporal signals. We conduct a comprehensive analysis of the
properties of ResFields and propose a matrix factorization technique to reduce
the number of trainable parameters and enhance generalization capabilities.
Importantly, our formulation seamlessly integrates with existing techniques and
consistently improves results across various challenging tasks: 2D video
approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF
reconstruction. Lastly, we demonstrate the practical utility of ResFields by
showcasing its effectiveness in capturing dynamic 3D scenes from sparse sensory
inputs of a lightweight capture system.
MyoDex: A Generalizable Prior for Dexterous Manipulation
Authors: name: Vittorio Caggiano, name: Sudeep Dasari, name: Vikash Kumar
Published: 2023-09-06 16:10:49

Human dexterity is a hallmark of motor control. Our hands can rapidly
synthesize new behaviors despite the complexity (multi-articular and
multi-joints, with 23 joints controlled by more than 40 muscles) of
musculoskeletal sensory-motor circuits. In this work, we take inspiration from
how human dexterity builds on a diversity of prior experiences, instead of
being acquired through a single task. Motivated by this observation, we set out
to develop agents that can build upon their previous experience to quickly
acquire new (previously unattainable) behaviors. Specifically, our approach
leverages multi-task learning to implicitly capture task-agnostic behavioral
priors (MyoDex) for human-like dexterity, using a physiologically realistic
human hand model - MyoHand. We demonstrate MyoDex's effectiveness in few-shot
generalization as well as positive transfer to a large repertoire of unseen
dexterous manipulation tasks. Agents leveraging MyoDex can solve approximately
3x more tasks, and 4x faster in comparison to a distillation baseline. While
prior work has synthesized single musculoskeletal control behaviors, MyoDex is
the first generalizable manipulation prior that catalyzes the learning of
dexterous physiological control across a large variety of contact-rich
behaviors. We also demonstrate the effectiveness of our paradigms beyond
musculoskeletal control towards the acquisition of dexterity in 24 DoF Adroit
Hand. Website: https://sites.google.com/view/myodex
SLiMe: Segment Like Me
Authors: name: Aliasghar Khani, name: Saeid Asgari Taghanaki, name: Aditya Sanghi, name: Ali Mahdavi Amiri, name: Ghassan Hamarneh
Published: 2023-09-06 17:39:05

Significant strides have been made using large vision-language models, like
Stable Diffusion (SD), for a variety of downstream tasks, including image
editing, image correspondence, and 3D shape generation. Inspired by these
advancements, we explore leveraging these extensive vision-language models for
segmenting images at any desired granularity using as few as one annotated
sample by proposing SLiMe. SLiMe frames this problem as an optimization task.
Specifically, given a single training image and its segmentation mask, we first
extract attention maps, including our novel "weighted accumulated
self-attention map" from the SD prior. Then, using the extracted attention
maps, the text embeddings of Stable Diffusion are optimized such that, each of
them, learn about a single segmented region from the training image. These
learned embeddings then highlight the segmented region in the attention maps,
which in turn can then be used to derive the segmentation map. This enables
SLiMe to segment any real-world image during inference with the granularity of
the segmented region in the training image, using just one example. Moreover,
leveraging additional training data when available, i.e. few-shot, improves the
performance of SLiMe. We carried out a knowledge-rich set of experiments
examining various design factors and showed that SLiMe outperforms other
existing one-shot and few-shot segmentation methods.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
  Tuning
Authors: name: Lili Yu, name: Bowen Shi, name: Ramakanth Pasunuru, name: Benjamin Muller, name: Olga Golovneva, name: Tianlu Wang, name: Arun Babu, name: Binh Tang, name: Brian Karrer, name: Shelly Sheynin, name: Candace Ross, name: Adam Polyak, name: Russell Howes, name: Vasu Sharma, name: Puxin Xu, name: Hovhannes Tamoyan, name: Oron Ashual, name: Uriel Singer, name: Shang-Wen Li, name: Susan Zhang, name: Richard James, name: Gargi Ghosh, name: Yaniv Taigman, name: Maryam Fazel-Zarandi, name: Asli Celikyilmaz, name: Luke Zettlemoyer, name: Armen Aghajanyan
Published: 2023-09-05 21:27:27

We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.
Physically Grounded Vision-Language Models for Robotic Manipulation
Authors: name: Jensen Gao, name: Bidipta Sarkar, name: Fei Xia, name: Ted Xiao, name: Jiajun Wu, name: Brian Ichter, name: Anirudha Majumdar, name: Dorsa Sadigh
Published: 2023-09-05 20:21:03

Recent advances in vision-language models (VLMs) have led to improved
performance on tasks such as visual question answering and image captioning.
Consequently, these models are now well-positioned to reason about the physical
world, particularly within domains such as robotic manipulation. However,
current VLMs are limited in their understanding of the physical concepts (e.g.,
material, fragility) of common objects, which restricts their usefulness for
robotic manipulation tasks that involve interaction and physical reasoning
about such objects. To address this limitation, we propose PhysObjects, an
object-centric dataset of 36.9K crowd-sourced and 417K automated physical
concept annotations of common household objects. We demonstrate that
fine-tuning a VLM on PhysObjects improves its understanding of physical object
concepts, by capturing human priors of these concepts from visual appearance.
We incorporate this physically-grounded VLM in an interactive framework with a
large language model-based robotic planner, and show improved planning
performance on tasks that require reasoning about physical object concepts,
compared to baselines that do not leverage physically-grounded VLMs. We
additionally illustrate the benefits of our physically-grounded VLM on a real
robot, where it improves task success rates. We release our dataset and provide
further details and visualizations of our results at
https://iliad.stanford.edu/pg-vlm/.
Doppelgangers: Learning to Disambiguate Images of Similar Structures
Authors: name: Ruojin Cai, name: Joseph Tung, name: Qianqian Wang, name: Hadar Averbuch-Elor, name: Bharath Hariharan, name: Noah Snavely
Published: 2023-09-05 17:50:36

We consider the visual disambiguation task of determining whether a pair of
visually similar images depict the same or distinct 3D surfaces (e.g., the same
or opposite sides of a symmetric building). Illusory image matches, where two
images observe distinct but visually similar 3D surfaces, can be challenging
for humans to differentiate, and can also lead 3D reconstruction algorithms to
produce erroneous results. We propose a learning-based approach to visual
disambiguation, formulating it as a binary classification task on image pairs.
To that end, we introduce a new dataset for this problem, Doppelgangers, which
includes image pairs of similar structures with ground truth labels. We also
design a network architecture that takes the spatial distribution of local
keypoints and matches as input, allowing for better reasoning about both local
and global cues. Our evaluation shows that our method can distinguish illusory
matches in difficult cases, and can be integrated into SfM pipelines to produce
correct, disambiguated 3D reconstructions. See our project page for our code,
datasets, and more results: http://doppelgangers-3d.github.io/.
AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image
  Collections
Authors: name: Yue Wu, name: Sicheng Xu, name: Jianfeng Xiang, name: Fangyun Wei, name: Qifeng Chen, name: Jiaolong Yang, name: Xin Tong
Published: 2023-09-05 12:44:57

Previous animatable 3D-aware GANs for human generation have primarily focused
on either the human head or full body. However, head-only videos are relatively
uncommon in real life, and full body generation typically does not deal with
facial expression control and still has challenges in generating high-quality
results. Towards applicable video avatars, we present an animatable 3D-aware
GAN that generates portrait images with controllable facial expression, head
pose, and shoulder movements. It is a generative model trained on unstructured
2D image collections without using 3D or video data. For the new task, we base
our method on the generative radiance manifold representation and equip it with
learnable facial and head-shoulder deformations. A dual-camera rendering and
adversarial learning scheme is proposed to improve the quality of the generated
faces, which is critical for portrait images. A pose deformation processing
network is developed to generate plausible deformations for challenging regions
such as long hair. Experiments show that our method, trained on unstructured 2D
images, can generate diverse and high-quality 3D portraits with desired control
over different properties.
Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon
  Manipulation
Authors: name: Yuanpei Chen, name: Chen Wang, name: Li Fei-Fei, name: C. Karen Liu
Published: 2023-09-02 16:55:48

Many real-world manipulation tasks consist of a series of subtasks that are
significantly different from one another. Such long-horizon, complex tasks
highlight the potential of dexterous hands, which possess adaptability and
versatility, capable of seamlessly transitioning between different modes of
functionality without the need for re-grasping or external tools. However, the
challenges arise due to the high-dimensional action space of dexterous hand and
complex compositional dynamics of the long-horizon tasks. We present Sequential
Dexterity, a general system based on reinforcement learning (RL) that chains
multiple dexterous policies for achieving long-horizon task goals. The core of
the system is a transition feasibility function that progressively finetunes
the sub-policies for enhancing chaining success rate, while also enables
autonomous policy-switching for recovery from failures and bypassing redundant
stages. Despite being trained only in simulation with a few task objects, our
system demonstrates generalization capability to novel object shapes and is
able to zero-shot transfer to a real-world robot equipped with a dexterous
hand. More details and video results could be found at
https://sequential-dexterity.github.io
One Wide Feedforward is All You Need
Authors: name: Telmo Pessoa Pires, name: António V. Lopes, name: Yannick Assogba, name: Hendra Setiawan
Published: 2023-09-04 21:30:21

The Transformer architecture has two main non-embedding components: Attention
and the Feed Forward Network (FFN). Attention captures interdependencies
between words regardless of their position, while the FFN non-linearly
transforms each input token independently. In this work we explore the role of
the FFN, and find that despite taking up a significant fraction of the model's
parameters, it is highly redundant. Concretely, we are able to substantially
reduce the number of parameters with only a modest drop in accuracy by removing
the FFN on the decoder layers and sharing a single FFN across the encoder.
Finally we scale this architecture back to its original size by increasing the
hidden dimension of the shared FFN, achieving substantial gains in both
accuracy and latency with respect to the original Transformer Big.
Gated recurrent neural networks discover attention
Authors: name: Nicolas Zucchet, name: Seijin Kobayashi, name: Yassir Akram, name: Johannes von Oswald, name: Maxime Larcher, name: Angelika Steger, name: João Sacramento
Published: 2023-09-04 19:28:54

Recent architectural developments have enabled recurrent neural networks
(RNNs) to reach and even surpass the performance of Transformers on certain
sequence modeling tasks. These modern RNNs feature a prominent design pattern:
linear recurrent layers interconnected by feedforward paths with multiplicative
gating. Here, we show how RNNs equipped with these two design elements can
exactly implement (linear) self-attention, the main building block of
Transformers. By reverse-engineering a set of trained RNNs, we find that
gradient descent in practice discovers our construction. In particular, we
examine RNNs trained to solve simple in-context learning tasks on which
Transformers are known to excel and find that gradient descent instills in our
RNNs the same attention-based in-context learning algorithm used by
Transformers. Our findings highlight the importance of multiplicative
interactions in neural networks and suggest that certain RNNs might be
unexpectedly implementing attention under the hood.
Contrastive Feature Masking Open-Vocabulary Vision Transformer
Authors: name: Dahun Kim, name: Anelia Angelova, name: Weicheng Kuo
Published: 2023-09-02 01:12:48

We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an
image-text pretraining methodology that achieves simultaneous learning of
image- and region-level representation for open-vocabulary object detection
(OVD). Our approach combines the masked autoencoder (MAE) objective into the
contrastive learning objective to improve the representation for localization
tasks. Unlike standard MAE, we perform reconstruction in the joint image-text
embedding space, rather than the pixel space as is customary with the classical
MAE method, which causes the model to better learn region-level semantics.
Moreover, we introduce Positional Embedding Dropout (PED) to address scale
variation between image-text pretraining and detection finetuning by randomly
dropping out the positional embeddings during pretraining. PED improves
detection performance and enables the use of a frozen ViT backbone as a region
classifier, preventing the forgetting of open-vocabulary knowledge during
detection finetuning. On LVIS open-vocabulary detection benchmark, CFM-ViT
achieves a state-of-the-art 33.9 APr, surpassing the best approach by 7.6
points and achieves better zero-shot detection transfer. Finally, CFM-ViT
acquires strong image-level representation, outperforming the state of the art
on 8 out of 12 metrics on zero-shot image-text retrieval benchmarks.
Hierarchical Masked 3D Diffusion Model for Video Outpainting
Authors: name: Fanda Fan, name: Chaoxu Guo, name: Litong Gong, name: Biao Wang, name: Tiezheng Ge, name: Yuning Jiang, name: Chunjie Luo, name: Jianfeng Zhan
Published: 2023-09-05 10:52:21

Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our https://fanfanda.github.io/M3DDM/.
Diffusion Generative Inverse Design
Authors: name: Marin Vlastelica, name: Tatiana López-Guevara, name: Kelsey Allen, name: Peter Battaglia, name: Arnaud Doucet, name: Kimberley Stachenfeld
Published: 2023-09-05 08:32:07

Inverse design refers to the problem of optimizing the input of an objective
function in order to enact a target outcome. For many real-world engineering
problems, the objective function takes the form of a simulator that predicts
how the system state will evolve over time, and the design challenge is to
optimize the initial conditions that lead to a target outcome. Recent
developments in learned simulation have shown that graph neural networks (GNNs)
can be used for accurate, efficient, differentiable estimation of simulator
dynamics, and support high-quality design optimization with gradient- or
sampling-based optimization procedures. However, optimizing designs from
scratch requires many expensive model queries, and these procedures exhibit
basic failures on either non-convex or high-dimensional problems.In this work,
we show how denoising diffusion models (DDMs) can be used to solve inverse
design problems efficiently and propose a particle sampling algorithm for
further improving their efficiency. We perform experiments on a number of fluid
dynamics design challenges, and find that our approach substantially reduces
the number of calls to the simulator compared to standard techniques.
ControlMat: A Controlled Generative Approach to Material Capture
Authors: name: Giuseppe Vecchio, name: Rosalie Martin, name: Arthur Roullier, name: Adrien Kaiser, name: Romain Rouffet, name: Valentin Deschaintre, name: Tamy Boubekeur
Published: 2023-09-04 16:18:49

Material reconstruction from a photograph is a key component of 3D content
creation democratization. We propose to formulate this ill-posed problem as a
controlled synthesis one, leveraging the recent progress in generative deep
networks. We present ControlMat, a method which, given a single photograph with
uncontrolled illumination as input, conditions a diffusion model to generate
plausible, tileable, high-resolution physically-based digital materials. We
carefully analyze the behavior of diffusion models for multi-channel outputs,
adapt the sampling process to fuse multi-scale information and introduce rolled
diffusion to enable both tileability and patched diffusion for high-resolution
outputs. Our generative approach further permits exploration of a variety of
materials which could correspond to the input image, mitigating the unknown
lighting conditions. We show that our approach outperforms recent inference and
latent-space-optimization methods, and carefully validate our diffusion process
design choices. Supplemental materials and additional details are available at:
https://gvecchio.com/controlmat/.
Compositional Diffusion-Based Continuous Constraint Solvers
Authors: name: Zhutian Yang, name: Jiayuan Mao, name: Yilun Du, name: Jiajun Wu, name: Joshua B. Tenenbaum, name: Tomás Lozano-Pérez, name: Leslie Pack Kaelbling
Published: 2023-09-02 15:20:36

This paper introduces an approach for learning to solve continuous constraint
satisfaction problems (CCSP) in robotic reasoning and planning. Previous
methods primarily rely on hand-engineering or learning generators for specific
constraint types and then rejecting the value assignments when other
constraints are violated. By contrast, our model, the compositional diffusion
continuous constraint solver (Diffusion-CCSP) derives global solutions to CCSPs
by representing them as factor graphs and combining the energies of diffusion
models trained to sample for individual constraint types. Diffusion-CCSP
exhibits strong generalization to novel combinations of known constraints, and
it can be integrated into a task and motion planner to devise long-horizon
plans that include actions with both discrete and continuous parameters.
Project site: https://diffusion-ccsp.github.io/