Orionnnn/hub-stats huggingface dataset:
Fast Registration of Photorealistic Avatars for VR Facial Animation
Authors: name: Chaitanya Patel, name: Shaojie Bai, name: Te-Li Wang, name: Jason Saragih, name: Shih-En Wei
Published: 2024-01-19 19:42:38

Virtual Reality (VR) bares promise of social interactions that can feel more
immersive than other media. Key to this is the ability to accurately animate a
photorealistic avatar of one's likeness while wearing a VR headset. Although
high quality registration of person-specific avatars to headset-mounted camera
(HMC) images is possible in an offline setting, the performance of generic
realtime models are significantly degraded. Online registration is also
challenging due to oblique camera views and differences in modality. In this
work, we first show that the domain gap between the avatar and headset-camera
images is one of the primary sources of difficulty, where a transformer-based
architecture achieves high accuracy on domain-consistent data, but degrades
when the domain-gap is re-introduced. Building on this finding, we develop a
system design that decouples the problem into two parts: 1) an iterative
refinement module that takes in-domain inputs, and 2) a generic avatar-guided
image-to-image style transfer module that is conditioned on current estimation
of expression and head pose. These two modules reinforce each other, as image
style transfer becomes easier when close-to-ground-truth examples are shown,
and better domain-gap removal helps registration. Our system produces
high-quality results efficiently, obviating the need for costly offline
registration to generate personalized labels. We validate the accuracy and
efficiency of our approach through extensive experiments on a commodity
headset, demonstrating significant improvements over direct regression methods
as well as offline registration.
EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
Authors: name: Koichi Namekata, name: Amirmojtaba Sabour, name: Sanja Fidler, name: Seung Wook Kim
Published: 2024-01-22 07:34:06

Diffusion models have recently received increasing research attention for
their remarkable transfer abilities in semantic segmentation tasks. However,
generating fine-grained segmentation masks with diffusion models often requires
additional training on annotated datasets, leaving it unclear to what extent
pre-trained diffusion models alone understand the semantic relations of their
generated images. To address this question, we leverage the semantic knowledge
extracted from Stable Diffusion (SD) and aim to develop an image segmentor
capable of generating fine-grained segmentation maps without any additional
training. The primary difficulty stems from the fact that semantically
meaningful feature maps typically exist only in the spatially lower-dimensional
layers, which poses a challenge in directly extracting pixel-level semantic
relations from these feature maps. To overcome this issue, our framework
identifies semantic correspondences between image pixels and spatial locations
of low-dimensional feature maps by exploiting SD's generation process and
utilizes them for constructing image-resolution segmentation maps. In extensive
experiments, the produced segmentation maps are demonstrated to be well
delineated and capture detailed parts of the images, indicating the existence
of highly accurate pixel-level semantic knowledge in diffusion models.
Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass
  Diffusion Transformers
Authors: name: Katherine Crowson, name: Stefan Andreas Baumann, name: Alex Birch, name: Tanishq Mathew Abraham, name: Daniel Z. Kaplan, name: Enrico Shippole
Published: 2024-01-21 21:49:49

We present the Hourglass Diffusion Transformer (HDiT), an image generative
model that exhibits linear scaling with pixel count, supporting training at
high-resolution (e.g. 1024 times 1024) directly in pixel-space. Building on
the Transformer architecture, which is known to scale to billions of
parameters, it bridges the gap between the efficiency of convolutional U-Nets
and the scalability of Transformers. HDiT trains successfully without typical
high-resolution training techniques such as multiscale architectures, latent
autoencoders or self-conditioning. We demonstrate that HDiT performs
competitively with existing models on ImageNet 256^2, and sets a new
state-of-the-art for diffusion models on FFHQ-1024^2.
UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with
  Authenticity Guided Textures
Authors: name: Mingyuan Zhou, name: Rakib Hyder, name: Ziwei Xuan, name: Guojun Qi
Published: 2024-01-20 01:55:17

Recent advances in 3D avatar generation have gained significant attentions.
These breakthroughs aim to produce more realistic animatable avatars, narrowing
the gap between virtual and real-world experiences. Most of existing works
employ Score Distillation Sampling (SDS) loss, combined with a differentiable
renderer and text condition, to guide a diffusion model in generating 3D
avatars. However, SDS often generates oversmoothed results with few facial
details, thereby lacking the diversity compared with ancestral sampling. On the
other hand, other works generate 3D avatar from a single image, where the
challenges of unwanted lighting effects, perspective views, and inferior image
quality make them difficult to reliably reconstruct the 3D face meshes with the
aligned complete textures. In this paper, we propose a novel 3D avatar
generation approach termed UltrAvatar with enhanced fidelity of geometry, and
superior quality of physically based rendering (PBR) textures without unwanted
lighting. To this end, the proposed approach presents a diffuse color
extraction model and an authenticity guided texture diffusion model. The former
removes the unwanted lighting effects to reveal true diffuse colors so that the
generated avatars can be rendered under various lighting conditions. The latter
follows two gradient-based guidances for generating PBR textures to render
diverse face-identity features and details better aligning with 3D mesh
geometry. We demonstrate the effectiveness and robustness of the proposed
method, outperforming the state-of-the-art methods by a large margin in the
experiments.
Single-View 3D Human Digitalization with Large Reconstruction Models
Authors: name: Zhenzhen Weng, name: Jingyuan Liu, name: Hao Tan, name: Zhan Xu, name: Yang Zhou, name: Serena Yeung-Levy, name: Jimei Yang
Published: 2024-01-22 18:08:22

In this paper, we introduce Human-LRM, a single-stage feed-forward Large
Reconstruction Model designed to predict human Neural Radiance Fields (NeRF)
from a single image. Our approach demonstrates remarkable adaptability in
training using extensive datasets containing 3D scans and multi-view capture.
Furthermore, to enhance the model's applicability for in-the-wild scenarios
especially with occlusions, we propose a novel strategy that distills
multi-view reconstruction into single-view via a conditional triplane diffusion
model. This generative extension addresses the inherent variations in human
body shapes when observed from a single view, and makes it possible to
reconstruct the full body human from an occluded image. Through extensive
experiments, we show that Human-LRM surpasses previous methods by a significant
margin on several benchmarks.
Scaling Face Interaction Graph Networks to Real World Scenes
Authors: name: Tatiana Lopez-Guevara, name: Yulia Rubanova, name: William F. Whitney, name: Tobias Pfaff, name: Kimberly Stachenfeld, name: Kelsey R. Allen
Published: 2024-01-22 14:38:25

Accurately simulating real world object dynamics is essential for various
applications such as robotics, engineering, graphics, and design. To better
capture complex real dynamics such as contact and friction, learned simulators
based on graph networks have recently shown great promise. However, applying
these learned simulators to real scenes comes with two major challenges: first,
scaling learned simulators to handle the complexity of real world scenes which
can involve hundreds of objects each with complicated 3D shapes, and second,
handling inputs from perception rather than 3D state information. Here we
introduce a method which substantially reduces the memory required to run
graph-based learned simulators. Based on this memory-efficient simulation
model, we then present a perceptual interface in the form of editable NeRFs
which can convert real-world scenes into a structured representation that can
be processed by graph network simulator. We show that our method uses
substantially less memory than previous graph-based simulators while retaining
their accuracy, and that the simulators learned in synthetic environments can
be applied to real world scenes captured from multiple camera angles. This
paves the way for expanding the application of learned simulators to settings
where only perceptual information is available at inference time.
DITTO: Diffusion Inference-Time T-Optimization for Music Generation
Authors: name: Zachary Novack, name: Julian McAuley, name: Taylor Berg-Kirkpatrick, name: Nicholas J. Bryan
Published: 2024-01-22 18:10:10

We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose
frame-work for controlling pre-trained text-to-music diffusion models at
inference-time via optimizing initial noise latents. Our method can be used to
optimize through any differentiable feature matching loss to achieve a target
(stylized) output and leverages gradient checkpointing for memory efficiency.
We demonstrate a surprisingly wide-range of applications for music generation
including inpainting, outpainting, and looping as well as intensity, melody,
and musical structure control - all without ever fine-tuning the underlying
model. When we compare our approach against related training, guidance, and
optimization-based methods, we find DITTO achieves state-of-the-art performance
on nearly all tasks, including outperforming comparable approaches on
controllability, audio quality, and computational efficiency, thus opening the
door for high-quality, flexible, training-free control of diffusion models.
Sound examples can be found at https://DITTO-Music.github.io/web/.
Make-A-Shape: a Ten-Million-scale 3D Shape Model
Authors: name: Ka-Hei Hui, name: Aditya Sanghi, name: Arianna Rampini, name: Kamal Rahimi Malekshan, name: Zhengzhe Liu, name: Hooman Shayani, name: Chi-Wing Fu
Published: 2024-01-20 00:21:58

Significant progress has been made in training large generative models for
natural language and images. Yet, the advancement of 3D generative models is
hindered by their substantial resource demands for training, along with
inefficient, non-compact, and less expressive representations. This paper
introduces Make-A-Shape, a new 3D generative model designed for efficient
training on a vast scale, capable of utilizing 10 millions publicly-available
shapes. Technical-wise, we first innovate a wavelet-tree representation to
compactly encode shapes by formulating the subband coefficient filtering scheme
to efficiently exploit coefficient relations. We then make the representation
generatable by a diffusion model by devising the subband coefficients packing
scheme to layout the representation in a low-resolution grid. Further, we
derive the subband adaptive training strategy to train our model to effectively
learn to generate coarse and detail wavelet coefficients. Last, we extend our
framework to be controlled by additional input conditions to enable it to
generate shapes from assorted modalities, e.g., single/multi-view images, point
clouds, and low-resolution voxels. In our extensive set of experiments, we
demonstrate various applications, such as unconditional generation, shape
completion, and conditional generation on a wide range of modalities. Our
approach not only surpasses the state of the art in delivering high-quality
results but also efficiently generates shapes within a few seconds, often
achieving this in just 2 seconds for most conditions.
StreamVoice: Streamable Context-Aware Language Modeling for Real-time
  Zero-Shot Voice Conversion
Authors: name: Zhichao Wang, name: Yuanzhe Chen, name: Xinsheng Wang, name: Zhuo Chen, name: Lei Xie, name: Yuping Wang, name: Yuxuan Wang
Published: 2024-01-19 23:05:05

Recent language model (LM) advancements have showcased impressive zero-shot
voice conversion (VC) performance. However, existing LM-based VC models usually
apply offline conversion from source semantics to acoustic features, demanding
the complete source speech, and limiting their deployment to real-time
applications. In this paper, we introduce StreamVoice, a novel streaming
LM-based model for zero-shot VC, facilitating real-time conversion given
arbitrary speaker prompts and source speech. Specifically, to enable streaming
capability, StreamVoice employs a fully causal context-aware LM with a
temporal-independent acoustic predictor, while alternately processing semantic
and acoustic features at each time step of autoregression which eliminates the
dependence on complete source speech. To address the potential performance
degradation from the incomplete context in streaming processing, we enhance the
context-awareness of the LM through two strategies: 1) teacher-guided context
foresight, using a teacher model to summarize the present and future semantic
context during training to guide the model's forecasting for missing context;
2) semantic masking strategy, promoting acoustic prediction from preceding
corrupted semantic and acoustic input, enhancing context-learning ability.
Notably, StreamVoice is the first LM-based streaming zero-shot VC model without
any future look-ahead. Experimental results demonstrate StreamVoice's streaming
conversion capability while maintaining zero-shot performance comparable to
non-streaming VC systems.
CheXagent: Towards a Foundation Model for Chest X-Ray Interpretation
Authors: name: Zhihong Chen, name: Maya Varma, name: Jean-Benoit Delbrouck, name: Magdalini Paschali, name: Louis Blankemeier, name: Dave Van Veen, name: Jeya Maria Jose Valanarasu, name: Alaa Youssef, name: Joseph Paul Cohen, name: Eduardo Pontes Reis, name: Emily B. Tsai, name: Andrew Johnston, name: Cameron Olsen, name: Tanishq Mathew Abraham, name: Sergios Gatidis, name: Akshay S. Chaudhari, name: Curtis Langlotz
Published: 2024-01-22 18:51:07

Chest X-rays (CXRs) are the most frequently performed imaging test in
clinical practice. Recent advances in the development of vision-language
foundation models (FMs) give rise to the possibility of performing automated
CXR interpretation, which can assist physicians with clinical decision-making
and improve patient outcomes. However, developing FMs that can accurately
interpret CXRs is challenging due to the (1) limited availability of
large-scale vision-language datasets in the medical image domain, (2) lack of
vision and language encoders that can capture the complexities of medical data,
and (3) absence of evaluation frameworks for benchmarking the abilities of FMs
on CXR interpretation. In this work, we address these challenges by first
introducing CheXinstruct - a large-scale instruction-tuning dataset
curated from 28 publicly-available datasets. We then present CheXagent -
an instruction-tuned FM capable of analyzing and summarizing CXRs. To build
CheXagent, we design a clinical large language model (LLM) for parsing
radiology reports, a vision encoder for representing CXR images, and a network
to bridge the vision and language modalities. Finally, we introduce
CheXbench - a novel benchmark designed to systematically evaluate FMs
across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
evaluations and qualitative reviews with five expert radiologists demonstrate
that CheXagent outperforms previously-developed general- and medical-domain FMs
on CheXbench tasks. Furthermore, in an effort to improve model transparency, we
perform a fairness evaluation across factors of sex, race and age to highlight
potential performance disparities. Our project is at
https://stanford-aimi.github.io/chexagent.html.
OK-Robot: What Really Matters in Integrating Open-Knowledge Models for
  Robotics
Authors: name: Peiqi Liu, name: Yaswanth Orru, name: Chris Paxton, name: Nur Muhammad Mahi Shafiullah, name: Lerrel Pinto
Published: 2024-01-22 18:42:20

Remarkable progress has been made in recent years in the fields of vision,
language, and robotics. We now have vision models capable of recognizing
objects based on language queries, navigation systems that can effectively
control mobile systems, and grasping models that can handle a wide range of
objects. Despite these advancements, general-purpose applications of robotics
still lag behind, even though they rely on these fundamental capabilities of
recognition, navigation, and grasping. In this paper, we adopt a systems-first
approach to develop a new Open Knowledge-based robotics framework called
OK-Robot. By combining Vision-Language Models (VLMs) for object detection,
navigation primitives for movement, and grasping primitives for object
manipulation, OK-Robot offers a integrated solution for pick-and-drop
operations without requiring any training. To evaluate its performance, we run
OK-Robot in 10 real-world home environments. The results demonstrate that
OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks,
representing a new state-of-the-art in Open Vocabulary Mobile Manipulation
(OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered
environments, OK-Robot's performance increases to 82%. However, the most
important insight gained from OK-Robot is the critical role of nuanced details
when combining Open Knowledge systems like VLMs with robotic modules. Videos of
our experiments are available on our website: https://ok-robot.github.io
WARM: On the Benefits of Weight Averaged Reward Models
Authors: name: Alexandre Ramé, name: Nino Vieillard, name: Léonard Hussenot, name: Robert Dadashi, name: Geoffrey Cideron, name: Olivier Bachem, name: Johan Ferret
Published: 2024-01-22 18:27:08

Aligning large language models (LLMs) with human preferences through
reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit
failures in the reward model (RM) to achieve seemingly high rewards without
meeting the underlying objectives. We identify two primary challenges when
designing RMs to mitigate reward hacking: distribution shifts during the RL
process and inconsistencies in human preferences. As a solution, we propose
Weight Averaged Reward Models (WARM), first fine-tuning multiple RMs, then
averaging them in the weight space. This strategy follows the observation that
fine-tuned weights remain linearly mode connected when sharing the same
pre-training. By averaging weights, WARM improves efficiency compared to the
traditional ensembling of predictions, while improving reliability under
distribution shifts and robustness to preference inconsistencies. Our
experiments on summarization tasks, using best-of-N and RL methods, shows that
WARM improves the overall quality and alignment of LLM predictions; for
example, a policy RL fine-tuned with WARM has a 79.4% win rate against a policy
RL fine-tuned with a single RM.
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning
  Capabilities
Authors: name: Boyuan Chen, name: Zhuo Xu, name: Sean Kirmani, name: Brian Ichter, name: Danny Driess, name: Pete Florence, name: Dorsa Sadigh, name: Leonidas Guibas, name: Fei Xia
Published: 2024-01-22 18:01:01

Understanding and reasoning about spatial relationships is a fundamental
capability for Visual Question Answering (VQA) and robotics. While Vision
Language Models (VLM) have demonstrated remarkable performance in certain VQA
benchmarks, they still lack capabilities in 3D spatial reasoning, such as
recognizing quantitative relationships of physical objects like distances or
size differences. We hypothesize that VLMs' limited spatial reasoning
capability is due to the lack of 3D spatial knowledge in training data and aim
to solve this problem by training VLMs with Internet-scale spatial reasoning
data. To this end, we present a system to facilitate this approach. We first
develop an automatic 3D spatial VQA data generation framework that scales up to
2 billion VQA examples on 10 million real-world images. We then investigate
various factors in the training recipe, including data quality, training
pipeline, and VLM architecture. Our work features the first internet-scale 3D
spatial reasoning dataset in metric space. By training a VLM on such data, we
significantly enhance its ability on both qualitative and quantitative spatial
VQA. Finally, we demonstrate that this VLM unlocks novel downstream
applications in chain-of-thought spatial reasoning and robotics due to its
quantitative estimation capability. Project website:
https://spatial-vlm.github.io/
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated
  Text
Authors: name: Abhimanyu Hans, name: Avi Schwarzschild, name: Valeriia Cherepanova, name: Hamid Kazemi, name: Aniruddha Saha, name: Micah Goldblum, name: Jonas Geiping, name: Tom Goldstein
Published: 2024-01-22 16:09:47

Detecting text generated by modern large language models is thought to be
hard, as both LLMs and humans can exhibit a wide range of complex behaviors.
However, we find that a score based on contrasting two closely related language
models is highly accurate at separating human-generated and machine-generated
text. Based on this mechanism, we propose a novel LLM detector that only
requires simple calculations using a pair of pre-trained LLMs. The method,
called Binoculars, achieves state-of-the-art accuracy without any training
data. It is capable of spotting machine text from a range of modern LLMs
without any model-specific modifications. We comprehensively evaluate
Binoculars on a number of text sources and in varied situations. Over a wide
range of document types, Binoculars detects over 90% of generated samples from
ChatGPT (and other LLMs) at a false positive rate of 0.01%, despite not being
trained on any ChatGPT data.
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
  Benchmark
Authors: name: Ge Zhang, name: Xinrun Du, name: Bei Chen, name: Yiming Liang, name: Tongxu Luo, name: Tianyu Zheng, name: Kang Zhu, name: Yuyang Cheng, name: Chunpu Xu, name: Shuyue Guo, name: Haoran Zhang, name: Xingwei Qu, name: Junjie Wang, name: Ruibin Yuan, name: Yizhi Li, name: Zekun Wang, name: Yudong Liu, name: Yu-Hsuan Tsai, name: Fengji Zhang, name: Chenghua Lin, name: Wenhao Huang, name: Wenhu Chen, name: Jie Fu
Published: 2024-01-22 13:34:34

As the capabilities of large multimodal models (LMMs) continue to advance,
evaluating the performance of LMMs emerges as an increasing need. Additionally,
there is an even larger gap in evaluating the advanced knowledge and reasoning
abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
designed to evaluate LMMs on tasks demanding college-level subject knowledge
and deliberate reasoning in a Chinese context. CMMMU is inspired by and
strictly follows the annotation and analysis pattern of MMMU.
  CMMMU includes 12k manually collected multimodal questions from college
exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
Engineering, like its companion, MMMU. These questions span 30 subjects and
comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
tables, music sheets, and chemical structures.
  CMMMU focuses on complex perception and reasoning with domain-specific
knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
indicating a large space for improvement. CMMMU will boost the community to
build the next-generation LMMs towards expert artificial intelligence and
promote the democratization of LMMs by providing diverse language contexts.