Orionnnn/hub-stats huggingface dataset:
PyNeRF: Pyramidal Neural Radiance Fields
Authors: name: Haithem Turki, name: Michael Zollhöfer, name: Christian Richardt, name: Deva Ramanan
Published: 2023-11-30 23:52:46

Neural Radiance Fields (NeRFs) can be dramatically accelerated by spatial
grid representations. However, they do not explicitly reason about scale and so
introduce aliasing artifacts when reconstructing scenes captured at different
camera distances. Mip-NeRF and its extensions propose scale-aware renderers
that project volumetric frustums rather than point samples but such approaches
rely on positional encodings that are not readily compatible with grid methods.
We propose a simple modification to grid-based models by training model heads
at different spatial grid resolutions. At render time, we simply use coarser
grids to render samples that cover larger volumes. Our method can be easily
applied to existing accelerated NeRF methods and significantly improves
rendering quality (reducing error rates by 20-90% across synthetic and
unbounded real-world scenes) while incurring minimal performance overhead (as
each model head is quick to evaluate). Compared to Mip-NeRF, we reduce error
rates by 20% while training over 60x faster.
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Authors: name: Albert Gu, name: Tri Dao
Published: 2023-12-01 18:01:34

Foundation models, now powering most of the exciting applications in deep
learning, are almost universally based on the Transformer architecture and its
core attention module. Many subquadratic-time architectures such as linear
attention, gated convolution and recurrent models, and structured state space
models (SSMs) have been developed to address Transformers' computational
inefficiency on long sequences, but they have not performed as well as
attention on important modalities such as language. We identify that a key
weakness of such models is their inability to perform content-based reasoning,
and make several improvements. First, simply letting the SSM parameters be
functions of the input addresses their weakness with discrete modalities,
allowing the model to selectively propagate or forget information along the
sequence length dimension depending on the current token. Second, even though
this change prevents the use of efficient convolutions, we design a
hardware-aware parallel algorithm in recurrent mode. We integrate these
selective SSMs into a simplified end-to-end neural network architecture without
attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5times
higher throughput than Transformers) and linear scaling in sequence length, and
its performance improves on real data up to million-length sequences. As a
general sequence model backbone, Mamba achieves state-of-the-art performance
across several modalities such as language, audio, and genomics. On language
modeling, our Mamba-3B model outperforms Transformers of the same size and
matches Transformers twice its size, both in pretraining and downstream
evaluation.
MoMask: Generative Masked Modeling of 3D Human Motions
Authors: name: Chuan Guo, name: Yuxuan Mu, name: Muhammad Gohar Javed, name: Sen Wang, name: Li Cheng
Published: 2023-11-29 19:04:10

We introduce MoMask, a novel masked modeling framework for text-driven 3D
human motion generation. In MoMask, a hierarchical quantization scheme is
employed to represent human motion as multi-layer discrete motion tokens with
high-fidelity details. Starting at the base layer, with a sequence of motion
tokens obtained by vector quantization, the residual tokens of increasing
orders are derived and stored at the subsequent layers of the hierarchy. This
is consequently followed by two distinct bidirectional transformers. For the
base-layer motion tokens, a Masked Transformer is designated to predict
randomly masked motion tokens conditioned on text input at training stage.
During generation (i.e. inference) stage, starting from an empty sequence, our
Masked Transformer iteratively fills up the missing tokens; Subsequently, a
Residual Transformer learns to progressively predict the next-layer tokens
based on the results from current layer. Extensive experiments demonstrate that
MoMask outperforms the state-of-art methods on the text-to-motion generation
task, with an FID of 0.045 (vs e.g. 0.141 of T2M-GPT) on the HumanML3D dataset,
and 0.228 (vs 0.514) on KIT-ML, respectively. MoMask can also be seamlessly
applied in related tasks without further model fine-tuning, such as text-guided
temporal inpainting.
Text-Guided 3D Face Synthesis -- From Generation to Editing
Authors: name: Yunjie Wu, name: Yapeng Meng, name: Zhipeng Hu, name: Lincheng Li, name: Haoqian Wu, name: Kun Zhou, name: Weiwei Xu, name: Xin Yu
Published: 2023-12-01 06:36:23

Text-guided 3D face synthesis has achieved remarkable results by leveraging
text-to-image (T2I) diffusion models. However, most existing works focus solely
on the direct generation, ignoring the editing, restricting them from
synthesizing customized 3D faces through iterative adjustments. In this paper,
we propose a unified text-guided framework from face generation to editing. In
the generation stage, we propose a geometry-texture decoupled generation to
mitigate the loss of geometric details caused by coupling. Besides, decoupling
enables us to utilize the generated geometry as a condition for texture
generation, yielding highly geometry-texture aligned results. We further employ
a fine-tuned texture diffusion model to enhance texture quality in both RGB and
YUV space. In the editing stage, we first employ a pre-trained diffusion model
to update facial geometry or texture based on the texts. To enable sequential
editing, we introduce a UV domain consistency preservation regularization,
preventing unintentional changes to irrelevant facial attributes. Besides, we
propose a self-guided consistency weight strategy to improve editing efficacy
while preserving consistency. Through comprehensive experiments, we showcase
our method's superiority in face synthesis. Project page:
https://faceg2e.github.io/.
DREAM: Diffusion Rectification and Estimation-Adaptive Models
Authors: name: Jinxin Zhou, name: Tianyu Ding, name: Tianyi Chen, name: Jiachen Jiang, name: Ilya Zharkov, name: Zhihui Zhu, name: Luming Liang
Published: 2023-11-30 21:44:39

We present DREAM, a novel training framework representing Diffusion
Rectification and Estimation-Adaptive Models, requiring minimal code changes
(just three lines) yet significantly enhancing the alignment of training with
sampling in diffusion models. DREAM features two components: diffusion
rectification, which adjusts training to reflect the sampling process, and
estimation adaptation, which balances perception against distortion. When
applied to image super-resolution (SR), DREAM adeptly navigates the tradeoff
between minimizing distortion and preserving high image quality. Experiments
demonstrate DREAM's superiority over standard diffusion-based SR methods,
showing a 2 to 3times  faster training convergence and a 10 to
20times reduction in necessary sampling steps to achieve comparable or
superior results. We hope DREAM will inspire a rethinking of diffusion model
training paradigms.
X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap
  Between Text-to-2D and Text-to-3D Generation
Authors: name: Yiwei Ma, name: Yijun Fan, name: Jiayi Ji, name: Haowei Wang, name: Xiaoshuai Sun, name: Guannan Jiang, name: Annan Shu, name: Rongrong Ji
Published: 2023-11-30 07:23:00

In recent times, automatic text-to-3D content creation has made significant
progress, driven by the development of pretrained 2D diffusion models. Existing
text-to-3D methods typically optimize the 3D representation to ensure that the
rendered image aligns well with the given text, as evaluated by the pretrained
2D diffusion model. Nevertheless, a substantial domain gap exists between 2D
images and 3D assets, primarily attributed to variations in camera-related
attributes and the exclusive presence of foreground objects. Consequently,
employing 2D diffusion models directly for optimizing 3D representations may
lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a
novel approach for high-quality text-to-3D content creation that effectively
bridges the gap between text-to-2D and text-to-3D synthesis. The key components
of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation
(CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically
incorporates camera information into the pretrained diffusion models by
employing camera-dependent generation for trainable parameters. This
integration enhances the alignment between the generated 3D assets and the
camera's perspective. AMA loss guides the attention map of the pretrained
diffusion model using the binary mask of the 3D object, prioritizing the
creation of the foreground object. This module ensures that the model focuses
on generating accurate and detailed foreground objects. Extensive evaluations
demonstrate the effectiveness of our proposed method compared to existing
text-to-3D approaches. Our project webpage:
https://xmuxiaoma666.github.io/Projects/X-Dreamer .
VideoBooth: Diffusion-based Video Generation with Image Prompts
Authors: name: Yuming Jiang, name: Tianxing Wu, name: Shuai Yang, name: Chenyang Si, name: Dahua Lin, name: Yu Qiao, name: Chen Change Loy, name: Ziwei Liu
Published: 2023-12-01 18:55:40

Text-driven video generation witnesses rapid progress. However, merely using
text prompts is not enough to depict the desired subject appearance that
accurately aligns with users' intents, especially for customized content
creation. In this paper, we study the task of video generation with image
prompts, which provide more accurate and direct content control beyond the text
prompts. Specifically, we propose a feed-forward framework VideoBooth, with two
dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine
manner. Coarse visual embeddings from image encoder provide high-level
encodings of image prompts, while fine visual embeddings from the proposed
attention injection module provide multi-scale and detailed encoding of image
prompts. These two complementary embeddings can faithfully capture the desired
appearance. 2) In the attention injection module at fine level, multi-scale
image prompts are fed into different cross-frame attention layers as additional
keys and values. This extra spatial information refines the details in the
first frame and then it is propagated to the remaining frames, which maintains
temporal consistency. Extensive experiments demonstrate that VideoBooth
achieves state-of-the-art performance in generating customized high-quality
videos with subjects specified in image prompts. Notably, VideoBooth is a
generalizable framework where a single model works for a wide range of image
prompts with feed-forward pass.
StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style
  Adapter
Authors: name: Gongye Liu, name: Menghan Xia, name: Yong Zhang, name: Haoxin Chen, name: Jinbo Xing, name: Xintao Wang, name: Yujiu Yang, name: Ying Shan
Published: 2023-12-01 03:53:21

Text-to-video (T2V) models have shown remarkable capabilities in generating
diverse videos. However, they struggle to produce user-desired stylized videos
due to (i) text's inherent clumsiness in expressing specific styles and (ii)
the generally degraded style fidelity. To address these challenges, we
introduce StyleCrafter, a generic method that enhances pre-trained T2V models
with a style control adapter, enabling video generation in any style by
providing a reference image. Considering the scarcity of stylized video
datasets, we propose to first train a style control adapter using style-rich
image datasets, then transfer the learned stylization ability to video
generation through a tailor-made finetuning paradigm. To promote content-style
disentanglement, we remove style descriptions from the text prompt and extract
style information solely from the reference image using a decoupling learning
strategy. Additionally, we design a scale-adaptive fusion module to balance the
influences of text-based content features and image-based style features, which
helps generalization across various text and style combinations. StyleCrafter
efficiently generates high-quality stylized videos that align with the content
of the texts and resemble the style of the reference images. Experiments
demonstrate that our approach is more flexible and efficient than existing
competitors.
HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion
  Models
Authors: name: Zhonghao Wang, name: Wei Wei, name: Yang Zhao, name: Zhisheng Xiao, name: Mark Hasegawa-Johnson, name: Humphrey Shi, name: Tingbo Hou
Published: 2023-11-30 02:33:29

This paper explores advancements in high-fidelity personalized image
generation through the utilization of pre-trained text-to-image diffusion
models. While previous approaches have made significant strides in generating
versatile scenes based on text descriptions and a few input images, challenges
persist in maintaining the subject fidelity within the generated images. In
this work, we introduce an innovative algorithm named HiFi Tuner to enhance the
appearance preservation of objects during personalized image generation. Our
proposed method employs a parameter-efficient fine-tuning framework, comprising
a denoising process and a pivotal inversion process. Key enhancements include
the utilization of mask guidance, a novel parameter regularization technique,
and the incorporation of step-wise subject representations to elevate the
sample fidelity. Additionally, we propose a reference-guided generation
approach that leverages the pivotal inversion of a reference image to mitigate
unwanted subject variations and artifacts. We further extend our method to a
novel image editing task: substituting the subject in an image through textual
manipulations. Experimental evaluations conducted on the DreamBooth dataset
using the Stable Diffusion model showcase promising results. Fine-tuning solely
on textual embeddings improves CLIP-T score by 3.6 points and improves DINO
score by 9.6 points over Textual Inversion. When fine-tuning all parameters,
HiFi Tuner improves CLIP-T score by 1.2 points and improves DINO score by 1.2
points over DreamBooth, establishing a new state of the art.
Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized
  Model Responses
Authors: name: Xiao Ma, name: Swaroop Mishra, name: Ariel Liu, name: Sophie Su, name: Jilin Chen, name: Chinmay Kulkarni, name: Heng-Tze Cheng, name: Quoc Le, name: Ed Chi
Published: 2023-12-01 18:31:28

Large language model (LLM) powered chatbots are primarily text-based today,
and impose a large interactional cognitive load, especially for exploratory or
sensemaking tasks such as planning a trip or learning about a new city. Because
the interaction is textual, users have little scaffolding in the way of
structure, informational "scent", or ability to specify high-level preferences
or goals. We introduce ExploreLLM that allows users to structure thoughts, help
explore different options, navigate through the choices and recommendations,
and to more easily steer models to generate more personalized responses. We
conduct a user study and show that users find it helpful to use ExploreLLM for
exploratory or planning tasks, because it provides a useful schema-like
structure to the task, and guides users in planning. The study also suggests
that users can more easily personalize responses with high-level preferences
with ExploreLLM. Together, ExploreLLM points to a future where users interact
with LLMs beyond the form of chatbots, and instead designed to support complex
user tasks with a tighter integration between natural language and graphical
user interfaces.
Merlin:Empowering Multimodal LLMs with Foresight Minds
Authors: name: En Yu, name: Liang Zhao, name: Yana Wei, name: Jinrong Yang, name: Dongming Wu, name: Lingyu Kong, name: Haoran Wei, name: Tiancai Wang, name: Zheng Ge, name: Xiangyu Zhang, name: Wenbing Tao
Published: 2023-11-30 17:57:34

Humans possess the remarkable ability to foresee the future to a certain
extent based on present observations, a skill we term as foresight minds.
However, this capability remains largely under explored within existing
Multimodal Large Language Models (MLLMs), hindering their capacity to learn the
fundamental principles of how things operate and the intentions behind the
observed subjects. To address this issue, we introduce the integration of
future modeling into the existing learning frameworks of MLLMs. By utilizing
the subject trajectory, a highly structured representation of a consecutive
frame sequence, as a learning objective, we aim to bridge the gap between the
past and the future. We propose two innovative methods to empower MLLMs with
foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning
(FIT), which are inspired by the modern learning paradigm of LLMs.
Specifically, FPT jointly training various tasks centered on trajectories,
enabling MLLMs to learn how to attend and predict entire trajectories from a
given initial observation. Then, FIT requires MLLMs to first predict
trajectories of related objects and then reason about potential future events
based on them. Aided by FPT and FIT, we build a novel and unified MLLM named
Merlin that supports multi-images input and analysis about potential actions of
multiple objects for the future reasoning. Experimental results show Merlin
powerful foresight minds with impressive performance on both future reasoning
and visual comprehension tasks.
Instruction-tuning Aligns LLMs to the Human Brain
Authors: name: Khai Loong Aw, name: Syrielle Montariol, name: Badr AlKhamissi, name: Martin Schrimpf, name: Antoine Bosselut
Published: 2023-12-01 13:31:02

Instruction-tuning is a widely adopted method of finetuning that enables
large language models (LLMs) to generate output that more closely resembles
human responses to natural language queries, in many cases leading to
human-level performance on diverse testbeds. However, it remains unclear
whether instruction-tuning truly makes LLMs more similar to how humans process
language. We investigate the effect of instruction-tuning on LLM-human
similarity in two ways: (1) brain alignment, the similarity of LLM internal
representations to neural activity in the human language system, and (2)
behavioral alignment, the similarity of LLM and human behavior on a reading
task. We assess 25 vanilla and instruction-tuned LLMs across three datasets
involving humans reading naturalistic stories and sentences. We discover that
instruction-tuning generally enhances brain alignment by an average of 6%, but
does not have a similar effect on behavioral alignment. To identify the factors
underlying LLM-brain alignment, we compute correlations between the brain
alignment of LLMs and various model properties, such as model size, various
problem-solving abilities, and performance on tasks requiring world knowledge
spanning various domains. Notably, we find a strong positive correlation
between brain alignment and model size (r = 0.95), as well as performance on
tasks requiring world knowledge (r = 0.81). Our results demonstrate that
instruction-tuning LLMs improves both world knowledge representations and brain
alignment, suggesting that mechanisms that encode world knowledge in LLMs also
improve representational alignment to the human brain.
Dolphins: Multimodal Language Model for Driving
Authors: name: Yingzi Ma, name: Yulong Cao, name: Jiachen Sun, name: Marco Pavone, name: Chaowei Xiao
Published: 2023-12-01 09:10:33

The quest for fully autonomous vehicles (AVs) capable of navigating complex
real-world scenarios with human-like understanding and responsiveness. In this
paper, we introduce Dolphins, a novel vision-language model architected to
imbibe human-like abilities as a conversational driving assistant. Dolphins is
adept at processing multimodal inputs comprising video (or image) data, text
instructions, and historical control signals to generate informed outputs
corresponding to the provided instructions. Building upon the open-sourced
pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's
reasoning capabilities through an innovative Grounded Chain of Thought (GCoT)
process. Then we tailored Dolphins to the driving domain by constructing
driving-specific instruction data and conducting instruction tuning. Through
the utilization of the BDD-X dataset, we designed and consolidated four
distinct AV tasks into Dolphins to foster a holistic understanding of intricate
driving scenarios. As a result, the distinctive features of Dolphins are
characterized into two dimensions: (1) the ability to provide a comprehensive
understanding of complex and long-tailed open-world driving scenarios and solve
a spectrum of AV tasks, and (2) the emergence of human-like capabilities
including gradient-free instant adaptation via in-context learning and error
recovery via reflection.
Towards Accurate Differential Diagnosis with Large Language Models
Authors: name: Daniel McDuff, name: Mike Schaekermann, name: Tao Tu, name: Anil Palepu, name: Amy Wang, name: Jake Garrison, name: Karan Singhal, name: Yash Sharma, name: Shekoofeh Azizi, name: Kavita Kulkarni, name: Le Hou, name: Yong Cheng, name: Yun Liu, name: S Sara Mahdavi, name: Sushant Prakash, name: Anupam Pathak, name: Christopher Semturs, name: Shwetak Patel, name: Dale R Webster, name: Ewa Dominowska, name: Juraj Gottweis, name: Joelle Barral, name: Katherine Chou, name: Greg S Corrado, name: Yossi Matias, name: Jake Sunshine, name: Alan Karthikesalingam, name: Vivek Natarajan
Published: 2023-11-30 19:55:51

An accurate differential diagnosis (DDx) is a cornerstone of medical care,
often reached through an iterative process of interpretation that combines
clinical history, physical examination, investigations and procedures.
Interactive interfaces powered by Large Language Models (LLMs) present new
opportunities to both assist and automate aspects of this process. In this
study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
evaluated 302 challenging, real-world medical cases sourced from the New
England Journal of Medicine (NEJM) case reports. Each case report was read by
two clinicians, who were randomized to one of two assistive conditions: either
assistance from search engines and standard medical resources, or LLM
assistance in addition to these tools. All clinicians provided a baseline,
unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
exhibited standalone performance that exceeded that of unassisted clinicians
(top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
arms, the DDx quality score was higher for clinicians assisted by our LLM
(top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
(McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
differential lists than those without its assistance. Our study suggests that
our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
accuracy in challenging cases, meriting further real-world evaluation for its
ability to empower physicians and widen patients' access to specialist-level
expertise.