Orionnnn/hub-stats huggingface dataset:
SpecNeRF: Gaussian Directional Encoding for Specular Reflections
Authors: name: Li Ma, name: Vasu Agrawal, name: Haithem Turki, name: Changil Kim, name: Chen Gao, name: Pedro Sander, name: Michael Zollhöfer, name: Christian Richardt
Published: 2023-12-20 15:20:25

Neural radiance fields have achieved remarkable performance in modeling the
appearance of 3D scenes. However, existing approaches still struggle with the
view-dependent appearance of glossy surfaces, especially under complex lighting
of indoor environments. Unlike existing methods, which typically assume distant
lighting like an environment map, we propose a learnable Gaussian directional
encoding to better model the view-dependent effects under near-field lighting
conditions. Importantly, our new directional encoding captures the
spatially-varying nature of near-field lighting and emulates the behavior of
prefiltered environment maps. As a result, it enables the efficient evaluation
of preconvolved specular color at any 3D location with varying roughness
coefficients. We further introduce a data-driven geometry prior that helps
alleviate the shape radiance ambiguity in reflection modeling. We show that our
Gaussian directional encoding and geometry prior significantly improve the
modeling of challenging specular reflections in neural radiance fields, which
helps decompose appearance into more physically meaningful components.
Repaint123: Fast and High-quality One Image to 3D Generation with
  Progressive Controllable 2D Repainting
Authors: name: Junwu Zhang, name: Zhenyu Tang, name: Yatian Pang, name: Xinhua Cheng, name: Peng Jin, name: Yida Wei, name: Wangbo Yu, name: Munan Ning, name: Li Yuan
Published: 2023-12-20 18:51:02

Recent one image to 3D generation methods commonly adopt Score Distillation
Sampling (SDS). Despite the impressive results, there are multiple deficiencies
including multi-view inconsistency, over-saturated and over-smoothed textures,
as well as the slow generation speed. To address these deficiencies, we present
Repaint123 to alleviate multi-view bias as well as texture degradation and
speed up the generation process. The core idea is to combine the powerful image
generation capability of the 2D diffusion model and the texture alignment
ability of the repainting strategy for generating high-quality multi-view
images with consistency. We further propose visibility-aware adaptive
repainting strength for overlap regions to enhance the generated image quality
in the repainting process. The generated high-quality and multi-view consistent
images enable the use of simple Mean Square Error (MSE) loss for fast 3D
content generation. We conduct extensive experiments and show that our method
has a superior ability to generate high-quality 3D content with multi-view
consistency and fine textures in 2 minutes from scratch. Code is at
https://github.com/junwuzhang19/repaint123.
StreamDiffusion: A Pipeline-level Solution for Real-time Interactive
  Generation
Authors: name: Akio Kodaira, name: Chenfeng Xu, name: Toshiki Hazama, name: Takanori Yoshimoto, name: Kohei Ohno, name: Shogo Mitsuhori, name: Soichi Sugano, name: Hanying Cho, name: Zhijian Liu, name: Kurt Keutzer
Published: 2023-12-19 18:18:33

We introduce StreamDiffusion, a real-time diffusion pipeline designed for
interactive image generation. Existing diffusion models are adept at creating
images from text or image prompts, yet they often fall short in real-time
interaction. This limitation becomes particularly evident in scenarios
involving continuous input, such as Metaverse, live video streaming, and
broadcasting, where high throughput is imperative. To address this, we present
a novel approach that transforms the original sequential denoising into the
batching denoising process. Stream Batch eliminates the conventional
wait-and-interact approach and enables fluid and high throughput streams. To
handle the frequency disparity between data input and model throughput, we
design a novel input-output queue for parallelizing the streaming process.
Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG),
which requires additional U-Net computation. To mitigate the redundant
computations, we propose a novel residual classifier-free guidance (RCFG)
algorithm that reduces the number of negative conditional denoising steps to
only one or even zero. Besides, we introduce a stochastic similarity
filter(SSF) to optimize power consumption. Our Stream Batch achieves around
1.5x speedup compared to the sequential denoising method at different denoising
levels. The proposed RCFG leads to speeds up to 2.05x higher than the
conventional CFG. Combining the proposed strategies and existing mature
acceleration tools makes the image-to-image generation achieve up-to 91.07fps
on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers
over 59.56x. Furthermore, our proposed StreamDiffusion also significantly
reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one
RTX4090, respectively.
Cached Transformers: Improving Transformers with Differentiable Memory
  Cache
Authors: name: Zhaoyang Zhang, name: Wenqi Shao, name: Yixiao Ge, name: Xiaogang Wang, name: Jinwei Gu, name: Ping Luo
Published: 2023-12-20 03:30:51

This work introduces a new Transformer model called Cached Transformer, which
uses Gated Recurrent Cached (GRC) attention to extend the self-attention
mechanism with a differentiable memory cache of tokens. GRC attention enables
attending to both past and current tokens, increasing the receptive field of
attention and allowing for exploring long-range dependencies. By utilizing a
recurrent gating unit to continuously update the cache, our model achieves
significant advancements in six language and vision tasks, including
language modeling, machine translation, ListOPs, image classification, object
detection, and instance segmentation. Furthermore, our approach surpasses
previous memory-based techniques in tasks such as language modeling and
displays the ability to be applied to a broader range of situations.
Mini-GPTs: Efficient Large Language Models through Contextual Pruning
Authors: name: Tim Valicenti, name: Justice Vidal, name: Ritik Patnaik
Published: 2023-12-20 00:48:13

In AI research, the optimization of Large Language Models (LLMs) remains a
significant challenge, crucial for advancing the field's practical applications
and sustainability. Building upon the foundational work of Professor Song Han's
lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via
contextual pruning. Our methodology strategically prunes the computational
architecture of traditional LLMs, like Phi-1.5, focusing on retaining core
functionalities while drastically reducing model sizes. We employ the technique
across diverse and complex datasets, including US law, Medical Q&A, Skyrim
dialogue, English-Taiwanese translation, and Economics articles. The results
underscore the efficiency and effectiveness of contextual pruning, not merely
as a theoretical concept but as a practical tool in developing domain-specific,
resource-efficient LLMs. Contextual pruning is a promising method for building
domain-specific LLMs, and this research is a building block towards future
development with more hardware compute, refined fine-tuning, and quantization.
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
Authors: name: Yixin Song, name: Zeyu Mi, name: Haotong Xie, name: Haibo Chen
Published: 2023-12-16 02:27:00

This paper introduces PowerInfer, a high-speed Large Language Model (LLM)
inference engine on a personal computer (PC) equipped with a single
consumer-grade GPU. The key underlying the design of PowerInfer is exploiting
the high locality inherent in LLM inference, characterized by a power-law
distribution in neuron activation. This distribution indicates that a small
subset of neurons, termed hot neurons, are consistently activated across
inputs, while the majority, cold neurons, vary based on specific inputs.
PowerInfer exploits such an insight to design a GPU-CPU hybrid inference
engine: hot-activated neurons are preloaded onto the GPU for fast access, while
cold-activated neurons are computed on the CPU, thus significantly reducing GPU
memory demands and CPU-GPU data transfers. PowerInfer further integrates
adaptive predictors and neuron-aware sparse operators, optimizing the
efficiency of neuron activation and computational sparsity. Evaluation shows
that PowerInfer attains an average token generation rate of 13.20 tokens/s,
with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a
single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier
server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x
while retaining model accuracy.
Generative Multimodal Models are In-Context Learners
Authors: name: Quan Sun, name: Yufeng Cui, name: Xiaosong Zhang, name: Fan Zhang, name: Qiying Yu, name: Zhengxiong Luo, name: Yueze Wang, name: Yongming Rao, name: Jingjing Liu, name: Tiejun Huang, name: Xinlong Wang
Published: 2023-12-20 18:59:58

The human ability to easily solve multimodal tasks in context (i.e., with
only a few demonstrations or simple instructions), is what current multimodal
systems have largely struggled to imitate. In this work, we demonstrate that
the task-agnostic in-context learning capabilities of large multimodal models
can be significantly enhanced by effective scaling-up. We introduce Emu2, a
generative multimodal model with 37 billion parameters, trained on large-scale
multimodal sequences with a unified autoregressive objective. Emu2 exhibits
strong multimodal in-context learning abilities, even emerging to solve tasks
that require on-the-fly reasoning, such as visual prompting and object-grounded
generation. The model sets a new record on multiple multimodal understanding
tasks in few-shot settings. When instruction-tuned to follow specific
instructions, Emu2 further achieves new state-of-the-art on challenging tasks
such as question answering benchmarks for large multimodal models and
open-ended subject-driven generation. These achievements demonstrate that Emu2
can serve as a base model and general-purpose interface for a wide range of
multimodal tasks. Code and models are publicly available to facilitate future
research.
InstructVideo: Instructing Video Diffusion Models with Human Feedback
Authors: name: Hangjie Yuan, name: Shiwei Zhang, name: Xiang Wang, name: Yujie Wei, name: Tao Feng, name: Yining Pan, name: Yingya Zhang, name: Ziwei Liu, name: Samuel Albanie, name: Dong Ni
Published: 2023-12-19 17:55:16

Diffusion models have emerged as the de facto paradigm for video generation.
However, their reliance on web-scale data of varied quality often yields
results that are visually unappealing and misaligned with the textual prompts.
To tackle this problem, we propose InstructVideo to instruct text-to-video
diffusion models with human feedback by reward fine-tuning. InstructVideo has
two key ingredients: 1) To ameliorate the cost of reward fine-tuning induced by
generating through the full DDIM sampling chain, we recast reward fine-tuning
as editing. By leveraging the diffusion process to corrupt a sampled video,
InstructVideo requires only partial inference of the DDIM sampling chain,
reducing fine-tuning cost while improving fine-tuning efficiency. 2) To
mitigate the absence of a dedicated video reward model for human preferences,
we repurpose established image reward models, e.g., HPSv2. To this end, we
propose Segmental Video Reward, a mechanism to provide reward signals based on
segmental sparse sampling, and Temporally Attenuated Reward, a method that
mitigates temporal modeling degradation during fine-tuning. Extensive
experiments, both qualitative and quantitative, validate the practicality and
efficacy of using image reward models in InstructVideo, significantly enhancing
the visual quality of generated videos without compromising generalization
capabilities. Code and models will be made publicly available.
3D-LFM: Lifting Foundation Model
Authors: name: Mosam Dabhi, name: Laszlo A. Jeni, name: Simon Lucey
Published: 2023-12-19 06:38:18

The lifting of 3D structure and camera from 2D landmarks is at the
cornerstone of the entire discipline of computer vision. Traditional methods
have been confined to specific rigid objects, such as those in
Perspective-n-Point (PnP) problems, but deep learning has expanded our
capability to reconstruct a wide range of object classes (e.g. C3PDO and PAUL)
with resilience to noise, occlusions, and perspective distortions. All these
techniques, however, have been limited by the fundamental need to establish
correspondences across the 3D training data -- significantly limiting their
utility to applications where one has an abundance of "in-correspondence" 3D
data. Our approach harnesses the inherent permutation equivariance of
transformers to manage varying number of points per 3D data instance,
withstands occlusions, and generalizes to unseen categories. We demonstrate
state of the art performance across 2D-3D lifting task benchmarks. Since our
approach can be trained across such a broad class of structures we refer to it
simply as a 3D Lifting Foundation Model (3D-LFM) -- the first of its kind.
HAAR: Text-Conditioned Generative Model of 3D Strand-based Human
  Hairstyles
Authors: name: Vanessa Sklyarova, name: Egor Zakharov, name: Otmar Hilliges, name: Michael J. Black, name: Justus Thies
Published: 2023-12-18 19:19:32

We present HAAR, a new strand-based generative model for 3D human hairstyles.
Specifically, based on textual inputs, HAAR produces 3D hairstyles that could
be used as production-level assets in modern computer graphics engines. Current
AI-based generative models take advantage of powerful 2D priors to reconstruct
3D content in the form of point clouds, meshes, or volumetric functions.
However, by using the 2D priors, they are intrinsically limited to only
recovering the visual parts. Highly occluded hair structures can not be
reconstructed with those methods, and they only model the ''outer shell'',
which is not ready to be used in physics-based rendering or simulation
pipelines. In contrast, we propose a first text-guided generative method that
uses 3D hair strands as an underlying representation. Leveraging 2D visual
question-answering (VQA) systems, we automatically annotate synthetic hair
models that are generated from a small set of artist-created hairstyles. This
allows us to train a latent diffusion model that operates in a common hairstyle
UV space. In qualitative and quantitative studies, we demonstrate the
capabilities of the proposed model and compare it to existing hairstyle
generation approaches.
TIP: Text-Driven Image Processing with Semantic and Restoration
  Instructions
Authors: name: Chenyang Qi, name: Zhengzhong Tu, name: Keren Ye, name: Mauricio Delbracio, name: Peyman Milanfar, name: Qifeng Chen, name: Hossein Talebi
Published: 2023-12-18 17:02:30

Text-driven diffusion models have become increasingly popular for various
image editing tasks, including inpainting, stylization, and object replacement.
However, it still remains an open research problem to adopt this
language-vision paradigm for more fine-level image processing tasks, such as
denoising, super-resolution, deblurring, and compression artifact removal. In
this paper, we develop TIP, a Text-driven Image Processing framework that
leverages natural language as a user-friendly interface to control the image
restoration process. We consider the capacity of text information in two
dimensions. First, we use content-related prompts to enhance the semantic
alignment, effectively alleviating identity ambiguity in the restoration
outcomes. Second, our approach is the first framework that supports fine-level
instruction through language-based quantitative specification of the
restoration strength, without the need for explicit task-specific design. In
addition, we introduce a novel fusion mechanism that augments the existing
ControlNet architecture by learning to rescale the generative prior, thereby
achieving better restoration fidelity. Our extensive experiments demonstrate
the superior restoration performance of TIP compared to the state of the arts,
alongside offering the flexibility of text-based control over the restoration
effects.
Tracking Any Object Amodally
Authors: name: Cheng-Yen Hsieh, name: Tarasha Khurana, name: Achal Dave, name: Deva Ramanan
Published: 2023-12-19 18:58:40

Amodal perception, the ability to comprehend complete object structures from
partial visibility, is a fundamental skill, even for infants. Its significance
extends to applications like autonomous driving, where a clear understanding of
heavily occluded objects is essential. However, modern detection and tracking
algorithms often overlook this critical capability, perhaps due to the
prevalence of modal annotations in most datasets. To address the scarcity of
amodal data, we introduce the TAO-Amodal benchmark, featuring 880 diverse
categories in thousands of video sequences. Our dataset includes amodal and
modal bounding boxes for visible and occluded objects, including objects that
are partially out-of-frame. To enhance amodal tracking with object permanence,
we leverage a lightweight plug-in module, the amodal expander, to transform
standard, modal trackers into amodal ones through fine-tuning on a few hundred
video sequences with data augmentation. We achieve a 3.3\% and 1.6\%
improvement on the detection and tracking of occluded objects on TAO-Amodal.
When evaluated on people, our method produces dramatic improvements of 2x
compared to state-of-the-art modal baselines.
MixRT: Mixed Neural Representations For Real-Time NeRF Rendering
Authors: name: Chaojian Li, name: Bichen Wu, name: Peter Vajda, name: Yingyan, name: Lin
Published: 2023-12-19 04:14:11

Neural Radiance Field (NeRF) has emerged as a leading technique for novel
view synthesis, owing to its impressive photorealistic reconstruction and
rendering capability. Nevertheless, achieving real-time NeRF rendering in
large-scale scenes has presented challenges, often leading to the adoption of
either intricate baked mesh representations with a substantial number of
triangles or resource-intensive ray marching in baked representations. We
challenge these conventions, observing that high-quality geometry, represented
by meshes with substantial triangles, is not necessary for achieving
photorealistic rendering quality. Consequently, we propose MixRT, a novel NeRF
representation that includes a low-quality mesh, a view-dependent displacement
map, and a compressed NeRF model. This design effectively harnesses the
capabilities of existing graphics hardware, thus enabling real-time NeRF
rendering on edge devices. Leveraging a highly-optimized WebGL-based rendering
framework, our proposed MixRT attains real-time rendering speeds on edge
devices (over 30 FPS at a resolution of 1280 x 720 on a MacBook M1 Pro laptop),
better rendering quality (0.2 PSNR higher in indoor scenes of the Unbounded-360
datasets), and a smaller storage size (less than 80% compared to
state-of-the-art methods).
FastSR-NeRF: Improving NeRF Efficiency on Consumer Devices with A Simple
  Super-Resolution Pipeline
Authors: name: Chien-Yu Lin, name: Qichen Fu, name: Thomas Merth, name: Karren Yang, name: Anurag Ranjan
Published: 2023-12-15 21:02:23

Super-resolution (SR) techniques have recently been proposed to upscale the
outputs of neural radiance fields (NeRF) and generate high-quality images with
enhanced inference speeds. However, existing NeRF+SR methods increase training
overhead by using extra input features, loss functions, and/or expensive
training procedures such as knowledge distillation. In this paper, we aim to
leverage SR for efficiency gains without costly training or architectural
changes. Specifically, we build a simple NeRF+SR pipeline that directly
combines existing modules, and we propose a lightweight augmentation technique,
random patch sampling, for training. Compared to existing NeRF+SR methods, our
pipeline mitigates the SR computing overhead and can be trained up to 23x
faster, making it feasible to run on consumer devices such as the Apple
MacBook. Experiments show our pipeline can upscale NeRF outputs by 2-4x while
maintaining high quality, increasing inference speeds by up to 18x on an NVIDIA
V100 GPU and 12.8x on an M1 Pro chip. We conclude that SR can be a simple but
effective technique for improving the efficiency of NeRF models for consumer
devices.
Customize-It-3D: High-Quality 3D Creation from A Single Image Using
  Subject-Specific Knowledge Prior
Authors: name: Nan Huang, name: Ting Zhang, name: Yuhui Yuan, name: Dong Chen, name: Shanghang Zhang
Published: 2023-12-15 19:07:51

In this paper, we present a novel two-stage approach that fully utilizes the
information provided by the reference image to establish a customized knowledge
prior for image-to-3D generation. While previous approaches primarily rely on a
general diffusion prior, which struggles to yield consistent results with the
reference image, we propose a subject-specific and multi-modal diffusion model.
This model not only aids NeRF optimization by considering the shading mode for
improved geometry but also enhances texture from the coarse results to achieve
superior refinement. Both aspects contribute to faithfully aligning the 3D
content with the subject. Extensive experiments showcase the superiority of our
method, Customize-It-3D, outperforming previous works by a substantial margin.
It produces faithful 360-degree reconstructions with impressive visual quality,
making it well-suited for various applications, including text-to-3D creation.
Jack of All Tasks, Master of Many: Designing General-purpose
  Coarse-to-Fine Vision-Language Model
Authors: name: Shraman Pramanick, name: Guangxing Han, name: Rui Hou, name: Sayan Nag, name: Ser-Nam Lim, name: Nicolas Ballas, name: Qifan Wang, name: Rama Chellappa, name: Amjad Almahairi
Published: 2023-12-19 18:53:01

The ability of large language models (LLMs) to process visual inputs has
given rise to general-purpose vision systems, unifying various vision-language
(VL) tasks by instruction tuning. However, due to the enormous diversity in
input-output formats in the vision domain, existing general-purpose models fail
to successfully integrate segmentation and multi-image inputs with coarse-level
tasks into a single framework. In this work, we introduce VistaLLM, a powerful
visual system that addresses coarse- and fine-grained VL tasks over single and
multiple input images using a unified framework. VistaLLM utilizes an
instruction-guided image tokenizer that filters global embeddings using task
descriptions to extract compressed and refined features from numerous images.
Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
represent binary segmentation masks as sequences, significantly improving over
previously used uniform sampling. To bolster the desired capability of
VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
dataset with 6.8M samples. We also address the lack of multi-image grounding
datasets by introducing a novel task, AttCoSeg (Attribute-level
Co-Segmentation), which boosts the model's reasoning and grounding capability
over multiple input images. Extensive experiments on a wide range of V- and VL
tasks demonstrate the effectiveness of VistaLLM by achieving consistent
state-of-the-art performance over strong baselines across all downstream tasks.
Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.