Orionnnn/hub-stats huggingface dataset:
Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using
  Sparse RGB Cameras
Authors: name: Hanzhang Tu, name: Ruizhi Shao, name: Xue Dong, name: Shunyuan Zheng, name: Hao Zhang, name: Lili Chen, name: Meili Wang, name: Wenyu Li, name: Siyan Ma, name: Shengping Zhang, name: Boyao Zhou, name: Yebin Liu
Published: 2024-05-23 17:59:45

In this paper, we present a low-budget and high-authenticity bidirectional
telepresence system, Tele-Aloha, targeting peer-to-peer communication
scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse
RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve
high-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms)
and robust distant communication. As the core of Tele-Aloha, we propose an
efficient novel view synthesis algorithm for upper-body. Firstly, we design a
cascaded disparity estimator for obtaining a robust geometry cue. Additionally
a neural rasterizer via Gaussian Splatting is introduced to project latent
features onto target view and to decode them into a reduced resolution.
Further, given the high-quality captured data, we leverage weighted blending
mechanism to refine the decoded image into the final resolution of 2K.
Exploiting world-leading autostereoscopic display and low-latency iris
tracking, users are able to experience a strong three-dimensional sense even
without any wearable head-mounted display device. Altogether, our telepresence
system demonstrates the sense of co-presence in real-life experiments,
inspiring the next generation of communication.
NeRF-Casting: Improved View-Dependent Appearance with Consistent
  Reflections
Authors: name: Dor Verbin, name: Pratul P. Srinivasan, name: Peter Hedman, name: Ben Mildenhall, name: Benjamin Attal, name: Richard Szeliski, name: Jonathan T. Barron
Published: 2024-05-23 17:59:57

Neural Radiance Fields (NeRFs) typically struggle to reconstruct and render
highly specular objects, whose appearance varies quickly with changes in
viewpoint. Recent works have improved NeRF's ability to render detailed
specular appearance of distant environment illumination, but are unable to
synthesize consistent reflections of closer content. Moreover, these techniques
rely on large computationally-expensive neural networks to model outgoing
radiance, which severely limits optimization and rendering speed. We address
these issues with an approach based on ray tracing: instead of querying an
expensive neural network for the outgoing view-dependent radiance at points
along each camera ray, our model casts reflection rays from these points and
traces them through the NeRF representation to render feature vectors which are
decoded into color using a small inexpensive network. We demonstrate that our
model outperforms prior methods for view synthesis of scenes containing shiny
objects, and that it is the only existing NeRF method that can synthesize
photorealistic specular appearance and reflections in real-world scenes, while
requiring comparable optimization time to current state-of-the-art view
synthesis models.
Neural Directional Encoding for Efficient and Accurate View-Dependent
  Appearance Modeling
Authors: name: Liwen Wu, name: Sai Bi, name: Zexiang Xu, name: Fujun Luan, name: Kai Zhang, name: Iliyan Georgiev, name: Kalyan Sunkavalli, name: Ravi Ramamoorthi
Published: 2024-05-23 17:56:34

Novel-view synthesis of specular objects like shiny metals or glossy paints
remains a significant challenge. Not only the glossy appearance but also global
illumination effects, including reflections of other objects in the
environment, are critical components to faithfully reproduce a scene. In this
paper, we present Neural Directional Encoding (NDE), a view-dependent
appearance encoding of neural radiance fields (NeRF) for rendering specular
objects. NDE transfers the concept of feature-grid-based spatial encoding to
the angular domain, significantly improving the ability to model high-frequency
angular signals. In contrast to previous methods that use encoding functions
with only angular input, we additionally cone-trace spatial features to obtain
a spatially varying directional encoding, which addresses the challenging
interreflection effects. Extensive experiments on both synthetic and real
datasets show that a NeRF model with NDE (1) outperforms the state of the art
on view synthesis of specular objects, and (2) works with small networks to
allow fast (real-time) inference. The project webpage and source code are
available at: https://lwwu2.github.io/nde/.
AlignGPT: Multi-modal Large Language Models with Adaptive Alignment
  Capability
Authors: name: Fei Zhao, name: Taotian Pang, name: Chunhui Li, name: Zhen Wu, name: Junjie Guo, name: Shangyu Xing, name: Xinyu Dai
Published: 2024-05-23 03:07:56

Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
the exploration of Artificial General Intelligence (AGI). The core of MLLMs
lies in their capability to achieve cross-modal alignment. To attain this goal,
current MLLMs typically follow a two-phase training paradigm: the pre-training
phase and the instruction-tuning phase. Despite their success, there are
shortcomings in the modeling of alignment capabilities within these models.
Firstly, during the pre-training phase, the model usually assumes that all
image-text pairs are uniformly aligned, but in fact the degree of alignment
between different image-text pairs is inconsistent. Secondly, the instructions
currently used for finetuning incorporate a variety of tasks, different tasks's
instructions usually require different levels of alignment capabilities, but
previous MLLMs overlook these differentiated alignment needs. To tackle these
issues, we propose a new multimodal large language model AlignGPT. In the
pre-training stage, instead of treating all image-text pairs equally, we assign
different levels of alignment capabilities to different image-text pairs. Then,
in the instruction-tuning phase, we adaptively combine these different levels
of alignment capabilities to meet the dynamic alignment needs of different
instructions. Extensive experimental results show that our model achieves
competitive performance on 12 benchmarks.
Distributed Speculative Inference of Large Language Models
Authors: name: Nadav Timor, name: Jonathan Mamou, name: Daniel Korat, name: Moshe Berchansky, name: Oren Pereg, name: Moshe Wasserblat, name: Tomer Galanti, name: Michal Gordon, name: David Harel
Published: 2024-05-23 02:14:17

Accelerating the inference of large language models (LLMs) is an important
challenge in artificial intelligence. This paper introduces distributed
speculative inference (DSI), a novel distributed inference algorithm that is
provably faster than speculative inference (SI) [leviathan2023fast,
chen2023accelerating, miao2023specinfer] and traditional autoregressive
inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
requiring no training or architectural modifications, and it preserves the
target distribution.
  Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
often do not have matching drafters that are sufficiently fast and accurate. We
show a gap: SI gets slower than non-SI when using slower or less accurate
drafters. We close this gap by proving that DSI is faster than both SI and
non-SI given any drafters. By orchestrating multiple instances of the target
and drafters, DSI is not only faster than SI but also supports LLMs that cannot
be accelerated with SI.
  Our simulations show speedups of off-the-shelf LLMs in realistic settings:
DSI is 1.29-1.92x faster than SI.
Thermodynamic Natural Gradient Descent
Authors: name: Kaelan Donatella, name: Samuel Duffield, name: Maxwell Aifer, name: Denis Melanson, name: Gavin Crooks, name: Patrick J. Coles
Published: 2024-05-22 16:47:03

Second-order training methods have better convergence properties than
gradient descent but are rarely used in practice for large-scale training due
to their computational overhead. This can be viewed as a hardware limitation
(imposed by digital computers). Here we show that natural gradient descent
(NGD), a second-order method, can have a similar computational complexity per
iteration to a first-order method, when employing appropriate hardware. We
present a new hybrid digital-analog algorithm for training neural networks that
is equivalent to NGD in a certain parameter regime but avoids prohibitively
costly linear system solves. Our algorithm exploits the thermodynamic
properties of an analog system at equilibrium, and hence requires an analog
thermodynamic computer. The training occurs in a hybrid digital-analog loop,
where the gradient and Fisher information matrix (or any other positive
semi-definite curvature matrix) are calculated at given time intervals while
the analog dynamics take place. We numerically demonstrate the superiority of
this approach over state-of-the-art digital first- and second-order training
methods on classification tasks and language model fine-tuning tasks.
Not All Language Model Features Are Linear
Authors: name: Joshua Engels, name: Isaac Liao, name: Eric J. Michaud, name: Wes Gurnee, name: Max Tegmark
Published: 2024-05-23 17:59:04

Recent work has proposed the linear representation hypothesis: that language
models perform computation by manipulating one-dimensional representations of
concepts ("features") in activation space. In contrast, we explore whether some
language model representations may be inherently multi-dimensional. We begin by
developing a rigorous definition of irreducible multi-dimensional features
based on whether they can be decomposed into either independent or
non-co-occurring lower-dimensional features. Motivated by these definitions, we
design a scalable method that uses sparse autoencoders to automatically find
multi-dimensional features in GPT-2 and Mistral 7B. These auto-discovered
features include strikingly interpretable examples, e.g. circular features
representing days of the week and months of the year. We identify tasks where
these exact circles are used to solve computational problems involving modular
arithmetic in days of the week and months of the year. Finally, we provide
evidence that these circular features are indeed the fundamental unit of
computation in these tasks with intervention experiments on Mistral 7B and
Llama 3 8B, and we find further circular representations by breaking down the
hidden states for these tasks into interpretable components.
DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale
  Synthetic Data
Authors: name: Huajian Xin, name: Daya Guo, name: Zhihong Shao, name: Zhizhou Ren, name: Qihao Zhu, name: Bo Liu, name: Chong Ruan, name: Wenda Li, name: Xiaodan Liang
Published: 2024-05-23 09:03:42

Proof assistants like Lean have revolutionized mathematical proof
verification, ensuring high accuracy and reliability. Although large language
models (LLMs) show promise in mathematical reasoning, their advancement in
formal theorem proving is hindered by a lack of training data. To address this
issue, we introduce an approach to generate extensive Lean 4 proof data derived
from high-school and undergraduate-level mathematical competition problems.
This approach involves translating natural language problems into formal
statements, filtering out low-quality statements, and generating proofs to
create synthetic data. After fine-tuning the DeepSeekMath 7B model on this
synthetic dataset, which comprises 8 million formal statements with proofs, our
model achieved whole-proof generation accuracies of 46.3% with 64 samples and
52% cumulatively on the Lean 4 miniF2F test, surpassing the baseline GPT-4 at
23.0% with 64 samples and a tree search reinforcement learning method at 41.0%.
Additionally, our model successfully proved 5 out of 148 problems in the Lean 4
Formalized International Mathematical Olympiad (FIMO) benchmark, while GPT-4
failed to prove any. These results demonstrate the potential of leveraging
large-scale synthetic data to enhance theorem-proving capabilities in LLMs.
Both the synthetic dataset and the model will be made available to facilitate
further research in this promising field.
Dense Connector for MLLMs
Authors: name: Huanjin Yao, name: Wenhao Wu, name: Taojiannan Yang, name: YuXin Song, name: Mengxi Zhang, name: Haocheng Feng, name: Yifan Sun, name: Zhiheng Li, name: Wanli Ouyang, name: Jingdong Wang
Published: 2024-05-22 16:25:03

Do we fully leverage the potential of visual encoder in Multimodal Large
Language Models (MLLMs)? The recent outstanding performance of MLLMs in
multimodal understanding has garnered broad attention from both academia and
industry. In the current MLLM rat race, the focus seems to be predominantly on
the linguistic side. We witness the rise of larger and higher-quality
instruction datasets, as well as the involvement of larger-sized LLMs. Yet,
scant attention has been directed towards the visual signals utilized by MLLMs,
often assumed to be the final high-level features extracted by a frozen visual
encoder. In this paper, we introduce the Dense Connector - a simple, effective,
and plug-and-play vision-language connector that significantly enhances
existing MLLMs by leveraging multi-layer visual features, with minimal
additional computational overhead. Furthermore, our model, trained solely on
images, showcases remarkable zero-shot capabilities in video understanding as
well. Experimental results across various vision encoders, image resolutions,
training dataset scales, varying sizes of LLMs (2.7B->70B), and diverse
architectures of MLLMs (e.g., LLaVA and Mini-Gemini) validate the versatility
and scalability of our approach, achieving state-of-the-art performance on
across 19 image and video benchmarks. We hope that this work will provide
valuable experience and serve as a basic module for future MLLM development.
Improved Distribution Matching Distillation for Fast Image Synthesis
Authors: name: Tianwei Yin, name: Michaël Gharbi, name: Taesung Park, name: Richard Zhang, name: Eli Shechtman, name: Fredo Durand, name: William T. Freeman
Published: 2024-05-23 17:59:49

Recent approaches have shown promises distilling diffusion models into
efficient one-step generators. Among them, Distribution Matching Distillation
(DMD) produces one-step generators that match their teacher in distribution,
without enforcing a one-to-one correspondence with the sampling trajectories of
their teachers. However, to ensure stable training, DMD requires an additional
regression loss computed using a large set of noise-image pairs generated by
the teacher with many steps of a deterministic sampler. This is costly for
large-scale text-to-image synthesis and limits the student's quality, tying it
too closely to the teacher's original sampling paths. We introduce DMD2, a set
of techniques that lift this limitation and improve DMD training. First, we
eliminate the regression loss and the need for expensive dataset construction.
We show that the resulting instability is due to the fake critic not estimating
the distribution of generated samples accurately and propose a two time-scale
update rule as a remedy. Second, we integrate a GAN loss into the distillation
procedure, discriminating between generated samples and real images. This lets
us train the student model on real data, mitigating the imperfect real score
estimation from the teacher model, and enhancing quality. Lastly, we modify the
training procedure to enable multi-step sampling. We identify and address the
training-inference input mismatch problem in this setting, by simulating
inference-time generator samples during training time. Taken together, our
improvements set new benchmarks in one-step image generation, with FID scores
of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the
original teacher despite a 500X reduction in inference cost. Further, we show
our approach can generate megapixel images by distilling SDXL, demonstrating
exceptional visual quality among few-step methods.
Semantica: An Adaptable Image-Conditioned Diffusion Model
Authors: name: Manoj Kumar, name: Neil Houlsby, name: Emiel Hoogeboom
Published: 2024-05-23 17:58:03

We investigate the task of adapting image generative models to different
datasets without finetuneing. To this end, we introduce Semantica, an
image-conditioned diffusion model capable of generating images based on the
semantics of a conditioning image. Semantica is trained exclusively on
web-scale image pairs, that is it receives a random image from a webpage as
conditional input and models another random image from the same webpage. Our
experiments highlight the expressivity of pretrained image encoders and
necessity of semantic-based data filtering in achieving high-quality image
generation. Once trained, it can adaptively generate new images from a dataset
by simply using images from that dataset as input. We study the transfer
properties of Semantica on ImageNet, LSUN Churches, LSUN Bedroom and SUN397.
RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance
Authors: name: Zhicheng Sun, name: Zhenhao Yang, name: Yang Jin, name: Haozhe Chi, name: Kun Xu, name: Kun Xu, name: Liwei Chen, name: Hao Jiang, name: Di Zhang, name: Yang Song, name: Kun Gai, name: Yadong Mu
Published: 2024-05-23 15:12:15

Customizing diffusion models to generate identity-preserving images from
user-provided reference images is an intriguing new problem. The prevalent
approaches typically require training on extensive domain-specific images to
achieve identity preservation, which lacks flexibility across different use
cases. To address this issue, we exploit classifier guidance, a training-free
technique that steers diffusion models using an existing classifier, for
personalized image generation. Our study shows that based on a recent rectified
flow framework, the major limitation of vanilla classifier guidance in
requiring a special classifier can be resolved with a simple fixed-point
solution, allowing flexible personalization with off-the-shelf image
discriminators. Moreover, its solving procedure proves to be stable when
anchored to a reference flow trajectory, with a convergence guarantee. The
derived method is implemented on rectified flow with different off-the-shelf
image discriminators, delivering advantageous personalization results for human
faces, live subjects, and certain objects. Code is available at
https://github.com/feifeiobama/RectifID.
Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
Authors: name: Shiqi Yang, name: Zhi Zhong, name: Mengjie Zhao, name: Shusuke Takahashi, name: Masato Ishii, name: Takashi Shibuya, name: Yuki Mitsufuji
Published: 2024-05-23 14:13:16

In recent years, with the realistic generation results and a wide range of
personalized applications, diffusion-based generative models gain huge
attention in both visual and audio generation areas. Compared to the
considerable advancements of text2image or text2audio generation, research in
audio2visual or visual2audio generation has been relatively slow. The recent
audio-visual generation methods usually resort to huge large language model or
composable diffusion models. Instead of designing another giant model for
audio-visual generation, in this paper we take a step back showing a simple and
lightweight generative transformer, which is not fully investigated in
multi-modal generation, can achieve excellent results on image2audio
generation. The transformer operates in the discrete audio and visual
Vector-Quantized GAN space, and is trained in the mask denoising manner. After
training, the classifier-free guidance could be deployed off-the-shelf
achieving better performance, without any extra training or modification. Since
the transformer model is modality symmetrical, it could also be directly
deployed for audio2image generation and co-generation. In the experiments, we
show that our simple method surpasses recent image2audio generation methods.
Generated audio samples can be found at
https://docs.google.com/presentation/d/1ZtC0SeblKkut4XJcRaDsSTuCRIXB3ypxmSi7HTY3IyQ
LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent
  Diffusion Models
Authors: name: Seyedmorteza Sadat, name: Jakob Buhmann, name: Derek Bradley, name: Otmar Hilliges, name: Romann M. Weber
Published: 2024-05-23 12:06:00

Advances in latent diffusion models (LDMs) have revolutionized
high-resolution image generation, but the design space of the autoencoder that
is central to these systems remains underexplored. In this paper, we introduce
LiteVAE, a family of autoencoders for LDMs that leverage the 2D discrete
wavelet transform to enhance scalability and computational efficiency over
standard variational autoencoders (VAEs) with no sacrifice in output quality.
We also investigate the training methodologies and the decoder architecture of
LiteVAE and propose several enhancements that improve the training dynamics and
reconstruction quality. Our base LiteVAE model matches the quality of the
established VAEs in current LDMs with a six-fold reduction in encoder
parameters, leading to faster training and lower GPU memory requirements, while
our larger model outperforms VAEs of comparable complexity across all evaluated
metrics (rFID, LPIPS, PSNR, and SSIM).
DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis
Authors: name: Yao Teng, name: Yue Wu, name: Han Shi, name: Xuefei Ning, name: Guohao Dai, name: Yu Wang, name: Zhenguo Li, name: Xihui Liu
Published: 2024-05-23 06:53:18

Diffusion models have achieved great success in image generation, with the
backbone evolving from U-Net to Vision Transformers. However, the computational
cost of Transformers is quadratic to the number of tokens, leading to
significant challenges when dealing with high-resolution images. In this work,
we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a
sequence model based on State Space Models (SSM), with the expressive power of
diffusion models for efficient high-resolution image synthesis. To address the
challenge that Mamba cannot generalize to 2D signals, we make several
architecture designs including multi-directional scans, learnable padding
tokens at the end of each row and column, and lightweight local feature
enhancement. Our DiM architecture achieves inference-time efficiency for
high-resolution images. In addition, to further improve training efficiency for
high-resolution image generation with DiM, we investigate ``weak-to-strong''
training strategy that pretrains DiM on low-resolution images (256times 256)
and then finetune it on high-resolution images (512 times 512). We further
explore training-free upsampling strategies to enable the model to generate
higher-resolution images (e.g., 1024times 1024 and 1536times 1536)
without further fine-tuning. Experiments demonstrate the effectiveness and
efficiency of our DiM.