Orionnnn/hub-stats huggingface dataset:
On Speculative Decoding for Multimodal Large Language Models
Authors: name: Mukul Gagrani, name: Raghavv Goel, name: Wonseok Jeon, name: Junyoung Park, name: Mingu Lee, name: Christopher Lott
Published: 2024-04-13 00:02:36

Inference with Multimodal Large Language Models (MLLMs) is slow due to their
large-language-model backbone which suffers from memory bandwidth bottleneck
and generates tokens auto-regressively. In this paper, we explore the
application of speculative decoding to enhance the inference efficiency of
MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
the need for image tokens and their associated processing components from the
draft model. Our experiments across three different tasks show that speculative
decoding can achieve a memory-bound speedup of up to 2.37times using a 115M
parameter language model that we trained from scratch. Additionally, we
introduce a compact LLaVA draft model incorporating an image adapter, which
shows marginal performance gains in image captioning while maintaining
comparable results in other tasks.
CompGS: Efficient 3D Scene Representation via Compressed Gaussian
  Splatting
Authors: name: Xiangrui Liu, name: Xinju Wu, name: Pingping Zhang, name: Shiqi Wang, name: Zhu Li, name: Sam Kwong
Published: 2024-04-15 04:50:39

Gaussian splatting, renowned for its exceptional rendering quality and
efficiency, has emerged as a prominent technique in 3D scene representation.
However, the substantial data volume of Gaussian splatting impedes its
practical utility in real-world applications. Herein, we propose an efficient
3D scene representation, named Compressed Gaussian Splatting (CompGS), which
harnesses compact Gaussian primitives for faithful 3D scene modeling with a
remarkably reduced data size. To ensure the compactness of Gaussian primitives,
we devise a hybrid primitive structure that captures predictive relationships
between each other. Then, we exploit a small set of anchor primitives for
prediction, allowing the majority of primitives to be encapsulated into highly
compact residual forms. Moreover, we develop a rate-constrained optimization
scheme to eliminate redundancies within such hybrid primitives, steering our
CompGS towards an optimal trade-off between bitrate consumption and
representation efficacy. Experimental results show that the proposed CompGS
significantly outperforms existing methods, achieving superior compactness in
3D scene representation without compromising model accuracy and rendering
quality. Our code will be released on GitHub for further research.
Compression Represents Intelligence Linearly
Authors: name: Yuzhen Huang, name: Jinghan Zhang, name: Zifei Shan, name: Junxian He
Published: 2024-04-15 17:03:41

There is a belief that learning to compress well will lead to intelligence.
Recently, language modeling has been shown to be equivalent to compression,
which offers a compelling rationale for the success of large language models
(LLMs): the development of more advanced language models is essentially
enhancing compression which facilitates intelligence. Despite such appealing
discussions, little empirical evidence is present for the interplay between
compression and intelligence. In this work, we examine their relationship in
the context of LLMs, treating LLMs as data compressors. Given the abstract
concept of "intelligence", we adopt the average downstream benchmark scores as
a surrogate, specifically targeting intelligence related to knowledge and
commonsense, coding, and mathematical reasoning. Across 12 benchmarks, our
study brings together 30 public LLMs that originate from diverse organizations.
Remarkably, we find that LLMs' intelligence -- reflected by average benchmark
scores -- almost linearly correlates with their ability to compress external
text corpora. These results provide concrete evidence supporting the belief
that superior compression indicates greater intelligence. Furthermore, our
findings suggest that compression efficiency, as an unsupervised metric derived
from raw text corpora, serves as a reliable evaluation measure that is linearly
associated with the model capabilities. We open-source our compression datasets
as well as our data collection pipelines to facilitate future researchers to
assess compression properly.
Megalodon: Efficient LLM Pretraining and Inference with Unlimited
  Context Length
Authors: name: Xuezhe Ma, name: Xiaomeng Yang, name: Wenhan Xiong, name: Beidi Chen, name: Lili Yu, name: Hao Zhang, name: Jonathan May, name: Luke Zettlemoyer, name: Omer Levy, name: Chunting Zhou
Published: 2024-04-12 20:28:14

The quadratic complexity and weak length extrapolation of Transformers limits
their ability to scale to long sequences, and while sub-quadratic solutions
like linear attention and state space models exist, they empirically
underperform Transformers in pretraining efficiency and downstream task
accuracy. We introduce Megalodon, a neural architecture for efficient sequence
modeling with unlimited context length. Megalodon inherits the architecture of
Mega (exponential moving average with gated attention), and further introduces
multiple technical components to improve its capability and stability,
including complex exponential moving average (CEMA), timestep normalization
layer, normalized attention mechanism and pre-norm with two-hop residual
configuration. In a controlled head-to-head comparison with Llama2, Megalodon
achieves better efficiency than Transformer in the scale of 7 billion
parameters and 2 trillion training tokens. Megalodon reaches a training loss of
1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code:
https://github.com/XuezheMax/megalodon
Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
  Environment from a Single Video
Authors: name: Hongchi Xia, name: Zhi-Hao Lin, name: Wei-Chiu Ma, name: Shenlong Wang
Published: 2024-04-15 14:32:32

Creating high-quality and interactive virtual environments, such as games and
simulators, often involves complex and costly manual modeling processes. In
this paper, we present Video2Game, a novel approach that automatically converts
videos of real-world scenes into realistic and interactive game environments.
At the heart of our system are three core components:(i) a neural radiance
fields (NeRF) module that effectively captures the geometry and visual
appearance of the scene; (ii) a mesh module that distills the knowledge from
NeRF for faster rendering; and (iii) a physics module that models the
interactions and physical dynamics among the objects. By following the
carefully designed pipeline, one can construct an interactable and actionable
digital replica of the real world. We benchmark our system on both indoor and
large-scale outdoor scenes. We show that we can not only produce
highly-realistic renderings in real-time, but also build interactive games on
top.
TransformerFAM: Feedback attention is working memory
Authors: name: Dongseong Hwang, name: Weiran Wang, name: Zhuoyuan Huo, name: Khe Chai Sim, name: Pedro Moreno Mengibar
Published: 2024-04-14 07:43:45

While Transformers have revolutionized deep learning, their quadratic
attention complexity hinders their ability to process infinitely long inputs.
We propose Feedback Attention Memory (FAM), a novel Transformer architecture
that leverages a feedback loop to enable the network to attend to its own
latent representations. This design fosters the emergence of working memory
within the Transformer, allowing it to process indefinitely long sequences.
TransformerFAM requires no additional weights, enabling seamless integration
with pre-trained models. Our experiments show that TransformerFAM significantly
improves Transformer performance on long-context tasks across various model
sizes (1B, 8B, and 24B). These results showcase the potential to empower Large
Language Models (LLMs) to process sequences of unlimited length.
Taming Latent Diffusion Model for Neural Radiance Field Inpainting
Authors: name: Chieh Hubert Lin, name: Changil Kim, name: Jia-Bin Huang, name: Qinbo Li, name: Chih-Yao Ma, name: Johannes Kopf, name: Ming-Hsuan Yang, name: Hung-Yu Tseng
Published: 2024-04-15 17:59:57

Neural Radiance Field (NeRF) is a representation for 3D reconstruction from
multi-view images. Despite some recent work showing preliminary success in
editing a reconstructed NeRF with diffusion prior, they remain struggling to
synthesize reasonable geometry in completely uncovered regions. One major
reason is the high diversity of synthetic contents from the diffusion model,
which hinders the radiance field from converging to a crisp and deterministic
geometry. Moreover, applying latent diffusion models on real data often yields
a textural shift incoherent to the image condition due to auto-encoding errors.
These two problems are further reinforced with the use of pixel-distance
losses. To address these issues, we propose tempering the diffusion model's
stochasticity with per-scene customization and mitigating the textural shift
with masked adversarial training. During the analyses, we also found the
commonly used pixel and perceptual losses are harmful in the NeRF inpainting
task. Through rigorous experiments, our framework yields state-of-the-art NeRF
inpainting results on various real-world scenes. Project page:
https://hubert0527.github.io/MALD-NeRF
HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing
Authors: name: Mude Hui, name: Siwei Yang, name: Bingchen Zhao, name: Yichun Shi, name: Heng Wang, name: Peng Wang, name: Yuyin Zhou, name: Cihang Xie
Published: 2024-04-15 17:59:31

This study introduces HQ-Edit, a high-quality instruction-based image editing
dataset with around 200,000 edits. Unlike prior approaches relying on attribute
guidance or human feedback on building datasets, we devise a scalable data
collection pipeline leveraging advanced foundation models, namely GPT-4V and
DALL-E 3. To ensure its high quality, diverse examples are first collected
online, expanded, and then used to create high-quality diptychs featuring input
and output images with detailed text prompts, followed by precise alignment
ensured through post-processing. In addition, we propose two evaluation
metrics, Alignment and Coherence, to quantitatively assess the quality of image
edit pairs using GPT-4V. HQ-Edits high-resolution images, rich in detail and
accompanied by comprehensive editing prompts, substantially enhance the
capabilities of existing image editing models. For example, an HQ-Edit
finetuned InstructPix2Pix can attain state-of-the-art image editing
performance, even surpassing those models fine-tuned with human-annotated data.
The project page is https://thefllood.github.io/HQEdit_web.
Learn Your Reference Model for Real Good Alignment
Authors: name: Alexey Gorbatovski, name: Boris Shaposhnikov, name: Alexey Malakhov, name: Nikita Surnachev, name: Yaroslav Aksenov, name: Ian Maksimov, name: Nikita Balagansky, name: Daniil Gavrilov
Published: 2024-04-15 10:44:31

The complexity of the alignment problem stems from the fact that existing
methods are unstable. Researchers continuously invent various tricks to address
this shortcoming. For instance, in the fundamental Reinforcement Learning From
Human Feedback (RLHF) technique of Language Model alignment, in addition to
reward maximization, the Kullback-Leibler divergence between the trainable
policy and the SFT policy is minimized. This addition prevents the model from
being overfitted to the Reward Model (RM) and generating texts that are
out-of-domain for the RM. The Direct Preference Optimization (DPO) method
reformulates the optimization task of RLHF and eliminates the Reward Model
while tacitly maintaining the requirement for the policy to be close to the SFT
policy. In our paper, we argue that this implicit limitation in the DPO method
leads to sub-optimal results. We propose a new method called Trust Region DPO
(TR-DPO), which updates the reference policy during training. With such a
straightforward update, we demonstrate the effectiveness of TR-DPO against DPO
on the Anthropic HH and TLDR datasets. We show that TR-DPO outperforms DPO by
up to 19%, measured by automatic evaluation with GPT-4. The new alignment
approach that we propose allows us to improve the quality of models across
several parameters at once, such as coherence, correctness, level of detail,
helpfulness, and harmlessness.
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse
  Controls to Any Diffusion Model
Authors: name: Han Lin, name: Jaemin Cho, name: Abhay Zala, name: Mohit Bansal
Published: 2024-04-15 17:45:36

ControlNets are widely used for adding spatial control in image generation
with different conditions, such as depth maps, canny edges, and human poses.
However, there are several challenges when leveraging the pretrained image
ControlNets for controlled video generation. First, pretrained ControlNet
cannot be directly plugged into new backbone models due to the mismatch of
feature spaces, and the cost of training ControlNets for new backbones is a big
burden. Second, ControlNet features for different frames might not effectively
handle the temporal consistency. To address these challenges, we introduce
Ctrl-Adapter, an efficient and versatile framework that adds diverse controls
to any image/video diffusion models, by adapting pretrained ControlNets (and
improving temporal alignment for videos). Ctrl-Adapter provides diverse
capabilities including image control, video control, video control with sparse
frames, multi-condition control, compatibility with different backbones,
adaptation to unseen control conditions, and video editing. In Ctrl-Adapter, we
train adapter layers that fuse pretrained ControlNet features to different
image/video diffusion models, while keeping the parameters of the ControlNets
and the diffusion models frozen. Ctrl-Adapter consists of temporal and spatial
modules so that it can effectively handle the temporal consistency of videos.
We also propose latent skipping and inverse timestep sampling for robust
adaptation and sparse control. Moreover, Ctrl-Adapter enables control from
multiple conditions by simply taking the (weighted) average of ControlNet
outputs. With diverse image/video diffusion backbones (SDXL, Hotshot-XL,
I2VGen-XL, and SVD), Ctrl-Adapter matches ControlNet for image control and
outperforms all baselines for video control (achieving the SOTA accuracy on the
DAVIS 2017 dataset) with significantly lower computational costs (less than 10
GPU hours).
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through
  Direct Preference Optimization
Authors: name: Navonil Majumder, name: Chia-Yu Hung, name: Deepanway Ghosal, name: Wei-Ning Hsu, name: Rada Mihalcea, name: Soujanya Poria
Published: 2024-04-15 17:31:22

Generative multimodal content is increasingly prevalent in much of the
content creation arena, as it has the potential to allow artists and media
personnel to create pre-production mockups by quickly bringing their ideas to
life. The generation of audio from text prompts is an important aspect of such
processes in the music and film industry. Many of the recent diffusion-based
text-to-audio models focus on training increasingly sophisticated diffusion
models on a large set of datasets of prompt-audio pairs. These models do not
explicitly focus on the presence of concepts or events and their temporal
ordering in the output audio with respect to the input prompt. Our hypothesis
is focusing on how these aspects of audio generation could improve audio
generation performance in the presence of limited data. As such, in this work,
using an existing text-to-audio model Tango, we synthetically create a
preference dataset where each prompt has a winner audio output and some loser
audio outputs for the diffusion model to learn from. The loser outputs, in
theory, have some concepts from the prompt missing or in an incorrect order. We
fine-tune the publicly available Tango text-to-audio model using diffusion-DPO
(direct preference optimization) loss on our preference dataset and show that
it leads to improved audio output over Tango and AudioLDM2, in terms of both
automatic- and manual-evaluation metrics.
Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and
  Training Strategies
Authors: name: Zichao Li, name: Cihang Xie, name: Ekin Dogus Cubuk
Published: 2024-04-12 02:04:34

This paper investigates the performance of the Contrastive Language-Image
Pre-training (CLIP) when scaled down to limited computation budgets. We explore
CLIP along three dimensions: data, architecture, and training strategies. With
regards to data, we demonstrate the significance of high-quality training data
and show that a smaller dataset of high-quality data can outperform a larger
dataset with lower quality. We also examine how model performance varies with
different dataset sizes, suggesting that smaller ViT models are better suited
for smaller datasets, while larger models perform better on larger datasets
with fixed compute. Additionally, we provide guidance on when to choose a
CNN-based architecture or a ViT-based architecture for CLIP training. We
compare four CLIP training strategies - SLIP, FLIP, CLIP, and CLIP+Data
Augmentation - and show that the choice of training strategy depends on the
available compute resource. Our analysis reveals that CLIP+Data Augmentation
can achieve comparable performance to CLIP using only half of the training
data. This work provides practical insights into how to effectively train and
deploy CLIP models, making them more accessible and affordable for practical
use in various applications.
COCONut: Modernizing COCO Segmentation
Authors: name: Xueqing Deng, name: Qihang Yu, name: Peng Wang, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2024-04-12 17:59:40

In recent decades, the vision community has witnessed remarkable progress in
visual recognition, partially owing to advancements in dataset benchmarks.
Notably, the established COCO benchmark has propelled the development of modern
detection and segmentation systems. However, the COCO segmentation benchmark
has seen comparatively slow improvement over the last decade. Originally
equipped with coarse polygon annotations for thing instances, it gradually
incorporated coarse superpixel annotations for stuff regions, which were
subsequently heuristically amalgamated to yield panoptic segmentation
annotations. These annotations, executed by different groups of raters, have
resulted not only in coarse segmentation masks but also in inconsistencies
between segmentation types. In this study, we undertake a comprehensive
reevaluation of the COCO segmentation annotations. By enhancing the annotation
quality and expanding the dataset to encompass 383K images with more than 5.18M
panoptic masks, we introduce COCONut, the COCO Next Universal segmenTation
dataset. COCONut harmonizes segmentation annotations across semantic, instance,
and panoptic segmentation with meticulously crafted high-quality masks, and
establishes a robust benchmark for all segmentation tasks. To our knowledge,
COCONut stands as the inaugural large-scale universal segmentation dataset,
verified by human raters. We anticipate that the release of COCONut will
significantly contribute to the community's ability to assess the progress of
novel neural networks.
Dataset Reset Policy Optimization for RLHF
Authors: name: Jonathan D. Chang, name: Wenhao Shan, name: Owen Oertell, name: Kianté Brantley, name: Dipendra Misra, name: Jason D. Lee, name: Wen Sun
Published: 2024-04-12 14:25:49

Reinforcement Learning (RL) from Human Preference-based feedback is a popular
paradigm for fine-tuning generative models, which has produced impressive
models such as GPT-4 and Claude3 Opus. This framework often consists of two
steps: learning a reward model from an offline preference dataset followed by
running online RL to optimize the learned reward model. In this work,
leveraging the idea of reset, we propose a new RLHF algorithm with provable
guarantees. Motivated by the fact that offline preference dataset provides
informative states (i.e., data that is preferred by the labelers), our new
algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing
offline preference dataset into the online policy training procedure via
dataset reset: it directly resets the policy optimizer to the states in the
offline dataset, instead of always starting from the initial state
distribution. In theory, we show that DR-PO learns to perform at least as good
as any policy that is covered by the offline dataset under general function
approximation with finite sample complexity. In experiments, we demonstrate
that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH)
dataset, the generation from DR-PO is better than that from Proximal Policy
Optimization (PPO) and Direction Preference Optimization (DPO), under the
metric of GPT4 win-rate. Code for this work can be found at
https://github.com/Cornell-RL/drpo.
On the Robustness of Language Guidance for Low-Level Vision Tasks:
  Findings from Depth Estimation
Authors: name: Agneet Chatterjee, name: Tejas Gokhale, name: Chitta Baral, name: Yezhou Yang
Published: 2024-04-12 15:35:20

Recent advances in monocular depth estimation have been made by incorporating
natural language as additional guidance. Although yielding impressive results,
the impact of the language prior, particularly in terms of generalization and
robustness, remains unexplored. In this paper, we address this gap by
quantifying the impact of this prior and introduce methods to benchmark its
effectiveness across various settings. We generate "low-level" sentences that
convey object-centric, three-dimensional spatial relationships, incorporate
them as additional language priors and evaluate their downstream impact on
depth estimation. Our key finding is that current language-guided depth
estimators perform optimally only with scene-level descriptions and
counter-intuitively fare worse with low level descriptions. Despite leveraging
additional data, these methods are not robust to directed adversarial attacks
and decline in performance with an increase in distribution shift. Finally, to
provide a foundation for future research, we identify points of failures and
offer insights to better understand these shortcomings. With an increasing
number of methods using language for depth estimation, our findings highlight
the opportunities and pitfalls that require careful consideration for effective
deployment in real-world settings
Probing the 3D Awareness of Visual Foundation Models
Authors: name: Mohamed El Banani, name: Amit Raj, name: Kevis-Kokitsi Maninis, name: Abhishek Kar, name: Yuanzhen Li, name: Michael Rubinstein, name: Deqing Sun, name: Leonidas Guibas, name: Justin Johnson, name: Varun Jampani
Published: 2024-04-12 17:58:04

Recent advances in large-scale pretraining have yielded visual foundation
models with strong capabilities. Not only can recent models generalize to
arbitrary images for their training task, their intermediate representations
are useful for other visual tasks such as detection and segmentation. Given
that such models can classify, delineate, and localize objects in 2D, we ask
whether they also represent their 3D structure? In this work, we analyze the 3D
awareness of visual foundation models. We posit that 3D awareness implies that
representations (1) encode the 3D structure of the scene and (2) consistently
represent the surface across views. We conduct a series of experiments using
task-specific probes and zero-shot inference procedures on frozen features. Our
experiments reveal several limitations of the current models. Our code and
analysis can be found at https://github.com/mbanani/probe3d.