Orionnnn/hub-stats huggingface dataset:
CMC-Bench: Towards a New Paradigm of Visual Signal Compression
Authors: name: Chunyi Li, name: Xiele Wu, name: Haoning Wu, name: Donghui Feng, name: Zicheng Zhang, name: Guo Lu, name: Xiongkuo Min, name: Xiaohong Liu, name: Guangtao Zhai, name: Weisi Lin
Published: 2024-06-13 17:41:37

Ultra-low bitrate image compression is a challenging and demanding topic.
With the development of Large Multimodal Models (LMMs), a Cross Modality
Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with
traditional codecs, this semantic-level compression can reduce image data size
to 0.1\% or even lower, which has strong potential applications. However, CMC
has certain defects in consistency with the original image and perceptual
quality. To address this problem, we introduce CMC-Bench, a benchmark of the
cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models
for image compression. This benchmark covers 18,000 and 40,000 images
respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000
subjective preference scores annotated by human experts. At ultra-low bitrates,
this paper proves that the combination of some I2T and T2I models has surpassed
the most advanced visual signal codecs; meanwhile, it highlights where LMMs can
be further optimized toward the compression task. We encourage LMM developers
to participate in this test to promote the evolution of visual signal codec
protocols.
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer
  Decoding
Authors: name: Zayd Muhammad Kawakibi Zuhri, name: Muhammad Farid Adilazuarda, name: Ayu Purwarianti, name: Alham Fikri Aji
Published: 2024-06-13 16:33:44

Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
caching, but can lead to major memory bottlenecks as model size, batch size,
and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
sharing, a novel approach extending KV sharing across transformer layers to
reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
significantly reduces memory usage with minimal performance loss, reducing KV
cache size down to a factor of 6x compared to MQA. These results highlight
MLKV's potential for efficient deployment of transformer models at scale. We
provide code at https://github.com/zaydzuhri/pythia-mlkv
Estimating the Hallucination Rate of Generative AI
Authors: name: Andrew Jesson, name: Nicolas Beltran-Velez, name: Quentin Chu, name: Sweta Karlekar, name: Jannik Kossen, name: Yarin Gal, name: John P. Cunningham, name: David Blei
Published: 2024-06-11 17:01:52

This work is about estimating the hallucination rate for in-context learning
(ICL) with Generative AI. In ICL, a conditional generative model (CGM) is
prompted with a dataset and asked to make a prediction based on that dataset.
The Bayesian interpretation of ICL assumes that the CGM is calculating a
posterior predictive distribution over an unknown Bayesian model of a latent
parameter and data. With this perspective, we define a hallucination
as a generated prediction that has low-probability under the true latent
parameter. We develop a new method that takes an ICL problem -- that is, a CGM,
a dataset, and a prediction question -- and estimates the probability that a
CGM will generate a hallucination. Our method only requires generating queries
and responses from the model and evaluating its response log probability. We
empirically evaluate our method on synthetic regression and natural language
ICL tasks using large language models.
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
Authors: name: Matthieu Futeral, name: Armel Zebaze, name: Pedro Ortiz Suarez, name: Julien Abadji, name: Rémi Lacroix, name: Cordelia Schmid, name: Rachel Bawden, name: Benoît Sagot
Published: 2024-06-13 00:13:32

Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. [2022] showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only in
English. There have been attempts to reproduce their results but the released
datasets are English-only. In contrast, current multilingual and multimodal
datasets are either composed of caption-like only or medium-scale or fully
private data. This limits mLLM research for the 7,000 other languages spoken in
the world. We therefore introduce mOSCAR, to the best of our knowledge the
first large-scale multilingual and multimodal document corpus crawled from the
web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
is sufficiently safe, diverse and of good quality. We additionally train two
types of multilingual model to prove the benefits of mOSCAR: (1) a model
trained on a subset of mOSCAR and captioning data and (2) a model train on
captioning data only. The model additionally trained on mOSCAR shows a strong
boost in few-shot learning performance across various multilingual image-text
tasks and benchmarks, confirming previous findings for English-only mLLMs.
Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven
  Text-to-Image Generation
Authors: name: Yufan Zhou, name: Ruiyi Zhang, name: Kaizhi Zheng, name: Nanxuan Zhao, name: Jiuxiang Gu, name: Zichao Wang, name: Xin Eric Wang, name: Tong Sun
Published: 2024-06-13 16:40:39

In subject-driven text-to-image generation, recent works have achieved
superior performance by training the model on synthetic datasets containing
numerous image pairs. Trained on these datasets, generative models can produce
text-aligned images for specific subject from arbitrary testing image in a
zero-shot manner. They even outperform methods which require additional
fine-tuning on testing images. However, the cost of creating such datasets is
prohibitive for most researchers. To generate a single training pair, current
methods fine-tune a pre-trained text-to-image model on the subject image to
capture fine-grained details, then use the fine-tuned model to create images
for the same subject based on creative text prompts. Consequently, constructing
a large-scale dataset with millions of subjects can require hundreds of
thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient
method to construct datasets for subject-driven editing and generation.
Specifically, our dataset construction does not need any subject-level
fine-tuning. After pre-training two generative models, we are able to generate
infinite number of high-quality samples. We construct the first large-scale
dataset for subject-driven image editing and generation, which contains 5
million image pairs, text prompts, and masks. Our dataset is 5 times the size
of previous largest dataset, yet our cost is tens of thousands of GPU hours
lower. To test the proposed dataset, we also propose a model which is capable
of both subject-driven image editing and generation. By simply training the
model on our proposed dataset, it obtains competitive results, illustrating the
effectiveness of the proposed dataset construction framework.
Commonsense-T2I Challenge: Can Text-to-Image Generation Models
  Understand Commonsense?
Authors: name: Xingyu Fu, name: Muyu He, name: Yujie Lu, name: William Yang Wang, name: Dan Roth
Published: 2024-06-11 17:59:48

We present a novel task and benchmark for evaluating the ability of
text-to-image(T2I) generation models to produce images that fit commonsense in
real life, which we call Commonsense-T2I. Given two adversarial text prompts
containing an identical set of action words with minor differences, such as "a
lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate
whether T2I models can conduct visual-commonsense reasoning, e.g. produce
images that fit "the lightbulb is unlit" vs. "the lightbulb is lit"
correspondingly. Commonsense-T2I presents an adversarial challenge, providing
pairwise text prompts along with expected outputs. The dataset is carefully
hand-curated by experts and annotated with fine-grained labels, such as
commonsense type and likelihood of the expected outputs, to assist analyzing
model behavior. We benchmark a variety of state-of-the-art (sota) T2I models
and surprisingly find that, there is still a large gap between image synthesis
and real life photos--even the DALL-E 3 model could only achieve 48.92% on
Commonsense-T2I, and the stable diffusion XL model only achieves 24.92%
accuracy. Our experiments show that GPT-enriched prompts cannot solve this
challenge, and we include a detailed analysis about possible reasons for such
deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation
benchmark for T2I commonsense checking, fostering advancements in real life
image generation.
MuirBench: A Comprehensive Benchmark for Robust Multi-image
  Understanding
Authors: name: Fei Wang, name: Xingyu Fu, name: James Y. Huang, name: Zekun Li, name: Qin Liu, name: Xiaogeng Liu, name: Mingyu Derek Ma, name: Nan Xu, name: Wenxuan Zhou, name: Kai Zhang, name: Tianyi Lorena Yan, name: Wenjie Jacky Mo, name: Hsiang-Hui Liu, name: Pan Lu, name: Chunyuan Li, name: Chaowei Xiao, name: Kai-Wei Chang, name: Dan Roth, name: Sheng Zhang, name: Hoifung Poon, name: Muhao Chen
Published: 2024-06-13 17:59:52

We introduce MuirBench, a comprehensive benchmark that focuses on robust
multi-image understanding capabilities of multimodal LLMs. MuirBench consists
of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
involve 10 categories of multi-image relations (e.g., multiview, temporal
relations). Comprising 11,264 images and 2,600 multiple-choice questions,
MuirBench is created in a pairwise manner, where each standard instance is
paired with an unanswerable variant that has minimal semantic differences, in
order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
results reveal that even the best-performing models like GPT-4o and Gemini Pro
find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
Open-source multimodal LLMs trained on single images can hardly generalize to
multi-image questions, hovering below 33.3% in accuracy. These results
highlight the importance of MuirBench in encouraging the community to develop
multimodal LLMs that can look beyond a single image, suggesting potential
pathways for future improvements.
EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal
  Prompts
Authors: name: Yucheng Han, name: Rui Wang, name: Chi Zhang, name: Juntao Hu, name: Pei Cheng, name: Bin Fu, name: Hanwang Zhang
Published: 2024-06-13 14:26:43

Recent advancements in image generation have enabled the creation of
high-quality images from text conditions. However, when facing multi-modal
conditions, such as text combined with reference appearances, existing methods
struggle to balance multiple conditions effectively, typically showing a
preference for one modality over others. To address this challenge, we
introduce EMMA, a novel image generation model accepting multi-modal prompts
built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA
seamlessly incorporates additional modalities alongside text to guide image
generation through an innovative Multi-modal Feature Connector design, which
effectively integrates textual and supplementary modal information using a
special attention mechanism. By freezing all parameters in the original T2I
diffusion model and only adjusting some additional layers, we reveal an
interesting finding that the pre-trained T2I diffusion model can secretly
accept multi-modal prompts. This interesting property facilitates easy
adaptation to different existing frameworks, making EMMA a flexible and
effective tool for producing personalized and context-aware images and even
videos. Additionally, we introduce a strategy to assemble learned EMMA modules
to produce images conditioned on multiple modalities simultaneously,
eliminating the need for additional training with mixed multi-modal prompts.
Extensive experiments demonstrate the effectiveness of EMMA in maintaining high
fidelity and detail in generated images, showcasing its potential as a robust
solution for advanced multi-modal conditional image generation tasks.
LRM-Zero: Training Large Reconstruction Models with Synthesized Data
Authors: name: Desai Xie, name: Sai Bi, name: Zhixin Shu, name: Kai Zhang, name: Zexiang Xu, name: Yi Zhou, name: Sören Pirk, name: Arie Kaufman, name: Xin Sun, name: Hao Tan
Published: 2024-06-13 17:51:00

We present LRM-Zero, a Large Reconstruction Model (LRM) trained entirely on
synthesized 3D data, achieving high-quality sparse-view 3D reconstruction. The
core of LRM-Zero is our procedural 3D dataset, Zeroverse, which is
automatically synthesized from simple primitive shapes with random texturing
and augmentations (e.g., height fields, boolean differences, and wireframes).
Unlike previous 3D datasets (e.g., Objaverse) which are often captured or
crafted by humans to approximate real 3D data, Zeroverse completely ignores
realistic global semantics but is rich in complex geometric and texture details
that are locally similar to or even more intricate than real objects. We
demonstrate that our LRM-Zero, trained with our fully synthesized Zeroverse,
can achieve high visual quality in the reconstruction of real-world objects,
competitive with models trained on Objaverse. We also analyze several critical
design choices of Zeroverse that contribute to LRM-Zero's capability and
training stability. Our work demonstrates that 3D reconstruction, one of the
core tasks in 3D vision, can potentially be addressed without the semantics of
real-world objects. The Zeroverse's procedural synthesis code and interactive
visualization are available at: https://desaixie.github.io/lrm-zero/.
Language Model Council: Benchmarking Foundation Models on Highly
  Subjective Tasks by Consensus
Authors: name: Justin Zhao, name: Flor Miriam Plaza-del-Arco, name: Amanda Cercas Curry
Published: 2024-06-12 19:05:43

The rapid advancement of Large Language Models (LLMs) necessitates robust and
challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
well their responses align with human preferences. However, many tasks such as
those related to emotional intelligence, creative writing, or persuasiveness,
are highly subjective and often lack majoritarian human agreement. Judges may
have irreconcilable disagreements about what constitutes a better response. To
address the challenge of ranking LLMs on highly subjective tasks, we propose a
novel benchmarking framework, the Language Model Council (LMC). The LMC
operates through a democratic process to: 1) formulate a test set through equal
participation, 2) administer the test among council members, and 3) evaluate
responses as a collective jury. We deploy a council of 20 newest LLMs on an
open-ended emotional intelligence task: responding to interpersonal dilemmas.
Our results show that the LMC produces rankings that are more separable,
robust, and less biased than those from any individual LLM judge, and is more
consistent with a human-established leaderboard compared to other benchmarks.
Real3D: Scaling Up Large Reconstruction Models with Real-World Images
Authors: name: Hanwen Jiang, name: Qixing Huang, name: Georgios Pavlakos
Published: 2024-06-12 17:59:08

The default strategy for training single-view Large Reconstruction Models
(LRMs) follows the fully supervised route using large-scale datasets of
synthetic 3D assets or multi-view captures. Although these resources simplify
the training procedure, they are hard to scale up beyond the existing datasets
and they are not necessarily representative of the real distribution of object
shapes. To address these limitations, in this paper, we introduce Real3D, the
first LRM system that can be trained using single-view real-world images.
Real3D introduces a novel self-training framework that can benefit from both
the existing synthetic data and diverse single-view real images. We propose two
unsupervised losses that allow us to supervise LRMs at the pixel- and
semantic-level, even for training examples without ground-truth 3D or novel
views. To further improve performance and scale up the image data, we develop
an automatic data curation approach to collect high-quality examples from
in-the-wild images. Our experiments show that Real3D consistently outperforms
prior work in four diverse evaluation settings that include real and synthetic
data, as well as both in-domain and out-of-domain shapes. Code and model can be
found here: https://hwjiang1510.github.io/Real3D/
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context
  Language Modeling
Authors: name: Liliang Ren, name: Yang Liu, name: Yadong Lu, name: Yelong Shen, name: Chen Liang, name: Weizhu Chen
Published: 2024-06-11 17:50:51

Efficiently modeling sequences with infinite context length has been a
long-standing problem. Past works suffer from either the quadratic computation
complexity or the limited extrapolation ability on length generalization. In
this work, we present Samba, a simple hybrid architecture that layer-wise
combines Mamba, a selective State Space Model (SSM), with Sliding Window
Attention (SWA). Samba selectively compresses a given sequence into recurrent
hidden states while still maintaining the ability to precisely recall memories
with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T
training tokens and show that Samba substantially outperforms the
state-of-the-art models based on pure attention or SSMs on a wide range of
benchmarks. When trained on 4K length sequences, Samba can be efficiently
extrapolated to 256K context length with perfect memory recall and show
improved token predictions up to 1M context length. As a linear-time sequence
model, Samba enjoys a 3.73x higher throughput compared to Transformers with
grouped-query attention when processing user prompts of 128K length, and 3.64x
speedup when generating 64K tokens with unlimited streaming. A sample
implementation of Samba is publicly available in
https://github.com/microsoft/Samba.
TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and
  Image-to-Video Generation
Authors: name: Weixi Feng, name: Jiachen Li, name: Michael Saxon, name: Tsu-jui Fu, name: Wenhu Chen, name: William Yang Wang
Published: 2024-06-12 21:41:32

Video generation has many unique challenges beyond those of image generation.
The temporal dimension introduces extensive possible variations across frames,
over which consistency and continuity may be violated. In this study, we move
beyond evaluating simple actions and argue that generated videos should
incorporate the emergence of new concepts and their relation transitions like
in real-world videos as time progresses. To assess the Temporal
Compositionality of video generation models, we propose TC-Bench, a benchmark
of meticulously crafted text prompts, corresponding ground truth videos, and
robust evaluation metrics. The prompts articulate the initial and final states
of scenes, effectively reducing ambiguities for frame development and
simplifying the assessment of transition completion. In addition, by collecting
aligned real-world videos corresponding to the prompts, we expand TC-Bench's
applicability from text-conditional models to image-conditional ones that can
perform generative frame interpolation. We also develop new metrics to measure
the completeness of component transitions in generated videos, which
demonstrate significantly higher correlations with human judgments than
existing metrics. Our comprehensive experimental results reveal that most video
generators achieve less than 20% of the compositional changes, highlighting
enormous space for future improvement. Our analysis indicates that current
video generation models struggle to interpret descriptions of compositional
changes and synthesize various components across different time steps.
Alleviating Distortion in Image Generation via Multi-Resolution
  Diffusion Models
Authors: name: Qihao Liu, name: Zhanpeng Zeng, name: Ju He, name: Qihang Yu, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2024-06-13 17:59:58

This paper presents innovative enhancements to diffusion models by
integrating a novel multi-resolution network and time-dependent layer
normalization. Diffusion models have gained prominence for their effectiveness
in high-fidelity image generation. While conventional approaches rely on
convolutional U-Net architectures, recent Transformer-based designs have
demonstrated superior performance and scalability. However, Transformer
architectures, which tokenize input data (via "patchification"), face a
trade-off between visual fidelity and computational complexity due to the
quadratic nature of self-attention operations concerning token length. While
larger patch sizes enable attention computation efficiency, they struggle to
capture fine-grained visual details, leading to image distortions. To address
this challenge, we propose augmenting the Diffusion model with the
Multi-Resolution network (DiMR), a framework that refines features across
multiple resolutions, progressively enhancing detail from low to high
resolution. Additionally, we introduce Time-Dependent Layer Normalization
(TD-LN), a parameter-efficient approach that incorporates time-dependent
parameters into layer normalization to inject time information and achieve
superior performance. Our method's efficacy is demonstrated on the
class-conditional ImageNet generation benchmark, where DiMR-XL variants
outperform prior diffusion models, setting new state-of-the-art FID scores of
1.70 on ImageNet 256 x 256 and 2.89 on ImageNet 512 x 512. Project page:
https://qihao067.github.io/projects/DiMR
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal
  Language Models
Authors: name: Yushi Hu, name: Weijia Shi, name: Xingyu Fu, name: Dan Roth, name: Mari Ostendorf, name: Luke Zettlemoyer, name: Noah A Smith, name: Ranjay Krishna
Published: 2024-06-13 17:59:31

Humans draw to facilitate reasoning: we draw auxiliary lines when solving
geometry problems; we mark and circle when reasoning on maps; we use sketches
to amplify our ideas and relieve our limited-capacity working memory. However,
such actions are missing in current multimodal language models (LMs). Current
chain-of-thought and tool-use paradigms only use text as intermediate reasoning
steps. In this work, we introduce Sketchpad, a framework that gives multimodal
LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts
planning and reasoning according to the visual artifacts it has drawn.
Different from prior work, which uses text-to-image models to enable LMs to
draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is
closer to human sketching and better facilitates reasoning. Sketchpad can also
use specialist vision models during the sketching process (e.g., draw bounding
boxes with object detection models, draw masks with segmentation models), to
further enhance visual perception and reasoning. We experiment with a wide
range of math tasks (including geometry, functions, graphs, and chess) and
complex visual reasoning tasks. Sketchpad substantially improves performance on
all tasks over strong base models with no sketching, yielding an average gain
of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a
new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial
reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in
https://visualsketchpad.github.io/.