Orionnnn/hub-stats huggingface dataset:
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video
  Dense Captioning
Authors: name: Lin Xu, name: Yilin Zhao, name: Daquan Zhou, name: Zhijie Lin, name: See Kiong Ng, name: Jiashi Feng
Published: 2024-04-25 19:29:55

Vision-language pre-training has significantly elevated performance across a
wide range of image-language applications. Yet, the pre-training process for
video-related tasks demands exceptionally large computational and data
resources, which hinders the progress of video-language models. This paper
investigates a straightforward, highly efficient, and resource-light approach
to adapting an existing image-language pre-trained model for dense video
understanding. Our preliminary experiments reveal that directly fine-tuning
pre-trained image-language models with multiple frames as inputs on video
datasets leads to performance saturation or even a drop. Our further
investigation reveals that it is largely attributed to the bias of learned
high-norm visual features. Motivated by this finding, we propose a simple but
effective pooling strategy to smooth the feature distribution along the
temporal dimension and thus reduce the dominant impacts from the extreme
features. The new model is termed Pooling LLaVA, or  in short.
 achieves new state-of-the-art performance on modern benchmark
datasets for both video question-answer and captioning tasks. Notably, on the
recent popular Video ChatGPT benchmark, PLLaVA achieves a score of 3.48 out of
5 on average of five evaluated dimensions, exceeding the previous SOTA results
from GPT4V (IG-VLM) by 9\%. On the latest multi-choice benchmark MVBench,
PLLaVA achieves 58.1\% accuracy on average across 20 sub-tasks, 14.5\% higher
than GPT4V (IG-VLM). Code is available at
https://github.com/magic-research/PLLaVA.
Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and
  Human Ratings
Authors: name: Olivia Wiles, name: Chuhan Zhang, name: Isabela Albuquerque, name: Ivana Kajić, name: Su Wang, name: Emanuele Bugliarello, name: Yasumasa Onoe, name: Chris Knutsen, name: Cyrus Rashtchian, name: Jordi Pont-Tuset, name: Aida Nematzadeh
Published: 2024-04-25 17:58:43

While text-to-image (T2I) generative models have become ubiquitous, they do
not necessarily generate images that align with a given prompt. While previous
work has evaluated T2I alignment by proposing metrics, benchmarks, and
templates for collecting human judgements, the quality of these components is
not systematically measured. Human-rated prompt sets are generally small and
the reliability of the ratings -- and thereby the prompt set used to compare
models -- is not evaluated. We address this gap by performing an extensive
study evaluating auto-eval metrics and human templates. We provide three main
contributions: (1) We introduce a comprehensive skills-based benchmark that can
discriminate models across different human templates. This skills-based
benchmark categorises prompts into sub-skills, allowing a practitioner to
pinpoint not only which skills are challenging, but at what level of complexity
a skill becomes challenging. (2) We gather human ratings across four templates
and four T2I models for a total of >100K annotations. This allows us to
understand where differences arise due to inherent ambiguity in the prompt and
where they arise due to differences in metric and model quality. (3) Finally,
we introduce a new QA-based auto-eval metric that is better correlated with
human ratings than existing metrics for our new dataset, across different human
templates, and on TIFA160.
List Items One by One: A New Data Source and Learning Paradigm for
  Multimodal LLMs
Authors: name: An Yan, name: Zhengyuan Yang, name: Junda Wu, name: Wanrong Zhu, name: Jianwei Yang, name: Linjie Li, name: Kevin Lin, name: Jianfeng Wang, name: Julian McAuley, name: Jianfeng Gao, name: Lijuan Wang
Published: 2024-04-25 07:29:17

Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of
GPT-4V, by enabling the model to associate visual objects with tags inserted on
the image. These tags, marked with alphanumerics, can be indexed via text
tokens for easy reference. Despite the extraordinary performance from GPT-4V,
we observe that other Multimodal Large Language Models (MLLMs) struggle to
understand these visual tags. To promote the learning of SoM prompting for
open-source models, we propose a new learning paradigm: "list items one by
one," which asks the model to enumerate and describe all visual tags placed on
the image following the alphanumeric orders of tags. By integrating our curated
dataset with other visual instruction tuning datasets, we are able to equip
existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our
finetuned SoM models on five MLLM benchmarks. We find that this new dataset,
even in a relatively small size (10k-30k images with tags), significantly
enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
Perhaps surprisingly, these improvements persist even when the visual tags are
omitted from input images during inference. This suggests the potential of
"list items one by one" as a new paradigm for training MLLMs, which strengthens
the object-text alignment through the use of visual tags in the training stage.
Finally, we conduct analyses by probing trained models to understand the
working mechanism of SoM. Our code and data are available at
https://github.com/zzxslp/SoM-LLaVA.
NeRF-XL: Scaling NeRFs with Multiple GPUs
Authors: name: Ruilong Li, name: Sanja Fidler, name: Angjoo Kanazawa, name: Francis Williams
Published: 2024-04-24 21:43:15

We present NeRF-XL, a principled method for distributing Neural Radiance
Fields (NeRFs) across multiple GPUs, thus enabling the training and rendering
of NeRFs with an arbitrarily large capacity. We begin by revisiting existing
multi-GPU approaches, which decompose large scenes into multiple independently
trained NeRFs, and identify several fundamental issues with these methods that
hinder improvements in reconstruction quality as additional computational
resources (GPUs) are used in training. NeRF-XL remedies these issues and
enables the training and rendering of NeRFs with an arbitrary number of
parameters by simply using more hardware. At the core of our method lies a
novel distributed training and rendering formulation, which is mathematically
equivalent to the classic single-GPU case and minimizes communication between
GPUs. By unlocking NeRFs with arbitrarily large parameter counts, our approach
is the first to reveal multi-GPU scaling laws for NeRFs, showing improvements
in reconstruction quality with larger parameter counts and speed improvements
with more GPUs. We demonstrate the effectiveness of NeRF-XL on a wide variety
of datasets, including the largest open-source dataset to date, MatrixCity,
containing 258K images covering a 25km^2 city area.
ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity
  Preserving
Authors: name: Jiehui Huang, name: Xiao Dong, name: Wenhui Song, name: Hanhui Li, name: Jun Zhou, name: Yuhao Cheng, name: Shutao Liao, name: Long Chen, name: Yiqiang Yan, name: Shengcai Liao, name: Xiaodan Liang
Published: 2024-04-25 17:23:43

Diffusion-based technologies have made significant strides, particularly in
personalized and customized facialgeneration. However, existing methods face
challenges in achieving high-fidelity and detailed identity (ID)consistency,
primarily due to insufficient fine-grained control over facial areas and the
lack of a comprehensive strategy for ID preservation by fully considering
intricate facial details and the overall face. To address these limitations, we
introduce ConsistentID, an innovative method crafted for
diverseidentity-preserving portrait generation under fine-grained multimodal
facial prompts, utilizing only a single reference image. ConsistentID comprises
two key components: a multimodal facial prompt generator that combines facial
features, corresponding facial descriptions and the overall facial context to
enhance precision in facial details, and an ID-preservation network optimized
through the facial attention localization strategy, aimed at preserving ID
consistency in facial regions. Together, these components significantly enhance
the accuracy of ID preservation by introducing fine-grained multimodal ID
information from facial regions. To facilitate training of ConsistentID, we
present a fine-grained portrait dataset, FGID, with over 500,000 facial images,
offering greater diversity and comprehensiveness than existing public facial
datasets. % such as LAION-Face, CelebA, FFHQ, and SFHQ. Experimental results
substantiate that our ConsistentID achieves exceptional precision and diversity
in personalized facial generation, surpassing existing methods in the MyStyle
dataset. Furthermore, while ConsistentID introduces more multimodal ID
information, it maintains a fast inference speed during generation.
Make Your LLM Fully Utilize the Context
Authors: name: Shengnan An, name: Zexiong Ma, name: Zeqi Lin, name: Nanning Zheng, name: Jian-Guang Lou
Published: 2024-04-25 17:55:14

While many contemporary large language models (LLMs) can process lengthy
input, they still struggle to fully utilize information within the long
context, known as the lost-in-the-middle challenge. We hypothesize that it
stems from insufficient explicit supervision during the long-context training,
which fails to emphasize that any position in a long context can hold crucial
information. Based on this intuition, our study presents information-intensive
(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer
dataset, where the answer requires (1) fine-grained information awareness on a
short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
and (2) the integration and reasoning of information from two or more short
segments. Through applying this information-intensive training on Mistral-7B,
we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
FILM-7B for utilizing long contexts, we design three probing tasks that
encompass various context styles (document, code, and structured-data context)
and information retrieval patterns (forward, backward, and bi-directional
retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
information from different positions in its 32K context window. Beyond these
probing tasks, FILM-7B significantly improves the performance on real-world
long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
Authors: name: Mostafa Elhoushi, name: Akshat Shrivastava, name: Diana Liskovich, name: Basil Hosmer, name: Bram Wasti, name: Liangzhen Lai, name: Anas Mahmoud, name: Bilge Acun, name: Saurabh Agarwal, name: Ahmed Roman, name: Ahmed A Aly, name: Beidi Chen, name: Carole-Jean Wu
Published: 2024-04-25 16:20:23

We present LayerSkip, an end-to-end solution to speed-up inference of large
language models (LLMs). First, during training we apply layer dropout, with low
dropout rates for earlier layers and higher dropout rates for later layers, and
an early exit loss where all transformer layers share the same exit. Second,
during inference, we show that this training recipe increases the accuracy of
early exit at earlier layers, without adding any auxiliary layers or modules to
the model. Third, we present a novel self-speculative decoding solution where
we exit at early layers and verify and correct with remaining layers of the
model. Our proposed self-speculative decoding approach has less memory
footprint than other speculative decoding approaches and benefits from shared
compute and activations of the draft and verification stages. We run
experiments on different Llama model sizes on different types of training:
pretraining from scratch, continual pretraining, finetuning on specific data
domain, and finetuning on specific task. We implement our inference solution
and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
checkpoints at https://github.com/facebookresearch/LayerSkip.
Tele-FLM Technical Report
Authors: name: Xiang Li, name: Yiqun Yao, name: Xin Jiang, name: Xuezhi Fang, name: Chao Wang, name: Xinzhang Liu, name: Zihan Wang, name: Yu Zhao, name: Xin Wang, name: Yuyao Huang, name: Shuangyong Song, name: Yongxiang Li, name: Zheng Zhang, name: Bo Zhao, name: Aixin Sun, name: Yequan Wang, name: Zhongjiang He, name: Zhongyuan Wang, name: Xuelong Li, name: Tiejun Huang
Published: 2024-04-25 14:34:47

Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal
  Models with Open-Source Suites
Authors: name: Zhe Chen, name: Weiyun Wang, name: Hao Tian, name: Shenglong Ye, name: Zhangwei Gao, name: Erfei Cui, name: Wenwen Tong, name: Kongzhi Hu, name: Jiapeng Luo, name: Zheng Ma, name: Ji Ma, name: Jiaqi Wang, name: Xiaoyi Dong, name: Hang Yan, name: Hewei Guo, name: Conghui He, name: Zhenjiang Jin, name: Chao Xu, name: Bin Wang, name: Xingjian Wei, name: Wei Li, name: Wenjian Zhang, name: Lewei Lu, name: Xizhou Zhu, name: Tong Lu, name: Dahua Lin, name: Yu Qiao
Published: 2024-04-25 17:59:19

In this report, we introduce InternVL 1.5, an open-source multimodal large
language model (MLLM) to bridge the capability gap between open-source and
proprietary commercial models in multimodal understanding. We introduce three
simple improvements: (1) Strong Vision Encoder: we explored a continuous
learning strategy for the large-scale vision foundation model -- InternViT-6B,
boosting its visual understanding capabilities, and making it can be
transferred and reused in different LLMs. (2) Dynamic High-Resolution: we
divide images into tiles ranging from 1 to 40 of 448times448 pixels
according to the aspect ratio and resolution of the input images, which
supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we
carefully collected a high-quality bilingual dataset that covers common scenes,
document images, and annotated them with English and Chinese question-answer
pairs, significantly enhancing performance in OCR- and Chinese-related tasks.
We evaluate InternVL 1.5 through a series of benchmarks and comparative
studies. Compared to both open-source and proprietary models, InternVL 1.5
shows competitive performance, achieving state-of-the-art results in 8 of 18
benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.
SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with
  Text-Rich Visual Comprehension
Authors: name: Bohao Li, name: Yuying Ge, name: Yi Chen, name: Yixiao Ge, name: Ruimao Zhang, name: Ying Shan
Published: 2024-04-25 17:39:35

Comprehending text-rich visual content is paramount for the practical
application of Multimodal Large Language Models (MLLMs), since text-rich
scenarios are ubiquitous in the real world, which are characterized by the
presence of extensive texts embedded within images. Recently, the advent of
MLLMs with impressive versatility has raised the bar for what we can expect
from MLLMs. However, their proficiency in text-rich scenarios has yet to be
comprehensively and objectively assessed, since current MLLM benchmarks
primarily focus on evaluating general visual comprehension. In this work, we
introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating
text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K
multiple-choice questions with precise human annotations, spanning three broad
categories: Charts, Maps, and Webs, each of which covers a wide spectrum of
text-rich scenarios in the real world. These categories, due to their inherent
complexity and diversity, effectively simulate real-world text-rich
environments. We further conduct a thorough evaluation involving 34 prominent
MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the
current limitations of MLLMs in text-rich visual comprehension. We hope that
our work can serve as a valuable addition to existing MLLM benchmarks,
providing insightful observations and inspiring further research in the area of
text-rich visual comprehension with MLLMs. The dataset and evaluation code can
be accessed at https://github.com/AILab-CVC/SEED-Bench.
Interactive3D: Create What You Want by Interactive 3D Generation
Authors: name: Shaocong Dong, name: Lihe Ding, name: Zhanpeng Huang, name: Zibin Wang, name: Tianfan Xue, name: Dan Xu
Published: 2024-04-25 11:06:57

3D object generation has undergone significant advancements, yielding
high-quality results. However, fall short of achieving precise user control,
often yielding results that do not align with user expectations, thus limiting
their applicability. User-envisioning 3D object generation faces significant
challenges in realizing its concepts using current generative models due to
limited interaction capabilities. Existing methods mainly offer two approaches:
(i) interpreting textual instructions with constrained controllability, or (ii)
reconstructing 3D objects from 2D images. Both of them limit customization to
the confines of the 2D reference and potentially introduce undesirable
artifacts during the 3D lifting process, restricting the scope for direct and
versatile 3D modifications. In this work, we introduce Interactive3D, an
innovative framework for interactive 3D generation that grants users precise
control over the generative process through extensive 3D interaction
capabilities. Interactive3D is constructed in two cascading stages, utilizing
distinct 3D representations. The first stage employs Gaussian Splatting for
direct user interaction, allowing modifications and guidance of the generative
direction at any intermediate step through (i) Adding and Removing components,
(ii) Deformable and Rigid Dragging, (iii) Geometric Transformations, and (iv)
Semantic Editing. Subsequently, the Gaussian splats are transformed into
InstantNGP. We introduce a novel (v) Interactive Hash Refinement module to
further add details and extract the geometry in the second stage. Our
experiments demonstrate that Interactive3D markedly improves the
controllability and quality of 3D generation. Our project webpage is available
at https://interactive-3d.github.io/.
MoDE: CLIP Data Experts via Clustering
Authors: name: Jiawei Ma, name: Po-Yao Huang, name: Saining Xie, name: Shang-Wen Li, name: Luke Zettlemoyer, name: Shih-Fu Chang, name: Wen-Tau Yih, name: Hu Xu
Published: 2024-04-24 17:59:24

The success of contrastive language-image pretraining (CLIP) relies on the
supervision from the pairing between images and captions, which tends to be
noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
a system of CLIP data experts via clustering. Each data expert is trained on
one data cluster, being less sensitive to false negative noises in other
clusters. At inference time, we ensemble their outputs by applying weights
determined through the correlation between task metadata and cluster
conditions. To estimate the correlation precisely, the samples in one cluster
should be semantically similar, but the number of data experts should still be
reasonable for training and inference. As such, we consider the ontology in
human language and propose to use fine-grained cluster centers to represent
each data expert at a coarse-grained level. Experimental studies show that four
CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
OpenCLIP on zero-shot image classification but with less (<35\%) training
cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
include new data experts. The code is available at
https://github.com/facebookresearch/MetaCLIP/tree/main/mode.
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster
  Pre-training on Web-scale Image-Text Data
Authors: name: Sachin Mehta, name: Maxwell Horton, name: Fartash Faghri, name: Mohammad Hossein Sekhavat, name: Mahyar Najibi, name: Mehrdad Farajtabar, name: Oncel Tuzel, name: Mohammad Rastegari
Published: 2024-04-24 05:13:28

Contrastive learning has emerged as a transformative method for learning
effective visual representations through the alignment of image and text
embeddings. However, pairwise similarity computation in contrastive loss
between image and text pairs poses computational challenges. This paper
presents a novel weakly supervised pre-training of vision models on web-scale
image-text data. The proposed method reframes pre-training on image-text data
as a classification task. Consequently, it eliminates the need for pairwise
similarity computations in contrastive loss, achieving a remarkable 2.7times
acceleration in training speed compared to contrastive learning on web-scale
data. Through extensive experiments spanning diverse vision tasks, including
detection and segmentation, we demonstrate that the proposed method maintains
high representation quality. Our source code along with pre-trained model
weights and training recipes is available at
https://github.com/apple/corenet.
MaGGIe: Masked Guided Gradual Human Instance Matting
Authors: name: Chuong Huynh, name: Seoung Wug Oh, name: Abhinav Shrivastava, name: Joon-Young Lee
Published: 2024-04-24 17:59:53

Human matting is a foundation task in image and video processing, where human
foreground pixels are extracted from the input. Prior works either improve the
accuracy by additional guidance or improve the temporal consistency of a single
instance across frames. We propose a new framework MaGGIe, Masked Guided
Gradual Human Instance Matting, which predicts alpha mattes progressively for
each human instances while maintaining the computational cost, precision, and
consistency. Our method leverages modern architectures, including transformer
attention and sparse convolution, to output all instance mattes simultaneously
without exploding memory and latency. Although keeping constant inference costs
in the multiple-instance scenario, our framework achieves robust and versatile
performance on our proposed synthesized benchmarks. With the higher quality
image and video matting benchmarks, the novel multi-instance synthesis approach
from publicly available sources is introduced to increase the generalization of
models in real-world scenarios.