Orionnnn/hub-stats huggingface dataset:
Exploring the Role of Large Language Models in Prompt Encoding for
  Diffusion Models
Authors: name: Bingqi Ma, name: Zhuofan Zong, name: Guanglu Song, name: Hongsheng Li, name: Yu Liu
Published: 2024-06-17 17:59:43

Large language models (LLMs) based on decoder-only transformers have
demonstrated superior text understanding capabilities compared to CLIP and
T5-series models. However, the paradigm for utilizing current advanced LLMs in
text-to-image diffusion models remains to be explored. We observed an unusual
phenomenon: directly using a large language model as the prompt encoder
significantly degrades the prompt-following ability in image generation. We
identified two main obstacles behind this issue. One is the misalignment
between the next token prediction training in LLM and the requirement for
discriminative prompt features in diffusion models. The other is the intrinsic
positional bias introduced by the decoder-only architecture. To deal with this
issue, we propose a novel framework to fully harness the capabilities of LLMs.
Through the carefully designed usage guidance, we effectively enhance the text
representation capability for prompt encoding and eliminate its inherent
positional bias. This allows us to integrate state-of-the-art LLMs into the
text-to-image generation model flexibly. Furthermore, we also provide an
effective manner to fuse multiple LLMs into our framework. Considering the
excellent performance and scaling capabilities demonstrated by the transformer
architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
based on the framework. We conduct extensive experiments to validate LI-DiT
across model size and data size. Benefiting from the inherent ability of the
LLMs and our innovative designs, the prompt understanding performance of LI-DiT
easily surpasses state-of-the-art open-source models as well as mainstream
closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
Midjourney V6. The powerful LI-DiT-10B will be available after further
optimization and security checks.
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and
  Instruction-Tuning Dataset for LVLMs
Authors: name: Ziyu Liu, name: Tao Chu, name: Yuhang Zang, name: Xilin Wei, name: Xiaoyi Dong, name: Pan Zhang, name: Zijian Liang, name: Yuanjun Xiong, name: Yu Qiao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-06-17 17:59:47

Generating natural and meaningful responses to communicate with multi-modal
human inputs is a fundamental capability of Large Vision-Language
Models(LVLMs). While current open-source LVLMs demonstrate promising
performance in simplified scenarios such as single-turn single-image input,
they fall short in real-world conversation scenarios such as following
instructions in a long context history with multi-turn and multi-images.
Existing LVLM benchmarks primarily focus on single-choice questions or
short-form responses, which do not adequately assess the capabilities of LVLMs
in real-world human-AI interaction applications. Therefore, we introduce MMDU,
a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning
dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and
multi-image conversations. We employ the clustering algorithm to ffnd the
relevant images and textual descriptions from the open-source Wikipedia and
construct the question-answer pairs by human annotators with the assistance of
the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and
27 turns, which is at least 5x longer than previous benchmarks and poses
challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs
using MMDU reveals that open-source LVLMs lag behind closed-source counterparts
due to limited conversational instruction tuning data. We demonstrate that
ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap,
generating longer and more accurate conversations, and improving scores on MMDU
and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our
contributions pave the way for bridging the gap between current LVLM models and
real-world application demands. This project is available at
https://github.com/Liuziyu77/MMDU.
CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language
  Model Pre-training
Authors: name: David Brandfonbrener, name: Hanlin Zhang, name: Andreas Kirsch, name: Jonathan Richard Schwarz, name: Sham Kakade
Published: 2024-06-15 15:28:02

Selecting high-quality data for pre-training is crucial in shaping the
downstream task performance of language models. A major challenge lies in
identifying this optimal subset, a problem generally considered intractable,
thus necessitating scalable and effective heuristics. In this work, we propose
a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering),
which leverages an empirical Bayes-inspired approach to derive a simple and
computationally efficient selection criterion based on the relative loss values
of two auxiliary models.
  In addition to the modeling rationale, we evaluate CoLoR-Filter empirically
on two language modeling tasks: (1) selecting data from C4 for domain
adaptation to evaluation on Books and (2) selecting data from C4 for a suite of
downstream multiple-choice question answering tasks. We demonstrate favorable
scaling both as we subselect more aggressively and using small auxiliary models
to select data for large target models. As one headline result, CoLoR-Filter
data selected using a pair of 150m parameter auxiliary models can train a 1.2b
parameter target model to match a 1.2b parameter model trained on 25b randomly
selected tokens with 25x less data for Books and 11x less data for the
downstream tasks.
  Code: https://github.com/davidbrandfonbrener/color-filter-olmo
  Filtered data:
https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4
How Do Large Language Models Acquire Factual Knowledge During
  Pretraining?
Authors: name: Hoyeon Chang, name: Jinho Park, name: Seonghyeon Ye, name: Sohee Yang, name: Youngkyung Seo, name: Du-Seong Chang, name: Minjoon Seo
Published: 2024-06-17 17:54:40

Despite the recent observation that large language models (LLMs) can store
substantial factual knowledge, there is a limited understanding of the
mechanisms of how they acquire factual knowledge through pretraining. This work
addresses this gap by studying how LLMs acquire factual knowledge during
pretraining. The findings reveal several important insights into the dynamics
of factual knowledge acquisition during pretraining. First, counterintuitively,
we observe that pretraining on more data shows no significant improvement in
the model's capability to acquire and maintain factual knowledge. Next, there
is a power-law relationship between training steps and forgetting of
memorization and generalization of factual knowledge, and LLMs trained with
duplicated training data exhibit faster forgetting. Third, training LLMs with
larger batch sizes can enhance the models' robustness to forgetting. Overall,
our observations suggest that factual knowledge acquisition in LLM pretraining
occurs by progressively increasing the probability of factual knowledge
presented in the pretraining data at each step. However, this increase is
diluted by subsequent forgetting. Based on this interpretation, we demonstrate
that we can provide plausible explanations for recently observed behaviors of
LLMs, such as the poor performance of LLMs on long-tail knowledge and the
benefits of deduplicating the pretraining corpus.
LLaNA: Large Language and NeRF Assistant
Authors: name: Andrea Amaduzzi, name: Pierluigi Zama Ramirez, name: Giuseppe Lisanti, name: Samuele Salti, name: Luigi Di Stefano
Published: 2024-06-17 17:59:59

Multimodal Large Language Models (MLLMs) have demonstrated an excellent
understanding of images and 3D data. However, both modalities have shortcomings
in holistically capturing the appearance and geometry of objects. Meanwhile,
Neural Radiance Fields (NeRFs), which encode information within the weights of
a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
widespread modality that simultaneously encodes the geometry and photorealistic
appearance of objects. This paper investigates the feasibility and
effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
general-purpose NeRF-language assistant capable of performing new tasks such as
NeRF captioning and Q\&A. Notably, our method directly processes the weights of
the NeRF's MLP to extract information about the represented objects without the
need to render images or materialize 3D data structures. Moreover, we build a
dataset of NeRFs with text annotations for various NeRF-language tasks with no
human intervention. Based on this dataset, we develop a benchmark to evaluate
the NeRF understanding capability of our method. Results show that processing
NeRF weights performs favourably against extracting 2D or 3D representations
from NeRFs.
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
  Dataset with One Trillion Tokens
Authors: name: Anas Awadalla, name: Le Xue, name: Oscar Lo, name: Manli Shu, name: Hannah Lee, name: Etash Kumar Guha, name: Matt Jordan, name: Sheng Shen, name: Mohamed Awadalla, name: Silvio Savarese, name: Caiming Xiong, name: Ran Xu, name: Yejin Choi, name: Ludwig Schmidt
Published: 2024-06-17 07:21:36

Multimodal interleaved datasets featuring free-form interleaved sequences of
images and text are crucial for training frontier large multimodal models
(LMMs). Despite the rapid progression of open-source LMMs, there remains a
pronounced scarcity of large-scale, diverse open-source multimodal interleaved
datasets. In response, we introduce MINT-1T, the most extensive and diverse
open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
trillion text tokens and three billion images, a 10x scale-up from existing
open-source datasets. Additionally, we include previously untapped sources such
as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
substantial engineering effort, sharing the data curation process and releasing
the dataset greatly benefits the community. Our experiments show that LMMs
trained on MINT-1T rival the performance of models trained on the previous
leading dataset, OBELICS. Our data and code will be released at
https://github.com/mlfoundations/MINT-1T.
Breaking the Attention Bottleneck
Authors: name: Kalle Hilsenbek
Published: 2024-06-16 12:06:58

Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
https://gitlab.com/Bachstelze/causal_generation.
Evaluating Open Language Models Across Task Types, Application Domains,
  and Reasoning Types: An In-Depth Experimental Analysis
Authors: name: Neelabh Sinha, name: Vinija Jain, name: Aman Chadha
Published: 2024-06-17 10:45:36

The rapid rise of Language Models (LMs) has expanded their use in several
applications. Yet, due to constraints of model size, associated cost, or
proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always
feasible. With open, smaller LMs emerging, more applications can leverage their
capabilities, but selecting the right LM can be challenging. This work conducts
an in-depth experimental analysis of the semantic correctness of outputs of 10
smaller, open LMs across three aspects: task types, application domains and
reasoning types, using diverse prompt styles. We demonstrate that most
effective models and prompt styles vary depending on the specific requirements.
Our analysis provides a comparative assessment of LMs and prompt styles using a
proposed three-tier schema of aspects for their strategic selection based on
use-case and other constraints. We also show that if utilized appropriately,
these LMs can compete with, and sometimes outperform, SOTA LLMs like
DeepSeek-v2, GPT-3.5-Turbo, and GPT-4o.
Task Me Anything
Authors: name: Jieyu Zhang, name: Weikai Huang, name: Zixian Ma, name: Oscar Michel, name: Dong He, name: Tanmay Gupta, name: Wei-Chiu Ma, name: Ali Farhadi, name: Aniruddha Kembhavi, name: Ranjay Krishna
Published: 2024-06-17 17:32:42

Benchmarks for large multimodal language models (MLMs) now serve to
simultaneously assess the general capabilities of models instead of evaluating
for a specific capability. As a result, when a developer wants to identify
which models to use for their application, they are overwhelmed by the number
of benchmarks and remain uncertain about which benchmark's results are most
reflective of their specific use case. This paper introduces Task-Me-Anything,
a benchmark generation engine which produces a benchmark tailored to a user's
needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
can programmatically generate a vast number of task instances. Additionally, it
algorithmically addresses user queries regarding MLM performance efficiently
within a computational budget. It contains 113K images, 10K videos, 2K 3D
object assets, over 365 object categories, 655 attributes, and 335
relationships. It can generate 750M image/video question-answering pairs, which
focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
critical insights: open-source MLMs excel in object and attribute recognition
but lack spatial and temporal understanding; each model exhibits unique
strengths and weaknesses; larger models generally perform better, though
exceptions exist; and GPT4o demonstrates challenges in recognizing
rotating/moving objects and distinguishing colors.
From Pixels to Prose: A Large Dataset of Dense Image Captions
Authors: name: Vasu Singla, name: Kaiyu Yue, name: Sukriti Paul, name: Reza Shirkavand, name: Mayuka Jayawardhana, name: Alireza Ganjdanesh, name: Heng Huang, name: Abhinav Bhatele, name: Gowthami Somepalli, name: Tom Goldstein
Published: 2024-06-14 17:59:53

Training large vision-language models requires extensive, high-quality
image-text pairs. Existing web-scraped datasets, however, are noisy and lack
detailed image descriptions. To bridge this gap, we introduce PixelProse, a
comprehensive dataset of over 16M (million) synthetically generated captions,
leveraging cutting-edge vision-language models for detailed and accurate
descriptions. To ensure data integrity, we rigorously analyze our dataset for
problematic content, including child sexual abuse material (CSAM), personally
identifiable information (PII), and toxicity. We also provide valuable metadata
such as watermark presence and aesthetic scores, aiding in further dataset
filtering. We hope PixelProse will be a valuable resource for future
vision-language research. PixelProse is available at
https://huggingface.co/datasets/tomg-group-umd/pixelprose
VideoLLM-online: Online Video Large Language Model for Streaming Video
Authors: name: Joya Chen, name: Zhaoyang Lv, name: Shiwei Wu, name: Kevin Qinghong Lin, name: Chenan Song, name: Difei Gao, name: Jia-Wei Liu, name: Ziteng Gao, name: Dongxing Mao, name: Mike Zheng Shou
Published: 2024-06-17 17:55:32

Recent Large Language Models have been enhanced with vision capabilities,
enabling them to comprehend images, videos, and interleaved vision-language
content. However, the learning methods of these large multimodal models
typically treat videos as predetermined clips, making them less effective and
efficient at handling streaming video inputs. In this paper, we propose a novel
Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
long-context, and real-time conversation within a continuous video stream. Our
LIVE framework comprises comprehensive approaches to achieve video streaming
dialogue, encompassing: (1) a training objective designed to perform language
modeling for continuous streaming inputs, (2) a data generation scheme that
converts offline temporal annotations into a streaming dialogue format, and (3)
an optimized inference pipeline to speed up the model responses in real-world
video streams. With our LIVE framework, we built VideoLLM-online model upon
Llama-2/Llama-3 and demonstrate its significant advantages in processing
streaming videos. For instance, on average, our model can support streaming
dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
also showcases state-of-the-art performance on public offline video benchmarks,
such as recognition, captioning, and forecasting. The code, model, data, and
demo have been made available at https://showlab.github.io/videollm-online.
THEANINE: Revisiting Memory Management in Long-term Conversations with
  Timeline-augmented Response Generation
Authors: name: Seo Hyun Kim, name: Kai Tzu-iunn Ong, name: Taeyoon Kwon, name: Namyoung Kim, name: Keummin Ka, name: SeongHyeon Bae, name: Yohan Jo, name: Seung-won Hwang, name: Dongha Lee, name: Jinyoung Yeo
Published: 2024-06-16 16:17:46

Large language models (LLMs) are capable of processing lengthy dialogue
histories during prolonged interaction with users without additional memory
modules; however, their responses tend to overlook or incorrectly recall
information from the past. In this paper, we revisit memory-augmented response
generation in the era of LLMs. While prior work focuses on getting rid of
outdated memories, we argue that such memories can provide contextual cues that
help dialogue systems understand the development of past events and, therefore,
benefit response generation. We present Theanine, a framework that augments
LLMs' response generation with memory timelines -- series of memories that
demonstrate the development and causality of relevant past events. Along with
Theanine, we introduce TeaFarm, a counterfactual-driven question-answering
pipeline addressing the limitation of G-Eval in long-term conversations.
Supplementary videos of our methods and the TeaBag dataset for TeaFarm
evaluation are in https://theanine-693b0.web.app/.
WPO: Enhancing RLHF with Weighted Preference Optimization
Authors: name: Wenxuan Zhou, name: Ravi Agrawal, name: Shujian Zhang, name: Sathish Reddy Indurthi, name: Sanqiang Zhao, name: Kaiqiang Song, name: Silei Xu, name: Chenguang Zhu
Published: 2024-06-17 17:59:13

Reinforcement learning from human feedback (RLHF) is a promising solution to
align large language models (LLMs) more closely with human values. Off-policy
preference optimization, where the preference data is obtained from other
models, is widely adopted due to its cost efficiency and scalability. However,
off-policy preference optimization often suffers from a distributional gap
between the policy used for data collection and the target policy, leading to
suboptimal optimization. In this paper, we propose a novel strategy to mitigate
this problem by simulating on-policy learning with off-policy preference data.
Our Weighted Preference Optimization (WPO) method adapts off-policy data to
resemble on-policy data more closely by reweighting preference pairs according
to their probability under the current policy. This method not only addresses
the distributional gap problem but also enhances the optimization process
without incurring additional costs. We validate our method on instruction
following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
but also establishes a remarkable length-controlled winning rate against
GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
model on the leaderboard. We will release the code and models at
https://github.com/wzhouad/WPO.
WildVision: Evaluating Vision-Language Models in the Wild with Human
  Preferences
Authors: name: Yujie Lu, name: Dongfu Jiang, name: Wenhu Chen, name: William Yang Wang, name: Yejin Choi, name: Bill Yuchen Lin
Published: 2024-06-16 20:53:25

Recent breakthroughs in vision-language models (VLMs) emphasize the necessity
of benchmarking human preferences in real-world multimodal interactions. To
address this gap, we launched WildVision-Arena (WV-Arena), an online platform
that collects human preferences to evaluate VLMs. We curated WV-Bench by
selecting 500 high-quality samples from 8,000 user submissions in WV-Arena.
WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet,
achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This
significantly outperforms other benchmarks like MMVet, MMMU, and MMStar.
  Our comprehensive analysis of 20K real-world interactions reveals important
insights into the failure cases of top-performing VLMs. For example, we find
that although GPT-4V surpasses many other models like Reka-Flash, Opus, and
Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces
challenges with subtle contextual cues, spatial reasoning, visual imagination,
and expert domain knowledge. Additionally, current VLMs exhibit issues with
hallucinations and safety when intentionally provoked. We are releasing our
chat and feedback data to further advance research in the field of VLMs.
MeshAnything: Artist-Created Mesh Generation with Autoregressive
  Transformers
Authors: name: Yiwen Chen, name: Tong He, name: Di Huang, name: Weicai Ye, name: Sijin Chen, name: Jiaxiang Tang, name: Xin Chen, name: Zhongang Cai, name: Lei Yang, name: Gang Yu, name: Guosheng Lin, name: Chi Zhang
Published: 2024-06-14 16:30:25

Recently, 3D assets created via reconstruction and generation have matched
the quality of manually crafted assets, highlighting their potential for
replacement. However, this potential is largely unrealized because these assets
always need to be converted to meshes for 3D industry applications, and the
meshes produced by current mesh extraction methods are significantly inferior
to Artist-Created Meshes (AMs), i.e., meshes created by human artists.
Specifically, current mesh extraction methods rely on dense faces and ignore
geometric features, leading to inefficiencies, complicated post-processing, and
lower representation quality. To address these issues, we introduce
MeshAnything, a model that treats mesh extraction as a generation problem,
producing AMs aligned with specified shapes. By converting 3D assets in any 3D
representation into AMs, MeshAnything can be integrated with various 3D asset
production methods, thereby enhancing their application across the 3D industry.
The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned
decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE,
then train the shape-conditioned decoder-only transformer on this vocabulary
for shape-conditioned autoregressive mesh generation. Our extensive experiments
show that our method generates AMs with hundreds of times fewer faces,
significantly improving storage, rendering, and simulation efficiencies, while
achieving precision comparable to previous methods.