Orionnnn/hub-stats huggingface dataset:
CoverBench: A Challenging Benchmark for Complex Claim Verification
Authors: name: Alon Jacovi, name: Moran Ambar, name: Eyal Ben-David, name: Uri Shaham, name: Amir Feder, name: Mor Geva, name: Dror Marcus, name: Avi Caciularu
Published: 2024-08-06 17:58:53

There is a growing line of research on verifying the correctness of language
models' outputs. At the same time, LMs are being used to tackle complex queries
that require reasoning. We introduce CoverBench, a challenging benchmark
focused on verifying LM outputs in complex reasoning settings. Datasets that
can be used for this purpose are often designed for other complex reasoning
tasks (e.g., QA) targeting specific use-cases (e.g., financial tables),
requiring transformations, negative sampling and selection of hard examples to
collect such a benchmark. CoverBench provides a diversified evaluation for
complex claim verification in a variety of domains, types of reasoning,
relatively long inputs, and a variety of standardizations, such as multiple
representations for tables where available, and a consistent schema. We
manually vet the data for quality to ensure low levels of label noise. Finally,
we report a variety of competitive baseline results to show CoverBench is
challenging and has very significant headroom. The data is available at
https://huggingface.co/datasets/google/coverbench .
Scaling LLM Test-Time Compute Optimally can be More Effective than
  Scaling Model Parameters
Authors: name: Charlie Snell, name: Jaehoon Lee, name: Kelvin Xu, name: Aviral Kumar
Published: 2024-08-06 17:35:05

Enabling LLMs to improve their outputs by using more test-time computation is
a critical step towards building generally self-improving agents that can
operate on open-ended natural language. In this paper, we study the scaling of
inference-time computation in LLMs, with a focus on answering the question: if
an LLM is allowed to use a fixed but non-trivial amount of inference-time
compute, how much can it improve its performance on a challenging prompt?
Answering this question has implications not only on the achievable performance
of LLMs, but also on the future of LLM pretraining and how one should tradeoff
inference-time and pre-training compute. Despite its importance, little
research attempted to understand the scaling behaviors of various test-time
inference methods. Moreover, current work largely provides negative results for
a number of these strategies. In this work, we analyze two primary mechanisms
to scale test-time computation: (1) searching against dense, process-based
verifier reward models; and (2) updating the model's distribution over a
response adaptively, given the prompt at test time. We find that in both cases,
the effectiveness of different approaches to scaling test-time compute
critically varies depending on the difficulty of the prompt. This observation
motivates applying a "compute-optimal" scaling strategy, which acts to most
effectively allocate test-time compute adaptively per prompt. Using this
compute-optimal strategy, we can improve the efficiency of test-time compute
scaling by more than 4x compared to a best-of-N baseline. Additionally, in a
FLOPs-matched evaluation, we find that on problems where a smaller base model
attains somewhat non-trivial success rates, test-time compute can be used to
outperform a 14x larger model.
Diffusion Models as Data Mining Tools
Authors: name: Ioannis Siglidis, name: Aleksander Holynski, name: Alexei A. Efros, name: Mathieu Aubry, name: Shiry Ginosar
Published: 2024-07-20 17:14:31

This paper demonstrates how to use generative models trained for image
synthesis as tools for visual data mining. Our insight is that since
contemporary generative models learn an accurate representation of their
training data, we can use them to summarize the data by mining for visual
patterns. Concretely, we show that after finetuning conditional diffusion
models to synthesize images from a specific dataset, we can use these models to
define a typicality measure on that dataset. This measure assesses how typical
visual elements are for different data labels, such as geographic location,
time stamps, semantic labels, or even the presence of a disease. This
analysis-by-synthesis approach to data mining has two key advantages. First, it
scales much better than traditional correspondence-based approaches since it
does not require explicitly comparing all pairs of visual elements. Second,
while most previous works on visual data mining focus on a single dataset, our
approach works on diverse datasets in terms of content and scale, including a
historical car dataset, a historical face dataset, a large worldwide
street-view dataset, and an even larger scene dataset. Furthermore, our
approach allows for translating visual elements across class labels and
analyzing consistent changes.
MMIU: Multimodal Multi-image Understanding for Evaluating Large
  Vision-Language Models
Authors: name: Fanqing Meng, name: Jin Wang, name: Chuanhao Li, name: Quanfeng Lu, name: Hao Tian, name: Jiaqi Liao, name: Xizhou Zhu, name: Jifeng Dai, name: Yu Qiao, name: Ping Luo, name: Kaipeng Zhang, name: Wenqi Shao
Published: 2024-08-05 17:56:41

The capability to process multiple images is crucial for Large
Vision-Language Models (LVLMs) to develop a more thorough and nuanced
understanding of a scene. Recent multi-image LVLMs have begun to address this
need. However, their evaluation has not kept pace with their development. To
fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
relationships, 52 tasks, 77K images, and 11K meticulously curated
multiple-choice questions, making it the most extensive benchmark of its kind.
Our evaluation of 24 popular LVLMs, including both open-source and proprietary
models, reveals significant challenges in multi-image comprehension,
particularly in tasks involving spatial understanding. Even the most advanced
models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
multi-faceted analytical experiments, we identify key performance gaps and
limitations, providing valuable insights for future model and data
improvements. We aim for MMIU to advance the frontier of LVLM research and
development, moving us toward achieving sophisticated multimodal multi-image
user interactions.
BioMamba: A Pre-trained Biomedical Language Representation Model
  Leveraging Mamba
Authors: name: Ling Yue, name: Sixue Xing, name: Yingzhou Lu, name: Tianfan Fu
Published: 2024-08-05 16:21:36

The advancement of natural language processing (NLP) in biology hinges on
models' ability to interpret intricate biomedical literature. Traditional
models often struggle with the complex and domain-specific language in this
field. In this paper, we present BioMamba, a pre-trained model specifically
designed for biomedical text mining. BioMamba builds upon the Mamba
architecture and is pre-trained on an extensive corpus of biomedical
literature. Our empirical studies demonstrate that BioMamba significantly
outperforms models like BioBERT and general-domain Mamba across various
biomedical tasks. For instance, BioMamba achieves a 100 times reduction in
perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test
set. We provide an overview of the model architecture, pre-training process,
and fine-tuning techniques. Additionally, we release the code and trained model
to facilitate further research.
RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented
  Generation
Authors: name: Daniel Fleischer, name: Moshe Berchansky, name: Moshe Wasserblat, name: Peter Izsak
Published: 2024-08-05 15:16:24

Implementing Retrieval-Augmented Generation (RAG) systems is inherently
complex, requiring deep understanding of data, use cases, and intricate design
decisions. Additionally, evaluating these systems presents significant
challenges, necessitating assessment of both retrieval accuracy and generative
quality through a multi-faceted approach. We introduce RAG Foundry, an
open-source framework for augmenting large language models for RAG use cases.
RAG Foundry integrates data creation, training, inference and evaluation into a
single workflow, facilitating the creation of data-augmented datasets for
training and evaluating large language models in RAG settings. This integration
enables rapid prototyping and experimentation with various RAG techniques,
allowing users to easily generate datasets and train RAG models using internal
or specialized knowledge sources. We demonstrate the framework effectiveness by
augmenting and fine-tuning Llama-3 and Phi-3 models with diverse RAG
configurations, showcasing consistent improvements across three
knowledge-intensive datasets. Code is released as open-source in
https://github.com/IntelLabs/RAGFoundry.
Operationalizing Contextual Integrity in Privacy-Conscious Assistants
Authors: name: Sahra Ghalebikesabi, name: Eugene Bagdasaryan, name: Ren Yi, name: Itay Yona, name: Ilia Shumailov, name: Aneesh Pappu, name: Chongyang Shi, name: Laura Weidinger, name: Robert Stanforth, name: Leonard Berrada, name: Pushmeet Kohli, name: Po-Sen Huang, name: Borja Balle
Published: 2024-08-05 10:53:51

Advanced AI assistants combine frontier LLMs and tool access to autonomously
perform complex tasks on behalf of users. While the helpfulness of such
assistants can increase dramatically with access to user information including
emails and documents, this raises privacy concerns about assistants sharing
inappropriate information with third parties without user supervision. To steer
information-sharing assistants to behave in accordance with privacy
expectations, we propose to operationalize contextual integrity
(CI), a framework that equates privacy with the appropriate flow of information
in a given context. In particular, we design and evaluate a number of
strategies to steer assistants' information-sharing actions to be CI compliant.
Our evaluation is based on a novel form filling benchmark composed of synthetic
data and human annotations, and it reveals that prompting frontier LLMs to
perform CI-based reasoning yields strong results.
ExoViP: Step-by-step Verification and Exploration with Exoskeleton
  Modules for Compositional Visual Reasoning
Authors: name: Yuxuan Wang, name: Alan Yuille, name: Zhuowan Li, name: Zilong Zheng
Published: 2024-08-05 03:22:10

Compositional visual reasoning methods, which translate a complex query into
a structured composition of feasible visual tasks, have exhibited a strong
potential in complicated multi-modal tasks. Empowered by recent advances in
large language models (LLMs), this multi-modal challenge has been brought to a
new stage by treating LLMs as few-shot/zero-shot planners, i.e.,
vision-language (VL) programming. Such methods, despite their numerous merits,
suffer from challenges due to LLM planning mistakes or inaccuracy of visual
execution modules, lagging behind the non-compositional models. In this work,
we devise a "plug-and-play" method, ExoViP, to correct errors in both the
planning and execution stages through introspective verification. We employ
verification modules as "exoskeletons" to enhance current VL programming
schemes. Specifically, our proposed verification module utilizes a mixture of
three sub-verifiers to validate predictions after each reasoning step,
subsequently calibrating the visual module predictions and refining the
reasoning trace planned by LLMs. Experimental results on two representative VL
programming methods showcase consistent improvements on five compositional
reasoning tasks on standard benchmarks. In light of this, we believe that
ExoViP can foster better performance and generalization on open-domain
multi-modal challenges.
The Impact of Hyperparameters on Large Language Model Inference
  Performance: An Evaluation of vLLM and HuggingFace Pipelines
Authors: name: Matias Martinez
Published: 2024-08-02 06:56:59

The recent surge of open-source large language models (LLMs) enables
developers to create AI-based solutions while maintaining control over aspects
such as privacy and compliance, thereby providing governance and ownership of
the model deployment process. To utilize these LLMs, inference engines are
needed. These engines load the model's weights onto available resources, such
as GPUs, and process queries to generate responses. The speed of inference, or
performance, of the LLM, is critical for real-time applications, as it computes
millions or billions of floating point operations per inference. Recently,
advanced inference engines such as vLLM have emerged, incorporating novel
mechanisms such as efficient memory management to achieve state-of-the-art
performance. In this paper, we analyze the performance, particularly the
throughput (tokens generated per unit of time), of 20 LLMs using two inference
libraries: vLLM and HuggingFace's pipelines. We investigate how various
hyperparameters, which developers must configure, influence inference
performance. Our results reveal that throughput landscapes are irregular, with
distinct peaks, highlighting the importance of hyperparameter optimization to
achieve maximum performance. We also show that applying hyperparameter
optimization when upgrading or downgrading the GPU model used for inference can
improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
respectively.
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data
  Assessment and Selection for Instruction Tuning of Language Models
Authors: name: Yulei Qin, name: Yuncheng Yang, name: Pengcheng Guo, name: Gang Li, name: Hang Shao, name: Yuchen Shi, name: Zihan Xu, name: Yun Gu, name: Ke Li, name: Xing Sun
Published: 2024-08-04 16:50:07

Instruction tuning plays a critical role in aligning large language models
(LLMs) with human preference. Despite the vast amount of open instruction
datasets, naively training a LLM on all existing instructions may not be
optimal and practical. To pinpoint the most beneficial datapoints, data
assessment and selection methods have been proposed in the fields of natural
language processing (NLP) and deep learning. However, under the context of
instruction tuning, there still exists a gap in knowledge on what kind of data
evaluation metrics can be employed and how they can be integrated into the
selection mechanism. To bridge this gap, we present a comprehensive review on
existing literature of data assessment and selection especially for instruction
tuning of LLMs. We systematically categorize all applicable methods into
quality-based, diversity-based, and importance-based ones where a unified,
fine-grained taxonomy is structured. For each category, representative methods
are elaborated to describe the landscape of relevant research. In addition,
comparison between latest methods is conducted on their officially reported
results to provide in-depth discussions on their limitations. Finally, we
summarize the open challenges and propose the promosing avenues for future
studies. All related contents are available at
https://github.com/yuleiqin/fantastic-data-engineering.
MiniCPM-V: A GPT-4V Level MLLM on Your Phone
Authors: name: Yuan Yao, name: Tianyu Yu, name: Ao Zhang, name: Chongyi Wang, name: Junbo Cui, name: Hongji Zhu, name: Tianchi Cai, name: Haoyu Li, name: Weilin Zhao, name: Zhihui He, name: Qianyu Chen, name: Huarong Zhou, name: Zhensheng Zou, name: Haoye Zhang, name: Shengding Hu, name: Zhi Zheng, name: Jie Zhou, name: Jie Cai, name: Xu Han, name: Guoyang Zeng, name: Dahai Li, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-08-03 15:02:21

The recent surge of Multimodal Large Language Models (MLLMs) has
fundamentally reshaped the landscape of AI research and industry, shedding
light on a promising path toward the next AI milestone. However, significant
challenges remain preventing MLLMs from being practical in real-world
applications. The most notable challenge comes from the huge cost of running an
MLLM with a massive number of parameters and extensive computation. As a
result, most MLLMs need to be deployed on high-performing cloud servers, which
greatly limits their application scopes such as mobile, offline,
energy-sensitive, and privacy-protective scenarios. In this work, we present
MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
integrating the latest MLLM techniques in architecture, pretraining and
alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
OCR capability and 1.8M pixel high-resolution image perception at any aspect
ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
support for 30+ languages, and (5) efficient deployment on mobile phones. More
importantly, MiniCPM-V can be viewed as a representative example of a promising
trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
are rapidly decreasing, along with the fast growth of end-side computation
capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
are becoming increasingly possible, unlocking a wider spectrum of real-world AI
applications in the near future.
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation
  with Multimodal Generative Pretraining
Authors: name: Dongyang Liu, name: Shitian Zhao, name: Le Zhuo, name: Weifeng Lin, name: Yu Qiao, name: Hongsheng Li, name: Peng Gao
Published: 2024-08-05 17:46:53

We present Lumina-mGPT, a family of multimodal autoregressive models capable
of various vision and language tasks, particularly excelling in generating
flexible photorealistic images from text descriptions. Unlike existing
autoregressive image generation approaches, Lumina-mGPT employs a pretrained
decoder-only transformer as a unified framework for modeling multimodal token
sequences. Our key insight is that a simple decoder-only transformer with
multimodal Generative PreTraining (mGPT), utilizing the next-token prediction
objective on massive interleaved text-image sequences, can learn broad and
general multimodal capabilities, thereby illuminating photorealistic
text-to-image generation. Building on these pretrained models, we propose
Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text
pairs to fully unlock their potential for high-aesthetic image synthesis at any
resolution while maintaining their general multimodal capabilities.
Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT),
transforming Lumina-mGPT into a foundation model that seamlessly achieves
omnipotent task unification. The resulting model demonstrates versatile
multimodal capabilities, including visual generation tasks like flexible
text-to-image generation and controllable generation, visual recognition tasks
like segmentation and depth estimation, and vision-language tasks like
multiturn visual question answering. Additionally, we analyze the differences
and similarities between diffusion-based and autoregressive methods in a direct
comparison.
VidGen-1M: A Large-Scale Dataset for Text-to-video Generation
Authors: name: Zhiyu Tan, name: Xiaomeng Yang, name: Luozheng Qin, name: Hao Li
Published: 2024-08-05 16:53:23

The quality of video-text pairs fundamentally determines the upper bound of
text-to-video models. Currently, the datasets used for training these models
suffer from significant shortcomings, including low temporal consistency,
poor-quality captions, substandard video quality, and imbalanced data
distribution. The prevailing video curation process, which depends on image
models for tagging and manual rule-based curation, leads to a high
computational load and leaves behind unclean data. As a result, there is a lack
of appropriate training datasets for text-to-video models. To address this
problem, we present VidGen-1M, a superior training dataset for text-to-video
models. Produced through a coarse-to-fine curation strategy, this dataset
guarantees high-quality videos and detailed captions with excellent temporal
consistency. When used to train the video generation model, this dataset has
led to experimental results that surpass those obtained with other models.
Language Model Can Listen While Speaking
Authors: name: Ziyang Ma, name: Yakun Song, name: Chenpeng Du, name: Jian Cong, name: Zhuo Chen, name: Yuping Wang, name: Yuxuan Wang, name: Xie Chen
Published: 2024-08-05 16:47:22

Dialogue serves as the most natural manner of human-computer interaction
(HCI). Recent advancements in speech language models (SLM) have significantly
enhanced speech-based conversational AI. However, these models are limited to
turn-based conversation, lacking the ability to interact with humans in
real-time spoken scenarios, for example, being interrupted when the generated
content is not satisfactory. To address these limitations, we explore full
duplex modeling (FDM) in interactive speech language models (iSLM), focusing on
enhancing real-time interaction and, more explicitly, exploring the
quintessential ability of interruption. We introduce a novel model design,
namely listening-while-speaking language model (LSLM), an end-to-end system
equipped with both listening and speaking channels. Our LSLM employs a
token-based decoder-only TTS for speech generation and a streaming
self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses
both channels for autoregressive generation and detects turn-taking in real
time. Three fusion strategies -- early fusion, middle fusion, and late fusion
-- are explored, with middle fusion achieving an optimal balance between speech
generation and real-time interaction. Two experimental settings, command-based
FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity
to diverse instructions. Our results highlight LSLM's capability to achieve
duplex communication with minimal impact on existing systems. This study aims
to advance the development of interactive speech dialogue systems, enhancing
their applicability in real-world contexts.
ProCreate, Dont Reproduce! Propulsive Energy Diffusion for Creative
  Generation
Authors: name: Jack Lu, name: Ryan Teehan, name: Mengye Ren
Published: 2024-08-05 04:10:52

In this paper, we propose ProCreate, a simple and easy-to-implement method to
improve sample diversity and creativity of diffusion-based image generative
models and to prevent training data reproduction. ProCreate operates on a set
of reference images and actively propels the generated image embedding away
from the reference embeddings during the generation process. We propose FSCG-8
(Few-Shot Creative Generation 8), a few-shot creative generation dataset on
eight different categories -- encompassing different concepts, styles, and
settings -- in which ProCreate achieves the highest sample diversity and
fidelity. Furthermore, we show that ProCreate is effective at preventing
replicating training data in a large-scale evaluation using training text
prompts. Code and FSCG-8 are available at
https://github.com/Agentic-Learning-AI-Lab/procreate-diffusion-public. The
project page is available at https://procreate-diffusion.github.io.