Orionnnn/hub-stats huggingface dataset:
MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
Authors: name: Jiacheng Chen, name: Tianhao Liang, name: Sherman Siu, name: Zhengqing Wang, name: Kai Wang, name: Yubo Wang, name: Yuansheng Ni, name: Wang Zhu, name: Ziyan Jiang, name: Bohan Lyu, name: Dongfu Jiang, name: Xuan He, name: Yuan Liu, name: Hexiang Hu, name: Xiang Yue, name: Wenhu Chen
Published: 2024-10-14 14:42:12

We present MEGA-Bench, an evaluation suite that scales multimodal evaluation
to over 500 real-world tasks, to address the highly heterogeneous daily use
cases of end users. Our objective is to optimize for a set of high-quality data
samples that cover a highly diverse and rich set of multimodal tasks, while
enabling cost-effective and accurate model evaluation. In particular, we
collected 505 realistic tasks encompassing over 8,000 samples from 16 expert
annotators to extensively cover the multimodal task space. Instead of unifying
these problems into standard multi-choice questions (like MMMU, MMBench, and
MMT-Bench), we embrace a wide range of output formats like numbers, phrases,
code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats,
we developed over 40 metrics to evaluate these tasks. Unlike existing
benchmarks, MEGA-Bench offers a fine-grained capability report across multiple
dimensions (e.g., application, input type, output format, skill), allowing
users to interact with and visualize model capabilities in depth. We evaluate a
wide variety of frontier vision-language models on MEGA-Bench to understand
their capabilities across these dimensions.
TVBench: Redesigning Video-Language Evaluation
Authors: name: Daniel Cores, name: Michael Dorkenwald, name: Manuel Mucientes, name: Cees G. M. Snoek, name: Yuki M. Asano
Published: 2024-10-10 09:28:36

Large language models have demonstrated impressive performance when
integrated with vision models even enabling video understanding. However,
evaluating these video models presents its own unique challenges, for which
several benchmarks have been proposed. In this paper, we show that the
currently most used video-language benchmarks can be solved without requiring
much temporal reasoning. We identified three main issues in existing datasets:
(i) static information from single frames is often sufficient to solve the
tasks (ii) the text of the questions and candidate answers is overly
informative, allowing models to answer correctly without relying on any visual
input (iii) world knowledge alone can answer many of the questions, making the
benchmarks a test of knowledge replication rather than visual reasoning. In
addition, we found that open-ended question-answering benchmarks for video
understanding suffer from similar issues while the automatic evaluation process
with LLMs is unreliable, making it an unsuitable alternative. As a solution, we
propose TVBench, a novel open-source video multiple-choice question-answering
benchmark, and demonstrate through extensive evaluations that it requires a
high level of temporal understanding. Surprisingly, we find that most recent
state-of-the-art video-language models perform similarly to random performance
on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.
Rethinking Data Selection at Scale: Random Selection is Almost All You
  Need
Authors: name: Tingyu Xia, name: Bowen Yu, name: Kai Dang, name: An Yang, name: Yuan Wu, name: Yuan Tian, name: Yi Chang, name: Junyang Lin
Published: 2024-10-12 02:48:34

Supervised fine-tuning (SFT) is crucial for aligning Large Language Models
(LLMs) with human instructions. The primary goal during SFT is to select a
small yet representative subset of training data from the larger pool, such
that fine-tuning with this subset achieves results comparable to or even
exceeding those obtained using the entire dataset. However, most existing data
selection techniques are designed for small-scale data pools, which fail to
meet the demands of real-world SFT scenarios. In this paper, we replicated
several self-scoring methods those that do not rely on external model
assistance on two million scale datasets, and found that nearly all methods
struggled to significantly outperform random selection when dealing with such
large-scale data pools. Moreover, our comparisons suggest that, during SFT,
diversity in data selection is more critical than simply focusing on high
quality data. We also analyzed the limitations of several current approaches,
explaining why they perform poorly on large-scale datasets and why they are
unsuitable for such contexts. Finally, we found that filtering data by token
length offers a stable and efficient method for improving results. This
approach, particularly when training on long text data, proves highly
beneficial for relatively weaker base models, such as Llama3.
LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large
  Multimodal Models
Authors: name: Junyan Ye, name: Baichuan Zhou, name: Zilong Huang, name: Junan Zhang, name: Tianyi Bai, name: Hengrui Kang, name: Jun He, name: Honglin Lin, name: Zihao Wang, name: Tong Wu, name: Zhizheng Wu, name: Yiping Chen, name: Dahua Lin, name: Conghui He, name: Weijia Li
Published: 2024-10-13 05:26:36

With the rapid development of AI-generated content, the future internet may
be inundated with synthetic data, making the discrimination of authentic and
credible multimodal data increasingly challenging. Synthetic data detection has
thus garnered widespread attention, and the performance of large multimodal
models (LMMs) in this task has attracted significant interest. LMMs can provide
natural language explanations for their authenticity judgments, enhancing the
explainability of synthetic content detection. Simultaneously, the task of
distinguishing between real and synthetic data effectively tests the
perception, knowledge, and reasoning capabilities of LMMs. In response, we
introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to
detect synthetic data across multiple modalities. LOKI encompasses video,
image, 3D, text, and audio modalities, comprising 18K carefully curated
questions across 26 subcategories with clear difficulty levels. The benchmark
includes coarse-grained judgment and multiple-choice questions, as well as
fine-grained anomaly selection and explanation tasks, allowing for a
comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6
closed-source models on LOKI, highlighting their potential as synthetic data
detectors and also revealing some limitations in the development of LMM
capabilities. More information about LOKI can be found at
https://opendatalab.github.io/LOKI/
Toward General Instruction-Following Alignment for Retrieval-Augmented
  Generation
Authors: name: Guanting Dong, name: Xiaoshuai Song, name: Yutao Zhu, name: Runqi Qiao, name: Zhicheng Dou, name: Ji-Rong Wen
Published: 2024-10-12 16:30:51

Following natural instructions is crucial for the effective application of
Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in
Large Language Models (LLMs), research on assessing and improving
instruction-following (IF) alignment within the RAG domain remains limited. To
address this issue, we propose VIF-RAG, the first automated, scalable, and
verifiable synthetic pipeline for instruction-following alignment in RAG
systems. We start by manually crafting a minimal set of atomic instructions
(<100) and developing combination rules to synthesize and verify complex
instructions for a seed set. We then use supervised models for instruction
rewriting while simultaneously generating code to automate the verification of
instruction quality via a Python executor. Finally, we integrate these
instructions with extensive RAG and general data samples, scaling up to a
high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further
bridge the gap in instruction-following auto-evaluation for RAG systems, we
introduce FollowRAG Benchmark, which includes approximately 3K test samples,
covering 22 categories of general instruction constraints and four
knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG
can seamlessly integrate with different RAG benchmarks. Using FollowRAG and
eight widely-used IF and foundational abilities benchmarks for LLMs, we
demonstrate that VIF-RAG markedly enhances LLM performance across a broad range
of general instruction constraints while effectively leveraging its
capabilities in RAG scenarios. Further analysis offers practical insights for
achieving IF alignment in RAG systems. Our code and datasets are released at
https://FollowRAG.github.io.
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large
  Vision-Language Models
Authors: name: Peng Xia, name: Siwei Han, name: Shi Qiu, name: Yiyang Zhou, name: Zhaoyang Wang, name: Wenhao Zheng, name: Zhaorun Chen, name: Chenhang Cui, name: Mingyu Ding, name: Linjie Li, name: Lijuan Wang, name: Huaxiu Yao
Published: 2024-10-14 04:15:00

Interleaved multimodal comprehension and generation, enabling models to
produce and interpret both images and text in arbitrary sequences, have become
a pivotal area in multimodal learning. Despite significant advancements, the
evaluation of this capability remains insufficient. Existing benchmarks suffer
from limitations in data scale, scope, and evaluation depth, while current
evaluation metrics are often costly or biased, lacking in reliability for
practical applications. To address these challenges, we introduce MMIE, a
large-scale knowledge-intensive benchmark for evaluating interleaved multimodal
comprehension and generation in Large Vision-Language Models (LVLMs). MMIE
comprises 20K meticulously curated multimodal queries, spanning 3 categories,
12 fields, and 102 subfields, including mathematics, coding, physics,
literature, health, and arts. It supports both interleaved inputs and outputs,
offering a mix of multiple-choice and open-ended question formats to evaluate
diverse competencies. Moreover, we propose a reliable automated evaluation
metric, leveraging a scoring model fine-tuned with human-annotated data and
systematic evaluation criteria, aimed at reducing bias and improving evaluation
accuracy. Extensive experiments demonstrate the effectiveness of our benchmark
and metrics in providing a comprehensive evaluation of interleaved LVLMs.
Specifically, we evaluate eight LVLMs, revealing that even the best models show
significant room for improvement, with most achieving only moderate results. We
believe MMIE will drive further advancements in the development of interleaved
LVLMs. We publicly release our benchmark and code in
https://mmie-bench.github.io/.
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
  Language Models
Authors: name: Bofei Gao, name: Feifan Song, name: Zhe Yang, name: Zefan Cai, name: Yibo Miao, name: Qingxiu Dong, name: Lei Li, name: Chenghao Ma, name: Liang Chen, name: Runxin Xu, name: Zhengyang Tang, name: Benyou Wang, name: Daoguang Zan, name: Shanghaoran Quan, name: Ge Zhang, name: Lei Sha, name: Yichang Zhang, name: Xuancheng Ren, name: Tianyu Liu, name: Baobao Chang
Published: 2024-10-10 14:39:33

Recent advancements in large language models (LLMs) have led to significant
breakthroughs in mathematical reasoning capabilities. However, existing
benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
truly challenging these models. To bridge this gap, we propose a comprehensive
and challenging benchmark specifically designed to assess LLMs' mathematical
reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
our dataset focuses exclusively on mathematics and comprises a vast collection
of 4428 competition-level problems with rigorous human annotation. These
problems are meticulously categorized into over 33 sub-domains and span more
than 10 distinct difficulty levels, enabling a holistic assessment of model
performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
in-depth analysis based on this benchmark. Our experimental results show that
even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
with highly challenging Olympiad-level problems, with 60.54% and 52.55%
accuracy, highlighting significant challenges in Olympiad-level mathematical
reasoning.
Think While You Generate: Discrete Diffusion with Planned Denoising
Authors: name: Sulin Liu, name: Juno Nam, name: Andrew Campbell, name: Hannes Stärk, name: Yilun Xu, name: Tommi Jaakkola, name: Rafael Gómez-Bombarelli
Published: 2024-10-08 18:03:34

Discrete diffusion has achieved state-of-the-art performance, outperforming
or approaching autoregressive models on standard benchmarks. In this work, we
introduce Discrete Diffusion with Planned Denoising (DDPD), a novel framework
that separates the generation process into two models: a planner and a
denoiser. At inference time, the planner selects which positions to denoise
next by identifying the most corrupted positions in need of denoising,
including both initially corrupted and those requiring additional refinement.
This plan-and-denoise approach enables more efficient reconstruction during
generation by iteratively identifying and denoising corruptions in the optimal
order. DDPD outperforms traditional denoiser-only mask diffusion methods,
achieving superior results on language modeling benchmarks such as text8,
OpenWebText, and token-based generation on ImageNet 256 times 256. Notably,
in language modeling, DDPD significantly reduces the performance gap between
diffusion-based and autoregressive methods in terms of generative perplexity.
Code is available at https://github.com/liusulin/DDPD.
Synth-SONAR: Sonar Image Synthesis with Enhanced Diversity and Realism
  via Dual Diffusion Models and GPT Prompting
Authors: name: Purushothaman Natarajan, name: Kamal Basha, name: Athira Nambiar
Published: 2024-10-11 08:27:25

Sonar image synthesis is crucial for advancing applications in underwater
exploration, marine biology, and defence. Traditional methods often rely on
extensive and costly data collection using sonar sensors, jeopardizing data
quality and diversity. To overcome these limitations, this study proposes a new
sonar image synthesis framework, Synth-SONAR leveraging diffusion models and
GPT prompting. The key novelties of Synth-SONAR are threefold: First, by
integrating Generative AI-based style injection techniques along with publicly
available real/simulated data, thereby producing one of the largest sonar data
corpus for sonar research. Second, a dual text-conditioning sonar diffusion
model hierarchy synthesizes coarse and fine-grained sonar images with enhanced
quality and diversity. Third, high-level (coarse) and low-level (detailed)
text-based sonar generation methods leverage advanced semantic information
available in visual language models (VLMs) and GPT-prompting. During inference,
the method generates diverse and realistic sonar images from textual prompts,
bridging the gap between textual descriptions and sonar image generation. This
marks the application of GPT-prompting in sonar imagery for the first time, to
the best of our knowledge. Synth-SONAR achieves state-of-the-art results in
producing high-quality synthetic sonar datasets, significantly enhancing their
diversity and realism.
GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Authors: name: Yuancheng Xu, name: Udari Madhushani Sehwag, name: Alec Koppel, name: Sicheng Zhu, name: Bang An, name: Furong Huang, name: Sumitra Ganesh
Published: 2024-10-10 17:58:24

Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.
SimpleStrat: Diversifying Language Model Generation with Stratification
Authors: name: Justin Wong, name: Yury Orlovskiy, name: Michael Luo, name: Sanjit A. Seshia, name: Joseph E. Gonzalez
Published: 2024-10-11 17:54:14

Generating diverse responses from large language models (LLMs) is crucial for
applications such as planning/search and synthetic data generation, where
diversity provides distinct answers across generations. Prior approaches rely
on increasing temperature to increase diversity. However, contrary to popular
belief, we show not only does this approach produce lower quality individual
generations as temperature increases, but it depends on model's next-token
probabilities being similar to the true distribution of answers. We propose
, an alternative approach that uses the language model itself to
partition the space into strata. At inference, a random stratum is selected and
a sample drawn from within the strata. To measure diversity, we introduce
CoverageQA, a dataset of underspecified questions with multiple equally
plausible answers, and assess diversity by measuring KL Divergence between the
output distribution and uniform distribution over valid ground truth answers.
As computing probability per response/solution for proprietary models is
infeasible, we measure recall on ground truth solutions. Our evaluation show
using SimpleStrat achieves higher recall by 0.05 compared to GPT-4o and 0.36
average reduction in KL Divergence compared to Llama 3.
MiRAGeNews: Multimodal Realistic AI-Generated News Detection
Authors: name: Runsheng Huang, name: Liam Dugan, name: Yue Yang, name: Chris Callison-Burch
Published: 2024-10-11 17:58:02

The proliferation of inflammatory or misleading "fake" news content has
become increasingly common in recent years. Simultaneously, it has become
easier than ever to use AI tools to generate photorealistic images depicting
any scene imaginable. Combining these two -- AI-generated fake news content --
is particularly potent and dangerous. To combat the spread of AI-generated fake
news, we propose the MiRAGeNews Dataset, a dataset of 12,500 high-quality real
and AI-generated image-caption pairs from state-of-the-art generators. We find
that our dataset poses a significant challenge to humans (60% F-1) and
state-of-the-art multi-modal LLMs (< 24% F-1). Using our dataset we train a
multi-modal detector (MiRAGe) that improves by +5.1% F-1 over state-of-the-art
baselines on image-caption pairs from out-of-domain image generators and news
publishers. We release our code and data to aid future work on detecting
AI-generated content.
I-Max: Maximize the Resolution Potential of Pre-trained Rectified Flow
  Transformers with Projected Flow
Authors: name: Ruoyi Du, name: Dongyang Liu, name: Le Zhuo, name: Qin Qi, name: Hongsheng Li, name: Zhanyu Ma, name: Peng Gao
Published: 2024-10-10 02:08:23

Rectified Flow Transformers (RFTs) offer superior training and inference
efficiency, making them likely the most viable direction for scaling up
diffusion models. However, progress in generation resolution has been
relatively slow due to data quality and training costs. Tuning-free resolution
extrapolation presents an alternative, but current methods often reduce
generative stability, limiting practical application. In this paper, we review
existing resolution extrapolation methods and introduce the I-Max framework to
maximize the resolution potential of Text-to-Image RFTs. I-Max features: (i) a
novel Projected Flow strategy for stable extrapolation and (ii) an advanced
inference toolkit for generalizing model knowledge to higher resolutions.
Experiments with Lumina-Next-2K and Flux.1-dev demonstrate I-Max's ability to
enhance stability in resolution extrapolation and show that it can bring image
detail emergence and artifact correction, confirming the practical value of
tuning-free resolution extrapolation.
ZeroComp: Zero-shot Object Compositing from Image Intrinsics via
  Diffusion
Authors: name: Zitian Zhang, name: Frédéric Fortier-Chouinard, name: Mathieu Garon, name: Anand Bhattad, name: Jean-François Lalonde
Published: 2024-10-10 17:45:12

We present ZeroComp, an effective zero-shot 3D object compositing approach
that does not require paired composite-scene images during training. Our method
leverages ControlNet to condition from intrinsic images and combines it with a
Stable Diffusion model to utilize its scene priors, together operating as an
effective rendering engine. During training, ZeroComp uses intrinsic images
based on geometry, albedo, and masked shading, all without the need for paired
images of scenes with and without composite objects. Once trained, it
seamlessly integrates virtual 3D objects into scenes, adjusting shading to
create realistic composites. We developed a high-quality evaluation dataset and
demonstrate that ZeroComp outperforms methods using explicit lighting
estimations and generative techniques in quantitative and human perception
benchmarks. Additionally, ZeroComp extends to real and outdoor image
compositing, even when trained solely on synthetic indoor data, showcasing its
effectiveness in image compositing.
Mentor-KD: Making Small Language Models Better Multi-step Reasoners
Authors: name: Hojae Lee, name: Junho Kim, name: SangKeun Lee
Published: 2024-10-11 17:53:27

Large Language Models (LLMs) have displayed remarkable performances across
various complex tasks by leveraging Chain-of-Thought (CoT) prompting. Recently,
studies have proposed a Knowledge Distillation (KD) approach, reasoning
distillation, which transfers such reasoning ability of LLMs through
fine-tuning language models of multi-step rationales generated by LLM teachers.
However, they have inadequately considered two challenges regarding
insufficient distillation sets from the LLM teacher model, in terms of 1) data
quality and 2) soft label provision. In this paper, we propose Mentor-KD, which
effectively distills the multi-step reasoning capability of LLMs to smaller LMs
while addressing the aforementioned challenges. Specifically, we exploit a
mentor, intermediate-sized task-specific fine-tuned model, to augment
additional CoT annotations and provide soft labels for the student model during
reasoning distillation. We conduct extensive experiments and confirm
Mentor-KD's effectiveness across various models and complex reasoning tasks.