Orionnnn/hub-stats huggingface dataset:
ColPali: Efficient Document Retrieval with Vision Language Models
Authors: name: Manuel Faysse, name: Hugues Sibille, name: Tony Wu, name: Gautier Viaud, name: Céline Hudelot, name: Pierre Colombo
Published: 2024-06-27 15:45:29

Documents are visually rich structures that convey information through text,
as well as tables, figures, page layouts, or fonts. While modern document
retrieval systems exhibit strong performance on query-to-text matching, they
struggle to exploit visual cues efficiently, hindering their performance on
practical document retrieval applications such as Retrieval Augmented
Generation. To benchmark current systems on visually rich document retrieval,
we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of
various page-level retrieving tasks spanning multiple domains, languages, and
settings. The inherent shortcomings of modern systems motivate the introduction
of a new retrieval model architecture, ColPali, which leverages the document
understanding capabilities of recent Vision Language Models to produce
high-quality contextualized embeddings solely from images of document pages.
Combined with a late interaction matching mechanism, ColPali largely
outperforms modern document retrieval pipelines while being drastically faster
and end-to-end trainable.
Towards Robust Speech Representation Learning for Thousands of Languages
Authors: name: William Chen, name: Wangyou Zhang, name: Yifan Peng, name: Xinjian Li, name: Jinchuan Tian, name: Jiatong Shi, name: Xuankai Chang, name: Soumi Maiti, name: Karen Livescu, name: Shinji Watanabe
Published: 2024-06-30 21:40:26

Self-supervised learning (SSL) has helped extend speech technologies to more
languages by reducing the need for labeled data. However, models are still far
from supporting the world's 7000+ languages. We propose XEUS, a Cross-lingual
Encoder for Universal Speech, trained on over 1 million hours of data across
4057 languages, extending the language coverage of SSL models 4-fold. We
combine 1 million hours of speech from existing publicly accessible corpora
with a newly created corpus of 7400+ hours from 4057 languages, which will be
publicly released. To handle the diverse conditions of multilingual speech
data, we augment the typical SSL masked prediction approach with a novel
dereverberation objective, increasing robustness. We evaluate XEUS on several
benchmarks, and show that it consistently outperforms or achieves comparable
results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS
sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT
2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or
pre-training data. Checkpoints, code, and data are found in
https://www.wavlab.org/activities/2024/xeus/.
SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix
Authors: name: Peng Dai, name: Feitong Tan, name: Qiangeng Xu, name: David Futschik, name: Ruofei Du, name: Sean Fanello, name: Xiaojuan Qi, name: Yinda Zhang
Published: 2024-06-29 08:33:55

Video generation models have demonstrated great capabilities of producing
impressive monocular videos, however, the generation of 3D stereoscopic video
remains under-explored. We propose a pose-free and training-free approach for
generating 3D stereoscopic videos using an off-the-shelf monocular video
generation model. Our method warps a generated monocular video into camera
views on stereoscopic baseline using estimated video depth, and employs a novel
frame matrix video inpainting framework. The framework leverages the video
generation model to inpaint frames observed from different timestamps and
views. This effective approach generates consistent and semantically coherent
stereoscopic videos without scene optimization or model fine-tuning. Moreover,
we develop a disocclusion boundary re-injection scheme that further improves
the quality of video inpainting by alleviating the negative effects propagated
from disoccluded areas in the latent space. We validate the efficacy of our
proposed method by conducting experiments on videos from various generative
models, including Sora [4 ], Lumiere [2], WALT [8 ], and Zeroscope [ 42]. The
experiments demonstrate that our method has a significant improvement over
previous methods. The code will be released at
https://daipengwa.github.io/SVG_ProjectPage.
Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical
  Reasoning
Authors: name: Zimu Lu, name: Aojun Zhou, name: Ke Wang, name: Houxing Ren, name: Weikang Shi, name: Junting Pan, name: Mingjie Zhan
Published: 2024-06-30 17:59:07

Direct Preference Optimization (DPO) has proven effective at improving the
performance of large language models (LLMs) on downstream tasks such as
reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
a method for automatically providing stepwise error supervision by creating
negative samples of mathematical reasoning rationales that start making errors
at a specified step. By applying these samples in DPO training, SCDPO can
better align the model to understand reasoning errors and output accurate
reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
solutions, empirically showing that it consistently improves the performance
compared to naive DPO on three different SFT models, including one existing SFT
model and two models we finetuned. Qualitative analysis of the credit
assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
identifying errors in mathematical solutions. We then apply SCDPO to an
InternLM2-20B model, resulting in a 20B model that achieves high scores of
88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
the great potential of our method.
RegMix: Data Mixture as Regression for Language Model Pre-training
Authors: name: Qian Liu, name: Xiaosen Zheng, name: Niklas Muennighoff, name: Guangtao Zeng, name: Longxu Dou, name: Tianyu Pang, name: Jing Jiang, name: Min Lin
Published: 2024-07-01 17:31:03

The data mixture for large language model pre-training significantly impacts
performance, yet how to determine an effective mixture remains unclear. We
propose RegMix to automatically identify a high-performing data mixture by
formulating it as a regression task. RegMix involves training a set of small
models with diverse data mixtures and fitting a regression model to predict
their performance given their respective mixtures. With the fitted regression
model, we simulate the top-ranked mixture and use it to train a large-scale
model with orders of magnitude more compute. To empirically validate RegMix, we
train 512 models with 1M parameters for 1B tokens of different mixtures to fit
the regression model and find the optimal mixture. Using this mixture we train
a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we
find performs best among 64 candidate 1B parameter models with other mixtures.
Further, our method demonstrates superior performance compared to human
selection and achieves results that match or surpass DoReMi, while utilizing
only 10% of the compute budget. Our experiments also show that (1) Data
mixtures significantly impact performance with single-task performance
variations of up to 14.6%; (2) Web corpora rather than data perceived as
high-quality like Wikipedia have the strongest positive correlation with
downstream performance; (3) Domains interact in complex ways often
contradicting common sense, thus automatic approaches like RegMix are needed;
(4) Data mixture effects transcend scaling laws, and our approach captures the
complexity by considering all domains together. Our code is available at
https://github.com/sail-sg/regmix.
DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
  Restoration Models
Authors: name: Chang-Han Yeh, name: Chin-Yang Lin, name: Zhixiang Wang, name: Chi-Wei Hsiao, name: Ting-Hsuan Chen, name: Yu-Lun Liu
Published: 2024-07-01 17:59:12

This paper introduces a method for zero-shot video restoration using
pre-trained image restoration diffusion models. Traditional video restoration
methods often need retraining for different settings and struggle with limited
generalization across various degradation types and datasets. Our approach uses
a hierarchical token merging strategy for keyframes and local frames, combined
with a hybrid correspondence mechanism that blends optical flow and
feature-based nearest neighbor matching (latent merging). We show that our
method not only achieves top performance in zero-shot video restoration but
also significantly surpasses trained models in generalization across diverse
datasets and extreme degradations (8times super-resolution and high-standard
deviation video denoising). We present evidence through quantitative metrics
and visual comparisons on various challenging datasets. Additionally, our
technique works with any 2D restoration diffusion model, offering a versatile
and powerful tool for video enhancement tasks without extensive retraining.
This research leads to more efficient and widely applicable video restoration
technologies, supporting advancements in fields that require high-quality video
output. See our project page for video results at
https://jimmycv07.github.io/DiffIR2VR_web/.
LiteSearch: Efficacious Tree Search for LLM
Authors: name: Ante Wang, name: Linfeng Song, name: Ye Tian, name: Baolin Peng, name: Dian Yu, name: Haitao Mi, name: Jinsong Su, name: Dong Yu
Published: 2024-06-29 05:14:04

Recent research suggests that tree search algorithms (e.g. Monte Carlo Tree
Search) can dramatically boost LLM performance on complex mathematical
reasoning tasks. However, they often require more than 10 times the
computational resources of greedy decoding due to wasteful search strategies,
making them difficult to be deployed in practical applications. This study
introduces a novel guided tree search algorithm with dynamic node selection and
node-level exploration budget (maximum number of children) calculation to
tackle this issue. By considering the search progress towards the final answer
(history) and the guidance from a value network (future) trained without any
step-wise annotations, our algorithm iteratively selects the most promising
tree node before expanding it within the boundaries of the allocated
computational budget. Experiments conducted on the GSM8K and TabMWP datasets
demonstrate that our approach not only offers competitive performance but also
enjoys significantly lower computational costs compared to baseline methods.
MIRAI: Evaluating LLM Agents for Event Forecasting
Authors: name: Chenchen Ye, name: Ziniu Hu, name: Yihe Deng, name: Zijie Huang, name: Mingyu Derek Ma, name: Yanqiao Zhu, name: Wei Wang
Published: 2024-07-01 12:22:46

Recent advancements in Large Language Models (LLMs) have empowered LLM agents
to autonomously collect world information, over which to conduct reasoning to
solve complex problems. Given this capability, increasing interests have been
put into employing LLM agents for predicting international events, which can
influence decision-making and shape policy development on an international
scale. Despite such a growing interest, there is a lack of a rigorous benchmark
of LLM agents' forecasting capability and reliability. To address this gap, we
introduce MIRAI, a novel benchmark designed to systematically evaluate LLM
agents as temporal forecasters in the context of international events. Our
benchmark features an agentic environment with tools for accessing an extensive
database of historical, structured events and textual news articles. We refine
the GDELT event database with careful cleaning and parsing to curate a series
of relational prediction tasks with varying forecasting horizons, assessing LLM
agents' abilities from short-term to long-term forecasting. We further
implement APIs to enable LLM agents to utilize different tools via a code-based
interface. In summary, MIRAI comprehensively evaluates the agents' capabilities
in three dimensions: 1) autonomously source and integrate critical information
from large global databases; 2) write codes using domain-specific APIs and
libraries for tool-use; and 3) jointly reason over historical knowledge from
diverse formats and time to accurately predict future events. Through
comprehensive benchmarking, we aim to establish a reliable framework for
assessing the capabilities of LLM agents in forecasting international events,
thereby contributing to the development of more accurate and trustworthy models
for international relation analysis.
E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
Authors: name: Sefik Emre Eskimez, name: Xiaofei Wang, name: Manthan Thakker, name: Canrun Li, name: Chung-Hsien Tsai, name: Zhen Xiao, name: Hemin Yang, name: Zirun Zhu, name: Min Tang, name: Xu Tan, name: Yanqing Liu, name: Sheng Zhao, name: Naoyuki Kanda
Published: 2024-06-26 01:38:37

This paper introduces Embarrassingly Easy Text-to-Speech (E2 TTS), a fully
non-autoregressive zero-shot text-to-speech system that offers human-level
naturalness and state-of-the-art speaker similarity and intelligibility. In the
E2 TTS framework, the text input is converted into a character sequence with
filler tokens. The flow-matching-based mel spectrogram generator is then
trained based on the audio infilling task. Unlike many previous works, it does
not require additional components (e.g., duration model, grapheme-to-phoneme)
or complex techniques (e.g., monotonic alignment search). Despite its
simplicity, E2 TTS achieves state-of-the-art zero-shot TTS capabilities that
are comparable to or surpass previous works, including Voicebox and
NaturalSpeech 3. The simplicity of E2 TTS also allows for flexibility in the
input representation. We propose several variants of E2 TTS to improve
usability during inference. See https://aka.ms/e2tts/ for demo samples.
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and
  Efficient Evaluation
Authors: name: Jinsheng Huang, name: Liang Chen, name: Taian Guo, name: Fu Zeng, name: Yusheng Zhao, name: Bohan Wu, name: Ye Yuan, name: Haozhe Zhao, name: Zhihui Guo, name: Yichi Zhang, name: Jingyang Yuan, name: Wei Ju, name: Luchen Liu, name: Tianyu Liu, name: Baobao Chang, name: Ming Zhang
Published: 2024-06-29 15:28:45

Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding
and reasoning abilities, often assessed through multiple-choice questions
(MCQs) that include an image, a question, and several options. However, many
benchmarks used for such evaluations suffer from systematic biases. Remarkably,
Large Language Models (LLMs) without any visual perception capabilities achieve
non-trivial performance, undermining the credibility of these evaluations. To
address this issue while maintaining the efficiency of MCQ evaluations, we
propose MMEvalPro, a benchmark designed to avoid Type-I errors through a
trilogy evaluation pipeline and more rigorous metrics. For each original
question from existing benchmarks, human annotators augment it by creating one
perception question and one knowledge anchor question through a meticulous
annotation process. MMEvalPro comprises 2,138 question triplets, totaling
6,414 distinct questions. Two-thirds of these questions are manually labeled
by human experts, while the rest are sourced from existing benchmarks (MMMU,
ScienceQA, and MathVista). Compared with the existing benchmarks, our
experiments with the latest LLMs and LMMs demonstrate that MMEvalPro is more
challenging (the best LMM lags behind human performance by 31.73%, compared
to an average gap of 8.03% in previous benchmarks) and more trustworthy (the
best LLM trails the best LMM by 23.09%, whereas the gap for previous
benchmarks is just 14.64%). Our in-depth analysis explains the reason for
the large performance gap and justifies the trustworthiness of evaluation,
underscoring its significant potential for advancing future research.
We-Math: Does Your Large Multimodal Model Achieve Human-like
  Mathematical Reasoning?
Authors: name: Runqi Qiao, name: Qiuna Tan, name: Guanting Dong, name: Minhui Wu, name: Chong Sun, name: Xiaoshuai Song, name: Zhuoma GongQue, name: Shanglin Lei, name: Zhe Wei, name: Miaoxuan Zhang, name: Runfeng Qiao, name: Yifan Zhang, name: Xiao Zong, name: Yida Xu, name: Muxi Diao, name: Zhimin Bao, name: Chen Li, name: Honggang Zhang
Published: 2024-07-01 13:39:08

Visual mathematical reasoning, as a fundamental visual reasoning ability, has
received widespread attention from the Large Multimodal Models (LMMs)
community. Existing benchmarks, such as MathVista and MathVerse, focus more on
the result-oriented performance but neglect the underlying principles in
knowledge acquisition and generalization. Inspired by human-like mathematical
reasoning, we introduce WE-MATH, the first benchmark specifically designed to
explore the problem-solving principles beyond end-to-end performance. We
meticulously collect and categorize 6.5K visual math problems, spanning 67
hierarchical knowledge concepts and five layers of knowledge granularity. We
decompose composite problems into sub-problems according to the required
knowledge concepts and introduce a novel four-dimensional metric, namely
Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
(CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
existing LMMs in visual mathematical reasoning and reveal a negative
correlation between solving steps and problem-specific performance. We confirm
the IK issue of LMMs can be effectively improved via knowledge augmentation
strategies. More notably, the primary challenge of GPT-4o has significantly
transitioned from IK to IG, establishing it as the first LMM advancing towards
the knowledge generalization stage. In contrast, other LMMs exhibit a marked
inclination towards Rote Memorization - they correctly solve composite problems
involving multiple knowledge concepts yet fail to answer sub-problems. We
anticipate that WE-MATH will open new pathways for advancements in visual
mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
available at https://github.com/We-Math/We-Math.
Iterative Nash Policy Optimization: Aligning LLMs with General
  Preferences via No-Regret Learning
Authors: name: Yuheng Zhang, name: Dian Yu, name: Baolin Peng, name: Linfeng Song, name: Ye Tian, name: Mingyue Huo, name: Nan Jiang, name: Haitao Mi, name: Dong Yu
Published: 2024-06-30 08:00:34

Reinforcement Learning with Human Feedback (RLHF) has achieved great success
in aligning large language models (LLMs) with human preferences. Prevalent RLHF
approaches are reward-based, following the Bradley-Terry (BT) model assumption,
which may not fully capture the complexity of human preferences. In this paper,
we explore RLHF under a general preference framework and approach it from a
game-theoretic perspective. Specifically, we formulate the problem as a
two-player game and propose a novel algorithm, iterative Nash policy
optimization (INPO). The key idea is to let the policy play against itself via
no-regret learning, thereby approximating the Nash policy. Unlike previous
methods, INPO bypasses the need for estimating the expected win rate for
individual responses, which typically incurs high computational or annotation
costs. Instead, we introduce a new loss objective that is directly minimized
over a preference dataset. We provide theoretical analysis for our approach and
demonstrate its effectiveness through experiments on various representative
benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
Arena-Hard, showing substantial improvement over the state-of-the-art iterative
algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
ablation study highlights the benefits of incorporating KL regularization for
response length control.
RaTEScore: A Metric for Radiology Report Generation
Authors: name: Weike Zhao, name: Chaoyi Wu, name: Xiaoman Zhang, name: Ya Zhang, name: Yanfeng Wang, name: Weidi Xie
Published: 2024-06-24 17:49:28

This paper introduces a novel, entity-aware metric, termed as Radiological
Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports
generated by AI models. RaTEScore emphasizes crucial medical entities such as
diagnostic outcomes and anatomical details, and is robust against complex
medical synonyms and sensitive to negation expressions. Technically, we
developed a comprehensive medical NER dataset, RaTE-NER, and trained an NER
model specifically for this purpose. This model enables the decomposition of
complex radiological reports into constituent medical entities. The metric
itself is derived by comparing the similarity of entity embeddings, obtained
from a language model, based on their types and relevance to clinical
significance. Our evaluations demonstrate that RaTEScore aligns more closely
with human preference than existing metrics, validated both on established
public benchmarks and our newly proposed RaTE-Eval benchmark.
Arboretum: A Large Multimodal Dataset Enabling AI for Biodiversity
Authors: name: Chih-Hsuan Yang, name: Benjamin Feuer, name: Zaki Jubery, name: Zi K. Deng, name: Andre Nakkab, name: Md Zahid Hasan, name: Shivani Chiranjeevi, name: Kelly Marshall, name: Nirmal Baishnab, name: Asheesh K Singh, name: Arti Singh, name: Soumik Sarkar, name: Nirav Merchant, name: Chinmay Hegde, name: Baskar Ganapathysubramanian
Published: 2024-06-25 17:09:54

We introduce Arboretum, the largest publicly accessible dataset designed to
advance AI for biodiversity applications. This dataset, curated from the
iNaturalist community science platform and vetted by domain experts to ensure
accuracy, includes 134.6 million images, surpassing existing datasets in scale
by an order of magnitude. The dataset encompasses image-language paired data
for a diverse set of species from birds (Aves), spiders/ticks/mites
(Arachnida), insects (Insecta), plants (Plantae), fungus/mushrooms (Fungi),
snails (Mollusca), and snakes/lizards (Reptilia), making it a valuable resource
for multimodal vision-language AI models for biodiversity assessment and
agriculture research. Each image is annotated with scientific names, taxonomic
details, and common names, enhancing the robustness of AI model training.
  We showcase the value of Arboretum by releasing a suite of CLIP models
trained using a subset of 40 million captioned images. We introduce several new
benchmarks for rigorous assessment, report accuracy for zero-shot learning, and
evaluations across life stages, rare species, confounding species, and various
levels of the taxonomic hierarchy.
  We anticipate that Arboretum will spur the development of AI models that can
enable a variety of digital tools ranging from pest control strategies, crop
monitoring, and worldwide biodiversity assessment and environmental
conservation. These advancements are critical for ensuring food security,
preserving ecosystems, and mitigating the impacts of climate change. Arboretum
is publicly available, easily accessible, and ready for immediate use.
  Please see the https://baskargroup.github.io/Arboretum/{project
website} for links to our data, models, and code.