Orionnnn/hub-stats huggingface dataset:
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal
  LLMs
Authors: name: Umberto Cappellazzo, name: Minsu Kim, name: Stavros Petridis
Published: 2025-03-09 00:02:10

Audio-Visual Speech Recognition (AVSR) leverages both audio and visual
modalities to enhance speech recognition robustness, particularly in noisy
environments. Recent advancements in Large Language Models (LLMs) have
demonstrated their effectiveness in speech recognition, including AVSR.
However, due to the significant length of speech representations, direct
integration with LLMs imposes substantial computational costs. Prior approaches
address this by compressing speech representations before feeding them into
LLMs. However, higher compression ratios often lead to performance degradation,
necessitating a trade-off between computational efficiency and recognition
accuracy. To address this challenge, we propose Llama-MTSK, the first
Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of
the audio-visual token allocation based on specific computational constraints
while preserving high performance. Our approach, inspired by Matryoshka
Representation Learning, encodes audio-visual representations at multiple
granularities within a single model, eliminating the need to train separate
models for different compression levels. Moreover, to efficiently fine-tune the
LLM, we introduce three LoRA-based Matryoshka strategies using global and
scale-specific LoRA modules. Extensive evaluations on the two largest AVSR
datasets demonstrate that Llama-MTSK achieves state-of-the-art results,
matching or surpassing models trained independently at fixed compression
levels.
WritingBench: A Comprehensive Benchmark for Generative Writing
Authors: name: Yuning Wu, name: Jiahao Mei, name: Ming Yan, name: Chenliang Li, name: SHaopeng Lai, name: Yuran Ren, name: Zijia Wang, name: Ji Zhang, name: Mengyue Wu, name: Qin Jin, name: Fei Huang
Published: 2025-03-07 08:56:20

Recent advancements in large language models (LLMs) have significantly
enhanced text generation capabilities, yet evaluating their performance in
generative writing remains a challenge. Existing benchmarks primarily focus on
generic text generation or limited in writing tasks, failing to capture the
diverse requirements of high-quality written contents across various domains.
To bridge this gap, we present WritingBench, a comprehensive benchmark designed
to evaluate LLMs across 6 core writing domains and 100 subdomains, encompassing
creative, persuasive, informative, and technical writing. We further propose a
query-dependent evaluation framework that empowers LLMs to dynamically generate
instance-specific assessment criteria. This framework is complemented by a
fine-tuned critic model for criteria-aware scoring, enabling evaluations in
style, format and length. The framework's validity is further demonstrated by
its data curation capability, which enables 7B-parameter models to approach
state-of-the-art (SOTA) performance. We open-source the benchmark, along with
evaluation tools and modular framework components, to advance the development
of LLMs in writing.
Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent
  Spaces
Authors: name: Souhail Hadgi, name: Luca Moschella, name: Andrea Santilli, name: Diego Gomez, name: Qixing Huang, name: Emanuele Rodolà, name: Simone Melzi, name: Maks Ovsjanikov
Published: 2025-03-07 09:51:56

Recent works have shown that, when trained at scale, uni-modal 2D vision and
text encoders converge to learned features that share remarkable structural
properties, despite arising from different representations. However, the role
of 3D encoders with respect to other modalities remains unexplored.
Furthermore, existing 3D foundation models that leverage large datasets are
typically trained with explicit alignment objectives with respect to frozen
encoders from other representations. In this work, we investigate the
possibility of a posteriori alignment of representations obtained from
uni-modal 3D encoders compared to text-based feature spaces. We show that naive
post-training feature alignment of uni-modal text and 3D encoders results in
limited performance. We then focus on extracting subspaces of the corresponding
feature spaces and discover that by projecting learned representations onto
well-chosen lower-dimensional subspaces the quality of alignment becomes
significantly higher, leading to improved accuracy on matching and retrieval
tasks. Our analysis further sheds light on the nature of these shared
subspaces, which roughly separate between semantic and geometric data
representations. Overall, ours is the first work that helps to establish a
baseline for post-training alignment of 3D uni-modal and text feature spaces,
and helps to highlight both the shared and unique properties of 3D data
compared to other representations.
RePO: ReLU-based Preference Optimization
Authors: name: Junkang Wu, name: Kexin Huang, name: Xue Wang, name: Jinyang Gao, name: Bolin Ding, name: Jiancan Wu, name: Xiangnan He, name: Xiang Wang
Published: 2025-03-10 15:11:07

Aligning large language models (LLMs) with human preferences is critical for
real-world deployment, yet existing methods like RLHF face computational and
stability challenges. While DPO establishes an offline paradigm with single
hyperparameter beta, subsequent methods like SimPO reintroduce complexity
through dual parameters (beta, gamma). We propose {ReLU-based Preference
Optimization (RePO)}, a streamlined algorithm that eliminates beta via two
advances: (1) retaining SimPO's reference-free margins but removing beta
through gradient analysis, and (2) adopting a ReLU-based max-margin loss that
naturally filters trivial pairs. Theoretically, RePO is characterized as
SimPO's limiting case (beta to infty), where the logistic weighting
collapses to binary thresholding, forming a convex envelope of the 0-1 loss.
Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO
and SimPO across multiple base models, requiring only one hyperparameter to
tune.
YOLOE: Real-Time Seeing Anything
Authors: name: Ao Wang, name: Lihao Liu, name: Hui Chen, name: Zijia Lin, name: Jungong Han, name: Guiguang Ding
Published: 2025-03-10 15:42:59

Object detection and segmentation are widely employed in computer vision
applications, yet conventional models like YOLO series, while efficient and
accurate, are limited by predefined categories, hindering adaptability in open
scenarios. Recent open-set methods leverage text prompts, visual cues, or
prompt-free paradigm to overcome this, but often compromise between performance
and efficiency due to high computational demands or deployment complexity. In
this work, we introduce YOLOE, which integrates detection and segmentation
across diverse open prompt mechanisms within a single highly efficient model,
achieving real-time seeing anything. For text prompts, we propose
Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines
pretrained textual embeddings via a re-parameterizable lightweight auxiliary
network and enhances visual-textual alignment with zero inference and
transferring overhead. For visual prompts, we present Semantic-Activated Visual
Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches
to bring improved visual embedding and accuracy with minimal complexity. For
prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy.
It utilizes a built-in large vocabulary and specialized embedding to identify
all objects, avoiding costly language model dependency. Extensive experiments
show YOLOE's exceptional zero-shot performance and transferability with high
inference efficiency and low training cost. Notably, on LVIS, with 3times
less training cost and 1.4times inference speedup, YOLOE-v8-S surpasses
YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6
AP^b and 0.4 AP^m gains over closed-set YOLOv8-L with nearly 4times less
training time. Code and models are available at
https://github.com/THU-MIG/yoloe.
MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for
  Complex Medical Reasoning
Authors: name: Xiangru Tang, name: Daniel Shao, name: Jiwoong Sohn, name: Jiapeng Chen, name: Jiayi Zhang, name: Jinyu Xiang, name: Fang Wu, name: Yilun Zhao, name: Chenglin Wu, name: Wenqi Shi, name: Arman Cohan, name: Mark Gerstein
Published: 2025-03-10 15:38:44

Large Language Models (LLMs) have shown impressive performance on existing
medical question-answering benchmarks. This high performance makes it
increasingly difficult to meaningfully evaluate and differentiate advanced
methods. We present MedAgentsBench, a benchmark that focuses on challenging
medical questions requiring multi-step clinical reasoning, diagnosis
formulation, and treatment planning-scenarios where current models still
struggle despite their strong performance on standard tests. Drawing from seven
established medical datasets, our benchmark addresses three key limitations in
existing evaluations: (1) the prevalence of straightforward questions where
even base models achieve high performance, (2) inconsistent sampling and
evaluation protocols across studies, and (3) lack of systematic analysis of the
interplay between performance, cost, and inference time. Through experiments
with various base models and reasoning methods, we demonstrate that the latest
thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in
complex medical reasoning tasks. Additionally, advanced search-based agent
methods offer promising performance-to-cost ratios compared to traditional
approaches. Our analysis reveals substantial performance gaps between model
families on complex questions and identifies optimal model selections for
different computational constraints. Our benchmark and evaluation framework are
publicly available at https://github.com/gersteinlab/medagents-benchmark.
Feature-Level Insights into Artificial Text Detection with Sparse
  Autoencoders
Authors: name: Kristian Kuznetsov, name: Laida Kushnareva, name: Polina Druzhinina, name: Anton Razzhigaev, name: Anastasia Voznyuk, name: Irina Piontkovskaya, name: Evgeny Burnaev, name: Serguei Barannikov
Published: 2025-03-05 15:33:52

Artificial Text Detection (ATD) is becoming increasingly important with the
rise of advanced Large Language Models (LLMs). Despite numerous efforts, no
single algorithm performs consistently well across different types of unseen
text or guarantees effective generalization to new LLMs. Interpretability plays
a crucial role in achieving this goal. In this study, we enhance ATD
interpretability by using Sparse Autoencoders (SAE) to extract features from
Gemma-2-2b residual stream. We identify both interpretable and efficient
features, analyzing their semantics and relevance through domain- and
model-specific statistics, a steering approach, and manual or LLM-based
interpretation. Our methods offer valuable insights into how texts from various
models differ from human-written content. We show that modern LLMs have a
distinct writing style, especially in information-dense domains, even though
they can produce human-like outputs with personalized prompts.
Should VLMs be Pre-trained with Image Data?
Authors: name: Sedrick Keh, name: Jean Mercat, name: Samir Yitzhak Gadre, name: Kushal Arora, name: Igor Vasiljevic, name: Benjamin Burchfiel, name: Shuran Song, name: Russ Tedrake, name: Thomas Kollar, name: Ludwig Schmidt, name: Achal Dave
Published: 2025-03-10 17:58:19

Pre-trained LLMs that are further trained with image data perform well on
vision-language tasks. While adding images during a second training phase
effectively unlocks this capability, it is unclear how much of a gain or loss
this two-step pipeline gives over VLMs which integrate images earlier into the
training process. To investigate this, we train models spanning various
datasets, scales, image-text ratios, and amount of pre-training done before
introducing vision tokens. We then fine-tune these models and evaluate their
downstream performance on a suite of vision-language and text-only tasks. We
find that pre-training with a mixture of image and text data allows models to
perform better on vision-language tasks while maintaining strong performance on
text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B
model, introducing visual tokens 80% of the way through pre-training results in
a 2% average improvement over introducing visual tokens to a fully pre-trained
model.
ProBench: Judging Multimodal Foundation Models on Open-ended
  Multi-domain Expert Tasks
Authors: name: Yan Yang, name: Dongxu Li, name: Haoning Wu, name: Bei Chen, name: Liu Liu, name: Liyuan Pan, name: Junnan Li
Published: 2025-03-10 03:29:18

Solving expert-level multimodal tasks is a key milestone towards general
intelligence. As the capabilities of multimodal large language models (MLLMs)
continue to improve, evaluation of such advanced multimodal intelligence
becomes necessary yet challenging. In this work, we introduce ProBench, a
benchmark of open-ended user queries that require professional expertise and
advanced reasoning. ProBench consists of 4,000 high-quality samples
independently submitted by professionals based on their daily productivity
demands. It spans across 10 fields and 56 sub-fields, including science, arts,
humanities, coding, mathematics, and creative writing. Experimentally, we
evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
that although the best open-source models rival the proprietary ones, ProBench
presents significant challenges in visual perception, textual understanding,
domain knowledge and advanced reasoning, thus providing valuable directions for
future multimodal AI research efforts.
SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and
  Multi-dimensional Evaluation for Automated Survey Writing
Authors: name: Xiangchao Yan, name: Shiyang Feng, name: Jiakang Yuan, name: Renqiu Xia, name: Bin Wang, name: Bo Zhang, name: Lei Bai
Published: 2025-03-06 17:15:48

Survey paper plays a crucial role in scientific research, especially given
the rapid growth of research publications. Recently, researchers have begun
using LLMs to automate survey generation for better efficiency. However, the
quality gap between LLM-generated surveys and those written by human remains
significant, particularly in terms of outline quality and citation accuracy. To
close these gaps, we introduce SurveyForge, which first generates the outline
by analyzing the logical structure of human-written outlines and referring to
the retrieved domain-related articles. Subsequently, leveraging high-quality
papers retrieved from memory by our scholar navigation agent, SurveyForge can
automatically generate and refine the content of the generated article.
Moreover, to achieve a comprehensive evaluation, we construct SurveyBench,
which includes 100 human-written survey papers for win-rate comparison and
assesses AI-generated survey papers across three dimensions: reference,
outline, and content quality. Experiments demonstrate that SurveyForge can
outperform previous works such as AutoSurvey.
AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via
  Reinforcement Learning and Reasoning
Authors: name: Bo Jiang, name: Shaoyu Chen, name: Qian Zhang, name: Wenyu Liu, name: Xinggang Wang
Published: 2025-03-10 17:59:42

OpenAI o1 and DeepSeek R1 achieve or even surpass human expert-level
performance in complex domains like mathematics and science, with reinforcement
learning (RL) and reasoning playing a crucial role. In autonomous driving,
recent end-to-end models have greatly improved planning performance but still
struggle with long-tailed problems due to limited common sense and reasoning
abilities. Some studies integrate vision-language models (VLMs) into autonomous
driving, but they typically rely on pre-trained models with simple supervised
fine-tuning (SFT) on driving data, without further exploration of training
strategies or optimizations specifically tailored for planning. In this paper,
we propose AlphaDrive, a RL and reasoning framework for VLMs in autonomous
driving. AlphaDrive introduces four GRPO-based RL rewards tailored for planning
and employs a two-stage planning reasoning training strategy that combines SFT
with RL. As a result, AlphaDrive significantly improves both planning
performance and training efficiency compared to using only SFT or without
reasoning. Moreover, we are also excited to discover that, following RL
training, AlphaDrive exhibits some emergent multimodal planning capabilities,
which is critical for improving driving safety and efficiency. To the best of
our knowledge, AlphaDrive is the first to integrate GRPO-based RL with planning
reasoning into autonomous driving. Code will be released to facilitate future
research.
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large
  Language Models
Authors: name: Wenxuan Huang, name: Bohan Jia, name: Zijie Zhai, name: Shaosheng Cao, name: Zheyu Ye, name: Fei Zhao, name: Yao Hu, name: Shaohui Lin
Published: 2025-03-09 20:06:45

DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning
capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by
this breakthrough, we explore how RL can be utilized to enhance the reasoning
capability of MLLMs. However, direct training with RL struggles to activate
complex reasoning capabilities such as questioning and reflection in MLLMs, due
to the absence of substantial high-quality multimodal reasoning data. To
address this issue, we propose the reasoning MLLM, Vision-R1, to improve
multimodal reasoning capability. Specifically, we first construct a
high-quality multimodal CoT dataset without human annotations by leveraging an
existing MLLM and DeepSeek-R1 through modality bridging and data filtering to
obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as
cold-start initialization data for Vision-R1. To mitigate the optimization
challenges caused by overthinking after cold start, we propose Progressive
Thinking Suppression Training (PTST) strategy and employ Group Relative Policy
Optimization (GRPO) with the hard formatting result reward function to
gradually refine the model's ability to learn correct and complex reasoning
processes on a 10K multimodal math dataset. Comprehensive experiments show our
model achieves an average improvement of sim6% across various multimodal
math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely
used MathVista benchmark, which is only 0.4% lower than the leading reasoning
model, OpenAI O1. The datasets and code will be released in:
https://github.com/Osilly/Vision-R1 .
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
Authors: name: Jongwoo Ko, name: Tianyi Chen, name: Sungnyun Kim, name: Tianyu Ding, name: Luming Liang, name: Ilya Zharkov, name: Se-Young Yun
Published: 2025-03-10 08:51:32

Despite the success of distillation in large language models (LLMs), most
prior work applies identical loss functions to both teacher- and
student-generated data. These strategies overlook the synergy between loss
formulations and data types, leading to a suboptimal performance boost in
student models. To address this, we propose DistiLLM-2, a contrastive approach
that simultaneously increases the likelihood of teacher responses and decreases
that of student responses by harnessing this synergy. Our extensive experiments
show that DistiLLM-2 not only builds high-performing student models across a
wide range of tasks, including instruction-following and code generation, but
also supports diverse applications, such as preference alignment and
vision-language extensions. These findings highlight the potential of a
contrastive approach to enhance the efficacy of LLM distillation by effectively
aligning teacher and student models across varied data types.
DreamRelation: Relation-Centric Video Customization
Authors: name: Yujie Wei, name: Shiwei Zhang, name: Hangjie Yuan, name: Biao Gong, name: Longxiang Tang, name: Xiang Wang, name: Haonan Qiu, name: Hengjia Li, name: Shuai Tan, name: Yingya Zhang, name: Hongming Shan
Published: 2025-03-10 17:58:03

Relational video customization refers to the creation of personalized videos
that depict user-specified relations between two subjects, a crucial task for
comprehending real-world visual content. While existing methods can personalize
subject appearances and motions, they still struggle with complex relational
video customization, where precise relational modeling and high generalization
across subject categories are essential. The primary challenge arises from the
intricate spatial arrangements, layout variations, and nuanced temporal
dynamics inherent in relations; consequently, current models tend to
overemphasize irrelevant visual details rather than capturing meaningful
interactions. To address these challenges, we propose DreamRelation, a novel
approach that personalizes relations through a small set of exemplar videos,
leveraging two key components: Relational Decoupling Learning and Relational
Dynamics Enhancement. First, in Relational Decoupling Learning, we disentangle
relations from subject appearances using relation LoRA triplet and hybrid mask
training strategy, ensuring better generalization across diverse relationships.
Furthermore, we determine the optimal design of relation LoRA triplet by
analyzing the distinct roles of the query, key, and value features within
MM-DiT's attention mechanism, making DreamRelation the first relational video
generation framework with explainable components. Second, in Relational
Dynamics Enhancement, we introduce space-time relational contrastive loss,
which prioritizes relational dynamics while minimizing the reliance on detailed
subject appearances. Extensive experiments demonstrate that DreamRelation
outperforms state-of-the-art methods in relational video customization. Code
and models will be made publicly available.
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue
  Learning
Authors: name: Jiazheng Liu, name: Sipeng Zheng, name: Börje F. Karlsson, name: Zongqing Lu
Published: 2025-03-10 07:32:53

Multimodal large language models (MLLMs), built on large-scale pre-trained
vision towers and language models, have shown great capabilities in multimodal
understanding. However, most existing MLLMs are trained on single-turn vision
question-answering tasks, which do not accurately reflect real-world human
conversations. In this paper, we introduce MMDiag, a multi-turn multimodal
dialogue dataset. This dataset is collaboratively generated through
deliberately designed rules and GPT assistance, featuring strong correlations
between questions, between questions and images, and among different image
regions; thus aligning more closely with real-world scenarios. MMDiag serves as
a strong benchmark for multi-turn multimodal dialogue learning and brings more
challenges to the grounding and reasoning capabilities of MLLMs. Further,
inspired by human vision processing, we present DiagNote, an MLLM equipped with
multimodal grounding and reasoning capabilities. DiagNote consists of two
modules (Deliberate and Gaze) interacting with each other to perform
Chain-of-Thought and annotations respectively, throughout multi-turn dialogues.
We empirically demonstrate the advantages of DiagNote in both grounding and
jointly processing and reasoning with vision and language information over
existing MLLMs.