Orionnnn/hub-stats huggingface dataset:
Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting
  LLMs Across Languages and Resources
Authors: name: Zihao Li, name: Shaoxiong Ji, name: Hengyu Luo, name: Jörg Tiedemann
Published: 2025-04-05 12:10:55

Large Language Models (LLMs) exhibit significant disparities in performance
across languages, primarily benefiting high-resource languages while
marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as
a promising approach to address this imbalance, although the relative
effectiveness of monolingual, bilingual, and code-augmented data strategies
remains unclear. This study systematically evaluates 36 CPT configurations
involving three multilingual base models, across 30+ languages categorized as
altruistic, selfish, and stagnant, spanning various resource levels. Our
findings reveal three major insights: (1) Bilingual CPT improves multilingual
classification but often causes language mixing issues during generation. (2)
Including programming code data during CPT consistently enhances multilingual
classification accuracy, particularly benefiting low-resource languages, but
introduces a trade-off by slightly degrading generation quality. (3) Contrary
to prior work, we observe substantial deviations from language classifications
according to their impact on cross-lingual transfer: Languages classified as
altruistic often negatively affect related languages, selfish languages show
conditional and configuration-dependent behavior, and stagnant languages
demonstrate surprising adaptability under certain CPT conditions. These nuanced
interactions emphasize the complexity of multilingual representation learning,
underscoring the importance of systematic studies on generalizable language
classification to inform future multilingual CPT strategies.
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
  Language Models
Authors: name: Hengyu Luo, name: Zihao Li, name: Joseph Attieh, name: Sawal Devkota, name: Ona de Gibert, name: Shaoxiong Ji, name: Peiqin Lin, name: Bhavani Sai Praneeth Varma Mantina, name: Ananda Sreenidhi, name: Raúl Vázquez, name: Mengjie Wang, name: Samea Yusofi, name: Jörg Tiedemann
Published: 2025-04-05 12:30:58

Large language models (LLMs) are advancing at an unprecedented pace globally,
with regions increasingly adopting these models for applications in their
primary language. Evaluation of these models in diverse linguistic
environments, especially in low-resource languages, has become a major
challenge for academia and industry. Existing evaluation frameworks are
disproportionately focused on English and a handful of high-resource languages,
thereby overlooking the realistic performance of LLMs in multilingual and
lower-resource scenarios. To address this gap, we introduce GlotEval, a
lightweight framework designed for massively multilingual evaluation.
Supporting seven key tasks (machine translation, text classification,
summarization, open-ended generation, reading comprehension, sequence labeling,
and intrinsic evaluation), spanning over dozens to hundreds of languages,
GlotEval highlights consistent multilingual benchmarking, language-specific
prompt templates, and non-English-centric machine translation. This enables a
precise diagnosis of model strengths and weaknesses in diverse linguistic
contexts. A multilingual translation case study demonstrates GlotEval's
applicability for multilingual and language-specific evaluations.
VAPO: Efficient and Reliable Reinforcement Learning for Advanced
  Reasoning Tasks
Authors: name: YuYue, name: Yufeng Yuan, name: Qiying Yu, name: Xiaochen Zuo, name: Ruofei Zhu, name: Wenyuan Xu, name: Jiaze Chen, name: Chengyi Wang, name: TianTian Fan, name: Zhengyin Du, name: Xiangpeng Wei, name: Gaohong Liu, name: Juncai Liu, name: Lingjun Liu, name: Haibin Lin, name: Zhiqi Lin, name: Bole Ma, name: Chi Zhang, name: Mofan Zhang, name: Wang Zhang, name: Hang Zhu, name: Ru Zhang, name: Xin Liu, name: Mingxuan Wang, name: Yonghui Wu, name: Lin Yan
Published: 2025-04-07 14:21:11

We present VAPO, Value-based Augmented Proximal Policy Optimization framework
for reasoning models., a novel framework tailored for reasoning models within
the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
Qwen 32B pre-trained model, attains a state-of-the-art score of
60.4. In direct comparison under identical experimental settings,
VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
and DAPO by more than 10 points. The training process of VAPO stands out for
its stability and efficiency. It reaches state-of-the-art performance within a
mere 5,000 steps. Moreover, across multiple independent runs, no training
crashes occur, underscoring its reliability. This research delves into long
chain-of-thought (long-CoT) reasoning using a value-based reinforcement
learning framework. We pinpoint three key challenges that plague value-based
methods: value model bias, the presence of heterogeneous sequence lengths, and
the sparsity of reward signals. Through systematic design, VAPO offers an
integrated solution that effectively alleviates these challenges, enabling
enhanced performance in long-CoT reasoning tasks.
Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language
  Models for Domain-Generalized Semantic Segmentation
Authors: name: Xin Zhang, name: Robby T. Tan
Published: 2025-04-04 05:44:45

Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained
traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong
generalization capabilities. However, existing DGSS methods often rely
exclusively on either VFMs or VLMs, overlooking their complementary strengths.
VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g.,
CLIP) provide robust text alignment but struggle with coarse granularity.
Despite their complementary strengths, effectively integrating VFMs and VLMs
with attention mechanisms is challenging, as the increased patch tokens
complicate long-sequence modeling. To address this, we propose MFuser, a novel
Mamba-based fusion framework that efficiently combines the strengths of VFMs
and VLMs while maintaining linear scalability in sequence length. MFuser
consists of two key components: MVFuser, which acts as a co-adapter to jointly
fine-tune the two models by capturing both sequential and spatial dynamics; and
MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by
incorporating image priors. Our approach achieves precise feature locality and
strong text alignment without incurring significant computational overhead.
Extensive experiments demonstrate that MFuser significantly outperforms
state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and
71.87 mIoU on real-to-real benchmarks. The code is available at
https://github.com/devinxzhang/MFuser.
DiaTool-DPO: Multi-Turn Direct Preference Optimization for
  Tool-Augmented Large Language Models
Authors: name: Sunghee Jung, name: Donghun Lee, name: Shinbok Lee, name: Gaeun Seo, name: Daniel Lee, name: Byeongil Ko, name: Junrae Cho, name: Kihyun Kim, name: Eunggyun Kim, name: Myeongcheol Shin
Published: 2025-04-02 05:47:28

Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in
real-world applications, but face challenges in handling incomplete queries and
out-of-scope requests. While existing approaches rely mainly on Supervised
Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method
that enhances TA-LLM's dialogue capabilities through Direct Preference
Optimization. We model TA-LLM interactions as a Markov Decision Process with 5
distinct dialogue states and categorize user queries into 3 types based on
their state transition trajectories. We automatically construct paired
trajectory datasets of correct and incorrect dialogue flows and introduce a
specialized objective loss for dialogue control. Our comprehensive evaluation
demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in
information gathering, 91% in tool call rejection) with substantial
improvements over baseline (44% and 9.6% respectively) while maintaining core
functionality. Our approach opens new possibilities for developing TA-LLMs that
can handle diverse real-world scenarios without requiring additional expert
demonstrations or human labeling.
SmolVLM: Redefining small and efficient multimodal models
Authors: name: Andrés Marafioti, name: Orr Zohar, name: Miquel Farré, name: Merve Noyan, name: Elie Bakouch, name: Pedro Cuenca, name: Cyril Zakka, name: Loubna Ben Allal, name: Anton Lozhkov, name: Nouamane Tazi, name: Vaibhav Srivastav, name: Joshua Lochner, name: Hugo Larcher, name: Mathieu Morlon, name: Lewis Tunstall, name: Leandro von Werra, name: Thomas Wolf
Published: 2025-04-07 17:58:57

Large Vision-Language Models (VLMs) deliver exceptional performance but
require significant computational resources, limiting their deployment on
mobile and edge devices. Smaller VLMs typically mirror design choices of larger
models, such as extensive image tokenization, leading to inefficient GPU memory
usage and constrained practicality for on-device applications.
  We introduce SmolVLM, a series of compact multimodal models specifically
engineered for resource-efficient inference. We systematically explore
architectural configurations, tokenization strategies, and data curation
optimized for low computational overhead. Through this, we identify key design
choices that yield substantial performance gains on image and video tasks with
minimal memory footprints.
  Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during
inference and outperforms the 300-times larger Idefics-80B model, despite an
18-month development gap. Our largest model, at 2.2B parameters, rivals
state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend
beyond static images, demonstrating robust video comprehension capabilities.
  Our results emphasize that strategic architectural optimizations, aggressive
yet efficient tokenization, and carefully curated training data significantly
enhance multimodal performance, facilitating practical, energy-efficient
deployments at significantly smaller scales.
T1: Tool-integrated Self-verification for Test-time Compute Scaling in
  Small Language Models
Authors: name: Minki Kang, name: Jongwon Jeong, name: Jaewoong Cho
Published: 2025-04-07 04:01:17

Recent studies have demonstrated that test-time compute scaling effectively
improves the performance of small language models (sLMs). However, prior
research has mainly examined test-time compute scaling with an additional
larger model as a verifier, leaving self-verification by sLMs underexplored. In
this work, we investigate whether sLMs can reliably self-verify their outputs
under test-time scaling. We find that even with knowledge distillation from
larger verifiers, sLMs struggle with verification tasks requiring memorization,
such as numerical calculations and fact-checking. To address this limitation,
we propose Tool-integrated self-verification (T1), which delegates
memorization-heavy verification steps to external tools, such as a code
interpreter. Our theoretical analysis shows that tool integration reduces
memorization demands and improves test-time scaling performance. Experiments on
the MATH benchmark demonstrate that, with T1, a Llama-3.2 1B model under
test-time scaling outperforms the significantly larger Llama-3.1 8B model.
Moreover, T1 generalizes effectively to both mathematical (MATH500) and
multi-domain knowledge-intensive tasks (MMLU-Pro). Our findings highlight the
potential of tool integration to substantially improve the self-verification
abilities of sLMs.
Clinical ModernBERT: An efficient and long context encoder for
  biomedical text
Authors: name: Simon A. Lee, name: Anthony Wu, name: Jeffrey N. Chiang
Published: 2025-04-04 22:14:12

We introduce Clinical ModernBERT, a transformer based encoder pretrained on
large scale biomedical literature, clinical notes, and medical ontologies,
incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
their textual descriptions. Building on ModernBERT the current state of the art
natural language text encoder featuring architectural upgrades such as rotary
positional embeddings (RoPE), Flash Attention, and extended context length up
to 8,192 tokens our model adapts these innovations specifically for biomedical
and clinical domains. Clinical ModernBERT excels at producing semantically rich
representations tailored for long context tasks. We validate this both by
analyzing its pretrained weights and through empirical evaluation on a
comprehensive suite of clinical NLP benchmarks.
BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose
  Estimation
Authors: name: Van Nguyen Nguyen, name: Stephen Tyree, name: Andrew Guo, name: Mederic Fourmy, name: Anas Gouda, name: Taeyeop Lee, name: Sungphill Moon, name: Hyeontae Son, name: Lukas Ranftl, name: Jonathan Tremblay, name: Eric Brachmann, name: Bertram Drost, name: Vincent Lepetit, name: Carsten Rother, name: Stan Birchfield, name: Jiri Matas, name: Yann Labbe, name: Martin Sundermeyer, name: Tomas Hodan
Published: 2025-04-03 17:55:19

We present the evaluation methodology, datasets and results of the BOP
Challenge 2024, the sixth in a series of public competitions organized to
capture the state of the art in 6D object pose estimation and related tasks. In
2024, our goal was to transition BOP from lab-like setups to real-world
scenarios. First, we introduced new model-free tasks, where no 3D object models
are available and methods need to onboard objects just from provided reference
videos. Second, we defined a new, more practical 6D object detection task where
identities of objects visible in a test image are not provided as input. Third,
we introduced new BOP-H3 datasets recorded with high-resolution sensors and
AR/VR headsets, closely resembling real-world scenarios. BOP-H3 include 3D
models and onboarding videos to support both model-based and model-free tasks.
Participants competed on seven challenge tracks, each defined by a task, object
onboarding setup, and dataset group. Notably, the best 2024 method for
model-based 6D localization of unseen objects (FreeZeV2.1) achieves 22% higher
accuracy on BOP-Classic-Core than the best 2023 method (GenFlow), and is only
4% behind the best 2023 method for seen objects (GPose2023) although being
significantly slower (24.9 vs 2.7s per image). A more practical 2024 method for
this task is Co-op which takes only 0.8s per image and is 25X faster and 13%
more accurate than GenFlow. Methods have a similar ranking on 6D detection as
on 6D localization but higher run time. On model-based 2D detection of unseen
objects, the best 2024 method (MUSE) achieves 21% relative improvement compared
to the best 2023 method (CNOS). However, the 2D detection accuracy for unseen
objects is still noticealy (-53%) behind the accuracy for seen objects
(GDet2023). The online evaluation system stays open and is available at
http://bop.felk.cvut.cz/
LiveVQA: Live Visual Knowledge Seeking
Authors: name: Mingyang Fu, name: Yuyang Peng, name: Benlin Liu, name: Yao Wan, name: Dongping Chen
Published: 2025-04-07 17:39:31

We introduce LiveVQA, an automatically collected dataset of latest visual
knowledge from the Internet with synthesized VQA problems. LiveVQA consists of
3,602 single- and multi-hop visual questions from 6 news websites across 14
news categories, featuring high-quality image-text coherence and authentic
information. Our evaluation across 15 MLLMs (e.g., GPT-4o, Gemma-3, and
Qwen-2.5-VL family) demonstrates that stronger models perform better overall,
with advanced visual reasoning capabilities proving crucial for complex
multi-hop questions. Despite excellent performance on textual problems, models
with tools like search engines still show significant gaps when addressing
visual questions requiring latest visual knowledge, highlighting important
areas for future research.
Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning
  Models
Authors: name: Ruikang Liu, name: Yuxuan Sun, name: Manyi Zhang, name: Haoli Bai, name: Xianzhi Yu, name: Tiezheng Yu, name: Chun Yuan, name: Lu Hou
Published: 2025-04-07 08:22:45

Recent advancements in reasoning language models have demonstrated remarkable
performance in complex tasks, but their extended chain-of-thought reasoning
process increases inference overhead. While quantization has been widely
adopted to reduce the inference cost of large language models, its impact on
reasoning models remains understudied. In this study, we conduct the first
systematic study on quantized reasoning models, evaluating the open-sourced
DeepSeek-R1-Distilled Qwen and LLaMA families ranging from 1.5B to 70B
parameters, and QwQ-32B. Our investigation covers weight, KV cache, and
activation quantization using state-of-the-art algorithms at varying
bit-widths, with extensive evaluation across mathematical (AIME, MATH-500),
scientific (GPQA), and programming (LiveCodeBench) reasoning benchmarks. Our
findings reveal that while lossless quantization can be achieved with W8A8 or
W4A16 quantization, lower bit-widths introduce significant accuracy risks. We
further identify model size, model origin, and task difficulty as critical
determinants of performance. Contrary to expectations, quantized models do not
exhibit increased output lengths. In addition, strategically scaling the model
sizes or reasoning steps can effectively enhance the performance. All quantized
models and codes will be open-sourced in
https://github.com/ruikangliu/Quantized-Reasoning-Models.
JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language
  Model
Authors: name: Yi Nian, name: Shenzhe Zhu, name: Yuehan Qin, name: Li Li, name: Ziyi Wang, name: Chaowei Xiao, name: Yue Zhao
Published: 2025-04-03 05:00:28

Multimodal large language models (MLLMs) excel in vision-language tasks but
also pose significant risks of generating harmful content, particularly through
jailbreak attacks. Jailbreak attacks refer to intentional manipulations that
bypass safety mechanisms in models, leading to the generation of inappropriate
or unsafe content. Detecting such attacks is critical to ensuring the
responsible deployment of MLLMs. Existing jailbreak detection methods face
three primary challenges: (1) Many rely on model hidden states or gradients,
limiting their applicability to white-box models, where the internal workings
of the model are accessible; (2) They involve high computational overhead from
uncertainty-based analysis, which limits real-time detection, and (3) They
require fully labeled harmful datasets, which are often scarce in real-world
settings. To address these issues, we introduce a test-time adaptive framework
called JAILDAM. Our method leverages a memory-based approach guided by
policy-driven unsafe knowledge representations, eliminating the need for
explicit exposure to harmful data. By dynamically updating unsafe knowledge
during test-time, our framework improves generalization to unseen jailbreak
strategies while maintaining efficiency. Experiments on multiple VLM jailbreak
benchmarks demonstrate that JAILDAM delivers state-of-the-art performance in
harmful content detection, improving both accuracy and speed.
Concept Lancet: Image Editing with Compositional Representation
  Transplant
Authors: name: Jinqi Luo, name: Tianjiao Ding, name: Kwan Ho Ryan Chan, name: Hancheng Min, name: Chris Callison-Burch, name: René Vidal
Published: 2025-04-03 17:59:58

Diffusion models are widely used for image editing tasks. Existing editing
methods often design a representation manipulation procedure by curating an
edit direction in the text embedding or score space. However, such a procedure
faces a key challenge: overestimating the edit strength harms visual
consistency while underestimating it fails the editing task. Notably, each
source image may require a different editing strength, and it is costly to
search for an appropriate strength via trial-and-error. To address this
challenge, we propose Concept Lancet (CoLan), a zero-shot plug-and-play
framework for principled representation manipulation in diffusion-based image
editing. At inference time, we decompose the source input in the latent (text
embedding or diffusion score) space as a sparse linear combination of the
representations of the collected visual concepts. This allows us to accurately
estimate the presence of concepts in each image, which informs the edit. Based
on the editing task (replace/add/remove), we perform a customized concept
transplant process to impose the corresponding editing direction. To
sufficiently model the concept space, we curate a conceptual representation
dataset, CoLan-150K, which contains diverse descriptions and scenarios of
visual terms and phrases for the latent dictionary. Experiments on multiple
diffusion-based image editing baselines show that methods equipped with CoLan
achieve state-of-the-art performance in editing effectiveness and consistency
preservation.
Are You Getting What You Pay For? Auditing Model Substitution in LLM
  APIs
Authors: name: Will Cai, name: Tianneng Shi, name: Xuandong Zhao, name: Dawn Song
Published: 2025-04-07 03:57:41

The proliferation of Large Language Models (LLMs) accessed via black-box APIs
introduces a significant trust challenge: users pay for services based on
advertised model capabilities (e.g., size, performance), but providers may
covertly substitute the specified model with a cheaper, lower-quality
alternative to reduce operational costs. This lack of transparency undermines
fairness, erodes trust, and complicates reliable benchmarking. Detecting such
substitutions is difficult due to the black-box nature, typically limiting
interaction to input-output queries. This paper formalizes the problem of model
substitution detection in LLM APIs. We systematically evaluate existing
verification techniques, including output-based statistical tests, benchmark
evaluations, and log probability analysis, under various realistic attack
scenarios like model quantization, randomized substitution, and benchmark
evasion. Our findings reveal the limitations of methods relying solely on text
outputs, especially against subtle or adaptive attacks. While log probability
analysis offers stronger guarantees when available, its accessibility is often
limited. We conclude by discussing the potential of hardware-based solutions
like Trusted Execution Environments (TEEs) as a pathway towards provable model
integrity, highlighting the trade-offs between security, performance, and
provider adoption. Code is available at
https://github.com/sunblaze-ucb/llm-api-audit