Orionnnn/hub-stats huggingface dataset:
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via
  Training-Time Test
Authors: name: Yuhui Li, name: Fangyun Wei, name: Chao Zhang, name: Hongyang Zhang
Published: 2025-03-03 18:59:04

The sequential nature of modern LLMs makes them expensive and slow, and
speculative sampling has proven to be an effective solution to this problem.
Methods like EAGLE perform autoregression at the feature level, reusing
top-layer features from the target model to achieve better results than vanilla
speculative sampling. A growing trend in the LLM community is scaling up
training data to improve model intelligence without increasing inference costs.
However, we observe that scaling up data provides limited improvements for
EAGLE. We identify that this limitation arises from EAGLE's feature prediction
constraints. In this paper, we introduce EAGLE-3, which abandons feature
prediction in favor of direct token prediction and replaces reliance on
top-layer features with multi-layer feature fusion via a technique named
training-time test. These improvements significantly enhance performance and
enable the draft model to fully benefit from scaling up training data. Our
experiments include both chat models and reasoning models, evaluated on five
tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with
about 1.4x improvement over EAGLE-2. The code is available at
https://github.com/SafeAILab/EAGLE.
Know You First and Be You Better: Modeling Human-Like User Simulators
  via Implicit Profiles
Authors: name: Kuang Wang, name: Xianfei Li, name: Shenghao Yang, name: Li Zhou, name: Feng Jiang, name: Haizhou Li
Published: 2025-02-26 09:26:54

User simulators are crucial for replicating human interactions with dialogue
systems, supporting both collaborative training and automatic evaluation,
especially for large language models (LLMs). However, existing simulators often
rely solely on text utterances, missing implicit user traits such as
personality, speaking style, and goals. In contrast, persona-based methods lack
generalizability, as they depend on predefined profiles of famous individuals
or archetypes. To address these challenges, we propose User Simulator with
implicit Profiles (USP), a framework that infers implicit user profiles from
human-machine conversations and uses them to generate more personalized and
realistic dialogues. We first develop an LLM-driven extractor with a
comprehensive profile schema. Then, we refine the simulation through
conditional supervised fine-tuning and reinforcement learning with cycle
consistency, optimizing it at both the utterance and conversation levels.
Finally, we adopt a diverse profile sampler to capture the distribution of
real-world user profiles. Experimental results demonstrate that USP outperforms
strong baselines in terms of authenticity and diversity while achieving
comparable performance in consistency. Furthermore, dynamic multi-turn
evaluations based on USP strongly align with mainstream benchmarks,
demonstrating its effectiveness in real-world applications.
S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following
  with Paralinguistic Information
Authors: name: Feng Jiang, name: Zhiyu Lin, name: Fan Bu, name: Yuhao Du, name: Benyou Wang, name: Haizhou Li
Published: 2025-03-07 02:07:00

The rapid development of large language models (LLMs) has brought significant
attention to speech models, particularly recent progress in speech2speech
protocols supporting speech input and output. However, the existing benchmarks
adopt automatic text-based evaluators for evaluating the instruction following
ability of these models lack consideration for paralinguistic information in
both speech understanding and generation. To address these issues, we introduce
S2S-Arena, a novel arena-style S2S benchmark that evaluates
instruction-following capabilities with paralinguistic information in both
speech-in and speech-out across real-world tasks. We design 154 samples that
fused TTS and live recordings in four domains with 21 tasks and manually
evaluate existing popular speech models in an arena-style manner. The
experimental results show that: (1) in addition to the superior performance of
GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly
trained model after text-speech alignment in speech2speech protocols; (2)
considering paralinguistic information, the knowledgeability of the speech
model mainly depends on the LLM backbone, and the multilingual support of that
is limited by the speech module; (3) excellent speech models can already
understand the paralinguistic information in speech input, but generating
appropriate audio with paralinguistic information is still a challenge.
LONGCODEU: Benchmarking Long-Context Language Models on Long Code
  Understanding
Authors: name: Jia Li, name: Xuyuan Guo, name: Lei Li, name: Kechi Zhang, name: Ge Li, name: Jia Li, name: Zhengwei Tao, name: Fang Liu, name: Chongyang Tao, name: Yuqi Zhu, name: Zhi Jin
Published: 2025-03-06 12:02:31

Current advanced long-context language models offer great potential for
real-world software engineering applications. However, progress in this
critical domain remains hampered by a fundamental limitation: the absence of a
rigorous evaluation framework for long code understanding. To gap this
obstacle, we propose a long code understanding benchmark LONGCODEU from four
aspects (8 tasks) to evaluate LCLMs' long code understanding ability required
for practical applications, including code unit perception, intra-code unit
understanding, inter-code unit relation understanding, and long code
documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6
general models and 3 code models). Our experimental results reveal key
limitations in current LCLMs' capabilities for long code understanding.
Particularly, the performance of LCLMs drops dramatically when the long code
length is greater than 32K, falling far short of their claimed 128K-1M context
windows. In the four aspects, inter-code unit relation understanding is the
most challenging for LCLMs. Our study provides valuable insights for optimizing
LCLMs and driving advancements in software engineering.
RuCCoD: Towards Automated ICD Coding in Russian
Authors: name: Aleksandr Nesterov, name: Andrey Sakhovskiy, name: Ivan Sviridov, name: Airat Valiev, name: Vladimir Makharev, name: Petr Anokhin, name: Galina Zubkova, name: Elena Tutubalina
Published: 2025-02-28 17:40:24

This study investigates the feasibility of automating clinical coding in
Russian, a language with limited biomedical resources. We present a new dataset
for ICD coding, which includes diagnosis fields from electronic health records
(EHRs) annotated with over 10,000 entities and more than 1,500 unique ICD
codes. This dataset serves as a benchmark for several state-of-the-art models,
including BERT, LLaMA with LoRA, and RAG, with additional experiments examining
transfer learning across domains (from PubMed abstracts to medical diagnosis)
and terminologies (from UMLS concepts to ICD codes). We then apply the
best-performing model to label an in-house EHR dataset containing patient
histories from 2017 to 2021. Our experiments, conducted on a carefully curated
test set, demonstrate that training with the automated predicted codes leads to
a significant improvement in accuracy compared to manually annotated data from
physicians. We believe our findings offer valuable insights into the potential
for automating clinical coding in resource-limited languages like Russian,
which could enhance clinical efficiency and data accuracy in these contexts.
ProReflow: Progressive Reflow with Decomposed Velocity
Authors: name: Lei Ke, name: Haohang Xu, name: Xuefei Ning, name: Yu Li, name: Jiajun Li, name: Haoling Li, name: Yuxuan Lin, name: Dongsheng Jiang, name: Yujiu Yang, name: Linfeng Zhang
Published: 2025-03-05 04:50:53

Diffusion models have achieved significant progress in both image and video
generation while still suffering from huge computation costs. As an effective
solution, flow matching aims to reflow the diffusion process of diffusion
models into a straight line for a few-step and even one-step generation.
However, in this paper, we suggest that the original training pipeline of flow
matching is not optimal and introduce two techniques to improve it. Firstly, we
introduce progressive reflow, which progressively reflows the diffusion models
in local timesteps until the whole diffusion progresses, reducing the
difficulty of flow matching. Second, we introduce aligned v-prediction, which
highlights the importance of direction matching in flow matching over magnitude
matching. Experimental results on SDv1.5 and SDXL demonstrate the effectiveness
of our method, for example, conducting on SDv1.5 achieves an FID of 10.70 on
MSCOCO2014 validation set with only 4 sampling steps, close to our teacher
model (32 DDIM steps, FID = 10.05).
An Empirical Study on Eliciting and Improving R1-like Reasoning Models
Authors: name: Zhipeng Chen, name: Yingqian Min, name: Beichen Zhang, name: Jie Chen, name: Jinhao Jiang, name: Daixuan Cheng, name: Wayne Xin Zhao, name: Zheng Liu, name: Xu Miao, name: Yang Lu, name: Lei Fang, name: Zhongyuan Wang, name: Ji-Rong Wen
Published: 2025-03-06 15:34:27

In this report, we present the third technical report on the development of
slow-thinking models as part of the STILL project. As the technical pathway
becomes clearer, scaling RL training has become a central technique for
implementing such reasoning models. We systematically experiment with and
document the effects of various factors influencing RL training, conducting
experiments on both base models and fine-tuned models. Specifically, we
demonstrate that our RL training approach consistently improves the Qwen2.5-32B
base models, enhancing both response length and test accuracy. Furthermore, we
show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
achieved a high performance level, it can be further refined through RL
training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
also explore the use of tool manipulation, finding that it significantly boosts
the reasoning performance of large reasoning models. This approach achieves a
remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
effectiveness in enhancing model capabilities. We release our resources at the
STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
Authors: name: Weigao Sun, name: Disen Lan, name: Tong Zhu, name: Xiaoye Qu, name: Yu Cheng
Published: 2025-03-07 14:17:45

Linear Sequence Modeling (LSM) like linear attention, state space models and
linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant
architectural improvements. In this paper, we introduce Linear-MoE, a
production-level system for modeling and training large-scale models that
integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules
for linear-complexity sequence modeling and MoE layers for sparsely activation,
aiming to offer high performance with efficient training. The Linear-MoE system
comprises: 1) Modeling subsystem, which provides a unified framework supporting
all instances of LSM. and 2) Training subsystem, which facilitates efficient
training by incorporating various advanced parallelism technologies,
particularly Sequence Parallelism designed for Linear-MoE models. Additionally,
we explore hybrid models that combine Linear-MoE layers with standard
Transformer-MoE layers with its Sequence Parallelism to further enhance model
flexibility and performance. Evaluations on two model series, A0.3B-2B and
A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining
competitive performance on various benchmarks, showcasing its potential as a
next-generation foundational model architecture. Code:
https://github.com/OpenSparseLLMs/Linear-MoE.
VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play
  Context Control
Authors: name: Yuxuan Bian, name: Zhaoyang Zhang, name: Xuan Ju, name: Mingdeng Cao, name: Liangbin Xie, name: Ying Shan, name: Qiang Xu
Published: 2025-03-07 17:59:46

Video inpainting, which aims to restore corrupted video content, has
experienced substantial progress. Despite these advances, existing methods,
whether propagating unmasked region pixels through optical flow and receptive
field priors, or extending image-inpainting models temporally, face challenges
in generating fully masked objects or balancing the competing objectives of
background context preservation and foreground generation in one model,
respectively. To address these limitations, we propose a novel dual-stream
paradigm VideoPainter that incorporates an efficient context encoder
(comprising only 6% of the backbone parameters) to process masked videos and
inject backbone-aware background contextual cues to any pre-trained video DiT,
producing semantically consistent content in a plug-and-play manner. This
architectural separation significantly reduces the model's learning complexity
while enabling nuanced integration of crucial background context. We also
introduce a novel target region ID resampling technique that enables any-length
video inpainting, greatly enhancing our practical applicability. Additionally,
we establish a scalable dataset pipeline leveraging current vision
understanding models, contributing VPData and VPBench to facilitate
segmentation-based inpainting training and assessment, the largest video
inpainting dataset and benchmark to date with over 390K diverse clips. Using
inpainting as a pipeline basis, we also explore downstream applications
including video editing and video editing pair data generation, demonstrating
competitive performance and significant practical potential. Extensive
experiments demonstrate VideoPainter's superior performance in both any-length
video inpainting and editing, across eight key metrics, including video
quality, mask region preservation, and textual coherence.
AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLM
Authors: name: Sunghyun Ahn, name: Youngwan Jo, name: Kijung Lee, name: Sein Kwon, name: Inpyo Hong, name: Sanghyun Park
Published: 2025-03-06 14:52:34

Video anomaly detection (VAD) is crucial for video analysis and surveillance
in computer vision. However, existing VAD models rely on learned normal
patterns, which makes them difficult to apply to diverse environments.
Consequently, users should retrain models or develop separate AI models for new
environments, which requires expertise in machine learning, high-performance
hardware, and extensive data collection, limiting the practical usability of
VAD. To address these challenges, this study proposes customizable video
anomaly detection (C-VAD) technique and the AnyAnomaly model. C-VAD considers
user-defined text as an abnormal event and detects frames containing a
specified event in a video. We effectively implemented AnyAnomaly using a
context-aware visual question answering without fine-tuning the large vision
language model. To validate the effectiveness of the proposed model, we
constructed C-VAD datasets and demonstrated the superiority of AnyAnomaly.
Furthermore, our approach showed competitive performance on VAD benchmark
datasets, achieving state-of-the-art results on the UBnormal dataset and
outperforming other methods in generalization across all datasets. Our code is
available online at github.com/SkiddieAhn/Paper-AnyAnomaly.
EuroBERT: Scaling Multilingual Encoders for European Languages
Authors: name: Nicolas Boizard, name: Hippolyte Gisserot-Boukhlef, name: Duarte M. Alves, name: André Martins, name: Ayoub Hammal, name: Caio Corro, name: Céline Hudelot, name: Emmanuel Malherbe, name: Etienne Malaboeuf, name: Fanny Jourdan, name: Gabriel Hautreux, name: João Alves, name: Kevin El-Haddad, name: Manuel Faysse, name: Maxime Peyrard, name: Nuno M. Guerreiro, name: Patrick Fernandes, name: Ricardo Rei, name: Pierre Colombo
Published: 2025-03-07 15:13:58

General-purpose multilingual vector representations, used in retrieval,
regression and classification, are traditionally obtained from bidirectional
encoder models. Despite their wide applicability, encoders have been recently
overshadowed by advances in generative decoder-only models. However, many
innovations driving this progress are not inherently tied to decoders. In this
paper, we revisit the development of multilingual encoders through the lens of
these advances, and introduce EuroBERT, a family of multilingual encoders
covering European and widely spoken global languages. Our models outperform
existing alternatives across a diverse range of tasks, spanning multilingual
capabilities, mathematics, and coding, and natively supporting sequences of up
to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering
insights into our dataset composition and training pipeline. We publicly
release the EuroBERT models, including intermediate training checkpoints,
together with our training framework.
SAGE: A Framework of Precise Retrieval for RAG
Authors: name: Jintao Zhang, name: Guoliang Li, name: Jinyang Su
Published: 2025-03-03 16:25:58

Retrieval-augmented generation (RAG) has demonstrated significant proficiency
in conducting question-answering (QA) tasks within a specified corpus.
Nonetheless, numerous failure instances of RAG in QA still exist. These
failures are not solely attributable to the limitations of Large Language
Models (LLMs); instead, they predominantly arise from the retrieval of
inaccurate information for LLMs due to two limitations: (1) Current RAG methods
segment the corpus without considering semantics, making it difficult to find
relevant context due to impaired correlation between questions and the
segments. (2) There is a trade-off between missing essential context with fewer
context retrieved and getting irrelevant context with more context retrieved.
  In this paper, we introduce a RAG framework (SAGE), to overcome these
limitations. First, to address the segmentation issue without considering
semantics, we propose to train a semantic segmentation model. This model is
trained to segment the corpus into semantically complete chunks. Second, to
ensure that only the most relevant chunks are retrieved while the irrelevant
ones are ignored, we design a chunk selection algorithm to dynamically select
chunks based on the decreasing speed of the relevance score, leading to a more
relevant selection. Third, to further ensure the precision of the retrieved
chunks, we propose letting LLMs assess whether retrieved chunks are excessive
or lacking and then adjust the amount of context accordingly. Experiments show
that SAGE outperforms baselines by 61.25% in the quality of QA on average.
Moreover, by avoiding retrieving noisy context, SAGE lowers the cost of the
tokens consumed in LLM inference and achieves a 49.41% enhancement in cost
efficiency on average. Additionally, our work offers valuable insights for
boosting RAG.
TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos
  via Diffusion Models
Authors: name: Mark YU, name: Wenbo Hu, name: Jinbo Xing, name: Ying Shan
Published: 2025-03-07 17:57:53

We present TrajectoryCrafter, a novel approach to redirect camera
trajectories for monocular videos. By disentangling deterministic view
transformations from stochastic content generation, our method achieves precise
control over user-specified camera trajectories. We propose a novel dual-stream
conditional video diffusion model that concurrently integrates point cloud
renders and source videos as conditions, ensuring accurate view transformations
and coherent 4D content generation. Instead of leveraging scarce multi-view
videos, we curate a hybrid training dataset combining web-scale monocular
videos with static multi-view datasets, by our innovative double-reprojection
strategy, significantly fostering robust generalization across diverse scenes.
Extensive evaluations on multi-view and large-scale monocular videos
demonstrate the superior performance of our method.
LoRACode: LoRA Adapters for Code Embeddings
Authors: name: Saumya Chaturvedi, name: Aman Chadha, name: Laurent Bindschaedler
Published: 2025-03-07 10:50:45

Code embeddings are essential for semantic code search; however, current
approaches often struggle to capture the precise syntactic and contextual
nuances inherent in code. Open-source models such as CodeBERT and UniXcoder
exhibit limitations in scalability and efficiency, while high-performing
proprietary systems impose substantial computational costs. We introduce a
parameter-efficient fine-tuning method based on Low-Rank Adaptation (LoRA) to
construct task-specific adapters for code retrieval. Our approach reduces the
number of trainable parameters to less than two percent of the base model,
enabling rapid fine-tuning on extensive code corpora (2 million samples in 25
minutes on two H100 GPUs). Experiments demonstrate an increase of up to 9.1% in
Mean Reciprocal Rank (MRR) for Code2Code search, and up to 86.69% for Text2Code
search tasks across multiple programming languages. Distinction in task-wise
and language-wise adaptation helps explore the sensitivity of code retrieval
for syntactical and linguistic variations.
R1-Searcher: Incentivizing the Search Capability in LLMs via
  Reinforcement Learning
Authors: name: Huatong Song, name: Jinhao Jiang, name: Yingqian Min, name: Jie Chen, name: Zhipeng Chen, name: Wayne Xin Zhao, name: Lei Fang, name: Ji-Rong Wen
Published: 2025-03-07 17:14:44

Existing Large Reasoning Models (LRMs) have shown the potential of
reinforcement learning (RL) to enhance the complex reasoning capabilities of
Large Language Models~(LLMs). While they achieve remarkable performance on
challenging tasks such as mathematics and coding, they often rely on their
internal knowledge to solve problems, which can be inadequate for
time-sensitive or knowledge-intensive questions, leading to inaccuracies and
hallucinations. To address this, we propose R1-Searcher, a novel
two-stage outcome-based RL approach designed to enhance the search capabilities
of LLMs. This method allows LLMs to autonomously invoke external search systems
to access additional knowledge during the reasoning process. Our framework
relies exclusively on RL, without requiring process rewards or distillation for
a cold start. % effectively generalizing to out-of-domain datasets and
supporting both Base and Instruct models. Our experiments demonstrate that our
method significantly outperforms previous strong RAG methods, even when
compared to the closed-source GPT-4o-mini.
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with
  Reinforcing Learning
Authors: name: Jiaxing Zhao, name: Xihan Wei, name: Liefeng Bo
Published: 2025-03-07 12:46:42

In this work, we present the first application of Reinforcement Learning with
Verifiable Reward (RLVR) to an Omni-multimodal large language model in the
context of emotion recognition, a task where both visual and audio modalities
play crucial roles. We leverage RLVR to optimize the Omni model, significantly
enhancing its performance in three key aspects: reasoning capability, emotion
recognition accuracy, and generalization ability. The introduction of RLVR not
only improves the model's overall performance on in-distribution data but also
demonstrates superior robustness when evaluated on out-of-distribution
datasets. More importantly, the improved reasoning capability enables clear
analysis of the contributions of different modalities, particularly visual and
audio information, in the emotion recognition process. This provides valuable
insights into the optimization of multimodal large language models.