Orionnnn/hub-stats huggingface dataset:
FinMTEB: Finance Massive Text Embedding Benchmark
Authors: name: Yixuan Tang, name: Yi Yang
Published: 2025-02-16 04:23:52

Embedding models play a crucial role in representing and retrieving
information across various NLP applications. Recent advances in large language
models (LLMs) have further enhanced the performance of embedding models. While
these models are often benchmarked on general-purpose datasets, real-world
applications demand domain-specific evaluation. In this work, we introduce the
Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart
to MTEB designed for the financial domain. FinMTEB comprises 64 financial
domain-specific embedding datasets across 7 tasks that cover diverse textual
types in both Chinese and English, such as financial news articles, corporate
annual reports, ESG reports, regulatory filings, and earnings call transcripts.
We also develop a finance-adapted model, FinPersona-E5, using a persona-based
data synthetic method to cover diverse financial embedding tasks for training.
Through extensive evaluation of 15 embedding models, including FinPersona-E5,
we show three key findings: (1) performance on general-purpose benchmarks shows
limited correlation with financial domain tasks; (2) domain-adapted models
consistently outperform their general-purpose counterparts; and (3)
surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated
dense embeddings in financial Semantic Textual Similarity (STS) tasks,
underscoring current limitations in dense embedding techniques. Our work
establishes a robust evaluation framework for financial NLP applications and
provides crucial insights for developing domain-specific embedding models.
Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the
  Limits of Embedding Space Capacity
Authors: name: Yuri Kuratov, name: Mikhail Arkhipov, name: Aydar Bulatov, name: Mikhail Burtsev
Published: 2025-02-18 17:08:45

A range of recent works addresses the problem of compression of sequence of
tokens into a shorter sequence of real-valued vectors to be used as inputs
instead of token embeddings or key-value cache. These approaches allow to
reduce the amount of compute in existing language models. Despite relying on
powerful models as encoders, the maximum attainable lossless compression ratio
is typically not higher than x10. This fact is highly intriguing because, in
theory, the maximum information capacity of large real-valued vectors is far
beyond the presented rates even for 16-bit precision and a modest vector size.
In this work, we explore the limits of compression by replacing the encoder
with a per-sample optimization procedure. We show that vectors with compression
ratios up to x1500 exist, which highlights two orders of magnitude gap between
existing and practically attainable solutions. Furthermore, we empirically show
that the compression limits are determined not by the length of the input but
by the amount of uncertainty to be reduced, namely, the cross-entropy loss on
this sequence without any conditioning. The obtained limits highlight the
substantial gap between the theoretical capacity of input embeddings and their
practical utilization, suggesting significant room for optimization in model
design.
You Do Not Fully Utilize Transformer's Representation Capacity
Authors: name: Gleb Gerasimov, name: Yaroslav Aksenov, name: Nikita Balagansky, name: Viacheslav Sinii, name: Daniil Gavrilov
Published: 2025-02-13 12:00:50

In contrast to RNNs, which compress previous tokens into a single hidden
state, Transformers can attend to all previous tokens directly. However,
standard Transformers only use representations from the immediately preceding
layer. In this paper, we show that this design choice causes representation
collapse and leads to suboptimal performance. To address this issue, we
introduce Layer-Integrated Memory (LIMe), a simple yet powerful approach that
preserves the model's overall memory footprint while expanding its
representational capacity by allowing access to hidden states from earlier
layers. Through extensive experiments across various architectures and
different lookup mechanisms, we demonstrate consistent performance improvements
on a wide range of tasks. Moreover, our analysis of the learned representation
dynamics and our exploration of depthwise circuits reveal how LIMe integrates
information across layers, pointing to promising directions for future
research.
Injecting Domain-Specific Knowledge into Large Language Models: A
  Comprehensive Survey
Authors: name: Zirui Song, name: Bin Yan, name: Yuhan Liu, name: Miao Fang, name: Mingzhe Li, name: Rui Yan, name: Xiuying Chen
Published: 2025-02-15 07:43:43

Large Language Models (LLMs) have demonstrated remarkable success in various
tasks such as natural language understanding, text summarization, and machine
translation. However, their general-purpose nature often limits their
effectiveness in domain-specific applications that require specialized
knowledge, such as healthcare, chemistry, or legal analysis. To address this,
researchers have explored diverse methods to enhance LLMs by integrating
domain-specific knowledge. In this survey, we provide a comprehensive overview
of these methods, which we categorize into four key approaches: dynamic
knowledge injection, static knowledge embedding, modular adapters, and prompt
optimization. Each approach offers unique mechanisms to equip LLMs with domain
expertise, balancing trade-offs between flexibility, scalability, and
efficiency. We discuss how these methods enable LLMs to tackle specialized
tasks, compare their advantages and disadvantages, evaluate domain-specific
LLMs against general LLMs, and highlight the challenges and opportunities in
this emerging field. For those interested in delving deeper into this area, we
also summarize the commonly used datasets and benchmarks. To keep researchers
updated on the latest studies, we maintain an open-source at:
https://github.com/abilliyb/Knowledge_Injection_Survey_Papers, dedicated to
documenting research in the field of specialized LLM.
Perovskite-LLM: Knowledge-Enhanced Large Language Models for Perovskite
  Solar Cell Research
Authors: name: Xiang Liu, name: Penglei Sun, name: Shuyan Chen, name: Longhan Zhang, name: Peijie Dong, name: Huajie You, name: Yongqi Zhang, name: Chang Yan, name: Xiaowen Chu, name: Tong-yi Zhang
Published: 2025-02-18 09:19:24

The rapid advancement of perovskite solar cells (PSCs) has led to an
exponential growth in research publications, creating an urgent need for
efficient knowledge management and reasoning systems in this domain. We present
a comprehensive knowledge-enhanced system for PSCs that integrates three key
components. First, we develop Perovskite-KG, a domain-specific knowledge graph
constructed from 1,517 research papers, containing 23,789 entities and 22,272
relationships. Second, we create two complementary datasets: Perovskite-Chat,
comprising 55,101 high-quality question-answer pairs generated through a novel
multi-agent framework, and Perovskite-Reasoning, containing 2,217 carefully
curated materials science problems. Third, we introduce two specialized large
language models: Perovskite-Chat-LLM for domain-specific knowledge assistance
and Perovskite-Reasoning-LLM for scientific reasoning tasks. Experimental
results demonstrate that our system significantly outperforms existing models
in both domain-specific knowledge retrieval and scientific reasoning tasks,
providing researchers with effective tools for literature review, experimental
design, and complex problem-solving in PSC research.
OctoTools: An Agentic Framework with Extensible Tools for Complex
  Reasoning
Authors: name: Pan Lu, name: Bowen Chen, name: Sheng Liu, name: Rahul Thapa, name: Joseph Boen, name: James Zou
Published: 2025-02-16 21:18:47

Solving complex reasoning tasks may involve visual understanding, domain
knowledge retrieval, numerical calculation, and multi-step reasoning. Existing
methods augment large language models (LLMs) with external tools but are
restricted to specialized domains, limited tool types, or require additional
training data. In this paper, we introduce OctoTools, a training-free,
user-friendly, and easily extensible open-source agentic framework designed to
tackle complex reasoning across diverse domains. OctoTools introduces
standardized tool cards to encapsulate tool functionality, a planner for both
high-level and low-level planning, and an executor to carry out tool usage. We
validate OctoTools' generality across 16 diverse tasks (including MathVista,
MMLU-Pro, MedQA, and GAIA-Text), achieving substantial average accuracy gains
of 9.3% over GPT-4o. Furthermore, OctoTools outperforms AutoGen, GPT-Functions
and LangChain by up to 10.6% when given the same set of tools. Through
comprehensive analysis and ablations, OctoTools demonstrates advantages in task
planning, effective tool usage, and multi-step problem solving.
Pre-training Auto-regressive Robotic Models with 4D Representations
Authors: name: Dantong Niu, name: Yuvan Sharma, name: Haoru Xue, name: Giscard Biamby, name: Junyi Zhang, name: Ziteng Ji, name: Trevor Darrell, name: Roei Herzig
Published: 2025-02-18 18:59:01

Foundation models pre-trained on massive unlabeled datasets have
revolutionized natural language and computer vision, exhibiting remarkable
generalization capabilities, thus highlighting the importance of pre-training.
Yet, efforts in robotics have struggled to achieve similar success, limited by
either the need for costly robotic annotations or the lack of representations
that effectively model the physical world. In this paper, we introduce ARM4R,
an Auto-regressive Robotic Model that leverages low-level 4D Representations
learned from human video data to yield a better pre-trained robotic model.
Specifically, we focus on utilizing 3D point tracking representations from
videos derived by lifting 2D representations into 3D space via monocular depth
estimation across time. These 4D representations maintain a shared geometric
structure between the points and robot state representations up to a linear
transformation, enabling efficient transfer learning from human video data to
low-level robotic control. Our experiments show that ARM4R can transfer
efficiently from human video data to robotics and consistently improves
performance on tasks across various robot environments and configurations.
PAFT: Prompt-Agnostic Fine-Tuning
Authors: name: Chenxing Wei, name: Yao Shu, name: Mingwen Ou, name: Ying Tiffany He, name: Fei Richard Yu
Published: 2025-02-18 13:46:47

While Large Language Models (LLMs) adapt well to downstream tasks after
fine-tuning, this adaptability often compromises prompt robustness, as even
minor prompt variations can significantly degrade performance. To address this,
we propose Prompt-Agnostic Fine-Tuning(PAFT), a simple yet effective approach
that dynamically adjusts prompts during fine-tuning. This encourages the model
to learn underlying task principles rather than overfitting to specific prompt
formulations. PAFT operates in two stages: First, a diverse set of meaningful,
synthetic candidate prompts is constructed. Second, during fine-tuning, prompts
are randomly sampled from this set to create dynamic training inputs. Extensive
experiments across diverse datasets and LLMs demonstrate that models trained
with PAFT exhibit strong robustness and generalization across a wide range of
prompts, including unseen ones. This enhanced robustness improves both model
performance and inference speed while maintaining training efficiency. Ablation
studies further confirm the effectiveness of PAFT.
Soundwave: Less is More for Speech-Text Alignment in LLMs
Authors: name: Yuhao Zhang, name: Zhiheng Liu, name: Fan Bu, name: Ruiyu Zhang, name: Benyou Wang, name: Haizhou Li
Published: 2025-02-18 14:36:39

Existing end-to-end speech large language models (LLMs) usually rely on
large-scale annotated data for training, while data-efficient training has not
been discussed in depth. We focus on two fundamental problems between speech
and text: the representation space gap and sequence length inconsistency. We
propose Soundwave, which utilizes an efficient training strategy and a novel
architecture to address these issues. Results show that Soundwave outperforms
the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks,
using only one-fiftieth of the training data. Further analysis shows that
Soundwave still retains its intelligence during conversation. The project is
available at https://github.com/FreedomIntelligence/Soundwave.
Magma: A Foundation Model for Multimodal AI Agents
Authors: name: Jianwei Yang, name: Reuben Tan, name: Qianhui Wu, name: Ruijie Zheng, name: Baolin Peng, name: Yongyuan Liang, name: Yu Gu, name: Mu Cai, name: Seonghyeon Ye, name: Joel Jang, name: Yuquan Deng, name: Lars Liden, name: Jianfeng Gao
Published: 2025-02-18 18:55:21

We present Magma, a foundation model that serves multimodal AI agentic tasks
in both the digital and physical worlds. Magma is a significant extension of
vision-language (VL) models in that it not only retains the VL understanding
ability (verbal intelligence) of the latter, but is also equipped with the
ability to plan and act in the visual-spatial world (spatial-temporal
intelligence) and complete agentic tasks ranging from UI navigation to robot
manipulation. To endow the agentic capabilities, Magma is pretrained on large
amounts of heterogeneous datasets spanning from images, videos to robotics
data, where the actionable visual objects (e.g., clickable buttons in GUI) in
images are labeled by Set-of-Mark (SoM) for action grounding, and the object
movements (e.g., the trace of human hands or robotic arms) in videos are
labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
that SoM and ToM reach great synergy and facilitate the acquisition of
spatial-temporal intelligence for our Magma model, which is fundamental to a
wide range of tasks as shown in Fig.1. In particular, Magma creates new
state-of-the-art results on UI navigation and robotic manipulation tasks,
outperforming previous models that are specifically tailored to these tasks. On
image and video-related multimodal tasks, Magma also compares favorably to
popular large multimodal models that are trained on much larger datasets. We
make our model and code public for reproducibility at
https://microsoft.github.io/Magma.
Revisiting the Test-Time Scaling of o1-like Models: Do they Truly
  Possess Test-Time Scaling Capabilities?
Authors: name: Zhiyuan Zeng, name: Qinyuan Cheng, name: Zhangyue Yin, name: Yunhua Zhou, name: Xipeng Qiu
Published: 2025-02-17 07:21:11

The advent of test-time scaling in large language models (LLMs), exemplified
by OpenAI's o1 series, has advanced reasoning capabilities by scaling
computational resource allocation during inference. While successors like QwQ,
Deepseek-R1 (R1) and LIMO replicate these advancements, whether these models
truly possess test-time scaling capabilities remains underexplored. This study
found that longer CoTs of these o1-like models do not consistently enhance
accuracy; in fact, correct solutions are often shorter than incorrect ones for
the same questions. Further investigation shows this phenomenon is closely
related to models' self-revision capabilities - longer CoTs contain more
self-revisions, which often lead to performance degradation. We then compare
sequential and parallel scaling strategies on QwQ, R1 and LIMO, finding that
parallel scaling achieves better coverage and scalability. Based on these
insights, we propose Shortest Majority Vote, a method that combines parallel
scaling strategies with CoT length characteristics, significantly improving
models' test-time scalability compared to conventional majority voting
approaches.
SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety
  Guardrails in Large Language Models
Authors: name: Seanie Lee, name: Dong Bok Lee, name: Dominik Wagner, name: Minki Kang, name: Haebin Seong, name: Tobias Bocklet, name: Juho Lee, name: Sung Ju Hwang
Published: 2025-02-18 02:51:17

Deploying large language models (LLMs) in real-world applications requires
robust safety guard models to detect and block harmful user prompts. While
large safety guard models achieve strong performance, their computational cost
is substantial. To mitigate this, smaller distilled models are used, but they
often underperform on "hard" examples where the larger model provides accurate
predictions. We observe that many inputs can be reliably handled by the smaller
model, while only a small fraction require the larger model's capacity.
Motivated by this, we propose SafeRoute, a binary router that distinguishes
hard examples from easy ones. Our method selectively applies the larger safety
guard model to the data that the router considers hard, improving efficiency
while maintaining accuracy compared to solely using the larger safety guard
model. Experimental results on multiple benchmark datasets demonstrate that our
adaptive model selection significantly enhances the trade-off between
computational cost and safety performance, outperforming relevant baselines.
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway
  Dynamic Dense Connections
Authors: name: Da Xiao, name: Qingye Meng, name: Shengping Li, name: Xingyuan Yuan
Published: 2025-02-13 10:26:27

We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective
method to address the limitations of residual connections and enhance
cross-layer information flow in Transformers. Unlike existing dense connection
approaches with static and shared connection weights, MUDD generates connection
weights dynamically depending on hidden states at each sequence position and
for each decoupled input stream (the query, key, value or residual) of a
Transformer block. MUDD connections can be seamlessly integrated into any
Transformer architecture to create MUDDFormer. Extensive experiments show that
MUDDFormer significantly outperforms Transformers across various model
architectures and scales in language modeling, achieving the performance of
Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches
Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B
in five-shot settings, while adding only 0.23% parameters and 0.4% computation.
Code in JAX and PyTorch and pre-trained models are available at
https://github.com/Caiyun-AI/MUDDFormer .
Multilingual Encoder Knows more than You Realize: Shared Weights
  Pretraining for Extremely Low-Resource Languages
Authors: name: Zeli Su, name: Ziyin Zhang, name: Guixian Xu, name: Jianing Liu, name: XU Han, name: Ting Zhang, name: Yushuang Dong
Published: 2025-02-15 16:53:10

While multilingual language models like XLM-R have advanced multilingualism
in NLP, they still perform poorly in extremely low-resource languages. This
situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen
support far fewer languages than XLM-R, making text generation models
non-existent for many languages in the world. To tackle this challenge, we
propose a novel framework for adapting multilingual encoders to text generation
in extremely low-resource languages. By reusing the weights between the encoder
and the decoder, our framework allows the model to leverage the learned
semantic space of the encoder, enabling efficient learning and effective
generalization in low-resource languages. Applying this framework to four
Chinese minority languages, we present XLM-SWCM, and demonstrate its superior
performance on various downstream tasks even when compared with much larger
models.
Continuous Diffusion Model for Language Modeling
Authors: name: Jaehyeong Jo, name: Sung Ju Hwang
Published: 2025-02-17 08:54:29

Diffusion models have emerged as a promising alternative to autoregressive
models in modeling discrete categorical data. Yet diffusion models that
directly work on discrete data space do not fully exploit the power of
iterative refinement, as the signals are lost during the transition between
discrete states. Existing continuous diffusion models for discrete data have
limited performance compared to discrete approaches, and the unclear link
between them restricts the development of diffusion models for discrete data.
In this work, we propose a continuous diffusion model for language modeling
that incorporates the geometry of the underlying categorical distribution. We
establish a connection between the discrete diffusion and continuous flow on
the statistical manifold, and building on the analogy, we introduce a simple
design for the diffusion process that generalizes previous discrete diffusion
models. We further propose a simulation-free training framework based on radial
symmetry and a simple technique to address the high dimensionality of the
manifold. Comprehensive experiments on language modeling benchmarks and other
modalities show that our method outperforms existing discrete diffusion models
and approaches the performance of autoregressive models. Codes available at
https://github.com/harryjo97/RDLM{https://github.com/harryjo97/RDLM}.
HealthGPT: A Medical Large Vision-Language Model for Unifying
  Comprehension and Generation via Heterogeneous Knowledge Adaptation
Authors: name: Tianwei Lin, name: Wenqiao Zhang, name: Sijing Li, name: Yuqian Yuan, name: Binhe Yu, name: Haoyuan Li, name: Wanggui He, name: Hao Jiang, name: Mengze Li, name: Xiaohui Song, name: Siliang Tang, name: Jun Xiao, name: Hui Lin, name: Yueting Zhuang, name: Beng Chin Ooi
Published: 2025-02-14 00:42:36

We present HealthGPT, a powerful Medical Large Vision-Language Model
(Med-LVLM) that integrates medical visual comprehension and generation
capabilities within a unified autoregressive paradigm. Our bootstrapping
philosophy is to progressively adapt heterogeneous comprehension and generation
knowledge to pre-trained large language models (LLMs). This is achieved through
a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is
complemented by a tailored hierarchical visual perception approach and a
three-stage learning strategy. To effectively learn the HealthGPT, we devise a
comprehensive medical domain-specific comprehension and generation dataset
called VL-Health. Experimental results demonstrate exceptional performance and
scalability of HealthGPT in medical visual unified tasks. Our project can be
accessed at https://github.com/DCDmllm/HealthGPT.
Multimodal Mamba: Decoder-only Multimodal State Space Model via
  Quadratic to Linear Distillation
Authors: name: Bencheng Liao, name: Hongyuan Tao, name: Qian Zhang, name: Tianheng Cheng, name: Yingyue Li, name: Haoran Yin, name: Wenyu Liu, name: Xinggang Wang
Published: 2025-02-18 18:59:57

Recent Multimodal Large Language Models (MLLMs) have achieved remarkable
performance but face deployment challenges due to their quadratic computational
complexity, growing Key-Value cache requirements, and reliance on separate
vision encoders. We propose mmMamba, a framework for developing
linear-complexity native multimodal state space models through progressive
distillation from existing MLLMs using moderate academic computational
resources. Our approach enables the direct conversion of trained decoder-only
MLLMs to linear-complexity architectures without requiring pre-trained
RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba
from trained Transformer and a three-stage distillation recipe, which can
effectively transfer the knowledge from Transformer to Mamba while preserving
multimodal capabilities. Our method also supports flexible hybrid architectures
that combine Transformer and Mamba layers for customizable
efficiency-performance trade-offs. Distilled from the Transformer-based
decoder-only HoVLE, mmMamba-linear achieves competitive performance against
existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further
improves performance significantly, approaching HoVLE's capabilities. At 103K
tokens, mmMamba-linear demonstrates 20.6times speedup and 75.8% GPU memory
reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5times speedup
and 60.2% memory savings. Code and models are released at
https://github.com/hustvl/mmMamba