Orionnnn/hub-stats huggingface dataset:
CLaMP 3: Universal Music Information Retrieval Across Unaligned
  Modalities and Unseen Languages
Authors: name: Shangda Wu, name: Zhancheng Guo, name: Ruibin Yuan, name: Junyan Jiang, name: Seungheon Doh, name: Gus Xia, name: Juhan Nam, name: Xiaobing Li, name: Feng Yu, name: Maosong Sun
Published: 2025-02-14 18:42:25

CLaMP 3 is a unified framework developed to address challenges of cross-modal
and cross-lingual generalization in music information retrieval. Using
contrastive learning, it aligns all major music modalities--including sheet
music, performance signals, and audio recordings--with multilingual text in a
shared representation space, enabling retrieval across unaligned modalities
with text as a bridge. It features a multilingual text encoder adaptable to
unseen languages, exhibiting strong cross-lingual generalization. Leveraging
retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
consisting of 2.31 million music-text pairs. This dataset is enriched with
detailed metadata that represents a wide array of global musical traditions. To
advance future research, we release WikiMT-X, a benchmark comprising 1,000
triplets of sheet music, audio, and richly varied text descriptions.
Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
MIR tasks, significantly surpassing previous strong baselines and demonstrating
excellent generalization in multimodal and multilingual music contexts.
Text-guided Sparse Voxel Pruning for Efficient 3D Visual Grounding
Authors: name: Wenxuan Guo, name: Xiuwei Xu, name: Ziwei Wang, name: Jianjiang Feng, name: Jie Zhou, name: Jiwen Lu
Published: 2025-02-14 18:59:59

In this paper, we propose an efficient multi-level convolution architecture
for 3D visual grounding. Conventional methods are difficult to meet the
requirements of real-time inference due to the two-stage or point-based
architecture. Inspired by the success of multi-level fully sparse convolutional
architecture in 3D object detection, we aim to build a new 3D visual grounding
framework following this technical route. However, as in 3D visual grounding
task the 3D scene representation should be deeply interacted with text
features, sparse convolution-based architecture is inefficient for this
interaction due to the large amount of voxel features. To this end, we propose
text-guided pruning (TGP) and completion-based addition (CBA) to deeply fuse 3D
scene representation and text features in an efficient way by gradual region
pruning and target completion. Specifically, TGP iteratively sparsifies the 3D
scene representation and thus efficiently interacts the voxel features with
text features by cross-attention. To mitigate the affect of pruning on delicate
geometric information, CBA adaptively fixes the over-pruned region by voxel
completion with negligible computational overhead. Compared with previous
single-stage methods, our method achieves top inference speed and surpasses
previous fastest method by 100\% FPS. Our method also achieves state-of-the-art
accuracy even compared with two-stage methods, with +1.13 lead of Acc@0.5 on
ScanRefer, and +2.6 and +3.2 leads on NR3D and SR3D respectively. The code
is available at
https://github.com/GWxuan/TSP3D{https://github.com/GWxuan/TSP3D}.
DarwinLM: Evolutionary Structured Pruning of Large Language Models
Authors: name: Shengkun Tang, name: Oliver Sieberling, name: Eldar Kurtic, name: Zhiqiang Shen, name: Dan Alistarh
Published: 2025-02-11 18:59:35

Large Language Models (LLMs) have achieved significant success across various
NLP tasks. However, their massive computational costs limit their widespread
use, particularly in real-time applications. Structured pruning offers an
effective solution by compressing models and directly providing end-to-end
speed improvements, regardless of the hardware environment. Meanwhile,
different components of the model exhibit varying sensitivities towards
pruning, calling for non-uniform model compression. However, a pruning
method should not only identify a capable substructure, but also account for
post-compression training. To this end, we propose \sysname, a method for
training-aware structured pruning. \sysname builds upon an evolutionary
search process, generating multiple offspring models in each generation through
mutation, and selecting the fittest for survival. To assess the effect of
post-training, we incorporate a lightweight, multistep training process within
the offspring population, progressively increasing the number of tokens and
eliminating poorly performing models in each selection stage. We validate our
method through extensive experiments on Llama-2-7B, Llama-3.1-8B and
Qwen-2.5-14B-Instruct, achieving state-of-the-art performance for structured
pruning. For instance, \sysname surpasses ShearedLlama while requiring
5times less training data during post-compression training.
ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generation
Authors: name: Rotem Shalev-Arkushin, name: Rinon Gal, name: Amit H. Bermano, name: Ohad Fried
Published: 2025-02-13 15:36:12

Diffusion models enable high-quality and diverse visual content synthesis.
However, they struggle to generate rare or unseen concepts. To address this
challenge, we explore the usage of Retrieval-Augmented Generation (RAG) with
image generation models. We propose ImageRAG, a method that dynamically
retrieves relevant images based on a given text prompt, and uses them as
context to guide the generation process. Prior approaches that used retrieved
images to improve generation, trained models specifically for retrieval-based
generation. In contrast, ImageRAG leverages the capabilities of existing image
conditioning models, and does not require RAG-specific training. Our approach
is highly adaptable and can be applied across different model types, showing
significant improvement in generating rare and fine-grained concepts using
different base models.
  Our project page is available at: https://rotem-shalev.github.io/ImageRAG
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of
  Small Multilingual Language Models for Low-Resource Languages
Authors: name: Daniil Gurgurov, name: Ivan Vykopal, name: Josef van Genabith, name: Simon Ostermann
Published: 2025-02-14 13:10:39

Low-resource languages (LRLs) face significant challenges in natural language
processing (NLP) due to limited data. While current state-of-the-art large
language models (LLMs) still struggle with LRLs, smaller multilingual models
(mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of
their capacity to low training data sizes. This study systematically
investigates parameter-efficient adapter-based methods for adapting mLMs to
LRLs, evaluating three architectures: Sequential Bottleneck, Invertible
Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and
structured knowledge from ConceptNet, we show that small adaptation datasets
(e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains
in intrinsic (masked language modeling) and extrinsic tasks (topic
classification, sentiment analysis, and named entity recognition). We find that
Sequential Bottleneck adapters excel in language modeling, while Invertible
Bottleneck adapters slightly outperform other methods on downstream tasks due
to better embedding alignment and larger parameter counts. Adapter-based
methods match or outperform full fine-tuning while using far fewer parameters,
and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3,
GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves
performance, pre-training data size remains the dominant factor, especially for
languages with extensive pre-training coverage.
Cluster and Predict Latents Patches for Improved Masked Image Modeling
Authors: name: Timothée Darcet, name: Federico Baldassarre, name: Maxime Oquab, name: Julien Mairal, name: Piotr Bojanowski
Published: 2025-02-12 20:17:10

Masked Image Modeling (MIM) offers a promising approach to self-supervised
representation learning, however existing MIM models still lag behind the
state-of-the-art. In this paper, we systematically analyze target
representations, loss functions, and architectures, to introduce CAPI - a novel
pure-MIM framework that relies on the prediction of latent clusterings. Our
approach leverages a clustering-based loss, which is stable to train, and
exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8%
accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes,
substantially outperforming previous MIM methods and approaching the
performance of the current state-of-the-art, DINOv2. We release all our code
and models.
AdaPTS: Adapting Univariate Foundation Models to Probabilistic
  Multivariate Time Series Forecasting
Authors: name: Abdelhakim Benechehab, name: Vasilii Feofanov, name: Giuseppe Paolo, name: Albert Thomas, name: Maurizio Filippone, name: Balázs Kégl
Published: 2025-02-14 15:46:19

Pre-trained foundation models (FMs) have shown exceptional performance in
univariate time series forecasting tasks. However, several practical challenges
persist, including managing intricate dependencies among features and
quantifying uncertainty in predictions. This study aims to tackle these
critical limitations by introducing adapters; feature-space transformations
that facilitate the effective use of pre-trained univariate time series FMs for
multivariate tasks. Adapters operate by projecting multivariate inputs into a
suitable latent space and applying the FM independently to each dimension.
Inspired by the literature on representation learning and partially stochastic
Bayesian neural networks, we present a range of adapters and
optimization/inference strategies. Experiments conducted on both synthetic and
real-world datasets confirm the efficacy of adapters, demonstrating substantial
enhancements in forecasting accuracy and uncertainty quantification compared to
baseline methods. Our framework, AdaPTS, positions adapters as a modular,
scalable, and effective solution for leveraging time series FMs in multivariate
contexts, thereby promoting their wider adoption in real-world applications. We
release the code at https://github.com/abenechehab/AdaPTS.
Agentic End-to-End De Novo Protein Design for Tailored Dynamics Using a
  Language Diffusion Model
Authors: name: Bo Ni, name: Markus J. Buehler
Published: 2025-02-14 14:07:54

Proteins are dynamic molecular machines whose biological functions, spanning
enzymatic catalysis, signal transduction, and structural adaptation, are
intrinsically linked to their motions. Designing proteins with targeted dynamic
properties, however, remains a challenge due to the complex, degenerate
relationships between sequence, structure, and molecular motion. Here, we
introduce VibeGen, a generative AI framework that enables end-to-end de novo
protein design conditioned on normal mode vibrations. VibeGen employs an
agentic dual-model architecture, comprising a protein designer that generates
sequence candidates based on specified vibrational modes and a protein
predictor that evaluates their dynamic accuracy. This approach synergizes
diversity, accuracy, and novelty during the design process. Via full-atom
molecular simulations as direct validation, we demonstrate that the designed
proteins accurately reproduce the prescribed normal mode amplitudes across the
backbone while adopting various stable, functionally relevant structures.
Notably, generated sequences are de novo, exhibiting no significant similarity
to natural proteins, thereby expanding the accessible protein space beyond
evolutionary constraints. Our work integrates protein dynamics into generative
protein design, and establishes a direct, bidirectional link between sequence
and vibrational behavior, unlocking new pathways for engineering biomolecules
with tailored dynamical and functional properties. This framework holds broad
implications for the rational design of flexible enzymes, dynamic scaffolds,
and biomaterials, paving the way toward dynamics-informed AI-driven protein
engineering.
We Can't Understand AI Using our Existing Vocabulary
Authors: name: John Hewitt, name: Robert Geirhos, name: Been Kim
Published: 2025-02-11 14:34:05

This position paper argues that, in order to understand AI, we cannot rely on
our existing vocabulary of human words. Instead, we should strive to develop
neologisms: new words that represent precise human concepts that we want to
teach machines, or machine concepts that we need to learn. We start from the
premise that humans and machines have differing concepts. This means
interpretability can be framed as a communication problem: humans must be able
to reference and control machine concepts, and communicate human concepts to
machines. Creating a shared human-machine language through developing
neologisms, we believe, could solve this communication problem. Successful
neologisms achieve a useful amount of abstraction: not too detailed, so they're
reusable in many contexts, and not too high-level, so they convey precise
information. As a proof of concept, we demonstrate how a "length neologism"
enables controlling LLM response length, while a "diversity neologism" allows
sampling more variable responses. Taken together, we argue that we cannot
understand AI using our existing vocabulary, and expanding it through
neologisms creates opportunities for both controlling and understanding
machines better.
Precise Parameter Localization for Textual Generation in Diffusion
  Models
Authors: name: Łukasz Staniszewski, name: Bartosz Cywiński, name: Franziska Boenisch, name: Kamil Deja, name: Adam Dziedzic
Published: 2025-02-14 06:11:23

Novel diffusion models can synthesize photo-realistic images with integrated
high-quality text. Surprisingly, we demonstrate through attention activation
patching that only less than 1% of diffusion models' parameters, all contained
in attention layers, influence the generation of textual content within the
images. Building on this observation, we improve textual generation efficiency
and performance by targeting cross and joint attention layers of diffusion
models. We introduce several applications that benefit from localizing the
layers responsible for textual content generation. We first show that a
LoRA-based fine-tuning solely of the localized layers enhances, even more, the
general text-generation capabilities of large diffusion models while preserving
the quality and diversity of the diffusion models' generations. Then, we
demonstrate how we can use the localized layers to edit textual content in
generated images. Finally, we extend this idea to the practical use case of
preventing the generation of toxic text in a cost-free manner. In contrast to
prior work, our localization approach is broadly applicable across various
diffusion model architectures, including U-Net (e.g., LDM and SDXL) and
transformer-based (e.g., DeepFloyd IF and Stable Diffusion 3), utilizing
diverse text encoders (e.g., from CLIP to the large language models like T5).
Project page available at https://t2i-text-loc.github.io/.
MRS: A Fast Sampler for Mean Reverting Diffusion based on ODE and SDE
  Solvers
Authors: name: Ao Li, name: Wei Fang, name: Hongbo Zhao, name: Le Lu, name: Ge Yang, name: Minfeng Xu
Published: 2025-02-11 14:57:33

In applications of diffusion models, controllable generation is of practical
significance, but is also challenging. Current methods for controllable
generation primarily focus on modifying the score function of diffusion models,
while Mean Reverting (MR) Diffusion directly modifies the structure of the
stochastic differential equation (SDE), making the incorporation of image
conditions simpler and more natural. However, current training-free fast
samplers are not directly applicable to MR Diffusion. And thus MR Diffusion
requires hundreds of NFEs (number of function evaluations) to obtain
high-quality samples. In this paper, we propose a new algorithm named MRS (MR
Sampler) to reduce the sampling NFEs of MR Diffusion. We solve the reverse-time
SDE and the probability flow ordinary differential equation (PF-ODE) associated
with MR Diffusion, and derive semi-analytical solutions. The solutions consist
of an analytical function and an integral parameterized by a neural network.
Based on this solution, we can generate high-quality samples in fewer steps.
Our approach does not require training and supports all mainstream
parameterizations, including noise prediction, data prediction and velocity
prediction. Extensive experiments demonstrate that MR Sampler maintains high
sampling quality with a speedup of 10 to 20 times across ten different image
restoration tasks. Our algorithm accelerates the sampling procedure of MR
Diffusion, making it more practical in controllable generation.
V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with
  Multi-Modal Large Language Models
Authors: name: Hsu-kuang Chiu, name: Ryo Hachiuma, name: Chien-Yi Wang, name: Stephen F. Smith, name: Yu-Chiang Frank Wang, name: Min-Hung Chen
Published: 2025-02-14 08:05:41

Current autonomous driving vehicles rely mainly on their individual sensors
to understand surrounding scenes and plan for future trajectories, which can be
unreliable when the sensors are malfunctioning or occluded. To address this
problem, cooperative perception methods via vehicle-to-vehicle (V2V)
communication have been proposed, but they have tended to focus on detection
and tracking. How those approaches contribute to overall cooperative planning
performance is still under-explored. Inspired by recent progress using Large
Language Models (LLMs) to build autonomous driving systems, we propose a novel
problem setting that integrates an LLM into cooperative autonomous driving,
with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and
benchmark. We also propose our baseline method Vehicle-to-Vehicle Large
Language Model (V2V-LLM), which uses an LLM to fuse perception information from
multiple connected autonomous vehicles (CAVs) and answer driving-related
questions: grounding, notable object identification, and planning. Experimental
results show that our proposed V2V-LLM can be a promising unified model
architecture for performing various tasks in cooperative autonomous driving,
and outperforms other baseline methods that use different fusion approaches.
Our work also creates a new research direction that can improve the safety of
future autonomous driving systems. Our project website:
https://eddyhkchiu.github.io/v2vllm.github.io/ .
Jailbreaking to Jailbreak
Authors: name: Jeremy Kritz, name: Vaughn Robinson, name: Robert Vacareanu, name: Bijan Varjavand, name: Michael Choi, name: Bobby Gogov, name: Scale Red Team, name: Summer Yue, name: Willow E. Primack, name: Zifan Wang
Published: 2025-02-09 20:49:16

Refusal training on Large Language Models (LLMs) prevents harmful outputs,
yet this defense remains vulnerable to both automated and human-crafted
jailbreaks. We present a novel LLM-as-red-teamer approach in which a human
jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or
other LLMs. We refer to the jailbroken LLMs as J_2 attackers, which can
systematically evaluate target models using various red teaming strategies and
improve its performance via in-context learning from the previous failures. Our
experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other
LLMs as J_2, achieving 93.0% and 91.0% attack success rates (ASRs)
respectively against GPT-4o (and similar results across other capable LLMs) on
Harmbench. Our work not only introduces a scalable approach to strategic red
teaming, drawing inspiration from human red teamers, but also highlights
jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard.
Specifically, an LLM can bypass its own safeguards by employing a jailbroken
version of itself that is willing to assist in further jailbreaking. To prevent
any direct misuse with J_2, while advancing research in AI safety, we
publicly share our methodology while keeping specific prompting details
private.
Large Language Diffusion Models
Authors: name: Shen Nie, name: Fengqi Zhu, name: Zebin You, name: Xiaolu Zhang, name: Jingyang Ou, name: Jun Hu, name: Jun Zhou, name: Yankai Lin, name: Ji-Rong Wen, name: Chongxuan Li
Published: 2025-02-14 08:23:51

Autoregressive models (ARMs) are widely regarded as the cornerstone of large
language models (LLMs). We challenge this notion by introducing LLaDA, a
diffusion model trained from scratch under the pre-training and supervised
fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
masking process and a reverse process, parameterized by a vanilla Transformer
to predict masked tokens. By optimizing a likelihood bound, it provides a
principled generative approach for probabilistic inference. Across extensive
benchmarks, LLaDA demonstrates strong scalability, outperforming our
self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
instruction-following abilities in case studies such as multi-turn dialogue.
Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
poem completion task. Our findings establish diffusion models as a viable and
promising alternative to ARMs, challenging the assumption that key LLM
capabilities discussed above are inherently tied to ARMs.
Diverse Inference and Verification for Advanced Reasoning
Authors: name: Iddo Drori, name: Gaston Longhitano, name: Mao Mao, name: Seunghwan Hyun, name: Yuke Zhang, name: Sungjun Park, name: Zachary Meeks, name: Xin-Yu Zhang, name: Ben Segev, name: Howard Yong, name: Nakul Verma, name: Avi Shporer, name: Alon Amit, name: Madeleine Udell
Published: 2025-02-14 07:22:25

Reasoning LLMs such as OpenAI o1, o3 and DeepSeek R1 have made significant
progress in mathematics and coding, yet find challenging advanced tasks such as
International Mathematical Olympiad (IMO) combinatorics problems, Abstraction
and Reasoning Corpus (ARC) puzzles, and Humanity's Last Exam (HLE) questions.
We use a diverse inference approach that combines multiple models and methods
at test time. We find that verifying mathematics and code problems, and
rejection sampling on other problems is simple and effective. We automatically
verify correctness of solutions to IMO problems by Lean, and ARC puzzles by
code, and find that best-of-N effectively answers HLE questions. Our approach
increases answer accuracy on IMO combinatorics problems from 33.3% to 77.8%,
accuracy on HLE questions from 8% to 37%, and solves 80% of ARC puzzles that
948 humans could not and 26.5% of ARC puzzles that o3 high compute does not.
Test-time simulations, reinforcement learning, and meta-learning with inference
feedback improve generalization by adapting agent graph representations and
varying prompts, code, and datasets. Our approach is reliable, robust, and
scalable, and in the spirit of reproducible research, we will make it publicly
available upon publication.