Orionnnn/hub-stats huggingface dataset:
Multimodal Language Modeling for High-Accuracy Single Cell
  Transcriptomics Analysis and Generation
Authors: name: Yaorui Shi, name: Jiaqi Yang, name: Sihang Li, name: Junfeng Fang, name: Xiang Wang, name: Zhiyuan Liu, name: Yang Zhang
Published: 2025-03-12 14:26:16

Pre-trained language models (PLMs) have revolutionized scientific research,
yet their application to single-cell analysis remains limited. Text PLMs cannot
process single-cell RNA sequencing data, while cell PLMs lack the ability to
handle free text, restricting their use in multimodal tasks. Existing efforts
to bridge these modalities often suffer from information loss or inadequate
single-modal pre-training, leading to suboptimal performances. To address these
challenges, we propose Single-Cell MultiModal Generative Pre-trained
Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT
effectively integrates the state-of-the-art cell and text PLMs, facilitating
cross-modal knowledge sharing for improved performance. To bridge the text-cell
modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes
extensive pre-training on 27 million cells -- the largest dataset for
multimodal cell-text PLMs to date. This large-scale pre-training enables
scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative
improvement of textual discrepancy for cell description generation, 20.5\%
higher accuracy for cell type annotation, and 4\% improvement in k-NN
accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
LocAgent: Graph-Guided LLM Agents for Code Localization
Authors: name: Zhaoling Chen, name: Xiangru Tang, name: Gangda Deng, name: Fang Wu, name: Jialong Wu, name: Zhiwei Jiang, name: Viktor Prasanna, name: Arman Cohan, name: Xingyao Wang
Published: 2025-03-12 05:55:01

Code localization--identifying precisely where in a codebase changes need to
be made--is a fundamental yet challenging task in software maintenance.
Existing approaches struggle to efficiently navigate complex codebases when
identifying relevant code sections. The challenge lies in bridging natural
language problem descriptions with the appropriate code elements, often
requiring reasoning across hierarchical structures and multiple dependencies.
We introduce LocAgent, a framework that addresses code localization through
graph-based representation. By parsing codebases into directed heterogeneous
graphs, LocAgent creates a lightweight representation that captures code
structures (files, classes, functions) and their dependencies (imports,
invocations, inheritance), enabling LLM agents to effectively search and locate
relevant entities through powerful multi-hop reasoning. Experimental results on
real-world benchmarks demonstrate that our approach significantly enhances
accuracy in code localization. Notably, our method with the fine-tuned
Qwen-2.5-Coder-Instruct-32B model achieves comparable results to SOTA
proprietary models at greatly reduced cost (approximately 86% reduction),
reaching up to 92.7% accuracy on file-level localization while improving
downstream GitHub issue resolution success rates by 12% for multiple attempts
(Pass@10). Our code is available at https://github.com/gersteinlab/LocAgent.
PlainQAFact: Automatic Factuality Evaluation Metric for Biomedical Plain
  Language Summaries Generation
Authors: name: Zhiwen You, name: Yue Guo
Published: 2025-03-11 20:59:53

Hallucinated outputs from language models pose risks in the medical domain,
especially for lay audiences making health-related decisions. Existing
factuality evaluation methods, such as entailment- and question-answering-based
(QA), struggle with plain language summary (PLS) generation due to elaborative
explanation phenomenon, which introduces external content (e.g., definitions,
background, examples) absent from the source document to enhance comprehension.
To address this, we introduce PlainQAFact, a framework trained on a
fine-grained, human-annotated dataset PlainFact, to evaluate the factuality of
both source-simplified and elaboratively explained sentences. PlainQAFact first
classifies factuality type and then assesses factuality using a
retrieval-augmented QA-based scoring method. Our approach is lightweight and
computationally efficient. Empirical results show that existing factuality
metrics fail to effectively evaluate factuality in PLS, especially for
elaborative explanations, whereas PlainQAFact achieves state-of-the-art
performance. We further analyze its effectiveness across external knowledge
sources, answer extraction strategies, overlap measures, and document
granularity levels, refining its overall factuality assessment.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of
  Experts
Authors: name: Shwai He, name: Weilin Cai, name: Jiayi Huang, name: Ang Li
Published: 2025-03-07 01:11:39

The Mixture of Experts (MoE) is an effective architecture for scaling large
language models by leveraging sparse expert activation, optimizing the
trade-off between performance and efficiency. However, under expert
parallelism, MoE suffers from inference inefficiencies due to imbalanced
token-to-expert assignment, where some experts are overloaded while others
remain underutilized. This imbalance leads to poor resource utilization and
increased latency, as the most burdened expert dictates the overall delay, a
phenomenon we define as the \textit{Straggler Effect}. To mitigate
this, we propose Capacity-Aware Inference, including two key techniques: (1)
\textit{Capacity-Aware Token Drop}, which discards overloaded tokens
to regulate the maximum latency of MoE, and (2) \textit{Capacity-Aware
Token Reroute}, which reallocates overflowed tokens to underutilized experts,
balancing the token distribution. These techniques collectively optimize both
high-load and low-load expert utilization, leading to a more efficient MoE
inference pipeline. Extensive experiments demonstrate the effectiveness of our
methods, showing significant improvements in inference efficiency, e.g., 0.2\%
average performance increase and a 1.94times inference speedup on
Mixtral-8times7B-Instruct.
Ideas in Inference-time Scaling can Benefit Generative Pre-training
  Algorithms
Authors: name: Jiaming Song, name: Linqi Zhou
Published: 2025-03-10 10:27:30

Recent years have seen significant advancements in foundation models through
generative pre-training, yet algorithmic innovation in this space has largely
stagnated around autoregressive models for discrete signals and diffusion
models for continuous signals. This stagnation creates a bottleneck that
prevents us from fully unlocking the potential of rich multi-modal data, which
in turn limits the progress on multimodal intelligence. We argue that an
inference-first perspective, which prioritizes scaling efficiency during
inference time across sequence length and refinement steps, can inspire novel
generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
concrete example, we demonstrate how addressing limitations in diffusion
models' inference process through targeted modifications yields a stable,
single-stage algorithm that achieves superior sample quality with over an order
of magnitude greater inference efficiency.
Inductive Moment Matching
Authors: name: Linqi Zhou, name: Stefano Ermon, name: Jiaming Song
Published: 2025-03-10 17:37:39

Diffusion models and Flow Matching generate high-quality samples but are slow
at inference, and distilling them into few-step models often leads to
instability and extensive tuning. To resolve these trade-offs, we propose
Inductive Moment Matching (IMM), a new class of generative models for one- or
few-step sampling with a single-stage training procedure. Unlike distillation,
IMM does not require pre-training initialization and optimization of two
networks; and unlike Consistency Models, IMM guarantees distribution-level
convergence and remains stable under various hyperparameters and standard model
architectures. IMM surpasses diffusion models on ImageNet-256x256 with 1.99 FID
using only 8 inference steps and achieves state-of-the-art 2-step FID of 1.98
on CIFAR-10 for a model trained from scratch.
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature
  Extraction
Authors: name: Huang Huang, name: Fangchen Liu, name: Letian Fu, name: Tingfan Wu, name: Mustafa Mukadam, name: Jitendra Malik, name: Ken Goldberg, name: Pieter Abbeel
Published: 2025-03-05 18:44:48

Vision-Language-Action (VLA) models aim to predict robotic actions based on
visual observations and language instructions. Existing approaches require
fine-tuning pre-trained visionlanguage models (VLMs) as visual and language
features are independently fed into downstream policies, degrading the
pre-trained semantic alignments. We propose OTTER, a novel VLA architecture
that leverages these existing alignments through explicit, text-aware visual
feature extraction. Instead of processing all visual features, OTTER
selectively extracts and passes only task-relevant visual features that are
semantically aligned with the language instruction to the policy transformer.
This allows OTTER to keep the pre-trained vision-language encoders frozen.
Thereby, OTTER preserves and utilizes the rich semantic understanding learned
from large-scale pre-training, enabling strong zero-shot generalization
capabilities. In simulation and real-world experiments, OTTER significantly
outperforms existing VLA models, demonstrating strong zeroshot generalization
to novel objects and environments. Video, code, checkpoints, and dataset:
https://ottervla.github.io/.
Collapse of Dense Retrievers: Short, Early, and Literal Biases
  Outranking Factual Evidence
Authors: name: Mohsen Fayyaz, name: Ali Modarressi, name: Hinrich Schuetze, name: Nanyun Peng
Published: 2025-03-06 23:23:13

Dense retrieval models are commonly used in Information Retrieval (IR)
applications, such as Retrieval-Augmented Generation (RAG). Since they often
serve as the first step in these systems, their robustness is critical to avoid
failures. In this work, by repurposing a relation extraction dataset (e.g.
Re-DocRED), we design controlled experiments to quantify the impact of
heuristic biases, such as favoring shorter documents, in retrievers like
Dragon+ and Contriever. Our findings reveal significant vulnerabilities:
retrievers often rely on superficial patterns like over-prioritizing document
beginnings, shorter documents, repeated entities, and literal matches.
Additionally, they tend to overlook whether the document contains the query's
answer, lacking deep semantic understanding. Notably, when multiple biases
combine, models exhibit catastrophic performance degradation, selecting the
answer-containing document in less than 3% of cases over a biased document
without the answer. Furthermore, we show that these biases have direct
consequences for downstream applications like RAG, where retrieval-preferred
documents can mislead LLMs, resulting in a 34% performance drop than not
providing any documents at all.
ObjectMover: Generative Object Movement with Video Prior
Authors: name: Xin Yu, name: Tianyu Wang, name: Soo Ye Kim, name: Paul Guerrero, name: Xi Chen, name: Qing Liu, name: Zhe Lin, name: Xiaojuan Qi
Published: 2025-03-11 04:42:59

Simple as it seems, moving an object to another location within an image is,
in fact, a challenging image-editing task that requires re-harmonizing the
lighting, adjusting the pose based on perspective, accurately filling occluded
regions, and ensuring coherent synchronization of shadows and reflections while
maintaining the object identity. In this paper, we present ObjectMover, a
generative model that can perform object movement in highly challenging scenes.
Our key insight is that we model this task as a sequence-to-sequence problem
and fine-tune a video generation model to leverage its knowledge of consistent
object generation across video frames. We show that with this approach, our
model is able to adjust to complex real-world scenarios, handling extreme
lighting harmonization and object effect movement. As large-scale data for
object movement are unavailable, we construct a data generation pipeline using
a modern game engine to synthesize high-quality data pairs. We further propose
a multi-task learning strategy that enables training on real-world video data
to improve the model generalization. Through extensive experiments, we
demonstrate that ObjectMover achieves outstanding results and adapts well to
real-world scenarios.
Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution
  Autonomous Driving VQA from Peru
Authors: name: Dunant Cusipuma, name: David Ortega, name: Victor Flores-Benites, name: Arturo Deza
Published: 2025-03-10 17:50:04

As multimodal foundational models start being deployed experimentally in
Self-Driving cars, a reasonable question we ask ourselves is how similar to
humans do these systems respond in certain driving situations -- especially
those that are out-of-distribution? To study this, we create the Robusto-1
dataset that uses dashcam video data from Peru, a country with one of the worst
(aggressive) drivers in the world, a high traffic index, and a high ratio of
bizarre to non-bizarre street objects likely never seen in training. In
particular, to preliminarly test at a cognitive level how well Foundational
Visual Language Models (VLMs) compare to Humans in Driving, we move away from
bounding boxes, segmentation maps, occupancy maps or trajectory estimation to
multi-modal Visual Question Answering (VQA) comparing both humans and machines
through a popular method in systems neuroscience known as Representational
Similarity Analysis (RSA). Depending on the type of questions we ask and the
answers these systems give, we will show in what cases do VLMs and Humans
converge or diverge allowing us to probe on their cognitive alignment. We find
that the degree of alignment varies significantly depending on the type of
questions asked to each type of system (Humans vs VLMs), highlighting a gap in
their alignment.
Exploiting Instruction-Following Retrievers for Malicious Information
  Retrieval
Authors: name: Parishad BehnamGhader, name: Nicholas Meade, name: Siva Reddy
Published: 2025-03-11 17:36:53

Instruction-following retrievers have been widely adopted alongside LLMs in
real-world applications, but little work has investigated the safety risks
surrounding their increasing search capabilities. We empirically study the
ability of retrievers to satisfy malicious queries, both when used directly and
when used in a retrieval augmented generation-based setup. Concretely, we
investigate six leading retrievers, including NV-Embed and LLM2Vec, and find
that given malicious requests, most retrievers can (for >50% of queries) select
relevant harmful passages. For example, LLM2Vec correctly selects passages for
61.35% of our malicious queries. We further uncover an emerging risk with
instruction-following retrievers, where highly relevant harmful information can
be surfaced by exploiting their instruction-following capabilities. Finally, we
show that even safety-aligned LLMs, such as Llama3, can satisfy malicious
requests when provided with harmful retrieved passages in-context. In summary,
our findings underscore the malicious misuse risks associated with increasing
retriever capability.
CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
  Audiovisual Narrative Processing
Authors: name: Jianxiong Gao, name: Yichang Liu, name: Baofeng Yang, name: Jianfeng Feng, name: Yanwei Fu
Published: 2025-03-10 05:39:43

In this paper, we introduce CineBrain, the first large-scale dataset
featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
stimulation. Recognizing the complementary strengths of EEG's high temporal
resolution and fMRI's deep-brain spatial coverage, CineBrain provides
approximately six hours of narrative-driven content from the popular television
series The Big Bang Theory for each of six participants. Building upon this
unique dataset, we propose CineSync, an innovative multimodal decoding
framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
significantly improving the reconstruction quality of complex audiovisual
stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
comprehensive evaluation protocol that assesses reconstructions across semantic
and perceptual dimensions. Experimental results demonstrate that CineSync
achieves state-of-the-art video reconstruction performance and highlight our
initial success in combining fMRI and EEG for reconstructing both video and
audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.
Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents
Authors: name: Haoyu Wang, name: Sunhao Dai, name: Haiyuan Zhao, name: Liang Pang, name: Xiao Zhang, name: Gang Wang, name: Zhenhua Dong, name: Jun Xu, name: Ji-Rong Wen
Published: 2025-03-11 17:59:00

Previous studies have found that PLM-based retrieval models exhibit a
preference for LLM-generated content, assigning higher relevance scores to
these documents even when their semantic quality is comparable to human-written
ones. This phenomenon, known as source bias, threatens the sustainable
development of the information access ecosystem. However, the underlying causes
of source bias remain unexplored. In this paper, we explain the process of
information retrieval with a causal graph and discover that PLM-based
retrievers learn perplexity features for relevance estimation, causing source
bias by ranking the documents with low perplexity higher. Theoretical analysis
further reveals that the phenomenon stems from the positive correlation between
the gradients of the loss functions in language modeling task and retrieval
task. Based on the analysis, a causal-inspired inference-time debiasing method
is proposed, called Causal Diagnosis and Correction (CDC). CDC first diagnoses
the bias effect of the perplexity and then separates the bias effect from the
overall estimated relevance score. Experimental results across three domains
demonstrate the superior debiasing effectiveness of CDC, emphasizing the
validity of our proposed explanatory framework. Source codes are available at
https://github.com/WhyDwelledOnAi/Perplexity-Trap.
AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion
  Models
Authors: name: Kwan Yun, name: Seokhyeon Hong, name: Chaelin Kim, name: Junyong Noh
Published: 2025-03-11 13:28:59

Despite recent advancements in learning-based motion in-betweening, a key
limitation has been overlooked: the requirement for character-specific
datasets. In this work, we introduce AnyMoLe, a novel method that addresses
this limitation by leveraging video diffusion models to generate motion
in-between frames for arbitrary characters without external data. Our approach
employs a two-stage frame generation process to enhance contextual
understanding. Furthermore, to bridge the domain gap between real-world and
rendered character animations, we introduce ICAdapt, a fine-tuning technique
for video diffusion models. Additionally, we propose a ``motion-video
mimicking'' optimization technique, enabling seamless motion generation for
characters with arbitrary joint structures using 2D and 3D-aware features.
AnyMoLe significantly reduces data dependency while generating smooth and
realistic transitions, making it applicable to a wide range of motion
in-betweening tasks.
Benchmarking AI Models in Software Engineering: A Review, Search Tool,
  and Enhancement Protocol
Authors: name: Roham Koohestani, name: Philippe de Bekker, name: Maliheh Izadi
Published: 2025-03-07 18:44:32

Benchmarks are essential for consistent evaluation and reproducibility. The
integration of Artificial Intelligence into Software Engineering (AI4SE) has
given rise to numerous benchmarks for tasks such as code generation and bug
fixing. However, this surge presents challenges: (1) scattered benchmark
knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3)
the absence of a uniform standard for benchmark development, and (4)
limitations of existing benchmarks. In this paper, we review 173 studies and
identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their
limitations, and expose gaps in practices. Based on our review, we created
BenchScout, a semantic search tool to find relevant benchmarks, using automated
clustering of the contexts from associated studies. We conducted a user study
with 22 participants to evaluate BenchScout's usability, effectiveness, and
intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5.
To advance benchmarking standards, we propose BenchFrame, a unified method to
enhance benchmark quality. As a case study, we applied BenchFrame to the
HumanEval benchmark and addressed its main limitations. This led to
HumanEvalNext, featuring (1) corrected errors, (2) improved language
conversion, (3) expanded test coverage, and (4) increased difficulty. We then
evaluated ten state-of-the-art code language models on HumanEval,
HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1
score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus,
respectively.
Referring to Any Person
Authors: name: Qing Jiang, name: Lin Wu, name: Zhaoyang Zeng, name: Tianhe Ren, name: Yuda Xiong, name: Yihao Chen, name: Qin Liu, name: Lei Zhang
Published: 2025-03-11 14:57:14

Humans are undoubtedly the most important participants in computer vision,
and the ability to detect any individual given a natural language description,
a task we define as referring to any person, holds substantial practical value.
However, we find that existing models generally fail to achieve real-world
usability, and current benchmarks are limited by their focus on one-to-one
referring, that hinder progress in this area. In this work, we revisit this
task from three critical perspectives: task definition, dataset design, and
model architecture. We first identify five aspects of referable entities and
three distinctive characteristics of this task. Next, we introduce HumanRef, a
novel dataset designed to tackle these challenges and better reflect real-world
applications. From a model design perspective, we integrate a multimodal large
language model with an object detection framework, constructing a robust
referring model named RexSeek. Experimental results reveal that
state-of-the-art models, which perform well on commonly used benchmarks like
RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple
individuals. In contrast, RexSeek not only excels in human referring but also
generalizes effectively to common object referring, making it broadly
applicable across various perception tasks. Code is available at
https://github.com/IDEA-Research/RexSeek
^RFLAV: Rolling Flow matching for infinite Audio Video generation
Authors: name: Alex Ergasti, name: Giuseppe Gabriele Tarollo, name: Filippo Botti, name: Tomaso Fontanini, name: Claudio Ferrari, name: Massimo Bertozzi, name: Andrea Prati
Published: 2025-03-11 11:18:47

Joint audio-video (AV) generation is still a significant challenge in
generative AI, primarily due to three critical requirements: quality of the
generated samples, seamless multimodal synchronization and temporal coherence,
with audio tracks that match the visual data and vice versa, and limitless
video duration. In this paper, we present , a novel transformer-based
architecture that addresses all the key challenges of AV generation. We explore
three distinct cross modality interaction modules, with our lightweight
temporal fusion module emerging as the most effective and computationally
efficient approach for aligning audio and visual modalities. Our experimental
results demonstrate that  outperforms existing state-of-the-art models
in multimodal AV generation tasks. Our code and checkpoints are available at
https://github.com/ErgastiAlex/R-FLAV.