Orionnnn/hub-stats huggingface dataset:
Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers
Authors: name: Zhiyuan Peng, name: Ting-ruen Wei, name: Tingyu Song, name: Yilun Zhao, name: Yi Fang
Published: 2025-07-08 17:56:28

Large Language Models (LLMs) have recently been applied to reranking tasks in
information retrieval, achieving strong performance. However, their high
computational demands often hinder practical deployment. Existing studies
evaluate the efficiency of LLM-based rerankers using proxy metrics such as
latency, the number of forward passes, input tokens, and output tokens.
However, these metrics depend on hardware and running-time choices (\eg
parallel or not, batch size, etc), and often fail to account for model size,
making it difficult to interpret and obscuring the evaluation of the
efficiency-effectiveness tradeoff. To address this issue, we propose
E2R-FLOPs, for LLM-based rerankers: ranking metrics per
PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for
hardware-agnostic throughput. Companied with the new metrics, an interpretable
FLOPs estimator is built to estimate the FLOPs of an LLM-based reranker even
without running any experiments. Based on the proposed metrics, we conduct
comprehensive experiments to evaluate a wide range of LLM-based rerankers with
different architecture, studying the efficiency-effectiveness trade-off and
bringing this issue to the attention of the research community.
The Landscape of Memorization in LLMs: Mechanisms, Measurement, and
  Mitigation
Authors: name: Alexander Xiong, name: Xuandong Zhao, name: Aneesh Pappu, name: Dawn Song
Published: 2025-07-08 01:30:46

Large Language Models (LLMs) have demonstrated remarkable capabilities across
a wide range of tasks, yet they also exhibit memorization of their training
data. This phenomenon raises critical questions about model behavior, privacy
risks, and the boundary between learning and memorization. Addressing these
concerns, this paper synthesizes recent studies and investigates the landscape
of memorization, the factors influencing it, and methods for its detection and
mitigation. We explore key drivers, including training data duplication,
training dynamics, and fine-tuning procedures that influence data memorization.
In addition, we examine methodologies such as prefix-based extraction,
membership inference, and adversarial prompting, assessing their effectiveness
in detecting and measuring memorized content. Beyond technical analysis, we
also explore the broader implications of memorization, including the legal and
ethical implications. Finally, we discuss mitigation strategies, including data
cleaning, differential privacy, and post-training unlearning, while
highlighting open challenges in balancing the minimization of harmful
memorization with utility. This paper provides a comprehensive overview of the
current state of research on LLM memorization across technical, privacy, and
performance dimensions, identifying critical directions for future work.
SAMed-2: Selective Memory Enhanced Medical Segment Anything Model
Authors: name: Zhiling Yan, name: Sifan Song, name: Dingjie Song, name: Yiwei Li, name: Rong Zhou, name: Weixiang Sun, name: Zhennong Chen, name: Sekeun Kim, name: Hui Ren, name: Tianming Liu, name: Quanzheng Li, name: Xiang Li, name: Lifang He, name: Lichao Sun
Published: 2025-07-04 16:30:38

Recent "segment anything" efforts show promise by learning from large-scale
data, but adapting such models directly to medical images remains challenging
due to the complexity of medical data, noisy annotations, and continual
learning requirements across diverse modalities and anatomical structures. In
this work, we propose SAMed-2, a new foundation model for medical image
segmentation built upon the SAM-2 architecture. Specifically, we introduce a
temporal adapter into the image encoder to capture image correlations and a
confidence-driven memory mechanism to store high-certainty features for later
retrieval. This memory-based strategy counters the pervasive noise in
large-scale medical datasets and mitigates catastrophic forgetting when
encountering new tasks or modalities. To train and evaluate SAMed-2, we curate
MedBank-100k, a comprehensive dataset spanning seven imaging modalities and 21
medical segmentation tasks. Our experiments on both internal benchmarks and 10
external datasets demonstrate superior performance over state-of-the-art
baselines in multi-task scenarios. The code is available at:
https://github.com/ZhilingYan/Medical-SAM-Bench.
R1-RE: Cross-Domain Relationship Extraction with RLVR
Authors: name: Runpeng Dai, name: Tong Zheng, name: Run Yang, name: Hongtu Zhu
Published: 2025-07-07 03:50:59

Relationship extraction (RE) is a core task in natural language processing.
Traditional approaches typically frame RE as a supervised learning problem,
directly mapping context to labels-an approach that often suffers from poor
out-of-domain (OOD) generalization. Inspired by the workflow of human
annotators, we reframe RE as a reasoning task guided by annotation guidelines
and introduce R1-RE, the first reinforcement learning with verifiable reward
(RLVR) framework for RE tasks. Our method elicits the reasoning abilities of
small language models for annotation tasks, resulting in significantly improved
OOD robustness. We evaluate our approach on the public Sem-2010 dataset and a
private MDKG dataset. The R1-RE-7B model attains an average OOD accuracy of
approximately 70%, on par with leading proprietary models such as GPT-4o.
Additionally, our comprehensive analysis provides novel insights into the
training dynamics and emergent reasoning behaviors of the RLVR paradigm for RE.
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Authors: name: Yuanzhe Hu, name: Yu Wang, name: Julian McAuley
Published: 2025-07-07 17:59:54

Recent benchmarks for Large Language Model (LLM) agents primarily focus on
evaluating reasoning, planning, and execution capabilities, while another
critical component-memory, encompassing how agents memorize, update, and
retrieve long-term information-is under-evaluated due to the lack of
benchmarks. We term agents with memory mechanisms as memory agents. In this
paper, we identify four core competencies essential for memory agents: accurate
retrieval, test-time learning, long-range understanding, and conflict
resolution. Existing datasets either rely on limited context lengths or are
tailored for static, long-context settings like book-based QA, which do not
reflect the interactive, multi-turn nature of memory agents that incrementally
accumulate information. Furthermore, no existing benchmarks cover all four
competencies. Therefore, we introduce MemoryAgentBench, a new benchmark
specifically designed for memory agents. Our benchmark combines reformulated
existing datasets with newly constructed ones, covering the above four memory
competencies, providing a systematic and challenging testbed for assessing
memory quality. We evaluate a diverse set of memory agents, ranging from simple
context-based and retrieval-augmented generation (RAG) systems to advanced
agents with external memory modules and tool integration. Empirical results
reveal that current methods fall short of mastering all four competencies,
underscoring the need for further research into comprehensive memory mechanisms
for LLM agents.
Preserving Privacy, Increasing Accessibility, and Reducing Cost: An
  On-Device Artificial Intelligence Model for Medical Transcription and Note
  Generation
Authors: name: Johnson Thomas, name: Ayush Mudgal, name: Wendao Liu, name: Nisten Tahiraj, name: Zeeshaan Mohammed, name: Dhruv Diddi
Published: 2025-07-03 01:51:49

Background: Clinical documentation represents a significant burden for
healthcare providers, with physicians spending up to 2 hours daily on
administrative tasks. Recent advances in large language models (LLMs) offer
promising solutions, but privacy concerns and computational requirements limit
their adoption in healthcare settings. Objective: To develop and evaluate a
privacy-preserving, on-device medical transcription system using a fine-tuned
Llama 3.2 1B model capable of generating structured medical notes from medical
transcriptions while maintaining complete data sovereignty entirely in the
browser. Methods: We fine-tuned a Llama 3.2 1B model using Parameter-Efficient
Fine-Tuning (PEFT) with LoRA on 1,500 synthetic medical
transcription-to-structured note pairs. The model was evaluated against the
base Llama 3.2 1B on two datasets: 100 endocrinology transcripts and 140
modified ACI benchmark cases. Evaluation employed both statistical metrics
(ROUGE, BERTScore, BLEURT) and LLM-as-judge assessments across multiple
clinical quality dimensions. Results: The fine-tuned OnDevice model
demonstrated substantial improvements over the base model. On the ACI
benchmark, ROUGE-1 scores increased from 0.346 to 0.496, while BERTScore F1
improved from 0.832 to 0.866. Clinical quality assessments showed marked
reduction in major hallucinations (from 85 to 35 cases) and enhanced factual
correctness (2.81 to 3.54 on 5-point scale). Similar improvements were observed
on the internal evaluation dataset, with composite scores increasing from 3.13
to 4.43 (+41.5%). Conclusions: Fine-tuning compact LLMs for medical
transcription yields clinically meaningful improvements while enabling complete
on-device browser deployment. This approach addresses key barriers to AI
adoption in healthcare: privacy preservation, cost reduction, and accessibility
for resource-constrained environments.
UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields
Authors: name: Fabian Perez, name: Sara Rojas, name: Carlos Hinojosa, name: Hoover Rueda-Chacón, name: Bernard Ghanem
Published: 2025-06-27 03:42:49

Neural Radiance Field (NeRF)-based segmentation methods focus on object
semantics and rely solely on RGB data, lacking intrinsic material properties.
This limitation restricts accurate material perception, which is crucial for
robotics, augmented reality, simulation, and other applications. We introduce
UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling
joint hyperspectral novel view synthesis and unsupervised material
segmentation. Our method models spectral reflectance via diffuse and specular
components, where a learned dictionary of global endmembers represents pure
material signatures, and per-point abundances capture their distribution. For
material segmentation, we use spectral signature predictions along learned
endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF
enables scene editing by modifying learned endmember dictionaries for flexible
material-based appearance manipulation. Extensive experiments validate our
approach, demonstrating superior spectral reconstruction and material
segmentation to existing methods. Project page:
https://www.factral.co/UnMix-NeRF.
On the rankability of visual embeddings
Authors: name: Ankit Sonthalia, name: Arnas Uselis, name: Seong Joon Oh
Published: 2025-07-04 16:03:31

We study whether visual embedding models capture continuous, ordinal
attributes along linear directions, which we term _rank axes_. We define a
model as _rankable_ for an attribute if projecting embeddings onto such an axis
preserves the attribute's order. Across 7 popular encoders and 9 datasets with
attributes like age, crowd count, head pose, aesthetics, and recency, we find
that many embeddings are inherently rankable. Surprisingly, a small number of
samples, or even just two extreme examples, often suffice to recover meaningful
rank axes, without full-scale supervision. These findings open up new use cases
for image ranking in vector databases and motivate further study into the
structure and learning of rankable embeddings. Our code is available at
https://github.com/aktsonthalia/rankable-vision-embeddings.
SeqTex: Generate Mesh Textures in Video Sequence
Authors: name: Ze Yuan, name: Xin Yu, name: Yangtian Sun, name: Yuan-Chen Guo, name: Yan-Pei Cao, name: Ding Liang, name: Xiaojuan Qi
Published: 2025-07-06 07:58:36

Training native 3D texture generative models remains a fundamental yet
challenging problem, largely due to the limited availability of large-scale,
high-quality 3D texture datasets. This scarcity hinders generalization to
real-world scenarios. To address this, most existing methods finetune
foundation image generative models to exploit their learned visual priors.
However, these approaches typically generate only multi-view images and rely on
post-processing to produce UV texture maps -- an essential representation in
modern graphics pipelines. Such two-stage pipelines often suffer from error
accumulation and spatial inconsistencies across the 3D surface. In this paper,
we introduce SeqTex, a novel end-to-end framework that leverages the visual
knowledge encoded in pretrained video foundation models to directly generate
complete UV texture maps. Unlike previous methods that model the distribution
of UV textures in isolation, SeqTex reformulates the task as a sequence
generation problem, enabling the model to learn the joint distribution of
multi-view renderings and UV textures. This design effectively transfers the
consistent image-space priors from video foundation models into the UV domain.
To further enhance performance, we propose several architectural innovations: a
decoupled multi-view and UV branch design, geometry-informed attention to guide
cross-domain feature alignment, and adaptive token resolution to preserve fine
texture details while maintaining computational efficiency. Together, these
components allow SeqTex to fully utilize pretrained video priors and synthesize
high-fidelity UV texture maps without the need for post-processing. Extensive
experiments show that SeqTex achieves state-of-the-art performance on both
image-conditioned and text-conditioned 3D texture generation tasks, with
superior 3D consistency, texture-geometry alignment, and real-world
generalization.
Should We Still Pretrain Encoders with Masked Language Modeling?
Authors: name: Hippolyte Gisserot-Boukhlef, name: Nicolas Boizard, name: Manuel Faysse, name: Duarte M. Alves, name: Emmanuel Malherbe, name: André F. T. Martins, name: Céline Hudelot, name: Pierre Colombo
Published: 2025-07-01 17:45:48

Learning high-quality text representations is fundamental to a wide range of
NLP tasks. While encoder pretraining has traditionally relied on Masked
Language Modeling (MLM), recent evidence suggests that decoder models
pretrained with Causal Language Modeling (CLM) can be effectively repurposed as
encoders, often surpassing traditional encoders on text representation
benchmarks. However, it remains unclear whether these gains reflect an inherent
advantage of the CLM objective or arise from confounding factors such as model
and data scale. In this paper, we address this question through a series of
large-scale, carefully controlled pretraining ablations, training a total of 30
models ranging from 210 million to 1 billion parameters, and conducting over
15,000 fine-tuning and evaluation runs. We find that while training with MLM
generally yields better performance across text representation tasks,
CLM-trained models are more data-efficient and demonstrate improved fine-tuning
stability. Building on these findings, we experimentally show that a biphasic
training strategy that sequentially applies CLM and then MLM, achieves optimal
performance under a fixed computational training budget. Moreover, we
demonstrate that this strategy becomes more appealing when initializing from
readily available pretrained CLM models (from the existing LLM ecosystem),
reducing the computational burden needed to train best-in-class encoder models.
We release all project artifacts at https://hf.co/MLMvsCLM to foster further
research.
VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity
  Classification
Authors: name: Cédric Bonhomme, name: Alexandre Dulaunoy
Published: 2025-07-04 14:28:14

This paper presents VLAI, a transformer-based model that predicts software
vulnerability severity levels directly from text descriptions. Built on
RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and
achieves over 82% accuracy in predicting severity categories, enabling faster
and more consistent triage ahead of manual CVSS scoring. The model and dataset
are open-source and integrated into the Vulnerability-Lookup service.
Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM
  Fine-Tuning Data from Unstructured Documents
Authors: name: Ziyang Miao, name: Qiyu Sun, name: Jingyuan Wang, name: Yuchen Gong, name: Yaowei Zheng, name: Shiqi Li, name: Richong Zhang
Published: 2025-07-05 11:38:59

Large language models (LLMs) have shown impressive performance on
general-purpose tasks, yet adapting them to specific domains remains
challenging due to the scarcity of high-quality domain data. Existing data
synthesis tools often struggle to extract reliable fine-tuning data from
heterogeneous documents effectively. To address this limitation, we propose
Easy Dataset, a unified framework for synthesizing fine-tuning data from
unstructured documents via an intuitive graphical user interface (GUI).
Specifically, Easy Dataset allows users to easily configure text extraction
models and chunking strategies to transform raw documents into coherent text
chunks. It then leverages a persona-driven prompting approach to generate
diverse question-answer pairs using public-available LLMs. Throughout the
pipeline, a human-in-the-loop visual interface facilitates the review and
refinement of intermediate outputs to ensure data quality. Experiments on a
financial question-answering task show that fine-tuning LLMs on the synthesized
dataset significantly improves domain-specific performance while preserving
general knowledge. The source code and installable package are available at
https://github.com/ConardLi/easy-dataset and have garnered over 9,000 GitHub
stars.
RoboBrain 2.0 Technical Report
Authors: name: BAAI RoboBrain Team, name: Mingyu Cao, name: Huajie Tan, name: Yuheng Ji, name: Minglan Lin, name: Zhiyu Li, name: Zhou Cao, name: Pengwei Wang, name: Enshen Zhou, name: Yi Han, name: Yingbo Tang, name: Xiangqi Xu, name: Wei Guo, name: Yaoxu Lyu, name: Yijie Xu, name: Jiayu Shi, name: Cheng Chi, name: Mengdi Zhao, name: Xiaoshuai Hao, name: Shanyu Rong, name: Zhengliang Cai, name: Bolun Zhang, name: Shuyi Zhang, name: Huaihai Lyu, name: Mengfei Du, name: Lingfeng Zhang, name: Xi Feng, name: Xiaodan Liu, name: Yance Jiao, name: Chenrui He, name: Mengsi Lyu, name: Zhuo Chen, name: Yulong Ao, name: Xue Sun, name: Zheqi He, name: Jingshu Zheng, name: Xi Yang, name: Donghai Shi, name: Kunchang Xie, name: Bochao Zhang, name: Shaokai Nie, name: Chunlei Men, name: Yonghua Lin, name: Zhongyuan Wang, name: Tiejun Huang, name: Shanghang Zhang
Published: 2025-07-02 17:05:33

We introduce RoboBrain 2.0, our latest generation of embodied vision-language
foundation models, designed to unify perception, reasoning, and planning for
complex embodied tasks in physical environments. It comes in two variants: a
lightweight 7B model and a full-scale 32B model, featuring a heterogeneous
architecture with a vision encoder and a language model. Despite its compact
size, RoboBrain 2.0 achieves strong performance across a wide spectrum of
embodied reasoning tasks. On both spatial and temporal benchmarks, the 32B
variant achieves leading results, surpassing prior open-source and proprietary
models. In particular, it supports key real-world embodied AI capabilities,
including spatial understanding (e.g., affordance prediction, spatial
referring, trajectory forecasting) and temporal decision-making (e.g.,
closed-loop interaction, multi-agent long-horizon planning, and scene graph
updating). This report details the model architecture, data construction,
multi-stage training strategies, infrastructure and practical applications. We
hope RoboBrain 2.0 advances embodied AI research and serves as a practical step
toward building generalist embodied agents. The code, checkpoint and benchmark
are available at https://superrobobrain.github.io.
Reviving Cultural Heritage: A Novel Approach for Comprehensive
  Historical Document Restoration
Authors: name: Yuyi Zhang, name: Peirong Zhang, name: Zhenhua Yang, name: Pengyu Yan, name: Yongxin Shi, name: Pengwei Liu, name: Fengjun Guo, name: Lianwen Jin
Published: 2025-07-07 15:26:17

Historical documents represent an invaluable cultural heritage, yet have
undergone significant degradation over time through tears, water erosion, and
oxidation. Existing Historical Document Restoration (HDR) methods primarily
focus on single modality or limited-size restoration, failing to meet practical
needs. To fill this gap, we present a full-page HDR dataset (FPHDR) and a novel
automated HDR solution (AutoHDR). Specifically, FPHDR comprises 1,633 real and
6,543 synthetic images with character-level and line-level locations, as well
as character annotations in different damage grades. AutoHDR mimics historians'
restoration workflows through a three-stage approach: OCR-assisted damage
localization, vision-language context text prediction, and patch autoregressive
appearance restoration. The modular architecture of AutoHDR enables seamless
human-machine collaboration, allowing for flexible intervention and
optimization at each restoration stage. Experiments demonstrate AutoHDR's
remarkable performance in HDR. When processing severely damaged documents, our
method improves OCR accuracy from 46.83\% to 84.05\%, with further enhancement
to 94.25\% through human-machine collaboration. We believe this work represents
a significant advancement in automated historical document restoration and
contributes substantially to cultural heritage preservation. The model and
dataset are available at https://github.com/SCUT-DLVCLab/AutoHDR.
StreamDiT: Real-Time Streaming Text-to-Video Generation
Authors: name: Akio Kodaira, name: Tingbo Hou, name: Ji Hou, name: Masayoshi Tomizuka, name: Yue Zhao
Published: 2025-07-04 18:00:01

Recently, great progress has been achieved in text-to-video (T2V) generation
by scaling transformer-based diffusion models to billions of parameters, which
can generate high-quality videos. However, existing models typically produce
only short clips offline, restricting their use cases in interactive and
real-time applications. This paper addresses these challenges by proposing
StreamDiT, a streaming video generation model. StreamDiT training is based on
flow matching by adding a moving buffer. We design mixed training with
different partitioning schemes of buffered frames to boost both content
consistency and visual quality. StreamDiT modeling is based on adaLN DiT with
varying time embedding and window attention. To practice the proposed method,
we train a StreamDiT model with 4B parameters. In addition, we propose a
multistep distillation method tailored for StreamDiT. Sampling distillation is
performed in each segment of a chosen partitioning scheme. After distillation,
the total number of function evaluations (NFEs) is reduced to the number of
chunks in a buffer. Finally, our distilled model reaches real-time performance
at 16 FPS on one GPU, which can generate video streams at 512p resolution. We
evaluate our method through both quantitative metrics and human evaluation. Our
model enables real-time applications, e.g. streaming generation, interactive
generation, and video-to-video. We provide video results and more examples in
our project website: <a href="https://cumulo-autumn.github.io/StreamDiT/">this
https URL.</a>