Orionnnn/hub-stats huggingface dataset:
MemOS: A Memory OS for AI System
Authors: name: Zhiyu Li, name: Shichao Song, name: Chenyang Xi, name: Hanyu Wang, name: Chen Tang, name: Simin Niu, name: Ding Chen, name: Jiawei Yang, name: Chunyu Li, name: Qingchen Yu, name: Jihao Zhao, name: Yezhaohui Wang, name: Peng Liu, name: Zehao Lin, name: Pengyuan Wang, name: Jiahao Huo, name: Tianyi Chen, name: Kai Chen, name: Kehang Li, name: Zhen Tao, name: Junpeng Ren, name: Huayi Lai, name: Hao Wu, name: Bo Tang, name: Zhenren Wang, name: Zhaoxin Fan, name: Ningyu Zhang, name: Linfeng Zhang, name: Junchi Yan, name: Mingchuan Yang, name: Tong Xu, name: Wei Xu, name: Huajun Chen, name: Haofeng Wang, name: Hongkang Yang, name: Wentao Zhang, name: Zhi-Qin John Xu, name: Siheng Chen, name: Feiyu Xiong
Published: 2025-07-04 17:21:46

Large Language Models (LLMs) have become an essential infrastructure for
Artificial General Intelligence (AGI), yet their lack of well-defined memory
management systems hinders the development of long-context reasoning, continual
personalization, and knowledge consistency.Existing models mainly rely on
static parameters and short-lived contextual states, limiting their ability to
track user preferences or update knowledge over extended periods.While
Retrieval-Augmented Generation (RAG) introduces external knowledge in plain
text, it remains a stateless workaround without lifecycle control or
integration with persistent representations.Recent work has modeled the
training and inference cost of LLMs from a memory hierarchy perspective,
showing that introducing an explicit memory layer between parameter memory and
external retrieval can substantially reduce these costs by externalizing
specific knowledge. Beyond computational efficiency, LLMs face broader
challenges arising from how information is distributed over time and context,
requiring systems capable of managing heterogeneous knowledge spanning
different temporal scales and sources. To address this challenge, we propose
MemOS, a memory operating system that treats memory as a manageable system
resource. It unifies the representation, scheduling, and evolution of
plaintext, activation-based, and parameter-level memories, enabling
cost-efficient storage and retrieval. As the basic unit, a MemCube encapsulates
both memory content and metadata such as provenance and versioning. MemCubes
can be composed, migrated, and fused over time, enabling flexible transitions
between memory types and bridging retrieval with parameter-based learning.
MemOS establishes a memory-centric system framework that brings
controllability, plasticity, and evolvability to LLMs, laying the foundation
for continual learning and personalized modeling.
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative
  Writing
Authors: name: Daniel Fein, name: Sebastian Russo, name: Violet Xiang, name: Kabir Jolly, name: Rafael Rafailov, name: Nick Haber
Published: 2025-07-01 14:10:36

Evaluating creative writing generated by large language models (LLMs) remains
challenging because open-ended narratives lack ground truths. Without
performant automated evaluation methods, off-the-shelf (OTS) language models
are employed as zero-shot judges, yet their reliability is unclear in this
context. In pursuit of robust evaluation for creative writing, we introduce
LitBench, the first standardized benchmark and paired dataset for creative
writing verification, comprising a held-out test set of 2,480 debiased,
human-labeled story comparisons drawn from Reddit and a 43,827-pair training
corpus of human preference labels. Using LitBench, we (i) benchmark zero-shot
LLM judges, (ii) train Bradley Terry and generative reward models, and (iii)
conduct an online human study to validate reward model rankings on newly
LLM-generated stories. Our benchmark identifies Claude-3.7-Sonnet as the
strongest off-the-shelf judge, reaching 73% agreement with human preferences;
among trained reward models, Bradley-Terry and Generative reward models both
attain an accuracy of 78%, outperforming all off-the-shelf judges. An online
human study further confirms that our trained reward models consistently align
with human preferences in novel LLM-generated stories. We release LitBench and
reward models at
https://huggingface.co/collections/SAA-Lab/litbench-68267b5da3aafe58f9e43461,
providing a vetted resource for reliable, automated evaluation and optimization
of creative writing systems.
Lost in Latent Space: An Empirical Study of Latent Diffusion Models for
  Physics Emulation
Authors: name: François Rozet, name: Ruben Ohana, name: Michael McCabe, name: Gilles Louppe, name: François Lanusse, name: Shirley Ho
Published: 2025-07-03 13:32:50

The steep computational cost of diffusion models at inference hinders their
use as fast physics emulators. In the context of image and video generation,
this computational drawback has been addressed by generating in the latent
space of an autoencoder instead of the pixel space. In this work, we
investigate whether a similar strategy can be effectively applied to the
emulation of dynamical systems and at what cost. We find that the accuracy of
latent-space emulation is surprisingly robust to a wide range of compression
rates (up to 1000x). We also show that diffusion-based emulators are
consistently more accurate than non-generative counterparts and compensate for
uncertainty in their predictions with greater diversity. Finally, we cover
practical design choices, spanning from architectures to optimizers, that we
found critical to train latent-space emulators.
How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation
  Models on Standard Computer Vision Tasks
Authors: name: Rahul Ramachandran, name: Ali Garjani, name: Roman Bachmann, name: Andrei Atanov, name: Oğuzhan Fatih Kar, name: Amir Zamir
Published: 2025-07-02 17:59:07

Multimodal foundation models, such as GPT-4o, have recently made remarkable
progress, but it is not clear where exactly these models stand in terms of
understanding vision. In this paper, we benchmark the performance of popular
multimodal foundation models (GPT-4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0
Flash, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard computer vision
tasks (semantic segmentation, object detection, image classification, depth and
surface normal prediction) using established datasets (e.g., COCO, ImageNet and
its variants, etc).
  The main challenges to performing this are: 1) most models are trained to
output text and cannot natively express versatile domains, such as segments or
3D geometry, and 2) many leading models are proprietary and accessible only at
an API level, i.e., there is no weight access to adapt them. We address these
challenges by translating standard vision tasks into equivalent text-promptable
and API-compatible tasks via prompt chaining to create a standardized
benchmarking framework.
  We observe that 1) the models are not close to the state-of-the-art
specialist models at any task. However, 2) they are respectable generalists;
this is remarkable as they are presumably trained on primarily image-text-based
tasks. 3) They perform semantic tasks notably better than geometric ones. 4)
While the prompt-chaining techniques affect performance, better models exhibit
less sensitivity to prompt variations. 5) GPT-4o performs the best among
non-reasoning models, securing the top position in 4 out of 6 tasks, 6)
reasoning models, e.g. o3, show improvements in geometric tasks, and 7) a
preliminary analysis of models with native image generation, like the latest
GPT-4o, shows they exhibit quirks like hallucinations and spatial
misalignments.
Eka-Eval : A Comprehensive Evaluation Framework for Large Language
  Models in Indian Languages
Authors: name: Samridhi Raj Sinha, name: Rajvee Sheth, name: Abhishek Upperwal, name: Mayank Singh
Published: 2025-07-02 16:07:54

The rapid advancement of Large Language Models (LLMs) has intensified the
need for evaluation frameworks that go beyond English centric benchmarks and
address the requirements of linguistically diverse regions such as India. We
present EKA-EVAL, a unified and production-ready evaluation framework that
integrates over 35 benchmarks, including 10 Indic-specific datasets, spanning
categories like reasoning, mathematics, tool use, long-context understanding,
and reading comprehension. Compared to existing Indian language evaluation
tools, EKA-EVAL offers broader benchmark coverage, with built-in support for
distributed inference, quantization, and multi-GPU usage. Our systematic
comparison positions EKA-EVAL as the first end-to-end, extensible evaluation
suite tailored for both global and Indic LLMs, significantly lowering the
barrier to multilingual benchmarking. The framework is open-source and publicly
available at https://github.com/lingo-iitgn/ eka-eval and a part of ongoing EKA
initiative (https://eka.soket.ai), which aims to scale up to over 100
benchmarks and establish a robust, multilingual evaluation ecosystem for LLMs.
CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for
  Multi-Organ Segmentation
Authors: name: Xinlei Yu, name: Chanmiao Wang, name: Hui Jin, name: Ahmed Elazab, name: Gangyong Jia, name: Xiang Wan, name: Changqing Zou, name: Ruiquan Ge
Published: 2025-06-29 07:05:27

Multi-organ medical segmentation is a crucial component of medical image
processing, essential for doctors to make accurate diagnoses and develop
effective treatment plans. Despite significant progress in this field, current
multi-organ segmentation models often suffer from inaccurate details,
dependence on geometric prompts and loss of spatial information. Addressing
these challenges, we introduce a novel model named CRISP-SAM2 with CRoss-modal
Interaction and Semantic Prompting based on SAM2. This model represents a
promising approach to multi-organ medical segmentation guided by textual
descriptions of organs. Our method begins by converting visual and textual
inputs into cross-modal contextualized semantics using a progressive
cross-attention interaction mechanism. These semantics are then injected into
the image encoder to enhance the detailed understanding of visual information.
To eliminate reliance on geometric prompts, we use a semantic prompting
strategy, replacing the original prompt encoder to sharpen the perception of
challenging targets. In addition, a similarity-sorting self-updating strategy
for memory and a mask-refining process is applied to further adapt to medical
imaging and enhance localized details. Comparative experiments conducted on
seven public datasets indicate that CRISP-SAM2 outperforms existing models.
Extensive analysis also demonstrates the effectiveness of our method, thereby
confirming its superior performance, especially in addressing the limitations
mentioned earlier. Our code is available at:
https://github.com/YU-deep/CRISP\_SAM2.git.
HalluSegBench: Counterfactual Visual Reasoning for Segmentation
  Hallucination Evaluation
Authors: name: Xinzhuo Li, name: Adheesh Juvekar, name: Xingyou Liu, name: Muntasir Wahed, name: Kiet A. Nguyen, name: Ismini Lourentzou
Published: 2025-06-26 17:59:12

Recent progress in vision-language segmentation has significantly advanced
grounded visual understanding. However, these models often exhibit
hallucinations by producing segmentation masks for objects not grounded in the
image content or by incorrectly labeling irrelevant regions. Existing
evaluation protocols for segmentation hallucination primarily focus on label or
textual hallucinations without manipulating the visual context, limiting their
capacity to diagnose critical failures. In response, we introduce
HalluSegBench, the first benchmark specifically designed to evaluate
hallucinations in visual grounding through the lens of counterfactual visual
reasoning. Our benchmark consists of a novel dataset of 1340 counterfactual
instance pairs spanning 281 unique object classes, and a set of newly
introduced metrics that quantify hallucination sensitivity under visually
coherent scene edits. Experiments on HalluSegBench with state-of-the-art
vision-language segmentation models reveal that vision-driven hallucinations
are significantly more prevalent than label-driven ones, with models often
persisting in false segmentation, highlighting the need for counterfactual
reasoning to diagnose grounding fidelity.
Heeding the Inner Voice: Aligning ControlNet Training via Intermediate
  Features Feedback
Authors: name: Nina Konovalova, name: Maxim Nikolaev, name: Andrey Kuznetsov, name: Aibek Alanov
Published: 2025-07-03 05:25:53

Despite significant progress in text-to-image diffusion models, achieving
precise spatial control over generated outputs remains challenging. ControlNet
addresses this by introducing an auxiliary conditioning module, while
ControlNet++ further refines alignment through a cycle consistency loss applied
only to the final denoising steps. However, this approach neglects intermediate
generation stages, limiting its effectiveness. We propose InnerControl, a
training strategy that enforces spatial consistency across all diffusion steps.
Our method trains lightweight convolutional probes to reconstruct input control
signals (e.g., edges, depth) from intermediate UNet features at every denoising
step. These probes efficiently extract signals even from highly noisy latents,
enabling pseudo ground truth controls for training. By minimizing the
discrepancy between predicted and target conditions throughout the entire
diffusion process, our alignment loss improves both control fidelity and
generation quality. Combined with established techniques like ControlNet++,
InnerControl achieves state-of-the-art performance across diverse conditioning
methods (e.g., edges, depth).
Self-Correction Bench: Revealing and Addressing the Self-Correction
  Blind Spot in LLMs
Authors: name: Ken Tsui
Published: 2025-07-03 16:41:30

Although large language models (LLMs) have become transformative, they still
make mistakes and can explore unproductive reasoning paths. Self-correction is
an important capability for a trustworthy LLM, particularly an autoregressive
LLM. While LLMs can identify error in user input, they exhibit a systematic
'Self-Correction Blind Spot' - failing to correct identical error in their own
outputs. To systematically study this phenomenon, we introduce Self-Correction
Bench, a systematic framework to measure this phenomenon through controlled
error injection at three complexity levels. Testing 14 models, we find an
average 64.5% blind spot rate. We find multiple evidences that this limitation
relates to training data composition: human training demonstrations
predominantly show error-free responses rather than error-correction sequences,
unlike RL-trained models that learn error correction through outcome feedback.
Remarkably, simply appending "Wait" reduces blind spots by 89.3%, suggesting
that the capability exists but requires activation. Our work highlights a
critical limitation in current LLMs and offers potential avenues for improving
their reliability and trustworthiness.
ZeCO: Zero Communication Overhead Sequence Parallelism for Linear
  Attention
Authors: name: Yuhong Chou, name: Zehao Liu, name: Ruijie Zhu, name: Xinyi Wan, name: Tianjian Li, name: Congying Chu, name: Qian Liu, name: Jibin Wu, name: Zejun Ma
Published: 2025-07-01 17:54:53

Linear attention mechanisms deliver significant advantages for Large Language
Models (LLMs) by providing linear computational complexity, enabling efficient
processing of ultra-long sequences (e.g., 1M context). However, existing
Sequence Parallelism (SP) methods, essential for distributing these workloads
across devices, become the primary bottleneck due to substantial communication
overhead. In this paper, we introduce ZeCO (Zero Communication Overhead)
sequence parallelism for linear attention models, a new SP method designed to
overcome these limitations and achieve end-to-end near-linear scalability for
long sequence training. For example, training a model with a 1M sequence length
across 64 devices using ZeCO takes roughly the same time as training with an
16k sequence on a single device. At the heart of ZeCO lies All-Scan, a new
collective communication primitive. All-Scan provides each SP rank with
precisely the initial operator state it requires while maintaining a minimal
communication footprint, effectively eliminating communication overhead.
Theoretically, we prove the optimaity of ZeCO, showing that it introduces only
negligible time and space overhead. Empirically, we compare the communication
costs of different sequence parallelism strategies and demonstrate that
All-Scan achieves the fastest communication in SP scenarios. Specifically, on
256 GPUs with an 8M sequence length, ZeCO achieves a 60\% speedup compared to
the current state-of-the-art (SOTA) SP method. We believe ZeCO establishes a
clear path toward efficiently training next-generation LLMs on previously
intractable sequence lengths.
Selecting and Merging: Towards Adaptable and Scalable Named Entity
  Recognition with Large Language Models
Authors: name: Zhuojun Ding, name: Wei Wei, name: Chenghao Fan
Published: 2025-06-28 08:28:52

Supervised fine-tuning (SFT) is widely used to align large language models
(LLMs) with information extraction (IE) tasks, such as named entity recognition
(NER). However, annotating such fine-grained labels and training
domain-specific models is costly. Existing works typically train a unified
model across multiple domains, but such approaches lack adaptation and
scalability since not all training data benefits target domains and scaling
trained models remains challenging. We propose the SaM framework, which
dynamically Selects and Merges expert models at inference time. Specifically,
for a target domain, we select domain-specific experts pre-trained on existing
domains based on (i) domain similarity to the target domain and (ii)
performance on sampled instances, respectively. The experts are then merged to
create task-specific models optimized for the target domain. By dynamically
merging experts beneficial to target domains, we improve generalization across
various domains without extra training. Additionally, experts can be added or
removed conveniently, leading to great scalability. Extensive experiments on
multiple benchmarks demonstrate our framework's effectiveness, which
outperforms the unified model by an average of 10%. We further provide insights
into potential improvements, practical experience, and extensions of our
framework.
Energy-Based Transformers are Scalable Learners and Thinkers
Authors: name: Alexi Gladstone, name: Ganesh Nanduru, name: Md Mofijul Islam, name: Peixuan Han, name: Hyeonjeong Ha, name: Aman Chadha, name: Yilun Du, name: Heng Ji, name: Jundong Li, name: Tariq Iqbal
Published: 2025-07-02 19:17:29

Inference-time computation techniques, analogous to human System 2 Thinking,
have recently become popular for improving model performances. However, most
existing approaches suffer from several limitations: they are modality-specific
(e.g., working only in text), problem-specific (e.g., verifiable domains like
math and coding), or require additional supervision/training on top of
unsupervised pretraining (e.g., verifiers or verifiable rewards). In this
paper, we ask the question "Is it possible to generalize these System 2
Thinking approaches, and develop models that learn to think solely from
unsupervised learning?" Interestingly, we find the answer is yes, by learning
to explicitly verify the compatibility between inputs and
candidate-predictions, and then re-framing prediction problems as optimization
with respect to this verifier. Specifically, we train Energy-Based Transformers
(EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy
value to every input and candidate-prediction pair, enabling predictions
through gradient descent-based energy minimization until convergence. Across
both discrete (text) and continuous (visual) modalities, we find EBTs scale
faster than the dominant Transformer++ approach during training, achieving an
up to 35% higher scaling rate with respect to data, batch size, parameters,
FLOPs, and depth. During inference, EBTs improve performance with System 2
Thinking by 29% more than the Transformer++ on language tasks, and EBTs
outperform Diffusion Transformers on image denoising while using fewer forward
passes. Further, we find that EBTs achieve better results than existing models
on most downstream tasks given the same or worse pretraining performance,
suggesting that EBTs generalize better than existing approaches. Consequently,
EBTs are a promising new paradigm for scaling both the learning and thinking
capabilities of models.
IntFold: A Controllable Foundation Model for General and Specialized
  Biomolecular Structure Prediction
Authors: name: The IntFold Team, name: Leon Qiao, name: Wayne Bai, name: He Yan, name: Gary Liu, name: Nova Xi, name: Xiang Zhang
Published: 2025-07-02 16:09:47

We introduce IntFold, a controllable foundation model for both general and
specialized biomolecular structure prediction. IntFold demonstrates predictive
accuracy comparable to the state-of-the-art AlphaFold3, while utilizing a
superior customized attention kernel. Beyond standard structure prediction,
IntFold can be adapted to predict allosteric states, constrained structures,
and binding affinity through the use of individual adapters. Furthermore, we
introduce a novel confidence head to estimate docking quality, offering a more
nuanced assessment for challenging targets such as antibody-antigen complexes.
Finally, we share insights gained during the training process of this
computationally intensive model.
Can LLMs Identify Critical Limitations within Scientific Research? A
  Systematic Evaluation on AI Research Papers
Authors: name: Zhijian Xu, name: Yilun Zhao, name: Manasi Patwardhan, name: Lovekesh Vig, name: Arman Cohan
Published: 2025-07-03 15:04:38

Peer review is fundamental to scientific research, but the growing volume of
publications has intensified the challenges of this expertise-intensive
process. While LLMs show promise in various scientific tasks, their potential
to assist with peer review, particularly in identifying paper limitations,
remains understudied. We first present a comprehensive taxonomy of limitation
types in scientific research, with a focus on AI. Guided by this taxonomy, for
studying limitations, we present LimitGen, the first comprehensive benchmark
for evaluating LLMs' capability to support early-stage feedback and complement
human peer review. Our benchmark consists of two subsets: LimitGen-Syn, a
synthetic dataset carefully created through controlled perturbations of
high-quality papers, and LimitGen-Human, a collection of real human-written
limitations. To improve the ability of LLM systems to identify limitations, we
augment them with literature retrieval, which is essential for grounding
identifying limitations in prior scientific findings. Our approach enhances the
capabilities of LLM systems to generate limitations in research papers,
enabling them to provide more concrete and constructive feedback.