Orionnnn/hub-stats huggingface dataset:
AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda
Authors: name: Mohd Nauman, name: Sravan Gvm, name: Vijay Devane, name: Shyam Pawar, name: Viraj Thakur, name: Kundeshwar Pundalik, name: Piyush Sawarkar, name: Rohit Saluja, name: Maunendra Desarkar, name: Ganesh Ramakrishnan
Published: 2025-11-04 08:53:21

Current large language models excel at broad, general-purpose tasks, but
consistently underperform when exposed to highly specialized domains that
require deep cultural, linguistic, and subject-matter expertise. In particular,
traditional medical systems such as Ayurveda embody centuries of nuanced
textual and clinical knowledge that mainstream LLMs fail to accurately
interpret or apply. We introduce AyurParam-2.9B, a domain-specialized,
bilingual language model fine-tuned from Param-1-2.9B using an extensive,
expertly curated Ayurveda dataset spanning classical texts and clinical
guidance. AyurParam's dataset incorporates context-aware, reasoning, and
objective-style Q&A in both English and Hindi, with rigorous annotation
protocols for factual precision and instructional clarity. Benchmarked on
BhashaBench-Ayur, AyurParam not only surpasses all open-source
instruction-tuned models in its size class (1.5--3B parameters), but also
demonstrates competitive or superior performance compared to much larger
models. The results from AyurParam highlight the necessity for authentic domain
adaptation and high-quality supervision in delivering reliable, culturally
congruent AI for specialized medical knowledge.
Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for
  Improving Video Generation
Authors: name: Jie Du, name: Xinyu Gong, name: Qingshan Tan, name: Wen Li, name: Yangming Cheng, name: Weitao Wang, name: Chenlu Zhan, name: Suhui Wu, name: Hao Zhang, name: Jun Zhang
Published: 2025-11-03 11:04:22

Recent studies have identified Direct Preference Optimization (DPO) as an
efficient and reward-free approach to improving video generation quality.
However, existing methods largely follow image-domain paradigms and are mainly
developed on small-scale models (approximately 2B parameters), limiting their
ability to address the unique challenges of video tasks, such as costly data
construction, unstable training, and heavy memory consumption. To overcome
these limitations, we introduce a GT-Pair that automatically builds
high-quality preference pairs by using real videos as positives and
model-generated videos as negatives, eliminating the need for any external
annotation. We further present Reg-DPO, which incorporates the SFT loss as a
regularization term into the DPO objective to enhance training stability and
generation fidelity. Additionally, by combining the FSDP framework with
multiple memory optimization techniques, our approach achieves nearly three
times higher training capacity than using FSDP alone. Extensive experiments on
both I2V and T2V tasks across multiple datasets demonstrate that our method
consistently outperforms existing approaches, delivering superior video
generation quality.
RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies
Authors: name: Adina Yakefu, name: Bin Xie, name: Chongyang Xu, name: Enwen Zhang, name: Erjin Zhou, name: Fan Jia, name: Haitao Yang, name: Haoqiang Fan, name: Haowei Zhang, name: Hongyang Peng, name: Jing Tan, name: Junwen Huang, name: Kai Liu, name: Kaixin Liu, name: Kefan Gu, name: Qinglun Zhang, name: Ruitao Zhang, name: Saike Huang, name: Shen Cheng, name: Shuaicheng Liu, name: Tiancai Wang, name: Tiezhen Wang, name: Wei Sun, name: Wenbin Tang, name: Yajun Wei, name: Yang Chen, name: Youqiang Gui, name: Yucheng Zhao, name: Yunchao Ma, name: Yunfei Wei, name: Yunhuan Yang, name: Yutong Guo, name: Ze Chen, name: Zhengyuan Du, name: Ziheng Zhang, name: Ziming Liu, name: Ziwei Yan
Published: 2025-10-20 17:59:14

Testing on real machines is indispensable for robotic control algorithms. In
the context of learning-based algorithms, especially VLA models, demand for
large-scale evaluation, i.e. testing a large number of models on a large number
of tasks, is becoming increasingly urgent. However, doing this right is highly
non-trivial, especially when scalability and reproducibility is taken into
account. In this report, we describe our methodology for constructing
RoboChallenge, an online evaluation system to test robotic control algorithms,
and our survey of recent state-of-the-art VLA models using our initial
benchmark Table30.
RiddleBench: A New Generative Reasoning Benchmark for LLMs
Authors: name: Deepon Halder, name: Alan Saji, name: Thanmay Jayakumar, name: Ratish Puduppully, name: Anoop Kunchukuttan, name: Raj Dabre
Published: 2025-10-28 19:58:24

Large Language Models have demonstrated strong performance on many
established reasoning benchmarks. However, these benchmarks primarily evaluate
structured skills like quantitative problem-solving, leaving a gap in assessing
flexible, multifaceted reasoning abilities that are central to human
intelligence. These abilities require integrating logical deduction with
spatial awareness and constraint satisfaction, which current evaluations do not
measure well. To address this, we introduce RiddleBench, a benchmark of 1,737
challenging puzzles in English designed to probe these core reasoning
capabilities. Evaluation of state-of-the-art models on RiddleBench shows
fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3,
and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and
63.16%). Analysis further reveals deep failures, including hallucination
cascades (accepting flawed reasoning from other models) and poor
self-correction due to a strong self-confirmation bias. Their reasoning is also
fragile, with performance degrading significantly when constraints are
reordered or irrelevant information is introduced. RiddleBench functions as a
diagnostic tool for these issues and as a resource for guiding the development
of more robust and reliable language models.
The Collaboration Gap
Authors: name: Tim R. Davidson, name: Adam Fourney, name: Saleema Amershi, name: Robert West, name: Eric Horvitz, name: Ece Kamar
Published: 2025-11-04 16:10:57

The trajectory of AI development suggests that we will increasingly rely on
agent-based systems composed of independently developed agents with different
information, privileges, and tools. The success of these systems will
critically depend on effective collaboration among these heterogeneous agents,
even under partial observability. Despite intense interest, few empirical
studies have evaluated such agent-agent collaboration at scale. We propose a
collaborative maze-solving benchmark that (i) isolates collaborative
capabilities, (ii) modulates problem complexity, (iii) enables scalable
automated grading, and (iv) imposes no output-format constraints, preserving
ecological plausibility. Using this framework, we evaluate 32 leading open- and
closed-source models in solo, homogeneous, and heterogeneous pairings. Our
results reveal a "collaboration gap": models that perform well solo often
degrade substantially when required to collaborate. Collaboration can break
down dramatically; for instance, small distilled models that solve mazes well
alone may fail almost completely in certain pairings. We find that starting
with the stronger agent often improves outcomes, motivating a "relay inference"
approach where the stronger agent leads before handing off to the weaker one,
closing much of the gap. Our findings argue for (1) collaboration-aware
evaluation, (2) training strategies developed to enhance collaborative
capabilities, and (3) interaction design that reliably elicits agents' latent
skills, guidance that applies to AI-AI and human-AI collaboration.
Don't Blind Your VLA: Aligning Visual Representations for OOD
  Generalization
Authors: name: Nikita Kachaev, name: Mikhail Kolosov, name: Daniil Zelezetsky, name: Alexey K. Kovalev, name: Aleksandr I. Panov
Published: 2025-10-29 15:20:10

The growing success of Vision-Language-Action (VLA) models stems from the
promise that pretrained Vision-Language Models (VLMs) can endow agents with
transferable world knowledge and vision-language (VL) grounding, laying a
foundation for action models with broader generalization. Yet when these VLMs
are adapted to the action modality, it remains unclear to what extent their
original VL representations and knowledge are preserved. In this work, we
conduct a systematic study of representation retention during VLA fine-tuning,
showing that naive action fine-tuning leads to degradation of visual
representations. To characterize and measure these effects, we probe VLA's
hidden representations and analyze attention maps, further, we design a set of
targeted tasks and methods that contrast VLA models with their counterpart
VLMs, isolating changes in VL capabilities induced by action fine-tuning. We
further evaluate a range of strategies for aligning visual representations and
introduce a simple yet effective method that mitigates degradation and yields
improved generalization to out-of-distribution (OOD) scenarios. Taken together,
our analysis clarifies the trade-off between action fine-tuning and the
degradation of VL representations and highlights practical approaches to
recover inherited VL capabilities. Code is publicly available:
https://blind-vla-paper.github.io
CodeClash: Benchmarking Goal-Oriented Software Engineering
Authors: name: John Yang, name: Kilian Lieret, name: Joyce Yang, name: Carlos E. Jimenez, name: Ofir Press, name: Ludwig Schmidt, name: Diyi Yang
Published: 2025-11-02 07:42:51

Current benchmarks for coding evaluate language models (LMs) on concrete,
well-specified tasks such as fixing specific bugs or writing targeted tests.
However, human programmers do not spend all day incessantly addressing isolated
tasks. Instead, real-world software development is grounded in the pursuit of
high-level goals, like improving user retention or reducing costs. Evaluating
whether LMs can also iteratively develop code to better accomplish open-ended
objectives without any explicit guidance remains an open challenge. To address
this, we introduce CodeClash, a benchmark where LMs compete in multi-round
tournaments to build the best codebase for achieving a competitive objective.
Each round proceeds in two phases: agents edit their code, then their codebases
compete head-to-head in a code arena that determines winners based on
objectives like score maximization, resource acquisition, or survival. Whether
it's writing notes, scrutinizing documentation, analyzing competition logs, or
creating test suites, models must decide for themselves how to improve their
codebases both absolutely and against their opponents. We run 1680 tournaments
(25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal
that while models exhibit diverse development styles, they share fundamental
limitations in strategic reasoning. Models also struggle with long-term
codebase maintenance, as repositories become progressively messy and redundant.
These limitations are stark: top models lose every round against expert human
programmers. We open-source CodeClash to advance the study of autonomous,
goal-oriented code development.
TabDSR: Decompose, Sanitize, and Reason for Complex Numerical Reasoning
  in Tabular Data
Authors: name: Changjiang Jiang, name: Fengchang Yu, name: Haihua Chen, name: Wei Lu, name: Jin Zeng
Published: 2025-11-04 03:13:02

Complex reasoning over tabular data is crucial in real-world data analysis,
yet large language models (LLMs) often underperform due to complex queries,
noisy data, and limited numerical capabilities. To address these issues, we
propose \method, a framework consisting of: (1) a query decomposer that breaks
down complex questions, (2) a table sanitizer that cleans and filters noisy
tables, and (3) a program-of-thoughts (PoT)-based reasoner that generates
executable code to derive the final answer from the sanitized table. To ensure
unbiased evaluation and mitigate data leakage, we introduce a new dataset,
CalTab151, specifically designed for complex numerical reasoning over tables.
Experimental results demonstrate that \method consistently outperforms existing
methods, achieving state-of-the-art (SOTA) performance with 8.79%, 6.08%, and
19.87% accuracy improvement on TAT-QA, TableBench, and \method, respectively.
Moreover, our framework integrates seamlessly with mainstream LLMs, providing a
robust solution for complex tabular numerical reasoning. These findings
highlight the effectiveness of our framework in enhancing LLM performance for
complex tabular numerical reasoning. Data and code are available upon request.
Can Visual Input Be Compressed? A Visual Token Compression Benchmark for
  Large Multimodal Models
Authors: name: Tianfan Peng, name: Yuntao Du, name: Pengzhou Ji, name: Shijie Dong, name: Kailin Jiang, name: Mingchuan Ma, name: Yijun Tian, name: Jinhe Bi, name: Qian Li, name: Wei Du, name: Feng Xiao, name: Lizhen Cui
Published: 2025-11-04 15:17:06

Large multimodal models (LMMs) often suffer from severe inference
inefficiency due to the large number of visual tokens introduced by image
encoders. While recent token compression methods, such as pruning and merging,
have shown promise in reducing redundancy, their evaluation remains fragmented
and inconsistent. In this work, we present UniPruneBench, a unified and
extensible benchmark for visual token pruning in multimodal LLMs. UniPruneBench
provides standardized protocols across six ability dimensions and ten datasets,
covering ten representative compression algorithms and three families of LMMs
(LLaVA-v1.5, Intern-VL3, and Qwen2.5-VL). Beyond task accuracy, it incorporates
system-level metrics such as runtime and prefilling latency to provide a
holistic view. Our experiments uncover several key findings: (1) random pruning
is a surprisingly strong baseline, (2) no single method consistently
outperforms others across scenarios, (3) pruning sensitivity varies
significantly across tasks, with OCR being most vulnerable, and (4) pruning
ratio is the dominant factor governing performance degradation. We believe
UniPruneBench will serve as a reliable foundation for future research on
efficient multimodal modeling.
Brain-IT: Image Reconstruction from fMRI via Brain-Interaction
  Transformer
Authors: name: Roman Beliy, name: Amit Zalcher, name: Jonathan Kogman, name: Navve Wasserman, name: Michal Irani
Published: 2025-10-29 21:21:54

Reconstructing images seen by people from their fMRI brain recordings
provides a non-invasive window into the human brain. Despite recent progress
enabled by diffusion models, current methods often lack faithfulness to the
actual seen images. We present "Brain-IT", a brain-inspired approach that
addresses this challenge through a Brain Interaction Transformer (BIT),
allowing effective interactions between clusters of functionally-similar
brain-voxels. These functional-clusters are shared by all subjects, serving as
building blocks for integrating information both within and across brains. All
model components are shared by all clusters & subjects, allowing efficient
training with a limited amount of data. To guide the image reconstruction, BIT
predicts two complementary localized patch-level image features: (i)high-level
semantic features which steer the diffusion model toward the correct semantic
content of the image; and (ii)low-level structural features which help to
initialize the diffusion process with the correct coarse layout of the image.
BIT's design enables direct flow of information from brain-voxel clusters to
localized image features. Through these principles, our method achieves image
reconstructions from fMRI that faithfully reconstruct the seen images, and
surpass current SotA approaches both visually and by standard objective
metrics. Moreover, with only 1-hour of fMRI data from a new subject, we achieve
results comparable to current methods trained on full 40-hour recordings.
BRAINS: A Retrieval-Augmented System for Alzheimer's Detection and
  Monitoring
Authors: name: Rajan Das Gupta, name: Md Kishor Morol, name: Nafiz Fahad, name: Md Tanzib Hosain, name: Sumaya Binte Zilani Choya, name: Md Jakir Hossen
Published: 2025-11-04 11:27:03

As the global burden of Alzheimer's disease (AD) continues to grow, early and
accurate detection has become increasingly critical, especially in regions with
limited access to advanced diagnostic tools. We propose BRAINS (Biomedical
Retrieval-Augmented Intelligence for Neurodegeneration Screening) to address
this challenge. This novel system harnesses the powerful reasoning capabilities
of Large Language Models (LLMs) for Alzheimer's detection and monitoring.
BRAINS features a dual-module architecture: a cognitive diagnostic module and a
case-retrieval module. The Diagnostic Module utilizes LLMs fine-tuned on
cognitive and neuroimaging datasets -- including MMSE, CDR scores, and brain
volume metrics -- to perform structured assessments of Alzheimer's risk.
Meanwhile, the Case Retrieval Module encodes patient profiles into latent
representations and retrieves similar cases from a curated knowledge base.
These auxiliary cases are fused with the input profile via a Case Fusion Layer
to enhance contextual understanding. The combined representation is then
processed with clinical prompts for inference. Evaluations on real-world
datasets demonstrate BRAINS effectiveness in classifying disease severity and
identifying early signs of cognitive decline. This system not only shows strong
potential as an assistive tool for scalable, explainable, and early-stage
Alzheimer's disease detection, but also offers hope for future applications in
the field.
Shorter but not Worse: Frugal Reasoning via Easy Samples as Length
  Regularizers in Math RLVR
Authors: name: Abdelaziz Bounhar, name: Hadi Abdine, name: Evan Dufraisse, name: Ahmad Chamma, name: Amr Mohamed, name: Dani Bouch, name: Michalis Vazirgiannis, name: Guokan Shang
Published: 2025-11-02 17:29:16

Large language models (LLMs) trained for step-by-step reasoning often become
excessively verbose, raising inference cost. Standard Reinforcement Learning
with Verifiable Rewards (RLVR) pipelines filter out ``easy'' problems for
training efficiency, leaving the model to train primarily on harder problems
that require longer reasoning chains. This skews the output length distribution
upward, resulting in a model that conflates ``thinking longer'' with
``thinking better''. In this work, we show that retaining and modestly
up-weighting moderately easy problems acts as an implicit length regularizer.
Exposing the model to solvable short-chain tasks constrains its output
distribution and prevents runaway verbosity. The result is
\emph{emergent brevity for free}: the model learns to solve harder
problems without inflating the output length,  despite the absence of
any explicit length penalization. RLVR experiments using this approach on
Qwen3-4B-Thinking-2507 (with a 16k token limit) achieve baseline
pass@1 AIME25 accuracy while generating solutions that are, on average, nearly
twice as short. The code is available at
https://github.com/MBZUAI-Paris/Frugal-AI{GitHub}, with datasets and
models on
https://huggingface.co/collections/MBZUAI-Paris/k2-think-mini-68dcfa8b114686a4bd3dc2bc{Hugging
Face}.
When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs
  Preference Dynamics in MLLMs
Authors: name: Zhuoran Zhang, name: Tengyue Wang, name: Xilin Gong, name: Yang Shi, name: Haotian Wang, name: Di Wang, name: Lijie Hu
Published: 2025-11-04 04:11:31

Multimodal large language models (MLLMs) must resolve conflicts when
different modalities provide contradictory information, a process we term
modality following. Prior work measured this behavior only with coarse
dataset-level statistics, overlooking the influence of model's confidence in
unimodal reasoning. In this paper, we introduce a new framework that decomposes
modality following into two fundamental factors: relative reasoning uncertainty
(the case-specific confidence gap between unimodal predictions) and inherent
modality preference( a model's stable bias when uncertainties are balanced). To
validate this framework, we construct a controllable dataset that
systematically varies the reasoning difficulty of visual and textual inputs.
Using entropy as a fine-grained uncertainty metric, we uncover a universal law:
the probability of following a modality decreases monotonically as its relative
uncertainty increases. At the relative difficulty level where the model tends
to follow both modalities with comparable probability what we call the balance
point, a practical indicator of the model's inherent preference. Unlike
traditional macro-level ratios, this measure offers a more principled and less
confounded way to characterize modality bias, disentangling it from unimodal
capabilities and dataset artifacts. Further, by probing layer-wise predictions,
we reveal the internal mechanism of oscillation: in ambiguous regions near the
balance point, models vacillate between modalities across layers, explaining
externally observed indecision. Together, these findings establish relative
uncertainty and inherent preference as the two governing principles of modality
following, offering both a quantitative framework and mechanistic insight into
how MLLMs resolve conflicting information.
When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for
  Visual Chain-of-Thought
Authors: name: Yiyang Zhou, name: Haoqin Tu, name: Zijun Wang, name: Zeyu Wang, name: Niklas Muennighoff, name: Fan Nie, name: Yejin Choi, name: James Zou, name: Chaorui Deng, name: Shen Yan, name: Haoqi Fan, name: Cihang Xie, name: Huaxiu Yao, name: Qinghao Ye
Published: 2025-11-04 18:00:51

We propose MIRA, a new benchmark designed to evaluate models in scenarios
where generating intermediate visual images is essential for successful
reasoning. Unlike traditional CoT methods that rely solely on text, tasks in
MIRA require models to generate and utilize intermediate images - such as
sketches, structural diagrams, or path drawings - to guide their reasoning
process. This setup closely mirrors how humans solve complex problems through
"drawing to think". To solve this, MIRA focuses on tasks that are intrinsically
challenging and involve complex structures, spatial relationships, or reasoning
steps that are difficult to express through language alone. To ensure that our
evaluation data is of high-quality, we include 546 multimodal problems,
annotated with intermediate visual images and final answers. We also propose a
unified evaluation protocol for MIRA that spans three levels of evaluation
input: direct input with image and question only, text-only CoT input with
image and thinking prompts, and Visual-CoT input with both annotated image
clues and textual thinking prompts. To probe the upper bound of model capacity
on our benchmark, we also report pass@k and majority voting accuracies under
different k settings. Experimental results show that existing multimodal large
language models, including strongest private models as well as strong
open-weight models, perform poorly when relying solely on textual prompts.
However, when intermediate visual cues are provided, model performance improves
consistently, yielding an average relative gain of 33.7% across all models and
tasks. We also probe the upper bound by expanding the search space and
designing textual prompts aligned with Visual-CoT, but both yield only limited
improvements compared to our Visual-CoT setting. These results underscore the
critical role of imagined visual information in enabling successful reasoning
on MIRA.