Orionnnn/hub-stats huggingface dataset:
WinT3R: Window-Based Streaming Reconstruction with Camera Token Pool
Authors: name: Zizun Li, name: Jianjun Zhou, name: Yifan Wang, name: Haoyu Guo, name: Wenzheng Chang, name: Yang Zhou, name: Haoyi Zhu, name: Junyi Chen, name: Chunhua Shen, name: Tong He
Published: 2025-09-05 17:59:47

We present WinT3R, a feed-forward reconstruction model capable of online
prediction of precise camera poses and high-quality point maps. Previous
methods suffer from a trade-off between reconstruction quality and real-time
performance. To address this, we first introduce a sliding window mechanism
that ensures sufficient information exchange among frames within the window,
thereby improving the quality of geometric predictions without large
computation. In addition, we leverage a compact representation of cameras and
maintain a global camera token pool, which enhances the reliability of camera
pose estimation without sacrificing efficiency. These designs enable WinT3R to
achieve state-of-the-art performance in terms of online reconstruction quality,
camera pose estimation, and reconstruction speed, as validated by extensive
experiments on diverse datasets. Code and model are publicly available at
https://github.com/LiZizun/WinT3R.
On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
Authors: name: Riccardo Lunardi, name: Vincenzo Della Mea, name: Stefano Mizzaro, name: Kevin Roitero
Published: 2025-09-04 08:43:27

Large Language Models (LLMs) effectiveness is usually evaluated by means of
benchmarks such as MMLU, ARC-C, or HellaSwag, where questions are presented in
their original wording, thus in a fixed, standardized format. However,
real-world applications involve linguistic variability, requiring models to
maintain their effectiveness across diverse rewordings of the same question or
query. In this study, we systematically assess the robustness of LLMs to
paraphrased benchmark questions and investigate whether benchmark-based
evaluations provide a reliable measure of model capabilities. We systematically
generate various paraphrases of all the questions across six different common
benchmarks, and measure the resulting variations in effectiveness of 34
state-of-the-art LLMs, of different size and effectiveness. Our findings reveal
that while LLM rankings remain relatively stable across paraphrased inputs,
absolute effectiveness scores change, and decline significantly. This suggests
that LLMs struggle with linguistic variability, raising concerns about their
generalization abilities and evaluation methodologies. Furthermore, the
observed performance drop challenges the reliability of benchmark-based
evaluations, indicating that high benchmark scores may not fully capture a
model's robustness to real-world input variations. We discuss the implications
of these findings for LLM evaluation methodologies, emphasizing the need for
robustness-aware benchmarks that better reflect practical deployment scenarios.
Behavioral Fingerprinting of Large Language Models
Authors: name: Zehua Pei, name: Hui-Ling Zhen, name: Ying Zhang, name: Zhiyuan Yang, name: Xing Li, name: Xianzhi Yu, name: Mingxuan Yuan, name: Bei Yu
Published: 2025-09-02 07:03:20

Current benchmarks for Large Language Models (LLMs) primarily focus on
performance metrics, often failing to capture the nuanced behavioral
characteristics that differentiate them. This paper introduces a novel
``Behavioral Fingerprinting'' framework designed to move beyond traditional
evaluation by creating a multi-faceted profile of a model's intrinsic cognitive
and interactive styles. Using a curated Diagnostic Prompt Suite and an
innovative, automated evaluation pipeline where a powerful LLM acts as an
impartial judge, we analyze eighteen models across capability tiers. Our
results reveal a critical divergence in the LLM landscape: while core
capabilities like abstract and causal reasoning are converging among top
models, alignment-related behaviors such as sycophancy and semantic robustness
vary dramatically. We further document a cross-model default persona clustering
(ISTJ/ESTJ) that likely reflects common alignment incentives. Taken together,
this suggests that a model's interactive nature is not an emergent property of
its scale or reasoning power, but a direct consequence of specific, and highly
variable, developer alignment strategies. Our framework provides a reproducible
and scalable methodology for uncovering these deep behavioral differences.
Project: https://github.com/JarvisPei/Behavioral-Fingerprinting
Why Language Models Hallucinate
Authors: name: Adam Tauman Kalai, name: Ofir Nachum, name: Santosh S. Vempala, name: Edwin Zhang
Published: 2025-09-04 21:26:31

Like students facing hard exam questions, large language models sometimes
guess when uncertain, producing plausible yet incorrect statements instead of
admitting uncertainty. Such "hallucinations" persist even in state-of-the-art
systems and undermine trust. We argue that language models hallucinate because
the training and evaluation procedures reward guessing over acknowledging
uncertainty, and we analyze the statistical causes of hallucinations in the
modern training pipeline. Hallucinations need not be mysterious -- they
originate simply as errors in binary classification. If incorrect statements
cannot be distinguished from facts, then hallucinations in pretrained language
models will arise through natural statistical pressures. We then argue that
hallucinations persist due to the way most evaluations are graded -- language
models are optimized to be good test-takers, and guessing when uncertain
improves test performance. This "epidemic" of penalizing uncertain responses
can only be addressed through a socio-technical mitigation: modifying the
scoring of existing benchmarks that are misaligned but dominate leaderboards,
rather than introducing additional hallucination evaluations. This change may
steer the field toward more trustworthy AI systems.
MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in
  3D CT Disease Detection, Understanding and Reporting
Authors: name: Yuheng Li, name: Yenho Chen, name: Yuxiang Lai, name: Jike Zhong, name: Vanessa Wildman, name: Xiaofeng Yang
Published: 2025-09-04 01:28:44

Radiologic diagnostic errors-under-reading errors, inattentional blindness,
and communication failures-remain prevalent in clinical practice. These issues
often stem from missed localized abnormalities, limited global context, and
variability in report language. These challenges are amplified in 3D imaging,
where clinicians must examine hundreds of slices per scan. Addressing them
requires systems with precise localized detection, global volume-level
reasoning, and semantically consistent natural language reporting. However,
existing 3D vision-language models are unable to meet all three needs jointly,
lacking local-global understanding for spatial reasoning and struggling with
the variability and noise of uncurated radiology reports. We present
MedVista3D, a multi-scale semantic-enriched vision-language pretraining
framework for 3D CT analysis. To enable joint disease detection and holistic
interpretation, MedVista3D performs local and global image-text alignment for
fine-grained representation learning within full-volume context. To address
report variability, we apply language model rewrites and introduce a Radiology
Semantic Matching Bank for semantics-aware alignment. MedVista3D achieves
state-of-the-art performance on zero-shot disease classification, report
retrieval, and medical visual question answering, while transferring well to
organ segmentation and prognosis prediction. Code and datasets will be
released.
LatticeWorld: A Multimodal Large Language Model-Empowered Framework for
  Interactive Complex World Generation
Authors: name: Yinglin Duan, name: Zhengxia Zou, name: Tongwei Gu, name: Wei Jia, name: Zhan Zhao, name: Luyi Xu, name: Xinzhu Liu, name: Hao Jiang, name: Kang Chen, name: Shuang Qiu
Published: 2025-09-05 17:22:33

Recent research has been increasingly focusing on developing 3D world models
that simulate complex real-world scenarios. World models have found broad
applications across various domains, including embodied AI, autonomous driving,
entertainment, etc. A more realistic simulation with accurate physics will
effectively narrow the sim-to-real gap and allow us to gather rich information
about the real world conveniently. While traditional manual modeling has
enabled the creation of virtual 3D scenes, modern approaches have leveraged
advanced machine learning algorithms for 3D world generation, with most recent
advances focusing on generative methods that can create virtual worlds based on
user instructions. This work explores such a research direction by proposing
LatticeWorld, a simple yet effective 3D world generation framework that
streamlines the industrial production pipeline of 3D environments. LatticeWorld
leverages lightweight LLMs (LLaMA-2-7B) alongside the industry-grade rendering
engine (e.g., Unreal Engine 5) to generate a dynamic environment. Our proposed
framework accepts textual descriptions and visual instructions as multimodal
inputs and creates large-scale 3D interactive worlds with dynamic agents,
featuring competitive multi-agent interaction, high-fidelity physics
simulation, and real-time rendering. We conduct comprehensive experiments to
evaluate LatticeWorld, showing that it achieves superior accuracy in scene
layout generation and visual fidelity. Moreover, LatticeWorld achieves over a
90times increase in industrial production efficiency while maintaining high
creative quality compared with traditional manual production methods. Our demo
video is available at https://youtu.be/8VWZXpERR18
WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning
Authors: name: Gagan Mundada, name: Yash Vishe, name: Amit Namburi, name: Xin Xu, name: Zachary Novack, name: Julian McAuley, name: Junda Wu
Published: 2025-09-05 01:54:50

Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated
impressive capabilities across various vision-language tasks. However, their
reasoning abilities in the multimodal symbolic music domain remain largely
unexplored. We introduce WildScore, the first in-the-wild multimodal symbolic
music reasoning and analysis benchmark, designed to evaluate MLLMs' capacity to
interpret real-world music scores and answer complex musicological queries.
Each instance in WildScore is sourced from genuine musical compositions and
accompanied by authentic user-generated questions and discussions, capturing
the intricacies of practical music analysis. To facilitate systematic
evaluation, we propose a systematic taxonomy, comprising both high-level and
fine-grained musicological ontologies. Furthermore, we frame complex music
reasoning as multiple-choice question answering, enabling controlled and
scalable assessment of MLLMs' symbolic music understanding. Empirical
benchmarking of state-of-the-art MLLMs on WildScore reveals intriguing patterns
in their visual-symbolic reasoning, uncovering both promising directions and
persistent challenges for MLLMs in symbolic music reasoning and analysis. We
release the dataset and code.
Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
Authors: name: Xingyue Huang, name: Rishabh, name: Gregor Franke, name: Ziyi Yang, name: Jiamu Bai, name: Weijie Bai, name: Jinhe Bi, name: Zifeng Ding, name: Yiqun Duan, name: Chengyu Fan, name: Wendong Fan, name: Xin Gao, name: Ruohao Guo, name: Yuan He, name: Zhuangzhuang He, name: Xianglong Hu, name: Neil Johnson, name: Bowen Li, name: Fangru Lin, name: Siyu Lin, name: Tong Liu, name: Yunpu Ma, name: Hao Shen, name: Hao Sun, name: Beibei Wang, name: Fangyijie Wang, name: Hao Wang, name: Haoran Wang, name: Yang Wang, name: Yifeng Wang, name: Zhaowei Wang, name: Ziyang Wang, name: Yifan Wu, name: Zikai Xiao, name: Chengxing Xie, name: Fan Yang, name: Junxiao Yang, name: Qianshuo Ye, name: Ziyu Ye, name: Guangtao Zeng, name: Yuwen Ebony Zhang, name: Zeyu Zhang, name: Zihao Zhu, name: Bernard Ghanem, name: Philip Torr, name: Guohao Li
Published: 2025-09-03 06:42:40

Recent advances in Large Language Models (LLMs) have shown that their
reasoning capabilities can be significantly improved through Reinforcement
Learning with Verifiable Reward (RLVR), particularly in domains like
mathematics and programming, where ground-truth correctness can be
automatically evaluated. However, extending this success to other
reasoning-intensive domains remains challenging due to the scarcity of
high-quality, verifiable datasets and the high cost of human supervision. In
this work, we introduce the Loong Project: an open-source framework for
scalable synthetic data generation and verification across a diverse range of
reasoning-intensive domains. The framework consists of two key components: (1)
LoongBench, a curated seed dataset containing 8,729 human-vetted examples
across 12 domains (e.g., Advanced Mathematics, Chemistry, Logic), each paired
with executable code and rich metadata; and (2) LoongEnv, a modular synthetic
data generation environment that supports multiple prompting strategies to
produce new question-answer-code triples. Together, these components form an
agent-environment loop that enables reinforcement learning, where an LLM-based
agent is rewarded for generating Chain-of-Thought (CoT) solutions that align
with code-executed answers. Empirically, we benchmark LoongBench on a broad
suite of both open-source and proprietary LLMs to evaluate domain coverage and
reveal performance bottlenecks. In addition, we conduct a comprehensive
analysis of synthetic data generated by LoongEnv, examining correctness,
difficulty, and diversity. Code and documentation are available at
https://github.com/camel-ai/loong.
Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
Authors: name: Yuan Xie, name: Tianshui Chen, name: Zheng Ge, name: Lionel Ni
Published: 2025-08-28 06:55:08

Long-form video understanding, characterized by long-range temporal
dependencies and multiple events, remains a challenge. Existing methods often
rely on static reasoning or external visual-language models (VLMs), which face
issues like complexity and sub-optimal performance due to the lack of
end-to-end training. In this paper, we propose Video-MTR, a reinforced
multi-turn reasoning framework designed to enable iterative key video segment
selection and question comprehension. Unlike traditional video reasoning
pipeline, which generate predictions in a single turn, Video-MTR performs
reasoning in multiple turns, selecting video segments progressively based on
the evolving understanding of previously processed segments and the current
question. This iterative process allows for a more refined and contextually
aware analysis of the video. To ensure intermediate reasoning process, we
introduce a novel gated bi-level reward system, combining trajectory-level
rewards based on answer correctness and turn-level rewards emphasizing
frame-query relevance. This system optimizes both video segment selection and
question comprehension, eliminating the need for external VLMs and allowing
end-to-end training. Extensive experiments on benchmarks like VideoMME, MLVU,
and EgoSchema demonstrate that Video-MTR outperforms existing methods in both
accuracy and efficiency, advancing the state-of-the-art in long video
understanding.
Delta Activations: A Representation for Finetuned Large Language Models
Authors: name: Zhiqiu Xu, name: Amish Sethi, name: Mayur Naik, name: Ser-Nam Lim
Published: 2025-09-04 17:59:06

The success of powerful open source Large Language Models (LLMs) has enabled
the community to create a vast collection of post-trained models adapted to
specific tasks and domains. However, navigating and understanding these models
remains challenging due to inconsistent metadata and unstructured repositories.
We introduce Delta Activations, a method to represent finetuned models as
vector embeddings by measuring shifts in their internal activations relative to
a base model. This representation allows for effective clustering by domain and
task, revealing structure in the model landscape. Delta Activations also
demonstrate desirable properties: it is robust across finetuning settings and
exhibits an additive property when finetuning datasets are mixed. In addition,
we show that Delta Activations can embed tasks via few-shot finetuning, and
further explore its use for model selection and merging. We hope Delta
Activations can facilitate the practice of reusing publicly available models.
Code is available at https://github.com/OscarXZQ/delta_activations.
Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth
Authors: name: Yang Wang, name: Chenghao Xiao, name: Chia-Yi Hsiao, name: Zi Yan Chang, name: Chi-Li Chen, name: Tyler Loakman, name: Chenghua Lin
Published: 2025-09-04 03:58:55

We introduce Drivelology, a unique linguistic phenomenon characterised as
"nonsense with depth", utterances that are syntactically coherent yet
pragmatically paradoxical, emotionally loaded, or rhetorically subversive.
While such expressions may resemble surface-level nonsense, they encode
implicit meaning requiring contextual inference, moral reasoning, or emotional
interpretation. We find that current large language models (LLMs), despite
excelling at many natural language processing (NLP) tasks, consistently fail to
grasp the layered semantics of Drivelological text. To investigate this, we
construct a small but diverse benchmark dataset of over 1,200 meticulously
curated examples, with select instances in English, Mandarin, Spanish, French,
Japanese, and Korean. Annotation was especially challenging: each of the
examples required careful expert review to verify that it truly reflected
Drivelological characteristics. The process involved multiple rounds of
discussion and adjudication to address disagreements, highlighting the subtle
and subjective nature of the Drivelology. We evaluate a range of LLMs on
classification, generation, and reasoning tasks. Our results reveal clear
limitations of LLMs: models often confuse Drivelology with shallow nonsense,
produce incoherent justifications, or miss the implied rhetorical function
altogether. These findings highlight a deeper representational gap in LLMs'
pragmatic understanding and challenge the assumption that statistical fluency
implies cognitive comprehension. We release our dataset and code to facilitate
further research in modelling linguistic depth beyond surface-level coherence.
NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware
  Embeddings
Authors: name: Or Shachar, name: Uri Katz, name: Yoav Goldberg, name: Oren Glickman
Published: 2025-09-04 08:42:23

We present NER Retriever, a zero-shot retrieval framework for ad-hoc Named
Entity Retrieval, a variant of Named Entity Recognition (NER), where the types
of interest are not provided in advance, and a user-defined type description is
used to retrieve documents mentioning entities of that type. Instead of relying
on fixed schemas or fine-tuned models, our method builds on internal
representations of large language models (LLMs) to embed both entity mentions
and user-provided open-ended type descriptions into a shared semantic space. We
show that internal representations, specifically the value vectors from
mid-layer transformer blocks, encode fine-grained type information more
effectively than commonly used top-layer embeddings. To refine these
representations, we train a lightweight contrastive projection network that
aligns type-compatible entities while separating unrelated types. The resulting
entity embeddings are compact, type-aware, and well-suited for nearest-neighbor
search. Evaluated on three benchmarks, NER Retriever significantly outperforms
both lexical and dense sentence-level retrieval baselines. Our findings provide
empirical support for representation selection within LLMs and demonstrate a
practical solution for scalable, schema-free entity retrieval. The NER
Retriever Codebase is publicly available at
https://github.com/ShacharOr100/ner_retriever
Few-step Flow for 3D Generation via Marginal-Data Transport Distillation
Authors: name: Zanwei Zhou, name: Taoran Yi, name: Jiemin Fang, name: Chen Yang, name: Lingxi Xie, name: Xinggang Wang, name: Wei Shen, name: Qi Tian
Published: 2025-09-04 17:24:31

Flow-based 3D generation models typically require dozens of sampling steps
during inference. Though few-step distillation methods, particularly
Consistency Models (CMs), have achieved substantial advancements in
accelerating 2D diffusion models, they remain under-explored for more complex
3D generation tasks. In this study, we propose a novel framework, MDT-dist, for
few-step 3D flow distillation. Our approach is built upon a primary objective:
distilling the pretrained model to learn the Marginal-Data Transport. Directly
learning this objective needs to integrate the velocity fields, while this
integral is intractable to be implemented. Therefore, we propose two
optimizable objectives, Velocity Matching (VM) and Velocity Distillation (VD),
to equivalently convert the optimization target from the transport level to the
velocity and the distribution level respectively. Velocity Matching (VM) learns
to stably match the velocity fields between the student and the teacher, but
inevitably provides biased gradient estimates. Velocity Distillation (VD)
further enhances the optimization process by leveraging the learned velocity
fields to perform probability density distillation. When evaluated on the
pioneer 3D generation framework TRELLIS, our method reduces sampling steps of
each flow transformer from 25 to 1 or 2, achieving 0.68s (1 step x 2) and 0.94s
(2 steps x 2) latency with 9.0x and 6.5x speedup on A800, while preserving high
visual and geometric fidelity. Extensive experiments demonstrate that our
method significantly outperforms existing CM distillation methods, and enables
TRELLIS to achieve superior performance in few-step 3D generation.
Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow
  Real Instructions?
Authors: name: Qinyan Zhang, name: Xinping Lei, name: Ruijie Miao, name: Yu Fu, name: Haojie Fan, name: Le Chang, name: Jiafan Hou, name: Dingling Zhang, name: Zhongfei Hou, name: Ziqiang Yang, name: Changxin Pu, name: Fei Hu, name: Jingkai Liu, name: Mengyun Liu, name: Yang Liu, name: Xiang Gao, name: Jiaheng Liu, name: Tong Yang, name: Zaiyuan Wang, name: Ge Zhang, name: Wenhao Huang
Published: 2025-09-04 15:03:02

Large Language Models (LLMs) achieve strong performance on diverse tasks but
often exhibit cognitive inertia, struggling to follow instructions that
conflict with the standardized patterns learned during supervised fine-tuning
(SFT). To evaluate this limitation, we propose Inverse IFEval, a benchmark that
measures models Counter-intuitive Abilitytheir capacity to override
training-induced biases and comply with adversarial instructions. Inverse
IFEval introduces eight types of such challenges, including Question
Correction, Intentional Textual Flaws, Code without Comments, and
Counterfactual Answering. Using a human-in-the-loop pipeline, we construct a
dataset of 1012 high-quality Chinese and English questions across 23 domains,
evaluated under an optimized LLM-as-a-Judge framework. Experiments on existing
leading LLMs demonstrate the necessity of our proposed Inverse IFEval
benchmark. Our findings emphasize that future alignment efforts should not only
pursue fluency and factual correctness but also account for adaptability under
unconventional contexts. We hope that Inverse IFEval serves as both a
diagnostic tool and a foundation for developing methods that mitigate cognitive
inertia, reduce overfitting to narrow patterns, and ultimately enhance the
instruction-following reliability of LLMs in diverse and unpredictable
real-world scenarios.