Orionnnn/hub-stats huggingface dataset:
ILIAS: Instance-Level Image retrieval At Scale
Authors: name: Giorgos Kordopatis-Zilos, name: Vladan Stojnić, name: Anna Manko, name: Pavel Šuma, name: Nikolaos-Antonios Ypsilantis, name: Nikos Efthymiadis, name: Zakaria Laskar, name: Jiří Matas, name: Ondřej Chum, name: Giorgos Tolias
Published: 2025-02-17 12:42:38

This work introduces ILIAS, a new test dataset for Instance-Level Image
retrieval At Scale. It is designed to evaluate the ability of current and
future foundation models and retrieval techniques to recognize particular
objects. The key benefits over existing datasets include large scale, domain
diversity, accurate ground truth, and a performance that is far from saturated.
ILIAS includes query and positive images for 1,000 object instances, manually
collected to capture challenging conditions and diverse domains. Large-scale
retrieval is conducted against 100 million distractor images from YFCC100M. To
avoid false negatives without extra annotation effort, we include only query
objects confirmed to have emerged after 2014, i.e. the compilation date of
YFCC100M. An extensive benchmarking is performed with the following
observations: i) models fine-tuned on specific domains, such as landmarks or
products, excel in that domain but fail on ILIAS ii) learning a linear
adaptation layer using multi-domain class supervision results in performance
improvements, especially for vision-language models iii) local descriptors in
retrieval re-ranking are still a key ingredient, especially in the presence of
severe background clutter iv) the text-to-image performance of the
vision-language foundation models is surprisingly close to the corresponding
image-to-image case. website: https://vrg.fel.cvut.cz/ilias/
Can a Single Model Master Both Multi-turn Conversations and Tool Use?
  CALM: A Unified Conversational Agentic Language Model
Authors: name: Emre Can Acikgoz, name: Jeremiah Greer, name: Akul Datta, name: Ze Yang, name: William Zeng, name: Oussama Elachqar, name: Emmanouil Koukoumidis, name: Dilek Hakkani-Tür, name: Gokhan Tur
Published: 2025-02-12 22:18:34

Large Language Models (LLMs) with API-calling capabilities enabled building
effective Language Agents (LA), while also revolutionizing the conventional
task-oriented dialogue (TOD) paradigm. However, current approaches face a
critical dilemma: TOD systems are often trained on a limited set of target
APIs, requiring new data to maintain their quality when interfacing with new
services, while LAs are not trained to maintain user intent over multi-turn
conversations. Because both robust multi-turn management and advanced function
calling are crucial for effective conversational agents, we evaluate these
skills on three popular benchmarks: MultiWOZ 2.4 (TOD), BFCL V3 (LA), and
API-Bank (LA), and our analyses reveal that specialized approaches excel in one
domain but underperform in the other. To bridge this chasm, we introduce CALM
(Conversational Agentic Language Model), a unified approach that integrates
both conversational and agentic capabilities. We created CALM-IT, a carefully
constructed multi-task dataset that interleave multi-turn ReAct reasoning with
complex API usage. Using CALM-IT, we train three models CALM 8B, CALM 70B, and
CALM 405B, which outperform top domain-specific models, including GPT-4o,
across all three benchmarks.
The Mirage of Model Editing: Revisiting Evaluation in the Wild
Authors: name: Wanli Yang, name: Fei Sun, name: Jiajun Tan, name: Xinyu Ma, name: Qi Cao, name: Dawei Yin, name: Huawei Shen, name: Xueqi Cheng
Published: 2025-02-16 15:57:55

Despite near-perfect results in artificial evaluations, the effectiveness of
model editing in real-world applications remains unexplored. To bridge this
gap, we propose to study model editing in question answering (QA) by
establishing a rigorous evaluation practice to assess the effectiveness of
editing methods in correcting LLMs' errors. It consists of QAEdit, a new
benchmark derived from popular QA datasets, and a standardized evaluation
framework. Our single editing experiments indicate that current editing methods
perform substantially worse than previously reported (38.5% vs. ~96%). Through
module analysis and controlled experiments, we demonstrate that this
performance decline stems from issues in evaluation practices of prior editing
research. One key issue is the inappropriate use of teacher forcing in testing
prevents error propagation by feeding ground truth tokens (inaccessible in
real-world scenarios) as input. Furthermore, we simulate real-world deployment
by sequential editing, revealing that current approaches fail drastically with
only 1000 edits. Our analysis provides a fundamental reexamination of both the
real-world applicability of existing model editing methods and their evaluation
practices, and establishes a rigorous evaluation framework with key insights to
advance reliable and practical model editing research.
Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with
  Reinforcement Learning
Authors: name: Egor Cherepanov, name: Nikita Kachaev, name: Alexey K. Kovalev, name: Aleksandr I. Panov
Published: 2025-02-14 20:46:19

Memory is crucial for enabling agents to tackle complex tasks with temporal
and spatial dependencies. While many reinforcement learning (RL) algorithms
incorporate memory, the field lacks a universal benchmark to assess an agent's
memory capabilities across diverse scenarios. This gap is particularly evident
in tabletop robotic manipulation, where memory is essential for solving tasks
with partial observability and ensuring robust performance, yet no standardized
benchmarks exist. To address this, we introduce MIKASA (Memory-Intensive Skills
Assessment Suite for Agents), a comprehensive benchmark for memory RL, with
three key contributions: (1) we propose a comprehensive classification
framework for memory-intensive RL tasks, (2) we collect MIKASA-Base - a unified
benchmark that enables systematic evaluation of memory-enhanced agents across
diverse scenarios, and (3) we develop MIKASA-Robo - a novel benchmark of 32
carefully designed memory-intensive tasks that assess memory capabilities in
tabletop robotic manipulation. Our contributions establish a unified framework
for advancing memory RL research, driving the development of more reliable
systems for real-world applications. The code is available at
https://sites.google.com/view/memorybenchrobots/.
Dyve: Thinking Fast and Slow for Dynamic Process Verification
Authors: name: Jianyuan Zhong, name: Zeju Li, name: Zhijian Xu, name: Xiangyu Wen, name: Qiang Xu
Published: 2025-02-16 15:11:19

We present Dyve, a dynamic process verifier that enhances reasoning error
detection in large language models by integrating fast and slow thinking,
inspired by Kahneman's Systems Theory. Dyve adaptively applies immediate
token-level confirmation System 1 for straightforward steps and comprehensive
analysis System 2 for complex ones. Leveraging a novel step-wise
consensus-filtered process supervision technique, combining Monte Carlo
estimation with LLM based evaluation, Dyve curates high-quality supervision
signals from noisy data. Experimental results on ProcessBench and the MATH
dataset confirm that Dyve significantly outperforms existing process-based
verifiers and boosts performance in Best-of-N settings.
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse
  Attention
Authors: name: Jingyang Yuan, name: Huazuo Gao, name: Damai Dai, name: Junyu Luo, name: Liang Zhao, name: Zhengyan Zhang, name: Zhenda Xie, name: Y. X. Wei, name: Lean Wang, name: Zhiping Xiao, name: Yuqing Wang, name: Chong Ruan, name: Ming Zhang, name: Wenfeng Liang, name: Wangding Zeng
Published: 2025-02-16 11:53:44

Long-context modeling is crucial for next-generation language models, yet the
high computational cost of standard attention mechanisms poses significant
computational challenges. Sparse attention offers a promising direction for
improving efficiency while maintaining model capabilities. We present NSA, a
Natively trainable Sparse Attention mechanism that integrates algorithmic
innovations with hardware-aligned optimizations to achieve efficient
long-context modeling. NSA employs a dynamic hierarchical sparse strategy,
combining coarse-grained token compression with fine-grained token selection to
preserve both global context awareness and local precision. Our approach
advances sparse attention design with two key innovations: (1) We achieve
substantial speedups through arithmetic intensity-balanced algorithm design,
with implementation optimizations for modern hardware. (2) We enable end-to-end
training, reducing pretraining computation without sacrificing model
performance. As shown in Figure 1, experiments show the model pretrained with
NSA maintains or exceeds Full Attention models across general benchmarks,
long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves
substantial speedups over Full Attention on 64k-length sequences across
decoding, forward propagation, and backward propagation, validating its
efficiency throughout the model lifecycle.
Better Embeddings with Coupled Adam
Authors: name: Felix Stollenwerk, name: Tobias Stollenwerk
Published: 2025-02-12 14:32:17

Despite their remarkable capabilities, LLMs learn word representations that
exhibit the undesirable yet poorly understood feature of anisotropy. In this
paper, we argue that the second moment in Adam is a cause of anisotropic
embeddings, and suggest a modified optimizer called Coupled Adam to mitigate
the problem. Our experiments demonstrate that Coupled Adam significantly
improves the quality of embeddings, while also leading to better upstream and
downstream performance on large enough datasets.
Show Me the Work: Fact-Checkers' Requirements for Explainable Automated
  Fact-Checking
Authors: name: Greta Warren, name: Irina Shklovski, name: Isabelle Augenstein
Published: 2025-02-13 08:56:25

The pervasiveness of large language models and generative AI in online media
has amplified the need for effective automated fact-checking to assist
fact-checkers in tackling the increasing volume and sophistication of
misinformation. The complex nature of fact-checking demands that automated
fact-checking systems provide explanations that enable fact-checkers to
scrutinise their outputs. However, it is unclear how these explanations should
align with the decision-making and reasoning processes of fact-checkers to be
effectively integrated into their workflows. Through semi-structured interviews
with fact-checking professionals, we bridge this gap by: (i) providing an
account of how fact-checkers assess evidence, make decisions, and explain their
processes; (ii) examining how fact-checkers use automated tools in practice;
and (iii) identifying fact-checker explanation requirements for automated
fact-checking tools. The findings show unmet explanation needs and identify
important criteria for replicable fact-checking explanations that trace the
model's reasoning path, reference specific evidence, and highlight uncertainty
and information gaps.
MagicArticulate: Make Your 3D Models Articulation-Ready
Authors: name: Chaoyue Song, name: Jianfeng Zhang, name: Xiu Li, name: Fan Yang, name: Yiwen Chen, name: Zhongcong Xu, name: Jun Hao Liew, name: Xiaoyang Guo, name: Fayao Liu, name: Jiashi Feng, name: Guosheng Lin
Published: 2025-02-17 18:53:27

With the explosive growth of 3D content creation, there is an increasing
demand for automatically converting static 3D models into articulation-ready
versions that support realistic animation. Traditional approaches rely heavily
on manual annotation, which is both time-consuming and labor-intensive.
Moreover, the lack of large-scale benchmarks has hindered the development of
learning-based solutions. In this work, we present MagicArticulate, an
effective framework that automatically transforms static 3D models into
articulation-ready assets. Our key contributions are threefold. First, we
introduce Articulation-XL, a large-scale benchmark containing over 33k 3D
models with high-quality articulation annotations, carefully curated from
Objaverse-XL. Second, we propose a novel skeleton generation method that
formulates the task as a sequence modeling problem, leveraging an
auto-regressive transformer to naturally handle varying numbers of bones or
joints within skeletons and their inherent dependencies across different 3D
models. Third, we predict skinning weights using a functional diffusion process
that incorporates volumetric geodesic distance priors between vertices and
joints. Extensive experiments demonstrate that MagicArticulate significantly
outperforms existing methods across diverse object categories, achieving
high-quality articulation that enables realistic animation. Project page:
https://chaoyuesong.github.io/MagicArticulate.
I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning
  in Diffusion Models
Authors: name: Zhenxing Mi, name: Kuan-Chieh Wang, name: Guocheng Qian, name: Hanrong Ye, name: Runtao Liu, name: Sergey Tulyakov, name: Kfir Aberman, name: Dan Xu
Published: 2025-02-12 05:30:08

This paper presents ThinkDiff, a novel alignment paradigm that empowers
text-to-image diffusion models with multimodal in-context understanding and
reasoning capabilities by integrating the strengths of vision-language models
(VLMs). Existing multimodal diffusion finetuning methods largely focus on
pixel-level reconstruction rather than in-context reasoning, and are
constrained by the complexity and limited availability of reasoning-based
datasets. ThinkDiff addresses these challenges by leveraging vision-language
training as a proxy task, aligning VLMs with the decoder of an encoder-decoder
large language model (LLM) instead of a diffusion decoder. This proxy task
builds on the observation that the LLM decoder shares the same input
feature space with diffusion decoders that use the corresponding
LLM encoder for prompt embedding. As a result, aligning VLMs with
diffusion decoders can be simplified through alignment with the LLM decoder.
Without complex training and datasets, ThinkDiff effectively unleashes
understanding, reasoning, and composing capabilities in diffusion models.
Experiments demonstrate that ThinkDiff significantly improves accuracy from
19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context
reasoning generation, with only 5 hours of training on 4 A100 GPUs.
Additionally, ThinkDiff demonstrates exceptional performance in composing
multiple images and texts into logically coherent images. Project page:
https://mizhenxing.github.io/ThinkDiff.
Intuitive physics understanding emerges from self-supervised pretraining
  on natural videos
Authors: name: Quentin Garrido, name: Nicolas Ballas, name: Mahmoud Assran, name: Adrien Bardes, name: Laurent Najman, name: Michael Rabbat, name: Emmanuel Dupoux, name: Yann LeCun
Published: 2025-02-17 14:27:14

We investigate the emergence of intuitive physics understanding in
general-purpose deep neural network models trained to predict masked regions in
natural videos. Leveraging the violation-of-expectation framework, we find that
video prediction models trained to predict outcomes in a learned representation
space demonstrate an understanding of various intuitive physics properties,
such as object permanence and shape consistency. In contrast, video prediction
in pixel space and multimodal large language models, which reason through text,
achieve performance closer to chance. Our comparisons of these architectures
reveal that jointly learning an abstract representation space while predicting
missing parts of sensory input, akin to predictive coding, is sufficient to
acquire an understanding of intuitive physics, and that even models trained on
one week of unique video achieve above chance performance. This challenges the
idea that core knowledge -- a set of innate systems to help understand the
world -- needs to be hardwired to develop an understanding of intuitive
physics.
Towards Data-Efficient Pretraining for Atomic Property Prediction
Authors: name: Yasir Ghunaim, name: Hasan Abed Al Kader Hammoud, name: Bernard Ghanem
Published: 2025-02-16 11:46:23

This paper challenges the recent paradigm in atomic property prediction that
links progress to growing dataset sizes and computational resources. We show
that pretraining on a carefully selected, task-relevant dataset can match or
even surpass large-scale pretraining, while using as little as 1/24th of the
computational cost. We introduce the Chemical Similarity Index (CSI), a novel
metric inspired by computer vision's Fr\'echet Inception Distance, for
molecular graphs which quantifies the alignment between upstream pretraining
datasets and downstream tasks. By selecting the most relevant dataset with
minimal CSI distance, we show that models pretrained on a smaller, focused
dataset consistently outperform those pretrained on massive, mixed datasets
such as JMP, even when those larger datasets include the relevant dataset.
Counterintuitively, we also find that indiscriminately adding more data can
degrade model performance when the additional data poorly aligns with the task
at hand. Our findings highlight that quality often outperforms quantity in
pretraining for atomic property prediction.
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Authors: name: Xinyu Zhang, name: Yuxuan Dong, name: Yanrui Wu, name: Jiaxing Huang, name: Chengyou Jia, name: Basura Fernando, name: Mike Zheng Shou, name: Lingling Zhang, name: Jun Liu
Published: 2025-02-17 17:24:14

Large language models demonstrate remarkable capabilities across various
domains, especially mathematics and logic reasoning. However, current
evaluations overlook physics-based reasoning - a complex task requiring physics
theorems and constraints. We present PhysReason, a 1,200-problem benchmark
comprising knowledge-based (25%) and reasoning-based (75%) problems, where the
latter are divided into three difficulty levels (easy, medium, hard). Notably,
problems require an average of 8.1 solution steps, with hard requiring 15.6,
reflecting the complexity of physics-based reasoning. We propose the Physics
Solution Auto Scoring Framework, incorporating efficient answer-level and
comprehensive step-level evaluations. Top-performing models like Deepseek-R1,
Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on
answer-level evaluation, with performance dropping from knowledge questions
(75.11%) to hard problems (31.95%). Through step-level evaluation, we
identified four key bottlenecks: Physics Theorem Application, Physics Process
Understanding, Calculation, and Physics Condition Analysis. These findings
position PhysReason as a novel and comprehensive benchmark for evaluating
physics-based reasoning capabilities in large language models. Our code and
data will be published at https:/dxzxy12138.github.io/PhysReason.
Large Language Models and Mathematical Reasoning Failures
Authors: name: Johan Boye, name: Birger Moell
Published: 2025-02-17 09:07:32

This paper investigates the mathematical reasoning capabilities of large
language models (LLMs) using 50 newly constructed high-school-level word
problems. Unlike prior studies that focus solely on answer correctness, we
rigorously analyze both final answers and solution steps to identify reasoning
failures. Evaluating eight state-of-the-art models - including Mixtral, Llama,
Gemini, GPT-4o, and OpenAI's o1 variants - we find that while newer models
(e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors
in spatial reasoning, strategic planning, and arithmetic, sometimes producing
correct answers through flawed logic. Common failure modes include unwarranted
assumptions, over-reliance on numerical patterns, and difficulty translating
physical intuition into mathematical steps. Manual analysis reveals that models
struggle with problems requiring multi-step deduction or real-world knowledge,
despite possessing broad mathematical knowledge. Our results underscore the
importance of evaluating reasoning processes, not just answers, and caution
against overestimating LLMs' problem-solving proficiency. The study highlights
persistent gaps in LLMs' generalization abilities, emphasizing the need for
targeted improvements in structured reasoning and constraint handling.
Language Complexity Measurement as a Noisy Zero-Shot Proxy for
  Evaluating LLM Performance
Authors: name: Birger Moell, name: Johan Boye
Published: 2025-02-17 09:09:58

Large Language Models (LLMs) have made significant strides in natural
language generation but often face challenges in tasks requiring precise
calculations and structural analysis. This paper investigates the performance
of state-of-the-art LLMs on language complexity measurement tasks, through the
computation of the LIX readability metric and Average Dependency Distance
(ADD). Using Swedish high school and university-level essays, we evaluate the
models' abilities to compute LIX scores and perform dependency parsing,
comparing their results to established ground truths. Our findings reveal that
while all models demonstrate some capacity for these tasks, ChatGPT-o1-mini
performs most consistently, achieving the highest accuracy in both LIX
computation and dependency parsing. Additionally, we observe a strong
significant correlation -0.875 p 0.026 (N=6) between the models' accuracy in
computing LIX and their overall performance on the Massive Multitask Language
Understanding (MMLU) benchmark. These results suggest that language complexity
measurement abilities can serve as a noisy zero-shot proxies for assessing the
general capabilities of LLMs, providing a practical method for model evaluation
without the need for extensive benchmarking datasets.
System Message Generation for User Preferences using Open-Source Models
Authors: name: Minbyul Jeong, name: Jungho Cho, name: Minsoo Khang, name: Dawoon Jung, name: Teakgyu Hong
Published: 2025-02-17 01:05:31

System messages play a crucial role in interactions with large language
models (LLMs), often serving as prompts to initiate conversations. Through
system messages, users can assign specific roles, perform intended tasks,
incorporate background information, specify various output formats and
communication styles. Despite such versatility, publicly available data are
often lack system messages and subject to strict license constraints in the
industry field. Manual labeling of publicly available data with system messages
that align with user instructions demands significant resources. In view of
such challenges, our work introduces SysGen, a pipeline for generating system
messages with better aligned assistant responses from the supervised
fine-tuning dataset without system messages. Training on SysGen data has
demonstrated substantial improvements in the alignment of model responses with
system messages and user instructions, as demonstrated across various
open-source models on the Multifacet benchmark, while maintaining minimal
impact on other unseen benchmarks such as Open LLM Leaderboard 2. Our
qualitative analysis highlights the importance of diverse system messages to
ensure better adaptability across different contexts.