Orionnnn/hub-stats huggingface dataset:
Grammars of Formal Uncertainty: When to Trust LLMs in Automated
  Reasoning Tasks
Authors: name: Debargha Ganguly, name: Vikash Singh, name: Sreehari Sankar, name: Biyao Zhang, name: Xuecen Zhang, name: Srinivasan Iyengar, name: Xiaotian Han, name: Amit Sharma, name: Shivkumar Kalyanaraman, name: Vipin Chaudhary
Published: 2025-05-26 14:34:04

Large language models (LLMs) show remarkable promise for democratizing
automated reasoning by generating formal specifications. However, a fundamental
tension exists: LLMs are probabilistic, while formal verification demands
deterministic guarantees. This paper addresses this epistemological gap by
comprehensively investigating failure modes and uncertainty quantification (UQ)
in LLM-generated formal artifacts. Our systematic evaluation of five frontier
LLMs reveals Satisfiability Modulo Theories (SMT) based autoformalization's
domain-specific impact on accuracy (from +34.8% on logical tasks to -44.5% on
factual ones), with known UQ techniques like the entropy of token probabilities
failing to identify these errors. We introduce a probabilistic context-free
grammar (PCFG) framework to model LLM outputs, yielding a refined uncertainty
taxonomy. We find uncertainty signals are task-dependent (e.g., grammar entropy
for logic, AUROC>0.93). Finally, a lightweight fusion of these signals enables
selective verification, drastically reducing errors (14-100%) with minimal
abstention, transforming LLM-driven formalization into a reliable engineering
discipline.
More Thinking, Less Seeing? Assessing Amplified Hallucination in
  Multimodal Reasoning Models
Authors: name: Chengzhi Liu, name: Zhongxing Xu, name: Qingyue Wei, name: Juncheng Wu, name: James Zou, name: Xin Eric Wang, name: Yuyin Zhou, name: Sheng Liu
Published: 2025-05-23 05:08:40

Test-time compute has empowered multimodal large language models to generate
extended reasoning chains, yielding strong performance on tasks such as
multimodal math reasoning. However, this improved reasoning ability often comes
with increased hallucination: as generations become longer, models tend to
drift away from image-grounded content and rely more heavily on language
priors. Attention analysis shows that longer reasoning chains lead to reduced
focus on visual inputs, which contributes to hallucination. To systematically
study this phenomenon, we introduce RH-AUC, a metric that quantifies how a
model's perception accuracy changes with reasoning length, allowing us to
evaluate whether the model preserves visual grounding during reasoning. We also
release RH-Bench, a diagnostic benchmark that spans a variety of multimodal
tasks, designed to assess the trade-off between reasoning ability and
hallucination. Our analysis reveals that (i) larger models typically achieve a
better balance between reasoning and perception, and (ii) this balance is
influenced more by the types and domains of training data than by its overall
volume. These findings underscore the importance of evaluation frameworks that
jointly consider both reasoning quality and perceptual fidelity.
CoDA: Coordinated Diffusion Noise Optimization for Whole-Body
  Manipulation of Articulated Objects
Authors: name: Huaijin Pi, name: Zhi Cen, name: Zhiyang Dou, name: Taku Komura
Published: 2025-05-27 17:11:50

Synthesizing whole-body manipulation of articulated objects, including body
motion, hand motion, and object motion, is a critical yet challenging task with
broad applications in virtual humans and robotics. The core challenges are
twofold. First, achieving realistic whole-body motion requires tight
coordination between the hands and the rest of the body, as their movements are
interdependent during manipulation. Second, articulated object manipulation
typically involves high degrees of freedom and demands higher precision, often
requiring the fingers to be placed at specific regions to actuate movable
parts. To address these challenges, we propose a novel coordinated diffusion
noise optimization framework. Specifically, we perform noise-space optimization
over three specialized diffusion models for the body, left hand, and right
hand, each trained on its own motion dataset to improve generalization.
Coordination naturally emerges through gradient flow along the human kinematic
chain, allowing the global body posture to adapt in response to hand motion
objectives with high fidelity. To further enhance precision in hand-object
interaction, we adopt a unified representation based on basis point sets (BPS),
where end-effector positions are encoded as distances to the same BPS used for
object geometry. This unified representation captures fine-grained spatial
relationships between the hand and articulated object parts, and the resulting
trajectories serve as targets to guide the optimization of diffusion noise,
producing highly accurate interaction motion. We conduct extensive experiments
demonstrating that our method outperforms existing approaches in motion quality
and physical plausibility, and enables various capabilities such as object pose
control, simultaneous walking and manipulation, and whole-body generation from
hand-only data.
GATE: General Arabic Text Embedding for Enhanced Semantic Textual
  Similarity with Matryoshka Representation Learning and Hybrid Loss Training
Authors: name: Omer Nacar, name: Anis Koubaa, name: Serry Sibaee, name: Yasser Al-Habashi, name: Adel Ammar, name: Wadii Boulila
Published: 2025-05-30 13:29:03

Semantic textual similarity (STS) is a critical task in natural language
processing (NLP), enabling applications in retrieval, clustering, and
understanding semantic relationships between texts. However, research in this
area for the Arabic language remains limited due to the lack of high-quality
datasets and pre-trained models. This scarcity of resources has restricted the
accurate evaluation and advance of semantic similarity in Arabic text. This
paper introduces General Arabic Text Embedding (GATE) models that achieve
state-of-the-art performance on the Semantic Textual Similarity task within the
MTEB benchmark. GATE leverages Matryoshka Representation Learning and a hybrid
loss training approach with Arabic triplet datasets for Natural Language
Inference, which are essential for enhancing model performance in tasks that
demand fine-grained semantic understanding. GATE outperforms larger models,
including OpenAI, with a 20-25% performance improvement on STS benchmarks,
effectively capturing the unique semantic nuances of Arabic.
LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements
  Generation
Authors: name: Chaeeun Kim, name: Jinu Lee, name: Wonseok Hwang
Published: 2025-05-28 09:02:41

Legal Case Retrieval (LCR), which retrieves relevant cases from a query case,
is a fundamental task for legal professionals in research and decision-making.
However, existing studies on LCR face two major limitations. First, they are
evaluated on relatively small-scale retrieval corpora (e.g., 100-55K cases) and
use a narrow range of criminal query types, which cannot sufficiently reflect
the complexity of real-world legal retrieval scenarios. Second, their reliance
on embedding-based or lexical matching methods often results in limited
representations and legally irrelevant matches. To address these issues, we
present: (1) LEGAR BENCH, the first large-scale Korean LCR benchmark, covering
411 diverse crime types in queries over 1.2M legal cases; and (2)
LegalSearchLM, a retrieval model that performs legal element reasoning over the
query case and directly generates content grounded in the target cases through
constrained decoding. Experimental results show that LegalSearchLM outperforms
baselines by 6-20% on LEGAR BENCH, achieving state-of-the-art performance. It
also demonstrates strong generalization to out-of-domain cases, outperforming
naive generative models trained on in-domain data by 15%.
Role-Playing Evaluation for Large Language Models
Authors: name: Yassine El Boudouri, name: Walter Nuninger, name: Julian Alvarez, name: Yvan Peter
Published: 2025-05-19 14:18:16

Large Language Models (LLMs) demonstrate a notable capacity for adopting
personas and engaging in role-playing. However, evaluating this ability
presents significant challenges, as human assessments are resource-intensive
and automated evaluations can be biased. To address this, we introduce
Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM
role-playing capabilities across four key dimensions: emotional understanding,
decision-making, moral alignment, and in-character consistency. This article
details the construction of RPEval and presents baseline evaluations. Our code
and dataset are available at https://github.com/yelboudouri/RPEval
Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual
  Large Language Models
Authors: name: Chaeyoung Jung, name: Youngjoon Jang, name: Jongmin Choi, name: Joon Son Chung
Published: 2025-05-27 08:22:56

The goal of this work is to enhance balanced multimodal understanding in
audio-visual large language models (AV-LLMs) by addressing modality bias
without requiring additional training. In current AV-LLMs, audio and video
features are typically processed jointly in the decoder. While this strategy
facilitates unified multimodal understanding, it may introduce modality bias,
where the model tends to over-rely on one modality due to imbalanced training
signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet
effective inference-time strategy that requires no additional training or
architectural modifications. FMD first performs modality-specific reasoning by
processing audio-only and video-only inputs through the early decoder layers (a
fork phase), and then merges the resulting hidden states for joint reasoning in
the remaining layers (a merge phase). This approach promotes balanced modality
contributions and leverages complementary information across modalities. We
evaluate our method on two representative AV-LLMs, VideoLLaMA2 and
video-SALMONN, using three benchmark datasets. Experimental results demonstrate
consistent performance improvements on tasks focused on audio, video, and
combined audio-visual reasoning, demonstrating the effectiveness of
inference-time interventions for robust multimodal understanding.
DexUMI: Using Human Hand as the Universal Manipulation Interface for
  Dexterous Manipulation
Authors: name: Mengda Xu, name: Han Zhang, name: Yifan Hou, name: Zhenjia Xu, name: Linxi Fan, name: Manuela Veloso, name: Shuran Song
Published: 2025-05-28 01:25:27

We present DexUMI - a data collection and policy learning framework that uses
the human hand as the natural interface to transfer dexterous manipulation
skills to various robot hands. DexUMI includes hardware and software
adaptations to minimize the embodiment gap between the human hand and various
robot hands. The hardware adaptation bridges the kinematics gap using a
wearable hand exoskeleton. It allows direct haptic feedback in manipulation
data collection and adapts human motion to feasible robot hand motion. The
software adaptation bridges the visual gap by replacing the human hand in video
data with high-fidelity robot hand inpainting. We demonstrate DexUMI's
capabilities through comprehensive real-world experiments on two different
dexterous robot hand hardware platforms, achieving an average task success rate
of 86%.
ViStoryBench: Comprehensive Benchmark Suite for Story Visualization
Authors: name: Cailin Zhuang, name: Ailin Huang, name: Wei Cheng, name: Jingwei Wu, name: Yaoqi Hu, name: Jiaqi Liao, name: Zhewei Huang, name: Hongyuan Wang, name: Xinyao Liao, name: Weiwei Cai, name: Hengyuan Xu, name: Xuanyang Zhang, name: Xianfang Zeng, name: Gang Yu, name: Chi Zhang
Published: 2025-05-30 17:58:21

Story visualization, which aims to generate a sequence of visually coherent
images aligning with a given narrative and reference images, has seen
significant progress with recent advancements in generative models. To further
enhance the performance of story visualization frameworks in real-world
scenarios, we introduce a comprehensive evaluation benchmark, ViStoryBench. We
collect a diverse dataset encompassing various story types and artistic styles,
ensuring models are evaluated across multiple dimensions such as different
plots (e.g., comedy, horror) and visual aesthetics (e.g., anime, 3D
renderings). ViStoryBench is carefully curated to balance narrative structures
and visual elements, featuring stories with single and multiple protagonists to
test models' ability to maintain character consistency. Additionally, it
includes complex plots and intricate world-building to challenge models in
generating accurate visuals. To ensure comprehensive comparisons, our benchmark
incorporates a wide range of evaluation metrics assessing critical aspects.
This structured and multifaceted framework enables researchers to thoroughly
identify both the strengths and weaknesses of different models, fostering
targeted improvements.
un^2CLIP: Improving CLIP's Visual Detail Capturing Ability via
  Inverting unCLIP
Authors: name: Yinqi Li, name: Jiahe Zhao, name: Hong Chang, name: Ruibing Hou, name: Shiguang Shan, name: Xilin Chen
Published: 2025-05-30 12:29:38

Contrastive Language-Image Pre-training (CLIP) has become a foundation model
and has been applied to various vision and multimodal tasks. However, recent
works indicate that CLIP falls short in distinguishing detailed differences in
images and shows suboptimal performance on dense-prediction and vision-centric
multimodal tasks. Therefore, this work focuses on improving existing CLIP
models, aiming to capture as many visual details in images as possible. We find
that a specific type of generative models, unCLIP, provides a suitable
framework for achieving our goal. Specifically, unCLIP trains an image
generator conditioned on the CLIP image embedding. In other words, it inverts
the CLIP image encoder. Compared to discriminative models like CLIP, generative
models are better at capturing image details because they are trained to learn
the data distribution of images. Additionally, the conditional input space of
unCLIP aligns with CLIP's original image-text embedding space. Therefore, we
propose to invert unCLIP (dubbed un^2CLIP) to improve the CLIP model. In this
way, the improved image encoder can gain unCLIP's visual detail capturing
ability while preserving its alignment with the original text encoder
simultaneously. We evaluate our improved CLIP across various tasks to which
CLIP has been applied, including the challenging MMVP-VLM benchmark, the
dense-prediction open-vocabulary segmentation task, and multimodal large
language model tasks. Experiments show that un^2CLIP significantly improves
the original CLIP and previous CLIP improvement methods. Code and models will
be available at https://github.com/LiYinqi/un2CLIP.
Harnessing Large Language Models for Scientific Novelty Detection
Authors: name: Yan Liu, name: Zonglin Yang, name: Soujanya Poria, name: Thanh-Son Nguyen, name: Erik Cambria
Published: 2025-05-30 14:08:13

In an era of exponential scientific growth, identifying novel research ideas
is crucial and challenging in academia. Despite potential, the lack of an
appropriate benchmark dataset hinders the research of novelty detection. More
importantly, simply adopting existing NLP technologies, e.g., retrieving and
then cross-checking, is not a one-size-fits-all solution due to the gap between
textual similarity and idea conception. In this paper, we propose to harness
large language models (LLMs) for scientific novelty detection (ND), associated
with two new datasets in marketing and NLP domains. To construct the
considerate datasets for ND, we propose to extract closure sets of papers based
on their relationship, and then summarize their main ideas based on LLMs. To
capture idea conception, we propose to train a lightweight retriever by
distilling the idea-level knowledge from LLMs to align ideas with similar
conception, enabling efficient and accurate idea retrieval for LLM novelty
detection. Experiments show our method consistently outperforms others on the
proposed benchmark datasets for idea retrieval and ND tasks. Codes and data are
available at https://anonymous.4open.science/r/NoveltyDetection-10FB/.
Time Blindness: Why Video-Language Models Can't See What Humans Can?
Authors: name: Ujjwal Upadhyay, name: Mukul Ranjan, name: Zhiqiang Shen, name: Mohamed Elhoseiny
Published: 2025-05-30 17:59:12

Recent advances in vision-language models (VLMs) have made impressive strides
in understanding spatio-temporal relationships in videos. However, when spatial
information is obscured, these models struggle to capture purely temporal
patterns. We introduce SpookyBench, a benchmark where information is
encoded solely in temporal sequences of noise-like frames, mirroring natural
phenomena from biological signaling to covert communication. Interestingly,
while humans can recognize shapes, text, and patterns in these sequences with
over 98% accuracy, state-of-the-art VLMs achieve 0% accuracy. This performance
gap highlights a critical limitation: an over-reliance on frame-level spatial
features and an inability to extract meaning from temporal cues. Furthermore,
when trained in data sets with low spatial signal-to-noise ratios (SNR),
temporal understanding of models degrades more rapidly than human perception,
especially in tasks requiring fine-grained temporal reasoning. Overcoming this
limitation will require novel architectures or training paradigms that decouple
spatial dependencies from temporal processing. Our systematic analysis shows
that this issue persists across model scales and architectures. We release
SpookyBench to catalyze research in temporal pattern recognition and bridge the
gap between human and machine video understanding. Dataset and code has been
made available on our project website: https://timeblindness.github.io/.
Point-MoE: Towards Cross-Domain Generalization in 3D Semantic
  Segmentation via Mixture-of-Experts
Authors: name: Xuweiyi Chen, name: Wentao Zhou, name: Aruni RoyChowdhury, name: Zezhou Cheng
Published: 2025-05-29 18:21:47

While scaling laws have transformed natural language processing and computer
vision, 3D point cloud understanding has yet to reach that stage. This can be
attributed to both the comparatively smaller scale of 3D datasets, as well as
the disparate sources of the data itself. Point clouds are captured by diverse
sensors (e.g., depth cameras, LiDAR) across varied domains (e.g., indoor,
outdoor), each introducing unique scanning patterns, sampling densities, and
semantic biases. Such domain heterogeneity poses a major barrier towards
training unified models at scale, especially under the realistic constraint
that domain labels are typically inaccessible at inference time. In this work,
we propose Point-MoE, a Mixture-of-Experts architecture designed to enable
large-scale, cross-domain generalization in 3D perception. We show that
standard point cloud backbones degrade significantly in performance when
trained on mixed-domain data, whereas Point-MoE with a simple top-k routing
strategy can automatically specialize experts, even without access to domain
labels. Our experiments demonstrate that Point-MoE not only outperforms strong
multi-domain baselines but also generalizes better to unseen domains. This work
highlights a scalable path forward for 3D understanding: letting the model
discover structure in diverse 3D data, rather than imposing it via manual
curation or domain supervision.
Vision Language Models are Biased
Authors: name: An Vo, name: Khai-Nguyen Nguyen, name: Mohammad Reza Taesiri, name: Vy Tuong Dang, name: Anh Totti Nguyen, name: Daeyoung Kim
Published: 2025-05-29 18:47:58

Large language models (LLMs) memorize a vast amount of prior knowledge from
the Internet that help them on downstream tasks but also may notoriously sway
their outputs towards wrong or biased answers. In this work, we test how the
knowledge about popular subjects hurt the accuracy of vision language models
(VLMs) on standard, objective visual tasks of counting and identification. We
find that state-of-the-art VLMs are strongly biased (e.g, unable to recognize a
fourth stripe has been added to a 3-stripe Adidas logo) scoring an average of
17.05% accuracy in counting (e.g., counting stripes in an Adidas-like logo)
across 7 diverse domains from animals, logos, chess, board games, optical
illusions, to patterned grids. Insert text (e.g., "Adidas") describing the
subject name into the counterfactual image further decreases VLM accuracy. The
biases in VLMs are so strong that instructing them to double-check their
results or rely exclusively on image details to answer improves counting
accuracy by only +2 points, on average. Our work presents an interesting
failure mode in VLMs and an automated framework for testing VLM biases. Code
and data are available at: vlmsarebiased.github.io.
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
Authors: name: Longze Chen, name: Renke Shan, name: Huiming Wang, name: Lu Wang, name: Ziqiang Liu, name: Run Luo, name: Jiawei Wang, name: Hamid Alinejad-Rokny, name: Min Yang
Published: 2025-05-30 04:15:06

Speculative decoding (SD) is a promising method for accelerating the decoding
process of Large Language Models (LLMs). The efficiency of SD primarily hinges
on the consistency between the draft model and the verify model. However,
existing drafting approaches typically require additional modules to be
trained, which can be challenging to implement and ensure compatibility across
various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping
strategy for self-speculative decoding. Unlike prior methods, CLaSp does not
require additional drafting modules or extra training. Instead, it employs a
plug-and-play mechanism by skipping intermediate layers of the verify model to
construct a compressed draft model. Specifically, we develop a dynamic
programming algorithm that optimizes the layer-skipping process by leveraging
the complete hidden states from the last verification stage as an objective.
This enables CLaSp to dynamically adjust its layer-skipping strategy after each
verification stage, without relying on pre-optimized sets of skipped layers.
Experimental results across diverse downstream tasks demonstrate that CLaSp
achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the
original distribution of the generated text.
HardTests: Synthesizing High-Quality Test Cases for LLM Coding
Authors: name: Zhongmou He, name: Yee Man Choi, name: Kexun Zhang, name: Jiabao Ji, name: Junting Zhou, name: Dejia Xu, name: Ivan Bercovich, name: Aidan Zhang, name: Lei Li
Published: 2025-05-30 01:00:34

Verifiers play a crucial role in large language model (LLM) reasoning, needed
by post-training techniques such as reinforcement learning. However, reliable
verifiers are hard to get for difficult coding problems, because a
well-disguised wrong solution may only be detected by carefully human-written
edge cases that are difficult to synthesize. To address this issue, we propose
HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this
pipeline, we curate a comprehensive competitive programming dataset HARDTESTS
with 47k problems and synthetic high-quality tests. Compared with existing
tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points
higher and recall that is 17.5 percentage points higher when evaluating
LLM-generated code. For harder problems, the improvement in precision can be as
large as 40 points. HARDTESTS also proves to be more effective for model
training, measured by downstream code generation performance. We will
open-source our dataset and synthesis pipeline at
https://leililab.github.io/HardTests/.