Orionnnn/hub-stats huggingface dataset:
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial
  Reasoning
Authors: name: Zhuohan Xie, name: Dhruv Sahnan, name: Debopriyo Banerjee, name: Georgi Georgiev, name: Rushil Thareja, name: Hachem Madmoun, name: Jinyan Su, name: Aaryamonvikram Singh, name: Yuxia Wang, name: Rui Xing, name: Fajri Koto, name: Haonan Li, name: Ivan Koychev, name: Tanmoy Chakraborty, name: Salem Lahlou, name: Veselin Stoyanov, name: Preslav Nakov
Published: 2025-06-03 06:44:42

Multi-step symbolic reasoning is critical for advancing downstream
performance on financial tasks. Yet, benchmarks for systematically evaluating
this capability are lacking. Existing datasets like FinQA and ConvFinQA
supervise only final numerical answers, without assessing intermediate
reasoning steps. To address this, we introduce FinChain, the first symbolic
benchmark designed for verifiable Chain-of- Thought (CoT) financial reasoning.
Spanning 54 topics across 12 financial domains, Fin- Chain offers five
parameterized templates per topic, each varying in reasoning complexity and
domain expertise required. Each dataset instance includes an executable Python
trace, enabling automatic generation of extensive training data and easy
adaptation to other domains. We also introduce ChainEval, a new metric for
automatic evaluation of both final answers and intermediate reasoning.
Benchmarking 30 LLMs on our dataset, we find that even state-of-the-art models
have considerable room for improvement in multi-step financial reasoning. All
templates and evaluation metrics for FinChain are available at https:
//github.com/mbzuai-nlp/finchain.
Survey of Active Learning Hyperparameters: Insights from a Large-Scale
  Experimental Grid
Authors: name: Julius Gonsior, name: Tim Rieß, name: Anja Reusch, name: Claudio Hartmann, name: Maik Thiele, name: Wolfgang Lehner
Published: 2025-06-04 10:41:37

Annotating data is a time-consuming and costly task, but it is inherently
required for supervised machine learning. Active Learning (AL) is an
established method that minimizes human labeling effort by iteratively
selecting the most informative unlabeled samples for expert annotation, thereby
improving the overall classification performance. Even though AL has been known
for decades, AL is still rarely used in real-world applications. As indicated
in the two community web surveys among the NLP community about AL, two main
reasons continue to hold practitioners back from using AL: first, the
complexity of setting AL up, and second, a lack of trust in its effectiveness.
We hypothesize that both reasons share the same culprit: the large
hyperparameter space of AL. This mostly unexplored hyperparameter space often
leads to misleading and irreproducible AL experiment results. In this study, we
first compiled a large hyperparameter grid of over 4.6 million hyperparameter
combinations, second, recorded the performance of all combinations in the
so-far biggest conducted AL study, and third, analyzed the impact of each
hyperparameter in the experiment results. In the end, we give recommendations
about the influence of each hyperparameter, demonstrate the surprising
influence of the concrete AL strategy implementation, and outline an
experimental study design for reproducible AL experiments with minimal
computational effort, thus contributing to more reproducible and trustworthy AL
research in the future.
Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Authors: name: Elias Abad Rocamora, name: Christian Schlarmann, name: Naman Deep Singh, name: Yongtao Wu, name: Matthias Hein, name: Volkan Cevher
Published: 2025-06-03 19:57:09

Adversarial input attacks can cause a significant shift of CLIP embeddings.
This can affect the downstream robustness of models incorporating CLIP in the
pipeline, such as text-to-image generative models or large vision language
models. While some efforts have been done towards making the CLIP image
encoders robust, the robustness of text encoders remains unexplored. In this
work, we cover this gap in the literature. We propose LEAF: an efficient
adversarial finetuning method for the text domain, with the ability to scale to
large CLIP models. Our models significantly improve the zero-shot adversarial
accuracy in the text domain, while maintaining the vision performance provided
by robust image encoders. When combined with text-to-image diffusion models, we
can improve the generation quality under adversarial noise. When employing our
robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
adversarial noise over standard CLIP models. Finally, we show that robust text
encoders facilitate better reconstruction of input text from its embedding via
direct optimization.
Rethinking the Stability-Plasticity Trade-off in Continual Learning from
  an Architectural Perspective
Authors: name: Aojun Lu, name: Hangjie Yuan, name: Tao Feng, name: Yanan Sun
Published: 2025-06-04 13:40:41

The quest for Continual Learning (CL) seeks to empower neural networks with
the ability to learn and adapt incrementally. Central to this pursuit is
addressing the stability-plasticity dilemma, which involves striking a balance
between two conflicting objectives: preserving previously learned knowledge and
acquiring new knowledge. While numerous CL methods aim to achieve this
trade-off, they often overlook the impact of network architecture on stability
and plasticity, restricting the trade-off to the parameter level. In this
paper, we delve into the conflict between stability and plasticity at the
architectural level. We reveal that under an equal parameter constraint, deeper
networks exhibit better plasticity, while wider networks are characterized by
superior stability. To address this architectural-level dilemma, we introduce a
novel framework denoted Dual-Arch, which serves as a plug-in component for CL.
This framework leverages the complementary strengths of two distinct and
independent networks: one dedicated to plasticity and the other to stability.
Each network is designed with a specialized and lightweight architecture,
tailored to its respective objective. Extensive experiments demonstrate that
Dual-Arch enhances the performance of existing CL methods while being up to 87%
more compact in terms of parameters.
Robust Neural Rendering in the Wild with Asymmetric Dual 3D Gaussian
  Splatting
Authors: name: Chengqi Li, name: Zhihao Shi, name: Yangdi Lu, name: Wenbo He, name: Xiangyu Xu
Published: 2025-06-04 03:40:33

3D reconstruction from in-the-wild images remains a challenging task due to
inconsistent lighting conditions and transient distractors. Existing methods
typically rely on heuristic strategies to handle the low-quality training data,
which often struggle to produce stable and consistent reconstructions,
frequently resulting in visual artifacts. In this work, we propose Asymmetric
Dual 3DGS, a novel framework that leverages the stochastic nature of these
artifacts: they tend to vary across different training runs due to minor
randomness. Specifically, our method trains two 3D Gaussian Splatting (3DGS)
models in parallel, enforcing a consistency constraint that encourages
convergence on reliable scene geometry while suppressing inconsistent
artifacts. To prevent the two models from collapsing into similar failure modes
due to confirmation bias, we introduce a divergent masking strategy that
applies two complementary masks: a multi-cue adaptive mask and a
self-supervised soft mask, which leads to an asymmetric training process of the
two models, reducing shared error modes. In addition, to improve the efficiency
of model training, we introduce a lightweight variant called Dynamic EMA Proxy,
which replaces one of the two models with a dynamically updated Exponential
Moving Average (EMA) proxy, and employs an alternating masking strategy to
preserve divergence. Extensive experiments on challenging real-world datasets
demonstrate that our method consistently outperforms existing approaches while
achieving high efficiency. Codes and trained models will be released.
AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment
Authors: name: Anastasiia Ivanova, name: Eva Bakaeva, name: Zoya Volovikova, name: Alexey K. Kovalev, name: Aleksandr I. Panov
Published: 2025-06-04 15:47:07

As a part of an embodied agent, Large Language Models (LLMs) are typically
used for behavior planning given natural language instructions from the user.
However, dealing with ambiguous instructions in real-world environments remains
a challenge for LLMs. Various methods for task ambiguity detection have been
proposed. However, it is difficult to compare them because they are tested on
different datasets and there is no universal benchmark. For this reason, we
propose AmbiK (Ambiguous Tasks in Kitchen Environment), the fully textual
dataset of ambiguous instructions addressed to a robot in a kitchen
environment. AmbiK was collected with the assistance of LLMs and is
human-validated. It comprises 1000 pairs of ambiguous tasks and their
unambiguous counterparts, categorized by ambiguity type (Human Preferences,
Common Sense Knowledge, Safety), with environment descriptions, clarifying
questions and answers, user intents, and task plans, for a total of 2000 tasks.
We hope that AmbiK will enable researchers to perform a unified comparison of
ambiguity detection methods. AmbiK is available at
https://github.com/cog-model/AmbiK-dataset.
CASS: Nvidia to AMD Transpilation with Data, Models, and Benchmark
Authors: name: Ahmed Heakl, name: Sarim Hashmi, name: Gustavo Bertolo Stahl, name: Seung Hun Eddie Han, name: Salman Khan, name: Abdulrahman Mahmoud
Published: 2025-05-22 17:48:53

We introduce CASS, the first large-scale dataset and model suite for
cross-architecture GPU code transpilation, targeting both source-level (CUDA
leftrightarrow HIP) and assembly-level (Nvidia SASS leftrightarrow AMD
RDNA3) translation. The dataset comprises 70k verified code pairs across host
and device, addressing a critical gap in low-level GPU code portability.
Leveraging this resource, we train the CASS family of domain-specific language
models, achieving 95% source translation accuracy and 37.5% assembly
translation accuracy, substantially outperforming commercial baselines such as
GPT-4o, Claude, and Hipify. Our generated code matches native performance in
over 85% of test cases, preserving runtime and memory behavior. To support
rigorous evaluation, we introduce CASS-Bench, a curated benchmark spanning 16
GPU domains with ground-truth execution. All data, models, and evaluation tools
are released as open source to foster progress in GPU compiler tooling, binary
compatibility, and LLM-guided hardware translation. Dataset and benchmark are
on
https://huggingface.co/datasets/MBZUAI/cass{blue{HuggingFace}},
with code at
https://github.com/GustavoStahl/CASS{blue{GitHub}}.
Voyager: Long-Range and World-Consistent Video Diffusion for Explorable
  3D Scene Generation
Authors: name: Tianyu Huang, name: Wangguandong Zheng, name: Tengfei Wang, name: Yuhao Liu, name: Zhenwei Wang, name: Junta Wu, name: Jie Jiang, name: Hui Li, name: Rynson W. H. Lau, name: Wangmeng Zuo, name: Chunchao Guo
Published: 2025-06-04 17:59:04

Real-world applications like video gaming and virtual reality often demand
the ability to model 3D scenes that users can explore along custom camera
trajectories. While significant progress has been made in generating 3D objects
from text or images, creating long-range, 3D-consistent, explorable 3D scenes
remains a complex and challenging problem. In this work, we present Voyager, a
novel video diffusion framework that generates world-consistent 3D point-cloud
sequences from a single image with user-defined camera path. Unlike existing
approaches, Voyager achieves end-to-end scene generation and reconstruction
with inherent consistency across frames, eliminating the need for 3D
reconstruction pipelines (e.g., structure-from-motion or multi-view stereo).
Our method integrates three key components: 1) World-Consistent Video
Diffusion: A unified architecture that jointly generates aligned RGB and depth
video sequences, conditioned on existing world observation to ensure global
coherence 2) Long-Range World Exploration: An efficient world cache with point
culling and an auto-regressive inference with smooth video sampling for
iterative scene extension with context-aware consistency, and 3) Scalable Data
Engine: A video reconstruction pipeline that automates camera pose estimation
and metric depth prediction for arbitrary videos, enabling large-scale, diverse
training data curation without manual 3D annotations. Collectively, these
designs result in a clear improvement over existing methods in visual quality
and geometric accuracy, with versatile applications.
VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code
  Generation
Authors: name: Yuansheng Ni, name: Ping Nie, name: Kai Zou, name: Xiang Yue, name: Wenhu Chen
Published: 2025-06-04 13:24:44

Large language models (LLMs) often struggle with visualization tasks like
plotting diagrams, charts, where success depends on both code correctness and
visual semantics. Existing instruction-tuning datasets lack execution-grounded
supervision and offer limited support for iterative code correction, resulting
in fragile and unreliable plot generation. We present VisCode-200K, a
large-scale instruction tuning dataset for Python-based visualization and
self-correction. It contains over 200K examples from two sources: (1) validated
plotting code from open-source repositories, paired with natural language
instructions and rendered plots; and (2) 45K multi-turn correction dialogues
from Code-Feedback, enabling models to revise faulty code using runtime
feedback. We fine-tune Qwen2.5-Coder-Instruct on VisCode-200K to create
VisCoder, and evaluate it on PandasPlotBench. VisCoder significantly
outperforms strong open-source baselines and approaches the performance of
proprietary models like GPT-4o-mini. We further adopt a self-debug evaluation
protocol to assess iterative repair, demonstrating the benefits of
feedback-driven learning for executable, visually accurate code generation.
Adapt before Continual Learning
Authors: name: Aojun Lu, name: Tao Feng, name: Hangjie Yuan, name: Chunhui Ding, name: Yanan Sun
Published: 2025-06-04 13:46:33

Continual Learning (CL) seeks to enable neural networks to incrementally
acquire new knowledge (plasticity) while retaining existing knowledge
(stability). While pre-trained models (PTMs) have become pivotal in CL,
prevailing approaches freeze the PTM backbone to preserve stability, limiting
their plasticity, particularly when encountering significant domain gaps in
incremental tasks. Conversely, sequentially finetuning the entire PTM risks
catastrophic forgetting of generalizable knowledge, exposing a critical
stability-plasticity trade-off. To address this challenge, we propose Adapting
PTMs before the core CL process (ACL), a novel framework that refines the PTM
backbone through a plug-and-play adaptation phase before learning each new task
with existing CL approaches (e.g., prompt tuning). ACL enhances plasticity by
aligning embeddings with their original class prototypes while distancing them
from others, theoretically and empirically shown to balance stability and
plasticity. Extensive experiments demonstrate that ACL significantly improves
CL performance across benchmarks and integrated methods, offering a versatile
solution for PTM-based CL.
VLMs Can Aggregate Scattered Training Patches
Authors: name: Zhanhui Zhou, name: Lingjie Chen, name: Chao Yang, name: Chaochao Lu
Published: 2025-06-04 06:46:06

One way to mitigate risks in vision-language models (VLMs) is to remove
dangerous samples in their training data. However, such data moderation can be
easily bypassed when harmful images are split into small, benign-looking
patches, scattered across many training samples. VLMs may then learn to piece
these fragments together during training and generate harmful responses at
inference, either from full images or text references. For instance, if trained
on image patches from a bloody scene paired with the descriptions "safe," VLMs
may later describe, the full image or a text reference to the scene, as "safe."
We define the core ability of VLMs enabling this attack as visual
stitching -- the ability to integrate visual information spread across
multiple training samples that share the same textual descriptions. In our
work, we first demonstrate visual stitching abilities in common open-source
VLMs on three datasets where each image is labeled with a unique synthetic ID:
we split each (image, ID) pair into {(patch,
ID)} pairs at different granularity for finetuning, and we find that
tuned models can verbalize the correct IDs from full images or text reference.
Building on this, we simulate the adversarial data poisoning scenario mentioned
above by using patches from dangerous images and replacing IDs with text
descriptions like ``safe'' or ``unsafe'', demonstrating how harmful content can
evade moderation in patches and later be reconstructed through visual
stitching, posing serious VLM safety risks. Code is available at
https://github.com/ZHZisZZ/visual-stitching.
BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
  Evaluation
Authors: name: Eunsu Kim, name: Haneul Yoo, name: Guijin Son, name: Hitesh Patel, name: Amit Agarwal, name: Alice Oh
Published: 2025-05-31 09:24:32

As large language models (LLMs) continue to advance, the need for up-to-date
and well-organized benchmarks becomes increasingly critical. However, many
existing datasets are scattered, difficult to manage, and make it challenging
to perform evaluations tailored to specific needs or domains, despite the
growing importance of domain-specific models in areas such as math or code. In
this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
researchers and developers to evaluate LLMs more effectively. BenchHub
aggregates and automatically classifies benchmark datasets from diverse
domains, integrating 303K questions across 38 benchmarks. It is designed to
support continuous updates and scalable data management, enabling flexible and
customizable evaluation tailored to various domains or use cases. Through
extensive experiments with various LLM families, we demonstrate that model
performance varies significantly across domain-specific subsets, emphasizing
the importance of domain-aware benchmarking. We believe BenchHub can encourage
better dataset reuse, more transparent model comparisons, and easier
identification of underrepresented areas in existing benchmarks, offering a
critical infrastructure for advancing LLM evaluation research.
MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in
  Videos
Authors: name: Kejian Zhu, name: Zhuoran Jin, name: Hongbang Yuan, name: Jiachun Li, name: Shangqing Tu, name: Pengfei Cao, name: Yubo Chen, name: Kang Liu, name: Jun Zhao
Published: 2025-06-04 16:33:41

The sequential structure of videos poses a challenge to the ability of
multimodal large language models (MLLMs) to locate multi-frame evidence and
conduct multimodal reasoning. However, existing video benchmarks mainly focus
on understanding tasks, which only require models to match frames mentioned in
the question (hereafter referred to as "question frame") and perceive a few
adjacent frames. To address this gap, we propose MMR-V: A Benchmark for
Multimodal Deep Reasoning in Videos. The benchmark is characterized by the
following features. (1) Long-range, multi-frame reasoning: Models are required
to infer and analyze evidence frames that may be far from the question frame.
(2) Beyond perception: Questions cannot be answered through direct perception
alone but require reasoning over hidden information. (3) Reliability: All tasks
are manually annotated, referencing extensive real-world user understanding to
align with common perceptions. (4) Confusability: Carefully designed distractor
annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos
and 1,257 tasks. Our experiments reveal that current models still struggle with
multi-modal reasoning; even the best-performing model, o4-mini, achieves only
52.5% accuracy. Additionally, current reasoning enhancement strategies
(Chain-of-Thought and scaling test-time compute) bring limited gains. Further
analysis indicates that the CoT demanded for multi-modal reasoning differs from
it in textual reasoning, which partly explains the limited performance gains.
We hope that MMR-V can inspire further research into enhancing multi-modal
reasoning capabilities.
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
Authors: name: Kejian Zhu, name: Shangqing Tu, name: Zhuoran Jin, name: Lei Hou, name: Juanzi Li, name: Jun Zhao
Published: 2025-06-04 16:33:44

The development of large language models (LLMs) depends on trustworthy
evaluation. However, most current evaluations rely on public benchmarks, which
are prone to data contamination issues that significantly compromise fairness.
Previous researches have focused on constructing dynamic benchmarks to address
contamination. However, continuously building new benchmarks is costly and
cyclical. In this work, we aim to tackle contamination by analyzing the
mechanisms of contaminated models themselves. Through our experiments, we
discover that the overestimation of contaminated models is likely due to
parameters acquiring shortcut solutions in training. We further propose a novel
method for identifying shortcut neurons through comparative and causal
analysis. Building on this, we introduce an evaluation method called shortcut
neuron patching to suppress shortcut neurons. Experiments validate the
effectiveness of our approach in mitigating contamination. Additionally, our
evaluation results exhibit a strong linear correlation with MixEval, a recently
released trustworthy benchmark, achieving a Spearman coefficient (rho)
exceeding 0.95. This high correlation indicates that our method closely reveals
true capabilities of the models and is trustworthy. We conduct further
experiments to demonstrate the generalizability of our method across various
benchmarks and hyperparameter settings. Code:
https://github.com/GaryStack/Trustworthy-Evaluation
DenseDPO: Fine-Grained Temporal Preference Optimization for Video
  Diffusion Models
Authors: name: Ziyi Wu, name: Anil Kag, name: Ivan Skorokhodov, name: Willi Menapace, name: Ashkan Mirzaei, name: Igor Gilitschenski, name: Sergey Tulyakov, name: Aliaksandr Siarohin
Published: 2025-06-04 03:06:08

Direct Preference Optimization (DPO) has recently been applied as a
post-training technique for text-to-video diffusion models. To obtain training
data, annotators are asked to provide preferences between two videos generated
from independent noise. However, this approach prohibits fine-grained
comparisons, and we point out that it biases the annotators towards low-motion
clips as they often contain fewer visual artifacts. In this work, we introduce
DenseDPO, a method that addresses these shortcomings by making three
contributions. First, we create each video pair for DPO by denoising corrupted
copies of a ground truth video. This results in aligned pairs with similar
motion structures while differing in local details, effectively neutralizing
the motion bias. Second, we leverage the resulting temporal alignment to label
preferences on short segments rather than entire clips, yielding a denser and
more precise learning signal. With only one-third of the labeled data, DenseDPO
greatly improves motion generation over vanilla DPO, while matching it in text
alignment, visual quality, and temporal consistency. Finally, we show that
DenseDPO unlocks automatic preference annotation using off-the-shelf Vision
Language Models (VLMs): GPT accurately predicts segment-level preferences
similar to task-specifically fine-tuned video reward models, and DenseDPO
trained on these labels achieves performance close to using human labels.