Orionnnn/hub-stats huggingface dataset:
Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health
  Information
Authors: name: Joshua Harris, name: Fan Grayson, name: Felix Feldman, name: Timothy Laurence, name: Toby Nonnenmacher, name: Oliver Higgins, name: Leo Loman, name: Selina Patel, name: Thomas Finnie, name: Samuel Collins, name: Michael Borowitz
Published: 2025-05-09 13:42:59

As Large Language Models (LLMs) become widely accessible, a detailed
understanding of their knowledge within specific domains becomes necessary for
successful real world use. This is particularly critical in public health,
where failure to retrieve relevant, accurate, and current information could
significantly impact UK residents. However, currently little is known about LLM
knowledge of UK Government public health information. To address this issue,
this paper introduces a new benchmark, PubHealthBench, with over 8000 questions
for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form
responses to public health queries, created via an automated pipeline. We also
release a new dataset of the extracted UK Government public health guidance
documents used as source text for PubHealthBench. Assessing 24 LLMs on
PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a
high degree of knowledge, achieving >90% in the MCQA setup, and outperform
humans with cursory search engine use. However, in the free form setup we see
lower performance with no model scoring >75%. Therefore, whilst there are
promising signs that state of the art (SOTA) LLMs are an increasingly accurate
source of public health information, additional safeguards or tools may still
be needed when providing free form responses on public health topics.
Bielik v3 Small: Technical Report
Authors: name: Krzysztof Ociepa, name: Łukasz Flis, name: Remigiusz Kinas, name: Krzysztof Wróbel, name: Adrian Gwoździej
Published: 2025-05-05 10:39:51

We introduce Bielik v3, a series of parameter-efficient generative text
models (1.5B and 4.5B) optimized for Polish language processing. These models
demonstrate that smaller, well-optimized architectures can achieve performance
comparable to much larger counterparts while requiring substantially fewer
computational resources. Our approach incorporates several key innovations: a
custom Polish tokenizer (APT4) that significantly improves token efficiency,
Weighted Instruction Cross-Entropy Loss to balance learning across instruction
types, and Adaptive Learning Rate that dynamically adjusts based on training
progress. Trained on a meticulously curated corpus of 292 billion tokens
spanning 303 million documents, these models excel across multiple benchmarks,
including the Open PL LLM Leaderboard, Complex Polish Text Understanding
Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter
model achieves results competitive with models 2-3 times its size, while the
1.5B model delivers strong performance despite its extremely compact profile.
These advances establish new benchmarks for parameter-efficient language
modeling in less-represented languages, making high-quality Polish language AI
more accessible for resource-constrained applications.
Bielik 11B v2 Technical Report
Authors: name: Krzysztof Ociepa, name: Łukasz Flis, name: Krzysztof Wróbel, name: Adrian Gwoździej, name: Remigiusz Kinas
Published: 2025-05-05 07:03:41

We present Bielik 11B v2, a state-of-the-art language model optimized for
Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
11B parameters using depth up-scaling, this model demonstrates exceptional
performance across Polish language benchmarks while maintaining strong
cross-lingual capabilities. We introduce two key technical innovations:
Weighted Instruction Cross-Entropy Loss, which optimizes learning across
diverse instruction types by assigning quality-based weights to training
examples, and Adaptive Learning Rate, which dynamically adjusts based on
context length. Comprehensive evaluation across multiple benchmarks
demonstrates that Bielik 11B v2 outperforms many larger models, including those
with 2-6 times more parameters, and significantly surpasses other specialized
Polish language models on tasks ranging from linguistic understanding to
complex reasoning. The model's parameter efficiency and extensive quantization
options enable deployment across various hardware configurations, advancing
Polish language AI capabilities and establishing new benchmarks for
resource-efficient language modeling in less-represented languages.
Sailing AI by the Stars: A Survey of Learning from Rewards in
  Post-Training and Test-Time Scaling of Large Language Models
Authors: name: Xiaobao Wu
Published: 2025-05-05 14:33:49

Recent developments in Large Language Models (LLMs) have shifted from
pre-training scaling to post-training and test-time scaling. Across these
developments, a key unified paradigm has arisen: Learning from Rewards, where
reward signals act as the guiding stars to steer LLM behavior. It has
underpinned a wide range of prevalent techniques, such as reinforcement
learning (in RLHF, DPO, and GRPO), reward-guided decoding, and post-hoc
correction. Crucially, this paradigm enables the transition from passive
learning from static data to active learning from dynamic feedback. This endows
LLMs with aligned preferences and deep reasoning capabilities. In this survey,
we present a comprehensive overview of the paradigm of learning from rewards.
We categorize and analyze the strategies under this paradigm across training,
inference, and post-inference stages. We further discuss the benchmarks for
reward models and the primary applications. Finally we highlight the challenges
and future directions. We maintain a paper collection at
https://github.com/bobxwu/learning-from-rewards-llm-papers.
G-FOCUS: Towards a Robust Method for Assessing UI Design Persuasiveness
Authors: name: Jaehyun Jeon, name: Jang Han Yoon, name: Min Soo Kim, name: Sumin Shim, name: Yejin Choi, name: Hanbin Kim, name: Youngjae Yu
Published: 2025-05-08 08:00:32

Evaluating user interface (UI) design effectiveness extends beyond aesthetics
to influencing user behavior, a principle central to Design Persuasiveness. A/B
testing is the predominant method for determining which UI variations drive
higher user engagement, but it is costly and time-consuming. While recent
Vision-Language Models (VLMs) can process automated UI analysis, current
approaches focus on isolated design attributes rather than comparative
persuasiveness-the key factor in optimizing user interactions. To address this,
we introduce WiserUI-Bench, a benchmark designed for Pairwise UI Design
Persuasiveness Assessment task, featuring 300 real-world UI image pairs labeled
with A/B test results and expert rationales. Additionally, we propose G-FOCUS,
a novel inference-time reasoning strategy that enhances VLM-based
persuasiveness assessment by reducing position bias and improving evaluation
accuracy. Experimental results show that G-FOCUS surpasses existing inference
strategies in consistency and accuracy for pairwise UI evaluation. Through
promoting VLM-driven evaluation of UI persuasiveness, our work offers an
approach to complement A/B testing, propelling progress in scalable UI
preference modeling and design optimization. Code and data will be released
publicly.
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions
Authors: name: Qingwen Bu, name: Yanting Yang, name: Jisong Cai, name: Shenyuan Gao, name: Guanghui Ren, name: Maoqing Yao, name: Ping Luo, name: Hongyang Li
Published: 2025-05-09 15:11:13

A generalist robot should perform effectively across various environments.
However, most existing approaches heavily rely on scaling action-annotated data
to enhance their capabilities. Consequently, they are often limited to single
physical specification and struggle to learn transferable knowledge across
different embodiments and environments. To confront these limitations, we
propose UniVLA, a new framework for learning cross-embodiment
vision-language-action (VLA) policies. Our key innovation is to derive
task-centric action representations from videos with a latent action model.
This enables us to exploit extensive data across a wide spectrum of embodiments
and perspectives. To mitigate the effect of task-irrelevant dynamics, we
incorporate language instructions and establish a latent action model within
the DINO feature space. Learned from internet-scale videos, the generalist
policy can be deployed to various robots through efficient latent action
decoding. We obtain state-of-the-art results across multiple manipulation and
navigation benchmarks, as well as real-robot deployments. UniVLA achieves
superior performance over OpenVLA with less than 1/20 of pretraining compute
and 1/10 of downstream data. Continuous performance improvements are observed
as heterogeneous data, even including human videos, are incorporated into the
training pipeline. The results underscore UniVLA's potential to facilitate
scalable and efficient robot policy learning.
Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM
  Reasoners With Verifiers
Authors: name: Kusha Sareen, name: Morgane M Moss, name: Alessandro Sordoni, name: Rishabh Agarwal, name: Arian Hosseini
Published: 2025-05-07 22:41:26

Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners,
such as GRPO or Leave-one-out PPO, abandon the learned value function in favor
of empirically estimated returns. This hinders test-time compute scaling that
relies on using the value-function for verification. In this work, we propose
RL^V that augments any ``value-free'' RL method by jointly training the LLM
as both a reasoner and a generative verifier using RL-generated data, adding
verification capabilities without significant overhead. Empirically, RL^V
boosts MATH accuracy by over 20\% with parallel sampling and enables
8-32times efficient test-time compute scaling compared to the base RL
method. RL^V also exhibits strong generalization capabilities for both
easy-to-hard and out-of-domain tasks. Furthermore, RL^V achieves
1.2-1.6times higher performance when jointly scaling parallel and sequential
test-time compute with a long reasoning R1 model.
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in
  Language Model Preference Learning
Authors: name: Tianjian Li, name: Daniel Khashabi
Published: 2025-05-05 04:54:44

Aligning language models with human preferences relies on pairwise preference
datasets. While some studies suggest that on-policy data consistently
outperforms off -policy data for preference learning, others indicate that the
advantages of on-policy data may be task-dependent, highlighting the need for a
systematic exploration of their interplay.
  In this work, we show that on-policy and off-policy data offer complementary
strengths in preference optimization: on-policy data is particularly effective
for reasoning tasks like math and coding, while off-policy data performs better
on open-ended tasks such as creative writing and making personal
recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach
to combine the complementary strengths of on-policy and off-policy preference
learning by simply mixing these two data sources. Our empirical results across
diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves
language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO
and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it
outperforms prior approaches that are much more complex in combining on- and
off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.
WaterDrum: Watermarking for Data-centric Unlearning Metric
Authors: name: Xinyang Lu, name: Xinyuan Niu, name: Gregory Kang Ruey Lau, name: Bui Thi Cam Nhung, name: Rachael Hwee Ling Sim, name: Fanyu Wen, name: Chuan-Sheng Foo, name: See-Kiong Ng, name: Bryan Kian Hsiang Low
Published: 2025-05-08 08:56:46

Large language model (LLM) unlearning is critical in real-world applications
where it is necessary to efficiently remove the influence of private,
copyrighted, or harmful data from some users. However, existing utility-centric
unlearning metrics (based on model utility) may fail to accurately evaluate the
extent of unlearning in realistic settings such as when (a) the forget and
retain set have semantically similar content, (b) retraining the model from
scratch on the retain set is impractical, and/or (c) the model owner can
improve the unlearning metric without directly performing unlearning on the
LLM. This paper presents the first data-centric unlearning metric for LLMs
called WaterDrum that exploits robust text watermarking for overcoming these
limitations. We also introduce new benchmark datasets for LLM unlearning that
contain varying levels of similar data points and can be used to rigorously
evaluate unlearning algorithms using WaterDrum. Our code is available at
https://github.com/lululu008/WaterDrum and our new benchmark datasets are
released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.
Vision-Language-Action Models: Concepts, Progress, Applications and
  Challenges
Authors: name: Ranjan Sapkota, name: Yang Cao, name: Konstantinos I. Roumeliotis, name: Manoj Karkee
Published: 2025-05-07 19:46:43

Vision-Language-Action (VLA) models mark a transformative advancement in
artificial intelligence, aiming to unify perception, natural language
understanding, and embodied action within a single computational framework.
This foundational review presents a comprehensive synthesis of recent
advancements in Vision-Language-Action models, systematically organized across
five thematic pillars that structure the landscape of this rapidly evolving
field. We begin by establishing the conceptual foundations of VLA systems,
tracing their evolution from cross-modal learning architectures to generalist
agents that tightly integrate vision-language models (VLMs), action planners,
and hierarchical controllers. Our methodology adopts a rigorous literature
review framework, covering over 80 VLA models published in the past three
years. Key progress areas include architectural innovations,
parameter-efficient training strategies, and real-time inference accelerations.
We explore diverse application domains such as humanoid robotics, autonomous
vehicles, medical and industrial robotics, precision agriculture, and augmented
reality navigation. The review further addresses major challenges across
real-time control, multimodal action representation, system scalability,
generalization to unseen tasks, and ethical deployment risks. Drawing from the
state-of-the-art, we propose targeted solutions including agentic AI
adaptation, cross-embodiment generalization, and unified neuro-symbolic
planning. In our forward-looking discussion, we outline a future roadmap where
VLA models, VLMs, and agentic AI converge to power socially aligned, adaptive,
and general-purpose embodied agents. This work serves as a foundational
reference for advancing intelligent, real-world robotics and artificial general
intelligence. >Vision-language-action, Agentic AI, AI Agents, Vision-language
Models
LiftFeat: 3D Geometry-Aware Local Feature Matching
Authors: name: Yepeng Liu, name: Wenpeng Lai, name: Zhou Zhao, name: Yuxuan Xiong, name: Jinchi Zhu, name: Jun Cheng, name: Yongchao Xu
Published: 2025-05-06 10:59:23

Robust and efficient local feature matching plays a crucial role in
applications such as SLAM and visual localization for robotics. Despite great
progress, it is still very challenging to extract robust and discriminative
visual features in scenarios with drastic lighting changes, low texture areas,
or repetitive patterns. In this paper, we propose a new lightweight network
called LiftFeat, which lifts the robustness of raw descriptor by
aggregating 3D geometric feature. Specifically, we first adopt a pre-trained
monocular depth estimation model to generate pseudo surface normal label,
supervising the extraction of 3D geometric feature in terms of predicted
surface normal. We then design a 3D geometry-aware feature lifting module to
fuse surface normal feature with raw 2D descriptor feature. Integrating such 3D
geometric feature enhances the discriminative ability of 2D feature description
in extreme conditions. Extensive experimental results on relative pose
estimation, homography estimation, and visual localization tasks, demonstrate
that our LiftFeat outperforms some lightweight state-of-the-art methods. Code
will be released at : https://github.com/lyp-deeplearning/LiftFeat.
Chain-of-Thought Tokens are Computer Program Variables
Authors: name: Fangwei Zhu, name: Peiyi Wang, name: Zhifang Sui
Published: 2025-05-08 05:32:36

Chain-of-thoughts (CoT) requires large language models (LLMs) to generate
intermediate steps before reaching the final answer, and has been proven
effective to help LLMs solve complex reasoning tasks. However, the inner
mechanism of CoT still remains largely unclear. In this paper, we empirically
study the role of CoT tokens in LLMs on two compositional tasks: multi-digit
multiplication and dynamic programming. While CoT is essential for solving
these problems, we find that preserving only tokens that store intermediate
results would achieve comparable performance. Furthermore, we observe that
storing intermediate results in an alternative latent form will not affect
model performance. We also randomly intervene some values in CoT, and notice
that subsequent CoT tokens and the final answer would change correspondingly.
These findings suggest that CoT tokens may function like variables in computer
programs but with potential drawbacks like unintended shortcuts and
computational complexity limits between tokens. The code and data are available
at https://github.com/solitaryzero/CoTs_are_Variables.
ICon: In-Context Contribution for Automatic Data Selection
Authors: name: Yixin Yang, name: Qingxiu Dong, name: Linli Yao, name: Fangwei Zhu, name: Zhifang Sui
Published: 2025-05-08 15:17:37

Data selection for instruction tuning is essential for improving the
performance of Large Language Models (LLMs) and reducing training cost.
However, existing automated selection methods either depend on computationally
expensive gradient-based measures or manually designed heuristics, which may
fail to fully exploit the intrinsic attributes of data. In this paper, we
propose In-context Learning for Contribution Measurement (ICon), a novel
gradient-free method that takes advantage of the implicit fine-tuning nature of
in-context learning (ICL) to measure sample contribution without gradient
computation or manual indicators engineering. ICon offers a computationally
efficient alternative to gradient-based methods and reduces human inductive
bias inherent in heuristic-based approaches. ICon comprises three components
and identifies high-contribution data by assessing performance shifts under
implicit learning through ICL. Extensive experiments on three LLMs across 12
benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
outperform full datasets by 5.42% points and exceed the best performance of
widely used selection methods by 2.06% points. We further analyze
high-contribution samples selected by ICon, which show both diverse tasks and
appropriate difficulty levels, rather than just the hardest ones.
Perception, Reason, Think, and Plan: A Survey on Large Multimodal
  Reasoning Models
Authors: name: Yunxin Li, name: Zhenyu Liu, name: Zitao Li, name: Xuanyu Zhang, name: Zhenran Xu, name: Xinyu Chen, name: Haoyuan Shi, name: Shenyuan Jiang, name: Xintong Wang, name: Jifang Wang, name: Shouzheng Huang, name: Xinping Zhao, name: Borui Jiang, name: Lanqing Hong, name: Longyue Wang, name: Zhuotao Tian, name: Baoxing Huai, name: Wenhan Luo, name: Weihua Luo, name: Zheng Zhang, name: Baotian Hu, name: Min Zhang
Published: 2025-05-08 03:35:23

Reasoning lies at the heart of intelligence, shaping the ability to make
decisions, draw conclusions, and generalize across domains. In artificial
intelligence, as systems increasingly operate in open, uncertain, and
multimodal environments, reasoning becomes essential for enabling robust and
adaptive behavior. Large Multimodal Reasoning Models (LMRMs) have emerged as a
promising paradigm, integrating modalities such as text, images, audio, and
video to support complex reasoning capabilities and aiming to achieve
comprehensive perception, precise understanding, and deep reasoning. As
research advances, multimodal reasoning has rapidly evolved from modular,
perception-driven pipelines to unified, language-centric frameworks that offer
more coherent cross-modal understanding. While instruction tuning and
reinforcement learning have improved model reasoning, significant challenges
remain in omni-modal generalization, reasoning depth, and agentic behavior. To
address these issues, we present a comprehensive and structured survey of
multimodal reasoning research, organized around a four-stage developmental
roadmap that reflects the field's shifting design philosophies and emerging
capabilities. First, we review early efforts based on task-specific modules,
where reasoning was implicitly embedded across stages of representation,
alignment, and fusion. Next, we examine recent approaches that unify reasoning
into multimodal LLMs, with advances such as Multimodal Chain-of-Thought (MCoT)
and multimodal reinforcement learning enabling richer and more structured
reasoning chains. Finally, drawing on empirical insights from challenging
benchmarks and experimental cases of OpenAI O3 and O4-mini, we discuss the
conceptual direction of native large multimodal reasoning models (N-LMRMs),
which aim to support scalable, agentic, and adaptive reasoning and planning in
complex, real-world environments.
Generating Physically Stable and Buildable LEGO Designs from Text
Authors: name: Ava Pun, name: Kangle Deng, name: Ruixuan Liu, name: Deva Ramanan, name: Changliu Liu, name: Jun-Yan Zhu
Published: 2025-05-08 17:58:18

We introduce LegoGPT, the first approach for generating physically stable
LEGO brick models from text prompts. To achieve this, we construct a
large-scale, physically stable dataset of LEGO designs, along with their
associated captions, and train an autoregressive large language model to
predict the next brick to add via next-token prediction. To improve the
stability of the resulting designs, we employ an efficient validity check and
physics-aware rollback during autoregressive inference, which prunes infeasible
token predictions using physics laws and assembly constraints. Our experiments
show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
designs that align closely with the input text prompts. We also develop a
text-based LEGO texturing method to generate colored and textured designs. We
show that our designs can be assembled manually by humans and automatically by
robotic arms. We also release our new dataset, StableText2Lego, containing over
47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
captions, along with our code and models at the project website:
https://avalovelace1.github.io/LegoGPT/.