Orionnnn/hub-stats huggingface dataset:
VEM: Environment-Free Exploration for Training GUI Agent with Value
  Environment Model
Authors: name: Jiani Zheng, name: Lu Wang, name: Fangkai Yang, name: Chaoyun Zhang, name: Lingrui Mei, name: Wenjie Yin, name: Qingwei Lin, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2025-02-26 07:52:02

Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI)
agents via Reinforcement Learning (RL) faces critical challenges:
environment-based RL requires costly interactions, while environment-free
methods struggle with distribution shift and reward generalization. We propose
an environment-free RL framework that decouples value estimation from policy
optimization by leveraging a pretrained Value Environment Model (VEM). VEM
predicts state-action values directly from offline data, distilling human-like
priors about GUI interaction outcomes without requiring next-state prediction
or environmental feedback. This avoids compounding errors and enhances
resilience to UI changes by focusing on semantic reasoning (e.g., Does this
action advance the user's goal?). The framework operates in two stages: (1)
pretraining VEM to estimate long-term action utilities and (2) guiding policy
exploration with frozen VEM signals, enabling layout-agnostic GUI automation.
Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art
performance in both offline and online settings, outperforming environment-free
baselines significantly and matching environment-based approaches without
interaction costs. Importantly, VEM demonstrates that semantic-aware value
estimation can achieve comparable performance with online-trained methods.
Scaling LLM Pre-training with Vocabulary Curriculum
Authors: name: Fangyuan Yu
Published: 2025-02-25 07:18:29

Modern language models rely on static vocabularies, fixed before pretraining,
in contrast to the adaptive vocabulary acquisition observed in human language
learning. To bridge this gap, we introduce vocabulary curriculum learning, an
approach that improves pretraining efficiency with log-linear scaling gains
relative to vocabulary size. Our method alternates between entropy-guided
vocabulary expansion and model optimization, enabling models to learn
transferable representations across diverse tokenization granularities. This
approach naturally gives rise to an optimal computation allocation pattern:
longer tokens capture predictable content, while shorter tokens focus on more
complex, harder-to-predict contexts. Experiments on small-scale GPT models
demonstrate improved scaling efficiency, reinforcing the effectiveness of
dynamic tokenization. We release our code to support further research and plan
to extend our experiments to larger models and diverse domains.
LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven
  Language Representation
Authors: name: Pengzhi Li, name: Pengfei Yu, name: Zide Liu, name: Wei He, name: Xuhao Pan, name: Xudong Rao, name: Tao Wei, name: Wei Chen
Published: 2025-02-25 15:42:34

In this paper, we introduce LDGen, a novel method for integrating large
language models (LLMs) into existing text-to-image diffusion models while
minimizing computational demands. Traditional text encoders, such as CLIP and
T5, exhibit limitations in multilingual processing, hindering image generation
across diverse languages. We address these challenges by leveraging the
advanced capabilities of LLMs. Our approach employs a language representation
strategy that applies hierarchical caption optimization and human instruction
techniques to derive precise semantic information,. Subsequently, we
incorporate a lightweight adapter and a cross-modal refiner to facilitate
efficient feature alignment and interaction between LLMs and image features.
LDGen reduces training time and enables zero-shot multilingual image
generation. Experimental results indicate that our method surpasses baseline
models in both prompt adherence and image aesthetic quality, while seamlessly
supporting multiple languages. Project page: https://zrealli.github.io/LDGen.
MLLMs Know Where to Look: Training-free Perception of Small Visual
  Details with Multimodal LLMs
Authors: name: Jiarui Zhang, name: Mahyar Khayatkhoei, name: Prateek Chhikara, name: Filip Ilievski
Published: 2025-02-24 18:54:40

Multimodal Large Language Models (MLLMs) have experienced rapid progress in
visual recognition tasks in recent years. Given their potential integration
into many critical applications, it is important to understand the limitations
of their visual perception. In this work, we study whether MLLMs can perceive
small visual details as effectively as large ones when answering questions
about images. We observe that their performance is very sensitive to the size
of the visual subject of the question, and further show that this effect is in
fact causal by conducting an intervention study. Next, we study the attention
patterns of MLLMs when answering visual questions, and intriguingly find that
they consistently know where to look, even when they provide the wrong answer.
Based on these findings, we then propose training-free visual intervention
methods that leverage the internal knowledge of any MLLM itself, in the form of
attention and gradient maps, to enhance its perception of small visual details.
We evaluate our proposed methods on two widely-used MLLMs and seven visual
question answering benchmarks and show that they can significantly improve
MLLMs' accuracy without requiring any training. Our results elucidate the risk
of applying MLLMs to visual recognition tasks concerning small details and
indicate that visual intervention using the model's internal state is a
promising direction to mitigate this risk.
Curie: Toward Rigorous and Automated Scientific Experimentation with AI
  Agents
Authors: name: Patrick Tser Jern Kon, name: Jiachen Liu, name: Qiuyi Ding, name: Yiming Qiu, name: Zhenning Yang, name: Yibo Huang, name: Jayanth Srinivasa, name: Myungjin Lee, name: Mosharaf Chowdhury, name: Ang Chen
Published: 2025-02-22 03:58:19

Scientific experimentation, a cornerstone of human progress, demands rigor in
reliability, methodical control, and interpretability to yield meaningful
results. Despite the growing capabilities of large language models (LLMs) in
automating different aspects of the scientific process, automating rigorous
experimentation remains a significant challenge. To address this gap, we
propose Curie, an AI agent framework designed to embed rigor into the
experimentation process through three key components: an intra-agent rigor
module to enhance reliability, an inter-agent rigor module to maintain
methodical control, and an experiment knowledge module to enhance
interpretability. To evaluate Curie, we design a novel experimental benchmark
composed of 46 questions across four computer science domains, derived from
influential research papers, and widely adopted open-source projects. Compared
to the strongest baseline tested, we achieve a 3.4times improvement in
correctly answering experimental questions.Curie is open-sourced at
https://github.com/Just-Curieous/Curie.
An Overview of Large Language Models for Statisticians
Authors: name: Wenlong Ji, name: Weizhe Yuan, name: Emily Getzen, name: Kyunghyun Cho, name: Michael I. Jordan, name: Song Mei, name: Jason E Weston, name: Weijie J. Su, name: Jing Xu, name: Linjun Zhang
Published: 2025-02-25 03:40:36

Large Language Models (LLMs) have emerged as transformative tools in
artificial intelligence (AI), exhibiting remarkable capabilities across diverse
tasks such as text generation, reasoning, and decision-making. While their
success has primarily been driven by advances in computational power and deep
learning architectures, emerging problems -- in areas such as uncertainty
quantification, decision-making, causal inference, and distribution shift --
require a deeper engagement with the field of statistics. This paper explores
potential areas where statisticians can make important contributions to the
development of LLMs, particularly those that aim to engender trustworthiness
and transparency for human users. Thus, we focus on issues such as uncertainty
quantification, interpretability, fairness, privacy, watermarking and model
adaptation. We also consider possible roles for LLMs in statistical analysis.
By bridging AI and statistics, we aim to foster a deeper collaboration that
advances both the theoretical foundations and practical applications of LLMs,
ultimately shaping their role in addressing complex societal challenges.
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More
  Challenging
Authors: name: Ahmed Elhady, name: Eneko Agirre, name: Mikel Artetxe
Published: 2025-02-25 16:09:38

We introduce WiCkeD, a simple method to increase the complexity of existing
multiple-choice benchmarks by randomly replacing a choice with "None of the
above", a method often used in educational tests. We show that WiCkeD can be
automatically applied to any existing benchmark, making it more challenging. We
apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight
LLMs. The performance of the models drops 12.1 points on average with respect
to the original versions of the datasets. When using chain-of-thought on 3 MMLU
datasets, the performance drop for the WiCkeD variant is similar to the one
observed when using the LLMs directly, showing that WiCkeD is also challenging
for models with enhanced reasoning abilities. WiCkeD also uncovers that some
models are more sensitive to the extra reasoning required, providing additional
information with respect to the original benchmarks. We relase our code and
data at https://github.com/ahmedselhady/wicked-benchmarks.
Prompt-to-Leaderboard
Authors: name: Evan Frick, name: Connor Chen, name: Joseph Tennyson, name: Tianle Li, name: Wei-Lin Chiang, name: Anastasios N. Angelopoulos, name: Ion Stoica
Published: 2025-02-20 18:58:07

Large language model (LLM) evaluations typically rely on aggregated metrics
like accuracy or human preference, averaging across users and prompts. This
averaging obscures user- and prompt-specific variations in model performance.
To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces
leaderboards specific to a prompt. The core idea is to train an LLM taking
natural language prompts as input to output a vector of Bradley-Terry
coefficients which are then used to predict the human preference vote. The
resulting prompt-dependent leaderboards allow for unsupervised task-specific
evaluation, optimal routing of queries to models, personalization, and
automated evaluation of model strengths and weaknesses. Data from Chatbot Arena
suggest that P2L better captures the nuanced landscape of language model
performance than the averaged leaderboard. Furthermore, our findings suggest
that P2L's ability to produce prompt-specific evaluations follows a power law
scaling similar to that observed in LLMs themselves. In January 2025, the
router we trained based on this methodology achieved the \#1 spot in the
Chatbot Arena leaderboard. Our code is available at this GitHub link:
https://github.com/lmarena/p2l.
Finding the Sweet Spot: Preference Data Construction for Scaling
  Preference Optimization
Authors: name: Yao Xiao, name: Hai Ye, name: Linyao Chen, name: Hwee Tou Ng, name: Lidong Bing, name: Xiaoli Li, name: Roy Ka-wei Lee
Published: 2025-02-24 04:22:57

Iterative data generation and model retraining are widely used to align large
language models (LLMs). It typically involves a policy model to generate
on-policy responses and a reward model to guide training data selection. Direct
Preference Optimization (DPO) further enhances this process by constructing
preference pairs of chosen and rejected responses. In this work, we aim to
scale up the number of on-policy samples via repeated random sampling to
improve alignment performance. Conventional practice selects the sample with
the highest reward as chosen and the lowest as rejected for DPO. However, our
experiments reveal that this strategy leads to a decline in performance
as the sample size increases. To address this, we investigate preference data
construction through the lens of underlying normal distribution of sample
rewards. We categorize the reward space into seven representative points and
systematically explore all 21 (C_7^2) pairwise combinations. Through
evaluations on four models using AlpacaEval 2, we find that selecting the
rejected response at reward position mu - 2sigma rather than the minimum
reward, is crucial for optimal performance. We finally introduce a scalable
preference data construction strategy that consistently enhances model
performance as the sample scale increases.
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
Authors: name: Hugo Pitorro, name: Marcos Treviso
Published: 2025-02-21 17:33:59

State space models (SSMs), such as Mamba, have emerged as an efficient
alternative to transformers for long-context sequence modeling. However,
despite their growing adoption, SSMs lack the interpretability tools that have
been crucial for understanding and improving attention-based architectures.
While recent efforts provide insights into Mamba's internal mechanisms, they do
not explicitly decompose token-wise contributions, leaving gaps in
understanding how Mamba selectively processes sequences across layers. In this
work, we introduce LaTIM, a novel token-level decomposition method for both
Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively
evaluate our method across diverse tasks, including machine translation,
copying, and retrieval-based generation, demonstrating its effectiveness in
revealing Mamba's token-to-token interaction patterns.
Introducing Visual Perception Token into Multimodal Large Language Model
Authors: name: Runpeng Yu, name: Xinyin Ma, name: Xinchao Wang
Published: 2025-02-24 18:56:12

To utilize visual information, Multimodal Large Language Model (MLLM) relies
on the perception process of its vision encoder. The completeness and accuracy
of visual perception significantly influence the precision of spatial
reasoning, fine-grained understanding, and other tasks. However, MLLM still
lacks the autonomous capability to control its own visual perception processes,
for example, selectively reviewing specific regions of an image or focusing on
information related to specific object categories. In this work, we propose the
concept of Visual Perception Token, aiming to empower MLLM with a mechanism to
control its visual perception processes. We design two types of Visual
Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding
Token. MLLMs autonomously generate these tokens, just as they generate text,
and use them to trigger additional visual perception actions. The Region
Selection Token explicitly identifies specific regions in an image that require
further perception, while the Vision Re-Encoding Token uses its hidden states
as control signals to guide additional visual perception processes. Extensive
experiments demonstrate the advantages of these tokens in handling spatial
reasoning, improving fine-grained understanding, and other tasks. On average,
the introduction of Visual Perception Tokens improves the performance of a 2B
model by 23.6\%, increasing its score from 0.572 to 0.708, and even outperforms
a 7B parameter model by 13.4\% (from 0.624). Please check out our repo
https://github.com/yu-rp/VisualPerceptionToken
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM
  Compression Preserve?
Authors: name: Zhenheng Tang, name: Xiang Liu, name: Qian Wang, name: Peijie Dong, name: Bingsheng He, name: Xiaowen Chu, name: Bo Li
Published: 2025-02-24 15:39:35

Motivated by reducing the computational and storage costs of LLMs, model
compression and KV cache compression have attracted much attention from
researchers. However, current methods predominantly emphasize maintaining the
performance of compressed LLMs, as measured by perplexity or simple accuracy on
tasks of common sense knowledge QA and basic arithmetic reasoning. In this
blog, we present a brief review of recent advancements in LLMs related to
retrieval-augmented generation, multi-step reasoning, external tools, and
computational expressivity, all of which substantially enhance LLM performance.
Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and
task, there exists a smaller lottery LLM capable of producing the same
performance as the original LLM with the assistance of multi-step reasoning and
external tools. Based on the review of current progress in LLMs, we discuss and
summarize the essential capabilities that the lottery LLM and KV cache
compression must possess, which are currently overlooked in existing methods.
K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs
Authors: name: Ziheng Ouyang, name: Zhen Li, name: Qibin Hou
Published: 2025-02-25 18:59:12

Recent studies have explored combining different LoRAs to jointly generate
learned style and content. However, existing methods either fail to effectively
preserve both the original subject and style simultaneously or require
additional training. In this paper, we argue that the intrinsic properties of
LoRA can effectively guide diffusion models in merging learned subject and
style. Building on this insight, we propose K-LoRA, a simple yet effective
training-free LoRA fusion approach. In each attention layer, K-LoRA compares
the Top-K elements in each LoRA to be fused, determining which LoRA to select
for optimal fusion. This selection mechanism ensures that the most
representative features of both subject and style are retained during the
fusion process, effectively balancing their contributions. Experimental results
demonstrate that the proposed method effectively integrates the subject and
style information learned by the original LoRAs, outperforming state-of-the-art
training-based approaches in both qualitative and quantitative results.
Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI
Authors: name: Syed Abdul Gaffar Shakhadri, name: Kruthika KR, name: Kartik Basavaraj Angadi
Published: 2025-02-24 12:15:07

We introduce Shakti VLM, a family of vision-language models in the capacity
of 1B and 4B parameters designed to address data efficiency challenges in
multimodal learning. While recent VLMs achieve strong performance through
extensive training data, Shakti models leverage architectural innovations to
attain competitive results with fewer tokens. Key advancements include
QK-Normalization for attention stability, hybrid normalization techniques, and
enhanced positional encoding. A three-stage training strategy further optimizes
learning efficiency. Evaluations show that Shakti-Shakti-VLM-1B and
Shakti-VLM-4B excel in document understanding, Visual Reasoning, OCR
extraction, and general multimodal reasoning. Our results highlight that high
performance can be achieved through model design and training strategy rather
than sheer data volume, making Shakti an efficient solution for
enterprise-scale multimodal tasks.
Scale-Distribution Decoupling: Enabling Stable and Effective Training of
  Large Language Models
Authors: name: Ya Wang, name: Zhijian Zhuo, name: Yutao Zeng, name: Xun Zhou, name: Jian Yang, name: Xiaoqing Li
Published: 2025-02-21 14:49:34

Training stability is a persistent challenge in the pre-training of large
language models (LLMs), particularly for architectures such as Post-Norm
Transformers, which are prone to gradient explosion and dissipation. In this
paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that
stabilizes training by explicitly decoupling the scale and distribution of the
weight matrix in fully-connected layers. SDD applies a normalization mechanism
to regulate activations and a learnable scaling vector to maintain
well-conditioned gradients, effectively preventing gradient explosion
and dissipation. This separation improves optimization efficiency,
particularly in deep networks, by ensuring stable gradient propagation.
Experimental results demonstrate that our method stabilizes training across
various LLM architectures and outperforms existing techniques in different
normalization configurations. Furthermore, the proposed method is lightweight
and compatible with existing frameworks, making it a practical solution for
stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
WebGames: Challenging General-Purpose Web-Browsing AI Agents
Authors: name: George Thomas, name: Alex J. Chan, name: Jikun Kang, name: Wenqi Wu, name: Filippos Christianos, name: Fraser Greenlee, name: Andy Toulis, name: Marvin Purtorab
Published: 2025-02-25 16:45:08

We introduce WebGames, a comprehensive benchmark suite designed to evaluate
general-purpose web-browsing AI agents through a collection of 50+ interactive
challenges. These challenges are specifically crafted to be straightforward for
humans while systematically testing the limitations of current AI systems
across fundamental browser interactions, advanced input processing, cognitive
tasks, workflow automation, and interactive entertainment. Our framework
eliminates external dependencies through a hermetic testing environment,
ensuring reproducible evaluation with verifiable ground-truth solutions. We
evaluate leading vision-language models including GPT-4o, Claude Computer-Use,
Gemini-1.5-Pro, and Qwen2-VL against human performance. Results reveal a
substantial capability gap, with the best AI system achieving only 43.1%
success rate compared to human performance of 95.7%, highlighting fundamental
limitations in current AI systems' ability to handle common web interaction
patterns that humans find intuitive. The benchmark is publicly available at
webgames.convergence.ai, offering a lightweight, client-side implementation
that facilitates rapid evaluation cycles. Through its modular architecture and
standardized challenge specifications, WebGames provides a robust foundation
for measuring progress in development of more capable web-browsing agents.
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Authors: name: Xilin Jiang, name: Sukru Samet Dindar, name: Vishal Choudhari, name: Stephan Bickel, name: Ashesh Mehta, name: Guy M McKhann, name: Adeen Flinker, name: Daniel Friedman, name: Nima Mesgarani
Published: 2025-02-24 03:06:45

Auditory foundation models, including auditory large language models (LLMs),
process all sound inputs equally, independent of listener perception. However,
human auditory perception is inherently selective: listeners focus on specific
speakers while ignoring others in complex auditory scenes. Existing models do
not incorporate this selectivity, limiting their ability to generate
perception-aligned responses. To address this, we introduce Intention-Informed
Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
(AAD-LLM), a prototype system that integrates brain signals to infer listener
attention. AAD-LLM extends an auditory LLM by incorporating intracranial
electroencephalography (iEEG) recordings to decode which speaker a listener is
attending to and refine responses accordingly. The model first predicts the
attended speaker from neural activity, then conditions response generation on
this inferred attentional state. We evaluate AAD-LLM on speaker description,
speech transcription and extraction, and question answering in multitalker
scenarios, with both objective and subjective ratings showing improved
alignment with listener intention. By taking a first step toward
intention-aware auditory AI, this work explores a new paradigm where listener
perception informs machine listening, paving the way for future
listener-centered auditory systems. Demo and code available:
https://aad-llm.github.io.