Orionnnn/hub-stats huggingface dataset:
Sparse Autoencoders Learn Monosemantic Features in Vision-Language
  Models
Authors: name: Mateusz Pach, name: Shyamgopal Karthik, name: Quentin Bouniot, name: Serge Belongie, name: Zeynep Akata
Published: 2025-04-03 17:58:35

Sparse Autoencoders (SAEs) have recently been shown to enhance
interpretability and steerability in Large Language Models (LLMs). In this
work, we extend the application of SAEs to Vision-Language Models (VLMs), such
as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
in vision representations. Our experimental results reveal that SAEs trained on
VLMs significantly enhance the monosemanticity of individual neurons while also
exhibiting hierarchical representations that align well with expert-defined
structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
applying SAEs to intervene on a CLIP vision encoder, directly steer output from
multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
model. These findings emphasize the practicality and efficacy of SAEs as an
unsupervised approach for enhancing both the interpretability and control of
VLMs.
Interpreting Emergent Planning in Model-Free Reinforcement Learning
Authors: name: Thomas Bush, name: Stephen Chung, name: Usman Anwar, name: Adrià Garriga-Alonso, name: David Krueger
Published: 2025-04-02 16:24:23

We present the first mechanistic evidence that model-free reinforcement
learning agents can learn to plan. This is achieved by applying a methodology
based on concept-based interpretability to a model-free agent in Sokoban -- a
commonly used benchmark for studying planning. Specifically, we demonstrate
that DRC, a generic model-free agent introduced by Guez et al. (2019), uses
learned concept representations to internally formulate plans that both predict
the long-term effects of actions on the environment and influence action
selection. Our methodology involves: (1) probing for planning-relevant
concepts, (2) investigating plan formation within the agent's representations,
and (3) verifying that discovered plans (in the agent's representations) have a
causal effect on the agent's behavior through interventions. We also show that
the emergence of these plans coincides with the emergence of a planning-like
property: the ability to benefit from additional test-time compute. Finally, we
perform a qualitative analysis of the planning algorithm learned by the agent
and discover a strong resemblance to parallelized bidirectional search. Our
findings advance understanding of the internal mechanisms underlying planning
behavior in agents, which is important given the recent trend of emergent
planning and reasoning capabilities in LLMs through RL
Scaling Laws in Scientific Discovery with AI and Robot Scientists
Authors: name: Pengsong Zhang, name: Heng Zhang, name: Huazhe Xu, name: Renjun Xu, name: Zhenting Wang, name: Cong Wang, name: Animesh Garg, name: Zhibin Li, name: Arash Ajoudani, name: Xinyu Liu
Published: 2025-03-28 14:00:27

Scientific discovery is poised for rapid advancement through advanced
robotics and artificial intelligence. Current scientific practices face
substantial limitations as manual experimentation remains time-consuming and
resource-intensive, while multidisciplinary research demands knowledge
integration beyond individual researchers' expertise boundaries. Here, we
envision an autonomous generalist scientist (AGS) concept combines agentic AI
and embodied robotics to automate the entire research lifecycle. This system
could dynamically interact with both physical and virtual environments while
facilitating the integration of knowledge across diverse scientific
disciplines. By deploying these technologies throughout every research stage --
spanning literature review, hypothesis generation, experimentation, and
manuscript writing -- and incorporating internal reflection alongside external
feedback, this system aims to significantly reduce the time and resources
needed for scientific discovery. Building on the evolution from virtual AI
scientists to versatile generalist AI-based robot scientists, AGS promises
groundbreaking potential. As these autonomous systems become increasingly
integrated into the research process, we hypothesize that scientific discovery
might adhere to new scaling laws, potentially shaped by the number and
capabilities of these autonomous systems, offering novel perspectives on how
knowledge is generated and evolves. The adaptability of embodied robots to
extreme environments, paired with the flywheel effect of accumulating
scientific knowledge, holds the promise of continually pushing beyond both
physical and intellectual frontiers.
Advances and Challenges in Foundation Agents: From Brain-Inspired
  Intelligence to Evolutionary, Collaborative, and Safe Systems
Authors: name: Bang Liu, name: Xinfeng Li, name: Jiayi Zhang, name: Jinlin Wang, name: Tanjin He, name: Sirui Hong, name: Hongzhang Liu, name: Shaokun Zhang, name: Kaitao Song, name: Kunlun Zhu, name: Yuheng Cheng, name: Suyuchen Wang, name: Xiaoqiang Wang, name: Yuyu Luo, name: Haibo Jin, name: Peiyan Zhang, name: Ollie Liu, name: Jiaqi Chen, name: Huan Zhang, name: Zhaoyang Yu, name: Haochen Shi, name: Boyan Li, name: Dekun Wu, name: Fengwei Teng, name: Xiaojun Jia, name: Jiawei Xu, name: Jinyu Xiang, name: Yizhang Lin, name: Tianming Liu, name: Tongliang Liu, name: Yu Su, name: Huan Sun, name: Glen Berseth, name: Jianyun Nie, name: Ian Foster, name: Logan Ward, name: Qingyun Wu, name: Yu Gu, name: Mingchen Zhuge, name: Xiangru Tang, name: Haohan Wang, name: Jiaxuan You, name: Chi Wang, name: Jian Pei, name: Qiang Yang, name: Xiaoliang Qi, name: Chenglin Wu
Published: 2025-03-31 18:00:29

The advent of large language models (LLMs) has catalyzed a transformative
shift in artificial intelligence, paving the way for advanced intelligent
agents capable of sophisticated reasoning, robust perception, and versatile
action across diverse domains. As these agents increasingly drive AI research
and practical applications, their design, evaluation, and continuous
improvement present intricate, multifaceted challenges. This survey provides a
comprehensive overview, framing intelligent agents within a modular,
brain-inspired architecture that integrates principles from cognitive science,
neuroscience, and computational research. We structure our exploration into
four interconnected parts. First, we delve into the modular foundation of
intelligent agents, systematically mapping their cognitive, perceptual, and
operational modules onto analogous human brain functionalities, and elucidating
core components such as memory, world modeling, reward processing, and
emotion-like systems. Second, we discuss self-enhancement and adaptive
evolution mechanisms, exploring how agents autonomously refine their
capabilities, adapt to dynamic environments, and achieve continual learning
through automated optimization paradigms, including emerging AutoML and
LLM-driven optimization strategies. Third, we examine collaborative and
evolutionary multi-agent systems, investigating the collective intelligence
emerging from agent interactions, cooperation, and societal structures,
highlighting parallels to human social dynamics. Finally, we address the
critical imperative of building safe, secure, and beneficial AI systems,
emphasizing intrinsic and extrinsic security threats, ethical alignment,
robustness, and practical mitigation strategies necessary for trustworthy
real-world deployment.
ShortV: Efficient Multimodal Large Language Models by Freezing Visual
  Tokens in Ineffective Layers
Authors: name: Qianhao Yuan, name: Qingyu Zhang, name: Yanjiang Liu, name: Jiawei Chen, name: Yaojie Lu, name: Hongyu Lin, name: Jia Zheng, name: Xianpei Han, name: Le Sun
Published: 2025-04-01 07:47:55

Multimodal Large Language Models (MLLMs) suffer from high computational costs
due to their massive size and the large number of visual tokens. In this paper,
we investigate layer-wise redundancy in MLLMs by introducing a novel metric,
Layer Contribution (LC), which quantifies the impact of a layer's
transformations on visual and text tokens, respectively. The calculation of LC
involves measuring the divergence in model output that results from removing
the layer's transformations on the specified tokens. Our pilot experiment
reveals that many layers of MLLMs exhibit minimal contribution during the
processing of visual tokens. Motivated by this observation, we propose ShortV,
a training-free method that leverages LC to identify ineffective layers, and
freezes visual token updates in these layers. Experiments show that ShortV can
freeze visual token in approximately 60\% of the MLLM layers, thereby
dramatically reducing computational costs related to updating visual tokens.
For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while
maintaining superior performance. The code will be publicly available at
https://github.com/icip-cas/ShortV
Scaling Analysis of Interleaved Speech-Text Language Models
Authors: name: Gallil Maimon, name: Michael Hassid, name: Amit Roth, name: Yossi Adi
Published: 2025-04-03 08:46:56

Existing Speech Language Model (SLM) scaling analysis paints a bleak picture.
They predict that SLMs require much more compute and data compared to text,
leading some to question the feasibility of training high-quality SLMs.
However, modern SLMs are often initialised from pre-trained TextLMs using
speech-text interleaving to allow knowledge transfer. This raises the question
- Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper
we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by
training several dozen and analysing the scaling trends. We see that under this
setup SLMs scale more efficiently with compute. Additionally, our results
indicate that the scaling-dynamics are significantly different than
textless-SLMs, suggesting one should allocate notably more of the compute
budget for increasing model size over training tokens. We also study the role
of synthetic data and TextLM model families in unlocking this potential.
Results suggest, that our scaled up model achieves comparable performance with
leading models on speech semantic metrics while using less compute and data
than other approaches. We open source models, samples, and data -
https://pages.cs.huji.ac.il/adiyoss-lab/sims.
GenPRM: Scaling Test-Time Compute of Process Reward Models via
  Generative Reasoning
Authors: name: Jian Zhao, name: Runze Liu, name: Kaiyan Zhang, name: Zhimu Zhou, name: Junqi Gao, name: Dong Li, name: Jiafei Lyu, name: Zhouyi Qian, name: Biqing Qi, name: Xiu Li, name: Bowen Zhou
Published: 2025-04-01 15:21:05

Recent advancements in Large Language Models (LLMs) have shown that it is
promising to utilize Process Reward Models (PRMs) as verifiers to enhance the
performance of LLMs. However, current PRMs face three key challenges: (1)
limited process supervision and generalization capabilities, (2) dependence on
scalar value prediction without leveraging the generative abilities of LLMs,
and (3) inability to scale the test-time compute of PRMs. In this work, we
introduce GenPRM, a generative process reward model that performs explicit
Chain-of-Thought (CoT) reasoning with code verification before providing
judgment for each reasoning step. To obtain high-quality process supervision
labels and rationale data, we propose Relative Progress Estimation (RPE) and a
rationale synthesis framework that incorporates code verification. Experimental
results on ProcessBench and several mathematical reasoning tasks show that
GenPRM significantly outperforms prior PRMs with only 23K training data from
MATH dataset. Through test-time scaling, a 1.5B GenPRM outperforms GPT-4o, and
a 7B GenPRM surpasses Qwen2.5-Math-PRM-72B on ProcessBench. Additionally,
GenPRM demonstrates strong abilities to serve as a critic model for policy
model refinement. This work establishes a new paradigm for process supervision
that bridges the gap between PRMs and critic models in LLMs. Our code, model,
and data will be available in https://ryanliu112.github.io/GenPRM.
Instruction-Guided Autoregressive Neural Network Parameter Generation
Authors: name: Soro Bedionita, name: Bruno Andreis, name: Song Chong, name: Sung Ju Hwang
Published: 2025-04-02 05:50:19

Learning to generate neural network parameters conditioned on task
descriptions and architecture specifications is pivotal for advancing model
adaptability and transfer learning. Existing methods especially those based on
diffusion models suffer from limited scalability to large architectures,
rigidity in handling varying network depths, and disjointed parameter
generation that undermines inter-layer coherence. In this work, we propose IGPG
(Instruction Guided Parameter Generation), an autoregressive framework that
unifies parameter synthesis across diverse tasks and architectures. IGPG
leverages a VQ-VAE and an autoregressive model to generate neural network
parameters, conditioned on task instructions, dataset, and architecture
details. By autoregressively generating neural network weights' tokens, IGPG
ensures inter-layer coherence and enables efficient adaptation across models
and datasets. Operating at the token level, IGPG effectively captures complex
parameter distributions aggregated from a broad spectrum of pretrained models.
Extensive experiments on multiple vision datasets demonstrate that IGPG
consolidates diverse pretrained models into a single, flexible generative
framework. The synthesized parameters achieve competitive or superior
performance relative to state-of-the-art methods, especially in terms of
scalability and efficiency when applied to large architectures. These results
underscore ICPG potential as a powerful tool for pretrained weight retrieval,
model selection, and rapid task-specific fine-tuning.
ZClip: Adaptive Spike Mitigation for LLM Pre-Training
Authors: name: Abhay Kumar, name: Louis Owen, name: Nilabhra Roy Chowdhury, name: Fabian Güra
Published: 2025-04-03 11:41:55

Training large language models (LLMs) presents numerous challenges, including
gradient instability and loss spikes. These phenomena can lead to catastrophic
divergence, requiring costly checkpoint restoration and data batch skipping.
Traditional gradient clipping techniques, such as constant or norm-based
methods, fail to address these issues effectively due to their reliance on
fixed thresholds or heuristics, leading to inefficient learning and requiring
frequent manual intervention. In this work, we propose ZClip, an adaptive
gradient clipping algorithm that dynamically adjusts the clipping threshold
based on statistical properties of gradient norms over time. Unlike prior
reactive strategies, ZClip proactively adapts to training dynamics without
making any prior assumptions on the scale and the temporal evolution of
gradient norms. At its core, it leverages z-score-based anomaly detection to
identify and mitigate large gradient spikes, preventing malignant loss spikes
while not interfering with convergence otherwise. Our code is available at:
https://github.com/bluorion-com/ZClip.
GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image
  Generation
Authors: name: Zhiyuan Yan, name: Junyan Ye, name: Weijia Li, name: Zilong Huang, name: Shenghai Yuan, name: Xiangyang He, name: Kaiqing Lin, name: Jun He, name: Conghui He, name: Li Yuan
Published: 2025-04-03 17:23:16

The recent breakthroughs in OpenAI's GPT4o model have demonstrated
surprisingly good capabilities in image generation and editing, resulting in
significant excitement in the community. This technical report presents the
first-look evaluation benchmark (named GPT-ImgEval), quantitatively and
qualitatively diagnosing GPT-4o's performance across three critical dimensions:
(1) generation quality, (2) editing proficiency, and (3) world
knowledge-informed semantic synthesis. Across all three tasks, GPT-4o
demonstrates strong performance, significantly surpassing existing methods in
both image generation control and output quality, while also showcasing
exceptional knowledge reasoning capabilities. Furthermore, based on the
GPT-4o's generated data, we propose a classification-model-based approach to
investigate the underlying architecture of GPT-4o, where our empirical results
suggest the model consists of an auto-regressive (AR) combined with a
diffusion-based head for image decoding, rather than the VAR-like
architectures. We also provide a complete speculation on GPT-4o's overall
architecture. In addition, we conduct a series of analyses to identify and
visualize GPT-4o's specific limitations and the synthetic artifacts commonly
observed in its image generation. We also present a comparative study of
multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the
safety implications of GPT-4o's outputs, particularly their detectability by
existing image forensic models. We hope that our work can offer valuable
insight and provide a reliable benchmark to guide future research, foster
reproducibility, and accelerate innovation in the field of image generation and
beyond. The codes and datasets used for evaluating GPT-4o can be found at
https://github.com/PicoTrex/GPT-ImgEval.
Audio-visual Controlled Video Diffusion with Masked Selective State
  Spaces Modeling for Natural Talking Head Generation
Authors: name: Fa-Ting Hong, name: Zunnan Xu, name: Zixiang Zhou, name: Jun Zhou, name: Xiu Li, name: Qin Lin, name: Qinglin Lu, name: Dan Xu
Published: 2025-04-03 12:44:41

Talking head synthesis is vital for virtual avatars and human-computer
interaction. However, most existing methods are typically limited to accepting
control from a single primary modality, restricting their practical utility. To
this end, we introduce ACTalker, an end-to-end video diffusion
framework that supports both multi-signals control and single-signal control
for talking head video generation. For multiple control, we design a parallel
mamba structure with multiple branches, each utilizing a separate driving
signal to control specific facial regions. A gate mechanism is applied across
all branches, providing flexible control over video generation. To ensure
natural coordination of the controlled video both temporally and spatially, we
employ the mamba structure, which enables driving signals to manipulate feature
tokens across both dimensions in each branch. Additionally, we introduce a
mask-drop strategy that allows each driving signal to independently control its
corresponding facial region within the mamba structure, preventing control
conflicts. Experimental results demonstrate that our method produces
natural-looking facial videos driven by diverse signals and that the mamba
layer seamlessly integrates multiple driving modalities without conflict.
Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual
  Editing
Authors: name: Xiangyu Zhao, name: Peiyuan Zhang, name: Kexian Tang, name: Hao Li, name: Zicheng Zhang, name: Guangtao Zhai, name: Junchi Yan, name: Hua Yang, name: Xue Yang, name: Haodong Duan
Published: 2025-04-03 17:59:56

Large Multi-modality Models (LMMs) have made significant progress in visual
understanding and generation, but they still face challenges in General Visual
Editing, particularly in following complex instructions, preserving appearance
consistency, and supporting flexible input formats. To address this gap, we
introduce RISEBench, the first benchmark for evaluating Reasoning-Informed
viSual Editing (RISE). RISEBench focuses on four key reasoning types: Temporal,
Causal, Spatial, and Logical Reasoning. We curate high-quality test cases for
each category and propose an evaluation framework that assesses Instruction
Reasoning, Appearance Consistency, and Visual Plausibility with both human
judges and an LMM-as-a-judge approach. Our experiments reveal that while
GPT-4o-Native significantly outperforms other open-source and proprietary
models, even this state-of-the-art system struggles with logical reasoning
tasks, highlighting an area that remains underexplored. As an initial effort,
RISEBench aims to provide foundational insights into reasoning-aware visual
editing and to catalyze future research. Though still in its early stages, we
are committed to continuously expanding and refining the benchmark to support
more comprehensive, reliable, and scalable evaluations of next-generation
multimodal systems. Our code and data will be released at
https://github.com/PhoenixZ810/RISEBench.
Efficient Model Selection for Time Series Forecasting via LLMs
Authors: name: Wang Wei, name: Tiankai Yang, name: Hongjie Chen, name: Ryan A. Rossi, name: Yue Zhao, name: Franck Dernoncourt, name: Hoda Eldardiry
Published: 2025-04-02 20:33:27

Model selection is a critical step in time series forecasting, traditionally
requiring extensive performance evaluations across various datasets.
Meta-learning approaches aim to automate this process, but they typically
depend on pre-constructed performance matrices, which are costly to build. In
this work, we propose to leverage Large Language Models (LLMs) as a lightweight
alternative for model selection. Our method eliminates the need for explicit
performance matrices by utilizing the inherent knowledge and reasoning
capabilities of LLMs. Through extensive experiments with LLaMA, GPT and Gemini,
we demonstrate that our approach outperforms traditional meta-learning
techniques and heuristic baselines, while significantly reducing computational
overhead. These findings underscore the potential of LLMs in efficient model
selection for time series forecasting.
Rethinking RL Scaling for Vision Language Models: A Transparent,
  From-Scratch Framework and Comprehensive Evaluation Scheme
Authors: name: Yan Ma, name: Steffi Chern, name: Xuyang Shen, name: Yiran Zhong, name: Pengfei Liu
Published: 2025-04-03 13:53:28

Reinforcement learning (RL) has recently shown strong potential in improving
the reasoning capabilities of large language models and is now being actively
extended to vision-language models (VLMs). However, existing RL applications in
VLMs often rely on heavily engineered frameworks that hinder reproducibility
and accessibility, while lacking standardized evaluation protocols, making it
difficult to compare results or interpret training dynamics. This work
introduces a transparent, from-scratch framework for RL in VLMs, offering a
minimal yet functional four-step pipeline validated across multiple models and
datasets. In addition, a standardized evaluation scheme is proposed to assess
training dynamics and reflective behaviors. Extensive experiments on visual
reasoning tasks uncover key empirical findings: response length is sensitive to
random seeds, reflection correlates with output length, and RL consistently
outperforms supervised fine-tuning (SFT) in generalization, even with
high-quality data. These findings, together with the proposed framework, aim to
establish a reproducible baseline and support broader engagement in RL-based
VLM research.