Orionnnn/hub-stats huggingface dataset:
API Agents vs. GUI Agents: Divergence and Convergence
Authors: name: Chaoyun Zhang, name: Shilin He, name: Liqun Li, name: Si Qin, name: Yu Kang, name: Qingwei Lin, name: Dongmei Zhang
Published: 2025-03-14 04:26:21

Large language models (LLMs) have evolved beyond simple text generation to
power software agents that directly translate natural language commands into
tangible actions. While API-based LLM agents initially rose to prominence for
their robust automation capabilities and seamless integration with programmatic
endpoints, recent progress in multimodal LLM research has enabled GUI-based LLM
agents that interact with graphical user interfaces in a human-like manner.
Although these two paradigms share the goal of enabling LLM-driven task
automation, they diverge significantly in architectural complexity, development
workflows, and user interaction models.
  This paper presents the first comprehensive comparative study of API-based
and GUI-based LLM agents, systematically analyzing their divergence and
potential convergence. We examine key dimensions and highlight scenarios in
which hybrid approaches can harness their complementary strengths. By proposing
clear decision criteria and illustrating practical use cases, we aim to guide
practitioners and researchers in selecting, combining, or transitioning between
these paradigms. Ultimately, we indicate that continuing innovations in
LLM-based automation are poised to blur the lines between API- and GUI-driven
agents, paving the way for more flexible, adaptive solutions in a wide range of
real-world applications.
Exploring the Vulnerabilities of Federated Learning: A Deep Dive into
  Gradient Inversion Attacks
Authors: name: Pengxin Guo, name: Runxi Wang, name: Shuang Zeng, name: Jinjing Zhu, name: Haoning Jiang, name: Yanran Wang, name: Yuyin Zhou, name: Feifei Wang, name: Hui Xiong, name: Liangqiong Qu
Published: 2025-03-13 08:08:44

Federated Learning (FL) has emerged as a promising privacy-preserving
collaborative model training paradigm without sharing raw data. However, recent
studies have revealed that private information can still be leaked through
shared gradient information and attacked by Gradient Inversion Attacks (GIA).
While many GIA methods have been proposed, a detailed analysis, evaluation, and
summary of these methods are still lacking. Although various survey papers
summarize existing privacy attacks in FL, few studies have conducted extensive
experiments to unveil the effectiveness of GIA and their associated limiting
factors in this context. To fill this gap, we first undertake a systematic
review of GIA and categorize existing methods into three types, i.e.,
optimization-based GIA (OP-GIA), generation-based GIA
(GEN-GIA), and analytics-based GIA (ANA-GIA). Then, we comprehensively
analyze and evaluate the three types of GIA in FL, providing insights into the
factors that influence their performance, practicality, and potential threats.
Our findings indicate that OP-GIA is the most practical attack setting despite
its unsatisfactory performance, while GEN-GIA has many dependencies and ANA-GIA
is easily detectable, making them both impractical. Finally, we offer a
three-stage defense pipeline to users when designing FL frameworks and
protocols for better privacy protection and share some future research
directions from the perspectives of attackers and defenders that we believe
should be pursued. We hope that our study can help researchers design more
robust FL frameworks to defend against these attacks.
PoseLess: Depth-Free Vision-to-Joint Control via Direct Image Mapping
  with VLM
Authors: name: Alan Dao, name: Dinh Bach Vu, name: Tuan Le Duc Anh, name: Bui Quang Huy
Published: 2025-03-10 09:34:05

This paper introduces PoseLess, a novel framework for robot hand control that
eliminates the need for explicit pose estimation by directly mapping 2D images
to joint angles using projected representations. Our approach leverages
synthetic training data generated through randomized joint configurations,
enabling zero-shot generalization to real-world scenarios and cross-morphology
transfer from robotic to human hands. By projecting visual inputs and employing
a transformer-based decoder, PoseLess achieves robust, low-latency control
while addressing challenges such as depth ambiguity and data scarcity.
Experimental results demonstrate competitive performance in joint angle
prediction accuracy without relying on any human-labelled dataset.
A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90%
  Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1
Authors: name: Zhaoyi Li, name: Xiaohan Zhao, name: Dong-Dong Wu, name: Jiacheng Cui, name: Zhiqiang Shen
Published: 2025-03-13 17:59:55

Despite promising performance on open-source large vision-language models
(LVLMs), transfer-based targeted attacks often fail against black-box
commercial LVLMs. Analyzing failed adversarial perturbations reveals that the
learned perturbations typically originate from a uniform distribution and lack
clear semantic details, resulting in unintended responses. This critical
absence of semantic information leads commercial LVLMs to either ignore the
perturbation entirely or misinterpret its embedded semantics, thereby causing
the attack to fail. To overcome these issues, we notice that identifying core
semantic objects is a key objective for models trained with various datasets
and methodologies. This insight motivates our approach that refines semantic
clarity by encoding explicit semantic details within local regions, thus
ensuring interoperability and capturing finer-grained features, and by
concentrating modifications on semantically rich areas rather than applying
them uniformly. To achieve this, we propose a simple yet highly effective
solution: at each optimization step, the adversarial image is cropped randomly
by a controlled aspect ratio and scale, resized, and then aligned with the
target image in the embedding space. Experimental results confirm our
hypothesis. Our adversarial examples crafted with local-aggregated
perturbations focused on crucial regions exhibit surprisingly good
transferability to commercial LVLMs, including GPT-4.5, GPT-4o,
Gemini-2.0-flash, Claude-3.5-sonnet, Claude-3.7-sonnet, and even reasoning
models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach
achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly
outperforming all prior state-of-the-art attack methods. Our optimized
adversarial examples under different configurations and training code are
available at https://github.com/VILA-Lab/M-Attack.
Studying Classifier(-Free) Guidance From a Classifier-Centric
  Perspective
Authors: name: Xiaoming Zhao, name: Alexander G. Schwing
Published: 2025-03-13 17:59:59

Classifier-free guidance has become a staple for conditional generation with
denoising diffusion models. However, a comprehensive understanding of
classifier-free guidance is still missing. In this work, we carry out an
empirical study to provide a fresh perspective on classifier-free guidance.
Concretely, instead of solely focusing on classifier-free guidance, we trace
back to the root, i.e., classifier guidance, pinpoint the key assumption for
the derivation, and conduct a systematic study to understand the role of the
classifier. We find that both classifier guidance and classifier-free guidance
achieve conditional generation by pushing the denoising diffusion trajectories
away from decision boundaries, i.e., areas where conditional information is
usually entangled and is hard to learn. Based on this classifier-centric
understanding, we propose a generic postprocessing step built upon
flow-matching to shrink the gap between the learned distribution for a
pre-trained denoising diffusion model and the real data distribution, majorly
around the decision boundaries. Experiments on various datasets verify the
effectiveness of the proposed approach.
Communication-Efficient Language Model Training Scales Reliably and
  Robustly: Scaling Laws for DiLoCo
Authors: name: Zachary Charles, name: Gabriel Teston, name: Lucio Dery, name: Keith Rush, name: Nova Fallen, name: Zachary Garrett, name: Arthur Szlam, name: Arthur Douillard
Published: 2025-03-12 20:04:38

As we scale to more massive machine learning models, the frequent
synchronization demands inherent in data-parallel approaches create significant
slowdowns, posing a critical challenge to further scaling. Recent work develops
an approach (DiLoCo) that relaxes synchronization demands without compromising
model quality. However, these works do not carefully analyze how DiLoCo's
behavior changes with model size. In this work, we study the scaling law
behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on
how algorithmic factors, including number of model replicas, hyperparameters,
and token budget affect training in ways that can be accurately predicted via
scaling laws. We find that DiLoCo scales both predictably and robustly with
model size. When well-tuned, DiLoCo scales better than data-parallel training
with model size, and can outperform data-parallel training even at small model
sizes. Our results showcase a more general set of benefits of DiLoCo than
previously documented, including increased optimal batch sizes, improved
downstream generalization with scale, and improved evaluation loss for a fixed
token budget.
Piece it Together: Part-Based Concepting with IP-Priors
Authors: name: Elad Richardson, name: Kfir Goldberg, name: Yuval Alaluf, name: Daniel Cohen-Or
Published: 2025-03-13 13:46:10

Advanced generative models excel at synthesizing images but often rely on
text-based conditioning. Visual designers, however, often work beyond language,
directly drawing inspiration from existing visual elements. In many cases,
these elements represent only fragments of a potential concept-such as an
uniquely structured wing, or a specific hairstyle-serving as inspiration for
the artist to explore how they can come together creatively into a coherent
whole. Recognizing this need, we introduce a generative framework that
seamlessly integrates a partial set of user-provided visual components into a
coherent composition while simultaneously sampling the missing parts needed to
generate a plausible and complete concept. Our approach builds on a strong and
underexplored representation space, extracted from IP-Adapter+, on which we
train IP-Prior, a lightweight flow-matching model that synthesizes coherent
compositions based on domain-specific priors, enabling diverse and
context-aware generations. Additionally, we present a LoRA-based fine-tuning
strategy that significantly improves prompt adherence in IP-Adapter+ for a
given task, addressing its common trade-off between reconstruction quality and
prompt adherence.
MinorBench: A hand-built benchmark for content-based risks for children
Authors: name: Shaun Khoo, name: Gabriel Chua, name: Rachel Shong
Published: 2025-03-13 10:34:43

Large Language Models (LLMs) are rapidly entering children's lives - through
parent-driven adoption, schools, and peer networks - yet current AI ethics and
safety research do not adequately address content-related risks specific to
minors. In this paper, we highlight these gaps with a real-world case study of
an LLM-based chatbot deployed in a middle school setting, revealing how
students used and sometimes misused the system. Building on these findings, we
propose a new taxonomy of content-based risks for minors and introduce
MinorBench, an open-source benchmark designed to evaluate LLMs on their ability
to refuse unsafe or inappropriate queries from children. We evaluate six
prominent LLMs under different system prompts, demonstrating substantial
variability in their child-safety compliance. Our results inform practical
steps for more robust, child-focused safety mechanisms and underscore the
urgency of tailoring AI systems to safeguard young users.
Discovering Influential Neuron Path in Vision Transformers
Authors: name: Yifan Wang, name: Yifei Liu, name: Yingdong Shi, name: Changming Li, name: Anqi Pang, name: Sibei Yang, name: Jingyi Yu, name: Kan Ren
Published: 2025-03-12 04:10:46

Vision Transformer models exhibit immense power yet remain opaque to human
understanding, posing challenges and risks for practical applications. While
prior research has attempted to demystify these models through input
attribution and neuron role analysis, there's been a notable gap in considering
layer-level information and the holistic path of information flow across
layers. In this paper, we investigate the significance of influential neuron
paths within vision Transformers, which is a path of neurons from the model
input to output that impacts the model inference most significantly. We first
propose a joint influence measure to assess the contribution of a set of
neurons to the model outcome. And we further provide a layer-progressive neuron
locating approach that efficiently selects the most influential neuron at each
layer trying to discover the crucial neuron path from input to output within
the target model. Our experiments demonstrate the superiority of our method
finding the most influential neuron path along which the information flows,
over the existing baseline solutions. Additionally, the neuron paths have
illustrated that vision Transformers exhibit some specific inner working
mechanism for processing the visual information within the same image category.
We further analyze the key effects of these neurons on the image classification
task, showcasing that the found neuron paths have already preserved the model
capability on downstream tasks, which may also shed some lights on real-world
applications like model pruning. The project website including implementation
code is available at https://foundation-model-research.github.io/NeuronPath/.
OmniPaint: Mastering Object-Oriented Editing via Disentangled
  Insertion-Removal Inpainting
Authors: name: Yongsheng Yu, name: Ziyun Zeng, name: Haitian Zheng, name: Jiebo Luo
Published: 2025-03-11 17:55:27

Diffusion-based generative models have revolutionized object-oriented image
editing, yet their deployment in realistic object removal and insertion remains
hampered by challenges such as the intricate interplay of physical effects and
insufficient paired training data. In this work, we introduce OmniPaint, a
unified framework that re-conceptualizes object removal and insertion as
interdependent processes rather than isolated tasks. Leveraging a pre-trained
diffusion prior along with a progressive training pipeline comprising initial
paired sample optimization and subsequent large-scale unpaired refinement via
CycleFlow, OmniPaint achieves precise foreground elimination and seamless
object insertion while faithfully preserving scene geometry and intrinsic
properties. Furthermore, our novel CFD metric offers a robust, reference-free
evaluation of context consistency and object hallucination, establishing a new
benchmark for high-fidelity image editing. Project page:
https://yeates.github.io/OmniPaint-Page/
TruthPrInt: Mitigating LVLM Object Hallucination Via Latent
  Truthful-Guided Pre-Intervention
Authors: name: Jinhao Duan, name: Fei Kong, name: Hao Cheng, name: James Diffenderfer, name: Bhavya Kailkhura, name: Lichao Sun, name: Xiaofeng Zhu, name: Xiaoshuang Shi, name: Kaidi Xu
Published: 2025-03-13 17:46:06

Object Hallucination (OH) has been acknowledged as one of the major
trustworthy challenges in Large Vision-Language Models (LVLMs). Recent
advancements in Large Language Models (LLMs) indicate that internal states,
such as hidden states, encode the "overall truthfulness" of generated
responses. However, it remains under-explored how internal states in LVLMs
function and whether they could serve as "per-token" hallucination indicators,
which is essential for mitigating OH. In this paper, we first conduct an
in-depth exploration of LVLM internal states in relation to OH issues and
discover that (1) LVLM internal states are high-specificity per-token
indicators of hallucination behaviors. Moreover, (2) different LVLMs encode
universal patterns of hallucinations in common latent subspaces, indicating
that there exist "generic truthful directions" shared by various LVLMs. Based
on these discoveries, we propose Truthful-Guided Pre-Intervention (TruthPrInt)
that first learns the truthful direction of LVLM decoding and then applies
truthful-guided inference-time intervention during LVLM decoding. We further
propose ComnHallu to enhance both cross-LVLM and cross-data hallucination
detection transferability by constructing and aligning hallucination latent
subspaces. We evaluate TruthPrInt in extensive experimental settings, including
in-domain and out-of-domain scenarios, over popular LVLMs and OH benchmarks.
Experimental results indicate that TruthPrInt significantly outperforms
state-of-the-art methods. Codes will be available at
https://github.com/jinhaoduan/TruthPrInt.
ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style
  Transfer
Authors: name: Bolin Chen, name: Baoquan Zhao, name: Haoran Xie, name: Yi Cai, name: Qing Li, name: Xudong Mao
Published: 2025-03-13 17:55:58

Style transfer involves transferring the style from a reference image to the
content of a target image. Recent advancements in LoRA-based (Low-Rank
Adaptation) methods have shown promise in effectively capturing the style of a
single image. However, these approaches still face significant challenges such
as content inconsistency, style misalignment, and content leakage. In this
paper, we comprehensively analyze the limitations of the standard diffusion
parameterization, which learns to predict noise, in the context of style
transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method
that enhances both content and style consistency by optimizing the LoRA weights
to predict the original image rather than noise. We also propose a two-step
training strategy that decouples the learning of content and style from the
reference image. To effectively capture both the global structure and local
details of the content image, we introduce a stepwise loss transition strategy.
Additionally, we present an inference guidance method that enables continuous
control over content and style strengths during inference. Through both
qualitative and quantitative evaluations, our method demonstrates significant
improvements in content and style consistency while effectively reducing
content leakage.
On the Limitations of Vision-Language Models in Understanding Image
  Transforms
Authors: name: Ahmad Mustafa Anis, name: Hasnain Ali, name: Saquib Sarfraz
Published: 2025-03-12 20:58:16

Vision Language Models (VLMs) have demonstrated significant potential in
various downstream tasks, including Image/Video Generation, Visual Question
Answering, Multimodal Chatbots, and Video Understanding. However, these models
often struggle with basic image transformations. This paper investigates the
image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by
Google. Our findings reveal that these models lack comprehension of multiple
image-level augmentations. To facilitate this study, we created an augmented
version of the Flickr8k dataset, pairing each image with a detailed description
of the applied transformation. We further explore how this deficiency impacts
downstream tasks, particularly in image editing, and evaluate the performance
of state-of-the-art Image2Image models on simple transformations.
PerCoV2: Improved Ultra-Low Bit-Rate Perceptual Image Compression with
  Implicit Hierarchical Masked Image Modeling
Authors: name: Nikolai Körber, name: Eduard Kromer, name: Andreas Siebert, name: Sascha Hauke, name: Daniel Mueller-Gritschneder, name: Björn Schuller
Published: 2025-03-12 13:14:51

We introduce PerCoV2, a novel and open ultra-low bit-rate perceptual image
compression system designed for bandwidth- and storage-constrained
applications. Building upon prior work by Careil et al., PerCoV2 extends the
original formulation to the Stable Diffusion 3 ecosystem and enhances entropy
coding efficiency by explicitly modeling the discrete hyper-latent image
distribution. To this end, we conduct a comprehensive comparison of recent
autoregressive methods (VAR and MaskGIT) for entropy modeling and evaluate our
approach on the large-scale MSCOCO-30k benchmark. Compared to previous work,
PerCoV2 (i) achieves higher image fidelity at even lower bit-rates while
maintaining competitive perceptual quality, (ii) features a hybrid generation
mode for further bit-rate savings, and (iii) is built solely on public
components. Code and trained models will be released at
https://github.com/Nikolai10/PerCoV2.
VisualPRM: An Effective Process Reward Model for Multimodal Reasoning
Authors: name: Weiyun Wang, name: Zhangwei Gao, name: Lianjie Chen, name: Zhe Chen, name: Jinguo Zhu, name: Xiangyu Zhao, name: Yangzhou Liu, name: Yue Cao, name: Shenglong Ye, name: Xizhou Zhu, name: Lewei Lu, name: Haodong Duan, name: Yu Qiao, name: Jifeng Dai, name: Wenhai Wang
Published: 2025-03-13 12:03:37

We introduce VisualPRM, an advanced multimodal Process Reward Model (PRM)
with 8B parameters, which improves the reasoning abilities of existing
Multimodal Large Language Models (MLLMs) across different model scales and
families with Best-of-N (BoN) evaluation strategies. Specifically, our model
improves the reasoning performance of three types of MLLMs and four different
model scales. Even when applied to the highly capable InternVL2.5-78B, it
achieves a 5.9-point improvement across seven multimodal reasoning benchmarks.
Experimental results show that our model exhibits superior performance compared
to Outcome Reward Models and Self-Consistency during BoN evaluation. To
facilitate the training of multimodal PRMs, we construct a multimodal process
supervision dataset VisualPRM400K using an automated data pipeline. For the
evaluation of multimodal PRMs, we propose VisualProcessBench, a benchmark with
human-annotated step-wise correctness labels, to measure the abilities of PRMs
to detect erroneous steps in multimodal reasoning tasks. We hope that our work
can inspire more future research and contribute to the development of MLLMs.
Our model, data, and benchmark are released in
https://internvl.github.io/blog/2025-03-13-VisualPRM/.
CoRe^2: Collect, Reflect and Refine to Generate Better and Faster
Authors: name: Shitong Shao, name: Zikai Zhou, name: Dian Xie, name: Yuetong Fang, name: Tian Ye, name: Lichen Bai, name: Zeke Xie
Published: 2025-03-12 15:15:25

Making text-to-image (T2I) generative model sample both fast and well
represents a promising research direction. Previous studies have typically
focused on either enhancing the visual quality of synthesized images at the
expense of sampling efficiency or dramatically accelerating sampling without
improving the base model's generative capacity. Moreover, nearly all inference
methods have not been able to ensure stable performance simultaneously on both
diffusion models (DMs) and visual autoregressive models (ARMs). In this paper,
we introduce a novel plug-and-play inference paradigm, CoRe^2, which comprises
three subprocesses: Collect, Reflect, and Refine. CoRe^2 first collects
classifier-free guidance (CFG) trajectories, and then use collected data to
train a weak model that reflects the easy-to-learn contents while reducing
number of function evaluations during inference by half. Subsequently, CoRe^2
employs weak-to-strong guidance to refine the conditional output, thereby
improving the model's capacity to generate high-frequency and realistic
content, which is difficult for the base model to capture. To the best of our
knowledge, CoRe^2 is the first to demonstrate both efficiency and effectiveness
across a wide range of DMs, including SDXL, SD3.5, and FLUX, as well as ARMs
like LlamaGen. It has exhibited significant performance improvements on HPD v2,
Pick-of-Pic, Drawbench, GenEval, and T2I-Compbench. Furthermore, CoRe^2 can be
seamlessly integrated with the state-of-the-art Z-Sampling, outperforming it by
0.3 and 0.16 on PickScore and AES, while achieving 5.64s time saving using
SD3.5.Code is released at https://github.com/xie-lab-ml/CoRe/tree/main.