Orionnnn/hub-stats huggingface dataset:
NetPress: Dynamically Generated LLM Benchmarks for Network Applications
Authors: name: Yajie Zhou, name: Jiajun Ruan, name: Eric S. Wang, name: Sadjad Fouladi, name: Francis Y. Yan, name: Kevin Hsieh, name: Zaoxing Liu
Published: 2025-06-03 14:04:22

Despite growing interest in domain-specific benchmarking of large language
models (LLMs) and agents, current evaluations remain limited to static,
small-scale datasets, especially in high-stakes tasks like network operations
that demand reliability for deployments. We present NetPress, an automated
benchmark generation framework for evaluating LLM agents in network
applications. NetPress introduces a unified abstraction with state and action,
enabling dynamic generation of diverse query sets along with corresponding
ground truths. At runtime, users can specify benchmark configurations to
generate millions of queries on the fly. In addition to dynamic benchmark
construction, NetPress integrates with network emulators to provide realistic
environment feedback, supporting comprehensive evaluation across correctness,
safety, and latency. We instantiate NetPress on three representative
applications, revealing interesting fine-grained differences in agent behavior
that static, correctness-only benchmarks often miss. NetPress moves LLM
evaluation toward realistic, scalable testing in infrastructure-centric
domains, helping close the gap between benchmark performance and real-world
deployment readiness. Code is available at
https://github.com/Froot-NetSys/NetPress.
Dynamic View Synthesis as an Inverse Problem
Authors: name: Hidir Yesiltepe, name: Pinar Yanardag
Published: 2025-06-09 17:59:47

In this work, we address dynamic view synthesis from monocular videos as an
inverse problem in a training-free setting. By redesigning the noise
initialization phase of a pre-trained video diffusion model, we enable
high-fidelity dynamic view synthesis without any weight updates or auxiliary
modules. We begin by identifying a fundamental obstacle to deterministic
inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and
resolve it by introducing a novel noise representation, termed K-order
Recursive Noise Representation. We derive a closed form expression for this
representation, enabling precise and efficient alignment between the
VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions
resulting from camera motion, we introduce Stochastic Latent Modulation, which
performs visibility aware sampling over the latent space to complete occluded
regions. Comprehensive experiments demonstrate that dynamic view synthesis can
be effectively performed through structured latent manipulation in the noise
initialization phase.
Hidden in Plain Sight: Probing Implicit Reasoning in Multimodal Language
  Models
Authors: name: Qianqi Yan, name: Hongquan Li, name: Shan Jiang, name: Yang Zhao, name: Xinze Guan, name: Ching-Chen Kuo, name: Xin Eric Wang
Published: 2025-05-30 21:47:28

Multimodal large language models (MLLMs) are increasingly deployed in
open-ended, real-world environments where inputs are messy, underspecified, and
not always trustworthy. Unlike curated benchmarks, these settings frequently
involve instructions that refer to missing objects or contradictory facts, rely
on ambiguous references, or request infeasible actions. In such cases, success
hinges not on task execution alone, but on a model's ability to detect when
something is silently wrong. This paper presents a systematic analysis of how
current MLLMs handle such implicit reasoning scenarios: cases where the flaw is
not explicitly stated but must be inferred from context. Using a curated
diagnostic suite spanning four categories of real-world failure modes, we
evaluate six MLLMs, including o3 and GPT-4o, and find that models frequently
fail to surface hidden issues, even when they possess the necessary perceptual
and reasoning skills. Explicit prompting reveals that the underlying
capabilities exist but are often suppressed in favor of user compliance. We
further show that simple inference-time interventions, such as cautious persona
prompting and, in particular, requiring a clarifying question, can dramatically
recover performance. Our findings highlight a persistent gap between reasoning
competence and behavioral compliance in current MLLMs and suggest practical
strategies for making these models more trustworthy in underconstrained
environments.
SAFEFLOW: A Principled Protocol for Trustworthy and Transactional
  Autonomous Agent Systems
Authors: name: Peiran Li, name: Xinkai Zou, name: Zhuohang Wu, name: Ruifeng Li, name: Shuo Xing, name: Hanwen Zheng, name: Zhikai Hu, name: Yuping Wang, name: Haoxi Li, name: Qin Yuan, name: Yingmo Zhang, name: Zhengzhong Tu
Published: 2025-06-09 09:04:37

Recent advances in large language models (LLMs) and vision-language models
(VLMs) have enabled powerful autonomous agents capable of complex reasoning and
multi-modal tool use. Despite their growing capabilities, today's agent
frameworks remain fragile, lacking principled mechanisms for secure information
flow, reliability, and multi-agent coordination. In this work, we introduce
SAFEFLOW, a new protocol-level framework for building trustworthy LLM/VLM-based
agents. SAFEFLOW enforces fine-grained information flow control (IFC),
precisely tracking provenance, integrity, and confidentiality of all the data
exchanged between agents, tools, users, and environments. By constraining LLM
reasoning to respect these security labels, SAFEFLOW prevents untrusted or
adversarial inputs from contaminating high-integrity decisions. To ensure
robustness in concurrent multi-agent settings, SAFEFLOW introduces
transactional execution, conflict resolution, and secure scheduling over shared
state, preserving global consistency across agents. We further introduce
mechanisms, including write-ahead logging, rollback, and secure caches, that
further enhance resilience against runtime errors and policy violations. To
validate the performances, we built SAFEFLOWBENCH, a comprehensive benchmark
suite designed to evaluate agent reliability under adversarial, noisy, and
concurrent operational conditions. Extensive experiments demonstrate that
agents built with SAFEFLOW maintain impressive task performance and security
guarantees even in hostile environments, substantially outperforming
state-of-the-art. Together, SAFEFLOW and SAFEFLOWBENCH lay the groundwork for
principled, robust, and secure agent ecosystems, advancing the frontier of
reliable autonomy.
Agents of Change: Self-Evolving LLM Agents for Strategic Planning
Authors: name: Nikolas Belle, name: Dakota Barnes, name: Alfonso Amayuelas, name: Ivan Bercovich, name: Xin Eric Wang, name: William Wang
Published: 2025-06-05 05:45:24

Recent advances in LLMs have enabled their use as autonomous agents across a
range of tasks, yet they continue to struggle with formulating and adhering to
coherent long-term strategies. In this paper, we investigate whether LLM agents
can self-improve when placed in environments that explicitly challenge their
strategic planning abilities. Using the board game Settlers of Catan, accessed
through the open-source Catanatron framework, we benchmark a progression of
LLM-based agents, from a simple game-playing agent to systems capable of
autonomously rewriting their own prompts and their player agent's code. We
introduce a multi-agent architecture in which specialized roles (Analyzer,
Researcher, Coder, and Player) collaborate to iteratively analyze gameplay,
research new strategies, and modify the agent's logic or prompt. By comparing
manually crafted agents to those evolved entirely by LLMs, we evaluate how
effectively these systems can diagnose failure and adapt over time. Our results
show that self-evolving agents, particularly when powered by models like Claude
3.7 and GPT-4o, outperform static baselines by autonomously adopting their
strategies, passing along sample behavior to game-playing agents, and
demonstrating adaptive reasoning over multiple iterations.
τ^2-Bench: Evaluating Conversational Agents in a Dual-Control
  Environment
Authors: name: Victor Barres, name: Honghua Dong, name: Soham Ray, name: Xujie Si, name: Karthik Narasimhan
Published: 2025-06-09 17:52:18

Existing benchmarks for conversational AI agents simulate single-control
environments, where only the AI agent can use tools to interact with the world,
while the user remains a passive information provider. This differs from
real-world scenarios like technical support, where users need to actively
participate in modifying the state of the (shared) world. In order to address
this gap, we introduce tau^2-bench, with four key contributions:
  1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both
agent and user make use of tools to act in a shared, dynamic environment that
tests both agent coordination and communication,
  2) A compositional task generator that programmatically creates diverse,
verifiable tasks from atomic components, ensuring domain coverage and
controlled complexity,
  3) A reliable user simulator tightly coupled with the environment, whose
behavior is constrained by tools and observable states, improving simulation
fidelity,
  4) Fine-grained analysis of agent performance through multiple ablations
including separating errors arising from reasoning vs
communication/coordination.
  In particular, our experiments show significant performance drops when agents
shift from no-user to dual-control, highlighting the challenges of guiding
users. Overall, tau^2-bench provides a controlled testbed for agents that
must both reason effectively and guide user actions.
Model Immunization from a Condition Number Perspective
Authors: name: Amber Yijia Zheng, name: Cedar Site Bai, name: Brian Bullins, name: Raymond A. Yeh
Published: 2025-05-29 17:59:48

Model immunization aims to pre-train models that are difficult to fine-tune
on harmful tasks while retaining their utility on other non-harmful tasks.
Though prior work has shown empirical evidence for immunizing text-to-image
models, the key understanding of when immunization is possible and a precise
definition of an immunized model remain unclear. In this work, we propose a
framework, based on the condition number of a Hessian matrix, to analyze model
immunization for linear models. Building on this framework, we design an
algorithm with regularization terms to control the resulting condition numbers
after pre-training. Empirical results on linear models and non-linear deep-nets
demonstrate the effectiveness of the proposed algorithm on model immunization.
The code is available at
https://github.com/amberyzheng/model-immunization-cond-num.
Self-Adapting Improvement Loops for Robotic Learning
Authors: name: Calvin Luo, name: Zilai Zeng, name: Mingxi Jia, name: Yilun Du, name: Chen Sun
Published: 2025-06-07 04:34:37

Video generative models trained on expert demonstrations have been utilized
as performant text-conditioned visual planners for solving robotic tasks.
However, generalization to unseen tasks remains a challenge. Whereas improved
generalization may be facilitated by leveraging learned prior knowledge from
additional pre-collected offline data sources, such as web-scale video
datasets, in the era of experience we aim to design agents that can
continuously improve in an online manner from self-collected behaviors. In this
work we thus propose the Self-Adapting Improvement Loop (SAIL), where an
in-domain video model iteratively updates itself on self-produced trajectories,
collected through adaptation with an internet-scale pretrained video model, and
steadily improves its performance for a specified task of interest. We apply
SAIL to a diverse suite of MetaWorld tasks, as well as two manipulation tasks
on a real robot arm, and find that performance improvements continuously emerge
over multiple iterations for novel tasks initially unseen during original
in-domain video model training. Furthermore, we discover that SAIL is
surprisingly robust regarding if and how the self-collected experience is
filtered, and the quality of the initial in-domain demonstrations. Through
adaptation with summarized internet-scale data, and learning through online
experience, we thus demonstrate a way to iteratively bootstrap a
high-performance video model for solving novel robotic tasks through
self-improvement.
SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward
  Models in LLMs
Authors: name: Michael J Ryan, name: Omar Shaikh, name: Aditri Bhagirath, name: Daniel Frees, name: William Held, name: Diyi Yang
Published: 2025-06-05 21:23:16

Recent calls for pluralistic alignment of Large Language Models (LLMs)
encourage adapting models to diverse user preferences. However, most prior work
on personalized reward models heavily rely on additional identity information,
such as demographic details or a predefined set of preference categories. To
this end, we introduce SynthesizeMe, an approach to inducing synthetic user
personas from user interactions for personalized reward modeling. SynthesizeMe
first generates and verifies reasoning to explain user preferences, then
induces synthetic user personas from that reasoning, and finally filters to
informative prior user interactions in order to build personalized prompts for
a particular user. We show that using SynthesizeMe induced prompts improves
personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining
SynthesizeMe derived prompts with a reward model achieves top performance on
PersonalRewardBench: a new curation of user-stratified interactions with
chatbots collected from 854 users of Chatbot Arena and PRISM.
CyberV: Cybernetics for Test-time Scaling in Video Understanding
Authors: name: Jiahao Meng, name: Shuyang Sun, name: Yue Tan, name: Lu Qi, name: Yunhai Tong, name: Xiangtai Li, name: Longyin Wen
Published: 2025-06-09 17:45:18

Current Multimodal Large Language Models (MLLMs) may struggle with
understanding long or complex videos due to computational demands at test time,
lack of robustness, and limited accuracy, primarily stemming from their
feed-forward processing nature. These limitations could be more severe for
models with fewer parameters. To address these limitations, we propose a novel
framework inspired by cybernetic principles, redesigning video MLLMs as
adaptive systems capable of self-monitoring, self-correction, and dynamic
resource allocation during inference. Our approach, CyberV, introduces a
cybernetic loop consisting of an MLLM Inference System, a Sensor, and a
Controller. Specifically, the sensor monitors forward processes of the MLLM and
collects intermediate interpretations, such as attention drift, then the
controller determines when and how to trigger self-correction and generate
feedback to guide the next round. This test-time adaptive scaling framework
enhances frozen MLLMs without requiring retraining or additional components.
Experiments demonstrate significant improvements: CyberV boosts Qwen2.5-VL-7B
by 8.3% and InternVL3-8B by 5.5% on VideoMMMU, surpassing the competitive
proprietary model GPT-4o. When applied to Qwen2.5-VL-72B, it yields a 10.0%
improvement, achieving performance even comparable to human experts.
Furthermore, our method demonstrates consistent gains on general-purpose
benchmarks, such as VideoMME and WorldSense, highlighting its effectiveness and
generalization capabilities in making MLLMs more robust and accurate for
dynamic video understanding. The code is released at
https://github.com/marinero4972/CyberV.
What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge
  Conflict on Large Language Models
Authors: name: Kaiser Sun, name: Fan Bai, name: Mark Dredze
Published: 2025-06-06 19:20:23

Large language models frequently rely on both contextual input and parametric
knowledge to perform tasks. However, these sources can come into conflict,
especially when retrieved documents contradict the model's parametric
knowledge. We propose a diagnostic framework to systematically evaluate LLM
behavior under context-memory conflict, where the contextual information
diverges from their parametric beliefs. We construct diagnostic data that
elicit these conflicts and analyze model performance across multiple task
types. Our findings reveal that (1) knowledge conflict has minimal impact on
tasks that do not require knowledge utilization, (2) model performance is
consistently higher when contextual and parametric knowledge are aligned, (3)
models are unable to fully suppress their internal knowledge even when
instructed, and (4) providing rationales that explain the conflict increases
reliance on contexts. These insights raise concerns about the validity of
model-based evaluation and underscore the need to account for knowledge
conflict in the deployment of LLMs.
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form
  Generation Tasks with Structured Checklists
Authors: name: Jie Ruan, name: Inderjeet Nair, name: Shuyang Cao, name: Amy Liu, name: Sheza Munir, name: Micah Pollens-Dempsey, name: Tiffany Chiang, name: Lucy Kates, name: Nicholas David, name: Sihan Chen, name: Ruxin Yang, name: Yuqian Yang, name: Jasmine Gump, name: Tessa Bialek, name: Vivek Sankaran, name: Margo Schlanger, name: Lu Wang
Published: 2025-06-02 01:39:02

This paper introduces ExpertLongBench, an expert-level benchmark containing
11 tasks from 9 domains that reflect realistic expert workflows and
applications. Beyond question answering, the application-driven tasks in
ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and
strict adherence to domain-specific requirements. Notably, each task in
ExpertLongBench includes a rubric, designed or validated by domain experts, to
specify task requirements and guide output evaluation. Furthermore, we propose
CLEAR, an evaluation framework that supports accurate evaluation of long-form
model outputs in our benchmark. To achieve fine-grained, expert-aligned
evaluation, CLEAR derives checklists from both model outputs and references by
extracting information corresponding to items in the task-specific rubric.
Checklist items for model outputs are then compared with corresponding items
for reference outputs to assess their correctness, enabling grounded
evaluation. We benchmark 11 large language models (LLMs) and analyze components
in CLEAR, showing that (1) existing LLMs, with the top performer achieving only
a 26.8% F1 score, require significant improvement for expert-level tasks; (2)
models can generate content corresponding to the required aspects, though often
not accurately; and (3) accurate checklist extraction and comparison in CLEAR
can be achieved by open-weight models for more scalable and low-cost usage.
Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
Authors: name: Akash Gupta, name: Amos Storkey, name: Mirella Lapata
Published: 2025-06-07 19:37:22

Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to
perform new tasks with minimal supervision. However, ICL performance,
especially in smaller LMMs, is inconsistent and does not always improve
monotonically with increasing examples. We hypothesize that this occurs due to
the LMM being overwhelmed by additional information present in the image
embeddings, which is not required for the downstream task. To address this, we
propose a meta-learning approach that provides an alternative for inducing
few-shot capabilities in LMMs, using a fixed set of soft prompts that are
distilled from task-relevant image features and can be adapted at test time
using a few examples. To facilitate this distillation, we introduce an
attention-mapper module that can be easily integrated with the popular LLaVA
v1.5 architecture and is jointly learned with soft prompts, enabling task
adaptation in LMMs under low-data regimes with just a few gradient steps.
Evaluation on the VL-ICL Bench shows that our method consistently outperforms
ICL and related prompt-tuning approaches, even under image perturbations,
improving task induction and reasoning across visual question answering tasks.
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech
  Evaluation
Authors: name: Noy Sternlicht, name: Ariel Gera, name: Roy Bar-Haim, name: Tom Hope, name: Noam Slonim
Published: 2025-06-05 14:06:51

We introduce Debate Speech Evaluation as a novel and challenging benchmark
for assessing LLM judges. Evaluating debate speeches requires a deep
understanding of the speech at multiple levels, including argument strength and
relevance, the coherence and organization of the speech, the appropriateness of
its style and tone, and so on. This task involves a unique set of cognitive
abilities that have previously received limited attention in systematic LLM
benchmarking. To explore such skills, we leverage a dataset of over 600
meticulously annotated debate speeches and present the first in-depth analysis
of how state-of-the-art LLMs compare to human judges on this task. Our findings
reveal a nuanced picture: while larger models can approximate individual human
judgments in some respects, they differ substantially in their overall judgment
behavior. We also investigate the ability of frontier LLMs to generate
persuasive, opinionated speeches, showing that models may perform at a human
level on this task.
Play to Generalize: Learning to Reason Through Game Play
Authors: name: Yunfei Xie, name: Yinsong Ma, name: Shiyi Lan, name: Alan Yuille, name: Junfei Xiao, name: Chen Wei
Published: 2025-06-09 17:59:57

Developing generalizable reasoning capabilities in multimodal large language
models (MLLMs) remains challenging. Motivated by cognitive science literature
suggesting that gameplay promotes transferable cognitive skills, we propose a
novel post-training paradigm, Visual Game Learning, or ViGaL, where MLLMs
develop out-of-domain generalization of multimodal reasoning through playing
arcade-like games. Specifically, we show that post-training a 7B-parameter MLLM
via reinforcement learning (RL) on simple arcade-like games, e.g. Snake,
significantly enhances its downstream performance on multimodal math benchmarks
like MathVista, and on multi-discipline questions like MMMU, without seeing any
worked solutions, equations, or diagrams during RL, suggesting the capture of
transferable reasoning skills. Remarkably, our model outperforms specialist
models tuned on multimodal reasoning data in multimodal reasoning benchmarks,
while preserving the base model's performance on general visual benchmarks, a
challenge where specialist models often fall short. Our findings suggest a new
post-training paradigm: synthetic, rule-based games can serve as controllable
and scalable pre-text tasks that unlock generalizable multimodal reasoning
abilities in MLLMs.