NetPress: Dynamically Generated LLM Benchmarks for Network Applications
Authors: name: Yajie Zhou, name: Jiajun Ruan, name: Eric S. Wang, name: Sadjad Fouladi, name: Francis Y. Yan, name: Kevin Hsieh, name: Zaoxing Liu

Despite growing interest in domain-specific benchmarking of large language
models (LLMs) and agents, current evaluations remain limited to static,
small-scale datasets, especially in high-stakes tasks like network operations
that demand reliability for deployments. We present NetPress, an automated
benchmark generation framework for evaluating LLM agents in network
applications. NetPress introduces a unified abstraction with state and action,
enabling dynamic generation of diverse query sets along with corresponding
ground truths. At runtime, users can specify benchmark configurations to
generate millions of queries on the fly. In addition to dynamic benchmark
construction, NetPress integrates with network emulators to provide realistic
environment feedback, supporting comprehensive evaluation across correctness,
safety, and latency. We instantiate NetPress on three representative
applications, revealing interesting fine-grained differences in agent behavior
that static, correctness-only benchmarks often miss. NetPress moves LLM
evaluation toward realistic, scalable testing in infrastructure-centric
domains, helping close the gap between benchmark performance and real-world
deployment readiness. Code is available at
https://github.com/Froot-NetSys/NetPress.