Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs
Authors: Hao Kang
Qingru Zhang
Han Cai
Weiyuan Xu
Tushar Krishna
Yilun Du
Tsachy Weissman
Venue: NeurIPS 2025 spotlight ()
Keywords: Machine Learning
Gaming
Model Compression
LLM Agents
Primary area: deep_learning
TL;DR: We present the first systematic study of lossy latency–quality trade-offs in LLM agents, introducing HFTBench and StreetFighter benchmarks, and proposing an adaptive mixed-precision framework for real-world latency-sensitive tasks.

Abstract: Large language models (LLMs) have shown remarkable performance across diverse reasoning and generation tasks, and are increasingly deployed as agents in dynamic environments such as code generation and recommendation systems. However, many real-world applications, such as high-frequency trading and real-time competitive gaming, require decisions under strict latency constraints, where faster responses directly translate into higher rewards. Despite the importance of this latency–quality trade-off, it remains underexplored in the context of LLM-based agents. In this work, we present the first systematic study of this trade-off in real-time decision-making tasks. To support our investigation, we introduce two new benchmarks: HFTBench, a high-frequency trading simulation, and StreetFighter, a competitive gaming platform. Our analysis reveals that optimal latency–quality balance varies by task, and that sacrificing quality for lower latency can significantly enhance downstream performance. To address this, we propose FPX, an adaptive framework that dynamically selects model size and quantization level based on real-time demands. Our method achieves the best performance on both benchmarks, improving win rate by up to 80% in Street Fighter and boosting daily yield by up to 26.52% in trading,  underscoring the need for latency-aware evaluation and deployment strategies for LLM-based agents. These results demonstrate the critical importance of latency-aware evaluation and deployment strategies for real-world LLM-based agents.

Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs
Hao Kang Georgia Institute of Technology hikng3428gatech. edu
Qingru Zhang Georgia Institute of Technology qzhang441@gatech. edu
Han Cai
NVIDIA Corporation heai hm@gmail. com
Weiyuan Xu University of California, Berkeley
Tushar Krishna
Yilun Du
Georgia Institute of Technology tusharβece. gatech. edu
Harvard University
y silundu@gmai... com
Tsachy Weissman
Stanford University
tsachy@stanford.cm
Abstract
Large language models (LLMs) have shown remarkable performance across diverse reasoning and generation tasks, and are increasingly deployed as agents in dynamic environments such as code generation and recommendation systems. However, many real-workd applications, such as high-frequency trading and real-time com- petitive gaming, require decisions under strict latency constraints, where faster responses directly translate into higher rewards. Despite the importance of this latency-quality trade-off, it remains underexplored in the context of LLLM-based agents. this work, we present the first systematic study this trade-off in real- time decision-making tasks. To support our investigation, we introduce two new benchmarks: Henchhhh high-frequency trading simulation, and StreetFighter, competitive gaming platform. Our analysis reveals that optimal latency -aantcyyyllltyyyyy balance varies by task, and that sacrificing quality for lower latency can signifi- cantly enhance downstream performance. To address this, propose FPX, adaptive framework that dynamically selects model size and quantization level based real-time demands. Our method achieves the best performance benchmarks, improving win rate by up to 80% Street Fighter and boosting daily yield by up 26.52% in trading, underscoring the need for latency-aware evalua- tion and deployment strategies for LLM-based agents. These results demonstrate the critical importance of latency-aware evaluation and deployment strategies for real-world LLM-based agents. Our benchmarks are available at Latency Sensitive Benchmarks.
both
1 Introduction
Large language models (LLMs) exhibit remarkable performance across various natural language processing (NLP) tasks and artificial intelligence (AI) applications, ranging from text generation to complex reasoning [OpenAI, 2023, Abdin et al., 2024, Team et al., 2025]. Beyond their standalone use, LLMs can be integrated into agent frameworks, enabling more sophisticated behaviors such decision-making multi-step reasoning, and planning [aao et al.. 2023, Shinn et al., 2023, Li et al., 2023, Du al., 2023]. In these settings, a LLM acts as a decision- maaning agent, generating its actions or responses and then receiving feedback or rewards from environment. Many of these
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.25
30
HFT
Win Rate over PM))
80
20
149
145
7B
Game
60
0.15
10-
0.10
40
Daly
0.05
0.00
10
150
200
300
400
600
400
500
600
700
1000
1300
1p16
1p4
Optimals
Latendy (ms)
Latency (ms)
(b) Street Fighter Latency-Aceuracy (c) HFTBench pareto curve
(a)
trade off
pareto curve
Figure 1: Latency-accuracy trade-offs across different model configurations and tasks. (a) FPX enables smooth and continuous trade-off between lateney and accuracy, allowing models meet diverse task-specific requirements. (b) In the Street Fighter benchmark, win rate first increases as latency decreases, peaking at a Pareto-optimal point, before dropping due to excessive accuracy loss. (c) Observation in HFTBench: daily yield improves with moderate latency reduction, but degrades when model accuracy is overly compromised.
agent tasks exhibit high tolerance for inference latency, where slow responses are acceptable as long the output quality remains high. Examples include code generation [Zhuo et al., 2024], mathematical problem solving [Xiao et al., 2023], and product recommendation [Wang et al., 2023], where correctness and completeness are prioritized over speed.
However, there is a different large class of real-time tasks that are highly sensitive to response latency and remains largely unexplored. These tasks often take place in dynamic environments that evolve continuously over time and are influenced by the agent's actions. such settings, response latency becomes critical factor in agent's overall performance. Fast and well-timed actions are essential for obtaining positive rewards, while delays often leads to missed opportunities or suboptimal outcomes. One prominent example is gaming. Competitive games, such as Street Fighter [Su, 2010] and StarCraft [Samvelyan et al., 2019], take place in wall-tmme environments where agents must perform multiple actions in timely manner to win. Faster agents are more likely to stay synchronized with environmental changes and maintain an advantage, while slower agents may fall behind while processing outdated observations. Similarly, robotic control in dynamic physical environments demands rapid perception-actiinn loops. Delayed responses in such settings, especially in htgh-stakes applications like autonomous driving, result in unsafe incorrect behavior.
Another important example is using LLM agents for feegfrryyynccy financial trading [He and Lin, 2022], where both low latency and high response quality are crucial. Stock exchanges match transactions based on real-time order flow, and faster trading agents can exploit arbitrage opportunities by acting before competitors respond. Prior research [Baron et al., 2019, He and Lin, 2022] in finance shows that trading latency directly impacts carning yields, motivating investment institutions to heavily invest in low-latency methods.
Across all these examples, both inference latency and response quality are critical for LLM agents to achieve strong performance. Either delayed or low-quality actions can lead to performance degradation or outright task failure. However, a fundamental trade-off exists between latency and quality when choosing models of different sizes compressing them to low precisions. As illustrated by Figure Ia, larger models typically generate higher-quality outputs but suffer from longer inference time, while smaller or highly compressed models offer faster inference speed at the expense of reduced output quality. Therefore, as shown in Figures 1b and Ic, there exists an optimal solution for this trade-off and effeectively balancing this trade-off is essential to optimize model performance in this type of real-world tasks.
In this paper, we are the first to systematically formulate and investigate the latency -uulliyy trade-off in the context of real-time ggeision-makinng by LLM agents. We define this class of tasks latency- sensitive agent decision tasks, where both output quality and response latency jointly determine the agent's overall performance. To evaluate model performance such latency-sensitive setting, we develop two novel real-time evaluation benchmarks: (i) HFTBench: a high-frequency trading system tailored to assess real-time trading decisions of LLMs; (ii) StreetFighter: competitive gaming platform that evaluate real-time gaming decisions of LLMs.

---

Based on our benchmarks, we observe that different tasks exhibit varying sensitivities to inference latency and output quality. As shown in Figure 1b and Ic, StreetFighter is more aatency-sensitive and less quality-
timely, even if suboptimal, actions often lead to winning outcomes due to the game's simple yet rapidly evolving dynamics.
contrast, trading tasks demand both high quality and low latency. Inaccurate decisions can result in significant financial losses, making response accuracy as crucial as speed. Confronting such diverse task requirements, it is inherently challenging to identify the optimal point along the latency-quality trade-off. Existing approaches, such as selecting among fixed model sizes or applying static low-precision quantization, typically offer only a limited set of discrete options, which fail to capture the fine-gn grained trade-offs required across real-worl real-world tasks. To enable fine-grained searching, we introduce FPX, an adaptive mrixed-precision inference framework that enables flexible control over inference latency while minimizing quality degradation. FPX jointly adjusts model size and dynamically mixes inference bitwidths across model layers to meet any specified latency target. Specifically, it hybridizes FP8 and FP4 inference kernels by selectively applying lower precision (FP4) to compression-toleres more sensitive components. This progressive and targeted quantization approach allows FPX to achieve continuous, fine-grained control across the latency-quality trade-off, effectively minimizing performance loss while satisfying diverse latency requirements. Our contributions are as follows:
layers while preserving FP8 for
Latency-Quality Trade-off. We are the first to systematically formulate and investigate the latency-qualit trade-off in the context of latency-sensitive agent decision tasks.
We introduce two novel benchmarks for evaluating LLM performance in the latency-sensitive settings: (1) high-frequency trading (HFT) system specifically tailored to LLMs, and (2) a competitive fighting game environment based on Street Fighter from the DIAMBRA platform [Palmas, 2022].
Latency-Sensitiv Evaluation Benchmarks:
Adaptive Mixed Precision Inference Framework. We propose adaptive mixed- precision inference framework that that enables flexible control over inference latency while minimizing quality degradation.
2 Background
2.1 Low Precision Inference to Reduce Latency
Recent advancements in hardware-supported low-precision inference, such as FP8 and FP4 [Micike- vicius et al., 2022, Li et al., 2025], offer significant improvements in both throughput and latency over standard full-pree full-precision inference (FP16). These methods employ floating point quantization (FP Quant) to map high- precision tensors to low-precision ones, reducing memory footprint of both model weights and activations [Li et al., 2025]. Given a tensor X, FP quantization rescales its entries and rounds them to values within a bounded range determined by bitwidth b:
X
max([X]]
if max([x]) > range, otherwise
Q(X) round
scalex
range,
scalex
Here, Q(X) is the quantized matrix and range, is determined by the bitwidth b, specifically 240 for FP8 [Micikevicius et al., 2022] and 6 for FP4. During inference, the forward pass in linear layers can be approximated as:
XW scalex scalew Q(X)((W)
As supported by hardware, low low-precision inference benefits from faster floating-point operations, improved memory bandwidth, and efficient datatype conversion. With substantially reduced memory footprint, low-precision inference can significantly lower end-to-end inference latency compared to FP16. For instance, FP8 typically provides up 2x latency speedup while maintaining near-lossless output quality, making it widely adopted. FP4, on the other hand, can yield up to 4x latency reduction, but often causes severe degradation in model performance, limiting its standalone application. Recent work such as SVDQuant [Li et al., 2025] attempts to mitigate the accuracy loss by combining it with low-rank corrections and smoothing. However, such approaches remain static and do not offer adaptive control over the latency- -uaalit trade-off in real-time, latency sensitive tasks.
-atency-sensitvve
2.2 Additional related work on throughput optimization
Another line of related work focuses on conventional serving scenarios, whose primary goal is to improve serving throughput while maintaining near-lossless performance. For example, systems such
3

---

as vLLM [Kwon et al., 2023] and SGLang [Zheng et al., 2024] achieve around 6.4x throughput improvements without compromising output quality. While such systems may reduce lateney in specific conditions (e.g., shared prefill structures in SGLangg, they are generally not designed to optimize latency in a task-specific manner. Other efforts, such as Al Metropolis [Xie al., 2024], build distributed cluster systems to accelerate agentic simulations through speculative execution of multiple agents. These approaches aim to maximize simulation throughput but are not tailored for latency-sensitive, real-world agent deployments. Separately, a substantial body of work explores integer quantization to improve serving throughput [Lee et al., 2024, Frantar al., 2023, Kang et al., 2024b, Zirui Liu et al., 2023]. Unlike hardware-supporte FP quantization, integer quantization typically requires highly costly dequantization operations during inference. While it enables larger batch sizes and improves overall throughput, the dequantization overhead significantly limits its effectiveness in reducing latency [Lin et al., 2024, Zhao et al., 2024, Kang et al., 2024a]. Other works [Tang al., 2023, Pandey et al., 2023] propose mixed-precision schemes combining integer and floating-pont point formats to halance throughput and accuracy. However, these methods remain static and lack the ability to provide fine-grained, dynamic control over LLM inference latency.
3 Latency-Sensitive Agent Decision Tasks
In this section, we formally define the latency-sensitive agent decision tasks, formulate its latency- quality trade-off. and introduce two real-time evaluation benchmarks: (i) HFTBench frequency trading system tailored to evaluate real-time trading decisions of LLMs, and (ii) Street- Fighter competitive gaming environment that assess real-time gaming decision of LLMs.
high-
3.1 Formulating Latency-Sensitive Agent Decision Tasks
Consider general setup of in which an LLM agent interacts with environment & to solve task. At time step i, the agent receives an environmental observation og E O. After spending A, time conducting inference, the agent responds action d++1; A following its decision policy Te:
a++A, (G()) where c {(og, ao+ (01, (1+A,)....... o.).
Here e is the context to the agent, for example, conversation between a user and the agent.
As introduced in Section I, conventional agent tasks exhibit high tolerance to LLM inference latency A, A simple case is single-step tasks such as one-hop question answering [Kwiatkowski et al., 2019] or document summarization [Shaham al., 2022]. where the agent generates single action given an initial input prompt on, and the outcome is evaluated purely based on the output quality:
r = R(a (on), where denotes a task-specific evaluation or reward function. A more complex case involves multi-step task-solving, such as multi-step mathematical reasoning code generation, where the agent produces a sequence of actions over time. In such tasks, the overall performance depends the cumulative quality of all outputs:
Σ R(on-sil le)
In both cases above, the environment is relatively static, and delayed responses acceptable as long as the agent maintains high response quality. Correctness is prioritized over speed.
However. many real-world tasks, such as gaming, robotic control, and tigh-frequency trading, take place in dynamic environments that evolve rapidly over time. This setting remains large unexplored and we name it lalency-sensitie agent decision tasks. In these tasks, a delayed actions at is often rendered ineffective or obsolete by the time it is executed under the updated environment state E leading to missed opportunities or degraded outcomes. In such setting, the agent is evaluated not only by what it decides, but also by how long decides. To succeed, it must produce actions that are both high-quality and timely. The reward thus becomes function of both the decision and its latency, evaluated under the evolved environment:
Σ R(acHsilEeaa.)
(5)
This formulation captures the core challenge of latency-sensitive tasks: enabling LLM agents to make fast and accurate decisions in environments where speed is critical accuracy.

---

3.2 HFTBench: High-Frequency Trading Benchmark
Latency and Quality in Financial Trading, High-frequency trading (HFT) involves rapidly submit- ting buy and sell orders to centralized exchanges, where transactions are strictly matched based on arrival time. In this setting, even millisecond-levt differences in reaction latency can significantly impact profitability. Temporary arbitrage opportunities often arise when short-term imbalances cause the bid-ask spread to widen. Agents that respond quickly can capitalize on these brief windows by buying at temporarily depressed prices or selling at elevated ones-befowe the market rebalances.
However, latency alone is insufficient. High-quality trading decisions rely on correctly interpreting market conditions, which often require processing multi-step patterns in historical prices, order book dynamics, and occasionally external signals such as policy announcements or financial news. While smaller LLMs benefit from lower latency, our experiments show that they often fail to capture such complex financial patterns, resulting in poor decisions that negate their speed advantage.
Benchmark Design. We construct a realistic backtesting simulation using historical per-second trad- ing data from Polygon.io [Polygon.io, 2024]. Each agent receives synchronized market observations at 1-second intervals and must decide whether to take action. To isolate the effect of decision latency, all agents have access the same information and observation windows.
When an arbitrage opportunity is detected, agents initiate inference. The simulated exchange ranks agents by their response time and assigns execution prices accordingly: faster agents secure more favorable prices. We implement a linearly decaying price model of time and price, where trading advantage diminishes with slower responses mimicking real-world queue- order execution.
Evaluation Protocol.
Each agent observes compact state containing prior execution prices, current bid-ask margins, available capital, and time remaining in the trading session. To avoid unnecessary LLM calls, inference is only triggered when the bid-ask margin exceeds a preset threshold b. Agents are evaluated by their cumulative daily yield, and a configuralle cooling window t is applied between evaluations to improve simulation efficiency.
3.3 Gaming Benchmark: Street Fighter
Latency Sensitivity in Competitive Games. In real-time competitive games such as Street Fighter and StarCraft, delayed actions can result immediate penalties, positional disadvantages, or even round losses. Unlike financial trading, where both decision quality and latency play important roles, these games are overwhelmingly latency-sensitive. our experiments(Figure 1b), agents with just a 20% reduction in response time consistently outperform their slower counterparts. Interestingly, the strategic depth of Street Fighter is relatively limited, and well-prompted small LLMs can produce effective actions, provided they respond quickly enough.
Benchmark Design. We build on top of DIAMBRA's simulation platform [Palmas, 2022] to support real-time Street Fighter matches with local model inference. To improve performance for compact models (c.g., <7B parameters), we augment the prompt with tailored few-shot examples specific to each character and scenario. This enhancement helps mitigate performance degradation from reduced model capacity.
Evaluation Protocol. Agents receive a concise game state that includes character-specifit move sets, recent action history, and a contextual prompt. We evaluate performance using the ELO rating system [Elo, 1967], where agents compete across multiple matches against a diverse set of opponents. ELO scores are updated dynamically reflect win-loss outcomes, providing a stable and interpretable metric for real-time decision quality under latency constraints.
3.4 Discussion
LLM Agents in Finance and Gaming, Recent works have explored the application of LLM-based agents in both financial trading and competitive gaming. In finance, FinMem [Yu et al., 2023] and FinAgents [Zhang et al., 2024] demonstrate that LLM agents outperform traditional reinforcement learning and rule-based strategies. This performance gain is attributed to the robustness of LLMs against overfitting and their unique ability to process unstructured inputs, such as policy updates or financial news, through in-context learning. However, these approaches are evaluated on static historical datasets and ignore the role of response timing, which crucial in real-time trading. In contrast, our nigh-frequency trading benchmark captures not only the agent's decision quality, but
5

---

Static env(DB, Web, etc)
Time sensitive env(Game, Trading, etc)
Task
Task
Task
C
Agent
Agent 9
Agent
Figure 2: Comparison of agentic LLM for Static environments like code generateion or research and time sensitive environments like trading and gaming. Environment is constantly changing with time and other agent's interaction. For such tasks, reward related to both quality and latency of agents.
also its response speed and the pricing gap it can exploit offering a more faithful simulation of real-world trading dynamics.
In the gaming domain, prior work has applied LLM agents to real-time strategy and fighting games such StarCraft and Street Fighter [Ma al., 2024, Palmas, 2022]. These studies primarily focus improving action quality and designing robust inference pipelines. However, they do not consider the inherent trade-off between latency and decision quality that governs real-time decision performance. Our benchmarks specifically emphasize this trade-off, providing a clearer understanding of how timing impacts success in latency-sensitive environments.
4 FPX: Adaptive Mixed Precision Inference Framework
In this section, we introduce FPX, our adaptive mixed-precision inference algorithm designed for latency-sensifivee agent decision tasks. As motivated in section 1, FPX dynamically adjusts precision at the operator level, switching between the matrix multiplication kernels of FP8 and FP4, to enable continuous fine-grained control over the latency-quality trade-off.
4.1 Adaptive Mixed-Precision Algorithm Design
The core goal of FPX is to balance latency and accuracy by selectively lowering the precision of only the most compression-tolenan
components in model. Instead of modifying full models or entire layers, we adopt a more granular precision control scheme that applies FP4 only to linear layers that can tolerate aggressive quantization, while preserving FP8 for more sensitive parts.
To ensure compatibility with a wide range of transformer architectures, we focus exclusively on optimizing matrix multiplication operators, which dominate inference latency in LLMs. These include query/key/value (QKV) projections, output projections, and feedforward layers. Other components, such normalization and attention mechanics, left untouched maintain functional correctness and deployment simplicity.
Importantly, because transformer linear layers exhibit similar structural and computational properties, latency gain from replacing FP8 with FP4 is approximately uniform across layers. This decouples precision assignment from latency impact and shifts the optimization focus entirely toward minimizing quality loss. To quantify the robustness of each linear layer to quantization, we compute a relative error metric E based activation outputs under FP16 and FP4 execution:
L4 fp16
fd l2
(6)
fp16
Here, ffp16 the output of layer under FP16 execution, and ffd is the output when the same input processed using an FP4 kernel. The normalized error ε captures the fidelity loss introduced by low-precision inference and serves as the basis for selecting compression candidates.
Given user-specified compression ratio 7 E [0, 1], we define precision assignment function 5(1) E {4, 8} for each linear layer I:
[ if Sy 8 otherwise"
8(1)
where S,, argmin SCC Σ |||=YL 16S
E1

---

Algorithm Adaptive FP4/FP8 Precision Assignment for Transformer Layers
Require:
Transformer model M with L linear layers L (f1....,4..) calibration dataset D, compression ratio zy [0, 1]
Ensure: Precision assignment function 6(1) E {4,8) for all E E
1: for all layer E C do
2: Run FP16 inference on to collect outputs ff116
3: Simulate FP4 output A using the same inputs
4: Compute relative quantization error:
fp16
A 1 l2
||
1p16
12
5: end for
6: Sort layers in ascending order of e
7:
Select S. as the 7 layers with the smallest ě1
8: for all layer E L do
9:
if ( S, then
8(1)
Assign FP4 to quantization-tolerant layers
11: else
6(1) +
Preserve FP8 for sensitive layers
13: end if
end for
15:
return 5
Here, S., denotes the subset of L layers with the lowest quantization error. This design ensures that FP4 is selectively applied to the most robust layers, enabling substantial latency gains while minimizing quality degradation.
4.2 Offline Calibration
To compute the layer-wise quantization error et, we perform one-time offline calibration using held-out language modeling dataset. Following standard practice in quantization research [Xiao et al., 2024, Hooper et al., 2024], this calibration phase estimates typical activation distributions (f Cenpenntly bi f inpprronniiii de Mee
Then, simulate FP4 execution by running each layer individually, replacing its FP16 kernel with an FP4 kernel, while keeping all other layers unchanged. This isolates the quantization impact at the layer level and yields reliable estimate of E for each candidate.
The complete precision assignment pipeline is summarized Algorithm 1.
Experiments
We evaluate our method on time-sensitive task benchmarks introduced in Section 3. We then perform an ablation study to analyze the lantecy qualltyy trade-off brought by FPX.
5.1 Experimental Setup
Models. To ensure fair comparison and reduce the complexity of the search space, conduct experiments on a family of models pretrained on similar datasets. Evaluating across heterogeneous model families could introduce biases due differences in pretraining quality, architecture, tokenizer design. Therefore, we focus on the Qwen2.5 model suite [Qwen et al., 2025], ranging from 1.5B 14B parameters.
Benchmark Configurations. For the high- high-frequency trading (HFT) benchmark, we evaluate on stock data from Nvidia and Amazon on August 5th, 2024. We follow the configuration introduced in Section 3.2, setting the profit threshold to 2% and the time window to 1 minute. The initial cash for agent is 10,000 dollars. For the gaming benchmark, conduct 40 matches between model pairs and compute win rates to derive EIO ratings.
Method Configurations. We discretize the compression ratio of FPX into steps of 0.1 to explore the trade-off between latency and accuracy across different benchmarks. We only report the best- performing setting for each model in each task. In experiments, fine-grainct changes in
7

---

Table 1: Evaluation results on latency-sensitive benchmarks. Our method achieves the best la- teney-reward trade-off across both tasks. Only shows top-6 results. More results are shown in appendix.
HFTBench
Model Parameter Size
Bitwidth Avg
Latency (ms)
Daily Yield (%)t
14B (ours)
7.2
713
26.52
14B
8
801
23.14
14B
16
1302
17.20
7B
16
619
-3.28
7B (ours)
7.6
386
-7.25
7B
8
394
-12.94
Street Fighter
Model Parameter Size
Bitwidth Avg
Latency (ms)
ELO Score
3B (ours)
6.8
195
5,99
7B (ours)
7.2
354
2.33
3B
8
222
2.19
3B
16
349
0.25
7B
8
394
-0.44
1.5B
8
142
-1.25
generally have minimal effect, indicating that our selected settings are near-optimal. All experiments are run on an RTX 5090 GPU unless otherwise specified. 14B models are served across multiple GPUs using model parallelism.
5.2 Baseline Techniques
Time-sensitive benchmarks are sensitive to both model quality and inference latency. Any quantization method that results slower inference than FP8 is excluded from consideration. We evaluate the following baselines:
FP16: A standard dense model with both activations and weights in 16-bit floating point. This serves as the upper baseline in quality but incurs the highest latency.
FP8: A widely adopted low-precision format for Hopper and newer GPU architectures, representing both activations and weights as 8-bit floating point. It typically offers near-lossless accuracy with significantly better efficiency than FP16.
FP4: highly compressed representation where both activations and weights are quantized bits and packed as 8-bit integers. This setting drastically improves efficiency but at the huge cost of model response quality. It is only available on blackwell architecture GPUs.
5.3 Evaluation Result
Table demonstrates that FPx, by dynamically trading off latency and quality through adaptive model size and bitwidth selection, achieves the highest daily yield on HFTBench and the best overall reward across both benchmarks.
High-Frequency Trading (HFTBench). This benchmark requires a careful balance between latency and response quality. We observe that larger models, such as 14B, outperform smaller alternatives due their stronger ability recognize profitable opportunities. In contrast, smaller models often fail to detect high-reward patterns or generate outputs that are too unreliable to be translated into effective trading decisions. FPX improves the latency the 14B model by compressing 20% of its linear layers into FP4, while preserving FP8 for the rest. This enables a favorable speed-quality trade-off, allowing 14B+FPX to achieve the highest daily yield among all candidates. Interestingly, we find that further reducing the latency of weaker models like 7B actually harms performance. Faster response does not help if the decisions themselves are poor, and can even increase the rate of loss.

---

Performance under different compression levels on Qwen2.5 models for HFTBench and Table 2:
Street Fighter." " means model performance is complete destroyed.
HFTBench Qwen2.5-14B
Gamma (?)
Lateney (ms)
PPL.
Daily Yield (%)1
0.0 (FP8)
801
4.55
23.14
4.92
26.52
0.4
623
6.71
12.93
0.6
558
0.00
0.8
503
0.00
(FP44
0.00
Street Fighter Qwen2.5-3B+FPX versus Qwen2.5-3B-FP16
Gamma (7)
Latency (ms)
PPL.
Winrate (%)1
0.0 (FP8)
222
6.85
72.5
0.2
207
7.03
77.5
0.3
200
9.02
80.0
0.4
192
11.59
62.5
0.6
178
17.42
12.5
0.8
153
0.0
(FP4)
0.0
StreetFighter. This task is highly latency-sensitive, yet quality still matters. Our method achieves the best performance with 3B model configured with 30% of layers in FP4 and 70% in FP8. Notably, although the fastest candidate, the 1.5B model with full FPS inference, has the lowest latency, performs poorly due to its limited decision makinn capability. Morcover, the environment itself imposes an upper bound on effective response rate. In StrectFighter, each character action takes a fixed amount of in-game time to complete, with effective frame rate of around 5 actions per second (i.e., 200ms per action). Any optimization that reduces model latency beyond this threshold yields no further benefit, the game cannot process actions faster than this limit.
5.4 Ablation Study
Latency-Quas Quality Trade-off of FPX We evaluate the Pareto frontier of the latency-quality trade-off' induced by FPX across both benchmarks and bitwidth configurations. Specifically, we apply FPX to the Qwen2.5 model family and compare against standard FP16 inference. Our results show that FPX effectively adapts each model's 's inference path between FP8 and FP4 regimes, dynamically balancing latency and accuracy. Notably, the optimal trade-off point varies by task and model: for instance, HFTBench with the 14B model, the best performance achieved at 0.2, while on Street Fighter with the 3B model, the optimal setting is 7 0.3. These findings highlight that latency-sensitiv decision-make tasks require task-specific latency-quality configurations, and FPX enables L.LM agents navigate this trade-off effectively.
Limitations and Conclusion
In this work, we present the first systematic study of the aieecy--ualiig trade-off for LLMM-based agents in latency-sensitive agent decision tasks. To support this investigation, introduce two real-time evaluation benchmarks: HFTBench, high-frequency trading simulator, and StreetFighter, competitive gaming environment. In both settings, rapid yet accurate decisions essential to achieving high downstream rewards.
To meet the heterogencous demands of these tasks, propose FPX, adaptive mixed-preeeeee precision infer- ence framework that dynamically adjusts model precision to optimize for task-specific latency-quality trade-offs. By selectively applying FP4 quantization to compression-tolenant layers while retaining FP8 for sensitive components, FPX enables fine- mne-grained latency control with minimal performance degradation.
Extensive experiments on Qwen2.5 model variants demonstrate that FPX consistently discovers favorable operating points that outperform fixed-precision baselines across both domains. Our ablation results further reveal that the optimal compression configuration varies significantly by task and model, underscoring the importance of latency-awan deployment strategies for LLM agents.

---

While FPX demonstrates strong empirical gains, it has limitations. Our current precision assignment operates the layer level for simplicity and compatibility. More fine-grained schemes, such as token-level precision control, may unlock better trade-offs, but require significantly more complex implementation and kernel support. We left this optimization for future works.
We hope that our benchmarks and findings encourage future research toward building efficient, adaptive LLM systems and algorithms that prioritize latency-awareness in real-world applications, rather than focusing solely on maximizing accuracy or model performance.
7 Acknowledgement
We are deeply grateful to Professor Tsachy Weissman for his guidance. We also thank Jinyan Su(PhD student from Cornell University) for valuable suggestions on refining the paper.
10

---

References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi. Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL https: att/axxivvv.o.... wrg/abs/ herg/bbs/2411 2412. 2.412.08900 08905. 1
Matthew Baron, Jonathan Brogaard, Björn Hagströmer, and Andrei Kirilenko. Risk and return in high-frequency trading. Journal of Financial and Quantitative Analysis, 54(3):993-1024, 2019. doi: 10.1017/50022100011800000
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. URL https:
//arxiv. org/ abs/2305. 14325.
Arpad E. Elo. The proposed usef rating system: Its development, theory, and applications. Chess Life, pages 21-28, August 1967. URL https: //tp::////c//oo//// suscfi-nycl... aodhosting, com/CL Cea//CLLNDD-OO AND-CR CR- ALL/ CL- ALLL1967/1967 08. pdf#page=26. wttp1/66__...paaaaeeeeeee
Accessed: 2025-05-05, 5
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https: org/ abs/ 2210. 17323. 4
Xue-Zhong He and Shen Lin. Reinforcement Learning Equilibrium in Limit Order Markets, Journal of Economic Dynamics and Control, 144(C), 2022. doi: 10. 10.10166ijcd::2222.044444 https://ideas htw.iiddasseppp....eeee aoppe..org/ /e/ee/e///aee httpa//eeeepoe////42222000011110
URL html.
Coleman Hooper, Schoon Kim. Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization, 2024. URL https: /tt////axiivvi/ org/ http:/abs/200.....
7
Hao Kang, Srikant Bharadwaj. James Hensman, Tushar Krishna, Victor Ruhle, and Saravan Rajmohan. Turboattention: Efficient attention approximation for high throughputs Ilms, 2024a. URL https: // //arxiv. . wwt./oo////////e//eooeoe / 08585.
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of I1m, 2024b. URL https: /arxxivxvvv/v//////////////.aav/ org/ ab43/403, 05527. 4
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 2019. doi: 10.1162/tacl a a_02276. URL wtgg/Q1--1022
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv wwwpeiiiivvor/aeeeooooo org/ abs/2309 06180.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024. URL https: /// www.iiooorr.... abs/ 2200. 02272.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52009 2023.
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng. Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models, 2025. URL https: //wxxvv..vv/////////////////aavl org/ abs/ 0bbs/211... 05007. 3
11

---

Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantizaaion and system co-design for efficient Ilm serving, 2024. URL
https://arxiv.. www.vv.rrg/bbss www.///////......////// 04532.
Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play starcraft ii: Benchmarks and a chain of summarization approach, 2024. URL https://arxi:: org/ abs/2312. 11865. 6
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. 7
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenth- waite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. Fp8 formats for deep learning, 2022. URL
/2099 05433. 3
OpenAl. Gpt-4 technical report, 2023. 1
Alessandro Palmas. Diambra arena: new reinforcement learning platform for research and experi- mentation, 2022. URL.https://aaxiit wwtp://.gg///b////////oee 10595. 3, 5, 6
Nilesh Prasad Pandey, Markus Nagel, Mart van Baalen, Yin Huang, Chirag Patel, and Tijmen Blankevoort. A practical mixed precision algorithm for post-training quantization, 2023. URL https:///rxxi org/ 220p./0397.........
Polygon.io. Polygon real-time financial market apis. https://polygon io, 2024. Accessed: 2025-04-29,
Qwen, :, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxix: org.abg..aa.....o.o........... 2212. 15115.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902:00900 2019. 2
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language sequences. arXiv preprint wtXXh:2201.035333, 2022. 4
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reftection. arXiv preprini arXiv:2303. 1366, 2(5):9, 2023.
Norman Makoto Su. Street fighter iv: braggadocio off and on-line. In Proceedings of the 2010 ACM ACM conference on Computer supported cooperative work, pages 361-370, 2010. 2
Chen Tang, Kai Ouyang, Zhi Wang, Yifei Zhu, Yaowei Wang, Wen Ji, and Wenwu Zhu. Mixed- precision neural network quantization via learned layer-wise wise importance, 2023. URL https: //arxiv. org/ abs 2203. 08368. 4
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete. Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa
12

---

Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic. Amit Vadi. András György, André Susano Pinto, Anil Das, Ankur Bupna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo. CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak- Plucińika, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, lan Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez. Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy. Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy. Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger. Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg. Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei. Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo., Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez. Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, -ean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https: // //aaaxiiivv.oo///aaaaaaaaaavvaaa org/ 2503. 19786. 1
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang. Yanbin Lu, and Yingzhen Yang. Recmind: Large language model powered agent for recommendation. arXiv preprint arXiv:2308. 14296, 2023. 2
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https: //arxiv. org/ /211 10438. 7
Ziyang Xiao, Dongxiang Zhang, Yangjun Wo, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. Chain-of-experts experts: When llms meet complex operations research problems. In The twelfth international conference on learning representations, 2023. 2
Zhiqiang Xie, Hao Kang, Ying Sheng, Tushar Krishna, Kayvon Fatahalian, and Christos Kozyrakis. Ai metropolis: Scaling large language model-based multi-agent simulation with out-of-order execution, 2024. URL https: //tpx//axivv.//// org/ absa /tab.////.....e 03519. 4
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, In The Eleventh International Conference on Learning Representations, 2023. URL httt::://// /twp://ee/e///ee//////// net/ form?? id= WE_vluYUL-X.
Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Denghui Zhang, Rong Liu, Jordan W. Suchow, and Khaldoun Khashanah. Finmem: A petformance-enhaaned 11m trading agent with layered memory and character design, 2023. URL h/tps://aaai
wwwp/ogg/////////e///oeeee/ abs////s//311/ 117433. 5
Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, Longtao Zheng, Xinrun Wang, and Bo An. A multimodal foundation agent for financial trading: Tooe-augmented, diversified, and generalist, 2024. URL https: //arxiv. org/ abs/ 18485. 5
Yilong Zhao, Chien Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving, 2024. URL https: /awxxviivvvvv////////a//aaaa orgg/as//g///ooooo 19102. 4
13

---

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. httpst / ht/p/axxvvv.ooo orgg/ abs/2312.077 07104. 4
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024. 2
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi : Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization. 2023. doi: 10. 2.144000..2222886..77777 R 2. 372882. 4
URL www.:htttt///e https://rgooi......./oooooo http:///rooooieett..111.... wwt.:/eett............. 13140/
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes it matches the contribution. We are the first the evaluate latency-quality trade-off in such latency-sensinn tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
This paper analyzes the trade-off of latency-sensitive agent decision tasks along with method. We evaluate the limitations of our method and leave this for future works.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
We provide detailed illustration and assumption for algorithms.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Yes will and we will keeping optimize our benchmark and method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
Ye we will opensource our benchmark and evaluation script of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
httpe//guuddee//eeSbmbmssse www.ssubbbsssooosccoo..oo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. cttps/cccppbllcccccccccc hopppulcc/guuddeeeeo guides/ CoodeSummis ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have specify the setting of our experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes we do.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes we have specificly pointed out what kind of hardware and platform we use for experiments experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http////eurr..peeeeee
Answer: [Yes]
Justification:
Yes we make sure preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: section.
Yes we discuss the positive societal and impacts of the work in the conclusion
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Yes we introduce two benchmarks and one adaptive quantization algorithm.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This project does not involve research with human nor crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htass///nneue htt//eerrss...e/ http///ips.c//oooeeeeeeeeeeeeeee / for what should or should be described.
21

---

A Visualization of HFTBench Data
Here we provide the high-low price per second for the data we have used for HFTBench tests in Figure 3. Red rectangle points out the buy-sell price gap in short time, which provide trading opportunity for agents. Such opportunity only happens in short time. Buying and selling decisions of other agents will decrease the gap quickly in miliseconds.
NVDA Per-Second Prices on 2024-08-05
102.5
High
100.0
Low
Price (S)
95.0
01-01 10
01-01 11
01-01 12
01-01
01-01
01-01
Time (hour)
AMZN Per-Second Prices on 2024-08-05
162.5
160.0
Men Ayan
Price (S)
157.5
155.0
High
152.5
Low
01-01
10
01-01 11
01-01 12
01-01 13
01-01 14
01-01 15
01-01 16
Time (hour)
Figure 3: Visualizations of HFTBench testing data.
B More Experiment Results for StreetFighter
Here we provide more results of StreetFighter. Competitors are run for round and calculate the ELO scores.
Table 3: Latency and yield comparison on StreetFighter.
Model Parameter Size
Bitwidth Avg
ELO Score( %)
6.8
5.65
3B
7.2
3.57
7B
6.8
2.33
7B
7.2
2.33
3B
8
2.18
3B
16
0.26
7B
8
-0.45
1.5B
16
-1.25
1.5B
8
-2,66
7B
16
-2.89
14B
8
-3.14
14B
16
-5.94
C Latency Profiling of Quantization method
We conduct a detailed latency profiling of various quantization methods on RTX 5090 GPUs. For the 14B model, we employ model parallelism across two GPUs. The results are summarized Table Our findings show that both FP8 and FP4 kernels yield substantial latency reductions compared to
22

---

the FP16 baseline. However, for the W4A16 configuration, where model weights are stored as 4-bit integers, the latency benefits are less pronounced, except in large models such as Qwen2.5-14B. This is likely due to the overhead introduced by data type conversion and dequantization. These results suggest that hybrid usage of FP8 and FP4 kernels a promising strategy for improving inference efficiency, particularly on large-scale models.
Table 4: Latency (ms) Comparison Across Quantization Schemes
Model
FP16
FP8
W4A16(int)
FP4
Qwen-1.5B
203
142
254
83
Qwen-3B
349
222
323
147
Qwen-7B
619
394
537
248
Qwen-14B (2x5090)
1302
801
792
492
23


Reviews:
[{"review_id": "pAd5naLbTg", "reviewer": "Reviewer_AkfL", "summary": "This paper establishes the first formal framework for latency-quality trade-offs in LLM agents operating in dynamic environments. It introduces \"latency-sensitive agent decision tasks\" where delayed actions incur real-time penalties , and contributes: (1) Two novel benchmarks—HFTBench (high-frequency trading simulator with Polygon.io data) and StreetFighter (real-time gaming platform); (2) FPX, an adaptive mixed-precision framework using layer-wise FP4/FP8 inference via offline calibration;", "questions": "- Threshold sensitivity: StreetFighter’s 200ms action ceiling appears game-specific. How would results scale to 100ms (e.g., autonomous driving) or 500ms (e.g., robotic control)?\n\n- MoE compatibility: Does FPX maintain efficacy for mixture-of-experts architectures? \n\n- The current experiment only tests a single agent, but the real trade/game involves a multi-party game. Is there a \"delayed arms race\" when multiple FPX agents are in parallel?", "limitations": "Authors acknowledge layer-wise control constraints. Additional discussion needed on: Multi-agent latency competition dynamics.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Experimental rigor is outstanding: granular ablation across Qwen2.5 models, Addresses a critical gap where prior agent frameworks ignore latency penalties. Benchmarks enable standardized evaluation for real-time AI systems. The task formulation and dual-benchmark design are groundbreaking.", "was_revised": false, "final_justification": "", "rebuttal": "## Question1: Threshold sensitivity study\nWe thank the reviewer for the thoughtful question regarding the action latency ceiling and its task dependence.\nTask-specific ceilings are realistic and often reflect physical or system-level constraints.\n\n In real-time environments like StreetFighter, planning latency beyond the 200ms window has diminishing returns, since **actions cannot be executed faster than the environment permits**. This aligns with the principle that planning should not outpace action, which holds in many closed-loop decision systems such as robotics and gaming.\n\n\nIn contrast, tasks like high-frequency trading behave differently: decisions can be executed nearly instantaneously, and latency directly determines priority in the execution queue. In such ranking-dominant tasks, **lower latency is always beneficial**, even far below any implicit ceiling. Our framework accommodates both settings by allowing compression to push latency toward the most beneficial region for the task.\n\nWhile we were unable to build new datasets with stricter or looser latency ceilings due to time constraints, we can indirectly study this question via **hardware variation**. The effective inference latency of each model varies by hardware, and thus changes how close each model comes to the latency ceiling.\n\n\nTo explore this, we evaluate the StreetFighter benchmark on NVIDIA H200, measuring ELO scores of FPX-compressed models across different Qwen3 model sizes. All models are evaluated using the same game rules (200ms ceiling), but their inference latency on H200 differs significantly. The results are shown in Figure R1T1, demonstrating how model hardware combinations influence outcomes relative to the ceiling. The reasoning that Qwen3-14B fp8 is the top winner is that the **latency is near 200ms at H200**. Though 4B and 8B models are faster than 14B models, they are not the top winners due to the **gaming latency ceiling**.\n### R1T1 StreetFighter with Qwen2.5 models on H200\n| Rank | Model Size | Bitwidth | Score | \n|------|------------|----------|-------------------|\n| 1    | 14B_8bit   | 8        | **0.824**         | \n| 2    | 32       | 16       | **0.549**         | \n| 3    | 14B        | 16       | **0.315**         | \n| 4    | 4B_8bit    | 8        | **0.299**         | \n| 5    | 32B        | 16       | **0.114**         | \n| 6    | 8B         | 16       | **0.025**         | \n\n## Question2: Does FPX maintain efficacy for mixture-of-experts architectures?\nWe thank the reviewer for the interesting question regarding FPX compatibility with mixture-of-experts (MoE) architectures.\n\n To evaluate this, we conducted an initial experiment using Qwen3-30B-A3B, a MoE model with 128 experts (2 active per forward pass). We applied our FPX calibration procedure as usual, assigning different precision levels to transformer blocks and expert layers independently. The experiment was run on 3 RTX 5090 GPUs, and follows the same evaluation setup used for the standard FPX StreetFighter benchmark. We found that prefilling time for such large moe models on 5090 is too time-consuming. So we apply prefill caching with FPX as well. Also, to embed FPX with MOE models, we see all experts in FFN layers as one united operator. We rank and compress them together at the same time.\n\nThe results are shown below in Table R1T1, and demonstrate that FPX continues to provide strong latency–quality trade-offs even in MoE models.\n\n### Table R1T2: MOE model on StreetFighter\n| Model |      Bitwidth Avg      |   Win Rate vs. FP16  |\n| :----------------------------: | :---------------------: | :---------------------: |\n| Qwen3-30B-A3B(ours)                 |           6.4                |    92.5%            |\n| Qwen3-30B-A3B                 |           8                |    82.5%            |\n| Qwen3-30B-A3B                 |           16                |    50%            |\n\nThese results demonstrate that FPX remains effective in the MoE setting, providing a substantial improvement in win rate (92.5%) compared to both FP16 and FP8 baselines, while achieving lower average bitwidth.\n This suggests that FPX can exploit the sparsity and modularity of MoE architectures to apply precision selectively, further enhancing both latency and decision quality.\nWe consider extending FPX with expert-aware calibration and dynamic routing–precision interaction an exciting direction for future work.\n\n## Question 3 and limitation: multiple FPX agents competition\nWe appreciate the reviewer’s question and would like to clarify that our current evaluation **already models a multi-agent competitive setting**, but has not been tested only with a single agent.\n\n In the StreetFighter benchmark, there are **two agents** competing with each other. Each agent controls a player in the game and runs inference **independently** on its own hardware and model. \n\nThis naturally reflects a “delayed arms race” dynamic: faster agents react more quickly and gain the upper hand, while slower ones are penalized. In fact, the incentive to reduce latency to outcompete an opponent is a key driving force behind our formulation.\nThat said, we acknowledge that fully simulating multiple FPX agents running in parallel on heterogeneous hardware, with timing feedback loops and cross-agent adaptation, would further enrich this evaluation. Due to current hardware constraints, **we leave multi-agent FPX interaction under joint optimization over two players as an exciting direction for future work**.\n\nWe believe that when multiple agents are deployed collaboratively or competitively, the latency–accuracy trade-off landscape becomes significantly more complex. In such settings, each agent’s optimal behavior may depend not only on its own constraints but also on the timing and strategy of others. This calls for the development of more sophisticated, coordination-aware algorithms that can adaptively control each agent’s compression level at runtime based on external observations or shared signals."}, {"review_id": "Fku5h9TnvQ", "reviewer": "Reviewer_2oJ3", "summary": "The authors address the latency–quality trade-off when deploying LLM agents in real-time environments. They (i) formalise latency-sensitive agent decision tasks, (ii) introduce two benchmarks—HFTBench for high-frequency trading and a StreetFighter gaming setup—and (iii) propose FPX, an adaptive mixed-precision framework that mixes FP8/FP4 at the layer level. FPX delivers smoother latency–accuracy Pareto curves than fixed-precision baselines, improving daily trading yield by ≈26 % and game win-rate by ≈80 % on their tasks .", "questions": "- While the paper provides a problem formulation, it lacks a unified evaluation metric. Metrics such as yield and latency or ELO and latency, are reported in separate columns. Do you consider introducing a normalized metric like reward ∕ latency to capture the trade-off?\n\n- The proposed FPX relies on manual grid search over the compression ratio γ in discrete 0.1 steps. Can the agent be extended to support self-adaptation of γ at runtime?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- This work is the first to address latency-aware evaluation for LLM agents, offering a principled formulation of the problem \n\n- The proposed FPX method outperforms fixed-precision baselines (FP8/FP16) across both tasks in terms of latency-normalized reward metrics, demonstrating a favorable speed–quality trade-off.\n\nWeaknesses\n\n- The StreetFighter evaluation is based on only 40 games per model pair, which may be insufficient for reliable ELO estimation\n- Although the checklist (Q7) states that error bars are included, Table 1 and Table 2 present only point estimates, with no variance or confidence intervals reported", "was_revised": false, "final_justification": "", "rebuttal": "## Question1: Do we consider introducing a normalized metric like reward ∕ latency to capture the trade-off?\nWe thank the reviewer for the insightful suggestion.\n\nWe agree that having a unified evaluation metric is appealing in theory. However, in our setting, **the task-specific reward already implicitly incorporates latency**, making a normalized “reward/latency” ratio redundant and potentially misleading.\n\nSpecifically:\n1. In HFTBench, the **daily yield is directly impacted by latency through the price decay function** (Equation 5 in our paper), where slower decisions result in worse execution prices and thus lower profit.\n2. In StreetFighter, the **win rate (and resulting ELO) reflects the agent's ability to respond quickly and accurately in a real-time environment**. Again, latency is implicitly encoded in the game outcome.\n\nTherefore, both reported metrics, daily yield and ELO, are already **final metrics that reflect the integrated effect of latency and accuracy for their respective tasks**.\n\nMoreover, we believe that a single normalized metric **may not generalize well across tasks**, since different latency-sensitive applications have **different latency-reward dynamics** (discussed in Section 5.3 and 5.4). A naive reward/latency ratio may fail to capture these nuances.\n\nWe agree that defining a general-purpose trade-off metric is an interesting direction for future work, especially for comparing across domains.\n\n## Question2: Can the agent be extended to support self-adaptation of γ at runtime?\n\nWe thank the reviewer for this valuable question. We confirm that **FPX can be extended to support runtime adaptation of the compression ratio γ**.\n\nAt a system level, this is feasible by storing multiple precision variants of the model or key layers, and dynamically selecting between them during inference. This allows the agent to change γ on the fly depending on the environmental context.\nTo demonstrate this, we implemented a simple rule-based policy in the StreetFighter benchmark:\n\n1. When the two players are far apart, precision is less critical, so the agent uses the faster FP8 model.\n2. When the players are close (within two action steps), the agent switches to the FPX-compressed model (γ = 0.3), which offers higher accuracy for fine-grained combat decisions.\n\nWe evaluated this self-adaptive model against fixed-γ models using 40-round tournaments, and report the number of wins in Table R2T1.\n\n### Table R2T1: Ablation study of self-adaptive and fixed trade-offs.\n| Model parameter size |      Self-adpt Win Round      |  Fixed Win Round      | \n| :----------------------------: | :---------------------: | :---------------------: |\n| 3B                           |           25                |           15              |\n| 7B                           |            22              |           18              |\n| 14B                           |            20               |           20              |\n\nWe found out that **self-adaptive is beneficial to most of the models**. But when the latency of response is far from latency action ceiling(200ms), the benefit of adaptive strategy is eliminated.\n\n## Weakness1: The StreetFighter evaluation is based on only 40 games per model pair, which may be insufficient for reliable ELO estimation.\nWe appreciate the reviewer’s concern regarding the statistical reliability of our ELO estimation.\n To address this, we conducted an extended evaluation using 400 rounds of head-to-head matches per model pair on the StreetFighter benchmark.\n\nThe updated results are shown below in Table R2T2. We observe that:\n\n1. The relative ranking of models remains consistent with our original 40-round results.\n2. The variance across blocks of 40 games is low, indicating that our earlier estimates were already directionally stable.\n\n### Table R2T2: ELO Scores Based on 400-Game Tournaments (StreetFighter)\n| Model parameter size |      Bitwidth Avg      |  ELO(400 rounds)   |  ELO(40 rounds)   |\n| :----------------------------: | :---------------------: | :---------------------: | :---------------------: |\n| 3B(ours)                 |           6.8                |           5.91              |    5.93                      |\n| 7B（ours)               |            7.2              |           2.31              |      2.33                    |\n| 3B                           |            8               |           2.23              |      2.19                         |\n| 3B                           |            16               |           0.27             |      0.25                         |\n| 7B                           |            8               |           -0.44             |      -0.44                        |\n| 1.5B                       |            8               |           -1.27             |      -1.25                        |\n\n## Weakness2: Error bar missing.\nWe thank the reviewer for pointing this out. You are correct. Our initial submission reported single-trial results without error bars, and we acknowledge this mistake in answering Checklist Question 7.\n\nTo address this, we have now repeated StreetFighter experiments 10 times(400 rounds in total), HFTBench 3 times, and report mean ± standard deviation across these runs. We will incorporate these revised results (with appropriate error bars) into the final version of the paper.\n\n### Table R2T3: StreetFighter ELO Scores with std error\n| Model parameter size |      Bitwidth Avg      |   ELO(40 rounds)   |\n| :----------------------------: | :---------------------: | :---------------------: |\n| 3B(ours)                 |           6.8                |    5.92 ± 0.13            |\n| 7B（ours)               |            7.2              |       2.31 ±  0.17         |\n| 3B                           |            8               |         2.21  ±  0.05        |\n| 3B                           |            16               |       0.26  ±  0.09        |\n| 7B                           |            8               |     -0.44   ±   0.12         |\n| 1.5B                       |            8               |      -1.26   ±  0.07          |\n### Table R2T4: HFTBench Daily Yield with std error\n| Model parameter size |      Bitwidth Avg      |   Daily Yield(%)   |\n| :----------------------------: | :---------------------: | :---------------------: |\n| 14B(ours)                 |           7.2                |    26.59 ± 3.21           |\n| 14B                        |            8              |       22.14 ± 2.03        |\n| 14B                           |            16               |         18.01 ± 3.95        |\n| 7B                           |            16               |       -3.28  ±  1.37        |\n| 7B                           |            8               |     -6.93   ±   2.38         |\n| 7B                       |            8               |      -10.02   ±  3.81          |"}, {"review_id": "ng2USL6tIq", "reviewer": "Reviewer_sJNR", "summary": "This paper presents a timely and valuable investigation into the latency-accuracy trade-off for LLM-based agents in dynamic environments. The work addresses a critical gap in deploying LLMs for latency-sensitive applications like high-frequency trading and competitive gaming. The novel benchmarks (HFTBench and StreetFighter) and adaptive framework constitute a significant contribution to the field.", "questions": "Given the importance of calibration data choice, how sensitive is FPX to domain shift (e.g., calibrating on financial/gaming data)? A simple ablation study demonstrating this sensitivity would provide valuable practical insights and significantly benefit the community.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n1.\tProblem Significance: The paper convincingly establishes the importance of latency-sensitive decision-making for LLM agents (§1, §3), highlighting underexplored domains (trading/gaming) where delays directly impact rewards. This addresses a critical real-world deployment challenge.\n2.\tBoth the proposed benchmark, framework and conclusion contribute to the community.\n\n\n\nWeakness\nAlthough the authors state that their evaluation is limited to the Qwen2.5 modes (ranging from 1.5B to 14B parameters) to ensure fair comparison and reduce search-space complexity, this raises significant concerns about the framework’s generalizability.", "was_revised": true, "final_justification": "The authors provided additional experiments that directly address my two main concerns.\n\n1. Robustness to calibration-data shift:  \n   • They re-calibrated FPX on the *Torchlight2* corpus (very different from Wikitext-2) and reported high Spearman correlations of the layer-sensitivity rankings (0.92 for Qwen-14B, 0.86 for Gemma-12B).  \n   • Downstream performance differences were small (≤ 2 pp win-rate on StreetFighter; ≤ 3 pp daily-yield on HFTBench), indicating that FPX is largely insensitive to domain shift in the calibration data.  \n\n2. Generalisability beyond Qwen models:  \n   • New results on the Gemma-3 family (4 B & 12 B) show consistent ELO improvements over FP8/FP16 baselines, confirming that FPX transfers to a distinct architecture.  \n\nThese additions satisfactorily resolve my reservations about generality. I therefore keep my overall recommendation as *borderline-accept* (score = 4) but with strengthened confidence that the work will be useful to the community.", "rebuttal": "## Question1: How sensitive is FPX to domain shift?\n\nWe thank the reviewer for the insightful question regarding the robustness of FPX calibration across data domains.\nTo evaluate this, we conducted an ablation study comparing the layer-wise compression sensitivity rankings εₗ (defined in Algorithm 1 in our paper) generated by FPX under two distinct calibration datasets: Wikitext-2 (used in our main paper) and Torchlight2, a corpus of in-game dialogues and narrative text extracted from the action RPG [Torchlight II](https://github.com/hmi-utwente/video-game-text-corpora/tree/master/Torchlight%20II). Torchlight2 offers a very different text distribution compared to Wikitext, reflecting domain-specific content from real-time game environments.\n\nFor both Qwen2.5-14B and Gemma3-27B, we observed **high Spearman rank correlations** between the layer rankings derived from the two datasets on the same model, **0.92 and 0.86**, respectively. This suggests that FPX yields stable and **transferable layer-wise precision assignments across domains**.\n\nTable R3T1 summarizes this result. In addition to correlation scores, we also compare downstream performance using the FPX strategy calibrated on Wikitext versus Torchlight2. We observe only **minor differences** in downstream task outcomes (e.g., win rate and daily yield), further reinforcing the **robustness of our approach to calibration domain shifts**. Win rate here denotes the outcome of a 40-round head-to-head match between two compressed models with different calibrated dataset. Each was calibrated using a different dataset (Wikitext vs. Torchlight) on the StreetFighter benchmark.\n\n### Table R3T1: Ablation study of calibration datasets.\n| Model |      correlation      | Win rate| Daily Yield(wikitext) |Daily Yield(TorchLight)|\n| :------: | :---------------------: | :----------: | :------------------------: |:---------: |\n| Qwen-14B |       0.92       |    50%     |    26.52    | 24.17|\n| Gemma3-12B |   0.86   |     52.5%     |    31.30    | 33.05|\n\nWe believe this empirical robustness is theoretically grounded:\n1. The quantization error in low-bit inference is primarily governed by the range and distribution of activations and weights in each layer.\n2. While input content may vary across datasets, the distributional characteristics of internal activations remain consistent per layer, especially in large pretrained LLMs.\n3. Furthermore, the gap between FP8 and FP4 is large, and dominates small shifts in activation variation across domains. Thus, the ranking of compression tolerance is preserved.\n\nIn contrast, sparsity-based compression may be more sensitive to input structure or task semantics. However, we intentionally avoid sparsity-based methods in this work due to their limited practical speedup, especially under realistic kernel implementations. In many latency-sensitive tasks, even perfect dynamic sparsity selection may not outperform well-optimized dense FP8 kernels.\n\nWe view extending FPX to other compression paradigms and better calibration algorithms (e.g., structured sparsity) as an important avenue for future work.\n\n## Weakness: Evaluation limited to the Qwen2.5 Models\nWe thank the reviewer for raising this important concern regarding generalizability. To address this, we provide ELO ranking results for the **Gemma3 model family on the StreetFighter benchmark**. The evaluation setup follows exactly the same procedure described in Section 5 of the paper:\n\nWe calibrate each model using the Wikitext dataset, apply FPX with compression ratio γ varying from 0.0 to 1.0 in steps of 0.1, and identify the best-performing γ for each model. Each model is then evaluated by playing 40 rounds of matches in the competitive tournament setting.\n\nThese results demonstrate that FPX remains effective on non-Qwen models, improving both 4B and 12B variants of Gemma3 in terms of downstream win-rate (as reflected by ELO score). **This supports the generalizability of our framework across different architectures**.\n\n### Table R3T2: Ablation study of Gemma3 family on Street Fighter.\n| Model parameter size |      Bitwidth Avg      | ELO score| \n| :----------------------------: | :---------------------: | :----------: |\n|4B(ours)                        |          6.8                 |    4.31       |\n|4B                                |             8                   |    2.19       |\n|12B(ours)                               |             16                   |    0.14       |\n|  4B                     |             6.4                   |    -1.32       |\n|12B                       |             8                   |    -2.14       |\n|12B                       |             16                   |    -2.97       |"}, {"review_id": "W3Y3FaUUt3", "reviewer": "Reviewer_oRvQ", "summary": "This work investigates real-time sequential decision making by LLM-based agents,\na scenario where both latency and quality of actions matter.\nIt first proposes two benchmarks, one for high-frequency trading and the other for the StreetFighter game.\nThen, it proposes FPX, a quantization method that allows choosing different quantization levels for adapting the same LLM to the needs of different tasks.\nEmpirical evaluations are conducted for such time-sensitive environments, where models of different sizes and quantization levels are compared.", "questions": "I'd suggest adding some concrete examples of prompts and responses in the proposed benchmarks. Is the LLM prompted to answer with the action directly (with a response length of just one or two tokens) for each action, in order to achieve latency within hundreds of ms?", "limitations": "Yes, the authors have discussed limitations of this work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n\n* This work studies a nice topic (i.e., latency-sensitive decision making by LLM agents) and proposes interesting benchmarks, pointing to promising directions for further research.\n\n* This work provides insights for LLM-based agent in latency-sensitive scenarios. One example is Eq (5), which points out that the reward depends on the rapidly changing environment. Another insight is that smaller latency for each action implies faster reaction to environmental change, and the ability to execute more actions within a fixed period of time, which can be crucial for latency-sensitive scenarios.\n\n* The proposed FPX mixed-precision quantization method allows the same model to adapt to different tasks by selecting the appropriate quantization levels.\n\n\n\n\n**Weaknesses:**\n\n\n\n* One limitation of FPX, as acknowledged by the authors in Sections 5.4 and 6, is that its hyperparameter gamma need to be tuned carefully for each <task, model> tuple  in order to achieve satisfactory performance.\n\n* The authors answer \"yes\" to Question 7 of the checklist regarding statistical significance. This is incorrect, as each number in the experiments is the result of a single trial (unless I missed something).\n\n\n\n* Writing can be further polished, for example:\n    * Typo: Line 261, \"lantecy\" --> \"latency\"\n    * Line 85 in the introduction where the authors claim the main contributions of this work: to be more precise, it should be \"LLM-based agent\" rather than just \"agent\".\n\n\n\n**Disclaimer:** \n\nDue to my limited familiarity with certain literature, I'm not sure about\n* whether this work is indeed \"**the first** to systematically formulate and investigate the latency–quality trade-off in the context of real-time decision-making by LLM agents\" (Line 60);\n* how novel and original is Algorithm 1 (which seems fairly natural and straightforward);\n* how reasonable and realistic is the benchmark design / problem formulation for quant trading (Lines 173 - 180).\n\nI believe that the AC and other reviewers can make better judgments on this aspects than I do.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I would maintain my rating.", "rebuttal": "## Question1: Concrete example of prompts\nThanks for your suggestion! Sure, we force LLM to answer with action directly for now to decrease the decoding step for better latency. The max generated token is set to 32 tokens.\nHere we provide an example prompt of our StreetFight Benchmark.\n```\nYou are the best and most aggressive Street Fighter III 3rd strike player in the world. \nYour character is {character}. Your goal is to beat the other opponent.\nif you are far from opponent, use Move Closer and Fireball more often.\n        If you are close to opponent or already move closer, try to use Punch and Kick more often.Megapunch, Hurricane, and other combinations uses more time but are more powerful. Use them when you are close to opponent and you are getting positive scores or winning. If you are getting negative scores or losing, try to Move away and use Kick.\nThe moves you can use are:\n{move_list}\n----\nExample 1:\nContext:\nYou are very far from the opponent. Move closer to the opponent. Your opponent is on the left.\nYour last action was Medium Punch. The opponent's last action was Medium Punch.\nYour current score is 108.0. You are winning. Keep attacking the opponent.\n\nYour Response:\n- Move closer\n- Move closer\n- Low Kick\n\nExample 2:\nContext:\nYou are close to the opponent. You should attack him.\nYour last action was High Punch. The opponent's last action was High Punch.\nYour current score is 37.0. You are winning. Keep attacking the opponent.\nYour Response:\n- High Punch\n- Low Punch\n- Hurricane\nNow you are provided the following context, give your response using the same format as in the example.\n\n```\nOne of the answers generated by Qwen2.5-3B is:\n```\n- Low Punch\n- High Punch\n- Move Away\n\n```\nPrompts are specially designed for HFTBench as well.\n## Weakness1: The Hyperparameter gamma needs to be tuned for each model and task.\nWe appreciate the reviewer’s observation and agree that the compression ratio γ is a task- and model-dependent hyperparameter that influences the latency–accuracy trade-off.\n\n However, we would like to emphasize that our primary contributions lie in **(1) defining and constructing realistic latency-sensitive benchmarks, and (2) systematically evaluating the trade-offs across models, bitwidths, and γ values on these benchmarks (see Sections 3 and 5).**\n\nBy exposing the Pareto frontier for various <task, model> configurations, our work provides the **first comprehensive empirical understanding of how inference latency interacts with decision quality in real-time agentic settings**. This understanding is crucial and was **previously missing from the literature**.\n\nWhile automated γ tuning or runtime adaptation is an important direction, we explicitly position this as **future work** (Section 6). Our goal in this paper is to **establish the problem setting, introduce an interpretable, practical framework (FPX), and demonstrate its effectiveness across representative real-world scenarios (trading and gaming)**. This lays the foundation for more sophisticated tuning or control mechanisms in follow-up work.\n\n## Weakness2: Statistical evaluation.\nWe thank the reviewer for pointing this out. You are correct. Our initial submission reported single-trial results without error bars, and we acknowledge this mistake in answering Checklist Question 7.\n\nTo address this, we have now repeated experiments with multiple independent runs and report the mean ± standard deviation across these runs. We will incorporate these revised results (with appropriate error bars) into the final version of the paper. StreetFighter runs 10 times. Each time consists of 40 rounds of fighting for each model pair. And we run HFTBench 4 times indepdently.\n### Table R4T1: StreetFighter ELO Scores with std\n| Model parameter size |      Bitwidth Avg      |   ELO(40 rounds)   |\n| :----------------------------: | :---------------------: | :---------------------: |\n| 3B(ours)                 |           6.8                |    5.92 ± 0.13            |\n| 7B（ours)               |            7.2              |       2.31 ±  0.17         |\n| 3B                           |            8               |         2.21  ±  0.05        |\n| 3B                           |            16               |       0.26  ±  0.09        |\n| 7B                           |            8               |     -0.44   ±   0.12         |\n| 1.5B                       |            8               |      -1.26   ±  0.07          |\n\n### Table R4T2: HFTBench Daily Yield with std\n| Model parameter size |      Bitwidth Avg      |   Daily Yield(%)   |\n| :----------------------------: | :---------------------: | :---------------------: |\n| 14B(ours)                 |           7.2                |    26.59 ± 3.21           |\n| 14B                        |            8              |       22.14 ± 2.03        |\n| 14B                           |            16               |         18.01 ± 3.95        |\n| 7B                           |            16               |       -3.28  ±  1.37        |\n| 7B                           |            8               |     -6.93   ±   2.38         |\n| 7B                       |            8               |      -10.02   ±  3.81          |\n\n## Disclaimer:\nWe appreciate the reviewer’s disclaimer and the thoughtful attention to prior work.\n\n### 1. We would like to clarify that, to the best of our knowledge, our work is indeed the first to systematically formulate and investigate the latency–quality trade-off for LLM agents operating in real-time environments.\n\nThis assessment is also supported by other reviewers: several have explicitly acknowledged the novelty of our task formulation and benchmark design, with one reviewer describing the dual-benchmark setup as “groundbreaking.”\n### 2. While Algorithm 1 may appear straightforward in isolation, it plays a central role in enabling our broader contribution, a practical and adaptive framework for latency-sensitive LLM inference.\n\nPrior work on model compression and mixed-precision typically focuses on **static throughput optimization or near-lossless accuracy**. In contrast, our FPX framework, summarized in Algorithm 1, introduces a **continuous, layer-wise mixed-precision assignment strategy** that is explicitly guided by **task-specific latency–reward trade-offs (see Section 4.1–4.2)**.\n\nWhat distinguishes our work is that latency is not just a systems constraint but is explicitly embedded in the reward function, as formalized in Equation 5 in our paper. Our benchmarks (HFTBench and StreetFighter) provide the **first concrete settings where latency directly impacts agent reward**, enabling us to evaluate FPX under realistic conditions.\n\nWe believe this integration of calibrated precision control with reward-sensitive deployment constitutes both a novel problem formulation and a practical contribution that opens new directions in LLM agent optimization.\n\n### 3. We appreciate the reviewer’s interest in the realism of our HFTBench design.\n We would like to clarify that our high-frequency trading benchmark is **not a synthetic simulation, but rather a realistic backtesting environment** constructed from real-world per-second historical market data collected via Polygon.io (see Section 3.2, Lines 173–180).\nAgents in HFTBench receive synchronized 1-second tick data, including price levels and bid–ask spreads, and must decide whether and when to act. Crucially, execution prices are assigned using a linearly decaying function of response latency, which mimics the real-world queue-based price execution model common in modern exchanges. This design allows us to faithfully capture the interaction between latency and reward—where faster agents consistently gain more favorable prices.\n\nWe believe this makes HFTBench a practically meaningful and realistic evaluation environment for latency-sensitive financial agents. It also opens the door for future extensions incorporating real multi-agent interaction or more granular tick-level data."}, {"review_id": "ofW3BmF3rd", "reviewer": "Reviewer_mdup", "summary": "This paper investigates the trade-off between inference latency and output quality in LLM-based agents, especially under real-time constraints. The authors identify that many applications (e.g., high-frequency trading, real-time gaming) require rapid decision-making, where even minor latency can degrade performance. To address this, they introduce two latency-sensitive benchmarks: HFTBench (for high-frequency trading) and StreetFighter (for competitive gaming). To optimize performance across diverse latency requirements, the authors propose FPX, an adaptive mixed-precision inference framework. FPX adjusts both model size and bitwidth (mixing FP8 and FP4) dynamically, applying lower precision to more quantization-tolerant layers. Experiments show that FPX achieves improved win rate and financial yield compared to static baselines, with up to 80% higher win rate in StreetFighter and 26.52% yield improvement in HFTBench.", "questions": "1. What is the most practically beneficial factor in your framework that enables LLM agents to operate more effectively in a border range of latency-sensitive tasks? \n2. How does FPX perform on real hardware that supports FP4 inference? Are there actual deployment results beyond simulation?\n3. Can you discuss whether any new low-precision design choices were made that specifically benefit latency-critical scenarios beyond standard mixed-precision strategies?\n4. Could you elaborate on how the precision configuration is applied during inference? Is it dynamically adjusted per input, per timestep, or fixed at deployment time? How feasible is this control pipeline in real-world systems with limited runtime flexibility?", "limitations": "Yes, limitation is clearly stated in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n+ Well-motivated by latency-sensitive applications like trading and gaming.\n+ Introduces two task-specific, real-time benchmarks.\n+ Empirically demonstrates that adjusting the latency–accuracy balance can improve downstream performance.\n\nWeaknesses\n- It is unclear whether the observed performance improvements translate to practical deployment gains in real-world latency-sensitive decision systems.\n- FP4 is only simulated, and the impact of actual hardware behavior is not evaluated, especially across non-Blackwell architectures.\n- The adaptive quantization method closely follows known practices (e.g., layer-wise calibration), with limited novelty in the core algorithmic contribution beyond application to LLM agents.", "was_revised": true, "final_justification": "Nice rebuttal. It resolves my concern. So I change my score to align with the other reviewers.", "rebuttal": "We thank the reviewer for your valuable suggestions for our paper. We would like to confirm that **all of our experiments are conducted on NVIDIA RTX 5090 GPUs with FP4 hardware support but not simulated**.\n## Question1: What is the practically beneficial factor of FPX?\nThe most practically beneficial factor in our FPX framework is its ability to provide a **continuous and fine-grained trade-off between generation accuracy and latency (illustrated clearly in Figure 1(a)).** Unlike previous compression algorithms that only support **discrete or very limited accuracy–latency trade-off points**, FPX allows **adaptive, layer-wise precision selection (FP8 and FP4 combinations in our experiments) tailored explicitly for latency-sensitive tasks**.\n\nIn latency-sensitive tasks, latency directly impacts the downstream reward and evaluation metrics(in section 3). **Different time sensitive tasks require different latency and accuracy to achieve the optimal point.** Hence, being able to apply a carefully tuned lossy optimization strategy that precisely meets task-specific demands is crucial. For example, gaming scenarios can benefit substantially from more aggressive precision reduction (lower latency at the cost of accuracy), while trading applications demand higher accuracy at moderately reduced latency. For example, in Table 2 in our paper, the smaller model with progressive quantization performs the best in StreetFights. While at the same time, bigger models with near-lossless perform well at HFTBench.\n\nFPX tackles this challenge by automatically calibrating and selecting near-optimal precision settings, **ensuring strong LLM performance across diverse latency-sensitive tasks**.\n## Question2: How does FPX perform on real hardware?\nWe understand the concern of the reviewer of our framework, but we would like to clarify that **none of the FP4 results in our paper are simulated**. All FP4 inference results, including latency and accuracy measurements, **were obtained from actual hardware experiments conducted on NVIDIA RTX 5090 GPUs with cuda kernel written by ourselves (as stated in Section 5.1 and Table 4 of our paper)**. NVIDIA's Blackwell architecture officially supports FP4 inference, as confirmed in the Blackwell documentation. Rubin, NVIDIA’s next-gen hardware, is confirmed to support FP4 inference as well at GTC2025. FP4 is going to be a common practical low-bit precision like FP8.\n## Question3: What is the new low-precision design of our paper?\nWe propose two key innovations beyond standard mixed-precision strategies, tailored for latency-critical applications:\n### 1. Offline Calibration for **Continuous Latency-Accuracy Trade-offs** (Section 4.2, Algorithm 1):\n\n We introduce a new offline calibration algorithm that estimates per-layer quantization error under FP4/FP8 execution by measuring the relative degradation in activation outputs. Based on this, we compute a precision assignment δ(l) that selectively applies FP4 to compression-tolerant linear layers and FP8 elsewhere. \n\nPrior calibrations (e.g., SmoothQuant) focus on **near-lossless optimization**, not on **adaptive latency–accuracy trade-offs required by real-world latency-aware agents**. In contrast, FPX supports **fine-grained, continuous control** over latency and accuracy, beyond fixed global settings in past approaches. This enables **task-specific adaptation**, identifying Pareto-optimal configurations (e.g., γ = 0.2 for HFT, γ = 0.3 for Street Fighter), as shown in Table 2 and Section 5.4.\n\n### 2.Custom Mixed Precision Inference Kernel with Automatic Dtype Fused Output Transfer (Implementation detail in Section 4.1):\nMixed-precision inference introduces **overhead** when **consecutive layers use different bitwidths**. For example, an FFN where the first layer runs in FP8 and the second in FP4. Prior implementations typically output FP16 and perform a separate conversion for the next layer, **incurring latency**, especially in single-batch inference.\n\nTo address this, we implement a custom FP8/FP4 kernel that fuses output quantization with memory storage, directly converting SMEM results into the target dtype (FP8/FP4) before writing to HBM. This eliminates separate dtype conversion and memory copy passes, **improving inference speed by at least 10% in mixed-bitwidth layers on RTX 5090.** Such fusion is essential in latency-critical scenarios where small overheads accumulate across many layers. FP4 is a new feature recently supported by NVIDIA this year. To our best knowledge, we are the **first** to implement kernel fusion between FP8 and FP4.\n### Table R5T1: Ablation study of kernel fusion on RTX5090.\n| Model |      Bitwidth Avg      | Fusion Speed up| \n| :---: | :--------: | :------: | \n| Qwen2.5-14B |       8       |    100%     | \n| Qwen2.5-14B |   7.6   |     112.7%     |  \n| Qwen2.5-14B |   7.2   |     119.3%     |  \n| Qwen2.5-14B |   6.8   |     123.4%     |  \n\n\n### 3. In addition, a core contribution of our work is the introduction of the first set of benchmarks specifically designed for latency-sensitive agent tasks, HFTBench and StreetFighter.\n\nThese benchmarks allow us to systematically study the **fundamental trade-off between latency and accuracy**, which is often overlooked in prior LLM evaluations. By doing so, we highlight a previously **underexplored challenge**: that in real-world deployments (e.g., trading, gaming), the performance of LLM agents is **not solely determined by output quality, but also by their response time**. This insight poses **a new systems and algorithm-level problem**, which is how to dynamically balance accuracy and latency under strict timing constraints.\n\n## Question4: FPX deployment and fitness in real-world system.\nAs discussed in Algorithm 1 of the paper, FPX first performs **offline calibration** on wikitext dataset to rank each layer's sensitivity to FP4 compression and get the precision assignment function. It then **adjusts the compression ratio γ(percentage of FP4 layers) according to the function** to achieve the desired latency-accuracy trade-off by running and calibrating on real sensitive task benchmarks. The parameter γ, as well as the precision assignment function, **is fixed at inference time.**\n\nThe deployment of FPX, combining FP8 and FP4 precision, is applicable to numbers of devices: it can run on any hardware supporting FP8 and FP4 inference. Currently, NVIDIA's Blackwell GPUs and certain AMD edge devices support FP4, and NVIDIA's upcoming Rubin GPUs will also provide FP4 compatibility. **This shows that FP4 will be a common low bit precision supported by hardware like FP8.**\n\n## Weakness1: Practical deployment gain on real-world systems.\nWe would like to clarify that all benchmarks in our benchmarks are grounded in **real-world data** designed to closely reflect practical deployment settings and our experiments are conducted on **real hardware**. \n\nSpecifically, StreetFighter is implemented using the DIAMBRA simulation engine, which mirrors the behavior of real-time fighting games and is deployable on personal computers. HFTBench, as detailed in Section 3.2, is constructed using real historical per-second market data from Polygon.io (e.g., NVDA, AMZN on 2024-08-05) and simulates trading under realistic latency-sensitive order execution rules.\n\nThis is a **backtesting-based evaluation**, not a synthetic simulation, where agents receive time-synchronized observations and are ranked by response latency, with execution prices decaying linearly as in **real-world** limit order books.\n\nWhile real-world latency-sensitive systems **vary in dynamics and constraints**, this motivates FPX to be **adaptive and general-purpose**. Rather than relying on fixed precision or handcrafted settings, FPX supports **continuous latency–accuracy trade-offs via tuning γ**. This flexibility allows engineers to adjust configurations for the latency budget of any target system, including trading, gaming, or robotics mentioned by other reviewers as well.\n\nIn short, our study not only uses realistic benchmarks based on real data, but also proposes a solution, FPX, that is broadly applicable across heterogeneous latency-sensitive domains with performance improvements.\n\n## Weakness2: FP4 is simulated.\nWe would like to clarify that FP4 inference in our experiments is **not simulated**.\n\n All results reported are obtained on RTX 5090 GPUs, which natively support FP4 inference. Our FP4 kernels are implemented and executed on the device, not emulated or simulated in software.\n\nTo further demonstrate cross-hardware generalizability, we additionally evaluate FPX on NVIDIA GB200, which also supports native FP4 execution. As shown in Table R5T2, FPX configurations with bitwidth averages below 8 continue to outperform the FP8 baseline, achieving up to 75% win rate on StreetFighter, validating that our method is effective **across multiple FP4-enabled architectures**.\n### Table R5T2: StreetFighter Ablation study of GB200.\n| Model |      Bitwidth Avg      | Win rate vs. FP16| \n| :------: | :-------------: | :--------: | \n| Qwen2.5-32B(ours) |       7.2       |    75%     | \n| Qwen2.5-32B(ours) |   7.6   |     70%     |  \n| Qwen2.5-32B |   8   |     62.5%     |  \n\n\n## Weakness3: Method lacks of novelty.\nWe respectfully disagree with the claim that our method lacks algorithmic novelty. While FPX builds on layer-wise calibration, its key innovation is in **adapting this to real-time agent settings**, where latency directly affects the reward (see Equation 5).\n\nWe propose a **continuous latency–accuracy trade-off mechanism** via a tunable parameter γ, enabling **inference-time latency control without retraining**, which is a feature **not supported** by prior quantization methods focused on static LLM serving.\n\nWe are also extending FPX to domains like **autonomous driving**, where early results show its effectiveness in managing sensor-to-action delay, highlighting its value as a general-purpose adaptive inference framework."}]

Author rebuttal:

Meta-review: This work contributes the first study on the tradeoff between latency and response generation for downstream task completion, together with the introduction of a couple benchmark environments for future investigation. The reviewers unanimously appreciated the framework and contributed insights, which justifies acceptance at this time.
Decision: Accept (spotlight)