Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Authors: Peijie Dong
Zhenheng Tang
Xiang Liu
Lujun Li
Xiaowen Chu
Bo Li
Venue: ICML 2025 poster (2025)
Keywords: Agent
LLM
Quantization
Pruning
Primary area: deep_learning->large_language_models
TL;DR: We present Agent Compression Benchmark to evaluate the Agentic Performance of Compressed LLMs.

Abstract: Post-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmarks focus narrowly on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities—workflow, tool use/function call, long-context understanding and real-world application. We introduce the Agent Compression Benchmark (ACBench), the first comprehensive benchmark for evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1) 12 tasks across 4 capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval), (2) 4-bit quantization (GPTQ, AWQ) and 50% pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B), standard (Qwen2.5-7B), and distilled reasoning LLMs (DeepSeek-R1-Distill). Our experiments reveal compression tradeoffs: 4-bit quantization preserves workflow generation and tool use (1%--3% drop) but degrades real-world application accuracy by 10%--15%. We introduce ERank, Top-k Ranking Correlation and Energy to systematize analysis. ACBench provides actionable insights for optimizing LLM compression in agentic scenarios, bridging the gap between algorithmic efficiency and real-world applicability.

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Peijie Dong" 1
Zhenheng Tang" 2
Xiang Liu
Lujun Li
Xiaowen Chu Bo Li
Abstract
strated transformative capabilities in domains ranging from code synthesis (Roziere et al.) 2023) and scientific re-
Post-training compression reduces the computa- tional and memory costs of large language mod- els (LLMs). enabling resource-efficien deploy- ment. However, existing compression bench- marks only focus on language modeling (c.g.. perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agen- tic wapabllttes-woorfflov, tool use/function call, long-context understanding and real-world appli- cation. We introduce the Agent Compression Benchmark (ACBench), the first comprehensive benchmark for evaluating how compression im- pacts LLMs' agentic abilities. ACBench spans (1) 12 tasks across capabilities (e.g., WorfBench for workflow generation, Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ) and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B), stan- dard (Qwen2.5 7B-32B), and distilled reason- ing LLMs
search (Li et al. 2024d) to multi-agent collaboration (Wang)
ct al. these advances, their practical deployment remains hindered by prohibitive computational and memory costs (Samsi et al. 2023b: Stojkooiic et al.) 2024). Post-training compres- sion techniques-notaabl pruning (Sun et al. 2024c Dong et al. 2024b) and quantization (Park et al. 2024) Dong et al.
2024 Li ct al. 2023a Wu et al. 2024b). Despite
2024c address this challenge by reducing model size by
up to 4x while preserving performance on standard bench- marks like perplexity and arithmetic tasks (Zhang et al. 2024d).
Existing compression evaluations focus narrowly on single- turn language modeling (e.g.. WikiText-2 perplexity) and natural language understanding (NLU) tasks (e.g.. GLUF accuracy) (Gong et al. 2024 Yang et al.
2024a Wang et al.
2025a Lai et al. 2025). ). However, real world agentic ap-
plications- such as robotic control (Li et al. 2023a) and nancial analytics (Yang al., 2023a demand capabilities that transcend these static benchmarks. Specifically, com- pressed LLMs must retain (1) multi-step planning (e.g., API call sequences in WebShop), (2) long-context coherence (e.g., 40k-token retrieval in Needle ieeeein--ie -n---------- (nn-Haystcck) (3) adaptive reasoning across conversational turns in workflow, and (4) seamless tool integration (e.g., external API). Prior work (Li et al. 2024e) evaluates quantized models on NLU tasks but overlooks the interplay between compression and these agentic capabilities critical oversight given their centrality to deployment scenarios requiring multi-turn in- teractions.
-I--ssiil, Our experi- ments reveal compression tradeoffs: 4-bit quan- tization preserves workflow generation and tool use (1%-3% drop) but degrades real-world ap- plication accuracy by 10%-15% We introduce ERank, Top-k Ranking Correlation and Energy to systematize analysis. ACBench provides action- able insights for optimizing LLM compression in agentic scenarios. The code can be found in
We make a holistic evaluation of both quantized and pruned LLMs using Agent Compression Benchmark (ACBench) to reveal the status quo across three dimensions: (1) Effects of compression on agent capabilities: Whether compressed LLMs still perform well on other essential agent capabili- ties, such as planning, reasoning, tool use, and long-context understanding; (2) Effect of compression on LLMs: We employ three novel metrics to systematically analyze the gap between compressed and uncompressed LLMs ER- ank, Top-K Ranking correlation, and energy. These metries help reveal how compression influences model outputs and
1. Introduction
Large language models (LLMs) (Bronn et al. 2020 Chiang have demon-
et al. 2023 OpenAl
2023
Team
2024a)
'Equal contribution 'The Hong Kong University of Science and Technology (GuangZhou) 'The Hong Kong University of Science and Technology. Correspondence to: Zhenheng Tang <zh- lang.ll@ ust.hk>. Xiaowen Chu <xwchu @ wxwchu@hkusteaeee gzu.dz.cn>>
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Table 1: The benchmarks and model families for evaluation. "Size" denotes the test size, while "Env" denotes the number of environments.
Section
Task & Ability
Benchmark
#Size/ffEnv
Model Family
Plan
553
InternI.M MM-2.5-7B,
Reason
6426
Qwen-2.5-7B,
Retrieve
64226
Sec.
Tool Use/T-Eval
Mistral-7B.
Understand
6753
DeepSeck-R1 BrrSSec--R--Diiil---ueeee -isstiill-Qwen
Instruct
2660
DeepSeek-RI DeepSeck-R1-DistilLLLLmm
Review
487
Function Call Tasks
1803
Qwen-2.5(1 (..BB(1.BB.00 5B-32B), L-2.57L.
Embodied Tasks
4048
Llama 3. 1..t-B.isstral---
Sec.
Workflow Generation
Problem-Solving Tasks
4257
DeepSeek-B R1 RI-Distill Qwen( 7B),
Open-Grounded Tasks
2281
DeepSeek-RI --ii-------iiiiiilaa------
Code Single-Doc QA Multi-Doc QA
Loc, RB-P
1000
NrtvQA, Qusper. MF-en HotporQA, 2WikiMQA, Musique, Dureader
750
800
Internl.M-2.5-777 Qwen-2.5(1.58, 3B,, 7B),
Summarization Few-shot Syathetic
QMSum, MultiNews, VCSum
600
Megrez-3B, MiniCPM-4B,
Sec.
TREC, TriviaQA, SAMSum, LSHT
800
Gemma-2B, Phi-3.5,
PRE, Pcount
600
DeepSeek-RI ---aa-R----aaa-BBB
GSMSK
8000
DeepSeck-RI-Disiiledl1... 7B)
LongGenBench
MMLU
15908
Information Extraction
Ncclle-inthh--Hassttc
40K
Embodied AI
ScienceWorld
90
Qwen-2.5(1.. (tnn-2.5(1.BB-BB)) Mnt.5LLM-2.5-7B..
Jericho
20
Game
Sec.
PDDI.
60
DeepSeek-RR RI D--stiilled- Distilled-Qwen- 1.5B,
Thol-Query
60
Tool Use
DeepSeek-RR Di-t-iilledd-L.aaa Jamm--
Tool-Operation
40
internal representations, providing insights into which com- pression configurations best preserve agent capabilities. (3) Impact of different compression approaches: The rela- tive effectiveness of quantization versus pruning methods, and their potential complementarity. remains unexplored in the context of agent tasks. A comprehensive comparison is needed to guide optimal compression strategy selection.
rank-based techniques to capture the distribution of model logits more efficiently. This tool helps identify the structural changes in the model's decision-making process induced by compression. (2) Top-K Ranking Correlation can mea- sure the ranking consistency between the top-k logits of compressed and uncompressed LLMs, providing insights into how well the compressed model preserves the origi- nal model's 's confidence its predictions. (2) Energy-based Analysis, inspired by Duttof-Distriiution (OOD) detection techniques (Lee al., 2018 Liu et al.) 2020), evaluates the distributional shifts in logit energies between the com- pressed and uncompressed models.
Specifically, we evaluate three categories of L.MMs: (1)
Small language models (<7B parameters) like Phi-3.5 (Ab-
al. 2024), MiniCPM (Hu et al. 2024), and Megrez (In
finigence, 2024); (2) Standard models (7B-32B parameters)
including Qwen2.5, InternL.M2.5, and Mistral-7B; and (3) Distilled Reasoning models like DeepSeek-RP I-Distilled se- rics. For agent tasks, we systematically evaluate four core agentic capabilities (Action Execution, Workflow Gener- ation, Long-Conter Understanding, and Real-world Ap- plication) that enable complex workflow execution while maintaining coherent behavior across extended interactions. To investigate the effects of compression configuration, we evaluate two mainstream compression methods: quantiza- tion (including GPTQ (Frantar et al. 2022) and AWQ (Ln ct al. )023) and pruning (including GPT (Frantar & Alistarh 2023a) and Wanda (Sun et al. 2024b)
2. Preliminaries
In this paper, we mainly focus on quantization and weight pruning as presented in Figure [b), with additional exper- iments on distilled DeepSeck-PR R1 series. Additionally, design three metrics to measure the gap between compressed and uncompressed LLMs.
Quantization reduces the memory and computational de- mands of neural networks by mapping full-precision values (e.g.. 16-bit floats) lower-bit integer representations, The affine transformation
To facilitate systematic analysis of compression impacts, we introduce three novel analytical tools: (1) Efficient
XFP16 - Z
Rank & Vetterli 2007:
Wei et al. 2024) leverages
XINT round
(1)
S
2

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
AB CD
Quantization
Action Execution
Real-World
Q1: Wher is the capiai of Franco?
Sim
GPTQ, AWQ, SmoothQuant
Function Call,
Web, Game, Embodied, SclencaWorid
Nice; B: Marseili,
Tool Use
Partst ynn
%
Oz: caacch box contains pencis
Quantized LLMs
and you have booes, how many pencils do you have total?
Workflow Build
Long Context
Previous LLM Quantization Benchmarks
Multi-Doc QA, GSMBK, Summarization, Code, weedle-i-thee-haystae Few-shot, etc.
Problem-Sotving Embodied Tasks, Function Call
?
11
S
Sparsification
The soice (Question) in
Multi-Tum
SparseGPT, Wanda
Fist, (Toal Name) gel...
My gow to [Thirkin).
(API Name] ootain....
Small LM
Reason LM
Normal-LLM
Compressed LLMs
Finall,
GemmaZb, PhiB.5,
R1 Dis. Qwan1 1.8,
Miatrai-78,
aulition.
R1 Liama 3B, Rt-Dis- DissQQei178
Qwan2.5-22
MiniOPM
InternLM2 5-7B
Agentic LLM Compression Benchmark
(a) Benchmark Comparison
(b) Compression Methods
(c) Overview of Agent Compression Benchmark
Figure 1: Overview of Agent Compression Benchmarks and Methods for Large Language Models (LLMs). (a) Benchmark Comparison: Illustrates the transition from single-turn quantized LLMs multi-turn compressed LLMs in agentic scenarios. (b) Compression Methods: Summarizes the techniques used for quantization (e.g., GPTQ, AWQ, SmoothQuant) and sparsification (e.g., SparseGPT, Wanda). (c) Overview of Agent Compression Benchmark: Provides a comprehensive view of the capabilities and components involved in agentic LLM compression, including action execution, workflow build, real-world applications, and long-context processing.
$ max (F16)) min ( (PI6)
sourced LLMs have not been well-studied. To fill the gap, focus on the distinct characteristics of agentic capabil- ities in this paper. To study the influences of compression on the agentic capabilities of LLMs, we mainly focus on the capabilities that can be influenced by compression. We classified the agentic capabilities into four main ones as shown in Figure (c) and Table
(2)
2N
preserves the dynamic range of the original tensor XFP16. where N specifies the target integer bit-width (e.g., 8 bits), S is the scaling factor, and Z aligns the integer zero-point with the floating-poin range. In this paper, we mainly focus on GPTQ (Frantar et al. 2022) and AWQ (Lin et al. 2023)
Weight Pruning removes redundant parameters to create sparse weight matrices, reducing model size and inference latency. Unstructured sparsity achieved via element-wise masking:
(1) Action Execution includes function call and tool use (Chen et al.] 2023c). Function call denotes that the agent can employ the user predefined functions to enhance the capabilities, while tool use denotes that the agent can use the external tools to enhance the capabilities. We focus on the six key capabilities: plan, reason, retrieve, understand, instruct, and review.
W W W, M.j- = { if Woll
otherwise
(2) Workflow Generation enables agents to break down complex tasks into executable sequences of steps (Wang) et al. 2024a This includes both single-task workflows where a direct solution path exists, and multi-step work- flows requiring intermediate planning and coordination across multiple tools or APIs. We mainly focus on four key tasks: function call, embodied, problem-solving, and open-grounded tasks.
where t governs the sparsity level. For hardware efficiency, structured pruning eliminates entire channels using the L1- norm Σ. c..i to identify less salient features, trading finer granularity for better computational alignment. In this paper, we mainly focus on SparseGPT (Frantar & Alistarh 2023a) and Wanda (Sun al. 2024b) with unstructured and 2:4 semi-structured settings.
2.1. Agentic Taxonomy
(3) Long- Cong-Cotextt Understanding: Due to the nature of multi-step workflows, the context length is obviously large compared with basic NLP tasks. In the near future, as the context length further increased, the long-context understanding will become key capability for agentic applications. We mainly focus on general long- long-context tasks like single- and multi-doc QA, summarization, and
Since there are already several benchmarks (Li et al. 2024e
Yang et al. 2024a| Gong et al. 2024 Wang et al.) 2024a b) that focus on the capabilities of LLMs like in-context learning, commonsense multi-step reasoning, mathemati- cal multi-step reasoning, instruction following, and self- calibration. However, the agentic capabilities of open-
3

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Weight Bits
KV Cache Bits
Weight Bits
KV Cache Bits
70
70
23.00
36
60
ERank
50
ERank
22.75
60
36.75
40
22:50
50
36
22:25
36,65
30
22.00
36
W3
144
KV3
KV16
W3
W/4
KVB
KV16
Figure 2: ERank analysis difference analysis for quantized LLaMA-2-7B (left) and Mistral-7B (right) models
code, etc., and the more challenging ones like LongGen- Bench (Liu et al. 2024a) (GSM8K and MMLU), Needle-in- The-Haystack.
3B [Infinggnnce 2024
Datasets. Instead of using standard datasets like previ- ous benchmarks such as lmme (Gong et al.) 2024). we uniquely focus on agentic tasks. We categorize the agentic tasks into three types: (1) Action Execution: T-Eval (Chen et al. 2023c). ToolBench (Qin et al. 2023a), and ToolAl-
(4) Real-world Application encompasses the agent's ability to operate in practical deployment scenarios (Wang al. 2025ab like e-commerce, robotics control, and scientific experimentation. This requires coordinating multiple ca- pabilities including tool use, planning, and environmental interaction. We evaluate performance on benchmarks like Game (Jericho, PDDL), and
paca Tang et al. 2023b); (2) Workflow Generation: Worf-
Bench (Qiao et al. 2024); ); (3) Long-Context Understanding:
LongBench (Bai et al.] 2024). LongGenBench (Liu et al.)
EmbodiedAI (Science practical tool use including Tool-Query and Tool-Operation These tasks can faithfully give feedback on how agents can process the real-world applications.
2024a). and Needle-ing Needle-in-thh-bare the-haystack (Kamradt, 2023); (4)
Real-world applications: AgentBoard (Ma al. 2024p. For
calibration data, ensure fair comparison, we follow the setting in previous works (Gong al.) 2024) and employ
the same subset of the Pile (Gao et al.) 2020) validation set.
2.2. Benchmarks and Models
We set the calibration data size to 128 and the sequence length to 512.
Our experimental evaluation focuses on compressed lan- guage models, encompassing both quantized and pruned variants. We assess their performance across four distinct categories of agent tasks: Action Execution, Workflow Gen- eration, Long-Context Understanding, and Real-world Ap- plication. In our investigation, we specifically prioritize compression algorithms that are compatible with contem- porary inference frameworks such as vLLM (Kwon al. 2023 thereby ensuring practical deployability and efficient model serving in real-world applications.
Compression Methods. We focus on post-training compression methods, including quantization and prun- ing. We use the following quantization methods: (1) GPTQ (Frantar et al. 2022); (2) AWQ (Lin et al., (002); (3) SmoothQuant (Xiao et al. 2023). We employ unstruc- tured, structured, and cemi-structured pruning methods.
SparseGPT (Frantar & Alistarh, 2023a) and Wanda (Sun) et al. 2024c) are used for unstructured and semi-structured pruning. ensure fairness and reproducibility, we set the temperature
LLMs. To ensure a comprehensive evaluation, we assess the performance of state-of-the-ard language models, pri- oritizing their latest versions to maintain relevance. Our analysis focuses on models with demonstrated agentic ca- pabilities, as earlier models generally lack these essen- tial features. We categorize the evaluated models into three groups: (1) Medium-scale models (7B parameters): InternL 2- (Cai et al.) 2024). Qwen2.5-7B (Yagg et al. 20246). and Mistral-7B (Jang et al. 2023a); (2) Knowledge-distilled models: DeepSeek- -I-ttekl----
2.3. Statistical Analysis
investigate the influences of compression on the LLMs and how the compression affects the LLMs, we employ three statistical analysis metries:
(1) Efficient Rank (ERank): The effective rank of non- zero matrix A E E is defined as
1.5B, DeepSeek-BR DeepSeek-RI-Distille RI-Dissill---enn and DeepSeek-RI Distill-Llama-53 (Team
cRank(A) exp
Σ 1=1 25=1 J
log
Q
2024a): (3) Efficient small-scale
models: MiniCPM3-4B (Hu al. 2024). Qwen-2.5-1.55 Qwen-2.5-3B (Yang et al.
(4)
2024b), Gemma-2-2b-it (Team]
where Q = min NN, d), and O2.0...... .o denote the sin- gular values of the matrix A. This measure quantifies the distribution of the singular values and provides a notion of
2024b), Phi-3.5-mini-3.8B
Abdin al. 2024), and Megrez-

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Spearman Correlation
0.9
Kendall's Tau
0.8
0.7
0.7
0.6
Top-3
Top-3
76a
Top-10
0.6
op-10
Top-20
Tog-20
0.5
10
20
30
10
30
Taken Position
Token Position
Figure 3: Comparison of format performance differences between (left) quantized and (right) sparse model architec- tures
Figure 4: Top-k Ranking Consistency Analysis for quan- tized Phi-3.5.
across these configurations. Specifically, the ERank de- creases the quantization precision reduces, suggesting loss of information and structural complexity in the weight matrices. This behavior highlights the impact of quantiza- tion on the intrinsic dimensionality the models' represen- tations, which may contribute to the performance trade-offs observed in downstream tasks. As shown in Table 2 this relationship holds across model scales from 125M to 6.7B parameters, where we observe ERank values ranging from 13.898 to 17.877 for 4-bit quantized models. Notably, the ERank values correlate positively with model accuracy, with larger models generally maintaining higher ERank values post-compression. For instance, the 6.7B model achieves the highest ERank of 17.877 and correspondingly strong accuracy of 0.360, while the 2.7B model shows lower ERank of 13.898 despite having comparable accuracy. This suggests that larger models may be more robust to compres- sion, maintaining their structural complexity better. The Diff-ERank metric, measuring the change in effective rank after compression, also shows consistent trend, increasing with model size from 1.410 to 2.280, indicating that larger models undergo more significant structural changes during compression while still preserving performance.
the matrix's effective dimensionality.
(2) Top-K Ranking Consistency: For a given input, let Th and 7/(0 be the sets of top-k tokens according to (o)) and (0)) respectively. We measure ranking consistency us- ing the Jaceard similarity:
1 n T (9) Js
I UT:
(3) Energy-based Analysis: Many OOD detection meth- ods (Lee et al.) 2018 Liu al. 2020) utilize the energy- based model to calculate the energy score the confidence of the model on the input data x. Let f(x): IR + R² be a discriminative neural classifier that maps an input E RD to K logits. For both original model f(²) and compressed model f(e), the categorical distribution is derived using the softmax function:
ef((x)/T
p(yxx)
(6)
ef(x)/T
where fy((x indicates the y-th logit corresponding to class label y, and T is the temperature parameter. The energy function Exx; f') for given input x is defined as:
Top-K Ranking Consistency Analysis. We quantitatively assess the discrepancy between compressed and uncom- pressed LLMs through top-k ranking consistency metrics. As depicted in Figure 4 we evaluate the ranking consis- tency using both Kendall's 's Tau and Spearman Correlation coefficients. Our analysis reveals a notable pattern: as k decreases from 10 to 3, the ranking consistency exhibits increasing instability and degradation. This finding is par- ticularly significant for LLM text generation, where the top-3 tokens are crucial as they represent the most probable next-token predictions. This degradation in ranking con- sistency for the most probable tokens provides potential explanation for the observed performance deterioration in downstream tasks when using compressed LLMs. Addition- ally, Figure demonstrates the strong Spearman correlation between perplexity and Top-k ranking correlation metrics across different model sizes and quantization levels, validat- ing the metrics' ability to capture meaningful performance characteristics.
K
E(x; f) - log
/xx/TT
1=1
We compare the energy distributions between original and compressed models using Ag = f(xxf()) (()) Exx; f())), where lower Ag indicates better preservation of model con- fidence patterns.
3. Statistical Analysis of Compression Effects
Efficient Rank Analysis. We visualize the ERank (Wei et al. 2024) of LLLMM-2-7B and Mistral-7B in Figure 2 We separately evaluate the ERank under varying weight quantization levels (W2, W3, W4, W8, W16) and KV Cache precisions (KV4, KV8, KV16). Our findings indicate that both models exhibit similar trends their effective rank
5

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
PPL and Topk R Correlations with Different Metrics
4.2. Effects of Compression
PPL
0.848
Correlation Coefficient
0.8
Topk R
0.756
Quantization Under Same Bit Budget. Figure 7 preeents comparative analysis of model performance under equiva- lent bit budgets. For instance, 7B model in FP16 and 14B model in FP8/INT8 occupy similar memory footprints, en- abling us to evaluate the effectiveness of quantization versus training larger models from scratch at FP16 precision. While this comparison cannot perfectly control for all variables such as training data and quantization methods, reveals important trends. For smaller model sizes (1 1.BBBB parame- ters), base models trained from scratch with BF16 precision generally demonstrate superior performance. However, as model size increases to around 7B parameters, quantized versions of larger models (e.g., AWQ-quantized 32B mod- els) begin to show significant advantages. This suggests that quantization becomes increasingly valuable for larger model architectures, highlighting its crucial role in making larger, more capable models practically deployable while maintaining strong performance characteristies.
0.664
0.6
0.551
0.457
0.417
0.4
0.323
0.2
0.096
0.084
0.0
HotporQA
TriviQA
News
Lcc
SciWorid
Figure 5: Spearman correlation analysis between perplexity and Top-k ranking correlation metrics across model sizes and quantization levels.
OPT
125M
1.3B
2.7B
6.7B
ACC
0,276
0.332
0.370
0.360
A Loss
5.734
6.138
6,204
6.258
Diff- Diff-ERank
1,410
2.140
2.338
2.280
ERank (4bit)
15.462
15,589
13.898
17.877
Energy
2.738
2.746
2.631
2.883
Model Compression Significantly Impacts Structured Output Generation. Our experimental results in Table demonstrate significant performance variations between JSON and string format outputs under different compression techniques. The analysis reveals that model performance consistently degrades more severely when generating JSON- structured outputs compared to string formats. As evidenced in Figure 3 this performance disparity is particularly pro- nounced in both I2t5rmL and Qwen2.5-7B architec- tures.
Table 2: Comparison of metrics across model sizes demon- strating correlation between ERank, Energy and model per- formance.
Energy-based Analysis. We visualize the distribution of energy scores of compressed and uncompressed L.MMs for a series of tokens. As shown in Appendix C.7 we observed that the quantized LLM has a distinct energy distribution compared with uncompressed ones in the initial stage, while they begin to merge in the subsequent stage. The higher neg- ative energy means high confidence in the prediction. The results show that in the initial decoding stage of the com- pressed LLM, the confidence distribution is polarized, with some over-confidence on certain tokens and some under- confidence on others. During the late decoding stage, both compressed and uncompressed LLMs regard tokens as low confidence. As shown in Table we observe consistent energy scores across different model sizes, indicating that this behavior is inherent to the compression process rather than model scale dependent.
Quantization Preserves Tool Use Capabilities Better Than Sparsification Methods. The empirical evidence presented in Table indicates that quantization approaches, specifically GPTQ and AWQ, maintain model performance more effectively than sparsification techniques such as SparseGPT and Wanda. However, it is noteworthy that Wanda (Sun et al. (002c) achieves comparable performance to quantization methods when implemented with unstruc- tured pruning parameters.
Tool Use Capabilities Vary Significantly Across Model Architectures. From Table g the experimental results re- veal substantial performance differentials among model ar- chitectures, with InternLLM..577B and Qwen2.5-7B demon- strating superior tool use capabilities compared to Mistral- 7B. This observation suggests that architectural advance- ments in newer models contribute significantly to enhanced tool use proficiency.
4. Evaluation on Action Execution
4.1. Experimental Setups
We evaluate the impact of quantization and pruning on LLMs' agent action execution capabilities, focusing on func- tion calling and tool use. We utilize T-Eval (Chen et al.] 2023c), which assesses six core competencies: planning, reasoning, retrieval, understanding, instruction following, and reviewing. Additional results can be found in Table in Appendix C.2
Distillation from a reasoning model lead to performance degradation in agent scenarios. Despite the theoretical advantages of incorporating explicit reasoning processes, our experiments with the DeepSeek-Ri-Distilee RI-Distileed variant of Qwen2.5-7B reveal degradation in performance. As shown
6

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentie Capabilities in LLM Compression
Internl -0-2.55--2.5778
Qwen-2,5-7B
InternI.M44 MM-2.57BB
Qwen-2.5-7B
Instruct
Instruct
Instrec
50
Undersan
Plan
Undertonn
Plan
Ohdlrssin
Pim
Undensand
Plan
Review
Rewse
Review
Resson
Retieve
Ketnive
IP16
Mag(Un)
Wanda(tla)
SpansGGPT(Un)))
EP16
AWQ
GPTQ
FPS
Figure 6: Performance Evaluation of IntermL.-22.5-7B and Qwen-2.5-7B Models Across Different Sparsification (left) and Quantization (right) Techniques for Tool Use
AWQ(INT4)
Base
SmoothQ(W8A8)
-1.5B Parameters
-38 Parameters
Parameters
-14B
Parameters
Average F1
0.6
0.6
0.6
0.4
0.4
0.4
0.4
1.58
78
38
38
7B
148
148
32B
Figure 7: Comparison of average Fl scores across different model sizes and configurations. The models are evaluated based their performance with approximately 1.5B, 3B, 7B, and 14B parameters. The configurations include AWQ(INT4), FP16, GPTQ(INT4), GPTQ(INTS) and SmoothQ(W8A8). Each bar represents the average Fl score achieved by the respective model configuration at different parameter sizes.
in Table 8 we observe a substantial decline in average accu- racy from 70% to 43.6%, contradicting conventional expec- tations regarding the benefits of knowledge distillation.
eration capabilities and execution performance in multi-tur conversational settings. The evaluation encompasses four distinct categories of tasks: function calling, embodied in- teraction, problem-solving and open-groundee scenarios. For compression techniques, evaluate three quantization methods AWQ (Lin al. (2023). GPTQ (Frantar et al.) 2022). and SmoothQuant (Xiao et al. 2023), alongside three pruning approaches Magnitude pruning, SparseGPT (Fran tar Alistarh (023a), and Wanda (Sun et al. 2024b) De- tailed experimental results are presented Table in Ap- pendix C.4
Compression Strategy Recommendations. Based empirical analysis, quantization emerges as the most effec- tive compression approach for maintaining tool use capabil- ities across diverse scenarios. When available, GPTQ with 4-bit precision or FP8 representation demonstrates supe- rior performance characteristics. For applications requiring sparsification, our findings suggest that Wanda with 2:4 spar- sity patterns provides an optimal trade-off between model performance and compression efficiency.
5.2. Effeets of Compression
5. Evaluation on Workflow Generation
Model Compression Shows Minimal Impact on Work- flow Generation Capabilities. Our experimental results, as presented in Table TO demonstrate that most compression methods maintain model performance within 5% degra- dation margin, with only magnitude-based pruning show- ing significant performance deterioration. This resilience suggests that workflow generation tasks primarily rely on high-level planning capabilities that remain largely intact
5.1. Experimental Setups
Our experimental evaluation is conducted using Worf- Bench (Qiao al., 2024), a comprehensive benchmark that employs graph-based workflow representation. This frame- work enables us systematically assess both workflow gen-

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentie Capabilities in LLM Compression
LongBench
T-Eval
WorfBench
AgentBoard
Single-Doc QA
Instruct
Altworld
Jericho
Seek Owen 78 120
DoopSees Ownn Owan
ScepSeok Owen Owen-10
DeepSieic Ownnn 78
52%
Qwenn
Code
Mbdii Mob QA
Review
Plan
Webshop
Scienceweria
22%
Pdal
Synthebe
Surpirerication
Underslamu
Aseson
ToolBenth
05
Tool-Operallon
Toal-query
Few shot
Retrieve
ToolAlpaca
Figure 8: Performance Comparison between DeepSeck-RI-Qwen22.. and Qwen2.5-7B across four evaluation benchmarks. The distilled version performs worse in most cases.
under compression. Among the evaluated architectures, Mistral-7B and Qwen2.5-7B demonstrate notably superior performance compared to InternL.M2.5-7B. indicating that architectural differences may be more influential than com- pression effects.
thermore, we observe that the uncompressed Megrez-3B demonstrates insufficient capability for workflow generation tasks, highlighting the importance of architectural design beyond parameter count.
Robust Performance Maintained for OS and Webshop Tasks Under Compression. Our analysis reveals that mod- els maintain approximately 80% performance across all compression methods for operating system and e-commerce tasks. This robustness likely stems from the structured na- ture of these domains and the relatively constrained action space. Based on these findings and considering the compute efficiency trade-offs, recommend GPTQ and AWQ optimal compression choices for these applications.
6. Evaluation on Long-Context Understanding
6.1. Experimental Setups
We evaluate long-context capabilities across three bench- marks: (1) LongBench (Bai et al., 2024), multi-task benchmark spanning Single/Multi-Doe QA, Code Compre- hension, Synthetic Reasoning, Summarization, and Few- shot Learning, with analyses of quantization (Table 3} sparsification (Table 4), and small/distilled models (Ta- ble 5 (2) LongGenBench (Liu al.) 2024a), designed for extreme-length generation with multi-shot in-context examples, tested via quantized/pruned models (Table 6 and lightweight architectures (Table (3) Needle-in tht Haystack (Kamradt, (003). retrieval-focuse task probing fine-grained contextual understanding, visualized across varying context lengths in Appendix C.5
Model Size Influences Compression Sensitivity in Spe- cialized Tasks. In Alfworld and Lumos tasks, which require fine-grained language understanding and complex reason- ing, we observe a clear correlation between model size and compression resilience. Smaller architectures, such as Qwen2.5-3B, experience substantial performance degrada- tion of up to 50% under GPTQ and AWQ compression. In contrast, larger models like Qwen2.5-32B maintain their per- formance under quantization, occasionally showing slight improvements. This suggests that larger models possess redundant capacity that helps preserve critical capabilities under compression.
6.2. Effects of Compression
Our experimental results demon- strate that quantization and sparsification exhibit minimal impact few-shot learning. synthetic task performance, and code completion capabilities for models exceeding 7B parameters. As evidenced by Table 3 and 4 most com- pression methods induce negligible performance degrada- tion, with the exception of magnitude-basee sparsification. However, smaller architectures (Qwen2.5-1.5B. Qwen2.55 3B, MiniCPM-4B, and Gemma-2B) exhibit significantly re- duced baseline capabilities, often failing to complete funda- mental tasks even prior to compression. Notably, the Merge- 3B model achieves state-of-the-art performance among sub- 7B models. The DeepSeek-RR R -Distillle series are particular sensitivity to compression, with substantial performance drop in single-document QA, multi-de document QA, and syn-
LongBench Analysis:
Distilled Models Shows inferior Performance than undis- tilled Version. Our evaluation of DeepSeck-R1 distilled models reveals significant performance regression compared to their undistilled versions. For example, DeepSeek-R Distilled-Qwen2.5-77 achieves only a 20% average score, substantially lower than the 44% achieved by the undistilled Qwen2.5-7B. This counter-intuitive finding sug- gests that current distillation techniques may not effectively preserve complex reasoning capabilities. Interestingly, the outperforms its larger 7B and 8B counterparts, indicating that model size alone does not determine distillation effectiveness. Fur-
smaller DeepSeek-RI-Nee
8

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
thetic tasks compared uncompressed ones.
progress and success rates in practical applications. As illustrated in Table 9 the distilled models experienced significant degradation in functional performance across practical tasks. For instance, the progress rate on the Pddl benchmark for the model dropped markedly from 33% to 1%. In contrast, smaller models such as Qwen2.5-3B. when enhanced with AWQ. achieved a 23% progress rate. This stark performance gap can be attributed to two key factors. First, the DeepSeek- R1 teacher model used for distillation lacked robust agentic capabilities in its training, particularly for tool use and prac- tical task execution. As a result, the distillation process could not effectively transfer these crucial agentic skills to the student model. Second, the limited model capacity creates an inherent trade-off during distillation when the process prioritizes core reasoning skills like mathematical problem-solving. it necessarily deprioritizes the preserva- tion of agentic capabilities like function calling and tool manipulation. As presented in Figure g we compare the undistilled Qwen2.5-7B with its distilled counterpart across LongBench, T-Eval, WorfBench, and AgentBoard, revealing consistent degradation in practical task performance despite improvements in traditional reasoning tasks. These findings highlight the need to develop distillation techniques that can better balance the preservation of both abstract reasoning and practical agentic capabilities.
LongGenBench Findings: In a more challenging bench- mark LongGenBench, AWQ consistently outperforms other compression methodologies. While Wanda and SparseGPT achieve competitive performance with AWQ on Qwen2.5- 7B (Table 7. this parity does not extend to InternLM2.5-- 7B, where only AWQ maintains baseline-equivalee formance. Intriguingly, compressed Qwen2.5-3B models demonstrate MMLU scores comparable to 7B counterparts, yet suffer catastrophic performance collapse in GSM8K reasoning (61%- + 11%). Smaller models (<3B) exhibit near-zero accuracy on both MMLU and GSM8K. A notable discrepancy emerges between Qwen2.5-7B (61% GSM8K, 64% MMI.U) and its DeepSeek-RI R1 R1-Distilled variant (near- zero scores), suggesting that reasoning patterns may signifi- cantly impact long-conted comprehension capabilities.
per-
Needle-in-the-Hayseee Evaluation: Both quantization and sparsification adversely affect long-context information retrieval, as detailed in Appendix C.5 Magnitude-basee
pruning induces particularly severe performance degrada- tion. Our 40K-contex experiments reveal consistent perfor- mance boundaries at 32K tokens across all compressed mod- els, suggesting architectural limitations inherent to LLM designs and training paradigms.
7. Evaluation on Real- World Applications
8. Conclusion
7.1. Experimental Sctup
In this paper, we introduce ACBench, the first benchmark for evaluating LLM compression's 's impact agentic capa- bilities, including action execution, workflow generation, long context understanding and some real-world applica- tions. By introducing ERank, Top-k Correlation, and En- ergy metrics, we systematize compression analysis for agent tasks. ACBench provides a practical tool for efficient LLM deployment, and inspires future LLM compression to focus more on real-world LLM based agentic applications.
Our empirical evaluation of compressed L.LMs encompasses real-world agent tasks utilizing the AgentBoard frame- work (Ma et al. 2024). The evaluation spans three primary domains: (1) Embodied AI, with focus on the Science- World environment for physical interaction simulation; (2) Game Interaction, incorporating both Jericho text-based adventures and PDDL planning scenarios; and (3) Tool Use, which examines both tool querying and operational capa- bilities. For more details, please refer to the experimental results in Table@of Appendix
9. Limitations
7.2. Effects of Compression
Our study is limited to examining post-training quantization and sparsification for agentic capabilities, and does not in- volve quantization-awar training (QAT) approaches. We restrict our analysis compression methods compatible with vLLM (Kwon al. 2023), excluding promising tech- niques like QuaRot (Ashkboos et al., (024) due to their com- putational overhead. Additionally, we employ default con- figurations throughout our experiments without exploring variations in parameters such as group size. Consequently, our findings and recommendations should be interpreted within the context of these specific experimental conditions and may not necessarily extend to other scenarios or tasks.
Compressed LLMs face significant challenges in han- dling most real-world scenarios. Through comprehensive evaluation detailed in Table we assessed various quanti- zation and pruning techniques applied to 7B LLMs. The results demonstrate that these compressed models generally exhibit substantial degradation in real-world task perfor- mance. Among the compression methods evaluated, only AWQ and Wanda maintained acceptable performance levels, while alternative approaches showed marked deterioration in capability across multiple task domains.
DeepSeek- DeepSeek-R1-Distillee -1-Dittille model series exhibited minimal
9

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Acknowledgments
integration into a wide range of industries and applications.
This work was partially supported by National Natural Sci- ence Foundation of China under Grant No. 62272122, the Guangzhou Municipal Joint Funding Project with Univer- sities and Enterprises under Grant No. 2024A0300616, Guangzhou Municipality Big Data Intelligence Key Lab (2023A03J0012), Hong Kong CRF grants under Grant No. C7004-22G and C6015-23G, a RGC RIF grant under contract R6021-20, RGC TRS grant under contract T43- 513/23N-2, RGC CRF grants under contracts C7004-22G, C1029-22G and C6015-23G, NSFC project under contract 62432008, and RGC GRF grants under contracts 16207922
References
Abdin, M., Jacobs, S. A., Awan, A. A., al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv, abs/2404. 14219, 2024.
Amirizaniani, M., Martin, F.,, Sivachenko, M., Mashhadi, A., and Shah, C. Can Ilms reason like humans? assessing theory of mind reasoning in Ilms for open-ended ques- tions. In CIKM, CIKM '24, pp. 34-44, 2024. ISBN 9798400704369.
and 16207423.
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. Agentharm: A benchmark for measuring harmfulness of LLM agents. arXiv preprint arXiv:2410.0902 2024.
Impact Statement
The findings of this work have significant implications for the future development and deployment of LLMs in real- world agentic applications. By systematically evaluating the impact of compression techniques on key agentic capabili- ties, our work sheds light on the complex trade-off's involved in compressing LLMs while preserving their performance in critical tasks such as planning, reasoning, tool use, and long-context understanding.
Asai, A.. Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self- rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2024.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T.,, and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated Ilms, arXiv preprint WrXiv:2444000466 0bs/24444004555 2024.
The introduction of the Agent Compression Benchmark (ACBench) provides foundational tool for the evaluation of compressed models in agentic settings, area that has been largely overlooked in prior research. This benchmark paves the way for more targeted research into optimizing LLM compression for specific agent tasks, helping to bridge the gap between state-of-the-ar compression methods and the practical requirements of real-world deployments.
Babakniya, S., Elkordy, A. R., Ezzeldin, Y. H., Liu, Q. Song. K.-B., El-Khamy, M., and Avestimehr, S. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06. 06522, abs/2308.06522, 2023.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang. J., Huang, Z., Du, Z., Liu, X., Zeng. A., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multitask benchmark for long context understanding. In ACL. pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/w1/2044.00 1/2024.acl-longg 172.
Our comprehensive comparison of quantization and pruning techniques highlights the importance of selecting appropri- ate compression configurations based on the specific needs of agentic tasks. The novel analytical tools we introduce, including ERank, Top-k Ranking Correlation and Energy, provide new insights into the internal workings of com- pressed models and enable a more granular understanding of the impact of compression on model performance and decision-making
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Pod- stawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, pp. 17682-17690, 2024.
This work has the potential to influence the design of more efficient LLMs, allowing for their deployment in resource- constrained environments where computational and memory resources are limited. Furthermore, the ability to maintain agentic capabilities even in compressed models will facili- tate the broader adoption of L.LMs in practical applications such as robotics, autonomous systems, and interactive AI agents. By providing systematic framework for evaluating and optimizing compressed L.LMs, we hope to contribute to the ongoing efforts to make AI models more accessible and capable in real-world scenarios, paving the way for their
Brown, T. Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 33:1877 2020.
Cai, Z., Cao, M., Chen, H., Chen, K., et al. Internim2 technical report. ArXiv, abs/2403. 17297, 2024.
Chakraborty, S., Ghosal, S.S Yin, M., Manocha, D., Wang, M., Bedi, A., and Huang, F. Transfer q-star Principled decoding for umm alignment. In NeurlPS, 2024.
10

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Chan, X., Wang. X., Yu, D., Mi, H., and Yu, D. Scaling syn- thetic data creation with 1, 1,000,000,000 personas. arXiv preprint ar arXiv:2406.
Dong, P., Li, L.., Zhong, Y., Du, D., Fan, R., Chen, Y., Tang, Z.., Wang, Q. Xue, W., Guo, Y., al. Stblim: Breaking the 1-bit barrier with structured binary Ilms. arXiv preprint arXiv:2408. 2.Xi::2408.0180333 2024c.
2024.
Chen, J., Yuan, S., Ye, R., Majumder, B. P., and Richardson, K. Put your money where your mouth is: Evaluating strategic planning and execution of Ilm agents in an auc- tion arena. arXiv preprint 23.ivv2330.00772 2023a.
Du, D., Gong, G., and Chu, X. Model quantization and hardware acceleration for vision transformers: A compre- hensive survey. ArXiv, abs/2405.00314 2024a.
Chen, J., Wang, X., Xu, R., Yuan, S., Zhang, Y., Shi, W., Xie, J., Li, S., Yang. R., Zhu, T., et al. From persona to personalization: A survey role-playing language agents. arXiv preprint arXiv:2404.18233 2024a.
Du, D., Zhang, Y., Cao, S., Guo, J., Cao, T., Chu, X., and
Xu, N. Bitdistiller: Unleashing the potential of sub-4-bit Ilms via self-distillation. In ACL, 2024b.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), NeurlPS, 2022.
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C.,, Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie, R., et al. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents, arXiv preprint arXiv:2308. 10848, 2023b.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv, abs/2301.00774, 2023a.
Chen, Y., Röder, D., Erker, J.-J., Hennig, L., Thomas, P., Möller, S.. and Roller, R. Retreeeal-uugmented knowl- edge integration into language models: A survey. In Li, S., Li, M., Zhang, M. J. Q. Choi, E., Geva, M., Hase, P., and Ji, H. (eds.), Proceedings of the 1st Workshop on Towards Knowledgeable Language Models [KNowL.MM 2024), pp- 45-63, 23,63, Bangkok, Thailand, August 2024b. Association for Computational Linguistics.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Krause, A., Brunskill, E.,, Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 10323-10337. PMLR, 2023b.
Chen, Z., Gao, Q. Bosselut, A., Sabharwal, A., and Richard- son, K. DISCO: Distilling counterfactuals with large lan- 2022. Chen, Z.. Du, W., Zhang, W., Liu, K., Liu, J., Zheng, M., Zhuo, J., Zhang, S., Lin, D., Chen, K., al. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint 23X2
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv, abs/2210. 17323, 2022.
guage models. arXiv preprint arXiv:2212.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, abs/2101.00022 2020.
2023c.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang. H., Zheng, L., Zhuang. S., Zhuang. Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http://viiunaabmmyy..o (accessed 14 April 2023), 2(3):6, 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv: 23Xiv:2312..0 23/2. 2023a.
Dasgupta, S., Cohn, T., and Baldwin, T. Cost-effective distillation of large language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), ACL 2023 Findings, pp. 7346-7354 Association for Computational Linguisties, 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K.,, Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. Retrieval- auggented generation for large language models: A survey, arXiv preprint arXiv:23 10997, 2023b.
Dong, P., Li, L., Liu, X., Tang, Z., Liu, X., Wang, Q., and Chu, X. Lpzero: Language model zero-cost proxy search from zero. EMNLP, 2024a.
Ge, Y., Ren, Y., Hua, W., Xu, S., Tan, J., and Zhang, Y. Llm os, agents as apps: Envisioning aios, agents and the aios-agent ecosystem. arXiv.org, 2023.
Dong, P., Li, Tang, Z., Liu, x., Pan, X., Wang, Q., and Chu, X. Pruner-zero zero: Evolving symbolic pruning metric from scratch for large language models. In ICML, pp. 11346-11374, 2024b.
Gehman, S., Gururangan, S., Sap, M.. Choi, Y., and Smith, N. A. Realtoxicitypromps: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009, /462, 2020.

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Gong, R., Yang, Y., Gu, S., Huang, Y., Zhang, Y., Liu, X., and Tao, D. Llme: Benchmarking large language model quantization with versatile compression toolkit. In EMNLP, 2024.
Huang, H., Qu, Y., Liu, J., Yang, M., and Zhao, T. An empirical study of llm-as-a-judge for Ilm evaluation: Fine- tuned judge models are task-specific classifiers. arXiv 2024a.
preprint arXivv2403.0288
Gu, H., Li, W., Li, L.. Zhu, Q. Lee, M., Sun, S., Xue, W., and Guo, Y.TT. Delta decompression for moe-based Ilms compression. ArXiv, abs/2502.17298. 2025.
Huang, Y., Zhong, W., Shi, E.,, Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q. Zheng, Z., and Wang. Y. Agents in software engineering: Survey, landscape, and vision. arXiv preprint arXiv:2409.0900 2024b.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Cortes, C., Lawrence, N.. Lee, D., Sugiyama, M., and Garnett, R. (eds.), NeurlPS, volume 28. Curran Asso- ciates, Inc., 2015.
Infinigence. Infini Infini-megrez: megrez: Megrez-3b and megrez- -bmoni models, 2024. GitHub repository containing models and related resources.
Jennings, N. R., Sycara, K., and Wooldridge, M. roadmap of agent research and development. Autonomous Agents and Multi-Agent Systems, 1:7-38, 1998.
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L.,, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint ar arXiv:2209.00844 2022.
Jeong, Soyeong, Baek, Jinheon, Cho, Sukmin, Hwang, Ju, S., Park, and C, Adaptive-rag: Learning to adapt reureeeal-augmmeneed large language models through ques- tion complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume I: Long Papers), pp. 7029-7043, 2024.
Hinton, G. E., Vinyals, O., and Dean, Distilling the knowledge in neural network. arXiv.org. 2015a.
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. ArXiv, ab3/1503,,,,,,, 2015b.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L.,, Lavril, T. Wang, T. Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv preprint, abs/2310.068255, 2023a.
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collabora- tive framework. arXiv preprint 2...ix:230..003522 2023.
Jiang, J., Zhou, K., Dong, Z., Ye, K., Zhao, W. X., and Wen, J.-R. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 EMNLP, pp. 9237-9251, 2023b.
Hong. S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J.. Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. Metagpt: Meta programming for a multi-agent collaborative frame- work. In ICLR, 2024.
Jiang, Z. Longrag: Enhancing retrieval-augmented generation with long-context 1lms. arXiv preprint WwXiv:2406.53194 2024.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang. Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv, abs/2404.06399 06399, 2024.
Kamradt, G. Needle in a haystack pressure testing llms, 2023.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: simple depth pruning for large language models. ICLR Work- shop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo), 2024.
Hua, W., Fan, L., Li, L., Mei, K., J., Ge, Y., Hemphill, L., and Zhang, Y. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint ar xrXiv:2311.17222 2023.
Kim, S., Hooper, C., Gholami, A., Dong, 7., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense- and-sparse quantization. CoRR, abs/2306.07629) 2023. doi: 10.48550AARXX2223000776222
Huang, D., Bu, Q. Zhang, J. M., Luck, M., and Cui, H. Agentcoder: Multi-agent- based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312. 130/0 2023.
Koa, JL. K., Ma, Y. Ng. R., and Chua, T.-S. Learning to gen- erate explainable stock predictions using self-reflective
12

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
large language models. In Proceedings of the ACM on Web Conference 2024, pp. 4304-4315 2024.
Explanations from large language models make small reasoners better. In 2nd Workshop on Sustainable AI, 2024c.
Kwon, W. Li, Z., Zhuang, S., Sheng, Y. Zheng. L., Yu, C. H., Gonzalez, E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Li, S., Huang. J., Zhuang. J., Shi, Y., Cai, X., Xu, M., Wang, X., Zhang, L.,, Ke, G., and Cai, H. ScilitlIm: How to adapt Ilms for scientific literature understanding. arXiv, 2024d.
Lai, K., Tang, Z.,, Pan, X., Dong, P., Liu, X., Chen, H., Shen, L., Li, B., and Chu, X. Mediator: Memory-efficient Ilm merging with less parameter conflicts and uncertainty based routing, 2025.
Li, S., Ning, X., Wang. L., Liu, T., Shi, X., Yan. S., Dai, G., Yang, H., and Wang, Y. Evaluating quantized large language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett. J., and Berkenkamp, F. (eds.), ICML, volume 235 of Proceedings of Machine Learning Research, pp. 18480-28522 PMLR, 21-27 July 2024e.
Leblond, R., Alayrae J.-B., Sifre, L.., Pislar, M., Lespiau, J.-B., Antonoglou, I., Simonyan, K., and Vinyals, O. Machine translation decoding beyond beam search. In EMNLP, 2021.
Li, W. Li, L., Lee, M., and Sun, S. Adaptive layer spar- sity for large language models via activation correlation assessment. In NeurlPS, 2024f.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), AAAL, pp. 13355- 13364. AAAI Press, 2024. doi: 10. A..1/AAA1118111 29237.
Li, X., Yu, P. Zhou, C., Schick, T., Levy, O., Zettlemoyer, L., Weston, J., and Lewis, M. Self-alignment with instruc- tion backtranslation. arXiv preprint arXiv:2308.0633 06255, 2023c.
Lee, D. and Tiwari, M. Prompt infection: Lim-to-Ilm prompt injection within multi-agent systems. arXiv preprint ar arXiv:2410.0728. 2024.
Li, X., Yu, P., Zhou, C., Schick, T., Levy, O., Zettlemoye, L., Weston, J. E., and Lewis, M. Self-alignment with instruction backtranslation. In ICIR, 2024g.
Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. arXiv, has/2200905463.. 2023d.
Lee, K., Lee, H., Lee, K., and Shin, J. Training confidence- calibrated classifiers for detecting out-of-distribution sam- ples. In ICLR, 2018
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for 11m compression and acceleration. CoRR, abs/2306.00978, 2023. doi: 10-485500/axxi22200.00077
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for "mind" exploration of large language model society. NeurlPS, 36:51991- 52008, 2023a.
Liu, J., Li, L., Xiang, T., Wang, B., and Qian, Y. Tera- Ilm: Token compression retrieval augmented large lan- guage model for inference cost reduction. arXiv preprint arXiv:2310.1553 2023a.
Li, J., Chen, J., Ren, R., Cheng, X., Zhao, X., Nie, J.-Y., and Wen, J.-R. The dawn after the dark: An empiri- cal study on factuality hallucination in large language models. In Ku, L.-... Martins, A., and Srikumar, V. (eds.), ACI., pp. 10879-10899, Bangkok, Thailand, Au- gust 2024a. Association for Computational Linguistics. doi: 1/.118653//1120222.cll
Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang. J., Su, H., Zhu, J., et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:231 1. 1..0233, 2023b.
Li, L., Dong. P., Li, A., Wei, Z., and Yang, Kd-zero: Evolving knowledge distiller for any teacher-student pairs. In NeurIPS, 2023b.
Liu, W., Wang, X., Owens, J. D., and Li, Y. Energy-based out-of-distiibution detection. NeurIPS, 2020.
Li, I.., Dong, P., Tang, 7., Liu, X., Wang, Q., Luo, W., Xue, W., fei Liu, Q., Chu, X., and Guo, Y.-T. Discovering spar- sity allocation for layer-wise pruning of large language models. In NeurIPS, 2024b.
Liu, X., Li, L., Li, C., and Yao, A. Norm: Knowledge distillation via n-to-one representation matching, 2023c. URL http:sss/aarrii..orr/aa///////// 13803
Liu, X., Dong, P., Hu, X., and Chu, X. Longgenbench: Long- context generation benchmark. ArXiv, abx/24100419, 2024a.
Li, S., Chen, J.. Shen, Y., Chen, Z., Zhang, X., Li, Z., Wang, H., Qian, J., Peng, B., Mao, Y., Chen, W., and Yan, X.
13

---

Can Compressed LLMS Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. Understanding LLMs: A comprehensive overview from training to infer- ence. arXiv preprint arXiv:2401.02038 2024b.
Patel, A., Raffel, C., and Callison- Burch, C. Datadreamer: A tool for synthetic data generation and reproducible Ilm workflows, 2024.
Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosse- lut, A., West, R., and Faltings, B. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.0194 2023.
Lu, Q.. Qiu, B.., Ding, L., Xie, L., and Tao, D. Error analysis prompting enables human-like like translation evaluation in large language models: A case study on chatgpt. ACL, 2023.
Pipitone, N. and Alami, G. H. Legalbench-rag: A bench- mark for retrieval -aagevee-- generation in the legal do- main. arXiv preprint arXiv:2408. 10343, 2024.
Luo, L., Li, Y.-F, Haf, R., and Pan, S. Reasoning on graphs: Faithful and interpretable large language model reasoning. In ICLR, 2023.
Qiao, S., Fang, R., Qiu, Z., Wang, X., Zhang, N., Jiang, Y.,
Xie, P., Huang, F., and Chen, H. Benchmarking agentic workflow generation. arXiv, abs/2410.07869, 2024.
Ma, C., Zhang, J., Zhu, Z., Yang. C., Yang. Y., Jin, Y., Lan, Z., Kong, L., and He, Agentboard: An analytical evaluation board of multi-turn Ilm agents, 2024.
Qin, Y., Hu, S., Lin, Y. Chen, W., Ding, N., Cui, G., Zeng,
Z., Huang, Y., Xiao, C., Han, C., Fung, Y. R., Su, Y., Wang. H., Qian, C., Tian, R., Zhu, K., Liang, S., Shen, X., Xu, B., Zhang, Z., Ye, Y., Li, B., Tang, Z., Yi, J., Zhu,
Ma, X., Fang, G., and Wang, X. Lim-pruner: On the struc- tural pruning of large language models. In Thirty-seventh Conference on NeurlPS, 2023.
Y., Dai, Z.,, Yan, L., Cong, X., Lu, Y., Zhao, W., Huang, Y., Yan, J., Han, X., Sun, X., Li, D., Phang, J., Yang, C., Wu, T. Ji, H., Liu, Z., and Sun, M. Tool learning with foundation models, 2023a.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang, Y., al. Self-refine: Iterative refinement with self-feedback. NeurlPS, 36, 2024.
Qin, Y. Liang, S. Ye, Y., Zhu, K., Yan, L., Lu, Y.-T,, Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z.., and Sun, M. Toolllm:
Molchanov, P., Mallya, A., Tyree, S., Frosio, I,, and Kautz, J. Importance estimation for neural network pruning. In CVPR, pp. 1 11264-18 11264-11272. Computer Vision Foundation / IEEE, 2019. doi: 10.1 D0.119//CVR..219.011.5
Facilitating large language models to master 16000+ real-world apis. ICLR, 2023b.
Rivera, C.,, Byrd, G., Paul, W., Feldman, T., Booker, M., Holmes, E., Handelman, D., Kemp, B., Badger, A., Schmidt, A., et al. Conceptagent: Llm-driven precon- dition grounding and tree search for robust task planning and execution. arXiv preprint arXiv:2410.06108, 2024.
OpenAl. Gpt-4 technical report, 2023.
Pan, J., Wang. J., and Li, G. Survey of vector database management systems. The VLDB Journal, 33(5):1591- 1615, July 2024. ISSN 1066-8888.
Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. 2007 15th European Signal Processing Conference, pp. 606-610, 2007.
Pan, L., Albalak, A., Wang, X., and Wang, W. Logic- LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of EMNLP, pp. 3806-3824, Singapore, December 2023. Association for Computational Linguisties.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Sachdeva, R., Tutek, M., and Gurevych, I. Catfood: Counter- factual augmented training for improving out-of-domain performance and calibration. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1576-1898, 2024.
Park, G., Park, B., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In ICLR, 2024.
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., and Gadepally, V. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-9, IEEE, 2023a.
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposiun on User Interface Software and Technology, pp. 1-22, 2023.
14

---

Can Compressed LLMS Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., and Gadepally, V. From words to watts: Benchmarking the energy costs of large language model inference, 2023b.
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. In ICLR, 2024b.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In ICLR, 2024c.
Schick, T., Dwivedi-YY Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. NeurIPS, 36:68539-685511 2023.
Tang. Zhenheng, Wang, Yuxin, Wang, Qiang, Chu, and Xiaowen. The impact of gpu dvfs the energy and performance of deep learning: An empirical study. In Proceedings of the Tenth ACM International Conference on Future Energy Systems, e-Energy '19, pp. 315-325, New York, NY, USA, 2019. Association for Comput- ing Machinery. ISBN 9781450366717. doi: 10.1145/ 3307772.3328315.
Schick, T., Dwivedi- Yu, J., Dess, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to tools. NeurlPS, 36, 2024.
Shao, H., Liu, B., and Qian, Y. One-shot sensitivity- aware mixed sparsity pruning for large language mod- cls. In ICASSP, pp. 11296-11300, 2024. doi: 10.1109) (CASSP48485.2044.110 10445737.
Tang, Zhenheng, Shi, Shaohuai, Chu, Xiaowen, Wang, Wei, Li, and Bo. Communication-effiiceenn distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.cm 0000:22003006007, 2020.
Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., and Ren, Z. Learning to use tools via cooperative and interactive agents. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Find- ings of EMNLP, pp. 10642-10657, Miami, Florida, USA, November 2024. Association for Computational Linguis- http://ww12222....nnn.gneooo
Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Cao, B., and Sun. L. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023a.
Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., and Sun, L. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023b.
tics. doi:
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforce- ment learning. NeurIPS, 36, 2024.
Team, D.-A. Deepseek-v3 technical report. ArXiv, Abs/2412.194377 2024a.
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. Lim-planner: Few-shot grounded plan- ning for embodied agents with large language models. In ICCV, pp. 3998-3009, 2023.
Team, G. Gemma 2: Improving open language models at a practical size. arXiv, abs/2408.001 18, 2024b.
Tesfatsion, L. Agent-based computational economics:
Overview and brief history. Artificial Intelligence, Learn- ing and Computation in Economics and Finance, pp. 41- 58, 2023.
Soudani, H., Kanoulas, E., and Hasibi, F. Fine tuning vs. re- trieval augmented generation for less popular knowledge. arXiv preprint arXiv:2403.01432 2024.
Tian, Z., Zhao, Y., Huang, Z., Wang, Y.-X., Zhang, N., and He, H. SeqPATE: Differentially private text generation via knowledge distillation. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in NeurIPS, 2022.
Stojkovic, J., Choukse, E., Zhang, C., Goiri, I., and Torrellas, Towards greener lims: Bringing energy-efficiency to the forefront of I1m inference, 2024.
Sun, H., Xu, W., Liu, W., Luan, J., Wang, B., Shang, S., Wen, J.-R., and Yan, Determir: Augmenting llm-based logical reasoning from indeterminacy to determinacy. In Ku, L.W Martins, A., and Srikumar, V. (eds.), ACL, pp. 9828 9862. Association for Computational Linguistics, August 2024a.
Tonmoy, S. M., Zaman, S. M., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.013133 2024.
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambham- pati, S. Large language models still can't 't plan (a bench- mark for iims on planning and reasoning about change). NeurIPS 2022 Foundation Models for Decision Mak- ing Workshop, 2022.
Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L., Shum, H.-Y., and Guo, J. Think-on- onngghhh: Deep and responsible reasoning of large language model on knowledge graph. In ICLR, 2023.
15

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L.., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. Transac- tions on Machine Learning Research, 2023a.
Wang, X., Alam, S., Wan, Z., Shen, H., and Zhang, M. Svd- Ilm v2: Optimizing singular value truncation for large language model compression. ArXiv, abs/2503.12340, 2025b.
Wang, P., Wang, Z., Li, Z., Gao, Y. Yin, B., and Ren, X. Scott: Self-consistent chain-of-thought distillation. In Rogers, A., Boyd-Graber, L., and Okazaki, N. (eds.), ACL 2023, pp. 5546-5555 Association for Computa- tional Linguistics, 2023b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of thought prompting elicits reasoning in large language models. NeurlPS, 35: 24824-24837, 2022a.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougee prompting elicits reasoning in large language models. NeurlPS, 35: 24824 24837, 2022b.
Wang, Q. Tang, Z., Jiang, 7.,, Chen, N., Wang, T., and He, B. Agenttaxo: Dissecting and benchmarking token distribution of IIm multi-agent systems. In ICLR 2025 Workshop on Foundation Models in the Wild, 2024a.
Wei, I.., Tan, Z., Li, C., Wang. J., and Huang, W. Diff-erank: A novel rank-based metric for evaluating large language models. In NeurIPS, 2024.
Wang, Q., Wang, T., Tang, Z., Li, Q., Chen, N., Liang, J., and He, B. Megaagent: A practical framework for autonomous cooperation in large-scale Ilm agent systems. ACL, 2024b.
Wen, Z., Tian, Z., Jian, Z., Huang. Z., Ke, P.,, Gao, Y., Huang, M., and Li, D. Perception of knowledge boundary for large language models through semi-open-endee question answering. ArXiv, abs/2405. 14383,, 2024.
Wang. Q., Wang, Z., Su. Y., Tong, H.. and Song, Y. Re- thinking the bounds of LLM reasoning: Are multi-agent discussions the key? arXiv preprint 18X1v:2402.18272, 2024c.
Wilkins, G., Keshav, S., and Mortier, R. Hybrid hetero- geneous clusters can lower the energy consumption of LLM inference workloads. In Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems, pp. 506-513, 2024.
Wang, Q., Wu, J., Tang, Z., Luo, B., Chen, N., Chen, W., and He, B. What limits Ilm-based human simulation: Lims or our design? arXiv preprint arXiv:2501.08579 2025a.
Wu, J., Chang, C.-C., Yu, T., He, Z., Wang, J., Hou, Y., and McAuley, J. Coral: Collaborative retrieval- augmented large language models improve long-tail recommendation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. -391-, 2024a.
Wang, S., Wei, Z., Choi, Y., and Ren, X. Symbolic work- ing memory enhances language models for complex rule application. In Proceedings of the 2024 EMNLP, pp- 17583-17604, 2024d.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08113 2023.
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H.. Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In ICLR, 2022.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. In ICLR 2024 Workshop Large Language Model (LLM) Agents, 2024b.
Wang, X., Wei, J., Schuurmans, D., Le. Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, ICI. 2023, Kiguli, Rwanda, May 1-5, 2023. OpenReview.net, 2023c.
Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z.,, Qiu, X., Li, Y., Lin, W., and Song. S. L. Dlah--lm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. Proc. VLDB Endow. 17(2):211-224, oct 2023. ISSN 2150-8097. doi: 10. 14778/3626292.3626300
Wang, X., Yang. Q. Qiu, Y., Liang. J., He, Q. Gu, Z., Xiao, Y., and Wang. W. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308. 11661, 2023d.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd-Ilm: Truncation-awant singular value decomposition for large language model compression. ArXiv, abs/2403.07378, 2024e.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training guantization for large language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and
16

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Scarlett, J. (eds.), ICML, volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099. PMLR, 2023.
Yuan, Z., Shang, Y., Song, Y., Wu, Q. Yan, Y., and Sun, G. Asvd: Activation-aware singular value decompo- sition for compressing large language models. ArXiv, abs/2312.05821, 2023.
Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: A benchmark for real-world planning with language agents. In ICMI., 2024.
Zhang, Kechi, Zhang, Huangzhao, Li, Ge, Li, Jia, Li, Zhuo, Jin, and Zhi. Toolcoder: Teach code generation models to use api search tools. arXiv preprint arXiv:2305.0409 2023a.
Xu, F., Lin, Q.. Han, J., Zhao, T., Liu, J., and Cambria, E. Are large language models really good logical reasoners? comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.09844 2023.
Zhang, F. Chen, B., Zhang, Y., Keung, J.. Liu, J., Zan,
D., Mao, Y., Lou, J.-G., and Chen, W. Repocoder: Repository- code completion through iterative re- trieval and generation. arXiv preprint axXiv2230.112577 2023b.
Xu, J., Fei, H., Pan, L.. Liu, Q., Lee, M. L.,, and Hsu, W. Faithful logical reasoning via symbolic chain-of-thought. In ACL, 2024.
Zhang, G., Yue, Y., Li, Z., Yun, S., Wan, G., Wang. K., Cheng, D., Yu, J. X., and Chen, T. Cut the crap: An economical communication pipeline for llm-based multi- agent systems. arXiv preprint vrXiv:2410.02500 2024a.
Xue, F., Fu, Y., Zhou, W., Zheng. Z., and You, Y. To repeat or not to repeat: Insights from scaling IIm under token- crisis. NeurIPS, 36, 2024.
Zhang, R., Wang, K., Liu, L., Wang, S., Cheng, H., Zhang, C., and Shen, Y. Lore: Low-rank compression for llms kv cache with progressive compression strategy, 2024b, URL
Yang, G., He, C., Guo, J.. Wu, J., Ding, Y., Liu, A., Qin, H., Ji, P., and Liu, X. Limebench: Benchmarking large language model compression for efficient deployment. ArXiv, abs22410.21352, 2024a.
//axiv///xxivv../ org wwtp:ba//aaaaaaaaaao......oaaa 03111
Zhang, S., Chen, Z., Shen, Y., Ding, M.. Tenenbaum, J. B., and Gan, C. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510 2023c.
Yang, H., Liu, X.-Y., and Wang. C. D. Fingpt: Open-source financial large language models, 2023a.
Yang, Q A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H. Lin, H., Yang, J., Tu, J. Zhang, J., Yang, J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K. Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y.-C., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., Qiu, Z., and Quan, S. Qwen2.5 technical report, arXiv, abs/2442.115115, 2024b.
Zhang, S., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T. Can transformers learn solve problems recursively? arXiv.org, 2023d.
Zhang, Y., Yang. S., Bai, C., Wu, Li, and Wang, Z. Towards efficient LLM grounding for embodied multi- agent collaboration. arXiv preprint arXiv:2405. 14314, 2024c.
Yang, R., Song, Li, Y., Zhao, S., Ge, Y. Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self- self-instruction. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), NeurlPS, volume 36, pp. 71005-72007. Curran Associates, Inc., 2023b.
Zhang, Y.. Zhao, L., Lin, M., Yunyun, S., Yao, Y., Han, X., Tanner, J., Liu, S., and Ji, R. Dynamic sparse no training: Training-free fine-tuning for sparse 11ms. In ICLR, 2024d.
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang. Y., Li, Z., Lin, Z., Xing, E., et al. Lmsys-chat- Im: A large-scale real-world I1m conversation dataset. arXiv preprint arXiv:2309. 1998, 2023a.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. NeurlPS, 36, 2024.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judg- ing LLM-ae-a with MT-bench and chatbot arena. NeurlPS, 36:46595-46622. 2023b.
You, J., Chung, J.W.., and Chowdhury, M. Zeus: Under- standing and optimizing gpu energy consumption of dnn training, 2022.
Zhou, A., Yan, K., Saapeentok--Rothmane M., Wang, H., and Wang, Y.-.. Language agent tree search unifies reasoning, acting, and planning in language models. In ICML, 2023.
Yu, Y., Ping, W., Liu, Z., Wang, B., You, J., Zhang, C., Shoeybi, M., and Catanzaro, B. Rankrag: Unifying con- text ranking with retrieval- augmente generation in llms. arXiv preprint orXiv:2407.0244 2024.
17

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Appendix
More Related Works
19
A.1
LLM Compression
19
A.2
LLM-based Agents
20
A.3
Reasoning
20
A.4 Planning
A.5
Tool Use
21
A.6
Retrieval Augmented Generation
22
A.7
Long Context
23
Detailed Experiment Settings
24
B.1
Details of Quantization and Sparsification
24
Details of Small Language Models and Distilled Models
24
B.3
Details of Tool Use Experiments
24
B.4
Details of Long Context Understanding Experiments
25
B.5
Details of Workflow Generation Experiments
25
B.6
Details of Real- World Applications Experiments
25
More Experiment Results
26
Long- Context Understanding
26
C.2
Tool Use
28
C,3
Real-World Applications
29
4
Workflow Generation
29
C,5
Needle-in-the-haysee Test
29
C.6
Logits Visualization
30
Energy Visualization
30
18

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
A. More Related Works
In this section, we provide more related works for the topics discussed in this paper.
A.1. LLM Compression
Broadly speaking, L.M compression comprises various techniques such as quantization, pruning, knowledge distillation, low-rank decomposition, and more. These methods aim to reduce the size and computational requirements of large language models without substantially sacrificing their performance. By employing these compression strategies, researchers and practitioners can make LLMs more accessible and deployable in resource-constrrined environments, thereby broadening their applicability across different platforms and devices.
Pruning. Pruning (Frantar& & Alistarh, 2023b Sun et al.,
2024c Shao et al. 2024 Zhang et al. 2024d Dong et al.) 2024b
Tang et al., 2020; Dong et al. [2024a] Lai et al. 2025) involves systematically removing less important parameters or connections within a large language model (LLM) to reduce its size and computational requirements. This technique helps enhance the model's efficiency without significantly compromising its performance by eliminating redundancies. Pruning can be categorized into several types, including unstructured pruning (Sun et al.
2024c] Shao et al. 2024p. which targets
individual weights, and structured pruning (Molchanov et al., 2019 Ma et al.
2023 Kim et al. 2024). which removes entire 2015 methods adjust the
neurons, chamnels, or layers. Additionally, dynamic pruning (Zhang al. sparsity level during training or inference, allowing for more flexible compression tailored to specific deployment needs. Research has shown that judicious pruning can maintain or even sometimes improve model generalization by removing overfitting parameters.
2024d Han et al.
Quantization. Quantization (Park et al. 2024 Frantar Alistarh| 2022 Lee et al., 2024 Du et al. 2024a Kim et al.
2023] Lin et al. 2023; Dong et al., 2024c Gu et al. 2025 Du et al. 2024b Li et al. 20241) reduces the precision of the model's weights and activations from high-precision formats (like 32-bit floating- poinnn to lower-bit representations (such as 8-bit integers). This reduction in numerical precision helps decrease the model's memory footprint and speeds up inference times, making LLMSs more deployable in coource-constrained environments. Quantization can be applied in various forms, including uniform quantization, which uses the same scale for all weights, and non-uniform quantization, which allows different scales for different layers or weight groups. Advanced techniques like quantization-awwe aware training (QAT) integrate the quantization process into the training phase, enabling the model to better adapt to lower precision and mitigate performance degradation. Research in post-training quantization (PTQ) explores methods to quantize pre-trained models without retraining, preserving accuracy while achieving significant compression.
Knowledge Distillation. Knowledge distillation (Hinton et al. 2015b Li et al. 2024b 2023b Liu al. 2023c Tian
et al. (022) is process where a smaller, student model is trained to replicate the behavior of a larger, teacher model. By transferring the knowledge from the teacher to the student, the resulting compressed model retains much of the performance
and capabilities of the original while being more efficient in terms of computation and memory usage. This technique involves training the student model using a combination of the original training data and the outputs (such as logits or probability distributions) produced by the teacher model. Various distillation strategies, including teacher-student learning and self-distillation, have been developed optimize the efficiency and effectiveness of the process. Recent advancements focus on multi-teacher distillation, where multiple teacher models contribute to the training of a single student model, enhancing the diversity and robustness of the compressed model.
Low-Rank Decomposition. Low-rank decomposition techniques (Yuan et al. 2023 Zhang et al.] 2024b Wang et al., 2024e 2025b) aim to approximate the weight matrices of LLMs with lower-rank matrices, effectively reducing the number of parameters and computational operations required during inference. By decomposing large matrices into products of smaller matrices, these methods exploit the inherent redundancy and low-rank structure present in many neural network architectures. Singular Value Decomposition (SVD) is a widely used technique for low-rank approximation, where matrix is factorized into three components with reduced dimensionality. Other methods, such as tensor decomposition and matrix factorization, have also been explored to facilitate efficient compression. Low-rank decomposition not only diminishes the model size but also accelerates inference by simplifying the mathematical operations involved, making it a valuable tool for deploying LLMs in real-time applications.
19

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
A.2. LLM-based Agents
2023 ) AutoGen (Wu 2023). AgentVerse (Chen et al. 2023b), and MegaAgent (Wang et al.] 2024b) are sophisticated Al systems that leverage the powerful natural language understanding and generation capabilities of LLMs as their core reasoning engine to perform complex, autonomous tasks through interaction with their environment. These agents typically integrate LLMs with additional components such as planning modules, tool use capabilities, and memory systems to create more versatile and capable autonomous systems. The combination of these elements allows goal-directed behavior, adapt to dynamic environments, and utilize external resources to achieve specified objectives.
Large Language Model (LLM)-based agents like MetaGPT (Hong et al.
agents to not only generate human-like text but also engage in
LLM-based agents operate by receiving inputs or prompts that define their goals or tasks (Chen al.) 2024a 2023a
Tesfatsion 2023 Wen et al. 2024 Chan al. 2024 Park et al. 2023), processing this information through the LLM to generate plans or actions (Chen et al.) 2023a Wang et al. 2024b c) and then executing these actions in their environment. For instance, an LLM agent designed for web navigation might interpret user queries, plan a sequence of web interactions to gather information, and utilize browser automation tools to execute these interactions effectively. Similarly, an L.LM agent tasked with information gathering can autonomously search, retrieve, and synthesize data from various sources to provide comprehensive insights or summaries.
Recent advancements in LLM-based agents have demonstrated remarkable abilities in diverse applications (Hong et al., 2023b Wang al. 2024blc). ranging from automated customer service bots that can handle complex queries to virtual assistants capable of managing schedules, composing emails, and even providing personalized recommendations. In more specialized domains, LLM agents have been employed for scientific research assistance (Jennings et al.) 1998 Chen et al. 2024a), coding support (Huang et al.) 2023 Zhang et al. 2023alb), b). and creative
2023 Wu et al., 2023 Chen et al.
content generation (Chen et al.
2024a 2023a), showcasing their flexibility and adaptability.
However, the development and deployment of LLM-based agents also raise important considerations regarding reliabil- ity (Zheng et al. 2023b Huang et al.) 2024a Xue et al..
2024 Tonmoy et al.) (0024), safety (Andriushchenko al.) 2024] Tonmoy et al. 2024), and ethical implications (Hua et al. 2023 Andriushchenko al., 2024). Ensuring that these agents act responsibly and ethically requires robust mechanisms for oversight, error correction, and adherence to guidelines that prevent misuse or unintended harmful behaviors. Moreover, the integration of planning and memory systems necessitates careful design to maintain data privacy, security, and compliance with regulatory standards.
Furthermore, performance optimization and resource management (Dasgupta et al., 2023]
Samsi et al. 2023a Zhang
ct al. 2024a| Xia et al.) 20233 Liu al. 2023a) are critical for the efficient functioning of LLM-based agents, especially
when deployed in environments with limited computational resources (Samsi et al., 2023a
Tang et al. 2019 Stojkovic
jet al., 2024: You et al.) 2022). Techniques such as distributed computing You et al. 2022 Wilkins et al.
2024). edge
computing (Babakniya et al., 2023), token optimization (Liu et al. 2023a Xue et al. 2024
Liu et al.) 2024b, Zhang et al.,
2024a), and model compression (Dong et al.] 2024b c Hinton et al. 2015a) strategies play a pivotal role in enhancing
the scalability and responsiveness of these agents, thereby facilitating their broader adoption across various industries and applications.
A.3. Reasoning
Reasoning capabilities (Paul et al.
2023 Wei et al. 2022b] Wang et al., 2024c 2022
Yao et al. 2024 Besta et al. 2024
Luo et al. 2023) Zhou et al.] 2023 Zhang et al. 2023d Xu et al. 2023) in L.MSs refer to their ability to process information
logically, make inferences, and arrive at conclusions through structured thought processes (Jiang et al.
2023b Sun et al..
2023 Zhou et al.] 2023 Han et al. 2022; Sun et al. (024a)). This encompasses various forms of reasoning. including
deductive reasoning (drawing specific conclusions from general principles) (Han et al. [022
Pan al. 2023), inductive
reasoning (forming generalizations based on specific observations) (Sun et al.
2024a] Xu et al.
2024 2023) and analogical
reasoning (applying knowledge from familiar situations to novel ones) (Xu et al.
2024
2023 Amirizaniani et al. 2024).
The sophistication of reasoning in L.MMs is primarily facilitated by their extensive training on diverse textual data, which
enables them to recognize patterns, understand contexts, and simulate logical progression (Han et al. 2022 Pan ct al. 2023
Xu et al. 2024).
Deductive reasoning Xu et al. 2023) in LLMSs allows them to apply established rules or premises derive specific outcomes. For example, given the premises "All humans are mortal" and "Socrates is a human," an LLM can deduce that "Socrates
is mortal." Inductive reasoning (Sun et al. 2024a Xu et al. 2024 2023) enables LLMs to make generalized statements
20

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
based on specific instances, such as inferring that "The sun will rise tomorrow" based on past observations. Analogical reasoning (Xu et al. 2024 2023: Amirizaniani et al. 2024) involves drawing parallels between different scenarios, aiding LLMs in transferring knowledge from one domain to another, which is particularly useful in problem-solving and creative tasks.
Recent research has demonstrated that LLMs can showcase sophisticated reasoning abilities through techniques like chain- of-thought prompting (Wei et al. 2022a; Yao et al. 2024, [Bsta et al. 2024), where the model generates intermediate reasoning steps that lead to the final answer. This not only improves the transparency of the model's 's decision- making process but also enhances the accuracy of its responses. Self-consistency (Wang al. [023c by 2022) checking (Koa et al.
2024. Shiin al., 2024 Madaan al. 2024 Li et al.
2024g; 2023c), another technique, involves generating multiple
reasoning paths (Leblond et al. 2021| Chakraborty et al.
2024 and selecting the most consistent or probable one, thereby
mitigating errors and improving reliability.
However, challenges persist in ensuring that LLMs maintain reliable and consistent logical reasoning across different
contexts and domains. Factors such as ambiguous inputs (Lee & Tiwari 2024 Lu et al. 2023 Gehman et al. 2020). lack of
real-world grounding (Zheng ct al. 2023a; Xie al. 2024] Qin al.)
2023b). and limitations in factual knowledge (Li
ct al. 2024a Chen al.
2022 Sachdeva et al. 2024) can impact the quality of reasoning. To address these issues, ongoing
research focuses integrating structured reasoning frameworks (Sun et al.
2023
Luo et al. [223] Jiang et al., 2023b),
enhancing model architectures to better capture logical relationships (Xu et al.
2024).
and incorporating external knowledge
sources (Jeong et al. 2024; Soudani et al. [2024] Gao et al. 2223a Chen et al.
2024b Asai et al. 2024 Wang et al. 2023d
Yu et al. 2024 that provide factual accuracy and context.
Moreover, the interpretability of reasoning in LLMs is a critical area of interest. Developing methods to visualize and understand the internal reasoning processes of these models can lead better insights into their decision-making mechanisms, facilitating improvements in both performance and trustworthiness. Ensuring that LLMs can reason effectively and transparently is essential for their application in high-stakes environments such as healthcare, law, and scientific research, where accurate and explainable reasoning is paramount.
A.4. Planning
Planning capabilities (Zhou al. 2023 Chen et al., 2023a Song et al.) 2023 Rivera et al. 2024 Valmeekam et al.,
2022 Zhang et al. [023c] in LLMs involve the ability to break down complex tasks into manageable steps and determine appropriate sequences of actions to achieve specific goals. This includes both high-level strategic planning and more detailed tactical planning, enabling LLMs to approach problems systematically and execute multi-step processes effectively. The integration of planning mechanisms enhances the autonomy and efficiency of LLM-based systems, making them more robust and adaptable to a variety of applications.
Recent advances have shown that LLMs can engage in various forms of planning, from simple task decomposition (Li et al. 2024c
Wang et al. 2024b), where a task is broken down into subtasks, to more complex hierarchical planning (Zhou et al. 2023 Wang et a 2024b) approaches that involve multiple layers of strategy and execution. Techniques like tree-of -fttuugght reasoning (Zhou et al., 2023) allow LLMs to explore different branches of possible actions and evaluate their outcomes, fostering more informed rreision-making (Xie et al. 2024). Recursive planning enables models to refine and adapt their plans based on intermediate results or feedback, enhancing their ability to handle dynamic and uncertain environments.
Applications of planning in LLMs span various domains, including roboties (Song et al., 2023: Wang et al., 2023a Zhang et al. (024c), where autonomous agents must navigate and interact with physical environments; software develop- ment (Huang et al. 2024b), where complex coding tasks require structured approaches (Zhang al. 2023c]; and creative industries (Xie et al. 2022). where planning is essential for content creation and project coordination. In each of these areas, the ability to plan effectively enables L.LMs to execute tasks with greater precision, efficiency, and reliability.
Continuous research and development are focused on refining planning algorithms (Hong et al. 2024 Zhou et al. 2023 Ge
al. 2023).
integrating advanced cognitive frameworks, and leveraging hybrid models that combine symbolic reasoning Pan
et al.
2023
Wang et al. 2024d) with neural network-based approaches.
A.5. Tool Use
Tool use capability refers to an LLM's ability to effectively utilize external tools and APIs (Zhang et al., 2023a Schick
a
2024
Liu et al. 2023b Patel et al. 2024 Tang et al.) 2023a Shi et al. 2024 Yang al. 2023b Qin al. 2023b to
21

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
accomplish tasks beyond its native capabilities. This includes interfacing with web browsers, calculators, databases, and other specialized software tools. By integrating external tools, L.LMs can extend their functionalities, enabling them to perform complex computational tasks, access up-to-date information, and interact with various systems seamlessly.
External tool integration involves several key components:
Selection of Appropriate Tools. LLMs must be capable of determining which tools are best suited for a given task. This involves understanding the requirements of the task and matching them with the functionalities offered by available tools. For instance, a task requiring numerical computations might prompt the use of a calculator API, while a task involving data
retrieval might utilize database query tool (Qin et al.] 2023b Shi et al. 2024 Schick et al. 2024 Liu et al. 2023b Zhang
et al. 2023a).
Constructing Valid API Calls. Once an appropriate tool is identified, the LLM must be able to construct valid API calls to interact with the tool. This requires understanding the syntax and parameters of the API, as well as the ability to format requests correctly (Qin et al. 2023b Tang et al. 2023a Shi et al. 2024). Proper API call construction ensures that the tool utilized effectively and returns the desired outcomes.
Interpreting Tool Outputs. After executing an API call, the LLM needs to interpret the output provided by the external tool. This involves parsing the returned data, understanding its structure, and integrating it with the ongoing task response generation (Zhang et al., 2023a [Schick et al. 2023). Effective interpretation enables the LLM to provide coherent and contextually appropriate results based on the tool's output.
Recent developments have shown significant progress in enhancing LLMs' tool use capabilities. Techniques such as tool-specific fine-tuning (Schick et al. [024] [iu et al. 2023b Tang et al. 2023a Shi et al.) (024), where models are trained on the usage patterns of specific tools, have improved the accuracy and reliability of tool interactions. Additionally, the incorporation of contextual understanding allows LLMs to dynamically select and switch between tools based on the task
requirements and the context in which the task is being performed.
Applications of tool use in LLMs are diverse and impactful (Zhang et al. 2023a Schick et al. 2024 Qin et al. 2023b). In customer service, LLMs equipped with tool use capabilities can access knowledge bases, perform troubleshooting steps, and process transactions autonomously, providing efficient and effective support. In data analysis, LLMs can interface with analytical tools and databases to perform complex queries, generate reports, and derive insights from large datasets. In creative industries, tool integration enables LLMs to interact with content creation software (Zhang et al. 2023a), facilitating tasks such as graphic design, video editing, and multimedia production (Liu et al. 2023b).
Furthermore, the ability to utilize tools enhances the problem--olving capabilities of LLMs, allowing them to address a broader range of challenges by leveraging specialized functionalities that are beyond their inherent design (Shi et al. 2024 Schick et al. 2024 Qin et al. 2023b). This synergy between LLMs and external tools leads to more robust, versatile, and capable AI systems that can perform intricate tasks with higher accuracy and efficiency.
A.6. Retrieval Augmented Generation
Aarrival-AAgmenteed Generation (RAG) is an advanced technique that enhances the capabilities of large language models (LLMs) by integrating external knowledge sources during the generation process. Unlike traditional LLMs that rely solely on the information encoded within their parameters, RAG leverages a retrieval component fetch relevant documents or
data from expansive databases or knowledge bases (Jeong et al.) 2024 Soudani al. 20244 Gao et al.) 2023a Chen et al.)
2024b Asai et al. 2024 Wang et al. 2023d Yu et al. 2024). This approach addresses the limitations of fixed- paramparreeter models, enabling them to access da-to-date and domain-speciff information dynamically.
The architecture of RAG typically comprises two primary components: a retriever and a generator (Jeong et al. 2024 Wang et al. 2023d). The retriever utilizes dense or sparse retrieval methods to identify and extract pertinent information related to a given query or prompt from the external corpus. Techniques such as dense vector representations (Pan et al. 2024), which encode semantic similarities, or traditional keyword-ber based approaches, can be employed to optimize the retrieval process. Once the relevant documents are retrieved, the generator, often powered by transformer-based architectures, assimilates this information to produce coherent and contextually accurate responses.
One of the significant advantages of RAG is its ability to mitigate issues related to factual inaccuracies and hallucinations commonly observed in generative models. By grounding the generation process in retrieved factual data, RAG models can produce more reliable and verifiable outputs. This is particularly beneficial for applications in areas such as healthcare, legal,
22

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
and scientific research, where precision and accuracy are paramount.
Recent advancements in RAG have focused on improving the integration between the retrieval and generation stages (Gao et al. 2023b). Innovations such as end-to-end training frameworks allow the retriever and generator to be optimized jointly, enhancing the overall performance and coherence of the generated content. Additionally, scalability challenges are being addressed by developing more efficient retrieval mechanisms that can handle vast and diverse datasets without compromising speed or accuracy.
Applications of Netrieval-Augmenee Generation are diverse and rapidly expanding. In customer service, RAG-powered chatbots can provide more informed and relevant responses by accessing company's knowledge base in real-time. In academic research (Wang et al., 2023d Zhang et al. 2023b Gao et al., (023b), RAG can assist in literature reviews by retrieving and summarizing pertinent studies (Pipitone & Alami 2024). Furthermore, in creative industries, such as content creation (Wu et al. (2224)) and journalism (Wang et al. 2023d RAG can aid in generating well-informed and contextually rich narratives Wu et al. 2024a).
A.7. Long Context
Processing and understanding long contexts is a formidable challenge in the realm of large language models (LLMs), Traditional transformer architectures, while effective in handling short to moderately long sequences, face scalability issues as the input length increases. This limitation arises primarily due to the quadratic complexity of self-attention mechanisms, which hampers efficiency and computational feasibility for extended texts.
To overcome these challenges, several innovative approaches and architectural modifications have been proposed to enhance the long-context capabilities of LLMs:
Sparse Attention Mechanisms Sparse attention reduces the computational burden by limiting the scope of attention to subset of tokens rather than considering all possible token pairs. Techniques such as Longformer's sliding window attention and BigBird's combination of global and random attention patterns enable models to process longer sequences efficiently while maintaining performance on downstream tasks.
Meeory--Augmette Networks Memory augmeeted models incorporate external memory structures that allow the model to store and retrieve information across longer spans. Transformer-Xie for instance, introduces a recurrence mechanism that reuses hidden states from previous segments, effectively extending the context window without a proportional increase in computational resources.
Hierarchical Models Hierarchical approaches decompose the processing of long texts into multiple levels of abstraction. By first encoding smaller chunks of text (e.g., sentences or paragraphs) and then aggregating these encodings at a higher level, models can capture long-range dependencies more effectively. This method not only enhances comprehension but also facilitates better information retention across lengthy documents.
Receptive Field Modifications Adjusting the receptive field of the attention mechanism allows models maintain a broadened scope of context without incurring significant computational costs. Techniques like dynamic attention span, where the model learns the optimal span of attention based the input, contribute to more flexible and efficient long-context processing.
Efficient Positional Encoding Positional encodings play crucial role in maintaining the order of tokens in sequences. For long contexts, standard positional encodings become less effective. Innovations such as relative positional encodings and rotary embeddings help preserve positional information over extended sequences, ensuring that the model retains a coherent understanding of the text structure.
Segment-Based Processing Breaking down long documents into manageable segments and processing them iteratively is another strategy to handle long contexts. This approach involves processing each segment independently while maintaining a summary or representation that captures the essential information from previous segments. Models like Reformer employ such techniques to sustain context over extended inputs.
23

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Adaptive Computation Adaptive computation strategies allow models allocate computational resources dynamically based the complexity and relevance of different parts of the input. By focusing more critical segments and allocating fewer resources to less pertinent sections, models can efficiently manage long contexts without compromising performance.
The ability to effectively handle long contexts is pivotal for numerous applications, including document summarization (Jiang 2024), in-depth question answering, narrative generation, and complex dialogue systems. Enhancing long-context processing not only improves the performance of LLMs on tasks requiring extended reasoning and memory but also expands their applicability across diverse domains where understanding large volumes of information is essential.
Continuous research and development in this are driving the evolution of more sophisticated models that can seamlessly integrate and interpret long-range dependencies, ultimately leading to more intelligent and contextually aware Al systems.
B. Detailed Experiment Settings
B.1. Details of Quantization and Sparsification
For our quantization and sparsification experiments, we evaluated several state-of-the-art compression methods across different model architectures:
Quantization Methods We employed two primary quantization approaches: (1) AWQ (Activation-aware Weight Quanti- zation) (Lin et al.) 2023), which adaptively determines quantization parameters based on activation patterns during inference; and (2) GPTQ (Frantar et al. 2022). post-training quantization method that uses second-ore -oderr information to optimize quantization parameters. We employed the LLMC (Gong et al. 2024) framework for unified quantization and aligned the settings in LLMC.
Sparsification Methods We investigated both structured and unstructured pruning approaches: (1) Magnitude Pruning, both unstructured (Mag(Un)) and structured 2:4 (Mag(2:4)) variants, which remove weights based on their absolute values; (2) Wanda (Sun et al. 2024b), applied in both unstructured (Wanda(Un)) and structured 2:4 (Wanda(2:4)) configurations, offering layer-wise adaptive pruning; and (3) SparseGPT (Frantar & Alistarh 2023a), implemented in unstructured (SparseGPT(Un)) and structured 2:4 (SparseGPT(2:4) formats, using gradient information for pruning decisions. We employed Wanda (Sun et al. 2024b) as the repository for Magnitude, Wanda, and SparseGPT.
All compression methods were applied to pre-trained models without additional fine-tuning. For structured pruning, we maintained the 2:4 sparsity pattern (50% sparsity) across all layers. Unstructured pruning targeted similar overall sparsity levels but allowed for flexible weight removal patterns.
B.2. Details of Small Language Models and Distilled Models
We evaluated several categories of compressed models:
Small Language Models (1) Qwen2.5-3B (Yang et al. 2024b), smaller variant of Qwen, evaluated with both AWQ
and GPTQ quantization; (2) Megrez-3B, a 3B parameter model showing strong performance on multi-doc QA tasks;
(3) MiniCPM-4B (Hu et al., 2024). a compact model optimized for efficiency; (4) Gemma-2B (Team, 2024b). Google's lightweight model focused on general-purpose tasks; and (5) Phi-3.5 (Li et al. 2023d), Microsoft's small-scale model emphasizing reasoning capabilities.
Distilled Models We examined three DeepSeck distilled models TTam 2024a): (1)
distilled version of I.LaMA, showing moderate performance on few-shot learning tasks; (2) Preesseet-RR-DDiiiil--eeee 1.5B, a heavily compressed version of Qwen, optimized for efficiency; and (3) distilled variant maintaining better performance than its smaller counterpart. All models were evaluated across various tasks, including long-context understanding, tool use, and real-world applications, to assess their capabilities and limitations under different scenarios.
larger
B.3. Details of Tool Use Experiments
We mainly employ the T-Eval (Chen et al. 2023c) to conduct experiments. In this dataset, tools were carefully selected from domains such as Research, Travel, Entertainment, Web, Life, and Financials, ensuring high availability and usage
24

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
rate, as well as complete documentation. The dataset includes 553 high-quality query-solution annotation pairs, resulting in total of 23,305 test cases across the Instruct, Plan, Reason, Retrieve, Understand, and Review subsets. The dataset is designed to provide a comprehensive and fine-grained evaluation of tool utilization capabilities, with an average of 5.8 tool calling steps per query, ensuring generalization and discrimination for tool utilization evaluation.
B.4. Details of Long Context Understanding Experiments
LongBench: LongBench (Bai ct al.) 2024 is a comprchensive benchmark for evaluating the long-context capabilitiess of LLMs. It consists of 12 tasks across five categories: Single-Doc QA, Multi-Doc QA, Summarization, Few-shot Learning, and Synthetic tasks. The benchmark features documents ranging from 2K to 32K tokens in length. Single-Doc QA tasks include NarrativeQA and Qasper, which test comprehension of long narratives and scientific papers. Multi-Doc QA encompasses HotpotQA and MultiNews, evaluating tross-document reasoning. The summarization tasks (e.g., GovReport, QMSum) assess models' ability to distill key information from lengthy documents. Few-shot learning tasks test the model's capacity to learn from examples within extended contexts, while synthetic tasks measure specific capabilities like information retrieval and counting in long sequences. The benchmark spans diverse domains including healthcare, legal, scientific research, and creative writing, providing a comprehensive evaluation of long-context understanding.
LongGenBench: LongGienBench (Liu et al. 2024a) is a synthetic benchmark designed to evaluate long-context generation capabilities of language models. Unlike existing long-context benchmarks that focus primarily on retrieval- tasks (e.g.. locating specific information within large input contexts), LongGenBench specifically targets a model's ability to produce coherent, contextually accurate, and structurally sound text over extended outputs. The benchmark supports customizable context length configurations and evaluates models on generating single, unified forg-form responses. This fills an important gap in long-context evaluation by providing standardized framework for assessing generation rather than just retrieval capabilities. The benchmark enables systematic analysis of how different model architectures handle the challenges of maintaining coherence and accuracy in long-form text generation.
theee-intthe-hhyttcck:: This benchmark (Kamradi 2023) specifically tests models' ability to locate and utilize critical information embedded within lengthy contexts. The test suite includes documents ranging from 4K to 64K tokens, with key information deliberately placed varying positions (beginning, middle, end) and with different levels of surrounding distractor content. Tasks include fact retrieval, where specific details must be located within long documents; reasoning chains, where multiple pieces of information must be connected across the document; and verification tasks, which test the model's s ability to find evidence supporting or contradicting given claims. The benchmark employs various document structures (narrative, technical, dialogue) and information patterns (explicit statements, implicit connections, numerical data) to comprehensively evaluate information retrieval capabilities in long contexts. Performance is measured not only on accuracy but also on efficiency in identifying relevant information without being misled by distractors.
B.5. Details of Workflow Generation Experiments
In this paper, we employ WorfBench (Qiao et al. 2024 to evaluate the workflow generation capabilities, which encompass planning and reasoning processes. WorfBench is a comprehensive benehmark containing 18k training samples, 2146 test samples, and 723 held-out tasks across four types of scenarios: function call, problem-solving, embodied planning, and open-grounded planning. The benchmark features multi-faceted scenarios and complex graph- sruph-structured workflows, addressing limitations of existing benchmarks that focus only on linear workflows. WorlBench is accompanied by WorfEval, systematic evaluation protocol that uses subsequence and subgraph matching algorithms to accurately quantify workflow generation abilities. This evaluation framework has revealed significant gaps between linear and graph planning capabilities in current LLMs, with even advanced models showing up to 15% performance differences. The benchmark's 's diverse scenarios and evaluation metrics make particularly suitable for assessing both the quality of generated workflows and their practical applicability in
tasks.
B.6. Details of Real- World Applications Experiments
In this paper, we employ the AgentBoard (Ma et al., 2024) to evaluate real-world applications. The benchmark is designed to comprehensively evaluate large language models (LLMs) as generalist agents, comprising diverse set of 9 unique tasks and 1013 environments. These tasks cover a wide range of scenarios, including embodied AI tasks (AIfWorld, ScienceWorld, BabyAD,, game environments (Jericho, PDDL), web-based tasks (WebShop, WebArena), and tool-oriented tasks (Tool-Query, Vool-Operation). Each environment is carefully crafted to ensure multi-round interactions and partially
25

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Method Performance Analysis (IntemLM-2.5-7BB)
100
80
Overall Score (%)
660
60
40
20
10.5
Instruct
Plan
Reason
Retrieve
Understand
Review
FP15
Mog(Un)
Wanda([In)
Sparso(iP((11))
Map(2:4)
Wanda(2:4)
SparseGPT(224)
Figure 9: Performance comparison of sparse compression methods on tool use tasks
observable characteristics, with subgoals defined or annotated to track detailed progress. The benchmark also introduces a fine Ame-grained progress rate metric capture incremental advancements, providing more nuanced evaluation beyond just the final success rate. Additionally, AgentBoard offers open-source evaluation framework with interactive visualization panel support detailed analysis of agent abilities across various dimensions.
C. More Experiment Results
C.1. Long-Context Understanding
Table presents comparative analysis of the AWQ and GPTQ quantization methods applied to two large language models (LLMss, M2.557 and Qwen2.5-7B, across various long-context understanding tasks sourced from LongBench.
For Intem the application of the AWQ results in significant performance enhancements tasks. Specificallyy, in Single-Doc QA tasks such as NrtvQA, Qasper, and MF-en, AWQ boosts the scores from 0 to 25.08, 29.27 to 45.16 16, and 22.92 to 47.72, respectively. Similarly, in Multi-Doc QA and Summarization tasks, AWQ consistently improves performance metrics, with notable increases in TriviaQA (from 48.1 to 84.69) and Code Completion (from 50 to 99.5). In contrast, the GPTQ method yields more modest improvements. While GPTQ enhances some scores, such as Qasper from 29.27 to 30.16 and MF-en from 22.92 to 30.04, the gains are less pronounced compared to AWQ. Additionally, GPTQ shows slight decreases in certain areas, such Lec, which drops from 58.2 56.62.
evaluated
Examining Qwen2.5-7B, both AWQ and GPTQ quantization methods demonstrate varying levels of performance enhance- ment. AWQ provides consistent, albeit smaller, improvements across most tasks. For example, Qasper increases from 13.12 to 14.18, and MF-en decreases slightly from 30.29 to 26.12. On the other hand, GPTQ offers substantial performance gains in several areas, notably increasing Qasper from 13.12 to 38.94 and MF-en from 30.29 to 47.53. However, similar to InternL.M..577B. GPTQ also results in slight performance declines in specific tasks, such as a reduction Lee from 62,28 to 57.44.
Overall, AWQ tends provide more stable and consistent improvements across broad range of tasks for both models, making it reliable choice for enhancing long-context understanding capabilities. GPTQ, while offering significant boosts in certain key areas, presents more variability in performance outcomes. These findings suggest that the choice between AWQ and GPTQ should be informed by the specific performance requirements and the nature of the tasks at hand.
Table presents a detailed comparison various sparsification methods applied to InternLM2.5-78 and Qwen2.5-7B across different long-context understanding tasks from LongBench. For InternLM2.5-7B, unstructured sparsification methods consistently outperform their structured (2:4) counterparts across most tasks. Among the unstructured approaches,
26

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Method Performance Analysis (IntemLLM-2.5-7B)
100
98
98.7
99
90
Overall Score (%)
80
70
76.
75,3
744
72.5
000
70.4
70
60
65
63.2
59.0
59.0
59.3
55.7
56
50
46.9
40
Plan
Reason
Retrieve
Understand
Instruct
Review
FP16
AWQ
GPTQ
FPS
Figure 10: Performance comparison quantization methods tool use tasks
Table Performance Comparison of AWQ and GPTQ Quantization Methods on Long-Conters Context Understanding Tasks from LongBench
Quantazation
Mf
Hespacts
SWRMMON
Doc
Dhomader
GooReport
Summaitzion
Maaliticon
VC3mm
TREE.
LSITT
Systhetic
0
29.27
22.92
4.43
22.51
15.26
12.8
16.00
12.15
24.15
9.82
41.5
45.1
18.53
21.75
50
58.2
AWRQ
25.08
45.16
47.72
36.52
36.18
25.6
26.26
25.54
24.17
25.82
11.21
69.5
$4.09
30.88
42.5
99.5
61.65
GPTQ
12:29
30.16
30.04
18.53
37.21
21.75
25.84
16.51
11.98
24.21
10.03
40.75
56.04
17.50
20.75
50
56.62
Qwes2.5.7B
26.12
11.84
33.16
21.72
1722
carTQ
28
47.55
$6.55
3331
34.5
18.55
72.5
42.08
Wanda(un) demonstrates superior performance, achieving the highest scores in several categories including NrtvQA (23.94),
Qasper (41.22 .2), ), and MF--n (45.07). The model also maintains strong performance synthetic tasks and code completion, with scores of 99.5 99. for PRE and 58.49 for Lee respectively. SparseGPT(un) follows closely behind, showing particularly strong results in Multi-Doc QA tasks.
For Qwen2.5-7B. the performance pattern differs notably. Magnitude pruning, both structured and unstructured, shows significant degradation across most tasks, with scores dropping to near-zero in many cases. However, SparseGPT and Wanda methods maintain relatively robust performance. Wanda(un) achieves the best overall performance, particularly in Single-Doc QA tasks (16.3 for NrtvQA) and code completion metries (45.14 14 for Lec). SparseGPT(un) also demonstrates strong performance, especially in few-shot learning tasks, achieving 89.35 TriviaQA and maintaining high scores synthetic tasks (92.54 on PRE).
Table 5 presents comprehensive evaluation of small language models and distilled DeepSeek models on LongBench. Among the small models, Qwen-3B demonstrates strong performance across most tasks, particularly when quantized with GPTQ, maintaining comparable performance to base model. For instance, GPTQ Qwen-3B achieves 21.39 on NrtvQA and 49.29 on MF-en, showing minimal degradation from the base model's 22.05 and 49.6 respectively. However, AWQ significantly impacts performance, with near-zero scores on several tasks.
Megrez-3B Infinigence 2024) stands among small models with exceptional performance multi-doc QA tasks (67.62 on HotpotQA, 57.7 on 2WikiMQA) and few-shot learning (82.5 on TREC). In contrast, newer models like MiniCPM-4B, Gemma-2B, and Phi-3.5 show limited capabilities on long-context tasks, particularly struggling with single-doc and multi- PrepSeek--wen--e
doc QA tasks. The distilled models PrepSeek-LLamm--ee moderate performance, with DeepSeck-LLame -B performing better on few-shot learning tasks (87.55 on TriviaQA) but all showing significant degradation QA and summarization tasks compared their teacher models.
SeepSeek-Qwen..
generally show
27

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Table 4: Performance Comparison of Sparsification Methods Across Different Tasks for InternLM2.5-7D and Qwen2.5-7B on LongBench
Spattication
Nevigh
Saagle-Dee
Quageer
Mi
HesporgA
TWILIMEN
MalliDoo QA
Domuater
OssHepent
QNiSan
Summetcation
Vi
TREC
Foresbe Leaming
SANGER
LISHI
Systheti
Code Compocton
Lod LLLA 7B
Mage2-4)
5.01
17.88
33.56
23.35
1562
6.74
30.11
20.5
66.85
3.06
36.76
Magian)
0:58
SpaneGPTius)
21.16
40.13
16.57
22.49
Wanday2-4)
19.34
34.45
25.81
27.6
27.49
148
31.63
42:25
99
0.5
31.45
Wandaton)
23396
40.21
45107
$2.05
34.00
20.1
29.3
29.03
25.23
25.87
18325
81.44
25.75
41
99.5
0
58.09
51 31.96
Mag:2:4)
2.45
1.85
2.92
0.25
Quest 5-28
Meghan)
0.54
0:23
4.77
1.51
2.9
0.53
9.99
1,45
1.23
201
11.3
36.5
21.87
10.90
14
U.S.
10.8
16.59
Spa
3:16
9.16
11.3
10.0
6.3
76.1
30.5
76.16
146%
84.76
41.75
75 96
5.12
90.41
SpaneGPTius)
4.95
12.59
27.35
11.82
12.27
6,39
29
33.52
22.67
25.77
16.47
69.5
89.25
45,51
39
92.54
3.11
29.7
42.03
2.21
6.32
23.25
8.91
9.27
3.4
32.43
28.61
24.01
26.06
16.17
76.5
84.39
63.1
44.5
83.23
2.46
39.34
42.09
Wandaton)
10.3
18.81
20.32
10.86
13.80
77
33.19
32.13
24.83
26.97
17.03
75.5
85.7
40,11
41.5
9339
1,02
45,14
46
Table 5: Performance Comparison of Small Language Models and Distilled DeepSeek Models on LongBench
Compeasoor
Singlo-hae
Quapet
laf--m
Mossaque
Daraakr
GavBapent
TRE
Thiik
Fra-shhi
SAMSS
LSHT
PRE
Synthetc Tadk
Complation
RB.
12.08
19.13
49
46.62
97-11
70.5
56.00
11:11
38.44
25.1
15
QwessWh
AWO
10:41
11.78
1.29
3.5
0
0
1.86
0.11
21.19
2.63
14.5
9.7
6:06
0
0
0
44.66
5.75
GPTQ
21.39
33.77
29.29
45.74
3902
23.9
32.61
34.1
26.44
25.62
15,89
58.5
53.01
44.72
41
25
2.5
50.41
51.47
21.22
37.77
49.38
41.74
32.58
23.24
30.84
30.58
24.87
25.8
15:85
74
$4.03
4402
35
29.5
2.5
37.56
45.67
Qwcn-1.36
AWR
11.99
15:56
182
4.72
0
3.70
0.06
23:36
2,59
13.5
10.16
6:36
0
0
0
28.22
GPPQ
13.04
13.21
1882
4.14
2.62
0.08
22.55
2.57
12.5
10.56
6.53
0
0
0
28.
4.84
Mommass
49
18.5
MinCPMM-4M
13:11
12.99
1.12
2,89
0
1.25
0.32
21.01
2.85
11
9:25
1.37
0
37.29
0.84
Gemma-2h
21:83
20:58
8.36
13.54
0042
0:51
7.08
1.55
22.35
2,46
26.25
19.95
8.5
0.75
56.6
13.00
Phi-3.
12.74
13.26
8.91
3.77
0.05
3.07
0.09
20.3
0.35
24.5
16.04
4.17
4.5
50.97
16.18
DS-LLams-8: DS-Qwen-1.8
Disiiled
16:31
27.5
23.42
57.5
1.5
Disiiled
12:32
6.5
15.02
3.04
32.32
4.75
Qwze-b
Dinille
9.77
7.68
12.7
36.8
22.18
13.14
56.5
33.59
23.75
0.83
Table presents the performance comparison of various quantization and sparsification methods on LongGenBench. For Qwen2.5-7B, AWQ achieves the best performance among all compression methods, with 52.67 GSM8K and 64.39 on MMLU, which is very close to the base model's 's performance (61.33 and 64.65 respectively). However, magnitude-basee pruming methods (both unstructured and 2:4) completely fail on these tasks, achieving 0 scores. Wanda and SparseGPT show moderate performance degradation, with unstructured variants generally performing better than 2:4 structured pruning. For InternLM2.5-7B. most compression methods lead to significant performance drops, with only AWQ maintaining reasonable performance (6.50 on GSMSK and 59.12 on MMLU compared to base model's 10.50 and 61.49).
Table 6: Performance Comparison of Quantization and Sparsification Methods LongGenBencc
LLM
Compression
GSM8K
MMLU
LLM
Compression
GSM8K
MMLU
Base
61.33
64.65
Base
10.50
61.49
Mag(Un)
0.00
0.00
Mag(Un)
1.67
4.91
Mag(2:4)
0.00
0.00
Mag(2:4)
0.00
0.00
Wanda(Un)
35.00
$7.54
Wanda(Un)
7.00
7.02
Qwen2.5-7B
Wanda(2:4)
2.17
37.02
IntemL
Wanda(2:4)
3.33
3.33
SparseGPT(Un)
18.00
58.86
SparseGPT(Unn)
6.00
10.09
SparseGPT(2:4)
7.67
43.16
SparseGPT(2:4)
0.17
1.23
GPTQ
47.50
37.28
GPTQ
10.18
AWQ
52.67
64.39
AWQ
6.50
59.12
C.2. Tool Use
In Table 8 we present a comprehensive evaluation of various compression methods across different L.MSs on tool use tasks. The results reveal several key findings: (1) InternLM2.5-7B demonstrates strong overall performance, with AWQ achieving the best results (68.6%) among compression methods, closely followed by GPTQ (71.8%) and the base model (71.4%). (2) For Mistral-7B, unstructured pruning methods (Mag(Un), SparseGPT(Unn, Wanda(Un) generally outperform their structured counterparts, though with lower absolute performance compared InternLM2.5-11 (3) Qwen2.5-1 5-1---11----- shows
28

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Table 7: Performance Comparison of Small Language Models and DeepSeek Distilled Models on LongGenBench
LLM
Compression
GSM8K
MMLU
GPTQ(W8)
3.67
32.89
Qwen2.5-1.5B
GPTQ(W4)
1.17
35.61
AWQ
2.17
16.49
GPTQ(W8)
11.83
54.39
Qwen2.5-3B
GPTQ(W4)
11.83
50.18
AWQ
14.67
51.93
Phi-3.5
3.33
-1.00
Small LM
Megrez-3b
1.67
6.14
DS DS-Qwen-7b
0.00
0.00
Distilled LM
DS- S5-Qwen-1.. 1.5b
0.00
0.09
DS-LLama-8b
3.67
2.28
impressive resilience compression, with SparseGPT(Un) maintaining performance close FP16 (77.0% 77.5%). (4) Among smaller models, Qwen2.5-7B exhibits strong performance with AWQ and GPTQ (around 70%), while distilled models generally underperform. (5) Across all models, string format tasks typically see better performance than JSON format, suggesting that compression methods better preserve natural language processing capabilities compared to structured format handling.
C.3. Real- World Applications
The AgentBoard evaluation results in Table 1 reveal several key insights about model compression performance. For Qwen2.5-7B, AWQ and GPTQ maintain relatively strong performance compared to the base model, with AWQ achieving 19.73% and 47.16% on Jericho and Tool-query tasks, respectively. However, magnitude- significant degradation, dropping to near across most tasks. For InternI.M2.5-775. the base model performs moderately with 14.23% on Jericho and 31.64% on Tool-query, but compression methods like Wanda and SparseGPT see notable drops in performance. Distillation approaches [DeepSeek-Qwen. poor performance across all tasks, suggesting that knowledge distillation may not be suitable for these complex agent-based scenarios. Overall, the results indicate that while quantization methods like AWQ and GPTQ can maintain reasonable performance, aggressive pruning and distillation lead to significant capability loss on agent-based tasks.
pruning (Mag) shows
DersSeekkQen:: 1.5B,
LLaaseee-LLLaa--B) show very
C.4. Workflow Generation
The WorfBench evaluation results in Table 10 reveal several key insights about model compression performance. For Qwen2.5-7B, quantization methods like AWQ and GPTQ maintain relatively strong performance with FI scores around 0.44-0.46, while pruning methods like Wanda and SparseGPT achieve similar F1 scores in the 0.46 range. However, magnitude-based pruning (Mag) shows complete degradation with 0.00 F1 scores across all tasks. Larger models like Qwen2.5-32B demonstrate better compression robustness, with AWQ and GPTQ maintaining Fl scores around 0.69-0.71 compared the base model's 0.72. Distillation approaches (DeepSeek-RI-Distee models) show significant performance drops with Fl scores between 0.20-0.43. The results suggest that while quantization methods can effectively compress models with minimal performance impact, aggressive pruning and distillation lead to substantial capability loss on workflow generation tasks.
C.5. Needle-in-the--haystcck Test
The needle-in-hayee in-haystack test results reveal several key insights about model compression methods and their impact on long-range attention capabilities. For InterLL--2.5-75 (Figure III we observe that while GPTQ and AWQ maintain relatively stable attention patterns similar to the base model, Wanda and SparseGPT exhibit some degradation in capturing long-range dependencies, particularly beyond the 20k token range. Magnitude pruning shows the most severe impact, with significant attention pattern distortion. For Qwen-2.5-7B (Figure 12 the results demonstrate better compression robustness,
29

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentie Capabilities in LLM Compression
IntemLM-2.5577 Base Model (Uncompressed)
IntemLL--2.5-7B with GPTQ
M--25-L with AWQ
IntemmLM-2.5--7B with Wanda (2:4)
MMterLLM-2.5-7B with Magnitude (2:4)
IntemLLM-2.5-7B with SparseGPT (2:4)
Figure 11: Visualization of attention patterns in needle-in- inaessstcckk tests across different compression methods for InternLM- 2.5-7B. The heatmaps demonstrate how various compression techniques affect the model's ability to maintain attention over long sequences (40k tokens), with darker colors indicating stronger attention weights.
with both GPTQ and AWQ preserving attention patterns remarkably well across the full 40k context. However, structured pruning methods like Magnitude and SparseGPT still show noticeable degradation in long-range attention. The smaller and distilled models (Figure 13) generally exhibit weaker long-range attention capabilities compared to their larger counterparts, SeepSeek-Qwen77B though DeepSeek- Qwen-7B shows promising results in maintaining attention patterns. These visualizations highlight the trade-offs between model compression and the preservation of long-range attention mechanisms, suggesting that careful selection of compression methods is crucial for maintaining model performance on long-context tasks.
C.6. Logits Visualization
Figure 14 presents a comprehensive visualization of logits distributions across different sequence positions in compressed models. The analysis reveals how the models' prediction confidence and token probability distributions evolve throughout the sequence. At early positions (token 2), we observe relatively focused distributions, indicating strong initial predictions. As we progress through the sequence (tokens 82 and 134), the distributions become more diverse, suggesting that the models maintain their ability to consider multiple plausible continuations. Notably, even at later positions (tokens 184, 225, and 246), the compressed models demonstrate stable logits patterns, indicating that they preserve their predictive capabilities across longer contexts. This visualization helps validate that model compression techniques maintain coherent probability distributions across sequence lengths.
C.7. Energy Visualization
Figure 15 illustrates the energy distributions across different sequence positions in compressed models. The visualization reveals interesting patterns in how the models' energy landscapes evolve throughout token generation. At the initial position (token 1), we observe a relatively concentrated energy distribution, suggesting focused model attention. As the sequence progresses through intermediate positions (tokens 64 and 117), the energy distributions show increased spread, indicating the models maintain flexibility in considering multiple possible token paths. At later positions (tokens 175, 244, and 252), the energy patterns remain stable and well-structured, demonstrating that the compressed models preserve their mmegy--based decision-making capabilities even in longer sequences. This analysis validates that our compression techniques successfully maintain the models' ability to capture meaningful energy landscapes across different sequence lengths.
30

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Qwen-2.5-2 2.5-7B Base Model (Uncompressed)
Owen-2.5-7B with GPTQ
Owen-2.5-7B with AWQ
Qwen- 2.5--2 with RTN
Owwn-225-7B with Magnitude (2:4)
Qwen-2.5-7B with SparseGPT (2:4)
Figure 12: Comparative analysis of attention patterns in Qwen-2.5-7B under different compression schemes. The visualiza- tions illustrate the preservation or degradation of attention mechanisms across 40k token context window, highlighting the varying impacts of different compression methods long-range dependency modeling.
DeepSeek SeepSeck-Qwen-1.35 Qwen-1.5B
DeepSeek TernSeek-Qee---P Qwen-7B
Qwen- L.5B with GPTQ (INT4)
Qwen-1.55 with GPTQ (INT8)
Qwen-3B with GPTQ (INT4)
Megrez-3B
Figure 13: Attention pattern analysis for smaller and distilled models across 40k context length. The visualizations demonstrate the varying capabilities of different model architectures and sizes in maintaining coherent attention patterns over long sequences, with particular focus on the effects of model distillation and parameter efficiency.
31

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Table 8: Performance Comparison of Quantization and Sparsification Methods Tool Use Tasks under String and JSON
Formats
Instruct
Plan
Reason
Retrieve String
Understand String
Review Choice
LLMs
Compression
Overall
String
Json
String
Json
String
Json
Json
Json
Mag(2:4)
27.2
13.3
29.3
0.6
41.4
5.0
73.2
1.6
62.8
7.9
10.5
24.8
Mag(Un)
57.8
73.2
27.7
23.1
59.2
19.6
86.4
20.6
73.2
24.2
60.6
47.8
SparseGPT(2:44)
73.2
33.3
52.0
71.1
41.6
35.7
Intern 202.5LM22577
SparseGPT(Unn)
84.5
80.3
42.4
62.3
63.6
39.7
90.9
47.4
74.7
41.5
57.3
62.2
Wanda(2:4)
78.5
46.4
50.6
11.2
59.6
21.9
95.8
18.1
59.7
21.4
61.4
47.7
Wanda(Un)
83.7
90.6
42.2
76.4
64.7
AWQ
98.6
98.7
48.5
45.3
65.8
46.5
85.6
66.7
79.6
56.3
63.2
68.6
FP16
98.6
98.6
44.3
73.7
67.5
51.0
85.7
66.4
81.9
56.0
70.4
72.2
FP8
98.6
98.6
44.3
50.3
83.3
65.5
79.4
55.1
70.4
71.4
GPTQ
96.5
98.4
53.9
76.2
66.9
44.5
93.7
56.9
81.2
49.5
72.5
71.8
Base
98.6
98.6
44.3
73.7
65.9
50.3
83.3
65.5
79.4
55.1
70.4
71.4
Mag(2:4)
3.9
0.3
22.5
0.7
38.1
0.5
45.4
0.2
1.0
11.7
Mag(Un)
0.4
4.2
44.9
33.4
42.5
3.8
37.1
7.1
0.0
6.0
54.4
21.3
SpurseGPT(2:4)
3.6
15.3
43.6
9.9
46.1
3.5
41.0
2.8
0.1
3.7
10.9
16.4
SparseGPT((Un)
92.0
40.8
59.2
56.6
45.8
8.2
28.9
13.9
0.0
13.7
77.8
39.7
Mistral
Wanda(2:4)
74.4
30.8
42.1
51.0
44.9
2.5
48.2
1.0
0.1
4,3
21.6
29.2
Wanda(Un)
88.8
27.3
61.8
70.1
47.0
7.5
29.3
9.8
0.1
68.4
38.1
AWQ
28.7
43.9
4.7
85.4
7.3
43.9
25.1
FP16
28.7
0.0
43.9
0.0
54.8
5.0
88.1
7.5
45.2
7.4
1.8
25.7
FP8
28.7
0.0
43.9
0.0
53.0
4.7
85.4
7.3
43.9
7.2
1.8
25.1
GPTQ
28.7
53.0
4.7
85.4
7.3
43.9
25.1
FP16
98.8
98.5
68.6
82.4
65.0
64.3
94.5
78.7
69.9
67.4
64.3
77.5
Qwen2.5-11 14B
Mag(2:4)
11.7
5.3
62.5
5.7
34.8
19.1
Mag(Un)
85.8
59.6
59.4
54.8
49.0
15.1
74.4
19.5
50.4
15.0
72.9
50.5
SpurseGPT(2:44)
80.0
43.9
52.1
7.0
60.4
31.1
73.0
24.2
68.3
27.0
83.0
50.0
SparseGPT((n))
98.3
98.5
67.4
89.4
77.2
67.6
67.3
77.0
Wanda(2:4)
90.0
63.3
61.0
73.8
61.0
45.2
90.4
51.5
69.6
43.2
78.6
66.1
AWQ
95.0
53.3
67.6
94.9
72.0
62.5
62.4
70.1
FP16
95.0
53.3
67.0
66.7
63.5
57.8
94.8
72.1
62.4
62.8
74.1
70.0
FP8
95.0
53.3
67.0
66.7
63.5
57.8
94.8
72.1
62.4
62.8
74.1
70.0
GPTQ
80.5
75.7
65.3
72.8
62.1
59.3
91.4
74.7
63.9
64.8
70.4
71.0
Qwen2
Mag(2:4)
0.0
0.0
7.0
0.0
14.8
0.0
20.0
0.0
0.9
0.1
0.6
3.9
Mag(Un) Un)
0.1
0.1
2.0
0.0
15.2
0.0
47.6
0.2
14.1
0.1
0.6
7.3
SparseGPT((:4)
27.5
23.4
59.2
6.1
55.8
33.2
90.2
34.5
68.4
32.0
60.0
44.6
SparseGPT((Un)
56.3
52.6
66.2
68.2
61.3
50.0
92.5
55.6
65.1
48.1
67.8
62.2
Wanda(2:4)
38.4
39.1
63.0
33.8
60.4
44.7
89.7
45.3
68.1
44.2
57.7
53.1
Wanda(Un)
68.2
51.6
65.3
32.6
62.0
54.1
95.2
65.2
68.5
57.9
64.3
62.3
DS-L ama--8
FP16
6.1
1.0
37.7
15.2
60.3
34.3
76.8
31.7
0.0
28.0
8.8
27.3
DS- DS-Qwen-1.5B
FP16
10.8
13.8
33.3
8.7
42.4
24.7
55.2
16.9
49.6
16.6
5.1
25.2
7B
FP16
44.6
54.0
42.2
22.8
42.3
71.6
37.8
66.7
34.0
9.2
43.6
32

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Table 9: Task Performance Comparison Across Models and Compression Methods on AgentBoard
Jericho
Pddl
Tool-query
Tool-Operation
Scienceworld
LLMs
Compression
P
P
P
S
S
S
8.21%
0.00%
33.56%
13.33%
44.00%
18.33%
20.00%
0.00%
26.79%
12.22%
Wanda(Un)
11.22%
0.00%
14.00%
3.33%
38.79%
8.33%
15.04%
0.00%
14.98%
6.67%
Wanda(2:4)
1.25%
0.00%
0.56%
0.00%
11.86%
0.00%
3.79%
0.00%
1.80%
0.00%
SparseGPT(Un)
17.39%
0.00%
20.13%
3.33%
35.87%
8.33%
16.58%
0.00%
14.78%
3.33%
Qwen2.5-7B
SpurseGPT(2:4)
5.71%
0.00%
7.46%
1.67%
20.41%
0.00%
7.46%
0.00%
7.89%
0.00%
Mag(Un)
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
Mag(2:4)
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
GPTQ
12.56%
0.00%
22.89%
10.00%
42.27%
15.00%
15.71%
0.00%
26.41 %
13.33%
AWQ
19.73%
0.00%
27.14%
13.33%
47.16%
25.00%
21.46%
0.00%
17.33%
5.56%
14.23%
0.00%
17.64%
0.00%
31.64%
8.33%
7.67%
0.00%
13.91 %
2.22%
Wanda(Un)
6.96%
0.00%
9.96%
1.67%
30.67%
3.33%
18.00%
0.00%
9.79%
1.11%
Wanda(2:4)
0.71%
0.00%
10.72%
1.67%
10.28%
1.67%
14.92%
0.00%
0.00%
0.00%
SparseGPT(Unn)
6,34%
0.00%
3.67%
0.00%
18.89%
0.00%
14.63%
0.00%
8.98%
2.22%
I22557
SparseGPT((:44
0.00%
0.00%
0.00%
0.00%
15.77%
0.00%
1.25%
0.00%
0.00%
0.00%
Mag(Un)
4.55%
0.00%
4.44%
1.67%
18.40%
0.00%
14.00%
0.00%
3.58%
0.00%
Mag(2:4)
4.58%
0.00%
0.00%
0.00%
11.03%
0.00%
1.33%
0.00%
0.00%
0.00%
GPTQ
12.68%
0.00%
25.29%
6.67%
21.69%
5.00%
14.21 %
0.00%
15.12%
4.44%
AWQ
11.22%
5.00%
18.67%
3.33%
34.48%
8.33%
2.08%
0.00%
13.81 %
3.33%
DS-Qwen-7b
Distilled
1.96%
0.00%
1.11%
0.00%
8.52%
0.00%
2.67%
0.00%
2.57%
0.00%
Qw-n--1-lbb
Distilled
3.21% %
0.00%
0.00%
0.00%
5.65%
0.00%
4.67%
0.00%
0.00%
0.00%
DS-LLama-8h 8b
Distilled
0.00%
0.00%
0.00%
0.00%
1.07%
0.00%
0.87%
0.00%
1.24%
0.00%
GPTQ
11.31%
0.00%
14.61%
3.33%
30.37%
11.67%
13.20%
0.00%
17.40%
3.20%
Qwen2,5-3B
AWQ
8.84%
0.00%
23.35%
6.67%
30.65%
6.67%
7.00%
0.00%
16.82%
2.22%
Table 10: Task Performance Comparison Across Models and Compression Methods on WoriBench
Alfworki
TeolAlpaca
ToolBench
Websshep
Average
L.Mss
Comperssion
R
P
R
Fl
R
R
F1
R
F]
SmoothQ[WSAA
0.46
0.56
0.48
0.51
0.64
0.57
0.77
0.92
0.81
0.03
0.04
0.00
0.00
0.00
0.00
0.51
0.70
0.75
0.43
0.45
0.44
GPTQUNT4))
0.48
11.54
11.00
0.77
0.83
11:02
01.02
0.00
0.69
11.74
01.47
0.44
AWQUINT4)
0.55
0.58
0.54
0.51
0.57
0.77
0.81
0.07
0.07
0.07
0.00
0.00
0.70
0.74
0.45
0.46
Qwen2.5-BB
Wanda(Un))
0.55
0.55
0.52
0.67
0.75
0.70
0.75
0.54
0.75
0.08
0.0%
0.08
0.00
0.00
0.00
0.79
0.69
0.73
0.47
0.45
0.46
Wimda(:44
0:53
0:49
0.49
0.63
0.68
0.65
0.82
0.84
0.79
0.09
0.09
0.08
0.00
0.00
0.00
0.78
0.67
0.71
0.47
0.46
0.46
0.49
0.62
0.78
0.07
0.74
0.48
0.60
0.45
0.50
0.59
0.66
0.63
0.82
0.81
0.78
0.09
0.10
0.09
0.00
0.00
0.00
0.82
0.70
0.75
0.49
0.45
0.46
Mag(2:4)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Qwen2.5.3B
GPTOGNTS)
0.39
0.36
0141
0..4
0.68
0.64
0.68
0.62
GPTQUNT4)
0.37
0.35
0.39
0.38
0.69
0.90
0.76
0.48
0.54
0.66
0.73
0.66
0.62
0.64
0.63
AWQUNT4)
0,41
0.42
0.40
0.42
0.38
0.40
0.75
0.96
0.78
0.57
0.71
0.61
0.69
0.76
0.69
0.28
0.30
0.28
0.52
0.57
0.53
Qwen2.5-328
GPTQUNTT))
0.52
0158
0.53
0.64
0.64
0.664
0.81
0.89
0.83
1.60
0.68
0.66
0.78
0.77
0.75
0.50
0.69
0.74
0.71
0.71
0.69
AWQUNT4)
0.69
0.72
0.74
0.55
0.62
0.57
0.69
0.68
0.68
0.84
0:91
0.85
0.74
0.71
0.70
0.76
0.79
0.75
0.83
0.71
0.75
0.73
0.74
0.72
Qwen2.5 14BB
GPTQXINT4)
0.28
0.23
0.34
0.36
0.38
0.57
0.36
0.54
0.42
0.43
0.40
0.40
0.24
0.30
0.26
0149
0.42
0.45
0.36
0.38
0.36
AWQUNT4)
0.60
0.58
0.57
0.76
0.67
0.71
0.81
0.89
0.62
0.66
0.66
0.63
0.70
0.66
0.66
0.30
0.70
0.74
0.72
0.69
0.69
0.55
0.59
0.71
0.82
0.62
0.73
0.67
0.72
0.71
0.71
Qwen2.5-1.58
GPTQUNTT))
0.62
0.52
0.54
0.66
0.71
0.68
0.80
0.84
0.79
0.44
0.66
0.50
0.44
0.65
0.51
0.31
0.70
0.75
0.63
0.68
0.63
GPTQUNT4))
0.63
0.45
0.51
0.70
0.69
0.69
0.77
0.54
0.78
0.45
0.64
0.50
0.50
0.73
0.58
0.52
0.71
0.76
0.65
0.68
0.64
AWQUNT4)
0.61
0.50
0.53
0.61
0.69
0.65
0.78
0.77
0.39
0.69
0.48
0.39
0.62
0.46
0.70
0.75
0.60
0.67
0.61
Mistri-7B--0.3
SmoceQ//Wss)
0.64
0.63
0.61
0.66
0.68
0.67
0.67
0.72
0.00
0.001
0.00
0.00
0.00
0.70
0.73
0.46
0.48
0.46
GPTVNITT)
0.62
0.63
0.61
1..7
0.70
0166
0.66
0.83
0.71
11000
0.001
0.00
0.00
0.00
0.00
0.77
0.70
1.72
0.45
00.45
0.45
GP (PTQUNNT))
0.54
0.57
0.54
0.66
0.69
0.68
0.67
0.86
0.73
0.00
0.00
0.00
0.00
0.00
0.00
0.50
0.70
0.74
0.45
0.47
0.45
AWQUNT4)
0:59
0.63
0.67
0.72
0.76
0.00
0.00
0.69
0.73
0.48
Winda(Un)
0.00
0.00
0.00
0.00
0.00
0.00
0.69
0.75
0.69
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.11
0.13
0.12
Winds(:44)
0.20
0.19
0.19
0.01
0.00
0.01
0.74
0.77
0.73
0.00
0.00
0.00
0.00
0.00
0.00
0.41
0.35
0.38
0.23
0.22
0.22
IntemLM-255
0:48
0.46
0.01
0.01
0.71
0.74
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.22
0.20
S2a)se)PT(22)
0.33
0.47
0.38
0.34
0.36
0.34
0.63
0.67
1.000
0.001
0.00
0.00
0.00
0.65
0.70
0.34
0.38
0.35
MagiUla
0.00
0.00
0.00
9.00
0.00
0.00
0.54
0.46
0.48
0.000
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.000
0.09
0.08
0.08
Mag(2.4)
0.50
0.00
0.00
0.00
0.00
01.00
0.68
0.65
0.64
1.00
0.00
0.00
0.00
0.00
0.00
0.53
0.45
0.48
0.20
0.18
0.19
Distilled
0.00
0.00
0.01
0.69
0.12
0.15
0.28
0.00
0.00
0.25
DS-RI- -iatill- Qwe-- 1.5B
Disilled
0.16
0.30
0.20
0.54
0.63
0.56
0.69
0.85
0.73
0.24
0.51
0.30
0.05
0.07
0.05
0.30
0.68
0.73
0.41
0.5
0.43
DS-R) n-iiill-- Iira-RR
Disilled
0.05
0.04
0.05
8.00
0.01
0.00
0.80
0.81
0.77
0.16
0.78
0.19
0.31
0.44
0.34
0.36
0.31
0.31
0.28
0.31
0.28
Megrex:
0.00
0.00
0.00
0.00
0.00
0.00
0.70
0.73
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.14
0.12
33

---

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Leght
Lognt
1:00
4.0
424
122
0.0
142
54
Logts
Lapss
Logits Distribution at token 2
Logits Distribution at token 82
Logits Distribution at token 134
Legits deetti plan Camparionn Weoitio
Loght BAIr butian Computtion Foctree
Lagas Cierri dien Comppe fone Poodrin
114
10-
684
Lodis
Logits Distribution at token 184
Logits Distribution at token 225
Logits Distribution at token 246
Figure 14: Logits distribution visualization across different sequence positions in compressed models
Every Botr burtan Comparien Portien
Soeryy Beet abon Compatioee Pootler
Chaltioo bork
<:0
Energy Distribution at token|
Energy Distribution at token 64
Energy Distribution at token 117
Enery Secribution Compariton Fedtle
keerg CotvEctic Our Compansinn Notiian 281
Enery Dueritutton Comparionn... Joett
cm
Coc
6.0
segatie Critge
Cernor
Energy Distribution at token 175
Energy Distribution at token 244
Energy Distribution at token 252
Figure 15: Energy distribution analysis at various sequence positions in compressed models
34


Reviews:
[{"review_id": "vCqUMbmzCu", "reviewer": "Reviewer_K5g3", "summary": "The authors introduce ACBench (Agent Compression Benchmark), a benchmark designed to evaluate how compression techniques (quantization and pruning) affect the agentic capabilities of large language models (LLMs), such as multi-step planning, workflow generation, tool use, and long-context understanding. They assess compression across three dimensions: the effect on agentic capabilities, internal model changes (using ERank, Top-K Ranking Correlation, and Energy-based Analysis), and the comparative effectiveness of compression methods (quantization vs. pruning). The authors argue that traditional benchmarks neglect real-world, multi-turn scenarios, making ACBench relevant for practical deployment. Their findings show that quantization preserves structured tasks well but significantly degrades performance on complex real-world applications, whereas pruning typically performs worse. Finally, they also found that distilled reasoning models performed poorly on agentic tasks.\n\n## update after rebuttal\nThank you for the rebuttal. I have upped my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough comments and recognition of our work. We appreciate that you acknowledge our experiments are “comprehensive and sound”. Please see our responses to your questions and concerns below. \n\n**Q1**: About the metrics.\n\n> Can you clarify if (and how) the novel metrics (ERank, Top-K Ranking Correlation, Energy-based Analysis) were validated against simpler, traditional benchmarks and metrics (e.g., perplexity, accuracy, simpler correlation metrics)?\n> \n\n***Ans for Q1***:  \n\nAs explained in the **Abstract and Introduction (lines 97-108)**, our proposed metrics **are not against** traditional evaluation metrics. Instead, they focus on compression explainability by revealing how compression influences LLM behavior and causes degradation. While traditional metrics only reflect the overall performance decay after compression, they cannot show **how compression specifically affects the model behavior.** For example, our Topk Ranking Consistency metric focuses on the inference feature of language models: when quantization/pruning causes he token ranking dislocation. For instance, in an instruction fine-tuning scenario, quantization can reverse the probability order of \"of course\" and \"sorry\" tokens, directly changing the output affective tendency. \n\nBesides, inspired by this question, we explored that **whether our metrics can be used not only to exploiting of compression, but also how to choose compression methods.**  We first conducted experiments on InternLM2.5-20B: \n\n| InternLM2.5 20B | PPL | Hotpot QA | TriviaQA | MultiNews | Lcc | SciWorld | Topk R |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| AWQ | 7.61 | 36.52 | 84.69 | 25.82 | 61.68 | 13.81 | 87.29 |\n| GPTQ | 7.59 | 18.53 | 56.04 | 24.21 | 56.62 | 15.21 | 84.15 |\n| Mag(Un) | 10.39 | 31.59 | 79.26 | 26.2 | 49.72 | 3.58 | 49.44 |\n| SparseGPT(Un) | 7.65 | 34.21 | 84.43 | 26.11 | 49.42 | 8.98 | 57.36 |\n| Wanda(Un) | 7.87 | 32.65 | 87.44 | 25.87 | 58.49 | 9.79 | 58.98 |\n\nThen, we compute the **correlation of traditional metric (acc, ppl) with topk ranking**, and find that the ranking consistency between ppl and topk ranking are relative high, meaning that topk ranking can reflect the evaluation performance in some extend. \n\n| Pearson | PPL | HotpotQA | TriviaQA | MultiNews | Lcc | SciWorld | Topk R |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| PPL | 1 | 0.098 | 0.088 | 0.417 | -0.551 | -0.848 | -0.636 |\n| Topk R | -0.636 | -0.323 | -0.457 | -0.664 | 0.756 | 0.928 | 1 |\n\nCompared with ppl, our topk ranking achieved generally better performance, indicating stronger predictive capability for downstream tasks.\n\n**Q2: About deeper insight of the unexpected degradation**\n\n> Can you elaborate on why distilled reasoning models unexpectedly underperform specifically on agentic tasks? Is this issue related directly to compression methods, or is it indicative of more general difficulties these models have with multi-step reasoning?\n> \n\n***Ans for Q2***:  The underperformance stems from two key factors: \n\n- **Knowledge Gap in R1:** Prior Deepseek V3/R1 lacked agentic capabilities (like tool use) until the recent V3 0324 version. Therefore, the distillation process couldn’t transfer the agentic knowledge from Teacher R1 to student Qwen. This distillation process mainly focus on reasoning and dialogue skills (evidenced by Qwen2.5 32B's improvement from 50.0%to 72.6% in AIME2024)[1].\n- **Capacity Tradeoff in Distillation**: The distilled models tested have limited capacity. When distillation priorizes core reasoning skills (math), agentic skills (function call) may be deprioritized. From the perspective of information bottleneck perspective, we formulate it as:\n  \n    $$\n    \\mathcal{L} _ {\\text{distill}} =I(\\theta_S; y_{\\text{reason}})-\\beta I(\\theta_S; y_{\\text{agent}})+\\lambda\\|\\theta_S\\|\n    $$\n    \n    where $I(\\theta_S; y_{reason})$ maximizes reasoning performance, $\\beta I(\\theta_S; y_{\\text{agent}})$ represents the penalty on agentic skill retention, $\\lambda\\|\\theta_S\\|$ enforces model compactness. So, for capacity-constrained LLMs, maximizing reasoning performance often suppresses agentic capabilities due to competition for parameter space. This suggests that larger models can mitigate this trade-off. In the Wanda (2:4) benchmark, Qwen2.5-14B experienced an 11.4% performance degradation, while Qwen2.5-7B saw a more significant drop of 17%(from Table 7).\n    \n\n[1] DeepSeek AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n**Q3:** About Figures\n\n> Figure 5 is hard to read, and Figure 6 seems incomplete?\n> \n\n***Ans for Q3***: We apologize for the clarity issues. For Fig.5, we improved readability by using darker color. For Fig. 6, we removed the redundant metric and add missing one(SmoothQ) to ensure completeness. You can  check the [anonymous link](https://anonymous.4open.science/r/ICML_ACBench_Rebuttal-B4DA/) over the updated figures.", "claims_and_evidence": "The paper’s core claim—that current benchmarks inadequately capture the performance impacts of compression on agentic capabilities—is convincingly supported by extensive experiments. However, the surprising underperformance of distilled models lacks deeper investigation or explanation, leaving uncertainty about whether this degradation arises specifically from compression or reflects a broader difficulty models have with multi-step tasks.", "methods_and_evaluation_criteria": "The proposed benchmark (ACBench) and evaluation criteria clearly address the identified gap by focusing explicitly on agentic capabilities—such as multi-step planning and long-context understanding—which current benchmarks neglect. The use of novel metrics (ERank, Top-K Ranking Correlation, Energy-based Analysis) to analyze internal model changes is well justified, although these metrics are not explicitly validated against established measures. It would be valuable to directly compare these novel metrics with traditional benchmarks and simpler metrics (such as perplexity, single-turn accuracy, or standard ranking correlations) to confirm whether they provide distinct additional insights. Additionally, the lack of explicit correlation analysis with simpler, existing benchmarks limits the ability to judge ACBench’s unique value, leaving open questions about whether observed performance differences arise specifically from compression or reflect more general model limitations on multi-step tasks.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is comprehensive and sound, evaluating a broad range of compressed models and techniques across diverse agentic tasks. However, a key limitation is that the experiments do not explicitly separate baseline model limitations from compression-specific degradation, nor do they compare or correlate performance systematically with simpler, traditional benchmarks.", "relation_to_broader_scientific_literature": "The key contribution of this paper—introducing the ACBench to systematically evaluate compression effects on agentic capabilities—addresses a specific gap in the broader literature. Prior research extensively studied compression techniques (quantization, pruning, distillation) mainly on traditional NLP benchmarks like GLUE or perplexity-based tasks. However, this literature rarely investigates how these techniques impact more complex, interactive, and multi-step tasks. ACBench builds upon recent work on evaluating and benchmarking LLMs’ agentic capabilities (such as workflow generation, long-context comprehension, and tool use), explicitly linking the compression literature to the emerging field of interactive agents. By doing so, the paper connects two previously distinct streams of research—model compression efficiency and agentic AI performance—offering insights particularly relevant for practical LLM deployments in resource-constrained, real-world scenarios.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "I like the general idea of the paper, but as expressed above, I am concerned about: 1) the lack of explicit validation and comparison of the novel metrics (ERank, Top-K Ranking Correlation, Energy-based Analysis) against simpler, existing benchmarks and metrics, leaving their unique added value unclear; and 2) the insufficient exploration and explanation of why distilled reasoning models underperform specifically on agentic tasks, raising ambiguity about whether this is due to compression techniques or more fundamental limitations of models in multi-step reasoning scenarios.", "other_comments_or_suggestions": "Figure 5 is hard to read, and Figure 6 seems incomplete?", "questions_for_authors": "1) Can you clarify if (and how) the novel metrics (ERank, Top-K Ranking Correlation, Energy-based Analysis) were validated against simpler, traditional benchmarks and metrics (e.g., perplexity, accuracy, simpler correlation metrics)?\n\n2) Can you elaborate on why distilled reasoning models unexpectedly underperform specifically on agentic tasks? Is this issue related directly to compression methods, or is it indicative of more general difficulties these models have with multi-step reasoning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3yniRydab", "reviewer": "Reviewer_DuJV", "summary": "Large language models (LLMs) have significantly advanced areas such as code synthesis and multi-agent collaboration; however, their practical deployment remains constrained due to substantial computational and memory requirements. Compression techniques, including pruning and quantization, effectively reduce model size but frequently neglect crucial agent capabilities like planning, coherence, and tool integration.\n\nThis paper proposes the Agent Compression Benchmark (ACB), aiming to comprehensively assess the effects of compression methods—pruning (SparseGPT, Wanda) and quantization (GPTQ, AWQ)—on LLMs. The benchmark specifically evaluates:\n\n1. Agent Capabilities: Action execution, long-context coherence, and tool integration.\n\n2. Model Impact: Assessed through ERank, Top-K Ranking Correlation, and energy-based analyses.\n\n3. Compression Comparison: Evaluates a range of models (from <7B up to 32B parameters) to provide insights for selecting optimal compression strategies without significantly compromising agent performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: About the practical utility of the proposed metrics:\n\n> the proposed metrics significantly enhance understanding of compression impacts is less convincing, as their practical utility remains unclear. \n\n**Ans for Q1**: . We would like to address each metric:\n\n- **ERank**: Diff-eRank[1] is a theoretically grounded method based on information theory and geometric principles. It analyzed differences between base and trained models using ERank. We extend it to quantized LLMs shows ERank's effectiveness with experimental results (extending Tab.1 of Diff-eRank), where higher ERank values correlate with better model performance (see below table).\n- **Top-K Ranking Correlation**: As detailed in Q1 to Reviewer K5g3, this metric provides meaningful insights by focusing on the model's behavior regarding top-k token ranking. The correlation analysis captures important aspects of the model's predictive distribution.\n- **Energy-based Analysis**: We find that compressed models' energy distributions gradually align with uncompressed ones over timesteps (Fig.14), showing that while quantization disrupts LLM representations, parameter redundancy compensates for the loss. Aggregated energy reflects this recovery and correlates with performance (see below table).\n\n| OPT          | 125M   | 1.3B   | 2.7B   | 6.7B   |\n| ------------ | ------ | ------ | ------ | ------ |\n| ACC          | 0.276  | 0.332  | 0.370  | 0.360  |\n| delta Loss   | 5.734  | 6.138  | 6.204  | 6.258  |\n| Diff-ERank   | 1.410  | 2.140  | 2.338  | 2.280  |\n| ERank (4bit) | 15.462 | 15.589 | 13.898 | 17.877 |\n| Energy       | 2.738  | 2.746  | 2.631  | 2.883  |\n\n[1] Lai Wei etc, Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models, NeurIPS 2024 \n\n\n**Q2**: About the connection between the metrics with traditional metric.\n\n> the connection between the experimental outcomes and the practical implications of the proposed metrics requires additional clarification. How the proposed metrics correlate with the specific benchmarks' performance metrics.\n\n***Ans for Q2***: Our proposed three metrics serve as complementary tools to traditional benchmarks, offering causal insights into performance variations. Specifically, they help explain why certain models achieve better or worse results on standard metrics like perplexity (PPL) or accuracy. Also, please refer to our response to **Reviewer K5g3 (Q1)**, where we provide additional experiments demonstrating how **Top-K Ranking Correlation** reflects changes in traditional benchmark performance. You can  check the [anonymous link](https://anonymous.4open.science/r/ICML_ACBench_Rebuttal-B4DA/) about the topk-ranking results.\n\n**Q3**: **Guiding Principles for Practitioners**\n\n> The paper lacks deeper insights and guiding principles on how practitioners might effectively leverage these extensive observations to make informed decisions.\n\n***Ans for Q3***: We have refined our guidelines to provide clearer, actionable insights for practitioners:  \n\n1. For Specific Agent Capabilities:  \n   - If targeting single capability like tool use, directly consult the task-specific results in Sections 4-7 to select the optimal compression method.  \n   - eg: For tool use, AWQ preserves JSON-structured outputs better than GPTQ (Fig. 3); for long-context, AWQ surpass GPTQ in most of cases.  \n\n2. For General-Purpose Agent Deployment:  \n   - Model Choice > Compression Method: Base model capability is critical. For instance, Qwen2.5 outperforms R1-Distill-Qwen2.5  in four capabilies (Tab. 8). Prioritize high-quality base models first.  \n   - Default to AWQ:  AWQ shows stable performance across all benchmarks (workflow, tool use, long-context).  \n   - If Quantization is Infeasible: Use Wanda (outperforms SparseGPT in 80% of cases).  \n   - Avoid R1-Distill for Agents: Despite its reasoning strengths, it fails in real-world agent tasks (Fig. 7). Use quantized base models instead.  \n\n3. For hybrid scenarios (e.g., workflow + tool use):  Start with Qwen2.5-7B (strong base) → Apply AWQ.\n\nYou can refer to [anonymous link](https://anonymous.4open.science/r/ICML_ACBench_Rebuttal-B4DA/) for guideline flow chart.\n\n**Q4**: About the importance：\n\n> Could you elaborate on the significance of your proposed metrics to the main contributions of the paper?\n\n***Ans for Q4***: \n\nThese metrics are foundational to our three key contributions:\n\n- ERank can reflect how the information are compressed from the perspective of information theory. **See Q1 to Reviewer DuJV**.\n- Topk Ranking Correlation is intuitive over how the compression influence sampling process. we also added additional experiments to show that topk ranking correlation have the ability to predict the downstream performance.  **See Q1 to Reviewer K5g3**.\n- Energy is helpful for us to understand what is going on during the decoding stages, where as the time goes, the distribution shifted to uncompressed one and become stable. **See Q1 to Reviewer DuJV**", "claims_and_evidence": "The primary goal of this paper—to extensively evaluate compression impacts across multiple dimensions—is clearly presented, and the paper thoroughly quantifies these effects across various models and settings. However, the claim that the proposed metrics (ERank, Top-K Ranking Correlation, and energy-based methods) significantly enhance understanding of compression impacts is less convincing, as their practical utility remains unclear.", "methods_and_evaluation_criteria": "The selected models, datasets, and metrics are comprehensive and well-chosen. Nevertheless, the paper does not clearly articulate how the proposed metrics meaningfully contribute to a broad quantitative analysis. More explanation is needed on how these metrics translate into practical insights across diverse workloads.", "theoretical_claims": "The theoretical justifications and statistical analyses provided by the authors are reasonable and supported effectively by motivational diagrams.", "experimental_designs_or_analyses": "As noted in the methods section, the experiments are comprehensive and rigorous. However, the connection between the experimental outcomes and the practical implications of the proposed metrics requires additional clarification.", "relation_to_broader_scientific_literature": "The significant contribution of this paper lies in presenting a curated benchmark suite for comprehensively evaluating different facets of model compression. However, the paper lacks deeper insights and guiding principles on how practitioners might effectively leverage these extensive observations to make informed decisions.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "See \"Other Comments or Suggestions\" below.", "other_comments_or_suggestions": "I find it challenging to clearly understand how the proposed metrics (ERank, Top-K Ranking Correlation, and energy-based metrics) correlate with the specific benchmarks' performance metrics. Certain compression techniques perform well in specific scenarios but underperform in others. Although trends in the proposed metrics are observable, it is unclear how practitioners should interpret these results practically to predict or understand performance across diverse workloads.", "questions_for_authors": "Could you elaborate on the significance of your proposed metrics to the main contributions of the paper? Specifically, why are these metrics important, and how can practitioners leverage them effectively to make informed decisions when choosing appropriate compression techniques for novel tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xksca9mmd2", "reviewer": "Reviewer_Sovg", "summary": "This is a very interesting paper that studies agentic capabilities in LLM compression. The authors have carefully selected a series of evaluation benchmarks that cover practical scenarios of agent manipulation to assess the performance drop after compression.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Sovg,\n\nThank you for your thorough and constructive review of our work. We sincerely appreciate your recognition of the experimental rigor and practical relevance of our evaluation benchmarks, as well as your encouraging feedback. Your insights strongly support our goal of providing actionable guidance on the trade-offs in LLM compression for agentic workflows.\n\nWe are pleased that the detailed results and visualizations in the appendix proved useful—ensuring methodological transparency was a key priority for us. Should you have any further suggestions or require additional clarifications for the final version, we would be happy to incorporate them.\n\nThank you once again for your time and thoughtful evaluation. We look forward to any additional comments you may have.\n\nBest regards, \n\nAuthors of #6400", "claims_and_evidence": "Yes, the claims are well supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed evaluation criteria are entirely reasonable.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The experimental design is extensive and well-structured, providing a detailed evaluation of all aspects of agentic abilities. It also tests across a variety of language models.", "relation_to_broader_scientific_literature": "This paper provides a valuable window for readers to understand the influence of compression on agentic workflows and serves as a guide on which compression methods to choose. It contains many valuable insights.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This paper is excellent. Well done!", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMRR4o230m", "reviewer": "Reviewer_vD1Z", "summary": "The ACBench framework proposed is designed to systematically evaluate the effects of compression on both agent capabilities and large language models (LLMs). It tests agent capabilities across key areas such as action execution, workflow generation, long-context understanding, and real-world application performance. For LLMs, it evaluates efficient rank, top-K ranking consistency, and energy-based analysis to measure the impact on model efficiency and output reliability. Additionally, ACBench analyzes the impact of different compression approaches, providing insights into their trade-offs and suitability for various tasks. This framework serves as a robust tool for optimizing compressed models while maintaining performance and practicality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: About Theory.\n\n> Theoretical claims are poor for this article. What is the relationship between the degradation of the compression has on LLMs and the degradation of the compression has on the Agentic Behaviors? \n> \n\n***Ans for Q1***: This paper is primarily an empirical study, as indicated by the title. To address this, we have developed a concise framework to explain how quantization errors propagate in sequential decision-making (agent scenarios). For details, please refer to this [anonymous link](https://anonymous.4open.science/r/ICML_ACBench_Rebuttal-B4DA/).\n\n**Q2**: About the setup for Action Execution\n\n> Action Execution aspect, lacks evaluation metrics setup.\n> \n\n***Ans for Q2***: The Action Execution metrics (Sec. 4) adopt T-Eval's setting, evaluating **Plan** (similarity/LIS), **Reason/Understand** (Sentence-BERT), **Retrieve** (exact match), **Instruct** (format/parameter accuracy), and **Review** (classification accuracy). \n\n**Q3**: About the Embodied AI tasks.\n\n> The evaluation in Workflow Generation and Real-World Application seem to have overshadowing parts about the Embodied AI task.\n> \n\n***Ans for Q3***:  In this paper, we employed **ScienceWorld** and **AlfWorld** as they inherently require structured, multi-step reasoning, making them ideal for evaluating *workflow generation*. BabyAI, while useful for basic instruction-following, lacks the complexity needed for higher-level planning assessment. Refer to Tab.8,9 and  [anonymous link](https://anonymous.4open.science/r/ICML_ACBench_Rebuttal-B4DA/) for more details.\n\n**Q4**: About the compression methods\n\n> The compression methods are limited to only five compression methods. LoRA and Distillation are not systematically analyzed. \n>\n> The passage use results from distilled model and original models to evaluate the distillation methods but not thoroughly analyzed.\n>\n>  Larger Models (only contains size from 1.5B to 7B) are not evaluated.\n\n***Ans for Q4***: As presented in Sec.2.2 and Sec.8, we prioritize these compression methods for the following reasons:\n\n1. **Practical Impact & Compatibility**: The selected methods (GPTQ, AWQ, SmoothQuant, SparseGPT, Wanda) are foundational and widely adopted in serving systems, which are supported by vLLM[3] and SGLang[4]. They are critical for real-world high-throughput serving (e.g., 10–24× speedups over HuggingFace).\n2. **Distillation/LoRA**:  While we include distilled models (e.g., R1-Distill series) in baseline comparisons, a systematic evaluation of distillation or LoRA lies beyond our focus on post-training compression. These techniques inherently require retraining or architectural modifications.\n3. **Model Scale**: We evaluate up to **Qwen2.5-32B** (Tab.9), but larger models (e.g., 70B) are prohibitively expensive (>1 month/run on 8 GPUs). Memory constraints in long-context agent tasks further limit scaling.\n\n**Q5**: About the evaluation\n\n> Four capabilities should be evaluated multi-step planning, long-context coherence, adaptive reasoning. How is each Benchmark related to these core aspects?\n> \n\n***Ans for Q5***: Thank you for raising this important point. We explicitly connect each benchmark to the core capabilities in the following ways:\n\n- **Multi-step planning** is evaluated in **Workflow (Sec. 5)** and **Real-World Applications (Sec. 7)**.\n- **Long-context coherence** is assessed in **Long-Context (Sec. 6)**.\n- **Adaptive reasoning** is demonstrated in both **Workflow (Sec. 5)** and **Real-World Applications (Sec. 7)**.\n\nIn line 35-52, we have already refined these tasks. We will further enhance clarity in the revised manuscript \n\n**Q6**: About the writing.\n\n> Some expressions are vague. For example, “Knowledge Distillation Shows Unexpected Performance Characteristics” is not a good summary sentence to conclude without showing what characteristic is.\n> \n\n***Ans for Q6***:   To address this, we have revised the statement to: \n\n> *\"Knowledge distillation from a reasoning model lead to performance degradation in agent scenarios.\"* \n\n\n\n**Q7**: About the Optimal Strategy:\n\n> Optimal Compression Strategy Recommendations proposed in Chapter 4 should be analyzed after all the aspects are evaluated\n> \n\n***Ans for Q7***: To align with the feedback, I propose refining the section title to **\"Compression Strategy Recommendations for Action Execution\"**. And for overall guidings, please refer to **Q3 to Reviewer DuJV**. \n\n\n\n**Q8**: About the \"Multi-Turn\".\n\n> The framework stresses the importance of “Multi-Turn” conversation testing, but it is not highlighted in the following experiments\n> \n\n***Ans for Q8***: Thank you for your feedback. The \"Multi-Turn\" is implicitly integrated into the experimental design: \n\n- In Sec. 5): Tasks in WorfBench inherently involve multi-turn interactions, as agents must dynamically plan and execute\n- In Sec.7, Benchmarks like Jericho and PDDL explicitly test multi-turn reasoning. For instance, agents must navigate 10+ conversational turns to solve complex puzzles", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "Theoretical claims are poor for this article.", "experimental_designs_or_analyses": "1.\tIt seems like the ACB mainly categorizes the existing Benchmark, while the one proposed by the essay, Action Execution aspect, lacks evaluation metrics setup. \n2.\tThe evaluation in Workflow Generation and Real-World Application seem to have overshadowing parts about the Embodied AI task.", "relation_to_broader_scientific_literature": "Integration of several Benchmarks and systematically propose the evaluation aspects for agentic behaviors.", "essential_references_not_discussed": "1.\tThe compression methods are limited to only five compression methods, namely GPTQ, AWQ, SmoothQuant, SparseGPT, Wanda. (For Quantization and Pruning)\n2.\tOther compression methods like LoRA and Distillation are not systematically analyzed. (The passage use results from distilled model and original models  to evaluate the distillation methods but not thoroughly analyzed.)\n3.\tLarger Models (only contains size from 1.5B to 7B) are not evaluated.", "other_strengths_and_weaknesses": "Strength\n1.\tThe essay conducts thorough and comprehensive experiments and provide rather abundant elaboration on the methods and benchmarks it uses. \n2.\tThe essay compares different models regarding different fields and uses multiple metrics to evaluate those agents’ ability.\n\nWeakness\n1.\tMaybe need categorization for difference Benchmarks, like what capabilities are each Benchmark is testing. The results analyzed seem irrelevant (For example, you mention that four capabilities should be evaluated multi-step planning, long-context coherence, adaptive reasoning. How is each Benchmark related to these core aspects?)\n2.\tSome expressions are vague. For example, “Knowledge Distillation Shows Unexpected Performance Characteristics” is not a good summary sentence to conclude without showing what characteristic is.\n3.\tWhat is the relationship between the degradation of the compression has on LLMs and the degradation of the compression has on the Agentic Behaviors? You seem to analyze them separately, but more correlation analysis is expected. \n4.\tOptimal Compression Strategy Recommendations proposed in Chapter 4 should be analyzed after all the aspects are evaluated\n5.\tThe framework stresses the importance of “Multi-Turn” conversation testing, but it is not highlighted in the following experiments", "other_comments_or_suggestions": "Please refer to the section Other Strengths and Weaknesses", "questions_for_authors": "Please refer to the section Other Strengths and Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new evaluation framework called ACBench. It assesses how post-training compression methods affect the agentic capabilities of large language models. The advantages include: (1) It addresses a clear gap in evaluating agent-like behaviors in LLMs post-compression; (2) it provides actionable insights for deploying compressed LLMs in real-world multi-turn scenarios; (3) it compares various compression techniques on small to medium models. However, the major concern is that this paper is empirical and lacks theoretical formalism to explain observed effects, especially compression vs agentic degradation. Another concern is that it focuses mostly on <32B models and omits other methods like LoRA or larger models. Their practical utility remains partially unclear to some reviewers. Overall, this is a good paper and should be considered if there is room in the program.
Decision: Accept (poster)