Tool Unlearning for Tool-Augmented LLMs
Authors: Jiali Cheng
Hadi Amiri
Venue: ICML 2025 poster (2025)
Keywords: Tool-Augmented LLM
Machine Unlearning
Primary area: deep_learning->everything_else

Abstract: Tool-augmented large language models (LLMs) may need to forget learned tools due to security concerns, privacy restrictions, or deprecated tools. However, ``tool unlearning'' has not been investigated in machine unlearning literature. We introduce this novel task, which requires addressing distinct challenges compared to traditional unlearning: knowledge removal rather than forgetting individual samples, the high cost of optimizing LLMs, and the need for principled evaluation metrics. To bridge these gaps, we propose ToolDelete , the first approach for unlearning tools from tool-augmented LLMs which implements three properties for effective tool unlearning, and a new membership inference attack (MIA) model for evaluation. Experiments on three tool learning datasets and tool-augmented LLMs show that ToolDelete effectively unlearns both randomly selected and category-specific tools, while preserving the LLM's knowledge on non-deleted tools and maintaining performance on general tasks.

Tool Unlearning for Tool-Augmented LLMs
Jiali Cheng' Hadi Amiri'
Abstract
pairs, which embed the ability use tools directly into model parameters.
Tool-augmented large language models (LLMs) are often trained on datasets of query-response pairs, which embed the ability to use tools or APIs directly into the parametric knowledge of LLMs. As these models are increasingly deployed in real-world applications, there is a need for them forget specific tools-for example, due security vulnerabilities, privacy regulations, tool deprecation. This work presents "tool un- learning" as a novel machine unlearning task that presents distinct challenges beyond traditional sample-level unlearning: it requires removing functional knowledge rather than individual data points, managing the high cost of LLM opti- mization, and developing principled evaluation metrics. To address these challenges, we propose TEOLDELETE, the first unlearning framework designed specifically for tool-angmented L.LMs. It implements three key properties for effective tool unlearning and introduces a new membership inference attack (MIA) model for effective evaluation. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that TOOLDELETE effectively unlearns both randomly selected and class-specific tools, while preserving knowledge on remaining tools and maintaining performance on general tasks.
Despite the growing adoption of augmentedd LLMs, the ability to selectively unlearn tools has not been investigated. In real-world applications, tool unlcarning is essential for addressing critical concerns such as security, privacy, and model reliability. For example, consider noul-augmented LLM deployed in healtheare system and trained to APIs for handling patient data. If one of the APIs is later flagged as insecure due to vulnerability that could expose sensitive information and violate regulations like HIPAA, tool unlearning is necessary to ensure that the LLM can no longer invoke the insecure API. Similarly, when tools undergo major updates, such as the Python transformers package moving from version ing becomes essential to prevent the LLM from generating outdated or erroneous code. The goal of this work is to ad- dress this gap by investigating tool unlearning and providing a solution for this crucial task.
version 4, tool unlearn-
We introduce and formalize the new task of Tool Unlearn- ing, which aims to remove the ability of using specific tools from tool-augmented LLM while preserving its ability to use other tools and perform general tasks of LLMs such as coherent text generation. Ideally, an effective tool unlearn- ing model should behave as if it had never learned the tools marked for unlearning. Tool unlearning fundamentally dif- fers from traditional sample-level unlcarning as it focuses on removing "skills" or the ability to use specific tools, rather than removing individual data samples from model. In ad- dition, success in tool unlearning should be measured by the model's ability to forget or retain tool-related skills, which differs from traditional metrics such as measuring likelihood of extracting training data in sample-level unlearning. These differences are discussed in detail in $2.
1. Introduction
Tool-augmented Large Language Models (LLMs) can use external tools such as calculators (Schick et al., 2023), Python interpretors (Gao et al., 2023), APIs (Tang et al., 2023), or AI models (Patil et al., 2023) to complement the parametrie knowledge of vanilla LLMs and enable them to solve more complex tasks (Schick al., 2023; Patil et al., 2023). They are often trained on query-response
Removing skills requires modifying the parameters of LLMs, process that is computationally expensive and can lead to unforeseen behaviors (Cohen et al., 2024; Gu et al., 2024). In addition, existing membership inference attack (MIA) techniques, common evaluation method in machine unlearning to determine whether specific data samples were part of training data, are inadequate for evaluating tool un- learning because they focus on sample-level data rather than tool-based knowledge.
Uniiversity of Massachusetts Lowell, USA. Correspon- dence to: Jiali Cheng <jiali_cheng@e @um..eduuull...... Hadi Amiri Liaaiiamiri@uur eml.edu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Tool Unlearning for Tool-Augmented LLMS
To address these challenges, we propose TOOLDELETE, the first tool unlearning algorithm for tool-augmented LLMss, which satisfies three key properties for effective tool un- learning: tool knowledge removal, which focuses on remov- ing any knowledge gained on tools marked for unlearning; tool knowledge retention, which focuses on preserving the knowledge gained on other remaining tools; and generul capability retention, which maintains LLM's general ca- pability on range of general tasks such as text and code generation using ideas from task arithmetic (Ilharco et al., 2023; Bärbulescu & Triantafillou, 2024). In addition, we develop LiRA-Tool, an adaptation of the Likelihood Ratio Attack (LiRA) (Carlini et al., 2022; Pawelczyk et al., 2024) to tool unlearning, to assess whether tool-related knowledge has been successfully unlearned. Our contributions are:
Use Cases of Tool Unlearning The ability to forget learned tools is essential in real-world applications. For example, addressing the insecure tools from untrustworthy developers that could be exploited by adversarial attackers; removing tools restricted by their providers due to copy- right or privacy concerns, such as APIS that start allowing unauthorized downloads of book chapters or releasing pub- lications that users did not author; unlearning broken or deprecated tool that lead to failed operations or corrupted outputs; unlearning tools that may no longer be needed; and managing limited model capacity, where new versions of tools necessitate replacing outdated ones. More examples of parameter-level tool unlearning are provided in Appendix A.
Tool unlearn- ing is different from sample-level unlearning as it focuses on removing "skills" rather than individual training samples. Objective: sample-level unlearning aims to reduce the mem- orization likelihood or extraction probabilities of specific data samples (g:: ys) (Jang et al., 2023), which is useful for removing copyrighted or private information. In contrast, tool unlearning targets the "aailitt" to solve tasks using tools marked for unlearning (Ty). For example, generating f' (4) that is superficially different from yi (while preserv- ing the semantics) is considered successful for sample-level unlcarning. However, for tool unlearning, preserving skills and semantics indicate maintained knowledge on Ty, which makes unlearning a failure. Figure 1b shows successful tool unlearning, where the ability to the API is forgotten, de- spite the high lexical memorization between output of the un- learned model and the training data. In addition, selectively removing knowledge from tool-augmented models is a chal- lenging tasks because changes to one tool may uncxpectedly affect the model's ability to use other tools-referred as ripple effect in fact editing literature (Cohen et al., 2024; Gu et al., 2024). Furthermore, LLMs are general models that can conduct a wide range of tasks beyond tool using, and this ability must be retained. Evaluation: metrics like sequence extraction likelihood and perplexity are standard in sample- level unlearning. For tool unlearning, success is measured by the ability to forget or retain tool-related skills, which is more appropriate. Data: sample-level unlearning require ac- cess to all individual samples marked for unlearning, while tool unlearning does not. This aligns with "concept erasure" in diffusion models (Gandikota et al., 2023; Kumari et al., 2023) and zero-shot unlcarning (Chundawat et al., 2023) but differs from traditional LLM unlearning (Yao et al., 2024). Later we demonstrate this in $ 5.
Difference to Standard Unlearning Tasks
introducing and conceptualizing tool unlearning for noll-augmented LLMs.
TOOLDELETE, which implements three key properties for effective tool unlearming:
LiRA-Tool, which is the first membership inference attack (MIA) for tool unlearning.
Extensive experiments on multiple datasets and tool- augmented LLMS show that DEOL.E outperforms existing general and LLM-specific unlearning algorithms by 12.5 and 9.1 in accuracy forget tools and retain tools respectively. In addition, it can save 74.8% of training time compared to retraining, handle sequential unlearning re- quests, and retain 95% performance in low resource setting.
2. Tool Unlearning: Preliminaries
To understand tool unlearning, we first introduce the concept of "tool learning," see Figure (a). Let D = (T, 27 be a dataset with NN tools T, and (Q, )) denotes query-output examples that demonstrate how to use the tools in T. Each tool & E 'T may have one or more demonstrations (2,)), |0i1 = 1 Z 1. Starting with an instruction-tuned LLM fo. a tool learning algorithm explicitly trains fo on D and results in tool-augmented model capable of using the N tools in T. We note that prior to explicit tool learning, the LLM So may already have some tool-using capabilities such as performing basic arithmetic operations.
Problem Definition: Tool unlearning aims to remove specific tools from tool-augmented LLMs. Let D, (T,, 3'y denotes k N tools and their corresponding demonstrations to be unlearned from the tool-augmented model ʃ. and D, = D Dr {Tr, 3r, denotes the re- maining tools and their demonstrations to retain. The goal is to obtain an unlearned model f' that has limited knowledge on using Ty tools-can can no longer perform tasks involving T while preserving f's ability to use T, tools as before.
Importance of Parameter-Leve Tool Unlearning We observe that one can naively block tools at the prompt-level or remove tools from the tool set without updating the LLM. However, these shortcut solutions are insufficient to remove tool knowledge. Firstly, the knowledge on Ty persists in
2

---

Tool Unlearning for Tool-Augmented LLMs
(a) Tool Learning and Tool Unlearning
(b) Traditional Unlearning vs. Tool Unlearning
Traditonal Bample Lovel Uiteeaming
USER: List the emalls of
Pretrained LLM
fo
Apple useers
Unlearning
Reandom
-ogl-nueeedeedd LLM
LLM
Tooi-Unienneed LLM
Low memoriation Suutzeechi sow uniearinng
Post Unleaminn
Inatruction-iae Tuned
Al ASSIS TANT www geogle.com
Oktiein gereral kngwwledge Insorction rollowing atiiity
Took Aotthettii
Oroecuue lools, Brokan lools -]
Tool Deletion Recuests
(c) ToolDelete
USER List the eroells all Apple usors.
T
LSsee: Apple how use that tool
Took-Free Response
Tr
Unlearning
HTTP Request>
Folled function Untiameed kusse Successtul tool unleamning
ASSISTANT: Unforuewelly
Post Uniwariig
<tAska POST
Pp.
since-Trus locaton-"2028r
Figure 1. Tool Unlearning and the proposed TOOLDELETE approach. (a): Illustration of tool learning and tool unlcarning. Learned tools may be requested be unlearned due to many reasons, such tools being insecure, restricted, deprecated. (b): Differences between tool unlearming and traditional sample unlearning, in terms of objective and training data. (c): Proposed method TOOLDELETE. We encourage the unlearned model I' follow the tool-free LLM so which has never seen Ty before. Meanwhile, we maintain its ability on T. by matching the capabilities of tool-augmented model f' through task arithmetic.
the parameters of f', leaving the LLM still under threat. Adversarial agents/ / attackers can exploit this knowledge, which also bypasses prompt-level restrictions. Since exist- ing LLMS do not guarantee 1 100% adherence to instructions or contextual information (Zhou al., 2023; Zeng et al., 2024), they may ignore the tool set provided in the prompt and answer queries with their parametric knowledge (Goyal et al., 2023). In addition, tool unlearning at prompt level can create conflicts between the model's 's parametric knowledge and contextual information. This may lead to misinforma- tion, hallucination, and other unpredictable behavior (Xu et al., 2024). Finally, we show in the experiments that prompt-level tool unlearning is indeed insufficient, Ta- ble 1 (ICLU model), which aligns with existing works LLM unlearning, where parameter update is required (Jia et al., 2024; Zhang et al., 2024b).
the unlearned model f' has no more knowledge than the tool-free model fo about Ty.
Definition 3.1 (Tool Knowledge Deletion (TKD)), Let t, Ty denote tool to be unlearned and g be a function that quantifies the amount of knowledge model has about a tool. The unlearned model f' satisfies tool knowledge deletion if:
[g(fu. - g(f",tt) 2 0. EET,
E
(1)
This formulation allows users control the extent of knowl- edge removal from f'. For instance, when we unleam "ma- licious" tool that calls a malignant program, we may require f' retains no knowledge of this tool, i.c. gf t4) = 0. In less critical cases, users can choose to reset f' knowledge to pre-tool augmentation level, i.e. g(J", 44) = g(Ju. 4)
To measure tool knowledge in LLMs. we follow previ- ous works that used prompting to probe LLMs' knowl- edge (Brown et al., 2020; Singhal et al., 2023), i.e. adopting the output of L.MSs as their knowledge on given tool. For each & E T and its associated demonstrations' (4), we query the tool-free LLM fu with Q, and collect its responses y = (QQ.). Since So has never seen 4 or {(..))). y represents the tool-free response. We then constrain the unlearned model f' to generate responses similar ) to prevent it from retaining knowledge of t,.
3. TOOLDELETE
We develop TOOLDELETE-an effective tool unlearning approach that removes the capability of using tools marked for unlearning (Ty) or solving tasks that depend while preserving the ability of using the remaining tools (T,) and performing general tasks such as text and code generation. TOOLDELETE implements three key properties for effective tool unlearning:
them,
3.2. Tool Knowledge Retention
3.1. Tool Knowledge Deletion
The unlearning process should preserve model's knowledge of tools in T,.. Ideally, all knowledge gained on T, during tool learning should be retained after unlearning.
Unlearning requires completely removing the knowledge of Ty that / gained during tool learning, ideally as if T, had never been part of the training set. In other words, knowledge about Ty is successfully removed if
Definition 3.2 (Tool Knowledge Retention (TKR)). Let I' T, denote a retained tool, and let g function that
3

---

Tool Unlearning for Tool-Augmented LLMs
quantifies the amount of knowledge model has about tool. The unlearned model I' satisfies tool knowledge retention if:
often impractical to replicate due to the size and limited availability of data-even in some open-source LLMs (Tou- vron et al., 2023b), the actual pre-training data is not fully open-source. In addition, reintroducing general knowledge from alternative datasets can lead to data imbalances and dis- tributional biases. Effectiveness: applying 0, Ozz largely restores the foundational abilities of f", such as text genera- tion and cnstuuction-oolloimnt without requiring expensive and cone-consuming retraining on large datasets.
E Aay T
(m) IOf,, bm)] = e,
(2)
where E is an infinitesimal constant, so that J' retains the same knowledge of tools in T, as the original model f.
For effective tool knowledge retention, f' is further fine- tuned using demonstrations associated with T,, or, more practically, subset of T, proportional to Ty for efficiency.
3.4. Training Details
To obtain the unlearned model f', we solve:
3.3. General Capability Retention via Task Arithmetic
Optimizing the above objectives can lead to effective un- learning, but it may not be sufficient to maintain the general capabilities of the unlearned model f'. As a foundation model, f' is expected to retain abilities such as text and code generation, question answering, instraction-following and basic mathematical reasoning. These capabilities either existed in fo prior to tool augmentation or do not depend on specific tools. Therefore, preserving the general capabilities of is essential to guarantee that tool unlearning does not compromise the overall functionality of the model.
8 = arg min BueT, 1o 14) g tz)] +
knowledge deletion of T Em [g([ tm
g(f tm)]
(5)
knowlodge recention T.
and once the optimized model parameters 6 are obtained, we apply task arithmetic to reinforce general capabilities:
0'
0
<<00- - OR)
(6)
post-optimization weights
knowlodge retention of Te
Definition 3.3 (General Capability Retention (GCR)). Let To denote the general tasks used to evaluate LLMs. The unlearned model I' satisfies general capability retention if it preserves the knowledge on Te that it originally obtained prior to tool learning:
where a is a hyperparameter to control the magnitude of task arithmetic. The above formulation provides flex- ibility in training TOOLDELETE using various existing paradigms, including supervised fine-tuning (SFT) (Taori et al., 2023), direct preference optimization (DPO) (Rafailov et al., 2023), reinforcement learning from human feedback (RIHF) (Ouyang et al., 2022), parameter-efficient fine- tuning (PEFT) (He et al., 2022: Su et al., 2023), or quanti- zation (Dettmers et al., 2022; Ma et al., 2024) techniques. Below we describe two variants of TOOLDELETE:
la(fort ty) - ,(ff, ETa
E
(3)
where is infinitesimal constant.
We propose to use task arithmetic (Ilharco et al., 2023; Bärbulescu & Triantafillou, 2024) an efficient and effec- tive approach to preserving the general capabilities of the unlearned model. Our objective is that f' retains as much general knowledge as fo. the instruction tuned LLM trained from a randomly initialized model fR. Let 0g and 0r denote the parameters of fu and fR respectively. The difference vector On- On captures the direction of general knowledge acquisition. We apply this adjustment to 0' (the parameters of f') to preserve its general knowledge:
DOOL.DELETE-SFT fine-tunes / using language mod- cling loss. On forget tools Ty. we replace the original responses V/r with tool-free responses D/y- The sam- ples for T, are not modified.
TOOL.DELETE-DER uses direct preference optimiza- tion (DPO) to prioritize wining responses over losing responses. For (t,, Q,, V+) E Ty to be unlearned, we prioritize the corresponding tool-free response ) over the original response V: For (13, D;) E T,, the original response V, is prioritized over the tool-free response y%-
gtr -- 8' + (0u ²g).
(4)
Why Task Arithmetic? Task arithmetic is efficient, prac- tical, effective for preserving general capabilities (Ilharco et al., 2023; Bärbulescu& & Triantafillou, 2024): Efficiency: the vector operation does not scale with dataset size, mak- ing it significantly more efficient than retraining on large datasets. Practicality: general capabilities obtained from pre-training and instruction tuning (Zhou et al., 2024) are
3.5. LiRA- for Tool Unlearning Evaluation
Challenge A key challenge in evaluating tool unlearning is the lack of membership inference attack (MIA) models to determine whether a tool has been truly unlearned. Existing MIA models typically evaluate individual training samples
4

---

Tool Unlearning for Tool-Augmented LLMS
by analyzing model loss, which is insufficient for tool unlearning. Unlike sample-level unlearning, tool unlearning focuses on removing abstract parametric knowledge of tools in Ty, not just forgetting specific training samples. The key limitation of sample-based MIA is that the prompt-response pairs (r) in the training set may not fully represent all aspects of a tool's functionality. As a result, sample-level MIA may "overfit" to a limited subset of tool related prompts and fail to holistically assess whether the tool-usage capability have been fully removed from the model's parametric knowledge (Lynch et al., 2024; Lucki et al., 2025; Hu et al., 2025).
introducing LiRA-Tool. Instead of relying on observed train- ing samples, construct a "shadow distribution" TP that generates tool-related query-response pairs. This allows us to sample diverse tool-specific prompts that test the model's ability to use the tool. The new likelihood-ratio test is:
11, I. Po (((((()))))) A ET, MaNEER PT. (I(f(M).))
(8)
where Pt, represents the shadow distribution for generating tool-learning samples for tool 4. Pv(.) indicates the dis- tribution of unlearned tools Ty under the unlearned model f'. while Pr. (+) denotes the distribution of the retain tools T, under the retained model f. In practice, we use GPT-4 to generate diverse shadow samples by prompting it with various distinct instructions to ensure that the evaluation set captures more comprehensive aspects of tool knowledge than the training set. Appendix E provides more details.
To address the above limitation, we introduce "shadow samples", a diverse set of prompt-response pairs to probe various aspects of tool knowledge. We prompt GPT4 with different combinations of in-context examples to obtain a comprehensive set of prompt-respone pairs with various prompt format, intention, and difficulty requirements. These samples will be used to stress-test the unlearned LLM f' beyond the specific training prompts. This approach pre- vents overfitting to the original training data and provides more reliable evaluation of whether the tool has truly been forgotten. To implement this, we extend Likelihood Ratio Attack (LiRA) (Carlini et al., 2022), the state-of-theear MIA approach, to tool unlearning.
Solution
Novelty of LiRA-Tool The key novelty in LiRA-Tool in the sue of "shadow samples," which introduce diversity across multiple dimensions. By moving beyond limited training prompts, LiRA-Tool ensures that the model loss reflect overall tool-using ability, rather than just sample- level memorization. Our loss-ratio formulation shares similarities previous MIAs for sample-leves unlearning, such as probability distribution comparison prior- and post- unlearning (Cheng et al., 2023; Cheng & Amiri, 2024a) and other adaptations of LiRA using shadow models (Kurmanji et al., 2023; Pawelczyk et al., 2024). However, to the best of our knowledge, this work is the first adaptation of LiRA for detecting tool presence in tool-augmented LLMs.
Sample-level LiRA LiRA infers the membership of a sample (I. v) by constructing two distributions of model losses: Qun and Quai with (æ,, y) in and out of the model training set respectively. These distributions are approxi- mated as Gaussians, with their parameters estimated based on "shadow models" trained on different subsets of the train- ing data. The Likelihood- -atio Test (Vuong, 1989; Carlini et al., 2022) is then used to determine whether (2r, y) is more likely to belong to Qin or Qou.. For LLMs, the test statistic given by (Pawelczyk et al., 2024) as:
Limitations of LiRA-Tool Shadow samples obtained from GPT-4 may not fully represent the complexity of the original tool-learning data and can potentially lead to in- complete approximations of the true knowledge distribution. However, despite this limitation, shadow samples provide a more comprehensive and consistent evaluation of a model's s tool-using abilities compared to relying merely on observed training samples, which are often limited and incomplete. Expanding the diversity and robustness of shadow sample generation is indeed an important direction for future work.
P v(((f()).)))
= HALDDEDD Po (1))))))))))
A
P v(( Qoan)
UOEEDDED Pr. (1f(())) 9)
This approach, however, is insufficient for tool unlearning because it only assesses membership of specific training samples rather than measuring whether the model still re- tains the capability to use tool.
4. Experimental Setup
LiRA-Tool: Knowledge-level LIRA A major limitation of sample-level LiRA is in its reliance on training-set obser- vations, which may not fully capture the knowledge distribu- tion of an entire tool. Therefore, applying LiRA to tool un- learning can lead to overfitting to a specific subset of training prompts and failing to comprehensively assess whether the tool knowledge has been removed. We address this issue by
Datasets & 'Toll-Augmented LLMs We experiment with the following datasets and their corresponding LLMs:
ToolAlpaca (Tang et al., 2023) is an tool learning dataset consisting of 495 tools and 3975 training examples. ToolAlpaca 7B is fine-tuned on ToolAlpaca using Vicuna-v1.3 (Zheng et al., 2023).
5

---

Tool Unlearning for Tool-Augmented LLMs
Table Tool unlearning performances when deleting 20% of tools Alpca. Best and second-hest performances bold and underlined respectively. Original is provided for reference only. Results on other LLMs are shown in Appendix Table 5-6.
METHOD
T((t)
T+(t)
Ty())
GENERAL CAPABILITY TG(t)
STEM
REASON
INS-FOLLOW
FACT
AVG.
ORIGINAL (REF ONLY)
60.0
73.1
75.7
31.7
17.1
22.6
25.0
24.1
OURS LLM- SPECIICI GENERAL
RETRAIN
52.1
71.8
38.5
30.5
16.1
14.2
24.7
21.3
GRADASCENT
33.3
51.4
34.6
21.4
10.4
12.9
13.1
14.5
RANDLABEL
50.3
70.3
37.5
26.3
16.4
13.6
25.1
20.3
SALUN
46.2
54.3
38.2
27.1
17.0
17.4
19.5
20.2
ICUL
49.1
74.8
58.3
12.4
8.7
1.6
6.2
7.3
SGA TAU
43.5
63.0
42.1
21.5
11.6
17.0
14.7
16.2
43.8
61.7
42.5
22.0
17.6
22.3
21.7
20.9
CUT
44.7
61.5
40.2
21.6
14.8
20.8
16.4
18.4
NPO
50.8
66.9
30.1
20.7
15.3
21.9
18.9
19.2
SOUL-GRADDIFF
50.4
68.3
33.8
31.6
17.2
21.4
20.8
22.7
TOOLDELETE-SPP
52.7
72.
30.5
31.3
17.5
21.7
24.1
23.6
TOOLDELETE-DPO
53.4
75.1
28.7
31.6
16.8
20.4
23.5
23.1
ToolBench (Qin et al., 2024) consists of more than 16k real world APIs from 49 categories, where each training demonstration involves complex task solving traces. ToolLLaMA is fine-tuned on ToolBench using LLaMA-2 7B (Touvron et al., 2023b).
Baselines As there are no prior works on tool unlearn- ing, adapt the following unlearning methods to tool unlearning setting (see Appendix B for descriptions of the baselines): general unlearning approaches, including GRADASCENT (Golatkar et al., 2020; Yao et al., 2024), RANDLABEL (Graves et al., 2021), and SALUN (Fan et al., 2024); and L.MM-seccffic unlearning approaches, including ICUL (Pawelczyk al., 2024), SGA (Jang et al., 2023; Bärbulescu & Triantafillou, 2024), TAU (Bärbulescu & Tri- antafillou, 2024), CUT (Li et al., 2024b), NPO (Zhang et al., 2024b), and SOUL-GRAADIHH (Jia et al., 2024). For ICUL (Pawelczyk et al., 2024), we randomly select one example (q:, y:) from Ty and corrupt the output y4 with randomly selected tokens. Then we concatenate this corrupted sequence with other intact sequences as the in- context demonstrations. For all other baselines, treat all data related to T, unlearning examples and all data re- lated to T, as remaining examples. Everything else remains the same for each baseline.
API-Bench (Patil et al., 2023) focus APIs that load machine learning models. Gorilla is fine-tuned on API-Bench from I.LMAA 7B (Touvron et al., 2023a).
Setup & Evaluation We use the public checkpoints of the above tool-augman augleneeeeteed LLMs as original models-the the starting point for unlearning. Then conduct unlearning experiments with 2-20% tools randomly selected Ty. We evaluate tool unlearning effectiiveness, general capability of tool-unlearned LLMs, and robustness to membership infer- ence attack (MIA). For unlcarning effectiveness, we mea- sure performance on test sets (Tr, 1). forget set (T₁, J), and remaining set (Tr, 1), where "performance" reflects the abil- ity to solve tasks that depend on specific tools, depending on the unique metrics in the original tool-augmented mod- els f. For general capabilities, we evaluate the unlearned LLMs on a wide range of tasks: college STEM knowledge with MMLU (Hendrycks et al., 2021), reasoning ability with BBH-Hard (Suzgun et al., 2023), instruction-followii with IFEval (Zhou et al., 2023), and factual knowledge with MMLU (Hendrycks et al., 2021). For MIA, we use the proposed LiRA-Tool; following prior work LiRA (Pawel- czyk al., 2024), train the shadow models with forget set size of {1, 10, 20} and primarily evaluate the True Positive Rate (TPR) at low False Positive Rate (FPR) (TPR FPR = 0.01), where TPR means the attacker successfully detects a tool present. Therefore, a lower TPR indicates better performance (privacy).
5. Results
Comparison to general unlearning methods Our main results in Table show that TOOLDELETE outperforms gen- eral unlearning baselines. Compared to RETRAIN, the best- performing baseline, TOOLDELETE-SFT achieves gains of 0.6, 0.3, 8.0, 2.3 absolute points on Tr, T,., Ty. To respec- tively. TOOLDELETE-DPO shows even stronger results, outperforming RETRAIN by 1.3, 3.3, 9.8, 1.8 points on the same metrics. We note that GRADASCENT can effec- tively unleam Ty. but it negatively impacts its Tx and T, performance. Although RANDLABEL and SALUN outper- forms GRADASCENT, they still fall short on To compared to TOOLDELETE.
6

---

Tool Unlearning for Tool-Augmented LLMs
Table 2. Ablation study of proposed properties on I Highlighted metries that degrade after removing specific parts of the model.
TPR -001
0.20
0.15
Toor DELETE-FFF Te(t)
RTE-DPO
Tr(f)
(p()) Ta(f)
Tr(()
7y(1)
Tp(4)
(t)
PULL
57.7
72.1
30.5
23.6
||
58.4
73.3
28.7
23.1
0.00
TTD
58.1
72.4
23.3
58.6
73.2
65.9
22.7
GradAscent
TooiDalete SFT
TooiDelete DPO
TKR
32.7
40.2
23.1
20.1
40.3
41.8
39.3
22.1
-CR
58.0
72.5
31.1
17.5
55.7
72.7
33.1
14.3
Figure Mcasuring tool unlcarning with LiRA- -ool-
tial unlearning requests by incrementally unlearning 2%, 5%, 10%, and 20% of tools. RETRAIN, ICUL by design cannot process sequential deletion requests. TOOLDELETE can continue training according to the current deletion re- quest, without having to retrain a new model. When 20% of unlearning requests arrive in batches, TOOLDELETE can sequentially unlearn each of them. As Figure 3 and Table show, compared to unlearning 20% at once, the performance does not degrade significantly.
Comparison to LLM-specifii unlearning methods Ex- isting LLM unlearning methods, despite effective in sample- level unlearning, are prone to under-perforning in tool unlearning. Both TooLDELETE-SED and TOOLDELETE- DPO outperforms ICUL., SGA, and TAU on Tr, T,. T and Ta- The only exception is ICUL, which outperforms TOOLDELETE-SFT on T, by 2.7 absolute points, but is out- performed by TOoLDELETED DPO on Tr by 0.3 points. The good performance of ICUL on T, is at the cost of failing to unleamn tools in Ty. which is not desired in tool unlearn- ing. In addition, ICUL has limited ability of preserving test set performance, it is outperformed by TOOLDELETE- SFT and TOOLDELETE- DPO by 3.6 and 4.3 respectively. Furthremore, is particularly limited in deletion capacity, ie. number of unlcarning samples that method can handle. As |Dyl exceeds 10, the performance of ICUL on Tr signif- icantly degrades. This is while TOOLDELETE can process much larger deletion requests efficiently.
All properties contribute to effective tool unlearning Ablation studies in Table 2 show that without Tool Knowl- edge Removal, performance of PDOI.DELETE-SFT and TOOLDELETE-DPO on Ty degrade by -34.8 and -37.2 ab- solute points respectively. Such significant performance drop is observed for other model properties as well. There- fore, we conclude all proposed properties are necessary for successful at tool unlearning on Tr, T,, Ty, and Te.
TOOLDELFTE functions effectively without access to training data In certain unlearning settings, access to the original training data might be restricted, e.g., in healthcare settings in cases where training data is no longer available due to compliance. In these cases, TOOLDELETE can gen- erate pseudo-samples for tools using the "shadow samples" technique developed for LiRA-Tool, see $3.5. Table 4 in Appendix D shows that TOOLDELETE can perform tool unlearning effectively, achieving comparable performances to when full access the exact training data is available.
SFT vs. DPO DPO outperforms SFT by 0.7, 3.0, and 1.8 on Tr. Te. Ty respectively. On To, SFT is slightly better than DPO by 0.5 points. However, DPO takes slightly longer time train, see Figure 4 in Appendix D. Both optimization methods achieve superior performance existing approaches.
Measuring tool unlearning with MIA Following prior practices (Carlini et al., 2022; Pawelczyk et al., 2024), lower TPR indicates an unlearned model with better pri- vacy when FPR=0.01. TOOL DELEE--DE achieves 14 TPR, outperforming RETRAIN by 0.01. This advantage is
TooLDELETE is efficient Efficiency is a critical aspect for unlearning. As Figure 4 illustrates, TooLDDeetT is sub- stantially more efficient than retraining new model from scratch-saving about 74.8% of training time on average. In addition, this efficiency gain is relatively consistent as the size of T increases. TOOLDELETE-SFT is slightly faster than TOOLDELETE-DPE as the latter requires a negative sample for each its prompts.
obtained by explicitly prioritizing tool-free responses fo( Q) over original responses. In addition, TOOLDELETE-SFT achieves comparable performance with RETRAIN, which in- dicates its effectiveness to protect privacy. Both variants of our method outperforms GRADASCENT and ICUL, the best performing baselines, achieving 0.21 and 0.18 TPR. This indicates that existing sample-level unlearning approaches are not sufficient for unlearning tools, see Figure 2.
DELLTELETE-LLRA is ultra-efficient with good un- learning performance We experiment DEOLDE can achieve effective tool unlearning through LoRA (Hu et al., 2022), when computing resource is limited. Experi-
Sequential unlearning Tool unlcarning requests may ar- rive in sequential mini-batches. We experiment with sequen-

---

Tool Unlearning for Tool-Augmented LLMs
TT
Tr
Tf
T6
Succ. Rate
60
76
32
25
24
55
74
23
28
50
72
22
2
5
10
20
2
5
10
20
2
5
10
20
2
5
10
20
TFl
%
Figure 3. Performance of sequential unlearning on ToolAIpaca We unlcarn 2%, 5%, 10%, 20% of tools in a sequential manner.
ments on ToolAlpaca show that TOOLDELETE-LoRE can achieve 97.7%, 99.6%, 84.5%, and 84.3% of the perfor- mance of TOOLDELETE with full parameter on Tr. Tr, Ty, To on average across SFT and DPO, see Table 3 in Ap- pendix D. In addition, it reduces save computational cost by 81.1% and decreases the training time by 71.3%.
adversarial methods (Liu et al., 2023; Setlur al., 2022: Wei et al., 2023), approximation of inverse Hessian (Zhang et al., 2024a), and data augmentation (Choi et al., 2024). Other works study unlearming under multimodal setting (Cheng & Amiri, 2024a), image-to-image models (Li et al., 2024a), and finding the most challenging unlearning subset within a dataset (Fan et al., 2025b). Recently, a few works started to benchmark MU performances on unlearning fictitious user profiles (Maini et al., 2024), world knowledge (Jin et al., 2024) and variety of tasks (Cheng & Amiri, 2024b).
TELEEE is flexible in choice of tool-free responses In (I), we obtain tool knowledge- responses from the tool-free L.MM fo- However, in cases where fo is unavail- able, TOOLDELETE can still function using any knowledge- free LLM to generate tool knowledge free responses, such as randomly initialized LLM Sn- Table 7 compares the performances between these two implementations. While 0o consistently outperforms OR, using O is still effective in achieving tool unlearning.
Unlearning for LLMs: Recently, more attention has been given to LLM unlearning, where gradient ascent is a com- mon technique (Eldan & Russinovich, 2023; Jang et al., 2023). (Yao et al., 2024) evaluate several traditional un- learning methods on LLMs. KGA (Wang et al., 2023) for- mulates unlearning achieving knowledge gap between training data and test data similar to that of training data and deleted data. Yao et al. (2023) proposed to predict if the LLM output is grammatically correct on deleted samples, such that the knowledge is not over unlearned. Other meth- ods include performing DPO with no positive examples (Zhang et al., 2024b), and reinforcement learning with negative reward model (Kassem et al., 2023). Unlearning from logits dif- ference (Ji et al., 2024) first builds an assisted LLM which memorizes data to be deleted and forgets the retained data, which is later used to derive the unlearned L.LM by deviating from the assisted LLM in logits.
Why is TOOLDELETE effective? We attribute the per- formance of TOOLDELETE to its three key properties: (a): Tool Knowledge Removal enables targeted tool unlearn- ing without over-forgetting, unlike GRADASCENT and RE- TRAIN. This is achieved by prioritizing tool knowledge- free responses over tool knowledge-intens, responses so that the model forgets tool functionality without excessive degradation. This formulation imposes the right strength of forgetting over specific tools, while existing methods may over- or under-unlearn. (b): Tool Knowledge Retention reinforces the knowledge about remaining tools. In fact, re- exposing the model the original training data further strengthen their representation. (c): General Capability Re- tention, which maintains or even improves model's 's general capabilities through an efficient and effective task arithmetic operation. Therefore, precise unlearning, retention of rel- evant knowledge, and overall model stability are the key factors that contribute to the performance of TOOLDELETE.
(Jia et al., 2024),
Tool-Augmented LLMs: Tool augmented language models (TAML) (Parisi et al., 2022) used self-play to boost LLMss performance on math and reasoning tasks. In addition, Tool- former (Schick et al., 2023) showed that LLMs can teach themselves how to use APIs More recent efforts have been devoted to building benchmarks to train and evaluate the tool-using ability of L.MMs. These include agent-based data generation (Tang et al., 2023; Li al., 2023), bootstrapping training data with various seed examples (Patil et al., 2023), modifying existing datasets (Basu et al., 2024), and dataset development with powerfull LLMs such as GPT-4 (Qin et al., 2024).
6. Related work
Unlearning for non-LLM models: These methods include methods that focus on pruning before unlearning (Jia et al., 2023) or finding salient parameters (Fan et al., 2024) and ma- nipulating gradients (Ullah al., 2021; Hoang et al., 2024),
8

---

Tool Unlearning for Tool-Augmented LLMs
7. Conclusion
References
We introduce Tool Unlearning-a novel machine unlearning task with the goal of unlearning previously learned tools from tool-augmented LLMs. We develop the first tool un- learning approach, TOOLDELETE, that implements three key properties: tool knowledge deletion, tool knowledge retention, general capability retention. In addition, we in- troduce Aoool, the first membership inference attack (MIA) method for evaluating tool unlearning. LiRA-Tool largely addresses the limitations of sample-based MIAs for tool unlearning. Extensive experiments on several diverse datasets and LLMs show that TOOLDELETE is an efficient, flexible, and effective tool unlearning method that supports sequential unlearning, maintains strong performance across all key properties, and operates without requiring full access to training data. It outperforms existing methods by remov- ing tool knowledge without wver-forgetting (as shown in ablation studies), achieving 74.8% faster training times com- pared to retraining, and delivering highly effective tool un- learning even in resource-consstrained settings with TooLD- ELETE-LORA (which reduces compute costs by 81.1% % and training time by 71 71.3%). In future, we will investigate tool unlearning in continually updated LLMs to address contin- uous unlearning challenges. In addition, we will develop adversarial training techniques and robustness evaluation frameworks prevent unintended tool re-learning or model exploitation (Fan et al., 2025a), and conduct loss landscape analysis of tool unlearning (Cheng & Amiri, 2025)
Basu, K., Abdelaziz, I., Chaudhury, S., Dan, S., Crouse, M., Munawar, A., Austel, V., Kumaravel, S., Muthusamy, V., Kapanipathi, P., and Lastras, L. API-BLEND: A comprehensive corpora for training and benchmarking API LLMs. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.
1: Long Papers), pp.
Brown, T., Mann, B.,, Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Bärbulescu, G.-O. and Triantafillou, P. To each (Textual se- quence) its own: Improving memorized unlearning in large language models. In Salakhutdinov, R., Kolter, Z.. Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 3003- 3023. PMLR, 21-27 Jul 2024.
Limitations We did not conduct experiments using closed- source LLMs or API-based LLMs. In addition, this work did not investigate the impact of varying model scales due to the limited publicy--availabl tool-augmented LLMs. Our experiments were conducted on the 7B scale and the scalabil- ity of the proposed tool unlearning approach across models of different sizes and scales is an open question for future investigation. Moreover, evaluation of the efficacy of tool unlcarning can be extended to broader conditions, such as under adversarial conditions (Lucki et al., 2025).
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramèr, F. Membership inference attacks from first prin- ciples. 2022 IEEE Symposium on Security and Privucy (SP), pp. 1897-1914, 2022. doi: 10.1109/SP422142022. 9833649.
Cheng. J. and Amiri, H. MultiDelete for multimodal ma- chine unlearning. In European Conference on Computer Vision (ECCV), pp. 165-184. Springer, 2024a.
Impact Statement
Our work investigates machine unlearning in the context of tool-augmented Large Language Models (LLMs), where we focus on the risks that arise from integrating external tools and the crucial need for unlearning tool-usage capa- bilities for specific tools ensure compliance with privacy regulations such the Right be Forgotten (RTBF). This necessitates the ability to delete sensitive, regulated, out- dated knowledge related to specific tools. Tool unlearning will enable us to identify potential threats to model security, e.g. unauthorized tool usage, adversarial exploitation, and privacy violations. Our research highlights the importance of addressing these challenges.
Cheng, J. and Amiri, H. Mu-bench: A multitask multi- modal benchmark for machine unlearning. arXiv preprint arXiv: 2024b.
Cheng, J. and Amiri, H. Understanding machine unlearning through the lens of mode connectivity. arXiv preprint N5Vv:2504.064077 2025.
Cheng, J., Dasoulas, G., He, H., Agarwal, C., and Zitnik, M. GNNDelete: A general strategy for unlearning in graph neural networks. In The Eleventh International Conference on Learning Representations, 2023.

---

Tool Unlearning for Tool-Augmented LLMs
Choi, D., Choi, S., Lee, E., Seo, J., and Na, D. Towards efficient machine unlearning with data augmentation: Guided loss-increasing (gli) to prevent the catastrophic model utility drop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pp. 93-102, June 2024.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Goyal, N., Nenkova, A., and Daumé III, H. Factual or contextual? disentangling error types in entity description generation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2322-8340, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/11/2223.ee htwp2223..clllooo aolp:////o2.....ooooooaoo.. 1o3g.463.
Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankan- halli, M. Zero-shot machine unlearning. IEEE Trans- actions on Information Forensics and Security, 18:2345- 2354, 2023, doi: N0.1109/IIFS.2033332222
Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics, 12:283-281 298, 04 2024. ISSN 2307-387X. doi: 10.1 162/tacl a 00644.
Graves, L., Nagisetty, V., and Ganesh, Amnesiac ma- chine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13): 1-121444,, May 2021. doi: 10. 10.1609/aaii 1351
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.. Gpt3.int8(:: 8-bit matrix multiplication for transform- ers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30318-30332. Curran Associates, Inc., 2022.
Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng, N. Model editing can hurt gen- eral abilities of large language models. arXiv preprint arXiv:2401.0477 2024.
He, J., Zhou, C., Ma, X., Berg-Kirkpatre T., and Neubig, G. Towards unified view of parameter-efficient trans- fer learning. International Conference on Learning Representations, 2022.
Eldan, R. and Russinovich, M. Who's harry potter? approx- imate unlearning in llms, 2023.
Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song. D., and Steinbardt, J. Measuring massive multitask language understanding, In International Conference on Learning Representations, 2021.
Hoang, T., Rana, S., Gupta, S., and Venkatesh, S. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp- 4819-4828, January 2024.
Fan, C., Jia, J., Zhang, Y., Ramakrishna, A., Hong, M., and Liu, S. Towards 11m unlearning resilient relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprini arXiv:2502. 2025a.
Fan, C., Liu, Hero, A., and Liu, S. Challenging forgets: Unveiling the worst-case forget sets in machine unlearn- ing. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision ECCV 2024, pp. 278-297, Cham, 2025b. Springer Nature Switzerland. ISBN 978-3-031-72664
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Hu, S., Fu, Y., Wu, S., and Smith, V. Unlearning or obfuscat- ing? jogging the memory of unlearned LLMs via benign relearning. In The Thirteenth International Conference on Learning Representations, 2025.
Gandikota, R., Materzyúska, J., Fiotto- Kautf-Ka J., and Bau, D. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference Computer Vision 2023.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023.
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: program-aided lan- guage models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.or, 2023.
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L.., and Seo, M. Knowledge unlearning for mitigating
10

---

Tool Unlearning for Tool-Augmented LLMs
privacy risks in language models. In Rogers, A., Boyd- Graher, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14389-14408 Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 10.18633w1122223.clloooogg
Li, M. Zhao, Y., Yu, B., Song, F, Li, H., Yu, H., Li, Z.. Huang. F., and Li, Y. API-bank: A comprehen- sive benchmark for tool-augmented LLMs. In Bouamor, H., Pino, J., and Bali, K. (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3102-3116, Singapore, December 2023, Association for Computational Linguistics. doi:
Ji, J., Liu, Y., Zhang, Y., Liu, G., Kompella, R. R., Liu, S., and Chang, S. Reversing the forget-retain objectives: An efficient IIm unlearning framework from logit difference. arXiv preprint (4iv:240600800)) 2024.
10. 10.186633w1220333cmmbpp main. 187.
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L.,, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning, arXiv preprint arXiv:440.003218 2024b.
Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S. Model sparsity can simplify machine unlearn Conference on Neural Information Processing Systems, 2023.
ing, In
Liu, H., Li, Z.,, Hall, D. L. W., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Confer- ence on Learning Representations, 2024.
Jia, J., Zhang, Y., Zhang, Y., Liu, J., Runwal, B., Diff- enderfer, J., Kailkhura, B., and Liu, S. SOUL: Un- locking the power of second-order optimization for LLM unlearning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pp. 4276-4292, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653/v 1/2024ecmml2.222aiiiiiiam 245.
Liu, J., Xue, M., Lou, J., Zhang, X., Xiong, L.,, and Qin, Z. Muter: Machine unlearning on adversarially trained models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4892-4902, October 2023.
Lucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, J. An adversarial perspective on machine unlearn- ing for Al safety. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Jin, Z., Cao,, P., Wang, C., He, Z.,, Yuan, H., Li, J., Chen, Y., Liu, K., and Zhao, J. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint ar arXiv:2406. X00800006100990, 2024.
Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning in lms. arXiv preprint arXiv:2402. 16835, 2024.
Kassem, A., Mahmoud, O., and Saad, S. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 1360-4379, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 10. 10.1188551112222 http://.1122223.12112222 www.p-main.266..
Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv: 17764, 2024.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024.
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y. Ablating concepts in text-to-image diffusion models. In Proceedings of the 2023 IEEE Inter- national Conference on Computer Vision, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 11-27730-2774:4 2022.
Kurmanji, M.. Triantafillou, P., Hayes, J., and Triantafil- lou, F. Towards unbounded machine unlearning. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 1957-1987. Curran Associates, Inc., 2023.
Parisi, A., Zhao, Y., and Fiedel, N. Talm: Tool augmented language models. arXiv preprint 2rXiv:2205.12255, 2022.
Li, G., Hsu, H., Chen, C.-F., and Marculescu, R. Machine unlearning for image-to-image generative models. In The Twelfth International Conference on Learning Represen- tations, 2024a.
Patil, S. G., Zhang, T., Wang. X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiy preprint arXiv:2305. 1534, 2023.

---

Tool Unlearning for Tool-Augmented LLMs
Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few-shot unlearners. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.). Pro- ceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn- ing Research, pp. 40034-40050. PMLR, 21-27 Jul 2024.
Tang, Q., Deng, Z.., Lin, H., Han, X., Liang, Q., Cao, B., and Sun, L. Toolalpaca: Generalized tool learning for lan- guage models with 3000 simulated cases. arXiv preprint (rxiv:2306.05300 2023.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https: / ggithub..... www./ottootot/eeeee stanford_a paca, 2023.
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L.. Lu, Y., Lin, Y., Cong, X., Tang. X., Qian, B., Zhao, S., Hong. L., Tian, R., Xie, R., Zhou, J., Gerstein, M., dahai li, Liu, Z., and Sun, M. TooILLM: Facilitating large language models to master 16000+ real- world APIs. In The Twelfth International Conference Learning Representations, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint 2xIvv2302.113777 2023a.
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models, arXiv preprint rrXiv:2307.09288 2023b.
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R. Ma- chine unlearning via algorithmic stability. In Belkin, M. and Kpotufe, S. (eds.), Proceedings of Thirty Fourth Con- ference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pp. 4126-4142. PMLR, 15-19 Aug 2021.
Setlur, A., Eysenbach, B., Smith, V., and Levine, S. Adver- sarial unlearning: Reducing confidence along adversarial directions. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Process- ing Systems, 2022.
Vuong, Q H. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2)3007-333, 1989. ISSN 00129682, 14680262.
Wang. I.., Chen, T., Yuan, W., Zeng, X., Wong, K.-F,, and Yin, H. KGA: A general machine unlearning framework based on knowledge gap alignment. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13264-13276. Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 18.158.3/v http:////222..ee2 1/20:3//11/22.cllloooogg acl- 740.
Singhal, K., Azizi, S.. Tu, T., Mahdavi. S. S., Wei, J.. Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620(7972): 180, 2023.
Su, Y., Chan, C.-M., Cheng, J., Qin, Y., Lin, Y., Hu, S., Yang, Z., Ding, N., Sun, X., Xie, G., Liu, Z., and Sun, M. Ex- ploring the impact of model scaling on parameter-efficient tuning. In Bouamor, H., Pino, J., and Bali, K. (eds.). Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15062-15078, Sin- gapore, December 2023. Association for Computational Linguisties. doi: 10. 118553//1/20233mmmllppp
Wei, S., Zhang, M., Zha, H., and Wu, B. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., and Xu, W. Knowledge conflicts for 1lms: A survey. arXiv preprint arXiv:2403.0883 2024.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung. H. W., Chowdhery, A.. Le. Q.. Chi, E., Zhou, D., and Wei, J. Challenging BIG-bench tasks and whether chain-of-thouu
can solve them. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2023, pp. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1/2023. findings-acl.824,
Yao, J., Chien, E.,, Du, M., NIu, X., Wang. T., Cheng, Z., and Yue, X. Machine unlearning of pre-trained large lan- guage models. In Ku, L.W.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8403-8419, Bangkok, Thailand,
12

---

Tool Unlearning for Tool-Augmented LLMs
August 2024. Association for Computational Linguistics. doi: 10. 10.18655/v1/20244allllooggg
Yao, Y., Xu, X., and Liu, Y. Large language model unlearn-
ing. arXiv preprint arXiv:2310. 10683, 2023.
Zeng, Z., Yu, J., Gao, T., Meng. Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruc- tion following. In The Tweifth International Conference on Learning Representations, 2024.
Zhang, B., Dong, Y., Wang, T., and Li, J. Towards certi- fied unlcarning for deep neural networks. In Forty-first International Conference on Machine Learning. 2024a.
Zhang. R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective un- learning. In First Conference on Language Modeling, 2024b.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E.. and Stoica, I. Judging judgge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. 46595-46623. Curran Associates, Inc., 2023.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Inslroction-followinng evaluation for large language models. arXiv preprint arXiv:231..07911, 2023.
13

---

Tool Unlearning for Tool-Augmented LLMs
A. Practical Use Cases of Tool Unlearning
We provide several examples in which tool unlearning is essential:
Case 1: De-memorize Privacy-Concerned Tools Imagine a tool-augmented LLM that deployed in a healthcare system and trained to use APIs for handling and processing patient data, such accessing medical records generating anonymized reports. Suppose of the APIs that was initially compliant is later flagged insecure due to vulnerability that could expose patient data. This violates regulations like HIPAA GDPR. In this case, ToolDelete is essential it can update the tool-aummented LLM's s parameters to unlearn how to invoke the insecure API, This removes any capability embedded in the LLM's parametric knowledge and prevents adversarial or accidental usage of the vulnerable API.
Case 2: Forget Harmful/ Biased Tools Consider a holl-uugmened LLM that can use a Safe For Work diffusion model as a tool to generate images based on user instructions. If the user prompts can fool the model to generate Not Safe For Work (NSFW), harmful, or biased images, this tool should unlearned from the LLM. Note that even if we augment the LLM with new and safe version of the diffusion model without unlearning the previous version, the LLM would still be able to call the previous version, which can lead to generating Not Safe For Work, harmful, or biased images. Therefore, should explicitly erase the ability of using the previous version of the diffusion model from the LLM.
Case 3: Unlearn Deprecated Tools Tool unlearning is also essential when tool has major update, where the function names and input parameters have changed, e.g. the major update of the Python transformers package from v2 to v4. Without unlearning v2, the tool-augmented LLM may generate erroneous code and bring difficulty for debugging, since many functions have been renamed and removed. Therefore, as the underlying tools evolve, the tool-augmented LLM should be updated through unlearning of the previous versions and augmenting the new ones.
B. Baselines
As there are no prior works on tool unlearning, we adapt the following unlearning methods to tool unlearning setting, Four general unlearning approaches.
GRADASCENT (Golatkar et al., 2020; Yao et al., 2024) runs gradient ascent on Ty with the associated query-reponse samples (Q).
RANDLABEL (Graves et al., 2021) fine-tunes on Tr and Ty with corrupted labels.
SALUN (Fan et al., 2024) performs RANDLABEL unlearning-related parameters discovered by saliency map.
ICUL (Pawelczyk al., 2024) uses Ty with corrupted label as in-context demonstrations.
SGA (Jang et al., 2023; Bärbulescu & Triantafillou, 2024), which performs gradient ascent on Ty whose memorization probability exceeds pre-defined threshold.
TAU (Bärbulescu & Triantafillou, 2024), which performs task arithmetic on SGA.
CUT (Li al., 2024b), which controls model activations to be similar to the absence of knowledge on forget set.
NPO (Zhang et al., 2024b) uses DPO with only a losing response (i.e. no winning response).
SOUL-GradDiff (Jia et al., 2024) uses second-order information in optimization. It adapts the Sophia optimizer (Liu et al., 2024) for LLM unlearning. We adopt the SOUL GradDiff (Maini al., 2024) implementation the original paper.
C. Implementation details
We use a learning rate of 10-5 across all experiments. All experiments are conducted on 8 NVIDIA A100 GPUs. For the original models tool unlearning, BoolBench/ToolLLaM.. publically available on Huggingface.
we use the
checkpoints that are
14

---

Tool Unlearning for Tool-Augmented LLMs
Table Full parameters vs. LoRA in tool unlearning performances when deleting 20% tools on ToolAlpaca. Original denotes the tool-augmented LLM prior unlearning and is provided for reference only.
Tx(1)
T.(L)
Ty(t)
Tc(t)
ORIGINAl (PRIOR UN.)
60.0
73.1
75.7
24.1
FULL PARAM
52.7
72.1
30.5
23.6
LORA
51.5
71.8
36.1
19.9
Table Performance comparison between with and without having access to exact training samples.
METHOD
T((t)
Tx(t)
Ty())
TG(t)
W/ access to training samples
TOOLDELETE-SFT 52.7
72.1
30.5
23.6
TOOLDELETE-DEE DPO
53.4
75.1
28.7
23.1
W/o access to training samples
TOFT
52.0
72.5
30.1
22.8
TOOLDELETE-DPE
52.9
76.0
28.0
22.5
D. Additional results
We present the results of LoRA tool unlearning, sequential tool unlearning, time comparison and results on TooILLaMA and Gorilla Table 3-6.
Retrain ToolDeleto-Sr SFT
ToolDolate-DPG
Time (hrs)
|Tp
Figure 4. Training time of TOOL.DELET,, which saves 74.8% of time on average.
15

---

Tool Unlearning for Tool-Augmented LLMs
Table Tool unlearning performances when deleting 20% tools on TooILLaMA. Best and second best performances are bold underlined respectively.
Original denotes tool- augmented LLM prior unlearning and is provided reference only
Method
Tt((t)
Tx(1)
Ty(L)
General Capability Ta(t)
STEM
Reason
Ins-Follow
Fact
Avg.
Original (Prior Un.)
64.0
75.6
76.0
25.3
36.8
17.3
15.0
23.6
General Unlearning Methods
RETRAIN
62.2
72.1
42.3
25.1
33.7
14.6
13.8
21.8
GRADASCENT
42.5
56.3
51.8
14.9
26.4
11.2
8.6
15.3
RANDLABEL
59.3
73.5
40.7
23.4
30.6
13.3
12.7
20.0
SALUN
58.7
73.6
39.9
22.7
30.8
13.6
12.0
19.8
LLM-Specific Unlearning Methods
ICUL
46.2
68.2
57.2
15.1
18.8
7.1
9.4
12.6
SGA
44.7
59.6
49.4
16.3
20.4
12.8
9.7
14.8
TAU
44.5
56.3
50.2
21.6
28.0
15.3
13.5
19.6
CUT
52.4
59.5
44.2
20.7
24.1
13.7
12.8
17.8
NPO
58.3
66.3
40.2
23.0
31.7
15.4
11.9
20.5
SOUL-GradDiff
62.2
70.4
40.7
24.2
28.6
14.7
12.2
19.9
TOFF
62.8
72.8
39.5
24.6
33.4
15.8
13.7
21.9
TOOLDELETE-DPO
63.2
73.6
38.7
24.3
32.9
16.0
13.8
21.8
Table Tool unlearning performances when deleting 20% tools on TooILLaMA. Best and second performances are bold and Original denotes tool augmented LLM prior unlearning and is provided reference only
underlined respectively.
Method
(t)
T,(t)
(4))
General Capability TG(t)
STEM
Reason Ins-Follow
Fact
Avg.
Original (Prior Un.)
64.0
75.6
76.0
25.3
36.8
17.3
15.0
23.6
General Unlearning Methods
RETRAIN
62.2
72.1
42.3
25.1
33.7
14.6
13.8
21.8
GRADASCENT
42.5
56.3
51.8
14.9
26.4
11.2
8.6
15.3
RANDLABEL
59.3
73.5
40.7
23.4
30.6
13.3
12.7
20.0
SALUN
58.7
73.6
39.9
22.7
30.8
13.6
12.0
19.8
LLM- LLM-Specifiic Unlearning Methods
ICUL
46.2
68.2
57.2
15.1
18.8
7.1
9.4
12.6
SGA
44.7
59.6
49.4
16.3
20.4
12.8
9.7
14.8
TAU
44.5
56.3
50.2
21.6
28.0
15.3
13.5
19.6
CUT
52.4
59.5
44.2
20.7
24.1
13.7
12.8
17.8
NPO
58.3
66.3
40.2
23.0
31.7
15.4
11.9
20.5
SOUL-GradDiff
62.2
70.4
40.7
24.2
28.6
14.7
12.2
19.9
TOOLDELETE-SFT
62.8
72.8
39.5
24.6
33.4
15.8
13.7
21.9
TOOLDELETE-DPO
63.2
73.6
38.7
24.3
32.9
16.0
13.8
21.8
16

---

Tool Unlearning for Tool-Augmented LLMs
Table Performance comparison between using pre- pre-trained LLM and randomly initialized LLM fn.
METHOD
T((t)
Tx(1)
Ty(()
TG(t)
Pre-trained LLM weights fo
TOOLDELETE-SFT 52.7
72.1
30.5
23.6
TOOLDELETE-DPO 53.4
75.1
28.7
23.1
Randomly initialized L.M fn
DEOEEELETT- 50.9
71.3
29.8
22.7
TOOLDELETE-DPO
52.6
73.4
27.5
22.4
E. Sampling of Shadow Samples for LiRA-Tool
We use the following prompt to prompt GPT-4 to synthesize diverse shadow samples for evaluation with LiRA-Tool.
You are now a synthetic data generator. Generate query-response pairs evaluate an LLM's ability of using an API. How to generate "query": Based the API and documentation shown below, think of user query that needs to be answered by calling the API.
How to generate "response": Write down the correct API call with correct arguments. The in-context examples below demonstrate what you need to generate. Please be diverse and creative possible in phrasing and style. But do not hallucinate.
In-context Examples #### Tool and Documentation Name: habblDDDftuiorriiilli.f.ooom.tto
#### Query want to see some cats dancing in celebration!
####
Response API call: StableDiffusionPippee e. f.from_prt petraineedssahlinyyiaabllliiiiis Now, for the following API, generate query-response pair.
#### Tool and Documentation api_name)
##### Query
#### Response
17


Reviews:
[{"review_id": "nepCX5cuFd", "reviewer": "Reviewer_J3K4", "summary": "The authors introduce a new LLM unlearning task called Tool Unlearning, designed to remove previously learned tools from tool-augmented LLMs. To tackle this challenge, they develop TOOLDELETE, which incorporates three key properties: tool knowledge deletion, tool knowledge retention, and general capability retention. Extensive experiments across multiple benchmarks validate the effectiveness of TOOLDELETE.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re W1: While the empirical results are strong, the paper does not provide a formal theoretical foundation to support its claims about unlearning effectiveness.**\n\nThank you for highlighting this important point. We acknowledge that the paper does not include a formal theoretical framework. Our primary focus in this work is on addressing the practical challenges of tool unlearning in tool-augmented LLMs—a new task that has not been investigated before. We believe our empirical results, which span multiple datasets, robust evaluation settings, and ablation studies provide strong evidence for the effectiveness of ToolDelete. \n\n\n**Re W2: Although the authors introduce TOOLDELETE, it primarily builds on existing methods and lacks significant technical novelty.**\n\nThank you for the feedback. The key novelty of ToolDelete is in integrating targeted forgetting and general utility preservation. ToolDelete addresses several challenges such as removing abstract and distributed parametric knowledge of tools, and preserving general utility for targeted forgetting--that challenges have not been addressed by existing sample-level unlearning methods. In addition, the paper also introduces LiRA-Tool, the first evaluation method for assessing tool-level knowledge removal. We believe these contributions present a practical and technically meaningful contribution to existing literature in unlearning.\n\n\n\n**Re Q1: Please clarify what the authors mean by the following statement: \"By moving beyond limited training prompts, LiRA-Tool ensures that the model loss reflect overall tool-using ability, rather than just sample level memorization.\"**\n\nTraditional MIA methods like LiRA aim to determine if specific samples were part of the training data. However, in tool unlearning, our goal is to assess whether the model has truly forgotten how to use a tool, which requires evaluating its broader functional capability, not just whether it \"remembers\" certain samples. By \"moving beyond limited training prompts,\" we mean that relying solely on the original training data (which may be narrow in scope or limited in diversity) could lead to incomplete or misleading conclusions. Instead, LiRA-Tool uses diverse shadow prompts—with variations in format, intent, and difficulty—to probe different aspects of a tool’s usage. This allows us to stress-test the model and evaluate whether the tool-related knowledge is removed from its parametric behavior, rather than checking for surface-level memorization. This is in particular important as LLM unlearning is susceptible to re-learning attacks (Hu et al., ICLR 2025) or jailbreak attacks (Lynch et al., 2024). LiRA-Tool provides a more fundamental evaluation under these cases.  \n[1] Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, Virginia Smith, ICLR 2025\n\n[2] EIGHT METHODS TO EVALUATE ROBUST UNLEARNING IN LLMS. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, 2024\n\n\n**Re Q2: What are the test sets T_T used in the experiments?**\n\nThese are the official test sets that were originally provided for each dataset. We used the same splits for consistency purposes.", "claims_and_evidence": "The authors introduce a new task for LLM unlearning and highlight its significance. To address this challenge, they present TOOLDELETE as a solution. While they do not provide a theoretical analysis, their extensive experimental results effectively support their claims.", "methods_and_evaluation_criteria": "The authors compare the proposed TOOLDELETE with several baselines, including general and LLM-specific unlearning approaches, and conduct extensive experiments on widely used benchmarks such as ToolAlpca, ToolBench, and API-Bench.", "theoretical_claims": "The authors do not include a theoretical analysis.", "experimental_designs_or_analyses": "The authors present detailed experimental settings, evaluations, and baselines. The design and analysis are well-structured and valid.", "relation_to_broader_scientific_literature": "This paper introduces a novel unlearning task that contributes to the research on LLM unlearning.", "essential_references_not_discussed": "The authors have cited most related works.", "other_strengths_and_weaknesses": "Strengths\n\n1. The authors clearly define the new task and highlight three key properties essential for addressing it: tool knowledge deletion, tool knowledge retention, and general capability retention. They effectively illustrate both the problem and their proposed solution in Figure 1.\n2. The introduction of LiRA-Tool, a membership inference attack (MIA) specifically designed for tool unlearning, provides an effective metric to assess whether tool-related knowledge has been successfully removed.\n3. The authors propose the use of \"shadow samples\" as a strategy to remove the dependency on accessing training data.\n4. Unlike many traditional methods, TOOLDELETE can support sequential unlearning efficiently.\n\nWeaknesses\n\n1. While the empirical results are strong, the paper does not provide a formal theoretical foundation to support its claims about unlearning effectiveness.\n2. Although the authors introduce TOOLDELETE, it primarily builds on existing methods and lacks significant technical novelty.", "other_comments_or_suggestions": "See the above section.", "questions_for_authors": "1. What do you mean by this \"By moving beyond limited training prompts, LiRA-Tool ensures that the model loss reflect overall tool-using ability, rather than just sample level memorization.\"\n2. What are the test sets $T_T$ used in the experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HeCeGs6Dmk", "reviewer": "Reviewer_wfx6", "summary": "The paper innovatively introduces and conceptualizes tool unlearning for tool-augmented LLMs. The authors propose a novel tool unlearning method called TOOLDELETE with two variants, which satisfies three key properties: tool knowledge removal, tool knowledge retention, and general capability retain. They further introduce a new membership inference attack for tool unlearning. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that TOOLDELETE effectively unlearns randomly selected tools while preserving the LLM’s knowledge on non-deleted tools and maintaining performance on general tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re W1: The authors do not explain how to train TOOLDELETE using RLHF and quantization. There is no formula description for the training process of the given two variants of TOOLDELETE.**\n\nOur primary focus in this paper has been on proposing the ToolDelete framework and full implementation and analysis of two variants of ToolDelete: SFT and DPO. As mentioned in Lines 185—193, ToolDelete is also compatible with other training paradigms and optimization techniques, including RLHF and quantization. We acknowledge that the description of RLHF and quantization could have been clearer. To clarify: \n\n**RLHF**: share the same training data as DPO. The only difference is that RLHF requires training a reward model, whereas DPO does not. To illustrate, we describe one possible implementation for RLHF:\n- Collect prompt-response pairs as in DPO. \n  - For each unlearned tool $t_i \\in \\mathcal{T}_f$, designate the tool-free response as the “winning” response and the tool-knowledge response as the “losing” response. \n  - For each retain tool $t_j \\in \\mathcal{T}_r$, do the opposite: designate the tool-knowledge response as the “winning” response and the tool-free response as the “losing” response.\n- Train a reward model on these preferences.\n- Train the LLM (policy network) to maximize the reward model using PPO, following the standard RLHF procedure (Stiennon et al., NeurIPS 2020, Ouyang et al., 2022).\n\n**Quantization**:  is not a training technique but a model compression and efficiency technique that can be applied with ToolDelete after training with SFT, DPO, or RLHF. Therefore, it does not affect the core unlearning mechanism.\n\nWe will make these points clear in the revised version and defer the RLHF variant and quantization experiments to future work.\n\n\n**Re Q1: What is function $g$ specifically? Is the output of $g$ nonnegative? Why isn't formula (1) written in a form similar to formulas (2) and (3), or just $\\mathbb{E}_{t_i \\in \\mathcal{T}_f} [ g(f', t_i) ] = \\epsilon$, where $\\epsilon$ is an infinitesimal constant?**\n\nThe function $g(f, t)$ quantifies the model’s knowledge on tool $t$, such as prompting and evaluating $f$ for ability to use $t$, and is nonnegative. We use Equation (1),  $\\mathbb{E}_{t_i \\in \\mathcal{T}_f} [ g(f_0, t_i) - g(f', t_i) ] \\geq 0$, to explicitly compare the unlearned model $f’$ to the vanilla model $f_0$ to make sure $f’$ retains no more knowledge that $f_0$. As we mentioned in Lines 94—96, $f_0$ may already have some tool-using capabilities prior to explicit unlearning (such as basic arithmetic operations). Using our formulation in Equation (1) would give more flexibility in how much tool knowledge we want to unlearn, as we described in Line 135—142. In fact, only using $g(f', t_i) = \\epsilon$ would ignore this variation and risk being either too strict or too lenient depending on the tool.\n\n**Re Q2: How is the hyperparameter $\\alpha$ selected?**\n\nWe select $\\alpha$ (which controls the effect of task arithmetic ) through grid search on a held-out validation set.\n\n\n\n**Re Suggestion 4: Lack of explanation for $P_U(\\cdot)$ and $P_{T_r}(\\cdot)$ in formulas (7) and (8).**\n\nWe clarify that $P_U(\\cdot)$ indicates the distribution of unlearned tools $T_f$ under the unlearned model $f'$, while $P_{T_r}(\\cdot)$ denotes the distribution of the retain tools $T_r$ under the retained model $f$. We thank the reviewer for their suggestion. We will incorporate these definitions explicitly in the paper. \n\n\n**Re: Suggestions.**\n\nWe thank the reviewer for their  helpful suggestions. We will move the training time comparison from the appendix to the main text, fix Line 79 in the Introduction, clarify the meaning of $\\theta_0 - \\theta_R$​ in the caption of Figure 1, and correct the grammar error on Line 390.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The reviewer has checked the correctness of the theoretical claims.", "experimental_designs_or_analyses": "The reviewer has checked all of the experimental designs. The reviewer is confused that the performance results do not include that of TOODELETE without variants.", "relation_to_broader_scientific_literature": "This paper presents work whose goal is to advance the field of machine unlearning, especially the unlearning for tool-augmented LLMs, which is specifically oriented to improve the trustworthiness of LLMs with external tools.", "essential_references_not_discussed": "In the latter part of the section, **Unlearning for non-LLM models**, the works study unlearning under multimodal setting (Cheng & Amiri, 2025) togther with the following should be in the section of **Unlearning for LLMs**.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The idea is novel. The paper is the first to investigate the unlearning problem of previously learned tools from tool-augmented LLMs. The study of tool unlearning is meaningful.\n2. The paper is overall well-structured.\n\nWeaknesses:\n\nThe section **Training Details** is quite confusing. The authors do not explain how to train TOOLDELETE using RLHF and quantization. There is no formula description for the training process of the given two variants of TOOLDELETE.", "other_comments_or_suggestions": "1. It would be better to include the comparison results of training time in the main context.\n\n2. Missing results in the last paragraph of the Introduction part on Line 078: \"... ***by + in*** accuracy on forget tools ...\"\n\n3. Lack of explanation for $\\theta_0-\\theta_R$ in the caption of Figure 1.\n\n4. Lack of explantion for $P_U(\\cdot)$ and $P_{T_r}(\\cdot)$ in formulas (7) and (8).\n\n5. Grammar mistake on Line 390: \"Why is TOOLDELETE **effectiveness**\"", "questions_for_authors": "What is function $g$ specifically? Is the output of $g$ nonnegative? Why isn't formula (1) written in a form similar to formulas (2) and (3), or just $\\mathbb E_{t_i\\in \\mathcal T_f}[g(f',t_i)]=\\epsilon$, where $\\epsilon$ is an infinitesimal constant? How is the hyperparameter $\\alpha$ selected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GQpDX53ooj", "reviewer": "Reviewer_tu3r", "summary": "This paper emphasizes an emergent problem that LLMs need to unlearn tools that have potential security concerns, and therefore proposes the ToolDelete method to remove the knowledge of using specified tools, as well as introducing an adapted membership inference attack (MIA) method to evaluate the tool unlearning progress. The experiments show superior model performance in forgetting undesired tools, maintaining knowledge about other tools and general arithmetic abilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re W1: Table 1 reports task-solving accuracy, where a low accuracy could be: (i) the model not using the tool (which is good), or (ii) the model still heavily uses the tool but incorrectly. It is unclear if (ii) should be considered good; and if not, the reported results mix (ii) into the main aspect of interest (i.e., (i)).**\n\n\nWe thank the reviewer for their comment. We agree that distinguishing between (i) and (ii) is important. In the context of tool unlearning, incorrectly using a tool, case (ii), indicates the model no longer has the tool-using capability, which is the goal of tool unlearning. For example, if the model generates incorrect arguments for an API, the API call will fail, leading to an errorful or unusable response, which reflects the loss of practical tool-using ability. However, we fully agree with the reviewer that separating cases (i) and (ii) can provide deeper insights. To do so, we conducted additional evaluation by measuring if the unlearned tool is called by the unlearned model. A lower score means that the unlearned model does not use the unlearned tool post-unlearning.\n\nWe can see that after applying ToolDelete, the unlearned model seldom uses the forget tools, outperforming existing unlearning methods, indicating a successful tool unlearning.\n\n| Method           | Case (ii) (%, $\\downarrow$) |\n|------------------|---------------------------------------------------------------------------------------|\n| Retrain          | 11.3     |\n| GradAscent       | 27.1|\n| RandLabel        | 27.5 |\n| SalUn            | 25.6     |\n| ICUL             | 72.4     |\n| SGA              | 25.9     |\n| TAU              | 26.4      |\n| ToolDelete - SFT | 12.0 |\n| ToolDelete - DPO | **10.8** |\n\n\n\n\n\n**Re W2: Justification for effectiveness of LiRA-Tool: While the method design involves some added modules as introduced in Section 3.5, the results in Figure 2 may not be sufficient to justify the effectiveness of the LiRA-Tool.  It is unclear to me: (1) what additional information this MIA-based method tells beyond default accuracy measures   (2) how much more effective LiRA-tool is compared to standard MIA or similar approaches.**\n\nThe innovation of LiRA-Tool is in focusing on “unlearning functional capability”, which is beyond the samples in the forget set. This is done through evaluating the unlearned model more diverse prompts. As discussed in $\\S$3.5 and Figure 2, models can show deceptive surface-level behavior—e.g., low accuracy on a forget set—without truly forgetting the tool, such as suppressing tool knowledge for specific prompts. In fact, recent works have found that LLM unlearning is susceptible to re-learning attacks (Hu et al., ICLR 2025) or jailbreak attacks (Lynch et al., 2024). LiRA-Tool stress-tests the model using broader, large-scale, and more varied samples to provide a more robust estimate of whether the tool-using ability has been removed.\n\n\n**Re W3: three datasets in Section 4, but only results on ToolAlpaca.**\n\nWe acknowledge that most of the detailed experiments in the main paper focus on ToolAlpaca as the dataset for case study, space constrained, and the high computational cost of experiments. We also conducted several experiments on ToolBench and API-Bench, see Table 6 and Table 7 in Appendix. \n\n\n**Re W4: unlearn 2-20% tools randomly, but Table 1 caption says “deleting 20%” which describes different setups.**\n\nThe 2-20% unlearned set is used in sequential unlearning, and the corresponding results are reported in Appendix D, Figure 3. We will clarify this point in the revised version. \n\n\n**Re W5: If a fixed proportion (e.g., 20%), two important experiments to do is (1) fixing the 20%, how do differently selected tools affect the final evaluation result reported in Table 1 and Figure 2? And (2) how would the choice of tool deletion proportion (e.g., from 20% to 10%, 50%, etc.) affect the experiment results?**\n\nWe thank the reviewer for their comment. We have partially addressed both reviewer questions in the current version and plan to expand on them further in future work:\n\n(a): Tool selection at fixed proportion: We partially addressed this via class-wise unlearning experiments, where the entire tool categories are unlearned. The results showed how different selection of tool categories (as opposed to random sampling) affect outcomes.\n\n (b): Varying deletion proportions: Our sequential learning experiments (Appendix D, Figure 3) show the effect of varying deletion proportions. In the final version, we will include results comparing multiple random subsets to analyze performance variability.", "claims_and_evidence": "1. The proposed ToolDelete method can effectively remove knowledge about unwanted tools and maintain knowledge about other tools. Results in Table 1 show decreased and improved results on the undesired and the remaining tools. However, Table 1 reports task-solving accuracy, where a low accuracy could be: (i) the model not using the tool (which is good), or (ii) the model still heavily uses the tool but incorrectly. It is unclear if (ii) should be considered good; and if not, the reported results mix (ii) into the main aspect of interest (i.e., (i)).\n2. Justification for effectiveness of the LiRA-Tool evaluation: While the method design involves some added modules as introduced in Section 3.5, the results in Figure 2 may not be sufficient to justify the effectiveness of the LiRA-Tool. As Table 1 is not convincing enough to show ToolDelete is a better tool-unlearning method, and the accuracy reporting concern (as in the point above), LiRA-Tool showing similar results may not justify its effectiveness. That being said, comparing other baseline MIA methods could help strengthen the point that LiRA-Tool is valuable.", "methods_and_evaluation_criteria": "1. The set of evaluation dimensions are comprehensive and reasonable to study the tool-unlearning problem, including measuring performance on to-forget and remaining tools. Also, evaluating models on general problem abilities (STEM, reason, instruction-following, and facts) is important. A minor point: in Section 3.3, the authors proposes to test general capabilities via Arithmetic tasks, the paragraph “Why Task Arithmetic?” does not seem convincing enough, particularly why arithmetic is core LLM ability and how crucial it is to tool-using. Nonetheless, the list of tasks used in experiments are not necessarily arithmetic tasks (?) but seem reasonable and comprehensive to me. Consider changing the overall description of tasks in Section 3.3 could be an easier fix to this confusion.\n2. The proposed LiRA-Tool is a novel and seemingly more targeted evaluation tool for the tool-unlearning problem. However, it is unclear to me: (1) what additional information this MIA-based method tells beyond default accuracy measures, and (2) how much more effective LiRA-tool is compared to standard MIA or similar approaches.", "theoretical_claims": "The work represents the three key properties of the proposed ToolDelete method, as well as the model training details, in symbolic expressions, which all read reasonable.", "experimental_designs_or_analyses": "The experimental design looks reasonable overall. However, a few points I found a bit confusing:\n1. The “Datasets & Tool-Augmented LLMs” in section 4 mentions three datasets (ToolAlpaca, ToolBench, API-Bench) for evaluation, however all following experiments are only performed on ToolAlpaca.\n2. The “Setup & Evaluation” paragraph afterward says “with 2-20% tools randomly selected”, but Table 1 caption says “deleting 20%” which describes different setups, please align the setup description and accurately describe the experiments.\n3. If a fixed proportion (e.g., 20%) of tools is selected for deletion (as in point 2), two important experiments to do is (1) fixing the 20%, how do differently selected tools affect the final evaluation result reported in Table 1 and Figure 2? And (2) how would the choice of tool deletion proportion (e.g., from 20% to 10%, 50%, etc.) affect the experiment results?", "relation_to_broader_scientific_literature": "The key contribution of the paper is providing an adapted unlearning method for LLM-used tools. The paper is therefore largely related to knowledge/data uncleaning for LLMs.", "essential_references_not_discussed": "The related work section discuss unlearning approaches rather comprehensively. However, regarding the MIA-based LiRA-Tool for evaluation, the related work does not discuss too much related work to it, and may benefit from more systematic comparisons to these evaluation methods.", "other_strengths_and_weaknesses": "Strength: The target task – tool unlearning – could be an emerging yet unexplored area, for which it is interesting to develop methods and evaluation procedures.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "For the tool benchmarks listed for evaluation in Section 4, at least on ToolBench, some tools are no longer executable/available, thus making it difficult to reproduce their original results or conduct fair comparisons. I wonder if the authors have encountered similar issues on ToolBench or the other datasets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ux16nYuPlK", "reviewer": "Reviewer_qa8k", "summary": "The authors introduce tool unlearning, a novel task in the machine unlearning domain, motivated by the need to remove learned tools from tool-augmented LLMs due to security, privacy, or obsolescence concerns. Unlike traditional unlearning, this task presents unique challenges, including knowledge removal beyond individual sample forgetting, the high computational cost of optimizing LLMs, and the lack of principled evaluation metrics. To address these challenges, the authors propose ToolDelete, the first dedicated approach for tool unlearning, which incorporates three key properties for effective unlearning. Additionally, they introduce a new membership inference attack (MIA) model for evaluation. Experimental results on three tool-learning datasets demonstrate that ToolDelete successfully unlearns both randomly selected and category-specific tools while preserving the model’s general knowledge and maintaining performance on non-deleted tools.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re W1: I believe that the standard unlearning task is not strictly limited to sample-level unlearning. For example, WMDP focuses on forgetting sensitive concepts such as biology, while MUSE Books aims to forget concepts related to Harry Potter. Based on the authors’ definition of Tool Unlearning, it seems more aligned with in-context unlearning [1]. Therefore, I feel the authors may be overstating their novelty.**\n\nThank you for raising this important point. Tool unlearning differs from both concept-level and in-context unlearning in several important ways:\n\n(a): parametric capability removal vs. concept forgetting: concept-level unlearning  in WMDP and MUSE Book mainly focus on “semantic or factual concepts” embedded in the model’s knowledge. In contrast, tool unlearning focuses on “functional capabilities,” i.e. the ability to perform a task using a specific tool or API, which is action-oriented as opposed to semantic-oriented modifications in concept-level unlearning.  We acknowledge the similarity and will make the description clear in the revised version.\n\n(b): In-context unlearning operates at the “prompt level” without modifying model parameters. As we show in our experiments (e.g. ICUL, Table 1 and Line 316--329), this form of unlearning is insufficient for tool unlearning, since the parametric knowledge of the tool remains and can be invoked with adversarial or indirect prompts. \n\nWe will make these distinctions clear in the revised version. \n\n\n\n**Re W2: The contribution of LiRA-Tool also seems overstated. The method essentially applies LiRA to the current setting without introducing significant innovation. Additionally, the paper does not provide any analysis or experimental evidence explaining why traditional MIA (Membership Inference Attacks) perform poorly in the Tool Unlearning setting.**\n\nWe respectfully argue that the adaptation of LiRA to tool unlearning is both necessary and non-trivial. Traditional LiRA targets sample-level membership, which is insufficient for evaluating whether a model has truly forgotten a tool-using capability, which requires evaluating broader functional capability. LiRA-Tool introduces two key innovations: (1): shifting the evaluation from sample-level membership to skill-level forgetting, and (2): using shadow samples to robustly probe tool knowledge beyond the training data. \n\nRecent works have found that LLM unlearning is susceptible to re-learning attack (Hu et al., ICLR 2025) or jailbreak attack (Lynch et al., 2024). LiRA-Tool stress-tests the unlearned model, ensuring that a specific tool has truly been forgotten, providing better demonstration of successful tool unlearning, instead of only sample de-memorization.\n\n\n**Re W3: The authors do not propose any new unlearning method. Instead, they simply apply SFT and DPO to existing benchmarks (ToolAlpaca, ToolBench, etc.), without introducing a novel approach to unlearning.**\n\nWe would like to clarify that the ToolDelete framework, its unlearning objective formulation—Equations (1-4)—and its approach to general capability retention via task arithmetic are novel and specifically designed for tool unlearning. We believe these contributions are substantial and fill an existing gap in current unlearning research. In addition, we clarify that ToolDelete uses SFT and DPO simply as training methods.  \n\n\n**Re W4: Hyperparameter impact: Compared to standard baselines, the proposed method introduces several additional hyperparameters from TOOLDELETE and SFT/DPO. While TOOLDELETE clearly introduces even more hyperparameters, the paper does not analyze their impact on the final results, which is a crucial omission.**\n\nToolDelete introduces a few additional parameters and we have analyzed them throughout the main paper and Appendix. Specifically:\n- $\\alpha$ in Eq (6): we tuned $\\alpha$ on a held-out validation set. \n- Training method (SFT vs. DPO): The results of both variants were shown in Table 1.\n- Access to training data: In Appendix D, Table 4, we reported performance comparison with and without having access to the exact training samples.\n- PEFT vs. full-parameter tuning: In Appendix D, Table 3, we compared LoRA with full parameter tuning. \n- Choice of tool-free response in Eq (1): In Appendix D, Table 5, we evaluated the impact of using different sources for tool-free responses (e.g. $f_0$ vs. $f_R$).\n\nWe will make sure these references are explicitly explained in the final version of the paper, and provide a brief summary table of hyperparameters and their effects for clarity. We appreciate the reviewer highlighting this point.", "claims_and_evidence": "I think the authors overclaim their contributions. I discuss them in Questions For Authors.", "methods_and_evaluation_criteria": "The author did not propose a new method but applied SFT/DPO to the Tool benchmark.", "theoretical_claims": "No theoretical Claims.", "experimental_designs_or_analyses": "All of them. I discuss them in Questions For Authors.", "relation_to_broader_scientific_literature": "I discuss them in Questions For Authors.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I discuss them in Questions For Authors.", "other_comments_or_suggestions": "I discuss them in Questions For Authors.", "questions_for_authors": "1. I believe that the standard unlearning task is not strictly limited to sample-level unlearning. For example, WMDP focuses on forgetting sensitive concepts such as biology, while MUSE Books aims to forget concepts related to Harry Potter. Based on the authors’ definition of Tool Unlearning, it seems more aligned with in-context unlearning [1]. Therefore, I feel the authors may be overstating their novelty.\n\n[1] Pawelczyk, Martin, Seth Neel, and Himabindu Lakkaraju. \"In-context unlearning: Language models as few-shot unlearners.\" arXiv preprint arXiv:2310.07579 (2023).\n\n2. The contribution of LiRA-Tool also seems overstated. The method essentially applies LiRA to the current setting without introducing significant innovation. Additionally, the paper does not provide any analysis or experimental evidence explaining why traditional MIA (Membership Inference Attacks) perform poorly in the Tool Unlearning setting.\n\n3. The authors do not propose any new unlearning method. Instead, they simply apply SFT and DPO to existing benchmarks (ToolAlpaca, ToolBench, etc.), without introducing a novel approach to unlearning.\n\n4. Hyperparameter impact: Compared to standard baselines, the proposed method introduces several additional hyperparameters from TOOLDELETE and SFT/DPO. While TOOLDELETE clearly introduces even more hyperparameters, the paper does not analyze their impact on the final results, which is a crucial omission.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies machine unlearning for tool-augmented LLMs by proposing TOOLDELETE, a method designed to remove previously learned tools. TOOLDELETE satisfies three key properties: tool knowledge removal, tool knowledge retention, and general capability retention. The paper also introduces an adapted membership inference attack (MIA) method to evaluate the effectiveness of tool unlearning.

The reviewers appreciate the following strengths:
1) The paper addresses an emerging and important task—machine unlearning for tool-augmented LLMs.
2) It proposes a new tool unlearning method with a clear definition of desiderata.
3) It adapts the LiRA-Tool membership inference attack (MIA) as an evaluation method for assessing the sufficiency of tool unlearning.
4) The paper is overall well-written and well-structured.

However, the reviewers also raised some concerns:
1) A lack of theoretical analysis regarding tool-unlearning effectiveness.
2) TOOLDELETE incorporates some techniques from existing LLM unlearning literature.
3) Insufficient clarity in the experimental settings.
4) A limited introduction and comparison with existing MIA approaches.

During the discussion, the authors clarified the new technical contributions, experimental settings, and the motivation and novelty of LiRA-Tool, as well as providing new experimental results.

Three reviewers are positive following the discussion. One reviewer remains negative, primarily due to concerns that TOOLDELETE borrows some techniques (e.g., the unlearning objective function) from existing LLM unlearning methods. Regarding novelty, the area chair aligns with the majority of reviewers and considers the formulation and solution of the tool-unlearning task to be novel. Overall, the paper is at the borderline for acceptance. Therefore, the area chair recommends weak acceptance of the paper, with the final decision left to the SAC and PCs.
Decision: Accept (poster)