BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models
Authors: Thierry Blankenstein
Jialin Yu
Zixuan Li
Vassilis Plachouras
Sunando Sengupta
Philip Torr
Yarin Gal
Alasdair Paren
Adel Bibi
Venue: ICLR 2026 Poster (2026)
Keywords: Tool Selection
LLM Agents
Fairness
Bias
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Agents backed by large language models (LLMs) increasingly rely on external tools drawn from marketplaces where multiple providers offer functionally equivalent options. This raises a critical fairness concern: systematic bias in tool selection can degrade user experience and distort competition by privileging certain providers over others. We introduce a benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to systematically evaluate tool-selection bias. Using this benchmark, we evaluate seven LLMs and show that substantial bias persists, with models either fixating on a single provider or disproportionately favoring tools that appear earlier in the context. To uncover the sources of this behavior, we conduct controlled experiments that isolate the effects of tool features, exposed metadata (name, description, and parameters), and pre-training exposure. We find that (1) semantic alignment between user queries and tool metadata is the strongest driver of selection; (2) small perturbations to tool descriptions can significantly shift choices; and (3) repeated pre-training exposure to a single endpoint amplifies provider-level bias. Finally, we propose a lightweight mitigation strategy that first filters tools to a relevant subset and then samples uniformly, substantially reducing selection bias while maintaining strong task coverage. Our results highlight tool-selection bias as a key obstacle to the fair deployment of tool-augmented LLM agents.

Published as a conference paper at ICLR 2026
BIASBUSTERS: UNCOVERING AND MITIGATING TOOL SELECTION BIAS IN LARGE LANGUAGE MODELS
Thierry Blankenstein', Jialin Yu-.2.1 Zixuan Li', Vassilis Plachouras",Sumann Sunando Sengupta", Philip Torr', Yarin Gal', Alasdair Paren', Adel Bibi UUniversity Oxford, Oxford, UK Microsoft, UK
CCorespooniing authors
ABSTRACT
Agents backed by large language models (LLMs) increasingly rely on external tools drawn from marketplaces where multiple providers offer functionally equiv- alent options. This raises critical fairness concern: systematic bias in tool selec- tion can degrade user experience and distort competition by privileging certain providers over others. We introduce benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to systematically evaluate tool- selection bias. Using this benchmark, evaluate seven LLMs and show that sub- stantial bias persists, with models either fixating on a single provider or dispropor- tionately favoring tools that appear earlier in the context. To uncover the sources of this behavior, we conduct controlled experiments that isolate the effects of tool features, exposed metadata (name, description, and parameters), and pre-training exposure. We find that (1) semantic alignment between user queries and tool meta- data is the strongest driver of selection; (2) small perturbations tool descriptions can significantly shift choices; and (3) repeated pre- pre-training exposure to single endpoint amplifies provider-level bias. Finally, we propose a lightweight mitiga- tion strategy that first filters tools to relevant subset and then samples uniformly, substantially reducing selection bias while maintaining strong task coverage. Our results highlight tool-selection bias a key obstacle to the fair deployment of tool-augmented LLM agents. Our code and benchmark are publicly available at https:: thg...huu...... cwm httmm/hhhrrrrrrr
see eeeeletiion ioactioo-ie ion-bias
INTRODUCTION
Large language models (LLMs) have transformed natural language processing, achieving near- human performance on tasks ranging from code generation creative writing (Naveed et al. [024] Luo et all: 2024). Yet LLMs camnot directly act in the world: they cannot query databases, fetch live information, or invoke external services. Additionally, their knowledge remains frozen at training time, leaving them prone to "hallucinations" when asked about events beyond their cutoff (Ji et al, 2023). Augmenting LLMs with external "tools" / APIs addresses these shortcomings by allowing models to delegate specialized functions to dedicated services (Qu et al. (222).. It endows LLMs with the ability act, core capability often associated with LLM agents (Chowa al. 2025). A crucial step within the typical tool-usage pipeline is the multi-stage tool-selection process: given an instruction the LLM, (i) retrieve a short list of the most relevant candidate tools based on the user query (with, c.g., highest semantic similarity) from a potentially large database of tools. (ii) insert their metadata into the prompt, (iii) have the LLM reason and pick one to solve (one of) the necessary user task(s) (see Appendix covering the entire tool-usage pipeline). However, this pro- cess introduces a new challenge: bias (see Figure []. An LLM may prefer certain tools not for their relevance or accuracy, but because of superficial metadata, i.e., tool names, descriptions, or prompt ordering. Such bias can degrade user experience by repeatedly selecting slow or unreliable ser- vices and can, therefore, also inflate operational costs. Additionally, under pey-per-request pricing for tool use, consider the scenario where such biases are systematic across frontier LLMs: if they consistently favor tools from single provider, this creates major market unfairness, disadvantag-

---

Published as a conference paper at ICLR 2026
Run #1:
Run #2:
User: What's the weather like in London?
User: What's the weather in Paris?
Available tools
Agent
Thought: need get the weather in London.
Tool Name: weatherapl com
Available tools:
Arguments: ("city": "London")
world weather online_api api. weatherapi com, weather forecast 14 days
Agent
Run #K:
User: How's the weather LA?
Available tools
Agent
Thought: need to fetch the weather Paris.
Tool Name: weatherapi ceaeherapi_com com Arguments: ("city": "Paris")
Thought: Let's fetch the weather LA.
Tool Name: weatherapi_com _cm com Arguments: ("city": "LA")
Figure/ 1: Tool-calling enables LLMs act through external services, but the selection process introduces bias. Models may favor certain tools based on superficial metadata or position rather than relevance (here "weatherapi.com is preferred), leading to a potential degraded user experience and unfair concentration of calls. If such biases are systematic across frontier LLMs, they risk distorting entire tool marketplaces, disadvantaging functionally equivalent competitors.
ing competing providers offering functionally equivalent services (RapidAPI 2025a Economize 2024). See Appendix D
covering the consequences of tool-selection bias in more detail.
In this paper, we make three key contributions.
First, present large-scale benchmark for measuring tool selection bias, which the first empirical study of tool-selection bias in LLMs. We introduce total variation-based metrics to quantify selection imbalances, assemble novel benchmark of clusters of tools with equivalent functionality, and conduct extensive experiments across multiple models and configurations. These tell us that bias exists to certain extent for all models tested; models either fixate on single provider or disproportionately prefer carlier-listed tools in context.
conduct
Second, after confirming the existence of bias, investigate the root causes of these biases. We define a rich set of tool-level features and use regression and permutation importance analyses to identify which factors predict selection. We further perturb metadata fields test their influence and probe whether continued pre-training on biased data can induce persistent preferences. Our findings show that semantic alignment between query and tool description is the strongest interpretable driver of tool choice, metadata interventions on descriptions reliably steer choices, and biased pre-training amplifies preferences. However, none of these fully explain the behavior.
commutttion-impottancce
Third, we propose a lightweight mitigation strategy: filtering the supplied tools to subset of relevant ones and sampling uniformly among them. This approach substantially reduces bias while still ensuring the user task can be solved.
By uncovering, explaining, and addressing tool-selection bias, our work illuminates a critical blind spot in tool-augmented LLM research. Our contributions go beyond diagnosis: we provide repro- ducible resources and a simple mitigation strategy that practitioners can adopt immediately. More broadly, aim to set foundation for fairer, more reliable tool-calling systems and precedent to judge tall-calling LLM applications not only by their accuracy but also by the equity of their interactions with external ecosystems.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Fairness & Bias in LLMs. Bias in large language models emerges from their opaque training on vast web-scale data, often carrying forward and amplifying human prejudices and reasoning short- cuts (Gallegos et al. 2024). Much of the literature has focused on identifying and mitigating these undesirable behaviors (Schick et al. 2021 | Gallegos et al. 22024; Bouchard 2024). Unwanted behav- iors in LLMs manifest as social bias, where stereotypes shape oulputs and lead unequal treatment of protected groups. or cognitive bias, reflecting human-like shortcuts such as anchoring or confir- mation effects (Lou & Sun 2024 Itzhak et af. 20244 O'Leary (025). Measuring these phenomena requires domain-specific metrics. For example, to measure how likely model is to associate neu- tral attribute with one group over another, one can utilize the normalized log-probabilit, bias score (Kurita al., 2019). More recently, formal certification frameworks have been proposed to quantify counterfactual bias over distributions of prompts with statistical guarantees (Chaudhary et al. 2025). Mitigation strategies are diverse and can be deployed at multiple stages (e.g.. at pre-/post-inference via augmentation of training data or rewriting of outputs or during training via fairness-aware ob- jectives) (Gallegos al., 2024). However, no single approach suffices for every scenario; effective debiasing requires interventions matched to the task's 's fairness goals and deployment context. We are not aware of prior work on bias in the tool-use paradigm.
Positional Bias in LLMs. Prior research has revealed significant positional selection biases in how LLMs answer multiple-choice questions. Pezeshkpour & Hruschka (2024) and Zheng et al. (2024) demonstrated that LLMs exhibit sensitivity to the ordering of answer options, with performance varying systematically based on option position. This was corroborated the investigation of Wei et al. 2024), who also found token sensitivity, where specific token characteristics at different posi- tions affect selection. Recently Wang et al. (2025), analyses the underlying model mechanisms that give rise to position bias and proposed mechanistic approach to eliminating it. In this work, study and investigate a broad range of potential biases in the critical context of tool selection.
Fairness in Tool Selection. Research on tool-selection bias is limited; however, some works in information retrieval address related issues such as positional or ranking bias (Dai et al. 2024 Ziems et al. 2024). Similarly, in code generation, favoritism for certain providers has been observed (Zhang et al. 2025b). However, these studies do not examine the specific setting of tool selection among providers. Recent studies do show that LLMs can develop systematic preferences in this setting, for example, for selecting tools with manipulated metadata (Mo et al.) 20255 Faghih et al., 2025| Zhang al. 2025a). Analogously, one study shows an advanced attack on tool selection which splits malicious tool entries into fragments optimized for retrieval and selection, achieving high success rates while evading defenses (Shi et al., 2025). Other recent work has focused on adversarial robustness of tool selection, proposing statistical certification of tool-selection accuracy under adaptive attacks (Yeon et al., 2025). These findings reveal that tool-selection mechanisms are highly fragile and susceptible to both naïve and advanced exploitation. In contrast to overt adversarial attacks, our study focuses on subtler sources of bias: small, non-adversarial diffcrences in phrasing or metadata that can still distort fairness in tool selection.
3 BIASBUSTERS: HOW WE UNCOVER, EXPLAIN, AND MITIGATE BIAS.
In this section, we present our end-to-end framework for uncovering and explaining tool-selection bias in LLMss We start in Section by introducing our formal definition of this bias. In Section 3.2 we describe how we generate a comprehensive benchmark of API clusters and user queries that enables systematic measurement of selection behavior. Section 3.3 details our analysis pipeline for explaining bias.
3.1 BIAS DEFINITION
Tool-selection bias captures the extent to which an LLM systematically favors certain APIs over others for reasons unrelated to their true utility. Formally:
Definition 3.1 Tool-selection bias is the systematic tendency of a model to favor certain APIs over others for reasons unrelated to the APIs' true relevance or utility for the task.
3

---

Published as a conference paper at ICLR 2026
To quantify this bias, compare model's empirical selection rates against ideal uniform choice. Consider a cluster of K APIs that can all solve a given query q. A perfectly unbi- ased model would select each API with probability 1/K, forming uniform distribution U. Let pAPI = (PAPI, ..... P) denote the expected empirical selection-rate distribution over the clus- ter under all orderings of the API list. We measure cluster-level bias via the total variation distance
K ÓAPI = 2 1 PAPI
1 K = TV(PAP) U).
Prior work has shown that list order itself can introduce bias: when multiple identical tools ap- pear in sequence, the first position is favored |Faghih et al. (2025). To capture this effect, we also record the expected selection-rate distribution over absolute positions, ppos, and compute 6pus 2 - ppos k Note that high positional bias ōpee can be mitigated by randomizing API order at prompt time, whereas API-level bias SAPI requires deeper intervention.
Combining both yields an overall bias metric: mmodel 6pm
3.2 DATASET GENERATION
We build on the TooILLM pipeline QQnn et al. 2024). which provides large repository of APIs scraped from RapidAPI (RapidAPI, 2025b). To measure tool-selection bias, we construct a bench- mark of functionally interchangeable APIs Specifically, we cluster APIs into groups performing the same task (e.g., weather forecasting or translation), and generate balanced, provider-agnostic user queries that all APIs in a cluster can answer. The final benchmark consists of 10 clusters, each containing 5 APIs and 100 queries, yielding 1,000 total cluster-query pairs. Running LLMs on this benchmark produces empirical API-selection distributions, from which we compute our bias metrics SAPI. ppss and modell. See Appendix A for clustering and query-generation details.
3.3 EXPLAINING BIAS
To pinpoint what drives tool-selection bias, we pursue three complementary analyses.
Attribute-Level Analysis. To test whether intrinsic API characteristics explain model preferences, we extract seven descriptive features for each API. Examples include semantic similarity between queries and descriptions, number of parameters, description length, readability, and promotional wording (See Table 3 for the full list). Our benchmark contains 10 clusters with 5 APIs each, giv- ing 50 APIs in total. For each API, we measure its empirical selection rate and pair this with its feature values. This produces a dataset of 50 (API, features, selection rate) entries per model. We then probe relationships between features and selection behavior in three ways. First, we compute Pearson correlations between tool features and selection rate to capture linear and monotonic asso- ciations. Second, we fit a linear regression per model to quantify the aggregate explanatory power (reported as R²) and inspect coefficients to understand the direction and relative weight of each feature. Third, we train random- forest regressors with cross- voos-vaaii to allow for non-linear in- teractions and obtain alternative measures of feature importance. This pipeline reveals which API attributes most strongly influence the model's choices.
Metadata Perturbation Experiments. To isolate the superficial cues that drive tool-selection pref- erences, we apply a series of controlled perturbations to tool metadata. Specifically, we utilize the following manipulations: (1) Fall name scramble. Replace every tool's name with a fresh 20-character random string, destroying any learned association tied to the literal name; (2) Name shuffle. Permute the tool names among APIs within each cluster so that names no longer align with their original endpoints; (3) Single-tool perturbation. Identify the most frequently chosen tool in each cluster and replace only its name with a random string; (4) Description and parameter scramble. Randomize each tool's descriptive text and parameter description(s) (but keep the orig- inal names intact) to test whether the semantic content beyond the name influences selection; (5) Description-only / Parameter-only scramble. Randomize only the tool descriptions (keeping pa- rameter descriptions intact), or only the parameter descriptions (keeping the tool description intact), to disambiguate their individual contributions; (6) Targeted description scramble seost-selected). Identify the most frequently chosen API in each cluster and scramble only its description to test whether degrading its semantics reduces its selection share; (7) Description transfer (most - least).
4

---

Published as a conference paper at ICLR 2026
Aval lable toola: APT R
tools:
Aval lable toola: API C
APT
APT A
API C
API
API B
Joer:
User: is the weather Paris?
User: What
What the weather in Paris?
What
the weather in Paris?
Figure Cyclie rotations one fixed tool list; each API appears at the top once.
Swap the most-selected API's description with the least-selected API while leaving other metadata untouched, assessing whether swapping semantic "advantage" transfers selection probability; and (8) Full scramble. Randomize each tool's descriptive text, parameter description(s), and tool names to test the effect of having minimal semantic signal in API metadata bias.
By re-running our selection experiments under these alterations, we quantify how much of the ob- served bias is attributable literal names, deeper semantic content in descriptions and parameters, and to relative contrast between a clean and corrupted endpoint.
Biased Continued Pre-Training. We also test whether pre-training data itself can induce tool- selection bias. To verify this hypothesis, we perform biased continued pre-training (CPT) on Qwen3- 8B using ~3.5M tokens deliberately saturated with single API's metadata. See Appendix C for additional details on corpus construction and training setup.
EXPERIMENTS
This section reports the empirical results from experiments. We first describe the experimental setup (Section 4.1). then characterize tool-selection behavior and quantifyy bias using our metrics (Section 4.2 Next, investigate the drivers of bias (Section 4.3) and finally, evaluate
approach to mitigate any observed bias in Section 4.4
4.1
EXPERIMENTAL SET-UP
Models. We evaluate diverse set of LLMs, including GPT-3.5-turbo and GPT-4.1-ming by OpenAl 2025). Claude 3.5 Sonnet by Anthropic (2025), DeepSeek- -ooeeek--3.----. V3.2-Exp (DeepSeek-Al et al. 2025).
Gemini 2.5 Flash (Google 2023).
(Qin et al. 2024), and Qwen3 models spanning
1.7B to 235B parameters Bai et al. 2023).
Reasoning, Each model prompted to produce short chain thought (Wei et al. 2022 followed by at most one tool call. This design makes outputs efficient to generate and casy to analyze.
Parameter Setup Dataset. Experiments use our benchmark of 10 clusters, each containing interchangeable APIs and 100 distinct user queries (Section 3.2 and Appendix A). Unless otherwise noted, decoding uses a temperature of 0.5 and a top-p of 1.0.
API ordering. As noted in previous work (Faghih al. (2025) )) LLMs prefer tools that appear earlier in the prompt. To control for this, we execute each query five times, each with different cyclic rotation of fixed API ordering (see Figure 2). This ensures that every API appears the top for given query exactly once.
4.2 How DO LLMS SELECT AMONG FUNCTIONALLY EQUIVALENT APIs?
Figure Jshows our first results: the empirical selection distributions of six LLMs over three clusters of functionally equivalent APIs (see Appendix G.2 for the full figure including all clusters). Choices are far from uniform. In some clusters (e.g., geocoding), all models concentrate heavily on a single API, whereas in others (e.g., language identification) the distributions that different models do not always prefer the same API, a point we explore further when analyzing alignment in selection behavior.
flatter. We also observe
5

---

Published as a conference paper at ICLR 2026
GPI-411 mini
Clusde
Gumiaai
DospSick
TooILLaMA
(2158)
Address Coordinates
Top News Headlines by Region
Language Identification
Selection Rate
Geoode Turkon
Boc Coood...
Gookro herzun....
Nocmdd...
Beak Taar Semoo.
Newcottilr
Thallgue
Nonaaaaa
What's Langung
Quakk Lanmagg...
Thnelanodd
Langong
Thinshis
Figure 3: Selection distributions for six LLMs across three clusters of functionally equivalent APIs. Each subplot corresponds to one cluster, with the x-axis indicating the API in the cluster and the y-axis showing the (mean) fraction of times each model chose that API over 500 inference runs; error bars indicate the standard deviation across three independent experimental runs. The optimal uniform selection rate highlighted.
Coordinates Address
Email Validation
QR Code Generation
TV distanoe ws. Unifom
0.8
06-
9.4
aPhas
OPE44 anter
Chaudo
Graitd
Demine
ToalLald
Qonu
aPPas
CRaido
Gruita
Dripied
ToatLalts
Gone
OPTaL
Chari
Gonita
Deppiedd
TodiLattA
Gone
OPT-L
opP
Figure 4: API- vs. positional bias by model for three clusters. Bars show total-variation deviation from uniform, where higher values indicate stronger bias.
Positional bias persists even when API-level preference is weak. summariiss API- vs. positional-bias (measured as total variation distance from uniform) across models and clusters. We notice two regimes: high API bias with low positional bias, or low API bias with high positional bias; a few exhibit both elevated. This pattern indicates that when API clearly dominates, models rely more on positional cues.
The extent of LLM bias in tool selection. To get idea of how biased each model is in their tool-selection behavior, we compute the positional bias ōpos. API bias 6API. and their average moodel for seven different models (Section 3.1). Table Sable|rrpooe reports these metrics averaged across all clusters and three runs. All models exhibit substantial bias: dmodel values are around 0.3-0.4, meaning that roughly 30-40% of the selection probability mass would have be redistributed to achieve fairness. GPT-4.1 mini is especially biased, with a combined metric of ~0.38. The least biased model in our suite Qwen (235B), which attains the lowest mmodel-
LLMs are (mostly) aligned their bias. To examine the similarity of selection behavior of the different models, we represent each model by vector obtained by concatenating its empirical selec- tion distributions over APIs across all clusters, then compute pairwise Pearson correlations between these vectors; the resulting matrix is shown in Figure [5] It shows that many of the models share similar bias patterns: GPT 4.1-mini, Claude, Gemini, DeepSeck, and Qwen3 (235B) tend to favor and disfavor the same APIs. GPT-3.5 and Tooll stand apart with consistently lower

---

Published as a conference paper at ICLR 2026
Table 1: Average cluster-level API bias θAPI. positional bias 6pos and combined bias model (mean across clusters and runs + std across runs).
Model
ÓAPI
8 6ps
model
Gemini 2.5 Flash
0.365
0.306
0.335
Claude 3.5 Sonnet
0.370
0.325
0.347
DeepSeek- -eepSeck-V3.2-Exp
0.249
0.5042.003
0.377
Qwen3 235B
0.330z.006
0.1682.004
0.249
TooILLaMA
0.2772.002
0.391
0.334
GPT-3.5-turbo
0.320s.uz2
0.336s.012
0.328
GPT-4.1 mini
0.33 0.088
0.423+.002
0.377
relations, suggesting qualitatively different selection behavior. This clustering of high correlations points to common drivers of bias. This could be due to shared set similar implicit decision rules. The relative divergence of GPT-3.5 and TooMLLaMA highlights that model architecture, capacity, or training objective can alter these tendencies, but overall alignment underscores that tool-selection bias patterns not isolated quirks but mostly reproducible phenomena across LLMs.
Ablation and Sensitivity Analysis. We also analyze the robustness of tool-selection bias across several factors (full results in Appendix(E. Temperaturr: raising temperature reduces bias modestly by softening extreme preferences. Top-p: increasing top-p has only a negligible effect. Model size: larger models exhibit noticeably less bias than smaller ones. API ordering: cyclic random permu- tations produce very similar outcomes, indicating intrinsic preferences dominate. System prompts: rewording restructuring prompts shifts which tools are favored, but does not remove bias. Toolset size: the composition of the toolset given in context is the primary driver of selection behavior, more than the size of the toolset alone (see Appendix D).
4.3 ARE PROL-SELECTIONS DRIVEN BY HUMAN-INTERPRETABLE HEURISTICS?
We investigate the drivers of bias along three complementary axes: (i) feature-level correlations between API attributes and selection rates, (ii) direct interventions on API metadata, and (iii) biased continued pre-training (CPT) test whether exposure alone can plant preferences. We summarise our results below, with full details in Appendix
Which API-level features predict selection rates? The results show a consistent pattern: (1) Semantic similarity between queries and API/ tool descriptions is the strongest predictor of selection (Table 4). By contrast, structural or stylistic attributes (e.g., parameter count, promotional wording) exhibit little consistent influence. (2) Linear regression reveals that surface-level semantic alignment the primary signal but leaves a lot unexplained (R² < 0.4 as can be seen in Figure 13 and (3)
Random forests fail to offer meaningful improvement.
How do metadata interventions affect API selection? Figure 5 shows the TV distance from the base selection distribution to the distribution obtained after a certain metadata perturbation, aver- aged over clusters and experimental runs. It shows a clear hierarchy: logically minimizing semantic signal by scrambling the description, name, and parameters causes the biggest shift. For Gemini, corrupting descriptions-paraneee
yields the second largest, most reliable shifts in selection (0.450 0.203 TV), followed by scrambling the description of the most popular tool (0.419 + 0. 0.179), and description swaps meaningfully steer choices (0.338 + 0.205). By contrast, name-only per- turbations are smaller and have higher variance, and parameters-onle are the least impactful. We similar results for GPT; description perturbation is more impactful than name/parameter pertur- bations. Overall sensitivity is higher for Gemini than GPT (mean TV 0.310 vs 0.234), indicating greater responsiveness metadata changes.
We show similar results across clusters in Figure 14 however, we observe that manipulating the description of certain tools has mixed effects from cluster cluster. It tells that the impact of

---

Published as a conference paper at ICLR 2026
Centan
CharGPT
Base
Mean TV distancs
Pbll
Parain
Torg. Dest.
Done.
Srop.. Dewc
Rand: Name
Skalf Name
Paran.
Tay Nome
Figure 5: Mean total-variation (TV) distance from the base selection distribution (no perturbation) to the distribution pertaining to each metadata perturbation (higher = larger shift). Blue bars show results for Gemini and orange bars for GPT-3.5. The error bars denote standard deviation across clusters. The run-to-run standard deviation is left out; max run-to-run variability of per-run mean TV was 0.084.
description tampering context-dependent: the same intervention can invert, redistribute, or barely change preferences, underscoring that tool choice emerges from multiple interacting cues.
Together, these patterns indicate that description-level semantics are the primary cues models to discriminate among functionally similar APIs. Name perturbation alone tends to inject noise without consistent effects. Finally, bias persists under minimal semantic signal (only leaving certain parameter schema fields intact). implying selection behavior sometimes relies on residual, non- obvious priors rather than solely on coherent, human-intens intrann-rtrrretbble heuristics. For an analysis of the impact of metadata perturbations on bias, see Appendix
Does additional pre-training exposure to one endpoint change selection distribution? We run biased continued pre-training on Qwen3-8B in which the training corpus is saturated with meta- data for a single target endpoint (Text Language by API-Ninjas All inference settings, prompts, and decoding parameters are held fixed; only the checkpoint changes (see full training and infer- ence hyperparameters in Appendix [D. We then compare selection rates within the same Language Identification cluster before CPT and after 11/3 ~2/3, and one full epoch of biased training (See Figure [7 for full figure). We observe that biased CPT substantially increases the selection of the exposed endpoint but does not fully determine choice. The target endpoint's 's share rises from 0.006 (base) 0.122 after 1/3 epoch, remains 0. at 2/3 epoch, and nudges to 0.128 at one epoch. This an absolute gain 12 percentage points (over 20x relative). Most of the shift realized early, suggesting a quickly saturating response. This demonstrates that biased exposure during training can directly shape tool-selection preferences favor the exposed endpoint. However, since the target never approaches dominant share, pre-training exposure explains only part of the bias, and additional factors still shape tool choice.
4.4 CAN WE MITIGATE THE OBSERVED BIAS?
After demonstrating the existence and causes of bias, we seek to mitigate it. Our approach is simple: models often recognize which APIs can solve certain task, but still exhibit skewed preferences among interchangeable endpoints. Hence, we propose to decouple recognition from selection via lightweight debiasing module. This module uses smaller LLM (Qwen3-14B), prompted to return only the subset of APIs from the candidate list that can solve the given query. We then choose uniformly at random from this subset, ensuring that each valid API has the same expected probability of selection. This eliminates positional or metadata- based favoritism while maintaining task coverage. See Appendix [ for benchmark details and evaluation metries. This section showcases the potential of the method and outlines how it can help.
We this model because its smaller size makes training tractable modest hardware.
8

---

Published as a conference paper at ICLR 2026
Table 2: Subset-selection performance for Qwen3 (14B) (overall and by ground-truth set size K).
By ground-truth set size K
Overall
K
Precision
Recall
Exact Set Match
Micro-Precision Micro-Recke
0.99641
2
300
1.0000
0.7717
0.5433
0.8856
3
200
0.9925
0.8850
0.7350
Exact Set Match Rate
0.6900
4
300
0.9940
0.9633
0.9100
200
1.0000
0.8610
0.5350
Befure Emniil Validation
After
Addrees Coordinates
Mulsi- Weather Forecast
0.5
Solction Hare
0.5
Gunnle Beneo
Gelleale
RraWWrr Gooad...
Adians Nontad...
Valican Stnsl
stad Walctrr
Laail Ventar
Drod Yabiln...
Emaall Elack Clacker
Weetbe oo
Whather
Gedlo
Figure 6: Selection distributions for GPT-4.1 mini with and without utilizing our mitigation method across three clusters. Each subplot corresponds one cluster, with the x-axis indicating the API in the cluster and the y-axis showing the fraction of times that API was chosen using the corresponding setup over 500 inference runs.
Subset selection avoids spurious inclusions of incorrect APIs while maintaining high coverage of correct APIs, thereby mitigating bias without sacrificing performance. Table 2 summarizes results for Qwen3-14B as the subset selector. Overall micro-precision is ~1000 (0.9964), meaning the selector almost never adds tools to the candidate set that cannot solve the user's task. This a desirable property as means that our mitigation module unlikely to output an incorrect tool, and therefore, performance will not suffer. Note that since all ground-trutt -trth set size classes have nearly the same number of queries, micro- and macro-prec precision are effectively equivalent; report micro-precision for simplicity. Micro-recall is ~0.89 (0.8856), so on average the selector itself not biased and retains most ground-truth tools, with an exact set match 0.69 across all instances. Broken out by the size of the ground-truth set K, precision remains essentially perfect across the board, while recall varies: it is strongest at K=4 (0.9633) and somewhat lower K=2 (0.7717) and K=5 (0.8610). The corresponding exact-match rates reflect the same pattern (notably 0,9100 at K=4). In practice, this means the subset filter very rarely introduces distracting tools (good for performance), but it can occasionally omit a true option when K small or large. Taken together, these results suggest subset selection is promising first line of defense: it does not lead to spurious inclusions (high precision) and maintains high coverage of correct APIs on average, and thus, less bias downstream (strong recall).
macro-precision
Our mitigation method successfully alleviates tool-selection bias. Figure the empirical selection distributions of GPT-4.1 mini before and after applying our mitigation method over three clusters of functionally equivalent APIs. Where choices were far from uniform before, after the mitigation method is applied notice even spread of selection share. However, in the weather forecasting cluster (right), we see that one API is still being underutilized, perhaps being an indica- tion that the model has difficult time using this API correctly even when it is the only API available. The effectiveness of the mitigation method is further exemplified in Table where a steep decrease in all our bias metrics after applying the mitigation method can be observed.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we introduced the first benchmark for evaluating tool selection bias in LLMs. Our results establish tool selection bias as real and potentially significant issue for tool-augmented (agentic) LLMs, with implications for user experience, operational cost, and marketplace fairness. This study offers a concrete starting point for understanding and mitigating this bias.
This study is limited by using 100 synthetically generated user queries cluster and narrow seven-feature set for selection-rad predictions, limited model/repeat coverage, and a focus on APIs from PapidAPI and English queries. Although our choices were constrained by compute (our setup already required ~500,000 inference runs), these factors may induce variance and restrict generality.
Limitations:
Future Work: Future work should scale queries and clusters and enrich features with deeper se- mantic and structural signals to raise predictive power beyond the modest observed It², In addition, deploying more expressive models (e.g.. boosted trees or deep nets) with cross-validation could capture higher-order interactions between tool features, further increasing their explanatory power.
We also believe it would be valuable to isolate the contribution of preference tuning RLHF) to tool-selection bias, since this might implicitly reward particular writing styles phrasings in tool names/descriptions and thereby skew selection among otherwise equivalent tools. Finally, richer analysis of models' internal decision processes could strengthen interpretability: while now al- low a single reasoning step before tool choice, preliminary inspection suggests models often do not explicitly justify why a specific tool selected. Allowing longer reasoning traces and conducting more systematic chain-of-thought analysis may yield clearer causal explanations of selection behav- ior, but leave this for future work due computational constraints. Lastly, broader replication across LLMs and runs would aid in quantifying variability.
ACKNOWLEDGMENTS
JY acknowledges support from Microsoft Ltd. AB and PT acknowledge the 2025 UK AISI Sys- temic Safety Grant and the UKRI Turing AI Fellowship (EP/W002981/1), AB, JY and PT are also affiliated with the Institute for Decentralized AI, which is supported by an AI Safety Fund grant. This work is supported by the European commissions' DVPS project. This work supported by the Schmidt AI 2050.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research did not involve identifiable human data or animals and therefore did not require ap- proval from an institutional ethics committee or review board. All experiments are conducted for scientific purposes only. The work does not involve or target any sensitive attributes such as gender, race, nationality, or skin color. Our study focuses on identifying and minimizing cool-selection bias in LLM agents, with the aim of improving the trustworthiness and safety of the deployment of LLM agents.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our work. We provide detailed descrip- tions of data and experimental setup in Section Our code is available as alpublic GitHub repository to facilitate replication.
Additional note. During initial experiments we observed unusually large standard deviation in DeepSeek's selection distributions for certain clusters. When we re-ran the experiments under the same configuration, this anomaly disappeared. Since we use the generic deepseex-chat endpoint for our LLM calls, which always serves the latest model version, one of the later runs was executed after silent model update and thus, ran with different model. All DeepSeek results reported in the current version the paper are based on a fresh set of runs collected using the same model, which removes the anomalous error bars.
REFERENCES
Anthropic. Meet Claude Anthropic, 2025, URL htt.tpp:/www.aaarrppic..oooo
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mci Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 166099 2023.
Dylan Bouchard. An actionable framework for assessing bias and fairness large language model use cases. arXiv preprint 10800 2024.
Isha Chaudhary, Qian Hu, Manoj Kumar, Morteza Ziyadi, Rahul Gupta, and Gagandeep Singh. Certifying counterfactual bias in lmms. ICLR, 2025.
Sadia Sultana Chowa, Riasad Alvi, Subhey Sadi Rahman, Md Abdur Rahman, Mohaimenul Azam Khan Raiaan, Md Rafqul Islam, Mukhtar Hussain, and Sami Azam. From language to action: A review of large language models as autonomous agents and tool users. arXiv preprint arXiv:2508. 17281, 2025.
Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. Bias and unfairness in information retrieval systems: New challenges in the Ilm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024.
DeepSeek-AI. AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang,
11

---

Published as a conference paper at ICLR 2026
Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. Xiv preprint arXiv:2501.12948 2025.
Economize. AWS API Gateway Pricing and 5 Strategies for Cost Reduction, 2024. URL https: 7/ //www. wcw.omize.. cloud/blog aws www.s-aa--ggaeeaa-ppriiiioo
Kazem Faghih, Wenxiao Wang, Yize Cheng, Siddhant Bharti, Gaurang Sriramanan, Sriram Bala- subramanian, Parsa Hosseini, and Soheil Feizi. Gaming tool preferences in agentic llms. arXiv preprint arXiv:2505. 18135, 2025.
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Der- noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Comput. Linguistics, 2024.
Google. Introducing Gemini: largest and most capable Al model. 2023.
URL https://
-ai/
Grand
API Marketplace Market (2025 2030),
2025.
URL
api agdimmarkeetplaccemmrrr------eee
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR, 2022.
Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. Instructed bias: Instruction- tuned language models exhibit emergent cognitive bias. arXiv preprint arXiv:2308.0022 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, 2019,
Jiaxu Lou and Yifan Sun. Anchoring bias in large language models: An experimental study, arXiv preprint ar arXiv:2412.0653 2024.
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilma Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello. Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona,
12

---

Published as a conference paper at ICLR 2026
Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K. Bizley, Sebas- tian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N. Apurva Ratan Murty, Leyla Souud-Khenissii Anna Behler, Chloe M. Hall, Jessica Dafilon, Sherry Dongqi Bao, and Bradley C. Love. Large language models surpass human experts in predicting neuro- science results. Nature Human Behaviour, 2024.
Kanghua Mo, Li Hu, Yucheng Long, and Zhihao Li. Attractive metadata attack: Inducing Ilm agents to invoke malicious tools. In Advances in Neural Information Processing Systems (NeurlPS), 2025.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435, 2024.
OpenAl. ChatGPT, 2025. URL https:// com httpaaaa.c..ccccatt///ererree/ee
Daniel E. O'Leary. Confirmation and specificity biases in large language models: An explorative
study, IEEE Intelligent Systems, 2025.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of op- tions in multiple-choice questions. NAACL, 2024.
Yujia Qin, Shihao Liang. Yining Ye, Kunlun Zhu, Lan Yan. Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. Tool learning with large language models: a survey. Frontiers of Computer Science, 2025.
RapidAPI, Monetizing your API on Rapidapi.com, 2025a. URL/ttts://Bccc..ap..a
com/ comm/docs//moetiiziie youhgggyourapppp -app-----a-------aapapp-p-------
RapidAPI. Nokia acquires Rapid technology and team!, 2025b. URL https://rap.dapi: com/
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 2021.
Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Prompt injection attack to tool selection in IIm agents. arXiv preprint arXiv:2504. 19793, 2025.
Ziqi Wang et al. Eliminating position bias of language models: A mechanistic approach. ICLR, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022.
Sheng-Lun Wei al. Unveiling selection biases: Exploring order and token sensitivity in large language models. ACL, 2024.
Jehyeok Yeon, Isha Chaudhary, and Gagandeep Singh. Quantifying distributional robustness of agentic tool-selection. arXiv preprint arXiv:2510.03992, 2025.
Rupeng Zhang, Haowei Wang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang. From allies to adversaries: Manipulating Ilm tool-calling through adversarial injection. NAACL (Long Papers), 2025a.
13

---

Published as a conference paper at ICLR 2026
Xiaoyu Zhang, Juan Zhai, Shiqing Ma. Qingshuang Bao, Weipeng Jiang, Qian Wang, Chao Shen, and Yang Liu. The invisible hand: Unveiling provider bias in large language models for code generation. Association for Computational Linguistics ACL 2025, 2025b.
Chujie Zheng et al. Large language models are not robust multiple choice selectors. ICLR, 2024.
Caleb Ziems, William Held, Jane Dwivedi- Yu, and Diyi Yang. Measuring and addressing indexical bias in information retrieval. In Findings of the Association for Computational Linguistics ACL 2024, 2024.
14

---

Published as a conference paper at ICLR 2026
A
MORE DETAILS ON DATA GENERATION
Equivalent
world www.tweweeeeeteeeeeeeeeeeeeeeeee wwwine_iieee weatherapi com weather forecast_ 1_ysas
What's the weather Parie"" the soadher
R Rapid RapidAPI 16000 APIs
Generate Queries
Benchmark Models
Cluater
Neaeures
codo generator apl codo generator V2
my "Create website
Figure 7: An overview of clustering and query generation pipeline.
We build the "tool-usage" evaluation pipeline introduced by Qin al. (0224) hereafter referred to as ToolLLM, owing to its wide adoption and extensibility. At its core, TooILLM assembles large catalog of real-world APIs scraped from RapidAPI spanning 49 functional categories (RapidAPI, 2025b). For each API, TooILLM provides JSON file containing the API's human-readable name, detailed description, and full parameter schema. this work, we leverage exactly that API repository but restrict our attention to the stages in which TooILLM selects among short list of retrieved candidates. Note that in TooILLM, set of closely-related APIs called a 'tool'. For example, geocoding tool could offer both forward geocoding and reverse geocoding API
We assemble benchmark in two stages: (1) clustering APIs into functionally equivalent groups, and (2) generating realistic user queries for each group (see Figure 7
Algorithm Generation Functionally-Eqqiivaeen API Clusters
Require:
API id to metadata map x Precomputed embeddings E,
List "general" APIs G = {(tool, tool_desc, api namen api_dese)); neighbor count K, max outlier loops R
1: C+0
2: for all (tool, tooldese, api_name, api_desc) E G do
3: Construct query text "tool: tool: desc api name: api desc
4:
Embed with ADA: Vg <- Embed(q)
5:
Compute cosine similarities: 8; +- cos(v,, E,) Vi
6: Select top-K unique tools with largest similarity and store in set TOP candidate +- [T[i] E T
7:
8:
for +- 1 to do
Prompt GPT-4 to detect outliers: outliers +- DetecctOuttirss(cnndidtt if outliers 0 then break 11: end if
9:
Remove outliers from candidate: candidate candidate outliers 13: end for
14: [candidate] > then
15: C C {candidate)
16: end
17:
end for
18:
return
Geocoding is the process of converting between "forward" geocoding maps address (e.g., "1600 Amphitheatre Parkway") to its latitude/longiudee, while "reverse" gencading maps given coordinate pair back structured postal address.
addresses and geographic coordinates:
15

---

Published as a conference paper at ICLR 2026
You are prempt-writing assistant. will give you a set of API endpoints (tool name description, endpoint name + description, and potentially the required parameters) that all perform the same underlying task. Please generate exactly (n) distinct, natural- language user queries that could be satisfied by ALL of these endpoints. *IInclude realistic sample values** for any required parameters (e.g. use "https::/exxmple. World" for a text field). Return them as JSON array of strings, with extra commentary.
for a URL, or "Hello
Jser:
Here are the endpoints:
Tool: WeatherNow Provides current weather information
Kndpcint: Current weather Returns temperature, humidity, and conditions for a given location.
Required parameters:
city (string) name of the city ccuntry (string) ISO country code
Figure 8: Example prompt used for query generation. The model outputs n natural-language queries that all listed endpoints can satisfy.
API clustering. We begin by embedding every endpoint's metadata (tool name, API name, descriptions, etc.) into shared vector space using pre-trained text wtttt-embediingadd----te-a
encoder (OpenAI's model). We then curate small set of "seed" APIs whose de- scriptions span a number of "general" tasks, such as text translation or weather forecasting. For each seed, we retrieve its top- K nearest neighbors in embedding space to form candidate cluster. To ensure true functional equivalence, we iteratively prompt GPT-4 to flag any outlier endpoints that cannot perform the same task as the rest; flagged APIs removed and the check repeats for up to pre-defined number of rounds. Any cluster that stabilizes with more than three members is retained. See Algorithm for overview of our clustering approach. Lastly, manually inspect and refine these clusters, yielding 10 high-quality groups of five APIs each.
Query generation. For each cluster, we prompt GPT-4 (see Figure queries that all members can satisfy. In batches of ten, the model generates candidate queries until collect 100 unique queries per cluster, filtering out duplicates. In cases where freeform gener- ation exhibits provider-specific bias (e.g., mentioning a particular vendor's feature), we switch to template-filling workflow: design small of generic templates with placeholders (e.g. "Get the latest news headlines for {country) } about ftopic}.") "), and ask GPT-4 to instantiate each template multiple times with realistic sample values.
produce natural-language
Final curation. All 1,000 generated queries are then reviewed by hand to remove any that inadver- tently favor a single provider or rely on specialized parameters. The resulting dataset consists of 10 clusters with 5 APIs each, and 100 balanced, provider-agnostic queries for each cluster.
Running each model over these prompts yields empirical selection distributions over APIs and list positions, from which compute our vilal-varition--aaee bias metrics ÒAPI, 6pos. and moddll This rigorously grounded benchmark enables precise measurement and comparison of tool-selection bias across models and settings.
B ATTRIBUTE-LEVEL ANALYSIS FEATURE TABLE
See Table for the list of features used in the analysis of Section 4.3
C MORE DETAILS ON THE BIASED CPT EXPERIMENT
We test whether pre- mee-trinnng data can cause tool-selection bias by doing biased continued pre- training (CPT) on single model. That is, we do additional next-token training text using
16

---

Published as a conference paper at ICLR 2026
Table API-level predictor features.
Feature
Description
avg simi cimilarity tool desc
Mean text similarity between cluster queries and the tool's de- scription.
avg.similarrty..pp..e.
Mean text similarity between cluster queries and each API's description.
age_days
Days since the API was first published.
desc.name.lengthe
Total character count of the API's name plus description. Number of required and optional parameters.
num_paran
flesch reading ease
Flesch reading-ease score of the combined descriptions. Count of positive or promotional words (e.g. "efficient," "ro- bust").
~3.5 tokens deliberately saturated with one endpoint's metadata (its name, description, and parameter info). After this exposure, we re-run the selection tasks and measure shifts in that endpoint's selection share.
To generate the biased corpus, we synthesize long-form prose with an external LLM (Gem- ini 2.5 Flash). We prompt it to produce single document of roughly 1.1-1.3k words written in randomly sampled style (c.g., blog note, Q&A memo, release note, how-to guide, troubleshooting checklist). The prompt requires frequent natural mentions of the target API name, the exact or faithfully paraphrased tool description, and the exact or paraphrased API deseription; it also requests inclusion of the endpoint path in about 60% of documents and parameter metadata in about 50%. This pipeline yields a large, stylistically varied corpus that is nevertheless saturated with the same endpoint's metadata.
We then run CPT on the same base model used elsewhere (i.e., Qwen3-8B) with parameter- efficient adapters (LoRA) attached (see Hu et al. (2022) for more details on LoRA). We keep tokenizer unchanged.
We evaluate pre/post CPT selection distributions on the original cluster, prompts, and infer- ence settings. The primary outcome is the shift in the target API's selection share. We also measure spillover: changes in the selection shares non-target APIs within the cluster. If biased CPT reliably increases the target API's selection share, this is evidence that portion of tool-selection bias originates from pre-training exposure.
D DETAILS ON THE IMPLEMENTATION AND EVALUATION OF THE MITIGATION METHOD
After showing the existence and possible causes of bias, seek to mitigate it. We pursue simple approach based on the following insight: Models often know which APIs can solve a task but can possibly exhibit biased choices among interchangeable endpoints. We decouple capability recognition from final selection via a lightweight debiasing module.
The debiasing module consists of a lightweight LLM (Qwen3 14B in our case) prompted to output the subset of APIs from the given candidate list that can solve the task given in the query. This way, we get a subset selector that outputs an array of the APIs that can complete the task. The system prompt constrains the output to an exact list with no prose. From the returned set S, we pick one API uniformly at random. This API then replaces the original API list and is used for the rest of the tool-usage pipeline.
this approach is successful, each API in S has an expected selection share 1/|S|, elimi- nating position/API favoritism at the choice stage. If the selector's true positive/negative rates high enough, the overall selection distribution approaches uniform even when original models were skewed. This, following definition, means the tool selection stage becomes unbiased by design.
17

---

Published as a conference paper at ICLR 2026
6.31
0.3
6.30
6.00
0.00
073
100
1.5
1.00
1.5
075
$113
155
(a) Temp. (GPT-3.5)
(b) Top-p (GPT-3.5)
(c) Model size (Qwen 3)
Figure 9: Sensitivity of the combined bias metric moddel model hyperparameters. Each point is mean over three independent runs (except for the top-p subplot); vertical bars show one standard deviation where available.
To evaluate this approach, we build a 1000-query benchmark with 8 API candidates per query and ground truth set indicating which K (2, 4, APIs sufficient (~250 items each). We report subset quality: precision, recall, and exact-set match. Note that the formula for exact-set match is given by EE= 1[S Gy] where G denotes the set of ground truth sets, S the set of selected subsets, and N the number of queries. Additionally note that bias can persist if the subset selector itself is biased; underselecting viable tools (false negatives) selecting unrelevant ones (false positives). Measures of recall and precision will tell us whether this is the case.
E MORE ELABORATE ABLATION AND SENSITIVITY ANALYSIS
Temperature. Raising temperature reduces combined bias. As shown in Figure Pa temperature goes from to 2, the mean &model for GPT-3.5 drops from about 0.350 to 0.285, a 6.5% absolute reduction. Figure [0 makes clear why: the overall selection patterns remain similar across temperatures, but higher temperatures soften extreme preferences. This suggests that increased stochasticity slightly mitigates bias, but does not eliminate it.
Top-p. Figure 9b shows how the combined bias Smodel for GPT-3.5 varies with the top-p cutoff. Increasing top-p from 0.7 to 1.0 yields small decrease in bias (from ~0.346 to ~0.340), suggesting that less aggressive truncation of the probability distribution slightly softens extreme tool preferences. The effect is noticeably weaker than the temperature change.
Model Size. In Figure 9c the combined bias dmodel is depicted for Qwen 3 with varying model size. It seems that larger models exhibit less bias, with a notable drop at 32B. This pattern suggests that larger models develop nuanced selection mechanisms which temper extreme preferences for certain APIs.
APT Ordering. Figure compares GPT-3.5's API selection under two different ordering schemes: cyclic rotations versus random permutations. Across all clusters, the choice distribution very similar: no API's selection rate shifts more than about ten percentage points. This indicates that the ordering of the APIs has some influence, but the dominant signal is the model's 's intrinsic preference. However, the small differences could also reflect the inherent noise from stochastic token sampling. and overall we argue that the tool-selection behavior is robust to either type of shuffling.
System Prompts. To evaluate how sensitive tool selection is to the phrasing and structure of the instructions given, we compare three variants of the system prompt: the original "Base" prompt, lightly reworded "Similar" prompt, and structurally different "Adjusted" prompt. Figure shows the resulting distributions for GPT-3.5.
Prompt wording shifts model preferences but does not remove bias. Reworded prompts amplify dominant choices and in some cases radically redistribute the selection shares. Elsewhere, effects are modest. Overall, framing and formatting can tilt the implicit ranking among functionally equivalent APIs, indicating that part of the observed bias is prompt-dependent even as a tendency to favor a subset of tools remains.
18

---

Published as a conference paper at ICLR 2026
Address Coodinates
Counlinates Address
Nows Headlines Region
Addros Grolocaton
S Btat
05
Per
Produce
Histury
Anabiis
Language
Sihetan Ban
APP
Renso
todke
Asar Slait.
Addl
Recload
AMd
Reselu
QR Code Geneation
Moltò Di Wodher Forectot
Sk Gear
Mocke
Figure 10: Selection distributions for GPT-3.5 using four different temperatures across ten clusters of functionally equivalent APIs. Each subplot corresponds to cluster, with the x-axis indicating the API in the cluster and the y-axis showing the fraction times that API was chosen by the respective model over 500 runs.
19

---

Published as a conference paper at ICLR 2026
Address Coadinates
Counlinates Adlress
News Headlioes Region
Address Geolocaton
05
S
Per
Histury
Anabais
Langugge
Schetan Bat
APP
Eodde
Budtbete
Avd
Anst
QR Code Geneation
Molts Day Worter Forectot
de
Mater
Cal
Figure 11: Selection distributions for GPT-3.5 using cyclic and random shuffling of APIs across ten representative clusters. Each subplot corresponds to cluster, with the x-axis indicating the API the cluster and the y-axis showing the fraction of times that API was chosen for that specific API ordering method over 500 runs.
MORE ELABORATE DISCUSSION ON THE EXPLANATION OF BIAS
We now expand on the analysis given in the main text surrounding the investigation of bias. We expand on the feature-level analysis, where we try to predict selection rates according intrinsic API attributes, and the perturbation experiments that directly intervene the API metadata to see which cues the models rely on during selection.
F.1 WHICH API-LEVEL FEATURES PREDICT SELECTION RATES?
We extract a common set of descriptive features from every API (see Section 3.3) and mean-center them investigate how being relatively high or low feature affects the API selection. These then paired with the empirical selection rates yielding a dataset of 50 examples for each L.M.. We then probe relationships between features and selection behavior in three ways. First, compute Pearson correlations to capture linear and monotonic associations. Second, we fit linear regression per model quantify the aggregate explanatory power (reported R²²) and inspect coefficients to understand the direction and relative weight of each feature. Third, we train random- forest regressors with validation allow for non-linear interactions and obtain alternative measures of feature importance.
Similarity between tool API description and query most correlated to selection rate. As Table X makes clear, the most predictive feature of API selection is semantic similar- ity between the query and the tool/API descriptions. Both avg-simil wtw.iimmaatttttoooooooooooooo and avg.simil wwr.lllrrittteeeeee
are consistently positively correlated with selection rates especialyy strong for Qwen and clear for Gemini; GPT shows the same pattern, albeit weaker with higher p-values. By contrast, structural stylistic attributes (e.g., parameter count, promo- tional wording) exhibit little consistent influence. Tool age (age days) shows a modest, broadly consistent negative correlation across models.
20

---

Published as a conference paper at ICLR 2026
Address Coodinates
Counlinttrs Address
News Headlines Region
Addres Grolocatin
S Buar
055
Girea
Geoad Per
Produie
Histury
Anabiss
Language
Schetin Ban
APP
bode
Asar plai..
Addl
QR Code Goneation
Molts Day Wodher Foreetot
Marke
Figure 12: Selection distributions for GPT-3.5 using different system prompts across ten clusters. Each subplot corresponds to one cluster, with the x-axis indicating the API in the cluster and the y-axis showing the fraction times that API was chosen using the corresponding system prompt over 500 runs.
21

---

Published as a conference paper at ICLR 2026
Table Correlation between API-level features and model selection rates. Each entry shows Pear- son with its p-value.
Feature
GPT-4.11 mini
Gemini
Qwen
avg.similarity..too..eee
(p)
-0.092(==0526)
0.2344p=0.10)
avg sey.simillritt... api desc
+0.11 (p=0.442)
+0.330kpm00119)
+0.41 (pm0.0003
age.days
-0.199(p=02201)
-0. (0.144(p=0.356)
-0. p0.26)))00296)
desc.name.lergth..sr
+0.044tp=07700
+0.103(g=0:477)
+0.038(p=07795)
num params
-0.065(p=0.553)
+0.038(p=0.995)
-0.185xp=0.198)
flesch.readdigg.eas positive.word.count
+0.160(p=0.267)
+0.176kp=0.22))
+0.098(p=0-996)
+0.126(p=0.884)
+0.087(p=0.547)
p0.093(=00.521)
Linear regression reveals that surface-level semantic alignment the primary signal but leaves lot unexplained. Figure 13 tells us that linear models explain only part of the variance: R modest-0.143 for GPT-4. I mini and 0.387 ToolLLaMA-leaving substantial error. Coeffi- cients show surface-level semantic alignment dominates: similarity between the query and tool/API descriptions has the largest positive weights for most models, with Qwen weighting both most strongly and Gemini emphasizing API-level descriptions. Unexpectedly, TooILLaMA gives neg- ative weight to tool-description similarity. Other features contribute little. Hence, semantic align- ment is important in driving selection but still gives incomplete explanation, implying nonlinear or omitted factors and motivating more flexible models (e.g., random forests).
CharPP (.222)
0.120)
0.226)
CharGPT 0.145)
DeepSrok
TooILaMMA
Linoa tngresscoo exeffitintt
ang simllatty dood dese
sinnlaatr app dess
daya
detc name longth simm
Hitm Rme
Beach roaling uase
positive word count
Figure 13: Linear regression feature weights used to predict API selection rates for six LLMs. Each group of bars corresponds to one API-level feature; different colors denote models, with their R² shown in the legend. Larger positive weights indicate features that increase the predicted selection rate.
Random forests fail to offer meaningful improvement. We fitted cadom-foress regressors with the same mean-centered features using both cross-validation and a held-out split, but predictive per- formance was poor, meaning the forests often did worse than a trivial constant baseline. This sug- gests the available features, at least in their current form and scale, don't contain enough signal or that some artifacts are overwhelming the gains from nonlinearity. Therefore, any ceature-importane estimates from these trees would be unreliable and we do not lean them for explanation. Future work could involve revisiting this with a richer feature set, more data, or alternative nonlinear mod- eling.
22

---

Published as a conference paper at ICLR 2026
F.2
How DO METADATA INTERVENTIONS AFFECT API SELECTION?
We saw that corrupting descriptions produces much larger and more stable effects on selection behavior than name-level perturbations, which sometimes noisy and unpredictable, in Section Figure [4] corroborates this across clusters. Name perturbations often leave distributions near- unchanged can make them drift unpredictably (c.g., Cluster where Cluster 1 positioned the scrambles frequently over- haul rankings: sometimes amplifying the dominant API (Cluster 2), other times causing dramatic re-ordering (Clusters 3). Name edits rarely produce comparably stable re-ranking.
4.3
top-left and Cluster 6 at the bottom-right), whereas descriptioo/parrmeet
Together, these patterns indicate that description-leve semantics (and, lesser extent, parameter semantics) are the primary cues models use to discriminate among functionally similar APIs. Name perturbation alone tend to inject noise without consistent effects. Finally, bias persists under minimal semantic signal (only names and schema fields), implying selection behavior sometimes relies on residual, non-obvious priors rather than solely on coherent, human- inttannnnntrrrtaale heuristics.
Manipulating the description of certain tools has mixed effects across clusters. Figure 14 (lower row) shows three behaviors when we manipulate descriptions. First, swapping the most- and least- popular tools' descriptions alone can dominate choice. Second, the same swap sometimes yields only a modest lift for the least-popular tool while unexpectedly altering the selection shares of unaffected tools (Cluster 5), suggesting the landscape is reconfigured rather than ranks simply exchanged. Third, in some clus- ters the swap has minimal effect (Cluster 6), implying other sug.-c.g. name priors or parameter schemas-anchor preferences.
invert their selection rates (Cluster 4), indicating description text
Targeted corruption of the most-selected tool's description has similarly inconsistent effects. In Cluster 4, scrambling collapses its share to near zero another tool absorbs the mass; in Clusters 5-6, corruption diffuses probability across competitors, producing a more even allocation. Over- all, description tampering often wields substantial influence, but the impact is context-dependent: the same intervention can invert, redistribute, or barely change preferences, underscoring that tool choice emerges from multiple interacting cues.
G ADDITIONAL FIGURES
G.1 CORRELATION IN SELECTION BETWEEN MODELS
Figure 15 shows that models exhibit varying degrees of correlation in their tool-selection patterns, suggesting shared but non-identical biases across families.
G.2 SELECTION DISTRIBUTIONS FOR ALL CLUSTERS
This subsection provides full version the figure referenced in the main text (Figure 3 It expands the subset plot to all ten clusters and keeps axes, counts, and error-bar conventions identical to the summary in Section 4.2 Use Figure 16 for detailed inspection per-cluster behavior.
G.3 FULL FIGURE RELATED TO THE CPT EXPERIMENT
Figure 17 shows how biased continued pre-training gradually increases preference for the exposed endpoint.
G.4 TABLE RELATED TO THE REDUCTION OF BIAS DUE TO MITIGATION
Table demonstrates that our mitigation substantially flattens selection distributions and reduces both API- and losition-level bias.
H EFFECT OF METADATA PERTURBATION ON BIAS
Relative to the base distributions, both models move farther from uniform (get more biased) when lower semantic signal (see bars corresponding the description parameter and full
23

---

Published as a conference paper at ICLR 2026
Basr
Rand. Nuzar
Dosc Parsun
Tay. Dos:
Swap. Dooc
Address
Coordinates
Address
Geolocation
WHOIS History
1.0
0.8
Scection Bate
6.0
Genood Forw:
Galkento
WayrWWr Gracod.
Grokos Formadd
Addhas Narnall
IP Clsdbnation
Goobralion
Free IP Gedee
IP Addma Cro
IF Gen Locato
Douat WHOT I....
Whese by APEN.
Nealy Rodster
WHOSS x2
Whods
Email Validation
Language Identiication
Multi-Day Weasher Forecust
Seletios Rate
6.2
0.0
Validate Eaval
Emsil Validaaoe
Emall Foriler
Email Valded...
Exail Choske
What's Langunge
Quink Langungo
Todl Langrage
Translato og
Trandate All
Foncas
WastherP11
Windd Hosther
Wenther
wmilter Encern
Figure 14: Selection distributions for Gemini under different name/ordering perturbations across six clusters of functionally equivalent APIs. Each subplot corresponds to one cluster; the x-axis lists the APIs and the y-axis shows the fraction of times the model under each condition selected that API, averaged over repeated runs. Error bars (when present) indicate the standard deviation across those repeats, making visible how robust variable the preferences are under the different perturbations. Tools whose names are in green are the most selected by the baseline, and those in red are the least selected. These are the tools that are targeted for the swapping and selected scramble experiments.
Table 5: Average cluster-level API bias SAPI. positional bias 8poe. and combined bias Smodel before and after mitigation.
Setup
θAPI
8pos
dmodel
Before
0.338
0.422
0.380
After
0.108
0.079
0.094
24

---

Published as a conference paper at ICLR 2026
1.00
GPT
1.00
0.12
0.10
0.28
0.12
0.29
0.75
GPT 4.1
0.33
1.00
0.50
0.73
0.64
0.64
0.25
0..00
Clande
0.12
0.59
1.00
0.67
0.47
0.62
0.12
0.25
Pearson r
Gemini
0.10
0.71
0.000
DeepSeek
0.28
0.64
0.47
0.49
1.00
0.40
0.08
-0.25
-0.50
Qwen
0.12
0.6-1
0.62
0.74
0.40
1.00
0.13
-0.75
T-LLaMA
0.29
0.12
0.13
-1.00
GPT 3.5
GPT 4.1
Clande
Gomini
DrapSeok
Qwen
T-LLaMA
Figure 15: Pearson correlation matrix between models' tool-selection bias patterns.
For Gemini, these manipulations yield the largest TV distances to uniform (x:0,42-0:33; GPT shows similar results. This suggests that when descriptions/harameters are corrupted, models amplify bias rather than flatten choices.
perturbations in Figure 18
Conversely, targeted edits to the most popular API tends to decrease bias. For Gemini, tar- geting the description of the most popular API leads to an average TVD slightly below the baseline and swapping the description between most- and least popular API leads to one that is substantially lower, indicating that weakening or transferring the strongest semantic cue moves the selection distribution toward uniform. Name-only manipulations have similar effects, but name scrambling does not increase bias as much.
I DETAILS ON CPT SETUP
Model and adapters. continue pre-training Qwen3-8B-Base using Unsloth with 4-bit load- ing. The maximum sequence length is 2048. We attach LoRA adapters with 16..29% trainable parameters. See Table for more info.
Table 6: Model/adapter configuration.
Base model
Max seq. length
2048
Quantization
4-bit (bitsandbytes)
LoRA hyperparameters
128, 32, dropout= 0
25

---

Published as a conference paper at ICLR 2026
GPT ved
Carde
Deçiees
Quee (3070)
Tup Regio
1.0
Selecton Bate
0.8
0.4
A0
Aiben
Domain History
Fmail Validation
Sentinent Analesis
Language Mettifration
1.0
Soledka Bate
86
bg
Rus todk
Reseo
QB Code Generation
Molti-ip Warthrr Foreentt
Selecion Rate
0.6
m0
For
Figure 16: Selection distributions for six LLMs across ten clusters functionally equivalent APIs. Each subplot corresponds one cluster, with the x-axis indicating the API in the cluster and the y- axis showing the (mean) fraction of times each model chose that API over 500 inference runs: error bars indicate the standard deviation across three independent experimental runs. This visualization highlights how different models exhibit systematic preferences for some APIs.
26

---

Published as a conference paper at ICLR 2026
Base Model
CPT Model (1/3 epoch)
Solettan Itate
000
00
What's Lanqugee
Quick Language Detectorr
Tes Luagrage by APl-Ninjes
Thandatee si
Tanelete All Linguues
What's is Languag
Quick Language Detectar
Tuat Langinge by API-Nigle
Thamlea ail
Tharlate All Langonge
CPT Model (2/3 spoch)
CPT Madel (1 opoch)
Salecion Kale
0.6
0.228
0.0
What's Laqquee
Quek Language Detecoir
Tea Lauguage by API-Nings
Thandateedd oi
Tomnlite All Languans
What's Longuuge
Quick Language Detectir
That Laurunge by API-Naal
Thamlanee et
Tomodate All Langongs
Figure 17: Selection rates for the Language Identification cluster across biased continued pre- training (CPT) checkpoints. Bars give the fraction that each endpoint was chosen across 500 in- ference runs. Panels show (top-left) base model, (top-right) CPT after 1/3 epoch, (bottom-left) 2/3 epoch, and (bottom-right) full epoch. The Text Language by API-Ninjas endpoint is highlighted, with its exact selection rate printed above its bar. Differences across panels visualize how biased CPT shifts tool choice over training.
Mon TV destane Unitom
hdil
Pheso
Tamg Das
Dee
Red.
Mott
Taae:
Sepp
Figure 18: Mean ootalvvriattins (TV) distance from the uniform selection distribution the distri- bution pertaining to each metadata perturbation (higher more deviation from uniform). Blue bars show Gemini and orange bars show GPT; error bars denote standard deviation across clusters, and single bars indicate perturbations not run for GPT. Dashed horizontal lines (in the corresponding model colors) mark each model's baseline TV-to-uniforn without perturbations.
Data. We use our corpus saturated with metadata of a single target endpoint (Text Language by API-Ninjas). The corpus contains ~3.5M tokens.
27

---

Published as a conference paper at ICLR 2026
Training.
Training uses Unsloth's trainer for one epoch with cosine LR scheduler and warmup. Optimizer 8-bit AdamW. We also set smaller LR for the embedding modules. See Table for more info.
Table 7: CPT training hyperparameters.
Epochs
Total steps (epoch)
153
Per-device batch size
Grad. accumulation
8
Effective batch size
Learning rate
5 10 (embeddings 5 10 6)
Scheduler Warmup Optimizer
cosine warmup ratio 0.1
adamw_8bit
Weight decay
0.0 step (base), 52 (R/1/3), 104 (R=2/3), 153 (1 epoch)
Checkpoints used
Evaluation (inference). For all checkpoints, we keep prompts and decoding fixed: temperature 0.5, top-p 1.0, and max_new. www.newwtttt new.tokens=512 We evaluate with the Language Identification cluster under circular shifts, aggregating the selection rates over 500 inference runs per checkpoint,
J EFFECT OF TOOL COUNT ON BIAS
To investigate how the number of available tools affects selection bias, we conducted an additional experiment using the Sentiment Analysis cluster (which contains five functionally equivalent A APIs).
From this cluster, constructed subsets of size K {2,3, 3, 4) using three selection strategies:
Best-to-worst: select the K APIs that were most frequently chosen in our initial K = experiments.
Worst-to-best: select the K least frequently chosen APIs
Random subsets: uniformly sample K tools from the cluster.
For each subset configuration, we re-ran the corresponding queries three times using Qwen3 (235B)
and computed the normalized API-level and positional bias metrics.
Weekeister
Jandees vierr
Awverge 250)
Figure 19: Normalized API-level and positional bias as function of cluster size K for the cuntiment-analysii cluster. Each curve corresponds to one subset-sclection strategy (best-to-worst, worst-to-best random), with error bars indicating variability across three independent runs. Bias is reported as percentage its theoretical maximum (1 1/K). The rightmost subplot shows the average of the normalized API-level and positional bias over K.
As can seen in Figure 19 we find that bias is highly sensitive to the specific subset of tools shown to the model. At K 2, both normalized ŌAPi and 6pos vary substantially across selection strate- gies, revealing strong instability. At moderate subset sizes (K = 3, 4), the variance decreases and the measured bias becomes more stable, although not uniformly smaller. Overall, the relationship
28

---

Published as a conference paper at ICLR 2026
between K and bias is non monotoni:::: depending which tools included, reducing the number of available tools can either amplify or attenuate bias. These findings indicate that the composition of the toolset is the primary driver of selection behavior, more than its size alone.
Finally, note that when K = 5 all methods trivially converge, since they expose the full cluster and therefore share identical toolsets. To fully characterize tool-count effects, future work should extend this analysis to larger clusters (e.g., K 6-10).
K EXPLANATION OF LLM TOOL USAGE PIPELINE
User Query (e.g. What's the woother in Pans?)
Final Response
Available tools: e.g. world oniine_an apl, wealherapp weather days
GETIPOST
Reskwerr
Loop urd al nacessary information
acquired
Figure 20: The overall workflow for tool usage with large language models. This figure illustrates the last three stages of tool usage: tool selection (via retriever and LLM selection), tool calling, and response generation (task planning is omitted). Note how the LLM's preliminary outputs need to be parsed to create a functional API call.
Large language models equipped with external tools typically follow a multi-stage decision and execution process. As summarized by prior work (Qu al. 2025). the standard tool-usage pipeline consists of four stages: task planning, tool selection, tool calling, and response generation. Figure 20 showcases the latter three of these stages.
In the first stage, the LLM interprets the user query, identifies the user's intent, and (if necessary) decomposes it into sub-tasks that can be handled by external tools. Next, in the tool selection stage, the model determines which tools are suitable for each sub-task. Modern systems often employ a retriever-basee based filtering step before LLM-basee selection: rather than presenting the descriptions of hundreds or thousands of available tools to the model (which is the case with large API hubs like RapidAPI), a retriever identifies the top-k most relevant candidates. This design widely adopted in practical deployments due to context-length limitations (Qu et al. 2025).
In the tool calling stage, the LLM chooses one of the retrieved tools and generates the corresponding API call, including the required parameters in the correct format. This step requires the model not only to select an appropriate tool but also to extract and structure the tool arguments accurately. Once the L.M generation is parsed, the tool executes and returns its output, the I.M may decide to proceed to the final stage or determine that more tools need to be called. In the final stage, the LLM integrates the returned tool results into coherent final answer to the user.
L CONSEQUENCES OF TOOL SELECTION BIAS
We have briefly touched the consequences of tool selection bias in the introduction. In this section, we will further elaborate our points on why tool selection bias matters to give the reader comprehensive view of the the effects and significance of the bias uncover.
L. ECONOMIC CONSEQUENCES
Problem Statement. A pay-per-request pricing model is common practice within tool marketplaces (see RapidAPI's pricing model or that of BridgeAPI). This means that usage an API directly corresponds to the amount of revenue the developer of that API makes. Ideally, when two more tools offer identical functionality. usage (and thus revenue) should also be split equally. This is not achieved when LLMS show consistent bias among functionality equivalent tools. This is not
29

---

Published as a conference paper at ICLR 2026
hypothetical concern. As show in experiments, some tools gets selected up 10 times more than others whilst being functionally identical (see the geocoding cluster in Figure I6. revenue is directly correlated with usage, the developer of this tool can then also expect 10 times higher revenue than the developer of the disadvantaged tool for seemingly no particular reason but more advantageous metadata phrasing (see section 4.3 investigating why tool-selection bias occurs).
Magnitude. The size of the tool-usage market hard to determine exactly. RapidAPI alone handles over 9 billion requests per month and recommends $0.00003 per API call at a minimum, quick calculation gives us at least $270.000 in developer revenue per month or $3.24 million per year on this platform alone (RapidAPI (2025a)) Looking more broadly, the global API marketplace market size was estimated at $18.00 billion in 2024 and is projected to reach $49.45 billion by 2030, growing at a CAGR of 18.9% from 2025 to 2030 VieandVieRReseane
(2025). From this, one can see that even small shifts in automated traffic can have meaningful economic impact. It's unclear how much of this API traffic currently due LLM agents, but with the increasing ubiquity of agents, expect this share to rise significantly. This makes tool selection bias tangible and pressing economic issue.
L.2 USER EXPERIENCE
Tool-selection bias can directly degrade user experience when an LLM consistently favors an API that is objectively slower, less accurate, or more costly than its functionally equivalent alternatives. In such scenarios, end users may experience higher latency, lower-quality responses, or unnecessary costs, despite the existence of equally capable tools that would have delivered better performance. A more balanced usage distribution across equivalent APIs mitigates these issues by ensuring that no single suboptimal tool is disproportionately selected purely due to incidental metadata positional biases. This leads to more stable, predictable, and higher-quality user outcomes.
L.3 SAFETY & RELIABILITY
Biased tool selection magnifies the system's vulnerability manipulated metadata and adversarial tools. Recent work on metadata-poisoning attacks (Mo et al. (2025)) shows that adversaries can craft strategically misleading tool names or deseriptions lure LLM agents into invoking harmful or unreliable APIs. When an LLM already exhibits strong, unintended preferences toward superfi- cial metadata cues, such attacks become significantly easier execute. this sense, selection bias is not merely an efficiency problem. It increases the surface area for adversarial exploitation and directly undermines the reliability of agentic systems. A more uniform or semantics-invariant se- lection process would provide stronger baseline defense by reducing the influence of manipulable metadata.
L.4 EROSION OF TRUST & ECOSYSTEM EFFECTS
A further systemic consequence of tool-selection bias the gradual erosion of trust in API market- places. If developers observe that LLM-mediated traffic does not meaningfully reflect the functional quality of their tools but instead hinges on arbitrary metadata preferences. they may perceive the marketplace as unfair or unpredictable. This creates incentives to bypass marketplaces entirely by hard-coding specific APIs into their applications. This outcome undermines the value proposition of marketplaces as neutral, competitive intermediaries. In the long term, this can reduce innova- tion, further distort competition, and create fragile ecosystems where a small number of arbitrarily preferred tools dominate traffic. Addressing tool-selection bias is therefore critical not only for indi- vidual user or developer outcomes, but for maintaining trust, participation, and healthy competition in the broader API economy.
30


Reviews:
[{"review_id": "ghGxcMYrxY", "reviewer": "Reviewer_CTUZ", "summary": "This paper studies tool-selection bias in tool-augmented LLMs, showing that models often favor certain APIs for reasons unrelated to utility. The authors build a benchmark of functionally equivalent tools, measure bias across major LLMs, and find semantic metadata and training exposure as key drivers. They further demonstrate that controlled metadata changes shift model choices and propose a simple mitigation (i.e., filter relevant tools then uniformly sample), that effectively reduces bias without harming performance.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1, This paper studies tool selection bias in LLMs, which is a very new and meaningful topic. \n\n2, The authors build a clear benchmark with multiple tool clusters and balanced queries. The experiments are systematic across several strong LLMs, and the evaluation metrics are reasonable.", "weaknesses": "1, The paper does not probe whether LLMs' own familiarity/knowledge about each API explains its selection bias. Current analyses focus on metadata features, perturbations, and exposure effects, but never explored the model’s prior knowledge of each candidate API, so it’s unclear whether choices reflect surface cues or genuine familiarity.\n\n2, The paper fixes the number of candidate APIs, leaving unexplored how bias scales or changes when the number of tools varies (e.g., 2,3,4...). This is important to understand whether larger API pools amplify or dilute preference concentration is important for real-world marketplaces where tool pool size varies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wy9ET5RTo5", "reviewer": "Reviewer_kirZ", "summary": "This paper investigates the causes behind tool selection biases, where LLMs exhibit biases towards certain tools based on their descriptions or positions in the prompt even when the tools are equivalent in terms of their functionality. The authors also propose a simple mitigation strategy to alleviate tool selection biases, and empirically show that it can reduce tool selection bias.", "questions": "1. As selection bias is a well-studied issue of LLMs. What are the special issues in tool selection bias compared to selection bias in general that require a dedicated research? Elaborate more on this can strengthen the positioning of this paper.\n\n2. Are the bias due to the fact that the tools are simply better described and understandable by LLMs? From your experiment results that \"semantic similarity between queries and tool descriptions is the strongest predictor of selection\", another way to interpret that is the descriptions are better aligned with the user query, rather than interpreting it as \"bias\". I would like to hear the authors' comments on this.\n\n3. I also have a fundamental question about the setting in this paper. From a practical point of view, if the APIs are equivalent in their functions (\"functionally interchangeable\" as descsribed in line 160), we don't have to include all of them in the prompt every single time. If practitioners need to consider fairness or balance usage, they just have to uniformly sample a tool from the equivalent tools of each functional cluster and put them into the prompt for LLMs to choose from. Therefore, I am curious about how important and realistic the setting is.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. In general, LLMs' selection bias is an important yet unsolved challenge despite extensively studied in recent literature. Within this broader context, tool selection bias is a less studied subproblem that worth further investigations. This work aims to address this gap.\n\n2. Measuring bias of API itself (δ_API) and positional bias (δpos) separately is reasonable and disentangle the biases arising from these two different sources.\n\n3. The idea behind the mitigation strategy, which recognizes the subset of tools that can address the given query and then performs uniform sampling from the subset, is intuitive and easy to implement.\n\n4. The authors evaluate a variety of LLMs and API endpoints.", "weaknesses": "1. (Informativeness of the findings) Selection biases and their mitigation strategies have been studied extensively in previous literature. For example, positional biases are well-known in the research community (see the \"missing references\" below). While I acknowledge that there are special issues unique to \"tool\" selection bias, such as tool descriptions and metadata, I think the contributions of this work are relatively limited considering the presence of existing works.\n\n2. (Missing references of selection bias) Following the previous points, I believe these highly relevant yet missing references need to be cited in this paper:\n\n[1] Pezeshkpour, Pouya, and Estevam Hruschka. \"Large language models sensitivity to the order of options in multiple-choice questions.\" arXiv preprint arXiv:2308.11483 (2023). NAACL 2024 Findings\n\n[2] Zheng, Chujie, et al. \"Large language models are not robust multiple choice selectors.\" arXiv preprint arXiv:2309.03882 (2023). ICLR 2024 Spotlight\n\n[3] Wei, Sheng-Lun, et al. \"Unveiling selection biases: Exploring order and token sensitivity in large language models.\" arXiv preprint arXiv:2406.03009 (2024). ACL 2024 Findings\n\n[4] Wang, Ziqi, et al. \"Eliminating position bias of language models: A mechanistic approach.\" arXiv preprint arXiv:2407.01100 (2024). ICLR 2025 Poster", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kA940C3dpE", "reviewer": "Reviewer_hsat", "summary": "The paper exposes and addresses the problem of tool selection bias in LLM agent systems. It provides a benchmark, root cause analyses, and a simple bias mitigation approach. The results show bias with respect to an ideal scenario of uniform tool selection and the effectiveness of the mitigation as it directly achieves the target tool selection distribution by adopting a uniform distribution.", "questions": "1. What is your LLM decoding scheme to create the tool selection distribution?\n2. What is the intuition behind \"averaging\" the API/tool and position probability distribution distances to form $\\delta_{model}$?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. I appreciate the problem benchmarked by the work - tool selection bias. I think that this is an important problem. \n2. The work is comprehensive - with a benchmark, exploratory root cause analyses, and a bias mitigation strategy.\n3. The experimental setup is pretty comprehensive, covering multiple SOTA models and scenarios.", "weaknesses": "1. Methodology concerns:\n    1. How does the LLM know that the tools are functionally equivalent when only metadata is exposed? From the LLM's perspective, the tools are not equivalent, and if the language of the metadata is more appealing, then it seems natural for the LLM to pick that. If instead of LLMs, we ask humans, then even their responses are not expected to produce uniform distribution over the tools. Hence, I think comparing with uniform distribution as the target doesn't seem appropriate. I would suggest comparing with some distribution learned from human responses, to make the LLMs respond similar to unbiased humans.\n    2. To create the tool selection distributions from the LLM responses, I think it would be more appropriate to sample from LLM response multiple times.\n    3. The method doesn't account for initial tool retriever biases, like [1], thus providing a partial picture of the tool selection bias.\n    3. Other distance metrics, such as KL-divergence can be used between the LLM tool selection and uniform probability distributions.\n    4. The proposed defense is a finer-grained retriever followed by an uninformed uniform distribution sampler. I wonder how it would perform compared to the initial retriever itself which produces a smaller slate of size K, directly followed by the uniform sampler.\n    5. I think that RLHF preference training could be a more potential source for the tool selection bias than pretraining and would recommend studying that as well, alongside continual pretraining.\n2. Results:\n    1. Main results of the root cause analyses are expected and not novel. We can expect LLMs to exploit higher similarity between query and tool metadata, as they are after all doing pattern recognition.\n    2. I recommend analyzing the chain of thought of models when the bias is high, to see if the bias might be intentional.\n3. Related works: There are some missing comparisons to prior works, such as [2] (shows that LLMs can be manipulated to select specific adversarial tools) and [3] (shows LLM bias beyond benchmarking). Also, [1] goes beyond benchmarking for tool selection, and I wonder how a similar analysis can be used to expose tool selection biases more rigorously.\n\n## References\n1. Quantifying Distributional Robustness of Agentic Tool-Selection\n2. From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection\n3. Certifying Counterfactual Bias in LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uIFhE33C1F", "reviewer": "Reviewer_xpw2", "summary": "This paper is the first work to study LLM tool-selection bias and provides a systematic evaluation across tasks and tools.\nThe authors design tasks that require invoking multiple candidate tools and measure how model choices vary across attributes and prompts. The core contributions of this paper include (1) a large-scale benchmark for measuring tool selection bias, (2) an in-depth analysis of the root cause of these biases, (3) a potential mitigation strategy.\nAlthough this paper discusses an interesting and potentially impactful bias in the LLM era, the authors have provided a limited discussion of the impact of such a bias and the risks and implications behind it, which reduces the readability of this paper.", "questions": "1. Fig. 3 shows a long error bar in DeepSeek and the `what's language` question. The authors could provide a brief explanation and discussion of the results corresponding to this unusually large standard deviation, which would help readers better understand the model's biases and potential anomalous behavior.\n2. Although this paper provides comprehensive experiments and analysis on the manifestations and sources of tool-selection bias, it lacks a discussion of the harmfulness and risks of this bias. The authors could supplement a discussion (for example, by providing figures to illustrate the value of the tool selection market) or conduct a questionnaire/interview to highlight the consequences of such a bias. This would further reinforce the significance and value of this work and echo the claim in the introduction and conclusion.\n3. Since not everyone has a background in LLM tool usage, it will be good if the authors could provide a detailed tool usage process in the related work or appendix, preferably with a visual flowchart, which will effectively enhance the presentation of this paper.\n4. Typo: There are no ChatGPT-3.5-turbo and ChatGPT 4.1-mini models. The authors may actually be referring to the GPT-3.5-Turbo and GPT 4.1-mini models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A large-scale benchmark for measuring tool selection bias\n2. In-depth analysis of tool selection bias (including testing the impact of pre-training data on tool selection bias through training), which can strengthen people's understanding of this issue\n3. Extensive experiments and analysis, providing valuable insights for future work on building fair and reliable agent systems", "weaknesses": "1. Lack of deep discussion of potential consequences of tool selection bias makes it difficult for readers to quickly get the effects and research significance of this paper.\n2. Lack of a clear explanation of the LLM tool usage pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)