Assessing biomedical knowledge robustness in large language models by query-efficient sampling attacks
Authors: Rui Patrick Xian
Alex Jihun Lee
Satvik Lolla
Vincent Wang
Russell Ro
Qiming Cui
Reza Abbasi-Asl
Venue: Accepted by TMLR ()

Abstract: The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. Understanding model vulnerabilities in high-stakes and knowledge-intensive tasks is essential to quantifying the trustworthiness of model predictions and regulating model use. The recent discovery of named entities as adversarial examples (i.e. adversarial entities) in natural language processing tasks raises questions about their potential impact on the knowledge robustness of pre-trained and finetuned LLMs in high-stakes and specialized domains. We examined the use of type-consistent entity substitution as a template for collecting adversarial entities for medium-sized billion-parameter LLMs with biomedical knowledge. To this end, we developed an embedding space, gradient-free attack based on powerscaled distance-weighted sampling for robustness evaluation, which has a low query budget and controllable coverage. Our method has favorable query efficiency and scaling over alternative approaches based on blackbox gradient-guided search, which we demonstrated for adversarial distractor generation in biomedical question answering. Subsequent failure mode analysis uncovered two regimes of adversarial entities on the attack surface with distinct characteristics. We also showed that entity substitution attacks can manipulate token-wise Shapley value explanations, which become deceptive in this setting. Our approach complements standard evaluations for high-capacity models and the results highlight the brittleness of domain knowledge in LLMs.

Published in Transactions on Machine Learning Research (12/2024)
Assessing biomedical knowledge robustness in
large language models by query-efficient sampling attacks
R. Patrick Xian 1.
Alex J. Leel
Satvik Lolla? Vincent Wang'
Russell Ro
Qiming Cui
Reza Abbasi-AsP.P
UC Sun Francisco UC Berkeley
"arpatrick Unnaarick@gmall..om Domaallcm.oooommm com,, reza awwasiassl@uss...dl
Reviewed on OpenReview: https: openreview. net/ formm? id=puol5Jy178
Abstract
The increasing depth of parametric domain knowledge in large language models (LLMs) is fu- eling their rapid deployment in real-world applications. Understanding model vulnerabilities in high-stakes and knowledge-intensive tasks is essential to quantifying the trustworthiness of model predictions and regulating model use. The recent discovery of named entities as adversarial examples (i.e. adversarial entities) in natural language processing tasks raises questions about their potential impact on the knowledge robustness of pre-trained and fino- luned LLMs in high-stakes and specialized domains. We examined the use of type-consistent entity substitution as a template for collecting adversarial entities for medium-sized billion- parameter LLMss with biomedical knowledge. To this end, we developed gradient-free free attack based on powerscaled distanco-weighted sampling for robustness evalu- ation, which has a low query budget and controllable coverage. Our met has favorable query efficiency and scaling over alternative approaches based on blackbox tradient-guided scarch, which we demonstrated for adversarial distractor generation in biomedical question answering. Subsequent failure mode analysis uncovered two regimes of adversarial entities on the attack surface with distinct characteristics. We also showed that entity substitution attacks can manipulate token-wise Shapley value explanations, which become deceptive in this setting. Our approach complements standard evaluations for high-capacity models and the results highlight the brittleness of domain knowledge in LLMs!
embedding space,
1 Introduction
LLMs such as pre-trained and finetuned generalist language models (GLMs) and their domain-adapted versions (Zhao al., 2023; Wang et al., 2023a) incorporating transformer architectures are emerging as the technological backbone of next-generation search engines and knowledge bases (Petroni et al., 2019; Sung et al., 2021). They are increasingly used in knowledge-intensive tasks by the general public without sufficient assessment of inherent issues in trustworthiness and safety (Clusmann el al., 2023; Spitale el al., 2023; Si et al., 2023). Adversarial examples are valuable probes for model vulnerability and robustness, and are essential for performance improvements and regulatory andits (Shayegani et al., 2023). Adversarial attacks are classified by query scaling behavior into low- and high-query-budget attacks (see Fig. la). Collecting adversarial data scale, either for adversarial training or robustness assessment, requires query-efficient attacks, first empirically studied in the vision domain (llyas al., 2018; Cheng al., 2019; Shukla et al., 2021). For language models, adversarial examples are generated through guided search (Alzantot et al., 2018), sampling and probabilistic hoth (Ren et al., 2019; Yang et al., 2020; Yan el al., 2022), collected dynamically by human operators (Jia & Liang 2017; Wallace et al., 2019b). Most of these methods require a
The code developed and datasets used for the work are available https:/ htapp:://ithuub.omm hatpa//c.ooooeeaaaali////a/aaaa

---

Published in Transactions on Machine Learning Research (12/2024)
large amount of model evaluations (in the hundreds to thousands beyond per instance) whitebox access to model internals, and their evaluations tend to focus on small models (with < 1B parameters) (Maheshwary et al., 2021; Yu et al., 2024), which are insufficient in the era of LLMs. Sampling-based attacks obviate the need for prodetermined search heuristics and can operate with only blackbox model access and a low query budget. Demonstrations of query-efficient altacks on LLMs meet an emerging demand for their continuous monitoring (Metaxa et al., 2021), which is particularly important for deploying LLMs specialized domains.
Question answering (QA) has been the proving ground in recent achievements of biomedical language models (BLMs), here referring to GLMs adapted to the biomedical domain (Singhal et al., 2023; Kung et al., 2023; Liévin et al., 2024; Saab et al., 2024). Knowledge and reasoning components in biomedical QA assess the understanding of diverse concepts, which are represented by domain-specific named entities (NEs). The present work aims to identify adversarial entities in biomedical QA for both GLMs and BLMs. While GLMs may have acquired general biomedical knowledge during model development, BLMs gain further domain knowledge primarily through finetuning on a variety of text data from clinical notes (Lchman al., 2023), electronic health records (Wornow et. al., 2023), to abstracts excerpts of scholarly publications (Tinn et al., 2023). Although QA is frequently employed paradigm for model performance evaluation (Gardner et al., 2019; Robinson & Wingate, 2023), currently, biomedical QA still lacks domain-specific robustness assessments and its use in real-world applications, albeit promising, is still in its infancy (Kung et al., 2023). Evaluating model robustness in high-stakes, specialized domains requires customized perturbations apart from traditional benchmarks (Cecchini et al., 2024). Deployable LLMs in the biomedical domain should have robust contextual knowledge and language understanding of domain-specite entities.
Adversarial attack settings
E
Low-query budget
Attack success rate
c
D
C'
B
A
High-query budget
Entities of the key (D), original (A,B,C) and adversarial (C') distractors in embedding space
Query budget (a)
(c)
50-year-old woman presents with severe headache and vomiting, She says that symptoms onset after attending a wine tasting at the local brewery. She says that her headache is mostly at the back of her head and that she has been nauseous
Original
Serum creatinine
Temperature
C: Creatine phosphokinase
and vomited twice. Past medical history depression diagnosed 20 years ago but now well-controlled with medication. She also has significant vitamin D deficiency. Current medications are phenelzine and a vitamin D supplement. The patient denies any smoking history. alcohol or recreational drug use. On physical examination. the patient is diaphoretic. Her pupils are dilated. Which of the following most likely to elevated in patient?
significant for
Blood pressure
Adversarial
Serum creatinine
B: Temperature
C: Parathyroid hormone
Blood pressure
(b)
Figure 1: Entity substitution attack on QA with adversarial distractors. (a) Typical query scaling curves low- and high-query-budget attack settings. (b) An adversarial distractor example found by type-consistent entity substitution (highlighted in blue). The correct (top) and incorrect (bottom) model responses (check- marked) before and after perturbation are included. (c) Illustration of the attack scheme in embedding space by PDWS for the example in (b). & represents the vocabulary set. D is the key to the question, A-C the original distractors, C' is an adversarial distractor at distance from the key D.
2

---

Published in Transactions on Machine Learning Research (12/2024)
In this work, we proposed and compared sampling- and search-based query-efficient adversarial attacks for evaluating the knowledge robustness of LLMs in the entity-rich, biomedical domain using type-consistent entity substitution (TCES). We investigated the setting where a large number of viable replacements (Ne > 1000) per attack cxist at only a small number of permissible text locations, which is provalent in specialized domains where the NEs have shared characteristies such as their type and (word or phrase) structure. The sctting differs from traditional synonym substitution (Liu et al., 2022; Yu et al., 2024), which have a significantly smaller number of viable replacements with matching word senses but can operate on many potential text locations. To relate to real-world scenarios (Apruzzese et al., 2023), we consider the attacks in the blackbox and hard-label setting with a limited query budget (B < Ne). We used the attacks to generate adversarial examples of distractors (see Fig. 1b), or adversarial distractors in biomedical QA tasks.
Our main contributions include four aspects: (i) Introduction of powerscaled distance-weighted sampling (PDWS) method in embedding space to generate adversarial entities. Our work reports the first use of DWS for adversarial attacks in text?. The attack schome has broad coverage of the embodding space centerod around an anchor point rillustrated the key D in Fig. Ic) and maps the entity-level robustness landscapes of different LLMs in two regimes (near and far). (ii) Introduction of the FDA-drugs and CTD-diseases datasets as the source vocabularies of the perturbation sets, along with an entity type-annotated version of existing biomedical QA datasets MedQA-USMLE (Jin et al., 2021) and MedMCQA (Pal et al., 2022). Using these data, we demonstrated the vulnerability of LLMs to two of the most common NEs in the biomedical domain (Wei et al., 2019): drug and disease names (see example in Fig. 1b). (iii) We unified sampling- and gradient besech-based methods in discrete text space and formulated TCES as an attack template, which we used for quantitative comparison of the scaling behaviors of the attacks in the same footing. We found that in the blackbox setting, sampling attacks attacks, which are gradient quality-limited within a fixed query budget. (iv) By investigating the relationship between Shapley value-based model explanations (Chen et al., 2023) and adversarial attacks, we uncovered the defining signatures of successful attacks as a result of entity substitution.
have favorable query scaling over gradient-based
2 Related works
Text substitution attacks Adversarial attacks in neural language models have been demonstrated from the character to the sentence level (Zhang al., 2020). The majority of text substitution attacks employ typos (Gao et al., 2018) and synonyms (Mozes et al., 2021), which effectively impose lexical constraints. Word frequency has been used as a metric to select adversarial examples in (Mozes et al., 2021). Synonym substitution attacks have been demonstrated in sentiment analysis (Liu et al., 2022; Yu et al., 2024). Alzan- tot et al. (2018) used a genelic algorithm to search for adversarial word substitution. The sensitivily of models to entity-level perturbations, including substitution and swapping, creates problematic situations such as knowledge conflicts (Longpre et al., 2021) for knowledge-intensiee tasks, compromisod performance in machine reading comprehension (Yan et al., 2022) and table interpretation tasks (Koleva el al., 2023).
Robustness in QA The robustness of QA models has been studied using perturbation methods (Jia & Liang, 2017) and meaning-preserving transformations (Gan & Ng, 2019; Elazar et. al., 2021). Sen & Salfari (2020) introduced set of perturbations to assess language model generalization between QA datasets. Richardson & Sabharwal (2020) proposed a distractor design method using proximity in semantic space to probe model understanding of word senses. Awadalla et al. (2022) examined the connection between in- and out-of-distribution robustness for different LLMs. The performance of LLMs is sensitive to the ordering of choices (Pezeshkpour & Hruschka, 2023; Zheng et al., 2024). Overall, the brittleness of knowledge in LLMs remains a primary limitation for their applications (Elazar et al., 2021; Augenstein et al., 2024).
Charactoristics of adversarial examples The defining characteristic of adversarial examples is the in- duction of false predictions ions in trained models that would otherwise predict accurately. Other characteristics include (i) their generality such as transferability and universality (Zou et al., 2023). Transferability refers to the effectiveness of the same example being adversarial to distinct models (Papernot et al., 2016). Universal
Thhooughout the text, we use DWS to refer to the class of sampling method that employs distance-dependent probability weights. The powerscaled DWS a type of DWS, so is the other version intraduced in Wu al. (2017).
3

---

Published in Transactions on Machine Learning Research (12/2024)
adversarial perturbations are reusable across many examples tested on the same model (Moosavi-Dezfooli et al., 2017; Wallace et al., 2019a). (ii) The quality of adversarial examples may be judged by their per- ceptibility to human (Dyrmishi et al., 2023) or simple defense mechanisms such as grammar check. (iii) Adversarial attacks can operate in low- and high-query-budget settings (Shukla et al., 2021) with different query scaling behaviors. (iv) Adversarial examples with specifically designed perturbations can obstruct the model explanation (Noppel & Wressnegger, 2024) and render it unusoful.
3
Entity-centric adversarial attacks
3.1 Adversarial distractor generation
with the context C, question stem S, a distractor set D = {do and a key k. The distractors are the wrong choices given to confuse test-takers, while the key the correct one (CH & Saha, 2020).
Definition 3.1. A multiple-choice question Q is represented by a tuple, Q=
Definition 3.2 (Adversarial distractor). Given the original (or unporturbed) and a perturbed multiple- choice question pair, Q = (C, D. kt) and Q' = S, D', k), which differ by only a distractor, D' (DDD') = {d'). The distractor d' is adversarial if the perturbed question elicits a false response from the model, while it answers correctly to the unperturbed question,
argmax Pre.M k,C, A argmnax PYLLM(( = k,C,S,
(1)
The threat model in Definition 3.2 refers to an untargeted attack, while the targeted version only requires restricting the answer to the perturbed question to a specific distractor. A simple way to construct the perturbed distractor set. D'' in O' is to substitute a distractor d by d', or D' = (D \ {d)) U (d'}. The substitution is subject to the constraint ||Q' 21 < e, where e the perturbation budget, and the norm assume the distractors contain only NEs and only one NE
|| refers to edit distance. For simplicity, each (such as in Fig. 1b). In Appendix A, we extend Eq. 1 to more general case with nultiple entities and additional non-entity text in the distractors.
Example 3.1. In the example of Fig. 1b, the stem S is the last sentence on the left side, "Which of the following is most likely to be elevated in this patient?" The context C is all the text before the stem, "A 50- year-old woman... Her pupils are dilated." The original distractor set D = {Serum creatinine, Temperature, Creatine phosphokinase, Blood pressure), the perturbed distractor set D' = (Serum creatinine, Temperature, Parathyroid hormone, Blood pressure), the key is Blood pressure, or equivalently, D
Unlike lext addition allacks that introduce distracting information (Jia & Liang, 2017; Wallace et al., 2019a), the edit constraint on Q that is usually enforced in text adversarial attacks (Roth et al., 2024) doesn't preclude large somantic perturbations at the entity level in the distractors. In this work, we draw perturbations from a finite perturbation set, denoted as E. == E\ \ (k), with & / being the entity dataset as illustrated in Fig. 1a. In practice, may be compiled from potential model use cases curated by regulatory bodies in model audit. Next, we discuss unified view of discovering adversarial entities in the embedding space using blackbox (or gradient-free) attack methods based on sampling and search.
3.2 Powerscaled distance- distance-weighted sampling
Definition 3.3 (Text span representations). Given the token sequence representation of a span e, we define its corresponding vector representation in text embedding of dimension as e E RR.
We consider sampling with distancc-dependennt probability weights from the embodding space of NEs. The concept of distance- sampling (DWS) was previously introduced in contrastive learning in computer vision, where the Euclidean distance between samples and an anchor point was used and the probability weights depend on the embedding dimensionality (Wu et al., 2017). We propose a more flexible version
For simplicity, in this work, we don't use different symbols distinguish between the content of distractor (or a key) from its index (e.g. A, B, C, D).
4

---

Published in Transactions on Machine Learning Research (12/2024)
independent of the embedding dimensionality using a powerscaled distance with tunable hyperparameter.
Specifically, for a key k, we assign distance-dependent probability weight Pj E [0, 1] to entity ej such that the distractor d is substituted by
(k))) d' w.p. PS h,
(2)
Here, li,,, eÿ) is distance metric between the anchor point k and eg, while the exponent n E R controls the shape of the probability weight distribution. When n > 0, as n increases, the entities further away from are more preferably weighted. When a < O, the sampling method employs inverse-distance-senn weights, and the entities closer k are more preferably weighted. In the subsequent evaluations, we use calculated in text embedding of choice, where k and are the respective vector representations of l and e. The formalism accommodates deterministie regime as limiting case?, when n +00, and Pj is vanishingly small for all but the entity furthest from k;
hikk, eÿ) CosineDist(k, eş)
d' = Ej s.t. ej = argmax h(k, ez)-
(3)
eyEE.
When O, Pj is the same for each entity in E.k, corresponding to uniform random sampling. A short proof of the limiting cases is given in Appendix B. Therefore, our approach bridges the probabilistic and deterministic approaches to sampling adversarial examples.
3.3 Sampling view of zeroth-order adversarial attack
Blackbox attacks using approximate gradients may proceed with zeroth-order optimization (2OO), where the gradients are estimated by finite difference to guide the search (Ilyas et al., 2018; Cheng et al., 2019). In the discrete text space, Berger et al. (2021) developed the DiscreteZ00 attack, which contains three components: the word importance ranker, the candidate sampler, and the gradient-based optimizer. It operates at the word level and has been validated on BERT-sized models for synonym substitution attacks. The last two components aim to find a replacement text span e' in each iteration through the gradient update rule
e'= (
(4)
Here, en refers the original text span, e' and en the vector representations of e' and co, respectively. À the learning rate, Vzo indicates the zeroth-order gradient estimator, and 2ZOLLM (cep) @zep) has the same dimensionality as e' or en. We use LL)((;;a) to denote the LLM text output with the unperturbed question @ (containing eo) as the input, then the multi-point (M 2 2) gradient estimate is constructed by querying the LLM M times and computing
ZZ co) = En
ALLM Q: en)
u
1
(LLM(Q LLL Q: en)) (e, ep) El,EEE..
Σ
M
(5)
Aeo
M
|le
eo
This is an example of the random directions stochastic approximation (Nesterov & Spokoiny, 2017), where u = (oyoee///leeee is randomly oriented vector in the embedding space corresponding to the neighboring entity éj- Berger et al. (2021) computed Eq. 5 by random sampling of neighbors (in candidate sampler) within similarity threshold for synonyms. The procedure was iterated once for all ranked attack locations. Given noisy gradient estimate VzoLLM( Q: eg) and discrete search space, we can rewrite Eq. effectively as deterministic DWS in the embedding space by
d' d'= s.t. ej argmin eg), cyEE.
(6)
where h(co, CosineDist(c.- ooteeeooooooisttoooo---AALLA--A- eo), oj), with e, as the anchor point, and argmin indicates the closest member within the text embedding. Using argmin is essential because the exact location hi eÿ)
4Th limit + -30 indicates replacing the distractor d with all times. Since not in the perturbation E.k, this limit not attained.
5

---

Published in Transactions on Machine Learning Research (12/2024)
in the discrete embedding space is usually not occupied. While a gradient estimate of sufficient quality is essential for successful attack, increasing M will count towards the query budget. Therefore, in the budgeted setting, a tradeoff should exist between the efficacy of the attack and the number of queries consumed for each gradient cstimation.
4
Implementing attacks on biomedical QA
4.1 Dataset selection
Entity datasets To relate to real-world scenarios, sourced vocabulary datasets of drug and disease names from existing public databases. The drug names dataset (FDA-drugs) contains over 2.3k unique entities from known drugs approved by the United States Food and Drug Administration (FDA) and curated by Drug Central (Ursu et al., 2017). FDA is a world authority that approves the legal distribution of drugs, therefore, the dataset represents the drug-entity names encountered in daily life. The disease names dataset (CTD-diseases) contains over 9.8k unique entities from the Comparative Toxicogenomics Database (Davis et al., 2009), which contains a comprehensive collection of uniquely documented chemical-gene-diiees interactions for humans. Both databascs are curated by domain experts through regular updates, and unlike traditional text corpora (Mohan & Li, 2019), the entilies in these two dalasets contain no redundancy. The 2023 version of the datasets was used after minor data processing.
Biomedical QA datasets We selected over 9.3k questions from the MedQA-USMLE (Jin et al., 2021) dataset and over 3.8k questions from the MedMCQA (Pal et al., 2022) dataset for benchmarking. Both datasets are entity-rich and cover a wide range of biomedical specialties and topics. The MedQA-USMLE dataset contains long-context questions that are used to assess medical students in the United States. The MedMCQA dataset contains short questions used in medical exams in India. They have been used in recent works to assess the biomedical knowledge in LLMs (Singhal al., 2023; Han et al., 2023; Liévin et al., 2024; Saab et al., 2024). Both datasets are publicly available and don't contain personal information. We annotated the NEs according to the entity types of the Unified Medical Language System (UMLS) (Bodenreider, 2004) using scispaCy (Ncumann et al., 2019). We then divided the QA datascts into drug- or discase-mention questions according to the UMLS entity types (see Appendix C) of NEs in distractors.
4.2 Type-consistent entity substitution (TCES)
Implementing attacks that target entities requires accounting for type consistency and compatibility with si- multancous multi-word (or span-level) operations, because of the inseparable nature of the entity components (e.g. growth hormone, acquired immunodeficiency syndrome). This effectively results in more substitution patterns (one to two words, two to one word, two to two words, two to three words, etc.) than the more common one-to-one synonym substitution (Mozes et al., 2021). We describe TCES in Algorithm 1 as a gen- eral and contextualized attack template for entities, with the goal of adversarially modifying model outputs with only one span-level substitution, specified by token boundaries. The template can accommodate single- or multi-query attacks and it applies to any task that may be formatted into QA (Gardner al., 2019).
The initial steps in TCES involve entity recognition and typing (see Appendix C). For distractor generation, the type-filtered entities in the choices (Entta))) are separated into key (Entrey) and distractor (Entaistre) entities. In the biomedical domain, the entity type information is readily available from the UMLS (Bo- denrcider, 2004) semantic groups, such as Chemicals & Drugs and Disorders used for demonstration later. The RankSelect step selects the text span to attack (Entvictim). In the PDWS attack, the type-matched entity the distractors closest to the key selected. The purpose is to maintain consistency and is not uniquely associated with the effects discussed later. In ZOO-based attacks, RankSelect corresponds to the word importance ranker. Then, the potential replacement entities selected using Sampler (Eq. 2 for PDWS or Fq. 6 for Discrete70O), which also removes any duplicates of the key in the vocabulary (Fatvoccb). Sampling is carried out without replacement from the corresponding perturbation set E. (constructed with
Shtter//anngeenttloogoo/owood Phttps//cctdaase..rgg htw.o//ooos...ottooooooossssss
6

---

Published in Transactions on Machine Learning Research (12/2024)
FDA-drugh or CTD-diseases datasets), equivalent to enforcing type consistency in the replacement enti- ties with the entity to attack (Enbvictim)- GoalFunc outputs a value from loss function or from directly comparing with the answer to determine if the attack is successful.
4.3 Attack execution on LLMs
Victim LLMs We selected both generalist (GLMs) and specialist (BLMs) models for robust- ness evaluation via blackbox adversarial attacks. The criterion we adopted here for a model to have biomedical knowledge is that it should have a base- line performance better than random guessing (e.g. accuracy of 0.25 for specific four-choice QA task) in the evaluated domain-speciff task. For GLMs, we used Instruction-fimetune T5 (Flan-T5) series models (Chung et al., 2024), and UL2 model (Tay et al., 2022) (Flan-UL2) For BLMs, we used MedAlpaca-7B (Han et al., 2023), 3 v2.0 from John Snow Labs (JSI-MeaI.M3-SSBV22 (LM2-Medtuund--133) (Ro- hanian et al., 2024). and Palmyra-Med-22 (Kamble & Alshikh, 2023) from Writer. The selected open- source models have sizes in ~ 1-20B range. Besides, we also evaluated GPT-3.5 from OpenAI.
Algorithm 1 TCES attack template for collecting ad- versarial entities in distractors.
Inputs: Question Q, LLM, entity type т, budget B. Internals: Nunber of choices Nah, text L, entity Ent, embedding Emb, key or correct answer k. Outputs: Number of querics and the replacement en- Lity, None if unsuccessful after all attempts.
ACESACAAEEER(( Model, T, B) Echoices: t Q.choices, Q.answer for c c Noh do Fch f- tchoices [c)
Entch f MERecognizer((an) Entzich TypeFilter T)
Entkey, Entdistre f- DolittyyLbbe(Enntaal
Entvictim <- RankSellect(anfary Entdistre: Emb)
i i
while i < B do perturb <- Sampler (Enlgey Eiltietim
Ent Emb)
Attack settings All models were evaluated at zero temperature or the non-sampling setting and model inference was conducted in the zero-shot set- ting with only basic prompt instructions (see the prompt structure in Appendix D). Model inference of Mallyyra-Med22000
Q' + Q (Entvietim + Entperturb) + Model(Q')
if GoalFunc(k', k) + then return Earturtuub
(Kamble & Alshikh, 2023) used 4-bit quantization to improve speed (Dettmers & Zettlemoyer, 2023). The evaluation scttings ra- sulted in our somewhat lower baseline accuracies than the reported ones which are often achieved with the help of few-shot prompting and prompt opti- mization. The DiscreteZOO attack (Berger et al., 2021) was originally implemented in the TEXTATTACK framework (Morris et al., 2020b), which updated to be compatible with span-level perturbation using its boundaries, multi-GPU split-model inference, and decoder-based billion-parameter LLMs by modifying the attack recipe. The following three aspects of the benchmarks are in our focus.
Enbyocab &- Entyneabl Entperturb
i 1+i+1
return
Query budget: We used fixed query budgets (B) for three main types of attacks: (i) Single-query sampling-based attacks were used as the reference because the DiscreteZOO attack requires a minimum of 3 model queries; (ii) Multi-query attacks used budget of 8 for reasonable compulational cost across all models and attack settings for both sampling- and search-based attacks; (iii) The query scaling trends of specific LLMs and different attack settings were investigated with a series of query budgets under 100 per input instance. Here, we also included the deterministic versions of PDWS to sample the nearest and farthest elements to an anchor point. The element was taken from the perturbation set and the distances (between entities) were calculated using cosine in a chosen text embedding. Their query-limited versions are referred to as B-ncarest and B-farthest element sampling, respectively, where all nearest or farthest elements were used sequentially to allack models in query scaling studies. Detailed definitions given in Appendix B.4.

---

Published in Transactions on Machine Learning Research (12/2024)
Text embedding: Both the PDWS and DiscreteZOO attacks require sampling from text embedding. The counter-fitted GloVe embedding (Mrkšić et al., 2016) was the default for DiscreteZOO, but GloVe is poor for representing biomedical vocabulary (Wang et al., 2018). We chose two recent embedding models with improved latent representations of text relations for the benchmarks: (i) CODER (Yuan et al., 2022), a SERT-structure. embedding model contrastively pretrained to distinguish UMLS knowledge graph concepts; (ii) GTE-base (Li al., 2023b) from contrastive learning of general web-scale text data².
Core hyperparameters: For single-query PDWS attacks, we tuned the hyperparameter n within the interval of [-50, 50] using grid search with a step of or 10 because the local maxima of ASR appear on the positive and negative sides. For multi-query PDWS attacks, hyperparameter was briefly re-tuned around its optimal value in the single-query attack. Most tuned hyperparameters fall within [-30, -5] and [5, 30). Operating the DiscreteZOO attack in a compatible setting for entities requires two key modifications: (i) Use a random word ranking, which allows the attack algorithm to operate in the low-query-budeee setting. (ii) Disable the similarity threshold for candidate point selection in gradient estimation because only entity type consistency is required. The ASR was greatly improved after this step. Benchmarks of the DiscreteZOO atlacks were carried out with these settings.
5 Finding and assessing adversarial entities
5.1 Two regimes of adversarial entities
Diversity quantification of adversarial entities The embedding space picture of the substitution attack in Fig. la provides a geometric interpretation of PDWS to examine different types of adversarial examples by tuning n. We use the Gini-Simpson index (Rao, 1982), JJSS = 1 Σ, E to quantify the diversity of adversarial examples and a low value indicates less diversity. Here, q3 E (0, 1] is the probability of occurrence of the entity e, among all instances of adversarial entities obtained from sampling, and Σ, 9 1. A low diversity index indicates high reusability and is a proxy for the universal adversarial attack regime -Moosavi-Dezfore et al., 2017; Wallace et al., 2019a).
Considering the joint effects of diversity and hyperparameter n on ASR in Fig. reveals two regimes: (i) adversarial entities semantically close to (n < 0) the key k have a greater diversity; (ii) adversarial entities semantically far from (n > 0) the key & are more reusable. TCES in either of these two regimes can lead to increase in ASR from random sampling (equivalent. to 0), as shown in Tables 1-2. Fig- 2 shows that larger positive n in Eq. 2 samples more universal adversarial entities than in other situations, leading to marked drop in the diversity index. Moreover, there exist local maxima of ASR in both regimes with finite n, which were discovered by hyperparameter tuning. We call the behavior the two-regime effect and it was observed in almost all models evaluated here in varying magnitudes for both datasets, thanks to the controllable spatial coverage of sampling methods incorporating distance information. In both single- and multi-query altacks, the benchmarks show that PDWS attacks in the n < regime tend to incur a greater change in model performance than in the n > regime, yet prominent counterexamples exist, including MedAlpaca-7B and Llama2-Medtumcd-110 evaluated on drug-mention questions, and Flan- T5-xd evaluated disease-mention questions in Table 1.
Semantic distortion by adversarial entities Besides ASR, evaluating the semantic distortion is another way to quantify the effectiveness of adversarial attacks. We estimated the semantic distortion of entire prompts in successful attacks before and after entity substitution, corresponding to Q and Q' in Definition 3.2. We calculated the average prompt semantic similarity (PSS) using SentenceTransf former (Reimers & Gurevych, 2019) with the RoBERTa-large model (all-roberta-large-->) At each distinct n, the PSS E [] with 1 being the most similar, is averaged over all successful attack instances to obtain the average PSS, quantitative measure of semantic distortion. Its dependence on the hyperparameter n for different models and datascts is shown in Fig. 2.
&CODE at http:////////nnnnnoo/o////////// https://wweeeuiicccccecccccccooe
ENG; GTF-base at http::///ugghhgfaee.////hhhhh base.
8

---

Published in Transactions on Machine Learning Research (12/2024)
Flan T5-xdl xal
Palmyra MaddyyraMMMMd-aa Mod-20B 20B
Fan-T5.xl
ModAlpace--7 7B
Modaa USNLE
VeaVisa
MedaCOA
- Crig Co
Diaase
Atback succass mate (1%)
Drg oz
Dessie
Atrock success rato (%)
Drg Co
Nocate Ca
Attack success rall
25
Attock success
Dag Co
Doode de
12-
12-
12-
0.
8.5
BiniSimpoonn index
6.5
Gini: Simp-immoon index
23-
Gin Swmppmmonnn
0.7
077
27
UEILLE
VedVook
(a)
(b)
(c)
(d)
Flan-TS-odl
Palmyra-Med-005
Flan-TS-dl
MadAlpaca-78
average serannti: simiarity
Avarage prompt somainic simlarty
Aver age promppt somarrtic similrity
0.6
0:90
0:4
0:34
2:04
12:44
Average
0.2
0.02
E.2
Oicaac
Davase
40
(c)
(f)
(h)
Figure 2: Powerscaled DWS of adversarial distractors exhibits a two-regime effect at negative and positive n values (see Eq. 2) in ASR (top) and diversily index (bottom) of replacement entities in successful allacks. Local maxima in ASR with a finite n are also present in each regime. The vertical dashed line indicates the location of random sampling. The observed similar behaviors are compared across models and datasets in (a) Flan-T5-xae on MedQA- USMLE, (b) Palmyra-Med MedMCQA. Disease and drug-mention questions separated by colors. The average
on MedQA-USMLE, (c) Flan-T5-xl on MedMCQA,
(d) Med Appcca--- prompt semantic similarity displayed in (e)-(h) is calculated for the successful attacks obtained from the corresponding allack settings in (a)-(d), respectively.
Adversarial entity characteristics The two-regime effect indicates that the success of the attacks may come from two sources that dominate at different locations the same attack surface, which refers to entity substitution in the present work.
At larger distances, the attack succeeds potentially due to the obscurity of the distractor entities in the question context, which can also profoundly impact the model performance (Ti et al., 2023a). This expla- nation may be further supported by examining the diversity index, TJGS, which shows the dominance of few replacement entities in successful attacks at large distances. We found these highly reusable adversar- ial entities by ranking their occurrences in successful attacks by powerscaled DWS. For drug names, the top-ranked adversarial entities include n-acctylglncosamine (an amino sugar and anti-inflammatory drug) and technetium To 99m exametazime (a radiopharmaceutical and contrast agent) For disease names, some of the chromosome deletion syndromes and rare disease names are more reusable than others. These
10 RxList drug index contains entries for maaestygluucossmie and technetium 99m exametazime.

---

Published in Transactions on Machine Learning Research (12/2024)
MedQA-USMLE
MedMCQA
Model (sise)
Entity
Baseline (Acc)
RandSa31 (ASR %)
PDWS&1 (-/+) +CODER (ASR %)
Baseline (Acc)
R (ASR %)
PDWSRI (-/-)
mention
+CODER (ASR %)
Flan-Tō-xl (3B)
Drugs
0.318 0.335
10.4
13.5 / 10.7
0.265
14.3
19.6
15.1
Diseases
11.0
16.1
12.5
0.261
23.8
25.7
31.8
Flan-T5-xxl (IIB)
Drugs
13.6
0.291
16.2
Diseases
0.345
12.2
20.0
12.8
0.376
15.2
26.6
23.9
Drugs
0.382
8.6
12.8
10.5
0.416
13.9
15.1
12.7
Flan-U1a (20B)
Discases
0.396
10.1
15.7 13.1
0.408
15.7
25.5
17.7
GPT-3.5 (175B)"
Drugs
0,438
9.1
11.2 / 10.7
Discases
0.453
8.6
11.5 / 8.8
MedAlpaca-7B
Drugs
0.479
7.2
8.2 / 9.6
0.148
13.9
16.2
15.9
Diseases
0.498
8.4
12.5 19.2
U.434
13.7
JSL-MedLLM8-SBv
Drugs Diseases
0.569
4.9
8.4 6.1
0.827
10.4
10.2
11.8
0.592
6.5
11.4 / 6.4
0.771
9.5
18.5
11.3
LM2-Medduune-1138
Drugs Discases
0.371
16.4
18.1
17.8
0.568
15.3
17.3
18.1
0.398
19.6
23.1
19.6
0.515
19.0
27.6
21.0
Palmyra-Med-20B
Drugs
0.382
12.6
13.3
13.5
0.614
10.2
13.8
12.7
Diseases
0.441
16.9
21.5
18.7
0.624
15.4
20.1
11.9
Table 1: Single-query (budget = 1) attack success rates (ASRs) on subsets of MedQA-USMLE and MedM- CQA datasets by perturbing the drug or disease entity mentions. Zero-shot baseline (Baseline) accuracy is included for reference next to the ASR sampling attacks using perturbation by random sampling (RandS), powerscaled weightee sampling (PDWS). The s_" and "+" signs indicate the two regimes (n < 0 and n > 0). For each model, the highest ASR for type of entity-mention question in each dataset is bolded. The model with asterisk (*) were only evaluated
dataset. to limit the computational cost.
Single-query attack success rate (%)
MedDA-USMLEEduu
RS
Generalist LM
Biomedical LM
lungle-query attack success rate (%)
85
Generalist LM
Biomedical LM
POWS in 0)
POWS (n 0)
POWS (n 0)
FOWS in 0)
20
15
20
15-
FlanTSlage
Flam- T5
Flan Timm
Flan- 2.
MedAApass-77 78
LM2 MMdtuadttned--- 3B
Palmyra 200
Plan-
Flan TSnii
MasAlpaca-78
LMR MostMnunn
Palmyra Manyya-Madd
Figure 3: Model robustness from single-query adversarial attacks using results in Table 1. The models (GLMs and BLMs) were evaluated on drug-mention questions from MedQA-USMLE (left) and disease- mention questions from MedMCQA (right) datasets. The GLMs and BLMs are ordered horizontally by their sizes. The bar colors distinguish between different attack methods. Perturbations by random sampling (RS) are in purple. Perturbations by powerscaled DWS (PDWS) in the n < 0 and n >> 0 regimes are in blue and grey, respectively.
10

---

Published in Transactions on Machine Learning Research (12/2024)
entities dominate the sampling results in the >> 0 regime for all LLMs, indicating their transferability and universality.
At small distances, semantic proximity (Mozes et al., 2021) reduces the distinguishability between the key and the distractor, thereby increasing the chance of model failure. Examples include similar drugs or diseases with insufficient details such as using diabetes as a distractor instead of the correct answer diabetes II (refers type-1l diabetes) can result in a successful attack.
Adversarial distractors result. in little semantic distortion al the prompt level (see Fig. 2e-h). The experiments on the MedMCQA dataset yield somewhat lower average PSS than MedQA-USSLLE because of the longer format of the latter dataset, leading to slightly larger distortions by entity substitution. Adversarial entities obtained in the n < regime results in higher average PSS compared to those in the n > regime, while there also a two-regime effect the average PSS tends to be the lowest around n 0 and higher when departs further from 0. The trend is consistent for different models and datasets demonstrated in Fig. 2.
MedQA-USMLE (ASR %)
McdMCQA (ASR %)
Model (size)
Eatity mention
RandS PDWS@8 (-/+)
D200as +CODER +GTE buse
DZOO@S
RandS 08
PDWS@S (-/+)
D200@S +CODER
D200@8
+CODERL
+CODER
+GTE base
Flan- (3B)
Drugs Discases
22.0
29.3 / 23.3
21.7
19.5
32.5
18.3
36.2
34.3
28.2
27.2
38.8 24.8
20.9
23.2
47.1
61.7
49.0
45.8
39.3
Flan-UL2 (20B)
Drugs Discuses
22.5
27.5
22.3
20.4
19.2
26.4
35.6
22.4
30.7
26.4
25.8
34.9 26.8
25.7
24.0
39.2
49.5
31.6
41.3
37.0
MedAlpaca-7B
Drugs Discases
16.8
21.1
22.3
22.8
20.5
27.9
37.1
/ 30.1
42.6
36.8
21.7
30.3
24.0
26.5
25.2
29.8
43.9 / 30.8
40.5
38.1
Pelmyra-Med-208
Drugs
30.1
31.4
31.3
29.1
29.6
28.5
37.3
31.9
35.4
35.0
Discases
33.2
35.1
30.2
32.3
34.5
41.1
48.4 / 39.6
46.3
50.1
Table 2: Multi-query (here budget = 8) attack success rates (ASR %) on various LLMs by perturbing the drug and disease entity mentions in MedQA-USMLE and MedMCQA datasets. The PDWS attack uses the embedding from CODER and DiscreteZOO (DZOO) attacks use either CODER or GTE- embedding. For each model, the highest ASR for type of entity-mention question in each dataset is bolded.
5.2 Scaling characteristics from adversarial entities
Size is not all for scaling model robustness. We compare model robustness using single-query ASR as the metric in Table 1 and Fig. 3. For the Flan-T5 series of GLMs (Chung et al., 2024) and Flan-UL2 (Tay et al., 2022), the single-qurry ASR is more pronounced in smaller and less performant models, indicating the improvement of robustness as the model's size grows. However, an opposite trend of robustness is seen in BLMs. It is worth noting that although times the number of parameters than MedAlpaca- 7B (Han et al., 2023), the larger model noticeably more sensitive to entity perturbation. In terms of ASR, the BLMs are comparable to the GLMs in each type of questions evaluated on. Another observation is that the ASR for models is asymmetrical under PDWS attacks. For most of the models, sampling adversarial entities with wwversedditance-scalle in Eq. 2) induces more performance drop than sampling with distance-scaled weights (n > 0 in Eq. 2), yet prominent countercxamples cxist. The attack performance is further investigated in the sequential, multi- query setting (budget = -8 in Table 2, where we also compared sampling- against search-based DiscreteZOO attacks. Here, PDWS attacks in the n < 0 regime are still largely the most effective.
halmyra-Med-2010 (Kamble & Alshikh, 2023) has about three
weights (n < 0
Sampling attacks has improved query scaling over blackbox gradient attacks in specialized text embedding. A query scaling study of adversarial attacks investigates attack success with varying query budgets per input instance (Shukla et al., 2021). It is a comprchensive evaluation of attack methods. Fig. 4 compares the query scaling of both sampling- and search-based methods for TCES. For low-query- budget attacks (Fig. la), each scaling curve exhibits a rapid rise phase and a plateau phase, separated by an inflection point. The key message here is that the advantage of PDWS attacks over DiscreteZOO depends on the model and the choice of text embedding. The PDWS attacks are overarchingly more effective for GLMs than D00, while for BLMs, the advantage is reduced. Moreover, using a specialized text
11

---

Published in Transactions on Machine Learning Research (12/2024)
100
60
Attack success rate (%)
Attack success rate (%)
50
30
Plan-T5 xl MedMCQQ disease
MedNpaca-78. USMiE-d
RS
PDWS CODER
20-
PDWS (n<0) CODER
PDWS (nc-0) CODER
PDWS (n>0) CODER
(nxc)) GTE-base
10
PDWS (n-0)
DiscreteZ00 CODER
DiscreteZ00 CODER
DiscreteZ00 GTE-base
Discrete200 GTE-bese
50
60
80
100
Query budget
Query budget (b)
(a)
100
60
Attack success rate (%)
Attack success rate (%)
30
Alan-T5xxl. x1x MedMCQA disease
MedAlpaca- 78,
(n=0)
PDWS [n<0] CODER
PDWS (n=0) CODER
20
PDWS (n=0) CODER
POWS (n=0) GTE-base
(n<0)
B-nearest CODER
B-nearest CODER
B-farthest CODER
8-farthest + CODER
B-nearest GTE base
B-nearest GTE-base
50
60
70
80
90
100
Query budget
Query budget
(d)
Figure 4: Example scaling curves of the query budget against ASR for disease-mention questions with (a) Flan-T5-xl model on MedMCQA datasct and (b) MedAlpaca model on MedQA-USMLA dataset. The curves are generated using TCES of disease names with the methods described in the legends. The DiscreteZOO attacks were run in the low-query-budger setting. Executing the random sampling (RS) attack doesn't 't require a text embedding, while the other attacks were evaluated with the CODER or GTE-base as text embedding. In (c) and (d), query scaling of the attacks based on B-nearest and B-farthest element sampling (B is query budget)) for the same datasets and models in (a) and (b) are compared with PDWS.
embedding further boosts the advantage PDWS attacks compared with a general text embedding due the difference in embodding space neighborhood. We expand the analysis in the following along three directions.
Query scaling depends on text embedding: Changing the text embedding affects the query scaling behavior of both PDWS and DiscreteZ00 attacks. Fig. 4 shows that the influence of the embedding choice PDWS is more pronounced in the rapid rise phase, while it inffuences the DiscreteZOO primarily in the plateau phase. Specifically, changing the text embedding from domain-specializet CODER (Yuan et al., 2022) to the general GTEE-bse (Li et al., 2023b), the performance advantage of PDWS over DiscreteZO0 attacks shrunk in Flan- T5-xl (Fig. 4a) and inverted in MedAlpaca- 7B (Fig. 4b).
Sampling hyperparameter controls scaling curve shape in PDWS attacks: The hyperparameter n changes The inflection point of the scaling curve. When the query budget passes the inflection point of the curve, random sampling starts to surpass PDWS (n > 0) in ASR. This a consequence of the distribution of adversarial entities the embedding space and the regional bias derived from the sampling weights. The examples in Fig. show that the largest performance gap between the PDWS and DiscreteZOO attacks appears around the inflection point of the respective query scaling curve.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Gradient quality affects query scaling in DiscreteZO0 attacks: The query complexity in blackbox gradient estimation is long-standing theoretical topic in derivative-free optimization (T (.'Euyyer & Yin, 1998) and online learning (Duchi et al., 2015). Nevertheless, how gradient quality affects query scaling remains less studied for adversarial attacks employing gradient-based discrete optimization. Gradient estimation consumess a large part of the total query budget, leading to a tradeoff between the gradient quality and terminal ASR attainable (at max budget) in the query scaling curve. We empirically explored the query scaling behavior of the DiscreteZOO attack in Appendix E with different candidate points in the multi-point gradient estimation, which in effect alters the gradient quality. The results show that having too few candidate points (i.e. small M in Eq. 5) leads to an early onset of the plateau phase, thereby limiting the highest attainable ASR, whereas having many candidate points can lead to slower growth of ASR in the rapid rise phase of the query scaling curve.
A 73-ycar-old man presents to the outpatient clinic complaining of chest pain with exertion. He states that resting for few minutes usually resolves the chest pain. Currently, he lakes 81 mg of aspirin daily. He has blood pressure of 127 /85 mm Hg and heart rate of 75/min. Physical examination reveals regular heart sounds and clear lung sounds bilateral. Which medication regimen below should be added?
A: Amlodipine daily. Sublingual
A: Amlodipine daily. Sublingual
nitroglycerin as needed.
nitroglycerin as needed.
B: Metoprolol and a statin daily. Sublingual nitroglycerin as needed. / C: Metoprolol and ranolazine daily. Sublingual nitroglycerin as needed. D: Amlodipine and a statin daily.
B: Metoprolol and statin daily. Sublingual nitroglycerin as needed. C:
and ranolazine daily. Sublingual nitroglycerin as needed.
D: Amlodipine and a statin daily. Sublingual nitroglycerin as needed. /
Sublingual nitroglycerin as needed.
[Context]: A 73-year-old presents the outpatient clinic complaining of chest pain with exertion. He states that resting for a few minutes usually resolves the chest pain. Currently, he takes 81 mg of aspirin daily. He has a blood pressure of 127/85 mm Hg and heart rate of 75/min. Physical examination reveals regular heart sounds and clear lung sounds bilateral.
[Context]: A 73-ycar-old man presents the outpatient clinic complaining of chest pain with exertion. He states that resting for a few minutes usually resolves the chest pain. Currently, he takes 81 mg of aspirin daily. He has a blood pressure of 127/85 mm Hg and heart rate of 75/min. Physical examination reveals regular heart sounds and clear lung sounds bilateral.
+
[Question]: Which medication regimen below should be added?
[Question]: Which medication regimen below should be added?
A: Amlodipine daily. Sublingual nitroglycerin as needed.
A: Amlodipine daily. Sublingual nitroglycerin
as needed.
B Metoprolol and statin daily. Sublingual
B: Metoprolol and statin daily. Sublingual
nitroglycerin as needed.
nitroglycorin as needed.
C: Metoprolol and ranolazine daily.
C: N-acetylglucosamine and ranolazine daily,
Sublingual nitroglycerin as needed.
Sublingual nitroglycerin as needed.
D: Amlodipine and a statin daily. Sublingual nitroglycerin as needed.
D Amlodipine and a statin daily. Sublingual nitroglycerin as needed.
Unperturbed
Adversarially perturbed
Figure 5: (Top) An entity substitution attack using n-acetylglneoosamine the replacement entity for meto- prolol creates an adversarial distractor. (Bottom) Heatmaps of loken-wise Shapley values for a question before and after the adversarial attack on choice C. The model prediction changes from the correct choice of B (unperturbed) to the incorrect choice of D (adversarially perturbed).
Deterministic sampling attacks are competitive proxies of their probabilistic counterparts. Performance comparison between B-nearest and B-farthest element sampling, the deterministic and query-limited versions of PDWS (corresponding to the n < and n > 0 regimes) are given Fig. 4c-d for the two example cases studied for the query scaling. The results show that the deterministic sampling tacks are already competitive against Discrete70O and also close in performance to PDWS
13

---

Published in Transactions on Machine Learning Research (12/2024)
in both specialized and general text embeddings. The difference is that the two types of deterministic sampling attacks examine only predefined regions in the embedding space and cannot explore the entire attack surface as PDWS would through tuning the hyperparameter.
5.3 Adversarial entities manipulate explainability
Post-hoc analysis is a common way to understand the rationale behind model decisions (Murdoch et al., 2019). Score-based leature importance such as that based on the Shapley values is routinely used to construct post-hoc explanation of model prediction through feature attribution (Chen al., 2023). The stability of model explanation is growing concern for their proper use (Alvarez-Melis & Jaakkola, 2018; Hancox-Li, 2020; Lin et al., 2023). Investigations on the susceplibility of LLM explanations to adversarial perturbation in QA are still lacking. For score- score-based feature importance, an explainer o(-) maps the m-dimensional feature x E R" to scores o(x) E RR".. In the current context, we assess the changes in feature importance represented by Shapley values (Chen et al., 2023) before (o(2) and after (ó(2') the adversarial attack by entity substitution Q + Q'. For convenience of discussion, we use Shapley profile to refer to ((Q) (or (())), the token-wise Shapley values calculated over (or Q').
We investigated the changes in the Shapley profile in the two regimes of adversarial entities for open-source LLMs, because the calculation requires access to model weights. We found that the token with the largest Shapley value (top-1 feature) is highly correlated with the model's prediction before and after adversarial perturbation, with an example shown in Fig. 5 produced using Flan-T5-large. This behavior agrees with the symbol-binding characteristic of LLMs in multiple-choice QA (Robinson & Wingate, 2023). Moreover, the Shapley profile typically exhibits pronounced changes before and after entity perturbation, especially in the text locations close to the perturbed entities (Fig. and Appendix E). Together, these observations indicate that the substitution attacks effectively manipulate model explanations, because the explanation after the attack aligns with the wrong prediction, as if they were correct. The behavior occurs despite unchanged context, stem, and key (C, S, and k), which should contain the most important information for understanding and answering the question (Chai & Jin, 2004).
6 Discussion and future work
The present study illustrates that TCES is a compact yet effective way to construct adversarial distractors, which leads to significant performance degradation in LLMs. Our results suggest that to minimize model queries, using a budget of up to the inflection point on the query scaling curve is the most cost-effective. The perturbed texts produced by TCES appear natural and are less likely to be detected using grammar check or topical filtering (Morris et al., 2020a). Therefore, the setting can simulate realistic scenarios where the "attacks" may be initiated by unsuspecting users, such as healthcare professionals using an LLM-based clinical docision support system (Liu et al., 2023). In the examples illustrated in Fig. 5 and Appendix E, ad- versarial distractors can lead to disease misdiagnosis or misprescription of medication, which are detrimental scenarios that may occur in algorithmic decision-making in biomedicine using L.MSs. Further improvement on the sampling attack performance may be achieved using learned samplers through rejection mechanisms (Narasimhan et al., 2024) or by finetuning the text embedding to improve the query efficiency of sampling. Alternatively, the semantic distance used for PDWS may be replaced with a concept distance (Choi et al., 2016) to improve attack performance on general text embeddings. Our approach may be integrated into inter- active platforms for adversarial data collection or online monitoring systems for open-ended hunan-machine conversation (Chao et al., 2023).
Future rescarch may focus on how to make LLMs more reliable through prompt engineering (Si et al., 2023: Nori et al., 2023) or leveraging retrieval from nonparametric knowledge bases (Weikum et al., 2021; Soman et al., 2023) or text sources (Wang et al., 2023b) to improve model generalization to long-tail entities. Besides vulnerability identification, the current work motivates adversarial defense strategics based on generalizod distances (La Malfa et al., 2020) to establish a tiered system for robustness assessment in user- fscr-facing and domain-oriented applications to mitigate catastrophic failures.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Acknowledgments
R.P.X. thanks N. Rethmeier for helpful technical discussions and we thank M. Collard for managing the computing resources. A.J.L. acknowlodges funding from NIH T32GM067547, the UCSF Discovery Fellows Program, and the Achievement Rewards for College Scientists (ARCS) Fellowship. R.A.-A would like to acknowledge funding from Weill Neurohub.
References
David Alvarez-Melis and Tommi S. Jaakkola. On the Robustness of Interpretability Methods, June 2018.
URL http://arxiv. orgg 200b//180008899
08049. arXiv:1806.08049 [es, stat].
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Chang, Generating Natural Langnage Adversarial Examples. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 2890-2896, Stroudsburg, PA, USA, 2018. Associa- orgganthologg
tion for Computational Linguisties. doi: 10. 10118633/v1/D118-3111 URL D18-1316.
Giovanni Apruzzese, Hyrum S. Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy. "Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Re- search and Practice. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) .ee....reeee.e..ee...e...e......
pp. 339 364, February 2023. doi: S41109/SaTMM.545775 0.003... URL hts//documnn/1113365
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Fil- ippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Ma- chine Intelligence, 6(8):852-866 August 2024. ISSN 2522-5839. doi: 10.1038/s42256 024-00881-z. URL Publisher: Nature Publishing Group.
attps://wwl www.:w.ntue....m.m wtt.e/occ/////ieeeeeeeeeeeeeeeee
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Tan Magnusson, Hannaneh Hajishirzi, and Ludwig Schmidt. Exploring The Landscape of Distributional Robustness for Question Answering Models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5971-5987, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2000 htt.:/.../.......nnnnn....e/ URL http:///aacaaaholgoorrrr//////// org/2022. f findings-emman 441.
Nathaniel Berger, Stefan Riezler, Sebastian Ebert, and Artem Sokolov. Don't Search for Search Method Simple Heuristics Suffice for Adversarial Text Attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8216-8224, Stroudsburg, PA, USA, 2021. Association for Computational Linguistics. doi: 10. 11653/v1/2001..mmm httm////22m.mmmmmmiia... Lemmlp-main.647 URL https: haap::////laaahoooy// org/ 2021. emnlp-main 647.
Bodenreider. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Research, 32(90001) :66D 270, jan 2004. ISSN 1362-4962. doi: / //academicc.ou oup. www.////////at/eee wwt.////arrraattcllooooooo/////p
URL https:
1093/000/ nar/gkh061.
David Cecchini, Arshaan Nazir, Kalyan Chakravarthy, and Veysel Kocaman. Holistic Evaluation of Large Language Models: Assessing Robustness, Accuracy, and Toxicity for Real-World Applications. In Anaelia Ovalle, Kai-Wei Chang, Yang Trista Cao, Ninareh Mehrabi, Jieyu Zhao, Aram Galstyan, Jwala Dhamala, Anoop Kumar, and Rahul Gupta (eds.), Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNILP 2024), pp. 109-117, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/vv 1005533//12222.aaa// 1/2022..rrustnlppp twwstruuuuppnt 1.11. URL https: https://aclanthologyy org/2024. trustnlp-1. 11.
Dhawaleswar Rao CH and Sujan Kumar Saha. Automatic Multiple Choice Question Generation From Text: A Survey. IEEE Transactions on Learning Technologies, 13(1):14-25, jan 2020. ISSN 1939-1382, doi: 10.
URL https://ieeexpllreeeeeee ieee arg//dccuentt/55555....
15

---

Published in Transactions on Machine Learning Research (12/2024)
Joyce Y. Chai and Rong Jin. Discourse Structure for Context Question Answering. In Proceedings of the Workshop on Progmalies of Question Answering al HI.T-NACL. 2004, pp. 23 23-30, Boston, Massachusetts, USA, May 2004, Association for Computational Linguistics. URL httpe://aallanthology/ wwwpoooooooyoo///--0//00
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. arXii,, pp. 2310.08419, oct 2023. URL
http://arxiv. org//os././..o.oo... 22t1//2111......... 08419.
Hugh Chen, Ian C. Covert, Scott M. Lundberg, and Su-In Lec. Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence, 5(6): 601, may 2023. ISSN 2522-5839. doi: 10. URL https://wwd http://wwwuuuuureroooocoorataaaa
1038//442556-033-06666
Minhao Cheng, Thong Le, Pin-Yo Yu Chen, Huan Zhang, JinFeng Yi, and Cho-Jui Hsieh. Query-Efficient Hard- label Black-box Attack: An Optimization-base Approach. September 2019. URL https::/opennreiie:
net/formmmPiderrJJGiRRg..
Youngduck Choi, Chill Yi-I. Chiu, and David Sontag. Learning Low-Dimensional Representations of Medical Concepts. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Transla- tional Science, 2016:41-50. 2016. ISSN 2153-4063.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yamping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instruction-Finettuee Language Models. Journal of Machine Learning Research, 25(7)): 1-53, 2024. ISSN 1533-7928. URL http://jmmt
wwtg/papers/v25/23-000 0870.html.
Jan Clusmann, Fiona R. Kolbinger, Hannah Sophie Muti, Zunamys Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P. Veldhuizen, Sophia J. Wagner, and Jakob Nikolas Kather. The future landscape of large language models in medicine. Communications Medicine, 3(1):141, October 2023. ISSN 2730-664X. doi: 10.
URL /tttps://ww...e nature. http://wwwwaaar.ooooooooooaoaaaa
A. Davis, C. G. Murphy, C. A. Saraceni-Richards. M. C. Rosenstein, C. Wiegers, and C. J. Mattingly. Comparative Toxicogenomics Database: a knowledgebase and discovery tool for chemical-gene-dieees networks. Nucleic Acids Research, D777 jan 2009. ISSN 0305-1048. doi: 10.1093/ nar/gkn580. URL https: htttp:://acadmmiee c. oup http:////////aataaaaooootto///// http://ee..e..eeo//o/0/0/0/////0
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit Inference Scaling Laws. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 7770-7774. PMLR, July 2023. URL httpss:/pproceedings mlr. wwt.:pprssss/eeeeeeeeee http://2aatmmeeaa..... html.
John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal Rates for Zero- Order Convex Optimization: The Power of Two Function Evaluations. IEEE Transactions on Information Theory, (1()):2788 2806, May 2015. ISSN 0018-9448, 1557-9654. doi: 10.1109 /IT TTT110/TTT..111.22221. 2015.24925.. URL http://ieeexplore. ieee. org/ /dtcuummett/055527/..
Salijona Dyrmishi, Salah Ghamizi, and Maxime Cordy. How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks. In Anna Rogers, Jor- dan Bryderr, and Naoaki Okazaki (ods.), Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 8822 8836, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: /18853//11/023.acll/ 491. URL https: /
wtt.://////..-11ee... 491.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transac- tions of the Association for Computational Linguistics, 9:1012-1033 1031, dec 2021. ISSN 2307-387X. doi:
16

---

Published in Transactions on Machine Learning Research (12/2024)
00.1162/tal_aa_0001 a 107384/Measuring and- and- htwpa///mioiigggoooooaaaaaaaaaaa
URL http:////ttctc.....aaaaaaaaaaaaa
Wee Chung Gan and Hwee Tou Ng. Improving the Robust of Question Answering Systems to Question Paraphrasing, In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6065 6075, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19-1610. URL https://wwat aclweb........... org/ org/annhoooogy/111 1611111100
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50-56. IEEE, may 2018. ISBN 978-1-5386-8276-0. doi: 251100/SPW22088000000 URL https: //ieeexplore. ieee. wet..e.eeeeeeeeeeeeeeeeeeeeeeeee ortp///cummettt/4426////
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. Question answering is format; when it useful?, September 2019. URL /ttp:://axxvvvvie org/ wwwp/ogg///////oo//eeee/e///// 1909. 11291. arXiv:1909.1120 [cs].
Tianyu Han, Lisa C. Adams, lens-Michalls Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löscr, Danicl Truhn, and Keno K. Bressem. MedAlpaca - An Open-Source Collection of Medical Con- versational AI Models and Training Data, October 2023. URL http: //axxiv www.aaiwww/aa.aaaaaaeoeaaeeooore wwgg/rag////////ee/eoo/ee ab3//344. 082477 arXiv:2304.0224 [cs).
Leif Hancox-Li. Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, pp. 640 647, New York, NY, USA, January 2020. ACM. ISBN 10. 11 1145/33511995 w3w.83511111..38866...
doi: 10. 10.1145/33310995337728
URL attps://di acm. org/
Andrew Iyass Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box Adversarial Attacks with Limited Queries and Information. In Proceedings of the 35th International Conference Machine Learning, pp. 2137 2146. PMLR, July 2018. URL https://proceedingg mll /tt.e/edddnna.......eee/////////
html.
Robin Jia and Percy Liang. Adversarial Examples for Exaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirioul Methods in Natural Language Processing, pp. 2021 2031, Stroudsburg, PA, USA, 2017. Association for Computational Linguistics. doi: 10.18863/v1/DI7-12155
URL http:://acl/ae wat...rggg.oo.. anthology/D17-1115
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Datasct from Medical Exams. Ayplied Sciences, 11(14):6421, jul 2021. ISSN 2076-3417. doi: 10.3390/app111466/ URL https://www.
Kiran Kamble and Waseem Alshikh. Palmyra-med: Instruction-based fine-tuning of llms enhancing medical domain performance, July 2023.
Aneta Koleva, Martin Ringsquandl, and Volker Tresp. Adversarial Attacks on Tables with Entity Swap. In Joint Proceedings of Workshops at the 49th International Conference on Very Large Data Bases (VLDB 2023), Vancouver, Conado, volume 3462 of CEUR Workshop Proceedings, 2023. URL https://ceur-ws: /e/VVoo-36622//
pdf.
Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng, Performance of ChatGPT on USMLE: Potential for Al-assisted medical education using large language models. PLOS Digital Health, 20200e00,, feb 2023. ISSN 2767-3170. doi: 10.1371/journal pdip.alllp...000000000000 0000198. URL https: org/ 10. 1 www.rnaal..di.....m 0000198.
Emanuele La Malfa, Min Wu, Luca Laurenti, Benjie Wang, Anthony Hartshorn, and Marta Kwiatkowska. Assessing Robustness of Text Classification through Maximal Safe Radius Computation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Compulational Linguisties: EMNLP 2020,
17

---

Published in Transactions on Machine Learning Research (12/2024)
pp. Online, November 2020. Association for Computational Linguisties. doi: 10.18653/v1/
2020.finding findings-emnlp.266 URL httpss///aclannhoooggg.. org/2020. Sindings-emmnn e.temmmmll...... 266.
Pierre L'Ecuyer and George Yin. Budget-Dependent Convergence Rate of Stochastic Approximation. SIAM Journal on Optimization, 8(1):217 247, 1998. doi:
URL httpp:
siam. hawm.oorg////o///ee 017.111117///0000224477773333 Publisher: Society for Industrial and Applied Mathematics.
Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. Do We Still Need Clinical Language Models? In Bobak J Mortazavi, Tasmie Sarker, Andrew Beam, and Joyce C Ho (eds.), Proceedings of the Conference on Health, Inference, and Learning, volume 209 Proceedings of Machine Learning Research, pp. 578 597. PMLR, 2023. URL tttps://proceedingg mlr wtt.ppeseee//eeeeseesseee html.
Xinzhe Li, Ming Liu, Shang Gao, and Wray Buntine. A Survey on Out-of-Distribution Evaluation of Neural NLP Models. In Proceedings of the Thárty-Second International Joint Conference Artificial Intelligence, pp. 6683-6691, California, aug 2023a. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-956692--0-- doi: 10.24963 ijeai.20233.. URL https: h/wp:://wwww http://wwwww/j////o//// ijcai. org/ proceedingss/2223777
Zehan Li, Xin Zhang. Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards General Text Embeddings with Multi-stage Contrastive Learning, August 2023b. URL http://arxits org/ abs/ 2308. 03281. arXiv:2308.0321 032281 [cs].
org/abs/
Chris Lin, lan Covert, and Su-In Lee. On the Robustness of Removal-Based Feature At- tributions. Advances in Newral Information Processing Systems, 11-79613-79666 De- cember 2023.
neurips. wewrips.cccppprrffiiee..... Siapperrflees/pppee/22////// 1b04 wwwp:adtt---bataat----ttoo Conferenne html.
URL
Aiwei Liu, Honghai Yu, Xuming Hu, Shu'ang Li, Li Lin, Fukun Ma, Yawen Yang, and Lijie Wen. Character- level White-Box Adversarial Attacks against. Transformers via Attachable Subwords Substitution. Tn Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7664 7676, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2000 emnlp- m2in.522. URL https://aaae https////aaaantoo.....oom org/2022. ewnll-main main.52n 522.
Siru Liu, Aileen P Wright, Barron L Patterson, Jonathan P Wanderer, Robert W Turer, Scott D Nelson, Allison B McCoy, Dean F Sittig, and Adam Wright. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. Journal of the American Medical Informalics Association, 30(7): 1237-1245, July 2023. ISSN 1527-974X. doi: 11993/jamia/ocaaa jamia/ocad072.
URL https://dor httt:://doi.org/1..... 0rg/10.1093/
Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3):100943,, March 2024. ISSN 2666- URL
3899. doi: 10.1016/j.patter. 2024.100943. pi1/998663892400424
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- Based Knowledge Conflicts in Question Answering. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7052 7063, Stroudsburg, PA, USA, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 we002..mmlppmae mainlpmminn565, URL https://aclanthologg org/
2021. emnlp-ma hta.ppmmai..6.....
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. A Strong Bascline for Query Efficient Attacks in a Black Box Setting. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scoll Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empérical Methods in Natural Language Processing, pp. 8396-8409, Online and Punta Cana, Dominican Republic, November 2021. Association for Compu- tational Linguistics. doi: 10.18653/vv (2221.eemmjyy main.mi........mm.a URL https://aclanthology org/2021. emnlp-main.661.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Danaë Metaxa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. Auditing Algorithms: Understanding Algorithmic Systems from the Outside In. Foundations and Trends in Human Computer Interaction, 14(4):272 344, 2021. ISSN 1551-3955. doi:
10.1561/1100000083. URL http: http://www. httppp/iiiaass..c//rrroooo////ee
Sunil Mohan and Donghui Li. MedMentions: A Large Biomedical Corpus Annolated with UMLS Concepts. In Automated Knowledge Base Construction (AKBC), 2019. URL SylxCx5pTQ.
Seyed-Mohsen Ghoosav-Deezfolle Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal Adversarial Perturbations. In 2017 TEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 86- 94. IEEE, jul 2017. ISBN 978-1-5386-0457- doi: /..1199/CVP..211.117 URL http://ieeexplore. leee. orgee.rg/cee httg//dcuumen//80950000
John Morris, Eli Lifland, Jack Lanchantin, Yangleng Ji, and Yanjun Qi. Reevaluating Adversarial Examples in Natural Language. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3829 3839, Online, November 2020a. Association for Computational Linguisties. doi: 10.18653/vll 203653/v1/202/ //2.//...nnn..dinggg-- org/2020. f indings-emnlp.30
URL https://aclantthllogn
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. In Qun Liu and David Schlangen (eds.); Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing: System Demonstrations, pp. 119-126, Online, October 2020b. Association for Computational Linguis- tics. doi: 10.18653/v http///122200m///// 202000eemmppddmmosssee.. URL https: hatpaa//aalaaahologo.oo wttphoollogyyog/2200000000 emnlp-demos .16.
Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis Griffin. Frequency-Guided Word Sub- stitutions for Detecting Textual Adversarial Examples. In Proceedings of the 16th. Conference of the European Chapter of the Association for Compulational Linguisties: Main Volume, pp. 171 186, Strouds- burg, PA, USA, 2021. Association for Computational Linguistics. doi: 2.11865/w1/20211 ma........... URL
org/2021, wttg//221eecc-mmmeeee eacl-main. 13.
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina M. Barah-Ba Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Counter-fitting Word Vectors to Linguis- tic Constraints. In Proceedings of the 2016 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 142-148, Stroudsburg, PA, USA, 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16 1018. //aclweb. org/ htt///ahooooo//11111000
URL http:
W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116 (44):22071 22080, oct 2019. ISSN 0027-8424. doi: 10.1073/ h..10733pnas.190000 190064499 URL attps://pnan org/ 10tp://///11111///ppaaaa 1900654116.
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Neha Gupta, and Sanjiv Kumar. Learning to Reject Meets Long-tail Learning. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL https://openrest
het//forumiiddtt22LLlll22
Yurii Nesterov and Vladimir Spokoiny. Random Gradient-Free Minimization of Convex Functions. Foun- dations of Computational Mathematics, 17()):527-566, April 2017. ISSN 1615-3383. doi: 10.1007/ s10206-015-9296-2 URL https:///ois http:/oo.o.rr.....000//000000000 0000000000000008-----9------
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 327, Stroudsburg, PA, USA, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ W19-5034. URL https://wwet www.:://wwwwwwwbbb...e.......... org/ wtg/anthollog//W1-5500.
19

---

Published in Transactions on Machine Learning Research (12/2024)
Maximilian Noppel and Christian Wressnegger. SoK: Explainable Machine Learning in Adversarial Envi- ronments. Tn 2024 TEEE Symposium on Securily and Privacy (SP), pp. 21. IEEE Computer Society, oct 2024. ISBN 9798350331301. doi: 10.1109/SP542633. 2024444022222220022.22. URL https: //www//wwwwwww computer
wwwp://ttttt..eee.r.mmommmmm//.. org/csdl/
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can Generalist Founda- tion Models Outcompete Special- Tuning? Case Study in Medicine. arXiv, pp. 2311. 1645,, nov 2023. URL http: http//aaxivvv..///e//eeeeeeeeel/ wwg.iii.orrgaeoo wttg/aa////11.....////g/ 16452.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: A Large-scale Multi- Subject Multi-Choice Dalaset for Medical domain Question Answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), Proceedings of the Conference Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248 260. PMLR, 2022. URL https: httpoce/pproceediggs mlr. wel.r.pres//vve pptps//eee/ppp//eeeee 2222aa..
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples. arXiv, pp. 1605.07277, may 2016. URL http://arxiv.. wrgg/ba///0..000
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language Models as Knowledge Bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 2463 2473, Stroudsburg, PA, USA, 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1/D1S 1250. URL https: //ww///wwwwco www.ww.wlw..... org/ /tt////toooogg/1-11200
Pouya Pezeshkpour and Estevam Hruschka. Large Language Models Sensitivity to The Order of Options in Multiple-Choite Questions. arXiu, pp. 2308.11483, aug 2023. URL http: http://wwwvw......./ooo/oo
11483.
C.Radhakrishna Rao. Diversity and dissimilarity coefficients: A unified approach. Theorelical Population URL https://
Biology, 21(1):24-43, feb 1982. ISSN 00405809. doi: 11100100000500002222000000 linkinghub. www.eeie..cooeeeeeeeeeeeeeeeeeee http://aw....eeeeeeee////////a00
Nils Reimers and Iryna Gurevych. Sentence-BEil BERT: Sentence Embeddings using Siamese BERT-Netwan Tn Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empèrical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 D19 1410. URL sttps://aclantthoopgg org/D19-1410
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency. In Proceedings of the 57th Annual Meeting of the Associa- tion for Compulational Linguisties, pp. 1085-1097 Stroudsburg, PA, USA, 2019. Association for Compu- tational Linguistics. doi: 10.18653/v1/P10 1103. URL https://wwa aclweb. org/ http//ahhoooog/P1-1000
Kyle Richardson and Ashish Sabharwal. What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge. Transactions of the Association for Computational Linguistics, 8:572-588, dec 2020. ISSN 2307-387X. doi: 10.1162/taa
mit. 00331. URL https: http::///rret..tt..essaaaaaaaraa wdt./acl/aariice///65aa
Joshua Robinson and David Wingate. Loveraging Large Language Modols for Multiple Choice Question A Answering. In The Eleventh International Conference on Leurning Represenlations, 2023. URL https:
//openreview. net/ net:/torrmm?ddeeyXpppaaaacc
Omid Rohanian, Mohammadmahdi Nouriborji, and David A. Clifton. Exploring the Effoctiveness of In- struction Tuning in Biomedical Language Processing, December 2024. URL attp://arxiv.org/aat arXiv:2401.0057 [cs).
20

---

Published in Transactions on Machine Learning Research (12/2024)
Tom Roth, Yansong Gao, Alsharif Abuadbba, Surya Nepal, and Wei Liu. Token-modification adversar- ial attacks for natural language processing: A survey. AT Communications, Pre-press:1-2 Jamuary 2024. ISSN 0921-7126. ai www.commicaaiion/aiiiooomo
doi: 10.3233/AIC-2000
URL Publisher: IOS Press.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulezyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yes Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, Daniel McDuff, Le Hou, Tomer Golany, Luyang Liu, Jcan-baptiste Alayrac, Neil Houlsby, Nenad Tomasev, Jan Freyberg, Charles Lau, Jonas Kemp, Jeremy Lai, Shekoofeh Azizi, Kimberly Kanada, SiWai Man, Kavila Kulkarni, Ruoxi Sun, Siamak Shakeri, Luheng He, Ben Caine, Albert Webson, Natasha Latysheva, Melvin Johnson, Philip Mansfield, Jian Lu, Ehud Rivlin, Jesper Anderson, Bradley Green, Rence Wong, Jonathan Krause, Jonathon Shlens, Ewa Dominowska, S. M. Ali Eslami, Katherine Chou, Claire Cui, Oriol Vinyals, Koray Kavukcuoglu, James Manyika, Jeff Dean, Demis Hassabis, Yossi Matias, Dale Webster, Joelle Barral, Greg Corrado, Christopher Semturs, S. Sara Mahdavi, Juraj Gottweis, Alan Karthikesalingam. and Vivek Natarajan. Capabilities of Gemini Models in Medicine, May 2024. URL http://arxx: org/ 18416. arXiv:2404.18116 [cs].
Priyanka Sen and Amir Saffari. What do Models Learn from Question Answering Datasets? In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2429 2438, Stroudsburg, PA, USA, 2020. Association for Computational Linguisties. doi: 10.18653/v1/ 9020.emnlp-main.1 190. URL https: http:://wwwwwlclc.. acw.eb. orgganthology//0000 emnlp-main. 190.
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks, October 2023. URL http://arxiv. org/ /2300. 10844. arXiv:2310. 10844 [cs].
Satya Narayan Shukla, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Simple and Efficient Hard Label Black-box Adversarial Atlacks in Low Query Budget Regimes. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, pp. 1461-1469, New York, NY, USA, August 2021. Association for Computing Machinery. ISBN 978-1-4503-8833-5. doi: 10.1145/3447548. 3467386. URL httpp:///don orto/0....g01... 1145/3447548. 2555/34447558..34735..
Chengici Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting GPT-3 To Be Reliable. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviie www.erreiie..eeeeorruraaaaaaaaae
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Coles, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gam ble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman Blaise Agüera Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomascv, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Sem- turs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowl- edge. Nature, 620(7972) 180, aug 2023. ISSN 0028-0836. doi: 100.1/////1111000--000000000 http://arxiv.on orw.ii..rrg/aaa. abs/ 2212..222222222 13138https: http://wwwwwww///.ee////ee nature. wttt:/ttcccccccc////eeeooccacctt 2e0.lle0///11580002-022020
URL
Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabricl Ccrono, Yongmei Shi, Angela Rizk-Jackson, Sharat Israni, Charlotte A Nelson, Sui Huang, and Sergio E Baranzini. Biomedical knowledge graph-enhanced prompt generation for large language models. arXiv, pp. 2311,, nov 2023. URL http: http///xxivv.rrr haagg//bs/22000 17330.
Giovanni Spitale, Nikola Biller- Andorno, and Federico Germani. Al model GPT-3 (dis)informs better than humans. Science Advances, 9(26), jun 2023. ISSN 2375-2548. doi: 10.1126 111/////cidv..dl// adh1850. URL htttp://www.scieee http://ceence.rg/ttoo//////o///t 1tt11/.1.11.1/aaarr.// 1a11////iiadv..a11855....
Mujeen Sung, Jinhyuk Lee, Scan Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. Can Language Models be Biomedical Knowledge Bases? In Proceedings of the 2021 Conference on Empirical Methods in Natural
21

---

Published in Transactions on Machine Learning Research (12/2024)
Language Processing, pp. 4734, Stroudsburg, PA, USA, 2021. Association for Computational Linguis- tics. doi: 00.18653/v1/2022 http://e.2221111mmmmaaa manlp-main.388. URL https://aclantthol00 000////0ooooooooo.//0000...1111 emnee-maaaaa....eee main ..in.3888.
org/2021.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations. 2022. URL attps://openreview nwt.toormmiddrrru//eeeo
Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Fine-tuning large neural language models for biomedical natural language processing, Pallerns, 4(4):100729, apr 2023. ISSN 26663899. doi: 10.1016/j patte.. patterr.023.111.... 2023.10072,
URL https://
Hinkinghub.elseee elsevier....... htt.a//ewe...eeeeeeeeeee/e//oo/o
Oleg Ursu, Jayme Holmes, Jeffrey Knockel, Cristian G. Bologa, Jeremy J. Yang, Stephen L. Mathias, Stuart J. Nelson, and Tudor Oprca. DrugCentral: online drug compendium. Nucleic Acids Research, 45(D1):D932 D939, jan 2017. ISSN 0305-1048, doi: 10.1093/nar/ /ar/gkw99/3. URL attps://academia oup.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner and Sameer Singh. Universal Adversarial Trig- gers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pp. 2153 2162, Stroudsburg, PA, USA, 2019a. Association for Computational Linguistics. doi: 10. 10.18653/v1 ///////11////// 1111111-112001 1221. URL https: //www. aclweb. wrg/anthologgyD19-222
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick Me If You Can Human-in-the-LLoop Generation of Adversarial Examples for Question Answering. Transactions of the Association for Computational Lènguistics, 7:387-400 401, nov 2019b. ISSN 2307-387X. doi: 10.1162/tacl
a 00279. URL https: /dirrect/////c//////// mit.
Benyou Wang. Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. Pre-trained Language Models in Biomedical Domain: A Systematic Survey. ACM Computing Surveys, 56(3):55:1 55:52, October 2023a. ISSN 0360-0300. doi: 10.1145/3611651. URL https://ddo owg/o0..///. 10./1...11 114.01615/33616.....
Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Bastegar-Mojarte Liwei Wang, Feichen Shen, Paul Kings- bury, and Honglang Liu. A comparison of word embeddings for the biomedical natural language processing. Journal of Biomedical Informatics, 87:12 20, November 2018. ISSN 1532-0464. doi: 10.1016/j.jbi.201. 09.008. URL https: //wp:////wwwwa/a
Yubo Wang, Xueguang Ma, and Wenhu Chen. Augmenting Black-box LLMs with Medical Text- books for Clinical Question Answering, September 2023b. URL http://arxiv, http:/awwivv..or/aae atg/abs/23099000 arXiv:2309.0222 0222222339 [cs].
Wei, Alexis Allot, Robert Leaman, and Zhiyong Lu. PubTator central: automated concept anno- tation for biomedical full text. articles. Nucleic Acids Research, (WW1).W5877W550, jul 2019. ISSN 0305- 1048. doi: 10.1093/nar/ /a/gg2339. URL attps://acader:: oup
Gerhard Weikum, Xin Luna Dong, Simon Razniewski, and Fabian Suchanek. Machine Knowledge: Creation and Curation of Comprchensive Knowledge Bascs. Foundations and TrendsS in Databases, 10(2-4):108- 490, 2021. ISSN 1931-7883. doi: 10.1561 /1900000064 URL http:/ /ttp::/www..r www...oobbbsssesscooo..o. Details/DBBS-04.
Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A. Pfeffer, Jason Fries, and Nigam H. Shah. The shaky foundations of large language models and foundation models for electronic health records. upj Digital Medicine, 6(1):135, jul 2023. ISSN 2398-6352. doi: 10.1038/ $41746-023-00879.5 URL https://www www.nnauuue....
Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2859-288 2867, October 2017. doi: D0.1109/TCCV.2011 URL https://ieeexpporee wewpplerreeeeeee.eeeeeeeeeeeeeee ieee.. htt.eeeaaeer..a.eeeeee/////////e 88237777
22

---

Published in Transactions on Machine Learning Research (12/2024)
Jun Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, and Xiang Ren. On the Robustness of Reading Comprehension Models to Entity Renaming. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 508-520, 520, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/2022 /2022 naacl-main.31 URL https: : http:p:s//acaaathhloio
wtt. naacl-main,
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael I Jordan. Greedy Attack and Gum- bel Attack: Generating Adversarial Examples for Discrete Data. Journal of Machine Learning Research, 21443)11-36, 2020. URL http://jann
httpppppprss/v/// 569.6699...0 html.
Zhen Yu, Zhenhua Chen, and Kun He. Query-Efficient Textual Adversarial Example Generation for Black- Box Attacks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp- 556-569, Mexico City, Mexico, June 2024. Association for Com- putational Linguistics. doi: 10.18653 /024 naacl-lng.31. URL httpa://aclanthologyy .or.ioyy..oooooooooooooooo org/2024. naacl-long.32 31.
Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. CODER: Knowledge-infused cross-lingual medical term embedding for term normalivation. Journal of Biomedicall Informalics, 126: 103983, feb 2022. ISSN 15320464. doi: 10.1016/j..i..2211... 2021. URL https: heles:///ikkinnhue elsevier.
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial Attacks on Deep-learning Models in Natural Language Processing. ACM Transactions on Intelligenl Systems and Technology, 11 (3):1-41, jun 2020. ISSN 2157-6904. 3374217.
doi: 00.1145/3374211 URL https : //dl//dll//// acm. www.aaa/c.ooooo......... org//oi/10..ooooooo 1145/
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A Survey of Large Language Models. arxi pp. 2303.18223, mar 2023. URL http: http///xxvvvv/////ee//ee abs//3//../...../ 18223.
Chujie Zheng, Hao Zhou, Fandong Meng Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference on Learning Representations, 2024. URL attps://openrevieee www.rrewwwwwwweeeeeeemmeeeeeee shwr
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Frodrikson. Universal and Transferable Advensarial Attacks on Aligned Language Models, December 2023. URL org/abs/2300 15043. arXiv:2307.15045 [es].
23


Reviews:
[{"review_id": "lDe0rTq0uh", "reviewer": "Reviewer_rYjV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- The research problem of evaluating the domain knowledge of large language models (LLMs) is highly significant.\n- The proposed method is query efficiency and can be scaled to larger model and benchmark.\n- The evaluation covers both sampling-based and search-based attack methods.\n\n\n## Weakness\n\n- The implementation of cosine distance in the NE model requires further discussion. What does cosine similarity of model embeddings represent in the biomedical context? Does a small distance reflect similar drug properties, or just token similarity (e.g., `Zolmitriptan` vs `Zolpidem`)? One way to investigate this is by evaluating the sentence similarity of drugs and disease descriptions.\n- The validity of adversarial distractors should be examined. Based on the results in Table 1, the PWS approach perform better when n < 0 in most cases, suggesting that the adversarial distractors are very close to the true answers in embedding space. This raises the possibility that these distractors could also be true.  The authors should provide a justification for this and conduct additional experiments.\n- Only few models are tested. The tested Flan series models are relatively weaker and do not follow the mainstream decoder-only architecture. Testing more powerful LLMs, such as LLaMA3, GPT or BLM in the leaderboard [1], would benefit the community and increase the impact of this work.\n- The work would benefit from including a defensive method that could enhance the robustness of BLMs against entity substitution attacks.\n\n\n## Ref\n[1] https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work propose a entity substitution attack to examine the vulnerabilities of biomedical LLMs. \nIt proposed an embedding-distance-weighted sampling (DWS) method to generate adversarial distractors, which is query efficiency and scalable.  \nExperiments demonstrate the effectiveness of DWS and the brittleness of biomedical knowledge of LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More powerful and decoder-only model should be evaluated, at least ChatGPT and a powerful BLM in the leaderboard.\n- Add the investigation and explaination of cosine distance in the biomedical context.\n- A justification or additional experiments about the validity of adversarial distractors. \n- It's better to provide a defensive method against the attack.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cuyyDIj18O", "reviewer": "Reviewer_HHFG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. TCES and PDWS for adversarial attacks on LLMs address the pressing need for model vulnerability assessment in specialized domains. TCES ensures entity consistency in biomedical contexts by maintaining semantic coherence, while PDWS efficiently explores embedding space to identify vulnerabilities with low query budgets. \n\n2. The proposed gradient-free method for adversarial entity generation improves attack success rate with a low-query-budget setting. This efficiency is particularly relevant for practical robustness evaluations of LLMs.\n\n3. The paper provides a rigorous experimental setup, evaluating multiple LLMs, such as Flan-UL2 and Palmyra-Med, across various biomedical datasets using metrics such as attack success rate (ASR), query efficiency, and model accuracy before and after adversarial perturbations. \n\n4. The findings carry significant implications for the deployment of LLMs in high-stakes applications, such as healthcare, where model robustness and reliability are paramount.\n\n*I am not familiar with entity substitution attacks in LLMs, but I hope to provide objective feedback on the paper's weaknesses and suggestions to improve the argumentation and presentation. If my suggestions are impractical or incorrect, I welcome discussion and corrections.*\n\n**Weaknesses**:\n\n1. This paper introduces the innovative PDWS, but the connection between DiscreteZOO in Sec. 3.3 and PDWS is not clear. Does the paper make any improvements to DiscreteZOO, or is it simply presented as an alternative and baseline to PDWS? Could DiscreteZOO be combined with PDWS to achieve better results?\n2. The experiments only compare PDWS to random sampling. I suggest including two additional baselines related to PDWS: the 'farthest' and 'closest' methods used in [1], as they represent two extreme cases of $n$. This would effectively demonstrate the advantages of distance-based sampling.\n3. I am curious why beam search and greedy search are not included as baseline methods for comparison. These methods are often known for high attack success rates, albeit at the cost of high query budgets.\n4. One of the key advantages of the proposed TCES framework is that type-consistent substitutions make perturbed text harder to detect. While this point is stated in the paper, there is no empirical evidence to support it. I suggest providing verification experiments, such as calculating BERTScore[2] for tokens or questions before and after perturbation, to strengthen this argument.\n\n**Reference**:\n\n[1]: Berger, N., Riezler, S., Sokolov, A., & Ebert, S. (2021). Don’t Search for a Search Method — Simple Heuristics Suffice for Adversarial Text Attacks. ArXiv, abs/2109.07926.\n\n[2]: Zhang, X., Zhao, J.J., & LeCun, Y. (2015). Character-level Convolutional Networks for Text Classification. Neural Information Processing Systems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the robustness of biomedical knowledge in large language models (LLMs) by using a novel query-efficient sampling approach to generate adversarial entities. Specifically, it introduces type-consistent entity substitution (TCES) to collect adversarial examples, leveraging a gradient-free, embedding-space method called powerscaled distance-weighted sampling (PDWS). The paper makes a significant contribution by demonstrating the susceptibility of both pre-trained generalist and fine-tuned domain-specific LLMs to adversarial entity attacks in biomedical question-answering (QA) tasks. Moreover, the research reveals two regimes of adversarial entities (near and far in embedding space) and demonstrates how entity substitutions can effectively deceive token-wise Shapley value explanations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would encourage the authors to address the “Weaknesses” listed above.", "broader_impact_concerns": "No concerns in this regard.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lkvmnHGnfy", "reviewer": "Reviewer_D6wa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- Hyperparameters of the presented methods are evaluated extensively \n- The paper makes precise claims with suitable experiments to back them up \n- The authors introduce practical attacks with a reasonable query budget\n- The method presented by the authors is suitable for generating difficult benchmark datasets in the biomedical Q&A domain, enabling researchers to investigate robust prediction methods\n\n**Weaknesses:*\n\n- The authors do not assess if recently proposed defenses to adversarial attacks in the LLM domain would improve the robustness of these models (However, I acknowledge that this might be out-of-scope of this work and do not expect the authors to provide such an experiment)\n- A white-box evaluation would have been interesting to provide some lower bound on the robustness of the models used in the study. I would expect 100% ASR to be possible based on results in the literature in other domains.\n-  The authors provide a method to investigate the robustness of LLMs to entity substitution. I believe the research community would have benefited from a \"fixed dataset\" with precomputed attacks as a sort of \"difficult\" benchmark in this domain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the adversarial robustness of LLMs in the biomedical domain to entity substitution. \n- For this purpose, they adapt distance weight sampling to a sampling-based attack and additionally investigate an attack using zeroth order optimization. \n- They demonstrate that currently available LLMs are vulnerable to adversarial entity substitutions and show that both attack types achieve higher attack success rates than random sampling (while still using a small query budget). \n- They investigate connections between the adversarial vulnerability of the models and the model size and the effect of adversarial entities on model explainability", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors provide one concrete example for the different variables in equation 3.1?\n- I did not find a formal definition of $e$. In general, the paper is missing formal definitions of variables in some places (including their dimensionality, etc.). \n- What is the original text span - $e_0$ can be understood as the embedding of the original text? \n- Is eq (5) normally used in an iterative setting? Maybe indicate in the equation\n- Would the authors consider open-sourcing an adversarial version of the base dataset as a “hard” \nbenchmark for biomedical Q&A (similar to ImageNet-A [1]). I believe this would be an interesting benchmark to measure the robustness of LLMs in pseudo-adversarial settings.\n\n[1] Hendrycks et al., \"Natural Adversarial Examples\" 2021", "broader_impact_concerns": "I do not have broader impact concerns. The presented attack methods are valuable in studying the robustness of LLMs. However, I do not see a practical risk of such attacks in deployment scenarios.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously agree that the paper is ready to be accepted.
Decision: Accept (poster)