Adversarial Robustness for Large Language NER models using Disentanglement and Word Attributions
Authors: Xiaomeng Jin
Bhanukiran Vinzamuri
Sriram Venkatapathy
Heng Ji
Pradeep Natarajan
Venue: EMNLP 2023 Findings (2023)
Keywords: Adversarial Robustness
Adversarial Attacks
Named Entity Recognition
Large Language Models
TL;DR: To enhance further adoption of LLM NER models, we conduct rigorous adversarial training using diverse examples synthesized from our technique combining disentanglement and word attributions.

Abstract: Large language models (LLM's) have been widely used for several applications such as question answering, text classification and clustering. While the preliminary results across the aforementioned tasks looks promising, recent work has dived deep into LLM's performing poorly for complex Named Entity Recognition (NER) tasks in comparison to fine-tuned pre-trained language models (PLM's). To enhance wider adoption of LLM's, our paper investigates the robustness of such LLM NER models and its instruction fine-tuned variants to adversarial attacks. In particular, we propose a novel attack which relies on disentanglement and word attribution techniques where the former aids in learning an embedding capturing both entity and non-entity influences separately, and the latter aids in identifying important words across both components. This is in stark contrast to most techniques which primarily leverage non-entity words for perturbations limiting the space being explored to synthesize effective adversarial examples. Adversarial training results based on our method improves the F1 score over original LLM NER model by 8% and 18% on CoNLL-2003 and Ontonotes 5.0 datasets respectively.

Adversarial Robustness for Large Language NER models using Disentanglement and Word Attributions
Xiaomeng Jin
Bhanukiran Vinzamuri Sriram Venkatapathy
Heng Ji Pradeep Natarajan'
University of Illinois, Urbana Champaign +Amazon Alexa A1
xjin17@illinois edu [vinzamub, vesriram, jihj, natarap]@amazzn.c com
Abstract
on NER tasks. However, these models tend hallucinate with even minor modifications in the in- put (Wang et al., 2023b). Recently, there has been a growing interest in developing adversarial attack- based techniques for NER models (Simoncini and Spanakis, 2021; Lin et al., 2021) to enhance NER models using word-level attacks. This is even more relevant in the context of LLM's, ss very recent study (Zhu al., 2023) demonstrated the lack of bustness in current LLM's with word-level attacks resulting in a significant performance drop 33%, However, despite resulting in a successful attack, many of the perturbed word-level attack candidates fail to qualify as adversarial examples, i.e., they are not semantically similar to original sentences as depicted in Figure 1.
Large language models (LLM's) have been widely used for several applications such as question answering. text classification and clus-
tering. While the preliminary results across the aforementioned tasks looks promising. re- cent work (Qin et al., 2023; Wang et al., 2023a) has dived deep into LLM's performing poorly for complex Named Entity Recognition (NER) tasks comparison to fine-tuned pre-trained language models (PLM's). To enhance wider adoption of L.MM's, our paper investigates the robustness of such LLM NER models and its instruction fine-tuned variants adversarial attacks. In particular, we propose a novel at- tack which relies on disentanglement and word attribution techniques where the former aids learning an embedding capturing both en- tity and non-entity influences separately, and the latter aids in identifying important words across components. This stark con- trast most techniques which primarily lever- age non-entity words for perturbations limiting the space being explored synthesize effec- tive adversarial examples. Adversarial training results based on our method improves the F1 score over original L.M NER model by 8% and 18% on CoNLL-2003 and Ontonotes 5.0 datasets respectively.
In addition, in resource constrained settings, it isn't 't frugal to explore the entire space of feasible word-level perturbations to devise good adversar- ial examples. This justifies the necessity of attack techniques which can explore the diverse space ef- ficiently. In our paper, this is mainly accomplished using two key levers, namely, disentanglement and word attribution techniques, respectively. Disen- tanglement (Higgins et al., 2017) is a technique which helps in separating the latent entity and con- text components of embedding space (Figure 2), making more congenial for word attribution function like Integrated Gradients (IG) (Sundarara- jan et al., 2017) to identify diverse, yet important words. The other subsequent steps include substi- tution of the selected words with entity and context substitution workflows (Figure 4), and selection of candidate adversarial examples based on their semantic similarity scores to the original text.
1 Introduction
Named Entity Recognition (NER) aims to iden- tify and categorize named entities mentioned in unstructured text into pre-defined categories such Person, Location, or Organization. In recent years, NER tasks (Malmasi et al., 2022) have be- come more challenging due the introduction of complex tagsets, which often leads to the failure existing NER systems accurately recognizing these entities. To address this, Large Language Models (LLMs) have emerged as powerful tools, delivering significant performance improvements
Experimental results indicate that to create successful attack, on average, our method quires 69% (details presented in Appendix A.1) lesser candidate adversarial sample generation than a state-of-the-ar technique like CLARE (Li et al., 2021) when evaluated on three popular
Work done while interning at Amazon Alexa Al

---

The main contributions in this paper are, datasets (CoNLL-2003 (Sang and De Meulder, 2003), Ontonotes 5.0 (Weischedel et al.. 2013), and MultiCoNER (Malmasi et al., 2022)). The re- sults also demonstrate that our method improves the F1 score over original BERT NER model by 8% and 18% on CoNLL-2003 and Ontonotes 5.0 re- spectively. Furthermore, in intrinsic evaluation of our adversarial examples generation approach CoNIL-22003, achieve a 10% higher attack success rate (percentage of generated adversarial examples causing label flip) at comparable modi- fication rate. In addition, our method successfully attacks theinstruction fine-tuned T5 NER model (Wang et al., 2022) on the MultiCoNER dataset.. resulting in 10% drop in the Fl score after the attack.
Grggnsl
*
One the hotoraabee seyondlgam rade Zhou Hothab...
Ofee homeud canth
the chansinite aorander
Figure 1: A PCA plot of an original sentence and the generated adversarial examples using different attack methods. The original sentence One of the two honor- able guests in the studio is Professor Zhou Hanhua from the Institute of Law of the Chinese Academy of Social Sciences., represented with the * are the adversarial examples from our method, and the X markers repre- sent the adversaria examples from RockNER. Across sentences we only retained modified spans and exclude identical content as compared to the original sentence.
We present a first of a kind architecture for synthesizing adversarial examples using novel disentanglement technique and several components such as word attribution, word substitution and semantic similarity.
one-hot input representation. However, these at- tacks have the problem that the character swapping generates spelling typos and the sentences are not semantically meaningful. Sentence-level attacks perform the operations by altering the input texts on the whole sentence. SCPN (Iyyer et al., 2018) 8) in- cludes a paraphrase generation under the guidance of a trained parser to label syntactic transformation.
Our novel disentanglement technique aids in generating a representation which signif- icantly enhances the effectiveness of our at- tacks as outlined in our ablation studies.
We present end-to-end results of improve- ments obtained through adversarial training on BERT, T5-based and LLAMA2 models us- ing the examples generated from our approach on multiple datasets.
Word-level attacks are more popular and effec- tive methods than the above-mentioned two meth- ods. There are also many effective word-level at- tacks on text classification tasks (Ribeiro et al., 2020; Das and Paik, 2022). (Liu et al., 2021) pro- poses an efficient local search algorithm to deter- mine the possible word substitutions. However, the adversarial attacks sequence-to-sequence models are not much explored. (Simoneini and Spanakis, 2021) extends TextAttack (Morris et al., 2020) framework that consists of multiple attack strategies via reformulating the goal functions to support NER tasks. RockNER (Lin et al., 2021) is a simple NER adversarial attack by perturbing both named entities and contexts in the original texts. However, the generated adversarial examples all have the issue of poor semantic equivalence to the original input sentences because of the high modi- fication rate, In contrast, we propose a word-level adversarial attack on NER models that effectively identifies important words via word attributions and generates the adversarial examples with lower modification rate.
The rest of the paper is structured as follows. Section 2 provides an overview of related work in this field, highlighting the unique aspects of our contribution. In Section 3, we introduce our ap- proach and discuss the individual technical compo- nents involved, including disentanglement, word attributions and semantic similarity. In Section we present attack and adversarial training results on three benchmark datasets followed by conclusions.
2 Related Work
The existing adversarial attack methods on NER tasks can be classified into three categories: Chavacter-level Attack, Word-level Attack, and Sentence-level Attack. The Character-level attacks generate adversarial examples by adding, deleting, replacing character in word in the natural language texts. HotFlip (Ebrahimi et al., 2018) performs a character-level attack by swapping char- acters based on their gradient with respect to a
Recently. large language models (LLM's) (Brown et al., 2020; Wang and Komatsuzaki, 2021;

---

Rorablo
Rerakde xil
Ronaldo (Enily)
Beckham Neymar
Messi will play nine metches against Manchester United
Ronaldo [PER] will
nine
play
Encode Agatne:
Disentanglo
against
Word Attribution
matches (Contexc)
Replace Words
games contests toummments
Rank Sentences
Ronaldo will play nine games against Universities
matches
Exsty word Castort uooe
Suntence Stndlenty
againet Manchester
Compartion
Ronaldo will play
United [ORG)
Manchester United (Entity)
against Barcefona
FC Barcelona Real Madrid CF Arsenal FC
Word Embeddings from LLM
Most Important Words
Input Sentences
Disentangled
New Words
Adversarial Examples
Representation
Figure 2: The architecture of our method as depicted here proceeds as follows. Given an input text, we first obtain word embeddings from language models that the NER model uses, then learn disentangled re representation for entity and context representations (context, entity in the UMAP embedding scatter plots). The disentangled representations are then used to compute scores via the IG word attribution function. The most important words are replaced with words following word substitution workflows. After generating sentences, rank the sentence similarity scores and output the most similar sentences as adversarial examples.
Touvron al., 2023), have brought attention that they outperform many NLP tasks and have been widely used for applications (Sallam, 2023). Nori et al. (2023) explore potential use of LLMs in med- ical education, assessment, and clinical practice. However, LLMs still underperform in some NLP tasks. (Wang et al., 2023a) study the robustness of in-context learning and propose an ICL attack on large language models by manipulating the demon- strations. In this paper, we focus on instruction fine-tuned language model on NER tasks and ex- amine its robustness under our proposed adversarial attack method.
of word representations, (2) Selection of impor- tant words using word attributions, (3) Substitution of the selected words with candidate alternatives, (4) Ranking and selection of candidate adversarial examples
3.1 Threat Model
The threat model composed of two key compo- nents the adversary and the defender
3.1.1 Goals of the adversary
The adversary's primary goal is to manipulate the NER model's predictions by introducing subtle changes to the input text that cause the model to misclassify named entities.
Disentanglement (Higgins et al., 2017) has been primarily used in the space of auto-encoders for learning latent factors which are mutually orthog- onal (diverse) w.r.t one another aid with in- terpretability primarily for image-based applica- tions. They have also been extended for text-based tasks (Zou et al., 2022) to leverage the latent com- ponents to learn more desired downstream repre- sentations. In this paper, we use disentanglement to segregate the entity and non-entity representations in the feature space.
3.1.2 Capabilities of the adversary
The adversary has deep understanding of the NER model's architecture and training data. The adver- sary has access the disentanglement and word attribution modules to sample entity and or non- entity component words. In addition, the adversary has access to two different word substitution work- flows for entities and context, respectively. Descrip- tion of these modules are provided in the following subsections within this section.
3 Our Method
3.1.3 Knowledge of the adversary
In this section, describe the threat model and the key technical components of our proposed adver- sarial NER framework. For the sake of brevity, we refer to non-entity as context and L.M NER as NER throughout the rest this paper. The approach con- sists of the following steps, (1) Disentanglement
The adversary has knowledge of the NER model's training data and can use this knowledge craft adversarial examples that are specifically designed to cause mis classsisiiiis through LLM halluci- nation and/or any other mechanisms.

---

3.1.4
Goals of the defender
The defender's primary goal is to protect the NER model from adversarial attacks and ensure that continues to make accurate predictions.
E
Decader
Enc
Dec
Latent Embeddings
3.1.5
Capabilities of the defender
Word Embeddings
Disentangled
The defender has access techniques for adversar- ial training, which help to increase the NER model's robustness attacks.
Representations
3.1.6 Knowledge of the defender
Discriminator
The defender has knowledge of the NER model's architecture and training data, well as any po- tential vulnerabilities that may be exploited by the adversary. The defender may also have knowledge of common adversarial attack strategies such as Bert-Attack (Li al., 2020), DeepWordBug (Gao et al., 2018), etc.
Figure The architecture of disentangled representa- tion learning framework which takes initial word feature embeddings as inputs, and use an auto-encoder to lear latent representation for entity words. discrimi- nator Dis is used to recover context embeddings C from entity embeddings E"
3.2
Disentanglement of Word Representations
Encoder. The encoder Enc takes entity word embeddings E as input, then learn a disentangled representation E' in the latent space. Note that the encoder is also aware of context word embeddings C when encoding E:
The primary objective of disentanglement is to mit- igate bias (between entity and non-entity words) in word selection for perturbation, thereby increasing the diversity of the generated adversarial examples. In Table I, we provide supporting evidence that in- dicates that NER models are highly biased towards context words during prediction. The word attri- bution function like IG picks 9%%9% of non-entity words from the CoNLL dataset for generating po- tential adversarial examples if disentanglement is not applied. This adversely affects the overall diver- sity of the generated adversarial examples thereby preventing exploration of potential vulnerabilities of the NER model to entity specific perturbations. Our approach of disentanglement mitigates this problem through generating adversarial examples by exploring the entire perturbation space across important entity and non-entity words in an effi- cient manner.
Enc(E:C) E..
Decoder. The decoder Dec takes E' as input and output E, which a new (disentangled) repre- sentation for E:
Dec FC:C= A.
(2)
Discriminator. The discriminator Dis tries to predict (recover) the context embeddings C using the hidden representation of entity words E':
Dis(E') =
(3)
Given the original features of input sentence, we first split word features into two sets: E for fea- tures of entity words and C for features of context words. Our goal is to learn a new representation for entity words, so that the features of entity words are independent of context words. The re- sulting representation (F, C') will be utilized for calculating word attribution scores in the next step. Following (Marx al., 2019), learn the dis- entangled representations by using an auto-encoder. The architecture of the auto-encoder illustrated in Figure 3, which consists of three neural networks: the encoder, the decoder, and the discriminator:
where Ô is the predicted result for C. However, since expect E' and C to be independent, the recovery error between Ö and C' should be max- imized. other words, the discriminator cannot recover any information of using E", which indi- cates that there is no correlation between C and E' (as well as Æ).
Training objective. Our training process aims to minimize the reconstruction mean squared error (MSE) between E and É, as well as maximizing the recovery error between C' and C:
= MSE(E, Ê) MSE(C, Õ) (4)

---

Dataset
CoNLL2003
Onto.Notes
Original Distribution of Context Distribution of Context as most important words before disentanglement Distribution of Context as most important words after disentanglement
83.2%
91.1% %
Foodaul
98.9%
96.1%
Ronaldo play
melthes againet Mandhester united.
Belping 30
Manchester Uniidd
Belorig
81.3%
91.4%
Madrid EP
FC Baroolon
Arserul
Word type distribution of datasets. The first row represents the original distribution of with the NER label O (context words) among all word labels in the two datasets followed by two representing the distribution of important context words identified by the IG word attribution function in two different settings. is clear that in comparison to before disentanglement (the second row), the representation learned using our balancing disentanglement technique (in third row) ob- tains distribution that aligns more closely with the original dataset distribution (first row).
Table I:
Entily Word Raplacament
itrronide will pay nine AbB)) ingainet
games
Montel
Context Word Replacement
Figure 4: Entity and Context Word Substitution Work- flows
3.4 Word Substitution of Selected Words
where 3 is balancing hyper-parameter. E and C are taken as the disentangled representations for entity and context words.
After identifying the most important words in a sentence, the next step is to determine potential substitutions for these words. We outline both placement workflows in Figure 4 and briefly de- scribe them below.
3.3 Selection of Important Words
To evaluate the impact of each word's feature on the NER model's output, we propose compute the word attribution scores (a.k.a feature importances or saliency scores) using Integrated Gradients (IG) (Sundararajan et al., 2017). IG calculates the gra- dient of the model's prediction output with respect to its input features and returns the attributions of output labels with respect to the input features. The assigned value each input feature is the impor- tance score to model outputs. The mathematical formulation for IG as follows
Named Entity Substitution To keep the entity la- bels unchanged, the replacement word for named entity word should belong to the same entity type the original entity word. Since Wikidata contains abundant structured knowledge and most of the en- tities have corresponding entries it, we utilize it as as an external knowledge base for word replace- ment. Specifically, similar to (Lin et al., 2021), to determine the substitution for given named entity, we first entity linking tools (Honnibal et al., 2020) link the named entity in Wikidata. If a matching entry found in Wiki- data, we collect the entries that have a belonged to relation with this entry and take them as the upper categories of the target named entity. We randomly sample at most 10 of the entries under the same upper categories possible word substitutions.
entry
Suppose we have a black-box machine learning model f, input E R" and baseline input x' E R" (for text models it could be embed- ding vector). The Integrated Gradient (IG) along the th dimension for the input and x' defined as:
As an example illustrated in Figure 4, the label of the target word Manchester United an orga- nization. After linking this entity to the correspond- ing entry Manchester United FC in the Wikidata knowledge base, we find that Manchester United FC belongs the category of Association Football Club. Within the same category, find other en- tries such FC Barcelona, Real Madrid CF, and Arsenal FC. These entities are possible substitu- tions to replace the original entity word Manch-
OF(x' + a(x x²))
IG,(x) = (I:- -
do.
a=0 dr,
(5)
We obtain ranked list of words in the sentence by sorting them based on their word attribution scores. The top-K among them are then selected for perturbations. The use of disentangled word representations (as described in the earlier section) enables a more balanced selection.

---

ester United.
Dataset
CoNLL
OntoNotes 59, 924
MultiCONER
Train
44.987
16778
Context Word Substitution We leverage an in- filling language modeling framework (ILM) (Don- ahue et al., 2020) that allows language models fill in the blanks given masked texts. We use a large language model (LLM) GPT-J (Wang and Komat- suzaki, 2021) to generate texts. Specifically, we mask the context word that needs to be replaced with [BLANK]. Then, take the masked sentence input to the LLM with a prompt that asks the model to generate possible candidate words to substitute the [BLANK] word. Finally, we collect the output words as substitutions for the context word, as illustrated in Figure 4.
Val
3. 466
528
871
Test
3, 684
8, 262
871
Table 2: Train, Validation and Test Splits.
4.2 Baseline Methods
We compare our proposed method with the follow- ing NER adversarial attack baseline methods:
RockNER (Lin et al., 2021) creates adversar- ial examples by operating at the entity level and replacing all target entities with other en- tities of the same semantic class in Wikidata. At the context level, it randomly masks up to 3 context words and uses pre-trained language models (PLM) generate word substitutions. Attck (Simoneini and Spanakis, 2021) is an attack framework against token classifica- tion models. The framework extends TextAt- tack (Morris et al., 2020) and contains mul- tiple adversarial attack strategies. We com- pare our attack method with 3 attacks in the framework, including Bert-Attack, CLARE, and Deep Wor
3.5 Sentence Ranking
For an input sentence, we identify target words to be substituted and replace them with new words, which gives us a set of new sentences. To decide which sentences should be output as the final adver- sarial samples, we compute the similarity between each new sentence and the original one using Uni- versal Sentence Encoder (USE) (Cer al., 2018). It encodes natural language texts into high dimen- sional vectors as text embedding. After encoding each pair of original and generated sentences, we compute their cosine similarity as their similarity score. Basically, given the original and new sen- tences S1 and Sz, the similarity score is computed by:
4.3 Experimental Setup
For the choice of the NER model, we follow the same settings as the baseline methods for fair com- parison. We utilize three different models for our experiments: a BERT base model cased (Laf- ferty et al., 2001; Devlin et al., 2019), T5 lan- guage model (Raffel et al., 2020) (T5-LARGE 770 M parameter model) and LLAMA 2-7B-CHAT model (Touvron et al., 2023). Architectures for BERT and T5 NER models are provided in the Appendix in Figure 7 and Figure 8, respectively. For the BERT NER model, we perform fine- tuning on both the CoNLL-2003 and OntoNotes datasets. Regarding the T5 model, we transform the NER task into an instruction fine-tuned model (T5 Instruction NER(Wang et al., 2022)) as out- lined in the Appendix in A.3. This is facilitated by converting the MultiCoNER dataset, into sentence- instruction pairs, we follow a similar approach to the sentences illustrated in Figure 6 in the Ap- pendix.
USE(S)) USE(S)) Sim, [USEE())) [SS((
(6)
4 Experiments
4.1 Datasets
We conduct experiments on three datasets, CoNLL-2003 (Sang and De Meulder, 2003), Ontonotes 5.0 (Weischedel et al., 2013), and Mul- tiCoNER (Malmasi et al., 2022). CoNLL-2003 is a NER dataset that consists of named entities clas- sified into four types: persons, locations, organiza- tions, and names of miscellaneous entities that do not belong to the previous three groups. Ontonotes 5.0 is a large corpus of news articles in three lan- guages, annotated with 18 types of named entities. MultiCONER a complex NER dataset that covers languages and has ane-egrined tagset with 36 entity types. In our experiments, we focus on the English track of these datasets.
To compute the word attributions, we freeze the parameters of encoder of the T5 model after fine- tuning, and add an additional classification layer after it, to create a new sequence labeling model. This T5 encoder only model is trained with a CRF

---

loss, and obtain word attributions and gener- ale adversarial examples by directly utilizing the predictions of this model. We evaluate the effec- tiveness of our attack methods based on multiple evaluation metrics as outlined below:
replace the most important words in Section 3.4 to output adversarial examples.
4.4 Benchmarking Results
In this section, we primarily benchmark the BERT NER, T5 Instruction NER and LLAMA models across different attack strategies.
4.3.1 Evaluation Metries
Attack Rate. This metric measures the suc- cess of an attack on a sentence. An attack is successful on a sentence if the NER model misclassifies at least one named entity in the sentence after the attack. We calculate the rate of successful attacks among all attacks.
4.4.1 Comparison against baselines
In Table 3, for the CoNLL-2003 dataset, we present a comparison the performance of our method against the five adversarial attack strategies within the SeqAttack framework. To ensure a fair compar- ison, we fine-tune our model to achieve the same Fl score the original Seq results (98%) before conducting the attack.
Modification Rate. The modification rate evaluates the changes made to the original sentence during the attack. is calculated as the percentage of different tokens between the original sentence and the modified sentence. Textual Similarity. Textual similarity is com- puted between the original sentence and the generated sentence. We use Universal Sen- tence Encoder (USE) to encode each sentence and compute cosine similarity between the two vectors.
From the results Table 3, we observe that our method shows a comparable level token modifi- cation rate (at around 22%) to both Bert-Attack and Deep Word However, our method outper- forms Bert-Attack by achieving upproximately 10% higher attack success rate. Furthermore, when com- pared to DeepWordBug our method achieves lower F1 score by 8% and a higher attack success rate by 24%. These results show the effectiveness of our method in achieving successful attacks with a lower level of modification rate.
4.3.2 Attack and Hyper-parameter Settings As shown in Table 3, to test the attack performance of our method, we report results different levels modification rate (maximum word or words). All displayed results are averaged across 3 random runs.
These significant improvements in Fl score and attack rate while having similar modification rates are attributed to the following reasons, our unique approach of combining effective disentan- gled representation learning with IG-hased targeted word attributions, and b) using semantic similar- ity based guardrails while generating adversarial examples.
For the BERT NER model, perform fine- tuning on a pre-trained bert-base-cased model with CRE. The training process utilizes batch size of 32 and a learning rate of 1e-5. We conduct train- ing for a total of 20 epochs on both the CoNLL- 2003 and OntoNotes datasets. In the case of the T5 model, we conduct a fine-tuning on the Multi- CoNER dataset. The training process has batch size of 16 and learning rate of 3e-4. The total training epoch is set to 10.
In Table 3, for the OutoNotes dataset, we com- pare the performance of RockNER and our method, For both RockNER and our method, successful attacks would result in lowering the Fl score com- pared to the baseline F1 score.
Note that RockNER chooses semantic-rich words and randomly masks tokens (3 max) in context-only attack, and replaces all named entities in entity-only attack. Different from RockNER, we compute disentangled representations and replace the most important or 3 words in the sentences. Comparing the Fl scores under attack, our method is much more effective than the baseline method on all three types of attacks. More impor- tantly, we observe that the Fl score for our method is 4% lower than the RockNER. This demonstrates that our method is better at selecting and replac-
The encoder, decoder and discriminator for com- puting the disentangled representations consist of two hidden layers with size = [64, 10] and 3 is set to 0.5. For IG word importance com- putation, we leverage the Captum library from Meta (Kokhlikyan et al., 2020). The learning rate is le-3, and the number of epochs is 25, We use Stochastic Gradient Descent optimizer with weight decay 1c-5.
After computing the word importance, we take the top K = 1, 3 important words to replace. Then,

---

Dataset
Attack Name
F1 Score,
Attack Ratet
Mod Rate!
Text Sim
Bert-Attack
79%
44%
22%
84%
CLARE
79%
37%
70%
86%
Deep WordBug-llay
87%
30%
21%
83%
RockNER
80%
51%
35%
64%
CONLL
82%
36%
11% (max word)
91%
Our Method
79%
54%
22% (max 3 words)
84%
Original
98%
RockNER
55%
37%
26% 6% (max word) 86%
66%
Our Method 65%
33%
OntoNotes
51%
42%
12% (max 3 words)
79%
Original
90.3%
Table 3: Results using different attack strategies on BERT NER model for CoNLL-2003 and OntoNotes datasets. The first 3 attack methods derived from Attactack. Numbers in the 4th and the Sth RockNER baseline. We generate adversarial examples allowing for maximum of or 3 word replacements. Lower score and modification rate along with higher attack rate and textual similarity indicate more superior attack strategy demonstrated by method.
derived using
Method
Attack Rater
Mod Ratel
RockNER
26.7% %
40.2%
Random
18.2%
19.7%
1.0
Beseline Training
Our Method
25.6%
19.7% %
Table 4: Attack Rate and Modification Rate for LLAMA2-7B-CHA. model on the CoNLL-2003 dataset.
0.8
F1 Score
0.6
ing the most important words which enables it to generate more effective adversarial examples.
4.4.2 Attack Results with LLAMA 2 model To examine the effectiveness of our attack method, we conduct experiments using LLAMA2-7B- CHAT model (Touvron et al., 2023) on the CoNL.L 2003 dataset. For this experiment, we allow a max- imum of word perturbation following our pro- posed framework. We compare our results with two baseline methods, RockNER and Random replace- ment of single word. Comparing the results in Table 4, we observe that both our proposed method and RockNER achieve higher attack success rate than random replacement. In addition, our method has a much lower modification rate compared to RockNER. After adversarial training, we observe that the robustness improves and F1 under attack increases by 3.1%.
0.2
0.0
CoNLL-2003
OntoNotes Dataset
MuINCONER
Figure 5: F1 scoret results three datasets. Blue columns and orange columns show the score under attack before and after adversarial training, respectively.
described above and allow max word perturba- tion, and generate adversarial examples from the original test set. On the new adversarial test set, the F1 score under attack drops to 37.96% with T5 instruction NER model. To better demonstrate the effectiveness of our method, we also use the base- line method RockNER to generate the adversarial test set, which replaces all named entities and al- lows max 3 word context words perturbation. The Fl score under attack '5 instruction NER model is 36.87% for RockNER. Our method reaches a comparable attack performance, whereas our mod-
4.4.3 Attack Results with T5 Instruction NER model
For the MultiCoNER dataset, the clean Fl score on the test set is 47.23% with the T5 Instruction NER model. We follow the same attack framework as

---

Disentanglement
F1 Score,
Attack Ratet
ure observe that adversarial training enhances model robustness across all datascts. Specifically,
Original 98%
CoNLL
without
86%
25%
for adversarial examples that allow maximum 3 word replacement on CoNLL-2003 dataset, the FI score under attack improves by approximately 8% after adversarial training. Similarly, the FI score under attack on the OntoNotes dataset demonstrates an improvement of about 18% after adversarial training. In the case of the MulriCoNER dataset, we observe a 4% improvement in the Fl score un- der attack after adversarial training.
with
82%
36%
Original
98%
OntoNotes
without
69%
28%
with 51%
42%
Table Attack Results for BERT NER model with and without disentanglement steps.
Word Selection
CoNLL
OntoNotes
Original
98%
90%
Random
87%
61%
TF-IDF
90%
58%
5
Conclusions
Kernel-SHAP
86%
61%
Integrated Gradients (IG)
82%
51%
LLM's for complex NER problems have been gain- ing lot of traction recently, and this work, we propose an adversarial attack based framework to make these NER models more robust to widen their adoption even further. Our end-to-end approach combines novel disentanglement technique with word attributions, substitution and semantic simi- larity generate adversarial examples. Our disen- tanglement method builds upon the idea of trying to learn an embedding by disentangling entity and non non-entity latent representations
Table 6: Fl scores for BERT NER model under attack using different strategies. (lower better)
ification rate is only 46% of the modification rate from RockNER. Further details on the setup here are provided in the Appendix in Section A.3.
4.4.4 Ablation Studies
Without disentanglement To assess the effective- ness of our disentanglement technique, we compare the attack performance on the BERT NER model with and without disentanglement steps. The re- sults, as illustrated in Table 5, demonstrate the im- pact of disentanglement on the attack success rate. With the disentanglement step, we observe reduc- tion in bias in word selection (Refer Table 1) im- proving diversity of adversarial samples generated and significant attack performance improvement. Other word selection methods To examine the effectiveness of IG, we also tested other word se- lection strategies. (I) Randomly select words for substitution. (2) Select words using TF-IDF val- as word importance.
Applying disentanglement before computing IG word attributions aids in ensuring that we are able to synthesize diverse set of adversarial exam- ples in extremely efficient manner demon- strated through lower candidate sample generation and modification rates. Experimental results across BERT, T5-based and LLAMA NER models on three benchmark datasets demonstrates that our method significantly outperforms competing base- line methods. Ablation studies highlight the im- portance of disentanglement and word attribution techniques. Limitations
(3) Select words using KernelSHAP (Lundberg and Lee, 2017) that uses a special weighted linear regression to compute feature importance. Compared with scores in Table 6, IG outperforms all other three word se- lection methods, which proves the effectiveness of using IG. The results that IG outperforms other three methods in non-linear models are also consis- tent with insights here (Modarres et al., 2018).
One of the key limitations of our work is that we have not explored the entire LLM landscape and would be keen to explore the decoder only models. Being able to evaluate our work across decoder only models strengthens some of the key claims made in this paper. In particular, we would like to investigate robustness of models with growing size.
4.4.5 Adversarial Training
Future avenues to investigate for us mainly in- clude building an end-to-end framework synthe- size attacks for LLM NER models where the word attributions and semantic similarity scoring are pur- sued as potential paths within a pipeline to identify
After generating adversarial examples from the training set using the workflow depicted Figure 2, we conduct adversarial training on the trained NER models on the three datasets. As illustrated in Fig-

---

the optimal attack. This will help in reducing the dependence on specific choices significantly.
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ethical Consideration
We acknowledge that our work is aligned with the ACL Code of the Ethics and will not raise ethical concerns. We do not use sensitive datasets/models that may cause any potential issues/risks.
Chris Donahue, Mina Lee, and Percy Liang. 2020. En abling language models to fill in the blanks. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2492- 2501.
Acknowledgement
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36.
We thank the anonymous reviewers for their help- ful suggestions. This research work is supported by Amazon Alexa AI, the Molecule Maker Lab Institute: an AI research institute program sup- ported by NSF under award No. 2019897 and No. 2034562, and DOE Center for Advanced Bioen- ergy and Bioproducts Innovation (U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research under Award Number DESC0018420), The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the Molecule Maker Lab Institute, or the U.S. Depart- ment of Energy. The U.S. Government is autho- rized to reproduce and distribute reprints for gov- ermmental purposes notwithstanding any copyright annotation therein.
Ji Gao. Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50-56.
Irina Higgins. Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAB VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1875-1885
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds. Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Roelitz-Richardson. 2020. Captum: A unified and generic model interpretability library for pytorch.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Steve Yuan, Chris Tar, et al. 2018 Universal sentence encoder. arXiv preprint 18X3::8803.1111
Mario Quajardo-Cespedd
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Sudeshna Das and Jiaul Paik. 2022. Resilience of named entity recognition models under adversarial attack. In Proceedings of the First Workshop on Dy- namic Adversarial Data Collection, pages 1-6.
Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming- Ting Sun, and Bill Dolan, 2021. Con- textualized perturbation for textual adversarial attack.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, Adversar-
and Xipeng Qiu. 2020. ial attack against BERT using BERT In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193 02--2202, Online. Association for Computational Linguistics.
'https: htttps://ww..clwwebbb org/portal/contentt acl-code-ethics

---

Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. RockNER: A simple method to create adversarial examples for evaluating the ro- bustmess of named entity recogmition models. In Pro-
Malik Sallam. 2023. Chatgpt utility in healthcare edu- cation, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, volume I1, page 887. MDPI.
ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 3728- 3737, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Erik Tjong Kim Sang and Fien De Meulder. 2003. In- troduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Shengeai Liu, Ning Lu, Cheng Chen, and Ke Tang. 2021. Efficient combinatorial optimization for word-level adversarial textual attack. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:98- 111.
Walter Simoncini and Gerasimos Spanakis. 2021. Seqat- tack: On adversarial attacks for named entity recog- nition. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 308-318.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions, Advances in neural information processing systems, 30.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Interna- tional conference on machine learning, pages 3319- 3328. PMLR.
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. Semeval-2022 task 11: Multilingual complex named entity recognition (multiconer).. In Proceedings of the 16th interna- tional workshop on semantic evaluation (SemEval- 2022), pages 1412-1437.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint Kii:22307009888
Charles Marx, Richard Phillips, Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasuurammanian. 2019. Disentangling influence: Using disentangled repre- sentations to audit model predictions. Advances in Neural Information Processing Systems, 32.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https:/ //gi how.b..com/kiggffol... mesh-transformerr-
Ceena Modarres, Mark Ibrahim, Melissa Louie, and John Paisley. 2018. Towards explainable deep learn- ing for eredit lending: case study, arXiv preprint arXiv: 1811.06471.
Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. 2023a. Adversarial demon- stration attacks on large language models. arXiv preprint 230.:2355.1955
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nip. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126.
Liwen Wang. Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, and Weiran Xu. 2022. In- structionner: A multi task instruction-bar based gener- ative framework for few-shot ner. arXiv preprint arXiv:2203.03903
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabili- ties of gpt-4 on medical challenge problems. arXiv preprint 2rXiv:2303.1333
Shuhe Wang. Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023h. Gpt-ner: Named entity recognition via large language models. arXiv preprint ar34v:2304. 10428.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.066 200.0667...
R Weischedel, M Palmer, M Marcus, E Hovy, S Pradhan, L Ramshaw, N Xue, A Taylor, Kaufman, Fran- chini, et al. 2013. Ontonotes release 5.0 1t12013119. linguistic data consortium, philadelphia, pa (2013).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (1(555585-5551.
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint 2xXi:2336.00522.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of nlp models with checklist. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912.
Yicheng Zou, Hongwei Liu, Tao Gui, Junzhe Wang, Qi Zhang, Meng Tang, Haixiang Li, and Daniel Wang. 2022. Divide and conquer: Text semantic matching with disentangled keywords and intents.

---

A Appendix
Teta "Japan ama began the defence of their Asian Cup title with a lucky 2-1 win against Syria in Group C championship match on Friday
A.1
Model Queries Comparison
We compare our attack performance with CLARE under SeqAttack framework. CLARE is word- level attack technique which generates highest- scoring candidate sentences from replacing, insert- ing, and merging new words into the original sen- tences. According to their experimental results in the paper, it reaches F1 score 79% and attack suc- cess rate 37% whereas our attack method gets F1 score 79% and attack success rate 54%. CLARE attack allows at most 512 model queries. It gener- ates all possible new sentences (usually 1000+) and checks if each new sentence attacks the NER model successfully. The attack success rate very low. Among the successful attack sentences, it takes on average about 33 model queries (33 new sentences) to reach a successful attack. Compared with our method, we only allow at most 10 candidate sen- tences generated from each original sentence. Then we check if any sentence is successful adversar- ial example which is much less than the baseline method.
AttackedTeext "Japan an began the defence of their Asian Cup title with a lucky 2-1 win against Syria in Group C. championship match on Friday
Attacked "Japan and began the defence of their Asian Cup title with lucky 2-1 1 win against Syria Group championship match on Friday
AttackedText "Japan ans began the defence of their Asian Cup title with lucky 2-1 1 win against Syria in Group C championship match on Friday (All generated examples failed attack)
From the outputs using CLARE, we can see that their attack strategy is to select multiple indices in the sentence for word-level operations. In the attacking sample, the attack method iteratively in- serting random words between Japan and began. These random insertion incurs syntax errors in the generated sentences. Comparing with this base- line, our method utilizes word attribution to find the most important word in the sentence more ef- ficiently. In addition, with the help of PLMs and knowledge bases, we are able to replace with more reasonable words.
Some example output from CLARE: Attacking sample: Japan began the defence of their Asian Cup title with a lucky 2-1 win against Syria in a Group C championship match on Friday.
AttackedText: "Japan 's began the defence of their Asian Cup title with lucky 2-1 win against Syria in a Group C championship match Friday
A.2 F1 score under attack results on OntoNotes dataset
AttackedText: "Japan 
 began the defence their Asian Cup title with lucky 2-1 win against Syria in a Group C championship match on Friday
In Table 8, for the OntoNotes dataset, we com- pare the performance of RockNER and our method under three different attack settings, namely, a) context-only attacks, b) entity-only attacks and c) context + entity attacks. The second column in this table represents the baseline Fl scores on the orig- inal test set before the attack was conducted. For both RockNER and our method, successful attacks would result in lowering the Fl score compared to the baseline Fl score.
TttackedText "Japan A began the defence of their Asian Cup title with a lucky 2-1 win against Syria in a Group C championship match Friday
AttackedText: "Japan a began the defence of their Asian Cup title with a lucky 2-1 win against Syria in a Group C championship match on Friday
Note that RockNER chooses semantic-rich words and randomly masks tokens (3 max) context-only attack, and replaces all named entities in entity-only attack. Different from RockNER, we compute disentangled representations and re- place the most important context words (3 max) for context-only attuck and the most important entity words (3 max) for entity-only attack.
AttackedText: "Japan ai began the defence of their Asian Cup title with lucky 2-1 win against Syria in a Group C championship match Friday
AttackedText: "Japan ain began the defence of their Asian Cup title with a lucky 2-1 win against Syria in a Group C championship match on Friday
Comparing the Fl scores under attack, our

---

Original Test Sentence
Ady Example (RockNER)
Adv Example (Our Method)
Sentenee: Dear viewers, the China News
Sentence: Dear viewers, the Hiwwe wie Driwwe
Sentence: Dear viewers, my China News
program will end here.
people will lose here.
program will end here.
Output: China News
Output: Hiwwe wie Driwwe is not an entity.
Output: China News is
an organization.
work-of-art.
Sentence: Relevant departments from Beijing Municipality
Sentence: Related departments from Markham
Sentence: Relevant departments from Berlin Municipality
promptly activated
promptly activated
promptly activated
emergency contingency plans.
emergency contingency members.
emergency contingency plans.
Output: Beijing Municipality is a geopolitical entity.
Output: Markham
Output: Berlin Municipality is an organization.
is not an entity.
Table 7: A case study on generated adversarial examples by our method. The sentences on the left are original test sentences from OntoNotex dataset with correctly predicted named entities (blue colored words). The sentences in the middle and on the right are generated adversarial examples using RockNER baseline and our method that cause the NER model make wrong predictions. All input sentences also come with the instruction Please extract entities and their types from the input sentence. The italic words are the candidate words got selected and replaced. The red labels are incorrect predictions after the adversarial attacks).
Ourput Stering
Sentenoe JUNY Bleiing Joar THADS intruction Planse serter erthwrr ANT Bich oper Norm the tnr.guaaeeernes.
Sentence: was the fifth actor to reprise the role followed by Ben Hardy 2013.
au Epprrta Groan
Adverasria Altack
Instruction NER
Sentence: Ceorge Stering joon 19H10) spocticako New Nork Yurk Rocc seeli Sywen. from the dertaiine
Dutput: George Steriig Person, Now York Yonkuby
BERT Encoder
Figure 6: Input sentence from MultiCoNER dataset and the adversarial example generated using attack method, wherein we attack instruction fine-tuned T5 NER model here. Following our attack, the NER model incorrectly classifies New York Yankees Location.
FC CRF Loss
O B-ARTIST I-ATIIST 000
method is much more effective than the baseline method on all three types of attacks. More impor- tantly, we observe that the Fl score for our method is 6% ~ 8% lower than the RockNER. This demon- strates that our method is better at selecting and replacing the most important words which enables it to generate more effective adversarial examples.
Figure 7: The architecture of the BERT-CRF NER model.
In this way, we can directly take the output logits of this T5-encoder only model for word attribution computation.
Figure 6 presents an example sentence gener- ated through an adversarial attack that successfully perturbs the NER model. The sentence on the top comes from the original training set. Along with the instruction Please extract entities and their types from the input sentence, the T5 Instruction NER model correctly predicts the NER labels in natural language texts format.
A.3 Adversarial Examples for Instruction Fine-tuned Model
We fine-tune a T5 language model for named en- tity recognition task on MultiCoNER dataset. As shown in Figure 8 on the left, T5 is an encoder- decoder model and we convert the NER dataset into text-to-text format. To compute the influence score, after fine-tuning the T5 model, as shown in Figure 8 on the right, we take its encoder only, freeze the parameters, and add one more linear layer after it to form a new sequence labeling model. Then, we train the last linear layer with CRF loss.
We propose an approach for word-level adver- sarial attacks that generates adversarial examples in a principle manner. In the example presented in Figure 6, the word 'John Sterling' is an entity word that influences the NER predictions. Then,

---

F1 Score
F1 Score
Model Name
Original test 90.6%
Context-only J Entity-only + Context + Entity +
BERT-CRF (RockNER)
85.8%
59.2%
54.6%
BERT-CRE (Ours)
90.3%
79.6%
51.1%
47.5%
Table 8: Fl score under attack results OntoNotes dataset. Both RockNER and method are evaluated under context-only attack, entity only attack, and context+entity attack settings. Comparing numbers row-wise against RockNER (first row) reveals that our method (second row) obtains lower Fl score compared to score on original test set under all three attack settings.
Santence He was the 10h actor to meris roin snlowed by Handy tyoes
method are more semantically similar to the orig- inal test sentence. This can be attributed to the fact that our method synthesizes diverse and effee- tive adversarial examples while preserving textual similarity. (keeping the modification rate low).
Sentence: followed Hardy
repren
15 Encoder
T5 Encoder
Output: Ben Handy ta artalll
00000000000O AATT 000
Figure 8: The architecture of two T5 models. On the left is the Instruction NER Model and the right the T5 Vanilla NER Model.
we replace it with relevant substitutes to generate the adversarial examples. The disentanglement of the latent representations of entity or non-entity word types prevents bias towards perturbations of particular word type.
A.4
Case Studies
We show the generated examples from OntoNotes dataset using the RockNER baseline method and our method in Table 7. The sentences on the left are from the original test set and the blue texts are correctly predicted named entities by the NER model. The sentences in the middle are the gen- erated adversarial examples using the RockNER bascline method. The sentences on the right are the generated adversarial examples following the workflow in Figure 2.
The italic words are selected be replaced from the original sentences. The red labels are wrong predictions by the NER model after the adversarial attacks.
In this experiment, we allow at most one word replacement and it could either be context or named entity word. From the output we can see that, using our method, the replacement of one word effectively causes the NER model to make wrong prediction.
From Table 7, we observe that in comparison to RockNER, the adversarial examples from our


Reviews:
[{"review_id": "N0pm8REwR3", "reviewer": "Reviewer_fP4e", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable inputs and feedback. We appreciate the reviewer for recognizing the soundness and effectiveness of our proposed method. Our response to your questions are as follows:\n\n**[Clarifications in the writeup]**: In Line 42, the LLM hallucination problem is mentioned as it is attributed as one of the primary reasons for the poor robustness of LLM NER’s (GPT-NER paper referenced on Line 761) which is relevant to the problem we are addressing in our paper. In Line 71, we wanted to guide the reader to look at Figure 2 which outlines the overall process flow while building a high level overview of the steps in the narrative. In Line 232, the three neural networks mentioned are an intrinsic part of the disentanglement module which fits within the bigger end-to-end architecture proposed in our paper. We will work towards fixing the references and making these subsections as well as parts of the experimental setup less verbose to improve the readability of the final version.\n\n**[Analysis of disentanglement]**: We agree with the reviewer that the primary validation of our hypothesis on the effectiveness of disentanglement and word attributions is obtained through the benchmarking done on the three datasets. However, beyond empirical validation, we also provide visual insights on how disentanglement modifies the representation through Figure 2. Eq (4) ensures that the output entity representation ($\\hat{E}$) after disentanglement in Figure 2 is preserving the necessary information from the original entity representation ($E$). Additional qualitative insights from the case study in Table 5 in the Appendix also provide more insight into the sentences generated post disentanglement representing preserved and modified components explicitly. We will consolidate this analysis even further as we expand upon this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem addressed is of interest to the research community. The approach is overall sounding and combines existing techniques in the literature (e.g., disentanglement via autoencoders, sentence embedding, and word attribution) in a simple and reasonable pipeline.", "reasons_to_reject": "The novelty of the paper is rather limited, the analyses of the disentangling mechanism are rather high-level (it is a given, never really analysed, what kind of information would be preserved once the representations of the entities are disentangled from the context, everything is justified only by the final performance on two datasets).\nThe experimental analyses, while written fluently, is difficult to follow and the language is at time verbose.\n\nSecondary: \nThe overall paper is well-written in English, but there are often logical gaps in the structure of the paragraphs.\nJust to mention a few in the Introduction, but several are spread throughout the paper: \nLine 042: There is a long introduction to LLM and their hallucination, without mentioning any possible link of this problem to the NER tasks;\nLine 071: it introduces the concept of \"steps\", but no \"process\" was so far mentioned;\nLine 193-195: it introduces \"for the sake of brevity\" abbreviations that are rather obscure and probably do not actually abbreviate much the text;\nLine 232: mentioned \"3 neural networks\" while this architecture is then discussed as an end-to-end architecture, without particular discontinuity (please refer to the literature about AE and VAE).\nSimilar problems made the reading of the experimental evaluation difficult to assess.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the limitations of Large Language Models (LLMs) in complex Named Entity Recognition (NER) tasks. In particular, it aims at improving the LLM robustness, introducing a pipeline mainly based on a novel adversarial attack disentangling entity and non-entity representations via an autoencoder approach to reduce the bias in the selection of words. This experimental assessment shows an improvement in terms of the F1 score of LLM NER models on datasets such as CoNLL-2003 and Ontonotes 5.0.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would invite the authors to rephrase the paragraphs to be less verbose to improve the overall readability of the paper.\nAdditional problems:\nPlease fix references to Figure following EMNLP former (e.g., Line 068)\nPlease fix references to work directly discussed (e.g., line 168)\nLine 058: remove contractions\nLine 078: remove \"an\""}}, {"review_id": "hkE87L6Qrd", "reviewer": "Reviewer_Yix7", "summary": "", "questions": "\n- In Figure 1, how do you perform PCA? Why do you think Rockner's results are worse? I can't tell which one is better.\n  In the disentanglement procedure, do you use the initial word embeddings, or hidden states generated by the model? \n- How do you define the context word? If the initial word embeddings are used, it would be rather strange since it has nothing to do with the model itself.\n- It is interesting to see that you can attack T5's decoder by attacking a surrogate model: T5's encoder+ CRF. The instruction tuning and sequence labeling paradigms are quite different. Why can such an approach work so well?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable inputs and feedback. We appreciate the reviewer for recognizing the effectiveness and comprehensive benchmarking of our proposed method. Our response to your questions are as follows:\n\n**[Comparison with Llama]:** We now provide a comparison with Llama2 also here.\n\n| Adversarial Attack Results      | Attack Rate | Modification Rate |\n| ----------- | ----------- | ----------- |\n| Our Method    |  25.6%   |   19.7%   |\n| Baseline (RockNER)   |  26.7%      |  40.2%    |\n| Random  |  18.2%      |   19.7%    |\n\nIn the table above, we report results using the Llama2-7b-chat-hf model on the CoNLL-2003 dataset. For this experiment, we allow a maximum of one word perturbation following our proposed framework. The first baseline method that we are comparing is RockNER, which by default substitutes all named entities as well as a maximum of three randomly selected context words. The second baseline method is the random replacement of a single word. Comparing the results of all three methods shown in the table, we observe that both our proposed method and RockNER achieve a higher attack success rate than random replacement. In addition, our method has a much lower modification rate (**19.7%**) compared to RockNER (40.2%), lower modification rate w.r.t original sentence indicates that the synthesized adversarial examples are semantically similar to the original sentence. After adversarial training, we observe that the model robustness improves and F1 under attack increases by **3.1%**.\n\n**[Motivation for disentanglement]:** We appreciate the reviewer thoroughly reading our paper, and we can move contents from Appendix F to the main paper to provide more motivation on disentanglement in the main paper. The qualitative analysis part is interpreted primarily from our case study in Table 5 in the Appendix. After disentanglement, we rely on the word attribution function to sample entity and/or non-entity (context) components. We have done a separate study in Table 6 in the Appendix where we substitute only entity or non-entity components exclusively to estimate the upper bound, and we observe consistent patterns of much lowered F1 score compared to RockNER. \n\n**[Size of T5 model]:** The size of the T5 model used in this paper is the t5-large 770 million parameter model.\n\n**[RockNER comparison in Figure 1]:** Figure 1, is meant to highlight the fact that methods such as RockNER do not preserve the semantic similarity while doing perturbation while our method maintains semantic closeness to the original example without trading off on attack effectiveness. Further information on the generation of the embeddings visualized through PCA in Figure 1 is provided in the response below.\n\n**[T5 experiment setup and results]:** The left part of Figure 8 in the Appendix depicts how we train the initial T5 encoder in the Instruction NER setup and export this encoder to the figure on the right. Subsequently, we obtain embeddings from this T5 encoder which are then disentangled, and we then use the Integrated Gradients word attribution technique based on the T5 encoder + CRF model to identify important entity and non-entity (context) components. Recent separate work on (*One Embedder, Any Task: Instruction-Finetuned Text Embeddings, Su et al., ACL 2023 Findings*) has shown the utility of such instruct embeddings across other tasks and we believe there are interesting commonalities worth investigating further here as they don’t specifically evaluate adversarial attack effectiveness using instruct embeddings like we do.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a word disentanglement approach to increase the performance of adversarial attacks. \n- Result exceed previous baselines and the experiments are comprehensive.", "reasons_to_reject": "- The motivation for the disentanglement is unclear. Related parts are set out in Appendix F, the very end of the manuscript. It should be highlighted and more quantitative and qualitative analysis is needed to support it. Why do context words affect the adversarial result, and why? Why disentangling such information may help the model produce better results? Since you have disentangled context information, is it possible to generate adversarial examples by simply substituting entity words? What is the upper bound of entity-word-only substitution? In my opinion, this is the most important contribution of this paper and needs further and deeper study.\n- Some details are missing from the manuscript. See questions.\n- I do not think this paper is related to LLM. BERT is not LLM. What is the model size of T5 you used? What is the result on modern LLMs, such as LLAMA?\n", "questions_to_authors": "\n- In Figure 1, how do you perform PCA? Why do you think Rockner's results are worse? I can't tell which one is better.\n  In the disentanglement procedure, do you use the initial word embeddings, or hidden states generated by the model? \n- How do you define the context word? If the initial word embeddings are used, it would be rather strange since it has nothing to do with the model itself.\n- It is interesting to see that you can attack T5's decoder by attacking a surrogate model: T5's encoder+ CRF. The instruction tuning and sequence labeling paradigms are quite different. Why can such an approach work so well?", "paper_topic_and_main_contributions": "This paper presents a novel attack that capitalizes on disentanglement and word attribution techniques, allowing for the learning of embeddings that capture both entity and non-entity influences separately. The latter approach is employed to identify key words that are important across both components. In contrast to most techniques, which primarily rely on manipulating non-entity words to generate adversarial examples, this approach expands the perturbation space to more effectively synthesize adversarial examples.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kyhTujJ0ho", "reviewer": "Reviewer_RiQ4", "summary": "", "questions": "- What is the connection of the title and the main contributions in this paper?\n- How does ChatGPT perform under the same attack perturbation?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable inputs and feedback. We appreciate the reviewer recognizing the novelty of our method and exhaustiveness of the paper. Our response to your questions are as follows:\n\n**[Comparison with ChatGPT]:** While we do not have the result with ChatGPT, alternatively we now provide a comparison with **Llama2** here.  \n\n| Adversarial Attack Results      | Attack Rate | Modification Rate |\n| ----------- | ----------- | ----------- |\n| Our Method    |  25.6%   |   19.7%   |\n| Baseline (RockNER)   |  26.7%      |  40.2%    |\n| Random  |  18.2%      |   19.7%    |\n\nIn the table above, we report results using the Llama2-7b-chat-hf model on the CoNLL-2003 dataset. For this experiment, we allow a maximum of one word perturbation following our proposed framework. The first baseline method that we are comparing is RockNER, which by default substitutes all named entities as well as a maximum of three randomly selected context words. The second baseline method is the random replacement of a single word. Comparing the results of all three methods shown in the table, we observe that both our proposed method and RockNER achieve a higher attack rate than random replacement. In addition, our method has a much lower modification rate (**19.7%**) compared to RockNER (40.2%), lower modification rate w.r.t original sentence indicates that the synthesized adversarial examples are semantically similar to the original sentence. After adversarial training, we observe that the model robustness improves and F1 under attack increases by **3.1%**.\n\n**[Threat model subsection and Error in Eq (5)]:** In the final version of the paper we will add a subsection on the threat model. We thank the reviewer for pointing out the missing $d\\alpha$ term in Eq (5) and we will address it in the final version.\n\n**[Context Word Substitution unclear]:** In the context word substitution, we mask the context word that needs to be replaced with [BLANK]. Then, we take the masked sentence as input to the LLM with a prompt such as “Please fill in the [BLANK] word with a single word, give me 5 possible candidate words: <MASKED SENT>”. Finally, we collect the word output as substitutions for the context word.\n\n**[Lack of proof validating disentanglement]:** Our framework is primarily driven by the intuition that applying disentanglement and attribution functions is a better approach for diversifying the adversarial example space hence assisting with adversarial robustness. The primary validation of this hypothesis is through the exhaustive empirical results and case study in the Appendix. We agree with the reviewer that theoretical analysis to study the limits of disentanglement and diversity in adversarial examples are good directions for future study.\n\n**[Acronym clarification]:** We can clarify in our writeup, but FOAK (First of a kind) is an existing public acronym which was not created by us in this paper. We will expand it in the final version and avoid usage of the acronym.\n\n**[Title and Contributions connection]:** We acknowledge the reviewer’s concern here, and we would like to clarify that the title was meant to highlight this completely unaddressed problem of adversarial robustness for LLM NER models rather than sounding as another proposed approach for an already existing problem. We believe the problem is of more importance which should be highlighted strongly and our solution is the first of a kind end-to-end pipeline to address this problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The research problem is interesting and practical. \n- The motivation of mechanism design is clear.  Although the introduced mechanisms may have early existed, their applications are motivated by these findings. It makes those designs technical sound and easy to follow. \n- Detailed case analysis and explanation.", "reasons_to_reject": "- Threat model is not clear. The current manuscript lacks a subsection normally titled Threat Model, that describes the considering threat model that contains: 1) goals of the adversary; 2) capabilities of the adversary; 3) knowledge of the adversary); 4) goals of the defender; 5) capabilities of the defender; 6) knowledge of the defender.\n- Formula (5) has an error and is missing an important item ${d\\alpha}$\n- The Context Word Substitution process is unclear. Please demonstrate the corresponding prompt or instruction.\n- The abbreviation ``FOAK’’ in Line 100 should be accompanied by the full name. Moreover, there is no need to use this abbreviation, since I cannot find any other occurrence of FOAK in this paper.\n- Lack of proof for validating the result of disentanglement.", "questions_to_authors": "- What is the connection of the title and the main contributions in this paper?\n- How does ChatGPT perform under the same attack perturbation?", "paper_topic_and_main_contributions": "This paper focuses on improving the robustness of LLM NER models. To construct qualified adversarial examples, they propose an attack method incorporating disentanglement and word attribution methods. The disentanglement technique mitigates the bias between entity and non-entity words. For the substitution of entity and context words, they introduce a knowledge base and LLM to accomplish this process. Extensive experiments on three datasets show the effectiveness of their proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6CUvwQTeOE", "reviewer": "Reviewer_MbYs", "summary": "", "questions": "NA", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable inputs and feedback. We appreciate the reviewer recognizing the novelty and effectiveness of our proposed method. Our response to your questions are as follows:\n\n**[Comparison with Llama]:** We now provide a comparison with Llama2 also here.\n\n| Adversarial Attack Results      | Attack Rate | Modification Rate |\n| ----------- | ----------- | ----------- |\n| Our Method    |  25.6%   |   19.7%   |\n| Baseline (RockNER)   |  26.7%      |  40.2%    |\n| Random  |  18.2%      |   19.7%    |\n\nIn the table above, we report results using the Llama2-7b-chat-hf model on the CoNLL-2003 dataset. For this experiment, we allow a maximum of one word perturbation following our proposed framework. The first baseline method that we are comparing is RockNER, which by default substitutes all named entities as well as a maximum of three randomly selected context words. The second baseline method is the random replacement of a single word. Comparing the results of all three methods shown in the table, we observe that both our proposed method and RockNER achieve a higher attack rate than random replacement. In addition, our method has a much lower modification rate (**19.7%**) compared to RockNER (40.2%), lower modification rate w.r.t original sentence indicates that the synthesized adversarial examples are semantically similar to the original sentence. After adversarial training, we observe that the model robustness improves and F1 under attack increases by **3.1%**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and easy to follow.\n\n* The proposed method, consisting of disentanglement and word attribution, is novel.", "reasons_to_reject": "The author frequently highlights LLM, but their experiments are not compared with LLM models such as GPT-3.5 or Llama.", "questions_to_authors": "NA", "paper_topic_and_main_contributions": "This paper proposes a novel adversarial attack for NER models. The attack consists of disentanglement and word attribution techniques to address resource-constrained attacks. Experiments show the effectiveness of the proposed methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method to generate adversarial examples based on disentanglement and word attribution techniques. Extensive experiments show the effectiveness of the proposed method. Based on the initial reviews, the reviewers found this paper working on an interesting problem, proposing a reasonable method, and conducting comprehensive experiments. However, there are many important details missing regarding, for example, the threat model, the context word substitution process, the motivation and analysis of disentanglement. Also, there is a question raised whether the tested models (i.e., BERT-base and T5-large) are really LLMs. 

In the rebuttal, the authors addressed most of the reviewers' concerns and answered detailed questions. Despite presenting additional results on Llama, the paper would be more interesting if the authors focused specifically on large language models as the NER robustness of pre-trained models has been studied in the literature while we know much less about how the emergent ability of LLMs (due to their enormous size) responds to the adversarial examples. Also, based on the threat model explained, the adversary works on the white-box scenario (with access to NER model's architecture and training data) of which the specific application value is not quite clear. 

If the paper is accepted, please consider fusing the rebuttals into the camera-ready version (especially on the threat model, the analysis of disentanglement, and the results on Llama) and revising the title (as discussed) so it better reflects the actual contributions of the paper.
Decision: Accept-Findings