Robustness of Named-Entity Replacements for In-Context Learning
Authors: Saeed Goodarzi
Nikhil Kagita
Dennis Minn
Shufan Wang
Roberto Dessi
Shubham Toshniwal
Adina Williams
Jack Lanchantin
Koustuv Sinha
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
in-context learning
named entities
robustness
natural language understanding
TL;DR: We find that LLM in-context learning can be non-robust with respect to named entity replacements in the demonstrations and/or query.

Abstract: A key feature of modern large language models (LLMs) is their ability to perform in-context learning, a prompting technique where query- answer demonstrations are shown before the final query. This allows for generalization to novel distributions at inference time where the LLM can learn new rules without parameter updates. However, the choice of demonstrations and their relationship to a particular query can have a profound impact on model accuracy, raising concerns about the true in-context generalization capabilities (Zhao et al., 2021). In this work, we explore the robustness of the in-context learning paradigm by focusing on entities. In particular, we seek to understand the robustness of LLM in-context learning with respect to named entity replacements. We discover a significant variance in downstream performance based on the choice of the named entities, across three popular reasoning tasks and two popular LLMs. Specifically, model accuracy on the test sets can fluctuate between -2.7 to +8.0 points depending on the choice of named entity replacements. Our analysis exposes the sensitivity of LLM in-context learning with respect to named entities, and offers a simple recipe to improve test performance by hyper-parameter tuning the named entities for a given dataset. Code and datasets for reproducing the results are publicly available.

Robustness of Named Entity Replacements for In-Context Learning
Saeed Goodarzi!
Nikhil Kagital
Dennis Minn
Shufan Wang'
Roberto Dessi
Shubham Toshniwal
Adina Williams
Jack Lanchantin :000
Koustuv Sinha
University of Massachusetts Amherst; "Universitat Pompeu Fabra
NVIDIA; FAIR, Meta
sgoodarzitae@umaen edu, koustuvs eewtassmmett...mm com
Abstract
Original Demonstration
[.]]
contect: Juson had 20 poliiopps Ha gave Denny some kolipopa. Now Jusion his hdlpppps
A key feature of modern large language models (LLMs) is their ability to perform in-context learning, a prompting technique where query- answer demonstrations are shown before the final query. This allows for generalization to novel distributions inference time where the LLM can learn new rules without parameter updates. However, the choice of demonstra- tions and their relationship a particular query can have a profound impact on model accu- racy, raising concerns about the true in-context generalization capabilities (Zhao al., 202 I).
question: How mony lollpappe did Jason give Denny?
answer: Jason staried with 20 tollpops. Then he had 12 after giving some to Denny, So he Donny 12 shor
[--]
contect: A tockery producses 60 tnoowss of bewad mach day Towo-theds of the inaoes aum sold in the momning and hal of what left sold ecually the afternon and evaniing. cuestion orc How mony icaves tread are sold the attemoon?
answer 88 2 - 40 lcaves bread are wold the mominng 60 40 loaves of
Eread ar leit 20x 2 40 loeoes bread ame sord affermoon, shive 40
Entity Replaced Demonstration
context: Dougles 20 lollpops. gave Donny some lollpaps. New Douglas has lolipops.
quisdar:: How many knlipape did Dougles give Danny?
lolipppp. Then So he gave Denny 20 12 8. anch [-]]
Douglas
giving Denny.
laaves bread each sold ecualyy bread sold answer 8D 2 = 40 loeves bruadd sold in the moming 60 40 20 lnuees of
A bakery
the looves and evening.
In this work, we explore the robustness of the in-context learning paradigm by focusing on entities. In particular, we seek understand the robustness of LLM in-context learning with dis-
momiing and hat what
question: How
afteroon??
Bread are left x0x1 1/222 10 laaves of bread ars said the afterconn. ssad 10
Figure 1: Named entity replacements for In-Context Learning. Example from FLANNTS5 on GSM8k (Cobbe et al., 2021). Here we replace all male entities (e") in demonstrations with Douglas, and observe a change in the query answer prediction (from the incorrect answer 40 to the correct answer 10). Only 1 out of k = 5 demonstrations (D) is depicted to conserve space, with the remaining demonstrations denoted by "[...]"'
respect to named entity replacements. cover significant variance in downstream per- formance based on the choice of the named entities, across three popular reasoning tasks and two popular LLMs. Specifically, model accuracy on the test sets can fluctuate between -2.7 +8.0 points depending on the choice of named entity replacements. Our analysis ex- poses the sensitivity of LLM in-context learn- ing with respect named entities, and offers simple recipe to improve test performance by tuning the named entities for a given dataset. Code and datasets for repro- ducing the results are publicly available.'
model parameters. However, model performance in the in-context setting can vary significantly with the choice of demonstrations. Previous work has uncovered performance instability when altering the demonstrations, particularly when the demon- strations change semantically (Schick and Schütze, 2020; Zhao et al., 2021; Perez et al., 2021; Gonen et al., 2022a). Naturally, question arises whether LLMS are robust to semantically preserving, mini- mal alterations to the demonstrations.
1 Introduction
Modern large language models (LLMs) routinely demonstrate in-context learning, where the model can solve a wide range of tasks from a few demon- stration examples (or exemplars) (Brown et al., 2020). In-context learning, or few-shot prompting, is a powerful learning paradigm it allows for fast adaptation new tasks without the need to train
In this paper, we seek to address this question using simple, semantically preserving alterations: named entity replacements. Because changing named entities should not, in principle, change the correct response to the query, the model is ex- pected to remain invariant to such modifications. For example, given the demonstration "Alice has
Phttys://githee com/ Son//ennisMinn: entities- in- context "Equal Leadership.

---

the ball. Alice went the kitchen. Where is the ball? kitchen.", ", the model should be invariant the choice of the named entity "Alice"-iee. model should behave the same if we swap all instances of "Alice" with another named entity, e.g. "Eve". However, we observe that the model behavior is not always robust to such alterations (Figure I)- prompting us to critically evaluate named entity invariance within in-context learning.
Appendix subsection A.1.
2.1 In-Context Learning
With in-context learning, a test sample is com- prised of a sequence of & demonstrations D d1, d2,...,44 dx. Each d; consists of a context, ques- tion and answer string. The final query q, consist- ing only of a context and question string, follows D. The goal of in-context learning is to predict the answer to g given small set of demonstrations at inference time. In other words, learning is done by eliciting activations and not changing model parameters.
We conduct experiments to investigate the ro- bustness of LLM in-context learning with respect to named entity replacements in three popular named entity heavy reasoning tasks, namely, bAbl, Weston et al. 2015; CLUTRR, Sinha et al. 2019; GSM8K, Cobbe et al. 2021, and two popular models, Flan- T5-XI (Chung al., 2022a) and GPT-J (Wang and Komatsuzaki, 2021). Our key findings are:
2.2 Modifying In-Context Learning with Named Entity Replacement
We seek to measure the change in accuracy when we replace named entities in the demonstrations (D) and/or query (q). We perform entity replace- ment in the following way.
Neither Flan-T5-XL nor GPT-J are robust to entity replacements for in-context learning. On bAbl tasks, the best replacement entity yields a +8.0 point performance increase (for Task 16), and the worst replacement entity yields -2.7 point performance drop (Task 12). Similarly, for CLUTRR, we observe 1.6 point improve- ment. For GSM8K, we only find <1 point in- crease/decrease with entity replacements.
We first extract the named entities from each demonstration and query using an off-the-shelf NER model, and flag the subject entity, eb, for placement. 2 We then select replacement entity e" from a large set of entities from Tzioumis (2018), which is composed of 4,250 unique first names. Fi- nally, we perform named entity replacement in one of the following three settings²: (D) replace all e" entities in the demonstrations, cd, Cdz...., Cag. (q) replace all e entities in the query, eg. or (D and q) replace all e" entities in both the query and demonstrations, eq2 Edy.... Cg :g.
Performance gains from best performing enti- ties are not random. To verify, we ran a boot- strap analysis and observe consistent gains with low variance.
Best/worst performing entities are task-specific.
Note that each demonstration d, can contain multiple instances of the entity e These settings allow us to isolate and evaluate the effect of named entities in different parts of an in-context learning sample on accuracy.
We also analyze frequency, token length and perplexity of replaced entities, and find none of these alone explain our results, introducing an open problem for future research.
We run multiple experiments using boostrap sampling (with replacement) on three sets of names, half labeled as 'male' and half as 'female'. 4 For each name e" in a particular bootstrap set, we re- place the original entities e" with e" in the test sam- ples according to one of the three previously men- tioned settings. We then evaluate the performance on the entity-replaced dataset. This procedure is repeated for each bootstrap set.
2 Experimental Setup
The goal of this study is to investigate the robust- ness of LLMs' in-context learning performance by changing entities in the demonstrations and queries. We consider a fixed set of prompt demon- strations aside from the named entities. By replac- ing the named entities and observing any accuracy changes, we can better understand the effect of cer- tain entities a model's accuracy. While many prior works have studied the robustness of LLMS with respect to perturbations (Ribeiro et al., 2020; Jin et al., 2020; Balasubramanian et al., 2020), we focus our analysis on in-context lcarning. more in-depth analysis of related work is provided in
"tttps://buuee ing co/dslim/ bert- Da-R-NRR, see Appendix Section A.3 for de- tails on the selection of co.
AAppendix Section A.4 describes each setting in detail.
"While our data source provided a binary gender anno- tation on names based on self-report, we note that this is a proxy and moreaver, that using names as a proxy for gender is indicative but can be noisy.

---

bAbl
only do replacements that preserve gender when a pronoun is contained in the sentence (see Appendix A.5). This step specifically addresses any potential uncontrolled effects on the performance from entity replacement, such as perplexity spikes that may results from swaps where the genders of e" and e" differ. and the model is expected to accommodate unexpected pronominal coreference with e".
100
Accuraty 186)
68
Y0
01
1.
92
75
02
855
3 Results
50
50
Performance Across Bootstrap Sets. We find a noticeable change accuracy with the best and worst performing entity compared to the baseline or median performing entity. Figure 2 shows Flan- T5-XL results on all validation datasets (analogous GPT-J results are in Figure 5 in the Appendix, and follow a similar pattern). This indicates that spe- cific entities can noticeably alter in-context learn- ing performance. Furthermore, the standard devi- ations for the median and best performing entity across bootstraps are low, suggesting that the selec- tion of different name sets will likely yield similar median and best performance. However, the varia- tion in the worst performing entities is large.
6/
40
02
62
20
Baseline
Median parteriing ty Best performing wittly
Wersst performinggg entiyy
Figure 2: Bootstrap results of Flan-T5-XI XL validation sets for 3 distinct sets of 200 names. Error bars represent the variation across bootstrap sets of names.
Performance Within a Single Bootstrap Set. To analyze the results in more detail, we measure the performance for Flan-T5-XL. and GPT-J single bootstrap set of 200 examples. We show the results for bAbl task 14 in Figure 3. When we replace names only in the demonstrations (D), the vari- ance in model performance lower compared to both when we replace names only in the query (q) and in both query and demonstration (D and q). The (D and q) setup exhibits the highest variance, as this scenario involves the maximum change in the tokens of the original prompt. Nevertheless, consistent variance from the baseline by simply replacing named entities in any of the three replacement settings. Other datasets exhibit a simi- lar behavior; see Appendix Figures 6 and 7.
Baseline results are reported using the original prompt demonstrations and query. However, our replacement strategy introduces frequency bias, e" is used to replace multiple demonstrations in a given input, resulting in artificially increasing the usage of given entity in an input. Thus, we also report the median accuracy of replacement, which serves as a proxy baseline for replacing all demonstrations and/or query with random entity.
2.3 Models and Datasets
We evaluate our entity replacement strategy on both an eecoder-decoder LLM (Flan-T5-XL, XL, 3B; Chung et al. 2022b) as well as decoder-only LLM (GPT- J, 6B; Wang and Komatsuzaki 2021). We test these models across the following datasets: bAbl (Tasks 1, 12, 14, 16; Weston et al. 2015), CLUTRR (Sinha et al., 2019), GSM8k (Cobbe et al., 2021). We chose these datasets primarily because they include a significant presence of named entities.
Best and Worst Entity Generalization. Thus far, we' ve used the valid set to demonstrate accuracy variation. To evaluate the best performing entities on the test set, we implement a majority voting method as follows. We first find the 10 best per- forming entities the valid set. For each sample in the test set, take the majority answer pre- diction from the 10 different best names, denoted "Maj @ 10". Table shows results entity replace- ments across the validation and test sets by replac-
For each d s we select five samples from the training set to serve as demonstrations. To maintain grammatical correctness and clarity,
Exceppt GSM8k, where we select 5 of the prompts from
Wai et al. (2022).

---

ing entities in the demonstration and query (both D and q). We observe a maximum 8.0 point accu- racy increase, seen on bAbl task 16. 6 The best and worst performing entities vary by dataset, but over- all entities generalize across the validation and test sets, with a uniform increase in accuracy from the
bAbl Task (Baseline Accuracy- 78)
Doviation of Accuracy from Baseline
Flan-T5
best entity, A list of these entities for each dataset and further evaluations are in Appendix Table 2.
For all datasets, there are entities with apprecia- ble deviations from the baseline, and these entities generalize across the validation and test splits. This finding indicates that the observed accuracy shifts are not coincidental and highlights the influence of entity swaps, including those that preserve overall example semantics, such as for names, model performance.
bAbl Task 14 (Baseline Accuracy= 70)
Devistion Accuracy from Baseline
GPT-J-60
4 Analysis
-4.
Why are the models not invariant to entity replace- ments? We conducted several analyses in attempts to find the root cause of this result, but they were largely inconclusive.
-8
D
D and
Does the frequeney of names affect how much they shift model performance? The lack of ro- bustness could well be explained by the frequency effects of named entities in the training corpus of the pre-trained model (Wei et al., 2021). We use the entity frequency data provided by Tzioumis (2018), and assessed performance changes across names with varying frequencies. We do not find any correlation between name frequency and the model performance post entity replacement (sce Appendix Section B.3).
Figure Accuracy for Flan-T5-XL and GPT-J-6B on bAbl task 14 for a single bootstrap set of 200 samples.
ter entity replacement to LLM performance, and yet again we are unable to find a statistical corre- lation (see Appendix Section B.5). While Gonen et al. (2022b) suggests improvement of model per- formance with decrease in perplexity, we observe certain entities displaying the opposite behavior.
Conclusion
Does the token length of names affect perfor- mance? Another hypothesis of the invariance could be due the inconsistent tokenization of named entities. Rare names can be tokenized into longer tokens hased on the learned subwords of the model. We analyzed token length post tokeniza- tion and compared it with model performance, but we are unable to find any meaningful relationship between the two (see Appendix Section B.4).
In this study, we find that LLMs exhibit consis- tent lack of robustness with respect to named entity replacement for the in-context learning setting, fur-
Dutaset
Baseline
Best Entity
Worst Entity
Valid
Test
Valid
Text (Maje mi
Valid
Test
bAbl 12
95.1
95.5
98.5
99.0
92.4 93.01-2.5
bAbl 14
76.1
77.5
78.4 (+2.3)
79.5 (+20]
75.2 (.09 76.6
bAbl
53.5
53.3
61.7 (48.2)
61.3 [480]
55.5
[-2.0 55.9 (426)
GSMSK
14.3
10.5
14.6
11.3 p00t)
12.7 (-16) 10.1 (-04)
Does improved performance result from de- crease in prompt perplexity due to entity re- placement? Finally, the surprisal of the LLM to certain entities could be a promising hypothesis to explain the change in model performance (Gonen et al., 2022b). We compare prompt perplexity af-
(42.8)
54.6 51.4 53.1 [+0.1]
Table 1: Performance of FLAN-TS-XL, with accuracy shift from baseline provided in [square brackets]. Entity replacements are done in the D and q setting, bAbl task reaches perfect accuracy and is thus omitted from this table. Performance between the validation and test datasets is consistent, and we observe a generalizable increase in accuracy across all datasets.
"Wo did not include bAbl Task because the haseline accuracy is already 100%, leaving no space for improvement.

---

ther questioning the process of reasoning employed by these models (Min et al., 2022). We demonstrate this across three axes: the complexity of the task, the model architecture, and which named entities are to be replaced (in the demonstrations, query, or both). The cause for this effect remains elusive, which opens up exciting research directions. One intriguing result is that the entities that can boost performance dataset specific. This points to potential avenues for enhancing LLM downstream performance by performing hyperparameter search over range of entities.
of gender-agnostic, gender-ambiguous, or gender- neutral names. Hopefully, additional work such as (Dev et al., 2021; Lauscher et al., 2022; Ovalle et al., 2023) will expand such resources, but re- gardless of these complications, our conclusions are still useful for trying to understand why some names can shift model performance under some settings.
References
2023. Genderize.io. cettpa:/ggedeereee io/. Accessed: June 15, 2023.
6 Limitations
Oshin Agarwal, Yinfei Yang, Byron Wallace, and Ani Nenkova. 2020. Entit--wiiched datasets: An approach auditing the in-domain robustness of named entity recognition models. arXiv preprint arXiv:2004.04123
We acknowledge that the selection of names in our study may not provide an unbiased representation of the complete set of possible names. The data provided by Tzioumis (2018) was derived from mortgage applications which may introduce bias against popular names within marginalized commu- nities. Moreover, this issue is compounded when we sub-sample the set of names due to computa- tional limitations. Additionally, we used a NER model available through Hugging Face to identify e". It was BERT-based (Devlin et al., 2018) and trained on CONLL-2008 corpus (Tjong Kim Sang and De Meulder, 2003) (Wolf al., 2019), and anecdotally observed that sometimes it struggles to identify rare names named entities, for example, it incorrectly identified "TreQuan" as "TreQ." We spot checked for this issue and removed names that posed problems to the best of our abilities.
Sriram Balasubramanian, Naman Jain, Gaurav Jin- dal, Abhijeet Awasthi, and Sunita Sarawagi. 2020. What's 's in a name? are BERT named entity represen- tations just as good for any other name? In Proceed- ings of the 5th Workshop on Representation Learning for NLP, pages 205-214, Online, Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao. Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022a. Scaling caaructioo-finetuun language mod- els. arXiv:2210.114:
7 Ethics and Broader Impact
In this work, we explore the effect of name swaps in an in-context learning setup and find that par- ticular names can increase accuracy. In our ef- forts to align names with their respective pronouns, we assumed binary gender, but acknowledge that other genders exist. We made this assumption be- cause our evaluated datasets used binary pronouns, and our names dataset was also only binarily an- notated. While our paper assumes name-gender associations, we do not want to discredit negative evidence. A name that is not explicitly associated with gender doesn't 't imply evidence of absence of association (imagine that only one person named "Alex" was sampled, and she was woman that doesn't mean that there aren't also men or nonbi- nary people called "Alex"). At the moment, there aren't 't many annotated name datasets that consists
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022b. Scaling catructiionfinetunae language mod- els. arXiv preprint arXiv:2210. 11.116.....
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.1 14168.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips, and Kai-Wei Chang.

---

2021. Harms of gender exclusivity and challenges in non- non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1968-1994, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Robert L Logan IV, Ivana Balažević. Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13::
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arXiv:2202.12887.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805.
Anaelia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai- Wei Chang, Aram Galstyan, Richard Zemel. and Rabul Gupta. 2023, "I'm fully who am": Towards centering transgender and non-binary voices to measure biases in open language genera- tion. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1246-1266.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and De- jing Dou. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
Ji Gao. Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50-56. IEEE.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In MIL- COM 2016-2016 IEEE Military Communications Conference, pages 49- 54. IEEE
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022a. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04033
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shor learning with language models. Ad- vances in neural information processing systems, 34:1 34:11054-11070.
Hila Gonen, Shauli Ravfogel, and Yoav Goldberg. 2022b. Analyzing gender representation in multi- lingual models. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 67-77, Dublin, Ireland. Association for Computational Lin- guistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912. Online. Association for Computational Lin- guistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? strong base- line natural language attack on text classification and entailment. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 8018-8022
Timo Schick and Hinrich Schütze. 2020. Exploit- ing cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07672
Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the modern world of pronouns: Identity- inclusive natural language processing beyond gen- der. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 1221- 1232, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Timo Schick and Hinrich Schütze. 2022. True few- shot learning with prompts-a world perspective. Transactions of the Association for Computational Linguistics, 10:716-731.
Taylor Shin, Yasaman Razeghi, Robert Logaan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.155:
Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. RockNER: A simple method create adversarial examples for evaluating the ro- bustness named entity recognition models. Pro- ceedings of the 202/ Conference on Empirical Meth- ods in Natural Language Processing, pages 3728- 3737, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. 2019. Clutrr: A diagnostic benchmark for inductive reasoning from text. arxiv preprint arxiv: 0917.7
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independde
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint 201::200000600000
named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.

---

Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don't t always say what they think: Unfaithful explana- tions chain-of-thought prompting. arXiv preprint arXiv:2305.04388
Konstantinos Tzioumis. 2018. Demographic aspects of first names. Scientific data, 5(1):1-9.
Ben Wang and Aran Komatsuzaki. 2021. GPTT J-6B: Billion Parameter Autoregressive Language Model. https::/githhr com/ watrofoloo/neesh-rasffoooo
Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan. Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng. Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. 2021. TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 1ith International Joint Conference on Natural Language Processing: System Demonstrations, pages 347-355, Online. Association for Computational Linguistics.
Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201. 1903.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M Rush, Bart Van Merrienboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint 1502.0050
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before Improv- ing few-shot performance of language models. In- lernational Conference Machine Learning, pages 12697-12706. PMLR.

---

A Appendix
in few-shot learning scenarios. In another study, Gonen al. (2022a) introduced novel technique for prompt generation that involves using a small set of manually created prompts and then applying paraphrasing and backtranslation to generate ad- ditional prompts automatically. Specifically, they utilized GPT-3 to perform the paraphrasing step. which involves rephrasing the original prompts while preserving their underlying meaning. The backtranslation step then involves translating the paraphrased prompts into another language and then back into the original language to generate new prompts that are semantically similar to the original prompts but with different phrasing and wording. This approach enables the generation of a large number of diverse prompts without requiring extensive manual effort, which can be particularly useful in few-shot learning scenarios where data is limited. Gonen et al. (2022a) also demonstrated an inverse relationship between the perplexity of the prompts and the model's performance across various tasks.
A.1 Related Work
Several works investigated the robustness of lan- guage models with respect to imput rewrites (Paper- not et al., 6016; Ebrahimi et al., 2017; Gao et al., 2018; Jin al., 2020). In particular, Balasubrama- nian et al. (2020) found that while fine-tuned BERT models were brittle in certain tasks, they demon- strated robustness against entity replacement for question answering. Other works have shown that many transformer- based models are insufficiently robust to name swaps (Ribeiro et al. 2020; Agarwal al. 2020; Lin al. 2021; Wang al. 2021, i.a.), Our work differs from these in that we focus on the in-context learning setting.
The ability to do in-context learning first emerged from Brown et al. (2020). Subsequently, many works sought to improve and understand the abilities of LLMss with respect to in-context learn- ing (Liu et al., 2021: Zhao et al., 2021). Perez et al. (2021); Logan IV et al. (2021); Schick and Schütze (2022); Turpin al. (2023) show improvement over the original paper in the the "true few-shot learning" setup where there is no access to a large test set to tune in-context learning hyperparameters. Despite the impressive results of this new paradigm, little has been understood regarding the generaliza- tion of certain in-context demonstrations.
Our work related to previous studies as we fo- cus on the true few-shot learning setup for the task of question answering and explore the robustness of NLP models against entity replacement. There are three main ways in which our work differs from previous studies. First, our investigation focuses analyzing the robustness of language models against entity replacement specifically for the task of question answering in the ICL setup, which has not been explored in previous studies. Second, we utilize left-to-right language models, which are more widely used and powerful than masked language models. Finally, our approach involves prompt-learning instead of fine-tuning, whereby the parameters of the language model are modi- fied to improve its performance on specific tasks. By leveraging the true few-shot learning setup and prompt-leariing techniques, we aim to develop a more robust and effective approach for few-shot question- cantiionnnnseringg.. tasks that can handle entity re- placement and other challenges more effectively.
Gonen et al. (2022a) bring to attention the im- portance of the specific prompts in model perfor- mance. They show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task. Min et al. (2022) show that the actual labels in the prompt are not very important. However, the labels still have a small impact in performance.
In recent years, several studies have explored the significance of prompt quality in few-shot learning, particularly in the context of guestion-answering tasks. One such study was conducted by Shin et al. (2020) who proposed AutoPrompt, an automated approach for prompt generation in few-shot learn- ing scenarios that has been shown to achieve the highest accuracy. The AutoPrompt model lever- ages gradient-based prompt search techniques to find the optimal prompt for a given task, thereby reducing the need for human intervention. Addi- tionally, AutoPrompt utilizes Masked Language Models (MLMs) to perform sentiment analysis and natural language inference tasks without the need for tueituning, highlighting the potential of MLMs
A.2 Dataset Details
bAbl (Weston et al., 2015): The bAbl dataset consists of 20 tasks that evaluate various aspects of language understanding by NLP models. In our experiments, we select four tasks representing varying degrees of difficulty from the bAbl dataset: task 1, task 12, task 14, and task 16.

---

context: Mary and [e travelled to the bathroom. John and [e travelled to the office. question: Where is le,.
ply this transformation k demonstrations: D d1, d2,..,, ..., dg and the final query 9, where the an- swer to q is omitted. We concatenate the format- ted demonstrations and query q. delimiting each string with a newline character followed by "###" and newline character. Figure 4 shows an in- context learning example.
answer: office ##
context: [e and Mary went back to the office.
Daniel and Sandra went to the bedroom. question: Where is le,
answer: bedroom ###
A.3 Entity Selection
context: Daniel and John moved to the hallway. [e and Daniel journeyed to the bathroom.
In general, we determine e" by selecting the first encountered entity within the question, as often represents the subject of the question. the ques- tion contains interrogative pronouns, such as "who," " no entities will be present within the question. In this case, we consider e" as the first entity encoun- tered within the answer since it typically represents the subject being referred to. If the subject of the question and answer is referred using pronouns then neither the question nor answer will contain any entities, we assume that noun ambiguity is not an issue and select the first encountered entity within the context as e..
question:
Where is [e,]?
answer:
Figure 4: Named entity replacement overview for in context learning. Here we show an in-context learning example with k=2 demonstrations before the final query. We consider 3 scenarios: replacing only the demonstra- tion entities (eai, e22), only the query entity (ey), or both (ca1, 6d2. fg). replacements are done with the same replacement entity e".
CLUTRR (Sinha et al., 2019): This dataset is similar to a subset of the bAbl dataset, but contains more diffieult multi-step questions. This allows to determine whether named entity replacement robustness affected by task difficulty. Given a multi-sentence context describing kinship relations between certain entities and a question, the task is to predict the kinship relation between the entities that can be inferred from the context but is not made explicit. We convert this into multiple- choice question where we expect the model to pick the answer from the four choices provided.
A44 Entity-replacemmnt Scenarios
In the demonstration-only replacement scenario, perform entity replacement exclusively on the demonstrations eş while keeping the query unmod- ified 4. By selectively replacing entities within the context, question, and answer of the demon- strations, we aim to examine the extent to which this replacement strategy influences the model's performance.
In the query-only replacement setting, we solely augment the entities within the query Gg while leav- ing the demonstrations unaltered Fig This al- lows us to investigate the impact of enhancing the query's 's entity representation on the model's 's ability to generate accurate responses.
GSM8k (Cobbe et al., 2021): This dataset is collection of elementary multi-step mathematical reasoning questions. We specifically selected this dataset due to the challenging nature of arithmetic operations, where existing models often struggle to achieve accurate results. We subsampled GSMSK to include only examples which contained identifi- able named entities.
Applying both demonstration and query replace- ment, we replace ed and eg in both the demon- strations and query simultaneously. By enriching the entity information in both the context and the question, we explore the combined effect of en- tity replacement on the overall performance of the model.
The Tzioumis (2018) dataset, contains of 4,250 unique first names, associated with their unigram frequency, and self-reported demographic informa- tion.
For each sample in the dataset, which is com- prised of context, question, and answer triplet, we transform into a formatted string. This trans- formation involves concatenating the field name, a colon, space, the respective value correspond- ing to that field, and newline character. We ap-
A.5 Ensuring Grammatical Correctness
To begin, we verify the pronouns associated with both 6 and e" using (gen, 2023). If the gender associated with e" corresponds with that of e", we proceed to substitute e" with e", However, there

---

is no alignment in gender, we refrain from altering ex and maintain its original form.
ing entity matched the gender of the name in the query (as replacements wouldn't 't occur otherwise). However, we chose not to adjust the test dataset to ensure that our accuracy improvement could be compared with other studies.
B Analysis
B.1 Analyzing Best/Worss performing entities In order to examine the generalizability of the best and worst performing entities in each dataset, we extracted named entities from the validation dataset and tested them on the test dataset. More specifi- cally, we replaced the named entities in the prompts of the validation dataset with set of names that included 100 male and 100 female names. We then calculated the accuracy for each name, selecting the 10 names that exhibited the highest performance and the 10 names that showed the lowest perfor- mance. Next, we replaced the named entities in the test dataset prompts with these 20 selected names and calculated the corresponding accuracy for the test dataset. Table 2 presents the best and worst performing entities for each dataset along with the absolute percentage of accuracy changes. The en- semble model functions by replacing the named entity in the prompt with the highest performing entity and then generating an answer at each step. After cycling through the best performing enti- ties, it selects the answer with the most votes as the final response. This approach is referred to as ma- jority @ 10. It's important to note that the bAbl task doesn't 't have a 'best performing entity" because the baseline accuracy is already 100%, leaving no room for improvement.
The consistency in performance between the val- idation and test datasets in CLUTRR is lower than that in the other datasets. This could be attributed to the structure of CLUTRR, where four options are provided in the prompt, from which the model chooses an answer. This setup introduces a de- gree of randomness to the generated answer, po- tentially leading to less consistency between the performance of the validation and test datasets.
B.2 LLM Architecture
The architecture of LLMs plays a crucial role in determining their behavior and performance, in- cluding their vulnerability to entity replacement. The design choices and architectural components of LLMs can influence how they process and gen- erate text, affecting their sensitivity to changes in named input entities. By comparing the Flan- T5-XL, which employs an encoder-decoder archi- tecture, with the GPT-J-6B, which represents a decoder-only architecture, we aimed to investigate the impact of LLM architecture on their brittle- ness towards entity replacement. The high variance in the performance of GPT-J (as shown in Fig 7) underscores that the brittleness towards entity re- placement is not exclusive to the encoder-decoder architecture but is also a feature of decoder-only architecture models.
Figure 8 provides an example of an generated answer for the GSM8k test dataset. The original prompt failed to produce the correct answer, while the transformed prompt was able to successfully answer the question. The absolute improvement of 0.8% in the accuracy of GSM8k through the majority @ 10 is remarkable, given the following facts: 1) The baseline aceuracy for this challenging dataset is 10.5%, making even small improve- ment significant. 2) The number of named entities in GSM8k is fewer than other datasets, meaning only a few words are replaced in the transformed prompt, as shown in Figure 8. The ratio of replaced words to the total number of words in the prompt is 0.032. 3) Of the 1319 examples in the GSM8k test dataset, 663 contain male name, 377 contain female name, and 279 don't contain any named entity the query. We could have achieved greater changes in percentage if we had only considered examples where the gender of the best perform-
B.3 Effect of entity frequency
Frequently recurring named entities in the web, which subsequently appear in the training data, can impact both the model's ppl and performance. En- tities appearing frequently in the training data are more likely to be well-represented in the model's understanding, contributing to lower ppl values. This occurs because the frequent presence of these entities reduces the unpredictability of the next word in a sequence, making it easier for the LLM to anticipate. The hypothesis is that names appearing more frequently would contribute to better model performance. To examine this hypothesis, the fre- quency of names, extracted from (Tzioumis, 2018), were categorized into five groups of equal num- bers, labeled G1 to G5, ranging from the most the least frequent names. Fig 9 presents the per- formance of Flan-T5 T5-XL across different datasets.

---

bAbl Task
bAbi Task 12
90
Baseline
Baseline
Worst performing entity
90
Worst performing entity
Median performing entily
Median performing entity
Best performing entity
Best performing entily
85
80
Accuracy
Accuracy
75
75
70
70
60
60
Demonstration
Both Query and Demonstration
Query
Demonstration
Bath Query and Demonstration
Query
bAbl Task 14
bAbl Task 16
80
60
55
50
70
Accuracy
Accuracy
65
60
Baseline
Baseline
Worst performing entity
30
Worst performing entity
Median performing entily
Median performing entily
Best performing entity
Best performing entity
50
25
Demonstration
Both Query and Demonstration
Query
Demonstration
Both Query and Demonstration
Query
Figure 5: Bootstrap results of GPT-J-6B for 3 distinct sets of names, each comprising 200 names. Error bars represent the variation across different name sets.

---

[Baseline Accuracy- 100)
12 [Baseline Accuracy
0.0
2.5
Deltation of Nccratty frrm Baseline
0.5
Deviattin of Accunery from Bleeeine
-2.5
-5.0
-20-
-25
-3.0
-10.0
-12.5
-15.0
Demoritration
Beth Query and Domonsstaaiio
Query
Demonstration
Both Query and Demonstration
Query
bAbi Task 14 (Baseline Accuracy= 78)
bAbi Task 16 (Baseline Accuracy= 56)
Devietion Acconary from thseline
10-
Devistion Accurccy
Demonstration Demonstration
Query
Demonstration
Demonstration
Query
Accuracy
Accuracy
Doviel ton Accurayy Baveiie
Dosiation Accuracy Basellre
Demorktration Both Query Demanstration
Query
Demanittration
Both Query Demonstration
Querry
Figure 6: Deviation accuracy from the baseline (without data augmentation) for Flan-T5-X across different datasets.

---

wAbi Task (Baseline Accuracy= 75)
bAbl Task 12 (Baseline Accuracy 77)
Doviaton of Accuracy from Baseline
Deviation of Accuracy from Basollie
2-
4-
Demonstration
Both Query and Demonetration
Query
Demonstration
Both Duery and Dem Demonstratinn
Duery
Taak (Baseline Accuracy-
bAbi (Basalina Accuracy
Devistoon of Accurrry from Bazaine
Deviston of Accurary from Basaine
2.5
5-
0.0
-2.5
0-
-10.0
Query
Query
Query Demonstration
Duery
Figure 7: Deviation in accuracy of GPT-J from the baseline across various bAbl tasks. Like the encoder-decoder architecture, the decoder-only architecture also exhibits brittleness towards entity replacement.
Original
Transfored
Douglas)
context chocolates
32 chocolates Creganlly, Leeh 52 chaoclates Her vader hadi So had Witer earing 30, they hod 76 36 sttre 35 pia
arowwer Originally, Lesh chocolales, tie had So had 42
After eating 35, they had 74 3 38, dreth 39
asa
connect Insan had pape gave Denny some toll polipope. Jasont has willoops. cucalion: many silloopa Aascol give Donny?
conteot: Bouglas tallpope, Hi gave Denny some lollipppe Now Bluugan has tolloops. question
loll ppop Dusplai give Denny? 12
Diwny W wehee
201 12 mede
act
Atd
connect Standil han frem loys. for Chitetman, te get two taps each from the mom and diec.
ronteot: Bousles hos them trys. For Che atmos te got two toys each from has mam and ded
question: How many 1oys doos have now?
Douglas queston: How many tays does have now?
Shawk
2 toys
get toys
wite
connect Michaal had 68 balls, On tauuday,, lost 28 tualls. wed wudressios, lost mom
contact: Douglai had 68 palf balls, On tuoudaay, he losi pelf buetss On wediisddee loxt more.
quention: kow many gott
qu queetoo many golf tottle
wnndewstay??
anower. Michsal
pulf
losing suosday.
Dooplaa
galf After losing taasday
Nit
connect Clivks mes $25. She boughe begais for wach
contect: Olivie $23, She bought fiee begala wech queetion His much money dowe she terve bett??
queston: How much money dass alse have inff?
answer: Olidk had dellors, togals pisllsio for dollars each be 5 dollors. the has
answer: Olivia had 23 corlars, bagels for dellors each will be 6 - dollon, she has
dollaaa
Biobhan
Slubi jewels.
Rismmndd sowlls.
quention: It Beymmnt har 4ll jeawrs, how many jeawis dons Sustton tave??
q inrtion Bougles hax 40 jeuvcs, how many jeowtss cose Swethan have? answer:
anower
Generated
prompt
Generated Answer Flan-Ts TL with transformed prompt
jowe's
Douplas's
more jove's than Douglas's
Figure 8: example of generated answer from the GSM8k dataset. In this instance, the original prompt fails to generate the correct answer, whereas the transformed prompt with the best-performing entity replaced successfully produces the correct response. This generated text is reproducible using a greedy decoding.

---

Best performing entity
Worst performing entity
Index
entity
Acc change (valid)
Ace change (lest)
entily
Acc change (valid)
Ace change (lest)
Penclope
Loreen
3.6
3.0
Raisn
4.0
-3.6
Dorcas
3.6
3.2
Ethel
-17
.1.5
Toby
Migdalia
bAbl task
Hoor
3,3
2.9
Azucena
-2.6
-3.2
valid baseline Ace 95.1
Nataliya
3,3
3,4
Bhavna
-2.6
-3.6
baseline
Nathaniel
Lucretia
3.3
3.3
Sanjeev
-1.9
1.7
Keiko
3.2
3.0
Armida
-1.8
-11
-1.7
Majority@10
3.5
Frich
2.9
2.0
Frederick
-1.4
-1.2
Jasqueline
2.6
1.8
Raisa
-1.3
-13
Zoltan
2.4
-066
Chrystal
2.3
1.5
Mecra
-0.5
-0.5
bAbl tack
Petm
2.2
1.1
baseline
test baseline Ace = 77.5
Claudio
2.2
1.0
Sanjecv
2.2
1.1
2.1
Prashant
2.1
2.3
Majority@1
2.0
Loyd
8.9
8.4
Armida
8.7
Brannon
8.7
9.5
Pravin
8.4
838
bAbl lask
Bozena
valid baseline Ace 53.5
Surinder
8.0
7.2
test huseline Acc 53.3
Bonita
8.0
7.1
Trena
Rosalia
7.9
6.2
Lespoldo
7.7
9.3
Majority
Douglas
0.5
0.9
Lespoldo
2.5
0.8
Juliann
0.4
0.1
Youssel'
-19
0.3
Kathloen
0.3
Epifanio
-1.7
Fimst
0.3
0.4
Sanjewv
-1.6
-08
GSMSKK
Toeja
0.3
0.0
Zoltun
-1.5
-03
valid baseline Ace = 14.3
Augela
0.2
0.2
Jaaawei
-1.4
4.4
baseline
Hina
Diego
0,2
0.4
Alphonse
-1.3
-0.7
Bradley
0.1
0.9
Dagoherto
-1.3
.0.5
Jonaihan
Nathuniel
-1.2
Majority @:10
0.8
Ashlee
4,4
1.2
Rob
-8.2
0.6
Penelope
3,4
3.4
Ahmad
-8.0
402
Allyson
3,2
1.0
Milai
-7.6
0.6
Chrystal
2.8
2.6
Frederick
-7.6
0.4
2.8
Martin
-7.4
valid baseline Ace- 59.0
Alisha
2.6
3.0
Jake
-7.4
-0.4
test baseline Acc = 53.0
Rosalba
2.4
1.8
Edward
-7.4
0.2
Jasqueline
2.2
2.0
Erik
-7.4
0,2
Zoltan
2.2
1.2
Giovanni
-7.4
-0.4
10
Dillon
2.0
1.4
Douglas
-7.2
-0.2
Majority 10
1.6
Table 2: Performance of the best and worst performing entities in the validation and test datasets. The green color signities consistent performance between the validation and test datasets, while the orange color indicates inconsistency. The entities, selected based on their performance in the validation dataset, show consistent behavior in the test dataset, suggesting that they generalize in each dataset. We have noted less than 10 entities in some columns because the remaining entities did not reduce the accuracy compared to the baseline.

---

Contrary to our hypothesis and expectations, there appears to be no correlation between the frequency names and the performance of LLMs entity replacement.
of the same named entity can contribute to lower ppl due the decreased uncertainty and increased predictability the next word in the sequence.
B.4 Effect of entity token length
The token length of a prompt can directly influence the performance of QA task, particularly as it relates to the computational demands on the model. A longer prompt, represented by larger number of tokens, increases the amount of data the model must process, potentially straining the model's ca- pabilities and affecting the overall efficiency and accuracy of the task. Changes the token length of a replaced name, which in turn modify the prompt's s overall token length due to multiple instances of the name within the prompt, provide an interesting variable for analysis. We delved into this potential effect by examining the model's performance in re- lation the token length of names, as illustrated in Fig 10. Contrary expectations, our study found no observable correlation between the model's per- formance and the token length of names. This suggests that, within the ranges examined, the to- ken length of names does not significantly affect the model's performance in QA tasks.
B.5 Perplexity Analysis
Our expectation was to observe strong inverse correlation between the perplexity and accuracy of the model across all datasets, according to the findings reported in (Gonen et al., 2022a). How- ever, we noted this inverse correlation only some datasets, such as the bAbl task 14 (as shown in Figure Fig 11), but not all of them. A plausible explanation for this inconsistency could be signifi- cant differences in our methodology compared to the approach adopted in (Gonen et al., 2022a). In study, limited alterations to substituting named entities within the prompts, as opposed to changing the entire prompt. Therefore, this dis- crepancy data augmentation techniques could potentially account for the lack of a significant cor- relation in some datasets.
Fig 12 shows the deviation of ppl from the base- line across various tasks. Regardless of the task, reduction in prompt ppl was observed when entities were replaced both in demonstrations and queries, compared the other two data augmen- tation settings. This can likely be attributed to the repeated usage the same replaced named entity within the prompt. The frequent repetition

---

100.0
98.5
Accuracy
Accuracy
90.0
96.5
36.0
Swapped names Query and Demonstration for bAbl Task
Swapped names Query and Demonstration for bAbi Task 12
Frequency names (G) most frequent gmuu)
Frequency names frequent group)
86
82-
Accurazy
80
76
Swapped Demonstration for bAb Task
50-
Swapped Demonstration bAbi Taak 16
72
G3
Frequency of namas (G1 is most frequent mmup)
Frequincy of names (G1 most frequent group)
66
64
Accuracy
62-
Swapped Query
Smapperd Query
GSMBK
G1
G3
G4
G1
G2
G3
Prequency of names (G1 is most trequent group)
Frequency of names (G1 most frequent group)
Figure 9: The performance of Flan-1 T5-XL XL in relation to the frequency of names derived from a demographic dataset. Contrary to expectations, no correlation appears exist between the frequency of names and performance Flan-T5-XL

---

100.0
n: 100
235
160
62
m: 15
R1 100
15
98.5
Accuracy
Accuracy
90.0
96.5
86
36.0
Swapped names Query and Demonstration for bAbi Task
Swapped names Query and Demonstration for bAbi Task 12
Token lengt of entlly using Flan-T5 tokenizer
Tokan length entity using Flan-TS takanizer
86
in: 100
335
82-
n::
n::
65
n1 235
11 188
I:: 62
15
Accurazy
80
76
Swapped Query Demonstration for bábl Tesk
50-
Swapped Demonstration bAbi Taak 16
72
Token length of entily using Flan-TS tokanizer
Token length of entity using RlanTT takenizer
60
15
66
64
In: $2
Aecuracy
Accuracy
62-
3.0
Swapped Query
Smapperd Query
GSMBK
Token langth of entitt using Plan-75 tokenzeer
Taken length entity using RonTS takenizer
Figure 10: The performance of various names in relation to their token length when using Flan- T5-XL. XL. There is apparent correlation between performance and the token length in names. quantity names corresponding to each group of token lengths is illustrated on the plot.

---

Demonstratios Only (lask12)
Query and Demansiration. (wsk12]
Query Only (lask12)
pootum 0.011
pootunm 0.30
poolin 0.0032
000
24
00
00:0000
cew
000
Figure 11: Correlation between prompt perplexity (ppl) and LLM performance for task 14 and GPT-J-6B across various data augmentation settings. The accuracy and perplexity of prompt for the baseline are shown by red lines. Although a statistically sigmificant inverse correlation between accuracy and ppl is observed in this dataset, this trend is not consistent across all datasets. The absence of this trend, contrary to expectation, could be attributed to the nature of our data augmentation approach, which only involved entity replacement instead of entire prompt modification.
Task1 (Baseline 3.00)
Task12 (Baseling- 3.311
0.3
2.4
Domoristation
seth Dusry Duncrstration
Query
Demordistinn
Bath Query Demerstatin
Qudt
Task24 (Bagelines 3.52)
Task16 (Baselines: 2.6
0.4
Devittion pooretaap seeeire
0.03
2.03
Doriation
-0.
-0.15
Donurittation
aoth Query Domenctration
Query
Demunstracnn
Bith Query and Demerstation
Quetr
Figure 12: Deviation of prompt ppl for GPT-J-6B from the baseline across different tasks. A greater reduction in prompt ppl is observed when entities are replaced in both demonstrations and queries, a trend likely driven by the recurrent usage of the same replaced named entity within the prompt.


Reviews:
[{"review_id": "zYjyUdAeji", "reviewer": "Reviewer_4TcR", "summary": "", "questions": "Refer to W3. Would like to see possible discussions on NE replacement on **different** parts of the exemplar.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The robustness of entity replacement in in-context learning is indeed an area that urgently needs exploration. Although the authors were unable to identify the root cause, this provides a basis for future prospects in the robustness of ICL for LLMs.\n\nS2. The writing is easy to follow with clear motivation. \n\nS3. The authors provide detailed steps and results in the appendix, which I believe interested readers could easily reproduce and get findings.", "reasons_to_reject": "W1. The authors failed to identify the root cause of the performance change due to entity replacement in ICL. However, it is not a major flaw, especially considering this paper is a short submission. \n\nW2. Possibly constrained by budget, the authors did not test the robustness of the OpenAI API in terms of entity replacement. I believe that as the model and data scale, the impact of entity replacement on validation results will become increasingly minor.\n\nW3. The comparison among `D only`, `D and q`, `q only` settings is not clear. Would like to see some possible reasons (discussions) on this, e.g. why `D only` and `q only` settings have different results on different benchmarks.", "questions_to_authors": "Refer to W3. Would like to see possible discussions on NE replacement on **different** parts of the exemplar.", "paper_topic_and_main_contributions": "This paper explores the robustness of in-context learning (ICL) for named-entity replacement. Specifically, the authors find that ICL exhibits bias towards specific entities in several entity reasoning test sets. The contributions are as follows:\n\n1. The authors conducted exhaustive experiments, attempting to contrast the performance differences in ICL for entity replacement using two underlying models, Flan-T5-XL and GPT-J.\n2. They designed reasonable constraints to ensure the semantic-preserving nature of entity replacement, such as the `Grammatical Correctness Check`.\n3. The authors did not find a significant influence (correlation) of entity frequency, prompt perplexity, and token length on ICL entity replacement.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YNSkWaR0cv", "reviewer": "Reviewer_iCLV", "summary": "", "questions": "Do you plan on open sourcing the code to facilitate reproducibility?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-Clear aim and presentation style\n-Experiments are performed on a quite large number of datasets making the results sound.\n-Good efforts to account for noise in the training set-up.\n", "reasons_to_reject": " - Premise that name changes shouldn’t alter the answer is not necessarily true. There are entities and first names (Jason (Bourne/Statham) Action Heroes) that are connected to certain characteristics, which should alter the answer probability. This phenomenon is often used to write jokes or tales (e.g. the Fox, Loki, The Crow, etc. for wittiness, or Ronaldinho for his football talent, Giovanni as an Italian, Huckleberry Finn). “The Fox has the ball. The Fox went to the kitchen. Where is the ball? Probably not in the kitchen because the Fox is a known symbol for trickery.” Furthermore, there are first names like Alice and Bob, who are often used in the context of word problems in mathematical tests.\n- The analysis of the problem seems superficial. Assuming the premise would be right the authors don’t account for contextual assumptions inherent in names. For example, the embedding of Johnson is probably more related to chemicals due to the company Johnson&Johnson in comparison to Wendy who is probably more related to food because of the fastfood chain. They should have grouped contextual similar first names and test for intra cluster differences. This semantic shift per first name would not necessarily show in a metric like perplexity (The results hint at correlation like this as the best performing entities vary per dataset.)\n-The authors don’t indicate if they share the dataset making the impact of the paper quite small\n", "questions_to_authors": "Do you plan on open sourcing the code to facilitate reproducibility?", "paper_topic_and_main_contributions": "The paper investigates how in-context learning of LLMs is impacted by entity changes. The authors contribute a sound experiment (several dataset and two sota models) with a short analysis of the results.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The authors have a good language and presentation style."}, "extra_text": {}}, {"review_id": "ueZpdXTTx1", "reviewer": "Reviewer_a4iF", "summary": "", "questions": "Question A. Can the entity replacement in the paper be applied to other types of entities? \n\nQuestion B. In Figure 2, why the entity replacement strategy D has a lower accuracy than other two strategies? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explore the robustness of the in-context learning paradigm by focusing on entities, which may be a new research direction for enhancing LLM downstream performance by searching over a range of entities.\n\nThe paper conducts extensive experiments on three datasets, and did an in-depth analysis.\n", "reasons_to_reject": "(1) The experiments and analysis are limited to first names, which will affect the extensibility and applicability of the methods.\n\n(2) More discussion should be provided in Section 3, which just describes the phenomenon. For example, In Figure 2, what’s the possible reason that the entity replacement strategy D has a lower accuracy than other two strategies. \n", "questions_to_authors": "Question A. Can the entity replacement in the paper be applied to other types of entities? \n\nQuestion B. In Figure 2, why the entity replacement strategy D has a lower accuracy than other two strategies? \n", "paper_topic_and_main_contributions": "This paper explores the robustness of the in-context learning paradigm by focusing on entities, and finds that LLMs exhibit a consistent lack of robustness to entity replacements for in-context learning.\n\nThe paper conducts the experiments on three datasets with both an encoder-decoder LLM (Flan-T5) and a decoder-only LLM (GPT-J), and also tries to explain the results by analyzing frequency, token length and perplexity of the replaced entities. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the robustness of LMs to choice of named entities (e.g. person name) and their effect showing that there is high variance in downstream performance when NEs are substituted. This is an interesting finding motivating the community to build stronger ways to validate models or build more robust models. The paper presents the findings only, but does not present any suggestion on how this could be improved. I think this is a reasonable contribution for a short paper. The paper presents other analyses regarding the frequency and token length.  There is a reasonable consensus among reviews that the the methodology is sound, but the excitement scores are mixed. I agree that the work has potential, but without mitigation, it is hard to take any action about this finding other than to check for it in other models.

Reviews iCLV raises some concerns that we do want models to be sensitive to NEs. Premise that name changes shouldn’t alter the answer is not necessarily true. I think this is an interesting discussion point that the authors should further address where name sensitivity is recommended and where models should be expected to be insensitive to this.
Decision: Accept-Findings