Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
Authors: Robert Litschko
Max Müller-Eberstein
Rob van der Goot
Leon Weber-Genzel
Barbara Plank
Venue: EMNLP 2023 Main (2023)
Keywords: Trustworthiness
Tasks
Evaluation
Skills
Trust

Abstract: Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model's functional capacity, and provide recommendations for more multi-faceted evaluation protocols.

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
Robert Litschko 1.3+
Max Müllerr-bersteinn* Rob van der Goot
Leon Weber!.3 Barbara Plank
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany
2 Department of Computer Science, IT University of Copenhagen, Denmark
3 Munich Center for Machine Learning (MCML), Munich, Germany (mamy, robv]@i tu. dk
{rlitschk, leonweber, bplank]@cis, de
Abstract
Language understanding is multi-faceted cog- nitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Tradition- ally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evalu- ation protocols. With the advent of large lan- guage models (LLMs) the community has wit- nessed dramatic shift towards general pur- pose, aask-agnottic approaches powered by gen- erative models. As a consequence, the tradi- tional compartmentalized notion of language tasks is breaking down, followed by an in- creasing challenge for evaluation and analysis. At the same time, LLMs are being deployed more real-world scenarios, including previ- ously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time rethink what constitutes tasks and model evaluation NLP, and pursue a more holistic view on language, placing trustworthiness the cen- ter. Towards this goal, we review existing com- partmentalized approaches for understanding the origins of model's functional capacity, and provide recommendations for more multi- faceted evaluation protocols.
Tosk
Exprossion
Expectation
formalizaaion
Datoset
Input
Output
Neric
Prediction
Evaluation
Model
Trust
Figure 1: Contemporary NLP Paradigm with lan- guage tasks formalized as datasets for which models pro- duce predictions. Recent LLMs break down this com- partmentalization (dashed lines), impacting all stages of the cycle. We argue that establishing trust requires re- thinking every facet of this framework, as formalization and evaluation become increasingly difficult.
as late 2022, we are witnessing a paradigm shift: Powerful LLMs, in the form of instruction-tune prompt-based generative models such as ChatGPT and GPT-4 (Wei et al., 2022a; Touvron et al., 2023b; Taori et al., 2023; OpenAI, 2023; Bubeck et al., 2023, inter alia), have found widespread adoption reaching far beyond the NLP community. Part of this success story is the casting of heterogeneous NLP tasks into sequence-to-sequence tasks (Raffel et al., 2020; Sanh et al., 2022; Wang et al., 2022b); which in turn enables extreme multi-task learning. and cross-task transfer learning.
"Trust arises from knowledge of origin us well as from knowledge of functional capacity."
Trustwortinesss Working Definition David G. Hays, 1979
1 Introduction
This is in stark contrast to the traditional com- partmentalized NLP paradigm (visualized in Fig- ure 1), wherein human- motivated language task with input expression and output expectation is clearly formalized into a dataser with machine- readable inputs and outputs. Both feature design and model development are highly task-specific- often manually curated. Paired with evaluation protocols for comparing model predictions with
Understanding natural language requires a multi- tude of cognitive capabilities which act holistically form meaning. Modeling this ability computa- tionally y is extremely difficult, thereby necessitating a compartmennalizzation of the problem into iso- lated tasks which are solvable with available meth- ods and resources (Schlangen, 2021). Undoubtedly
human-motivated
Equal contribution.

---

human expectations via formalized metrics or qual- itative judgement, this general methodology has been widely adopted and trusted. ! However, with contemporary LLMs this compartmemnalization is breaking down-having severe impacts on all stages of the cycle. Therefore, a persistent and crit- ical question regains importance: How can trust be established between the human and the model?
DI. Knowledge about Model Input. In the be- ginnings of NLP, researchers followed strict, task- specific formalizations and had precise control over which "ingredients" go into model training and inference (i.e., manual feature engineering). Neu- ral models have caused a shift towards learning representations, improving performance at the cost of interpretability. While analogy tasks (Mikolov et al., 2013) have enabled analyses of how each word-level representation is grounded, contempo- rary representations have moved to the subword level, and are shared across words and different lan- guages, obscuring our knowledge of the origin of their contents, and requiring more complex lexical semantic probing (Vulié et al., 2020, 2023). This is amplified in today's instruction-based paradigm in which tasks are longer formalized by NLP re- searchers and expert annotators but are formulated as natural language expressions by practitioners and end users (Ouyang et al., 2022). The cognitive process of formalizing raw model inputs into ML features has been incrementally outsourced from the human to the representation learning algorithm, during which we lose knowledge over functional capacity.
As early as 44 years ago, Hays (1979) offers an attempt and provides a definition of trustworthiness (cf. quote). Today, the topic of trustworthiness is an ongoing discussion deserving special attention (Baum et al., 2017; Fisenstein, 2022; Clarke et al., 2023). We argue that to establish trust, it is time to rethink how we deal with tasks and their evaluation. Why now? It is getting increasingly hard to predict a priori when we can expect models trained on web-scale data to work well. Were we to live in a hypothetical world with full knowledge of origin and functional capacity, then each task instance could be routed to the right model(s) to not only tap into the LLMs' full potential, but to also enable trust in their predictions. Today, the absence of this knowledge directly linked to our lack of trust in deploying models in real-world scenarios.
In this position paper, we synthesize contem- porary work distributed throughout different sub- fields of NLP and ML into a conceptual framework for trust, guided by Hays (1979)'s 's definition and centered around knowledge facets guiding prin- ciple for all aspects of the model development and evaluation cycle. We outline high-level desiderata ($2), and suggest directions on how to gain trust, by providing starting points of facets ($3) aimed to stipulate uptake and discussion. In $4 we discuss how trustworthiness relates to user trust.
D2. Knowledge about Model Behaviour. In the old compartmentalized view of NLP, higher-level tasks are typically broken down into pipelines of subtasks (Manning et al., 2014), where inspect- ing intermediate outputs improves our knowledge about model behaviour. Recently however, LLMs are usually trained on complex tasks in an end-to- end fashion (Glasmachers, 2017), which makes it more difficult expose intermediate outputs and analyze error propagation. Over time we have gained powerful black-box models, but have lost the ability to interpret intermediate states and de- cision boundaries, thus increasing uncertainty and complexity. Because as of today, we cannot build models that always provide factually correct, up-to- date information, models at a large scale, in where reliability and transparency are key. In this regard, pressing questions are e.g., how hallucina- tion and memorization behaviour can be explained (Dziri et al., 2022; Mallen et al., 2023), how models behave when trained on many languages (Conneau et al., 2020; Choenni et al., 2023), what internal features are overwritten when trained on differ- 'While not without deficiencies, evaluation protocols were arguably more heterogeneous and established than today w.r.t. quantitative/uuaiini evaluation, human judgements etc.
2 Desiderata for Trustworthy LLMs
LLMs today pose a conundrum: They are seem- ingly universally applicable, having high functional capacity, however, the larger the model, the less we appear to know about the origins of its capabilities. How did we get here, which aspects contribute to trustworthiness, and what did we lose the way? In the following. we aim provide brief his- tory of central trust desiderata (D1-4), discussing how our knowledge of functional capacity and its origins has changed over time.
cannot trust to employ these
scenarios,
We refer to ingredients as explicit inputs and L.M''s para- metric knowledge (De Cao et al., 2021; Mallen et al., 2023).

---

ent tasks sequentially (catastrophic forgetting; e.g..
train-dev-test splits and overestimating the capa- bilities of LLMs. To compound matters further, models are not only trained on natural, but also on generated data, and unknown data provenance is also becoming an issue as annotators start to use LLMs (Veselovsky et al., 2023). LLMs trained on data generated by other LLMs can lead to a "curse of recursion" where (im-)probable events are over/uundrrstimatted (Shumailov et al., 2023).
McCloskey and Cohen, 1989; French, 1999), how to improve models' ability to know when they do not know (model uncertainty, e.g., Li et al., 2022a), or how do LLMs utilize skills and knowledge dis- tributed in their model parameters.
D3. Knowledge of Evaluation Protocols. The emergence of LLMs has raised the question of how to evaluate general-purpose models. Many recent efforts have followed the traditional NLP evaluation paradigm and summarized LLM per- formance into evaluation metrics across existing benchmark datasets (Sanh et al., 2022; Wang et al., 2022b; Scao et al., 2022: Wei et al., 2022a; Touvron et al., 2023a). This estimates LLM performance for tasks covered by the benchmark datasct and thus establishes trust when applying the model to the same task. However, the situation is different when L.MSs used solve tasks outside of the benchmark, which is often the case for real-world usage of LLMs (Ouyang et al., 2022). Then, the expected performance becomes unclear and bench- mark results become insufficient to establish trust. One proposal to solve this issue is to evaluate on a wide variety of task-agnostic user inputs and report an aggregate metric (Ouyang et al., 2022; Chung et al., 2022; Wang et al., 2023b; Dettmers et al., 2023). This approach has the potential cover wider range of use cases, however, it relies mostly on manual preference annotations from human la- belers or larger LLMs which is costly and has no accepted protocol yet.
What Can We Do to Gain Trust Now and in Future?
In a world where generative LLMs seemingly dom- inate every benchmark and are claimed to have reached human-level performance many tasks, advocate that now is the time to treat trust first-class citizen and place it at the center of model development and evaluation. To operational- ize the concept of trust, we denote with knowledge facets (henceforth, facets) all factors that improve our knowledge of functional capacity and knowl- edge of origin. Facets can be local (instance) or global (datasets, tasks). They refer to 1) descriptive knowledge such as meta-data or data/task prove- nance, and 2) inferred knowledge; for example which skills are exploited. We next propose con- crete suggestions on how facets can help us gain trust in LLMs based on the desiderata in $2.
Explain Skills Required versus Skills Employed. It is instructive to think of prompt-based genera- tive LLMs as instance-level problem solvers and, as such, we need to understand a- a-priori the nec- essary skills for solving instances (local facets) as well as knowing what skills are actually em-- ployed during inference. Most prior work aims to improve our understanding of tasks and the skills acquired to solve them by studying models trained specifically for each task, and can be broadly clas- sified into: (i) linguistically motivated approaches and (ii) model-driven approaches (D1). Linguis- tic approaches formalize skills cognitive abili- ties, which are studied, e.g., through probing tasks (Adi al., 2017: Conneau al., 201 8; Amini and Ciaramita, 2023), checklists (Ribeiro et al., 2020) and linguistic profiling (Miaschi et al., 2020, 2021; sarti et al., 2021). Model-driven approaches at- tribute regions in the model parameter space to skills (Ansell et al., 2022; Wang et al., 2022a; Ponti
D4. Knowledge of Data Origin. So far, we discussed trust desiderata from the viewpoint of knowledge of functional capacity. Next to this, a model's behaviour is also largely influenced by its training data. Knowledge about data provenance helps make informed decisions about whether a given LLM a good match for the intended use case. Therefore, open access data must be pri- oritized. In compartmentalized NLP, models are trained and evaluated on well-known, manually cu- rated, task-specific datasets. Today's models are instead trained on huskhheterogeneouu corpora at web scale, typically of unknown provenance. For novel tasks, this means we do not know how well relevant facets (e.g., language, domain) are repre- sented in the training data. For existing tasks, is unclear if the model has seen test instances in their large training corpora (i.c., test data Icakage; Piktus et al., 2023), blurring the lines between traditional
'For example, GPT-4 reportedly passed the bar exam and placed top at GRF exams, see https://openai.com research/gpt-

---

al., 2023; Ilharco al., 2023). The former can be seen as describing global facets (i.e., the overall functional capacity of black-box models), while the latter identifies local facets (i.e., skill regions in model parameters). To establish trust, we need to know what skills are required to solve instances, which is different from which skills are exercised by a model at inference time, as described next. Besides knowlege about skills needed to solve a task, it is important to gain knowledge about what skills are actually being applied by an LLM. This is linked to explainability and transparency, corre- sponding to (i) understanding the knowledge" that goes into the inference process (D1), and (ii) the inference process itself in terms of applied skills (D2), e.g., examinations of LLMs' "thought pro- cesses". Regarding (i), existing work includes at- tributing training instances model predictions (Pruthi et al., 2020; Weller et al., 2023) and ex- plaining predictions through the lens of white-box models (Frosst and Hinton, 2017; Aytekin, 2022; Hedderich et al., 2022). They are, however, of- ten grounded in downstream task data and thus do not provide insights connected to the knowledge memorized by LLMs during pre-training (global facets). Regarding (ii), existing approaches include guiding the generation process through intermedi- ate steps (Wei et al., 2022c; Wang et al., 2023a; Li et al., 2023) and pausing the generation process to call external tools (Schick et al., 2023; Shen et al., 2023; Paranjape et al.. 2023; Mialon et al., 2023). Their shortcoming is that they operate on the in- put level, and similarly do not capture cases where pre-existing, model-interna knowledge is applied. Furthermore, prior work has shown that LLMs fol- low the path of least resistance. That is, neural networks are prone to predict the right thing for the wrong reasons (McCoy et al., 2019; Schramowski et al., 2020), which can be caused by spurious cor- relations (Eisenstein, 2022). 5 On the path to gain- ing trust, we advocate for LLMs that are able to attribute their output to internal knowledge and the skills used to combine that knowledge. Alterna- tively, LLMS could be accompanied by white-box explanation models that (are at least a proxy) for explaining the inference process.
NLP papers proposing new model is beat previous models on a certain quantitative bench- mark. We argue that if datasets and metrics are well-designed and weellggounded in skills/capabili- ties, they can be used as an indicator of progress. 6 On the other hand, findings from negative results might be obscured without faceted quantitative analysis: even when obtaining lower scores on benchmark, sub-parts of an NLP problem may be better solved compared the baseline, but go unnoticed (D3). We therefore cannot trust reported SOTA results as long as the facets that explain how well cuob-rrbblmms are solved remain hidden. Complementary to holistic quantitative explana- tions, as proposed by HELM (Liang et al., 2022), we call for holistic qualitative evaluation where benchmarks come with standardized qualitative evaluation protocols, which facilitates comparable qualitative analysis. This proposal inspired by the manually-curated GLUE diagnosties annota- tions (Wang et al., 2018), which describe examples by their linguistic phenomena. 7 Recycling existing tasks and augmenting them with diagnostic sam- ples to study LLMS provides a very actionable di- rection for applying existing compartmentalization in a more targeted trustworthy way. Diagnostics samples should ideally represent the full spectrum of cognitive abilities required to solve task. De- signing these samples is however a complex task. We hypothesize that the set of required skills varies between tasks and should ideally be curated by expert annotators.
Be Explicit about Data Provenance. In ML, it is considered good practice use stratified data splits to avoid overestimation of performance on de- v/test splits based on contamination. Traditionally, this stratification was done based on, c.g., source, time, author, language (cross-lingual) or domain (cross-domain). Recent advances have hinted at LLMs' ability to solve new tasks, and even to ob- tain new, i.e., emergent abilities (Wei et al., 2022b). These are in fact similar cross-X' settings, where X' is no longer a property the level of dataset sampling, but of the broader task setup. We call for always employing a cross-X' setup (D4); whether is based on data sampling, tasks, or capabilities- urging practitioners to make this choice explicit. Transparency about data provenance and test data
Facilitate Representative and Comparable Qual- itative Analysis. Today, the standard target for
"Including acquired knowledge such common sense and world knowledge (Li al., 2022;; De Bruyn et al., 2022). 5"The sentiment of movie should he invariant to the identity of the actors in the movic" (Eisenstein, 2022)
6 Note that baseline comparisons can still be obscured by unfair comparisons (Ruffinelli al., 2020). ? "https: http:///llubbeebeeeee com com/ddianossticss

---

leakage improve our trust in reported results. In practice, these data provenance facets are also valu- able for identifying inferred knowledge such as estimated dataset/instance difficulty (Swayamdipta et al., 2020; Rodriguez et al., 2021; Ethayarajh et al., 2022), especially when used in conjunction with the aforementioned diagnostic facets.
Specifically, we need to know what users do with the model output across multiple interactions (c.g.. verify, fact check, revise, accept). For example, González et al. (2021) investigate the connection between explanations (D2) and user trust in the con- text of question answering systems. In their study users are presented with explanations in different modalities and either accept (trust) reject (don't trust) candidate answers. Similarly, Smith-Renner et al. (2020) discuss how generated explanations can promote over-reliance or undermine user trust. A closely related question how the faithfulness of explanations affect user trust (Atanasova et al., 2023; Chiesurin et al., 2023). For a comprehensive overview on user trust we refer to the recent survey by Bach et al. (2022).
Data provenance also important when draw- ing conclusions from benchmark results (D3). Tedeschi et al. (2023) question the notion of su- perhuman performance and claims of tasks being solved (i.e., overclaiming model capabilities), and criticize how benchmark comparisons "do not in- centivize a deeper understanding of the systems' performance". The authors discuss how external factors can cause variation in human-leve perfor- mance (incl. annotation quality) and lead to unfair comparisons. Similarly, underclaiming LLMs ca- pabilities also obfuscates our knowledge of their functional capacity (Bowman, 2022). Additionally, in recent study domain experts find the accuracy of LLMs to be mixed (Peskoff and Stewart, 2023). It is therefore important be explicit about the limitations of benchmarks (Raji al., 2021) and faithful in communicating model capabilities. At the same time, it is an ongoing discussion whether reviewers should require (i.e, disincentivize the ab- sence of) closed-source baseline models such as ChatGPT and GPT-4, which do not meet our trust desiderata (Rogers et al., 2023). Closed-source models that sit behind APIs typically evolve over time and have unknown data provenance, thus lack- ing both knowledge of origin (D4), and the consis- tency of its functional capacity. Consequently, they make untrustworthy baselines and should not be used as an isolated measure of progress.
While such controlled studies using human feed- back are cost and time intensive, the minimum viable alternative for establishing trust may simply be the publication of a model's input-output history. In contrast to standalone metrics and cherry-picked qualitative examples, access to prior predictions en- ables post-hoc knowledge of model behaviour (D2), even without direct access to the model. This de- mocratizes the ability to verify functional capacity and helps end users seeking to understand how well model works for their task.
In summary, evaluating user trust is an integral part of trustworthiness and goes hand in hand with careful qualitative analyses and faceted quantitative evaluation. Towards this goal, we believe LLM development needs to be more human-centric.
Conclusions
In this position paper, we emphasize that the de- mocratization of LLMs calls for the need to rethink tasks and model evaluation, placing trustworthi- ness at its center. We adopt a working definition of trustworthiness and establish desiderata required to improve our knowledge of LLMs ($2), followed by suggestions on how trust can be gained by outlin- ing directions guided by what we call knowledge facets ($3). Finally, we draw connection between trustworthiness as knowledge facets and user trust as means to evaluate their impact on human-Al collaboration ($4).
4 Trustworthiness and User Trust
So far we have discussed different avenues for im- proving our knowledge about LLM's functional capacity and origin, paving the way for establish- ing trustworthiness. From a user perspective it is essential to not only understand knowledge facets but also how they empirically impact user trust in a collaborative environment. This is especially im- portant in high-risk scenarios such as in the medical and legal domain. One could argue, if LLMs such ChatGPT are already widely adopted, do we al- ready trust LLMs (too much)? To better understand user trust we need interdisciplinary research and user experience studies on human-Al collaboration.
Limitations
To limit the scope of this work, we did not discuss the topics of social and demographic biases (Gira et al., 2022), discrimination of minority groups

---

(Lauscher et al., 2022) and hate speech as factors influencing our trust in LLMs. Within our proposed desiderata, this facet would fall under 'Knowlegge of Data Origin' ($2), in terms of understanding where model-internal knowledge and the associated biases originate from (D4).
of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 283-294, Toronto, Canada. Association for Computational Linguistics.
Caglar Aytekin. 2022, Neural networks are decision trees. arXiv preprinl arXiv:2210005/88.
Our proposed multi-faceted evaluation protocols rely strongly on human input-either via quali- tative judgements and/or linguistically annotated diagnostic benchmarks ($3). We acknowledge that such analyses require more time and resources com- pared to evaluation using contemporary, automatic metrics, and may slow down the overall research cycle. While we believe that slower, yet more delib- erate analyses are almost exclusively beneficial to establishing trust, our minimum effort alternative of publishing all model predictions can also be used to build user trust ($4). This simple step closely mir- rors the scientific method, where hypotheses must be falsifiable by anyone (Popper, 1934). Identify- ing even a single incorrect prediction for a similar task in a model's prediction history, can already tell us plenty about the model's trustworthiness.
Tita Alissa Bach, Amna Khan, Harry Hallock, Gabriela Beltrão, and Sonia Sousa. 2022. A systematic liter- ature review of user trust in ai an-eneeled systems: An hci perspective. International Journal of Human- Computer Interaction, pages 1-16.
Kevin Baum, Maximilian A. Köhl, and Eva Schmidt. 2017. Two challenges for CI trustworthiness and how to address them. Proceedings of the 1st Workshop on Explainable Computational Intelligence (XCI 2017), Dundee, United Kingdom. Association for Computational Linguistics.
Samuel Bowman. 2022. The dangers of underclaim- ing: Reasons for caution when reporting how NLP systems fail. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484-7499, Dublin, Ireland, Association for Computational Linguisties.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4, arXiv preprint 1xXi:23033.1212..
Acknowledgements
We thank the anonymous reviewers for their in- sightful comments. This research is supported by the Independent Research Fund Denmark (DFF)
Sabrina Chiesurin, Dimitris Dimakopoulos, Marco An- tonio Sobrevilla Cabezudo, Arash Eshghi, Ioannis Papaioannou, Verena Rieser, and loannis Konstas. 2023. The dangers of trusting stochastic parrots: Faithfulness and trust in open-domain conversational question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 947- 959, Toronto, Canada. Association for Computational Linguistics.
Sapere Aude grant 9063-00077B and ERC Consol- idator Grant DIALECT 101043235.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi,
and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representa- tions.
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. 2023. How do languages influence each other? study- ing cross cross-lingual data sharing during Ilm fine-tuning. arXiv preprint arXiv:2305. /3286.
Afra Amini and Massimiliano Ciaramita. 2023. Prob- ing in context: Toward building robust classitiers via probing large language models. arXiv preprint arXiv:2305. 141711.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finttueel language models. CoRR, abs/2210.1 11416.
Alan Ansell, Edcardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1778-1790 1796. Dublin, Ireland. Association for Computational Lin- guistics.
Pepa Atanasova, Oana-Maria Camburu, Christina Li- oma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness tests for natural language explanations. In Proceedings
Charles LA Clarke, Gianluca Demartini, Laura Dietz, Guglielmo Faggioli, Matthias Hagen. Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast,

---

Ian Soboroff, et al. 2023. 4.2 hme: A spectrum of human- machine collaboravll relevance judgment frameworks. Frontiers of Information Access Experi- mentation for Research and Education, page 41.
Robert M. French. 1999, Catastrophic forgetting in con- nectionist networks. Trends in cognitive sciences. 3(4): 123-135.
Nicholas Frosst and Geoffrey Hinton. 2017. Distilling neural network into a soft decision tree. arXiv preprint arXiv:111100974.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 39-69, Dublin, Ireland. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- plc, Loic Barrault, and Marco Baroni. 2018. What you can cram into single S& 1 vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Pupers), pages 2126-2136, Melbourne, Aus-
Tobias Glasmachers. 2017. Limits of end-to-end learn- ing. In Asian conference on machine learning, pages 17-32. PMLR.
Ana Valeria González, Gagan Bansal, Angela Fan, Yashar Mehdad, Robin Jia, and Srinivasan Iyer. 2021. Do explanations help users detect errors in open- domain QA? an evaluation of spoken vs. visual ex- planations. In Findings of the Association for Com- putational Linguisticx: ACL-IJCNLP 2021, pages 1103-116, 16, Online. Association for Computational Linguistics.
tralia. Association for Computational Linguistics.
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2022. 20Q: Overlap- free world knowledge benchmark for language models. In Pro- ceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). pages 494-508, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
David G. Hays. 1979. Applications. In 17th Annual Meeting of the Association for Computational Lin- guistics, pages 89-89, La Jolla, California, USA. Association for Computational Linguisties.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 202] Conference on Empirical Meth- ods in Natural Language Processing, pages 6491- 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Michael A Hedderich, Jonas Fischer, Dietrich Klakow, and Jilles Vreeken. 2022. Label-descriptive patterns and their application to characterizing classification errors. In International Conference on Machine Learning, pages 8691-8707. PMLR.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms. CoRR, 1at./2355.4314
Gabriel Ihaaroo, Marco Tulio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannameh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learn- ing Representations.
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy, 2022. On the origin of ballucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 5271-5284 Seattle, United States. Association for Computational Linguistics.
Anne Lauscher, Federico Bianchi, Samuel R. Bowman, and Dirk Hovy. 2022. SocioProbe: What, when. and where language models learn about sociodemo- graphics. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7901-7918, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jacob Eisenstein. 2022. Informativeness. and invariance:
Two perspectives on spurious correlations in natural language. In P Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4326-4331, Seattle, United States. Association for Computational Linguisties.
Dongfang Li, Baotian Hu, and Qingeai Chen. 2022a. Calibration meets explanation: A simple and effec- tive approach for model confidence estimates. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2775- 2784, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kawin Ethayarajh. Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008, PMLR
Xiang Lorraine Li, Adhigunaa Kuncoro, Jordan Hoff- mann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. 2022b. A systematic investi- gation of commonsense knowledge in large language models. In Proceedings of the 2022 Conference on

---

Empirical Methods in Natural Language Processing, pages 11838-11855, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 40-47, Online. Association for Computational Linguistics.
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing. Shafiq Joty, and Soujanya Poria. 2023. Chain of knowledge: A framework for ground- ing large language models with structured knowledge bases. arviv preprint ar Xrviv:2305. 13269.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.091111
R OpenAL. 2023. Gpt-4 technical report. arXiv, pages 2303-08774.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non- non- parametric mem- ories. In Proceedings of the ólst Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 9802-9822, Toronto, Canada. Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguis- tics: System Demonstrations, pages 55-60, Balti- more, Maryland. Association for Computational Lin- guistics.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023, Art: Automatic multi- step reasoning and tool-use for large language mod- els. arXiv preprint arXiv:2303.099: 09014.
Denis Peskoff and Brandon Stewart. 2023. Credible without credit: Domain experts assess generative lan- guage models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guisties (Volume Short Papers), pages 427-438. Toronto, Canada. Association for Computational Lin- guistics.
Michael McCloskey and Neal J Cohen, 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. In Psychology of learn- ing and motivation, volume 24, pages 109-165. Else- vier.
Tom McCoy, Ellic Pavlick, and Tal Linzen. 2019, Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428- -344,, Florence, Italy. Association for Computational Linguistics.
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023. The roots search tool: Data transparency for 1lms. arXiv preprint arXiv:2302.1400:
Edcardo Maria Ponti, Alessandro Sordoni, Yoshua Ben- gio, and Siva Reddy. 2023. Combining parameter- efficient modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 687-702, Dubrovnik, Croatia. Associ- ation for Computational Linguistics.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. 2023. Augmented language models: survey, arXiv preprint arXiv:2302.0784.
Alessio Miaschi, Dominique Brunato, Felice Orlttt,, and Giulia Venturi. 2020. Lin- guistic profiling of neural language model. In Pruceedings of the 28th International Conference on Computational Linguisties. pages 745-756, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Karl Popper. 1934, Karl Popper: Logik der Forschung. Mohr Sieheck, Tübingen, Germany.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influ- ence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 13:19990-19930
Alessio Miaschi, Dominique Brunato, Felice Orleta,, and Giulia Venturi. 2021. What makes my model perplexed? a linguistic investi- gation on neural language models perplexity. In
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits

---

of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21()):54855551)
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, Frangois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier. Samson Tan, Pedro Ortiz Suarez, Vie- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji. Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 116b-pat parameter open-access multilingual language model. CoRR, abs/221 1.05 100.
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. Ai and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks, vol- ume Curran.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd- Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leader- boards? In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 Iith International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 4503-4503 Online. Association for Computa- tional Linguistics.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint
David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference Natu- ral Language Processing ( (Volume Short Papers), pages 670-674, Online. Association for Computa- tional Linguistics.
Anna Rogers, Niranjan Balasubramanian, Leon Der- czynski, Jesse Dodge, Alexander Koller, Sasha Luc- cioni, Maarten Sap, Roy Schwartz, Noah A Smith, and Emma Strubell. 2023. Closed ai models make bad baselines.
Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You can teach an old dog new tricks! on training knowledge graph embeddings. In Inter- national Conference on Learning Representations.
Patrick Schramowski. Wolfgang Stammer, Stefano Teso, Anna Brugger. Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kris- tian Kersting. 2020. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476-486.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen. Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abhcesht Sharma, An- drea Santilli. Thibault Févry, Jason Alan Fries. Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li. Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580
Ili Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papemot, and Ross Anderson. 2023. The curse of recursion: Training on gen- erated data makes models forget. arXiv preprint arxiv:2305. 17493.
Alison Smith-Renner, Ron Fan, Melissa Birchfield, Tongshuang Jordan Boyd-Graber, Daniel S Weld, and Leah Findlater. 2020. No explainability without accountability: An empirical study of explanations and feedback in interactive ml. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems, pages 1-13.
Gabriele Sarti, Dominique Brunato, and Felice Orletaa. 2021. That looks hard: Characteriz- ing linguistic complexity in humans and language models. In Proceedings of the Workshop on Cogni- rive Modeling and Computational Linguistics. pages 48 48-60, Online. Association for Computational Lin- guistics.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping

---

and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, Association for Computa- tional Linguistics.
A multi-task task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P Analyzing and Interpreting Neural Networks for NLP. pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpaca: strong, replicable instruction-folllwwe ford Center for Research Foundation Models. hipps//crfm. stanford. edu/2023/03/1 3/alpaca. himl, 3(6).7.
Xiaozhi Wang. Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022a. Finding skill neurons in pre- mre-trained transformer- models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 11 132-11 1552, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
model. Stan-
language
Simone Tedeschi, Johan Bos, Thiery Declerck, Jan Hajie. Daniel Hershcovich, Eduard Hovy, Alexan- der Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What's the meaning of superhuman perfor- mance in today's ' NLU? In Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pages 12471- 12491, Toronto, Canada. Association for Computa- tional Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Fd H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023b. How far can camels go? exploring the state of instruction tuming on open resources. CoRR, abs/2306.0475/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- AntieeeAneeIeee Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume lample. 2023a. Llama: Open and efficient foundation language models. CoRR, 1bs/2302.13971.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson. Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel. Mehrad Moradshahi, Mihir Parmar. Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. SupperNaturalllssrretiio
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023b. Llama: Open and effi- cient foundation language models. arXiv preprint 139.::
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023, Artificial artificial artificial intel- ligence: Crowd workers widely use large language models for text production tasks. arXiv preprint (XXiv23006008899
General- ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5085-5100000 5109,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tvan Vulié, Goran Glavaš, Fangyu Liu, Nigel Collier, Edoardo Maria Ponti, and Anna Korhonen. 2023. Probing cross-lingual lexical knowledge from mul- tilingual sentence encoders. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics. pages 2081- 2097, Dubrovnik, Croatia. Association for Computa- tional Linguisties.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Ivan Vulié, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222 22407 Online. Association for Computa- tional Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto. Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. 2022h. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica-
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2018. GLUE:
tion.

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022c. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. according to..." prompting language mod- els improves quoting from pre-training data, arXiv preprint 2305 13252.


Reviews:
[{"review_id": "F9mxBz3Krz", "reviewer": "Reviewer_H6uV", "summary": "", "questions": "Question A. In the \"Explain Skills Required versus Skills Employed\" part of Section 3, I understand that the first paragraph is about \"Skills required\" and the second paragraph is about \"Skills Employed\". Why do you discuss the approachs attributing regions in the model parameters as identifing \"Skills Required\" rather than \"Skill Employed\"? I thought they are finding which neurons/skills are activated/employed during task solving. \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your concrete suggestions, and the detailed review of our work. We are highly encouraged by your positive feedback regarding the execution of our desiderata analysis.\n\nRegarding Weakness 1: We fully agree that the best use of additional space would be to be more concrete on explaining desiderata, please also refer to our response to Reviewer 8KkL. One of our primary goals is to propose trust desiderata as a thought framework for stimulating and guiding discussions in the community. To this end, we hope to inspire possible research directions for future uptake. We also agree that parts of the Limitations would fit also in the main content. Given the extra space in the final version we will revisit the paper structure. \n\nRegarding Question A: This indeed appears to be an error in the writeup, we will revise the paper in the final version. Thank you for highlighting this. We also appreciate the concrete suggestions for presentation improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper performs excellently as a position paper. The introduction's discourse on why the emergence of Large Language Models (LLMs) has made trustworthy NLP more important is appropriate and convincing. The analysis of the desiderata is also well executed.\n\n(2) Not only does this article provide new insights and proposals, but it also comprehensively surveys and organizes existing work within a limited space. I believe that by following the framework provided by this paper to read the cited works, readers will gain a deeper understanding of the research landscape.", "reasons_to_reject": "(1) The part of proposed actions (Section 3) is not clear and specific enough to me. It would be better to use more space and add some examples or illustrations for them. It will also be helpful to explicitlt split the parts about background and related work between the parts of proposed actions. Also, the proposal in the Limitations section is interesting and I think it should be placed into the content. This position paper might be better to be presented in long paper.", "questions_to_authors": "Question A. In the \"Explain Skills Required versus Skills Employed\" part of Section 3, I understand that the first paragraph is about \"Skills required\" and the second paragraph is about \"Skills Employed\". Why do you discuss the approachs attributing regions in the model parameters as identifing \"Skills Required\" rather than \"Skill Employed\"? I thought they are finding which neurons/skills are activated/employed during task solving. \n", "paper_topic_and_main_contributions": "This paper argues that with the rise of large language models (LLMs), it is becoming increasingly difficult to establish trust in their capabilities since the traditional NLP handle compartmentalized tasks and evaluations but LLMs are applied extremely broadly. It proposes to place trust at the center of model development and evaluation, and rethink task formalization and evaluation around the concept of \"knowledge facets\" - factors that improve understanding of a model's origins and functional capacities. This paper outlines desiderata for trustworthy LLMs, including knowledge of model inputs, behavior, evaluation protocols, and data origin. It also proposes some potential solutions, including to explain required versus employed skills, facilitate qualitative analysis, and be explicit about data provenance.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1) Line 306: duplicate \"linguistic\".\n(2) Line 340: missing preposition between \"knowledge\" and \"LLMs\".\n(3) Some references are duplicate, such as Wei et al., 2022a and 2022b, and Touvron et al., 2023a and 2023b."}}, {"review_id": "0DleENPmzV", "reviewer": "Reviewer_kEVc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding Weakness 1: Thank you for this suggestion. In Figure 1 we show the traditional NLP compartments with clearly defined evaluation protocols, task definitions and datasets, indicated with dashed lines, effectively describing the “before” state. In the current state of NLP those boxes tend to disappear as datasets are combined and tasks and evaluation protocols are conflated. We will update the diagram to better highlight the paradigm shift from NLP compartments towards general purpose, task-agnostic approaches (L008-L012). \n\nRegarding Weakness 2: Given the extra space in the final version, we will improve the overall clarity of each desiderata. In particular, in D1 we refer to knowledge about “ingredients” that goes into the model’s decision making (L104-107, Footnote 2), this includes unwanted ingredients such as social biases and propaganda (L344-348). In D2 we refer to knowledge about the skills that are used to process those ingredients into model outputs, e.g., by being able to interpret decision boundaries (L130), this includes unwanted skills learned from spurious correlations (L275). D3 and D4 refer to our knowledge that help us gauge the true capability of models. \n\nRegarding Weakness 3: Thank you for the feedback. We will improve the overall clarity of our paper and also highlight the implications on future research more explicitly by including a discussion on incentive structures and concrete examples of possible research directions. Please refer to our reply to 8KkL’s question and our response regarding suggestions on attributing model output to internal knowledge, building diagnostics datasets and conducting user trust studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "--This paper synthesizes contemporary work distributed throughout different subfields of NLP and ML into a conceptual framework for trust.\n\n--This paper provides a different and valuable perspective for analyzing LLMs.\n", "reasons_to_reject": "--The authors point out that recent LLMs break down the compartmentalization (dashed lines) in Figure 1, impacting all stages of the cycle. Can a corresponding Figure be provided to offer readers a more visually informative comparison between the two?\n\n--In Section 2, each trust desideratum should be accompanied by its corresponding explanation, making it easier for readers to understand. For example, \"Knowledge about Model Input refers to...\"\n\n--In Section 3, the conclusions drawn from the analysis of existing work, which provide insightful implications for future research, should be presented more explicitly and clearly.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper emphasizes that the democratization of LLMs calls for the need to rethink tasks and model evaluation, with a focus on placing trustworthiness at the center of these considerations.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pmR8mcEAxa", "reviewer": "Reviewer_8KkL", "summary": "", "questions": "At the risk of being somewhat philosophical, but my understanding is that the paper rests on the premise that NLP systems should be explainable, but the word \"trustworthy NLP systems\" doesn't feel like the right goal, given that lay humans trust AI a lot already (too much?), despite having no idea how it works. What are the authors' thoughts on that distinction?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to write such a thorough review and for sharing your thoughts. We interpret this as a positive sign given that one of our main goals of this position paper is to stimulate discussions in the research community. Please find below our replies to your specific concerns and questions, we will use the extra space in the final version of the paper to incorporate your valuable feedback. \n\nRegarding your questions on L277, we acknowledge attributing model output to internal knowledge and skills used is a challenging goal, and only one step towards a multi-faceted evaluation, yet an important one. We believe attribution requires a deeper understanding of model behavior, including (but not limited to) localizing skills and knowledge in the model’s parameter space. Prior work towards this goal has been done in the context of localizing internal factual knowledge [1-4] and localizing task-specific parameter regions (L235-237). Extending this to other types of knowledge (e.g., common sense knowledge, social biases) presents opportunities for future work, as well as opportunities to uncover what can and what cannot be done with attribution methods, therefore we deem it an important facet. Furthermore, viewing tasks as mixtures of skills allows future work in another general direction, namely extending methods for task representations towards the skill and instance-level. Thank you for your comment, we will make sure to be more specific on the general directions. \n\nRegarding your question on L306, thank you for your positive feedback and very good question. Generally, diagnostics samples should ideally represent the full spectrum of required cognitive abilities required to solve a task. This is however a complex task. We hypothesize that the set of required skills varies between tasks. Representative test examples and probes should be manually curated by expert annotators (L353-356) as opposed to cherry-picked test examples (L365), but the crafting remains an open problem. Here, we will add Raji et al. [7] to point out the limitations of benchmarking to measure general language understanding capabilities. As there exists no framework, we call for more work in this direction. Recent work on collecting explanations on ambiguous and uncertain cases (e.g. [6]) can be taken as an actionable starting point. \n\nRegarding incentive structures: Thank you for raising this important point. We agree that the manuscript benefits from explicitly touching upon current incentive structures. We will add a discussion (e.g. expand D.4 with the current lack of incentives on data transparency, and extend the discussion in L356 with examples like the transparency issue). We believe that adding this helps the discussion on how incentives structures can be changed and we thank the reviewer for this constructive feedback. For instance, we believe that rethinking tasks and model evaluation with trustworthiness in mind can be a valuable conceptual framework to revisit incentive structures. For example, an ongoing discussion is whether reviewers should require (i.e, disincentivize the absence of) closed-source baseline models such as ChatGPT and GPT-4, which don’t meet our trust desiderata [5]. In addition, there is a lot to gain from rethinking benchmarks, slowing down research (L356-362) and we call for incentivizing more elaborate evaluation protocols and systematic qualitative evaluation (L299-303). \n\nRegarding the Question: Thank you for this insightful question! We would like to clarify that explainability alone is not sufficient, our perspective on trust and our trust desiderata go beyond current explainability methods and beyond the users' perspective. For example, we argue that we cannot trust reported results in the absence of data provenance information (L290-297, L326-327). Overall, we believe however that it is important to  make the crucial distinction between the widespread adoption of LLMs and the user’s trust in those models. Whether end users trust (too much?) LLMs, and whether and how more trustworthy systems impact user trust, is an intriguing question. To get a better understanding of this phenomenon we need interdisciplinary collaboration and user experience studies (human-AI collaboration). Specifically, to answer the question about user trust we would need to know what users do with the model output across multiple interactions (longitudinal, e.g., verify, fact check, revise, accept) and do, e.g., user interviews. An interesting research question given our desiderata could be: How does user trust compare to systems that implement our desiderata vs. systems that do not (control group). We will include this discussion on the distinction between trustworthiness and user trust in the final version. \n\nThank you for the suggested paper reference, we agree that CoT literature is indeed relevant to our work (L262-265). We will add it to the paper, together with the references discussed here. \n\n[1] “Locating and Editing Factual Associations in GPT” Meng et al. (NeurIPS’22)\\\n[2] “Mass-Editing Memory in a Transformer” Meng et al. (ICLR’23)\\\n[3] “Editing Factual Knowledge in Language Models” De Cao et al. (EMNLP’22)\\\n[4] “When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories” Mallen et al. (ACL’23)\\\n[5] “Closed AI Models Make Bad Baselines” Anna Rogers (Blog Post, 2023)\\\n[6] \"Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation\" Jian & de Marneffe (2023, arXiv)\\\n[7] \"AI and the Everything in the Whole Wide World Benchmark\" Raji et al., (NeurIPS 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a solid outline of how the shift to instruction-tuned NLP systems requires new evaluation paradigms.\n- The use of Hays' definition and the four desiderate described make sense to me.\n- The listed directions and recommendations make sense to me.", "reasons_to_reject": "- The main reason to reject is the following: while I agree and appreciate most of the arguments made in the paper and future directions towards more trustworthy NLP, the paper's impact and usefulness for the community could be improved by making the recommendations more concrete and fleshed out. \n  - For example, on L277, this feels like an ambitious and complex thing to ensure that LLMs be able to attribute their outputs to internal knowledge... How do authors believe this could be done? Or at least, what are some general directions towards that?\n  - Another example, on L306, authors suggest the use of diagnostic samples to study LLMs behavior, which I love; but how should NLP practitioners actually craft these? Is there a framework for crafting these? What is some prior work that has done something similar, that practitioners could rely upon?\n- A second concern is the lack of discussion around incentive structures of current landscape of NLP research, including industry profit and risks of harms. Specifically, there are several reasons why the current big players of LLMs have directly acknowledged that transparency is antithetical to their profit incentives (e.g., the OpenAI GPT-4 report describes the training data details as OpenAI's own \"intellectual property,\" as a reason for not sharing those details). Additionally, some of these desiderata have their own potential risks (for example, releasing an entire training dataset for an LLM could include releasing copyrighted materials, or issues of data sovereignty where data is in languages that should be stewarded only by folks who speak those languages). I wish authors could at least acknowledge these incentives and potentially even mention ways to change these incentives.\n\n**Edit: post-rebuttal update:** Changed my scores from 3/3 to 4/4", "questions_to_authors": "At the risk of being somewhat philosophical, but my understanding is that the paper rests on the premise that NLP systems should be explainable, but the word \"trustworthy NLP systems\" doesn't feel like the right goal, given that lay humans trust AI a lot already (too much?), despite having no idea how it works. What are the authors' thoughts on that distinction?", "paper_topic_and_main_contributions": "This position paper outlines current issues of applying standard NLP task and evaluation paradigms to the new generation of large LMs (GPT-4 etc.). Using David Hays' definition of trustworthiness, they outline 4 desiderata for trustworthy models: knowledge of inputs, knowledge of behavior, knowledge of evaluation protocols, and knowledge of data origins. Based on these, authors discuss some avenues for improving trustworthiness of NLP systems.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In section 3, it feels like the recent work on chain-of-thought and improving the logical consistency of CoT is relevant, for example:\n- Jung, Jaehun, et al. \"Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022."}}]

Author rebuttal:

Meta-review: This position paper investigated an important question, i.e. how should we evaluate LLMs. Using David Hays' definition of trustworthiness, the authors outline 4 desiderata for trustworthy models. The authors discuss some avenues for improving the trustworthiness of NLP systems. It is basically an excellent position paper and well motivated by the introduction section. The analysis of the desiderata is also well executed. It also comprehensively surveys and organizes existing work within a limited space.
Decision: Accept-Main