Venue: NAACL 2024

# Elastic Weight Removal for Faithful and Abstractive

Dialogue Generation

Nico Daheim\({}^{1}\)   Nouha Dziri\({}^{2}\)  Mrinmaya Sachan\({}^{3}\)

**Iryna Gurevych\({}^{1}\)   Edoardo M. Ponti\({}^{4}\)**

###### Abstract

Generating factual responses is a crucial requirement for dialogue systems. To promote more factual responses, a common strategy is to ground their responses in relevant documents that inform response generation. However, common dialogue models still often hallucinate information that was not contained in these documents and is therefore unfaithful. In this work, we propose to alleviate such hallucinations by 'subtracting' the parameters of a model trained to hallucinate from a dialogue response generation model in order to 'negate' the contribution of such hallucinated examples from it. Extensive automatic and human evaluation shows favourable results when compared to state-of-the-art methods that combine the distributions of multiple models, such as DE Experts Liu et al. (2021), and others that change the training procedure, such as Quark Lu et al. (2022a). Finally, we show how we can not only reduce hallucinations but also discourage extractive responses, which are often a consequence of reducing hallucinations by encouraging copy-pasting of document spans. We publicly release our code for reproducibility and facilitating further research.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7096–7112 June 16-21, 2024 ©2024 Association for Computational Linguistics

[https://github.com/UKPLab/naacl2024-ewr](https://github.com/UKPLab/naacl2024-ewr)

## 1 Introduction

Current-day large language models (LLMs) impressively generate coherent, grammatical, and seemingly meaningful text, but are prone to hallucinating incorrect information. While grounding them in relevant documents can alleviate this Shuster et al. (2021), models still tend to generate information that conflicts these documents, which would again be classified as hallucination Dziri et al. (2022a). This raises major safety concerns. Such hallucinations could impair student learning, or proliferate convincing-but-inaccurate news articles. Therefore, ensuring trustworthiness is crucial for the safe deployment of LLMs at scale, particularly in high-stakes domains.

Modelling solutions to mitigate hallucination often take inspiration from methods used to discourage other undesirable behaviours in LLMs, for example, contradictions Keskar et al. (2019), repetitions Lu et al. (2022a), or toxicity Illarco et al. (2023). One group of methods achieves this by fine-tuning an LLM conditioned on special tokens Niu and Bansal (2018); Keskar et al. (2019), which can be assigned to model generations by a learned reward model during training Lu et al. (2022a). Another re-weights the predictive distribution with models that are specialised for positive or negative behaviour Liu et al. (2021); Dahelim et al. (2022), called 'experts' or 'anti-experts' respectively. While successful, these methods are either inefficient to train, as a large number of generations needs to be sampled during training, or inefficient in inference, as multiple models have to be stored and evaluated. In this work, we explore a different family of methods Choubey et al. (2023); Ilharco et al. (2023) that uses modular deep learning Ponti et al. (2021); Pfeiffer et al. (2023)

Figure 1: Constructed example of responses \(\mathbf{u}_{T+1}\) that are i) hallucinated (words contradicting the knowledge \(\mathcal{K}\) in red); ii) faithful but not abstractive (longest copied \(n\)-gram in blue); and iii) both \(\mathbf{F}\)aithful and \(\mathbf{A}\)bstractive based on Wizard-of-Wikipedia Dinan et al. (2019).



by interpolating parameters without altering the model architecture. This is efficient during inference, because only one interpolated model needs to evaluated, and for training the models that are interpolated no new data needs to be sampled during the training procedure. Concretely, a new model is obtained as the weighted difference between a pretrained LLM and a model finetuned from it, for example, as an anti-expert (Ilharco et al., 2023). One drawback of this strategy is that parameters are weighted uniformly even though they might have differing contributions to hallucinations. Furthermore, it might result in catastrophic interference between the specialised models (McCloskey and Cohen, 1989). To address this, we propose Elastic Weight Removal (EWR), a novel method for parameter interpolation that weights the importance of each parameter by using the Fisher Information Matrix (FIM) as a measure of importance, similar to previous works in continual learning (Kirkpatrick et al., 2017), sample-efficient learning (Ponti et al., 2019), or merging models for different tasks (Matena and Raffel, 2022). In our experiments, we show how this can be used to discourage hallucinations by first training an anti-expert on synthetically created data and then interpolating it with the baseline model.

We compare our method with state-of-the-art methods for removing hallucinations and other undesired behaviours, which we adapt to removing hallucinations. Namely, we adapt Quark (Lu et al., 2022), DExperts (Liu et al., 2021), and task arithmetic (Choubey et al., 2023; Ilharco et al., 2023). Our findings show consistent improvements in faithfulness, which can be combined with those of others, such as CTRL (Rashkin et al., 2021). Oftentimes, an increase in faithfulness comes at an increase in extractiveness from copy-pasting document spans into the response. Based on this insight, we finally highlight how EWR can be extended to reducing hallucinations and extractiveness at the same time. Our results are confirmed using a human evaluation with the Attributable to Identified Source (AIS) framework (Rashkin et al., 2023). We will release the code for all methods and metrics in a comprehensive framework.

## 2 Background

The goal of dialogue response generation is to continue a dialogue \({\bf u}_{1}^{T}\coloneqq({\bf u}_{1},\ldots,{\bf u}_{T})\) of \(T\) turns by generating a new turn \({\bf u}_{T+1}\). Here, each turn \({\bf u}_{t}\) is just a sequence of \(N_{t}\) tokens \([{\bf u}_{t}]_{1}^{N_{t}}\in\mathcal{V}^{N_{t}}\) from the model vocabulary \(\mathcal{V}\). In document-grounded response generation, \({\bf u}_{T+1}\) is grounded in one or more documents \(\hat{\mathcal{K}}\subseteq\mathcal{K}\) from a document knowledge base \(\mathcal{K}\), meaning that \(\hat{\mathcal{K}}\) informs the information content of \({\bf u}_{T+1}\). Therefore, \({\bf u}_{T+1}\) should also faithfully reflect it. This means that neither contradicting nor unverifiable information should be added. In this work, we assume that \(\hat{\mathcal{K}}\) is given.

A common strategy for generating \({\bf u}_{T+1}\) is using language generators that model the distribution

\[p_{\mathbf{\theta}}({\bf u}_{T+1}~{}|~{}{\bf u}_{1}^{T},\hat{ \mathcal{K}})=\\ \prod_{n=1}^{N_{T+1}}p_{\mathbf{\theta}}([{\bf u}_{T+1}]_{n}~{}|~{}[ {\bf u}_{T+1}]_{1}^{n-1},{\bf u}_{1}^{T},\hat{\mathcal{K}}), \tag{1}\]

parameterised by weights \(\mathbf{\theta}\), for next-token prediction paired with a search algorithm like beam search. We focus on different methods of obtaining \(\mathbf{\theta}\) while maintaining the same model architecture.

### Parameter Combination for Faithful Generation

Previous works have explored combining model parameters with different goals, for example, to increase robustness (Gao et al., 2022) but also to promote or discourage different behaviours by merging specifically trained model instances (Ilharco et al., 2023).

In this work, we use it to discourage hallucinations in dialogue models. By letting \(\Theta=\{\mathbf{\theta}_{1},\dots,\mathbf{\theta}_{N}\}\), where \(\mathbf{\theta}_{i}\in\mathbb{R}^{d}\), denote the parameters of a set of models that should be merged and share a common architecture, and \(\mathbf{\lambda}_{i}\in\mathbb{R}^{d}\) their respective scaling factors, many such methods can be expressed by:

\[\mathbf{\theta}^{\prime}=\sum_{i=1}^{N}\frac{\mathbf{\lambda}_{i}\odot\mathbf{\theta}_{i}}{ Z}, \tag{2}\]

where \(\odot\) denotes element-wise multiplication and \(Z\) can be used to re-scale parameters.

One such method is task arithmetic (Ilharco et al., 2023), which bases on the idea that essential information about a task can be captured by the change of the parameter values between pretrained initialisation \(\mathbf{\theta}_{0}\) and the finetuned \(\mathbf{\theta}_{\text{ft}}\), called task vector. Given this information, the behaviour needed for this task can be added to the model \(\mathbf{\theta}_{0}\) by adding a task vector and also removed by subtracting it. Concretely, the task vector can be expressed as:

\[\mathbf{\tau}\coloneqq\mathbf{\theta}_{\text{ft}}-\mathbf{\theta}_{0}.\] (3) 

Then, task arithmetic (Ilharco et al., 2023) uses the following for model combination:

\[{\mathbf{\theta}^{\prime}=\mathbf{\theta}_{0}+\sum_{i}\lambda_{i}\tau_{i}}, \tag{4}\]

where the scalar \(\lambda_{i}\) promotes the behaviour captured by \(\mathbf{\tau}_{i}\) if \(\lambda_{i}>0\) and discourages it if \(\lambda_{i}<0\).

We will use the latter to discourage hallucinations by training a model to hallucinate and then discouraging its behaviour through subtraction. Intuitively, this can be understood as a form of data removal (Daheim et al., 2024), where unwanted training data is (approximately) removed from a model by subtracting a model trained on this data. We will refer to such a model as 'anti-expert' (\(\mathbf{\theta}_{\text{AE}}\)) and then use the following task arithmetic:

\[{\mathbf{\theta}^{\prime}=\mathbf{\theta}_{0}-\lambda\cdot\mathbf{\tau}}\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}

 

large \({\bf f}_{\theta_{0}}\) are preserved and parameters with large \({\bf f}_{r_{1}}\) are changed more due to their contribution to negative behaviour. When an expert model is added, as well, it is only possible to obtain a similar rewrite when the sign of the corresponding \(\alpha_{i}\) is flipped in the denominator, i.e. \(Z:=\lambda_{0}\cdot{\bf f}_{\theta_{0}}+\sum_{i}(-\lambda_{i})\cdot{\bf f}_{ r_{i}}\). We have found this to be more stable empirically. However, it can introduce divisions by \(0\) which can be avoided by adding a small constant. Finally, we have found calculating the Fisher at \(\mathbf{\tau}\) to perform well empirically, even though calculating it at \(\mathbf{\theta}_{\text{AE}}\) or \(\mathbf{\theta}_{\text{E}}\), respectively, is theoretically better grounded. Next, we describe how we train the expert and anti-expert models. Pseudocode for EWR is shown in Appendix A.1.

### Training Data for (Anti-)Experts

The subtraction of a model trained to hallucinate in EWR can be understood as removing training examples which contain hallucinations. Similarly, adding a model trained on abstractive examples can be seen as data addition. Therefore, the data used to train the (anti-)expert models also determines the behaviour of EWR. In particular, the data for the anti-expert influences which kinds of hallucinations can be reduced: precisely those that are captured in the anti-expert training data.

We use different strategies to create hallucinated examples \(\mathcal{D}^{\mathrm{AE}}\). For Wizard-of-Wikipedia (WoW), we use all examples from Faithdial Dziri et al. (2022a) which humans rated as hallucinations according to the BEGIN taxonomy Dziri et al. (2022c). That is, we use the hallucinated examples from the training data which we want to remove. Since such annotations often do not exist for other data, we try lightweight data augmentation techniques to artificially create hallucinated data which should capture different forms of hallucinations. We find that replacing the ground-truth documents by randomly sampled ones performs similarly to using human hallucination annotations. Potentially, this forces the model to hallucinate, as the input does not contain the correct information for the response. We use this strategy for all other datasets than WoW. CaPE and DEXerts (which we introduce in detail in the following Section 4.2) also use a faithfulness expert in addition to a hallucination anti-expert. For training this expert, we use responses that are assigned an entailment token when training CTRL, because such examples are unlikely to contain hallucinations.

To create a dataset of abstractive examples \(\mathcal{D}^{\text{E}}\), we use the density and coverage metrics introduced in Grusky et al. (2018). Coverage measures the ratio of unigrams from the grounding documents that appear in the response and density measures the average length of copied text spans. Intuitively, we would like to have low density, because this indicates paraphrasing, but such examples might be hallucinated. Therefore, we pick examples that also have high coverage to ensure that the information from the document is used. We do this by splitting the dataset into buckets and assigning low, medium, and high density or coverage tokens to them, similar to Keskar et al. (2019), and taking the high density examples. Future work can explore further methods for data augmentation.

## 4 Experiments

We experiment on multiple datasets outlined in Section 4.1. We compare EWR to CaPE and task arithmetic, as well as a set of other unlearning methods, which we apply for faithful dialogue generation for the first time. Furthermore, we compare to state-of-the-art methods for faithful dialogue generation. We list these baselines in Section 4.2. Crucially, parameter combination can be added independently on top of many of the other baselines.

All experiments are implemented using Huggingface transformers Wolf et al. (2020) and models are initialised from publicly available FlanT5 checkpoints Longpre et al. (2023), which we have found to perform substantially better than previously introduced encoder-decoder models like BART Lewis et al. (2020) or T5 Raffel et al. (2020). We organise our experiments using Sisyphus Peter et al. (2018) and release configuration files to reproduce our results. Further experimental details, such as learning rate or number of epochs, are given in Appendix B.1. We use beam search with a beam size of 10 for decoding.

### Datasets

We evaluate all methods on Wizard-of-Wikipedia Dinan et al. (2019), WoW), an open-domain dataset for information-seeking dialogue where turns are grounded in Wikipedia snippets. WoW contains a _seen_ and an _unseen_ split. Furthermore, we use the DSTC9 Kim et al. (2020) extension of Multi-Woz 2.1 Eric et al. (2020), which augments the original dialogues by turns that are grounded in short FAQ documents. For further experiments, we 

use DSTC11 (Zhao et al., 2023; Kim et al., 2023), which extends DST59 to multi-document settings, and FaithDial (Dziri et al., 2022a), which is a dehallucinated subset of WoW. Statistics are shown in Appendix B.2.

### Baselines

CTRLRashkin et al. (2021)Keskar et al. (2019) introduce CTRL, which uses a sequence of control tokens \({\bf c}\) to steer the model towards desirable generations:

\[p_{\mathbf{\theta}}({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1},\hat{\mathcal{ K}},{\bf c}). \tag{10}\]

Rashkin et al. (2021) adapt the model in Equation (10) to promote faithfulness in document-grounded dialogue by introducing _entailment_, _lexical overlap_ and _first-person_ tokens. We employ the first two. Entailment indicates whether the response is entailed by the documents, determined by an MNLI model, and lexical overlap splits the responses into three buckets according to low, medium, and high lexical overlap. CTRL is trained on examples from all three buckets and both entailment labels but only conditioned on desired ones at inference time (high-overlap and entailment).

QuarkLu et al. (2022a)uses a similar strategy as CTRL for unlearning. The difference is that not only the original training data but also model generations which are taken after each epoch are augmented with special tokens and used for training. Noting this similarity to CTRL, we therefore employ the same tokens to adapt it to faithful dialog generation, allowing for a direct comparison.

DExperts Liu et al. (2021) makes use of an expert and anti-expert model in order to reduce toxicity. The expert model is trained to generate non-toxic text and the anti-expert to generate toxic text. However, instead of combining models in parameter space, as in our method, they are combined at inference time as a density ratio:

\[p({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1},\hat{\mathcal{K}})\propto \tag{11}\] \[p_{\mathbf{\theta}_{0}}({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1},\hat{ \mathcal{K}})\cdot\frac{p_{\mathbf{\theta}_{\text{E}}}({\bf u}_{T+1}~{}|~{}{\bf u} ^{T}_{1},\hat{\mathcal{K}})}{p_{\mathbf{\theta}_{\text{AE}}}({\bf u}_{T+1}~{}|~{}{ \bf u}^{T}_{1},\hat{\mathcal{K}})}.\]

Tokens with high expert probability are encouraged and tokens with high anti-expert probability are discouraged. We use the same expert and anti-expert models as in CaPE to adapt it to faithful dialog generation and fairly compare both methods.

Noisy Channel Model Dahheim et al. (2022) introduce a noisy channel model for documented dialogue:

\[p({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1},\hat{\mathcal{K}})\propto \tag{12}\] \[p_{\mathbf{\theta}_{1}}(\hat{\mathcal{K}}~{}|~{}{\bf u}^{T}_{1},{\bf u }_{T+1})\cdot p_{\mathbf{\theta}_{2}}({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1}).\]

Here, \(p_{\mathbf{\theta}_{1}}(\hat{\mathcal{K}}~{}|~{}{\bf u}^{T}_{1},{\bf u}_{T+1})\) can be seen as a faithfulness and \(p_{\mathbf{\theta}_{2}}({\bf u}_{T+1}~{}|~{}{\bf u}^{T}_{1})\) as a fluency expert. We use their reranking method to rescore generations obtained from our baseline model.

Dial-BeInfo Razumovskaaia et al. (2023)uses two strategies for behavioural finetuning: 1) adding

\begin{table}
\begin{tabular}{l|c c c c c c|c c c c c} \hline  & \multicolumn{4}{c}{WeWoWe} & \multicolumn{4}{c}{DISTO} \\  & BLEU(\(\downarrow\)) & Critic(\(\downarrow\)) & \(Q^{2}(\uparrow)\) & BERT(\(\uparrow\)) & Ft(\(\uparrow\)) & Dens.(\(\downarrow\)) & BLEU(\(\uparrow\)) & Critic(\(\downarrow\)) & \(Q^{2}(\uparrow)\) & BER(\(\uparrow\)) & Ft(\(\uparrow\)) & Dens.(\(\downarrow\)) \\ Model & \(({\bf y},\hat{\mathbf{g}})\) & & & \(({\bf y},\hat{\mathbf{X}})\) & & \(({\bf y},\hat{\mathbf{Y}})\) & & & \(({\bf y},\hat{\mathbf{Y}})\) & & & \(({\bf y},\hat{\mathbf{X}})\) & \\ \hline Flan-TS & 18.5 & 24.3 & 76.2 & 84.4 & 78.6 & 12.4 & 18.5 & 6.2 & 62.3 & 61.3 & 45.2 & 1.73 \\ + TA & 19.1 & 19.4 & 75.9 & 82.2 & 74.4 & 11.1 & 18.5 & **2.5** & 79.6 & 63.6 & 53.9 & 2.80 \\ + EWR & 18.1 (\%) & 18.1 (\%) & 27.8 (\%) & 86.2 (\%) & 86.0 (\%) & 11.5 (\%) & 20.0 (\%) & 4.3 (\%) & 78.4 (\%) & 55.6 (\%) & 32.2 (\%) \\ CapFE & 18.8 & 13.2 & 78.2 & 83.7 & 75.9 & 11.2 & 17.3 & 2.3 & 72.5 & 63.3 & 52.6 & 2.63 \\ + EWbr & 19.0 (\%) & 9.4 (\%) & 78.7 (\%) & 88.2 (\%) & 83.0 (\%) & 13.6 (\%) & 16.7 (\%) & 26.6 (\%) & 79.2 (\%) & 64.3 (\%) & 54.0 (\%) & 27.6 (\%) \\ CTRL & **19.5** & 10.3 & 83.9 & 87.8 & 82.3 & 13.9 & 17.6 & 5.3 & 79.8 & 64.5 & 57.8 & 3.30 \\ + TA & 19.3 & 8.9 & 82.7 & 87.0 & 81.2 & 13.0 & 18.0 & **1.2** & **89.5** & **66.5** & **63.6** & 4.53 \\ + EWR & 18.4 (\%) & **57.9** (\%) & 86.8 (\%) & 91.3 (\%) & 87.7 (\%) & 16.3 (\%) & 19.4 (\%) & 23.3 (\%) & 85.3 (\%) & 65.5 (\%) & 60.6 (\%) & 3.80 (\%) \\ \hline Digreps & 18.0 & 14.8 & 79.6 & 87.0 & 82.2 & 14.3 & 17.1 & 2.9 & 74.9 & 63.6 & 55.7 & 2.83 \\ Quark & 17.2 & 7.9 & **91.9** & **92.6** & **90.2** & 18.6 & 19.0 & 5.7 & 73.1 & 62.7 & 49.8 & 2.03 \\ Noisy Channel & 18.4 & 24.0 & 78.6 & 85.0 & 79.8 & 13.1 & 18.6 & 5.1 & 67.1 & 62.7 & 48.4 & 2.18 \\ Dial-Belfno & 18.2 & 23.0 & 78.2 & 85.6 & 80.7 & 13.1 & 19.5 & 5.7 & 74.9 & 63.0 & 51.7 & 2.86 \\ Loss Truncation & 18.4 & 24.9 & 77.3 & 84.0 & 77.9 & 12.0 & 18.4 & 4.9 & 63.0 & 61.2 & 45.6 & 1.68 \\ \end{tabular}
\end{table}
Table 1: Main results on WoW\({}_{\text{seen}}\) and DSTC9 indicating: i) performance in dialogue generation comparing true \(\hat{\mathbf{y}}\) and predicted \({\bf y}\) responses (BLEU); ii) faithfulness of predicted response \(y\) to ground-truth knowledge \(\hat{\mathcal{K}}\) (Critic, \(Q^{2}\), BERT, F1); 3) abstractiveness (Dens.). We report several baselines adapted for faithful generation and show how Task Arithmetic (TA) and Elastic Weight Removal (EWR, ours) can be deployed on top of vanilla pre-trained models, like Flan-T5, or on top of other methods like CTRL. Relative improvements and degradations are indicated in green and red, respectively.

 

distractor documents to the ground-truth knowledge to teach the model to discern relevant information in the input and 2) creating unanswerable responses without any relevant knowledge to teach the model to not attempt to answer them. We use the first strategy with \(n=2\) distractor documents.

Loss truncation Kang and Hashimoto (2020) proposes to remove high-loss examples to improve language generation. In our setting, it is reasonable to assume that such examples are more likely hallucinated than low-loss examples. Therefore, we adopt this strategy, similar to prior works on document-grounded dialogue Dziri et al. (2022a).

### Metrics

We measure the lexical similarity of the generated and the ground-truth responses with the sacrebleu Post (2018) implementation of BLEU Papineni et al. (2002). To evaluate faithfulness, we employ the hallucination critic introduced by Dziri et al. (2022a)1, which classifies responses as hallucinated or not, \(Q^{2}\) Honovich et al. (2021), which uses a question generation and question answering pipeline, as well as token-level F1 and BERTScore Zhang* et al. (2020)2. To measure abstractiveness, we again use Density Grusky et al. (2018). Further details are found in Appendix B.3.

Footnote 1: [https://huggingface.co/McGill-NLP/roberta-large-faithcriteria](https://huggingface.co/McGill-NLP/roberta-large-faithcriteria).

Footnote 2: We use the _deberta-large-mmli_ checkpoint.

## 5 Results

We first introduce our main results on WoW and DSTC9 in Section 5.1. Then, we characterise trade-offs between faithfulness and abstractiveness in Section 5.2 before discussing the controllability of model interpolation in Section 5.3. Finally, we discuss ablations on various datasets in Section 5.4, provide a qualitative analysis in Section 5.5, and report human evaluation results in Section 6.

### Main Results on Faithfulness

We start with results for de-hallucinated models using Flan-T5base in Table 1. Results with Flan-T5large are found in the Appendix C.1 and show a similar trend: subtracting anti-experts from various base models can improve faithfulness at minor degradation in other metrics. Increases in faithfulness from EWR are often stronger than from task arithmetic, except for Flan-T5base on DSTC9, especially in terms of BERT and token-level F1, but can also lead to decreased BLEU. EWR on top of CTRL provides state-of-the-art performance in faithfulness, comparable to strong baselines like Quark. While the additional faithfulness expert used in CaPE generally improves over using only an anti-expert, we observe fast degradation in terms of BLEU and BERTScore on DSTC9, potentially stemming from comparatively small amounts of expert training data after partitioning the dataset.

CTRL and Quark confirm the effectiveness of control tokens and iteratively applying them to model generations during training. DE experts and noisy channel reranking are mostly outperformed by EWR, task arithmetic, and CaPE, except for Flan-T5base on WoW. This is notable, as they require keeping multiple models but all others use just one at inference time. Nevertheless, the performance of noisy channel model reranking increases with beam size Daheim et al. (2022) which we keep identical for all methods. Behavioural fine

Figure 2: Metrics for EWR with Flan-T5base on WoWseen. (a) Faithfulness and abstractiveness can be traded-off by varying both the influence of the abstractiness expert (a) and hallucination anti-expert (b).

 

tuning improves performance on both datasets and could be combined with EWR. Loss truncation, on the other hand, mainly improves results on DSTC9, but not as much on WoW, possibly because examples containing opinions might not be truncated.

Improvements of CTRL and Quark are much more conspicuous in WoW than DSTC9. We attribute this to the fact that in DSTC9, the ground-truth documents are FAQs, in which the question might not be as important for the control tokens. Furthermore, gold responses contain follow-up questions at every turn, which might decrease the effectiveness of the special tokens and might affect automatic metrics.

Nevertheless, our results in Table 1 also illustrate that increased faithfulness comes at the cost of increased extractiveness, as measured by Density. We investigate this further in the next subsection.

### Faithfulness-Abstractiveness Trade-Off

As our main experiments show that improvements in faithfulness also increase extractiveness, we now outline experiments using an additional abstractiveness expert to reduce this effect. Figure 2 a highlights our results on WoW using Flan-T5base, when only varying the scaling factor of the abstraction expert. From the plot, it emerges that we can control the trade-off between faithfulness and abstractiveness to improve over the baseline in both dimensions, in the interval indicated by the greyed area. To further quantify this trade-off, which has also been described in related works Dziri et al. (2022); Daheim et al. (2022); Aksitov et al. (2023), we use the ratio of the length of the longest common subsequence between \({\bf u}_{T+1}\) and \(\hat{\mathcal{K}}\) and the length of \({\bf u}_{T+1}\) (LCS). We plot the dependency of LCS and Critic in Figure 3 for Flan-T5base-based models on WoW. There is a clear trend towards more extractiveness with increased faithfulness but a better Critic score does not always imply an increase in LCS.

### Scaling Factors & Controllability

Next, we assess how much control EWR provides over faithfulness scores within an acceptable range of BLEU, which measures overall performance. Figure 2 b highlights that there is a larger region of factors along which faithfulness constantly improves within a narrow range of BLEU scores. However, corresponding to the previously discussed trade-off, density increases with faithfulness, indicating that the scaling factor also controls how much of the knowledge is copied into the response.

### Generalisation to Additional Datasets

In this section, we study the performance of EWR in challenging settings, namely on: i) unseen topics that require generalisation (WoW unseen), ii) multi-document corpora (DSTC11), and iii) cleaned training and test data that does not contain hallucinations in ground-truth annotations (FaithDial). We report the results in Table 2.

In summary, we observe the following: 1) EWR shows improvements in all settings, especially in terms of generalisation and in a multi-document setting. Furthermore, we can even improve faithfulness metrics when training and evaluating on the cleaned FaithDial dataset. 2) task arithmetic can improve results on multi-document corpora and some metrics on the unseen set but fails to improve

Figure 3: Improvements in faithfulness (Critic) tend to incur an increase in extractiveness (LCS) on WoW.

 

BERT F1 and F1 on WoW unseen and FaithDial.

### Qualitative Analysis

We provide here a qualitative analysis that highlights which forms of hallucinations are reduced on WoW and DSTC9 when using EWR instead of Flan-T5base. Examples are shown in Table 6. On WoW, hallucinated examples are often made up of opinions and disclosures, as shown in Dziri et al. (2022), where, for example, around 60% of the partially-hallucinated ground-truth responses of a WoW audit were annotated as "Disclosure" according to the VRM taxonomy Stiles (1992). We find that using EWR reduces the amount of such personal opinions. Furthermore, the number of statements that conflict the given ground-truth knowledge is reduced. These make up the remaining around 40% of annotated partial hallucinations in FaithDial's WoW audit, classified as "Eification" in VRM. This highlights how EWR performs an approximate form of training data removal, where the types of hallucinations present in the training data of the anti-expert are reduced.

On DSTC9, we find that, similarly to WoW, less conflicting statements are generated. For example, Flan-T5base often generates "No" or "Yes" when a specific type of question appears, such as whether a place sells alcohol, even though the knowledge says the opposite. Furthermore, EWR gives more specific responses. The first DSTC9 example in Table 6 shows this: Flan-T5base only mentions that pets on a train should fit into a specifically-sized basket but EWR mentions that they can also simply be kept on a short lead.

## 6 Human Evaluation

In addition to the automatic evaluation, we conduct a human evaluation on WoW and DSTC9 with the help of three expert annotators 3, using the Attributable to Identified Source (AIS) framework Rashkin et al. (2023). First, we ask them to score responses as attributable (A) only if all their content can be attributed to the knowledge that grounds the dialogue response. Furthermore, we ask annotators to rate cooperativeness (C), i.e. the ability of the model to connect with and follow up on user turns on a 3-point Likert scale. Here, 1 indicates a response that does not cooperate with the dialogue, 2 a response that brings the dialogue forward, and 3 a response that acknowledges the previous utterances and responds with a follow-up question. Lastly, annotators rate paraphrasing (P) on a binary scale, where 2 indicates non-trivial paraphrasing of the knowledge and 1 substantial copying. Detailed instructions can be found in Appendix B.4.

Footnote 3: All annotators are graduate students in NLP and paid above minimum wage.

Table 3 shows the results for the A, C, and P categories with agreements of \(0.61,0.51,0.53\), respectively, in terms of Fleiss' \(\kappa\). Generally, we observe that human evaluation results for attributability confirm results based on automatic faithfulness metrics as they display similar patterns. In particular, all methods improve over vanilla Flan-T5, with CTRL and Quark performing similarly on average and outperforming each other on the two different datasets. Task arithmetic and EWR give improvements over CTRL on both datasets. Most notably, EWRCTRL improves over all other methods, including task arithmetic and Quark, by a statistically significant margin in human evaluation.

Our results also emphasize the trade-off between faithfulness and both paraphrasing (which reflects abstractiveness) and cooperativeness. Increased attributability often leads to a decrease in both other criteria. Nevertheless, EWR with a faithfulness anti-expert and an abstraction expert, labelled EWRabs, improves both paraphrasing and attributability on WoW and attributability on both datasets compared to vanilla Flan-T5. While EWRabs does not outperform this baseline in paraphrasing on DSTC9, we believe that this stems from the way the expert dataset \({\mathcal{D}^{\text{E}}}\) is constructed, related to the comparatively less strong performance of Quark and CTRL. As the ground-truth responses in DSTC9 contain longer follow-up questions, it is likely that density-based binning does not pick up nuances, such as the difference between non-paraphrased responses and follow-up questions independent from the knowledge.

\begin{table}
\begin{tabular}{|l|c c c c|c c|} \hline Model & \multicolumn{3}{c|}{WoW} & \multicolumn{3}{c|}{DSTC9} \\  & A (?) & C (?) & P (!) & A (?) & C (?) & P (?) \\ \hline Flan-Tsbase & 72.3 & 1.74 & 1.19 & 89.7 & 2.83 & 1.71 \\ + EWRab & 75.1 & 1.62 & 1.25 & 94.7* & 2.41 & 1.49 \\ CTRL & 85.5* & 1.58 & 1.12 & 94.7* & 2.72 & 1.42 \\ + TA & 88.8* & 1.58 & 1.16 & 97.0* & 2.63 & 1.40 \\ + EWR & 96.8† & 1.50 & 1.08 & 98.0† & 2.50 & 1.36 \\ Quark & 93.1� & 1.51 & 1.05 & 86.0 & 2.89 & 1.66 \\ \hline \end{tabular}
\end{table}
Table 3: Human evaluation on \(218\) examples annotated by \(3\) expert annotators each. We measure attributability (A), Co-cooperativeness (C), and paraphrasing (P). * indicates significance wrt. Flan-T5base and † wrt. to the next best method with \(p<0.05\).

 

Related Work

Hallucination in LMsThe impressive abilities of LMs are offset by the potential for generating hallucinated text [12, 30, 47] which sparked an increasing interest in tackling this problem in the context of grounded language generation [15], encompassing several tasks such as data-to-text generation [68, 9], machine translation [31, 32], summarisation [26], generative question answering [12], and dialogue generation [13, 14, 15, 16, 17, 18, 19]. Different studies aim to address the issue of hallucination by either developing automatic metrics to detect it [68], or by identifying potential causes, such as out-of-domain generalisation, noisy training data, and exposure bias [31, 32, 33, 34].

For neural dialogue models it has been shown that retrieving relevant knowledge can reduce -but not completely eliminate- hallucinations [69]. Therefore, different methods have been proposed to tackle it, such as token-level critics [13], or control token- [15] and reranking-based methods [16]. Lastly, as hallucinations in training data can greatly exacerbate those in models [13], a hallucination-free dialogue benchmark has been proposed [13].

Controllable text generationDifferent works steer model behaviour by controlled generation, for example by combining models at decoding time [14] or in parameter space [12], conditioning on reward tokens assigned to model generations in training [14] or the initial training data [17, 16]. Finally, different methods constrain text generation with logical constraints [14, 15] or by forcing specific words to appear [18].

## 8 Conclusion & Future Work

We introduce Elastic Weight Removal (EWR), a novel method for steering the behaviour of language generation models by combining their parameters with those of (anti-)experts, weighted by Fisher Information. We show how EWR can be used to reduce hallucinations in document-grounded dialogue response generation across different settings. We compare it to other state-of-the-art methods, many of which we adapt to faithful response generation for the first time. Automated metrics and human evaluation show that EWR improves faithfulness over multiple baselines, and can furthermore provide complementary improvements with them. Moreover, we show that faithfulness comes at the expense of abstraction. Therefore, we combine an abstraction expert with the hallucination anti-expert to promote responses that are both more faithful and abstractive than the baseline.

The main contribution of this work is that it outlines an unexplored way of promoting faithfulness in document-grounded dialogue by using experts and anti-experts not at inference time--and thereby incurring significant overhead--but rather to navigate the parameter space towards an improved set of parameters without altering the model architecture. This opens up many potential areas for future work, such as controlling for further dimensions, or developing more sophisticated data augmentation techniques to create data for (anti-)experts.

## Acknowledgements

This project received funding from the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.

## 9 Limitations

One limitation of our work is that we assume the ground-truth knowledge \(\hat{\mathcal{K}}\) to be given. This assumption does not hold in general, when a dialogue system is used, because for a new user query it is unknown. We might then expect that our method stays more faithful to the retrieved knowledge, too, but could generate erroneous responses to the user query if this knowledge is incorrect.

A further limitation is the scale at which we conduct experiments, which do not go beyond 1B parameters due to the large number of baselines that we evaluate on multiple corpora. On the other hand, models used in production are often significantly larger, often having tens of billions of parameters.

Connected to this, many of such models are 

now trained using parameter-efficient finetuning techniques, which either introduce a new subset of model parameters that are trained, while all existing ones are kept fixed, or train a subset of model parameters. Our method should be amenable to this setting, because the task vector will also be \(0\) for parameters that are not trained. However, we did not experiment using parameter-efficient finetuning techniques in this work.

Finally, we only evaluate a small set of (data augmentation) techniques for creating hallucinated and abstractive data and future work could evaluate more such methods.

While we only study english datasets, we expect the techniques to be similarly applicable for other languages.

## Ethics and Broader Impact Statement

Our work relies on LLMs to generate responses in dialogue. Since such LLMs are prone to producing errors, it can not be guaranteed that our methods also do not produce erroneous outputs, such as hallucinations, or output toxic or biased data. However, this work aims to mitigate hallucinations and therefore we think that there is no direct ethical concern.

## References

* Aksitov et al. (2023) Renat Aksitov, Chung-Ching Chang, David Reitter, Siamak Shakeri, and Yun-Hsuan Sung. 2023. Characterizing attribution and fluency tradeoffs for retrieval-augmented large language models. _CoRR_, abs/2302.05578.
* Amari (1998) Shun-ichi Amari. 1998. Natural Gradient Works Efficiently in Learning. _Neural Computation_, 10(2):251-276.
* Ansell et al. (2022) Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulic. 2022. Composable sparse fine-tuning for crosslingual transfer. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1778-1796, Dublin, Ireland. Association for Computational Linguistics.
* Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lorenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet Vo, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-718, Nusa Dua, Bali. Association for Computational Linguistics.
* Choubey et al. (2023) Prafulla Kumar Choubey, Alex Fabbri, Jesse Vig, Chien-Sheng Wu, Wenhao Liu, and Nazneen Rajani. 2023. CaPE: Contrastive parameter ensembling for reducing hallucination in abstractive summarization. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 10755-10773, Toronto, Canada. Association for Computational Linguistics.
* Daheim et al. (2024) Nico Daehim, Thomas Mollenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2024. Model merging by uncertainty-based gradient matching. In _International Conference on Learning Representations_.
* Dahem et al. (2022) Nico Daehim, David Thulke, Christian Dugast, and Hermann Ney. 2022. Controllable factuality in document-grounded dialog systems using a noisy channel model. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 1365-1381, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auil, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In _International Conference on Learning Representations_.
* Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5055-5070, Online. Association for Computational Linguistics.
* Dziri et al. (2022a) Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022a. FaithDial: A Faithful Benchmark for Information-Seeking Dialogue. _Transactions of the Association for Computational Linguistics_, 10:1473-1490.
* Dziri et al. (2021) Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 2197-2214, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Dziri et al. (2022b) Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022b. On the origin of hallucinations in conversational models: Is it the datasets or the models? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5271-5285, Seattle, United States. Association for Computational Linguistics 

* Dziri et al. (2022c) Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022c. Evaluating attribution in dialogue systems: The BEGIN benchmark. _Transactions of the Association for Computational Linguistics_, 10:1066-1083.
* Eric et al. (2020) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 422-428, Marseille, France. European Language Resources Association.
* Gao et al. (2022) Yingbo Gao, Christian Herold, Zijian Yang, and Hermann Ney. 2022. Revisiting checkpoint averaging for neural machine translation. In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_, pages 188-196, Online only. Association for Computational Linguistics.
* Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers)_, pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
* Guerreiro et al. (2023) Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, and Andre Martins. 2023. Optimal transport for unsupervised hallucination detection in neural machine translation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13766-13784, Toronto, Canada. Association for Computational Linguistics.
* Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q\({}^{2}\): Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
* Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In _International Conference on Learning Representations_.
* Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. _ACM Computing Surveys_, 71:387.
* Kang and Hashimoto (2020) Daniel Kang and Tatsumori B. Hashimoto. 2020. Improved natural language generation via loss truncation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 718-731, Online. Association for Computational Linguistics.
* Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. _CoRR_, abs/1909.05858.
* Kim et al. (2020) Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani-Tur. 2020. Beyond domain APIs: Task-oriented conversational modeling with unstructured knowledge access. In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 278-289, 1st virtual meeting. Association for Computational Linguistics.
* Kim et al. (2023) Seokhwan Kim, Spandana Gella, Chao Zhao, Di Jin, Alexandros Papangelis, Behnam Hedayatnia, Yang Liu, and Dilek Z Hakkani-Tur. 2023. Task-oriented conversational modeling with subjective knowledge track in dstc11. In _Proceedings of The Eleventh Dialog System Technology Challenge_, pages 274-281.
* Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieram Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. _Proceedings of the national academy of sciences_, 114(13):3521-3526.
* Kunstner et al. (2019) Frederrik Kunstner, Philipp Hennig, and Lukas Balles. 2019. Limitations of the empirical fisher approximation for natural gradient descent. _Advances in neural information processing systems_, 32.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2021) Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Songfang Huang. 2021. Addressing semantic drift in generative question answering with auxiliary extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 942-947.
* Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayadmidpta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 

6691-6706, Online. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _CoRR_, abs/1907.11692.
* Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The final collection: Designing data and methods for effective instruction tuning. _CoRR_, abs/2301.13688.
* Lu et al. (2022a) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022a. QUARK: Controllable text generation with reinforced unlearning. In _Advances in Neural Information Processing Systems_.
* Lu et al. (2022b) Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowam Zellers, Noah A. Smith, and Yejin Choi. 2022b. NeuroLogic a*esque decoding: Constrained text generation with lookahead heuristics. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 780-799, Seattle, United States. Association for Computational Linguistics.
* Lu et al. (2021) Ximing Lu, Peter West, Rowan Zelters, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4288-4299, Online. Association for Computational Linguistics.
* Martens (2020) James Martens. 2020. New insights and perspectives on the natural gradient method. _Journal of Machine Learning Research_, 21(146):1-76.
* Matena and Raffel (2022) Michael S Matena and Colin Raffel. 2022. Merging models with fisher-weighted averaging. In _Advances in Neural Information Processing Systems_.
* McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In _Psychology of Learning and Motivation_, volume 24, pages 109-165. Elsevier.
* Niu and Bansal (2018) Tong Niu and Mohit Bansal. 2018. Polite Dialogue Generation Without Parallel Data. _Transactions of the Association for Computational Linguistics_, 6:373-389.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bihuwan Dingra, Diyi Yang, and Dipjan Das. 2020. ToToTt: A controlled table-to-text generation dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1173-1186, Online. Association for Computational Linguistics.
* Pascual et al. (2021) Damian Pascual, Beni Egeressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer. 2021. A plug-and-play method for controlled text generation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3973-3997, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Peter et al. (2018) Jan-Thorsten Peter, Eugen Beck, and Hermann Ney. 2018. Susyphus, a workflow manager designed for machine translation and automatic speech recognition. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 84-89, Brussels, Belgium. Association for Computational Linguistics.
* Pfeiffer et al. (2023) Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, and Edoardo Ponti. 2023. Modular deep learning. _Transactions on Machine Learning Research_, 12/2023. Survey Certification.
* Ponti et al. (2021) Edoardo M. Ponti, Ivan Vulic, Ryan Cotterell, Marinela Parovic, Roi Reichart, and Anna Korhoonen. 2021. Parameter space factorization for zero-shot learning across tasks and languages. _Transactions of the Association for Computational Linguistics_, 9:410-428.
* Ponti et al. (2019) Edoardo Maria Ponti, Ivan Vulic, Ryan Cotterell, Roi Reichart, and Anna Korhonen. 2019. Towards zero-shot language modeling. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2900-2910, Hong Kong, China. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaiao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language processing task solver? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 1339-1384, Singapore. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67 

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring Attribution in Natural Language Generation Models. _Computational Linguistics_, pages 1-64.
* Rashkin et al. (2021) Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 704-718, Online. Association for Computational Linguistics.
* Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1172-1183, Online. Association for Computational Linguistics.
* Razumovskaia et al. (2022) Evgeniia Razumovskaia, Ivan Vulic, and Anna Korhonen. 2022. Data augmentation and learned layer aggregation for improved multilingual language understanding in dialogue. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2017-2033, Dublin, Ireland. Association for Computational Linguistics.
* Razumovskaia et al. (2023) Evgeniia Razumovskaia, Ivan Vulic, Pavle Markovic, Tomasz Cichy, Qian Zheng, Tsung-Hsien Wen, and Pawel Budzianowski. 2023. Dial beinfo for faithfulness: Improving factuality of information-seeking dialogue via behavioural fine-tuning. _CoRR_, abs/2311.09800.
* Schraudolph (2002) Nicol N Schraudolph. 2002. Fast curvature matrix-vector products for second-order gradient descent. _Neural Computation_, 14(7):1723-1738.
* Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Stiles (1992) William B Stiles. 1992. _Describing talk: A taxonomy of verbal response modes_. Newbury Park: Sage Publications.
* Sun et al. (2022) Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuandjing Huang. 2022. BERTScore is unfair: On social bias in language model-based metrics for text generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3726-3739, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Thoppilan et al. (2020)Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jia, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixu Steven Zheng, Amin Ghafouri, Marccle Menegali, Yanping Huang, Maxim Kristku, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokom, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Helstern, Meredith Ringel Morris, Tulse Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Buttyna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzwell, Blaise Agueracracas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog applications. _CoRR_, abs/2201.08239.
* Wang and Sonnrich (2020) Chaojun Wang and Rico Sonnrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3544-3552, Online. Association for Computational Linguistics.
* Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Phtu, Canwen Xu, Tevein Le Scao, Sylvian Guugger, Mariana Darme, Quentin Lobest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhang* et al. (2020) Tianyi Zhang*, Vasha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: evaluating text generation with BERT. In _International Conference on Learning Representations_.
* Zhao et al. (2023) Chao Zhao, Spandan Gella, Seokhwan Kim, Di Jin, Devamanyu Hazarika, Alexandros Papangelis, Behnam Hedayatnia, Mahdi Namazifar, Yang Liu, and Dilek Hakkani-Tur. 2023. "what do others think?": Taskoriented conversational modeling with subjective knowledge. In _Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 309-323, Prague, Czechia. Association for Computational Linguistics 



## Appendix A Details on Method

### Pseudocode

```
Input Dialogues \({\cal D}\), hallucinated anti-expert dataset \({\cal D}^{\text{AE}}\), abstractive expert dataset \({\cal D}^{\text{B}}\), initial parameter set \(\boldsymbol{\theta}\) Output \(\boldsymbol{\theta}^{\prime}\)  \(\boldsymbol{\theta_{0}}~{}\leftarrow\texttt{finetune}(\boldsymbol{\theta},{ \cal D})\) \(\boldsymbol{\theta_{\text{AE}}}\leftarrow\texttt{finetune}(\boldsymbol{\theta}_{0} ,{\cal D}^{\text{AE}})\) \(\tau_{\text{AE}}~{}\leftarrow\texttt{\theta_{AE}}-\boldsymbol{\theta}_{0}\) \(\boldsymbol{\theta_{\text{E}}}~{}\leftarrow\texttt{finetune}(\boldsymbol{ \theta}_{0},{\cal D}^{\text{E}})\) \(\tau_{\text{AE}}~{}\leftarrow\texttt{\theta_{BE}}-\boldsymbol{\theta}_{0}\) \(\boldsymbol{\ell_{0}}~{}\leftarrow\frac{1}{|{\cal D}|}\sum_{{\cal D}=1}^{{ \cal D}}(\nabla\log p_{\boldsymbol{\theta}_{0}}(u_{T+1}\mid u_{1}^{T},\hat{{ \cal K}}))^{2}\) \[\mathds{f}_{\tau_{\text{AE}}}~{}\leftarrow\frac{1}{|{\cal D}^{ \text{AE}}|}\sum_{{\cal D}^{\text{AE}}}(\nabla\log p_{\tau_{\text{AE}}}(u_{ T+1}\mid u_{1}^{T},\hat{{\cal K}}))^{2}\] \[\mathds{f}_{\tau_{\text{E}}}~{}\leftarrow\frac{1}{|{\cal D}^{ \text{B}}|}\sum_{{\cal D}^{\text{E}}}(\nabla\log p_{\tau_{\text{AE}}}(u_{T+1} \mid u_{1}^{T},\hat{{\cal K}}))^{2}\] \[\boldsymbol{\theta}^{\prime}~{}\leftarrow\frac{\lambda_{o}\, \mathbf{f_{0i}}_{\boldsymbol{0}}\cdot\lambda_{\text{BE}}\,\mathsf{f_{r_{BI}}} \cdot\tau_{\text{BB}}+\lambda_{\text{E}}\,\mathsf{f_{r_{I}}}\cdot\tau_{\text{ B}}}{{\cal Z}}\]
```

**Algorithm 1** Pseudocode for removing hallucinations and promoting abstraction with EWR. Note that we apply \((\cdot)^{2}\) element-wise.

Algorithm 1 outlines the steps for using EWR to reduce hallucinations while promoting abstractiveness. Concretely, a dialogue response generation model is trained first. Then, an anti-expert and expert model are trained on hallucinated and abstractive (and not hallucinated) data, respectively. Both models are the subtracted and added to the dialogue response generation model, respectively, but weighted by Fisher information. The Fisher information is estimated by its diagonal with a squared gradient approximation over the training, where labels are sampled. We have found that calculating this by parameterising the model with the task vectors \(\boldsymbol{\tau_{\text{AE}}}\) and \(\boldsymbol{\tau_{\text{E}}}\) performs empirically well, but it is theoretically better motivated to calculate it at the anti-expert \(\boldsymbol{\theta_{\text{AE}}}\) or expert \(\boldsymbol{\theta_{\text{E}}}\), respectively. Both strategies provided similar performance in our experience.

## Appendix B Details on Experiments & Evaluation

### Further Experimental Details

All models, with the exception of Quark and (anti-)experts, which we train for 5 epochs, are trained for 10 epochs using an initial learning rate of \(6.25e{-5}\), linear learning rate decay without warmup, and a batch size of 32, following prior work (Daehim et al., 2022). We take checkpoints after each epoch and pick the one with smallest validation loss. For Task Arithmetic and EWR we do a grid search to determine the scaling factors on a validation set on WoW, FaithDial, and DSTC9. For DSTC11 we did not perform such a grid set because we only had a validation but not a test set, and the hyperparameters seemed to be consistent across datasets. We chose \(1.0\) for Task Arithmetic and \(0.15\) for EWR for all experiments with only a hallucination anti-expert, since these factors performed best. We use Flan-T5base and Flan-T5large with 250M and 780M parameters, respectively. We use the checkpoints that are available on the huggingface hub under [https://huggingface.co/google/flan-t5-base](https://huggingface.co/google/flan-t5-base) and [https://huggingface.co/google/flan-t5-large](https://huggingface.co/google/flan-t5-large). All experiments are performed on NVIDIA A100 or V100 GPUs and each model takes at most half a day to finetune.

All code for reproducing the experiments will be made publicly available in a comprehensive software repository under Apache License 2.04.

Footnote 4: [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)

### Further Details on Datasets

In this section we provide details on the splits of all used datasets. The statistics are shown in Table 4. For Wizard-of-Wikipedia, we have used the train, dev and both test splits (seen and unseen). For DSTC11 we have only used validation split, because the test set was not yet available at the time of our experiments.

For the hallucination anti-expert model, the training data is exactly the same size as for the document-grounded response generation model, just with the knowledge switched out. For all expert models we subsample the data according to the assigned control tokens which depend on the used metric and NLI model.

All datasets are in English and might therefore represent predominantly the demographics of

\begin{table}
\begin{tabular}{|l|c||c|c|} \hline Dataset & \#train & \#val & \#test \\ \hline WoW (Dinan et al., 2019) & 83247 & 4444 & [4356, 4380] \\ DSTC9 (Kim et al., 2020) & 19184 & 2673 & 1981 \\ FaithDial (Zirzi et al., 2022a) & 18357 & 3417 & 3539 \\ DSTC11 (Zhao et al., 2023) & 14768 & 2129 & - \\ \hline \end{tabular}
\end{table}
Table 4: Dataset statistics showing the number of train, validation, and test examples counted in number of utterances. For WoW test, we first show the seen and then unseen split in curly brackets. For DSTC11, the test set was not available yet at the time of writing.



english-speaking countries. WoW was collected by crowdsourcing dialogues in a roleplaying game. DSTC9 was collected by asking crowdworkers to fill in dialogues from MultiWoZ 2.1 Eric et al. (2020). DSTC11 was collected using crowdworkers on Amazon MTurk, who stem from the USA, Canada, and Great Britain Zhao et al. (2023). Finally, FaithDial was created by asking crowdworkers, also on Amazon MTurk, to clean dialogues from WoW Dziri et al. (2022a).

### Further Details on Used Metrics

We evaluate BLEU Papineni et al. (2002) on the corpus-level using the sacrebleu package Post (2018). Other metrics are calculated on an example-level and averaged to obtain a global score. Concretely, for critic model taken from Dziri et al. (2022a), this means that we classify each utterance as hallucination or not, with \(1\) indicating hallucination and \(0\) otherwise. The score is averaged over these classifications and can therefore be seen as calculating the percentage of hallucinated examples in the model predictions. The model used for this is finetuned from RoBERTA Liu et al. (2019) and released as part of Dziri et al. (2022a). It is openly available on the huggingface hub and can be found under [https://huggingface.co/McGill-NLP/roberta-large-faithcritic](https://huggingface.co/McGill-NLP/roberta-large-faithcritic). For \(Q^{2}\)Honovich et al. (2021), a pipeline of steps is performed for each generated example to arrive at a score. First, answer candidates are determined for the generated response, which often correspond to spans of entities. Then, questions are generated for each answer candidate and answered based on the knowledge documents. If the answer is the same by string match, a score of \(1\) is assigned. If there is no string match, a score of \(1\) is assigned if an NLI model judges one answer to entail the other, and a score of \(0\) otherwise. Questions are also filtered, and if no valid question is found, entailment between the knowledge and the generated response is calculated as a fallback. We base our implementation on the open-source implementation found in [https://github.com/orhonovich/q-squared](https://github.com/orhonovich/q-squared) which was released with Honovich et al. (2021) and will open-source our reproduction under Apache License 2.0.

Our adoption of density Grusky et al. (2018) calculates the average squared length of extractive spans that were copied from the knowledge documents into the generated response. We average the densities of all predictions. Similarly, F1 calculates the token-level overlap between generated response and document, and we again take the average over predictions. Again, all the implementations of these metrics will be made publicly available by us.

For BertScore Sun et al. (2022), we use the open-source implementation found at [https://github.com/Tiiiger/bert_score](https://github.com/Tiiiger/bert_score) and use the 'debetra-large-mnli' checkpoint, which was recommended at the time of implementation.

### Details on Human Evaluation

In this section, we detail the instructions and recruitment for our human evaluation. All of the annotators are graduate students in NLP from one of the authoring institutions and are all paid well above minimum-wage. All annotators voluntarily agreed to participating in our study and were informed, and agree to, that no personal data would be released and only the human judgements would be stored. The annotators were instructed to score 218 randomly sampled examples generated with different models from WoW and DSTC9 according to three criteria: Faithfulness, Coherence, and Paraphrasing, abbreviated with F, C, and P, respectively, in Table 3. The instructions for Faithfulness follow the well-established Attributable to Identified source framework (AIS) Rashkin et al. (2023). We follow the exact definitions from their work and show these as guidelines to the annotators, who were instructed to carefully read the paper. This is feasible, because all annotators have graduate-level knowledge of NLP. Following the frame work, we instructed users to only annotate interpretable responses, others were to be left out. Then, a score of one should be assigned if the conditions in Rashkin et al. (2023, Definition 8) are met. We repeat the definition here verbatim for completeness and refer the reader to their work for more information about the framework.

**Definition 1**.: _AIS, full definition Rashba et al. (2021) A pair \((s,c)\), where \(s\) is a sentence and \(c_{!},t\) is a pair consisting of a linguistic context and a time, is Attributable to Identified Sources (AIS) iff the following conditions hold:_

1. _The systems provides a set of parts_ \(P\) _of some underlying corpus_ \(K\)_, along with_ \(S\) 

2. \(s\) _in the context_ \(c\) _is interpretable (i.e.,_ \(E(c,s)\neq\text{NULL}\)_._
3. _The explicature_ \(E(c,s)\) _is a standalone proposition._
4. _The pair_ \((E(c,s),t)\) _is attributable to_ \(P\)_._

_The pair_ \(E(c,s),t)\) _is attributable to a set of parts_ \(P\) _of some underlying corpus_ \(K\) _iff: A generic hearer will, with a chosen level of confidence, affirm the following statement: "According to_ \(P\)_,_ \(E(c,s)\)_, where_ \(E(c,s)\) _is interpreted relative to time t."_

According to this, a binary label is assigned, where \(1\) indicates 'faithful' and 0 'not faithful'. We only make a slight change in definition for DSTC9, where the FAQ documents are short and give relevant information to a customer in customer service conversations, for example, for hotel booking. The change is as follows: "If important information for the user in \(K\) is left out, the response should be scored as 'not faithful'."

For Coherence, we ask the annotators to only score such responses that were annotated with \(1\) in the previous step on a 3-point Likert scale. The instructions are as follows:

* The response is highly co-operative and, for example, explicitely acknowledges the previous turn (e.g. ""Yes,...".) and contains a follow-up question.
* The response follows up logically to the previous dialog and / or shows some degree of co-operativeness.
* The response is standalone and does not follow-up logically to the previous dialog.

Here, the listing item (e.g. "3:") indicates the rating.

For Paraphrasing, we chose a two-point scale with the following instructions:

* Response paraphrases the evidence to a sufficient extent.
* The response copy-pastes the evidence into the response verbatim or almost verbatim.

As noted in Section 6, we achieve agreements of \(0.61,0.51,0.53\), respectively, in terms of Fleiss' \(\kappa\), for the three categories above in order of writing.

## Appendix C Further Results

### Additional Experiments Using Flan-T5large

Table 5 shows results obtained using the same setup as in Section 5.1 but using Flan-T5large instead of Flan-T5base. We find the results from the smaller checkpoint to be confirmed and find much larger improvements for EWR on DSTC9 than using the base checkpoint. Again, parameter interpolation methods can be used effectively to reduce hallucinations at minor costs of fluency and abstractiveness, also on top of other methods that promote faithfulness. However, we find CTRL and Quark less effective for DSTC9, potentially because the overlap and entailment tokens have more errors than in WoW due to the structure of the used FAQ documents.

#### c.1.1 Qualitative Examples

Qualitative examples for EWR and a Flan-T5base baseline on WoW and DSTC9 are shown in Table 6. A discussion is found in Section 5.5 



\begin{table}
\begin{tabular}{|l|p{284.5pt}|} \hline  & \multicolumn{3}{c|}{WoW} \\ \(u_{t}\) & now, do you know the name of his first novel? \\ \(\hat{\mathcal{K}}\) & Sparks was born in Omaha, Nebraska and wrote his first novel, “The Passing”, in 1985, while a student at the University of Notre Dame. \\ Flan-T5base & Yes, it was called ”The Passing” and was published in 1985. \\ EWR & Yes, it was called “The Passing” and it was written while he was a student at Notre Dame. \\ \(u_{T}\) & Interesting! Has she been a vegan for her entire life? I think it would be hard to give up meat! \\ \(\hat{\mathcal{K}}\) & Many people object to eating meat out of respect for sentient life. \\ Flan-T5base & I don’t think she’s been a vegan for a long time. Many people object to eating meat out of respect for sentient life. \\ EWR & I don’t know how long she’s been a vegan, but I know many people object to eating meat out of respect for sentient life. \\ \hline DSTC9 \\ \(u_{t}\) & What is the exact size limit of pets onboard \\ \(\hat{\mathcal{K}}\) & You may bring 2 domestic petals, they must be \text@underline{1} kept on a short lead, or \text@underline{2} carried in a basket size 85x60x60cm. \\ Flan-T5base & \text@underline{2} They must be able to be carried in a basket sized 85x60x60cm. Is there anything else I can help you with? \\ EWR & \text@underline{1} Pets must be kept on a short lead or \text@underline{2} carried in a basket size 85x60x60cm. Is there anything else I can help you with? \\ \(u_{t}\) & Do they serve alcohol? \\ \(\hat{\mathcal{K}}\) & Frascati serves beer and wine only. \\ Flan-T5base & No, they only serve beer and wine. Anything else I can do for you? \\ EWR & Yes, they serve beer and wine. Is there anything else I can help you with? \\ \hline \end{tabular}
\end{table}
Table 6: Qualitative examples for Flan-T5base and EWR on WoW and DSTC9. We find that faithfulness is improved due to various reasons. On WoW, EWR tends to generate less conflicting and opinionated outputs. On DSTC9, EWR generates more specific outputs that, for example, point out alternative options, and less generic responses that are often hallucinated.

\begin{table}
\begin{tabular}{l|l l l l l l l l l l l l} \hline  & \multicolumn{7}{c}{WeWoW} \\  & BLEU(\(\uparrow\)) & Critic(\(i\)) & \(Q^{2}(\uparrow)\) & BERT(\(\uparrow\)) & F1(\(\uparrow\)) & Dens.(\(j\)) & BLEU(\(\uparrow\)) & Critic(\(j\)) & \(Q^{2}(\uparrow)\) & BERT(\(\uparrow\)) & F1(\(\uparrow\)) & Dens.(\(j\)) \\ Model & \((y,\bar{y})\) & & & \((y,\hat{\mathcal{K}})\) & & & \((y,\hat{\mathcal{K}})\) & & & & \((y,\hat{\mathcal{K}})\) & & \\ \hline Flan-TS\({}_{\text{large}}\) & 18.6 & 26.7 & 77.8 & 83.8 & 77.5 & 12.3 & 18.6 & 6.9 & 64.0 & 61.2 & 44.7 & 1.81 \\ + TA & 19.1 & 16.7 & 80.2 & 84.6 & 77.8 & 12.6 & 19.0 & 3.7 & 74.3 & 64.4 & 55.6 & 3.50 \\ + EWR & 17.3 & 16.9 & 80.3 & 88.3 & 83.3 & 83.9 & 14.9 & 19.1 & 28.0 & **28.3** & **83.0** & **64.8** & **57.3** & **3.48** \\ CaPE & 19.0 & 13.0 & 79.5 & 83.7 & 75.4 & 11.3 & 17.2 & 4.3 & 73.3 & 64.4 & 53.2 & 2.82 \\ + EWR & 18.2 & (14.6 & 9.3 / 13.5) & 80.4 (14.8) & 88.4 (15.2) & 84.9 (15.2) & 16.2 (14.6) & 1.1 (14.3) & 74.9 (14.1) & 64.1 (14.3) & 54.1 (14.9) & 30.0 (14.6) \\ CTRL & **19.8** & 11.3 & 82.0 & 87.3 & 81.5 & 13.4 & **19.5** & 6.8 & 77.4 & 63.8 & 52.7 & 2.73 \\ + TA & 19.2 & 7.2 & 84.3 & 86.8 & 80.6 & 13.0 & 19.3 & 2.6 & 79.3 & **65.9** & 57.5 & 3.37 \\ + EWR & 18.6 & (14.2 & **7.0** / 14.8 & 85.8 (14.8) & 90.5 (14.2) & 86.8 (14.6) & 18.1 (14.6) & **0.8** / 14.6 (14.6) & **0.8** / 14.6 (14.