Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs
Authors: Yatin Nandwani
Vineet Kumar
Dinesh Raghu
Sachindra Joshi
Luis A. Lastras
Venue: EMNLP 2023 Main (2023)
Keywords: docuemnt grounded dialogs
dialog response generation
faithful response generation

Abstract: A major concern in using deep learning based generative models for document-grounded dialogs is the potential generation of responses that are not faithful to the underlying document. Existing automated metrics used for evaluating the faithfulness of response with respect to the grounding document measure the degree of similarity between the generated response and the document's content. However, these automated metrics are far from being well aligned with human judgments. Therefore, to improve the measurement of faithfulness, we propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. PMI quantifies the extent to which the document influences the generated response -- with a higher PMI indicating a more faithful response. We build upon this idea to create a new decoding technique that incorporates PMI into the response generation process to predict more faithful responses. Our experiments on the BEGIN benchmark demonstrate an improved correlation of our metric with human evaluation. We also show that our decoding technique is effective in generating more faithful responses when compared to standard decoding techniques on a set of publicly available document-grounded dialog datasets.

Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs
Yatin Nandwani, Vineet Kumar, Dinesh Raghu, Sachindra Joshi and Luis A. Lastras IBM Research, AI
vineekudein, di diraghul@in, jsachindein, lastrasl@us) ibm. com
Abstract
Creating free my Social Security account takes than minutes, you up chunge your direct depocit gives you many other online services.
major concern using deep learning based generative models for dbcumeent-roounded di- alogs is the potential generation of responses are not faithful to underlying document. Existing automated metrics used for evaluating the faithfulness of response with respect to the grounding document measure degree sim- ilarity between the generated response and the document's content. However, these automated metries far from being well aligned with human judgments. Therefore, to improve the measurement of faithfulness, we propose a new metrie that utilizes (Conditional) Point-wise Mutual Information (PMI) between the gener- ated response
Dialog Historry
Hi, is the social security account free of charge?
Next Responses
Yes, the social sccurity account is free
Yes, course
Figure An example document grounded dialog with two types of responses: sentential response (R2) and non-sentential response (R1).
source document, condi- tioned on the dialogue. PMI quantifies the tent to which the document influences the gen- erated response with higher PMI indicating a more faithful response. We build upon this idea to create a new decoding technique that incorporates PMI into the response generation process to predict more faithful responses. Our experiments on the BEGIN benchmark demon- strate improved correlation of metric with human evaluation. We also show that our decoding technique effective in generating more faithful responses when compared to stan- dard decoding techniques on a set of publicly dialog datasets.
have been proposed. These metrics take as input the agent generated response and the grounding document to quantify faithfulness. These are based on lexical overlap (e.g., BLEU, unigram-F1), se- mantic overlap (BERTScore) sifier (Dziri al., 2022a). Recently, Honovich al. (2021) proposed Q², a metric that measures faith- fullness using automatic question generation and question answering.
a trained clas-
available
A major limitation of existing metrics is that they ignore the crucial dialog history when measuring faithfulness of responses. Even though, in many cases, the dialog history provides essential context that necessary for a complete understanding the response. To illustrate this point, let's consider two responses, textitR1 and R2 as depicted in Fig- ure 1. Response R/ is self-contained and can be comprehended without relying on the dialog his- tory. On the other hand, response R2 is dependent on the dialog history and can only be fully under- stood when considering the preceding conversation. Unfortunately, current automated metries do not take into account the dialog history, leading to their failure in evaluating responses that are not self-
1 Introduction
Document- -rouudded dialog agents converse with users based on information present document provided to them. These agents are expected to be factually consistent or faithful to the grounding document and refrain from generating content that cannot be verified using the document. As most existing document- dboumenntgroundded dialog agents (Prab- humoye et al., 2021; Wu al., 2021) are built by fine-tuning large language models, ensuring faith- ful response generation is major challenge.
To measure the ability of dialog agents to gen- erate faithful responses, several automatic metrics

---

contained. Responses like R2 often lack domain- specific words, making similarity-based metrics like unigram-Fl and BERTScore ineffective. Ad- ditionally, generating aresttin-aaswer pairs using such responses typically captures incomplete infor- mation, rendering metrics like Q2 as imadequate measures.
2 Related Work
In this work, focus primarily faithfulness aspect of the generated responses with respect the grounding document. It is crucial to distinguish between faithfulness and hallucination (Maynez et al., 2020) in evaluating responses. A response is considered faithful only when all the informa- tion it contains can be verified or inferred from the grounded document. On the other hand, a response is considered hallucination if it provides false or fabricated information. It is important to note that there can be responses that are not hallucina- tions but are still unfaithful. In such cases, the information provided may not be false, but it can- not be verified using the grounded document as a reference. It is important to point out that the set of faithful responses is a subset of responses that not hallucinations. In this section, discuss related work in faithfulness, followed by brief dis- cussion on Mutual Information in conversational settings.
To overcome this problem, we propose a new metric that quantifies the faithfulness of gener- ated response with respect to both the document and the dialog history. Our metric is grounded in information theoretic concepts and captures the as- sociation of the response with the given document using Conditional Pointwise Mutual Information (CPMI). We call our metric PMI-FAITH, which uses CPMI between the generated response and the document, conditioned on the dialogue history, for quantifying faithfulness. PMI-FAITH captures the intuition that for response to be grounded in the document, the probability of its generation given the document should be higher than the probability of its generation without the document.
Researchers have used various terms such as faithfulness (Cao et al., 2018), factual consistency (Cao et al., 2020; Santhanam et al., 2021), factual accuracy (Goodrich et al., 2019), fidelity (Chen et al., 2020), attribution (Rashkin al., 2021a) and hallucination (i.e., the lack of faithfulness) (Xiao and Wang, 2021) 1) to define and quantify faithfulness of a model's generated text to a given knowledge.
A significant advantage of our metric PMI-FAITH is that it can be factorized the same way as the likelihood of a response can be factorized in auto regressive models. We take advantage of this property to propose a novel decoding objective, PMI-DECODE. The goal of PMI-DECODE is to maximize not just the response's ' likelihood but score that combines its likelihood and faithfulness.
Most of the works focusing on evaluating faith- fulness propose to train classifier for the task (Goodrich et al., 2019; Kryseinski al., 2020; Dziri et al., 2022a). Whereas our proposed met- ric doesn't 't require any training and is agnostic to the underlying data. Recently, Honovich et al. (2021) proposed Q² for quantifying faithfulness. It uses question generator to first generate question- answer (QA) pairs from the generated response. Then QA system is used to find an answer, to the generated question, from the document. Finally, NLI system is used compare the two answers. Though Q² uses the given document to check the faithfulness of a response, ignores the dialog his- tory. Thus, may fail at handling responses that are non-self contained as depicted in Figure 1. Our metric PMI- FAI-FAITH addresses this issue.
To summarize, our contributions are threefold:
1. We propose PMI-FAITH, a novel metric which quantifies faithfulness as a conditional PMI between the response and the document given the dialog history.
2. We propose novel decoding objective, PMI-DECODE, which can aid in generating faithful responses.
3. Our experiments show that PMI-FAITH cor- relates with human judgments better than any existing metrics on the BEGIN benchmark (Dziri et al., 2022b). We also show that us- ing PMI-DECODE as the objective generates more faithful responses than standard likeli- hood objective on three standard document- grounded dialog datasets.
Many recent works (Dziri et al., 2022b; Hon- ovich et al., 2022) have released different bench- marks that can be used to evaluate the performance of faithfulness metries. While Honovich et al. aim to standardize benchmark datasets across different
We release our code for further use by the research community.
'httts://ggthhin http:////ggttuu..oommaaaaaa/r//e

---

generation tasks, Dziri et al. focus on document- grounded dialogues, and thus we use their bench- mark to compare our metric with various baselines. Li et al. (2016) and Paranjape and Manning (2021) use mutual information to prevent the con- versational models from generating generic re- sponses (such "Sorry, I'm not sure about this topic"). Contemporary to work, Ren et al. (2023) propose to use a Conditional Pointwise Mu- tual Information (CPMI) based metric to evaluate relevance of a generated response with respect to a reference hypothesis for open-domain response generation. In contrast, our work is the first to use CPMI as metric for evaluating the faithfulness of response given document and dialogue history in bolcummnn--groundee response generation.
where F represents logical entailment. A faithfulness metric should quantify the faith- fulness of the response r to the document d and dialogue history h. In general, such a metric should take d and h as its input and compute score, F(r, d, h) R, such that higher value of F(r, d, h) indicates more faithful response.
Approach
In this section, we describe our proposed metric for faithfulness - PMI-FAITH. We then propose decoding strategy PMI-DECODE based on our metric, with the objective of generating relevant and faithful responses.
4.1 PMI-FAITH
3 Background
PMI-FAITH based the information-theoriii concept of Pointwise Mutual Information. We use the notion of CPMI between generated response and the document d given the context h to capture the influence of the document in generating the response. We define our metric, PMI-FAITH, for faithfulness of the response r to the document d as:
In this section, we first review the task of document- grounded dialog respouse generation, followed by the definition of the faithfulness metric.
Document Grounded Response Generation: Let dialog history h [w,, Um] be sequence of m utterances in the dialog so far and be the doc- ument on which the next response is grounded. The task of document-grounded dialog response generation is to predict the next response, r (r172... FT), one token time, given the dialog history and the document d. Here, Vi,r; C V. where V is the vocabulary of all possible tokens. The underlying model learns a probability distribu- tion dr h) over all possible responses r V+, where V is the space of all the sequences having one or more tokens from vocabulary V.
PMI-FAITH(r, h, CPMI(r; d
P(r|d, h) P(r, log P(r[h)
= log P(rlh)
(2)
Mathematically. PMI is measure of the strength of the association between two random events. A positive value of CPMI in eq. (2) implies that the probability of generating the response given the document and the dialogue history is higher than the probability of generating the response given only the dialogue history. Hence, the response is likely to be grounded in the document. On the other hand, if the response r is not faithful to the document d, the probability of its generation given the document and the dialogue history is likely to be similar to the probability of its generation with- out the document, resulting in lower value of PMI-FAITH. We pre-trained language mod- els such as BLOOM (Scao et al., 2022) or GPT2 (Radford et al., 2019), to compute these conditional probabilitics P d, h) and P(r[h).
Typically, this distribution is factorized over the tokens of as:
T P(r/d, h) II t=1
(1)
Most of the existing defini- tions (and metrics) for faithfulness focus mainly on document and response r but ignore the his- tory h (Dziri et al., 2022a,b). This may be for the sake of uniformity across different tasks such as summarization. grounded dialogue generation, and paraphrase generation. We qualify the definition of faithfulness specifically for the task of document- grounded dialogue generation. Formally, a re- sponse r is considered 'faithful' to given doc- ument and the dialogue history h iff d, F=
Faithfulness Metric:
4.2 PMI-DECODE
PMI-DECODE is decoding strategy whose objec- tive is to generate responses that are both relevant and faithful. Typically, the goal of any decoding

---

strategy is select a response that has the maxi- mum (log) likelihood:
lows:
PMI-FAITH(TIH-I) d,
log P(ri=-1,
arg max log P(r h)
P(mx--1d,,h
TEV+
log P(r
P(u|d, h, r1x-1)
+ log P( r1.t-)
The objective of PMI-DECODE is to select re- sponse that is highly likely and faithful. This is achieved by maximizing combination of likeli- hood and faithfulness quantified using an appropri- ate metric F. With a E [0, 1]. and linear scoring function, we get:
- PMMI-FATH(TIA-I.) h,
(6)
+ CPMI( d r1a-1)
By using PMI-FAITH as F eq. (4), and drop- ping the two terms which are independent of v from eq. (5) and eq. (6), the objective of greedy decoding using the
objective is expressed as:
r arg max(1- a) log h) oF h) reV+
arg max (1 a) log P(uld. h, r1a-1)
VEV
(7)
aCPMI(U;; d/h, ria-1)
With an auto-regressive model that generates the response one token at a time, we use decoding strategies, such as greedy decoding, beam search, nucleus sampling (Holtzman et al., 2020), or beam sampling a heuristic find the maxima. For ease of description, we use the greedy decoding be- low, though our approach is agnostic to the choice of heuristic for maximising the objective function. It just modifies the standard ligeeliolooo objec- tive with an additional term corresponding to faith- fulness. Our choice of PMI-FAITH as function F for quantification of faithfulness keeps the decod- ing heuristic tractable as shown below.
To compute CPMI in eq. (7), the same language model be used to get the conditional proba- bilities P( d, h, rit-1) and P(o Fit-11) by sep- arately passing d,h, Fı:t 1 and h, Fıa 1, respec- tively, through the model.
We observed that using CPMI in the scoring function sometimes results in selecting tokens from the document which may interfere with the gram- mar. To mitigate this, instead of maximizing over the entire vocabulary V at each step f, propose to maximize only over the 'top p' subset from the likelihood distribution, Vp.,- defined as the mini- mum cardinality subset of tokens with the sum of their probabilities as p. We call this top masking:
With eq. (3) as the objective. greedy decoding would sample the next token 7t follow:
rt = arg max (1 ai) log P(uld, h Tia 1) VEV,.
(8)
aCPMI ( dh, ria-1)
F arg max log P(r 1, b, VEV
The intuition here is that while CPMI has pos- itive influence on generating more faithful re- (5) sponse, may negatively impact the grammatical structure. Therefore by restricting the vocabulary to Vp.,, we use only highly probable tokens to form a response and thus are likely to generate responses that are faithful as well as grammatically correct.
arg max log P(r vCV
+ log h, Fia
In eq. (5), the likelihood term has been factorized and notice that its first term is independent of the next token candidate and thus be dropped while taking arg max. Not all faithfulness metrics can be decomposed the same way as the likelihood term. One advantuge of PMI-FAITH is that it can be decomposed the same way as likelihood as fol-
5 Experiments
Our experiments answer two research questions: PMI-FAITH: How does our novel metric per- form when compared exisitng metrics on a standard benchmark (section 5.2)?

---

2. PMI-DECODE: Does our proposed decoding technique generate responses that are more faithful compared to vanilla decoding tech- niques, while still maintaining relevance (sec- tion 5.4).?
Metric
Precision
Recall
F1
Accuracy
U-FI BLEU
0.401
0.785
0.531
0.677
0.478
0.479
0,479
0.757
RougeL
0.487
0.552
0.518
0.760
BERTScore
0.459
0.673
0.546
0.739
FaithCritic
0.684
0.492
0.573
0.829
Q²
0.517
0.744
0.610
0.779
5.1 PMI-FAITH: Experimental Setup
UPMI-FAITI
0.592
0.704
0.643
0.818
PMI-FAITH
0.607
0.818
0.697
0.834
Dataset: We experiment using recently proposed BEGIN benchmark (Dziri al., 2022b) for eval- uating the ability of PMI-FAITH to identify faith- ful responses. This benchmark uses three docu- ment grounded datasets, viz, CMU-DoG (Zhou et al., 2018), TopicalChat (Gopalakrishnan et al.. 2019), and WoW (Dinan et al., 2019). It contains 11, 059 responses generated by three different mod- els, GPT2 (Radford et al., 2019), DoHA (Prabhu- moye et al., 2021) and T5 (Raffel et al., 2020), on randomly selected samples from test splits of the 3 datasets. Each generated response is anno- tated by humans and classified into either 'Fully- attributable', "Generic', or 'Not fully-attributable,, Overall, 23.3% of the total response have been clas- sified FFully auuly-attributaale by human annotators, Evaluation Metries: Our objective is to iden- tify faithful responses. Accordingly, we consider atttibyattbbb as the positive class and both the neg-
Table 1: Performance of various faithfulness metrics on the BEGIN Benchmark.
2002), and RougeL (Lin, 2004) to capture lex- ical overlap between d and generated response BERTSccre(ZZann et al., 2020) to capture r's semantic similarity with d. We use the code? provided by Honovich et al. (2021) for all the above baselines." 3 We also compare faith- critic"(Dziir et al., 2022a), which is a ara-trained classifier to predict faithfulness of a response.
faith-
Training Details: To measure PMI-FAITH, we need to compute two conditional probabilities: P(r h), and P(r[h). To do so, we use pretrained LLMs available off the shelf from huggingface li- brary (Wolf et al., 2019). To quantify the impact of using one language model over the other, we com- pute the performance of PMI-FAITH using eight LLMs of varying sizes: five BLOOM (Scao et al., 2022) models with up to 7 billion parameters, and three GPT2 (Radford et al., 2019) models up GPT2-large (774 million). We observe robust and consistent performance with a variability of only 0.02 points in the FI score. Hence, for all further experiments, we use BLOOM-560m. Unconditional variant of PMI-Faith: To quantify the impact of dialogue history h PMI-FAITH, we also variant of it called un- conditional PMI between a response and a docu- ment, i.e., UPMI-FAITH = log P(r log P(r). to measure faithfulness.
'Generic' and "Not fully-attributaalee ative class. We use the same evaluation setup as recommended in Honovich et al. (2021). For each metric, we first normalize the score using min-max normalization. The min and the max scores are identified from the dev set. We then identify an optimum threshold for each metric as the one that achieves the best Fl on the dev set. Finally, during test, the identified min, max and thresholds classify response faithful. The thresholds, min and max for each metric are reported in Ap- pendix B. Once we have the predicted class, we then compute precision, recall, score and accu- racy achieved by each metric. As done in Honovich et al. (2021), we also report calibration-free metrics that don't 't require any normalization. In addition to Spearman's 's and Pearson's correlation with human annotations, also report AUROC for various faithfulness metrics.
5.2 PMI-FAITH: Experimental Results
Table reports the precision, recall, Fl score, and accuracy achieved by different metrics on the test split of BEGIN benchmark. We first observe that PMI-FAITH performs better than UPMI-FAITHE clearly demonstrating the advantage of using the
Baselines: We compare our model against Q the state-of-the-art metric. We also compare against various lexical and semantic similarity- based, and trained classifiers as faithfulness metries. Specifically,, we use Uni (U- FI), SacreBLEU(Post, 2018; Papineni et al.,
"hups://gituub.c www/orhonovichlg hhugs://gttubb.mm/bboooocccchhh
*For BERTSccore, we change underlying model to the current best serrrssffudberttt--aaeeeee
'htps://hugiingfcc. hot.:aalc...o//aaaaaaaaaaa Piroherta-large- faitheritic

---

Document
Document
Football has become Brady religion, said Chopra, a filmmaker whose latest iroject Tom Time, behind the scenes documentary series about the quarterback' 's preparations for this past NFL season...
...Home Alone is 1990 American comedy film written produced John Hughes directed Chris Columbus
Conversation:
Conversation:
Perhaps the documentary series Tom vs Time will shed some light Toms remaining time.
What genre it 7
guess it about time, he has been coaching forever and has gotten older and has fewer years left to live. think that is his plan.
it is comedy
Reasoning
Reasoning
Question:
What about?
Question:
What is called?
Response Ans:
time
Response Ans: comedy
Document Ans:
time, behind the scenes...
alone
attlybatt response
Prull--atrriutabbl response
Figure 2: A Fully attributable' (right) and 'Not fully attributable" (left) example, incorrectly classified by Q? and correctly classified by PMI-FAITH.
CMU
TC
WoW
similarity based metrics achieve poor performance, with accuracy worse than even the majority class classifier's ' accuracy of 76.7%.
BERTScore
0.292
0.432
0.612
FaithCritic
0.156
0.039
0.794
Q²
0.543
0.487
0.705
Next, notice that all metrics, except faith- critie, have higher recall than precision, indicating that they tend to be lenient while classifying a re- sponse as faithful, whereas faitheritic tends be conservative and classifies most of the responses as not faithful. Comparing the next two best metrics, we observe that Q² has better Fl and recall but worse accuracy and precision than faithcritic.
UPMI-FAII
0.539
0.542
0.733
PMI FAITH
0.663
0.584
0.771
Table 2: Fl score over each of the three contributing datasets the BEGIN, viz., CMU-DoG (CMU), Topi- calChat (TC), and Wizards of Wikipedia (WoW).
Spearman Cor:
Pearson Cor.
AUROC
To identify dataset specific biases, Table ports Fl score separately for each of the three con- tributing datasets. We observe that PMI-FAITH achieves the highest F1 on CMU-DoG and Top- icalChat with more than 12% and 9.7% absolute gain, respectively, over the other metrics. Faith- Critic achieves the best F1 score on WoW, whereas its Fl TopicalChat and CMU-DoG is quite low. This over-fitting on WoW is because faitheritic is learned metric, and the training data for has been adapted from WoW, and its low performance on the other two datasets demonstrates its lack generalization.
F1-U
0.449
0.447
0,804
BLEU
0.238
0.388
0.663
RougeL
0.412
0.465
0.781
BERTScore
0.433
0.479
0.796
FaithCritic
0.479
0.479
0.712
Q?
0.447
0.448
0.795
UPMI-FAITE FAITH
0,528
0.485
0.861
PMI-FXITH 0.597
0.650
0.907
Table 3: Spearman and Pearson correlation with human annotations in the BEGIN Benchmark, and AUROC of various faithfulness metries.
dialogue history while measuring faithfulness. We then observe that both UPMI-FAITH and PMI-FAITH perform better than all other faith- fulness metrics by considerable margin across all reported performance measures, with the ab- solute gains ranging from 21.8% 8.7% in F1 score. Even against the strong baseline of Q². PMI-FAITH achieves an absolute gain of 5.6% and 8.7% in accuracy and Fl score, respectively. As expected, all the lexical overlap and semantic
To understand the correlation of various faith- fulness metrics with human judgement, report three calibration metrics in table In all three metrics, we observe that PMI-Faith better aligned to human judgements than the other measures faithfulness.
Subjective Analysis: The state-of-the-ard metric, º identifies whether response faithful or not using two steps. the first step, it generates a set

---

of questions based on the response. In the second step, it uses a question answering system gen- erate two responses for each question: one based on the response and one based on the document. If both the answers match, then the response is considered faithful to the document. We now dis- cuss the shorteoming of Q'and how PMI-FAITH overcomes it using two examples.
Decode
Faithfulness
Relevance
Obj.
Method
PMI-F
Q2
BLEU
RougeL
MultiDoe2Dial
Beam Search
Stand. PMI-D
0.59
0.63
30.56
0,488
0.64
0.65
28.95
0.473
Beam
Stand.
0.58
0.62
30.50
0.491
Sampling
PMI-D
0.63
0.66
30.69
0.488
TopicalChat
Figure shows the two examples where PMI-FAITH correctly identifies the faithfulness (or lack of it) whereas the strongest baseline Q² fails to do so. In the case of Fuully-attributall response (right), the pronoun 'ir' in the response is an anaphora, referring back to the antecedent 'home alone' (movie name), which is difficult to infer without the dialogue context. However, Q2 doesn't 't take the dialogue history into account, and thus considers the pronoun it in the response as a cataphor, referring to its postcedent 'comedy'. As result, the QA system correctly answers the generated question "What is it called?' with the postcedent "comedy', when presented with the re- sponse, and correctly outputs its antecedent (home alone) when presented with the document. But the overall Q² system fails, as the two answers do not match. On the other hand, by virtue of considering dialogue history during computation, PMI-FAITH has information that the question is about the genre and not the movie name, and hence it can correctly classify the response 'Fully-attributable.
Beam
Stand. PMI-D
0.49
0.68
6.63
0.219
Search
0.57
0.73
5.65
0.197
Beam
Stand.
0.49
0.67
6.28
0.214
Sampling
PMI-D
0.54
0.72
6.02
0.207
FailhDial
Ream Search
Stand. PMI-D
0.54
0.83
13.53
0.404
0.63
0.87
12.38
0.389
Beam Sampling
Stand.
0.52
0.82
13.30
0.398
PMI-D
0.61
0.87
12.38
0.388
Table 4: Faithfulness and relevance metrics computed for various decoding techniques three datasets.
Doc2Dial (Feng et al., 2021), TopicalChat (Gopalakrishnan et al., 2019) and FaithDial (Dziri et al., 2022b). Each dialog in MultiDoc2Dial (MD2D) is between a user and an agent. Only the agent has access to the documents. So, we only model the agent responses for this dataset. TopicalChat (TC) consists of dialogs between two parties, where each party may have different set of documents on the same topics. We use the 'rare" version of the dataset and filtered utterances tagged as 'personal knowledge", FaithDial (FD), a faithful adaptation of WoW(Dinan al., 2019), in which one participant can ask a wide range of questions and the other participant can only provide informa- tion from Wikipedia. Some statistics of the three datasets are in Table 6.
The other example highlights two issues: (1) when the response is partially hallucinated, ques- tion generation system may generate question from just the faithful part of the response and may in- correctly declare the whole response faithful. In this example, most of the response contains an opinion, which is not faithful to the document, but the QG system focused on 'I guess it is about time'. (2): the other issue that the NI.I system fails to capture that the single word answer 'time' from the response is not entailed by the long answer from the document, resulting in the incorrect pre- diction by the overall system. On the other hand, PMI-FAITH considers the response as a whole, instead of separately focusing on parts of it. As a result, it is correctly able to identify the given response as not faithful.
A Algorithms: For each of the three datasets, we separately finetune a BART-Large (Lewis et al., 2019)$ model using the code made available by Dziri et al. (2022a). As baselines, we use two de- coding techniques that use the standard likelihood as the objective function: (1) beam search and (2) beam sampling. Both the techniques use a beam size of 4. We compare these baselines with their variants that uses our PMI-DECODE (PMI-D) ob- jective. We use the values of a and top-p masking that achieved the highest sum of RougeL and nor- malized PMI-FAITH on the dev set.
5.3 PMI-DECODE: Experimental Setup
Datasets: We perform our experiments on three document-grounded dialog datasets: Multi-
htt httts: httggg////huginaaae..... co/ co//aacebooo/bart-larrg hhttps://githhe thub. www/Mcti http///ccll-NLP/Faiite

---

PMIF
RougeL
Grammatical errors (in %
0.25
0.5
0.25
0.5
0
0.25
0.5
0.6
0.52
0.58
0.59
0.59
0.40
0.40
0.39
0.38
6.9
10.2
11.8
12.2
0.75
0.52
0.60
0.61
0.60
0.40
0.39
0.38
0.36
6.4
12.0
16.3
18.6
0.9
0.52
0.61
0.62
0.57
0.40
0.39
0.36
0.30
7.5
15.0
24.2
45.8
1
0.52
0.61
0.56
0.34
0.40
0.38
0.30
0.05
7.4
16.5
54.8
Table 5: Faithfulness, relevance, and grammatical errors of the responses generated by beam sampling using different configurations of and top p masking on the FaithDial dataset
Num. samples
Avg. words
decreases. Simultaneously, the relevance decreases
with an increase in a. The highest level of faith- fulness is observed when a = 0.5 and p = 0.6. Meanwhile, the highest relevance is achieved when the CPMI is not used in the decoding objective (a 0), or when both and have low values.
Train
Dev.
Test
Doc.
Hist.
Resp.
MD2D
24,603
4,699
4,567
166
93
18
TC
131,555
8,183
8,301
241
199
20
FD
18,357
3,417
3,539
23
69
18
Various statistics of the three document grounded dialog datasets.
Table 6:
We have gained two valuable insights from ta- ble 5. The first insight reveals that there are spe- cific configurations, such as p 0.6 and a 0.25, which achieve the same level of relevance (0.40) as the standard variant while generating more faithful responses. The second insight highlights a signif- icant drop in relevance when solely focusing on PMI scores (a 1 and p 1). Therefore, if the goal is to generate responses with nearly equivalent relevance, utilising smaller values for a along with masking value of 0.6 recommended.
Multi-Doc2Dial
Topical Chat
Fai
Rel
Gra
Fai
Rel
Gra
Standard
0.52
0.72
0.96
0.69
0.70
0,96
PMI-D
0.60
0.75
0.92
0.80
0.67
0.93
Table 7: Human evaluation of responses generated using beam search with different decoding objectives. We evaluate faithfulness (Fai), relevance (Rel) and grammar (Gra).
To demonstrate the impact of top-p masking on the grammar. we also report the percentage of gram- matically incorrect responses for different values of and top-p masking in table 5. We use GECToR (Omelianchuk et al., 2020). grammatical error correction method, to find if generated response is grammatically correct or not. We can easily see that for any value of > 0, the grammatical errors reduce significantly with a reduction top p. We do not report any value for a = and top = as the responses with this configuration are not even in proper English. For example, one of the responses generated with = 1 and top p is "Cheyne Lauren sisters Vel Lauren wear Ralph indo Austrian Ralph linesauxricting Ren therapies Combat Rarity glamorous".
5.4 PMI-DECODE: Experimental Results
We compare the decoding strategies in Table using various automated metries for faithfulness (PMI-FAITH, Q²) and relevance (BLEU, RougeL). The general trend is that the PMI-DECODE gener- ates more faithful responses compared the stan- dard variant and the improvement in faithfulness comes a the cost of relevance.
To gain a better understanding of the correlation between faithfulness and relevance, we conducted experiments involving different configurations of the control parameters of PMI-D (a and top-p masking). The results of these experiments are pre- sented in the table 5, showeasing the corresponding normalized PMIF and RougeL metrics for each configuration.
Human Evaluation: We perform human evalu- ation experiments to compare (1) relevance, (2) faithfuulness, and (3) grammar. All three dimen- sions were categorically labeled agree, neu- tral, disagree. We sampled 100 random (doc- ument, dialog history, response) tuples, 50 each from MD2D and TC. We evaluate the responses generated by beam search using two objectives:
For given fixed value of a, as the p value in- creases, the faithfulness of the responses also in- creases. However, this improvement in faithfulness comes at the cost of decreased relevance. On the other hand, for fixed value, as a increases, the Faithfulness initially increases and then gradually

---

Document
What happens had lapse of insurance coverage during active duty? You are required maintain continuous coverage or surrender your plates before deployed. you were unable do so DMV has a procedure in place exempt you from administrative actions upon your return. You must provide notarized statement that the vehicle was not used during the time in question, copy your military ID, proof current insurance or surrender of plates, and signed and dated copies of deployment and return papers, or DD-214 if separated from service...
Context
User: lost my insurance while on active duty and have some questions
You are required to maintain continuous coverage or surrender your plates before being deployed. Were you unable to keep your insurance while active duty?
Agent:
User: yes, it just wasn't on my mind don't get notices and anything like that while deployed
Greedy
Agent: Unfortunately, no relevant information is found.
PMI:
DMV has procedure in place to exempt you from admunistrative actions upon your return. You must provide notarized statement that the vehicle was not used during the time in question, a copy of your military ID proof of current insurance or surrender of plates, and signed and dated copies of deployment and return papers, or DD-214 if separated from service.
Agent:
Table An example from the test set of Hulti-Doc2Dial dataset where Greedy generates 'I don't know' response and PMI-DECODE generates relevant and faithful response.
standard and PMI-DECODE. Out of six in house annotators used (3 per dataset), four were experts in dialog research and two were beginners. Refer to appendix A for more details.
Subjective Analysis: Table 8 presents an example from MD2D where standard likelihood based beam search decoding returns a generic response ('no info. found") which is present in around 1800 train- ing samples. The same model returns the correct response when PMI-DECODE objective is used instead of just likelihood, demonstrating the capa- bility of PMI-DECODE to shift the score in favour of the words present in the document.
The results are summarized in Table 7. For each dimension, we report the percentage of responses that were rated agree. As expected, PMI-DECODE generates more faithful responses compared to greedy, We observe a 15% improvement in faithful- ness compared to greedy decode on both datasets. Further, PMI-DECODE improves relevance on MD2D but slightly deteriorates TC. Manual analysis revealed that the improvement of relevance on MD2D is primarily due inherent solution mul- tiplicity (Nandwani et al., 2021) in most dialogues, where more than one correct response is possible, but the metrics capture just one.
6 Conclusion
In this paper, we present novel metric, PMI-FAITH, to measure faithfulness of responses generated by document grounded dialog systems. It uses conditional PMI between the response and the document given the dialog history to quantify faith- fulness. We extend the idea of PMI-FAITHtO pro- pose a novel decoding objective, PMI-DECODE which encourages responses to be faithful to the given document by maximizing both the likelihood and faithfulness of the decoded response. Our ex- periments on the BEGIN benchmark prove that our proposed metric better correlates with human judgments compared existing metrics. On three (bocument-groundee dialog datasets, our novel de- coding objective generates more faithful responses than the standard likelihood objective, as measured using automated metrics and a human study.
As PMI-DECODE maximises not just the like- lihood of responses, but a combination of likeli- hood and faithfulness, we expected the responses to contain grammatical errors compared to greedy decode. To counter this issue, we proposed to use a weighted combination of likelihood and faithful- ness during decode, with higher weight on like- lihood. We also restricted the vocabulary during each decode step to just the top-p subset. The hu- man study shows that these mitigation techniques helped in reducing the grammatical mistakes made by PMI-DECODE. We see that the grammar is only slightly inferior to greedy on both the datasets.
Limitations
We use Fleiss Kappa (Fleiss and Cohen, 1973) to measure the inter-annotator agreement, which is substantial for relevance (0.63) and faithfulness (0.63), and almost perfect (0.88) for grammar.
Though our decoding objective generates more faithful responses, we observed its inability re- spond to generic chit-chat or pleasantrics, like 'Hello!' or 'Good-bye'. It is possible to combine it

---

with other techniques, like training with CTRL to- kens (Rashkin al., 2021b), which can enable it to generate both generic as well as faithful responses depending upon the dialogue context. But identify- ing when to generate a particular kind of response may require more insights and we leave this overall thread for future work. Next, to compute CPMI, we need to pass d, h, and h separately to the decoder. Though it be done in parallel, but it may still reduce the throughput of the overall system by half. Finally, as demonstrated by the human evaluation, PMI-DECODE at times generates grammatically incorrect responses, even though the pre-trained language models are very good at generating flu- ent and coherent English. While we presented two knobs: a and lop p masking to overcome this, we believe there could be other ways of handling this.
Wang. 2020. Logic2text: High-fidelig natural lan- guage generation from logical forms. In Findings of the Association for Computational Linguisties: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 2111. Association for Computational Linguis- tics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge- agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net
conversational
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Os- mar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022a. FaithDial: A Faithful Benchmark for Information Seeking Dialogue. Transactions of the Association for Computational Linguistics, 10:1473- 1490.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022b. Evaluating attribution in dialogue sys- tems: The BEGIN benchmark. Trans. Assoc. Com- put. Linguistics, 10:1066-1083.
Ethics Statement
Our work does not introduce any new ethical con- cerns per se, other than the ones already faced by large language models. Our decoding objective works on top of any trained language model and generates the text which is more faithful to a given input document. This can act as a double-edged sword: one hand, if the document itself contains profanity, it may enhance the model's likelihood of generating similar content. But on the other hand, providing a valid document may also reduce the in- herent likelihood of the model to generate profane content. Therefore, we recommend using it with responsibility and caution.
Song Feng. Siva Sankalp Patel, Hui Wan, and Sachin- dra Joshi. 2021. Multidoc2dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- rual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6162-6176. Association for Computational Linguistics
Joseph L. Fleiss and Jacob Cohen. 1973. The equiva- lence of weighted kappa and the intraclass correlation coefficient as measures of reliability, Educational and Psychological Measarement, 33:613 619.
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the f'actual accuracy of gener- ated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Diseovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 166-175, ACM.
References
Meng Cao. Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstrac- tive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 6251-6258. Association for Computational Linguistics.
Karthik Gopalakrishnan. Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Türr 2019. Topical-Chat: Towards Knowledge-Grruunde Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Second- AAAI Conference Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pages 4784-4791. AAAI Press.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom. Idan Szpektor, Avinatan Hassidim. and Yossi Matias. 2022. TRUE: re-evaluating factual
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang

---

consistency evaluation. In Proceedings of the Sec- ond DialDoc Workshop on Document -rruunndd Dia- logue and Conversational Question Answering. Di- alDoc ACL 2022, Dublin, Ireland, May 26, 2022, pages 161-175. Association for Computational Lin- guistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 5-1-118, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Sq"2S: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November. 2021, pages 7856-7870. Association for Computational Linguis- tics.
Ashwin Paranjape and Christopher Manning. 2021. Human- Mokman-like informative conversations: Better ac- knowledgements using conditional mutual informa- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 768-781, Online. Association for Com- putational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332 9346, Online. Association for Computa- tional Linguistics.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W. Black, and Ruslan Salakhutdi- nov. 2021. Focused attention improves document- grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIIT 2021, Online, June 6-11, 11, 2021, pages 4274-4287 Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. CoRR, abs/1910.13461.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Eiwersit-prromotie ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text- to-txt transformer. Journal of Machine Learning Research, 21 21(140):1-67.
Chin Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das. Slav Petrov, Gau- rav Singh Tomar, Tulia Turc, and David Reitter. 202 Measuring attribution in natural language generation models. CORR, abs/2112.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. On faithfulness and factuality in abstractive summarization. ArXiv, abs/2005.00661.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021 Increasing faithfulness dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACL/JCNNP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 704-718. Association for Computational Linguistics.
Yatin Nandwani, Deepanshu Jindal, Mausam, and Parag Singla. 2021. Neural learning of -fay solu- tions for combinatorial problems in structured output spaces. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenRevie..net.
in mnowledge-groundde
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR grammatical error correction: Tag, not rewrite. In Proceedings of the Fifieenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-170, Seattle, WA, USA - Online. Association for Computational Linguistics.
Liliang Ren, Mankeerat Sidhu, Qi Zeng, Revanth Gangi Reddy, Heng Ji, and ChengXiang Zhai. 2023. C-PMI: Conditional pointwise mutual information for turn-level dialogue evaluation. In Proceedings of the Third DialDoc Workshop on Document- Document-grundeed

---

Dialogue and Conversational Question Answering. pages 80-85, Toronto, Canada. Association for Com- putational Linguisties.
solid foundation in Machine Learning (ML), and Natural Language Processing (NLP). Out of six in- house annotators used, four were experts dialog research and two were beginners. Each annota- tor had completed at least one formal course in ML/NLP. The exact qualifications and experience of the six annotators are given below:
Sashank Santhanam, Behnam Hedayatmia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study factual correctness knwwledge-grouuee abs/21 10.05456.
response generation. CoRR.
Annotator- is postgraduate degree holder with more than decades of experience NLP research.
Teven Le Scao, Angela Fan, Christopher Akiki, and al. 2022. Bloom: A 176b-parameter open-access multilingual language model.
Annotator-2 2 and annotator-3 are PhD degree holders with more than years of experience in AL.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the-ane art natural language processing. CoRR, abs/1910.03771.
Annotator-4 is an undergraduate degree holder with more than 5 years of experience in NLP research.
Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel-Kedzionsk Jianfeng Gao, Hannaneh Hajishirzi, Mari Ostendorf, and Bill Dolan. 2021. A controllable model of grounded response generation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thil Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 14085-14093, AAAI Press.
Annotator-5 and annotator-6 6 are undergradu- ate degree holders with about 2 years of expe- rience in Al research.
All the annotators provided their consent over an appropriate official communication channel, e.g., official email or slack channels.
The following were the instructions provided to the human evaluators.
Yijun Xiao and William Yang Wang. 2021. On hal- lucination and predictive uncertainty in conditional language generation. Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 2734-2744. Association for Computational Linguistics.
What is the task? There are 50 incomplete dialogs along with a document over which the dialog is grounded on. For each (document, incomplete di- alog) pair we provide the next response predicted by different dialog systems (shuffled in random order). You are requested to judge the response generated by these 2 systems along three dimen- sions: faithfulness, relevance and grammar. Each dimension has to be evaluated using the following scale: Agree (A), Neutral (N), and Disagree (D). How to judge relevance? Relevance measures how apt is the response given the dialog context and the knowledge. Please select agree when the re- sponse apt and does not convey any incorrect in- formation. Select neutral when is hard to decide whether it is right or wrong and disagree otherwise. How to judge faithfulness? The faithfulness of a response is only dependent on the grounding docu- ment and it is independent of the dialog. A system response can be marked disagree for relevance and still be marked agree for faithfulness. Please select agree when the complete response can be inferred from the document. Select neutral when it is hard to decide whether it can be inferred from the docu- ment or not and disagree when a major portion of
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W. Black. 2018. A dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, Brussels, Belgium, October 31 November 4, 2018, pages 708-713. Association for Computational Linguistics.
A Human Evaluation
The screenshot of sample task is shown in Figure
3.
For our human evaluation study, we ensured the quality and expertise of our annotators by selecting individuals who are fluent in English and have a

---

Conversation-1
Document
Withdraw Your Application
Sometimes life changes occur after you submit your application. You have up 12 months to withdrew your app ication, if you chenge your mind. You will be required to repay any benefits you ve already reselved. Learn more about Withdrawing Your Social Security Retirement Application
Dialog
User: What can't access my status online?
Agent: You can always call us 1-800-72 090-772-10 2 TTY 1-800 325 077- from 8.00 a.m. to 5.30 p.m, Monday through Friday.
User: Noted, Can estimate my retirement benefite somehow?
Agent: Yes, you can estimate your retirement benefite.
User: is also possible to withdraw my apolication?
Faithfulness
Relevance
Grammar
Agent:Yes, life chenges occur efter you submit your application, You have up to 12 months to withdraw your application if you change your mind!
A OO OD
OA ON
OA
Yeen soretlimes life changes occur after you submit your applization. You have up to 12 manths to withdraw your application, if you change your mind"
O O NONN O
O ON OD
OA
Figure 3: An example of the task provided to the human judges.
the response cannot be inferred from the document. For the case where the response something like "No information is present". The judgement should be agree if there is no information about that document provided and "disagree" there infor- mation available in the document, but the system didn't 't pick it up. For cases where the user initiates chit-chat (say the user says "hi, how are you"), the agent responds with chit-chat ("I am doing good"), please mark faithfulness as neutral.
report the threshold used on the normalized metrics for the faithfulness classification task. These are the thresholds than achieved the hightest Fl on the dev set.
Min
Max
Threshold
U-FI BLEU RougeL
0
0.070
0
0.039
0.202
BERTScore
-0.639
0..40)
FaithCritic
0
Q²
0
0.625
How to judge grammar? The grammar score for response is independent of the dialog or the docu- ment. A system response can be marked disagree for relevance and still be marked agree for gram- mar. Please select agree when the response looks like how an expert human writes. Select neutral when there a major issue with how the response reads but still understandable and disagree when the response makes no sense.
UPMI FAITH
-8.092
4,286
0.759
OMI-FAITH
-2.069
4.334
0..34
Table 9: The min and max values used for normalizing each metric and the threshold used for the classifying faithfulness of response using the metric.
B Faithfulness Metric Normalization
The minimum and maximium values used for normalizing various metrics are shown in Table 9. These are the minimum and maximum values achieved by each metric on the dev set. We also


Reviews:
[{"review_id": "NdMBiaRnAC", "reviewer": "Reviewer_AFXM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer AFXM,\n\nThank you for a positive review. We are glad that you found our detailed comparison with s-o-t-a useful.  Below we attempt to address your concern regarding the lack of human evaluation of PMI-Faith. In case we have misunderstood your concern, please let us know and we will be happy to clarify.\n\n**Reason-to-Reject-1:** *There is no report on human evaluation comparing PMI-FAITH to other metrics. It seems that additional proof is required to substantiate the effectiveness of this metric and the author's argument that previous automated metrics are far from being well aligned with human judgments.*\n\nWe are indeed evaluating PMI-FAITH and other metrics using human annotations. For this, we use BEGIN benchmark (Dziri et. al., 2022). Each sample in this benchmark has a model-generated response that has been annotated for faithfulness by *humans*. As our goal is to design a metric that mimics human behaviour, we measure how correlated are the existing and proposed metrics to the human judgement. In the paper (Table 1), we show that PMI-Faith is the best metric as it matches with human judgement 83.4% times (accuracy). \n\nWe have also computed Spearman correlation, Pearson correlation and area under ROC curve (AUROC) for various metrics. This shows that PMI-Faith has the highest correlation with human judgements. \n\n| **Metric**        | **F1-U** | **BLEU** | **RougeL** | **BERTScore** | **FaithCritic** | **Q2** | **PMI-Faith** |\n|-------------------|:--------:|:--------:|:----------:|:-------------:|:---------------:|:------:|:---------:|\n| **Spearman Cor.** |   0.449  |   0.238  |    0.412   |     0.433     |      0.479      |  0.447 |   **0.597**   |\n| **Pearson Cor.**  |   0.447  |   0.388  |    0.465   |     0.479     |      0.479      |  0.448 |  **0.650**   |\n| **AUROC**         |   0.804  |   0.663  |    0.781   |     0.796     |      0.712      |  0.795 |   **0.907**   |\n\nWe will add this table to our revised manuscript", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors not only demonstrated improved performance of the proposed method, but also provided detailed analyses in comparison with state-of-the-art methods. It appears that the proposed metric and the decoding strategy based on this metric are effective for evaluating factuality in dialogue. PMI-FAITH can also be easily computed using only a pre-trained model, which implies the wide acceptability of this method. Finally, it was proven that PMI-D has superior faithfulness according to human evaluation.", "reasons_to_reject": "There is no report on human evaluation comparing PMI-FAITH to other metrics. It seems that additional proof is required to substantiate the effectiveness of this metric and the author's argument that previous automated metrics are far from being well aligned with human judgments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors proposed Point-wise Mutual Information (PMI) between the generated response and the source document to measure the faithfulness of response with respect to the grounding document considering the dialogue history.\nIn addition, they proposed a new decoding strategy based on the PMI to generate more faithful responses.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "92f5jseVSS", "reviewer": "Reviewer_RmAJ", "summary": "", "questions": "A. Please indicate the V+ in line 180 (Section 3). It's quite confusing while reading without a clear explanation. \nB. Is there any clear reason why you compare the evaluation metrics with the classification task by deciding the threshold? Is there any benchmark in comparing evaluation metrics? Why didn't you normalize the score directly from the raw evaluation results? \nC. I understand that the Q^2 metric is SOTA, but I think it would be better to show the examples which got incorrect with Faithcritic but correct with your metric as well in subjective analysis. \nD. The p-masking was proposed to mitigate grammar issues, but the experimental results in Table 5 only show the relevance and faithfulness differences. I would recommend adding grammar-related metrics to show its improvements. \nE. Who were the human evaluators? graduate students? undergraduate? fluent in English? I need more details. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RmAJ, \n\nThank you for a detailed review of our manuscript and for pointing out missing references and typos.  We will include the missing references in the camera ready version.  We also thank you for suggesting additional evaluations, specially automated evaluation of grammar, that will further strengthen our paper. We address your concerns and answer your questions below:\n\n**Reason-to-Reject-1:** *I'm not sure the evaluation setting is validated. Is there any clear reason why you compare the evaluation metrics with the classification task by deciding the threshold? Is there any benchmark in comparing evaluation metrics? Why didn't you normalize the score directly from the raw evaluation results?*\n\nThank you for raising this concern. As suggested, we have added a few more standard ways for comparing the evaluation metrics below.\n\nOur evaluation setting of using classification task for comparing faithfulness metrics is same as the “Response-Level Evaluation” proposed by Q2 (Honovich et. al., 2021). We used a threshold as Honovich et. al. suggested the following in Section 5.1 of their paper - “*We note that the threshold was arbitrarily selected for the purpose of demonstrating Q2’s ability in separating consistent from inconsistent content, and properly tuning it by splitting the data into development and test sets may improve the results further.*” So, we selected an optimal threshold for all the metrics using the small dev set provided in the BEGIN benchmark. \n\nWe use BEGIN bechmark which is a standard dataset for comparing various faithfulness metrics.  Each sample in the BEGIN benchmark has a model-generated response that has been annotated for faithfulness by *humans*. As our goal is to design a metric that mimics human behaviour, we measure how close are the existing and proposed metric to human judgement of faithfulness. In the same way as Q2 (Table 5 in Honovich et. al., 2021), we report three calibration-free metrics to quantify the alignment of various metrics with human judgements. In all 3 metrics, we see that PMI-Faith is better aligned to human judgements than the existing metrics.\n\n| **Metric**        | **F1-U** | **BLEU** | **RougeL** | **BERTScore** | **FaithCritic** | **Q2** | **PMI-Faith** |\n|-------------------|:--------:|:--------:|:----------:|:-------------:|:---------------:|:------:|:---------:|\n| **Spearman Cor.** |   0.449  |   0.238  |    0.412   |     0.433     |      0.479      |  0.447 |   0.597   |\n| **Pearson Cor.**  |   0.447  |   0.388  |    0.465   |     0.479     |      0.479      |  0.448 |   0.650   |\n| **AUROC**         |   0.804  |   0.663  |    0.781   |     0.796     |      0.712      |  0.795 |   0.907   |\n\n**Reason-to-Reject-2:**  *Human evaluation needs more details to validate the PMI-D's performance. WoW dataset is removed from the human evaluation while WoW is included in automatic evaluation.*\n\nFor our human evaluation study, we ensured the quality and expertise of our annotators by selecting individuals who are fluent in English and have a solid foundation in Machine Learning (ML), and Natural Language Processing (NLP). Each annotator had completed at least one formal course in ML/NLP. The exact qualifications and experience of the six annotators are given below:\n- annotator-1 is a postgraduate degree holder with more than 2 decades of experience in NLP research.\n- annotator-2 and annotator-3 are PhD degree holders with more than 5 years of experience in AI.\n- annotator-4 is an undergraduate degree holder with more than 5 years of experience in NLP research.\n- annotator-5 and annotator-6 are undergraduate degree holders with about 2 years of experience in AI research.\n\nShould the reviewer require more specific details, we are more than willing to furnish them with the necessary information.\n\nAs we used our in-house conversational AI experts for the experiments, we were only able to get it annotated for 2 datasets due to resource constraints and hence we randomly chose two datasets for this experiment. \n\n**Reason-to-Reject-3:**  *I think the related works that PMI is adapted in a conversational setting to indicate the differences.*\n\nWe thank the reviewer for pointing us to missing related and contemporary works. We will add the missing references in the updated manuscript as given below.\n\nMutual information has been previously used in conversational settings. Mutual information has been used to prevent the conversational models from generating generic responses (such as “Sorry, I’m not sure about this topic”) [1,2]. Contemporary to our work, Liliang et al (2023) propose to use a Conditional Pointwise Mutual Information (C-PMI) based metric to evaluate **relevance** of a generated response with respect to a reference hypothesis for open-domain response generation. In contrast, our work is the first to use C-PMI as a metric for evaluating **faithfulness** of a response given a document and dialogue history in a document-grounded response generation. \n\n[1] Ren, Liliang, et al. \"C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation.\" Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering. 2023.\n\n[2] Li, Jiwei, et al. \"A Diversity-Promoting Objective Function for Neural Conversation Models.\" Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.\n\n[3] Paranjape, Ashwin, and Christopher D. Manning. \"Human-like informative conversations: Better acknowledgements using conditional mutual information.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n**Question-A:** *Please indicate the V+ in line 180 (Section 3). It's quite confusing while reading without a clear explanation.*\n\nWe apologize for the confusion caused. $V^+$ refers to the space of all the sentences having one or more token from vocabulary $V$. We will clarify this in the updated manuscript. \n\n**Question-B:** *Is there any clear reason why you compare the evaluation metrics with the classification task by deciding the threshold? Is there any benchmark in comparing evaluation metrics? Why didn't you normalize the score directly from the raw evaluation results?*\n\nPlease refer to Reason-to-Reject-1\n\n**Question-C:** *I understand that the Q^2 metric is SOTA, but I think it would be better to show the examples which got incorrect with Faithcritic but correct with your metric as well in subjective analysis.*\n\nSome examples where Faithcritic is  incorrect and PMI-Faith is correct.\n\n*Example-1*\n\n- Document: ...Home Alone is a 1990 American comedy film written and produced by John Hughes and directed by Chris Columbus ...\n\n- Dialog History: User: What genre is it ? :)\n\n- Response: it is a comedy\n\n- Gold Label: Faithful\n\n*Example-2*\n\n- Document:  …American football, referred to as football in the United States and Canada, and also known as gridiron football or simply gridiron, is a team sport played by two teams of eleven players on a rectangular field with goalposts at each end….\n\n- Dialog History: User: How many players are on the field at once?\n\n- Response: 12 players .\n\n- Gold Label: Not faithful\n\n**Question-D:** *The p-masking was proposed to mitigate grammar issues, but the experimental results in Table 5 only show the relevance and faithfulness differences. I would recommend adding grammar-related metrics to show its improvements.*\n\nThank you for suggesting this evaluation. Following your comment, we used GECToR [1], a grammatical error correction method, to find if the response is grammatically correct or not. The 2D grid below contains the percentage of grammatically incorrect responses for different values of $\\alpha$ and $top-p$ mask. We can easily see that for any value of $\\alpha > 0$, the grammatical errors reduce significantly with reduction in $top-p$. \n\n\n| **$\\alpha$** |     **0**    | **0.25** | **0.5** | **1** |\n|-------------|--------------|----------|---------|-------|\n|   **top-p**   |         |      |     |   |\n|   **0.6**   |     6.9%     |   10.2%  |  11.8%  | 12.2% |\n|   **0.75**  |     6.4%     |   12.0%  |  16.3%  | 18.6% |\n|   **0.9**   |     7.5%     |   15.0%  |  24.2%  | 45.8% |\n|    **1**    |     7.4%     |   16.5%  |  54.8%  |   -   | \n\nWe do not report any value for $\\alpha = 1$ and $top-p = 1$ as the responses there are not even in proper English.  An example response from the $\\alpha = 1$ and $top-p = 1$ setup is shown below:\n\n“Cheyne Lauren sisters Vel Lauren wear Ralph indo Austrian Ralph linesauxricting Ren therapies Combat Rarity glamorous”\n\n[1] Omelianchuk, Kostiantyn, et al. \"GECToR–Grammatical Error Correction: Tag, Not Rewrite.\" Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. 2020.\n\n**Question-E:** *Who were the human evaluators? graduate students? undergraduate? fluent in English? I need more details.*\n\nPlease refer to Reason-to-Reject-2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Their suggestion involves a measurement which makes use of Conditional Point-wise Mutual Information (C-PMI) between the response produced and the source document. This measurement is influenced by the ongoing conversation and aims to assess faithfulness.\n\n2) They develop a decoding method that integrates PMI into the process of generating responses. This integration enhances the ability to anticipate and generate responses that are more faithful to the source document.", "reasons_to_reject": "A. I'm not sure the evaluation setting is validated. (Please refer to Question B). Why did you evaluate generation metrics with a classification setting? Please give us some benchmark settings. \nB. Human evaluation needs more details to validate the PMI-D's performance. WoW dataset is removed from the human evaluation while WoW is included in automatic evaluation. \nC. I think the related works that PMI is adapted in a conversational setting to indicate the differences. ", "questions_to_authors": "A. Please indicate the V+ in line 180 (Section 3). It's quite confusing while reading without a clear explanation. \nB. Is there any clear reason why you compare the evaluation metrics with the classification task by deciding the threshold? Is there any benchmark in comparing evaluation metrics? Why didn't you normalize the score directly from the raw evaluation results? \nC. I understand that the Q^2 metric is SOTA, but I think it would be better to show the examples which got incorrect with Faithcritic but correct with your metric as well in subjective analysis. \nD. The p-masking was proposed to mitigate grammar issues, but the experimental results in Table 5 only show the relevance and faithfulness differences. I would recommend adding grammar-related metrics to show its improvements. \nE. Who were the human evaluators? graduate students? undergraduate? fluent in English? I need more details. ", "paper_topic_and_main_contributions": "1) They propose a metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue in respect of faithfulness. \n2) They build a decoding technique that incorporates PMI into the response generation process to predict more faithful responses.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line # 58, please apply textit style\n2. line # 338, dev -> please change into development set. or dev. as an abbreviation. "}, "extra_text": {"Missing_References": "[1] C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation\n- I know it is published after the submission. But I think you should include the paper as well in the review process just in case for the acceptance.\n\n[2] Human-like informative conversations: Better acknowledgements using conditional mutual information\n\n"}}, {"review_id": "rvZwaYcQkj", "reviewer": "Reviewer_FY11", "summary": "", "questions": "- I am not sure if \"dialogue history\" is the real matter as you have noted in the Introduction. According to the proposed methodology, the information disparity derived by the \"document\" seems to be the key aspect in guaging faithfulness. Have you done any additional analysis on the importance of considering dialogue history? Although dialogue history is considered in the proposed CPMI, it seems that CPMI can be implemented without dialogue history, and it seems that this can also produce quite meaningful results.\n\n- In equation 2, were P(r|d, h) and P(r|h) estimated by different models? I find that this paper utilized fine-tuned model for such probabilities, but I wonder that whether you intended to use respective model or not. If different, P(r|d, h) and P(r|h) would be better to be noted like P(r|d, h, \\theta) and P(r|h, \\phi).\n\n- In equation 2, I find that p(r|h) can be vanished to 0 and may yield CPMI explosion. Is there any cases in practical use concerning it?\n\n- In equation 5, it seems that r_t = arg max_{v∈V} log P(v|r1:t−1,d, h) in the first stage (paper described r_t = arg max_{v∈V} log P(r1:t−1, v|d, h)). The generation probability of the t^th token is grounded at \"1:t-1\"^th tokens, and it seems that the generation probability of \"1:t-1\" is not considered together. If wrong, please let me know. \n\n- In implementing PMI–DECODE based inference, it seems that lagging will inevitably occur. How much is the speed difference from normal decoding?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer FY11, \n\nWe thank you for a careful review of our work and for asking pertinent questions. We appreciate that you found our problem formulation clear and liked the clear distinction between faithfulness and hallucination in the related works.  We are glad that you didn’t find any special reason to reject our work. Below we try our best to answer your questions. \n\n**Question-A:** *I am not sure if \"dialogue history\" is the real matter as you have noted in the Introduction. According to the proposed methodology, the information disparity derived by the \"document\" seems to be the key aspect in gauging faithfulness. Have you done any additional analysis on the importance of considering dialogue history? Although dialogue history is considered in the proposed CPMI, it seems that CPMI can be implemented without dialogue history, and it seems that this can also produce quite meaningful results.*\n\nYou are absolutely right in pointing out that PMI-Faith aims to capture the information disparity derived by the document.  We note that none of the existing metrics exploit dialogue history to quantify faithfulness and hence the emphasis in the introduction.\n\nWe condition on dialogue history while computing the conditional probabilities as it is crucial for a good performance. We thank you for suggesting an alternative PMI based metric that doesn’t condition on dialogue history to demonstrate this aspect. As suggested, we use PMI, defined as  $log (P(r | d) / P(r) )$, as an additional metric for quantifying faithfulness and call it Unconditional-PMI-Faith (U-PMI-F). We observe that it indeed performs slightly inferior to PMIFaith (in terms of F1) and better than all other metrics. The tables below are a replica of Table 1 and Table 2 with an additional row for U-PMI-F:\n\nTable 1: Performance of various faithfulness metrics on the BEGIN Benchmark\n\n|    **Metric**   | **Precision** | **Recall** |    **F1** | **Accuracy** |\n|:---------------:|--------------:|-----------:|----------:|-------------:|\n| **FaithCritic** | **68.4%**     | 49.2%      | 57.3%     | 82.9%        |\n|      **Q2**     | 51.7%         | 74.4%      | 61.0%     | 77.9%        |\n| **U-PMI-F** | 59.2%         | 70.4%      | 64.3%     | 81.8%        |\n|    **PMI-F**    | 60.7%         | **81.8%**  | **69.7%** | **83.4%**    |\n\nTable 2: F1 score over each of the three contributing datasets in the BEGIN, viz., CMU–DoG (CMU), TopicalChat (TC), and Wizards of Wikipedia (WoW).\n\n| **Metric**      | **CMU**   | **TC**    | **WoW**   |\n|-----------------|-----------|-----------|-----------|\n| **FaithCritic** | 15.6%     | 3.9%      | **79.4%** |\n| **Q2**          | 54.3%     | 48.7%     | 70.5%     |\n| **U-PMI-F** | 53.9%     | 54.2%     | 73.3%     |\n| **PMI-F**       | **66.3%** | **58.4%** | 77.1%     |\n\n**Question-B:** *In equation 2, were P(r|d, h) and P(r|h) estimated by different models? I find that this paper utilized fine-tuned model for such probabilities, but I wonder that whether you intended to use respective model or not. If different, P(r|d, h) and P(r|h) would be better to be noted like P(r|d, h, \\theta) and P(r|h, \\phi).*\n\nBoth $P(r | d, h)$ and $P(r | h)$ were estimated using the same model and hence we dropped dependence on model parameters for conciseness. \n\nIn the experiments, we used finetuned models only for generating responses in section 5.3 (PMI–DECODE: Experimental Setup), but not for computing PMI-Faith. \n\n**Question-C:** *In equation 2, I find that p(r|h) can be vanished to 0 and may yield CPMI explosion. Is there any cases in practical use concerning it?*\n\nTheoretically, it may explode. But practically, we never observed this when computing PMI-Faith. Since we are evaluating responses generated by a language model, and computing the probabilities also using a language model, albeit a different one, we do not observe $P( r | h)$ close to 0 for the generated responses.\n\nOn the other hand, in PMI-Decode, a very low value of $P(v | h, r_{1:t-1})$ may result in an arbitrary token getting selected and this is one of the issues that our masking strategy elegantly prevents from happening. \n\n**Question-D:** *In equation 5, it seems that r_t = arg max_{v∈V} log P(v|r1:t−1,d, h) in the first stage (paper described r_t = arg max_{v∈V} log P(r1:t−1, v|d, h)). The generation probability of the t^th token is grounded at \"1:t-1\"^th tokens, and it seems that the generation probability of \"1:t-1\" is not considered together. If wrong, please let me know.*\n\nYour understanding is absolutely correct and both the equations are equivalent.  Consider:\n\n $r_t = arg max_{v∈V} log P(r_{1:t−1}, v|d, h))$\n\nNow applying Bayes rule, we can represent $P(r_{1:t-1} , v | d, h )$ in the RHS as $P(r_{1:t-1} | d, h) * P(v | r_{1:t-1}, d, h)$. We get: \n\n$r_t = arg max_{v∈V} ( log P(r_{1:t-1} | d, h)  + log P(v | r_{1:t-1}, d, h) )$\n\nSince $P(r_{1:t-1} | d, h)$ is independent of $v$, it can be dropped from RHS, resulting in exactly what you described:\n\n$r_t = arg max_{v∈V} (log P(v | r_{1:t-1}, d, h) )$\n\n**Question-E:** *In implementing PMI–DECODE based inference, it seems that lagging will inevitably occur. How much is the speed difference from normal decoding?*\n\nWe need to compute both $P(r | d, h)$, and $P (r | h)$ at every step of the decoder. In our implementation, we compute both these probabilities in parallel on the GPU by batching them together in the same request. Specifically, we make use of the LogitProcessor class in HuggingFace to combine the probabilities at every decode step. In our experiments, we observed that PMI Decode with beam size 4 takes 429 ms per example whereas standard beam search takes 312 ms per example on a A100 GB GPU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Problem statement and aiming objective are very clearly described: distinction between faithfulness and hallucination\n- New state-of-the art performance metric for estimating faithfulness of generated dialogue\n- Novel decoding strategy for enhancing faithfulness in dialogue generation", "reasons_to_reject": "Nothing special, but I wonder the following questions.", "questions_to_authors": "- I am not sure if \"dialogue history\" is the real matter as you have noted in the Introduction. According to the proposed methodology, the information disparity derived by the \"document\" seems to be the key aspect in guaging faithfulness. Have you done any additional analysis on the importance of considering dialogue history? Although dialogue history is considered in the proposed CPMI, it seems that CPMI can be implemented without dialogue history, and it seems that this can also produce quite meaningful results.\n\n- In equation 2, were P(r|d, h) and P(r|h) estimated by different models? I find that this paper utilized fine-tuned model for such probabilities, but I wonder that whether you intended to use respective model or not. If different, P(r|d, h) and P(r|h) would be better to be noted like P(r|d, h, \\theta) and P(r|h, \\phi).\n\n- In equation 2, I find that p(r|h) can be vanished to 0 and may yield CPMI explosion. Is there any cases in practical use concerning it?\n\n- In equation 5, it seems that r_t = arg max_{v∈V} log P(v|r1:t−1,d, h) in the first stage (paper described r_t = arg max_{v∈V} log P(r1:t−1, v|d, h)). The generation probability of the t^th token is grounded at \"1:t-1\"^th tokens, and it seems that the generation probability of \"1:t-1\" is not considered together. If wrong, please let me know. \n\n- In implementing PMI–DECODE based inference, it seems that lagging will inevitably occur. How much is the speed difference from normal decoding?", "paper_topic_and_main_contributions": "- Faithfulness metric for the response generation, considering dialogue history. Proposal of the state-of-the art metric, PMI–FAITH.\n- Faithfulness-considering decoding strategy, PMI–FAITH, and significant enhancement on the BEGIN benchmark", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 58: textitR1\n- line 587: PMI–FAITHto"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A Point-wise Mutual Information-based technique to measure and improve the faithfulness of generated response with the grounding document and dialogue history. The paper is well written. Please add addressed feedback in the final version.
Decision: Accept-Main