Causal Document-Grounded Dialogue Pre-training
Authors: Yingxiu Zhao
Bowen Yu
Bowen Li
Haiyang Yu
Jinyang Li
Chao Wang
Fei Huang
Yongbin Li
Nevin L. Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Document-grounded dialogue
task specific pretraining
causal effect
TL;DR: We construct a causally complete dataset for document-grounded dialogue, and propose a causally-perturbed pre-training strategy.

Abstract: The goal of document-grounded dialogue (DocGD) is to generate a response by anchoring the evidence in a supporting document in accordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly enhanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To address this, we present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora. Additionally, we propose a causally-perturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect. Experiments conducted on three benchmark datasets demonstrate that our causal pre-training yields substantial and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings.

Causal Document-Grounded Dialogue Pre-training
Yingxiu Zhao', Bowen Yu'; Haiyang Yu², Bowen Li", Jinyang Li?, Chao Wang', Fei Huang', Yongbin Li", Nevin L. Zhang'
1 The Hong Kong University of Science and Technology, 2 Alibaba Group
{yzhaocx, Jeconnc ust.hk, hk, libowen. ne@gmail.com (yubowen. yiw, yifii. wey.eeiyy..... why, rww.i. ww, huang, h shuide. lyylyallbaba-inng
Abstract
clothing allowance:
If your clothing has been damaged, you may be able get money each year to help you buy new clothes. This is d
The goal of document-groundder dialogue (DocGD) is to generate response by anchor- ing the evidence in supporting document in ac- cordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly hanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre- pre-training approach that explicitly captures the causal relationships. To address this, we present the first causally- complete dataset construction strategy for de- veloping million-scale DocGD pre-training cor- pora. Additionally, we propose a causally- perturbed pre-training strategy to better cap- ture causality by introducing perturbations the variables and optimizing the overall causal effect. Experiments conducted on three bench- mark datasets demonstrate that our causal pre- training yields substantial and consistent im- provements in fully-supervised, low-resource, few-shot, and zero-shot settings'.
disability compensation ... clathing allowance.
How to get these benefits?
You'll need to file claim for disability compensation.
heard about a disability benefit for clothing, it true?
help
How to apply for the allowance my dad eligible?
You'll I1 need to file clamm for disability compensation.
He will need to file claim for disability compensation.
Figure 1: Causal graph of an example in DocGD, where four variables are causally connected: document d, evi- dence dialogue context e, and response
on downstream tasks, by mitigating the gaps be- tween pre-training and fine-tuning due different data distributions and training objectives (Mengge et al., 2020; Liu et al., 2021: Bai et al., 2022). Nonetheless, the study of pre-training for DocGD is hindered due to the difficulty of modeling the causal effects, which is prominent characteris- tic of DocGD. As shown in Figure/ 1, the task requires the model identify evidence in doe- ument based on the dialogue context, and then utilize the grounding evidence to generate a corre- sponding response. This process involves the inter- play of four variables that are causally connected. To attain precise modeling of causal effects during tre-training, two challenges must be overcome: (1) the scarcity of large-scale and causally-coppler DocGD datasets for pre-training, as opposed di- alogue generation tasks that have the advantage of utilizing conversational data from various so- cial media sources (Zhang et al., 2020), (2) the traditional likelihood objective (e.g., Raffel et al.
1 Introduction
Goal-oriented dialogue, focusing on assisting users in achieving their goals through natural language interactions, has made significant progress recent years (Peng et al., 2021; He et al., 2022). Nonethe- less, these systems frequently encounter constraints in providing information that extends beyond what can be obtained from particular databases or do- mains. To address this issue, researchers have proposed the goal-oriented Document-Groundee Dialogue (DocGD) task (Feng et al., 2020, 2021), which leverages documents as the external knowl- edge source to support dialogue systems in meeting the diverse information needs users.
Recently, task-specifie pre-training has shown an extraordinary ability to boost performances
Corresponding author.
'The datascts and code will be made publicly available.

---

(2020)) being insufficient capture the causal re- lationships among variables.
effectiveness of our pre-training corpus construc- tion and puusally-petturbed pre-training strategy. Especially, CausalDD even outperforms GPT-3.5.
For the first challenge, we propose a novel strat- egy for building DocGD pre-training corpus, We define dataset as causally-complete if it in- cludes all the variables related to task and encom- passes all reasonable causal relationships among these variables. Our strategy involves two steps. Firstly, we transform Wikipedia documents into dialogues by generating pseudo-user utterances and modifying evidence in the documents to serve as agent responses. Secondly, we extract ground- ing documents embedded in URLs and insert vir- tual evidence to supplement dialogues from Reddit, Both steps guarantee that the datasets are causally- complete, and they complement one another, as the former has authentic evidence with synthetic dialogues, while the latter possesses authentic dia- logues with synthetic evidence.
2
Related Work
2.1
Causal Inference For NLP
Causal Inference is statistical modeling tool that has been applied in explanatory analysis better understand the relationships between vari- ables (Glymour et al., 2016; Kuang et al., 2020; Feder et al., 2022). In the context named entity recognition, Zeng et al.2020 sought to eliminate spurious correlations between context and entity tokens by replacing entities with counterfactual tokens. Wu et al.2020 similarly utilized counterfac- tual samples in sentiment classification, replacing causal terms with their antonyms. Our research endeavors to explore DocGD from causal per- spective, presenting the causal relationships among DocGD variables for the first time.
To tackle the second challenge, we propose a causally-perturbel pre-training strategy that en- hances the modeling of causality in our pre-training datasets. Our approach entails introducing causal interventions to both the document and evidence variables while optimizing the total effect of re- sponses for different causes. The total effect com- prises the natural direct effect (NDE) and total indi- rect effect (TIE) (Niu et al., 2021). In essence, the NDE quantifies the impact of irrelevant sentences in the supporting document, while the TIE captures the influence of evidence (detailed explanations in $3.3). Our objective is twofold: enhance the model's resilience perturbations irrelevant sen- tences by minimizing the NDE, and to promote re- liance on evidence in generating dialogue responses by maximizing the TIE. To achieve this, we retain relevant evidence while perturbing the remaining parts of the document to improve response consis- tency when using two versions, thus reducing the NDE. Additionally, we eliminate evidence from the document while preserving other information, sub- sequently decreasing the likelihood of generating original responses, thus maximizing the TIE.
2.2
Dialogue
Goal-oriented dialogue generation grounded in doc- uments is challenging and realistic task (Ma al., 2020; Yu et al., 2022: Zhang et al., 2023). Re- searchers have increasingly utilized documents in a more flexible manner to improve the fluency and informativeness of model-generated responses, in- cluding in tasks such as Machine Reading Compre- hension, Convention Question Answering, and the focus of this paper, DocGD, To support the devel- opment of models for these tasks, various datasets have been proposed, including CoQA (Reddy al., 2019) 9), QuAC (Choi et al., 2018). DoQA (Cam- pos et al., 2020), Wizard (Dinan et al., 2018), Doc2dial (Feng et al., 2020), Multioc2Dial (Feng et al., 2021) and Doc2bot (Fu et al., 2022).
However, the high annotation requirements for document-grronnde dialogues have limited the scale of available annotated data. To address this issue, Li et al. (2020) express the document knowl- edge as latent variables and devise a variational approach to achieve zero-resource knowledge- grounded dialogue generation. Li et al. (2021) ho- mogenize different sources of knowledge (e.g., dic- tionaries, or knowledge graphs) into unified rep- resentation alleviate reliance on single source. Gao et al. (2022) develop a prompt-connected multi-task learning to unify DocGD tasks. Zhang et al. (2023) propose coarse-to-fin tonfine knowledge selection to improve knowledge retrieval among
Overall, we refer to the two aforementioned strategies jointly as Causal Document DocumenntGGrounded Dialogue (CausalDD). We thoroughly conduct ex- periments and analyses on three DocGI bench- mark datasets. Our results, obtained through fully- supervised, few-shot, low-resource, and zero-shot scenarios, and evaluated by both automatic and human assessment, convincingly demonstrate the

---

Reddit
WilDlalo
Now fies the freshman relone to eating tobes??
pupper
onsrrental supertsed dits, stadents dio isua riist the proper propodion..
Inoticedttat mony pees wennwightt today.
Nos, being ovenweigtt i bad for
Thicls because when ctudent go off collage, they fhoe on pndependento
pess hcolth
peogle
heatts
Inpaintee
Do you know how k
You can throse a papee
Paraphoasg
More tres to persent of cooleeeatveeoss stelles routinlly eet fost fooch instodd fists and regessllss, rescorch shoee.
What's pood
Artides Freehmson 15
Feerdse
both
sld, That's easily
superitedd studeons shio ceusily aigert proper proportio
acvavved with dog: Nuarly warr dag would bunelt from at leask sotora ting, or good choee afrer holl or Fridee, the old. fy mennwais tnsert cpeartuiity chow and play.
many independence percont colloge studiots commoniiy
Thi beteeuse eusertt off not experioncd before. Research iassshoon
o papprr rok, you cre ofee your Imsractiie toys don't have In experale...
repeet feity fruits and vegetables.
Figure 2: Two complementary datasets WikiDialog and Reddit built for DoeGD, which are causally complete.
Datasets
Dialogues
Documents
Total Turns
3.2
Causally-complete Dataset Construction
WikiDialog
1.00M
0.12M
3.00M
While task-specific pre-training has been exten- sively researched in various domains and proven effective, there is lack of research on pre-training for DocGD. The challenge lies in constructing pre-training DocGD corpus that captures causal relationships among relevant variables. Merging a dialog corpus with unverified documents with- out careful considcration of causality can result in missing variables or weak causal connections. As a consequence, models may learn spurious features, such as generating responses solely based on the dialogue context while ignoring the document d. Our analysis ($4.7) demonstrates that this significantly degrade performance.
Reddit
1.00M
L.0MM
1.39M
All
2.00M
1.12M
4,39M
Table 1: Statisties of CausalDD pre-training corpora.
multiple documents. These approaches omit pre- training on DocGD and merely initialize the pa- rameters with general language models such as T5 (Raffel et al., 2020). Thus, how to effectively pre-train the DocGD model is still open problem. In this paper, we give the answer from the perspec- tive of causal inference and demonstrate that causal pre-training effective in various DocGD settings.
3 Methodology
To overcome this challenge, we propose pre- training data construction strategy that utilizes high-quality documents from Wikipedia to create a causally-complete dataset with virtual dialogues. We further complement the corpus by leveraging real- world dialogues from Reddit to construct an- other dataset with virtual external knowledge. (Re- fer data statistics in Table 1).
In this section, we first take a casual-effect perspec- tive on the DocGD task in $3.1. We then propose two strategies to overcome the challenges discussed in $1: (1) a dataset construction strategy in $3.2 for building a causally-complete pre-training cor- pus; (2) a causally-perturbed pre-training strategy in $3.3 for better modeling causality of DocGD.
3.2.1 Causally-complete WikiDialog
3.1 A Causal-Effect Perspective on DocGD
Wikipedia offers wealth of excellent articles, of- authored or edited by experts who have invested considraable time and effort into ensuring clarity, accuracy, and addressing readers' queries. A dis- tinctive feature of Wikipedia is that each page is dedicated to describing a specific entity. Therefore, when a user inquires about one entity from an agent, the corresponding page can be considered as the source of information for the agent's 's response. We utilize this property to convert Wikipedia into two-person DocGD. Given a page, denoted as d (c1, C2,......... Cm), consisting of m sen- tences, each sentence is treated as evidence e, rep- resenting an agent's response in an m-round di-
The DocGD task commonly formulated se- quential process comprising two sub-tasks: knowl- edge grounding and response generation (Feng et al., 2020; Gao et al., 2022). In knowledge grounding, a text segment denoted as e, is iden- tified within the supporting document d, based on the dialogue context c. This segment serves as the evidence for generating the subsequent response Consequently, four variables c, d, e, r are causally interrelated: The causal paths + r and + directly influence the response, while an indirect effect occurs through the intermediary variable e. See Fig. for an example causal graph.

---

Document Diolog
context (c) and response (r), but lacking the docu- ment (d) and evidence (e) for DocGD. We observe that many submissions on Reddit contain URLs. These URLs often lead to web pages such as news articles that provide specific information related to the discussed topics. Therefore, we can crawl Reddit submissions that include URLs, using the content pointed to by the URL as the document (d), while utilizing the conversations and replies under the submission as and respectively.
0.00
000
00o
Response
Document
Pel i ExDI
D
Evldencc o
Pus LTIE
Document
Figure 3: The diagram demonstrates causally-perturbel pre-training for CausalDD. The central document on the left represents the original document d, while the top and bottom documents represent perturbed documents d and d used for optimizing NDE and TIE, respectively.
Remark. Why is the above construction causally complete? First, the evidence e a paraphrase of response r, so a causal relationship can be estab- lished between and Given that r is a natural response to the dialogue context on Reddit, causal paths c + and the random insertion of evidence e into the docu- ment d, d can thus become the cause of e, and then the cause path d >> r could also be established.
alogue. To complete the dialogue, we employ dialogue inpainter (Dai et al., 2022a) (see A.1) generate missing user utterances, which interleaved with the agent's responses. The in- painter generates a pseudo session Sinpainter (u1, E1, u2, E2, Um, em), where ui is the i-th generated utterance for the user, and C; is used as the agent's response. In our causally-complete WikiDialog, we first copy the first two turns from the inpainter. For the third turn, we randomly se- lect one turn from the remaining turns the ut- terance and grounding knowledge. To enhance naturalness, we employ a well-trained paraphrase model (see $A.2) to rewrite the evidence ez into the agent's response r3, creating the dialogue sequence 8 = (u1,c1: C1, u2, G2 Uy, r3). ) The dialogue context c c (u1,e1, u2, e2, u3), while the response is a paraphrased rendition of the evidence é3.
cxist. Furthermore, through
3.3 Causal Pre- Training Framework
DoeGD-specific Pre-Training To better capture the interdependence between evidence and re- sponse r, as described by Gao et al. (2022), we
adopt the most efficient and straightforward fine- tuning method to sequentially generate the e and r based on the dialogue context c and associated doc- ument d, instead of retrieving evidence and feeding to the model to generate responses in pipeline manner. We align our pre-training task with fine- tuning by optimizing the following objective:
Do = (dger))C
log(po(esr(d:c)) (1)
Remark. Why the above construction causally complete? First, the evidence is an exact sentence in the document d, and the dialogue context e is generated by the dialogue inpainter based on e, so the evidence e can be uniquely determined in d by that is, e is the effect of {c, d]. Considering response is paraphrase of e, so is the direct cause of r, and the causal paths e -> and dl >> can also be implicitly established.
where C' is the constructed causally-complete cor- pora in $3.2, A is optimized during pre-training.
Causally- Pre-Training To facilitate the causal modeling of DocGD, we propose causally-perturhed pre-training strategy by intro- ducing causal perturbations variables of DocGD and evaluating the outcomes under different causes.
Here, we utilize a common measurement, causal effect, to compare two potential outcomes for the same variable under two distinct treatment condi- tions (Rubin, 1978; Robins, 1986). Supposed that the random variable X assigned with the observed value x,, X x,, represents "under no-treatment condition" and X x* represents "under treat- ment condition" (Niu et al., 2021). The total ef- fect (TE) of treatment X x on the variable compares two hypothetical situations X = x and
3.2.2 Causally-complete Reddit
Despite having a causally-complete dataset like WikiDialog, the generated virtual dialogues may not fully align with the distribution of real-world human conversations. To address this issue, we pro- pose supplementing the pre-training corpus with diverse and realistic conversations. We consider Reddit, a popular online platform for person-to- person discussions, as a valuable source of dialogue

---

==x,, which denoted TE Yg- Y. In DocGD, we aim to estimate the effect of the docu- ment on the identification of evidence e and the generation of response r. We denote X = x to rep- resent the original document d (under no-treatment condition), and X = x² to represent applying per- turbations to the document (under treatment condi- tion). We use Y to denote the generated sequence of evidence and response. More precisely, we fur- ther divide the document d into two parts, the sen- tence where the evidence span lies and the other sentences {die) outside the evidence scope, i.c., d = e U e) Hence, the total effect in DocGD can be written as:
retained. Secondly, maximize TIE, we intro- duce the following unlikelihood loss:
CTIE =
Σ
log - p(((er)d;c)) (6)
(diGe))
unlikehihood
The situation d here represents removing of evi- dence from the document d, i.e., d ed After the removal, we aim decrease the model's probability of generating tokens in the ground truth evidence e and response T.
Total Pretraining Objective Overall, our pre- training objective is the sum of standard DocGD loss and our newly proposed causally-perturbed losses as follows:
TE- (dhho (ddce).
(2)
C DDeGGD + LNDE + CITE After pre-training, we fine-tune the obtained model on downstream datasets by optimizing DoeGD in Eq. 1 following Gao et al. (2022).
(7)
We adopt the decomposition in Niu et al. (2021) and adapt it to our causal DocGD scenario. Con- cretely, TE can be decomposed into the sum of the natural direct effect (NDE) and total indirect effect (TIE). NDE expresses the increase in the outcome Y when {die) changes {d
Experiments
4.1 Datasets
(3) We evaluate the effectiveness of our CausalDD for DocGD in both English and Chinese. For En- glish, we use the two causally-complete datasets constructed $3.2 for pre-training and evaluate the performances on two goal-oriented document- grounded dialogue datasets: Doc2dial (Feng et al., 2020) and MultiDoc2dial (Feng et al., 2021). Doc2dial (Feng et al., 2020) contains 3,474 dia- logues with 44, 14.149 turns for training and 661 di- alogues with 8,539 turns for evaluation Multi- Doc2dial (Feng et al., 2021) contains 4,796 dia- logues with an average of 14 turns grounded in 488 documents, with multiple documents supporting each dialogue in four different domains.
NDE = ,e Y(drehe
TIE is the difference between TE and NDE:
TIE = et Yidre)".e
(4)
For an ideal causal-aware DocGD pre-trained model, variations in d' should not significantly affect the model's output, while the presence of evidence within d determines the model's ability to identify evidence and generate responses that are relevant to the dialogue context. In other words, Y [ddyee",e should be indistinguishable from Y(Ax)). while ((dte)> e- needs to differ significantly from Yddie)",.e .- Hence,
necessary to improve the ro- bustness of our model against perturbations on vari- ables by minimizing NDE, and promote reliance on evidence in the generation of dialogue responses by maximizing TIE. (See the illustration of causally- perturbed pre-training of CausalDD in Figure 3.) To achieve this, we design two causally- perturbed pre-training objectives. Firstly, we use Kullback-Leible divergence to measure NDE:
For Chinese, we utilize translation model (Wei et al., 2022) to translate the English pre-training data into Chinese, and evaluate the performance on Chinese DocGD dataset Doc2bot (Fu et al., 2022). Doc2bot (Fu al., 2022) has samples of Chinese conversations that are natural, coherent, and grounded in diverse documents. Although the translation model may impact the quality of Chi- nese pre-training data, we have observed significant improvements in our approach across three Chinese pre-trained backbones. We leave constructing better Chinese pre-training data for future work.
LNDE =
Σ (d.c,e.)ec
(5)
where 7 -=c e (d refers to disturbing the doc- ument by randomly deleting or inserting some sentences while keeping the evidence sentence e in
Since we cannot access the test set, we report results on the development set for comparison following previous work (Gao et al., 2022).

---

Model
F1
and then generate corresponding responses. Hence, following Glass et al. (2022), we first train an ad- ditional retrieval model and a ranking model locate documents relevant to dialogues as the ex- ternal knowledge for downstream fine-tuning (see details in Appendix B.1).
BERTQA
42.6
36.7
BERT-PR-large
44.2
38.9
RoBERTa-PR-large
55.7
54.6
Multi-Sentence
56.1
57.4
DIALKI
65.9
57.4
UniGDD
65.6
76.4
GPT-3.5
57.3
CausalDD
66.0
77.3
4.3 Baselines
Dterge
67.0
78.1
We compare CausalDD with several strong base- lines, including UniGDD (Gao al., 2022) and many commonly measured methods specific to each dataset. UniGDD utilizes the pre-trained T5 model(Raffel et al., 2020) as the initialization and optimizes D in Eq. on downstream datasets, which also serves our most relevant baseline. Furthermore, meticulously design the instruc- tion assess the performance of GPT-3.5. See more baselines and details Appendix B.2.
Table Results on Doc2dial knowledge identification.
Model
BLEU
DIALKI-BART-buS
25.8
PAAERTa-PR-largeBAAAR
39.6
RoBERTa--lrge++TS-hen
40.7
UniGDD
42.4
GPT-3.5
3.57
CDusalDD
43.0
CausalDDiarge
42.5
Table Results on Doc2dial response generation.
4.4 Metries
In concurrence with prevalent measurements (Gao et al., 2022; Feng et al., 2020, 2021), we utilize the metrics of Exact Match (EM) and token-level Fl for the identification of evidence and BLEU (Pap- ineni al., 2002) for the generation of responses.
4.2 Implementation Details
For pre-training, we use the pre-trained T5-base and more powerful T5-large (Raffel et al., 2020) to initialize CausalDD for English DocGD. For Chinese DocGD, try three types of initializa- tion: T5-Mengzi (Zhang al., 2021), mT5 (Xue et al., 2021), and T5-Randeng (Wang al., 2022). CausalDD is pre-trained on 4 80G NVIDIA A100 GPUs with a maximum learning rate of le-5 and a warm-up ratio of 0.1 for one epoch. The batch size per iteration is set to 8, and we use the A op- timizer with parameters betal 0.9, beta2 = 0.98. and epsilon 1e-6. The total pre-training time is about 60 hours. After CausalDD pre-training. the model is then fine-tuned on the Doc2dial, Mul- tiDoc2dial, and Doc2bot datasets following Gao et al. (2022), with batch size of and training epochs of 5. Note that both MultiDoc2dial and Doc2bot require the model to first retrieve relevant documents, identify evidence in those documents,
4.5 Main Results
Fully-Supervised As shown in Tables 2, 3, 8, and 4, our proposed CausalDD method outper- forms the baseline model on all evaluation met- rics in both English and Chinese datasets. The improvements of CausalDD over other baselines are statistically significant with p-value < 0.05 un- der t-test. Regardless of the language or different T5 parameter initialization, CausalDD consistently surpasses the strongest baseline UniGDD.
Initialized with a larger T5, CausalDD further enhances the performance compared CausalDD, albeit with slight decrease in BLEU score. To investigate this phenomenon, we com- pute the distinct scores with Dist-n following et al. (2015) to evaluate generated responses on the Doc2Dial dataset, shown in Table 7. We discover that the large model tends to generate more di- verse responses, which may differ from the expres- sions of manually annotated answers. Furthermore, observe that the popular large language model GPT-3.5 performs poorly on the existing DocGD datasets, despite including task instructions and cases its prompt. Upon analyzing GPT-3.5's pre- dictions, we find that its underperformance stems
Model
F1
EM
BLEU
UniGDD Rundeng
38.6
39.6
15.3
MlusalDDranden
42.9
41.8
22.3
UniGDD,,rr
40.6
41.5
17.7
Caus
48.2
47.4
22.0
UniGDD Mengri
44.2
44.9
CausalDD
47.8
25.4
Table 4: Results on Doc2bot.

---

Few-Shot 50-Shot
Low-Resource
Dataset
5-Shot
100-Shot
1%
5%
10%
UniGDD
13.7 25.8 (12.11)
14,8 27.4 (12.61)
17.1 29.8 (12.71)
17.8
36.9
39.8
Doc2dial
CausalDD
34.7 (16.91)
43.4 (6.51)
46.9 (7.11)
UniGDD
13.5
20.1
19.8
23.9
33.9
34.5
MultiDoc2dial
CausalDD
29.8 (16.31)
29.7 (9.6 1)
30.6 (10.81)
34.4 (10.51)
42.0 (8.11)
44.3 (9.81)
0.00
0.01
0,10
0.00
2.3 24.8 (22.51)
12.6
Doc2bot
CausalDDMemga
12.3 (12.31)
12.7 (12.71)
14.9 (14.81)
13.7 (13.71)
31.0 (18.41)
Table 3: Few-Shot and Low-resource results for knowledge identification (F1 score).
Few-Shot 50-Shot
Low-Resource
Dataset
Model
5-Shot
100-Shot
1%
UniGDD
2.00
2.27 11.2 (8.9 T)
3.07
5.07
14.6
15.6
Doc2dial
CausalD>))
(9.11)
11.6
14.9 (9.8t)
16.1 (1-5t)
18.7 (3.11)
UniGDD
2.20
2.31
3,37
6.98
14.1
12.7
MultiDoc2diial
CausalDDD
(11.8))))))
(11.8)
13.9 (10.5t)
13.5 (6.5t)
(0.61)
(4.31)
UniGDD
0.00
0.00
0.00
0.00
2,10
2.31
Doc2bot
CausallD
(2.501))
(2.611)
(3.701)
(3.007)
(10.2))
13.1 (10.8t)
Table Few-Shot Low-resource results response generation (BLBU score).
Dist-1
Dist-2
Dist-3
Dist-4
Model
F1
EM
BLEU
UniGDD
0.0736
0.3191
0.5055
0.6049
D toleen -nq
40.0
22.3
15.7
CD
0.0736
0.3198
0.5079
0.6081
Dutruct -nq
39.8
22.3
16.6
CausalDDtarge
0.0749
0.3308
0.5299
0.6347
token
26.4
Dstruct_ -ft
43.5
26.1
19.5
Table 7: Distinct scores of responses Doc2Dial
D tokenn -rr-clsftt
42,1
25.0
18.4
Dutruct -T-cls-ft
43.5
26.2
19.8
from failure to adhere strictly to the given doc- ument's content for generating responses and pro- viding evidence. Instead, it suffers from a severe hallucination issue, generating irrelevant content. We present the results of our human evaluation in Section 4.9.
CPI-NLP
47.3
R3
43.3
31.1
G4
44.6
Re3FDD
33.5
UniGDD
45.8
31.8
GPT-3.5
30.7
CausalDD
63.7
49.3
33,9
CausalDDrarge 64.5
51.0
33.6
We also observe that the model's improvements more significant on Doc2bot in Table 4. We speculate that it's because the training data of Doc2bot is smaller than that of Doc2dial and Mul- tiDoc2dial, and our model obtains better initializa- tion for DocGD through causal pre- tre-training, thus being more data-efficient for fine-tuning.
Table 8: Results on MultiDoc2dial.
ment of 9.7 points is achieved in these settings, indicating that our method is more effective with limited human- data, which is particularly important for real-world applications.
Few-Shot Low-Resource To verify the above speculation, we further conduct experiments on three datasets under few-shot and low-resource set- tings. We consider few-shot settings with only 5, 50, and 100 training examples and low-resource settings with 1 %6 5%, and 10% of the original train- ing datasets to train the models. The results are shown in Table 5, 13 and 6 for knowledge identifi- cation and response generation tasks, respectively (see more results in Appendix B.3). We can tice that the improvements of CausalDD are more significant in scenarios with such small amount training data. Specifically, an average improve-
Zero-Shot We evaluate the performances on three datasets under the zero-shot setting. Results in Table indicate that CausalDD achieves superior performances
UniGDD without any training samples of downstream tasks. This verifies the high quality of our constructed causally-complete corpora, and the good initialization provided by CausalDD for downstream tne-tuning:
4.6 Ablation Studies
To evaluate the contribution of each component in CausalDD, we conduct a series of ablation studies on two language datasets: Doc2dial and

---

30 2.00
25
(a) Convergence Speed
(b) Multiple Turns
(c) Multiple Evidence
Figure 4: Compared the UniGDD model, CausalDD has faster convergence time, better modeling of dialog
history, and stronger grounding of multiple evidence.
Dataset
Model
F1
EM
BLEU
Model
EM
F1
Doc2dial
UniGDD
13.9
0.00
1.98
UniGDD
65.6
76.4
CausalDD
26.1
1.43
10.9
CausalDD CausalDDincompieet incomplets
65.3
76.4
MultiDoc2dial
Ca
UniGDD
30.0
14.3
2.76
0.00
2.32
13.9
CausalDD compiete
65.8
77.1
UninDDssensai
0.00
0.00
0.00
Doc2Bol
CausalDDaceuse
11.46
5.25
1.98
Table 11:
Performance comparison with causal-
incomplete and complete pre-training data on Doc2dial
Table 9: Results under the zero-shot setting.
To assess the effectiveness of our created com- plementary datasets, we also carry out case study that compares the responses of CausalDD trained with various pre-training data (Appendix B.4).
Model
Doc2dial
Doc2bot
F1
FM
BI.EU
F1
EM
BLEU
Wikipedia
76.9
65.7
42.5
46.8
46.5
23.6
+ Reddit
77.2
65.8
42.7
47.6
47.0
24.8
4.7
Effects of Causally-complete Data
+ NDE
77.2
65.9
43.0
47.5
47.2
24.0
TIF
77.0
66.0
42.8
46.8
46.6
23.9
The creation of causally-complete pre-training data is one of the contributions of this paper. But (1) is causally-complete data really necessary for DocGD pre-training?
Table 10: Ablation study of CausalDD Doc2Dial.
(2) what problems would arise if part of the causality the pre-training data was missing? To address these two ques- tions, we build a causally-incomplete pre-training dataset by removing the introduced evidence e from the previously-built Reddit dataset (i.e, the document d {d)ee)). Then pre-training task is to generate responses based solely docu- ments and dialogue context c, without identifying knowledge first. We also pre-train model using a causally-complete Reddit dataset for compari- son. The results of Table 11 indicate that perfor- mance degrades when pre-training data cannot ade- quately model causal connections. The comparison with UniGDD (i.c.. initialized with the original pre- trained T5) demonstrates that pre-training introduces bias, resulting in discrep- ancy between pre-training and fine-tuning.
Dot2Bot. The baseline for these studics is pre- training CausalDD exclusively on the causally- complete WikiDialog dataset. We then assess the impact of adding additional components, including (I) supplementing WikiDialog with constructed Reddit dataset (+Reddit), (2) minimizing the NDE loss in Eq. 5 during pre-training (+NDE), and (3) maximizing the TIE loss in Eq. (+TTE).
Results in Table 10 indicate that: (1) introducing a causally-complete Reddit containing real-world dialogues enhance the ability of the model to iden- tify knowledge and generate better responses; (2) optimizing NDE to enhance the consistency of the model outputs with different support documents can enhance the robustness of the model; (3) op- timizing TIE to prevent the normal output of the model when removing evidence from documents increases the model's reliance the grounding ev- idence. These results validate that each component has positive effect on CD, leading to its better capability of modeling causal relationships among DocGD variables.
4.8 Other Benefits of Causal Pre-training
In addition to overall performance improvement, we also observe some additional benefits brought by causal pre-training: (1) faster convergence speed

---

(Figure 4(a)), our model achieves good results in the first epoch due having more DocDG- specific initialization parameters compared to gen- eral pre-training; (2) better modeling of dialog history (Figure 4(b),, we find better performance across all turns when we divided the Doc2bot test set based on the number of turns in the dialog his- tory; (3) better ability to ground complex evi- dence in Doc2bot, many samples require the model ground multiple relevant segments in the docu- ment, and as the number of relevant evidence in- creases, CausalDD still shows better performance compared to UniGDD (Figure 4(c)).
DocGD, To the best of our knowledge, this is the first work that analyzes DocGD from causal per- spective. Extensive experiments and analyses ver- ify that our causal pre-training method CausalDD significantly improves performance under fully- supervised, few-shot, and low-resource settings, while also accelerating convergence and enhancing the ability to handle complex cases.
6 Limitations
Despite the fact that CausalDD has demonstrated its superior performance on three benchmarks, it still has few limitations. Firstly, the pre-training data we construct is generated by models such the dialogue inpainter and paraphrase model. De- spite the large size of our causal-complete datasets, the data quality is slightly inferior to manually an- notated data. We will also consider constructing data corpus through large language models like Li et al. (2023); Zhao et al. (2023). Secondly, there are other tasks such as knowledge graph grouuddd dialogue, and our proposed pre-training data con- struction strategy may not be applicable, Lastly, the effectiveness of task-specific pre-training will de- crease as the amount of labeled data increases, so if a large amount of DocGD labeled data is provided, the performance gains brought from our approach may be marginal.
4.9 Human Evaluation
To evaluate the performance of CausalDD against strong baselines, we randomly select 100 evalua- tion instances in Doc2Dial and request five human annotators to perform pairwise comparisons two factors: (1) Relevance: indicating which response is more pertinent and relevant the user's inquiry, and (2) Informativeness: determining which an- swer is more informative. See Table 12.
Win
Tie
Lose
CausalDD vs. UniGDD
Relevance
42
55
Informativeness
43
47
10
Ca vs. GPT-3,5
Relevance
61
34
7 Ethical Statement
Informativeness
27
20
53
This paper constructs a pre-training dataset for DocGD, and we discuss some related ethical considerations here. First, regards to intellectual property, the Wikipedia corpus and Reddit dump used as data sources are both freely available for research use, with the Wikipedia corpus shared under the CC BY-SA 3.0 license" and the Reddit dump shared for research purposes (Baumgartner et al., 2020). Second, have taken measures control potential risks by ensuring that the texts in Wikipedia do not contain private information. Additionally, we have ensured that the conversation data from Reddit does not include any personal information and that the topics discussed are public and harmless. Third, for human evaluation on the downstream Doc2Dial task, we hire five annotators to score 400 instances in total. The hourly pay is set to 15 US$ per person, higher than the local statutory minimum wage.
Table 12: Human Evaluation
Our method exhibits an apparent edge over UniGDD in two aspects when compared to base- lines, highlighting the ability of CausalDD to ef- fectively leverage rich document text to generate more suitable responses by capturing the causal relationship among variables. While GPT-3.5 can produce more informative responses, it depicts less consistency with the document and user's inquiry, implying the presence of hallucination issues.
5 Conclusion
In this paper, we demonstrate that modeling com- plete causal relationships among variables (i.e., documents, dialogue contexts, evidence, and re- sponses) is necessary for pre-training for document- grounded dialogue task (DocGD). We propose a strategy for creating causally-complete pre- training datasets and design a -eentullypeeturredd pre-training strategy to model the causality of

---

References
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Fisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Trunsactions of the Association for Computational Linguisticss 10:1 138-1158.
Sai Vamsi Alisetti. 2020.
Paraphrase- Generator. https: www.....c.ooooomom.. ccm/Vamm1995//////
Paraphrase-Generaton
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for amr parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 6001-6015.
Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. Multidoc2dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162-6176.
Srijan Bansal, Suraj Tripathi, Sumit Agarwal, Sireesh Gururaja, Aditya Srikanth Veerubhotla, Ritam Dutt, Teruko Mitamura, and Eric Nyberg. 2022. R3: : Re- fined retriever-reade pipeline for multidoc2dial. In Proceedings of the Second DialDoc Workshop on Document- Document-grounded Dialogue and Conversational Question Answering. pages 148-154, Dublin, Ireland. Association for Computational Linguistics.
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel. Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A dialogue dataset.
goal-oriented secumenntggrondeee In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 81 18-8128.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the inter- national AAAI conference on web and social media, volume 14, pages -30.
Haomin Fu, Yeqin Zhang. Haiyang Yu, Jian Sun, Fei Huang, Luo Si, Yongbin Li, and Cam-Tu Nguyen. 2022. Doc2bot: Accessing heterogeneous doc- uments via conversational bots. arXiv preprint arXiv:2210.11060.
Chang Gao, Wenxuan Zhang, and Wai Lam. 2022. Unigdd: unified generative framework for goal- oriented Hocument-groundee dialogue. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 599-605.
Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Mi- lan Deriu. Mark Cieliebak, and Eneko Agirre. 2020. Doqa-accessing domain-specific faqs via conversa- tional qa. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7302-7314.
Michael Glass, Gactano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2g: Retrieve, rerank, generate. arXiv preprint NXV:22007006600
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih. Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018, Quae: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184.
Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016. Causal inference in statistics: primer. John Wiley & Sons.
Nico Daheim, David Thulke, Christian Dugast, and Hermann Ney. 2021. Cascaded span extraction and response generation for document-gen grruuddd dialog. In Proceedings of the 1st Workshop Document- grounded Dialogue and Conversational Question An- swering (DialDoc 2021), pages 57-62.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang. Min Yang. Fei Huang, Luo Si, et al. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi- supervised learning and explicit policy injection. In Proceedings of the AAAI Conference on Artificial Intelligence.
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent 7hao, Aida Amini. Qazi Mamunur Rashid, Mike Green, and Kelvin Guu, 2022a. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, pages 4558 4586. PMLR
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunox, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), P), pages 6769-6781.
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. 2022b. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, pages 4558-4586. PMLR.
Jacob Devlin Ming-We Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: khowledge-powered conversational agents. In International Conference on Learning Representations.
Kun Kuang, Lian Li, Zhi Geng, Lei Xu, Kun Zhang, Beishui Liao, Huaxin Huang, Peng Ding, Wang Miao, and Zhichao Jiang. 2020. Causal inference. Engi- neering, pages 253-263.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke 7eutlmmyer. 2019. Bart: De- noising soquence-to-sqquene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461.
Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12700- 12710.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- generation
täschel, et al. 2020. Retrieval-augmente for knowledge-intensive nip tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9477
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Buildingtask bots at scale with transfer learning and machine teaching. Transactions of the Association for Compulational Linguistics, 9:807-822 824.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, divensity-promoting
and Bill Dolan. 2015, A diversity- promoting objec- tive function for neural conversation models. arXiv prepriat arXiv: 1510.03055.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., (1410)): 1-67.
Kun Li, Tianhua Zhang, Liping Tang, Junan Li, Hongyuan Lu, Xixin Wu, and Helen Meng. 2022. Grounded dialogue generation with cross-encoding re-ranker, grounding span prediction, and passage dropout. In Proceedings of the Second DialDoc Workshop on Document-grouee grounded Dialogue and Con- versational Question Answering, pages 123-129, Dublin, Ireland. Association for Computational Lin- guistics.
Siva Reddy, Dangi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
James Robins. 1986. new approach to causal in- ference in mortality studies with a sustained expo- sure period application to control of the healthy worker survivor effect. Mathematical modelling, 7(9- 12):1393-1511
Linxiao Li. Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020. Zero-resource knowledge-groundee dialogue generation. Advances in Neural Information Processing Systems, 33:8475- 8485.
Donald B Rubin. 1978. Bayesian inference for causal effects: The role of randomization. The Annals of statistics, pages 34-58.
Minghao Li. Yingxiu Zhao, Feifan Song. Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin L.. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. EMNLP.
Junjie Wang. Yuxiang Zhang, Lin Zhang, Ping Yang. Xinyu Gao, Ziwei Wu, Xiaoqum Dong, Junqing He, Jianheng Zhuo Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiao- jun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang. 2022. Fengshenbang 1.0: Be- ing the foundation of chinese cognitive intelligence. CoRR, abs/2209.02977
Yu Li. Baolin Peng, Yelong Shen. Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2021.
nowwlegeegroundeee dialogue generation with a unified knowledge representation. arXiv preprint arXiv:2112.07924.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2021. Tapex: Table pre-training via learning neural sql executor. In International Conference on Learning Representations.
Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. 2022. Learning to gen- eralize to more: Continuous semantic augmentation for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 7930-7944.
Longxuan Ma. Wei-Nan Zhang, Mingda Li, and Ting Liu. 2020. A survey of document grounded dialogue systems (dgds). arXiv preprint arXiv:2004. 13818.
Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-fine pre- training for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6345- 6354.
Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court's 's view generation with causality. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 763-780.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, -heng Hua, and Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In
Zeqiu Wu, Bo-Ru Lu, Hannanch Hajishirzi, and Mari Ostendorf. 2021. Dialki: Knowledge identification in conversational systems through dialogue- document

---

contextualization. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 1852-1863.
A Method Details
A.1 Dialogue Inpainter
The goal of a dialogue inpainting is to task a partial dialog to generate complete dialog. A dialogue inpainter is trained using the following dialogue construction task (Dai al., 2022b): Given a com- plete dialog, d (u1, 22,"" wr), we randomly mask one utterance u, yielding a partial dialogue: dm (t) = (N1, 14-1, o, U441,".. ,r,) With this partial dialogue as the input, we train T5 to predict u4 with the following objective:
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng and Meng Jiang. 2022. text generation. ACM
survey of knowledge-enhane Computing Surveys (CSUR).
L - Σ
Eumaallggee (e(ue dm (1)),, dED
(8)
Xiangji Zeng. Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020. Counterfactual generator: A weakly- supervised method for named entity recognition. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 7270-7280.
where D is a corpus of complete dialogs and it is randomly sampled utterance from d. We then use the trained inpainter to transform a document into a dialog. Suppose a document = =(81, 82, ,m)) image each sentence 8; is an utterance spoken by an agent in a dialogue with a user. We ask the in- painter to complete the following partial dialogue: (o, $1, 9, 9,, , sm). Each utterance from the imagined user starts masked and is responded to by the agent with a sentence from the document. We use the model autoregressively: generate úi and replace the first mask o, feed (4i), $1, o, $2) to com- plete the second mask. We continue the process until all masks are filled and the dialog is complete.
Shiwei Zhang, Yiyang Du, Guanzhong Liu, Zhao Yan, and Yunbo Cao. 2022. G4: Grounding-guien goal- oriented dialogues generation with multiple docu- ments. In Proceedings of the Second DialDoc Work- shop on
Dialogue and Con- versational Question Answering, pages 108-114, Dublin, Ireland. Association for Computational Lin- guistics.
Yeqin Zhang, Haomin Fu, Cheng Fu, Haiyang Yu, Yong- bin Li, and Cam-Tu Nguyen, 2023. Coarse-to-fin knowledge selection for document grounded dialogs.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William Dolan. 2020. Dialogpt: Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278.
A.2 Paraphrase Model
We adopt well-trained paraphrase model from Alisetti (2020) to transform sentence into another sentence with similar semantics. Specifically, the model takes an English sentence as input and pro- duces set of paraphrased sentences. We randomly select one sentence, and use it as the virtual utter- ance for causal- Wikidialogue and the virtual evi- dence for causal-Reddit, respectively.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet inge- nious pre- pre-trained models for chinese.
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L. Zhang. 2023. A preliminary study of the intrinsic relationship be- tween complexity and alignment.
B Experiments Details
B.1 Details of Retrieval and ranking
Because the Multidoc2dial and Doc2bot datasets do not give the document that the current dialog needs to be grounded but require the model to find the relevant document in the document corpus Z, so introduce an additional retrieval model and ranking model to find the most relevant document of the current dialogue context Retrieve. We use DPR (Karpukhin et al., 2020) as our retriever, which projects dialog context and documents to

---

shared space using two BERT encoders (Ken- ton and Toutanova, 2019)). During retrieval, we perform a maximum inner-product search with FAISS (Lewis et al., 2020). Formally, we retrieve K most relevant document 2 K = [1,,,, C for dialogue as:
B.2 Baselines
In Doc2dial, for the task of knowledge identifica- tion, we compare CausalDD with several strong baselines, including UniGDD (Gao et al., 2022), BERTQA (Kenton and Toutanova, 2019), BERT- PR (Daheim et al., 2021). RoBERTa-PR (Daheim et al., 2021), Multi-Sentence (Wu et al., 2021), and DIALKI (Wu et al., 2021). The other mod- els formulate knowledge identification chine reading comprehension task and extract the grounding span from the document. For the re- sponse generation task, we compare CausalDD with UniGDD and several pipeline methods, includ- ing DIALKI+BART that uses DIALKI for knowl- edge identification and BART (Lewis et al., 2019) for response generation, 1-BERTT-PR+BART, and RoBERTa+T5 (Gao et al., 2022).
x { E #/topk (BERT()) (E)T(2)))
(9)
ma-
The goal of retrieval training is to develop coder that maps given dialogue d and all relevant documents into an embedding space such that the dialogue close in proximity to its corresponding ground-trute document 2+. During training, we would like to maximize Preer (z 10, Z):
exp(sim(q: )))
Pretr(2 (2 1q,2) = Σ:Er exp(sim(q, =))
(10)
In MultiDoc2dial, we first use the same Retrieval and Ranking module as Re2G (Glass et al., 2022) to obtain relevant documents input for UniGDD and CausalDD. We also compare series of base- lines set up by Feng al. (2021), which use BM25 and multiple DPR variances retrievers, and use BART-large pre-trained on the CNN dataset the generation module. Moreover, we compare our method CausalDD with recent methods: R3 (Bansal et al., 2022), G4 (Zhang et al., 2022), and CPII-NLP (Li et al., 2022) proposed in the DialDoc Workshop and Re3FiD (Zhang et al., 2023), con- sidering these three methods did not provide the exact-match results, we leave blanks in the Table 8. In Doc2bot, we mainly compare with UniGDD and use three different T5 pre-trained models T5- Mengzi (Zhang et al., 2021), mT5 (Xue et al., 2021), and T5-Randeng (Wang et al., 2022). to initialize for more comprehensive comparison. The prompt for GPT-3.5 is carefully designed to match the input-output format of the training dataset of Doc2Dial. Examples and test input will be filled in the brackets the instruction.
where sim 2) is the cosine similarity between the normalized embeddings of the dialogue and doc- ument, generated by the BERT encoder. In order to perform contrastive learning, a set of negative documents must be sampled it is not feasible enumerate all other evidence. This done by using the BM25 algorithm to retrieve the most dif- ficult negative clue for each positive clue and then placing them into batches of 128 instances. The training loss is then calculated as the negative log- likelihood for the positive document.
Rank. The ranker use is based on the sequence-per classification. The dialogue q and each candidate document 2; E ZK are input to- gether to a BERT followed by a projection layer and Sigmoid function to calculate the ranking score of zę:
- Siggooi((Lnnea((BERTY. @ g))}
(11)
The training of the ranker begins by gathering the initial retrieval results on the training set. The top 36 samples (excluding the ground-truth evi- dence =) returned by the retrieval module are used as negative examples, and the ranker model trained to distinguish positive cases from negative cases.
Document-grounded dialogue task aims to identify evidence from a supporting document for a dialogue between a user and an agent for answering the user's question. Then the agent replies to the user based evidence. Here are two examples for your reference. In the example-input, <last- turn refers the last utterance of the user. After <last-turn> part, a reverse order of dialogue between <user> and
During inference, we first the retrieval model obtain the relevant document list and then use the rank model to identify the most relevant one as the supporting document. Note that we use the same retrieval and ranking model for Da and our baseline UniGDD.
the retrieved

---

<agent> is provided. After the signal </title>, the supporting document is provided. Specifically, you need to first retrieve the evidence (i.e. (groundion)))))) from the document in the test-input based on the dialogue and the user's s query. Sentences after <grounding> must be the exact same string in the document (including the spaces and punctuation). Then you should continually generate the response <agent)) based on the evidence as an agent to reply to the user. Example-1: (example1} Example-2: (example2) Test-Input: (test-input) Test-Output:
B.3
More Results of Few-Shot & Low Resource
We present experimental results of CausalDD and the strongest baseline UniGDD in Table 13 and 6.
B.4 Case Study
To assess the effectiveness of created comple- mentary datasets, compare the responses of CausalDD under various data scenarios: after pre- training only on WikiDialog, only on Reddit, and on their combined corpora followed by Doc2dial fine-tuning.
From the case in Figure 5, we can refer that:
UniGDD able accurately identify the grounding evidence in the supporting docu- ment, however, the generated response is just simple copy of the evidence.
After training solely on WikiDialog, the pre- dicted response more fluent and more con- sistent with the dialogue context rather than the copy of the evidence. This verifies the high quality of our constructed causally complete WikiDialog.
After training solely on Reddit, the response is more colloquial while retaining high quality.
Pre-training the complementary datasets (i.e., WikiDialog + Reddit) with CausalDD, the generated response is more precise and natural compared with the ground truth. This demonstrates that constructing complemen- tary datasets that are both causally complete yields better performances for downstream tasks' fine-tuning.

---

Few- Shott
Low-Resaurce
Dataset
Model
5-Shot
50-Shot
100-Shot
1%
5%
10%
UniGDD
0.00
0.00
0.00
0.38
6.11
3.43
Doc2dial
CausalDD
1.50 (1.50f)
1.95 (1.95))
2.49 (2.491)
6. (5.781)
15.4 (9.291)
20.1 (16.71)
UniGDD
0.00
0.24
0.07
0.00
9,42
3.95
MultiDoc2diall
CausalDD
2,93 (2.931)
3.02 (2.787)
3.31 (3.241)
7.24 (7.241)
14.5 (5.081)
17.1 (13.21)
DniGDDssengai
0,00
0.00
0.00
0.00
1.18 21.0 (19.8))
11.7
Doc2bot
Causal Daiemga
5.83 (5.83))
5.99 (5.997)
6.36 (6.361)
5.52 (5.521)
28.7 (17.01)
Table 13: Few-Shot and Low-resource results for knowledge identification (EM score).
Dialogue Context
filled of the information the Retirement Estimator and it took long time. When back from answering the door, of the information was gone. What happened?
bad. Were you gone long
Yes
Supporting Document
How Can You Each Page? For security there time limits viewing each You will receive a worning ofter 25 minutes without doing anything, and you will be able to extend your time page.
warning on page, you and work that will be lost.
page. you not, your
Response:
UniGDD:
security there time limits viewing each You will receive warning after minutes without doing anything and you will be able to extend your time on the page.
WikiDiolog:
Then you should know that there are time limits viewing each page. You will receive a warning 25 minutes doing anything you
extend your
page.
Reddit:
Okay, security reasons
viewing each page. will warning after 25 minutes without doing anything and will be able to extend your time on the page.
WikiDiolog Reddit (CausalDD):
Do you know that for security reasons there are time limits for viewing pages?
Ground Truth:
For reasons of security, there are time limits for viewing each page.
Figure 5: Case study for Doc2dial,


Reviews:
[{"review_id": "tegqky3PmY", "reviewer": "Reviewer_y1SZ", "summary": "", "questions": "1. In Table 3, a marginal reduction in performance is observed for Large CausalDD. Does this observation potentially imply that the BLEU metric inadequately assesses the model's proficiency? I think that some more suitable evaluation metrics should be replaced for experimentation.\n2. Have there been attempts to experiment with popular decoder-only architecture models, considering that the models chosen all adopt the encoder-decoder architecture of the T5 model?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer y1SZ,\n\nWe truly appreciate your constructive comments. We address your concerns as follows:\n\n> **W1.1: The upper limit of data quality is constrained by the dialogue inpainter and paraphrase model.**\n\nWe appreciate your attention to the data quality constraints imposed by the dialogue inpainter and paraphrase model. We would like to emphasize three points:\n\n1. Our dataset's pretraining has yielded notable advancements, even within these limitations. Our method, CausalDD, surpasses the strongest baseline around 10pts under few-shot and low-resource settings. This highlights the effectiveness of our dataset in driving performance enhancements beyond its inherent constraints.\n\n2. Our meticulously select state-of-the-art inpainter and paraphrase models, which have been extensively trained on diverse and large-scale datasets (e.g. MS Marco, PAQ, PublicDialog, OR-QuAC, QReCC, etc.). These models exhibit a remarkable ability to recreate contextually coherent and grammatically accurate sentences. Moreover, the data quality has already been rigorously evaluated by human experts.  Five annotators assessed the quality of 500 samples in the dataset, yielding an average score larger than 0.9 (with a maximum score of 1).  This ensures that the generated sentences align closely with natural human discourse, thereby enhancing the credibility of our dataset.\n\n3. Our dataset is thoughtfully constructed from two distinct sources, Wikipedia and Reddit. These sources complement each other, where Wikipedia consists of authentic documents and virtual dialogues, and Reddit consists of authentic dialogues and virtual documents. The bulk of the dataset is composed of authentic and original content, ensuring the dataset's richness and utility for further research.\n\n> **W1.2: Moreover, the singular nature of data sources raises concerns about their potential lack of robust generalizability.**\n1. Our dataset is thoughtfully constructed from two distinct sources: Wikipedia and Reddit. These sources collectively cover a wide array of domains, including but not limited to education, health, history, culture, society, and science. This diversity ensures a comprehensive representation of various subject areas, mitigating concerns about a singular nature and enhancing the dataset's potential for robust generalizability.\n\n2. It's worth noting that the superior performance of our method on three benchmarks (in both English and Chinese) can serve as evidence of the dataset's robustness.\n\n> **Q1: More suitable evaluation metrics should be replaced for experimentation.**\n\nWe appreciate your valuable comments and would like to address your concerns with the following points:\n\n1. In Section 4.5 of our paper, we have analyzed the reason for the marginal reduction in BLEU scores for the large CausalDD model. Our investigation reveals that larger models tend to generate more diverse responses, which may differ from the expressions of manually annotated answers. To shed light on this aspect, we have included a diversity evaluation by computing distinct scores in Appendix B6.\n\n|                | Dist-1 | Dist-2 | Dist-3 | Dist-4 |\n|----------------|:------:|:------:|:------:|:------:|\n| UniGDD         | 0.0736 | 0.3191 | 0.5055 | 0.6049 |\n| CausalDD       | 0.0736 | 0.3198 | 0.5079 | 0.6081 |\n| CausalDD_large | 0.0749 | 0.3308 | 0.5299 | 0.6347 |\n\n2. We want to emphasize that we have utilized all the metrics employed in the compared baselines to evaluate those methods. We acknowledge the limitation of BLEU score, so we further conduct human evaluation to measure the performance of causal pretraining. We randomly select 100 evaluation instances in Doc2Dial and request five human annotators to perform pairwise comparisons on two factors: (1) Relevance: indicating which response is more pertinent and relevant to the user’s inquiry, and (2) Informativeness: determining which answer is more informative.\nThe results are shown in the following table:\n\n|    　                       | 　              | Win | Tie  | Lose |\n|-----------------------------|-----------------|-----|------|------|\n|     CausalDD vs. UniGDD     | Relevance       | 42  | 55   | 3    |\n|                             | Informativeness | 43  | 47   | 10   |\n| CausalDD_large vs. CausalDD | Relevance       | 45  | 50   | 5    |\n|                             | Informativeness | 48  | 43   | 9    |\n\n> **Q2. Any attempts to experiment with popular decoder-only architecture models?**\n\n1. We directly adopt T5 as the backbone model following our strongest baseline UniGDD for the DocGD task. Moreover, we also compare other encoder-decoder model like BART and pipeline-manner methods (identifying evidence using BERT first and using the evidence as the input to generate responses).\n\n2. Kindly note that our proposed causal pretraining is model-agnostic. Given resource limitations, we will explore decoder-only large language models like LLAMA as future work, leveraging their capabilities to enhance the generalization of our causal pretraining.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper endeavors to explore DocGD from a causal perspective, presenting the causal relationships among DocGD variables for the first time.\n2. This paper present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora. \n3. This paper also contributes a causally-complete DocGD pre-training dataset constructed from Wikipedia and Reddit.", "reasons_to_reject": "1. The upper limit of data quality is constrained by the dialogue inpainter and paraphrase model. Moreover, the singular nature of data sources raises concerns about its potential lack of robust generalizability.", "questions_to_authors": "1. In Table 3, a marginal reduction in performance is observed for Large CausalDD. Does this observation potentially imply that the BLEU metric inadequately assesses the model's proficiency? I think that some more suitable evaluation metrics should be replaced for experimentation.\n2. Have there been attempts to experiment with popular decoder-only architecture models, considering that the models chosen all adopt the encoder-decoder architecture of the T5 model?\n", "paper_topic_and_main_contributions": "1. This paper present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora.\n2. Additionally, it propose a causally-perturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect.\n3. The causal pre-training is experimented on three benchmark datasets, yielding varying degrees of improvement across fully-supervised, low-resource, few-shot, and zero-shot settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kN6vAfxzsx", "reviewer": "Reviewer_oY7u", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oY7u,\n\nWe truly appreciate your valuable and constructive comments. We address your concerns as follows:\n\n> **W1: The paper needs more evidence to explain how the pre-training works for Document-Grounded Dialogue task, specifically.**\n\n1. In the realm of DocGD, the common practice involves initial general pre-training followed by fine-tuning on downstream datasets. Our approach, CausalDD, introduces a distinctive second-phase pre-training paradigm. It bridges the gap between general pre-training and downstream fine-tuning. In contrast to UniGDD's direct fine-tuning of a T5 model pre-trained on general tasks, CausalDD's intermediary pre-training phase has demonstrated significant enhancements. This is particularly noteworthy in scenarios with few-shot, low resource, and even zero-shot settings, where the improvements are striking.\n\n2. To further verify the effectiveness of our pre-training approach, Section 4.8 of the paper presents dedicated analyses of the benefits brought about by our causal pre-training (causalDD): (1) faster convergence speed compared to general pretraining; (2) better modeling of dialogue history: we find better performance across all turns when we divided the Doc2bot test set based on the number of turns in the dialog history in Figure 4(b); (3) a better ability to ground complex evidence: as the number of relevant evidence increases, CausalDD shows better performance compared to baselines.\n\n> **W2: Authors might need to make a comparison with other existing pre-training methods for the Document-Grounded Dialogue task and different formulations of unsupervised causal datasets.**\n\n1. Kindly note that our work presents the first effort in proposing pre-training tailored for DocGD, distinct from existing studies merely about downstream fine-tuning. We have conducted a thorough comparison with the strongest baseline UniGDD, which utilizes the general pre-training checkpoint as the initialization. Our approach consistently outperforms the baselines across three downstream datasets and various resource settings, underscoring its efficacy.\n\n2. Our work endeavors to explore DocGD from a causal perspective, presenting the causal relationships among DocGD variables for the first time. DocGD task contains four variables: causes (document, evidence, dialogue context) and effect (response). Constructing causally-complete datasets should consider all these variables, which leads to a supervised pretraining corpus.\n\n3. We will explore unsupervised causal datasets and design new pretraining objectives for future work. We thank you for this valuable suggestion, which will undoubtedly contribute to the broader applicability of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation for creating causally-complete datasets and causally-perturbed pre-training are strong and has novelty.\n2. Achieves outperformed performance on 3 benchmarks and even better than GPT-3.5 on two of them.\n3. It makes a thorough study in different scenarios and demonstrates effectiveness of causally-perturbed pre-training.", "reasons_to_reject": "1. The paper needs more evidence to explain how the pre-training works for Document-Grounded Dialogue task, specifically.\n2. Authors might need to make comparison with other existing pre-training methods for Document-Grounded Dialogue task and different formulations of unsupervised causal datasets. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel way to create causally-complete datasets from Wikipedia and Reddit to address data scarcity problem in pre-training and causally-perturbed pre-training in addition to optimize the total effect of responses for different causes for Document-Grounded Dialogue task. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. L92: twofold -> two-fold"}, "extra_text": {}}, {"review_id": "zqhFDhf6a1", "reviewer": "Reviewer_DA1s", "summary": "", "questions": "How about performance on GPT-4? Also, what is the GPT-3.5 version used here?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DA1s,\n\nWe truly appreciate your constructive comments. We address your concerns as follows:\n\n> **W1: For causality, the effects of all external factors are not measured.**\n\nWe would like to address your concern from two perspectives:\n\n1. From a causal perspective of the DocGD task, there are four variables involved: dialogue context, document, evidence (causes), and response (effect). To assess causality, we perform perturbations on both a direct variable (document) and an indirect variable (evidence) to measure their impact on the response. By perturbing the document, we aim to enhance model robustness, while perturbing the evidence aims to reinforce the model's reliance on evidence. This multi-pronged approach enables a more holistic evaluation of the effects of external factors on the causal relationships within DocGD. Moreover, we also measure the effects of the Reddit dataset for causal pre-training.\n\n2. The ablation study in Table 8 of the paper also verifies how variations in the causal variables influence the generated responses:\n\n    (1) Introducing a causally-complete Reddit containing real-world dialogues enhances the ability of the model to identify knowledge and generate better responses; \n\n    (2) Perturbing document to enhance the consistency of the model outputs enhances the robustness of the model; \n\n    (3) Perturbing evidence by removing it from documents increases the model’s reliance on the grounding evidence. \n\nThese results validate that each component has a positive effect on CausalDD, leading to its better capability of modeling causal relationships among DocGD variables.\n\n> **Q1: How about performance on GPT-4? Also, what is the GPT-3.5 version used here?**\n\nThe GPT-3.5 version we used is gpt-3.5-turbo. Considering the budget constraints, we evaluate the performance of GPT-4 on 200 instances of Doc2dial and provide the results of 200 samples in the following table:\n\n|               |  EM  |   F1  | BLEU |\n|---------------|:----:|:-----:|:----:|\n| GPT-3.5-turbo | 51.0 | 61.94 | 3.02 |\n| GPT-4         | 77.0 | 83.03 | 5.32 |\n| CausalDD      | 70.3 |  80.5 | 42.8 |\n\nHere are our analyzes:\n\n1. We notice that GPT-4 improves GPT-3.5 by a large margin regarding the evidence identification task in DocGD.\n\n2. While GPT-4 does demonstrate improved performance in terms of EM and F1 metrics, it's important to note that our proposed causal pretraining (CausalDD) achieves competitive results with comparatively smaller parameters, using models T5-200M and T5-large 770M. This efficient utilization of parameters underscores the efficacy of our method, especially in resource-constrained scenarios.\n\n3. The BLEU score is not a good evaluation metric to measure the performance provided by large language models. Here are two reasons: (1) We need to carefully design the prompt to meet the output format and explain the role that ChatGPT plays. (2) Usually, responses generated by ChatGPT are more informative than the ground truth. However, the BLEU score only measures the overlap of n-grams between the generated predictions and the references.\n\nIf you have any further questions, please feel free to raise them directly. We are committed to addressing all your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They proposed causality-based method for document-ground response generation. \n2. A large scale pre-training dataset with causal relation is also proposed.\n", "reasons_to_reject": "1. For causality, the effects of all external factors are not measured.", "questions_to_authors": "How about performance on GPT-4? Also, what is the GPT-3.5 version used here?", "paper_topic_and_main_contributions": "The authors propose a causally-complete dataset to develop a pre-training scale corpora. Then they further designed a causally-perturbed pre-training method to capture the causality within the variables. They demonstrate the methods with extensive experiments", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the provided reviews, there are several positive aspects of the paper. Review 1 highlights the contribution of exploring DocGD from a causal perspective, presenting the causal relationships among variables for the first time. Additionally, the paper proposes a causally-complete dataset construction strategy and a causally-perturbed pre-training strategy. Review 2 also emphasizes the novelty of the proposed methods for creating causally-complete datasets and causally-perturbed pre-training. The paper achieves outperformed results on benchmark datasets and demonstrates effectiveness in different scenarios.

However, Review 1 mentions concerns about the data quality being constrained by the dialogue inpainter and paraphrase model, as well as potential lack of robust generalizability due to the singular nature of data sources. Review 2 suggests that the paper needs more evidence to explain how the pre-training works for Document-Grounded Dialogue task and suggests comparisons with other existing pre-training methods.

Overall, the soundness and excitement scores from the three reviews range from 3 to 4, indicating the paper provides sufficient support for its major claims/arguments, has some merits, and reports state-of-the-art results. However, there are key weaknesses and areas for improvement, such as providing more evidence and comparisons with other methods.

The contributions and strong motivation of the paper make it a valuable addition to the field. However, additional evidence and clearer explanations are needed to address concerns and improve the clarity of the paper.
Decision: Accept-Main