Follow the Knowledge: Structural Biases and Artefacts in Knowledge Grounded Dialog Datasets
Year: 2023

Abstract:
Crowd-sourcing has been one of the primary ways to curate conversational data, specially for certain scenarios like grounding in knowledge. In this setting, using online platforms like AMT, non-expert participants are hired to converse with each other, following instructions which try to guide the outcome towards the desired format. The resulting data then is used for different parts of dialog modelling like knowledge selection and response selection/generation. In this work, we take a closer look into two of the most popular knowledge grounded dialog (KGD) datasets. Investigating potential biases and artefacts in knowledge selection labels, we observe that in many cases the ‘knowledge selection flow’ simply follows the order of presented knowledge pieces. In Wizard of Wikipedia (the most popular KGD dataset) we use simple content-agnostic models based on this bias to get significant knowledge selection performance. In Topical-Chat we see a similar correlation between the knowledge selection sequence and the order of entities and their segments, as provided to crowd-source workers. We believe that the observed results, question the significance and origin of the presumed dialog-level attributes like ‘knowledge flow’ in these crowd-sourced datasets.

heading: 1 Introduction
text: Since the introduction of data hungry methods into dialog modeling, sizeable datasets have become an essential asset for researchers of the field. While generic conversational data can be harvested in large quantities from already existing resources like movie subtitles (Lison and Tiedemann, 2016) or website forums (Lowe et al., 2015), more specific datasets usually need to be curated under supervision. Knowledge grounded dialog is one of the fields that has remarkably benefited from crowdsourced datasets like Wizard of Wikipedia or WoW (Dinan et al., 2018), Holl-E (Moghe et al., 2018)
and Topical-Chat (Gopalakrishnan et al., 2019), which offer grounded utterances generated by nonexpert annotators or Turkers. During the curation, participants are commonly asked to first choose a knowledge piece (or no knowledge) from a provided pool and then use the selected piece to ground their next utterance on.
One common attribute in these datasets is the sizeable difference between the knowledge selection performance with and without access to the uttered response. The phenomenon –referred to as the prior-posterior gap– is demonstrated in Table 1 (from Kim et al. (2020)) for the WoW dataset. Looking for ways to improve the prior performance, studies have tried to design methods to capture higher-order patterns beyond the limiting (and seemingly insufficient) turn-level scope. A natural candidate for this, is modeling the ‘knowledge flow’; i.e. how the history of knowledge selection affects the next selection.
In this work we investigate the potential spurious origins of ‘knowledge flow’ in crowd-sourced KGD datasets. Focusing on the most popular resources in the field, i.e. WoW, we show that competitive results can be obtained in the knowledge selection task, using very simple structural heuristics. We also show that these rudimentary patterns are not an isolated case and can be found in other knowledge grounded dialog datasets like Topical-
109
Chat, which –in our opinion– connects knowledge selection patterns to dataset curation choices and design.
While dataset artifacts and their relation to the curation process have been widely studied in NLU tasks and especially NLI (Nangia et al., 2021; Gururangan et al., 2018), it is an under-studied topic in dialog modeling. We hope our work draws attention to the issue and contributes to having better dialog datasets, which we believe is necessary for properly modeling higher-order dialog attributes.

heading: 2 Knowledge Grounded Conversation

heading: 2.1 Problem Formulation
text: In general, the question of knowledge grounded dialog (KGD) modelling is defined over dialog and knowledge datasets Dd = {(Ci, ri)}Ni=1 and Dk = {(kj)}Mj=1 where ∀i ∈ {1, ..., N}, Ci and ri represent context and response for a specific dialog turn, and ∀j ∈ {1, ...,M}, kj is a knowledge piece (e.g. a sentence or paragraph). In most recent datasets, Dd and Dk are provided as parallel, which allows for a simpler formalization over D = {(Ci,Ki, ri)}Ni=1, where Ki (or knowledge pool) is a subset of Dk, and often includes one or more ‘gold truth’ (KGi ), i.e. the knowledge piece(s) picked by the annotator during data curation.
The problem of knowledge selection (KS) in this context means designing a model fks to identify the relevant knowledge piece(s) in Ki: fks(Ki) = KGi . Ideally fks provides a ranking over Ki which can be used to retrieve top-k results for response generation.

heading: 2.2 Popular Datasets
text: The problem of modeling open-domain knowledge grounded conversation attracted increasing attention since the introduction and release of large scale crowd-sourced knowledge grounded dialog datasets with parallel dialog and knowledge corpora. Table 2 shows selected details of the four
most popular KGD datasets: Wizard of Wikipedia (Dinan et al., 2018) includes conversations between a wizard (with access to knowledge) and an apprentice (no knowledge access) grounded on Wikipedia articles. CMU_DoG (Zhou et al., 2018) and Holl-E (Moghe et al., 2018) contain dialogs about movies grounded on Wikipedia information plus descriptions for 3 key scenes (CMU_DoG) or a selection of movie’s plot, reviews, comments and facts (Holl-E). Finally Topical-Chat (Gopalakrishnan et al., 2019) includes conversations on various ‘entities‘ and grounded in a combination of Wikipedia article, fun facts and news articles, in both symmetric and asymmetric knowledge access scenarios.
Among these, WoW is by far The most cited dataset in the field which can be attributed to qualities like proper size, gold knowledge labels and multi-topic knowledge pool. It is also the only dataset with dynamic pool, meaning that the knowledge choices are updated at each turn. TopicalChat is another popular resource which creates distinction with pre-defined scenarios for knowledge access between collocutors. However it only provides section-level (and not sentence-level) labels for knowledge selection, which makes it less convenient for supervised knowledge selection modeling.

heading: 2.3 Knowledge Selection Methods
text: The popular approach of breaking the KGD problem into the knowledge selection (KS) and response generation (RG) tasks, became mainstream with WoW. Along with the dataset, the release paper (Dinan et al., 2018) also proposed a baseline model (Transformer MemNet) which addressed KGD in these two steps, acquiring 22.5% and 12.2% accuracy for knowledge selection on the seen and unseen test sets accordingly1.
One of the first approaches to improve on this,
1In the seen set -unlike the unseen- dialog ‘topics’ are shared with the training set.
was addressing and exploiting the prior-posterior gap, which uses the posterior knowledge distribution to provide additional learning signals for the KS module, usually via a KL-divergence loss (Lian et al., 2019; Chen et al., 2020; Zhan et al., 2021a; Lotfi et al., 2021).
But probably the most popular approach is trying to address the problem on the dialog level (rather than turn level), and model higher-order ‘flows’ or sequential patterns that could guide the knowledge selection process. Li et al. (2019) used an incremental transformer to incorporate the knowledge selection history. Jiang et al. (2020) enhanced the posterior signal by modeling the ‘topic drift’. Kim et al. (2020) introduced sequential latent knowledge selection to incorporate the selection history. Zheng et al. (2020) took a more specific approach by providing a positive bias for new or different knowledge choices. Meng et al. (2020) explicitly modeled ‘knowledge tracking’ and ‘knowledge shifting’ during a conversation while Meng et al. (2021) tried to incorporate speakers’ initiative. Zhan et al. (2021b) used conditional random fields to model knowledge transition and Wu et al. (2021) leveraged the document structure to provide dialog-contextualized passage encodings while adding an auxiliary loss to capture the history of dialog-document connections. Li et al. (2022) used document semantic graphs to guide the knowledge selection, and Yang et al. (2022) proposed a
topic-shift aware knowledge selector. More recently models like RAG (Lewis et al., 2020) and FID (Kim et al., 2020) improved the question answering performance by shifting the final knowledge selection to the decoding process. Extending this to dialog (which was implemented differently by Lin et al. (2020)), studies have incorporated the fine-grained decoding-stage selection for better knowledge grounding (Shuster et al., 2021) or combined it with the posterior signal (Paranjape et al., 2021).
Table 3 summarizes a selection of these approaches, which mostly try to incorporate dynamic knowledge patterns by modeling attributes like topic shift, knowledge transition, knowledge tracking/shifting, knowledge difference etc.

heading: 3 Knowledge Selection Biases and Artefacts
text: Our main objective in this work is to explore the structural biases and artefacts in the knowledge selection labels of popular KGD datasets. For this, we use different methods depending on the way knowledge pools are constructed and presented to crowd-source workers, but in both cases, we essentially investigate the same hypothesis:
Crowd-source workers often base their knowledge selection on the structure and order of the knowledge pool, as presented to them.
In other words, when selecting the knowledge piece for the next utterance, they tend to just follow the knowledge document and pick the ‘next’ item, instead of coming up with a more sophisticated ‘flow’. In the following sections, we explore this hypothesis separately for Wizard of Wikipedia and Topical-Chat.

heading: 3.1 Wizard of Wikipedia (WoW)
text: As mentioned before, in WoW, dialogs happen between a ‘wizard’ and ‘apprentice’, with the former having access to unstructured knowledge. For each dialog either the wizard or apprentice is picked to choose the topic and speak first (the other player receives the topic information). The conversation begins and at each turn the wizard (system) can select from a knowledge pool which has been curated from a collection of Wikipedia articles via basic retrieval methods. Then the (potentially) selected knowledge piece is used by the wizard to generate the next utterance (system response). Out of 83247 wizard turns in the training set, 77523 (93%) are ‘knowledge-grounded’; i.e. turns where the annotator has chosen a knowledge piece to ground their next utterance on.
Figure 1 shows how the knowledge pool is created for the wizard: At each turn, the last two utterances are used as queries by a TF-IDF retrieval module to get 14 (7 for each) relevant articles from a Wikipedia collection (title + first paragraphs). The dialog-topic article (title + first 10 sentences) is added to this set to create the final pool, which on average contains 63 knowledge sentences from up to 15 passages or articles2. Figure 5 (Appendix A) shows how this pool is presented to annotators.
To investigate our hypothesis, we consider 3 heuristic content-agnostic models for knowledge selection:
• Topic-First (T0): Picks the first sentence of the dialog-topic article in all turns.
• Topic-Next (T+): Starts from the dialog-topic article’s first sentence, but proceeds to the next sentence at each successive turn.
• Last-Next (L+): Picks the next sentence in the (gold) passage that was selected in the previous turn3.
2Since each passage corresponds to an article with a unique topic, passage, article and topic can be used interchangeably in the WoW context.
3If not available, the model returns the next (unused) sen-
T0 is a static (‘flow-less’) model. T+ precisely and strictly follows the topic-article’s narrative for dialog grounding, and L+ exploits the knowledge selection history for the next move.
Table 4 shows the performance of these 3 models in selecting the gold passage and sentence from the knowledge pool. T0 does not score very high but it offers a strong baseline for knowledge selection in WoW. In particular the T0 performance on the unseen test set already beats a handful of the models in Table 3 including the original baseline (18.9 vs. 12.2). Adding the basic ‘flow’ (T+) significantly improves the KS performance (an additional ~6% accuracy), and following the L+ selection policy adds another 5% boost to accuracy, making the content-agnostic L+ model highly competitive among the KS models. These performances show a strong bias towards picking the dialog-topic article among the passages, as well as picking the ‘next’ sentence within the current passage.

heading: 3.2 Topical-Chat
text: Unlike WoW, in Topical-Chat (Gopalakrishnan et al., 2019) the partners do not have explicitly defined roles. Instead, the authors leveraged infor-
tence in the dialog-topic article (e.g. in the first turn, the dialog-topic article’s first sentence will be selected.)
mation asymmetry to implicitly cause both partners to serve dual roles of a teacher and a participant which more accurately reflects real-world conversations.
For each dialog, 3 entities (from a pool of 300) were picked, plus their Wikipedia lead section, 8- 10 fun facts, and a news article referencing all 3. These resources then were divided or modified according to one of 4 configurations (Figure 6 in Appendix B) to provide 2 identical (conf. A and B) or different (conf. C and D) knowledge pools. Finally Mechanical Turk workers were partnered up and assigned to these reading sets, and asked to chat about them for at least 20 turns. To present the reading set, information about an entity E (i.e. Wikipedia sections and fun facts) were displayed as a group titled Factual Section (FS), and the news article about the entities was chunked into 4 similar-sized sections (AS1-4). Turkers were asked to specify the knowledge source (FS1-3, AS1-4 and/or Personal Knowledge (PK)) used to generate their message at each turn. Selecting Personal Knowledge as the source means that the utterance is not grounded in external knowledge.
Figure 2 shows how the grounding evolves as
conversations proceed in TopicalChat. Knowledge sections are arranged along the Y-axis, and point sizes represent normalized (per turn) frequency, or density. We can see that:
1. Grounding is mainly done on Factual Sections (FS), rather than Article Sections (AS).
2. The first part of the Article (AS1) is used significantly more than the rest for grounding, mainly in the beginning of the conversation4.
3. As the conversation proceeds, the grounding density peak moves from FS1 to FS2 and FS3.
4. Personal Knowledge has a higher density in the beginning and ending turns which agrees with greeting patterns.
In other words, the ‘average’ TopicalChat conversation is likely to follow the PK-AS1-FS1-FS2FS3-PK pattern for grounding. Since the numbering of entities and corresponding Factual Sections (i.e. FS1-3) in each conversation is independent
4This usually corresponds to opening utterances like Do you know/Have you heard about X?
of the Article content5, the observed pattern suggests that the grounding order is biased towards a pre-determined arbitrary parameter.
To have a more fine-grained view of this pattern, we exploit the response-knowledge overlap and employ a pre-trained sentence embedding model to estimate the gold knowledge sentence within the gold section (details in Appendix B). We then use these labels to expand each FS section into FSwiki (the Wikipedia part) and FS-ff-{0-4} (the 5 fun facts). Figure 3 shows the resulting chart (limited to AS1, FS1, FS2 and FS3 sections), which demonstrates the same overall tendency of ground-
5As opposed to –for example– numbered by importance or coverage order in the supporting Article.
ing on later sentences/sections as the conversation proceeds. The straight line is the linear regression fit (assuming sections’ order as their value; i.e. 1, 2, ..., 19) with the slope and r2 value of 0.41 and 0.19 respectively (The slope of the diagonal line is 0.95).
The proceeding pattern is better illustrated in Ridgeline plots. For this, we switch the axes and invert the Y-axis direction so that the conversation starts at the top of the Y-axis. The result (Figure 4) displays a dispersing distribution with a clear tendency to ‘move’ forward; i.e. towards later sentences/sections. As Figure 7 (Appendix B) shows, this is shared in different knowledge configurations (i.e. A, B, C, D in Figure 6) with slight fluctuations.

heading: 3.3 WoW: A Markov View
text: As the final experiment, we revisit WoW dataset with a more complicated knowledge selection strategy. Inspired by the noticeable performance of content-agnostic models, we now consider a Markov model for KS with two stochastic parameters; Passage (P ), and Line (L), with the following domains:
P ∈ {None, DTopic, Other} L ∈ {None, Next, Other} (1)
P can be None (no grounding), DTopic (grounding on the dialog-topic passage) or Other (grounding on any other passage). Similarly, L can be None (no grounding), Next (picking the next sentence in the selected passage)6, or Other (picking any other sentence. These choices follow the observed biases towards picking the dialog-topic article among the passages, and the ‘next’ sentence within the current passage, as discussed in 3.1.
Using this model, we can calculate initial state and transition probabilities from the WoW training set. Table 5 shows P0 and L0 probabilities for the initial state (S0); i.e first turn. 0/0 and 0/1 refer to the first turns in which wizard (0/0) or apprentice (0/1) start the conversation. As one can see, there is a strong bias towards picking the dialog-topic passage (DTopic) which is expected, especially for 0/0 where DTopic is the only grounding choice. More interesting is the tendency to start from the first sentence, especially when DTopic is chosen as passage (random probability: ~0.1).
Table 6 shows the transition probabilities for Passage (P ) and Line (L) between successive states
6Here Next is meant with respect to the grounding history; i.e. picking up from the last time the passage was visited. In the case of no grounding memory (first-order Markov), L starts from 0 every time the grounding topic changes.
(full grounding memory), which demonstrates a strong tendency to ‘stay’ in DTopic (~0.75) and an overall preference for picking the Next sentence ((random probability: ~0.19).
Equation 1 along with the P0, L0, P and L values provides a fine-grained content-agnostic distribution (CAG) over the knowledge choices at each turn, which can be used in combination with any content-aware (CAW) KS model. Here we examine three ways to do so (all CAW models are based on RoBERTa-base):
• Ensemble: We simply use the CAG predictions in a mean-value ensemble.
• TokenCues: Instead of directly incorporating the CAG values, we provide corresponding bias cues as special tokens in the input sequences. In particular we add <topic>, <next> and <prev_next> to respectively mark the topic-article sentences, the successive sentence in each passage (w.r.t the last visited one in that passage) and the sentence after the one selected in the previous turn.
• Both: We use the token-cues model in combination with CAG in a mean-value ensemble.
Table 7 shows the KS performance of these variations compared with the conventional sequence classification approach (Baseline). As one can see,
incorporating the content-agnostic knowledge (directly or indirectly), results in a significant performance improvement. Moreover it seems that the transformer model is capable of learning the KS biases once proper cues are provided in the training data: the TokenCues model matches the Baseline Ensemble while gaining only marginal improvement from the explicit CAG values.

heading: 4 Discussion and Conclusion
text: In this work we investigated the potential knowledge selection biases and artefacts in two popular KGD datasets. Our central governing hypothesis was that crowd-source workers tend to simply follow the structure and order of knowledge pieces, as presented to them. For the WoW dataset, we showed that using this hypothesis, content-agnostic models can achieve noticeable knowledge selection performance, and combined with simple sequence classification training are able to compete with sophisticated solutions. For Topical-Chat we observed a noisy alignment between the KS sequence and the order of entities and their segments, as provided to crowd-sources.
Although following the existing order of knowledge pieces is not strange or unexpected (at least within one document), we believe that the way knowledge options are curated and presented to crowd-source workers can be an exacerbating factor. All 4 datasets provide a large number of retrieved knowledge pieces at each turn (usually more than 60) which is statistically beneficial to the dataset, but it could also encourage an ‘easy solution’ regime in which annotators opt for the safe and convenient choice of following the already existing structure of knowledge articles, instead of trying to create and maintain a novel ‘flow’. In its extreme case, this leads to conversations similar to reciting an article line by line7.
In terms of dialog modeling, these results can suggest that the origin and significance of higherorder attributes in the dataset can be questioned. In particular, the concept of ‘flow’ as governing the dialog-level pattern of knowledge selection seems to be rooted substantially in the structure of knowledge documents. This does not rule out the ex-
7There are also case-specific factors. for example in WoW the utterance-based knowledge pieces are subject to change at each turn, and therefore there is no guarantee that the passage used for grounding in the current turn will be present in the provided pool for the next turn. This makes grounding on the dialog-topic article a safe choice, since it is always in the pool.
istence or learnability of genuine patterns/flows, but the very low human performance for this task (~17%; Table 1) imposes a serious higher-bound on its discerning power; i.e. in most cases, there seems to be not enough semantic cues in the conversational history to uniquely and clearly bound it to a single knowledge piece.
Although the ultimate goal in KGD modeling is generating proper responses (and not mastering the knowledge selection part), but in order to model higher-order and dialog-level conversational phenomena, we probably need better datasets. One important factor in producing such resources is considering the process from annotators’ point of view, and how design choices (e.g. annotation interface and instructions, size of knowledge pool, etc.) can persuade them towards or away from ‘easy solution’ regimes which are prone to artifacts. Another approach is providing explicit ‘scenarios’ for the way dialogs are supposed to unfold. This is how DuConv (Wu et al., 2019) and NaturalConv (Wang et al., 2021) datasets (both Chinese) have been curated, but whether this mitigates the problem or introduces new artifacts should be studied.

heading: 5 Limitations
text: The main limitation of our work is its focus on English datasets. While this was due to their popularity and extensive usage (and our limited language skills), it overlooks datasets like DuConv (Wu et al., 2019) and NaturalConv (Wang et al., 2021) (both Chinese) which employ more explicit annotation instructions regarding dialog ‘path’ and topic transitions. Studying the way these restrictions affect conversational attributes, is necessary for a more comprehensive understanding of the problem.
Another limitation is the lack of an empirical investigation on how/if these artefacts and biases affect the final objective of KGD modeling, i.e. response generation. This of course is not easy in the absence of a less biased dataset, but synthetic datasets –which have become much better in quality and flexibility thanks to large language models– can probably provide reliable estimations, which we plan to explore in future studies.

heading: Acknowledgements
text: This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

heading: A Appendix: WoW Interface
text: Figure 5 shows the annotation interface used in curating Wizard of Wikipedia.

heading: B Appendix: Topical-Chat
text: Considering the absence of sentence-level gold labels in Topical-Chat, we exploit the responseknowledge overlap and employ a pre-trained sentence embedding model to estimate the gold knowledge sentence within the gold section. More specifically, we use the all-mpnet-base-v2 model from the ‘sentence-transformers’ library (Reimers and Gurevych, 2019) which shows the highest performance on benchmarks, and pick the sentence which has the highest cosine similarity with the response. Manually checking the performance on a small subset (500 grounded samples) shows an error rate of 18% (accuracy =~82%) of which 10% is due to incorrect gold section labels. Enforcing an acceptance similarity threshold of 0.2, filters out 13% of samples including 88% of errors, which improves the accuracy to 93%. We apply this setting to the train set8, and –to keep conversations in reasonable lengths (and therefore less likely to be damaged by the filtering)–, we remove dialogs with less than 80% of accepted utterances. This, results in a more reliable subset of 7922 (out of 8,628) conversations, with 150564 utterances.
8We consider the first 20 utterances in each dialog, which is the minimum required length during crowd-sourcing.

heading: Future Work
text: This of course is not easy in the absence of a less biased dataset, but synthetic datasets –which have become much better in quality and flexibility thanks to large language models– can probably provide reliable estimations, which we plan to explore in future studies