SlowGuess/NAACL_2021_OCR huggingface dataset:
The presentation of this dialogue is divided over the two following subsections; the first gives the warm-up necessary for the second. Subsection 4.1 illustrates how positive evidence of understanding is provided, and no examples of CRs are presented here. Subsection 4.2's goal, on the other hand, is to illustrate CRs in different modalities, so here we focus on negative evidence.
4.1 Positive evidence
At the beginning of this dialogue, the DG is instructing the DF to find the rebreather. As part of this task, they have to press a button in order to open a door as shown in Figure 1. The figure shows a dialogue fragment and a screenshot of the shared view when the fragment starts. The turns which provide positive evidence at levels 3 and 4
$DG(1)$ : see that button straight ahead of you?
DF(2): mhm
DG(3): hit that one
$DF(4)$  :ok
Figure 1: Example of the view shared by the dialogue participants and fragment from the SCARE corpus
are shown in boldface. If evidence for proposal is followed by a turn that is not evidence of uptake (of the proposal) then we say that the turn is a CR.
The dialogue fragment reproduced below starts when the DG is trying to get the DF to press the button that is straight ahead in their current view; this button opens the cabinet where the rebreather is located. As part of this project, the DG first makes sure that the DF identifies this button using the sub-dialogue constituted by (1) and (2). Once the button is identified, the short instruction in (3) suffices to convey the goal of the joint project, namely hitting this button; this is acknowledged at level 4 in turn (4) when the DF presses the button.
4.2 Negative evidence
Now we turn to an extended example, extracted from the SCARE corpus, of clarification requests at different levels. Between square brackets we indicate forms of non-linguistic communication. The DG utters an instruction in (1). In turn (2) the DF makes explicit an obstacle at level 3 that must be solved before putting the rebreather in the cabinet, namely identifying cabinet 9; in doing so he proposes this task. In turn (3) the DG proposes to identify cabinet 9 by first identifying its location. Turn (4) is evidence of uptake of turn (3) — the DG answers his own question — but it is also evidence of the proposal: get back to the starting room.
$DG(1)$  : we have to put it in cabinet nine [pause]  $DF(2)$  : yeah they're not numbered [laughs]
DG(3): [laughs] where is cabinet nine
DG(4): it's kinda like back where you started so
DF(5): ok so I have to go back through here?
$DG(6)$  : yeah
DF(7): and around the corner?
DG(8): right
DF(9): and then do I have to go back up the steps?
DG(10): yeah
DF(11): alright this is where we started
DG(12): ok so your left ca-[pause] the left one
DF(13): so how do I open it?
DF(14): one of the buttons?
DG(15): yeah, it's the left one
DF(16): makes sense
DF(17): alright so we put it in cabinet nine
Of the 17 turns, 9 were uttered by the DF and 8 by the DG. From the 9 turns by the DF, 5 of them are CRs at level 4 and one at level 3. Turn (2) is a CR of instruction (1). Turns (5), (7) and (9) are CRs of instruction (4). Utterance (11) shows positive evidence at level 4 of instruction (4) so this instruction cannot be further clarified following the recipe we defined in Section 3. Turns (13) and (14) are CRs of utterance (12). The positive evidence at level 4 of instruction (12) is completed by a physical action of the DF in the game world: opening the cabinet by pressing the left button while uttering (16). Finally, turn (17) together with the corresponding physical action are positive evidence at level 4 of instruction (1).
5 Comparative analysis of clarifications
In this section, we identify and discuss a number of pressures that interact in order to determine the number and type of CRs that occur in dialogue; we also explain why it makes sense (although it may seem counter-intuitive at first sight) that too much uncertainty will tend to lower the number of CRs.
The distribution and types of CRs found in a corpus depend on the characteristics of the task that the dialogues in the corpus are addressing. Previous clarification corpus studies (Purver, 2004; Rieser and Moore, 2005; Rodriguez and Schlangen, 2004) have required expensive and detailed annotations by linguists who also evaluated the quality of the datasets. Purver (2004) annotates more than 10K turns of the BNC corpus, which contains English dialogue transcriptions of topics of general interest in multiparty dialogue such as meetings. These annotations were used to build a dialogue system that could make and understand relevant clarifications related to different modalities (Purver, 2006). (Rieser and Moore, 2005) and (Rodriguez
and Schlangen, 2004) did similar annotations on task-oriented dialogue corpora. (Rieser and Moore, 2005) looked for CRs in a corpus of English task-oriented human-human dialogue called Communicator. The corpus consists of travel reservation dialogues between a client a travel agent. The interactions occur by phone; the participants do not have a shared view of the task. The corpus comprises 31 dialogues of 67 turns each (on average), from which  $4.6\%$  of the turns are CRs.  $12\%$  of CRs found were classified as level 4 CRs; such as the following:
Client: You know what the conference might be downtown Seattle so I may have to call you back on that.
Agent: Okay. Did you want me to wait for the hotel then?
In this corpus the world validation is informational not physical as in the Bielefeld data that we turn to now.
(Rodriguez and Schlangen, 2004) looked for CRs in a corpus of German task-oriented human-human dialogue called Bielefeld. The dialogues occur in a instruction giving task for building a model plane. The interactions occur face to face; the participants have a shared-view of the task. The corpus consists of 22 dialogues, with 180 turns each (on average), from which  $5.8\%$  of the turns are CRs.  $22\%$  of CRs found were classified as level 4 CRs, such as the following:
DG: Turn it on.
DF: By pushing the red button?
We analyzed the SCARE corpus while watching the associated videos and we classified the clarification requests according to the levels of communication using the decision procedure explained in Section 3.7 We found that  $6.5\%$  of the turns are CRs. Of these,  $65\%$  belong to level 4 of Table 2, and  $31\%$  belong to level 3 (most of them related to reference resolution). Only  $2\%$  of the CRs were acoustic (level 2) since the channel used was very reliable, and another  $2\%$  had to do with establishing contact (level 1).
The SCARE corpus presents slightly more CRs (at  $6.5\%$  ) than the corpora analyzed in previous work (which reported that  $4\% - 6\%$  of the dialogue turns were CRs). Furthermore, in contrast to the
<sup>7</sup>We will release our annotations to the research community upon request.
BNC corpus study (Purver, 2004), most CRs in the SCARE corpus occurred at level 4. What task characteristics might have caused the observed differences?
We hypothesize that the following six characteristics account for the larger proportion of CRs at level 4 that we find in the SCARE corpus. Task oriented dialogues (unlike general interest dialogues) are constrained by the task, thus the hearer may have a better hypothesis of what the problem is with the source utterance. He also has a clear motivation for asking for clarifications when the utterance does not fit his model of the task. Dialogues situated in an instruction giving task show an asymmetry between the knowledge that the dialogue participants (DPs) have about the task. The Direction Giver (DG) knows how the task has to be done and the Direction Follower (DF) doesn't. Hence, it is to be expected that the DF will have doubts about the task which (both DPs know) can only be answered by the DG. In symmetric dialogues, it might not be clear who has what information and then the DPs might not know who can answer the CRs. Immediate world validation seems to play a role as well. Dialogues that interleave linguistic actions and informational or physical actions exhibit immediate world validation of the interpretations. If an instruction fails in the world, the DF will ask for clarification. When the DPs have a shared view of the task, the DP that is acting on the world knows that the other participant is observing him and verifying his actions and then will try to be sure of what he has to do before doing it. If he is not sure he will ask. Long dialogues tend to increase the percentage of clarifications (more than 100 turns) because DPs prefer to ask questions when they have a good hypothesis to offer. The longer the interaction, the more background is shared by the DPs and the easier it will be to come up with a good hypothesis. Finally, if there are actions in some modality that are irreversible, then they will clarify more until they are sure of what they have to do.
6 Discussion and objections
Humans switch between clarifications grounded in different modalities seamlessly and we have argued they do so systematically; in effect they do so by following a recipe for grounding classifications. We obtained this recipe by granting a role to both perceptual and collaborative grounding in clarification requests. This we did by examining
Clark (1996)'s action ladder of communication and Ginzburg, Purver and colleagues (2012)'s classification of clarification phenomena, and combining the concept of level taken from the ladder of communication with Gabsdil (2003)'s test for clarification requests. We reframed Clark's downward evidence and upwards completion properties for multimodal interactions.
This gave us the following: given an utterance, a subsequent turn is its clarification grounded in modality m if it cannot be preceded by positive evidence of understanding in m. This provides a unified way to frame clarification mechanisms and their interactions across modalities — something we view as useful in its own right given the scattered literature on clarification mechanisms. However we also suggested that this recipe was suitable for learning from data collected by crowdsourcing. We supported this by examining the claim that clarifications are rare in dialogue datasets (Ginzburg, 2012), and that current data-hungry algorithms cannot learn them. We argued that whether they are rare or not depends on pragmatic factors of the conversation and the modality of the grounded clarification. Moreover, along the way we noted a number of practical issues — work with large dialogues, don't just provide annotators with dialogue fragments, take future conversational history into account when annotating — that we think could have an important impact on learnability.
Below we list some possible objections to our proposal. We also include our responses in the hope that this will motivate further debate on these issues in the community.
**Objection:** I still don't have a feel for how much we will gain from this when it comes to a practical, realistic use case; in particular, for an end-to-end system rather than an NLP pipeline.
Response: Being able to identify and annotate a turn as a clarification request can help an end-to-end system learn to apply the mechanisms of collaborative grounding to subdialogs, which have rules that differ from modality to modality.
**Objective:** The biggest problem I see is that the distinction of the different levels (which the correct annotation relies on) might not be clear-cut (in particular when considering that crowdsourced annotations usually come from non-experts). I have no idea what quality we get, nor what inter-annotator agreement figures we can expect.
Response: Our methodology unifies and refines
CHARACTERISTICS	BNC FRAGMENT	COMMUNICATOR	BIELEFELD	SCARE
Task	Chit-chat	Travel reservation	Building	Moving
Shared view	Yes (meetings)	No (on the phone)	Yes (face-to-face)	Yes (3D game)
Participants	More than two	Two	Two	Two
World validation	Common ground	Informational	Physical	Simulated
Information Flow	Symmetrical	Symmetrical	Symmetrical	Asymmetrical
Total # turns	10466	2098	3962	11350
Avg dialogue length	30	67	180	800
% of CRs/turns	3.1	4.6	5.8	6.8
% CRs level 1	10	3	0	3
% CRs level 2	31	32	12	9
% CRs level 3	47	40	50	32
% CRs level 4	2	12	22	53
% CRs other	10	13	16	4
Table 4: Comparing the number of CRs at each level in four corpus studies
previous methodologies for which inter-annotator agreement has been reported in certain corpora: E.g., .70 for the Bielefeld corpus (Rodriguez and Schlangen, 2004), and .75 for the BNC corpus (Purver, 2004). Our methodology refines Clark (1996) 4-level classification by grounding each level (previously only described by means of examples) to 4 different modalities relevant for situated dialog: socioperception, hearing, vision and movement. This new grounded characterization should improve previous inter-annotator agreement. Using our extended methodology we report a .84 kappa for the SCARE.
**Objective:** The corpora that are being investigated are all very domain-specific and relatively small in terms of numbers of dialogues (but with a large average number of turns). This means that even if we were to obtain annotation quality figures, it would still raise the question of what general conclusions we can draw from this.
Response: We share this concern; our goal with this paper is to motivate more work in this area. We believe that this objection actually lends support to our insistence on the importance of a more fine-grained analysis of grounding mechanisms. Our methodology generalizes to domains that ground the communicative intent in the modalities of socioperception, hearing, vision and movement. Examples are robots and virtual assistants, where the dialog partners share a sensible environment. Our argument is that better conceptualizations of clarification subdialogs are needed so that models are able to identify them, distinguish the different types ruled by the different modalities, and learn the
structures that govern them.
This paper urges the community to address a research gap: how clarification mechanisms can be learned from data. We believe that novel research methodologies which highlight the importance of the role of clarification mechanisms in communicative intent are needed for this. So we presented an annotation methodology, based on a theoretical analysis of clarification requests, which unified a number of previous accounts.
But to conclude, a different note. As dialogue systems get better at negotiating meaning with clarifications, future work will need to seriously consider how people relate to conversationally-gifted artificial agents. Studies of how users feel when interacting with dialogue systems (Brave et al., 2005; Portela and Granell-Canut, 2017) found that systems can have a psychological impact on users; thus it will become increasingly important to consider the risks of users developing social or emotional bonds with more sophisticated system (thereby affecting their well-being in unforeseen ways) and of users being emotionally manipulated by them. Socioperceptive dialogue systems could turn out to have very sharp teeth indeed.
We thank the anonymous reviewers for their detailed reviews and insightful comments.
In this paper we have not trained machine learning models so we have used negligible computing power. We have not collected a new dataset so we have not used crowdsourcing. The annotation of the SCARE corpus was done by one of the authors and a friend who likes the work and was not economically rewarded. As we noted in the papers conclusion, there are important ethical issues that future work on this area will need to consider. But there are also more immediate discuss ethical considerations and we turn to these now.
First, the datasets that we use in this paper are described in (Purver, 2004; Rodríguez and Schlangen, 2004; Rieser and Moore, 2005; Stoia, 2007). The dataset in (Purver, 2004) contains spoken British English dialogues collected during meetings. The dataset used in (Rieser and Moore, 2005) is a fragment of the Carnegie Mellon Communicator Corpus (Bennett and Rudnicky, 2002), and is in American English. In these dialogues, an experienced travel agent is making reservations for trips that people in the Carnegie Mellon Speech Group were taking in the upcoming months. There is no information to whether the dialogue participants were rewarded or notified about the dataset collection. The dataset in (Rodríguez and Schlangen, 2004) includes dialogues in which one participant gives instructions in German to the other to build a model plane. Finally, the Scare corpus (Stoia, 2007) is an American English corpus collected using students at Ohio State University; they were paid to participate in the experiment.
Future work in this area will need to collect new datasets that reflect the interactions between different types of clarifications in different modalities. Usually such collections are crowdsourced, which raises ethical concerning fair wages and number of hits per day. We would like to encourage the community to value datasets in languages other than English in order to model different strategies for indicating the source of the clarification (prosody, syntactic construction, etc). Last but not least, computing power and carbon footprint should be considered. Machine learning models trained on long multimodal dialogue histories may get very big very fast. We need models that learn to summarize dialogue histories for the sake of the environment and the budget of low-income researchers.
Malihe Alikhani and Matthew Stone. 2020. Achieving common ground in multi-modal dialogue. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 10-15. ACL.
Jens Allwood. 1995. An activity based approach to pragmatics. In Harry Bunt and William Black, editors, Abduction, Belief and Context in Dialogue: Studies in Computational Pragmatics, pages 47-80. John Benjamins Publishers.
John Austin. 1962. How to do Things with Words: The William James Lectures delivered at Harvard University in 1955. Oxford University Press.
Christina L. Bennett and Alexander I. Rudnicky. 2002. The Carnegie mellon communicator corpus. In *Interspeech*. ISCA.
Dan Bohus and Eric Horvitz. 2014. Managing humanrobot engagement with forecasts and... um... hesitations. In Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14, page 2-9, New York, NY, USA. ACM.
Scott Brave, Clifford Nass, and Kevin Hutchinson. 2005. Computers that care: Investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 62(2):161-178.
Herbert Clark. 1996. Using Language. Cambridge University Press, New York.
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 4466-4475. IEEE Computer Society.
Alexandre Denis. 2010. Generating referring expressions with reference domain theory. In Proceedings of the Sixth International Natural Language Generation Conference. ACL.
David DeVault and Matthew Stone. 2009. Learning to interpret utterances using dialogue history. In Proceedings of the 12th Conference of the European Chapter of the ACL, pages 184-192. ACL.
David DeVault and David Traum. 2013. Approximation of incremental understanding of explicit utterance meaning using predictive models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 1092-1099. ACL.
Myroslava Dzikovska, Peter Bell, Amy Isard, and Johanna Moore. 2012. Evaluating language understanding accuracy with respect to objective outcomes in a dialog system. In Proceedings of the Conference European Chapter of the Association for Computational Linguistics, pages 471-481. ACL.
Myroslava Dzikovska, Johanna Moore, Natalie Steinhauer, and Gwendolyn Campbell. 2010. The impact of interpretation problems on tutorial dialogue. In Proceedings of the Conference of the Association for Computational Linguistics, pages 43-48. ACL.
Arash Eshghi, Christine Howes, Eleni Gregoromichelaki, Julian Hough, and Matthew Purver. 2015. Feedback in conversation as incremental semantic update. In Proceedings of the 11th International Conference on Computational Semantics, pages 261-271, London, UK. ACL.
Raquel Fernández. 2006. Non-sentential utterances in dialogue: Classification, resolution and use. Ph.D. thesis, University of London.
Raquel Fernández, Jonathan Ginzburg, and Shalom Lappin. 2007. Classifying non-sentential utterances in dialogue: A machine learning approach. Computational Linguistics, 33(3):397-427.
Mary Ellen Foster, Andre Gaschler, and Manuel Giuliani. 2017. Automatically classifying user engagement for dynamic multi-party human-robot interaction. International Journal of Social Robotics, 9(5):659-674.
Malte Gabsdil. 2003. Clarification in spoken dialogue systems. In Proceedings of the AAAI Spring Symposium on Natural Language Generation in Spoken and Written Dialogue, pages 28-35.
Bart Geurts. in press. Quantity implicatures. Cambridge University Press.
Jonathan Ginzburg. 2012. The Interactive Stance. Oxford Press.
Jonathan Ginzburg and Raquel Fernández. 2010. Computational models of dialogue. In Alexander Clark, Chris Fox, and Shalom Lappin, editors, Handbook of Computational Linguistics and Natural Language Processing. Blackwell.
Paul Grice. 1975. Logic and conversation. In P. Cole and J. Morgan, editors, Syntax and Semantics, volume 3, pages 41-58. Academic Press, New York.
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1):335-346.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Daniel Jurafsky. 2004. Pragmatics and computational linguistics. In *Handbook of Pragmatics*, pages 3-28. Blackwell, Oxford.
Nikolina Koleva, Martin Villalba, Maria Staudte, and Alexander Koller. 2015. The impact of listener gaze on predicting reference resolution. In Proceedings