SlowGuess/NAACL_2021_OCR huggingface dataset:
Luciana Benotti
Universidad Nacional de Córdoba CONICET, Argentina
luciana.benotti@unc.edu.ar
Patrick Blackburn
Philosophy and Science Studies  
IKH, Roskilde University, Denmark  
patrickb@ruc.dk
In order to interpret the communicative intents of an utterance, it needs to be grounded in something that is outside of language; that is, grounded in world modalities. In this paper we argue that dialogue clarification mechanisms make explicit the process of interpreting the communicative intents of the speaker's utterances by grounding them in the various modalities in which the dialogue is situated. This paper frames dialogue clarification mechanisms as an understudied research problem and a key missing piece in the giant jigsaw puzzle of natural language understanding. We discuss both the theoretical background and practical challenges posed by this problem, and propose a recipe for obtaining grounding annotations. We conclude by highlighting ethical issues that need to be addressed in future work.
Clarifications are crucial to robust dialogues, and pragmatic factors — notably those shaped by the world modalities situating the conversation — have a key role to play. Referring expressions have in vision a modality in which to ground clarifications concerning objects in the world (de Vries et al., 2017); navigation instructions have in movement a modality in which to ground clarifications concerning collaborative wayfinding (Thomason et al., 2019). Clarifications grounded in situationally relevant modalities boost the redundancy required to learn to use language without explicit supervision, as they make explicit the process of negotiating the communicative intent. But despite its importance, work on clarification remains scattered.
Humans switch between clarifications grounded in different modalities seamlessly but (we shall argue) systematically. Our discussion is based around a general recipe for detecting grounded clarifications; we work towards this in Section 2 by first reviewing the distinction between perceptual and
collaborative grounding, and then discussing clarification mechanisms, Clark (1996)'s action ladder of communication, and Ginzburg, Purver and colleagues (2012)'s classification of clarification phenomena. In Section 3 we draw these threads together and present the central idea:
Given an utterance  $U$ , a subsequent turn is its clarification grounded in modality m if it cannot be preceded by positive evidence of understanding of  $U$  in m.
This provides a unified way to frame clarification mechanisms and their interactions across various modalities; a graphical specification of the recipe it gives rise to can be found in Figure 2 of the supplementary material. It covers clarifications grounded in moving, grabbing and changing the physical world: these have traditionally been considered plain-old-questions (Purver et al., 2018), but we view them as useful clarification ingredients. $^{1}$  In Sections 4 and A we test the practical implications of our recipe by identifying and characterizing (according to their modalities) the clarifications in a corpus of long dialogues in English. In Section 5 we turn to the claim that clarifications are rare in dialogue datasets (Ginzburg, 2012), and that current data-hungry algorithms cannot learn them. We argue that whether they are rare or not depends on pragmatic factors of the conversation and the modality of the grounded clarification, and discuss the impact of six such factors. After presenting potential objections and our responses in Section 6, we conclude in Section 7 by noting ethical issues raised by socioperceptive dialogue systems that will need to be addressed. $^{2}$
<sup>1</sup>We are suspicious of the common assumption that requests for information regarding references that are grounded in vision (e.g. the red or the blue jacket?) are clarifications, whereas requests for information grounded in other modalities are not (e.g. do I take the stairs up or down?).
See also the supplement on ethical considerations.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4065-4077 June 6-11, 2021. ©2021 Association for Computational Linguistics
2 Theoretical Background
We begin by reviewing the theoretical background on grounding and clarification mechanisms. We then examine two schemes proposed to characterize clarifications according to their conversational function: one focuses on the problem of anchoring utterance parameters into the conversational history, the other emphasizes a multimodal ladder of actions co-temporal with dialogue turn-taking. We are interested in the potential contributions of both towards a recipe for annotating clarification mechanisms.
2.1 Collaborative and perceptual grounding
Collaborative grounding is the process of seeking and providing incremental evidence of mutual understanding through dialogue. When the speaker believes that the dialogue is on track, positive evidence of understanding is provided in different forms (depending on the channel of communication) such as explicit acknowledgements, and via backchannels such as nods, eye contact, etc. Negative evidence of understanding signals that something needs negotiation before the dialogue partners can commit; clarification requests are the prototypical example of negative evidence.
Collaborative grounding is distinct from perceptual (or symbol) grounding (Harnad, 1990; He et al., 2016; Tan and Bansal, 2019; Lu et al., 2020). The perceptual grounding literature deals with capabilities enabling symbols to be linked with perceptions, and is rooted in situationally relevant modalities such as vision. Collaborative grounding, on the other hand, deals with the dynamics of conversation (the ongoing exchange of speaker and hearer roles) and is rooted in situationally relevant aspects of socioperception. Alikhani and Stone (2020) note several basic mechanisms that contribute to collaborative grounding, including those for dealing with joint attention (Koller et al., 2012; Koleva et al., 2015; Tan et al., 2020), engagement (Bohus and Horvitz, 2014; Foster et al., 2017), turn taking and incremental interpretation (Schlangen and Skantze, 2009; Selfridge et al., 2012; DeVault and Traum, 2013; Eshghi et al., 2015) corrections and clarifications (Villalba et al., 2017; Ginzburg and Fernandez, 2010) and dialogue management (DeVault and Stone, 2009; Selfridge et al., 2012). These mechanisms have been studied for different kinds of applications (Denis, 2010; Dzikovska et al., 2010, 2012). Both collaborative and perceptual grounding are
important (all relevant modalities are potentially important) and in this paper we bring them together under an umbrella we call grounded clarification.
2.2 Clarification mechanisms
Clarification requests (CRs) and their answers are the prototypical clarification mechanisms (CMs), pieces of dialogue that participants use to signal lack of understanding and to trigger negotiation. CMs are used in all kinds of dialogue and are influenced by the type of interaction, the dialogue participants, and the context in which the conversation occurs. Interest in CMs by the artificial intelligence community dates back to the start of the century, and has typically focused on mechanisms for human-computer dialogue systems (Gabsdil, 2003; Purver, 2004; Rodriguez and Schlangen, 2004; Rieser and Moore, 2005; Skantze, 2007). In sociolinguistics and discourse analysis, on the other hand, the interest in CMs (or repairs, as they are usually called there) has focused on human-human conversation for over three decades now; see (Schegloff, 1987) for a representative example.
How CMs can be learned from data remains understudied. Rao and Daumé III (2018) rank clarification requests of stackoverflow articles according to their usefulness: a good clarification question is one whose expected answer will be useful, which means that the clarification highlighted important information missing from the initial request for help; we share this view, but differ from Rao and Daumé III, in that we focus on CMs and their responses occurring in multiturn dialogue.
It may seem plausible to expect that clarification requests will be realized as questions; however, corpus studies indicate that their most frequent realization is in declarative form (Jurafsky, 2004). Indeed, the form of a clarification request (Rodriguez and Schlangen, 2004) is not a reliable indicator of the function that the clarification request is playing. Neither does form unambiguously indicate whether a dialogue contribution is a CR or not. The surface form of explicit negotiations of meaning in dialogue are frequently non-sentential utterances (Fernández, 2006; Fernández et al., 2007). These include the prototypical positive and negative evidence of grounding (acknowledgements and clarification requests (Stoyanchev et al., 2013)) but also less-well-known forms such as self-corrections, rejections, and modifiers (Purver, 2004; Purver et al., 2018). These observations indicate that we face
significant challenges if we want to train a system to seek or supply clarification effectively.
2.3 Clarifications grounded in parameters
Ginzburg, Purver and colleagues (henceforth G&P) proposed the first scheme to classify the functions of CRs; see (Purver et al., 2003; Purver, 2006; Ginzburg, 2012). The G&P classification uses the categories shown on Table 1. The idea driving this work is that CRs are caused by problems arising during the anchoring of utterance parameters into the conversational history.
CATEGORY	OBSTACLE	EXAMPLES
Repetition	Cannot identify a surface parameter	What did you say?
Clausal	Uncertain value for a clausal dialogue history parameter	Are you asking if BO SMITH left?
Intended	The hearer can find no value for a parameter	Who is Bo?
Correction	The hearer thinks that the speaker made a mistake and offers an alternative realization	Did you mean to say ‘Bro’?
The G&P classification has been criticized (Rodriguez and Schlangen, 2004; Rieser and Moore, 2005) because, in practice, it seems difficult to decide what the category of a particular CR is; that is, CRs are usually ambiguous in this classification. In fact, G&P recognize this issue themselves, pointing out that CRs that do not repeat (part of) the content of the source utterance (that is, the utterance that is being clarified) can exhibit all three readings.
However, G&P's classification is only ambiguous if only the past, but not the future, conversational history is taken into account. It is crucial to analyze the CR response in order to disambiguate the CR category. Sometimes the immediate linguistic context gives the clue necessary for disambiguation: whereas a repetition reading permits the responder to the CR to repeat her utterance verbatim, a clausal confirmation usually receives a yes/no answer, and an intended content reading requires the responder to reformulate in some way. Hence, the turn of the responder (and the subsequent reaction of the participant originally making the CR) can disambiguate among readings. Consider the following example from (Purver, 2004). The example shows a case where George's initial
clausal interpretation is incorrect (the initiator is not satisfied), and a constituent reading is required (Anon cannot find a value for Spunyarn).
George: you always had er er say every foot he had with a piece of spunyarn in the wire
Anon: Spunyarn?
George: Spunyarn, yes
Anon: What's spunyarn?
George: Well that's like er tarred rope
In other situations, the immediate linguistic context will not be enough (for instance, a reformulation can be a good response to all three types of CRs) and then the whole conversational history might need to be analyzed in order to disambiguate. This makes G&P's classification difficult to use in annotation studies where the annotators only get shallow, partial, localized views of the dialogues.
2.4 Clarifications grounded in modalities
The second classification we shall examine puts the conversational action modality in the central role; it has been used in formal approaches to handling clarifications in dialogue systems (Gabsdil, 2003; Rodríguez and Schlangen, 2004; Rieser and Moore, 2005). This classification is based on the four-level model of conversational action independently developed by (Allwood, 1995) and (Clark, 1996). Here, we use Clark's terminology; his model is reproduced in Table 2.
Table 1: CR classification scheme by P&G
L	SPEAKER A'S ACTIONS	ADDRESSE B'S ACTIONS
4	Propose project w to B	Uptake A's proposal w
3	Intend that B does i	Recognize i from A
2	Present signal s to B	Perceive signal s from A
1	Execute behavior t for B	Attend to behavior t from A
Table 2: Ladder of actions involved in communication
Clark proposed this model in order to move from Austin's controversial classification<sup>3</sup> of speech acts (Austin, 1962) to a ladder of actions which characterizes not only the actions that are performed in language use (as Austin's does) but also their inter-relationships. Clark (1996) defines a ladder of actions as a set of co-temporal actions which provide upward causality and downward evidence. Let us discuss these using Table 2; we will call the speaker Anna and the addressee Barny. Suppose that Anna tells Barny to sit down. We might say that Anna is performing just one action: asking
3For discussion of the controversies around Austin's classification of speech acts see (Clark, 1996)
Barny to sit down. But it is easy to argue that she is performing four distinct, though co-temporal, actions — actions beginning and ending simultaneously. These actions are in a causal relation going up the ladder (from level 1 up to level 4): Anna must get Barny to attend her behavior t (level 1) in order to get him to hear the words she is presenting in her signals (level 2). Anna must succeed at that in order to get Barny to recognize what she means (level 3), and she must succeed at that in order to get Barny to uptake the project she is proposing (level 4). In short, causality (do something in order to get some result) climbs up the ladder; this property Clark calls upward causality.
The different levels are related to different human modalities. We say that level 1 is grounded into socioperception, an ability that humans developed for collaboration that is crucial for achieving joint attention (Tomasello et al., 2005). Level 2 is grounded in hearing if we use speech as our communication channel. Level 3 is grounded in vision when it involves recognizing referents in the real world. Level 4 is grounded in kinesthetic when it involves moving and acting in the real world. The classification, along with obstacles that the addressee may face in the various modalities during the interpretation of a conversational action, is shown in Table 3. In the rest of the paper we will refer to these modalities using the level number.
L	MODALITY	EXAMPLES
4	Kinesthetic	Do I take the stairs up or down?
3	Vision	The red or the blue jacket?
2	Hearing	What did you say?
1	Socioperception	Are you talking to me?
Table 3: Ladder of actions grounded in modalities.
Humans systematically use the evidence provided by this ladder. Observing Barny sitting down is good evidence that he did not refuse to uptake (level 4) but also recognized what Anna intended and identified the chair (level 3). That is also evidence that she got Barny to hear her words (level 2), and evidence that she got him to attend to her (level 1). That is, evidence trickles down the ladder; Clark calls this the downward evidence property.
If Barny repeats verbatim what Anna said (e.g. suppose she spoke in Spanish and he repeats the word sientate), then Anna has good evidence that he heard what she said (level 2). However, that is not necessarily evidence that he has recognized her intention; there might be an obstacle in level 3
(for instance, Barny might not know Spanish). If there is such an obstacle, she would have completed levels 1 and 2 while failing to complete not only level 3 but also level 4 (it is rather unlikely that Barny would sit down right after hearing Anna — and even if he did, this would not be because he was uptaking Anna's project). A high level action in the ladder can only be completed by executing all the actions in the lower levels. This property Clark calls upward completion.
If you tell somebody something, you expect a reaction from him. If he doesn't answer, you might think that he didn't hear you, that he doesn't want to answer, or that he thinks you are talking to somebody else. None of these situations is very agreeable; humans don't like wasting effort, or being ignored. In order not to annoy the speaker, the addressee has two options: either he shows evidence in level 4 (and then, by downward evidence, the speaker knows that all the levels succeeded), or he indicates the obstacle in executing the action (in any level). Clarifications are the tools that addressee can use to make the obstacle explicit.
3 A grounded clarification recipe
In this section we draw these threads together under the heading grounded clarification. First, what is a clarification? Our starting proposal, which we will modify, is the following: given an utterance  $U$ , a subsequent turn is its clarification if it cannot be preceded by positive evidence of  $U$ . Note that this proposal implicitly embodies a procedure for annotating clarifications, one which could be crowdsourced: Is this a clarification? Check whether it can be preceded by positive evidence!
Our starting proposal is a modified version of Gabsdil (2003)'s test for CRs. Gabsdil says that CRs (as opposed to other kinds of dialogue contributions) cannot be preceded by explicit acknowledgments. For example:
Lara: There's only two people in the class.
a) Matthew: Two people?
b) (*) Matthew: Ok, Two people?
(BNC, taken from (Purver et al., 2003))
Gabsdil argues that (a) in the example above is a CR because (b) is odd (we mark odd turns with  $(^{*})$  in examples). In (b), Matthew first acknowledges Lara's turn and only then indicates that her turn contains information that he finds controversial.4
4This could be a felicitous response, but it would require
On the other hand, (b) in the example below is fine and hence (a) is not a CR: the lieutenant acknowledges the sergeant's turn and then moves on to address what has become the most pressing topic in the conversation:
Sergeant: There was an accident sir
a) Lieutenant: Who is hurt?
b) Lieutenant: Ok. Who is hurt?
Adapted from (Traum, 2003, p.391)
However Gabsdil's original test incorrectly discards cases that we view as CRs. Consider the following example:
G: I want you to go up the left hand side of it towards the green bay and make it a slightly diagonal line, towards, sloping to the right.
F: Ok. So you want me to go above the carpenter?
Adapted from (Gabsdil, 2003, p.30)
The problem is that the level of positive evidence contributed by F's acknowledgment is ambiguous. For instance, the Ok could (conceivably) mean:
- Ok, so you want to talk to me (level 1).
- Ok, I heard you (level 2).
- Ok, I saw what you are referring to (level 3).
- Ok, I did it (level 4, the highest level).
Thus we modify Gabsdil's test to make it levelsensitive. In order to signal that all the levels have been successful and that no CR related to any of them is expected, the simple acknowledgment needs to be replaced by positive evidence in the highest level. This works for Gabsdil's example:
(*)  $F$ : Ok, I did it. So you want me to go above the carpenter?
Here So you want me to go above the carpenter? is either weird or far more likely to be interpreted as a question about an action that comes after F has successfully followed G's instruction. That is: it could be interpreted as F taking the initiative and proposing the next move, rather than as clarifying G's instruction. Whether this is plausible would be determined by the following turns.
More generally, if the addressee wants to uptake the speaker's proposal then he or she has two options: either to give positive evidence at the highest modality (and then, by downward closure, the
marked intonation to induce a backtracking effect.
speaker knows that all lower levels succeeded) or to explicitly indicate the problem using a clarification (at any level). Table 3 illustrates, for each level and modality, possible CRs. We are not exhaustive about all the modalities that could happen in reality. We list four of them here but there could be more depending on the task.
This approach to CR identification and classification is useful not only for instructions but also for other types of utterances. The following is an extension of Grice's classic implicature example (physical actions are between square brackets):
A: I am out of petrol.
B: There is a garage around the corner.
A: [A goes to the garage and then meets B again] (*) A: Ok, I got petrol at the garage. Do you think the garage was open?
Adapted from (Grice, 1975, p.311)
After acknowledging a contribution at level 4 (which A's Ok, I got petrol at the garage clearly does) it is really hard to go on and ask a CR about that contribution (A's Do you think the garage was open? is a bizarre follow-up — it could perhaps be interpreted as sarcastic).
Thus our modified proposal for identifying clarifications is the following: given an utterance  $U$ , a subsequent turn is its clarification grounded in modality  $m$  if it cannot be preceded by positive evidence of understanding of  $U$  in  $m$ . Like the earlier version, this implicitly embodies a annotation procedure. Let's see how it works.
4 Grounded clarification annotation
In this section we evaluate our recipe and the modality-based classification it gives rise to. We do so by using it to annotate a small dataset, the SCARE corpus (Stoia, 2007). Before delving into the details of the classification, we describe the pragmatic influences that the dialogue participants are under in this dataset.
The SCARE corpus consists of fifteen English spontaneous dialogues situated in an instruction giving task. The dialogues vary in length, with a
For a detailed graphical specification of our recipe, see Figure 2 in the supplementary material. Notice that the utterances are stored in a stack in Figure 2 because the clarification does not need to be immediately after its source. While an utterance is at the top of the stack it can be clarified, no matter how many turns in between have happened. That way an utterance can be clarified many times.
<sup>6</sup>The corpus is available at http://slate.cse.ohio-state.edu/quake-corpora/scare/.
minimum of 400 turns and a maximum of 1500; hence, the dialogues are much longer than other datasets grounded in vision and action where dialogues typically have less than 10 turns on average (de Vries et al., 2017; Thomason et al., 2019). The dialogues were collected using the QUAKE environment, a first-person virtual reality game (so there is immediate world validation). The task consists of a direction giver (DG) instructing a direction follower (DF) on how to complete several tasks in a simulated game world. The corpus contains the collected audio and video, as well as word-aligned transcriptions.
The DF had no prior knowledge of the world map or tasks and relied on his partner, the DG, to guide him on completing the tasks (so the DPs have asymmetric knowledge of the task). The DG had a map of the world and a list of tasks to complete. The partners spoke to each other through headset microphones. As the participants collaborated on the tasks, the DG had instant feedback about the DF's location in the simulated world, because the game engine displayed the DF's first person view of the world on both the DG's and DF's computer monitors (so the DPs share a view of the task). Finally, the DPs were punished (they were told they would receive less money for performing the experiment) if they pressed the wrong buttons or put things in the wrong cabinets.
We present a sample interaction from the SCARE corpus. During this dialogue fragment, the dialogue participants were performing one of the tasks of the SCARE experiment specified: hide the rebreather in cabinet 9.