Introduction
If you're good at replying to a single request, are you also likely to be good at doing dialogue? Much current work seems to assume that the answer to this question is yes, in that it attempts a scaling up from single pairs of utterance plus response to longer dialogues: See, e.g., the work on neural chatbots following on from BIBREF0, where the main evaluation metric is “next utterance retrieval”; and on visual dialogue BIBREF1, which views itself as a natural extension of visual question answering BIBREF2.
If you assume, however, that dialogue crucially is a joint project between its participants in a way that single exchanges are not, you're likely to put more focus on coordination phenomena BIBREF3, but may end up with settings that combine multiple language capabilities in ways that current methods cannot yet seem to capture. (See, for example, the dialogues collected in BIBREF4.) In this short paper, we contribute a type of setting that introduces such coordination phenomena, while still allowing for control of the complexity of the resulting interaction.
Visual Dialogue as Example of the Scaling Up Approach
Figure FIGREF2 shows an example interaction with the original Visual Dialogue system BIBREF1. The competence of the system is impressive from a multimodal grounding perspective — it gets right several questions aiming at different aspects of the image. It is also clear, however, that this is a modest step beyond single-shot visual question answering BIBREF2. It seems that here the (human) questioner is doing all the work of keeping the dialogue alive, and there is little that suggest that the answerer is keeping any state about the dialogue. Later work by BIBREF6 on the “visual dialogue” dataset BIBREF1 indeed identified co-reference in the questions as the main issue that distinguishes this setting from one-shot question answering.
One shortcoming of this setting—that the questioner was not provided with a good reason for why they are asking questions in the first place—was addressed in some related work: In the GuessWhat? setting introduced by BIBREF7 at around the same time as Visual Dialogue, a questioner is asking polar questions about an image, with the goal of identifying an object known only to the questioner. In the ALICE variant of the visual dialogue setting, the questioner is asking questions with the later goal in mind of identifying the image from a set also containing distractor images BIBREF8. These variants of the general setting provide purpose to the questioner, but not to the answerer, which is the target of the modelling effort; and, crucially, it does not give the dialogue a joint purpose, a shared sense of semantic ownership of the interaction, which is a central feature of most genres of human interaction BIBREF3.
Coming back to the visual dialogue setting, it can be assumed that the crowd workers that created the original data did try to orient themselves to the usual maxims that govern conversational behaviour. However, being constrained by the rigid roles of questioner and answerer, and with the perceptual task being so easy for them, a need for dealing with miscommunication never arose for them and hence no such strategies can be learned from that data. That this is missing from the resulting agents can easily been shown in cases where something goes wrong, but normal repair mechanisms BIBREF9 are not available, as in the example interaction we created shown in Figure FIGREF5.
Agreement Games
In Herbert Clark's (BIBREF3) model of dialogue, the mutual need for ensuring understanding—“sufficient to current purposes”—is the main structuring force in dialogue. As a metaphor for this interaction management process, Clark uses the notion of a “secondary track”, on which the constant negotiation of this understanding happens. This can be done through quite subtle methods, such as simply producing a continuation that displays through its fit an understanding of the previous utterance, as well as through specially designed markers (such as feedback utterances like “uhu”). This model has been influential in the design of spoken dialogue systems BIBREF13, BIBREF14, where it has been incorporated in the design of dialogue state update rules. In the data-driven era, however, it seems to have become less well known, and many datasets almost appear as to be designed in such a way as to limit the possibility of grounding interactions.
The idea behind our setting of “agreement games” is to make this secondary track more prominent and hence more easy to pick up from the data, by making reaching mutual understanding on the answer to the game question an explicit goal. Or, in Clark's term, the grounding criterion for answering the question is raised so as to make reaching mutual understanding on it an explicit, rather than as normally implicit, goal.
The representational challenge is that it is an abstract object—the understanding of the discourse—that is jointly constructed, purely through verbal actions.
Agreement Games ::: More formally
An Agreement Game is a dialogue game with two regular participants, $\mathcal {P} = \lbrace P_1, P_2\rbrace $, and a disinterested third participant, $N$ (for Nature). $N$ poses a question $Q$ to the players $\mathcal {P}$, and provides them with information $I$ required to answer the question; possibly split up over the players. If $I$ contains visual information, we call the game a Grounded Agreement Game.
The players can exchange messages in an unrestricted way. The game ends when one of the players explicitly proposes an answer $A$ and the other player explicity agrees with the proposal. As the answer $A$ will be based on a construal of $I$, the agreement on $A$ is also an agreement on that construal. Optionally, a reward can be given to the players after they have provided their joint answer, tied to some measure of quality of $A$.
We illustrate the concept by discussing some instantiations that we have recently experimented with.
Some Examples ::: The MeetUp Game
In the MeetUp game BIBREF17, BIBREF18, the two participants are presented with an environment through which they can (separately) navigate and which is represented to static photographs or real indoor scenes (e.g., a picture of a bedroom, a kitchen, etc.) Their goal is to meet up in the same room, of a type previously told to them. (E.g., they might be told at the start of the game: “your goal is to meet up in a room of type kitchen.”) As the positions of the players are not represented graphically, the only way they can be sure of whether they have reached that goal is by conversing (via chat messages). Once they have come to the conclusion that they are in the same room, they can end the game by each sending a special signal. If they are indeed in such a winning constellation, they will receive a bonus.
Unlike the Visual Dialogue setting discussed above, this setting ensures informational symmetry between the participants (both have access to the same type of information; but not the same information, as they can't “see” each other). More importantly, however, the constraint that the game only ends if they both agree ensures a “committment symmetry”, where the success of the game must be ensured by both participants. The design also provides for a clear “relevance place” at which an opportunity arises for semantic negotiation, namely, before the final decision is made. An example of this is shown in the example below. (The number in the parentheses indicate the time, relative to the beginning of the interaction, when the utterance was made.)
. B (00:00:34): okay I think I'm there if I understand utility room
B (00:00:42): It has a washer and dryer
A (00:00:46): I was wondering too. This is sorta like a laundry room.
A (00:00:55): This has pet bowl on the floor below a window.
B (00:01:00): ok... let us keep looking
A (00:01:22): And a small kids looking suit hanging on the wall. And a big banner above the window.
B (00:01:33): Are you saying a utility room is like a laundry room?
B (00:02:00): let me find you
A (00:02:07): Google says, a room equipped with appliances for washing and other domestic work.
A (00:02:09): So I think so.
Some Examples ::: The MatchIt Game
The MatchIt Game (Ilinykh et al., forthcoming) is a yet further simplified visual game. Here, the goal simply is to decide whether you and your partner are both looking at the same image (of the same genre as in MeetUp). In that sense, it is a reduction of the MeetUP game to the final stage, taking out the navigation aspect. As example SECREF12 shows, this can similarly lead to meta-semantic interaction, where classifications are revised. As SECREF12 shows, even in cases where a decision can be reached quickly, there can be an explicit mutual confirmation step, before the (silent) decision signal is sent.
. B (00:00:25): white kitchen?
A (00:00:25): im in a bathroom
B (00:00:28): ah
B (00:00:32): well wait
B (00:00:38): there is something that looks like a big bath
B (00:00:44): is it all white?
A (00:00:54): yes its white and I see a bit of a kitchen
A (00:01:11): yes
B (00:01:11): are you sure it's a bathroom lol
A (00:01:16): no its not a bathroom haha
. A (00:00:24): i see stairs
B (00:00:25): I see a staircase with a bike with wicker basket at the bottom of the staircase
B (00:00:31): do you have a bike?
A (00:00:39): no bike
B: (00:00:46): okay..it is different
A (00:00:54): yes
Some Examples ::: The Concept Learning Game
A third setting that we have explored BIBREF19 brings conceptual negotiation more clearly into the foreground. In that game, the players are presented with images of birds of particular species and are tasked with coming up with a description of common properties. Again, the final answer has to be approved by both participants. As SECREF13 shows, this can lead to an explicit negotiation of conceptual content.
.
Conclusions
We have argued that some prominent current dialogue settings lack room for the occurence of coordination phenomena prevalent in natural dialogue. We have shown a simple condition that brings out the need for coordination in an explicit way, but still can easily be added to controlled (and controllable) dialogue settings.

Question: Do the authors perform experiments using their proposed method?
Answer: Yes