Venue: NAACL 2019

# Beyond Task Success:

A Closer Look at Jointly Learning to See, Ask, and GuessWhat

Ravi Shekhar\({}^{\dagger}\), Aashish Venkatesh\({}^{*}\), Tim Baumgartner\({}^{*}\), Elia Bruni\({}^{*}\), Barbara Plank\({}^{\blacktriangledown}\), Raffaella Bernardi\({}^{\dagger}\) and Raquel Fernandez\({}^{*}\)

\({}^{\dagger}\)University of Trento, "University of Amsterdam, "IT University of Copenhagen

ravi.shekhar@unitn.it aashishv@outlook.com baumgaertner.t@gmail.com elia.bruni@gmail.com bap1@itu.dk raffaella.bernardi@unitn.it raquel.fernandez@uva.nl

###### Abstract

We propose a grounded dialogue state encoder which addresses a foundational issue on how to integrate visual grounding with dialogue system components. As a test-bed, we focus on the _GuessWhat?!_ game, a two-player game where the goal is to identify an object in a complex visual scene by asking a sequence of yes/no questions. Our visually-grounded encoder leverages synergies between guessing and asking questions, as it is trained jointly using multi-task learning. We further enrich our model via a cooperative learning regime. We show that the introduction of both the joint architecture and cooperative learning lead to accuracy improvements over the baseline system. We compare our approach to an alternative system which extends the baseline with reinforcement learning. Our in-depth analysis shows that the linguistic skills of the two models differ dramatically, despite approaching comparable performance levels. This points at the importance of analyzing the linguistic output of competing systems beyond numeric comparison solely based on task success.1

Footnote 1: Equal contribution by R. Shekhar and A. Venkatesh.

## 1 Introduction

Over the last few decades, substantial progress has been made in developing dialogue systems that address the abilities that need to be put to work during conversations: Understanding and generating natural language, planning actions, and tracking the information exchanged by the dialogue participants. The latter is particularly critical since, for communication to be effective, participants need to represent the state of the dialogue and the common ground established through the conversation Stalnaker (1978); Lewis (1979); Clark (1996).

In addition to the challenges above, dialogue is often situated in a perceptual environment. In this study, we develop a dialogue agent that builds a representation of the context and the dialogue state by integrating information from both the _visual_ and _linguistic_ modalities. We take the _Guesswhat??_game de Vries et al. (2017) as our tested, a two-player game where a Questioner faces the task of identifying a target object in a visual scene by asking a series of yes/no questions to an Oracle. We model the agent in the Questioner's role.

To model the Questioner, previous work relies on two independent models to learn to ask questions and to guess the target object, each equipped with its own encoder de Vries et al. (2017); Strub et al. (2017); Zhu et al. (2017); Lee et al. (2017); Shekhar et al. (2018); Zhang et al. (2018). We propose an end-to-end architecture with a single _visually-grounded dialogue state encoder_ (cf. Figure 1). Our system is trained jointly in a supervised learning setup, extended with a cooperative learning (CL) regime: By letting the model play the game with self-generated dialogues, the components of the Questioner agent learn to better perform the overall Questioner's task in a cooperative manner Das et al. (2017b) have explored the use of CL to train two visual dialogue agents that receive joint rewards when they play a game successfully. To our knowledge, ours is the first ap

Figure 1: Our questioner model with a single visually grounded dialogue state encoder.



proach where cooperative learning is applied to the internal components of a grounded conversational agent.

Our cooperative learning regime can be seen as an interesting alternative to reinforcement learning (RL)--which was first applied to _GuessWhat?!_ by Strub et al. (2017)--because it is entirely differentiable and computationally less expensive to train than RL. Little is known on how this learning approach compares to RL not only regarding task success, but also in terms of the quality of the linguistic output, a gap we seek to fill in this paper. In particular, our contributions are:2

Footnote 2: Code and supplementary material are available at [https://wista-unitn-uva.github.io](https://wista-unitn-uva.github.io).

* The introduction of a single visually-grounded dialogue state encoder jointly trained with the guesser and question generator modules to address a foundational question of how to integrate visual grounding with dialogue system components; this yields up to 9% improvement on task success.
* The effectiveness of cooperative learning, which yields an additional increase of 8.7% accuracy, while being easier to train than RL.
* A first in-depth study to compare cooperative learning to a state-of-the-art RL system. Our study shows that the linguistic skills of the models differ dramatically, despite approaching comparable task success levels. This underlines the importance of linguistic analysis to complement solely numeric evaluation.

## 2 Related Work

Task-oriented dialogue systemsThe conventional architecture of task-oriented dialogue systems includes a pipeline of components, and the task of tracking the dialogue state is typically modelled as a partially-observable Markov decision process Williams et al. (2013); Young et al. (2013); Kim et al. (2014) that operates on a symbolic dialogue state consisting of predefined variables. The use of symbolic representations to characterise the state of the dialogue has some advantages (e.g., ease of interfacing with knowledge bases), but it has also some key disadvantages: the variables to be tracked have to be defined in advance and the system needs to be trained on data annotated with explicit state configurations.

Given these limitations, there has been a shift towards neural end-to-end systems that learn their own representations. Early works focus on non-goal-oriented chatbots Vinyals and Le (2015); Sordoni et al. (2015); Serban et al. (2016); Li et al. (2016a, b). Bordes et al. (2017) propose a memory network to adapt an end-to-end system to task-oriented dialogue. Recent works combine conventional symbolic with neural approaches Williams et al. (2017); Zhao and Eskenazi (2016); Rastogi et al. (2018), but all focus on language-only dialogue. We propose a visually grounded task-oriented end-to-end dialogue system which, while maintaining the crucial aspect of the interaction of the various modules at play in a conversational agent, grounds them through vision.

Visual dialogue agentsIn recent years, researchers in computer vision have proposed tasks that combine visual processing with dialogue interaction. Pertinent datasets created by Das et al. (2017) and de Vries et al. (2017) include _VisDial_ and _GuessWhat?!_, respectively, where two participants ask and answer questions about an image. While impressive progress has been made in combining vision and language, current models make simplifications regarding the integration of these two modalities and their exploitation for task-related actions. For example, the models proposed for _VisDial_ by Das et al. (2017) concern an image guessing game where one agent does not see the target image (thus, no multimodal understanding) and is required to 'imagine' it by asking questions. The other agent does see the image, but only responds to questions without the need to perform additional actions.

In _GuessWhat?!_, the Questioner agent sees an image and asks questions to identify a target object in it. The Questioner's role hence involves a complex interaction of vision, language, and guessing actions. Most research to date has investigated approaches consisting of different models trained independently de Vries et al. (2017); Strub et al. (2017); Zhu et al. (2017); Lee et al. (2017); Shekhar et al. (2018); Zhang et al. (2018). We propose the first multimodal dialogue agent for the _GuessWhat?!_ task where all components of the Questioner agent are integrated into a joint architecture that has at its core a _visually-grounded dialogue state encoder_ (cf. Figure 1).

Reinforcement learning for visual dialogue agents was introduced by Das et al. (2017) for 

_VisDial_ and by Strub et al. (2017) for _GuessWhat?!_. Our joint architecture allows us to explore a simpler solution based on cooperative learning between the agent's internal modules (see Section 5 for details).

## 3 Task and Data

The _GuessWhat?!_ game de Vries et al., 2017) is a simplified instance of a referential communication task where two players collaborate to identify a referent--a setting used extensively in human-human collaborative dialogue Clark and WilkesGibbs, 1986; Yule, 1997; Zarriess et al., 2016).

The _GuessWhat?!_ dataset3 was collected via Amazon Mechanical Turk by de Vries et al. (2017). The task involves two human participants who see a real-world image, taken from the MSCOO dataset Lin et al. (2014). One of the participants (the Oracle) is assigned a target object in the image and the other participant (the Questioner) has to guess it by asking Yes/No questions to the Oracle. There are no time constraints to play the game. Once the Questioner is ready to make a guess, the list of candidate objects is provided and the game is considered successful if the Questioner picks the target object. The dataset consists of around 155k English dialogues about approximately 66k different images. Dialogues contain on average 5.2 questions-answer pairs.

Footnote 3: Dataset: [https://guesswhat.ai/downloadad](https://guesswhat.ai/downloadad).

## 4 Models

We focus on developing an agent who plays the role of the Questioner in _GuessWhat?!_.

### Baseline Model

As a baseline model (BL), we consider our own implementation of the best performing system put forward by de Vries et al. (2017). It consists of two independent models: a Question Generator (QGen) and a Guesser. For the sake of simplicity, QGen asks a fixed number of questions before the Guesser predicts the target object.

QGen is implemented as an Recurrent Neural Network (RNN) with a transition function handled with Long-Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997), on which a probabilistic sequence model is built with a Softmax classifier. At each time step in the dialogue, the model receives as input the raw image and the dialogue history and generates the next question one word at a time. The image is encoded by extracting its VGG-16 features Simonyan and Zisserman (2014). In our new joint architecture (described below in Section 4.2), we use ResNet152 He et al. (2016) features instead of VGG, because they tend to yield better performance in image classification and are more efficient to compute. For the baseline model it turns out that the original VGG-16 features lead to better performance (41.8% accuracy for VGG-16 vs. 37.3% with ResNet152 features). While we use ResNet152 features in our models, we keep the original VGG-16 feature configuration as de Vries et al. (2017), which constitutes a stronger baseline.

The Guesser model exploits the annotations in the MS-COCO dataset Lin et al. (2014) to represent candidate objects by their object category and their spatial coordinates. This yields better performance than using raw image features in this case, as reported by de Vries et al. (2017). The objects' categories and coordinates are passed through a Multi-Layer Perceptron (MLP) to get an embedding for each object. The Guesser also takes as input the dialogue history processed by its own dedicated LSTM. A dot product between the hidden state of the LSTM and each of the object embeddings returns a score for each candidate object.

The model playing the role of the Oracle is informed about the target object \(o_{target}\). Like the Guesser, the Oracle does not have access to the raw image features. It receives as input embeddings of the target object's category, its spatial coordinates, and the current question asked by the Questioner, encoded by a dedicated LSTM. These three embeddings are concatenated and fed to an MLP that gives an answer (Yes or No).

### Visually-Grounded Dialogue State Encoder

In line with the baseline model, our Questioner agent includes two sub-modules, a QGen and a Guesser. As in the baseline, the Guesser guesses after a fixed number of questions, which is a parameter tuned on the validation set. Our agent architecture differs from the baseline model by de Vries et al.: Rather than operating independently, the language generation and guessing modules are connected through a common _grounded dialogue state encoder_ (GDSE) which combines linguistic and visual information as a prior for the two modules. Given this representation, we will 

refer to our Questioner agent as GDSE.

As illustrated in Figure 1, the encoder receives as input representations of the visual and linguistic context. The visual representation consists of the second to last layer of ResNet152 trained on ImageNet. The linguistic representation is obtained by an LSTM (LSTM\({}_{e}\)) which processes each new question-answer pair in the dialogue. At each question-answer \(QA_{t}\), the last hidden state of LSTM\({}_{e}\) is concatenated with the image features \(I\), passed through a linear layer and a _tanh_ activation to result in the final layer \(h_{t}\):

\[h_{t}=\tanh\left(W\cdot[\text{LSTM}_{e}(qa_{1:t-1});~{}I]\right) \tag{1}\]

where \([\cdot;\cdot]\) represents concatenation, \(I\in\mathbb{R}^{2048\times 1}\), \(\text{LSTM}_{e}\in\mathbb{R}^{1024\times 1}\) and \(W\in\mathbb{R}^{512\times 3072}\) (identical to prior work except for tuning the ResNet-specific parameters). We refer to this final layer as the _dialogue state_, which is given as input to both QGen and Guesser.

As illustrated in Figure 2, our QGen and Guesser modules are like the corresponding modules by de Vries et al. (2017), except for the crucial fact that they receive as input the same grounded dialogue state representation. QGen employs an LSTM (LSTM\({}_{q}\)) to generate the token sequence for each question conditioned on \(h_{t}\), which is used to initialise the hidden state of LSTM\({}_{q}\). As input at every time step, QGen receives a dense embedding of the previously generated token \(w_{i-1}\) and the image features \(I\):

\[p(w_{i})=p(w_{i}|w_{1},...,w_{i-1},h_{t},I) \tag{2}\]

We optimise QGen by minimising the Negative Log Likelihood (NLL) of the human dialogues and use the Adam optimiser Kingma and Ba (2015):

\[\mathcal{L}_{Q}=\sum_{i}-\log p(w_{i}) \tag{3}\]

Thus, in our architecture the LSTM\({}_{q}\) of QGen in combination with the LSTM\({}_{e}\) of the Encoder form a sequence-to-sequence model Sutskever et al. (2014), conditioned on the visual and linguistic context -- in contrast to the baseline model, where question generation is performed by a single LSTM on its own.

The Guesser consists of an MLP which is evaluated for each candidate object in the image. It takes the dense embedding of the category and the spatial information of the object to establish a representation \(r_{j}\in\mathbb{R}^{512\times 1}\) for each object. A score is calculated for each object by performing the dot product between the dialogue state \(h_{t}\) and the object representation. Finally, a softmax over the scores results in a probability distribution over the candidate objects:

\[p(o_{j})=\frac{e^{h_{t}^{T}\cdot r_{j}}}{\sum_{j}e^{h_{t}^{T}\cdot r_{j}}} \tag{4}\]

We pick the object with the highest probability and the game is successful if \(o_{guess}=o_{target}\), where \(o_{guess}=\operatorname*{arg\,max}_{j}p(o_{j})\). As with QGen, we optimise the Guesser by minimising the NLL and again make use of Adam:

\[\mathcal{L}_{G}=-\log p(o_{target}) \tag{5}\]

The resulting architecture is fully differentiable. In addition, the GDSE agent faces a multi-task optimisation problem: While the QGen optimises \(\mathcal{L}_{Q}\) and the Guesser optimises \(\mathcal{L}_{G}\), the parameters of the Encoder (\(W\), LSTM\({}_{e}\)) are optimised via both \(\mathcal{L}_{Q}\) and \(\mathcal{L}_{G}\). Hence, both tasks faced by the Questioner agent contribute to the optimisation of the dialogue state \(h_{t}\), and thus to a more effective encoding of the input context.

## 5 Learning Approach

We first introduce the supervised learning approach used to train both BL and GDSE, then our cooperative learning regime, and finally the reinforcement learning approach we compare to.

### Supervised Learning

In the baseline model, the QGen and the Guesser modules are trained autonomously with supervised learning (SL): QGen is trained to replicate

Figure 2: Question Generation and Guesser modules.

 

human questions and, independently, the Guesser is trained to predict the target object. Our new architecture with a common dialogue state encoder allows us to formulate these two tasks as a multi-task problem, with two different losses (Eq. 3 and 5 in Section 4.2). These two tasks are not equally difficult: While the Guesser has to learn the probability distribution of the set of possible objects in the image, QGen needs to fit the distribution of natural language words. Thus, QGen has a harder task to optimize and requires more parameters and training iterations. We address this issue by making the learning schedule task-dependent. We call this setup _modulo-n_ training, where \(n\) indicates after how many epochs of QGen training the Guesser is updated together with QGen.

Using the validation set, we experimented with \(n\) from 5 to 15 and found that updating the Guesser every 7 epochs worked best. With this optimal configuration, we then train GDSE for 100 epochs (batch size of 1024, Adam, learning rate of 0.0001) and select the Questioner module best performing on the validation set (henceforth, GDSE-SL or simply SL).

### Cooperative Learning

Once the model has been trained with SL, new training data can be generated by letting the agent play new games. Given an image from the training set used in the SL phase, we generate a new training instance by randomly sampling a target object from all objects in the image. We then let our Questioner agent and the Oracle play the game with that object as target, and further train the common encoder using the generated dialogues by backpropagating the error with gradient descent through the Guesser. After training the Guesser and the encoder with generated dialogues, QGen needs to 'readapt' to the newly arranged encoder parameters. To achieve this, we re-train QGen on the human data with SL, but using the new encoder states. Also here, the error is backpropagated with gradient descent through the common encoder.

Regarding _modulo-n_, in this case QGen is updated at every \(n^{\mathrm{th}}\) epoch, while the Guesser is updated at all other epochs; we experimented with \(n\) from 3-7 and set it to the optimal value of 5. The GDSE previously trained with SL is further trained with this cooperative learning regime for 100 epochs (batch size of 256, Adam, learning rate of 0.0001), and we select the Questioner module performing best on the validation set (henceforth, GDSE-CL or simply CL).

### Reinforcement Learning

Strub et al. (2017) proposed the first extension of BL de Vries et al. (2017) with deep reinforcement learning (RL). They present an architecture for end-to-end training using an RL policy. First, the Oracle, Guesser, and QGen models are trained independently using supervised learning. Then, QGen is further trained using a policy gradient.

We use the publicly available code and pre-trained model based on Sampling Strub et al. (2017), which resulted in the closest performance to what was reported by the authors.4 This is the RL model we use throughout the rest of the paper.

Footnote 4: Their result of 53.3% accuracy published in Strub et al. (2017) is obsolete, as stated on their GitHub page ([https://github.com/GuessWhatGame/guesswhat](https://github.com/GuessWhatGame/guesswhat)) where they report 56.5% for sampling and 58.4% for greedy search. By running their code, we could only replicate their results with sampling, obtaining 56%, while greedy and beam search resulted in similar or worse performance. Our analysis showed that greedy and beam search have the additional disadvantage of learning a smaller vocabulary.

### Experimental Details

We use the same train (70%), validation (15%), and test (15%) splits as de Vries et al. (2017). The test set contains new images not seen during training. We use two experimental setups for the number of questions to be asked by the question generator, motivated by prior work: 5 questions (5Q) following de Vries et al. (2017), and 8 questions (8Q) as in Strub et al. (2017). As noted in Section 3, on average, there are 5.2 questions per dialogue in the _GuessWhat?_ data set.

For evaluation, we report task success in terms of accuracy Strub et al. (2017). To neutralize the effect of random sampling in training CL, we trained the model 3 times. RL is tested 3 times with sampling. We report means and standard deviation (for some tables these are provided in the supplementary material; see footnote 2).

## 6 Results

Table 1 reports the results for all models. There are several take-aways.

**Grounded joint architecture** First of all, our visually-grounded dialogue state encoder is effective. GDSE-SL outperforms the baseline by de Vries et al. (2017) significantly in both setups (absolute accuracy improvements of 6.6% 

and 9%). To evaluate the impact of the multi-task learning aspect, we did an ablation study and used the encoder-decoder architecture to train the QGen and Guesser modules independently. With such a decoupled training we obtain lower results: 44% and 43.7% accuracy for 5Q and 8Q, respectively. Hence, the multi-task component brings an increase of up to 6% over the baseline.5

Footnote 5: While de Vries et al. (2017) originally report an accuracy of 46.8%, this result was later revised to 40.8%, as clarified on their GitHub page. Our own implementation of the base-line system achieves an accuracy of 41.2%.

Cooperative learning and RLThe introduction of the cooperative learning approach results in a clear improvement over GDSE-SL: +8.7% (8Q: from 49.7 to 58.4) and +5.9% (with 5Q). Despite its simplicity, our GDSE-CL model achieves a task success rate which is comparable to RL: In the 8Q setup, GDSE-CL reaches an average accuracy of 58.4 versus 56.3 for RL, giving CL a slight edge in this setup (+2.1%), while in the 5Q setup RL is slightly better (+2.5%). Overall, the accuracy of the CL and RL models is close. The interesting question is how the linguistic skills and strategy of these two models differ, to which we turn in the next section.

We compared to Strub et al. (2017), but RL has also been put forward by Zhang et al. (2018), who report 60.7% accuracy (5Q). This result is close to our highest GDSE-CL result (60.8 \(\pm\)0.51, when optimized for 10Q).6 Their RL system integrates several partial reward functions to increase coherence, which is an interesting aspect. Yet their code is not publicly available. We leave the comparison to Zhang et al. (2018) and adding RL to GDSE to future work.

Footnote 6: Since our aim is to compare to the best setup for BL (5Q) and RL (8Q), we do not report our results with 10Q in Table 1.

## 7 Analysis

In this section, we present a range of analyses that aim to shed light on the performance of the models. They are carried out on the test set data using the 8Q setting, which yields better results than the 5Q setting for the GDSE models and RL. Given that there is only a small difference in accuracy for the baseline with 5Q and 8Q, for comparability we analyse dialogues with 8Q also for BL.

### Quantitative Analysis of Linguistic Output

We analyse the language produced by the Questioner agent with respect to three factors: (1) lexical diversity, measured as type/token ratio over all games, (2) question diversity, measured as the percentage of unique questions over all games, and (3) the number of games with questions repeated verbatim. We compute these factors on the test set for the models and for the human data (H).

As shown in Table 2, the linguistic output of SL & CL is closer to the language used by humans: Our agent is able to produce a much richer and less repetitive output than both BL and RL. In particular, it learns to use a more diverse vocabulary, generates more unique questions, and repeats questions within the same dialogue at a much lower rate than the baseline and RL: 93.5% of the games played by BL contain at least one verbatim question repetition, for RL this happens in 96.47% of the cases, whereas for SL and CL this is for only 55.8% and 52.19% of the games, respectively.

### Dialogue Strategy

To further understand the variety of questions asked by the agents, we classify questions into different types. We distinguish between questions that aim at getting the category of the target object (entity questions, e.g., _'is it a vehicle?'_) and questions about properties of the queried objects (attribute questions, e.g., _'is it square?'_ or _'are

\begin{table}
\begin{tabular}{l l l} \hline \hline Model & 5Q & 8Q \\ \hline Baseline & 41.2 & 40.7 \\ GSDE-SL & 47.8 & 49.7 \\ GDSE-CL & 53.7 (\(\pm\).83) & 58.4 (\(\pm\).12) \\ RL & 56.2 (\(\pm\).24) & 56.3 (\(\pm\).05) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Test set accuracy for each model (for setups with 5 and 8 questions). GDSE-SL is our grounded supervised learning system, GDSE-CL the cooperative learning setup, and RL the results we obtain with the reinforcement learning system by Strub et al. (2017).

\begin{table}
\begin{tabular}{l l l l} \hline \hline  & Lexical & Question & \% Games with \\  & diversity & diversity & repeated Q’s \\ \hline BL & 0.030 & 1.60 & 93.50 \\ SL & 0.101 & 13.61 & 55.80 \\ CL & 0.115 (\(\pm\).02) & 14.15 (\(\pm\)3.0) & 52.19 (\(\pm\)4.7) \\ RL & 0.073 (\(\pm\).00) & 1.04 (\(\pm\).03) & 96.47 (\(\pm\).04) \\ H & 0.731 & 47.89 & — \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of the linguistic output of all models with the 8Q setting and of humans (H) in all test games.

 

they standing?_). Within attribute questions, we make a distinction between color, shape, size, texture, location, and action questions. Within entity questions, we distinguish questions whose focus is an object category or a super-category (see the supplementary material for example questions). The classification is done by manually extracting keywords for each question type from the human dialogues, and then applying an automatic heuristic that assigns a class to a question given the presence of the relevant keywords.7 This procedure allows us to classify 91.41% of the questions asked by humans. The coverage is higher for the questions asked by the models: 98.88% (BL), 94.72% (SL), 94.11% (CL) and 99.51 % (RL).8

Footnote 7: A question may be tagged with several attribute classes if keywords of different types are present. E.g., “_Is it the white one on the left?_” is classified as both colon and location.

The statistics are shown in Table 3. We use Kullback-Leibler (KL) divergence to measure how the output of each model differs from the human distribution of fine-grained question classes. The baseline's output has the highest degree of divergence: For instance, the BL model does never ask any shape or texture questions, and hardly any size questions. The output of the RL model also differs substantially from the human dialogues: It asks a very large number of location questions (74.8% vs. 40% for humans). Our model, in contrast, generates question types that resemble the human distribution more closely.

We also analyse the structure of the dialogues in terms of the sequences of question types asked. As expected, both humans and models almost always start with an entity question (around 97% for BL, SL and CL, 98.7% for RL, and 78.48% for humans), in particular a super-category (around 70% for BL, SL and CL, 84% for RL, and 52.32% for humans). In some cases, humans start by asking questions directly about an attribute that may easily distinguish an object from others, while this is very uncommon for models. Figure 3 shows an example: The human dialogue begins with an attribute question (_'does it have cereal on it?'_), which in this case is not very effective and leads to a change in strategy at turn 4. The CL model starts by asking an object question (_'is it a donut?'_) while the RL model begins with a more generic super-category question (_'is it food?'_).

We check how the answer to a given question type affects the type of the follow-up question. In principle, we expect to find that question types that are answered positively will be followed by more specific questions. This is indeed what we observe in the human dialogues, as shown in Table 4. For example, when a super-category question is answered positively, humans follow up with an object or attribute question 89.56% of the time. This trend is mirrored by all models.

Overall, the models also learn the strategy to move from an object to an attribute question when an object question receives a Yes answer. The BL, SL, and CL models do this to a lesser extent than humans, while the RL model systematically

\begin{table}
\begin{tabular}{l l c r r|r r} \hline Question type & BL & SL & CL & RL & H \\ \hline
**entity** & **49.00** & **48.07** & **46.51** & **23.99** & **38.11** \\ super-cat & 19.6 & 12.38 & 12.58 & 14.00 & 14.51 \\ object & 29.4 & 35.70 & 33.92 & 9.99 & 23.61 \\
**attribute** & **49.88** & **46.64** & **47.60** & **75.52** & **53.29** \\ color & 2.75 & 13.00 & 12.51 & 0.12 & 15.50 \\ shape & 0.00 & 0.01 & 0.02 & 0.003 & 0.30 \\ sizie & 0.02 & 0.33 & 0.39 & 0.024 & 1.38 \\ texture & 0.00 & 0.13 & 0.15 & 0.013 & 0.89 \\ location & 47.25 & 37.09 & 38.54 & 74.80 & 40.00 \\ action & 1.34 & 7.97 & 7.60 & 0.66 & 7.59 \\
**Not classified** & **1.12** & **5.28** & **5.90** & **0.49** & **8.60** \\ \hline KL (wrt human) & 0.953 & 0.042 & 0.038 & 0.396 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 3: Percentage of questions per question type in all the test set games played by humans (H) and the models with the 8Q setting, and KL divergence from human distribution of fine-grained question types.

Figure 3: Game example where Gbase-CL succeeds and RL fails at guessing the target object (framed).

 

transitions to attributes (in 99.46% of cases), using mostly location questions as pointed out above. For example (Figure 3), after receiving an affirmative answer to the object question 'is it a donut?' both CL and RL shift to a location question. Once location is established, CL moves on to other attributes while RL keeps asking the same location question, which leads to failure. Further illustrative examples are given in the supplementary material.

### Analysis of the CL Learning Process

In order to better understand the effect of the cooperative learning regime, we trace the evolution of linguistic factors identified above over the CL epochs. As illustrated in Figure 4 (a) and (b), through the epochs the CL model learns to use a richer vocabulary and more diverse questions, moving away from the levels achieved by BL and RL, overpassing SL and moving toward humans.

The CL model progressively produces fewer repeated questions within a dialogue, improving over SL in the last few epochs, cf. Figure 4 (c). Finally, (d) illustrates the effect of modulo-\(n\) training: As the model is trained on generated dialogues, its linguistic output drifts away from the human distribution of question types; every 5\({}^{th}\) epoch QGen is trained via supervision, which brings the model's behaviour closer back to human linguistic style and helps decrease the drift.

## 8 Conclusion

We present a new visually-grounded joint Questioner agent for goal-oriented dialogue. First, we show that our architecture archives 6-9% accuracy improvements over the _GuessWhat??!_ baseline system [10]. This way, we address a foundational limitation of previous approaches that model guessing and questioning separately.

Second, our joint architecture allows us to propose a two-phase cooperative learning approach (CL), which further improves accuracy. It results in our overall best model and reaches state-of-the-art results (cf. Section 6). We compare CL to the system proposed by Strub et al. (2017) which extends the baseline with reinforcement learning (RL). We find that the two approaches (CL and RL) achieve overall relatively similar task success rates. However, evaluating on task success is only one side of the coin. Finally and most importantly, we propose to pursue an in-depth analysis of the quality of the dialogues by visual conversational agents, which is an aspect often neglected in the literature. We analyze the linguistic output of the two models across three factors (lexical diversity, question diversity, and repetitions) and find them to differ substantially. The CL model uses a richer vocabulary and inventory of questions, and produces fewer repeated questions than RL. In contrast, RL highly relies on asking location questions, which might be explained by a higher reliance on spatial and object-type information explicitly given to the Guesser and Oracle models. Limiting rewards to task success or other rewards not connected to the language proficiency does not stimulate the model to learn rich linguistic skills, since a reduced vocabulary and simple linguistic structures may be an efficient strategy to succeed at the game.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Question type shift & BL & SL & CL & RL & H \\ \hline super-cat \(\rightarrow\) obj/att & 89.05 & 92.61 & 89.75 & 95.63 & 89.56 \\ object \(\rightarrow\) attribute & 67.87 & 60.92 & 65.06 & 99.46 & 88.70 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Proportion of question type shift vs. no type shift in consecutive questions \(Q_{t}\to Q_{t+1}\) where \(Q_{t}\) has received a Yes answer.

Figure 4: Evolution of linguistic factors over 100 training epochs for our GDSE-CL model. Note: lexical and question diversity of the human data fall outside the range in (a) / (b). The same is the case with KL for BL in (d).

 

Overall, the presence of repeated questions remains an important weakness of all models, resulting in unnatural dialogues. This shows that there is still a considerable gap to human-like conversational agents. Looking beyond task success can provide a good basis for extensions of current architectures, e.g., Shekhar et al. (2018) add a decision-making component that decides when to stop asking questions which results in less repetitive and more human-like dialogues. Our joint architecture could easily be extended with such a component.

## Acknowledgements

The work carried out by the Amsterdam team was partially funded by the Netherlands Organisation for Scientific Research (NWO) under VIDI grant nr. 276-89-008, _Asymmetry in Conversation_. We thank the University of Trento for generously funding a research visit by Barbara Plank to CIMeC that led to part of the work presented in this paper. In addition, we kindly acknowledge the support of NVIDIA Corporation with the donation to the University of Trento of the GPUs used in our research.

## References

* Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal oriented dialog. In _Proceedings of ICLR_.
* Clark (1996) Herbert H Clark. 1996. _Using Language_. Cambridge University Press.
* Clark and Wilkes-Gibbs (1986) Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. _Cognition_, 22(1):1-39.
* Das et al. (2017a) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M.F. Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual Dialog. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* Das et al. (2017b) Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, and Dhruv Batra. 2017b. Learning cooperative visual dialog agents with deep reinforcement learning. In _International Conference on Computer Vision (ICCV)_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kim et al. (2014) Dongho Kim, Catherine Breslin, Pirros Tsiakoulis, M Gasic, Matthew Henderson, and Steve Young. 2014. Inverse reinforcement learning for micro-turn management. In _Fitteenth Annual Conference of the International Speech Communication Association_.
* Kingma and Ba (2015) D. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of ICLR_.
* Lee et al. (2017) Sang-Woo Lee, Yujung Heo, and Byoung-Tak Zhang. 2017. Answerer in questioner's mind for goal-oriented visual dialogue. In _NIPS Workshop on Visually-Grounded Interaction and Language (ViGL)_.
* Lewis (1979) David Lewis. 1979. Scorekeeping in a language game. _Journal of Philosophical Logic_, 8(1):339-359.
* Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In _Proceedings of NAACL-2016_, pages 110-119.
* Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. In _Proceedings of EMNLP_.
* Lin et al. (2014) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollar, P., and C. L. Zitnick. 2014. Microsoft COCO: Common objects in context. In _Proceedings of ECCV (European Conference on Computer Vision)_.
* Rastogi et al. (2018) Abhinav Rastogi, Raghav Gupta, and Dilek Hakkani-Tur. 2018. Multi-task learning for joint language understanding and dialogue state tracking. In _Proceedings of SIGdial_.
* Serban et al. (2016) Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_.
* Shekhar et al. (2018) Ravi Shekhar, Tim Baumgartner, Aashish Venkatesh, Elia Bruni, Raffaella Bernardi, and Raouel Fernandez. 2018. Ask no more: Deciding when to guess in referential visual dialogue. In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_, pages 1218-1233.
* Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Technical Report arXiv:1409.1556, Visual Geometry Group, University of Oxford.
* Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015.

 

A neural network approach to context-sensitive generation of conversational responses. In _Proceedings of NAACL-HLT_, pages 196-205.
* Stalnaker (1978) Robert Stalnaker. 1978. Assertion. In P. Cole, editor, _Pragmatics_, volume 9 of _Syntax and Semantics_. New York Academic Press.
* Strub et al. (2017) Florian Strub, Harm de Vries, Jeremie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_, pages 3104-3112.
* Vinyals and Le (2015) Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. In _ICML Deep Learning Workshop_.
* de Vries et al. (2017) Harr de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! Visual object discovery through multi-modal dialogue. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Williams et al. (2017) Jason Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: Practical and efficient end-to-end dialog control with supervised and reinforcement learning. In _Proceedings of ACL-2017_. Association for Computational Linguistics.
* Williams et al. (2013) Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In _Proceedings of the SIGDIAL 2013 Conference_, pages 404-413.
* Young et al. (2013) Steve Young, Milica Gasic, Blaise Thomson, and Jason D Williams. 2013. POMDP-based statistical spoken dialog systems: A review. _Proceedings of the IEEE_, 101(5).
* Yule (1997) George Yule. 1997. _Referential communication tasks_. Routledge.
* Zarriess et al. (2016) Sina Zarriess, Julian Hough, Casey Kennington, Ramesh Manvinakurike, David DeVault, Raquel Fernandez, and David Schlangen. 2016. Pentoref: A corpus of spoken references in task-oriented dialogues. In _10th edition of the Language Resources and Evaluation Conference_.
* Zhang et al. (2018) Junjie Zhang, Qi Wu, Chunhua Shen, Jian Zhang, Jianfeng Lu, and Anton van den Hengel. 2018. Goal-oriented visual question generation via intermediate rewards. In _Proceedings of the European Conference of Computer Vision (ECCV)_.
* Zhao and Eskenazi (2016) Tiancheng Zhao and Maxine Eskenazi. 2016. Towards end-to-end learning for dialog state trading and management using deep reinforcement learning. In _Proceedings of SIGDIAL-2016_.
* Zhu et al. (2017) Yan Zhu, Shaoting Zhang, and Dimitris Metaxas. 2017. Interactive reinforcement learning for object grounding via self-talking. In _NIPS Workshop on Visually-Grounded Interaction and Language (ViGLL)_.