Introduction
Deep neural networks have been successfully applied to several computer vision tasks such as image classification BIBREF0 , object detection BIBREF1 , video action classification BIBREF2 , etc. They have also been successfully applied to natural language processing tasks such as machine translation BIBREF3 , machine reading comprehension BIBREF4 , etc. There has also been an explosion of interest in tasks which combine multiple modalities such as audio, vision, and language together. Some popular multi-modal tasks combining these three modalities, and their differences are highlighted in Table TABREF1 .
Given an image and a question related to the image, the vqa challenge BIBREF5 tasked users with selecting an answer to the question. BIBREF6 identified several sources of bias in the vqa dataset, which led to deep neural models answering several questions superficially. They found that in several instances, deep architectures exploited the statistics of the dataset to select answers ignoring the provided image. This prompted the release of vqa 2.0 BIBREF7 which attempts to balance the original dataset. In it, each question is paired to two similar images which have different answers. Due to the complexity of vqa, understanding the failures of deep neural architectures for this task has been a challenge. It is not easy to interpret whether the system failed in understanding the question or in understanding the image or in reasoning over it. The CLEVR dataset BIBREF8 was hence proposed as a useful benchmark to evaluate such systems on the task of visual reasoning. Extending question answering over images to videos, BIBREF9 have proposed MovieQA, where the task is to select the correct answer to a provided question given the movie clip on which it is based.
Intelligent systems that can interact with human users for a useful purpose are highly valuable. To this end, there has been a recent push towards moving from single-turn qa to multi-turn dialogue, which is a natural and intuitive setting for humans. Among multi-modal dialogue tasks, visdial BIBREF10 provides an image and dialogue where each turn is a qa pair. The task is to train a model to answer these questions within the dialogue. The avsd challenge extends the visdial task from images to the audio-visual domain.
We present our modelname model for the avsd task. modelname combines a hred for encoding and generating qa-dialogue with a novel FiLM-based audio-visual feature extractor for videos and an auxiliary multi-task learning-based decoder for decoding a summary of the video. It outperforms the baseline results for the avsd dataset BIBREF11 and was ranked 2nd overall among the dstc7 avsd challenge participants.
In Section SECREF2 , we discuss existing literature on end-to-end dialogue systems with a special focus on multi-modal dialogue systems. Section SECREF3 describes the avsd dataset. In Section SECREF4 , we present the architecture of our modelname model. We describe our evaluation and experimental setup in Section SECREF5 and then conclude in Section SECREF6 .
Related Work
With the availability of large conversational corpora from sources like Reddit and Twitter, there has been a lot of recent work on end-to-end modelling of dialogue for open domains. BIBREF12 treated dialogue as a machine translation problem where they translate from the stimulus to the response. They observed this to be more challenging than machine translation tasks due the larger diversity of possible responses. Among approaches that just use the previous utterance to generate the current response, BIBREF13 proposed a response generation model based on the encoder decoder framework. BIBREF14 also proposed an encoder-decoder based neural network architecture that uses the previous two utterances to generate the current response. Among discriminative methods (i.e. methods that produce a score for utterances from a set and then rank them), BIBREF15 proposed a neural architecture to select the best next response from a list of responses by measuring their similarity to the dialogue context. BIBREF16 extended prior work on encoder-decoder-based models to multi-turn conversations. They trained a hierarchical model called hred for generating dialogue utterances where a recurrent neural network encoder encodes each utterance. A higher-level recurrent neural network maintains the dialogue state by further encoding the individual utterance encodings. This dialogue state is then decoded by another recurrent decoder to generate the response at that point in time. In followup work, BIBREF17 used a latent stochastic variable to condition the generation process which aided their model in producing longer coherent outputs that better retain the context.
Datasets and tasks BIBREF10 , BIBREF18 , BIBREF19 have also been released recently to study visual-input based conversations. BIBREF10 train several generative and discriminative deep neural models for the visdial task. They observe that on this task, discriminative models outperform generative models and that models making better use of the dialogue history do better than models that do not use dialogue history at all. Unexpectedly, the performance between models that use the image features and models that do no use these features is not significantly different. As we discussed in Section SECREF1 , this is similar to the issues vqa models faced initially due to the imbalanced nature of the dataset, which leads us to believe that language is a strong prior on the visdial dataset too. BIBREF20 train two separate agents to play a cooperative game where one agent has to answer the other agent's questions, which in turn has to predict the fc7 features of the Image obtained from VGGNet. Both agents are based on hred models and they show that agents fine-tuned with rl outperform agents trained solely with supervised learning. BIBREF18 train both generative and discriminative deep neural models on the igc dataset, where the task is to generate questions and answers to carry on a meaningful conversation. BIBREF19 train hred-based models on GuessWhat?! dataset in which agents have to play a guessing game where one player has to find an object in the picture which the other player knows about and can answer questions about them.
Moving from image-based dialogue to video-based dialogue adds further complexity and challenges. Limited availability of such data is one of the challenges. Apart from the avsd dataset, there does not exist a video dialogue dataset to the best of our knowledge and the avsd data itself is fairly limited in size. Extracting relevant features from videos also contains the inherent complexity of extracting features from individual frames and additionally requires understanding their temporal interaction. The temporal nature of videos also makes it important to be able to focus on a varying-length subset of video frames as the action which is being asked about might be happening within them. There is also the need to encode the additional modality of audio which would be required for answering questions that rely on the audio track. With limited size of publicly available datasets based on the visual modality, learning useful features from high dimensional visual data has been a challenge even for the visdial dataset, and we anticipate this to be an even more significant challenge on the avsd dataset as it involves videos.
On the avsd task, BIBREF11 train an attention-based audio-visual scene-aware dialogue model which we use as the baseline model for this paper. They divide each video into multiple equal-duration segments and, from each of them, extract video features using an I3D BIBREF21 model, and audio features using a VGGish BIBREF22 model. The I3D model was pre-trained on Kinetics BIBREF23 dataset and the VGGish model was pre-trained on Audio Set BIBREF24 . The baseline encodes the current utterance's question with a lstm BIBREF25 and uses the encoding to attend to the audio and video features from all the video segments and to fuse them together. The dialogue history is modelled with a hierarchical recurrent lstm encoder where the input to the lower level encoder is a concatenation of question-answer pairs. The fused feature representation is concatenated with the question encoding and the dialogue history encoding and the resulting vector is used to decode the current answer using an lstm decoder. Similar to the visdial models, the performance difference between the best model that uses text and the best model that uses both text and video features is small. This indicates that the language is a stronger prior here and the baseline model is unable to make good use of the highly relevant video.
Automated evaluation of both task-oriented and non-task-oriented dialogue systems has been a challenge BIBREF26 , BIBREF27 too. Most such dialogue systems are evaluated using per-turn evaluation metrics since there is no suitable per-dialogue metric as conversations do not need to happen in a deterministic ordering of turns. These per-turn evaluation metrics are mostly word-overlap-based metrics such as BLEU, METEOR, ROUGE, and CIDEr, borrowed from the machine translation literature. Due to the diverse nature of possible responses, world-overlap metrics are not highly suitable for evaluating these tasks. Human evaluation of generated responses is considered the most reliable metric for such tasks but it is cost prohibitive and hence the dialogue system literature continues to rely widely on word-overlap-based metrics.
The avsd dataset and challenge
The avsd dataset BIBREF28 consists of dialogues collected via amt. Each dialogue is associated with a video from the Charades BIBREF29 dataset and has conversations between two amt workers related to the video. The Charades dataset has multi-action short videos and it provides text descriptions for these videos, which the avsd challenge also distributes as the caption. The avsd dataset has been collected using similar methodology as the visdial dataset. In avsd, each dialogue turn consists of a question and answer pair. One of the amt workers assumes the role of questioner while the other amt worker assumes the role of answerer. The questioner sees three static frames from the video and has to ask questions. The answerer sees the video and answers the questions asked by the questioner. After 10 such qa turns, the questioner wraps up by writing a summary of the video based on the conversation.
Dataset statistics such as the number of dialogues, turns, and words for the avsd dataset are presented in Table TABREF5 . For the initially released prototype dataset, the training set of the avsd dataset corresponds to videos taken from the training set of the Charades dataset while the validation and test sets of the avsd dataset correspond to videos taken from the validation set of the Charades dataset. For the official dataset, training, validation and test sets are drawn from the corresponding Charades sets.
The Charades dataset also provides additional annotations for the videos such as action, scene, and object annotations, which are considered to be external data sources by the avsd challenge, for which there is a special sub-task in the challenge. The action annotations also include the start and end time of the action in the video.
Models
Our modelname model is based on the hred framework for modelling dialogue systems. In our model, an utterance-level recurrent lstm encoder encodes utterances and a dialogue-level recurrent lstm encoder encodes the final hidden states of the utterance-level encoders, thus maintaining the dialogue state and dialogue coherence. We use the final hidden states of the utterance-level encoders in the attention mechanism that is applied to the outputs of the description, video, and audio encoders. The attended features from these encoders are fused with the dialogue-level encoder's hidden states. An utterance-level decoder decodes the response for each such dialogue state following a question. We also add an auxiliary decoding module which is similar to the response decoder except that it tries to generate the caption and/or the summary of the video. We present our model in Figure FIGREF2 and describe the individual components in detail below.
Utterance-level Encoder
The utterance-level encoder is a recurrent neural network consisting of a single layer of lstm cells. The input to the lstm are word embeddings for each word in the utterance. The utterance is concatenated with a special symbol <eos> marking the end of the sequence. We initialize our word embeddings using 300-dimensional GloVe BIBREF30 and then fine-tune them during training. For words not present in the GloVe vocabulary, we initialize their word embeddings from a random uniform distribution.
Description Encoder
Similar to the utterance-level encoder, the description encoder is also a single-layer lstm recurrent neural network. Its word embeddings are also initialized with GloVe and then fine-tuned during training. For the description, we use the caption and/or the summary for the video provided with the dataset. The description encoder also has access to the last hidden state of the utterance-level encoder, which it uses to generate an attention map over the hidden states of its lstm. The final output of this module is the attention-weighted sum of the lstm hidden states.
Video Encoder with Time-Extended FiLM
For the video encoder, we use an I3D model pre-trained on the Kinetics dataset BIBREF23 and extract the output of its Mixed_7c layer for INLINEFORM0 (30 for our models) equi-distant segments of the video. Over these features, we add INLINEFORM1 (2 for our models) FiLM BIBREF31 blocks which have been highly successful in visual reasoning problems. Each FiLM block applies a conditional (on the utterance encoding) feature-wise affine transformation on the features input to it, ultimately leading to the extraction of more relevant features. The FiLM blocks are followed by fully connected layers which are further encoded by a single layer recurrent lstm network. The last hidden state of the utterance-level encoder then generates an attention map over the hidden states of its lstm, which is multiplied by the hidden states to provide the output of this module. We also experimented with using convolutional Mixed_5c features to capture spatial information but on the limited avsd dataset they did not yield any improvement. When not using the FiLM blocks, we use the final layer I3D features (provided by the avsd organizers) and encode them with the lstm directly, followed by the attention step. We present the video encoder in Figure FIGREF3 .
Audio Encoder
The audio encoder is structurally similar to the video encoder. We use the VGGish features provided by the avsd challenge organizers. Also similar to the video encoder, when not using the FiLM blocks, we use the VGGish features and encode them with the lstm directly, followed by the attention step. The audio encoder is depicted in Figure FIGREF4 .
Fusing Modalities for Dialogue Context
The outputs of the encoders for past utterances, descriptions, video, and audio together form the dialogue context INLINEFORM0 which is the input of the decoder. We first combine past utterances using a dialogue-level encoder which is a single-layer lstm recurrent neural network. The input to this encoder are the final hidden states of the utterance-level lstm. To combine the hidden states of these diverse modalities, we found concatenation to perform better on the validation set than averaging or the Hadamard product.
Decoders
The answer decoder consists of a single-layer recurrent lstm network and generates the answer to the last question utterance. At each time-step, it is provided with the dialogue-level state and produces a softmax over a vector corresponding to vocabulary words and stops when 30 words were produced or an end of sentence token is encountered.
The auxiliary decoder is functionally similar to the answer decoder. The decoded sentence is the caption and/or description of the video. We use the Video Encoder state instead of the Dialogue-level Encoder state as input since with this module we want to learn a better video representation capable of decoding the description.
Loss Function
For a given context embedding INLINEFORM0 at dialogue turn INLINEFORM1 , we minimize the negative log-likelihood of the answer word INLINEFORM2 (vocabulary size), normalized by the number of words INLINEFORM3 in the ground truth response INLINEFORM4 , L(Ct, r) = -1Mm=1MiV( [rt,m=i] INLINEFORM5 ) , where the probabilities INLINEFORM6 are given by the decoder LSTM output, r*t,m-1 ={ll rt,m-1 ; s>0.2, sU(0, 1)
v INLINEFORM0 ; else . is given by scheduled sampling BIBREF32 , and INLINEFORM1 is a symbol denoting the start of a sequence. We optimize the model using the AMSGrad algorithm BIBREF33 and use a per-condition random search to determine hyperparameters. We train the model using the BLEU-4 score on the validation set as our stopping citerion.
Experiments
The avsd challenge tasks we address here are:
We train our modelname model for Task 1.a and Task 2.a of the challenge and we present the results in Table TABREF9 . Our model outperforms the baseline model released by BIBREF11 on all of these tasks. The scores for the winning team have been released to challenge participants and are also included. Their approach, however, is not public as of yet. We observe the following for our models:
Since the official test set has not been released publicly, results reported on the official test set have been provided by the challenge organizers. For the prototype test set and for the ablation study presented in Table TABREF24 , we use the same code for evaluation metrics as used by BIBREF11 for fairness and comparability. We attribute the significant performance gain of our model over the baseline to a combination of several factors as described below:
Our primary architectural differences over the baseline model are: not concatenating the question, answer pairs before encoding them, the auxiliary decoder module, and using the Time-Extended FiLM module for feature extraction. These, combined with using scheduled sampling and running hyperparameter optimization over the validation set to select hyperparameters, give us the observed performance boost.
We observe that our models generate fairly relevant responses to questions in the dialogues, and models with audio-visual inputs respond to audio-visual questions (e.g. “is there any voices or music ?”) correctly more often.
We conduct an ablation study on the effectiveness of different components (eg., text, video and audio) and present it in Table TABREF24 . Our experiments show that:
Conclusions
We presented modelname, a state-of-the-art dialogue model for conversations about videos. We evaluated the model on the official AVSD test set, where it achieves a relative improvement of more than 16% over the baseline model on BLEU-4 and more than 33% on CIDEr. The challenging aspect of multi-modal dialogue is fusing modalities with varying information density. On AVSD, it is easiest to learn from the input text, while video features remain largely opaque to the decoder. modelname uses a generalization of FiLM to video that conditions video feature extraction on a question. However, similar to related work, absolute improvements of incorporating video features into dialogue are consistent but small. Thus, while our results indicate the suitability of our FiLM generalization, they also highlight that applications at the intersection between language and video are currently constrained by the quality of video features, and emphasizes the need for larger datasets.

Question: At which interval do they extract video and audio frames?
Answer: Unanswerable

Question: Do they use pretrained word vectors for dialogue context embedding?
Answer: Yes

Question: Do they train a different training method except from scheduled sampling?
Answer: Answer with content missing: (list missing) 
Scheduled sampling: In our experiments, we found that models trained with scheduled sampling performed better (about 0.004 BLEU-4 on validation set) than the ones trained using teacher-forcing for the AVSD dataset. Hence, we use scheduled sampling for all the results we report in this paper.

Yes.