,  ,  ,  ,  ,  ,  ,  have generated descriptive sentences from images with the help of Deep Networks. There have been many works for solving Visual Dialog  ,  ,  ,  ,  . A variety of methods have been proposed by  ,  ,  ,  ,  ,  for solving VQA task including attention-based methods  ,  ,  ,  ,  ,  ,  . However, Visual Question Generation (VQG) is a separate task which is of interest in its own right and has not been so well explored  . This is a vision based novel task aimed at generating natural and engaging question for an image.  proposed a method for continuously generating questions from an image and subsequently answering those questions. The works closely related to ours are that of  and  . In the former work, the authors used an encoder-decoder based framework whereas in the latter work, the authors extend it by using a variational autoencoder based sequential routine to obtain natural questions by performing sampling of the latent variable.

Whereas for a little girl, the questions could be related to age rather than the action. How can one have widely varying context provided for generating questions? To solve this problem, we use the context obtained by considering exemplars, specifically we use the difference between relevant and irrelevant exemplars. We consider different contexts in the form of Location, Caption, and Part of Speech tags. The human annotated questions are (b) for the first image and (a) for the second image. Our method implicitly uses a differential context obtained through supporting and contrasting exemplars to obtain a differentiable embedding. This embedding is used by a question decoder to decode the appropriate question. As discussed further, we observe this implicit differential context to perform better than an explicit keyword based context. The difference between the two approaches is illustrated in Figure FIGREF2 . This also allows for better optimization as we can backpropagate through the whole network. We provide detailed empirical evidence to support our hypothesis. As seen in Figure FIGREF1 our method generates natural questions and improves over the state-of-the-art techniques for this problem. To summarize, we propose a multimodal differential network to solve the task of visual question generation.

We explored using the CONV5 having feature dimension 14x14 512, FC7 having 4096 and FC8 having feature dimension of 365 of places365. Ablation Analysis Sampling Exemplar: KNN vs ITML Our method is aimed at using efficient exemplar-based retrieval techniques. We have experimented with various exemplar methods, such as ITML  based metric learning for image features and KNN based approaches. We observed KNN based approach (K-D tree) with Euclidean metric is a efficient method for finding exemplars. Also we observed that ITML is computationally expensive and also depends on the training procedure. The table provides the experimental result for Differential Image Network variant with k (number of exemplars) = 2 and Hadamard method: Question Generation approaches: Sampling vs Argmax We obtained the decoding using standard practice followed in the literature  . This method selects the argmax sentence. Also, we evaluated our method by sampling from the probability distributions and provide the results for our proposed MDN-Joint method for VQG dataset as follows: How are exemplars improving Embedding In Multimodel differential network, we use exemplars and train them using a triplet loss.

Variations of Proposed Method	While, we advocate the use of multimodal differential network for generating embeddings that can be used by the decoder for generating questions, we also evaluate several variants of this architecture. These are as follows: Tag Net: In this variant, we consider extracting the part-of-speech (POS) tags for the words present in the caption and obtaining a Tag embedding by considering different methods of combining the one-hot vectors. Further details and experimental results are present in the supplementary. This Tag embedding is then combined with the image embedding and provided to the decoder network. Place Net: In this variant we explore obtaining embeddings based on the visual scene understanding. This is obtained using a pre-trained PlaceCNN  that is trained to classify 365 different types of scene categories. We then combine the activation map for the input image and the VGG-19 based place embedding to obtain the joint embedding used by the decoder. Differential Image Network: Instead of using multimodal differential network for generating embeddings, we also evaluate differential image network for the same. In this case, the embedding does not include the caption but is based only on the image feature. We also experimented with using multiple exemplars and random exemplars.

While the caption generates scene description, our proposed method generates semantically meaningful and novel questions. Examples for Figure 1 of main paper: First Image:- Caption- A young man skateboarding around little cones. Our Question- Is this a skateboard competition? Second Image:- Caption- A small child is standing on a pair of skis. Our Question:- How old is that little girl? Intuition behind Triplet Network: The intuition behind use of triplet networks is clear through this paper  that first advocated its use. The main idea is that when we learn distance functions that are “close” for similar and “far” from dissimilar representations, it is not clear that close and far are with respect to what measure. By incorporating a triplet we learn distance functions that learn that “A is more similar to B as compared to C”. Learning such measures allows us to bring target image-caption joint embeddings that are closer to supporting exemplars as compared to contrasting exemplars. Analysis of Network Analysis of Tag Context Tag is language based context. These tags are extracted from caption, except question-tags which is fixed as the 7 'Wh words' (What, Why, Where, Who, When, Which and How).

We also used learning rate decay to decrease the learning rate on every epoch by a factor given by: INLINEFORM25 where values of a=1500 and b=1250 are set empirically. Ablation Analysis of Model While, we advocate the use of multimodal differential network (MDN) for generating embeddings that can be used by the decoder for generating questions, we also evaluate several variants of this architecture namely (a) Differential Image Network, (b) Tag net and (c) Place net. These are described in detail as follows: Differential Image Network For obtaining the exemplar image based context embedding, we propose a triplet network consist of three network, one is target net, supporting net and opposing net. All these three networks designed with convolution neural network and shared the same parameters. The weights of this network are learnt through end-to-end learning using a triplet loss. The aim is to obtain latent weight vectors that bring the supporting exemplar close to the target image and enhances the difference between opposing examples. More formally, given an image INLINEFORM26 we obtain an embedding INLINEFORM27 using a CNN that we parameterize through a function INLINEFORM28 where INLINEFORM29 are the weights of the CNN.

We empirically evaluated whether an explicit approach that uses the differential set of tags as a one-hot encoding improves the question generation, or the implicit embedding obtained based on the triplet network. We observed that the implicit multimodal differential network empirically provided better context for generating questions. Our understanding of this phenomenon is that both target and supporting exemplars generate similar questions whereas contrasting exemplars generate very different questions from the target question. The triplet network that enhances the joint embedding thus aids to improve the generation of target question. These are observed to be better than the explicitly obtained context tags as can be seen in Figure FIGREF2 . We now explain our method in detail.

Multimodal Differential Network	The proposed Multimodal Differential Network (MDN) consists of a representation module and a joint mixture module. We used an efficient KNN-based approach (k-d tree) with Euclidean metric to obtain the exemplars. This is obtained through a coarse quantization of nearest neighbors of the training examples into 50 clusters, and selecting the nearest as supporting and farthest as the contrasting exemplars. We experimented with ITML based metric learning  for image features. Surprisingly, the KNN-based approach outperforms the latter one. We also tried random exemplars and different number of exemplars and found that INLINEFORM0 works best. We provide these results in the supplementary material. We use a triplet network  ,  in our representation module. We refereed a similar kind of work done in  for building our triplet network. The triplet network consists of three sub-parts: target, supporting, and contrasting networks. All three networks share the same parameters. Given an image INLINEFORM0 we obtain an embedding INLINEFORM1 using a CNN parameterized by a function INLINEFORM2 where INLINEFORM3 are the weights for the CNN.

And the best result comes when we take 3 Wh-Words as context and apply the Hadamard Model with concatenating the 3 WH-words. Also in Table TABREF34 we have shown the results when we take more than one words as context. Here we show that for 3 words i.e 3 nouns, 3 verbs and 3 Wh-words, the Concatenation model performs the best. In this table the conv model is using 1D convolution to combine the tags and the joint model combine all the tags. Analysis of Context: Exemplars In Multimodel Differential Network and Differential Image Network, we use exemplar images(target, supporting and opposing image) to obtain the differential context. We have performed the experiment based on the single exemplar(K=1), which is one supporting and one opposing image along with target image, based on two exemplar(K=2), i.e. two supporting and two opposing image along with single target image. similarly we have performed experiment for K=3 and K=4 as shown in table- TABREF35 .

Multimodal Differential Network for Visual Question Generation	Generating natural questions from an image is a semantic task that requires using visual and language modality to learn multimodal representations. Images can have multiple visual and language contexts that are relevant for generating questions namely places, captions, and tags. In this paper, we propose the use of exemplars for obtaining the relevant context. We obtain this by using a Multimodal Differential Network to produce natural and engaging questions. The generated questions show a remarkable similarity to the natural questions as validated by a human study. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU, METEOR, ROUGE, and CIDEr).

Our contributions are: (1) A method to incorporate exemplars to learn differential embeddings that captures the subtle differences between supporting and contrasting examples and aid in generating natural questions. (2) We provide Multimodal differential embeddings, as image or text alone does not capture the whole context and we show that these embeddings outperform the ablations which incorporate cues such as only image, or tags or place information. (3) We provide a thorough comparison of the proposed network against state-of-the-art benchmarks along with a user study and statistical significance test.

Introduction	To understand the progress towards multimedia vision and language understanding, a visual Turing test was proposed by  that was aimed at visual question answering  . Visual Dialog  is a natural extension for VQA. Current dialog systems as evaluated in  show that when trained between bots, AI-AI dialog systems show improvement, but that does not translate to actual improvement for Human-AI dialog. This is because, the questions generated by bots are not natural (human-like) and therefore does not translate to improved human dialog. Therefore it is imperative that improvement in the quality of questions will enable dialog agents to perform well in human interactions. Further,  show that unanswered questions can be used for improving VQA, Image captioning and Object Classification. An interesting line of work in this respect is the work of  . Here the authors have proposed the challenging task of generating natural questions for an image. One aspect that is central to a question is the context that is relevant to generate it. However, this context changes for every image. As can be seen in Figure FIGREF1 , an image with a person on a skateboard would result in questions related to the event.

It measure human-likeness, that is the sentence is written by human or not. The consensus is measured, how often n-grams in the predicted question are appeared in the reference question. If the n-grams in the predicted question sentence is appeared more frequently in reference question then question is less informative and have low CIDer score. We provide our results using all these metrics and compare it with existing baselines.

we have used same evaluation metric is used in machine translation. BLEU  is the first metric to find the correlation between generated question with ground truth question. BLEU score is used to measure the precision value, i.e That is how much words in the predicted question is appeared in reference question. BLEU-n score measures the n-gram precision for counting co-occurrence on reference sentences. we have evaluated BLEU score from n is 1 to 4. The mechanism of ROUGE-n  score is similar to BLEU-n,where as, it measures recall value instead of precision value in BLEU. That is how much words in the reference question is appeared in predicted question.Another version ROUGE metric is ROUGE-L, which measures longest common sub-sequence present in the generated question. METEOR  score is another useful evaluation metric to calculate the similarity between generated question with reference one by considering synonyms, stemming and paraphrases. the output of the METEOR score measure the word matches between predicted question and reference question. In VQG, it compute the word match score between predicted question with five reference question. CIDer  score is a consensus based evaluation metric.

It is known that using a triplet network, we can learn a representation that accentuates how the image is closer to a supporting exemplar as against the opposing exemplar  ,  . The Joint embedding is obtained between the image and language representations. Therefore the improved representation helps in obtaining an improved context vector. Further we show that this also results in improving VQG. Are exemplars required? We had similar concerns and validated this point by using random exemplars for the nearest neighbor for MDN. (k=R in table TABREF35 ) In this case the method is similar to the baseline. This suggests that with random exemplar, the model learns to ignore the cue. Are captions necessary for our method? This is not actually necessary. In our method, we have used an existing image captioning method  to generate captions for images that did not have them. For VQG dataset, captions were available and we have used that, but, for VQA dataset captions were not available and we have generated captions while training. We provide detailed evidence with respect to caption-question pairs to ensure that we are generating novel questions.

Multimodal Differential Network [1] MDN INLINEFORM0 Finding Exemplars: INLINEFORM1 INLINEFORM2 Compute Triplet Embedding: INLINEFORM3 INLINEFORM4 Compute Triplet Fusion Embedding : INLINEFORM5 INLINEFORM6 INLINEFORM7 Compute Triplet Loss: INLINEFORM8 Compute Decode Question Sentence: INLINEFORM9 INLINEFORM10 —————————————————– Triplet Fusion INLINEFORM11 , INLINEFORM12 INLINEFORM13 :Image feature,14x14x512 INLINEFORM14 : Caption feature,1x512 Match Dimension: INLINEFORM15 ,196x512 INLINEFORM16 196x512 If flag==Joint Fusion: INLINEFORM17 INLINEFORM18 , [ INLINEFORM19 (MDN-Mul), INLINEFORM20 (MDN-Add)] If flag==Attention Fusion : INLINEFORM21 Semb INLINEFORM22 Dataset and Training Details Dataset We conduct our experiments on two types of dataset: VQA dataset  , which contains human annotated questions based on images on MS-COCO dataset. Second one is VQG-COCO dataset based on natural question  . VQA dataset VQA dataset  is built on complex images from MS-COCO dataset.

Experiments	We evaluate our proposed MDN method in the following ways: First, we evaluate it against other variants described in section SECREF19 and SECREF10 . Second, we further compare our network with state-of-the-art methods for VQA 1.0 and VQG-COCO dataset. We perform a user study to gauge human opinion on naturalness of the generated question and analyze the word statistics in Figure FIGREF22 . This is an important test as humans are the best deciders of naturalness. We further consider the statistical significance for the various ablations as well as the state-of-the-art models. The quantitative evaluation is conducted using standard metrics like BLEU  , METEOR  , ROUGE  , CIDEr  . Although these metrics have not been shown to correlate with `naturalness' of the question these still provide a reasonable quantitative measure for comparison. Here we only provide the BLEU1 scores, but the remaining BLEU-n metric scores are present in the supplementary. We observe that the proposed MDN provides improved embeddings to the decoder. We believe that these embeddings capture instance specific differential information that helps in guiding the question generation. Details regarding the metrics are given in the supplementary material.

This is illustrated in figure FIGREF43 . Tag net The tag net consists of two parts Context Extractor & Tag Embedding Net. This is illustrated in figure FIGREF45 . Extract Context: The first step is to extract the caption of the image using NeuralTalk2  model. We find the part-of-speech(POS) tag present in the caption. POS taggers have been developed for two well known corpuses, the Brown Corpus and the Penn Treebanks. For our work, we are using the Brown Corpus tags. The tags are clustered into three category namely Noun tag, Verb tag and Question tags (What, Where, ...). Noun tag consists of all the noun & pronouns present in the caption sentence and similarly, verb tag consists of verb & adverbs present in the caption sentence. The question tags consists of the 7-well know question words i.e., why, how, what, when, where, who and which. Each tag token is represented as a one-hot vector of the dimension of vocabulary size. For generalization, we have considered 5 tokens from each category of the Tags.

Tag Embedding Net: The embedding network consists of word embedding followed by temporal convolutions neural network followed by max-pooling network. In the first step, sparse high dimensional one-hot vector is transformed to dense low dimension vector using word embedding. After this, we apply temporal convolution on the word embedding vector. The uni-gram, bi-gram and tri-gram feature are computed by applying convolution filter of size 1, 2 and 3 respectability. Finally, we applied max-pooling on this to get a vector representation of the tags as shown figure FIGREF45 . We concatenated all the tag words followed by fully connected layer to get feature dimension of 512. We also explored joint networks based on concatenation of all the tags, on element-wise addition and element-wise multiplication of the tag vectors. However, we observed that convolution over max pooling and joint concatenation gives better performance based on CIDer score. INLINEFORM30 Where, T_CNN is Temporally Convolution Neural Network applied on word embedding vector with kernel size three. Place net Visual object and scene recognition plays a crucial role in the image.

Conclusion	In this paper we have proposed a novel method for generating natural questions for an image. The approach relies on obtaining multimodal differential embeddings from image and its caption. We also provide ablation analysis and a detailed comparison with state-of-the-art methods, perform a user study to evaluate the naturalness of our generated questions and also ensure that the results are statistically significant. In future, we would like to analyse means of obtaining composite embeddings. We also aim to consider the generalisation of this approach to other vision and language tasks. Supplementary Material Section SECREF8 will provide details about training configuration for MDN, Section SECREF9 will explain the various Proposed Methods and we also provide a discussion in section regarding some important questions related to our method. We report BLEU1, BLEU2, BLEU3, BLEU4, METEOR, ROUGE and CIDER metric scores for VQG-COCO dataset. We present different experiments with Tag Net in which we explore the performance of various tags (Noun, Verb, and Question tags) and different ways of combining them to get the context vectors.

Approach	In this section, we clarify the basis for our approach of using exemplars for question generation. To use exemplars for our method, we need to ensure that our exemplars can provide context and that our method generates valid exemplars. We first analyze whether the exemplars are valid or not. We illustrate this in figure FIGREF3 . We used a pre-trained RESNET-101  object classification network on the target, supporting and contrasting images. We observed that the supporting image and target image have quite similar probability scores. The contrasting exemplar image, on the other hand, has completely different probability scores. Exemplars aim to provide appropriate context. To better understand the context, we experimented by analysing the questions generated through an exemplar. We observed that indeed a supporting exemplar could identify relevant tags (cows in Figure FIGREF3 ) for generating questions. We improve use of exemplars by using a triplet network. This network ensures that the joint image-caption embedding for the supporting exemplar are closer to that of the target image-caption and vice-versa.

Method	The task in visual question generation (VQG) is to generate a natural language question INLINEFORM0 , for an image INLINEFORM1 . We consider a set of pre-generated context INLINEFORM2 from image INLINEFORM3 . We maximize the conditional probability of generated question given image and context as follows: DISPLAYFORM0  where INLINEFORM0 is a vector for all possible parameters of our model. INLINEFORM1 is the ground truth question. The log probability for the question is calculated by using joint probability over INLINEFORM2 with the help of chain rule. For a particular question, the above term is obtained as: INLINEFORM3  where INLINEFORM0 is length of the sequence, and INLINEFORM1 is the INLINEFORM2 word of the question. We have removed INLINEFORM3 for simplicity. Our method is based on a sequence to sequence network  ,  ,  . The sequence to sequence network has a text sequence as input and output. In our method, we take an image as input and generate a natural question as output. The architecture for our model is shown in Figure FIGREF4 .

Here, places in the image are labeled with scene semantic categories  , comprise of large and diverse type of environment in the world, such as (amusement park, tower, swimming pool, shoe shop, cafeteria, rain-forest, conference center, fish pond, etc.). So we have used different type of scene semantic categories present in the image as a place based context to generate natural question. A place365 is a convolution neural network is modeled to classify 365 types of scene categories, which is trained on the place2 dataset consist of 1.8 million of scene images. We have used a pre-trained VGG16-places365 network to obtain place based context embedding feature for various type scene categories present in the image. The context feature INLINEFORM31 is obtained by: INLINEFORM32 Where, INLINEFORM33 is Place365_CNN. We have extracted INLINEFORM34 features of dimension 14x14x512 for attention model and FC8 features of dimension 365 for joint, addition and hadamard model of places365. Finally, we use a linear transformation to obtain a 512 dimensional vector.

Inference	We made use of the 1250 validation images to tune the hyperparameters and are providing the results on test set of VQG-COCO dataset. During inference, We use the Representation module to find the embeddings for the image and ground truth caption without using the supporting and contrasting exemplars. The mixture module provides the joint representation of the target image and ground truth caption. Finally, the decoder takes in the joint features and generates the question. We also experimented with the captions generated by an Image-Captioning network  for VQG-COCO dataset and the result for that and training details are present in the supplementary material.

Similarly, We obtain context vectors INLINEFORM0 & INLINEFORM1 for the supporting and contrasting exemplars. Details for other fusion methods are present in supplementary.The aim of the triplet network  is to obtain context vectors that bring the supporting exemplar embeddings closer to the target embedding and vice-versa. This is obtained as follows: DISPLAYFORM0  where INLINEFORM0 is the euclidean distance between two embeddings INLINEFORM1 and INLINEFORM2 . M is the training dataset that contains all set of possible triplets. INLINEFORM3 is the triplet loss function. This is decomposed into two terms, one that brings the supporting sample closer and one that pushes the contrasting sample further. This is given by DISPLAYFORM0  Here INLINEFORM0 represent the euclidean distance between the target and supporting sample, and target and opposing sample respectively. The parameter INLINEFORM1 controls the separation margin between these and is obtained through validation data.

It contains a total of 204721 images, out of which 82783 are for training, 40504 for validation and 81434 for testing. Each image in the MS-COCO dataset is associated with 3 questions and each question has 10 possible answers. So there are 248349 QA pair for training, 121512 QA pairs for validating and 244302 QA pairs for testing. We used pre-trained caption generation model  to extract captions for VQA dataset. VQG dataset The VQG-COCO dataset  , is developed for generating natural and engaging questions that are based on common sense reasoning. This dataset contains a total of 2500 training images, 1250 validation images and 1250 testing images. Each image in the dataset contains 5 natural questions. Training Configuration We have used RMSPROP optimizer to update the model parameter and configured hyper-parameter values to be as follows: INLINEFORM23 to train the classification network . In order to train a triplet model, we have used RMSPROP to optimize the triplet model model parameter and configure hyper-parameter values to be: INLINEFORM24 .

Further details, pseudocode and results regarding these are present in the supplementary material.

Ablation Analysis	We considered different variations of our method mentioned in section SECREF19 and the various ways to obtain the joint multimodal embedding as described in section SECREF10 . The results for the VQG-COCO test set are given in table TABREF24 . In this table, every block provides the results for one of the variations of obtaining the embeddings and different ways of combining them. We observe that the Joint Method (JM) of combining the embeddings works the best in all cases except the Tag Embeddings. Among the ablations, the proposed MDN method works way better than the other variants in terms of BLEU, METEOR and ROUGE metrics by achieving an improvement of 6%, 12% and 18% in the scores respectively over the best other variant.

Mixture Module: Other Variations Hadamard method uses element-wise multiplication whereas Addition method uses element-wise addition in place of the concatenation operator of the Joint method. The Hadamard method finds a correlation between image feature and caption feature vector while the Addition method learns a resultant vector. In the attention method, the output INLINEFORM35 is the weighted average of attention probability vector INLINEFORM36 and convolutional features INLINEFORM37 . The attention probability vector weights the contribution of each convolutional feature based on the caption vector. This attention method is similar to work stack attention method  . The attention mechanism is given by: DISPLAYFORM0 where INLINEFORM38 is the 14x14x512-dimensional convolution feature map from the fifth convolution layer of VGG-19 Net of image INLINEFORM39 and INLINEFORM40 is the caption context vector. The attention probability vector INLINEFORM41 is a 196-dimensional vector. INLINEFORM42 are the weights and INLINEFORM43 is the bias for different layers. We evaluate the different approaches and provide results for the same. Here INLINEFORM44 represents element-wise addition. Evaluation Metrics Our task is similar to encoder -decoder framework of machine translation.

We have experimented with Noun tag, Verb tag and 'Wh-word' tag as shown in tables. Also, we have experimented in each tag by varying the number of tags from 1 to 7. We combined different tags using 1D-convolution, concatenation, and addition of all the tags and observed that the concatenation mechanism gives better results. As we can see in the table TABREF33 that taking Nouns, Verbs and Wh-Words as context, we achieve significant improvement in the BLEU, METEOR and CIDEr scores from the basic models which only takes the image and the caption respectively. Taking Nouns generated from the captions and questions of the corresponding training example as context, we achieve an increase of 1.6% in Bleu Score and 2% in METEOR and 34.4% in CIDEr Score from the basic Image model. Similarly taking Verbs as context gives us an increase of 1.3% in Bleu Score and 2.1% in METEOR and 33.5% in CIDEr Score from the basic Image model.

Perceptual Realism	A human is the best judge of naturalness of any question, We evaluated our proposed MDN method using a `Naturalness' Turing test  on 175 people. People were shown an image with 2 questions just as in figure FIGREF1 and were asked to rate the naturalness of both the questions on a scale of 1 to 5 where 1 means `Least Natural' and 5 is the `Most Natural'. We provided 175 people with 100 such images from the VQG-COCO validation dataset which has 1250 images. Figure FIGREF30 indicates the number of people who were fooled (rated the generated question more or equal to the ground truth question). For the 100 images, on an average 59.7% people were fooled in this experiment and this shows that our model is able to generate natural questions.

Statistical Significance Analysis	We have analysed Statistical Significance  of our MDN model for VQG for different variations of the mixture module mentioned in section SECREF10 and also against the state-of-the-art methods. The Critical Difference (CD) for Nemenyi  test depends upon the given INLINEFORM0 (confidence level, which is 0.05 in our case) for average ranks and N (number of tested datasets). If the difference in the rank of the two methods lies within CD, then they are not significantly different and vice-versa. Figure FIGREF29 visualizes the post-hoc analysis using the CD diagram. From the figure, it is clear that MDN-Joint works best and is statistically significantly different from the state-of-the-art methods.

Baseline and State-of-the-Art	The comparison of our method with various baselines and state-of-the-art methods is provided in table TABREF26 for VQA 1.0 and table TABREF27 for VQG-COCO dataset. The comparable baselines for our method are the image based and caption based models in which we use either only the image or the caption embedding and generate the question. In both the tables, the first block consists of the current state-of-the-art methods on that dataset and the second contains the baselines. We observe that for the VQA dataset we achieve an improvement of 8% in BLEU and 7% in METEOR metric scores over the baselines, whereas for VQG-COCO dataset this is 15% for both the metrics. We improve over the previous state-of-the-art  for VQA dataset by around 6% in BLEU score and 10% in METEOR score. In the VQG-COCO dataset, we improve over  by 3.7% and  by 3.5% in terms of METEOR scores.

We also get good results on the VQA dataset (as shown in Table TABREF26 ) which shows that our method doesn't necessitate the presence of ground truth captions. We train our model separately for VQG-COCO and VQA dataset.

Cost function	Our objective is to minimize the total loss, that is the sum of cross entropy loss and triplet loss over all training examples. The total loss is: DISPLAYFORM0  where INLINEFORM0 is the total number of samples, INLINEFORM1 is a constant, which controls both the loss. INLINEFORM2 is the triplet loss function EQREF13 . INLINEFORM3 is the cross entropy loss between the predicted and ground truth questions and is given by: INLINEFORM4  where, INLINEFORM0 is the total number of question tokens, INLINEFORM1 is the ground truth label. The code for MDN-VQG model is provided .

Our model contains three main modules, (a) Representation Module that extracts multimodal features (b) Mixture Module that fuses the multimodal representation and (c) Decoder that generates question using an LSTM-based language model. During inference, we sample a question word INLINEFORM0 from the softmax distribution and continue sampling until the end token or maximum length for the question is reached. We experimented with both sampling and argmax and found out that argmax works better. This result is provided in the supplementary material.

Dataset	We conduct our experiments on Visual Question Generation (VQG) dataset  , which contains human annotated questions based on images of MS-COCO dataset. This dataset was developed for generating natural and engaging questions based on common sense reasoning. We use VQG-COCO dataset for our experiments which contains a total of 2500 training images, 1250 validation images, and 1250 testing images. Each image in the dataset contains five natural questions and five ground truth captions. It is worth noting that the work of  also used the questions from VQA dataset  for training purpose, whereas the work by  uses only the VQG-COCO dataset. VQA-1.0 dataset is also built on images from MS-COCO dataset. It contains a total of 82783 images for training, 40504 for validation and 81434 for testing. Each image is associated with 3 questions. We used pretrained caption generation model  to extract captions for VQA dataset as the human annotated captions are not there in the dataset.

The caption INLINEFORM4 results in a caption embedding INLINEFORM5 through an LSTM parameterized by a function INLINEFORM6 where INLINEFORM7 are the weights for the LSTM. This is shown in part 1 of Figure FIGREF4 . Similarly we obtain image embeddings INLINEFORM8 & INLINEFORM9 and caption embeddings INLINEFORM10 & INLINEFORM11 . DISPLAYFORM0  The Mixture module brings the image and caption embeddings to a joint feature embedding space. The input to the module is the embeddings obtained from the representation module. We have evaluated four different approaches for fusion viz., joint, element-wise addition, hadamard and attention method. Each of these variants receives image features INLINEFORM0 & the caption embedding INLINEFORM1 , and outputs a fixed dimensional feature vector INLINEFORM2 . The Joint method concatenates INLINEFORM3 & INLINEFORM4 and maps them to a fixed length feature vector INLINEFORM5 as follows: DISPLAYFORM0  where INLINEFORM0 is the 4096-dimensional convolutional feature from the FC7 layer of pretrained VGG-19 Net  . INLINEFORM1 are the weights and INLINEFORM2 is the bias for different layers. INLINEFORM3 is the concatenation operator.

Decoder: Question Generator	The role of decoder is to predict the probability for a question, given INLINEFORM0 . RNN provides a nice way to perform conditioning on previous state value using a fixed length hidden vector. The conditional probability of a question token at particular time step INLINEFORM1 is modeled using an LSTM as used in machine translation  . At time step INLINEFORM2 , the conditional probability is denoted by INLINEFORM3 , where INLINEFORM4 is the hidden state of the LSTM cell at time step INLINEFORM5 , which is conditioned on all the previously generated words INLINEFORM6 . The word with maximum probability in the probability distribution of the LSTM cell at step INLINEFORM7 is fed as an input to the LSTM cell at step INLINEFORM8 as shown in part 3 of Figure FIGREF4 . At INLINEFORM9 , we are feeding the output of the mixture module to LSTM. INLINEFORM10 are the predicted question tokens for the input image INLINEFORM11 . Here, we are using INLINEFORM12 and INLINEFORM13 as the special token START and STOP respectively.

The softmax probability for the predicted question token at different time steps is given by the following equations where LSTM refers to the standard LSTM cell equations: INLINEFORM14  Where INLINEFORM0 is the probability distribution over all question tokens. INLINEFORM1 is cross entropy loss.

Related Work	Generating a natural and engaging question is an interesting and challenging task for a smart robot (like chat-bot). It is a step towards having a natural visual dialog instead of the widely prevalent visual question answering bots. Further, having the ability to ask natural questions based on different contexts is also useful for artificial agents that can interact with visually impaired people. While the task of generating question automatically is well studied in NLP community, it has been relatively less studied for image-related natural questions. This is still a difficult task  that has gained recent interest in the community. Recently there have been many deep learning based approaches as well for solving the text-based question generation task such as  . Further,  have proposed a method to generate a factoid based question based on triplet set {subject, relation and object} to capture the structural representation of text and the corresponding generated question. These methods, however, were limited to text-based question generation. There has been extensive work done in the Vision and Language domain for solving image captioning, paragraph generation, Visual Question Answering (VQA) and Visual Dialog.  ,  ,  proposed conventional machine learning methods for image description.

Question: How do the authors define a differential network?
Answer: The proposed Multimodal Differential Network (MDN) consists of a representation module and a joint mixture module.