Ensemble based discriminative models for Visual Dialog Challenge 2018
This manuscript describes our approach for the Visual Dialog Challenge 2018. We use an ensemble of three discriminative models with different encoders and decoders for our final submission. Our best performing model on 'test-std' split achieves the NDCG score of 55.46 and the MRR value of 63.77, securing third position in the challenge.
Introduction Models Models ::: LF-RCNN Models ::: MN-RCNN Models ::: MN-RCNN-Wt Models ::: Ensembling Experiments and Results Conclusion
Visual dialog BIBREF0 is an interesting new task combining the research efforts from Computer Vision, Natural Language Processing and Information Retrieval. While BIBREF1 presents some tips and tricks for VQA 2.0 Challenge, we follow their guidelines for the Visual Dialog challenge 2018. Our models use attention similar to BIBREF2 to get object level image representations from Faster R-CNN model BIBREF3. We experiment with different encoder mechanisms to get representations of conversational history.
Common to all the models, we initialize our embedding matrix with pre-trained Glove word vectors of 300 dimensions using 6B tokens . Out of 11319 tokens present in the dataset, we found 188 tokens missing from the pre-trained Glove embeddings, so we manually map these tokens to words conveying semantically similar meaning, e.g. we map over ten variations of the word “yes” - misspelled or not picked up by tokenizer - “*yes", “yesa", “yess", “ytes", “yes-", “yes3", “yyes", “yees", etc. For image features, we extract Faster R-CNN features with ResNet-101 backbone trained on Visual genome BIBREF4 dataset, similar to BIBREF2. We use an adaptive number of object proposals per-image ranging from 10 to 100 generated using a fixed confidence threshold and each object is then associated with 2048-dimensional mean-pooled features using ROI pooling. We use discriminative decoding throughout our models. We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0.
Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture.
Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6)
Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
We ensembled final layer's log-softmax output - which is a distribution over candidate answers for each round (Figure FIGREF7). We use the three models described above and take the mean of the results (we also tried taking maximum of the results but found mean to perform better). We also tried ensembling a subset of the above three models, but found the combination of all three to outperform the rest.
We used Pytorch BIBREF5 for implementation. In our experiments, we find that fine-tuning initialized Glove embeddings performed better than frozen embeddings. Object level representations play a critical role to generate a correct response from the model. Eventually, we use an ensemble of all the models described above for our final submission. Table TABREF9 summarizes our results on validation set while Table TABREF8 on Test-Standard split.
We experimented with discriminative models for our submission. Object level image representations gave a huge uplift in the evaluation metrics. Bi-directional GRUs constantly performed better than uni-directional LSTMs with Memory Networks outperforming Late fusion encoders for encoding conversational history. We even found that fine-tuning Glove embeddings performed better than their counterparts. Our final submission is an ensemble of three discriminative models and achieve the NDCG of 55.46 on test-std.
What was the baseline? Which three discriminative models did they use?
6f024d4c952fbb53b6f8cf27020659aa4d9a6590 d2a0142150cc3788475572f82458ef88087bd7ac
c1fbdd7a261021041f75fbe00a55b4c386ebbbb4 c1fbdd7a261021041f75fbe00a55b4c386ebbbb4
f7c6919c984185022d59e0f9767fadf601a92206
c1018a31c3272ce74964a3280069f62f314a1a58
LF-RCNN MN-RCNN MN-RCNN-Wt
We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0. Models ::: LF-RCNN Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture. Models ::: MN-RCNN Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6) Models ::: MN-RCNN-Wt Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0.

Models ::: LF-RCNN
Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture.

Models ::: MN-RCNN
Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6)

Models ::: MN-RCNN-Wt
Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
LF-RCNN MN-RCNN MN-RCNN-Wt
For image features, we extract Faster R-CNN features with ResNet-101 backbone trained on Visual genome BIBREF4 dataset, similar to BIBREF2. We use an adaptive number of object proposals per-image ranging from 10 to 100 generated using a fixed confidence threshold and each object is then associated with 2048-dimensional mean-pooled features using ROI pooling. We use discriminative decoding throughout our models. We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0. Models ::: LF-RCNN Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture. Models ::: MN-RCNN Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6) Models ::: MN-RCNN-Wt Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
We use discriminative decoding throughout our models.

We first describe our models individually and then the ensembling technique that we employ. Models ::: LF-RCNN
Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history.  Models ::: MN-RCNN
Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Models ::: MN-RCNN-Wt
Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. 
392ffb4d60dcacc0267497c8778052997df80493 488488449cf0b67be13b4ed9a5ed9e0683d7402a
34c35a1877e453ecaebcf625df3ef788e1953cc4 c1018a31c3272ce74964a3280069f62f314a1a58
Figure 2. MN-RCNN model Figure 1. LF-RCNN model as defined in Section 2. Number indicates the output dimension of each layer. Figure 4. Ensemble using mean of log probabilities from the three individual models. Table 1. Results for the challenge on test-std. We took an ensemble of best performing models for the final submission. Table 2. Results on validation set. We show the impact of using additional gated linear layer in decoder. Compare MN with MNWt.
2-Figure2-1.png 2-Figure1-1.png 3-Figure4-1.png 3-Table1-1.png 3-Table2-1.png