\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{comment}
\usepackage{amsmath,amssymb} %
\usepackage{color}

\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}

\usepackage{epsfig}
\usepackage{ mathrsfs }
\usepackage{xcolor}
\usepackage{tablefootnote}
\usepackage{ stmaryrd }
\usepackage[ruled,vlined,linesnumbered]{algorithm2e}
\usepackage{tabularx} %
\usepackage{multirow}
\usepackage{array}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\usepackage{supertabular}
\usepackage{enumitem}
\usepackage{ dsfont }
\usepackage[toc,page]{appendix}
\DeclareMathOperator*{\argmax}{arg\,max}
\newcommand\red[1]{{\color{red}#1}}

\newcommand\brown[1]{{\color{brown}#1}}
\newcommand{\tuong}[1]{\brown{#1}}

\usepackage{pifont}%
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%

\def\R{{\mathbb R}}
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

\renewcommand{\baselinestretch}{0.98} \normalsize
\usepackage{floatrow}
\begin{document}
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{1}  %

\title{Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering} %
\titlerunning{MILQT}
\author{Tuong Do\inst{1} \and
Binh X. Nguyen\inst{1} \and
Huy Tran\inst{1} \and
Erman Tjiputra\inst{1} \and
Quang D. Tran\inst{1}\and
Thanh-Toan Do\inst{2}
}
\authorrunning{Tuong Do et al.}
\institute{AIOZ, Singapore \\ \email{\{tuong.khanh-long.do,binh.xuan.nguyen,huy.tran,\\erman.tjiputra,quang.tran\}@aioz.io} \and University of Liverpool \\
\email{thanh-toan.do@liverpool.ac.uk}}
\maketitle
\begin{abstract}
Different approaches have been proposed to Visual Question Answering (VQA). However, few works are aware of the behaviors of varying joint modality methods over question type prior knowledge extracted from data in constraining answer search space, of which information gives a reliable cue to reason about answers for questions asked in input images. In this paper, we propose a novel VQA model that utilizes the question-type prior information to improve VQA by leveraging the multiple interactions between different joint modality methods based on their behaviors in answering questions from different types. The solid experiments on two benchmark datasets, i.e., VQA 2.0 and TDIUC, indicate that the proposed method yields the best performance with the most competitive approaches.
\keywords{visual question answering, multiple interaction learning.}
\end{abstract}

\section{Introduction}

The task of Visual Question Answering (VQA) is to provide a correct answer to a given question such that the answer is consistent with the visual content of a given image. The VQA research raises a rich set of challenges because it is an intersection of different research fields including computer vision, natural language processing, and reasoning. 
Thanks to its wide applications, the VQA has attracted great attention in recent years~\cite{VQA,Xu2016AskAA,Yang2016StackedAN,bottom-up2017,Kim2018BilinearAN,MTL_QTA}. This also leads to the presence of large scale datasets~\cite{VQA,vqav22016,Kushal2018Tdiuc} and evaluation protocols~\cite{VQA,Kushal2018Tdiuc}.   

There are works that consider types of question as the side information which gives a strong cue to reason about the answer \cite{2017AgrawalPriorVQA,MTL_QTA,kafle2016answer}. However, the relation between question types and answers from training data have not been investigated yet. Fig.~\ref{fig:distribution_graph} shows the correlation between question types and some answers in the  VQA 2.0 dataset \cite{vqav22016}. It suggests that a question regarding the quantity should be answered by a number, not a color. The observation indicated that the prior information got from the correlations between question types and answers open an answer search space constrain for the VQA model. The search space constrain is useful for VQA model to give out final prediction and thus, improve the overall performance. 
The Fig.~\ref{fig:distribution_graph} is consistent with our observation, e.g., it clearly suggests that a question regarding the quantity should be answered by a number, not a color.
\begin{figure}
    \centering
    \includegraphics[width = \columnwidth*8/9, keepaspectratio=True]{Distribution_graph.png}
    \caption{The distribution of candidate answers in each question type in VQA 2.0. 
    }
    \label{fig:distribution_graph}
\end{figure}
\begin{figure*}[!t]
    \centering
    \includegraphics[width=\textwidth*9/10, keepaspectratio=true]{diff_attentions.png}
 \caption{Examples of attention maps of different attention mechanisms. BAN~\cite{Kim2018BilinearAN} and SAN~\cite{Yang2016StackedAN} identify different visual areas when answering questions from different types. \cmark\ and \xmark\ indicate correct and wrong answers, respectively.}
    \label{fig:diff_attentions}
\end{figure*}

In current state-of-the-art VQA systems, the joint modality component plays an important role since it would learn meaningful joint representations  between linguistic and visual inputs~\cite{Xu2016AskAA,Yang2016StackedAN,bottom-up2017,Kim2018BilinearAN,dense-attention,tan2019lxmert}.
Although different joint modality methods or attention mechanisms have been proposed, we hypothesize that each method may capture different aspects of the input. That means different attentions may provide different answers for questions belonged to different question types. 
Fig.~\ref{fig:diff_attentions} shows examples in which the attention models (BAN~\cite{Kim2018BilinearAN} and SAN~\cite{Yang2016StackedAN}) attend on different regions of input images when dealing with questions from different types. Unfortunately, most of recent VQA systems are based on single attention models~\cite{Xu2016AskAA,Yang2016StackedAN,bottom-up2017,Kim2018BilinearAN,MTL_QTA,Fukui2016MultimodalCB}. From the above observation, it is necessary to develop a VQA system which leverages the power of different attention models to deal with questions from different question types.

In this paper, we propose a multiple interaction learning with question-type prior knowledge (MILQT) which extracts the question-type prior knowledge from questions to constrain the answer search space and leverage different behaviors of multiple attentions in dealing with questions from different types. 

Our contributions are summarized as follows. 
(i) We propose a novel VQA model that leverages the question-type information to augment the  VQA loss.
(ii) We identified that different attentions shows different performance in dealing with questions from different types and then leveraged this characteristic to rise performance through our designed model.
(iii) The extensive experiments show that the proposed model yields the best performance with the most competitive approaches in the widely used  VQA 2.0~\cite{vqav22016} and TDIUC~\cite{Kushal2018Tdiuc} datasets.

\section{Related Work}
\textbf{Visual Question Answering}. 
In recent years, VQA has attracted a large attention from both computer vision and natural language processing communities. 
The recent VQA researches mainly focus on the development of different attention models. In~\cite{Fukui2016MultimodalCB}, the authors proposed the Multimodal Compact Bilinear (MCB) pooling by projecting the visual  and linguistic features to a higher dimensional space and then convolving both vectors efficiently by using element-wise product in Fast Fourier Transform space. 
In \cite{Yang2016StackedAN}, the authors proposed Stacked Attention Networks (SAN) which locate, via multi-step reasoning, image regions that are relevant to the question for answer prediction. In~\cite{bottom-up2017,tip-trick}, the authors employed the top-down attention that learns an attention weight for each image region by applying non-linear transformations on the combination of image features and linguistic features. In~\cite{dense-attention}, the authors proposed a dense, symmetric attention model that allows each question word attends on image regions and each image region attends on question words. In~\cite{Kim2018BilinearAN} the authors proposed Bilinear Attention Networks (BAN) that find bilinear attention distributions to utilize given visual-linguistics information seamlessly. Recently, in \cite{tan2019lxmert} the authors introduced Cross Modality Encoder Representations (LXMERT) to learn the alignment/ relationships between visual concepts and language semantics.

Regarding the question type, previous works have considered question-type information to improve VQA results. 
Agrawal et al. \cite{2017AgrawalPriorVQA} trained a separated question-type classifier to classify input questions into two categories, i.e., Yes-No and non Yes-No. Each category will be subsequently processed in different ways. In the other words, the question type information is only used for selecting suitable sub-sequence processing.
Shi et al. \cite{MTL_QTA} also trained a question-type classifier to predict the question type. The predicted one-hot question type is only used to weight the importance of different visual features. 
Kafle et al. \cite{kafle2016answer} also used question type to improve the performance of VQA prediction. Similar to \cite{2017AgrawalPriorVQA}, the authors separately trained a classifier to predict the type of the input question. The predicted question type is then used to improve VQA prediction through a Bayesian inference model.
 
In our work, different from~\cite{2017AgrawalPriorVQA}, \cite{MTL_QTA} and \cite{kafle2016answer}, question types work as the prior knowledge, which constrain answer search space through loss function. Additionally, we can further identify the performance of different joint modality methods over questions from different types.
Besides, through the multiple interaction learning, the behaviors of the joint modality methods are utilized on giving out the final answer which further improve VQA performance.

\section{Methodology}
\begin{figure*}
    \centering
    \includegraphics[width=\textwidth*8/10, keepaspectratio=true]{vqa-net-diagram.png}
    \caption{The proposed MILQT for VQA. 
    }
    \label{fig:framework}
\end{figure*}
The proposed  multiple interaction learning with question-type prior knowledge (MILQT) is illustrated in Fig.~\ref{fig:framework}. 
Similar to the most of the VQA systems \cite{Kim2018BilinearAN,Yang2016StackedAN,bottom-up2017}, multiple interaction learning with question-type prior knowledge (MILQT) consists of the joint learning solution for input questions and images, followed by a multi-class classification over a set of predefined candidate answers. However, MILQT allows to leverage multiple joint modality methods under the guiding of question-types to output better answers.

As in Fig.~\ref{fig:framework}, MILQT consists of two modules: Question-type awareness $\mathcal{A}$, and Multi-hypothesis interaction learning $\mathcal{M}$. The first module aims to learn the question-type representation, which is further used to enhance the joint visual-question embedding features and to constrain answer search space through prior knowledge extracted from data. Based on the question-type information, the second module aims to identify the behaviors of multiple joint learning methods and then justify adjust contributions to giving out final predictions.

In the following, we describe the representation of input questions and images in Section~\ref{subsec:rep}. Section~\ref{subsec:qt-awa} presents the Question-type awareness module $\mathcal{A}$. Section~\ref{subsec:interaction} presents the Multi-hypothesis interaction learning module $\mathcal{M}$. 
Section~\ref{subsec:overall-loss} presents the multi-task loss for entire model training. 
\subsection{Input Representation}
\label{subsec:rep}
\textbf{Question representation.} 
Given an input question, follow the recent state-of-the-art~\cite{bottom-up2017,Kim2018BilinearAN}, we trim the question to a maximum of 12 words. The questions that are shorter than 12 words are zero-padded. Each word is then represented by a 600-D vector that is a concatenation of the 300-D GloVe word embedding \cite{pennington2014glove} and the augmenting embedding from training data as ~\cite{Kim2018BilinearAN}. This step results in a sequence of word embeddings with size of $12 \times 600$ and is denoted as $f_w$ in Fig~\ref{fig:framework}. In order to obtain the intent of question, the $f_w$ is passed through a Gated Recurrent Unit (GRU)~\cite{2014ChoGRU} which results in a 1024-D vector representation $f_q$ for the input question.

\textbf{Image representation.}
There are several object detectors have been proposed in the literature, of which outputs vary in size and location. Inspired by recent advances of VQA~\cite{bottom-up2017,MTL_QTA,tip-trick}, we use bottom-up attention, i.e. an object detection which takes as FasterRCNN \cite{Ren2015FasterRCNN} backbone, to extract image representation. At first, the input image is passed through bottom-up networks to get  $K \times 2048$ bounding box representation which is denotes as $f_v$ in Fig. \ref{fig:framework}.
\subsection{Question-type Awareness}  
\label{subsec:qt-awa}
\textbf{Question-type classification.}
This component in module $\mathcal{A}$ aims to learn the question-type representation.
Specifically, aforementioned component takes the question embedding $f_q$ as input, which is then passed through several fully-connected (FC) layers and is ended by a softmax layer which produces a probability distribution $h$ over $P$ question types, where $P$ depends on the dataset, i.e., $P$ equals $3$ for VQA 2.0~\cite{vqav22016} and equals $12$ for TDIUC~\cite{Kushal2018Tdiuc}. The question type embedding $f_{qt}$ extracted from question-type classification component will be combined with the attention features to enhance the joint semantic representation between the input image and question, while the predicted question type will be used to augment the VQA loss. 

\textbf{Multi-level multi-modal fusion.}
Unlike the previous works that perform only one level of fusion between linguistic and visual features that may limit the capacity of these models to learn a good joint semantic space. In our work, a multi-level multi-modal fusion that encourages the model to learn a better joint semantic space is introduced which takes the question-type representation got from question-type classification component as one of inputs.

\textit{First level multi-modal fusion:}
The first level fusion is similar to previous works~\cite{bottom-up2017,Kim2018BilinearAN,Yang2016StackedAN}. Given visual features $f_v$, question features $f_{q}$, and any joint modality mechanism (e.g., bilinear attention~\cite{Kim2018BilinearAN}, stacked attention~\cite{Yang2016StackedAN}, bottom-up~\cite{bottom-up2017} etc.), 
we combines visual features with question features and learn attention weights to weight for visual and/or linguistic features. Different attention mechanisms have different ways for learning the joint semantic space. The detail of each attention mechanism can be found in the corresponding studies~\cite{Yang2016StackedAN,Kim2018BilinearAN,bottom-up2017}. 
The output of first level multi-modal fusion is denoted as $f_{att}$ in the Fig.~\ref{fig:framework}. 

\textit{Second level multi-modal fusion:}
In order to enhance the joint semantic space, the output of the first level multi-modal fusion $f_{att}$ is combined with the question-type feature $f_{qt}$, which is the output of the last FC layer of the ``Question-type classification'' component. 
We try two simple but effective operators, i.e. \textit{element-wise multiplication --- EWM} or \textit{element-wise addition --- EWA}, to combine $f_{att}$ and $f_{qt}$. The output of the second level multi-modal fusion, which is denoted as $f_{att-qt}$ in Fig.~\ref{fig:framework}, can be seen as an attention representation that is aware of the question-type information. 

Given an attention mechanism, the $f_{att-qt}$ will be used as the input for a classifier that predicts an answer for the corresponding question. This is shown at the ``Answer prediction'' boxes in the Fig.~\ref{fig:framework}. 

\textbf{Augmented VQA loss.} The introduced loss function takes model predicted question types and prior knowledge question types from data to identify the answer search space constraints when the model outputs predicted answers.

\textit{Prior computation.} 
In order to make the VQA classifier pay more attention on the answers corresponding to the question type of the input question, we use the statistical information from training data to identify the relation between the question type and the answer.
The Alg.~\ref{alg:mapping} presents the calculation of the prior information between the question types and the answers. 
To calculate the prior, we firstly make statistics of the frequency of different question types in each VQA candidate answer. This results in a matrix  $m_{qt-ans}$ (lines 2 to 4). 
We then column-wise normalize the matrix $m_{qt-ans}$ by dividing elements in a column by the sum of the column (lines 5 to 7). 
 
\begin{algorithm}
\label{alg:mapping}
\DontPrintSemicolon
\SetAlgoLined
\SetKwInOut{Input}{Input}\SetKwInOut{Output}{Output}

\Input{$Q$: number of questions in training set.\\
$P$: number of question types.\\
$A$: number of candidate answers.\\
$qtLabels \in  \{1,...,P\}^{Q \times  1}$: type labels of questions in training set. \\
$ansLabels \in \{1,...,A\}^{Q \times 1}$: answer labels of questions in training set.}
\Output{$m_{qt-ans}$ $\in \R^{P
\times A}$: relational prior of question types and answers.}
$m_{qt-ans} = zeros(P,A)$ /* init   $m_{qt-ans}$ with all zero values */\;
\For {$q = 1 \rightarrow Q$}{
   $m_{qt-ans} [qtLabels[q], ansLabels[q]]$ += 1 \;
}
\For {$a = 1 \rightarrow A$}{
    $m_{qt-ans}[:,a]$ = $normalize (m_{qt-ans}[:,a])$ \\
}
\caption{Question type - answer relational prior computation}
\end{algorithm}
  
\textit{Augmented VQA loss function design $l_{vqa}$.} 
Let $y_i \in \R^{A \times 1}$, $g_i \in \R^{A \times 1}$, $h_i \in \R^{P \times 1}$ be the VQA groundtruth answer, VQA answer prediction, and the question-type prediction of the $i^{th}$ input question-image, respectively. 
Given the question, our target is to increase the chances of possible answers corresponding to the question type of the question.
To this end, we first define the weighting (question-type) awareness matrix $m_{awn}$ by combining the predicted question-type $h_i$ and the prior information $m_{qt-ans}$ as follows:
\begin{equation}
    m_{awn} = {h_i}^T m_{qt-ans}
    \label{eq:m_awn}
\end{equation}
This weighting matrix is used to weight the VQA groundtruth $y_i$ and VQA answer prediction $g_i$ to as follows:
\begin{equation}
\hat{y}_i=  m_{awn}^{T} \odot y_i
\end{equation}
\begin{equation}
\hat{g}_i= m_{awn}^{T} \odot g_i
\end{equation}
where $\odot$ is the element-wise product. As a result, this weighting increases the chances of possible answers corresponding to the question type of the question. Finally, the VQA loss $l_{vqa}$ is computed as follows: \\

\begin{equation}
\begin{aligned}
\label{eq:vqaloss}
 &l_{vqa} = -  \frac{1}{QA}\sum_{i=1}^{Q}\sum_{j=1}^{A} \hat{y}_{ij} \log (\sigma(\hat{g}_{ij}))+ (1-\hat{y}_{ij})\log(1-\sigma(\hat{g}_{ij}))\\
\end{aligned}
\end{equation}
where $Q$ and $A$ are the number of training questions and candidate answers; $\sigma$ is the element-wise sigmoid function. (\ref{eq:vqaloss}) is a \textit{soft} cross entropy loss and has been shown to be more effective than softmax in VQA problem~\cite{tip-trick}. 

It is worth noting that when computing the weighting matrix $a_{awn}$ in (\ref{eq:m_awn}), instead of using the predicted question type $h_i$, we can also use the groundtruth question type. 
However, we found that there are some  inconsistency between the groundtruth question types and the groundtruth answers. For example, in VQA 2.0 dataset, most of questions started by ``how many" are classified with the question type ``number", and the answers to these questions are numeric numbers. However, there are also some exceptions. For example, the question \textit{``How many stripes are there on the zebra?''} is annotated with the  groundtruth question-type  ``number" but its annotated groundtruth answer is ``many", which is not a numeric number. By using groundtruth question type to augment the loss, the answer to that question is likely a numeric number, which is an incorrect answer compared to the groundtruth answer. In order to make the model robust to these exceptions, we use the predicted question type to augment the VQA loss. Using the predicted question type can be seen as a self-adaptation mechanism that allows the system to adapt to exceptions.
In particular, for the above example, the predicted question type may not be necessary ``number'' and it can be ``other''.
\subsection{Multi-hypothesis interaction learning}
\label{subsec:interaction}
As presented in Fig.~\ref{fig:framework}, MILQT allows to utilize multiple hypotheses (i.e., joint modality mechanisms). Specifically, we propose a multi-hypothesis interaction learning design $\mathcal{M}$ that takes answer predictions produced by different joint modality mechanisms and interactively learn to combine them.
Let $g \in \R^{A \times J}$ be the matrix of predicted probability distributions over $A$ answers from the $J$ joint modality mechanisms. $\mathcal{M}$ outputs the distribution $\rho \in \R^{A}$, which is calculated from $g$ through Equation (\ref{eq:multi-hypothesis}).
\begin{equation}
\begin{aligned}
&\rho  = \mathcal{M} \left(g,w_{mil}\right) = \sum_{j}\left(m^T_{qt-ans}w_{mil} \odot g\right)
\end{aligned}
\label{eq:multi-hypothesis}
\end{equation}
$w_ {mil} \in \mathds{R}^{P \times J}$ is the learnable weight which control the contributions of $J$ considered joint modality mechanisms on predicting answer based on the guiding of $P$ question types; $\odot$ denotes Hardamard product.

\subsection{Multi-task loss} 
\label{subsec:overall-loss}
In order to train the proposed MILQT, we define a multi-task loss to jointly optimize the question-type classification, the answer prediction of each individual attention mechanism, and the VQA loss (\ref{eq:vqaloss}). 
Formally, our multi-task loss is defined as follows:
\begin{equation}
l = \alpha_1\sum_{j=1}^{k} l_{H_j} +\alpha_2 l_{vqa} + \alpha_3 l_{qt}
\label{eq:final_loss}
\end{equation}
where $\alpha_1, \alpha_2, \alpha_3$ are parameters controlling the importance of each loss; $l_{qt}$ is the question-type classification loss; $l_{H_j}$ is the answer prediction loss of $j^{th}$ mechanism over $J$ joint modality methods; $l_{vqa}$ is the introduced VQA loss augmented by the predicted question type and the prior information defined by (\ref{eq:vqaloss}). 

\section{Experiments}
\subsection{Dataset and implementation detail}
\textbf{Dataset.} We conduct the experiments on two benchmark VQA datasets that are VQA 2.0~\cite{vqav22016} and TDIUC~\cite{Kushal2018Tdiuc}.  The VQA 2.0 dataset is the most popular and is widely used in VQA problem. In VQA 2.0 dataset, questions are divided into three question types, i.e., ``Yes-No'', ``Number'' and ``Other'' while the TDIUC dataset has 12 different question types.  

As standardly done in the literature, we use the standard VQA accuracy metric \cite{VQA} when evaluating on VQA 2.0 dataset and Arithmetric  MPT  as well as Harmonic MPT proposed in \cite{Kushal2018Tdiuc} when evaluating on TDIUC\footnote{In \cite{Kushal2018Tdiuc}, the authors show that using Arithmetric  MPT and Harmonic MPT is more suitable than the standard VQA accuracy metric \cite{VQA} when evaluating on TDIUC.}. 

\textbf{Implementation detail. }
\label{subsec:implement}
Our proposed MILQT is implemented using PyTorch \cite{paszke2017automaticPyTorch}. The experiments are conducted on a single NVIDIA Titan V with 12GB RAM. 

\begin{figure*}[!t]
    \centering
    \includegraphics[width=\textwidth*9/10, keepaspectratio=true]{exp_examples.png}
    \caption{Example results of SAN \cite{Yang2016StackedAN}, BAN \cite{Kim2018BilinearAN}, and our method on the validation set of VQA 2.0. In all cases, the proposed method produces better attention maps. It also produce more accurate answers than compared methods (second row).}
    \label{fig:exp_figure}
\end{figure*}
In all experiments, the learning rate is set to $10^{-3}$ (or $7\times 10^{-4}$ if using Visual Genome \cite{visualgenome} as augmenting data) and batch size is set to $256$. The number of detected bounding boxes  is set to $50$ when extracting visual features. 
The GRU \cite{2014ChoGRU} for question embedding has one layer with $1024$-D hidden state and processes words in forward order. 
During training, except image representations $f_v$, other components are trained end-to-end with the multi-task loss (\ref{eq:final_loss}). AdaMax optimizer \cite{Kingma2014AdamAM} is used to train our model.
\begin{table}[!t]
\begin{center}
\small
\begin{tabular}{l c} 
\hline
 \begin{tabular}[l]{@{}l@{}}\textbf{Models}\end{tabular}                                                                  &\textbf{VQA score}\\
\hline
\multicolumn{2}{c}{\textbf{Contribution of question type awareness}}     \\ 
BAN-2-Counter     \cite{Kim2018BilinearAN}                                                    &65.25  \\
\quad + add                                                                    &65.68\\
\quad\quad  + prior                                                                   &66.04\\

\quad + mul                                                                    &65.80\\
\quad\quad + prior                                                                    &66.13\\
\hline
\multicolumn{2}{c}{\textbf{Contribution of hypothesis interaction learning}}     \\
BAN-2-Counter \cite{Kim2018BilinearAN}                                                                  &65.25  \\

\quad + BAN-2 \cite{Kim2018BilinearAN}                                                                 &66.15\\

\quad  + SAN \cite{Yang2016StackedAN}                                                                  &65.64\\

\hline
\multicolumn{2}{c}{\textbf{Whole model testing}}     \\
BAN-2-Counter         \cite{Kim2018BilinearAN}                                                          &65.25  \\

\quad + BAN-2 \cite{Kim2018BilinearAN} + Mul + prior                                                                   &66.31\\

\quad + SAN \cite{Yang2016StackedAN} + Mul + prior                                                                    &66.48\\

\hline

\end{tabular}
\end{center}
\caption{Contributions of the proposed components and the whole model on the VQA 2.0 validation set.}
\label{tab:valeval}
\end{table}

\begin{table}[!t]
\small
\begin{center}
\begin{tabular}{|c|c|c|c|c|}
\hline
{Models}    & BAN-2        & \begin{tabular}[c]{@{}c@{}}BAN-2-\\ Counter \    \end{tabular}  & \begin{tabular}[c]{@{}c@{}}Averaging\\ Ens.\end{tabular} & \begin{tabular}[c]{@{}c@{}}Interaction\\ Learning\end{tabular}   \\ \hline
{Accuracy} & 65.36 & 65.25                                                & 65.61                                              & {66.15} \\ \hline
\end{tabular}
\end{center}
\caption{Performance on VQA 2.0 validation set  where BAN2 \cite{Kim2018BilinearAN} and BAN-2-Counter \cite{Kim2018BilinearAN} are ensembled using  averaging ensembling and the proposed interacting learning. 
}
\label{tab:ens}
\end{table}
\subsection{Ablation study}
To evaluate the contribution of question-type awareness $\mathcal{A}$ module and multi-hypothesis interaction learning $\mathcal{M}$ in our method, we conduct ablation studies when training on the train set and testing on the validation set of VQA 2.0 \cite{vqav22016}.

Starting with the BAN glimpse 2 with counter sub-module (BAN-2-Counter) \cite{Kim2018BilinearAN} as the baseline, we show the effectiveness of proposed modules when they are integrated into the baseline. 
The counter sub-module \cite{Zhang2018LearningToCount} is used in the baseline to prove the extendability of proposed model on supporting ``Number" question. However, any sub-modules can also be applied, e.g., relational reasoning sub-module  \cite{2017SantoroRelationalNet} to support for ``Yes/No" and ``Other" questions. It is worth noting that in order to make a fair comparison, we use the same visual features and question embedding features for both BAN-2-Counter baseline and our model. 
\begin{table}[!t]
\begin{center}
\small
\begin{tabular}{|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Question\\ types\end{tabular}}} & \multicolumn{3}{c|}{\textbf{Correlation scores}}   \\ \cline{2-4} 
                                                                                   & \textbf{BAN-Counter} & \textbf{BAN} & \textbf{SAN} \\ \hline
\textit{Yes/No}                                                                    & 0.40                     & 0.55            & 0.05             \\ \hline
\textit{Numbers}                                                                   & 0.55                     & 0.23            & 0.22             \\ \hline
\textit{Others}                                                                    & 0.35                     & 0.38             & 0.27             \\ \hline
\end{tabular}
\end{center}
\caption{The correlation scores extracted from $w_{mil}$ of MILQT. The extracted information got from model trained in VQA 2.0 train set.}
\label{tab:corr}
\end{table}

\begin{table*}[!t]
\centering
\small
\begin{center}
\begin{tabular}{l| c| c c c|c |c c c} 
\hline
\multirow{2}{*}{\textbf{Models}} 
&\multicolumn{4}{c|}{\textbf{VQA - test-dev}} &\multicolumn{4}{c}{\textbf{VQA - test-std}}     \\
\cline{2-9}
&\textbf{Overall} &\textbf{Yes/No} &\textbf{Nums} &\textbf{Other}
&\textbf{Overall} &\textbf{Yes/No} &\textbf{Nums} &\textbf{Other}\\
\hline
SAN \cite{Yang2016StackedAN}                                                                      &64.80	&79.63	&43.21	&57.09	&65.21	&80.06	&43.57	&57.24 \\
Up-Down \cite{bottom-up2017}                                                                  &65.32 &81.82 &44.21 &56.05 &65.67 &82.20 &43.90 &56.26 \\
\begin{tabular}[c]{@{}c@{}}CMP \cite{tan2019lxmert}\    \end{tabular}  
&68.7	&84.91	&50.15	&59.11	&69.23	&85.48	&49.53	&59.6\\
Pythia \cite{Jiang2018PythiaVT}                                                                      &70.01 &86.12 &48.97 &61.06 &70.24 &86.37 &48.46 &61.18 \\
BAN   \cite{Kim2018BilinearAN}                                                                      &70.04	&85.42 &54.04 &60.52 &70.35 &85.82 &53.71 &60.69 \\

\begin{tabular}[c]{@{}c@{}}LXMERT\cite{tan2019lxmert} \\     \end{tabular}  
&\textbf{72.4}	&88.3	&54.2	&62.9	&\textbf{72.5}	&88.0	&56.7	&65.2\\
\hline
\textbf{MILQT}  
&70.62	&86.47	&54.24	&60.79	&70.93	&86.80	&53.79	&61.03\\
\hline
\end{tabular}
\end{center}
\caption[Test-dev and test-standard results on VQA 2.0 dataset with single-models of different methods]
{Comparison to the state of the arts on the test-dev and test-standard of VQA 2.0. For fair comparison, in all setup except LXMERT which uses BERT \cite{Devlin2019BERTPO} as question embedding, Glove embedding and GRU are leveraged for question embedding and Bottom-up features are used to extract visual information. CMP, i.e.Cross-Modality with Pooling, is the LXMERT with the aforementioned setup.
}
\label{tab:VQA}
\end{table*}

\textbf{The effectiveness of question-type awareness and prior information proposed in Section~\ref{subsec:qt-awa}.} The first section in Table \ref{tab:valeval} shows that by having second level multi-modal fusion (Section~\ref{subsec:qt-awa})  which uses element-wise multiplication (\textit{+mul}) to combine the question-type feature $f_{qt}$ and the attention feature $f_{att}$, the overall performance increases from $65.25\%$ (baseline) to $65.80\%$. 
 By further using the predicted question type and the prior information (\textit{+prior}) to augment the VQA loss, the performance increases to $66.13\%$ which is $+0.88\%$ 
 improvement over the baseline.
The results in the first section in Table \ref{tab:valeval} confirm that combining question-type features with attention features helps to learn a better joint semantic space, which leads to the performance boost over the baseline. These results also confirm that using the predicted question type and the prior provides a further boost in the performance. 
We also find out that using EWM provides better accuracy than EWA at the second level fusion.
\textbf{The effectiveness of multi-hypothesis interaction learning proposed in Section~\ref{subsec:interaction}.}  The second section in Table \ref{tab:valeval} shows the effectiveness when leveraging different joint modality mechanisms by using multi-hypothesis interaction learning. By using  BAN-2-Counter \cite{Kim2018BilinearAN} and BAN-2 \cite{Kim2018BilinearAN} (BAN-2-Counter + BAN-2), the overall performance is $66.15\%$ which is $+0.9\%$ improvement over the BAN-2-Counter baseline.

Table \ref{tab:corr} illustrates the correlation between different joint modality mechanisms and question types. This information is extracted from $w_{mil}$ which identify the contributions of each mechanism in giving final VQA results guiding by the question type information.

The results in Table \ref{tab:VQA} indicate that some joint modality methods achieve better performance in some specific question types, e.g., joint modality method BAN outperform other methods in Number question type by a large margin. The correlation in Table \ref{tab:corr} and performance in Table \ref{tab:VQA} also indicates that the MILQT model tends to leverage the contribution of joint methods proportional to their performance in each specific question type. Besides, the results in Table \ref{tab:ens} indicate that under the guiding of question type, $\mathcal{M}$ module produce better performance when comparing with none-use solution or the weighted sum method \cite{li2019regat} in which the predictions of different joint modality mechanisms are summed up and the answer with highest score are considered as the final answer.

\textbf{The effectiveness of the entire proposed model.}
The third section in Table \ref{tab:valeval} presents results when all components (except the visual feature extractor) are combined in a unified model and are trained end-to-end. To verify the effectiveness of the proposed framework, we conduct two configurations. In the first configuration, we use two joint modality mechanisms BAN-2-Counter and BAN-2, the EWM in the second level multi-modal fusion, and the predicted question type together with the prior information to augment the loss. 
The second configuration is similar to the first configuration, except that we use BAN-2-Counter and SAN in interaction learning.
The third section on Table \ref{tab:valeval} shows that both configurations give the performance boost over the baseline. The second configuration achieves better performance, i.e., $66.48\%$ accuracy,  which outperforms over the baseline BAN-2-Counter $+1.23\%$. Table \ref{tab:valeval} also show that using  ``question-type awareness" gives further boost over using interaction learning only, i.e., the performance of ``BAN-2-Counter + SAN + Mul + prior" (66.48) outperforms the performance of ``BAN-2-Counter + SAN" (65.64). 
Fig.~\ref{fig:exp_figure}  presents some visualization results of our second configuration and other methods on the VQA 2.0 validation set.

\textbf{Question-type classification analysis}
The proposed MILQT is a model which allows joint training between question-type classification and VQA answer classification. The effectiveness of multi-task learning helps to improve performance in both tasks. To further analyze the effectiveness of MILQT in the question-type classification, we provide in this section the question type classification on TDIUC dataset. We follow QTA~\cite{MTL_QTA} to calculate the accuracy, i.e., the overall accuracy is the number of correct predictions over the number of testing questions, across all categories. 

The results are presented in Table \ref{tab:state-of-the-art-qt}. Our MILQT uses BAN-2 \cite{Kim2018BilinearAN}, BAN-2-Counter~\cite{Kim2018BilinearAN}, and SAN~\cite{Yang2016StackedAN} in the interaction learning, element-wise multiplication in the second level of multi-modal fusion, and the predicted question type with prior information to augment the VQA loss. Compare to the state-of-the-art QTA~\cite{MTL_QTA}, our MILQT outperforms QTA for most of question types. In overall, we achieve state-of-the-art performance on question-type classification task on TDIUC dataset with $96.45\%$ accuracy. 

It is worth noting that for the ``Utility and Affordances" category, the question type classification accuracy is $0\%$ for both QTA and MILQT. It is because the imbalanced data problem in TDIUC dataset. The ``Utility and Affordances" category has only $\approx 0.03\%$ samples in the dataset. Hence this category is strongly dominated by other categories when learning the question type classifier. 
Note that, there are cases in which questions belonging to the ``Utility and Affordances" category have similar answers with questions belonging to other categories. Thus, the data becomes less bias w.r.t. answers (in comparing to question categories).
This explains why although both MILQT and QTA have $0\%$ accuracy for the ``Utility and Affordances" on the question category classification, both of them achieve some accuracy on the VQA classification (see Table \ref{tab:state-of-the-art-qt}). 

\begin{table}[!t]
\begin{center}
\small
\begin{tabular}{|l|c |c|} 
\hline
\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}\textbf{Question-type accuracy}\end{tabular}} & \multicolumn{2}{c|}{\textbf{Reference Models}}     \\ 
\cline{2-3}
    &\textbf{QTA \cite{MTL_QTA}}&
    \begin{tabular}[l]{@{}l@{}}\textbf{MILQT}\end{tabular}\\
\hline
\begin{tabular}[c]{@{}c@{}}\textbf{Scene Recognition} \end{tabular} &99.40 &\textbf{99.84} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Sport Recognition}\end{tabular} &73.08 &\textbf{85.81} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Color Attributes} \end{tabular} &86.10 &\textbf{89.60} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Other Attributes} \end{tabular} &77.76 &\textbf{85.03} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Activity Recognition}\end{tabular} &13.18 &\textbf{16.43} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Positional Recognition}\end{tabular} &89.52 &\textbf{89.55} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Sub-Object Recognition}\end{tabular} &98.96 &\textbf{99.42} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Absurd}\end{tabular} &\textbf{95.46} &95.12 \\
\begin{tabular}[c]{@{}c@{}}\textbf{Utility and Affordances}\end{tabular} &00.00 &00.00 \\
\begin{tabular}[c]{@{}c@{}}\textbf{Object Presence}\end{tabular} &\textbf{100.00} &\textbf{100.00} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Counting}\end{tabular} &99.90 &\textbf{99.99}\\
\begin{tabular}[c]{@{}c@{}}\textbf{Sentiment Understanding}\end{tabular} &60.51 &\textbf{67.82} \\
\hline
\begin{tabular}[c]{@{}c@{}}\textbf{Overall}\end{tabular} &95.66 &\textbf{96.45} \\
\hline
\end{tabular}
\end{center}
\caption{The comparative question-type classification results between MILQT and state-of-the-art QTA \cite{MTL_QTA} on the TDIUC validation set.}
\label{tab:state-of-the-art-qt}
\end{table}

\begin{table*}[!t]
\centering
\small
\begin{center}
\begin{tabular}{|l|c c c|c|} 
\hline
\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}\textbf{Score}\end{tabular}} & \multicolumn{4}{c|}{\textbf{Reference Models}}     \\ 
\cline{2-5}
    &\textbf{QTA-M \cite{MTL_QTA}}&
    \textbf{MCB-A \cite{Kushal2018Tdiuc}}&
    \textbf{RAU \cite{Kushal2018Tdiuc}}&
    \begin{tabular}[l]{@{}l@{}}\textbf{MILQT}\end{tabular}\\
\hline
\begin{tabular}[c]{@{}c@{}}\textbf{Scene Recognition} \end{tabular} &93.74 &93.06 &93.96 &\textbf{94.74} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Sport Recognition}\end{tabular} &94.80 &92.77 &93.47 &\textbf{96.47} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Color Attributes} \end{tabular} &57.62 &68.54 &66.86 &\textbf{75.23} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Other Attributes} \end{tabular} &52.05 &56.72 &56.49 &\textbf{61.93} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Activity Recognition}\end{tabular} &53.13 &52.35 &51.60 &\textbf{65.03} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Positional Recognition}\end{tabular} &33.90 &35.40 &35.26 &\textbf{42.31} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Sub-Object Recognition}\end{tabular} &86.89 &85.54 &86.11 &\textbf{89.63} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Absurd}\end{tabular} &\textbf{98.57} &84.82 &96.08 &88.95 \\
\begin{tabular}[c]{@{}c@{}}\textbf{Utility and Affordances}\end{tabular} &24.07 &35.09 &31.58 &\textbf{38.60} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Object Presence}\end{tabular} &94.57 &93.64 &94.38
 &\textbf{96.21} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Counting}\end{tabular} &53.59 &51.01 &48.43 &\textbf{62.41} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Sentiment Understanding}\end{tabular} &60.06 &\textbf{66.25} &60.09 &64.98 \\
\hline
\begin{tabular}[c]{@{}c@{}}\textbf{Arithmetic MPT}\end{tabular} &66.92 &67.90 &67.81 &\textbf{73.04} \\
\begin{tabular}[c]{@{}c@{}}\textbf{Harmonic MPT}\end{tabular} &55.77 &60.47 &59.00 &\textbf{66.86} \\
\hline
\end{tabular}
\end{center}
\caption{The comparative results between the proposed model and other models on the validation set of TDIUC. 
}
\label{tab:TDIUC}
\end{table*}
\subsection{Comparison to the state of the art}
\textbf{Experiments on VQA 2.0 test-dev and test-standard.}
We evaluate MILQT on the test-dev and test-standard of VQA 2.0 dataset \cite{vqav22016}. 
To train the model, similar to previous works~\cite{Yang2016StackedAN,tip-trick,Jiang2018PythiaVT,Kim2018BilinearAN}, we use both training set and validation set of VQA 2.0. We also use 
the Visual Genome~\cite{visualgenome} as additional training data. 

MILQT consists of three joint modality mechanisms, i.e., {BAN-2}, {BAN-2-Counter}, and {SAN} accompanied with the EWM for the multi-modal fusion, and the predicted question type together with the prior information to augment the VQA loss.
Table~\ref{tab:VQA} presents the results of different methods on test-dev and test-std of VQA 2.0.
The results show that our MILQT yields the good performance with the most competitive approaches. 

\textbf{Experiments on TDIUC.} 
In order to prove the stability of MILQT, we evaluate MILQT on TDIUC dataset \cite{Kushal2018Tdiuc}. 
The results in Table \ref{tab:TDIUC}  show  that the proposed model  establishes the state-of-the-art results on both evaluation metrics Arithmetic MPT and Harmonic MPT \cite{Kushal2018Tdiuc}. Specifically, our model significantly outperforms the recent QTA~\cite{MTL_QTA}, i.e., on the overall, we improve over QTA $6.1\%$ and $11.1\%$ with Arithemic MPT and Harmonic MPT metrics, respectively. It is worth noting that the results of QTA~\cite{MTL_QTA} in Table \ref{tab:TDIUC}, which are cited from \cite{MTL_QTA}, are achieved when  \cite{MTL_QTA} used the one-hot \textit{predicted question type} of testing question to weight visual features. When using \textit{the groundtruth question type} to weight visual features, \cite{MTL_QTA} reported  $69.11\%$ and $60.08\%$ for Arithemic MPT and Harmonic MPT metrics, respectively. Our model also outperforms these performances a large margin, i.e., the improvements are $3.9\%$ and $6.8\%$ for Arithemic MPT and Harmonic MPT metrics, respectively. 
 
We also note that for the question type ``Absurd", we get lower performance than QTA \cite{MTL_QTA}. For this question type, the question is irrelevant  with the image content. Consequently, this question type does not help to learn a joint meaningful embedding between the input question and image. This explains for our lower performance on this question type.

\section{Conclusion}
We present a multiple interaction learning with question-type prior knowledge for constraining answer search space--- MILQT that takes into account the question-type information to improve the VQA performance at different stages. The system also allows to utilize and learn different attentions under a unified model in an interacting manner. The extensive experimental results show that all proposed components improve the VQA performance. We yields the best performance with the most competitive approaches on VQA 2.0 and TDIUC dataset.

\bibliographystyle{splncs04}
\bibliography{egbib}
\end{document}


Peer reviews:

content: confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature
rating: 7: Good paper, accept
review: 

#### 1. [Summary] In 2-3 sentences, describe the key ideas, experiments, and their significance.
The paper proposes a method which constrains search space by using question type information as prior information and utilizes different attentions to obtain better results.

#### 2. [Strengths] What are the strengths of the paper? Clearly explain why these aspects of the paper are valuable.
- Search space constraints according to question types
- Using multiple attention mechanisms
- Performance and better attention maps

#### 3. [Weaknesses] What are the weaknesses of the paper? Clearly explain why these aspects of the paper are weak.
- Question types are prior knowledge yet not visual prior knowledge.

#### 4. [Overall rating] Paper rating
7

#### 5. [Justification of rating] Please explain how the strengths and weaknesses aforementioned were weighed in for the rating. 


#### 6. [Detailed comments] Additional comments regarding the paper (e.g. typos or other possible improvements you would like to see for the camera-ready version of the paper, if any.)

- Fig.3: Some of the notations are not visible. In addition, you can show modules with dashed areas with different colors.
- Table 4: Why did you make your results as bold?
- The effectiveness of multi-hypothesis interaction learning proposed
in Section 3.3: The explanation in the subsection makes confusion because the order of showing the results. It can be better to have a paragraph for each result (table).

- Limitations and failing cases.
- Time and memory usage
- Will you share the code and models?
Typos:
- L80: constraint
- L.436: not fitting in the line
- L.590:modality
title: Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering

content: confidence: 3: The reviewer is fairly confident that the evaluation is correct
rating: 7: Good paper, accept
review: [Summary] In 2-3 sentences, describe the key ideas, experiments, and their significance.

The authors propose a VQA method that jointly optimizes question answering and answer type classification using an ensemble of existing attention-based VQA methods. Prior information about the answer types is integrated into a weighted loss. Modalities are fused by first merging visual and linguistic features, then merging in question type features.

[Strengths] What are the strengths of the paper? Clearly explain why these aspects of the paper are valuable.

The authors use prior information on answer types while allowing the model to account for outlier questions. The authors perform extensive experiments to show all aspects of their works.

[Weaknesses] What are the weaknesses of the paper? Clearly explain why these aspects of the paper are weak.

It is not clear how significant the individual performance increases of each contribution are. Crucial design choices for the VQA loss are not motivated (e.g. where to integrate awareness matrix). Presentation needs work (typos, grammar, typesetting).

[Overall rating] Paper rating: Accept (tentative rating, subject to revision until deadline)

[Detailed comments] Additional comments regarding the paper (e.g. typos or other possible improvements you would like to see for the camera-ready version of the paper, if any.)

- Please review the paper for typos, incorrect grammar and typesetting. Specifically lines 40 (grammar), 114, 178, 260 "constraints", 264, 287 (capitalizing VQA), 389 "standardly", 436 end of line, 590 "modaality"
- Table 4: your method is not the highest (LXMERT is), so please do not use bold numbers. See also the claim on line 594.
- Algorithm 1 should be expressed in math, like the other equations.
- Source for claim on line 153
title: An ensemble of VQA methods with prior knowledge on question types