# Multiple Context Learning Networks for Visual Question Answering

Authors: Pufen Zhang; Hong Lan; Muhammad Asim Khan
Journal: Scientific Programming
 (2022)

---

## Abstract

A novel Multiple Context Learning Network (MCLN) is proposed to model multiple contexts for visual question answering (VQA), aiming to learn comprehensive contexts. Three kinds of contexts are discussed and the corresponding three context learning modules are proposed based on a uniform context learning strategy. Specifically, the proposed context learning modules are visual context learning module (VCL), textual context learning module (TCL), and visual-textual context learning module (VTCL). The VCL and TCL, respectively, learn the context of objects in an image and the context of words in a question, allowing object and word features to own intra-modal context information. The VTCL is performed on the concatenated visual-textual features that endows the output features with synergic visual-textual context information. These modules work together to form a multiple context learning layer (MCL) and MCL can be stacked in depth for deep context learning. Furthermore, a contextualized text encoder based on the pretrained BERT is introduced and fine-tuned, which enhances the textual context learning at the feature extraction stage of text. The approach is evaluated by using two benchmark datasets: VQA v2.0 dataset and GQA dataset. The MCLN achieves 71.05% and 71.48% overall accuracy on the test-dev and test-std sets of VQA v2.0, respectively. And an accuracy of 57.0% is gained by the MCLN on the test-standard split of GQA dataset. The MCLN outperforms the previous state-of-the-art models and the extensive ablation studies examine the effectiveness of the proposed method.


---

## Body

## 1. Introduction

An artificial intelligence agent must be able to understand not only the semantics of text but also the content of images. Most multimodal tasks involving image and text modalities require this ability; these tasks include grounding referring expressions [1], image captioning [2], image-text matching [3], and visual question answering (VQA) [4, 5]. Compared to other multimodal tasks, VQA is more complex that requires associating visual content in the image with the semantic meaning in the question, together with visual reasoning to make the correct answer. In addition, VQA has a wide range of applications in practice, such as assisting the blind and early childhood education [6]. Considering the challenges and significance of VQA, visual question answering has attracted much attention in computer vision and natural language processing communities.In the early stages, the VQA methods adopted the convolutional neural networks (CNN) [7] and recurrent neural networks (RNN) [8] to extract the global image and text features [9, 10], respectively. However, the extracted global features are not fine-grained. With the development of deep learning, some fine-grained VQA approaches proposed employing attention mechanisms to locate the question keywords and the image objects related to the answers [11–13]. For example, question-guided visual attention on image regions was first proposed in [11]. Following that, a large variety of attention-based variations including co-attention and compositional attention have been proposed for VQA [14, 15]. Although aforementioned attention-based approaches have dramatically ameliorated the performance of VQA, the context information on image and question are not considered by these methods.As context reflects the high-order interactive information between entities (image objects or question words) and helps to distinguish the targets from other entities, several attention-based VQA models have begun to emphasize multimodal context learning. For example, to capture the visual context information, two shallow context learning models, ReGAT [16] and v-AGCN [17], are proposed. They both structured the object relation graph on the image and used the graph attention network [18] to generate the visual context-aware image representations. Furthermore, LGCN learned the deep visual context representation through iterative message passing the conditioned textual input on the fully connected image object graph [19]. However, these models only consider the context for image modality. More recently, some deep models have attempted to learn the extra textual context. DFAF [20] used the intra-modality attention flow to capture the intra-modal contexts within each modality. Two deep co-attention models, namely, MCAN [21] and MEDAN [22], have been introduced to VQA task to capture the visual and textual contexts based on the encoder-decoder framework.Despite the effectiveness of these methods, we still find that more comprehensive context learning is rarely explored. On one hand, a special context, named visual-textual context and expressed as the synergic context-dependence under two modal information, is almost ignored by the above shallow and deep context learning models. On the other hand, how to explore an accordant context learning way integrating multicontext learning into a unified framework and modelling deep multiple contexts, there is still an open question. Besides, in terms of text representations, most existing methods use RNN architecture to extract the text features. However, due to long-term dependence between words, the RNN cannot sufficiently capture the textual context at the textual features extraction stage. To tackle these issues, we propose a novel Multiple Context Learning Network (MCLN) to learn comprehensive contexts for VQA. In this paper, three contexts containing an ignored visual-textual context and two intra-modal contexts (i.e., the visual context in the image modality and textual context in the text modality) are explored. The key-query attention mechanism [23] is able to model the context-dependence of all entities (objects or words) in an entity set. Inspired by this, we adopt key-query attention to uniformly model the multiple context-dependence and propose the corresponding context learning modules. Figure 1 shows the proposed uniform multiple contexts learning strategy. The proposed visual context learning module (VCL), textual context learning module (TCL), and visual-textual context learning module (VTCL) are constructed based on the key-query attention. Firstly, two intra-modal contexts are learned through the VCL and TCL, respectively. Then, the image and question features with intra-modal context information are concatenated to form the visual-textual features, which will be fed into VTCL to extract the visual-textual context information. After that, by modular composition of the three modules, the multiple contexts learning layer (MCL) is structured. Furthermore, we also tend to capture the deep context information. By cascading MCL in depth, the deeper level and more complex context learning can be reached handily. Additionally, the pretrained Bidirectional Encoder Representations from Transformers (BERT) [24] models the textual context by stacking the bidirectional encoder. To further enhance the textual context learning ability of MCLN at the textual features extraction stage and provide better contextualized representation for textual features, a BERT-based text encoder is introduced and fine-tuned to learn contextualized text representations, resulting in better VQA performance. Finally, the Multiple Context Learning Network that consists of cascaded MCL layers is proposed. The preprint is shown in [25].Figure 1
A diagram of the proposed context learning method, which simultaneously learns multiple contexts by using a uniform context learning framework. VCL and TCL model the intra-modal contexts in the MCL layer, while VTCL models the visual-textual context. For image features (V) and question feature (T), the handled features with context information are (V)′ and (T)′, respectively.The main contributions of this study can be summarized as follows:(1)
We explore multiple contexts including visual context, textual context, and visual-textual context ignored by previous methods for VQA task. Particularly, we propose corresponding context learning module based on a uniform context learning strategy and compose the context learning modules to structure a multiple context learning layer (MCL), which can model and learn multiple contexts.(2)
We attempt to consider the deep contexts and the proposed MCLN is a deep model stacking several MCL layers, meaning that it can capture deep-level context information.(3)
To further consider more comprehensive context learning, we adopt the pretrained BERT as a contextualized text encoder and fine-tune its learning rate during the training of MCLN, enhancing the textual context at the textual features extraction stage.(4)
Evaluation results on two benchmark VQA datasets demonstrate that MCLN achieves state-of-the-art performance. The proposed MCLN achieves 71.05% and 71.48% overall accuracies for test-dev and test-std sets on the VQA v2.0 dataset [4], respectively. On the GQA [5], the overall performance reaches 56.8% for test-dev set and 57.0% for test set. In addition, adequate ablation studies are conducted to quantitatively and qualitatively prove the significance of different modules in the proposed model, verifying the effectiveness of the proposed MCLN architecture.The rest of the paper is organized as follows. In Section2, the proposed Multiple Context Learning Networks (MCLN) are presented. In Section 3, the experimental results and analysis are provided. Finally, the paper is concluded in Section 4.


## 2. Multiple Context Learning Networks for VQA

As common practice, identifying a correct answer related to the given image-question pair from a set of candidate answers is a typical formulation of VQA:(1)a^=argmaxa∈Apθa|I,Qwhere the model predicts the correct answer a^ from the candidate answers A for a given image I and question Q pair, and θ is the learned parameters of model. Without loss of generality, the proposed method also follows this convention.Figure2 illustrates the overall pipeline of the proposed MCLN, which consists of a series of subnetworks: (a) image and question representation, which includes image encoder and text encoder. The image encoder detects objects in an image and extracts the object features, while text encoder embeds the question words and encodes the word features; (b) multiple contexts learning that performs multicontext learning and it is composed of multiple learning layers by the cascaded way; and (c) multimodal fusion and answer prediction that fuses image representation and question representation to predict a correct answer.Figure 2
Overall flowchart of the MCLN that consists of three subnetworks. (a) Image and question representation; (b) multiple context learning; (c) multimodal fusion and answer prediction.


### 2.1. Image and Question Representation

#### 2.1.1. Image Encoder

Inspired by humans using bottom-up attention to process visual information [13], we extract the object-level features for a given image. Our image encoder encapsulates three main steps: (i) a forward pass of a Faster R-CNN [26] object detector pretrained on the Visual Genome dataset [27] for extracting the K most salient objects O=oii=1K within a given image I, thereby achieving the bottom-up attention; (ii) a ResNet-101 network [28] that extracts the feature map of detected objects, followed by a mean pooling layer that generates the 2048-dimensional vector representation ri for the original object oi; and (iii) a linear transformation layer to project the object feature ri into a 768-dimensional vector.(2)ri=Faster−RCNNI,i∈1,K,vi=Wvri+bv.Wv∈R2048×768 and bv∈R768 are the parameter of the linear layer. Finally, image I is represented as visual object features set V=vi∈R768i=1K.


#### 2.1.2. Text Encoder

RNN-based text encoder first tokenizes questionQ into words and uses a maximum of L words to trim the question. Then, the l-th word in the question is further embedded into a 300-dimensional vector representation el∈R300 by using the GloVe word embeddings [29]. The questions shorter than L words are padded at the end with zero vectors. Thus, Q is initialized as a word embeddings sequence E=el∈R300l=1L. Here, a one-layer and 768-dimensional long short-term memory network (LSTM) [10] is utilized, which scans the word embeddings sequence E from e1 to eL and picks up current el into its unit to obtain the word feature tl.(3)t1,…,tl,…tL=LSTMe1,…,el,…eL.To enhance the textual context at the textual features extraction stage, we also introduce a contextual text encoder based on the pretrained BERT model [23] and fine-tune it to extract the contextualized word features. The BERT-based text encoder is represented as(4)t1,…,tl,…tL=BERTQ.Finally, the questionQ is represented as word features set T=tl∈R768l=1L.


### 2.2. Multiple Context Learning Layers

The key-query attention mechanism [23] can model context-dependence of all entities (objects or words) across whole entities set. Inspired by this, we adopt key-query attention to uniformly model the multiple context-dependence and propose the corresponding context learning modules. Using visual context learning as an example, Figure 3 shows the design of the visual context learning module (VCL), and the implementation will be discussed in detail below.Figure 3
The detailed design of proposed visual context learning module. (a) Key-query attention, (b) multihead attention, (c) visual context learning module.
(a)(b)(c)To capture the visual context of all objects, we adopted the key-value attention to model the context-dependence of objects. Obtained image object features setV can be denoted as a features matrix V∈RK×768 and its i-th row represents the feature of i-th object. The matrix V would be converted to key matrix KV, query matrix QV, and value matrix VV by three linear transformations.(5)KV=VWK,QV=VWQ,VV=VWV,where WK, WQ, and WV are the linear transformation parameters that calculate the key, query, and value matrices. Then, calculate the dot products between query and key matrices to get the attention weights matrix AV.(6)AV=softmaxQVKVTd,where d is a normalization constant. In this way of dot product, the contextual path dependency distance between all objects is set to 1. So AV∈RK×K reflects the context-dependence of all objects and stores the attention weights between all objects. The row attention weights Ai:V represent the context-dependence of i-th object in the image. Therefore, the visual context feature for i-th object can be obtained by calculating a weighted combination of the Ai:V and all object features, whereas the form of matrices for the visual context feature of all objects can be stated as follows:(7)ATTV=softmaxQVKVTdVV.In addition, the multihead attention based on key-value attention is performed onV.(8)MultiHeadV=head1⋯headHWO,headh=ATThV=softmaxVWhVVWhKTdhVWhV.headh is h-th head key-query attention. WhK∈R768×dh, WhQ∈R768×dh, and WhV∈R768×dh are the parameter matrices of headh. WO is the projection matrix for all heads and || represents concatenation of all heads. dh is the dimensionality of the output features from each head and usually making dh = 768/H. The multihead attention allows the model to jointly attend to context information from different representation subspaces, improving the representation capacity of features.After acquiring the visual context features of all objects using multihead attention, residual connection followed by layer normalization is applied to integrate the visual context into the object features.(9)V^=LNV+MultiHeadV,where LN (·) represents layer normalization. Aiming to further adjust the object representations, the position wise feed-forward network (FFN) transforms object features V^ with two fully connected layers. And it can be described as(10)FFNV^=ReLuV^W1+b1W2+b2,Where the W1,b1,W2, and b2 are the parameters of FFN, and its hidden and output dimensions are 768 × 4 and 768, respectively. ReLu is the rectified linear unit. In addition, the residual connection and layer normalization are also applied after FFN to facilitate optimization.By performing the above procedure, the visual contexts that represent high-order interaction and context-dependence among all objects can be captured. Such procedure is simplified by the visual context learning module (VCL) as(11)V′=VCLV.The textual context learning module can work in the same manner on the word featuresT, but with a different set of parameters to be learned; thus details are omitted and it is also simplified as(12)T′=TCLT.Since the contextual path dependency distance between all entities is 1, the visual-textual context can be easily modelled by feeding the connected visual-textual feature matrixV′‖T′∈RK+M×768 into the visual-textual context learning module (VTCL) with the same context learning mechanism.(13)Vout,Tout=VTCLV′T′.Then, we combine three context learning modules to form a multiple context learning layer (MCL). Finally, such MCL layer is stacked in depth to learn the deeper and more high-level context information, and the output of the last layer is the input of the next layer, expressed asVN,TN=MCLNVN−1,TN−1.


### 2.3. Multimodal Fusion and Answer Prediction

AfterN MCL layers, the output image object features VN and question word features TN not only contain the intra-modal contexts but also capture the inter-modal context dependencies. To distinguish important entities from context information and fuse multimodal features, an attention model with an FFN (its hidden and output dimensions are 768 and 1) is designed for VN and TN to obtain attended image feature v and question feature q. Taking image object features VN as an example, the attended image features v are calculated as follows:(14)v=∑i=1KaiviN,whereαi=softmaxFFNviN.αi is the learned object attention weight for i-th object. The learned word attention weight βl and the attended question feature q can be obtained using an independent attention model by analogy.After obtaining image featurev and question feature q, two linear transformations are implemented on v and q, and using the addition to get joint representation z. Finally, z is fed into a fully connected layer followed by a sigmoid function to generate the answer vector p∈RA.(15)z=LNWvTv+WqTq,p=sigmaidWzz+bz.WvT∈R1024×768, WqT∈R1024×768, and Wz∈RA×1024 are three linear projection matrixes, bz is the bias parameter, sigmoid (·) is used for classification, and A is the number of candidate answers.


## 2.1. Image and Question Representation

### 2.1.1. Image Encoder

Inspired by humans using bottom-up attention to process visual information [13], we extract the object-level features for a given image. Our image encoder encapsulates three main steps: (i) a forward pass of a Faster R-CNN [26] object detector pretrained on the Visual Genome dataset [27] for extracting the K most salient objects O=oii=1K within a given image I, thereby achieving the bottom-up attention; (ii) a ResNet-101 network [28] that extracts the feature map of detected objects, followed by a mean pooling layer that generates the 2048-dimensional vector representation ri for the original object oi; and (iii) a linear transformation layer to project the object feature ri into a 768-dimensional vector.(2)ri=Faster−RCNNI,i∈1,K,vi=Wvri+bv.Wv∈R2048×768 and bv∈R768 are the parameter of the linear layer. Finally, image I is represented as visual object features set V=vi∈R768i=1K.


### 2.1.2. Text Encoder

RNN-based text encoder first tokenizes questionQ into words and uses a maximum of L words to trim the question. Then, the l-th word in the question is further embedded into a 300-dimensional vector representation el∈R300 by using the GloVe word embeddings [29]. The questions shorter than L words are padded at the end with zero vectors. Thus, Q is initialized as a word embeddings sequence E=el∈R300l=1L. Here, a one-layer and 768-dimensional long short-term memory network (LSTM) [10] is utilized, which scans the word embeddings sequence E from e1 to eL and picks up current el into its unit to obtain the word feature tl.(3)t1,…,tl,…tL=LSTMe1,…,el,…eL.To enhance the textual context at the textual features extraction stage, we also introduce a contextual text encoder based on the pretrained BERT model [23] and fine-tune it to extract the contextualized word features. The BERT-based text encoder is represented as(4)t1,…,tl,…tL=BERTQ.Finally, the questionQ is represented as word features set T=tl∈R768l=1L.


## 2.1.1. Image Encoder

Inspired by humans using bottom-up attention to process visual information [13], we extract the object-level features for a given image. Our image encoder encapsulates three main steps: (i) a forward pass of a Faster R-CNN [26] object detector pretrained on the Visual Genome dataset [27] for extracting the K most salient objects O=oii=1K within a given image I, thereby achieving the bottom-up attention; (ii) a ResNet-101 network [28] that extracts the feature map of detected objects, followed by a mean pooling layer that generates the 2048-dimensional vector representation ri for the original object oi; and (iii) a linear transformation layer to project the object feature ri into a 768-dimensional vector.(2)ri=Faster−RCNNI,i∈1,K,vi=Wvri+bv.Wv∈R2048×768 and bv∈R768 are the parameter of the linear layer. Finally, image I is represented as visual object features set V=vi∈R768i=1K.


## 2.1.2. Text Encoder

RNN-based text encoder first tokenizes questionQ into words and uses a maximum of L words to trim the question. Then, the l-th word in the question is further embedded into a 300-dimensional vector representation el∈R300 by using the GloVe word embeddings [29]. The questions shorter than L words are padded at the end with zero vectors. Thus, Q is initialized as a word embeddings sequence E=el∈R300l=1L. Here, a one-layer and 768-dimensional long short-term memory network (LSTM) [10] is utilized, which scans the word embeddings sequence E from e1 to eL and picks up current el into its unit to obtain the word feature tl.(3)t1,…,tl,…tL=LSTMe1,…,el,…eL.To enhance the textual context at the textual features extraction stage, we also introduce a contextual text encoder based on the pretrained BERT model [23] and fine-tune it to extract the contextualized word features. The BERT-based text encoder is represented as(4)t1,…,tl,…tL=BERTQ.Finally, the questionQ is represented as word features set T=tl∈R768l=1L.


## 2.2. Multiple Context Learning Layers

The key-query attention mechanism [23] can model context-dependence of all entities (objects or words) across whole entities set. Inspired by this, we adopt key-query attention to uniformly model the multiple context-dependence and propose the corresponding context learning modules. Using visual context learning as an example, Figure 3 shows the design of the visual context learning module (VCL), and the implementation will be discussed in detail below.Figure 3
The detailed design of proposed visual context learning module. (a) Key-query attention, (b) multihead attention, (c) visual context learning module.
(a)(b)(c)To capture the visual context of all objects, we adopted the key-value attention to model the context-dependence of objects. Obtained image object features setV can be denoted as a features matrix V∈RK×768 and its i-th row represents the feature of i-th object. The matrix V would be converted to key matrix KV, query matrix QV, and value matrix VV by three linear transformations.(5)KV=VWK,QV=VWQ,VV=VWV,where WK, WQ, and WV are the linear transformation parameters that calculate the key, query, and value matrices. Then, calculate the dot products between query and key matrices to get the attention weights matrix AV.(6)AV=softmaxQVKVTd,where d is a normalization constant. In this way of dot product, the contextual path dependency distance between all objects is set to 1. So AV∈RK×K reflects the context-dependence of all objects and stores the attention weights between all objects. The row attention weights Ai:V represent the context-dependence of i-th object in the image. Therefore, the visual context feature for i-th object can be obtained by calculating a weighted combination of the Ai:V and all object features, whereas the form of matrices for the visual context feature of all objects can be stated as follows:(7)ATTV=softmaxQVKVTdVV.In addition, the multihead attention based on key-value attention is performed onV.(8)MultiHeadV=head1⋯headHWO,headh=ATThV=softmaxVWhVVWhKTdhVWhV.headh is h-th head key-query attention. WhK∈R768×dh, WhQ∈R768×dh, and WhV∈R768×dh are the parameter matrices of headh. WO is the projection matrix for all heads and || represents concatenation of all heads. dh is the dimensionality of the output features from each head and usually making dh = 768/H. The multihead attention allows the model to jointly attend to context information from different representation subspaces, improving the representation capacity of features.After acquiring the visual context features of all objects using multihead attention, residual connection followed by layer normalization is applied to integrate the visual context into the object features.(9)V^=LNV+MultiHeadV,where LN (·) represents layer normalization. Aiming to further adjust the object representations, the position wise feed-forward network (FFN) transforms object features V^ with two fully connected layers. And it can be described as(10)FFNV^=ReLuV^W1+b1W2+b2,Where the W1,b1,W2, and b2 are the parameters of FFN, and its hidden and output dimensions are 768 × 4 and 768, respectively. ReLu is the rectified linear unit. In addition, the residual connection and layer normalization are also applied after FFN to facilitate optimization.By performing the above procedure, the visual contexts that represent high-order interaction and context-dependence among all objects can be captured. Such procedure is simplified by the visual context learning module (VCL) as(11)V′=VCLV.The textual context learning module can work in the same manner on the word featuresT, but with a different set of parameters to be learned; thus details are omitted and it is also simplified as(12)T′=TCLT.Since the contextual path dependency distance between all entities is 1, the visual-textual context can be easily modelled by feeding the connected visual-textual feature matrixV′‖T′∈RK+M×768 into the visual-textual context learning module (VTCL) with the same context learning mechanism.(13)Vout,Tout=VTCLV′T′.Then, we combine three context learning modules to form a multiple context learning layer (MCL). Finally, such MCL layer is stacked in depth to learn the deeper and more high-level context information, and the output of the last layer is the input of the next layer, expressed asVN,TN=MCLNVN−1,TN−1.


## 2.3. Multimodal Fusion and Answer Prediction

AfterN MCL layers, the output image object features VN and question word features TN not only contain the intra-modal contexts but also capture the inter-modal context dependencies. To distinguish important entities from context information and fuse multimodal features, an attention model with an FFN (its hidden and output dimensions are 768 and 1) is designed for VN and TN to obtain attended image feature v and question feature q. Taking image object features VN as an example, the attended image features v are calculated as follows:(14)v=∑i=1KaiviN,whereαi=softmaxFFNviN.αi is the learned object attention weight for i-th object. The learned word attention weight βl and the attended question feature q can be obtained using an independent attention model by analogy.After obtaining image featurev and question feature q, two linear transformations are implemented on v and q, and using the addition to get joint representation z. Finally, z is fed into a fully connected layer followed by a sigmoid function to generate the answer vector p∈RA.(15)z=LNWvTv+WqTq,p=sigmaidWzz+bz.WvT∈R1024×768, WqT∈R1024×768, and Wz∈RA×1024 are three linear projection matrixes, bz is the bias parameter, sigmoid (·) is used for classification, and A is the number of candidate answers.


## 3. Experiment

### 3.1. Datasets and Evaluation Metric

VQA v2.0 dataset is the most commonly used large-scale VQA dataset [4], which contains 1.1 M questions asked by humans and 10 answers are collected for each image-question pair from human annotators. The answer with the highest number of occurrences will be regarded as the correct answer. The dataset is divided into three parts: a training set containing 80 K images and 444 K questions, a validation set containing 40 K images and 214 K questions, and a test set containing 80 K images and 448 K questions. Additionally, based on the answer category, all questions are divided into three types: yes/no, number, and other; we use the following accuracy as the evaluation metric for answering quality.(16)accuracyanswer=minprovidedannotators3,1.To demonstrate the generalization of approach, we further evaluate the model on the GQA dataset [4]. GQA dataset is the latest large-scale VQA dataset containing more than 110 K images and 22 M questions [5]. The dataset is divided randomly into proportions of 87%, 12%, and 1% for train, validation, and test sets, respectively.


### 3.2. Experimental Setup

#### 3.2.1. Universal Setup

The proposed MCLN is implemented by PyTorch and all the experiments are conducted on a workstation; the experimental environment is shown in Table1. The universal hyperparameters of MCLN model that are used in the experiments are listed as follows. For the pretrained Faster R-CNN, we follow the strategies in [13] to set its parameters and obtain a dynamic number of objects K ∈ [10, 100]. For textual features, we use either a LSTM with a 300-dimensional GloVe word embedding size or a pretrained BERT model with the 768-dimensional embedding size. In all context learning modules, we set the number of heads H as 12, so the latent dimensionality for each head is dh = d/12 = 64. Adam Optimizer [31] is used to optimize the model with 64 batch sizes.Table 1
Experimental environment.
ConfigurationDetailsOperating systemUbuntu 18.04RAM64 GGraphic processing unitNVIDIA GeForce GTX 2080 TiProgramming languagePython 3.6Deep learning frameworkPyTorch 1.0.1Architecture platformCUDA10.0 and CUDNN7.4


#### 3.2.2. Setup for VQA v2.0

The answers with an occurrence rate less than 8 times in the training and validation sets are discarded, which resulted inA=3129 candidate answers. The maximum length of tokenized words is L = 14. Moreover, the number of MCL layers is N ∈ {1, 2, 3, 4}. For the choice of learning rate lr, the warm-up strategy is employed. Specifically, the initial learning rate is 2.5 × 10−5 and grows by 2.5 × 10−5 at each epoch till it reaches 1 × 10−4 at epoch 4. After 10 epochs, the learning rate is decreased by 1/5 for every epoch up to 12 epochs. Since there exist multiple correct answers for a question in the VQA v2.0 dataset, the binary cross-entropy loss (BCE) is applied to optimize MCLN:(17)BCE loss=−∑i=1Ayilogpi+1−yilog1−pi,where yi is the given score by the datasets and pi is the predicted score by the MCLN model and corresponds to the i-th element in the answer vector p.


#### 3.2.3. Setup for GQA

The GQA dataset providedA=1878 candidate answers. The maximum length of tokenized words is L = 29. We also use the same learning rate strategy on the GQA dataset, while after 8 epochs the learning rate is decreased by 1/5 for each up to 10 epochs. For the loss function, we choose the cross-entropy loss (CE) to optimize our model on GQA dataset.(18)CE loss=−∑i=1Ayilogpi.


### 3.3. Ablation Studies

On the validation sets of VQA v2.0 and GQA, the proposed MCLN will execute extensive ablation studies from three different aspects: (1) the effectiveness of the three context learning modules in the proposed MCL network architecture; (2) the impact of different layers on the performance of MCLN; (3) the effect of enhancing the textual context.


#### 3.3.1. Effect of the Context Learning Module

As mentioned, three modules are designed to perform corresponding context learning. To demonstrate the effectiveness of the proposed context learning modules, we employ the MCLN model with LSTM to quantitatively ablate three modules. Table2 shows the results of various ablated versions of the model on the VQA v2.0 and GQA validation sets. It has been discovered that induction and elimination of any module have an impact on the performance. The optimum results are only achieved by integrating all three components. Results indicate that every context learning module contributes significantly to VQA performance and demonstrating the effectiveness of context learning modules. The result on VTCL module is noteworthy. For the models on VQA v2.0 dataset, model 4 only with the VTCL module obtains 62.07% overall accuracy, which outperforms model 1 more than 7.47% and higher performance is gained compared to model 2 and model 3. Furthermore, when the VTCL module is disabled in model 7 compared to model 8, a rapid decrease of 9.92% is observed. It indicates that the visual-textual context ignored by prior approaches is crucial to VQA.Table 2
The results of ablating the context learning modules on VQA v2.0 and GQA validation sets.
ModelModuleGQAVQA v2.0AllAllY/NNumOther1Without all53.0854.6069.7936.0247.502Only VCL53.4555.1369.8236.0947.993Only TCL53.5055.5369.8236.4449.724Only VTCL58.6362.0779.7942.6753.735TCL + VTCL63.8865.1782.8844.6857.156VCL + VTCL59.0462.7279.3343.2955.237VCL + TCL53.7255.7671.0136.3749.298Full modules64.4865.6883.4045.5757.53


#### 3.3.2. Effect of MCL Layers

Next, we stack the full-module MCL layer in depth to evaluate the effect of MCL layers. Figure4 shows the performance of the MCLN models with the different numbers of MCL layers N ∈ {1, 2, 3, 4} on the VQA v2.0 validation set. Regarding the performance, we observed two phenomena on the overall accuracy: (1) as increasing N, the performances of MCLN models steadily improve and finally saturate at a certain number N = 3; (2) the one-layer model does not perform as well as the deeper models, but there is a quite improvement over N = 1 while a subtle decrease over N = 3 when N = 4. In addition, the performance of the MCLN with the different number of MCL layers on the GQA validation set is shown in Figure 5. Similar experimental phenomena are observed and the best performance appears at N = 2. From the experimental results, it can be seen that deeper MCL layers usually obtain higher overall accuracy. We attribute it to deep context learning, which captures more complex context information and achieves a better contextual understanding of the image and question contents. However, comparing the shallow model and deep model, the number of layers increases and the parameters of the model rise as well. As a result, the model with deeper MCL layers is difficult to be optimized and suffers from a larger risk of overfitting the training set due to the higher number of parameters. Therefore, the overall performance decreases at N = 4 and N = 3 on the VQA v2.0 and GQA, respectively.Figure 4
The overall and per-type accuracies of the MCLN with different MCL layers on the VQA v2.0 validation set. (a) Yes/no, (b) number, (c) others, and (d) overall.
(a)(b)(c)(d)Figure 5
The overall accuracies of the MCLN with different MCL layers on the GQA validation set.


#### 3.3.3. Effect of Enhancing the Textual Context

Since the pretrained BERT [23] was trained on large text corpus and models the textual context by stacking the bidirectional encoder, to enhance the textual context at the textual features extraction stage, a contextualized text encoder based on BERT is introduced and fine-tuned. The results of fine-tuning the weight of pretrained BERT with different learning rates are shown in Tables 3 and 4. On the VQA v2.0, the best performance is achieved at lr × 0.1 under one-layer MCL. With this learning rate, by increasing the MCL layers, the performance grows and also saturates at three MCL layers. On the GQA dataset, due to the differences in datasets and experimental settings, the best performance is achieved by fine-tuning the learning rate of BERT to lr × 0.01 and setting the MCL layer to 2. Compared with the MCLN-LSTM using three-layer MCL, 1% overall accuracy is improved by the MCLN-BERT using three-layer MCL and lr × 0.1 learning rate on the VQA v2.0 validation set. On the GQA validation set, when the MCL layer is set to two and fine-tuning the learning rate of BERT to lr × 0.01, the gain is 0.11% compared with MCLN-BERT and MCLN-LSTM. From these experimental and compared results, it can be found that BERT is compatible with MCLN and conducive to boosting the performance by fine-tuning it to enhance textual context at the textual features extraction stage.Table 3
The results of fine-tuning BERT on VQA v2.0 validation set.
Modellr×AllY/NNumOtherN = 1, BERT0.00165.2182.6145.7457.13N = 1, BERT0.0166.1284.0845.9757.80N = 1, BERT0.166.6185.0946.1457.98N = 2, BERT0.167.5285.1849.0958.97N = 3, BERT0.167.8685.3549.7559.27N = 4, BERT0.167.8085.7349.9458.94N = 3, LSTM-66.8684.6248.8558.34Table 4
The results of fine-tuning BERT on GQA validation set.
Modellr×AllN = 1, BERT0.00164.25N = 1, BERT0.0164.63N = 1, BERT0.164.51N = 2, BERT0.0165.22N = 3, BERT0.0165.11N = 2, LSTM-65.01


### 3.4. Visualization Analysis

Assuming that, after processing the object and word features through multiple MCL layers, the keywords in a question and the relevant image objects related to the answer can be well distinguished by the multimodal fusion and answer prediction networks according to the multiple context information. Thus, to intuitively illustrate the effectiveness of MCL layers, we selected three models to visualize the learned attention weights by equation (14): the MCLN-w/o without the MCL layer, the MCLN-1 with one-layer MCL, and the MCLN-3 with three MCL layers. The top-2 object attention weights and all word attention weights are visualized in Figure 6. As can be seen from the visualization of attention weights, MCL contributes to focusing on relevant objects and keywords. For the correctly predicted example by three models, although the MCLN-w/o can locate the elephants, the MCLN-1 and MCLN-3 can focus on the target objects more fine-grained and the target objects with the higher attention weight value. The incorrectly predicted examples by the MCLN-w/o model are more complex and require a comprehensive understanding of the contexts of image and question. Due to the lack of MCL, MCLN-w/o is not able to accurately locate keywords and relevant objects according to the context information, resulting in wrong answers. For example, MCLN-w/o mainly pays attention to the “people” but ignores other keywords (e.g., “many” and “board”), which means that the context-dependency of “people on board” cannot be captured. In addition, compared with MCLN-1, more reasonable attention weights are usually learned by the MCLN-3. As shown in Figure 6, MCLN-2 mainly attend the keywords like “people,” “on,” and “board” in the second question. In the third question, the mainly attended keywords by the MCLN-2 are “person,” “white,” and “holding.” In other words, the deep-level textual contexts like “people on board” and “white person holding” are captured by MCLN-3.Figure 6
Visualizations of the learned attention weights on the VQA v2.0 dataset. The colors ranging from clear to red on image objects or words denote the attention weights from 0 to 1.


### 3.5. Comparison with State-of-the-Art Methods

In Tables5 and 6, we compare our MCLN model with the current state-of-the-art (SOTA) models on the GQA and VQA v2.0, respectively. And the text encoder of compared models is based on RNN architecture. The proposed MCLN-LSTM is based on RNN architecture, while the text encoder of proposed MCLN-BERT is based on BERT and it enhances the textual context learning at the textual features extraction stage. Furthermore, two proposed MCLN are deep context learning models; they not only consider two intra-modal contexts but also learn the ignored visual-textual context.Table 5
Comparison with the current state-of-the-art methods on GQA test datasets.
ModelTest-devTestCNN + LSTM [5]—46.6BUTD [13]—49.7MAC [15]—54.1LCGN [19]55.856.1OCCAM [31]56.256.3MCLN-LSTM56.456.6MCLN-BERT56.857.0Table 6
Comparison with previous state-of-the-art methods on VQA v2.0 test dataset.
ModelTest-devTest-stdAllY/NNumOtherAllBUTD [13]65.3281.8244.2156.0565.67MFH [32]68.7685.3149.5659.89-Counter [33]68.0983.1451.6258.9768.09v-AGCN [17]65.9482.5845.1256.7166.17ReGAT [16]70.2786.0854.4260.3370.58DFAF [20]70.2286.0953.3260.4970.34MCAN [21]70.6386.8253.2660.7270.90MEDAN [22]70.6087.1052.6960.5671.01MCLN-LSTM70.2685.9553.1860.7270.63MCLN-BERT71.0587.4353.2861.0871.48On the GQA dataset, the compared results of two MCLN models and the SOTA models are shown in Table5. Among them are CNN + LSTM [5], BUTD [13], and MAC [15] without any context learning. LCGN [19] is a deep context learning model and OCCAM [31] is a shallow context learning model, but they both only take into account the context of the image modality. The highest accuracy of the model without context learning is 54.1% and the highest accuracy of the model with visual context learning is 56.3% on the test set. For the MCLN-LSTM also using RNN to extract the word features, compared with MAC and OCCAM, the accuracy is 2.5% and 0.3% higher, respectively, while for the MCLN-BERT employing BERT, higher gains, 2.9% and 0.7% are achieved compared with MAC and OCCAM. Additionally, MCLN-LSTM and MCLN-BERT both overperform the SOTA models, proving the excellent performance of the proposed model on the GQA.On the test sets of VQA v2.0, the proposed models are compared with SOTA models to further verify their advantage. The compared models include three models without context learning (BUTD [13], MFH [32], Counter [33]), two shallow models that only learn the visual context (ReGAT [16], v-AGCN [17]), and three deep models with intra-modal context learning (DFAF [20], MCAN [21], MEDAN [22]). Table 6 shows the compared experiment results on the VQA v2.0 test-dev and test-std sets. For the RNN-based MCLN-LSTM model, compared with three methods that ignore the context learning, on the test-dev set, 4.94%, 1.5%, and 2.17% overall accuracy are improved by the MCLN-LSTM. ReGAT only models the visual context on the image modality, which combines various relation learning models and the Counter model [33]. By comparison, the single MCLN-LSTM not only is higher than ReGAT on test-std, but also has an advantage in the number of models. The DFAF model ignoring the visual-textual context captures intra-modal contexts via two intra-modality attention flows and uses two separate inter-modality attention flows to achieve the inter-modal interactions, but the two separate inter-modality information flows do not reflect the synergic context dependence of image objects and question words under the two-modality information. On the contrary, the MCLN-LSTM not only captures intra-modal contexts, but also achieves the synergic visual-textual context learning as well as the inter-modal interactions through a single VTCL module. As shown in Table 3, MCLN-LSTM is 0.04 and 0.29 points higher than DFAF on test-dev and test-std. By using the encoder-decoder architecture, MCAN and MEDAN achieve optimal performance. Even though the overall accuracy of the proposed MCLN-LSTM model is lower than MCAN and MEDAN, where BERT is introduced to enhance the textual context at the textual features extraction stage, the MCLN-BERT model overperforms these two models on the overall accuracy. The results indicate that MCLN considering comprehensive contexts is superior. Furthermore, compared with other state-of-the-art models, the significant improvements are obtained by MCLN-BERT and it gains 71.05% and 71.48% overall accuracies on the test-dev and test sets, respectively.From the compared experiment results on GQA and VQA v2.0 dataset, it can be seen that the proposed MCLN gains better results due to the fact that it considers more comprehensive contexts. On one hand, MCLN not only captures two intra-modal contexts but also learns the ignored visual-textual context. On the other hand, MCLN enhances the textual context learning at the textual features extraction stage.


## 3.1. Datasets and Evaluation Metric

VQA v2.0 dataset is the most commonly used large-scale VQA dataset [4], which contains 1.1 M questions asked by humans and 10 answers are collected for each image-question pair from human annotators. The answer with the highest number of occurrences will be regarded as the correct answer. The dataset is divided into three parts: a training set containing 80 K images and 444 K questions, a validation set containing 40 K images and 214 K questions, and a test set containing 80 K images and 448 K questions. Additionally, based on the answer category, all questions are divided into three types: yes/no, number, and other; we use the following accuracy as the evaluation metric for answering quality.(16)accuracyanswer=minprovidedannotators3,1.To demonstrate the generalization of approach, we further evaluate the model on the GQA dataset [4]. GQA dataset is the latest large-scale VQA dataset containing more than 110 K images and 22 M questions [5]. The dataset is divided randomly into proportions of 87%, 12%, and 1% for train, validation, and test sets, respectively.


## 3.2. Experimental Setup

### 3.2.1. Universal Setup

The proposed MCLN is implemented by PyTorch and all the experiments are conducted on a workstation; the experimental environment is shown in Table1. The universal hyperparameters of MCLN model that are used in the experiments are listed as follows. For the pretrained Faster R-CNN, we follow the strategies in [13] to set its parameters and obtain a dynamic number of objects K ∈ [10, 100]. For textual features, we use either a LSTM with a 300-dimensional GloVe word embedding size or a pretrained BERT model with the 768-dimensional embedding size. In all context learning modules, we set the number of heads H as 12, so the latent dimensionality for each head is dh = d/12 = 64. Adam Optimizer [31] is used to optimize the model with 64 batch sizes.Table 1
Experimental environment.
ConfigurationDetailsOperating systemUbuntu 18.04RAM64 GGraphic processing unitNVIDIA GeForce GTX 2080 TiProgramming languagePython 3.6Deep learning frameworkPyTorch 1.0.1Architecture platformCUDA10.0 and CUDNN7.4


### 3.2.2. Setup for VQA v2.0

The answers with an occurrence rate less than 8 times in the training and validation sets are discarded, which resulted inA=3129 candidate answers. The maximum length of tokenized words is L = 14. Moreover, the number of MCL layers is N ∈ {1, 2, 3, 4}. For the choice of learning rate lr, the warm-up strategy is employed. Specifically, the initial learning rate is 2.5 × 10−5 and grows by 2.5 × 10−5 at each epoch till it reaches 1 × 10−4 at epoch 4. After 10 epochs, the learning rate is decreased by 1/5 for every epoch up to 12 epochs. Since there exist multiple correct answers for a question in the VQA v2.0 dataset, the binary cross-entropy loss (BCE) is applied to optimize MCLN:(17)BCE loss=−∑i=1Ayilogpi+1−yilog1−pi,where yi is the given score by the datasets and pi is the predicted score by the MCLN model and corresponds to the i-th element in the answer vector p.


### 3.2.3. Setup for GQA

The GQA dataset providedA=1878 candidate answers. The maximum length of tokenized words is L = 29. We also use the same learning rate strategy on the GQA dataset, while after 8 epochs the learning rate is decreased by 1/5 for each up to 10 epochs. For the loss function, we choose the cross-entropy loss (CE) to optimize our model on GQA dataset.(18)CE loss=−∑i=1Ayilogpi.


## 3.2.1. Universal Setup

The proposed MCLN is implemented by PyTorch and all the experiments are conducted on a workstation; the experimental environment is shown in Table1. The universal hyperparameters of MCLN model that are used in the experiments are listed as follows. For the pretrained Faster R-CNN, we follow the strategies in [13] to set its parameters and obtain a dynamic number of objects K ∈ [10, 100]. For textual features, we use either a LSTM with a 300-dimensional GloVe word embedding size or a pretrained BERT model with the 768-dimensional embedding size. In all context learning modules, we set the number of heads H as 12, so the latent dimensionality for each head is dh = d/12 = 64. Adam Optimizer [31] is used to optimize the model with 64 batch sizes.Table 1
Experimental environment.
ConfigurationDetailsOperating systemUbuntu 18.04RAM64 GGraphic processing unitNVIDIA GeForce GTX 2080 TiProgramming languagePython 3.6Deep learning frameworkPyTorch 1.0.1Architecture platformCUDA10.0 and CUDNN7.4


## 3.2.2. Setup for VQA v2.0

The answers with an occurrence rate less than 8 times in the training and validation sets are discarded, which resulted inA=3129 candidate answers. The maximum length of tokenized words is L = 14. Moreover, the number of MCL layers is N ∈ {1, 2, 3, 4}. For the choice of learning rate lr, the warm-up strategy is employed. Specifically, the initial learning rate is 2.5 × 10−5 and grows by 2.5 × 10−5 at each epoch till it reaches 1 × 10−4 at epoch 4. After 10 epochs, the learning rate is decreased by 1/5 for every epoch up to 12 epochs. Since there exist multiple correct answers for a question in the VQA v2.0 dataset, the binary cross-entropy loss (BCE) is applied to optimize MCLN:(17)BCE loss=−∑i=1Ayilogpi+1−yilog1−pi,where yi is the given score by the datasets and pi is the predicted score by the MCLN model and corresponds to the i-th element in the answer vector p.


## 3.2.3. Setup for GQA

The GQA dataset providedA=1878 candidate answers. The maximum length of tokenized words is L = 29. We also use the same learning rate strategy on the GQA dataset, while after 8 epochs the learning rate is decreased by 1/5 for each up to 10 epochs. For the loss function, we choose the cross-entropy loss (CE) to optimize our model on GQA dataset.(18)CE loss=−∑i=1Ayilogpi.


## 3.3. Ablation Studies

On the validation sets of VQA v2.0 and GQA, the proposed MCLN will execute extensive ablation studies from three different aspects: (1) the effectiveness of the three context learning modules in the proposed MCL network architecture; (2) the impact of different layers on the performance of MCLN; (3) the effect of enhancing the textual context.


### 3.3.1. Effect of the Context Learning Module

As mentioned, three modules are designed to perform corresponding context learning. To demonstrate the effectiveness of the proposed context learning modules, we employ the MCLN model with LSTM to quantitatively ablate three modules. Table2 shows the results of various ablated versions of the model on the VQA v2.0 and GQA validation sets. It has been discovered that induction and elimination of any module have an impact on the performance. The optimum results are only achieved by integrating all three components. Results indicate that every context learning module contributes significantly to VQA performance and demonstrating the effectiveness of context learning modules. The result on VTCL module is noteworthy. For the models on VQA v2.0 dataset, model 4 only with the VTCL module obtains 62.07% overall accuracy, which outperforms model 1 more than 7.47% and higher performance is gained compared to model 2 and model 3. Furthermore, when the VTCL module is disabled in model 7 compared to model 8, a rapid decrease of 9.92% is observed. It indicates that the visual-textual context ignored by prior approaches is crucial to VQA.Table 2
The results of ablating the context learning modules on VQA v2.0 and GQA validation sets.
ModelModuleGQAVQA v2.0AllAllY/NNumOther1Without all53.0854.6069.7936.0247.502Only VCL53.4555.1369.8236.0947.993Only TCL53.5055.5369.8236.4449.724Only VTCL58.6362.0779.7942.6753.735TCL + VTCL63.8865.1782.8844.6857.156VCL + VTCL59.0462.7279.3343.2955.237VCL + TCL53.7255.7671.0136.3749.298Full modules64.4865.6883.4045.5757.53


### 3.3.2. Effect of MCL Layers

Next, we stack the full-module MCL layer in depth to evaluate the effect of MCL layers. Figure4 shows the performance of the MCLN models with the different numbers of MCL layers N ∈ {1, 2, 3, 4} on the VQA v2.0 validation set. Regarding the performance, we observed two phenomena on the overall accuracy: (1) as increasing N, the performances of MCLN models steadily improve and finally saturate at a certain number N = 3; (2) the one-layer model does not perform as well as the deeper models, but there is a quite improvement over N = 1 while a subtle decrease over N = 3 when N = 4. In addition, the performance of the MCLN with the different number of MCL layers on the GQA validation set is shown in Figure 5. Similar experimental phenomena are observed and the best performance appears at N = 2. From the experimental results, it can be seen that deeper MCL layers usually obtain higher overall accuracy. We attribute it to deep context learning, which captures more complex context information and achieves a better contextual understanding of the image and question contents. However, comparing the shallow model and deep model, the number of layers increases and the parameters of the model rise as well. As a result, the model with deeper MCL layers is difficult to be optimized and suffers from a larger risk of overfitting the training set due to the higher number of parameters. Therefore, the overall performance decreases at N = 4 and N = 3 on the VQA v2.0 and GQA, respectively.Figure 4
The overall and per-type accuracies of the MCLN with different MCL layers on the VQA v2.0 validation set. (a) Yes/no, (b) number, (c) others, and (d) overall.
(a)(b)(c)(d)Figure 5
The overall accuracies of the MCLN with different MCL layers on the GQA validation set.


### 3.3.3. Effect of Enhancing the Textual Context

Since the pretrained BERT [23] was trained on large text corpus and models the textual context by stacking the bidirectional encoder, to enhance the textual context at the textual features extraction stage, a contextualized text encoder based on BERT is introduced and fine-tuned. The results of fine-tuning the weight of pretrained BERT with different learning rates are shown in Tables 3 and 4. On the VQA v2.0, the best performance is achieved at lr × 0.1 under one-layer MCL. With this learning rate, by increasing the MCL layers, the performance grows and also saturates at three MCL layers. On the GQA dataset, due to the differences in datasets and experimental settings, the best performance is achieved by fine-tuning the learning rate of BERT to lr × 0.01 and setting the MCL layer to 2. Compared with the MCLN-LSTM using three-layer MCL, 1% overall accuracy is improved by the MCLN-BERT using three-layer MCL and lr × 0.1 learning rate on the VQA v2.0 validation set. On the GQA validation set, when the MCL layer is set to two and fine-tuning the learning rate of BERT to lr × 0.01, the gain is 0.11% compared with MCLN-BERT and MCLN-LSTM. From these experimental and compared results, it can be found that BERT is compatible with MCLN and conducive to boosting the performance by fine-tuning it to enhance textual context at the textual features extraction stage.Table 3
The results of fine-tuning BERT on VQA v2.0 validation set.
Modellr×AllY/NNumOtherN = 1, BERT0.00165.2182.6145.7457.13N = 1, BERT0.0166.1284.0845.9757.80N = 1, BERT0.166.6185.0946.1457.98N = 2, BERT0.167.5285.1849.0958.97N = 3, BERT0.167.8685.3549.7559.27N = 4, BERT0.167.8085.7349.9458.94N = 3, LSTM-66.8684.6248.8558.34Table 4
The results of fine-tuning BERT on GQA validation set.
Modellr×AllN = 1, BERT0.00164.25N = 1, BERT0.0164.63N = 1, BERT0.164.51N = 2, BERT0.0165.22N = 3, BERT0.0165.11N = 2, LSTM-65.01


## 3.3.1. Effect of the Context Learning Module

As mentioned, three modules are designed to perform corresponding context learning. To demonstrate the effectiveness of the proposed context learning modules, we employ the MCLN model with LSTM to quantitatively ablate three modules. Table2 shows the results of various ablated versions of the model on the VQA v2.0 and GQA validation sets. It has been discovered that induction and elimination of any module have an impact on the performance. The optimum results are only achieved by integrating all three components. Results indicate that every context learning module contributes significantly to VQA performance and demonstrating the effectiveness of context learning modules. The result on VTCL module is noteworthy. For the models on VQA v2.0 dataset, model 4 only with the VTCL module obtains 62.07% overall accuracy, which outperforms model 1 more than 7.47% and higher performance is gained compared to model 2 and model 3. Furthermore, when the VTCL module is disabled in model 7 compared to model 8, a rapid decrease of 9.92% is observed. It indicates that the visual-textual context ignored by prior approaches is crucial to VQA.Table 2
The results of ablating the context learning modules on VQA v2.0 and GQA validation sets.
ModelModuleGQAVQA v2.0AllAllY/NNumOther1Without all53.0854.6069.7936.0247.502Only VCL53.4555.1369.8236.0947.993Only TCL53.5055.5369.8236.4449.724Only VTCL58.6362.0779.7942.6753.735TCL + VTCL63.8865.1782.8844.6857.156VCL + VTCL59.0462.7279.3343.2955.237VCL + TCL53.7255.7671.0136.3749.298Full modules64.4865.6883.4045.5757.53


## 3.3.2. Effect of MCL Layers

Next, we stack the full-module MCL layer in depth to evaluate the effect of MCL layers. Figure4 shows the performance of the MCLN models with the different numbers of MCL layers N ∈ {1, 2, 3, 4} on the VQA v2.0 validation set. Regarding the performance, we observed two phenomena on the overall accuracy: (1) as increasing N, the performances of MCLN models steadily improve and finally saturate at a certain number N = 3; (2) the one-layer model does not perform as well as the deeper models, but there is a quite improvement over N = 1 while a subtle decrease over N = 3 when N = 4. In addition, the performance of the MCLN with the different number of MCL layers on the GQA validation set is shown in Figure 5. Similar experimental phenomena are observed and the best performance appears at N = 2. From the experimental results, it can be seen that deeper MCL layers usually obtain higher overall accuracy. We attribute it to deep context learning, which captures more complex context information and achieves a better contextual understanding of the image and question contents. However, comparing the shallow model and deep model, the number of layers increases and the parameters of the model rise as well. As a result, the model with deeper MCL layers is difficult to be optimized and suffers from a larger risk of overfitting the training set due to the higher number of parameters. Therefore, the overall performance decreases at N = 4 and N = 3 on the VQA v2.0 and GQA, respectively.Figure 4
The overall and per-type accuracies of the MCLN with different MCL layers on the VQA v2.0 validation set. (a) Yes/no, (b) number, (c) others, and (d) overall.
(a)(b)(c)(d)Figure 5
The overall accuracies of the MCLN with different MCL layers on the GQA validation set.


## 3.3.3. Effect of Enhancing the Textual Context

Since the pretrained BERT [23] was trained on large text corpus and models the textual context by stacking the bidirectional encoder, to enhance the textual context at the textual features extraction stage, a contextualized text encoder based on BERT is introduced and fine-tuned. The results of fine-tuning the weight of pretrained BERT with different learning rates are shown in Tables 3 and 4. On the VQA v2.0, the best performance is achieved at lr × 0.1 under one-layer MCL. With this learning rate, by increasing the MCL layers, the performance grows and also saturates at three MCL layers. On the GQA dataset, due to the differences in datasets and experimental settings, the best performance is achieved by fine-tuning the learning rate of BERT to lr × 0.01 and setting the MCL layer to 2. Compared with the MCLN-LSTM using three-layer MCL, 1% overall accuracy is improved by the MCLN-BERT using three-layer MCL and lr × 0.1 learning rate on the VQA v2.0 validation set. On the GQA validation set, when the MCL layer is set to two and fine-tuning the learning rate of BERT to lr × 0.01, the gain is 0.11% compared with MCLN-BERT and MCLN-LSTM. From these experimental and compared results, it can be found that BERT is compatible with MCLN and conducive to boosting the performance by fine-tuning it to enhance textual context at the textual features extraction stage.Table 3
The results of fine-tuning BERT on VQA v2.0 validation set.
Modellr×AllY/NNumOtherN = 1, BERT0.00165.2182.6145.7457.13N = 1, BERT0.0166.1284.0845.9757.80N = 1, BERT0.166.6185.0946.1457.98N = 2, BERT0.167.5285.1849.0958.97N = 3, BERT0.167.8685.3549.7559.27N = 4, BERT0.167.8085.7349.9458.94N = 3, LSTM-66.8684.6248.8558.34Table 4
The results of fine-tuning BERT on GQA validation set.
Modellr×AllN = 1, BERT0.00164.25N = 1, BERT0.0164.63N = 1, BERT0.164.51N = 2, BERT0.0165.22N = 3, BERT0.0165.11N = 2, LSTM-65.01


## 3.4. Visualization Analysis

Assuming that, after processing the object and word features through multiple MCL layers, the keywords in a question and the relevant image objects related to the answer can be well distinguished by the multimodal fusion and answer prediction networks according to the multiple context information. Thus, to intuitively illustrate the effectiveness of MCL layers, we selected three models to visualize the learned attention weights by equation (14): the MCLN-w/o without the MCL layer, the MCLN-1 with one-layer MCL, and the MCLN-3 with three MCL layers. The top-2 object attention weights and all word attention weights are visualized in Figure 6. As can be seen from the visualization of attention weights, MCL contributes to focusing on relevant objects and keywords. For the correctly predicted example by three models, although the MCLN-w/o can locate the elephants, the MCLN-1 and MCLN-3 can focus on the target objects more fine-grained and the target objects with the higher attention weight value. The incorrectly predicted examples by the MCLN-w/o model are more complex and require a comprehensive understanding of the contexts of image and question. Due to the lack of MCL, MCLN-w/o is not able to accurately locate keywords and relevant objects according to the context information, resulting in wrong answers. For example, MCLN-w/o mainly pays attention to the “people” but ignores other keywords (e.g., “many” and “board”), which means that the context-dependency of “people on board” cannot be captured. In addition, compared with MCLN-1, more reasonable attention weights are usually learned by the MCLN-3. As shown in Figure 6, MCLN-2 mainly attend the keywords like “people,” “on,” and “board” in the second question. In the third question, the mainly attended keywords by the MCLN-2 are “person,” “white,” and “holding.” In other words, the deep-level textual contexts like “people on board” and “white person holding” are captured by MCLN-3.Figure 6
Visualizations of the learned attention weights on the VQA v2.0 dataset. The colors ranging from clear to red on image objects or words denote the attention weights from 0 to 1.


## 3.5. Comparison with State-of-the-Art Methods

In Tables5 and 6, we compare our MCLN model with the current state-of-the-art (SOTA) models on the GQA and VQA v2.0, respectively. And the text encoder of compared models is based on RNN architecture. The proposed MCLN-LSTM is based on RNN architecture, while the text encoder of proposed MCLN-BERT is based on BERT and it enhances the textual context learning at the textual features extraction stage. Furthermore, two proposed MCLN are deep context learning models; they not only consider two intra-modal contexts but also learn the ignored visual-textual context.Table 5
Comparison with the current state-of-the-art methods on GQA test datasets.
ModelTest-devTestCNN + LSTM [5]—46.6BUTD [13]—49.7MAC [15]—54.1LCGN [19]55.856.1OCCAM [31]56.256.3MCLN-LSTM56.456.6MCLN-BERT56.857.0Table 6
Comparison with previous state-of-the-art methods on VQA v2.0 test dataset.
ModelTest-devTest-stdAllY/NNumOtherAllBUTD [13]65.3281.8244.2156.0565.67MFH [32]68.7685.3149.5659.89-Counter [33]68.0983.1451.6258.9768.09v-AGCN [17]65.9482.5845.1256.7166.17ReGAT [16]70.2786.0854.4260.3370.58DFAF [20]70.2286.0953.3260.4970.34MCAN [21]70.6386.8253.2660.7270.90MEDAN [22]70.6087.1052.6960.5671.01MCLN-LSTM70.2685.9553.1860.7270.63MCLN-BERT71.0587.4353.2861.0871.48On the GQA dataset, the compared results of two MCLN models and the SOTA models are shown in Table5. Among them are CNN + LSTM [5], BUTD [13], and MAC [15] without any context learning. LCGN [19] is a deep context learning model and OCCAM [31] is a shallow context learning model, but they both only take into account the context of the image modality. The highest accuracy of the model without context learning is 54.1% and the highest accuracy of the model with visual context learning is 56.3% on the test set. For the MCLN-LSTM also using RNN to extract the word features, compared with MAC and OCCAM, the accuracy is 2.5% and 0.3% higher, respectively, while for the MCLN-BERT employing BERT, higher gains, 2.9% and 0.7% are achieved compared with MAC and OCCAM. Additionally, MCLN-LSTM and MCLN-BERT both overperform the SOTA models, proving the excellent performance of the proposed model on the GQA.On the test sets of VQA v2.0, the proposed models are compared with SOTA models to further verify their advantage. The compared models include three models without context learning (BUTD [13], MFH [32], Counter [33]), two shallow models that only learn the visual context (ReGAT [16], v-AGCN [17]), and three deep models with intra-modal context learning (DFAF [20], MCAN [21], MEDAN [22]). Table 6 shows the compared experiment results on the VQA v2.0 test-dev and test-std sets. For the RNN-based MCLN-LSTM model, compared with three methods that ignore the context learning, on the test-dev set, 4.94%, 1.5%, and 2.17% overall accuracy are improved by the MCLN-LSTM. ReGAT only models the visual context on the image modality, which combines various relation learning models and the Counter model [33]. By comparison, the single MCLN-LSTM not only is higher than ReGAT on test-std, but also has an advantage in the number of models. The DFAF model ignoring the visual-textual context captures intra-modal contexts via two intra-modality attention flows and uses two separate inter-modality attention flows to achieve the inter-modal interactions, but the two separate inter-modality information flows do not reflect the synergic context dependence of image objects and question words under the two-modality information. On the contrary, the MCLN-LSTM not only captures intra-modal contexts, but also achieves the synergic visual-textual context learning as well as the inter-modal interactions through a single VTCL module. As shown in Table 3, MCLN-LSTM is 0.04 and 0.29 points higher than DFAF on test-dev and test-std. By using the encoder-decoder architecture, MCAN and MEDAN achieve optimal performance. Even though the overall accuracy of the proposed MCLN-LSTM model is lower than MCAN and MEDAN, where BERT is introduced to enhance the textual context at the textual features extraction stage, the MCLN-BERT model overperforms these two models on the overall accuracy. The results indicate that MCLN considering comprehensive contexts is superior. Furthermore, compared with other state-of-the-art models, the significant improvements are obtained by MCLN-BERT and it gains 71.05% and 71.48% overall accuracies on the test-dev and test sets, respectively.From the compared experiment results on GQA and VQA v2.0 dataset, it can be seen that the proposed MCLN gains better results due to the fact that it considers more comprehensive contexts. On one hand, MCLN not only captures two intra-modal contexts but also learns the ignored visual-textual context. On the other hand, MCLN enhances the textual context learning at the textual features extraction stage.


## 4. Conclusions

This article presents a novel framework, the Multiple Context Learning Network (MCLN), to model multiple context learnings for visual question answering. The MCLN exploits three types of contexts, the visual context and textual context in the intra-modalities, and the visual-textual context, to learn a context-aware representation through the corresponding context learning module. The core idea of the context learning module is to establish the contextual dependency of all entities in an entity set by using key-query attention mechanism. Our context learning approach is simple but extremely effective. The MCLN is better able to learn complex contexts and improve VQA performance by composing three context learning modules to construct MCL layer and stacking such layer in depth. Furthermore, BERT-based text encoder is introduced and fine-tuned to facilitate the textual context learning at the textual features extraction stage. Experimental results on two large-scale benchmark datasets show that our proposed method outperforms the previous state-of-the-art methods. And the extensive ablation studies demonstrate the effectiveness of context learning modules, stacking MCL layers, and enhancing the textual context. However, stacking MCL layers will increase the training time of the model and make the model difficult to be optimized in practical experiments; we will optimize our model to achieve a rapid network framework in a future study. In addition, we plan to promote the proposed context learning method to other multimodal tasks such as image captioning and image-text matching.



---

*Source: 4378553-2022-02-09.xml*