## 3 Model
### Multi-View Attention Network
#### 3.2.3 Modality Alignment Module  
Given the output representations of the Context Matching module \(e^{\mathcal{C}}\in\mathbb{R}^{2d_{w}\times 1}\) and Topic Aggregation module \(\{e_{i}^{\mathcal{T}}\}_{i=1}^{n_{q}}\)\(\in\mathbb{R}^{2d_{w}\times n_{q}}\), the Modality Alignment module aligns them with visual features \(\{v_{k}\}_{k=1}^{n_{q}}\)\(\in\mathbb{R}^{d_{w}\times n_{q}}\) via two-step alignments. First, topic-view alignment performs soft alignment between heterogeneous modalities at topic level before mapping high-level contextual representation with the visual features. We utilize dot-product attention to represent the visual relevant topic-aggregation embeddings as follows:  
\[z_{k}^{\mathcal{T}} =f_{\ell}^{\mathcal{W}}(e_{i}^{\mathcal{T}})^{\top}f_{\ell}^{ \mathcal{T}}(v_{k}) \tag{7a}\] \[a_{ik}^{\mathcal{T}} =exp(z_{ik}^{\mathcal{T}})/\sum_{i=1}^{n_{q}}exp(z_{ik}^{\mathcal{ T}})\] (7b) \[\bar{e}_{k}^{\mathcal{T}} =\sum_{i=1}^{n_{q}}a_{ik}^{\mathcal{T}}e_{i}^{\mathcal{T}}, \tag{7c}\]  
where \(f_{\ell}^{\mathcal{W}}(\cdot)\) and \(f_{\
u}^{\mathcal{W}}(\cdot)\) are non-linear transformation functions to embed two different modality representations into the same \(d_{f}\) dimensions. Then, we obtain the fused feature vectors by concatenating the visual and attended topic-aggregation features and using a Multi-Layer Perceptron (MLP):  
\[m_{k}^{\mathcal{T}}=\text{MLP}([v_{k},\bar{e}_{k}^{\mathcal{T}}]). \tag{8}\]  
Note that \(\{m_{k}^{\mathcal{T}}\}_{k=1}^{n_{q}}\in\mathbb{R}^{d_{w}\times n_{q}}\) is a topic-view aligned representation that is aligned information across the visual contents with the salient question topics. This is passed to the context-view aligning step as follows:  
\[\hat{m}_{k}^{\mathcal{C}} =[m_{k}^{\mathcal{T}},e^{\mathcal{C}}] \tag{9a}\] \[z_{k}^{\mathcal{C}} =\mathbf{W}^{\top}(\text{L2 Norm}(f_{m}^{\mathcal{C}}(\hat{m}_{k}^ {\mathcal{C}})\circ f_{\ell}^{\mathcal{C}}(e^{\mathcal{C}})))+b\] (9b) \[a_{k}^{\mathcal{C}} =\text{softmax}(z_{k}^{\mathcal{C}})\] (9c) \[\hat{m}^{\mathcal{C}} =\sum_{k=1}^{n_{q}}a_{k}^{\mathcal{C}}\hat{m}_{k}^{\mathcal{C}}, \tag{9d}\]  
where \(f_{m}^{\mathcal{C}}(\cdot)\) and \(f_{\ell}^{\mathcal{C}}(\cdot)\) are non-linear transformation functions which convert \(\hat{m}_{k}^{\mathcal{C}}\) and \(e^{\mathcal{C}}\) into the same \(d_{f}\) dimensions and \(\mathbf{W}\in\mathbb{R}^{d_{f}\times 1}\) is a projection matrix. Note that \(\hat{m}^{\mathcal{C}}\in\mathbb{R}^{(d_{w}+2d_{w})\times 1}\) is a context-view aligned representation, which is realigned using context-matching representation. These multiple alignment processes allow the model to understand the semantic intent of the question with complementary views, and effectively align the corresponding heterogeneous multimodal inputs. Finally, this enhanced feature is fed into a single-layer Feed-Forward Neural Network (FFNN) with a ReLU activation function.  
\[m^{enc}=\text{max}(0,\mathbf{W}^{\top}\hat{m}^{\mathcal{C}}+b), \tag{10}\]  
where \(\mathbf{W}\in\mathbb{R}^{(d_{w}+2d_{t})\times d_{enc}}\) and \(b\in\mathbb{R}^{d_{enc}\times 1}\) are trainable parameters. Note that \(m^{enc}\in\mathbb{R}^{d_{enc}\times 1}\) is the multi-view aligned representation, which is fed into either a discriminative or generative decoder.