# Incorporating Concreteness in Multi-Modal Language Models with Curriculum Learning
## 4 Experiments
The best result is obtained when both multi-modal pre-training and attentive pooling mechanisms are used, although the performance is consistent across all configurations. In terms of accuracy, there is a 1.01% difference between the best performing model (with multi-modal pre-training and attentive pooling) and the worst (with fully connected layer and without multi-modal pre-training). Performance difference becomes more significant in F1: a 3.37% increase can be observed between the best and worst-performing models (model with multi-modal pre-training and attentive pooling, and model without multi-modal pre-training with a fully connected layer, respectively, similar to the previous case). One can better analyze performance differences with ablation studies. Table 6 reports the relative improvements of each component. Each column represents the percentage \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & **Multi-Modal** & \multirow{2}{*}{**Combination Method**} & \multirow{2}{*}{**Accuracy**} & \multirow{2}{*}{**F1**} & \multirow{2}{*}{**Precision**} & \multirow{2}{*}{**Recall**} \\  & **Pre-Training** & & & & & \\ \hline [PERSON] + Resnet & ✗ & FC & 53.12 & 50.71 & 54.07 & 53.12 \\ [PERSON] & ✓ & FC & 53.17 & 52.79 & 53.34 & 53.17 \\ [PERSON] + Resnet & ✗ & AP & 53.56 & 52.91 & 53.69 & 53.56 \\ [PERSON] + Resnet & ✓ & AP & 54.13 & 54.08 & 54.07 & 54.13 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Model performance on VQA dataset v2. (FC = Fully-connected, AP = Attentive pooling). increase in relative performance when the feature/component in the row is replaced or enhanced by the feature/component in the column. The results show that multi-modal pre-training increases the model's performance regardless of the underlying fusion mechanism (Fully-connected or attentive pooling). It leads to a 4.1% increase when used with fully connected layers and leads to a 2.21% increase when used with attentive pooling networks. Similarly, the attentive pooling mechanism improves the performance of the model in both cases: When the fully-connected layer is replaced with attentive pooling, it amounts to an increase of 4.34% without multi-modal pre-training and an increase of 2.44% with multi-modal pre-training. Additionally, from the first row, we can conclude that replacing FC with an attentive pooling mechanism is slightly more beneficial than using FC together with multi-modal pre-training. Overall, as the results suggest, using both attentive pooling and multi-modal pre-training proved to be useful and led to an increase in performance up to 6.65% compared to the baseline model. Table 7 shows the performance of the multi-modal models described in Section 2 on the VQA task. We share the results on version 1 and version 2, though it would only be fair to compare the models that run on the same version. The models that run on both versions (stacked attention network (SAN) and GVQA) suggest that a performance difference between 3-7% can be expected between the versions, most likely due to the effect of language priors. Human baselines, obtained on the 3000 samples in the training set of the v1 dataset, are also provided in the top part. Although human baselines are on v1 and our performance is on the v2 version of the dataset, our 54.13% accuracy indicates that the model can perform similarly to humans when given only questions and corresponding captions without images. Compared to the other models, ours performed better than the earlier models but cannot reach the success obtained by the state-of-the-art model (VilBERT), which has 70.92% accuracy. VilBERT processes paired visiolinguistic data in the architecture of BERT to exploit visual grounding in a task-agnostic way. It should be noted that there are subtle but vital differences between our model and the VilBERT model. The main focus of VilBERT is to process text and image streams in parallel