# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 5 Results Discussion
We also observed the best performance in a six-layer architecture, when we only cross interfaced the top three stacks, with the lower three just processing and sending their outputs to the ones above them. We believe that this is because this increased depth starts to cause optimization issues. The models that first encoded the image features using deep encoders (FC3, FC5) initially had high scores, especially in the BLEU-4 metrics before dropping. This is because in the earlier stages of the training process, the deep fully connected nature of the model allows it to easily map individual aspects of input features to the captions more efficiently, but as the model gets more refined and the focus turns to mapping an understanding the relationship between the individual feature and the captions, the deep nature becomes a hindrance to image representation. It starts experiencing optimization issues. The captions produced at this point, initially containing accurate but repetitive patterns, start becoming syntactically inaccurate. It does eventually converge, but produces the worst captions. Graphs highlighting the SPICE vs. BLEU-1 scores are shown in Figure 10(c). It shows that even with the high number of accurate unigrams generated (BLEU-1), the resulting sentences did not correlate well with human generated captions (SPICE). Comparison modelsWhen FC4 is compared to the comparison models as in Table 3, the results using the fully connected layers are higher than the compared models which consist of state-of-the-art LSTM and CNN-based models. They are a few points behind the performance of the best transformer models in many of the metrics, but almost at par with them in the unigram-based BLEU scores. This shows that with some more optimization, performance could be at par with the current best models that employ self-attention to encode images and text. The lower performance in the other metrics especially the CIDEr and SPICE scores correlates with the caption results which show that while the fully connected models are able to pick up on the individual salient information available in the images or text, they are not as capable as intra-attention networks in maintaining long distance dependency and relevancy information. As a result, while the captions generated are accurate, they are noticeably shorter, which leaves out some of the details. To compensate for the reduction in the amount of context about the relationships between each caption token and the other tokens, we implement a deeper neural network to capture more of the \"intra-caption\" relevancy semantics. The results obtained by FC4, especially the SPICE scores show that this allowed much better captions to be generated as more information was learned by the layers. There are a number of other experiments that we carried out where we kept image sequence length constant while varying the caption sequence length and vice-versa. These experiments mostly showed very similar memory consumption, usually with the self-attention variant only slightly better. This is particularly surprising observation given that the fully connected feed-forward variant has about 70% more parameters and the saved models 70% larger than the self-attention variant. Part of the surprisingly good performance of the feed-forward model could be because it is quite similar to a convolutional neural network. A feed-forward neural network can be seen as a special type of CNN with a single channel and full receptive field. The similar performance of the two models in terms of memory usage in spite of the feed-forward one having a lot more parameters implies that the self-attention model actually has a worse performance per parameter. This means that if the feed-forward algorithm is refined to decompose into fewer parameters, its memory consumption can be significantly less than that of the self-attention mechanism. From the experiments, we make some recommendations and guidelines for dealing with fully connected layers in vision-language settings, specifically image captioning. 1.