# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 5 Results Discussion
We do not apply masking in the proposed models and show that our model inference works properly and believe that the model actually benefits from having current and previous information during training. In language generation, the context of a word may not only depend on the words before it, but also those that may come after it. Almost all supervised sequence generation deep learning based models use teacher forcing, and not using masking as in our case is simply an extreme case of teacher forcing. The concern that the deep learning models, which are adept at finding and exploiting any patterns will simply learn to predict the next word in the input distribution by looking at the next word in the sequence is not completely valid, at least, not in our models. The model is still able to learn and capture the semantics in the input representation well enough that it is able to predict the next word at inference time. This is proven by the accurate and relevant captions that are auto-regressively generated during inference, with deeper decoder feed-forward variants performing better at caption generation because they are able to learn to create a better representation of the input caption features. When we performed an explicit comparison between the performance of the model with and without using the look-ahead mask, between our best model, FC4 and its counterpart based on a self-attention mechanism (FC-SA), we discovered inconsistencies between the results. While both models do actually operate with or without the look-ahead mask, the difference was the level of variability between results, which seems to point to the likelihood that using a look-ahead mask does also help reduce on the stochastic nature of the model. Judged solely on the results of the CIDEr evaluation metric, based on only four complete runs of the models (train + test) due to time and computational restraints, there was an average of a 6% variance in the results for the FC-SA compared to 2% for the FC4 model. Over the four runs, the CIDEr scores, without using a look-ahead mask were 127.7, 128.8, 121.1, 123.0 and 122.0, 121.9 124.4, 124.2 for the FC-SA and the FC4 model, respectively. When a look-ahead mask was used, the score varied by less than 1% for both, i.e., 128.7, 128.8, 128.7, 128.8 and 124.3, 124.4, 124.1, 124.4, for the FC-SA and the FC4 model, respectively. These results show that the look-ahead mask is more important in self-attention based mechanisms than it is in feed-forward based mechanisms. It further shows that using it also improved on consistency between the results obtained. From a practical point of view, in terms of captions generated, the difference is much smaller. We believe that this is due to the beam search algorithm that we use, combined with the sampling method that is based on a softmax probability distribution over a predetermined vocabulary set, ensuring that the most appropriate token is chosen. This means that there needs to be a much bigger difference between the evaluation metric scores for the quality of the captions to start decreasing. Stacked LayersIncreasing the number of layers is not a sure way to improve performance and may be detrimental to the performance of transformer models as shown by [49]. In image captioning, ref. [25] got their best performance when they stacked three encoder and three decoder models, and showed a significant drop in performance when they tried to stack more decoder layers. In our experiments using linear models we actually had mixed results. The models usually performed better, the more modules we stacked, up to a certain point where there was little to no improvement. Our best model was the three-layer FC4 model, but the same model with six layers, performed about the same, in fact marginally worse in some metrics. In the case of the base models (FC0, FC1, and FC2), the six-layer model performed almost the same as the 12-layer model, but both performed much better than the three-layer variant.