# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 5 Results Discussion
This comes out to an average of about 39.7 compared to 43.9 images per second for the feed-forward and self-attention variant, respectively, on the GPU server. Since the inference part is one that is likely to be processed locally on a desktop pc or laptop, we repeated the test on an Intel i7-6700 HQ processor with an Nvidia GeForce-GTX 960M GPU, where it took 16.2 min (5.1 images per second) compared to 15.8 min (5.2 images per second) for the feed-forward and self-attention variant respectively. A large chunk of the processing time for this sequential task is spent in dealing with copying and moving images in memory and so it is not surprising that there is a negligible difference in the time required to process each image for either one of the two networks. The disparity only becomes non-negligible when creating captions for several thousands of images, on a sufficiently powerful machine. \begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{1}{c}{**GT:** a large semi truck pulling a blue tractor behind it.} \\
**FC0:** a large truck is driving down a traffic light. \\
**FC1:** a large dump truck driving down a traffic light. \\
**FC6:** a construction truck carrying a crane on the road. \\
**FC4:** a construction truck driving down a traffic light on the road \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Sample captions generated by the models. Memory usage:The graphs in Figure 3 show the memory usage of our best model (FC4) compared to a variant that uses self-attention layers (FC-SA). From the Figure 3a it can be observed that the feed-forward variant only uses slightly more memory as the batch sized increase from 10 to 80. At a batch size of 100, there is a sharp change in the memory usage of both models. FC4 shows a sharp decrease in memory usage while the usage for FC-SA stops increasing rapidly, almost flattening out. This is particularly fascinating since it contains about 70% more parameters than the self-attention based counterpart. We think that the 100 batch size created ideal conditions for the GPU we used, wherein, the batch size saturates the CUDA cores and the GPU performance is most efficient. This ideal performance continues to the next tested batch size. The FC4 (feed-forward) variant benefits most, using 25% less memory at that ideal batch size. This shows that the simplicity of the proposed model, containing fewer components reduces on the computational complexity of the setup. Figure 3b contains a graph that shows the memory consumption of the two compared models (FC4 and FC-SA) at different caption sequence lengths. As can be seen, as the sequence length of the captions increases, the quadratic nature of the self-attention mechanism starts manifesting itself. A repeated observation concerning the feed-forward variant is that at some point, the GPU is able to efficiently represent the linear layers and this results in a significant drop in memory consumption. The self-attention Q-K-V mechanism is also made up of linear layers organized in a matrix form, which also benefits from the GPU efficiency at that ideal sequence length, resulting in a reduction in its memory usage too. This is can be interpreted from the less steep slope. Its quadratic nature still causes it to continue increasing its memory consumption at a faster rate than its feed-forward counterpart. A GPU is very adept at handling the matrix multiplications which constitute most of the work done during the training process. The linear transformations in both variants benefit greatly from this. Non-masked caption embedding representation:One of the major changes to the network architecture was the utilization of a non-masked fully connected layer to capture the semantic information and token relationships of the caption embeddings in place of intra-attention. In previous implementations of transformer-based text generation tasks, the self-attention layer over the captions is masked so that the network only has access to previous tokens when learning to predict a future token.