# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 4 Experiments and Results
### Evaluation Studies  
We perform memory consumption and training time per epoch tests (results shown in Table 1), and implement several model variants (configurations shown in Table 2 and Figure 5). The evaluation metric results of our best model compared to the comparison models are shown in Table 3 while those of the model variants are displayed in Table 4. Details of the model variants and evaluation studies are discussed below: where **X** refers to the input feature set of either text or image features. (i) To _evaluate the memory consumption_ of our model to a similar one using self-attention, we designed a variant that we refer to as FC-SA. In this variant, the encoder feed-forward network is replaced by a self-attention network, and the decoder one, replaced by a masked self-attention layer. The GPU memory consumption with the batch size  
Figure 4: The Graphs above show various evaluation metrics across the training epochs for select model variants corresponding to: (**a**) BLEU-1, BLEU-4 (**b**) CIDEr, (**c**) METEOR, and (**d**) SPICE scores. set to 10, 20, 40, 50, 80, 100 and 150 for both our best model (FC4) and the self-attention based variant FC-SA are shown in Figure 3a. 2. To _evaluate the memory consumption_ at different sequence lengths, just as above, we use the FC4 and FC-SA variants. Since it is just a test of memory consumption and not a test of caption generation, we duplicate the captions to create longer captions, i.e., every caption is a multiple of its original length. So we measure memory usage at multiples of the caption length, i.e., (1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8) * caption length. Basically, given a caption sequence \(X=\{x_{1},x_{2},...,x_{n}\}=X*1.0\), then, \(X*1.5=\{x_{1},x_{2},...,x_{n},x_{1},x_{2},...,x_{\frac{n}{2}}\}\) where \(n\) is the original sequence length. The results are shown in Figure 3b. We also report the inference time required to generate captions for 5000 randomly selected images. 3. To investigate the performance of the model before and after using a look-ahead mask, we ran the FC4 and FC-SA variants with and without the mask. 4. To evaluate the time complexity of the introduction of the feed-forward layers, we build 2 variants of our model that are comprised of only the encoder. One variant contains the self-attention layers. In the other variant, the self-attention layers are replaced by feed-forward layers. The time taken to produce the image feature representation is noted and shown in Table 1.
5. FC0, FC1 and FC2 _investigate the effect of change in number of layers or stacks_ on the base variant model FC0. FC0 is designed with the \"normal\" configuration as shown in Table 2 where input features (2048-dimensional) are directly transformed into the model dimension (512-dimensional) and forwarded to the next stage, i.e., position-wise feed-forward stage, without any other changes. The experiments covered 3-, 6- and 12-layer variants. A layer here refers to one whole encoder or decoder section. Varying the number of layers may improve or degrade the performance of the model. While it could initially improve performance because of the additional discriminatory information mined, the increase in the complexity may result in a reduction in performance, and an explosion in the size of the model and training time. This is discussed further in Section 5. 6. FC3 is used to investigate _mapping the input image features into a higher dimensional space._\(\mathbf{X}\in\mathbb{R}^{N\times D}\xrightarrow{\text{mapped}}\mathbf{X}^{\prime}\in\mathbb{R}^{N\times D^{ \prime}}\), where \(D^{{}^{\prime}}>D\). This is the initial building block for the next couple of experiments. In this experiment, we try to find out the effect of mapping the input image features (2048-dimensional) into a higher dimensional space (4096-dimensional), an arrangement that mimics an over-complete autoencoder. The aim is to see if it creates a better representation of the input features, which would then lead to better captions.