# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 1 Introduction
In M2 [25], the authors employ a multi-level meshed encoder and decoder increasing complexity and working within the limitations of the quadratic nature of the self-attention technique. Transformers are more computationally expensive, more data hungry and memory intensive architectures. In order to study possible ways to alleviate some of the above-mentioned problems, while still maintaining a competitive performance, in this paper, we implement and analyze transformer architectures that utilize an encoder composed solely of fully connected feed-forward layers and a decoder that also uses a deep fully connected feed-forward network to encode caption embeddings as shown in Figures 1 and 2. The feed-forward architecture is a simple yet powerful architecture that can adequately model the input textual and image feature representations. The complexity of the feed-forward only encoder section is linear instead of quadratic. In order for it to handle image captioning which involves conditional sequence generation conditioned on the contextual image and textual information, we retain intra-attention only in the cross-attention layer. Furthermore, the input image features used are the output of a Faster R-CNN [27] network, which significantly helps to reduce the dimensions of the input image feature vectors, while retaining and highlighting (via regional proposals) most of its semantic information. Our main contributions are summarized below:  
* We propose transformer-based systems that, in the first stage, utilize a feed-forward encoder using deep fully-connected layers and, in the next stage, a decoder that uses feed-forward layers to encode caption embeddings. We study the memory usage at different batch sizes, the captions generated and the implication of the evaluation metric scores. * We propose, implement and perform a comprehensive analysis of several combinations of fully connected layers on image captioning models in order to add to the body of knowledge available about the use of simpler networks, in this case, linear layersin transformer-based vision-language systems, which is an area that has hardly been explored. * We make a comparative analysis of the use of fully connected layers in vision language models in comparison to self-attention layers and also show that masking of the decoder input features is unnecessary in feed-forward based caption encoding by implementing an unmasked decoder input transformer module. * We make recommendations and offer tips based on the observations made during the study in order to foster further research in the area of developing simpler transformer models for vision-language applications. * The MSCOCO benchmark dataset is used to evaluate the performance of the proposed models, which demonstrate competitive results despite the simplicity of the models. The models also show a shorter training time, and lower memory usage at higher batch sizes in spite of containing more parameters. Lower memory requirements and training time allow for use in lower end systems. Figure 1: Model complexity reduction: Contrasting the two methods of creating the decoder feature representation that is sent to the cross-attention layer. On the left is the default scaled product attention method that requires masking. The figure on the right shows the much simpler and direct fully connected (FC) layers, with no look-ahead masking or self-attention.