# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 2 Background
### Image Captioning  
The quantity and quality of image data generated or used by both users and researchers keeps increasing due to new technologies. This is because of the availability of better image capturing hardware such as smart phones, and the popularity of social media networks where images are uploaded. Companies now need to store more data because of the quality (and by extension, size) of the images being generated. From a computer vision perspective, people are now taking more random images in random places surrounded by random objects, which requires processing by more sophisticated algorithms. This stands in contrast to images in the past, which were taken in studios or well-planned in advance and thus contained more predictable objects. This presents a greater challenge for computer vision tasks such as image captioning. As a result, there are several techniques that have been devised to reduce them to their simplest form while retaining as much of the salient information as possible, for example, by detecting and only retaining the most important features using techniques such as autoencoders and principal component analysis. The other alternative has been to devise more complicated algorithms. As a result, deep learning models have been getting more complicated, bigger and a lot of the state-of-the-art performance of a number of models such as GPT, BERT can be attributed to the amount data used to train them. Some methods using simpler architectures have been proposed to reverse this trend such as using Fourier transforms and fully connected layers [7; 33] to get performances close to the state-of-the-art. Modern image captioning methods are deep learning based, and treat the task as a sequence-to-sequence image-to-text translation task where the image features are input in the encoder in an encoder-decoder architecture which produces a fixed length feature representation vector of the image which is fed to the decoder. The decoder is the language model and is trained in an auto-regressive manner where the image feature vector and previously generated words are used to predict the next word. This technique is usually accomplished by CNN encoders and LSTM decoders [8; 9; 11; 12; 13], CNN encoders and CNN decoders [19; 20] and lately transformers [21; 22; 23; 25]. While transformers have reached several milestones in vision, language and vision-language tasks, they have a lot of shortcomings when implemented in their most popular format as proposed by the original authors [4]. For long sequences as is the case with images and usually with text too, it is very expensive to train transformers both due to the memory and computational requirements. Transformer-only vision or vision-language models [24; 34] that do not use convolutional image features as input, require a lot of image training data, as much as needing millions more images [26] to achieve the same results as convolutional based models in tasks such as classification. This is because of the initial unfocused nature of the transformer, which requires it to use some data to learn where to focus, and what to focus on. The inductive biases in convolutional neural networks (due to the kernels) and in recurrent networks (due to their inherent sequential nature) bias them towards an area on which to focus and so they only need to learn what to focus on, and not where to focus. As a result, a number of transformer-based captioning methods vision-language models use pre-training methods [22; 23] in order to achieve competitive results. In this work, we explore the effect of replacing the self-attention layer with fully connected layers and concretely analyze its impact on caption generation. We also suggest a set of guidelines and recommendations based on our observations as pertaining to the aforementioned research. Previous works only concentrate on self-attention [21; 25], recurrent [8; 9; 11] or convolutional [19; 20] implementations on the image captioning task, or analyze the scenario in same domain sequence applications [7].