# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 1 Introduction  
By allowing for the hierarchical representation of features, with complex features described in subsequent layers by successively simpler features, i.e., multiple levels of abstraction, deep learning algorithms have led to many breakthroughs in representation learning dependent tasks. This has resulted in many state-of-the-art achievements in areas such as object detection and recognition, speech recognition, Natural Language Processing, computer vision and vision-language tasks such as image captioning. The fundamental building block of deep learning networks is a neuron which is combined in different combinations, across multiple layers to create more and more powerful networks. This includes simple supervised learning algorithms like Multi-Layer Perceptrons (MLPs), through to more complex architectures such as Recurrent Neural Networks (RNNs) [1], Long Short-Term Memory (LSTMs) [2], Convolutional Neural Networks (CNNs) [3] and Transformers [4]. With the emergence of massive models such as the Bidirectional Encoder Representations from Transformers (BERT) [5], and Generative Pre-trained Transformer (GPT-3) [6], the trend has been to go bigger, more complex and using of more data. The results show that this has been positively rewarding with these systems achieving state-of-the-art results in every field they target. Recently, works such as [7] have shown that it is possible to achieve competitive results in text classification and translation tasks while using simpler architectures. These architectures alleviate some of the limitations inherent in the more complex ones such as the high computational and high memory requirements that result from, for example, the quadratic nature of self-attention mechanisms that limit sequence length. In our paper, where we use image captioning as proof of concept, we completely discard the self-attention mechanism in the image encoders, and the masked self-attention mechanism in the decoder and introduce a series of feed-forward layers in a regular deep feed-forward framework, and evaluate the most optimum way to represent the input features. We aren't attempting to achieve a new state-of-the-art score, but provide a qualitative analysis of the resulting captions, an empirical analysis of the evaluation metrics, and memory usage, providing a practical insight into the effect of this substitution in vision-language tasks while also demonstrating competitive results with the much simpler architecture. In image captioning tasks, a machine is designed to automatically generate captions describing an input image. To achieve this, an image feature extractor and a language model are needed. The language model has commonly been implemented by using LSTMs [8; 9; 10; 11; 12; 13] and image feature extraction through a state-of-the-art CNN architecture like ResNet [14], VGGNet [15], Inception [16] pre-trained on a large dataset like ImageNet [17]. In this arrangement, the image captioning task is treated as a sequence-to-sequence \"translation\" task, a technique inspired by the Neural Machine Translation [18]. Due to their sequential nature, which inhibits parallelization and them being slow, alternatives to LSTMs have been explored and implemented resulting in fully convolutional [19; 20] and fully attentive architectures such as transformers. First proposed in the paper \"Attention is all you need\" [4], transformers consist of a stack of self-attention and linear layers in an encoder-decoder architecture. Transformer-based models like [21; 22; 23; 24; 25] have achieved state-of-the-art results in image captioning. To achieve this [22; 23; 24] use a data-intensive pre-training step to achieve good results. Transformer-only architectures for image captioning like [24] also require massive amounts of pre-training data, and so use a pre-trained encoder from [26], otherwise they report very poor results when training from scratch.