# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 3 Method
### MLPs in Image Captioning  
Using MLP sub-modules, our objective is to demonstrate and analyze transformer-based vision-language models, specifically, image captioning models. The models we analyze and propose use linear layers to encode and transform the image and caption embeddings rather than using self-attention yet still generate satisfactory captions and evaluation scores comparable to their self-attention-based counterparts. Transformer models [4], as used in image captioning tasks, consist of two parts: an encoder and decoder section. The encoder creates a fixed vector representation of the input image which is sent to the auto-regressive decoder, which uses this representation togetherwith the caption embeddings representation to create a probability distribution over the vocabulary. From this probability distribution, the next word in the caption sequence is sampled, i.e., to predict the next token \(x_{t+1}\), we compute the \(P(x_{t+1}|I,x_{1},x_{2},...,x_{t})\) where \(I\) represents the image features and \(x_{1},x_{2},...,x_{t}\) are the previously generated word tokens. Unlike the typical models that consist of a self-attention network to compute the relevance or attention score of each image feature in relation to the other features, we use linear transformations in conjunction with activation function non-linearities to learn a representation of the input. The image input features are transformed through one or several fully connected layers as in (2) and (3), respectively. \[\mathbf{X}_{i}=\phi(\mathbf{W}^{\mathrm{T}}\mathbf{x}_{n}+b), \tag{2}\]  
where \(\phi,\mathbf{W},b\) are the ReLu activation function, weight and bias of a fully connected layer, \(\mathbf{x}_{i}\in\mathbb{R}^{N_{i}\times D}\) is the input image sequence with \(N_{i}\), \(D\) being the number of input features and the dimension of each input feature, respectively,  
\[\mathbf{X}_{i}=g(\mathbf{x}_{n}), \tag{3}\]  
where \(g(\mathbf{x}_{n})=\phi\Big{(}\mathbf{W}_{(L)}...\phi\Big{(}\mathbf{W}_{(1)} \mathbf{x}_{n}+b_{1}\Big{)}+b_{L}\Big{)}\). \(L\) is the layer number in a multi-layer setup. For the deeper models as in (3), akin to autoencoders, the input features \(\mathbf{x}_{n}\) can be transformed into latent space \(h=f(\mathbf{x}_{n})\) then the features reconstructed \(r=g(h)\). In an under-complete architecture \(h\) is of a lower dimension than \(\mathbf{x}_{n}\), whereas it is of a higher dimension in an over-complete architecture. Image features, as shown in Figure 2 are taken from the output of a Faster-R-CNN model. Each feature \(\mathbf{x}_{i}\) is \(\in\mathbb{R}^{N\times D}\) where the dimension of the features \(D=2048\) and \(N\), which is set to 50 is the number of selected features, i.e., regional proposals per image. Because of their source (output of Faster-R-CNN), positional encodings are not used. This is due to the fact that the features are a bag of detections from regional proposals which do not preserve the spatial and geometric location information of each regional proposal. There is thus little to no benefit in using positional encodings in the encoder. Encoder section: Similar to [4], the stack of encoders is arranged such that the output of one encoder is used as the input of the other next decoder, but different from them, as in [25], instead of only using the output of the final decoder as the final image vector representation, we save the output of each encoder, which will be shared with the corresponding decoder. Consequently, we have the same number of encoders as decoders. The final transformation in the encoder is another linear layer, which is composed of two fully connected layers. The time complexity is used to depict the amount of computer time required to run an algorithm.