# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 3 Method
### MLPs in Image Captioning
The computational complexity of the self-attention mechanism is \(\mathcal{O}\big{(}n^{2}\cdot d\big{)}\) whereas that of CNNs and RNNs is \(\mathcal{O}\big{(}k\cdot n\cdot d^{2}\big{)}\) and \(\mathcal{O}\big{(}n\cdot d^{2}\big{)}\), respectively, where \(n\) is the sequence length and \(d\) is the dimension of the features, the \(k\) variable in the CNN complexity represents the convolutional kernel size. The feed-forward network complexity is function of the number of layers, number of neurons per layer and the feature dimensions. While the number of neurons in the input layer corresponds to the feature dimension, it may differ for the hidden and output layers. These computations for a fully connected layer are dominated by the matrix operations and are computed as \(\mathbf{N}_{1}\times\mathbf{N}_{2}\), where \(\mathbf{N}_{1}\in\mathbb{R}^{n_{1}\times n_{2}}\) is the dimension of the input matrix and \(\mathbf{N}_{2}\in\mathbb{R}^{n_{2}\times n_{3}}\) is the dimension of the output matrix. In comparison, as shown in (4), in addition to the quadratic calculation of the relevancy scores, there are also 4 matrix multiplications for the Queries, Keys and Value matrices. Decoder section: In image captioning applications, captions are both the targets and at the same time one of the inputs. Captions are entered into the decoder in a parallel manner, but usually masked in order to ensure that the decoder can only see the previously generated tokens. In our implementation, where we encode and extract caption information using a linear transformation with ReLu activations, we found that maskingwas unnecessary. This is explained in detail in Section 5 and shown in Figure 1. Briefly, as in the standard implementation, caption embeddings are concatenated with sinusoidal positional embeddings. This ordinarily would then go to a masked self-attention layer, but we use a deep fully connected layer to encode these caption features, without masking future captions, i.e., when predicting the token at a position, the model has access to future and past tokens, i.e., a non-masked caption embedding representation with positional contextual information added. Cross-attention/image-caption translation section: During the training phase, this learns the relationship between the image feature representation and the caption embeddings representation, in an autoregressive manner, so as to produce a new feature representation that can be used to predict the next word in a caption sequence. First off, the linearly transformed caption embeddings from the previous layer are sent to the cross-attention layer. Owing to the fact that this section requires a cross comparison between image features and caption representations, and an attention mechanism where the caption embeddings are used as Queries (Q) over the image feature representations (Keys (K)-Values (V) pair), the possible linear transformations required and attention mechanisms would be more complicated than a self-attention mechanism. We argue that this is the only part of the original transformer architecture that greatly benefits from intra-attention mechanisms. As such, we resolved to use it in only this section of the model. The Query (Q) vectors from the caption embeddings, and the Key (K)-Value (V) vectors from the image feature representation are used to compute the relevance score of each image feature to other features, to produce the attention matrix as shown in (4). See Figure 2 for the visual representation of the previous statement. \[Attention\left(Q,K,V\right)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})\;V \tag{4}\]  
where \(Q=X_{i}*W_{Q}\), \(K=X_{i}*W_{K}\), \(V=X_{i}*W_{V}\). \(X_{i}\) represents the encoder input features \(x_{1}...x_{n};W_{Q},W_{K},W_{V}\) are learned projection matrices for the queries, keys and values, respectively. Multi-head attention as well as residual connections, are employed to send information from the input of one layer to the next in order to reduce information loss. This loss may occur due to the many transformations undergone by the data.