# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 2 Background
### Masking in Transformer Decoders  
In LSTM and GRU sequence generation, their inherent sequential nature ensures that the model only has access to past tokens and the previous hidden state. In transformer-based language models, the text embeddings are input in parallel during the training phase. The decoder is trained in an autoregressive manner, where the previously generated words are used as part of the input to predict the next word. In current implementations of transformer-based text generation tasks, as shown in Figure 1, the self-attention layer over the captions is masked so that the network only has access to previous tokens when learning to predict a future token. According to [4], masking is required because of the parallel nature of inputs to the transformer. Due to this parallelism, the auto-regressive model has access to the full ground truth caption sequence which it may exploit and learn to output the next word in the input sequence, instead of predicting based on the already generated tokens. The authors argue that, at each time step, it will simply look at the input sequence and output the next word, instead of producing a probability distribution from which the next token can be sampled. So in order to achieve this, as in Figure 1, a mask is added to the input sequence representation. The mask is basically a matrix with negative infinity \((-\infty)\) in the position of the elements that you don't want attended to, in order to ensure that they have no influence on the output representation. When the softmax operation is then applied to the resultant matrix, the high scores will be boosted and the negative infinity scores zeroed out. We use fully connected linear layers rather than self-attention to encode both the image feature representation and caption embeddings, without masking the caption input during training as explained in Sections 3.2 and 5 and more formally below.  
During the training phase, assume we have an input caption sequence \(\mathbf{X}=\{x_{1},x_{2},x_{3},...,x_{n}\}\), where \(x_{1}:n\) are the individual sequence tokens of a sequence of length \(n\). In a sequential model like an RNN, \(x_{1}\) is given as an input and the model trained to produce \(x_{2}\). In the next iteration \(x_{1},x_{2}\) are given as inputs to train the model to generate \(x_{3}\), and so on. As can be seen, the sequential nature prevents the model from looking ahead to see and condition on future tokens. On the other hand, the common transformer architecture [4] uses the full sequence \(\{x_{1},x_{2},x_{3},...,x_{n}\}\) as input, i.e., passed in parallel. To prevent future tokens from being processed, a look-ahead mask, e.g., \(\{\{-\infty_{1},-\infty_{2},-\infty_{3},...,-\infty_{n}\}\}\) is added to the input sequence and a softmax operation performed as in (1).  
\[\mathbf{X}=\{x_{1},x_{2},x_{3},...,x_{n}\}+\{0,-\infty_{2},- \infty_{3},...,-\infty_{n}\} \tag{1}\] \[\mathbf{X}=Softmax\{x_{1},-\infty_{2},-\infty_{3},...,-\infty_{n}\}\] \[\mathbf{X}=\{x_{1},0,0,...,0\}.\]  
For subsequent operations, the mask is modified so as to mask the required values, i.e., \(\{0,-\infty_{2},-\infty_{3},...,-\infty_{n}\},\{0,0,-\infty_{2},...,-\infty_{ n}\},\{0,0,0,...,-\infty_{n}\}\) and so on, as required. In this paper we show that when feed-forward layers are used in place of self-attention layers, masking is not required, thereby reducing on the complexity of the model. The idea for this partly arose from the fact that residual connections in a transformer model are used in such a way that the unmasked input information is added to the masked input information in the \"Add $ Norm\" section of the decoder transformer. The model should still be able to exploit this to learn something about future tokens, thus rendering the benefits of the look-ahead masking procedure not as significant as it is currently assumed.