# An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
## 3 Method  
### Mlp  
A multi-layer perceptron can be described as a mathematical function that maps input values into output values to provide a new representation of the input values. Arranged in different combinations across several layers (deep), they allow for automatic feature engineering, with features learned across multiple levels of increasing abstraction making them capable of tackling complex data. Given a set of \(N\) features \(\mathbf{X}=\{x_{1},x_{2},...,x_{N}\}\), \(\mathbf{X}\in\mathbb{R}^{N\times D}\) where \(N\) is the number of features and \(D\) is the dimension of each feature, an MLP learns the parameters of a function \(f*\) that maps the input features to a target (e.g., a category) \(y\) and can be described by \(y=\phi(\sum_{N=1}^{N}\mathbf{W}_{i}\mathbf{X}_{i}+b)=\phi(\mathbf{W}^{T} \mathbf{X}+b)\) where \(\mathbf{W}\) represents the learnable weights, \(b\) the bias and \(\phi\) a non-linear activation function that is used to introduce non-linearities to the network that enables it to be able to map and learn non-linear patterns in the data. In a feed-forward topology (basically a directed acyclic Graph), an MLP consists of an input layer with nodes that correspond to the input feature dimension, one or more hidden layers, and finally an output layer. The neurons in the hidden and output layers compute a weighted summation of the input features and then applies a non-linear activation function. After predicting an output that is compared to the expected output (supervised learning), the loss is computed and the weights updated by back-propagation.  
An image is comprised of matrix of pixels. To use this with an MLP, the image will usually be resized, scaled normalized transforming them to have a mean and standard deviation 0.0 and 1.0 respectively. A 28 \(\times\) 28 image for example, will be reshaped to a feature vector of size 748 which will be fed to the MLP. In our method, we process the input features using a Faster R-CNN network and use the output as the input features to the encoder. Directly feeding input features of a color image to an MLP would significantly increase the computational complexity of the model since even a simple 64 \(\times\) 64 3-channel color image would lead to 12,288 weights to a single neuron. For this reason, we use a 2048-dimensional feature vector from a Faster R-CNN network. The output of the network is a set of 2048-dimensional feature vectors, corresponding to the features detected in each image.  
A neural network only understands numbers, typically it is fed a matrix of features. In order to deal with text, every word needs to be represented by a vector. To accomplish this, a vocabulary is defined that contains all the unique words in the input corpus, or at least a subsection of words with a frequency above some arbitrary threshold. Each of the words in the vocabulary can then be represented by a one-hot-encoded vector, and subsequently, each of the words in an input sentence can now be represented by this sparse one-hot-encoded vector. More commonly though, the words are represented by word embeddings. Word embeddings retain the structure of the text, and therefore the context in which they were used. This has the added benefit of identifying similar words and representing them in similar representations within a predefined vector space. A real-valued vector representation of the text is learned using a predefined fixed sized vocabulary learned from a large corpus. To achieve this, each word is mapped to one vector and learned creating a dense vector representation containing hundreds of dimensions that when learned like a neural network captures the meaning of words around them [40].