# An Attentive Fourier-Augmented Image-Captioning Transformer
## 3 Material and Methods
### Method  
In our proposed model, AFCT, the input features at the decoder are 2048-dimensional image feature vectors from a faster R-CNN model trained on the visual genome dataset. These region proposal features undergo a linear transformation that projects them into a lower dimensional encoder representational space, reducing their dimension from 2048 to 512. From the 512-dimensional image features, a set of Fourier transform features are extracted. These features are concatenated to the original image regional proposal features, forming a richer representation of the input image features over which a multi-head self-attention network is applied to generate relevance scores. Each subsequent encoder or decoder uses the output of the previous layer as its input. No positional embeddings are used for the encoder but they are retained for the decoder. In the encoder section, [PERSON] et al. [9] described the task as a query (Q), key (K), and value (V) problem in which for every input token, a Q, K, and V vector is generated by applying three matrices that are learned during the training process. For each item in the input sequence, an attention or relevance score is calculated by doing a dot product of the query vector with the key vector. The output will be the product of the softmax of the relevance scores (attention vectors) and the weighted average of the values. These attention vectors describe how important each part of the input sequence is to the other parts and to itself. The output of this sub-module is then passed through a position-wise feed-forward network, producing a feature vector representation of the inputs that is then sent to the decoder. The decoder receives this encoded vector through an encoder-decoder cross-attention sub-module. The caption embeddings, concatenated to the positional embeddings, are fed to the auto-regressive decoder module first via a masked multi-head self-attention sub-module. The output of the section is then sent to the encoder-decoder sub-module from which the relationships between the image representations and captions are determined, matched, and mapped. Then, through a position-wise feed-forward network, the output of the layers are sent to a feed-forward layer, onwards to a classification linear layer, and finally to a softmax layer in which the probability distribution over the input vocabulary is the output. A normalization layer is applied after every sub-module and residual connections between the layers reduce the amount of information lost as it propagates through the layers. This is formally described below. Given an input sequence, \(x_{n}\), where \(n\in[0,N-1]\), the features first undergo a linear transformation into:  
\[X_{i}=\phi(x_{n}W)+b, \tag{1}\]  
where \(\phi,W\), and \(b\) correspond to the ReLu activation function, weight, and bias of a fully connected layer. Fourier transforms are then used to extract more of the less obvious fine-grained salient information. A discrete Fourier transform (DFT) can be used when you have signals only known at specific \(N\) instants, as is our case. Therefore, for the input sequence, the DFT can be computed by:  
\[X_{k}=\sum_{n=0}^{N-1}x_{n}e^{-\frac{2\pi i}{N}nk},\quad 0\leq k\leq N-1, \tag{2}\]  
where \(X_{k}\) is the new representation generated by the Fourier transform from the input sequence tokens \(x_{n}\). The DFT in Equation (2) can be computed using a fast Fourier transform (FFT), which is an algorithm for computing the DFT of a sequence or its inverse. Inspired by the techniques proposed in [14], we performed a 1D Fourier transform on the input image sequence in the hidden dimension and a 1D Fourier transform in the sequence dimension, and then passed the output through two linear transforms as shown below and further demonstrated in Figure 3:\[ \begin{split} X_{fft}=Real(\mathcal{F}_{seq}(\mathcal{F}_{hidden}(x)))\\ X^{{}^{\prime}}_{fft}=\phi((\phi(X_{fft}W_{f_{1}})+b_{1})W_{f_{2}})+b _{2},\end{split}