# A Review of Transformer-Based Approaches for Image Captioning
## 4 Transformers in Captioning
### Image Transformer  
[PERSON] et al. [90] proposed an image transformer, whose core idea is to increase the width of the original transformer layer, designed for machine translation, and make it more suitable for the structure of images. In their image transformer, each layer has several sub-transformers that capture the spatial relationships between the image regions. The encoding method makes use of a visual semantic graph as well as a spatial graph. They use a transformer layer to combine them without external relationship or attribute detectors.  
[PERSON] et al. [90] distinguish between single-stage and two-stage attention-based methods. The single-stage methods are those in which attention is only applied at the decoding step with the decoder attending to the most relevant regions. Two-stage methods use bottom-up attention and top-down attention [35]. The bottom-up uses object detection based methods to select the most relevant regions; top-down attention then attends to those detected regions. Although the two-stage methods improve on the single-stage methods, they have the limitation that the detected regions are isolated and their relationships are not modeled. This limitation is tackled by scene graph based models. However, the scene graph models use auxiliary or external models to detect and build the scene graphs. [PERSON] et al. [90] introduce a spatial graph encoding transformer layer, which considers the spatial relationships between the various detected regions in an image.  
The model considers three categories of spatial relationships, namely parent, neighbor and child relationships between the various regions of an image. These categories are based on the amount of overlap between the regions. Neighbors of a query region are those regions with no overlap or with overlap below a set threshold; a parent region contains a query region, whereas a child region is contained by the query region. The spatial relationships between region pairs are captured using graph adjacency matrices. For any two regions, \(l\) and \(m\), the graph adjacency matrices are defined as represented in Equation (4). \(\mathbf{\Omega}_{p}\), \(\mathbf{\Omega}_{n}\) and \(\mathbf{\Omega}_{c}\) are the parent, neighbor and child node adjacency matrices, respectively, and \(\epsilon\) is a given threshold.  
\[Area_{l}=\frac{Area(l\cap m)}{Area(l)}\text{ and }Area_{m}=\frac{Area(l\cap m)}{ Area(m)}\]  
\[\mathbf{\Omega}_{p}[l,m]=\begin{cases}1,\text{ if }Area_{l}\geqslant\epsilon \text{ and }Area_{l}\ >Area_{m}\\ 0,\text{ otherwise}\end{cases} \tag{4}\]  
Figure 7: Simplified X-Linear Attention Network. The encoder and decoder utilize the X-Linear Attention Block. The decoder also relies on an LSTM. Decoder outputs at each time step therefore feed into the decoding operation of the next time step.  
\[\mathbf{\Omega}_{c}[l,m]=\mathbf{\Omega}_{p}[m,l]\]  
\[\text{with}\ \sum_{i\in\{p,n,c\}}\mathbf{\Omega}_{i}[l,m]=1\]  
As shown in Figure 8, the original transformer layer is widened by adding three parallel sub-transformer layers, each being responsible for a subcategory of spatial relationships. The decoder incorporates an LSTM and its structure is also widened to correspond to the encoder; its output is obtained via a gated linear unit.  
Since they are incorporating graph information into the transformer, the model is similar to other graph-extraction techniques. However, they use a transformer, unlike other graph-extraction techniques [81; 82]. The authors point out that their model is more computationally efficient since the other scene graph extracting models fuse semantic and spatial scene graphs, and require auxiliary models to first build the scene graph.