# A Review of Transformer-Based Approaches for Image Captioning
## 4 Transformers in Captioning
### Image Captioning: Transforming Objects into Words  
[PERSON] et al. [79] introduce the object relation transformer. This transformer employs the concept of geometric attention to incorporate information about the spatial relationships between various objects in an image. Geometric attention had earlier been used for object detection [80]. The approach of the authors involves utilizing the size ratio of the bounding boxes of different objects and the difference of the bounding box coordinates to deduce object relationship features. Their model is shown in Figure 3.  
Figure 3: Object relation transformer. Appearance and geometry features are extracted from the image based on the object detector’s regions.  
Figure 2: Architecture of the captioning transformer with stacked attention modules and multi-level supervision.  
The feature detector is a Faster R-CNN object detector [15], which is used to extract appearance and geometry features, similar to the approach of [PERSON] et al. [80]. The transformer decoder is similar to that introduced by [PERSON] et al. [36]. However, for each encoding layer, the attention scores are modified by multiplying by the geometric attention weights,  
\[\mathbf{\Omega}_{A}=\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{k}}} \tag{1}\]  
where \(\mathbf{Q}\) and \(\mathbf{K}\) are the queries and keys, respectively, and \(\mathbf{\Omega}_{A}\) is an \(N\times N\) attention weight matrix, whose elements \(\omega_{A}^{mn}\) are the appearance-based attention weights between the \(m^{th}\) and \(n^{th}\) token. Relative geometry is incorporated by multiplying the appearance-based attention weights by a learned function of their relative position and size. The geometric attention weights, \(\omega_{G}^{mn}\), are first calculated based on the geometric features of the bounding boxes (center coordinates, widths and heights). The combined attention weights are then given by  
\[\omega^{mn}=\frac{\omega_{G}^{mn}\text{exp}(\omega_{A}^{mn})}{\sum_{l=1}^{N} \omega_{G}^{ml}\text{exp}(\omega_{A}^{ml})} \tag{2}\]  
where \(\omega_{A}^{mn}\) and \(\omega_{G}^{ml}\) are the elements of the attention and geographic attention weights, respectively.  
[PERSON] et al. [81], who followed a similar approach, used a scene graph representation for the encoding, which helps capture object relationships. [PERSON] et al. [82] introduced a Graph Convolutional Network plus LSTM (GCN-LSTM) architecture that incorporates semantic and spatial object relationships into the image encoder. [PERSON] et al. [83] develop a HIerarchy Parsing (HIP) architecture that parses images into multi-level structure consisting of the global level, regional level features and instance level features based on semantic segmentation. The hierarchical structure is fed into a Tree-LSTM to generate captions. However, none of these models ([81; 82; 83]) make use of transformers.  
[PERSON] et al. [79] report better performance than [PERSON] et al. [81] and [PERSON] et al. [82] on the CIDEr-D metric. However, the [PERSON] et al. [79] model uses a transformer whereas [81] and [82] do not. The use of a transformer is a significant factor contributing to the better performance. It is worth noting that [83] outperforms [79] on the CIDEr metric, which highlights the effectiveness of their hierarchical parsing approach.