Introduction
Image captioning—the task of providing a natural language description of the content within an image—lies at the intersection of computer vision and natural language processing. As both of these research areas are highly active and have experienced many recent advances, the progress in image captioning has naturally followed suit. On the computer vision side, improved convolutional neural network and object detection architectures have contributed to improved image captioning systems. On the natural language processing side, more sophisticated sequential models, such as attention-based recurrent neural networks, have similarly resulted in more accurate image caption generation.
Inspired by neural machine translation, most conventional image captioning systems utilize an encoder-decoder framework, in which an input image is encoded into an intermediate representation of the information contained within the image, and subsequently decoded into a descriptive text sequence. This encoding can consist of a single feature vector output of a CNN (as in BIBREF0 ), or multiple visual features obtained from different regions within the image. In the latter case, the regions can be uniformly sampled (e.g., BIBREF1 ), or guided by an object detector (e.g., BIBREF2 ) which has been shown to yield improved performance.
While these detection based encoders represent the state-of-the art, at present they do not utilize information about the spatial relationships between the detected objects such as relative position and size. This information can often be critical to understanding the content within an image, however, and is used by humans when reasoning about the physical world. Relative position, for example, can aid in distinguishing “a girl riding a horse” from “a girl standing beside a horse”. Similarly, relative size can help differentiate between “a woman playing the guitar” and “a woman playing the ukelele”. Incorporating spatial relationships has been shown to improve the performance of object detection itself, as demonstrated in BIBREF3 . Furthermore, in machine translation encoders, positional relationships are often encoded, in particular in the case of the Transformer BIBREF4 , an attention-based encoder architecture. The use of relative positions and sizes of detected objects, then, should be of benefit to image captioning visual encoders as well, as evidenced in Figure FIGREF1 .
In this work, we propose and demonstrate the use of object spatial relationship modeling for image captioning, specifically within the Transformer encoder-decoder architecture. This is achieved by incorporating the object relation module of BIBREF3 within the Transformer encoder. The contributions of this paper are as follows:
Related Work
Many early neural models for image captioning BIBREF5 , BIBREF6 , BIBREF7 , BIBREF0 encoded visual information using a single feature vector representing the image as a whole and hence did not utilize information about objects and their spatial relationships. Karpathy and Fei-Fei in BIBREF8 , as a notable exception to this global representation approach, extracted features from multiple image regions based on an R-CNN object detector BIBREF9 and generated separate captions for the regions. As a separate caption was generated for each region, however, the spatial relationship between the detected objects was not modeled. This is also true of their follow-on dense captioning work BIBREF10 , which presented an end-to-end approach for obtaining captions relating to different regions within an image. Fang et al. in BIBREF11 generated image descriptions by first detecting words associated with different regions within the image. The spatial association was made by applying a fully convolutional neural network to the image and generating spatial response maps for the target words. Here again, the authors do not explicitly model any relationship between the spatial regions.
A family of attention based approaches BIBREF1 , BIBREF12 , BIBREF13 to image captioning have also been proposed that seek to ground the words in the predicted caption to regions in the image. As the visual attention is often derived from higher convolutional layers from a CNN, the spatial localization is limited and often not semantically meaningful. Most similar to our work, Anderson et al. in BIBREF2 addressed this limitation of typical attention models by combining a “bottom-up” attention model with a “top-down” LSTM. The bottom-up attention acts on mean-pooled convolutional features obtained from the proposed regions of interest of a Faster R-CNN object detector BIBREF14 . The top-down LSTM is a two-layer LSTM in which the first layer acts as a visual attention model that attends to the relevant detections for the current token and a language LSTM that generates the next token. The authors demonstrated state-of-the-art performance for both visual question answering and image captioning using this approach, indicating the benefits of combining features derived from object detection with visual attention. Again, spatial information is not utilized, which we propose in this work via geometric attention, as first introduced by Hu et al. for object detection in BIBREF3 . The authors used bounding box coordinates and sizes to infer the importance of the relationship of pairs of objects, the assumption being that if two bounding boxes are closer and more similar in size to each other, then their relationship is stronger.
Recent developments in NLP, namely the Transformer architecture BIBREF4 have led to significant performance improvements for various tasks such as translation BIBREF4 , text generation BIBREF15 , and language understanding BIBREF16 . In BIBREF17 , the Transformer was applied to the task of image captioning. The authors explored extracting a single global image feature from the image as well as uniformly sampling features by dividing the image into 8x8 partitions. In the latter case, the feature vectors were fed in a sequence to the Transformer encoder. In this paper we propose to improve upon this uniform sampling by adopting the bottom-up approach of BIBREF2 . The Transformer architecture is particularly well suited as a bottom-up visual encoder for captioning since it does not have a notion of order for its inputs, unlike an RNN. It can, however, successfully model sequential data with the use of positional encoding, which we apply to the decoded tokens in the caption text. Rather than encode an order to objects, our Object Relation Transformer seeks to encode how two objects are spatially related to each other and weight them accordingly.
Proposed Approach
Figure FIGREF5 shows an overview of the proposed image caption algorithm. We first use an object detector to extract appearance and geometry features from all the detected objects in the image. Thereafter we use the Object Relation Transformer to generate the caption text. Section SECREF7 describes how we use the Transformer architecture BIBREF4 in general for image captioning. Section SECREF13 explains our novel addition of box relational encoding to the encoder layer of the Transformer.
Object Detection
Following BIBREF2 , we use Faster R-CNN BIBREF14 with ResNet-101 BIBREF18 as the base CNN for object detection and feature extraction. Using intermediate feature maps from the ResNet-101 as inputs, a Region Proposal Network (RPN) generates bounding boxes for object proposals. Using non-maximum suppression, overlapping bounding boxes with an intersection-over-union (IoU) exceeding a threshold of 0.7 are discarded. A region-of-interest (RoI) pooling layer is then used to convert all remaining bounding boxes to the same spatial size (e.g. INLINEFORM0 2048). Additional CNN layers are applied to predict class labels and bounding box refinements for each box proposal. We further discard all bounding boxes where the class prediction probability is below a threshold of 0.2. Finally, we apply mean-pooling over the spatial dimension to generate a 2048-dimensional feature vector for each object bounding box. These feature vectors are then used as inputs to the Transformer model.
Standard Transformer Model
This section describes how apply the Transformer architecture of BIBREF4 to the task of image captioning. The Transformer model consists of an encoder and a decoder, both of which are composed of a stack of layers (in our case 6). Our architecture uses the feature vectors from the object detector as inputs and generates a sequence of words (i.e., the image caption) as outputs.
Every image feature vector is first processed through an input embedding layer, which consists of a fully-connected layer to reduce the dimension from 2048 to INLINEFORM0 followed by a ReLU and a dropout layer. The embedded feature vectors are then used as input tokens to the first encoder layer of the Transformer model. We denote INLINEFORM1 as the n-th token of a set of INLINEFORM2 tokens. For encoder layers 2 to 6, we use the output tokens of the previous encoder layer as the input to the current layer.
Each encoder layer consists of a multi-head self-attention layer followed by a small feed-forward neural network. The self-attention layer itself consists of 8 identical heads. Each attention head first calculates a query INLINEFORM0 , key INLINEFORM1 and value INLINEFORM2 for each of the INLINEFORM3 tokens given by DISPLAYFORM0
where INLINEFORM0 contains all the input vectors INLINEFORM1 stacked into a matrix and INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 are learned projection matrices.
The attention weights for the appearance features are then computed according to DISPLAYFORM0
where INLINEFORM0 is an INLINEFORM1 attention weight matrix, whose elements INLINEFORM2 are the attention weights between the m-th and n-th token. Following the implementation of BIBREF4 , we choose a constant scaling factor of INLINEFORM3 , which is the dimension of the key, query, and value vectors. The output of the head is then calculated as DISPLAYFORM0
Equations EQREF8 to EQREF10 are calculated for every head independently. The output of all 8 heads are then concatenated to one output vector, INLINEFORM0 , and multiplied with a learned projection matrix INLINEFORM1 , i.e., DISPLAYFORM0
The next component of the encoder layer is the point-wise feed-forward network (FFN), which is applied to each output of the attention layer. DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 , INLINEFORM3 are the weights and biases of two fully connected layers. In addition, skip-connections and layer-norm are applied to the outputs of the self-attention and the feed-forward layer.
The decoder then uses the generated tokens from the last encoder layer as input to generate the caption text. Since the dimensions of the output tokens of the Transformer encoder are identical to the tokens used in the original Transformer implementation, we make no modifications on the decoder side. We refer the reader to the original publication BIBREF4 for a detailed explanation of the decoder.
Object Relation Transformer
In our proposed model, we incorporate relative geometry by modifying the attention weight matrix INLINEFORM0 in Equation EQREF9 . We multiply the appearance based attention weights INLINEFORM1 of two objects INLINEFORM2 and INLINEFORM3 , by a learned function of their relative position and size. We use the same function that was first introduced in BIBREF3 to improve the classification and non-maximum suppression stages of a Faster R-CNN object detector.
First we calculate a displacement vector INLINEFORM0 for two bounding boxes INLINEFORM1 and INLINEFORM2 from their geometry features INLINEFORM3 and INLINEFORM4 (center coordinates, width, and heights) as DISPLAYFORM0
The geometric attention weights are then calculated as follows DISPLAYFORM0
where Emb(.) calculates a high-dimensional embedding following the functions INLINEFORM0 described in BIBREF4 , where sinusoid functions computed for each value of INLINEFORM1 . In addition, we multiply the embedding with the learned vector INLINEFORM2 to project down to a scalar and apply the ReLU non-linearity. The geometric attention weights INLINEFORM3 are then incorporated into the attention mechanism according to DISPLAYFORM0
where INLINEFORM0 are the appearance based attention weights from Equation EQREF9 and INLINEFORM1 are the new combined attention weights.
The output of the head can be calculated as follows DISPLAYFORM0
where INLINEFORM0 is the INLINEFORM1 matrix, whose elements are given by INLINEFORM2 .
The Bounding Box Relational Encoding diagram in Figure FIGREF5 shows the multi-head self-attention layer of the Object Relation Transformer. Equations EQREF14 to EQREF17 are represented with the relation boxes.
Implementation Details
Our algorithm was developed in PyTorch using the image captioning implementation in BIBREF19 as our basis. We ran our experiments on NVIDIA Tesla V100 GPUs on AWS. Our best performing model is pre-trained for 30 epochs with a softmax cross-entropy loss, using the ADAM optimizer with learning rate defined as in the original Transformer paper with 20000 warmup steps, and a batch size of 10. We trained for an additional 30 epochs using self-critical reinforcement learning BIBREF20 optimizing for CIDEr-D score, and did early-stopping for best performance on the validation set (of 5000 images). On a single GPU the training with cross-entropy loss and the self-critical training take about 1 day and 3.5 days, respectively.
The models compared in sections SECREF22 - SECREF29 are evaluated after training for 30 epochs, with ADAM optimization with the above learning rate schedule, and with batch size 15.
Dataset and Metrics
We trained and evaluated our algorithm on the Microsoft COCO (MS-COCO) 2014 Captions dataset BIBREF21 . We report results on the Karpathy validation and test splits BIBREF8 , which are commonly used in other image captioning publications. The dataset contains 113K training images with 5 human annotated captions for each image. The Karpathy test and validation sets contain 5K images each. We evaluate our models using the CIDEr-D BIBREF22 , SPICE BIBREF23 , BLEU BIBREF24 , METEOR BIBREF25 , and ROUGE-L BIBREF26 metrics. While it has been shown experimentally that BLEU and ROUGE have lower correlation with human judgments than the other metrics BIBREF23 , BIBREF22 , the common practice in the image captioning literature is to report all the mentioned metrics.
Comparative Analysis
We compare our proposed algorithm against the best results from a single model of the self-critical sequence training (Att2all) BIBREF20 and the Bottom-up Top-down (Up-Down) BIBREF2 algorithm. Table TABREF21 shows the metrics for the test split as reported by the authors. Following the implementation of BIBREF2 , we fine-tune our model using the self-critical training optimized for CIDEr-D score BIBREF20 and apply beam search with beam size 5, achieving a 6.8% relative improvement over the previous state-of-the-art.
Positional Encoding
Our proposed geometric attention can be seen as a replacement for the positional encoding of the original Transformer network. While objects do not have an inherent notion of order, there do exist some simpler analogues to positional encoding, such as ordering by object size, or left-to-right or top-to-bottom based on bounding box coordinates. We provide a comparison between our geometric attention and these object orderings in Table TABREF23 . For box size, we simply calculate the area of each bounding box and order from largest to smallest. For left-to-right we order bounding boxes according to the x-coordinate of their centroids. Analogous ordering is performed for top-to-bottom using the centroid y-coordinate. Based on the CIDEr-D scores shown, adding such an artificial ordering to the detected objects decreases the performance. We observed similar decreases in performance across all other metrics (SPICE, BLEU, METEOR and ROUGE-L).
Ablation Study
Table TABREF25 shows the results for our ablation study. We show the Bottom-Up and Top-Down algorithm BIBREF2 as our baseline algorithm. The second row replaces the LSTM with a Transformer network. The third row includes the proposed geometric attention. The last row includes beam search with beam size 2. The contribution of the Object Relation Transformer is small for METEOR, but significant for CIDEr-D and the BLEU metrics. Overall we can see the most improvements on the CIDEr-D and BLEU-4 score.
Geometric Improvement
In order to demonstrate the advantages of the geometric relative attention layer, we performed a more detailed comparison of the Standard Transformer against the Object Relation Transformer. For each of the metrics, we performed a two-tailed t-test with paired samples in order to determine whether the difference caused by adding the geometric relative attention layer was statistically significant. The metrics were computed for each individual image in the test set for each of the Transformer models, so that we are able to run the paired tests. In addition to the standard evaluation metrics, we also report metrics obtained from SPICE by splitting up the tuples of the scene graphs according to different semantic subcategories. For each subcategory, we are able to compute precision, recall, and F-scores. The reported measures are the F-scores computed by taking only the tuples in each subcategory. More specifically, we report SPICE scores for: Object, Relation, Attribute, Color, Count, and Size BIBREF23 . Note that for a given image, not all SPICE subcategory scores might be available. For example, if the reference captions for a given image have no mention of color, then the SPICE Color score is not defined and therefore we omit that image from that particular analysis. In spite of this, each subcategory analyzed had at least 1000 samples. For this experiment, we did not use self-critical training for either Transformer and they were both run with a beam size of 2.
The metrics computed over the 5000 images of the test set are shown in Tables TABREF27 and TABREF28 . We first note that for all of the metrics, the Object Relation Transformer presents higher scores than the Standard Transformer. The score difference was statistically significant (using a significance level INLINEFORM0 ) for CIDEr-D, BLEU-1, ROUGE-L (Table TABREF27 ), Relation, and Count (Table TABREF28 ). The significant improvements in CIDEr-D and Relation are in line with our expectation that adding the geometric attention layer would help the model in determining the correct relationships between objects. In addition, it is interesting to see a significant improvement in the Count subcategory of SPICE, from 11.30 to 17.51. Image captioning methods in general show a large deficit in Count scores when compared with humans BIBREF23 , while we're able to show a significant improvement by adding explicit positional information. Some example images and captions illustrating these improvements are presented in Section SECREF29 .
Qualitative Analysis
To illustrate the advantages of the Object Relation Transformer relative to the Standard Transformer, we present example images with the corresponding captions generated by each model. The captions presented were generated using the following setup: both the Object Relation Transformer and the Standard Transformer were trained without self-critical training and both were run with a beam size of 2 on the 5000 images of the test set. We chose examples for which there were was a marked improvement in the score of the Object Relation Transformer relative to the Standard Transformer. This was done for the Relation and Count subcategories of SPICE scores. The example images and captions are presented in Tables TABREF30 and TABREF30 . The images in Table TABREF30 illustrate an improvement in determining when a relationship between objects should be expressed, as well as in determining what that relationship should be. An example of correctly determining that a relationship should exist is shown in the third image of Table TABREF30 , where the two chairs are actually related to the umbrella, by being underneath it. Additionally, an example where the Object Relation Transformer correctly infers the type of relationship between objects is shown in the first image of Table TABREF30 , where the man in fact is not on the motorcycle, but is working on it. The examples in Table TABREF30 specifically illustrate the Object Relation Transformer's marked ability to better count objects.
Conclusion
We present the Object Relation Transformer, a modification of the conventional Transformer specifically adapted to the task of image captioning. The proposed Transformer encodes 2D position and size relationships between detected objects in images, building upon the bottom-up and top-down image captioning approach. Our results on the MS-COCO dataset demonstrate that the Transformer does indeed benefit from incorporating spatial relationship information, most evidently when comparing the relevant sub-metrics of the SPICE captioning metric. We also present qualitative examples of how incorporating this information can yield captioning results demonstrating better spatial awareness.
At present, our model only takes into account geometric information in the encoder phase. As a next step, we intend to incorporate geometric attention in our decoder cross-attention layers between objects and words. We aim to do this by explicitly associating decoded words with object bounding boxes. This should lead to additional performance gains as well as improved interpretability of the model.

Question: What are the common captioning metrics?
Answer: the CIDEr-D BIBREF22 , SPICE BIBREF23 , BLEU BIBREF24 , METEOR BIBREF25 , and ROUGE-L BIBREF26 metrics