Title: LEARNING TO GENERATE GROUNDED VISUAL CAPTIONS WITHOUT LOCALIZATION SUPERVISION

Abstract: When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.

1. INTRODUCTION:

Vision and language tasks, such as visual captioning, combine linguistic descriptions with data from real-world scenes. Deep learning models for such tasks have achieved great success, driven in part by the development of attention mechanisms that focus on various objects in the scene while generating captions. The resulting models, however, are known to have poor grounding performance (Liu et al., 2017) , leading to undesirable behaviors such as object hallucinations (Rohrbach et al., 2018) , despite having high captioning accuracy. That is, they often do not correctly associate generated words with the appropriate image regions (e.g., objects) in the scene, resulting in models that lack interpretability.

Several existing approaches have tried to improve the grounding of captioning models. One class of methods generate sentence templates with slot locations explicitly tied to specific image regions. These slots are then filled in by visual concepts identified by off-the-shelf object detectors (Lu et al., 2018) . Other methods have developed specific grounding or attention modules that aim to attend to the correct region(s) for generating visually groundable word. Such methods, however, rely on explicit supervision for optimizing the grounding or attention modules (Liu et al., 2017; Zhou et al., 2019) and require bounding box annotations for each visually groundable word.

In this work, we propose a novel cyclical training regimen that is able to significantly improve grounding performance without any grounding annotations. The key insight of our work is that current models use attention mechanisms conditioned on the hidden features of recurrent modules such as LSTMs, which leads to effective models with high accuracy but entangle grounding and decoding. Since LSTMs are effective at propagating information across the decoding process, the network does not necessarily need to associate particular decoded words with their corresponding image region(s). However, for a captioning model to be visually grounded, the model has to predict attentional weights without knowing the word to localize.

Based on this insight, we develop a cyclical training regimen to force the network to ground individual decoded words: decoding -→ localization -→ reconstruction. Specifically, the model of the decoding stage can be any state-of-the-art captioning model; in this work, we follow GVD (Zhou et al., A woman with a black hat is holding a cup .

A woman with a black hat is holding a cup .

Localizing … Matched:

Captioning …

Localization:

Figure 1 : Visual captioning models are often not visually-grounded. As human, we perform localization to check whether the generated caption is visually-grounded. If the localized image region is incorrect, we update the model. However, without the ground-truth grounding annotation, how does the model know the localized region is incorrect? To overcome this issue, we propose to perform localization and reconstruction to regularize the captioning model to be visually-grounded without relying on the grounding annotations.

2019) to extend the widely used Up-Down model (Anderson et al., 2018) . At the localization stage, each word generated by the first decoding stage is localized through a localizer, and the resulting grounded image region(s) are then used to reconstruct the ground-truth caption in the final stage. Both decoding and reconstruction stages are trained using a standard cross-entropy loss. Key to our method, both stages share the same decoder, thereby causing the localization stage to guide the decoder to improve its attention mechanism. Our method is simple and only adds a fully-connected layer to perform localization. During inference, we only use the (shared) decoder, thus we do not add any computational cost.

We benchmark our proposed method on the challenging Flickr30k Entities image captioning dataset (Plummer et al., 2015) and the ActivityNet-Entities video captioning dataset (Zhou et al., 2019) on both captioning and grounding performances. In addition to the existing grounding metric that calculate the grounding accuracy for each object class (Zhou et al., 2019) , we further include a grounding metric that compute grounding accuracy for each generated sentence. This new metric on each sentence removes the stringency of the original evaluation metric (as we discuss in Sec. 4) and provides an alternative way of measuring the grounding performance.

Despite the simplicity of our proposed method, we are able to significantly surpass prior unsupervised models quantitatively and qualitatively on both datasets. We achieve around 18% relative improvements in terms of bridging the gap between the unsupervised baseline and supervised methods on Flickr30k Entites and around 34% on ActivityNet-Entities. We further find that our method can even outperform the supervised method on infrequent words, owing to its self-supervised nature.

Contributions summary.:

We propose object re-localization as a form of self-supervision for grounded visual captioning and present a cyclical training regimen that re-generates sentences after re-localizing the objects conditioned on each word, implicitly imposing grounding consistency. We evaluate our proposed approach on both image and video captioning tasks. We show that the proposed training regime can boost grounding accuracy over a state-of-the-art baseline, enabling grounded models to be trained without bounding box annotations, while retaining high captioning quality across two datasets and various experimental settings. Our code will be publicly released and can be found in supplemental.

2. RELATED WORK:

Visual captioning. Neural models for visual captioning have received significant attention recently (Anderson et al., 2018; Ma et al., 2018; Lu et al., 2018; Donahue et al., 2015; Venugopalan et al., 2015; Rohrbach et al., 2017b; Venugopalan et al., 2017; Rohrbach et al., 2017a; Shetty et al., 2017; Park et al., 2019) . Most current state-of-the-art models contain attention mechanisms, allowing the process to focus on subsets of the image when generating the next word. These attention mechanisms can be defined over spatial locations (Vinyals et al., 2015) , semantic metadata (Li et al., 2018; Yao et al., 2017; You et al., 2016; Zhou et al., 2017) or a predefined set of regions extracted via a region proposal network (Ma et al., 2018; Zanfir et al., 2016; Anderson et al., 2018; Lu et al., 2018; Das et al., 2013; Kulkarni et al., 2013) . In the latter case, off-the-shelf object detectors are first used to extract object proposals (Ren et al., 2015; He et al., 2017) and the captioning model then learns to dynamically attend over them when generating the caption.

Visual grounding. Although attention mechanisms are generally shown to improve captioning quality and metrics, it has also been shown that they don't really focus on the same regions as a human would (Das et al., 2017) . This make models less trustworthy and interpretable, and therefore creating grounded image captioning models, i.e., models that accurately link generated words or phrases to specific regions of the image, has recently been an active research area. A number of approaches have been proposed, e.g., for grounding phrases or objects from image descriptions (Rohrbach et al., 2016; Hu et al., 2016; Xiao et al., 2017; Deng et al., 2018; Zhou et al., 2019; Zhang et al., 2019) , grounding visual explanations (Hendricks et al., 2018) , visual co-reference resolution for actors in video (Rohrbach et al., 2017a) , or improving grounding via human supervision (Selvaraju et al., 2019) .

Recently, Zhou et al. (2019) presented a model with self-attention based context encoding and direct grounding supervision that achieves state-of-the-art results in both the image and video tasks. They exploit ground-truth bounding box annotations to significantly improve the visual grounding accuracy.

In contrast, we focus on reinforcing the visual grounding capability of the existing captioning model via a cyclical training regimen without using bounding box annotations and present a method that can increase grounding accuracy while maintaining comparable captioning performance with state of the arts.

Cyclical training. Cycle consistency (Wang et al., 2013; Zhu et al., 2017; He et al., 2016; Chen & Lawrence Zitnick, 2015) has been used recently in a wide range of domains, including machine translation (He et al., 2016) , unpaired image-to-image translation (Zhu et al., 2017) , visual question answering (Shah et al., 2019) , question answering (Tang et al., 2018) , image captioning (Chen & Lawrence Zitnick, 2015) , video captioning (Wang et al., 2018; Duan et al., 2018) , captioning and drawing (Huang et al., 2018) as well as domain adaptation (Hosseini-Asl et al., 2019) . While the cyclical training regime has been explored vastly in both vision and language domains, it has not yet been used for enforcing the visual grounding capability of a captioning model.

3. METHOD:

Notation. For a visual captioning task, we denote the input image as I (or input video as V ) and the target sentence as S. Each image (or video) is represented by spatial feature map(s) extracted by a ResNet-101 model and a bag of regions obtained from Faster-RCNN (Ren et al., 2015) as

R = [r 1 , r 2 , ..., r N ] ∈ R d×N .

The target sentence is represented as a sequence of one-hot vectors y * t ∈ R s , where T is the sentence length, t ∈ 1, 2, ..., T , and s is the dictionary size.

3.1. BASELINE:

We reimplemented the model used in GVD (Zhou et al., 2019) without self-attention for region feature encoding (Ma et al., 2018; Vaswani et al., 2017) 

EQUATION

where [; ] denotes concatenation, and W e are learned parameters. We omit the Attention LSTM input hidden and cell states to avoid notational clutter in the exposition.

The Language LSTM uses the hidden state h A t from the Attention LSTM to dynamically attend on the bag of regions R for obtaining visual representations of the image rt to generate a word y t . where W aa and W a are learned parameters. The conditional probability distribution over possible output words y t is computed as:

EQUATION

EQUATION

where y 1:t-1 is a sequence of outputs (y 1 , ..., y t-1 ). We refer the Language LSTM and the output logit layer as the complete language decoder.

3.2. OVERVIEW:

Our goal is to enforce the generated caption to be visually grounded, i.e., attended image regions correspond specifically to individual words being generated, without ground-truth grounding supervision. Towards this end, we propose a novel cyclical training regimen that is comprised of decoding, localization, and reconstruction stages, as illustrated in Figure 2 .

The intuition of our method is that the baseline network is not forced to generate a correct correspondence between the attended objects and generated words, since the LSTMs can learn priors in the data instead of looking at the image or propagate information forward which can subsequently be used to generate corresponding words in future time steps. The proposed cyclical training regimen, in contrast, aims at enforcing visual grounding to the model by requiring the language decoder (Eq. 3) to rely on the localized image regions rl t to reconstruct the ground-truth sentence, where the localization is conditioned only on the generated word from the decoding stage. Our cyclical method can therefore be done without using any annotations of the grounding itself. Specifically, let y d t = D d (r t ; θ d ) be the initial language decoder with parameters θ d (Eq. 3), trained to sequentially generate words y d t . Let G(y d t ; θ g ) define a localizer unit with parameters θ g , that learns to map (ground) each generated word to region(s) in the image, i.e., rl t = G(y d t , R; θ g ). Finally, let y l t = D l (r l t ; θ l ) be a second decoder, that is required to reconstruct the ground-truth caption using the localized region(s), instead of the attention computed by the decoder itself. We define the cycle:

EQUATION

where D d and D l share parameters. Although parameters are shared, the inputs for the two language decoders differ, leading to unique LSTM hidden state values during a run. Note that the Attention LSTMs and logit layers in the two stages also share parameters, though they are omitted for clarity.

Through cyclical joint training, both D d and D l are required to generate the same ground-truth sentence. They are both optimized to maximize the likelihood of the correct caption:

EQUATION

During training, the localizer regularizes the region attention of the reconstructor and the effect is further propagated to the baseline network in the decoding stage, since the parameters of Attention LSTM and Language LSTM are shared for both decoding and reconstruction stages. Note that the gradient from reconstruction loss will not backprop to the decoder D d in the decoding stage since the generated words used as input to the localizer are leafs in the computational graph. The network is implicitly regularized to update its attention mechanism to match with the localized image regions rt → rl t . In Sec. 4.3, we demonstrate that the localized image regions rl t indeed have higher attention accuracy than rt when using ground-truth words as inputs for the localizer.

3.3. CYCLICAL TRAINING:

We now describe each stage of our cyclical model in detail, as illustrated in Figure 3 .

Decoding. We first use the baseline model presented in Sec. 3.1 to generate a sequence of words

y = [y d 1 , y d 2 , ..., y d T ],

where T is the ground-truth sentence length. Localization. Following the decoding process, a localizer G is then learned to localize the image regions from each generated word y t .

EQUATION

where e t is the embedding for the word generated during decoding stage at step t, r n is the image representation of a region proposal, and W e and W l are the learned parameters. Based on the localized weights β t , the localized region representation can be obtained by rl t = Rβ. Reconstruction. Finally, the shared language decoder D l relies on the localized region representation rl t to generate the next word. The probability over possible output words is:

EQUATION

Given the target ground truth caption y * 1:T and our proposed captioning model parameterized with θ, we minimize the following cross-entropy losses:

L CE (θ) = -λ 1 T t=1 log(p θ (y * t |y * 1:t-1 ))1 (y * t =y d t ) decoding loss -λ 2 T t=1 log(p θ (y * t |y * 1:t-1 ))1 (y * t =y l t ) reconstruction loss (8)

where λ 1 and λ 2 are weighting coefficient selected on the validation split.

4. EXPERIMENTS:

Datasets. We use the Flickr30k Entities image dataset (Plummer et al., 2015) and the ActivityNet-Entities video dataset (Zhou et al., 2019) for evaluating our proposed approach. Flickr30k Entities contains 275k annotated bounding boxes from 31k images associated with natural language phrases. Each image is annotated with 5 crowdsourced captions. ActivityNet-Entities contains 15k videos with 158k spatially annotated bounding boxes from 52k video segments.

Captioning evaluation metrics. We measure captioning performance using four language metrics, including BLEU (Papineni et al., 2002) , METEOR (Banerjee & Lavie, 2005) , CIDEr (Vedantam et al., 2015) , and SPICE (Anderson et al., 2016) .

Grounding evaluation metrics. Following the grounding evaluation from GVD (Zhou et al., 2019) , we measure the attention accuracy on generated sentences, denoted by F1 all and F1 loc . In F1 all , a region prediction is considered correct if the object word1 is correctly predicted and also correctly localized. We also compute F1 loc , which only considers correctly-predicted object words. Please see illustration of the grounding metrics in Appendix A.1.

In the original formulation, the precision and recall for the two F1 metrics are computed for each object class, and it is set to zero if an object class has never been predicted. The scores are computed for each object class and averaged by the total number of classes. Such metrics are extremely stringent as captioning models are generally biased toward certain words in the vocabulary, given the long-tailed distribution of words. In fact, both the baseline and proposed method generate about 45% of the annotated object words within the val set in Flickr30k Entities. The grounding accuracy of the other 55% of the classes are therefore zero, making the averaged grounding accuracy seemingly low.

Measuring grounding per generated sentence. Instead of evaluating grounding on each object class (which might be less intuitive), we include a new grounding evaluation metric per sentence to directly reflect the grounding measurement of each generated sentence. The metrics are computed against a pool of object words and their ground-truth bounding boxes (GT bbox) collected across five GT captions on Flickr30k Entities (and one GT caption on ActivityNet-Entities). We use the same Prec all , Rec all , Prec loc , and Rec loc as defined previously, but their scores are averaged on each of the generated sentence. As a result, the F1 loc per sent measures the F1 score only on the generated words. The model will not be punished if some object words are not generated, but it also needs to maintain diversity to achieve high captioning performance.

4.1. IMPLEMENTATION AND TRAINING DETAILS:

Region proposal and spatial features. Following GVD (Zhou et al., 2019) , we extracted 100 region proposals from each image (video frame) and encode them via the grounding-aware region encoding.

Please refer to Appendix A.5 for more implementation details.

Training. We train the model with ADAM optimizer (Kingma & Ba, 2015) . The initial learning rate is set to 1e -4. Learning rates automatically drop by 10x when the CIDEr score is saturated. The batch size is 32 for Flickr30k Entities and 96 for ActivityNet-Entities. We learn the word embedding layer from scratch for fair comparisons with existing work (Zhou et al., 2019) .

4.2. CAPTIONING AND GROUNDING PERFORMANCE COMPARISON:

Flickr30k Entities. We first compare the proposed method with our baseline with or without grounding supervision on the Flickr30k Entities test set (see Table 1 ). To train the supervised baseline, we train the attention mechanism as well as add the region classification task using the ground-truth grounding annotation, similar to GVD (Zhou et al., 2019) . We train the proposed baselines and our method on the training set and choose the best performing checkpoints based on their CIDEr score on the val set. Our experimental results are reported by averaging across five runs on the test set. We report only the mean of the five runs to keep the table uncluttered. When compared to the existing state of the arts, our proposed baselines achieve comparable captioning evaluation performances and grounding accuracy. Using the resulting supervised baseline as the upper bound, our proposed method with cyclical training statistically achieves around 20 to 25% relative grounding accuracy improvements for both F 1 all and F 1 loc and 10 to 15% for F 1 all per sent and F 1 loc per sent without utilizing any grounding annotations or additional computation during inference.

ActivityNet-Entities. We adapt our proposed baselines and method to the ActivityNet-Entities video dataset (see Table 2 ). We can see that our proposed method significantly improved the grounding accuracy around 25% to 30% relative grounding accuracy improvements for both F 1 all and F 1 loc and around 40% for F 1 all per sent and F 1 loc per sent .

4.3. ANALYSIS:

Are localized image regions better than attended image regions during training? Given our intuition described in Sec. 3, we expect the decoder to be regularized to update its attention mechanism to match with the localized image regions rt → rl t . This indicates that the localized image regions should be more accurate than the attended image regions by the decoder during training. To verify this, we compute the attention accuracy for both decoder and localizer over ground-truth sentences following (Rohrbach et al., 2016; Zhou et al., 2018) . The attention accuracy for localizer is 20.4% and is higher than the 19.3% from the decoder at the end of training, which confirms our hypothesis.

Grounding performance when using a better object detector. In Table 1 and 2 we showed that our proposed method significantly improved the grounding accuracy for both image and video captioning. These experimental settings follow the widely used procedure for visual captioning systems: extract regional proposal features and generate visual captions by attending to those extracted visual features. left) and proposed approach (right). Our proposed method is able to generate more descriptive sentences while selecting the correct regions for generating the corresponding words.

One might ask, what if we have a better object detector that can extract robust visual representation that are better aligned with the word embeddings? Will visual grounding still an issue for captioning?

To answer this, we ran two sets of experiments (Table 3 ): (1) Perfect object detector: we replace the ROIs by ground-truth bbox and represent the new ROIs by learning embedding features directly from ground-truth object words associated with each ground-truth bbox. This experiment gives an estimate of the captioning and grounding performance if we have (almost) perfect ROI representations (though unrealistic). We can see that the fully-supervised method achieves an F1 all of only 23%, which further confirms the difficulty of the metric and the necessity of our grounding metric on a per sentence level (note that F1 loc per sent shows 90%). We can also see that baseline (unsup.) still leaves room for improvement on grounding performance. Surprisingly, our method improved both captioning and grounding accuracy and surpasses the fully-supervised baseline except on the F1 loc per sent . We find that it is because the baseline (sup.) overfits to the training set, while ours is regularized from the cyclical training. Also, our generated object words are more diverse, which is critical for F1 all and F1 loc .

(2) Grounding-biased object detector: we extract ROI features from an object detector pre-trained on Flickr30k. Thus, the ROI features and their associated object predictions are biased toward the annotated object words but do not generalize to predict diverse captions compared to the original object detector trained from Visual Genome, resulting in lower captioning performance. We can see that our proposed method still successfully improves grounding and maintains captioning performance in this experiment setting as well.

How does the number of annotations affect grounding performance? In Figure 4 , we present the average F1-score on the Flickr30k Entities val set when grouping classes according to their frequency of appearance in the training set3 . We see that, unsurprisingly, the largest difference in grounding accuracy between the supervised and our proposed cyclical training is for the 50 most frequently appearing object classes, where enough training data exists. As the number of annotated boxes decreases, however, the difference in performance diminishes, and cyclical training appears to be more robust. Overall, we see that the supervised method is biased towards frequently appearing objects, while grounding performance for the proposed approach is more balanced among classes.

Qualitative analysis. We additionally conduct qualitative analysis for comparing the baseline (Unsup.) and the proposed method in Figure 5 . Each highlighted word has a corresponding image region annotated on the original image. The image regions are selected based on the region with the maximum attention weight in α t . We can see that our proposed method significantly outperformed the baseline (Unsup.) in terms of both the quality of the generated sentence and grounding accuracy.

In addition, we also discuss a number of correct and incorrect examples of our proposed method in Figure 8 in the Appendix. Please refer to the Appendix A.4 for further discussions on the qualitative results and the complete sequence of attended image regions of examples in Figure 5 .

5. CONCLUSION:

Working from the intuition that typical attentional mechanisms in the visual captioning task are not forced to ground generated words since recurrent models can propagate past information, we devise a novel cyclical training regime to explicitly force the model to ground each word without grounding annotations. Our method only adds a fully-connected layer during training, which can be removed during inference, and we show thorough quantitative and qualitative results demonstrating around 20% or 30% relative improvements in visual grounding accuracy over existing methods for image and video captioning tasks. Table 4 : Grounding performance when using better object detector on the Flickr30k Entities test set (results are averaged three runs). Fully-supervised method (Sup.) is used as upper bound, thus its numbers are not bolded.

Ground-truth::

A man is hiking while holding a water bottle.

A man wearing a hat and hiking shoes is hiking.

Object words: {man, bottle, hat, shoes}

Predicted:

A man wearing a hat is hiking with a dog. 

A APPENDIX A.1 GROUNDING EVALUATION METRICS ILLUSTRATED:

To help better understand the grounding evaluation metrics used in this work, we illustrated the grounding evaluation metrics in Figure 6 .

We define the number of object words in the generated sentences as A, the number of object words in the GT sentences as B, the number of correctly predicted object words in the generated sentences as C and the counterpart in the GT sentences as D, and the number of correctly predicted and localized words as E (see illustration of the grounding metrics in Appendix A.1). A region prediction is considered correct if the object word is correctly predicted and also correctly localized (i.e., IoU with GT box > 0.5). We then compute two version of the precision and recall as Prec The original grounding evaluation metric proposed in GVD (Zhou et al., 2019) average the grounding for each object class. We additionally calculate the grounding accuracy for each generated sentence as demonstrated in the figure . From this example, we can see that while P recision all counts dog as a wrong prediction for the dog object class, the P recision loc only cares if man and hat are predicted and correctly localizer (IoU > 0.5).

Under review as a conference paper at ICLR 2020 

A.2 ADDITIONAL ANALYSIS:

Should we explicitly make attended image regions to be similar to localized image regions? One possible way to regularize the attention mechanism of the decoder is to explicitly optimize rt → rl t via KL divergence over two soft-attention weights α t and β t . The experimental results are shown in Table 5 (Attention consistency). We use a single run unsupervised baseline with a fix random seed as baseline model for ablation study. We can see that when explicitly forcing the attended regions to be similar to the localized regions, both the captioning performance and the grounding accuracy remain similar to the baseline (unsup.). We conjecture that this is due to the noisy localized regions at the initial training stage. When forcing the attended regions to be similar to noisy localized regions, the Language LSTM will eventually learn to not rely on the attended region at each step for generating sequence of words. To verify, we increase the weight for attention consistency loss and observed that it has lower grounding accuracy (F1 all = 3.2), but the captioning will reach similar performance while taking 1.5x longer to reach convergence.

Is using only the generated word for localization necessarily? Our proposed localizer (Eq. 6 and Figure 3 ) relies on purely the word embedding representation to locate the image regions. This forces the localizer to rely only on the word embedding without biasing it with the memorized information from the Attention LSTM. As shown in the Table 5 (localizer using h A ), although this achieves comparable captioning performance, it has lower grounding accuracy improvement compared to our proposed method.

Can words that are not visually-groundable handled differently? In the proposed method, all the words are handled the same regardless of whether they are visually-groundable or not. Yet, typically words that are nouns or verbs are more likely to be grounded, and words like "a", "the", etc, are not visually-groundable.

We explored a few method variants to handle nouns and verbs differently. Mainly, we explored with two variants. Cyclical (1): the reconstruction loss is only computed when the target word is either nouns or verbs. Cyclical (2): the localized region representation will be invalid (set to zero) if the target word is neither nouns nor verbs.

The experimental results are shown in Table 6 , 7, and 8. For the first variant, Cyclical (1), we observed that the captioning performance stays the same while grounding accuracy has a small improvement. On the other hand, for the second variant, Cyclical (2), we can see that all captioning scores are improved over baseline with CIDEr improved 2.4. We can also see that grounding accuracy on per sentence basis further improved as well. We then conducted further experiments on both ActivityNet-Entities and Flickr30k Entities with unrealistically perfect object detector, but the improvements however are not consistent. In summary: on the Flickr30k Entities test set, we observed that CIDEr is better and grounding per sentence better, on the ActivityNet-Entities val set, A man sits on a chair in front of a lake.

A man in a red shirt is standing on a wooden platform.

A girl in a purple sweater is jumping on rocks.

A man in a yellow jacket and blue helmet riding a bike.

An Asian woman is holding a red umbrella and walking down the sidewalk.

A man in a black shirt is holding up a flag.

A young girl in a pink shirt and jeans is walking down a brick wall.

A man in an orange shirt and a hat is standing next to a blue wall. 

A.4 ADDITIONAL QUALITATIVE RESULTS:

In Figure 8 , we show a number of correct and incorrect examples of our proposed method. We observe that while the model is able to generate grounded captions for the images, it may sometimes overlook the semantic meaning of the generated sentences, for example, "A young girl [...] walking down a brick wall". Similarly, the model can overlook the spatial relationship between the objects, for instance, "A man [...] is holding up a flag". While a flag is present in the scene and was able to be successfully located with the corresponding word, the man in a black shirt is spatially far from the flag.

In Figure 9 , 10, 11, 12, 13, 14, 15, and 16 , we illustrated the sequence of attended image region when generating each word for a complete image description. At each step, only the top-1 attended image region is shown. This is the same as how the grounding accuracy is measured. Please see the description for Figure 9 -16 for further discussions on the qualitative results. Figure 9 : A group of men in white uniforms are standing in a field with a crowd watching. We can see that our proposed method attends to the sensible image regions for generating visually-groundable words, e.g., man, uniforms, field, and crowd. Interestingly, when generating standing, the model pays its attention on the image region with a foot on the ground.

Figure 10 : A young girl wearing a winter hat and a purple coat is smiling at the camera. The proposed method is able to select the corresponding image regions to generate girl, hat, and coat correctly. We have also observed that the model tends to localize the person's face when generating camera.

Figure 11 : A white horse with a rider in a blue helmet and white shirt jumping over a hurdle. While the model is able to correctly locate objects such as horse, rider, helmet, shirt, and hurdle, it mistakenly describes the rider as wearing a blue helmet, while it's actually black, and with white shirt while it's blue.

Figure 12 : A man in a red shirt is standing on a wooden platform. Our method correctly attends on the correct regions for generating man, shirt, and platform. Figure 15 : A girl in a white shirt and black pants is jumping on a red couch. Our method is able to ground the generated descriptive sentence with the correct grounding on: girl, shirt, pants, and couch.

Figure 16 : A man in a blue robe walks down a cobblestone street. Our method grounds the visuallyrelevant words like: man, robe, and street. We can also see that it is able to locate the foot on ground for walks.

:

The object words are words in the sentences that are annotated with corresponding image regions.

We group the

object classes in 10 groups, sorted by the number of annotated bounding boxes.

Figures and Tables:

Figure 2: Proposed cyclical training regimen: decoding -→ localization -→ reconstruction. The decoder attends to the image regions and sequentially generate each of the output words. The localizer then uses the generated words as input to locate the image regions. Finally, the shared decoder during reconstruction stage uses the localized image regions to regenerate a sentence that matches with the ground-truth sentence.

Figure 3: Proposed model architecture (left) and how the model operates during decoding, localization, and reconstruction stages (right). During the decoding stage, the soft-attention module uses the hidden state of the Attention LSTM to compute attention weights on image regions. During the localization and reconstruction stage, the soft-attention module instead uses the generated word from decoding stage to compute attention weights on image regions.

Figure 4: Average F1 all -score per class as a function of class frequency.

Figure5: Generated captions and corresponding visual grounding regions with comparison between baseline (left) and proposed approach (right). Our proposed method is able to generate more descriptive sentences while selecting the correct regions for generating the corresponding words.

Figure 6: Illustration of Grounding metrics.

man and woman, each wearing a white shirt, are walking down the sidewalk.

Figure 7: Demonstration of our human evaluation study on grounding. Each human subject is required to rate which method (A or B) has a better grounding on each highlighted word.

Figure 8: Correct (top) examples and examples with errors (bottom) from the proposed method.

Figure13: A man in a yellow jacket and blue helmet riding a bike. The proposed method correctly generates a descriptive sentence while precisely attending to the image regions for each visuallygroundable words: man, jacket, helmet, and bike.

Figure14: A man in an orange shirt and a hat is standing next to a blue wall. While our method is able to ground the generated sentence on the objects like: man, shirt, hat, and wall , it completely ignores the person standing next to the man in the orange cloth.

Performance comparison on the Flickr30k Entities test set: ATT-FCN(You et al., 2016), NBT(Lu et al., 2018), Up-Down(Anderson et al., 2018), GVD(Zhou et al., 2019), and Baseline is our reimplementation of GVD. *: our results are averaged across five runs. Only numbers reported by multiple runs are considered to be bolded.

Table content: <table><tr><td/><td>Grounding</td><td/><td colspan="3">Captioning Evaluation</td><td/><td/><td colspan="2">Grounding Evaluation</td><td/></tr><tr><td>Method</td><td>supervision</td><td>B@1</td><td>B@4</td><td>M</td><td>C</td><td>S</td><td>F1all</td><td>F1loc</td><td>F1all per sent</td><td>F1loc per sent</td></tr><tr><td>ATT-FCN</td><td/><td>64.7</td><td>19.9</td><td>18.5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NBT</td><td/><td>69.0</td><td>27.1</td><td>21.7</td><td>57.5</td><td>15.6</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Up-Down</td><td/><td>69.4</td><td>27.3</td><td>21.7</td><td>56.6</td><td>16.0</td><td>4.14</td><td>12.3</td><td>-</td><td>-</td></tr><tr><td>GVD (w/o SelfAttn)</td><td/><td>69.2</td><td>26.9</td><td>22.1</td><td>60.1</td><td>16.1</td><td>3.97</td><td>11.6</td><td>-</td><td>-</td></tr><tr><td>GVD</td><td/><td>69.9</td><td>27.3</td><td>22.5</td><td>62.3</td><td>16.5</td><td>7.77</td><td>22.2</td><td>-</td><td>-</td></tr><tr><td>Baseline*</td><td/><td>69.0</td><td>26.8</td><td>22.4</td><td>61.1</td><td>16.8</td><td>8.44 (+100%)</td><td>22.78 (+100%)</td><td>27.37 (+100%)</td><td>63.19 (+100%)</td></tr><tr><td>Baseline*</td><td/><td>69.1</td><td>26.0</td><td>22.1</td><td>59.6</td><td>16.3</td><td>4.08 (+0%)</td><td>11.83 (+0%)</td><td>13.20 (+0%)</td><td>31.83 (+0%)</td></tr><tr><td>Cyclical*</td><td/><td>69.4</td><td>26.9</td><td>22.3</td><td>60.8</td><td>16.6</td><td>5.11 (+24%)</td><td>14.15 (+21%)</td><td>15.15 (+14%)</td><td>35.56 (+12%)</td></tr><tr><td/><td>Grounding</td><td/><td colspan="3">Captioning Evaluation</td><td/><td/><td colspan="2">Grounding Evaluation</td><td/></tr><tr><td>Method</td><td>supervision</td><td>B@1</td><td>B@4</td><td>M</td><td>C</td><td>S</td><td>F1all</td><td>F1loc</td><td>F1all per sent</td><td>F1loc per sent</td></tr><tr><td>GVD</td><td/><td>23.0</td><td>2.27</td><td>10.7</td><td>44.6</td><td>13.8</td><td>0.28</td><td>1.13</td><td>-</td><td>-</td></tr><tr><td>GVD (w/o SelfAttn)</td><td/><td>23.2</td><td>2.28</td><td>10.9</td><td>45.6</td><td>15.0</td><td>3.70</td><td>12.7</td><td>-</td><td>-</td></tr><tr><td>GVD</td><td/><td>23.9</td><td>2.59</td><td>11.2</td><td>47.5</td><td>15.1</td><td>7.11</td><td>24.1</td><td>-</td><td>-</td></tr><tr><td>Baseline*</td><td/><td>23.1</td><td>2.13</td><td>10.7</td><td>45.0</td><td>14.6</td><td>7.30 (+100%)</td><td>25.02 (+100%)</td><td>17.88 (+100%)</td><td>60.23 (+100%)</td></tr><tr><td>Baseline*</td><td/><td>23.2</td><td>2.22</td><td>10.8</td><td>45.9</td><td>15.1</td><td>3.75 (+0%)</td><td>12.00 (+0%)</td><td>9.41 (+0%)</td><td>31.68 (+0%)</td></tr><tr><td>Cyclical*</td><td/><td>23.7</td><td>2.45</td><td>11.1</td><td>46.4</td><td>14.8</td><td>4.68 (+26%)</td><td>15.84 (+29%)</td><td>12.60 (+38%)</td><td>44.04 (+43%)</td></tr></table>

Performance comparison on the ActivityNet-Entities val set: GVD(Zhou et al., 2019) and Baseline is our reimplementation of GVD.

Table content: <table/>

Grounding performance when using better object detector on the Flickr30k Entities test set (see Table4for complete version).

Table content: <table/>

Model ablation study on the Flickr30k Entities val set.

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="3">Captioning Eval.</td><td colspan="2">Grounding Eval.</td></tr><tr><td/><td/><td/><td>#</td><td/><td>M</td><td>C</td><td>S</td><td>F1all</td><td>F1loc</td></tr><tr><td/><td/><td colspan="2">Baseline (Unsup.)</td><td/><td>22.3</td><td>62.1</td><td>16.0</td><td>4.18</td><td>11.9</td></tr><tr><td/><td/><td/><td>Cyclical</td><td/><td>22.2</td><td>62.2</td><td>16.2</td><td>5.63</td><td>14.6</td></tr><tr><td/><td/><td colspan="3">-Attention consistency</td><td>22.3</td><td>61.8</td><td>16.2</td><td>4.19</td><td>11.3</td></tr><tr><td/><td/><td colspan="2">-Localizer using h A</td><td/><td>22.2</td><td>61.8</td><td>16.1</td><td>4.58</td><td>11.3</td></tr><tr><td/><td/><td colspan="3">Captioning Evaluation</td><td/><td/><td/><td colspan="2">Grounding Evaluation</td></tr><tr><td>Method</td><td>B@1</td><td>B@4</td><td>M</td><td>C</td><td/><td>S</td><td>F1 all</td><td>F1 loc</td><td>F1 all per sent</td><td>F1 loc per sent</td></tr><tr><td>Baseline</td><td>69.1</td><td>26.0</td><td>22.1</td><td colspan="2">59.6</td><td>16.3</td><td>4.08</td><td>11.83</td><td>13.20</td><td>31.83</td></tr><tr><td>Cyclical</td><td>69.4</td><td>26.9</td><td>22.3</td><td colspan="2">60.8</td><td>16.6</td><td>5.11</td><td>14.15</td><td>15.15</td><td>35.56</td></tr><tr><td>Cyclical (1)</td><td>69.7</td><td>27.0</td><td>22.2</td><td colspan="2">60.1</td><td>16.5</td><td>5.14</td><td>14.32</td><td>15.36</td><td>36.33</td></tr><tr><td>Cyclical (2)</td><td>69.9</td><td>27.5</td><td>22.4</td><td colspan="2">62.0</td><td>16.6</td><td>5.13</td><td>13.99</td><td>16.30</td><td>38.45</td></tr></table>

Performance comparison on the Flickr30k Entities test set. All results are averaged across five runs.

Table content: <table/>

Peer review (iclr2020):
# 1. Summary The paper deals with the problem of learning grounded captions from images without joint text-location information, but instead texts (words in captions) and locations are provided independently and the model needs to figure out their link. The model is built upon GVD (Zhou et al., 2019): each word generated by the encoder is grounded to the locations provided by the region proposal module (Up-Down model (Anderson et al., 2018)), used to reconstruct the ground-truth caption. My weak reject decision was guided by the following strengths and weaknesses of the paper. Strengths: * The reconstruction formulation of the problem is interesting and relevant for the task * Proposed new metric to measure grounding performance Weaknesses: * Questionable motivations: it is not clear what application is grounding text to the image useful for? * Marginal (not statistically significant?) improvement on image captioning metrics by using grounded text (proposed method) compared to not grounding (GVD) * Limited novelty: extension of GVD, where attention is removed and object locations are used instead # 2. Clarity and Motivation The paper is generally well written, however there are some concerns on motivations. One concern is related to the motivations of the paper. The authors use grounding as a proxy to improve image captioning results, which improvement is marginal wrt GVD (see Table 1). Why do we need to localize text if this has very marginal impact on the captioning metrics? It is missing the link between the potential applications where the localization of words is relevant. The authors claim that they do not uses any grounding annotation, however the pre-trained Faster-RCNN has been trained using annotations which consist of bounding boxes + categories. Therefore, the model do (in an implicit way) rely on grounding annotations, especially because there might be an overlap between the words (classes) used to pretrain the detector and the words in the captions. The authors should assess if this ovelap/bias in the pre-trained Faster-RCNN exists or not. Some other questions are still to be answered: * How are the regions R parametrized? Is it the visual representation or bounding box locations? * What happens to words that are not grounded to the image (e.g., verbs or articles)? Do you have a special way to deal with those? * What is the intuition of multiplying word embedding and region embeddings to generate z in Eq. 6? # 3. Novelty The proposed method is an extension of the existing model GVD, where the attentional module is removed and its functionality is replaced by the cyclical training mode with reconstruction of the object locations. From the technical point of view this is limited novelty, but still an interesting improvement of the model; however the experiments and results do not support the claim that using such model improves image captioning result in a significant way. One way to answer to this question would have been by showing an application where the outputted locations are used for downstream tasks. # 4. Experimentation The experiments are carried out in a scrupulous way, by showing the comparing with GVD (with and without attention; with and without grounding supervision). The non-convincing part of them (as mentioned above already) is the fact that the improvements on these datasets might be non significant for image captioning. For example, let*s consider the image captioning results in Table 1 (Flickr30k Entities): cyclical have a max improvement of 0.7 (CIDER) and min of 0 (B@4) when compared with GVD without grounding supervision. There is an obvious huge improvement on the grounding evaluation, which is obvious since GVD does not do it explicitly. The same trend is in Table 2. These results are not convincing, combined by the fact that it is not clear in which applications one would want a very accurate grounded text. # Minor Points * Sec. 3.1: it is not clear that the Language LSTM is the decoder. Please explicitly say it before Eq. 1 * Caption of Table 3: which dataset is this?