Learning to Generate Grounded Visual Captions without Localization Supervision
Authors: Chih-Yao Ma
Yannis Kalantidis
Ghassan AlRegib
Peter Vajda
Marcus Rohrbach
Zsolt Kira
Venue:  (2020)
Keywords: image captioning
video captioning
self-supervised learning
visual grounding

Abstract: When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.

Under review as a conference paper at ICLR 2020
LEARNING TO GENERATE GROUNDED VISUAL CAPTIONS WITHOUT LOCALIZATION SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
When automatically generating sentence description for image video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.
INTRODUCTION
Vision and language tasks, such as visual captioning. combine linguistic descriptions with data from real-world scenes. Deep learning models for such tasks have achieved great success, driven in part by the development of attention mechanisms that focus on various objects the scene while generating captions. The resulting models, however, are known to have poor grounding performance (Liu et al., 2017), leading to undesirable behaviors such as object hallucinations (Rohrbach et al., 2018), despite having high captioning accuracy. That is, they often do not correctly associate generated words with the appropriate image regions (e.g., objects) in the scene, resulting in models that lack interpretability.
Several existing approaches have tried to improve the grounding of captioning models. One class of methods generate sentence templates with slot locations explicitly tied to specific image regions. These slots then filled in by visual concepts identified by off-the-shelf object detectors (Lu et al., 2018). Other methods have developed specific grounding or attention modules that aim to attend to the correct region(s) for generating visually groundable word. Such methods, however, rely on explicit supervision for optimizing the grounding or attention modules (Liu et al., 2017; Zhou et al., 2019) and require bounding box annotations for each visually groundable word.
In this work, we propose a novel cyclical training regimen that is able to significantly improve grounding performance without any grounding annotations. The key insight of our work is that current models use attention mechanisms conditioned on the hidden features of recurrent modules such as L.STMs, which leads to effective models with high accuracy but entangle grounding and decoding. Since LSTMs are effective at propagating information across the decoding process, the network does not nccessarily need to associate particular decoded words with their corresponding image region(s). However, for captioning model to be visually grounded, the model has to predict attentional weights without knowing the word to localize.
Based this insight, we develop cyclical training regimen to force the network to ground individual decoded words: decoding - localization - reconstruction. Specifically, the model of the decoding stage can be any fate--ffthee.e the--rrrttttttt- captioning model; in this work, we follow GVD (Zhou et al.,

---

Under review as a conference paper at ICLR 2020
Captioning
momenn with Black
Localization
Final Result
De haldnn cuR
ACEIN
WorTian
hal
Noman?
1s
THE
16
cup?
Localizing
Matched
Model updale!!
Matched
Awaman with a DOtCh RIE holding CLUT
Figure I: Visual captioning models are often not tion check whether the generated caption is ggruuddygro If the localized image region is incorrect, we update the model. However, without the ground-trut grounding annotation, how does the model know the localized region is incorrect? To overcome this issue, we propose perform localization and reconsiruction to regularize the captioning model to be iisually-grounder without relying on the grounding annotations.
As human, we perform localiza-
2019) to extend the widely used Up-Down model (Anderson al., 2018). At the localization stage, each word generated by the first decoding stage is localized through a localizer, and the resulting grounded image region(s)
then used reconstruct the ground-truth caption in the final stage. Both decoding and reconstruction stages are trained using a standard cross-entropy loss. Key to our method, both stages share the same decoder, thereby causing the localization stage guide the decoder to improve its attention mechanism. Our method is simple and only adds fully-connected layer to perform localization. During inference, we only use the (shared) decoder, thus do not add any computational cost.
We benchmark our proposed method on the challenging Flickr30k Entities image captioning dataset (Plummer et al., 2015) and the ActivityyNe--ntitiis video captioning dataset (Zhou et al., 2019) on both captioning and grounding performances. that calculate the grounding accuracy for each object class (Zhou et al., 2019), we further include grounding metric that compute grounding accuracy for each generated sentence. This new metric on each sentence removes the stringency of the original evaluation metric (as we discuss in Sec. 4) and provides an alternative way of measuring the grounding performance.
addition the existing grounding metric
Despite the simplicity of our proposed method, we able significantly surpass prior unsupervised models quantitatively and qualitatively on both datasets. We achieve around 18% relative improve- ments in terms of bridging the gap between the unsupervised baseline and supervised methods Flickr30k Entites and around 34% on Nettvity-et-Entiine outperform the supervised method on infrequent words, owing to its self-supervised nature.
We further find that our method can even
Contributions summary. We propose object re-localization as a form of self-supervision for grounded visual captioning and present cyclical training regimen that re-generates sentences after re-localizing the objects conditioned on each word, implicitly imposing grounding consistency. We evaluate our proposed approach on both image and video captioning tasks. We show that the proposed training regime can boost grounding accuracy over statc-of-the-as bascline, enabling grounded models to be trained without bounding box annotations, while retaining high captioning quality across two datasets and various experimental settings. Our code will be publicly released and can be found in supplemental.
RELATED WORK
Visual captioning. Neural models for visual captioning have received significant attention recently (Anderson et al., 2018; 8; Ma et al., 2018; Lu et al., 2018; Donahue et al., 2015; Venugopalan et al., 2015; Rohrbach et al., 2017b; Venugopalan al., 2017; Rohrbach et al., 2017a; Shetty al., 2017; Park et al., 2019). Most current state-of ofate-of-the-ale the-ahh art models contain attention mechanisms, allowing the process to foeus subsets of the image when generating the next word. These attention mechanisms
2

---

Under review as a conference paper at ICLR 2020
can be defined over spatial locations (Vinyals et al., 2015), semantic metadata (Li et al., 2018; Yao et al., 2017; You et al., 2016; Zhou et al., 2017) or a predefined set of regions extracted via a region proposal network (Ma et al., 2018; Zanfir et al., 2016; Anderson et al., 2018; Lu et al., 2018; Das et al., 2013; Kulkarni et al., 2013). In the latter case, off-the-shelf object detectors are first used to extract object proposals (Ren et al., 2015; He al., 2017) and the captioning model then learns to dynamically attend over them when generating the caption.
Visual grounding. Although attention mechanisms are generally shown to improve captioning quality and metrics, it has also been shown that they don't really focus on the same regions as a human would (Das et al., 2017). This make models less trustworthy and interpretable. and therefore creating grounded image captioning models, i.e., models that accurately link generated words or phrases to specific regions of the image, has recently been an active research area. A number of approaches have been proposed, e.g., for grounding phrases or objects from image descriptions (Rohrbach et al., 2016; Hu et al., 2016; Xiao et al., 2017: Deng et al., 2018; Zhou al., 2019; Zhang et al., 2019), grounding visual explanations (Hendricks et al., 2018), visual co-reference resolution for actors in video (Rohrbach et al., 2017a), or improving grounding via human supervision (Selvaraju et al., 2019). Recently, Zhou et al. (2019) presented model with self self-attention based context encoding and direct grounding supervision that achieves state-of -frthh--rt results in both the image and video tasks. They exploit ground-truth bounding box annotations to significantly improve the visual grounding accuracy. In contrast, we focus on reinforcing the visual grounding capability of the existing captioning model via a cyclical training regimen without using bounding box annotations and present a method that can increase grounding accuracy while maintaining comparable captioning performance with state of the arts.
Cyclical training. Cycle consistency (Wang et al., 2013; Zhu et al., 2017; He et al., 2016; Chen & Lawrence Zitnick, 2015) has been used recently in a wide range of domains, including machine translation (He et al., 2016), unpaired mmage-to-image translation (Zhu et al., 2017), visual question answering (Shah et al., 2019), question answering (Tang et al., 2018), image captioning (Chen & Lawrence Zitnick, 2015), video captioning (Wang et al., 2018; Duan et al., 2018), captioning and drawing (Huang et al., 2018) as well as domain adaptation (Hosseini-Ass et al., 2019). While the cyclical training regime has been explored vastly in both vision and language domains, it has not yet been used for enforcing the visual grounding capability of a captioning model.
METHOD
Notation. For visual captioning task, we denote the input image as / (or input video as V)) and the target sentence as S. Each image (or video) is represented by spatial feature map(s) extracted by ResNet-101 model and a bag of regions obtained from FaNter-RCNN (Ren et al., 2015) as R- [r, T2, ..., TN] E Id The target sentence is represented as a sequence of one-hot vectors y. E R*, where T is the sentence length, I E 1,2, ... T. and is the dictionary size.
3.1 BASELINE
We reimplemented the model used in GVD (Zhou et al., 2019) without self-attention for region feature encoding (Ma et al., 2018; Vaswani et al., 2017) as our baseline. It is an extension of the state-of-the-as Up-Down (Anderson al., 2018) model with the grounding -awar region encoding (see Appendix A.5). Specifically, our baseline model uses two LSTM modules: Attention LSTM and Language LSTM. The Attention L.TTM identifies which visual representation in the image is needed for the Language LSTM to generate the next word. It encodes the global image feature Vg. previous hidden state output of the Language LSTM h 1. and the previous word embedding e,- into the hidden state h
h = LST ((vgi ii- e.
WeBr-1+
(1)
where [:] denotes concatenation, and W. are learned parameters. We omit the Attention LSTM input hidden and cell states to avoid notational clutter in the exposition.
The Language LSTM uses the hidden state h from the Attention LSTM dynamically attend on the bag of regions R for obtaining visual representations of the image é. to generate word ye.
zt,n Waglanhi (W, W.,h, rw), at = softmax(z1).
è: Ro.
3

---

Under review as a conference paper at ICLR 2020
Decoding
Decockng
Localitation
Locoltaraa
person
stthigg
ateon
Decoding loss
man
sitting
bench
Attended ROls
Locowlzed ROls
Figure 2: Proposed cyclical training regimen: decoding - localization + reconstruction. The decoder attends to the image regions and sequentially generate each of the output words. The localizer then uses the generated words imput locate the image regions. Finally, the shared decoder during reconstruction stage uses the localized image regions to regenerate a sentence that matches with the ground-truth sentence.
where W on and Wa are learned parameters. The conditional probability distribution over possible output words ye is computed as:
h L.ST I.ST Long((Fe hz]), p(y = softmax (W,h!")
where y1:r- is a sequence of outputs (yI, ...... 11-1). We refer the Language LSTM and the output logit layer as the complete language decoder.
3.2 OVERVIEW
Our goal is to enforce the generated caption to be visually grounded, i.e., attended image regions correspond specifically to individual words being generated, without ground-truth grounding supervi- sion. Towards this end, we propose novel cyclical training regimen that comprised of decoding, localization, and reconstruction stages, as illustrated in Figure 2.
The intuition of our method that the bascline network is not forced to generate a correct correspon- dence between the attended objects and generated words, since the LSTTSs can learn priors in the data instead of looking at the image or propagate information forward which can subsequently be used to generate corresponding words in future time steps. The proposed cyclical training regimen, in contrast, aims at enforcing visual grounding to the model by requiring the language decoder (Eq. 3) to rely on the localized image regions of to reconstruct the ground-truth sentence, where the localization is conditioned only on the generated word from the decoding stage. Our cyclical method can therefore be done without using any annotations of the grounding itself.
Specifically, let y;º² = (fr; 0x) be the initial language decoder with parameters 0g (Eq. 3). trained to sequentially generate words y Let ((y map (ground) each generated word to region(s) in the image, i.e., r = ( R: 0g). Finally, let y D' (f!: (t) be a second decoder, that is required to reconstruct the ground-truth caption using the localized region(s), instead of the attention computed by the decoder itself. We define the cycle:
; (ly) define a localizer unit with parameters 0g. that learns
v = D" ("GG( 0a), R;0,);00) =2.
where D' and D' share parameters. Although parameters shared, the inputs for the two language decoders differ, leading to unique LSTM hidden state values during run. Note that the Attention LSTMs and logit layers in the two stages also share parameters, though they are omitted for clarity.
Through cyclical joint training, both Dd and D' are required to generate the same ground-truth sentence. They are both optimized to maximize the likelihood the correct caption:
8 arg max
Σ log p 0a) + arg max
Σ log p(vl:
0g
0,
During training, the localizer regularizes the region attention of the reconstructor and the effect is further propagated the baseline network in the decoding stage, since the parameters of Attention
4

---

Under review as a conference paper at ICLR 2020
Soltattentio
Logit
Decoding stage
A
Localizer
Switch
Localization Recon. stage
(decoding
Attention LSTM
hf
Localizer
Localizor
WoY+
W
W4y--
Figure Proposed model architecture (left) and how the model operates during decoding, localization, and reconstruction stages (right). During the decoding stage, the soft-attention module uses the hidden state of the Attention LSTM to compute attention weights on image regions. During the localization and reconstruction stage, the soft-attention module instead uses the generated word from decoding stage to compute attention weights on image regions.
LSTM and Language L.TT are shared for both decoding and reconstruction stages. Note that the gradient from reconstruction loss will not backprop to the decoder DDd in the decoding stage since the generated words used as input to the localizer are leafs the computational graph. The network is implicitly regularized to update its attention mechanism to match with the localized image regions f + F/. In Sec. 4.3, we demonstrate that the localized image regions rf indeed have higher attention accuracy than f. when using ground-truth words inputs for the localizer.
3.3 CYCLICAL TRAINING
We now describe each stage of our cyclical model in detail, as illustrated in Figure 3.
Decoding. We first use the baseline model presented Sec. 3.1 to generate a sequence of words [yf ....., y where T is the ground-truth sentence length.
Localization. Following the decoding process, a localizer g is then learned to localize the image regions from each generated word yt-
e, = Way. 21,m (Wie,) Ta and B, softmax(z().
(6)
where is the embedding for the word generated during decoding stage step r, is the image representation of a region proposal, and W. and W; are the learned parameters. Based on the localized weights 3r, the localized region representation can be obtained by rf R.3.
Reconstruction. Finally, the shared language decoder 'D' relies on the localized region representation i! generate the next word. The probability over possible output words is:
h LLLSMMMaan (FF): 11,"), p(vl vi. = softmax Boftmax(W..f33 (W.hf).
(7)
Given the target ground truth caption y and our proposed captioning model parameterized with we minimize the following cross-entropy losses:
log(pe vv vi. (w= CCe(O) - Σ
log(pe (yū v)) 4 (w)=
(8)
decoding loss
reconstuction loss
where d and d2 are weighting coefficient selected on the validation split.
4 EXPERIMENTS
Datasets. We use the Flickr30k Entities image dataset (Plummer et al., 2015) and the ActivityNet- Entities video dataset (Zhou et al., 2019) for evaluating our proposed approach. Flickr30k Entities
5

---

Under review as a conference paper at ICLR 2020
contains 275k annotated bounding boxes from 31k images associated with natural language phrases. Each image is annotated with 5 crowdsourced captions. ActivityNet- Entiiiiiss contains 15k videos with 158k spatially annotated bounding boxes from 52k video segments.
Captioning evaluation metries. We measure captioning performance using four language metrics, including BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016).
Grounding evaluation metries. Following the grounding evaluation from GVD (Zhou et al., 2019), we measure the attention accuracy on generated sentences, denoted by al and Fl In Flan, a region prediction is considered correct if the object word! is correctly predicted and also correctly localized. We also compute Fliec. which only considers correctly- object words. Please see illustration of the grounding metries in Appendix A.I.
In the original formulation, the precision and recall for the two Fl metrics are computed for each object class, and it set to zero if an object class has never been predicted. The scores are computed for each object class and averaged by the total number of classes. Such metrics are extremely stringent as captioning models are generally biased toward certain words in the vocabulary, given the long-tailed distribution of words. In fact, both the baseline and proposed method generate about 45% of the annotated object words within the val set in Flickr30k Entities. The grounding accuracy of the other 55% of the classes are therefore zero, making the averaged grounding accuracy seemingly low.
Measuring grounding per generated sentence. Instead of evaluating grounding on each object class (which might be less intuitive), we include a new grounding evaluation metric per sentence to directly reflect the grounding measurement of each generated sentence. The metrics are computed against a pool of object words and their ground-truth bounding boxes (GT bbox) collected across five GT captions on Flickr30k Entities (and one GT caption on ActivityNet-Entitien Precan, Recan, Preciec and Rec as defined previously, but their scores are averaged on each of the generated sentence. As a result, the Fl per sent measures the Fl score only on the generated words. The model will not be punished if some object words are not generated, but it also needs to maintain diversity to achieve high captioning performance.
We use the same
4.1 IMPLEMENTATION AND TRAINING DETAILS
Region proposal and spatial features. Following GVD (Zhou et al., 2019), we extracted 100 region proposals from each image (video frame) and encode them via the grounding aware region encoding. Please refer to Appendix A.5 for more implementation details.
Training. We train the model with ADAM optimizer (Kingma & Ba, 2015). The initial learning rate is set to le 4. Learning rates automatically drop by 10x when the CIDEr score is saturated. The batch size is 32 for Flickr30k Entities and 96 for ActivityNet--ntities, We learn the word embedding layer from scratch for fair comparisons with existing work (Zhou et al., 2019).
4.2 CAPTIONING AND GROUNDING PERFORMANCE COMPARISON
Flickr30k Entities. We first compare the proposed method with our baseline with or without grounding supervision on the Flickr30k Entities test set (see Table 1). To train the supervised baseline, we train the attention mechanism as well as add the region classification task using the ground-truth grounding annotation, similar to GVD (Zhou et al., 2019). We train the proposed baselines and our method on the training set and choose the best performing checkpoints based on their CIDEr score on the val set. Our experimental results are reported by averaging across five runs on the test set. We report only the mean of the five runs to keep the table uncluttered. When compared to the existing state of the arts, our proposed baselines achieve comparable captioning evaluation performances and grounding accuracy. Using the resulting supervised baseline as the upper bound, our proposed method with cyclical training statistically achieves around 20 to 25% relative grounding accuracy improvements for both Flal and Flae and 10 to 15% for Flall peilppersentt_ and T' loe_per_sent without utilizing any grounding annotations or additional computation during inference.
ActivityNet-Entitte We adapt our proposed baselines and method to the Activity dataset (see Table 2). We can see that our proposed method significantly improved the grounding
video
The object words are words in the sentences that annotated with corresponding image regions.

---

Under review as a conference paper at ICLR 2020
Grounding supervision
Captoonnng Evalaation
Grounding Evaluation
B8
R94
Flai
Fl
po ses
per.ee
ATT-FCN
19.9
18.5
27.1
21.7
57.5
Up Down
27.3
21.7
56.6
16.0
4.14
12.3
GVD (wio SelrAm))
60.3
26,9
22.1
60.1
16.1
3,97
11.6
GVD
69.9
27.3
22.5
62.3
16.5
7,77
22.2
Biseline
09.0
26,8
22.4
61.1
16.8
8.44
1008))
22,78 (4 (009))
27.57 (0008))
(1-199()1005)
Riseline
00.1
26.0
22.1
59.6
16.3
4,08
(-0%)
11.83 (-0%)
13.20 (40%)
51.83 (40%)
Cyclical*
69.4
26,9
22.3
60.8
16.6
5.11 (+249))
14.15 (+219)
15.15 (+14%)
38.56 (+12%)
Table I: Performance comparison on the Flickr30k Entities test set: ATT-FCN (You al., 2016), NBT (Lu et al., 2018), Up-Down (Anderson et al., 2018), GVD (Zhou et al., 2019), and Baseline is our reimplementation of GVD. $: our results are averaged across five runs, Only numbers reported by multiple runs are considered to be bolded.
Grounding
Captioning Evaluation
Grounding Evaluation
Method
supervisson
BB1
B64
M
C
Flai
Fl
Fla
Flac
GVD
23.0
2.27
10.7
44.6
13.8
0,28
1,13
GVD (wio SelrAma)
23.2
2.28
10.9
45.6
15.0
3.70
12.7
GVD
23,9
2,59
11.2
47.5
15.1
7.11
24.1
Biseline"
23.1
2.13
10.7
45.0
14.6
(+
25.02 (+ (000))
17.88 (+ (00%))
60.23 (+
Biseline"
23.2
2.22
10.8
45.9
15.1
3:75 (-0%))
12,00 (+0%)
9.41 (+0%)
51.68 (+0%)
Cyclical
23.7
2.45
11.1
46.4
14,8
4.68 (+26%)
15.80 (+29%)
12,60 (+38%)
44,04 (+13))
Table 2: Performance comparison on the ActivityNet-Entites val set: GVD (Zhou al., 2019) and Baseline is our reimplementation of GVD. *: our results are averaged across five runs. Only numbers reported by multiple runs are considered be bolded.
accuracy around 25% to 30% relative grounding accuracy improvements for both Flall and Flac and around 40% for Flall per_sent and F'looeper__ _loo_per_scnt-
4.3 ANALYSIS
Are localized image regions better than attended image regions during training? Given intuition described in Sec. 3, we expect the decoder to be regularized to update its attention mechanism to match with the localized image regions Å, + R/. This indicates that the localized image regions should be more accurate than the attended image regions by the decoder during training. To verify this, we compute the attention accuracy for both decoder and localizer over ground-truth sentences following (Rohrbach et al., 2016; Zhou et al., 2018). The attention accuracy for localizer is 20.4% % and is higher than the 19.3% from the decoder the end of training, which confirms our hypothesis.
Grounding performance when using better object detector. In Table I and 2 we showed that our proposed method significantly improved the grounding accuracy for both image and video captioning. These experimental settings follow the widely used procedure for visual captioning systems: extract regional proposal features and generate visual captions by attending to those extracted visual features.
Fla, Baseline
Grouding supervisoo
Capraning liall
Groending lival.
awerage F1-meosure
Fl.e-Cycical
Fla.
Flocyece
Fla Supeeviedd
Carcalistioall
detector
Baseline
25.3
76.5
22.5
23.09
52.83
90.76
77.81
Cyclical
NI.2
$4.54
81.56
Grounding-hivedd object detector
Reudine
21.3
58.
155
33.95
66.96
Baxdin:
17.51
42.84
Cydicl
21.2
52.0
154
6.8
1965
$8.25
Classes grouped by frequency (in descending order)
5330
3EL-30
Joc-ce
Table 3: Grounding performance when using better object detector on the Flickr30k Entities test set (see Table 4 for complete version).
Figure 4: Average Flan-score per class as function of class frequency.

---

Under review as a conference paper at ICLR 2020
Baseling:
Proposed: uroup while unitooma standing hold wattshing
Beseline youung gir Blue alting snow.
Proposed: young girweartg winder smilng comerrs
Baselina: Jumpling obstacte
Proposed: white and wtutl shir Jampinng Over hurte.
Baseline: Four skrers are skingg down a Anowy mountiioo
Proposad:: siller jumping
Snowy hill while other shiers watch.
Figure 5: Generated captions and corresponding visual grounding regions with comparison between baseline (left) and proposed approach (right). Our proposed method is able to generate more descriptive sentences while selecting the correct regions for generating the corresponding words.
One might ask, what if we have better object detector that can extract robust visual representation that are better aligned with the word embeddings? Will visual grounding still an issue for captioning?
To answer this, we ran two of experiments (Table 3): (1) Perfect object detector: we replace the ROIS by ground-truth bbox and represent the new ROIs by learning embedding features directly from ground-truth object words associated with each ground-truth bbox. This experiment gives an estimate of the captioning and grounding performance if we have (almost) perfect ROI representations (though unrealistic). We can see that the fully-supervised method achieves Flal of only 23%, which further confirms the difficulty of the metric and the necessity of our grounding metric on per sentence level (note that Fl per sent shows 90%). We can also see that baseline (unsup.) still leaves room for improvement on grounding performance. Surprisingly, our method improved both captioning and grounding accuracy and surpasses the fully-supervised baseline except on the Fl Ine_per_sent We find that it is because the bascline (sup.) overfits to the training set, while ours is regularized from the eyclical training. Also, our generated object words are more diverse, which is critical for Flail extract ROI features from object detector pre-trained on Flickr30k. Thus, the ROI features and their associated object predictions are biased toward the annotated object words but do not generalize predict diverse captions compared the original object detector trained from Visual Genome, resulting in lower captioning performance. We can see that our proposed method still successfully improves grounding and maintains captioning performance in this experiment setting as well.
and F1 loc- (2) Grounding-biased object detector:
How does the number of annotations affect grounding performance? In Figure 4, we present the average Fl Fl-score on the Flickr30k Entities val set when grouping classes according to their frequency of appearance in the training set'. We see that, unsurprisingly, the largest difference in grounding accuracy between the supervised and our proposed cyclical training is for the 50 most frequently appearing object classes, where enough training data exists. As the number of annotated boxes decreases, however, the difference in performance diminishes, and cyclical training appears to be more robust. Overall, we see that the supervised method is biased towards frequently appearing objects, while grounding performance for the proposed approach is more balanced among classes.
Qualitative analysis. We additionally conduct qualitative analysis for comparing the baseline (Unsup.) and the proposed method in Figure 5. Each highlighted word has corresponding image region annotated on the original image. The image regions are selected based on the region with the maximum attention weight in ag,, We can see that our proposed method significantly outperformed the baseline (Unsup.) in terms of both the quality of the generated sentence and grounding accuracy. In addition, we also discuss number of correct and incorrect examples of our proposed method in Figure 8 in the Appendix. Please refer to the Appendix A.4 for further discussions on the qualitative results and the complete sequence of attended image regions of examples in Figure 5.
We group the 460 object classes in 10 groups, sorted by the number of annotated bounding boxes.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
Working from the intuition that typical attentional mechanisms in the visual captioning task are not forced ground generated words since recurrent models can propagate past information, devise novel cyclical training regime to explicitly force the model to ground each word without grounding annotations. Our method only adds a fully-connected layer during training. which can be removed during inference, and we show thorough quantitative and qualitative results demonstrating around 20% or 30% relative improvements in visual grounding accuracy over existing methods for image and video captioning tasks.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice:
Semantic propositional image caption evaluation. In European Conference on Computer Vision, pp. 382-398. Springer, 2016.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6, 2018.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, pp. 65-72, 2005.
Xinlei Chen and C Lawrence Zitnick. Mind's eye: A recurrent visual representation for image caption generation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 2422-2431, 2015.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look the same regions? Computer Vision and Image Understanding, 163:90-100, 2017.
Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso. A thousand frames in just few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2634-2641, 2013.
Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. Visual grounding via accumulated attention. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 7746-7755, 2018.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Mareus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2625-26334 2015.
Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, and Junzhou Huang. Weakly supervised dense event captioning in videos. In Advances in Neural Information Processing Systems (NeurIPS). pp- 3063-3073, 2018.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. Advances Neural Information Processing Systems (NeurIPS), pp- 820-828, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 2980-2988. IEEE, 2017.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual expla- nations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 264-279, 2018.

---

Under review as a conference paper at ICLR 2020
Ehsan Hosseini-As!. Astl, Yingbo Zhou, Caiming Xiong, and Richard Socher. Augmented cyclic adversar- ial learning for low resource domain adaptation. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng. Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4555-4564, 2016.
Qiuyuan Huang, Pengchuan Zhang, Dapeng Wu, and Lei Zhang. Turbo learning for captionbot and drawingbot. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6456-6466, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference Learning Representations (ICLR), 2015.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Girish Kulkarni, Visruth Premraj. Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2891-29)) 2013.
Ychao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. Jointly localizing and describing events for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 1492-7500, 2018
Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. Attention correctness in neural image captioning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7219-7228, 2018.
Chih- Yao Ma, Asim Kadav, lain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf, Attend and interact: Higher-order object interactions for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311 311-318. Association for Computational Linguistics, 2002.
Jae Sung Park, Marcus Rohrbach, Trevor Darrell, and Anna Rohrbach. Adversarial inference for multi-sentence video description. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), 2019.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting cisiiontt--hraase correspondences for richer image-to-senteed toseeeee-ee models. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 2641-2649, 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 91-99, 2015.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference Computer Vision (ECCV), pp. 817-834. Springer, 2016.
Anna Rohrbach, Mareus Rohrbach, Siyu Tang, Seong Joon Oh, and Bernt Schiele. Generating descriptions with grounded and co-referenced people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4979-4989, 2017a
10

---

Under review as a conference paper at ICLR 2020
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 123(1):94-12) 2017b.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4035-4045, 2018.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Dhruv Batra, and Devi Parikh. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Rakshith Shetty, Mareus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4135-4144, 2017.
Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, and Ming Zhou. Learning to collaborate for question answering and asking. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long Papers), volume 1, pp. 1564-1574, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pp. 199986008, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4566 4575-4575, 2015.
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sruence-vviees to text. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4534 4542, 2015.
Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko. Captioning images with diverse objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5753-5761, 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3156-3164, 2015.
Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1622-76315 2018.
Fan Wang, Qixing Huang, and Leonidas J. Guibas. Image co-segmentation via consistent functional maps. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
Fanyi Xiao, Leonid Sigal, and Yong Jae Lee. Wuakly-supervised visual grounding of phrases with linguistic structures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2017.
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting image captioning with attributes. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 22-29, 2017.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4659, 2016.

---

Under review as a conference paper at ICLR 2020
Mihai Zanfir, Elisabeta Marinoiu, and Cristian Sminchisescu. Spatio-temporal attention models for grounded video captioning. Asian Conference on Computer Vision, pp. 104-119, 2016.
Yundong Zhang, Juan Carlos Niebles, and Alvaro Soto. Interpretable visual question answering by visual grounding from attention supervision mining. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 349-357. IEEE, 2019.
Luowei Zhou, Chenliang Xu, Parker Koch, and Jason J Corso. Watch what you just said: Image captioning with text-conditional attention. In Proceedings of the Thematic Workshops of ACM Multimedia 2017, pp. 305-313. ACM, 2017.
Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly-supervised video object grounding from text by loss weighting and object interaction. In British Machine Vision Conference (BMVC), 2018.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. Grounded video description. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEFE International Conference Computer Vision (ICCV) pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2020
Grounding
Captioning Evalusioo
Grounding Evaluation
Method
supervision
B@1
BB4
C
Flat
Flix
Flav
Unrealistically perfect object detector
(-100%)
(+100%)
(+100%)
75.1
76.3
22.0
(4056)
48 (4086
(4055)
77.81 (441%)
Cyclical
76.7
32.8
25.8
80.2
22.7
25.27 (+188%)
54.54 (+142%)
46.98 (+46%)
81.56 (+29%)
Grounding- biased object detector
Boseline
65.9
23.4
(+1001)
23,95 (+10006)
(+100%)
Baseline
(+08))
(+0%)
(+0%)
Cyclical
65.5
23.3
21.2
52.0
15.4
6.87 (+405))
19.65 (+339)
20.82 (427%)
50.25 (+31%8)
Table 4: Grounding performance when using better object detector the Flickr30k Entities test set (results are averaged three runs). Fully-supervised method (Sup.) is used as upper bound, thus its numbers are not bolded.
Ground-truth:
Predicted:
man is hiking while holdingg water bottle. wearing and hiking shoes hiking.
man wearing a hat hiking with dog.
Object words: (man, bottle, hat, shaes)
Object words: [man, hat, dog)
Grounding Metrics
Grounding
Aceraged pir object
Averaged generated unteence
Precision pent
0.33
A
E
E
(man, hat, dog)
Precisionai
Recall
B
Recall ail.er sont
0.25
B: [man, bottle, hat, shoes)
Flali 0.10
C: (man, har]
(man, hat)
E: (man) (|oU> > 0.5)
Precision per Jnd C
0.5
Precision
Recalloe
Recal per sent
Flaxe per sent
Figure 6: Illustration of Grounding metrics.
A APPENDIX
A.1 GROUNDING EVALUATION METRICS ILLUSTRATED
To help better understand the grounding evaluation metrics used in this work, we illustrated the grounding evaluation metrics in Figure
We define the number of object words in the generated sentences A, the number of object words in the GT sentences as B, the number of correctly predicted object words the generated sentences as C and the counterpart in the GT sentences as D, and the number of correctly predicted and localized words as (see illustration considered correct if the object word is correctly predicted and also correctly localized (i.e., IoU with E
the grounding metrics in Appendix A. A region prediction is
GT box > 0.5). We then compute two version the precision and recall as Precau A' Recall
E c
E
Prec
and Recwe
The original grounding evaluation metric proposed in GVD (Zhou et al., 2019) average the grounding for each object class. We additionally calculate the grounding accuracy for each generated sentence demonstrated in the figure. From this example, a wrong prediction for the dog object class, the Precision,,r only cares if man and hat are predicted and correctly localizer (loU > 0.5).
that while Precisional counts dog as
13

---

Under review as a conference paper at ICLR 2020
Captioning Eval.
Grounding Eval.
M
S
Flai
Fla
Baseline (Unsup.)
22.3
62.1
16.0
4.18
11.9
Cyclical
22.2
62.2
16.2
5.63
14.6
consistency
4.19
Localizer using
22.2
61.8
16.1
4.58
11.3
Table 5: Model ablation study on the Flickr30k Entities val set.
Cuptioning Evaluation
Grounding Evaluation
Method
B@l I
B@4
M
C
S
Fl
Flie
Flalt per sen
Flue sen
Baseline
69.1
26.0
22.1
59.6
16.3
4.08
11.83
13.20
31.83
Cyclical
69.4
26.9
22.3
60.8
16.6
5.11
14.15
15.15
35.56
Cyclical (1)
69.7
27.0
22.2
60.1
16.5
5,14
14.32
15.36
36.33
Cyclical (2)
69.9
27.5
22.4
62.0
16.6
5.13
13.99
16.30
38.45
Table 6:
Performance comparison on the Flickr30k Entities test set. All results are averaged across five runs.
A.2 ADDITIONAL ANALYSIS
Should we explicitly make attended image regions to be similar to localized image regions? One possible way regularize the attention mechanism of the decoder is to explicitly optimize f + fl via KL divergence over two soft-attention weights a and B.. The experimental results shown in Table (Attention consistency). We use a single run unsupervised baseline with fix random seed as baseline model for ablation study. We can see that when explicitly forcing the attended regions be similar the localized regions, both the captioning performance and the grounding accuracy remain similar to the baseline (unsup.). We conjecture that this is due to the noisy localized regions at the initial training stage. When forcing the attended regions to be similar noisy localized regions, the Language LSTM will eventually learn to not rely on the attended region at each step for generating sequence of words. To verify, we increase the weight for attention consistency loss and observed that it has lower grounding accuracy (FI,n = 3.2), but the captioning will reach similar performance while taking 1 1.5x longer to reach convergence.
Is using only the generated word for localization necessarily? Our proposed localizer (Eq. 6 and Figure 3) relies on purely the word embedding representation to locate the image regions. This forces the localizer to rely only the word embedding without biasing it with the memorized information from the Attention LSTM. As shown in the Table (localizer using A),, although this achieves comparable captioning performance, it has lower grounding accuracy improvement compared to our proposed method.
Can words that not visually-groundan handled differently? In the proposed method, all the words are handled the same regardless of whether they are grouudyy- or not. Yet. typically words that are nouns or verbs are more likely be grounded, and words like "a". "the", etc, are not visually-groundable.
We explored a few method variants handle nouns and verbs differently. Mainly, we explored with two variants. Cyclical (I): the reconstruction loss is only computed when the target word is either nouns or verbs. Cyclical (2): the localized region representation will be invalid (set to zero) the target word is neither nouns nor verbs.
The experimental results are shown in Table 6, 7, and For the first variant, Cyclical (1), we observed that the captioning performance stays the same while grounding accuracy has small improvement. On the other hand, for the second variant, Cyclical (2), we can see that all captioning scores are improved over baseline with CIDEr improved 2.4. We can also that grounding accuracy on per sentence basis further improved as well. We then conducted further experiments on both Setivityyee-Entittee and Flickr30k Entities with unrealistically perfect object detector, but the improvements however are not consistent. summary: on the Flickr30k Entities test set, we observed that CIDEr is better and grounding per sentence better, the ActivityNet- Entitiis val set,
14

---

Under review as a conference paper at ICLR 2020
Captioning Evaluation
Grounding Evaluation
Method
B@I
B@4
S
Flall
Flioc
Flali_per__e sear
Fliee_errssen
Baseline
23.2
2.22
10.8
45.9
15.1
3.75
12.00
9.41
31.68
Cyclical
23.7
2,45
11.1
46.4
14.8
4.68
15.84
12.60
44.04
Cyclical (2)
23.9
2,58
11.2
46.6
14.8
4.48
15.01
11.53
40.30
Table 7: Performance comparison the ActivityNet--nntite val set. All results averaged across five runs.
Captioning Evaluation
Grounding Evaluation
Method
B@1
B@4
M
C
S
Flatl
Fl
Flall per
Fl per
Unrealistically perfect object detector
Baseline
75.1
32.1
25.2
76.3
22.0
20.82
48.74
43.21
77.81
Cyclical
76.7
32.8
25.8
80.2
22.7
25.27
54.54
46.98
81.56
Cyclical (2)
75.8
32.2
25.6
79.0
22.4
25.65
55.81
48.99
85.99
Table 8:
Grounding performance when using better object detector the Flickr30k Entities test set (results are averaged three runs).
the captioning performances are about the same but grounding accuracy became worse, and on the Flickr30k Entities test set with unrealistically perfect object detector, captioning performances are slightly worse but grounding accuracy improved.
A.3 HUMAN EVALUATION GROUNDING
We conduct human evaluation on the perceptual quality of the grounding. We asked 10 human subjects to pick the best among two grounded regions (by baseline and Cyclical) for each word. The subjects have three options to choose from: 1) grounded region A is better, 2) grounded region B better, and 3) they are about the same (see Figure 7 for example). Authors or other colleagues familiar with the proposed method were excluded from the study. Each of the human subjects given 25 images, each with a varying number of groundable words. Each image was presented to two different human subjects in order to able measure inter-rater agreement. To avoid being biased towards the object words defined in the dataset for automatic grounding evaluation, for the study we define word be groundable it either a noun or verb. The order of approaches was randomized for each sentence.
Our experiment on the Flickr30k Entities val set showed that: 28.1% of words more grounded by Cyclical, 24.8% of words are more grounded by baseline, and 47.1% % of words are similarly grounded.
We also measured inter-rater agreement between each pair of human subjects: 72.7% of ratings are the same, 4.9% of ratings are the opposite, and 22.4% of ratings could be ambiguous (e.g.. one chose better, the other chose they are about same).
We would also like to make note that the grounded words judged to be similar largely consisted of very easy or impossible cases. For example, words like mountain, water, street, etc, are typically rated to be "about the same" since they usually have many possible boxes and is very easy for both models to ground the words correctly. On the other hand, for visually ungroundable cases, e.g., stand appears a lot and the subject would choose about the same since the image does not cover the fact that the person's 's feet are on the ground.
We that the human study results follow the grounding results presented in the paper and show an improvement in grounding accuracy for the proposed method over a strong baseline. The improvement achieved without grounding annotations extra computation test time.
15

---

Under review as a conference paper at ICLR 2020
A man and woman, each wearing white shirt, walking down the sidewalk.
shirt
sidewalk
A
man
shirt
sidewalk
B
is
A is better.
is better.
B is better.
B is better.
and are about the same
A and B are about the same
and about the
Figure Demonstration our human evaluation study on grounding. Each human subject is required to rate which method (A B) has better grounding each highlighted word.
A man red shirt standing wooden pletform.
A men In yellow lacket and blue helmel
A man an orange shirt and
man sits on a chair in frant of lake.
hal standing next to a blue wall
gid purple sweater jumping mokks.
gir pink shirt and waiking down brick wall,
An Aslan woman holdingg red walking sidewalk.
man black shirt is holding up a flag.
Figure 8: Correct (top) examples and examples with errors (bottom) from the proposed method.
A.4 ADDITIONAL. QUALITATIVE RESULTS
In Figure 8, we show a number of correct and incorrect examples of our proposed method. We observe that while the model is able generate grounded captions for the images, it may sometimes overlook the semantic meaning of the generated sentences, for example, "A young girl [...] walking down a brick wall". Similarly, the model can overlook the spatial relationship between the objects, for instance, "A man [...] is holding up flag". While flag is present in the scene and able to be successfully located with the corresponding word, the man a black shirt is spatially far from the flag.
In Figure 9, 10, 11, 12, 13, 14, 15, and 16, we illustrated the sequence of attended image region when generating each word for complete image description. At each step, only the top-1 attended image region is shown. This is the same as how the grounding accuracy measured. Please see the description for Figure 9 16 for further discussions the qualitative results.
16

---

Under review as a conference paper at ICLR 2020
A.5 ADDITIONAL IMPLEMENTATION DETAILS
Region proposal features. We use Faster-RCNN model (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017) for region proposal and feature extraction. In practice, besides the region proposal features, we also use the Conv features (comv4) extracted from an ImageNet pre- trained ResNet-101. Following GVD (Zhou al., 2019), the region proposals are represented using the grounding aware region encoding, which is the concatenation of i) region feature, ii) region-class similarity matrix, and iii) location embedding.
For region-class similarity matrix, we define a set of object classifiers Wes and the region-class similarity matrix can be computed as M, = softmax (W R). which captures the similarity between regions and object classes. We omit the ReLU and Dropout layer after the linear embedding layer for clarity. We initialize We using the weight from the last linear layer of an object classifiers pre-trained on the Visual Genome dataset (Krishna et al., 2017).
For location embedding, we use 4 values for the normalized spatial location. The 4-D feature is then projected to dg 300-D location embedding for all the regions.
Software and hardware configuration. Our code is implemented in PyTorch. All experiments were on the 1080Ti, 2080Ti, and Titan Xp GPUs.
Network architecture. The embedding dimension for encoding the sentences is 512. We use dropout layer with ratio 0.5 after the embedding layer. The hidden state size of the Attention and Language LSTM are 1024. The dimension of other learnable matrices are: We E gd.x512 W. R 1024x512 W. eu E R512x1 W. R 1024xd, Wi R512x612 where the vocabulary size d, 8639 for Flickr30k Entities and 4905 for ActivityNet--ntiite
A.5.1 TRAINING DETAILS.
The hyper-parameters \ and X are set after tlyper-paramete search between and 1.
Flickr30k Entities. Images are randomly cropped to 512 x 512 during training, and resized to 512 x 512 during inference. Before entering the proposed cyclical training regimen, the decoder was pre-trained for about 35 epochs. The total training epoch with the cyclical training regimen is around 80 epochs. The total training time takes about 1 day.
AetivityNet--nniitee Before entering the proposed cyclical training regimen, the decoder pre- trained for about 50 epochs. The total training epoch with the cyclical training regimen is around 75 epochs. The total training time takes about 1 day.
17

---

Under review as a conference paper at ICLR 2020
with
croad
wotchirg g
Figure 9: A group of in white uniforms are slanding field with a crowd watching. We that our proposed method attends to the sensible image regions for generating visually-groundable words, e.g., man, uniforms, field, and crowd. Interestingly, when generating standing, the model pays its attention on the image region with a foot on the ground
yourg
rid
wariin
winter
hat
and
purple
cost
is
smiling
the
camera
Figure 10: young girl wearing winter hat and a purple coat is smiling the camera. The proposed method is able to select the corresponding image regions to generate girl, hat, and coat correctly. We have also observed that the model tends localize the person's 's face when generating camera.
18

---

Under review as a conference paper at ICLR 2020
helmeet
and
shirt
turdle
Figure 11: A white horse with rider blue helmet and white shirt jumping over hurdle. While the model is able to correctly locate objects such as horse, rider, helmet, shirt, and hurdle, it mistakenly describes the rider as wearing a blue helmet, while it's actually black, and with white shirt while it's blue.
man
red
shirt
standing
acoden
platform
Figure 12: man in a red shirt standing on a wooden platform. Our method correctly attends on the correct regions for generating man, shirt, and platform.
19

---

Under review as a conference paper at ICLR 2020
yellw
jocket
Figure 13: A man in yellow jacket and blue helmet riding a bike. The proposed method correctly generates descriptive sentence while precisely attending to the image regions for each visually- groundable words: man, jacket, helmet, and bike.
Shir
Figure 14: A man in an orange shirt and hat is standing next to blue wall. While our method is able to ground the generated sentence on the objects like: man, shirt, hat, and wall it completely ignores the person standing next to the man in the orange cloth.
20

---

Under review as a conference paper at ICLR 2020
sii
white
shir
and
black
jumping
Figure 15: A girl in white shirt and black pants is jumping on a red couch. Our method is able to
ground the generated descriptive sentence with the correct grounding on: girl, shirt, pants, and couch.
bur
rob
walks
down
cobblestone
Street
Figure 16: A man in blue robe walks down a cobblestone street. Our method grounds the visually- relevant words like: man, robe, and street. We can also see that is able to locate the foot on ground for walks.
21


Reviews:
[{"review_id": "SJeXvXSl9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an architecture that grounds words from a captioning model, but without requiring explicit per-word grounding training data. Instead, they show that it is sufficient to use cycle consistency, verifying that by predicting word->grounding->word the two words are the same. \n\nGeneral: \n\nCycle consistency has been shown to be very useful in replacing explicit paired data, for eample in image-to-image translation (CycleGAN, or the more recent FUNIT). This paper takes it to the domain of vision and language. While the novelty is not very large it seems like a solid step in an interesting direction.  Evaluated on both image and video captioning with substantial localization improvement in the specific relevant eval settings. \n\nSpecific comments: \n\n-- In Table 1, the part of \"Caption Evaluation\" the proposed method is in bold, but it seems that \"Up-Down\" method out-performs the proposed method in B@1 and B@4. \n\n-- Are words that are not nouns/verbs (the/a/are/with/etc) handled differently? It doesn't really make sense to localize them just like object words. \n\n-- The localization model is linear? What would be the effect of richer models on localization accuracy? \n\n-- Qualitative analysis: It would have been useful to add evaluations by human-raters to measure the perceptual quality of the localization. \n\n-- Error analysis? examples and analysis  of failure cases? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg1_5l1cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses captioning generation for images and videos, by proposing a novel cyclical training regimen consisting of three steps: decoding, localization, and reconstruction. The experimental results show that the performance on image captioning and video captioning are improved without grounding supervision.\n\nI lean to accept this paper. The motivation using cyclic feedback itself is not so novel for language generation, but focusing on grounding without localization supervision for visual captioning is interesting. The experimental results show that the proposed method can boost performance both qualitatively and qualitatively. I have several comments and questions below.\n- Why do the authors introduce GVD without self-attention as a baseline? Table 1 and 2 show that removing self-attention degrades the performance. If the combination of self-attention in GVD and cyclical training proposed in this paper is complementary to each other, it does help to improve the overall accuracy.  \n- While the authors develop a cyclical training pipeline, including decoding, localization, and reconstruction, Figure 1 does not show which part corresponds to the decoding phase. The authors should clarify it to make the paper easier to be understood.\n- Equation (5) seems to be strange. $\\theta^*$, a sum of two parameters for each arg max operator, doesn't guarantee that each term in the right side of Eq. (5) keeps its max. This equation seems to be a conceptual one, and the actual training would be performed according to Eq. (7). Therefore, the experimental results might not be influenced by the error in Eq. (5).\n- $\\hat{r}^l_t=\\beta_t^\\top R$ between Eq. (6) and Eq. (7) means that $\\hat{r}^l_t$ is a row vector while $r_n$ seems to be a column vector. Since $R = [r_1, r_2, ..., r_N]$ for $N$ regions, $\\hat{r}^l_t= R \\beta_t$ seems to be appropriate. The authors should correct it. I have a similar comment for $\\hat{r}_t = \\alpha_t^\\top R$ in Eq. (2).\n- In the caption of Table 5, the number equal to or smaller than ten should be spelled out; \"5 runs\" should be \"five runs.\" There are similar errors, such as \"5 GT captions\" and \"1 GT caption\" in Sec. 4.\n- The format of items in References is not consistent.\n- According to Sec. A.4.1, $\\lambda_1$ and $\\lambda_2$ are tuned between 0 and 1. How are the experimental results sensitive to these hyperparameters? Additional experiments using different $\\lambda_1$ and $\\lambda_2$ would be helpful.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygE12YaKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# 1. Summary\nThe paper deals with the problem of learning grounded captions from images without joint text-location information, but instead texts (words in captions) and locations are provided independently and the model needs to figure out their link. The model is built upon GVD (Zhou et al., 2019): each word generated by the encoder is grounded to the locations provided by the region proposal module (Up-Down model (Anderson et al., 2018)), used to reconstruct the ground-truth caption.     \n\nMy weak reject decision was guided by the following strengths and weaknesses of the paper.\n\nStrengths:\n* The reconstruction formulation of the problem is interesting and relevant for the task\n* Proposed new metric to measure grounding performance\n      \nWeaknesses:\n* Questionable motivations: it is not clear what application is grounding text to the image useful for?\n* Marginal (not statistically significant?) improvement on image captioning metrics by using grounded text (proposed method) compared to not grounding (GVD)\n* Limited novelty: extension of GVD, where attention is removed and object locations are used instead\n     \n      \n# 2. Clarity and Motivation\nThe paper is generally well written, however there are some concerns on motivations. \n\nOne concern is related to the motivations of the paper. The authors use grounding as a proxy to improve image captioning results, which improvement is marginal wrt GVD (see Table 1). Why do we need to localize text if this has very marginal impact on the captioning metrics? It is missing the link between the potential applications where the localization of words is relevant. \n\nThe authors claim that they do not uses any grounding annotation, however the pre-trained Faster-RCNN has been trained using annotations which consist of bounding boxes + categories. Therefore, the model do (in an implicit way) rely on grounding annotations, especially because there might be an overlap between the words (classes) used to pretrain the detector and the words in the captions. The authors should assess if this ovelap/bias in the pre-trained Faster-RCNN exists or not.\n\nSome other questions are still to be answered:\n* How are the regions R parametrized? Is it the visual representation or bounding box locations?\n* What happens to words that are not grounded to the image (e.g., verbs or articles)? Do you have a special way to deal with those?\n* What is the intuition of multiplying word embedding and region embeddings to generate z in Eq. 6? \n\n\n# 3. Novelty\nThe proposed method is an extension of the existing model GVD, where the attentional module is removed and its functionality is replaced by the cyclical training mode with reconstruction of the object locations. From the technical point of view this is limited novelty, but still an interesting improvement of the model; however the experiments and results do not support the claim that using such model improves image captioning result in a significant way. One way to answer to this question would have been by showing an application where the outputted locations are used for downstream tasks.\n\n\n# 4. Experimentation\nThe experiments are carried out in a scrupulous way, by showing the comparing with GVD (with and without attention; with and without grounding supervision). The non-convincing part of them (as mentioned above already) is the fact that the improvements on these datasets might be non significant for image captioning. For example, let's consider the image captioning results in Table 1 (Flickr30k Entities): cyclical have a max improvement of 0.7 (CIDER) and min of 0 (B@4) when compared with GVD without grounding supervision. There is an obvious huge improvement on the grounding evaluation, which is obvious since GVD does not do it explicitly. The same trend is in Table 2. \nThese results are not convincing, combined by the fact that it is not clear in which applications one would want a very accurate grounded text.\n\n\n# Minor Points\n* Sec. 3.1: it is not clear that the Language LSTM is the decoder. Please explicitly say it before Eq. 1\n* Caption of Table 3: which dataset is this?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a cyclical training scheme for grounded visual captioning, where a localization model is trained to identify the regions in the image referred to by caption words, and a reconstruction step is added conditioned on this information. This extends prior work which required grounding supervision. 

While the proposed approach is sensible and grounding of generated captions is an important requirement, some reviewers (me included) pointed out concerns about the relevance of this paper's contributions. I found the authors’ explanation that the objective is not to improve the captioning accuracy but to refine its grounding performance without any localization supervision a bit unconvincing -- I would expect that better grounding would be reflected in overall better captioning performance, which seems to have happened with the supervised model of Zhou et al. (2019). In fact, even the localization gains seem rather small: “The attention accuracy for localizer is 20.4% and is higher than the 19.3% from the decoder at the end of training.” Overall, the proposed model is an incremental change on the training of an image captioning system, by adding a localizer component, which is not used at test time. The authors' claim that “The network is implicitly regularized to update its attention mechanism to match with the localized image regions” is also unclear to me -- there is nothing in the loss function that penalizes the difference between these two attentions, as the gradient doesn’t backprop from one component to another. Sharing the LSTM and Language LSTM doesn’t imply this, as the localizer is just providing guidance to the decoder, but there is no reason this will help the attention of the original model. 

Other natural questions left unanswered by this paper are:
- What happens if we use the localizer also in test time (calling the decoder twice)? Will the captions improve? This experiment would be needed to assess the potential of this method to help image captioning.
- Can we keep refining this iteratively?
- Can we add a loss term on the disagreement of the two attentions to actually achieve the said regularisation effect?

Finally, the paper [1] (cited by the authors) seems to employ a similar strategy (encoder-decoder with reconstructor) with shown benefits in video captioning.

[1] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7622–7631, 2018.

I suggest addressing some of these concerns in a revised version of the paper.
Decision: Reject