Learning Compact Reward for Image Captioning
Authors: Nannan Li
Zhenzhong Chen
Venue:  (2020)
Keywords: image captioning
adversarial learning
inverse reinforcement learning
vision
language

Abstract: Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.

Under review as a conference paper at ICLR 2020
LEARNING COMPACT REWARD FOR IMAGE CAPTION- ING
Anonymous authors
Paper under double-blind review
ABSTRACT
Adversarial learning has shown its advances in generating natural and diverse de- scriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in sentence, well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.
INTRODUCTION
Image captioning is task of generating descriptions of given image in natural language. In general encoder-decoder structure (Vinyals et al. 2015), image features are encoded in CNN and decoded into caption in word by word manner. Based the loss function, standard approaches to the problem could be divided into three categories: MLE (Maximum Likelihood Estimation), RL (Reinforcement Learning) and GAN (Generative Adversarial Network).
Early proposed methods were based on MLE function and made improvements by designing specific model structure (Xu al. 2015). MLE adopts the cross-entropy loss and learns a one-hot distribution for each word in the sentence. By maximizing the probability of the ground truth word whilst suppressing other reasonable vocabularies, the probability distribution learned by MLE tends to be sparse and the generated captions have limited diversity (Dai et al. (017). On the other hand, RL has advantages in boosting the model performance by optimizing the handcrafted metrics (Rennie et al., 2017; Liu et al.) 2017;; Chen al. 2019). However, due to the reward hacking problem, RL maximizes the reward in an unintended way and fails to produce human-like descriptions (Li let al. 2019a). Considering naturalness and diversity of the generated captions, GAN has raised attention in image captioning for its capability of producing descriptions that are indistinguishable from
ones DDai et al. 2017 Shetty al. 2017 Chen et al. 2019 Dognin et al. 2019).
In image captioning, the generator of GAN learns true data distribution by maximizing the reward function learned from discriminator, and the discriminator distinguishes the generated sample from the true data. The adversarial training converges to an equilibrium point (i.e., Nash equilibrium) at which both the generator and discriminator cannot improve (Goodfellow al. 2014).. As shown in Figure [] the learned distribution of GAN is closer to the ground truth distribution than that of other methods (i.e., .LEE and RL) on different splits However, previous work of adversarial networks in image captioning gives one reward function D for a complete sentence consisting of n words. This strategy causes the reward ambiguity problem (Ng et al., 1999) since which word(s) causes the reward to increase or decrease is not accounted for, and thus there are many optimal policies that determine the sentence can explain one reward. An example that each image in MS COCO has five ground truth captions. Although these captions may vary in formats, each caption has the same reward value in GAN. From the perspective on the system level, learning sentence- reward from different image-caption pairs is analogous learning reward of trajectory from different system dynamics, which makes the discriminator unable to distinguish the true reward functions from those shaped by the environment dynamics (Fu al. 2018).

---

Under review as a conference paper at ICLR 2020
430
200
131
(a) Standard split (Karpathy Fei-Fei 2015)
(b) Robust split et al. 2018
Figure 1: Comparison of word count ratios Shetty et al. 2017 on two splits of MS COCO. x axis is the [test frequency) // (ttrain frequency) } of a word and y axis is the word count of the corresponding ratio. GT represents ground truth distribution.
Facing the challenge, we adopt AIRL (Fu al. 2018) to solve the reward ambiguity problem by disentangling reward for each action (i.e., word in a sentence) from different image-caption pairs and learning a compact reward function. compact means words with similar semantics, such as children and kids, correspond to close reward values. Driven by the compact reward function of the discriminator, the generator learns the optimal policy and thus produces qualitative descriptions. However, there still two major problems to address: 1) AIRL is difficult to converge Nash equilibrium using policy gradient (See Section 4.2 for details); 2) AIRL is designed without mode control, and thus the outputs have limited diversity, which is commonly encountered issue called mode collapse (Mirza & Osindero 2014).
In this paper, we propose refined AIRL method to learn compact reward function for each word, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. The refined method makes possible to reach the equilibrium point for non-concave model function of the generator. In addition, a conditional term is introduced in the loss function to mitigate mode collapse and to increase the diversity the generated descriptions. Both the caption evaluator (i.e., discriminator) (Cui et al., 2018 Sharif et al. [201 8) and the generator are cast into this unified framework, where the discriminator evaluates captions using a learned compact reward function, and the generator produces qualitative image descriptions. We demonstrate the effectiveness of our method in the experiments.
RELATED WORK
Image Captioning. The development of image captioning can be summarized into two directions:
model structure design (Lu et al. 2017 Yao et al. (0188) and loss function construction (Rennie et al. 2017 Ren et al., 2017).
the methods based on model structure design, attention mechanism and
the fusion of visual and semantic information are the key focus. [u al. (2017) 2018) proposed
sentinel gate to learn adaptive attention between visual content and non-visual text. Yao et al. (2018) explored the role of visual relationship in image captioning. On the other hand, methods based on loss function construction focus on optimization of the loss function. Rennie et al. (2017) optimized on non-differentiable evaluation metric using policy gradient, and improved scores of these metrics on various models. |Ren et al. (2017) designed an embedding reward under actor-critic reinforcement learning. Similarly, we address the construction of loss functions, and thus our algorithm can be built on existing model structures. See Appendix F for short discussion about different loss functions.
Adversarial Methods for Image Captioning. Adversarial methods are known for producing plau- sible samples by training the generator and the discriminator in an adversarial manner (Goodfellow) cet al., 2014). In image captioning, the discriminator is formed as a binary classifier that distinguishes the generated sentence from the ground truth, while the generator produce captions that can fool the discriminator. Conditional GAN was proposed in (Dai et al. 2017) to improve the naturalness and diversity of generated captions. CNN and RNN based discriminators 2019). However, existing methods estimate a reward function for the complete sentence consisting of words, where multiple optimal policies that determine the sentences can correspond to reward
introduced in (Chen et al.
2

---

Under review as a conference paper at ICLR 2020
Ng et al. 1999). Thus the learned reward is ambiguous and ill-defined. We solve this problem by recovering a compact reward function for each word in the sentence under refined AIRL framework. Although AIRL has been utilized to solve problems in other fields (Wang al. 2018: Li al.) 2019b, Shi et al. 2018), we are the first to make algorithmic improvements to AIRL such that Nash equilibrium can be reached even for non-concave model function of the generator, and that diversity of the outputs can be increased.
3 ADVERSARIAL INVERSE REINFORCEMENT LEARNING
Due to the high variance estimate of a full sentence and the reward ambiguity problem, instead of learning reward for complete sentence, we could learn reward distribution pal (., 5r) for each word-state pair (Wr, 8t) so that the true reward can be recovered at optimality (Fu et al. 2018). 8) the following, introduce how AIRL disentangles reward for each word-state pair (w,, 8t).
AIRL an adversarial reward learning algorithm based on Maximum first proved that Maximum Maximum-Entropy- Entropy-IRe -RL is mathematically equivalent to GAN under special form of the discriminator:
Maximum-Entropy--e Entropy-IRL. Finn et al. (2016)
Po We, St)
Do (We, 8t) = po po(We: 82) + π (W:,
(2) where (o st) is the actual probability distribution estimated by the discriminator. <(w,, 8:) is the policy distribution produced by the generator, which also called vocabulary distribution under the context of image captioning.
po (wr, 8e) = exp ( 8t))
The goal estimate po(wy, 8t) that approximates the true data distribution Piue (wre 8t), as well as to learn optimal vocabulary distribution æ that maximizes the reward. Subsequently, considering reward ambiguity problem, [Fu et al. (2018) further extended the theory to AIRL by adding a reward shaping term hep into fe(we, St), and thus disentangled reward from different system dynamics (i.e., image-caption pairs):
fo 8t) = go(we,
(W,, $1; $(+1) + 7/pp($+11) - (sp)s:)
(3)
where go denotes the reward approximator that recovers the true reward up to a constant, and hep is the reward shaping term that preserves the optimal y is constant in range (0, 1].
In the context of divergence minimization, the adversarial process can be represented as a min-max game (Mescheder & Geiger 2017):
min max Buy Dlog (Da.p(w tue " _.4o))] Emn (log (1 Du.p(wr: (w))))))) 0.P
where Ptrue is the true data distribution and π is the vocabulary distribution learned by the generator. (witee .s is the word-state pair of the true data.
Despite of the capability of AIRL in disentangling reward for each word, it is difficult for the above AIRL algorithm to converge to Nash equilibrium and to produce diverse outputs through adversarial training (See Section 4.2 for details). These issues can result in non-optimal solution and lack of diversity of the generated descriptions. Aiming to learn the optimal compact reward as well as diverse captions, we refine the loss function to shift the stationary point towards Nash equilibrium and to mitigate mode collapse in the two-player game.
4 LEARNING COMPACT REWARD FOR IMAGE CAPTIONING
To address the problems discussed above, we refine the loss function to: /) find a compact reward function that is optimal; 2) increase diversity of the generated captions. In particular, constant term used to solve 1) by shifting the stationary point to Nash equilibrium, and a conditional term is introduced to solve 2) by utilizing mode control techniques. Our algorithm is detailed in Algorithm 1, where n is the sentence length and N denotes number of iterations.
In the following notations, 0 and oo are the parameters of the discriminator, and & represents the parameter the generator. w. and s, denote the frh. word and its corresponding hidden state vector, respectively. For better clarity, policy मe is hereinafter referred to vocabulary distribution.
3

---

Under review as a conference paper at ICLR 2020
Algorithm I: Refined ARIL
Initialize the vocabulary distribution Te. and discriminator fe.p-
for iteration in {1,.., N) do
Obtain caption {wf ,..., w from the ground truth.
Collect generated caption {wi,... Wn using the vocabulary distribution Ti.
D8 sigmoii fe, [T)(
Update (0, via Eq.
5 for the discriminator.
Update vb via Eq.
16 for the generator.
end
4.1 DISCRIMINATOR
The objective of the discriminator is to distinguish the true caption from the generated one. At time t, the discriminator maximizes the divergence in Eq. (4) by
(0, .p) = -olog (w true
log De. (w,, $t)) w.--
(5)
where Pine is the distribution of the true caption, and ne is the vocabulary distribution estimated by the generator. Da computed as Eq. ①1 and Eq. ( where the discriminator learns the reward function fe.p for De.pp and the generator estimates the vocabulary distribution T. for Do.e- respectively.
4.2 GENERATOR
In the following, Da is represented as below (Fu et al. 2018) using Eq. (I) and Eq.
(w, 92) sigmoid (fo. (w), 8.) log(m)))
(6)
Given word we that sampled from the vocabulary distribution Ty. the generator maximizes (o, s,) by
Ia(+) = - Bro, way (log (Do (w., s:)) - log Da (wr,
= BBrr ~T (f (w,, $t) log((T;)))
Using REINFORCE algorithm (Sutton Barto 1998). the gradient V La becomes:
Vela = Σ (fosp(ue,, st) - log(( V + t V (Jo.p (we, 8t) - log(T+))
=-En Tv. To 1
(fo. st) log T.)
V uu + V (f
(We, 31)
log(TT))
Teb
(8)
(fo,p(ue, log(
T
V
(fo st) log(TT))
Tp
(fo st) log( - V 거 Tt.
When the generator converges (i.e., V = 0), there exists two stationary points: V. = 0 and log(m.) = fo.p (w). st) 1. If Nash equilibrium can be reached at optimality, Da.p should converge to 0.5 when L. = O. Thus it's only possible for the first point to reach Nash equilibrium since Da.e = sigmoid (Sp.o [e.p 8.) log(TT,)) -oggm)(Tm)) = sigmoid(1) f 0.5 the second point. However, even for the first point, Nash equilibrium exists only for concave xØ. requiring Hessian of the gradient vector filed being positive definite (Mescheder & Geiger 2017). To relax this constraint, a constant
term is added into the expectation in Eq. (7)
(+(1) Eno, [fo (We, se) log(m²) + 1]
(9)
1 Tw
V V L
(wr, St) log(m V "
(10)
to expand the feasible region of reaching Nash equilibrium by shifting the second stationary point to (w:) sigmoid(0) 0.5. The constant term makes it possible achieve Nash equilibrium
4

---

Under review as a conference paper at ICLR 2020
for non-concave To, which also makes it easier for the adversarial model to converge... It is noted that the constant term can also be regarded as baseline in REINFORCE, except it is utilized to centralize the stationary point instead of reducing variance of the estimation.
In practice, mode collapse occurs when the generator produces a single or limited modes, which exhibits little diversity in image captioning. To mitigate mode collapse (Mirza & Osindero 2014 and increase the diversity of the generated captions, we add ground truth data into the generator as a conditional term:
L (16))= - -u I (wi, st) (Tp) ) + +1 E Bu up up( (w/ sS( log( Me = (Soopp (W2, St) log(T log( (Tx)) - (Ja true ,)) - log(m((1)) log( (T
+ 1]
(1I)
where T is the approximated true caption distribution in the generator, and I Euguerii ( is the con- ditional term. See Appendix G for the deduction details. The coefficient of log(m is symmetrical to the coefficient of (og(TT) and is updated adaptively in the training process. The conditional term helps in strengthening the generator in the adversarial training. When Dirue D Dgen the gradient of the true data becomes larger than that of the generated one (V xap Le > VngLi and thus the generator further increases the probability of the true word (mi))) Otherwise (i.e., Dime Dgen). the generator prefers sampling its self-generated words to fool the discriminator. By switching between the true words and the generated ones, the generator maintains informative gradient during the adversarial training (Peng al. (219). Note that adding the conditional term does not change the model's 's convergence to Nash equilibrium since T T
true at the second stationary point.
EXPERIMENTS
In the experiments, we validate the effectiveness of the proposed algorithm by answering three questions: 1) Is the caption evaluator (i.e., discriminator) capable of learning compact reward? 2) Driven by the learned reward, is the caption generator effective to produce qualitative captions? 3) How does our algorithm perform when built on or compared with existing methods?
To answer 1), we first tested the compactness of the Iearned reward by observing performance of the collected top-k captions. Then we explored the correlation between the Iearned reward and the human evaluation results. To answer 2), we evaluated the quality of the generated caption on its content, diversity and grammar. To answer 3), we built our algorithm on existing learning methods and compared their performance. We also conducted ablation experiments to demonstrate the importance of each component of our algorithm.
5.1 IMPLEMENTATION DETAILS
We conducted experiments the well-known benchmark dataset MS COCO (Chen al. 2015). The dataset has 123,287 labeled images and each image has at least human annotated captions as reference. To assess the robustness of our algorithm, we use two splits of the COCO dataset: standard split (Karpathy & Fei- 2015) which is created by randomly picking test images, and robust split (Lu et 20118) which is organized maximize difference of the co-occurrence distribution between the training and test set. The robust split is recently proposed and is more challenging due to its distribution difference between the training and test set. The standard split has (13287/5000/255 train/val/test images and the robust split has 110234/3911 /9138 train/val/test images.
We implement our algorithm using Adam optimizer (Kingma Ba 2014) with fixed learning rate 10 Our vocabulary size is fixed to 10,000 including a special start sign <BOS>am an end sign <EOS>. In the generator, the number of hidden nodes of every layer is 512. For simplicity, the discriminator has the same model structure as the generator except having one additional layer for estimating htp For fair comparison, all the methods in MLE, RL, GAN, AIRL and rAIRL were produced using the same image features and model structure in (Anderson et al. 2018). Specifically, RL is the self-critical algorithm in (Rennie et al. (017). GAN is the adversarial algorithm in (Dai & Lin 2017) that learns sentence-level reward. AIRL the standard adversarial inverse reinforcement learning method in (Fu et al.) 2018). and rAIRL is the proposed method. Note that our scores of MLE lower on the standard split but higher on the robust split than (Anderson al. 2018) because
5

---

Under review as a conference paper at ICLR 2020
1) we used fixed number of the bounding box (i.e., 36) for simplicity; 2) the hyperparameters were tuned to adapt to both splits and thus are not exactly the same with (Anderson et al. 2018
5.2 PERFORMANCE OF THE RECOVERED REWARD
Table 1:
Correlation between the reward differences and semantic differences by replacing given word with a similar word (RP_ () and distinct word (RP ___,, respectively.
Standard Split
Robust Split
Method
RP S
RP
RP
RP D
RL
0.03
0.01
-0.10
0,00
GAN
0.07
0.21
0.04
0.18
AIRL
0.15
0.11
0.30
0.20
TAIRL
0.54
0.30
0.51
0.31
Compactness. Compactness means that the reward values should be close for similar words and different for distinct words. For example, kid can also be referred as little boy or little girl, and thus their reward values should be close to each other in the discriminator: To see the correlation between the reward differences and semantic differences, we replace given word w, in the generated caption with a similar word Wsimilar and a distinct word Wdiistints, respectively. Specifically, in a generated caption, the first word that belongs to the COCO 80 class (Lu al. 2018) is replaced. A sentence is discarded if no word can be replaced. The words within the same class are considered to be similar (such as bike and bicycle), and the words that belong to difference classes are distinct (such as man and bike). For RL, since it maximizes handcrafted reward (SPICE (Anderson al. 2016) in our experiment) instead of learning a reward function, the reward difference is the variation of SPICE before and after replacement. For geward-learinng methods, the reward difference is the variation of the learned reward. The semantic difference is the Euclidean distance between the Glove embedding vectors of two words (Pennington et al. 2014). The results are reported in Table [ Higher correlation indicates better compactness. RL serves as a baseline in that the handcrafted reward SPICE compares n-gram overlapping without considering the semantic difference. The reward differences of rAIRL. correlate the best with the semantic differences for both similar words and distinct words, proving the compactness of the learned reward. It's also noted that due to the reward ambiguity problem, the reward differences of GAN poorly correlate with the semantic differences, especially for similar words.
Table 2: Sentence-level correlation with human evaluation. All p-value (not shown) are less than 0.001.
Correctness
Throughness
Method
Peasson
Spearman
Kendall
Peason
Spearman
Kendail
SPICE
0.44
0.45
0.39
0.45
0.46
0.38
GAN
0.12
0.11
0.15
0.12
0.11
0.15
AIRL
0.04
0.06
0.08
0.05
0.06
0.07
rAIRL
0.43
0.40
0.35
0.40
0.37
0.34
SAICE-SPICE
0.47
0.46
0.41
0.46
0.47
0.39
Correlation with human evaluation. As a caption evaluator, the discriminator learns go that recovers the true reward up to constant at optimality (Fu et al.] 2018)) We explore the correlation between the recovered reward and the human evaluation scores from the COMPOSITE dataset (Aditya et al. (017). where the Amazon Mechanical Turk (AMT) workers evaluate two aspects of the captions (i.c., correctness and throughness) at a range of 1-5, see Appendix [ for details of the human evaluation process. The correlation is evaluated using Pearson p. Kendall's 's and Spearman's 's r correlation coefficients. In Table 2 the reward of AIRL/AIILL is the sum of the word-wise reward Go. and the reward of tAIRL+SPICE is linear combination of ge and the SPICE score. Among the reward-lcarning methods, AIRL poorly correlates with human, whereas the proposed rAIRL improves AIRL on all the correlation metrics, especially on the Pearson correlation (from 0.04 to 0.43). Furthermore, simple combination of SPICE and the recovered reward leads to an increased correlation with the human scores, which proves the capacity of the discriminator a caption evaluator.
_ass_name____ txt

---

Under review as a conference paper at ICLR 2020
5.3 EVALUATION ON THE GENERATED CAPTIONS.
Table 3: Evaluation scores on generated captions. The best score is in bold font and the second best score is underlined. SPICE the handcrafted evaluation metric. CHAIR, and CHAIR: represent the object hallucination ratio at sentence level and instance level, respectively. HE indicates human evaluation. VC indicates vocabulary coverage and NS is the ratio of novel sentences.
Standard Split
Robust Split
Motbod
SPICE
CHAIR,
CHAIR
IIE
VC
NS
SPICE
CHAIR,
CHAIR;
11E
VC
NS
MLE
19.0
6.0
12.4
49.7
18.6
19.1
16.9
18.0
12.5
58.8
RI.
20.7
11.4
8.5
8.7
11.4
88.5
18.1
25.2
20.4
6.4
12.7
87.3
GAN
18.3
6.4
19.9
13.4
75.0
16.8
17.3
15.2
20.2
15.3
75.6
AIRL
17.3
12.7
10.3
14.0
12.3
67.3
16.7
22.7
18.5
14.8
15.6
73.8
TAIRL
20.4
5.5
13.6
26.1
18.7
14.3
15.8
76.5
Content correctness. For comprehensive evaluation of the content correctness, the results of both the handcrafted metries and human studies are shown in Table [ For the handcrafted metrics, we report scores of SPICE and the recently proposed CHAIR, and CHAIR; since they correlate well with human (Anderson et al. 2016; Rohrbach et al., 2018). SPICE computes similarity with the ground truth captions based on scene graph whilst CHIATR, and CHIAR; indicate ratio of hallucinated objects. The full results of other handcrafted metries can be found in Appendix E Compared with non-adversarial methods (i.c., MLE, RL), existing adversarial net (GAN) does not perform well on SPICE due the reward ambiguity problem, whereas our rAIRL improves GAN (from 16.8 to 18.7) by disentangling reward for each word, and even outperforms RL (from 18.1 1 to 18.7) on the robust split. The lowest scores on CHIAIR, and CHIAR; suggest that object hallucination is less likely in rAIRL. As for the human evaluation, HE in Table 3 iindicates the percentage of captions that are considered the best among the five methods. See Appendix for details of the human evaluation process. The descriptions generated by our rAIRL are considered the best for over 40% images, whilst RL has the lowest scores that are less than 10%.
Diversity. The diversity of captions is evaluated on a corpus level, indicating to what extent the generated captions of different images have diverse expressions. The results are presented in Table 3) VC indicates vocabulary coverage, which is the number of vocabularies of the generated captions over number vocabularies of the ground truth captions. NS represents ratio of novel sentence, which is the ratio of sentences that do not appear in the training set. The fact that RL has high ratio of novel sentence (88.5 088.5//87.. 7.3%% ) but low vocabulary coverage (11.4% (2)) suggests that it uses high-frewn freeurre words (such as "in a", "of a")) to reconstruct captions that appear to be different from the training set (Li et al., 2019a). See Appendix for a few examples. Our rAIRL improves AIRL on the diversity metrics and outperforms other learning methods on vocabulary coverage, indicating its capability of generating diverse descriptions on a corpus level.
Table 4: Percentage of different grammar errors found in the generated captions. Re represents Redundancy, AE is Agreement Error, AM denotes Article Misuse and IS Incomplete Sentence
Standard Split
Robust Split
Method
Total
AE
Total
Re
AF
IS
MLE
0.78
0.04
0.56
0.14
0.04
0.57
0.04
0.26
0.16
0.10
5.64
0.90
0
3.36
1.38
4.67
0.19
0.02
0.69
GAN
1.24
0.62
0.18
0.06
0.38
2.40
1.10
0.40
0.26
0.63
AIRL
0.04
0.62
0.70
0.32
1.20
0.10
0.27
0.72
0.12
FAIRL
0.75
0.14
0,20
0.21
0.20
0.57
0.14
0.17
0.16
0.10
Grammar. We used LonguageTool Pto check grammar of the generated captions. Table shows percentage of sentences that have grammar errors found by LanguageTool: I) Redundancy means repeated phrases in sentence; 2) Agreement Error means subject-verb agreement error, such as "people is"; 3) Article Misuse denotes inappropriate usage of indefinite articles, such as using "a" before uncountable nouns plural words; 4) Incomplete Sentence refers to incomplete sentence that lacks a subject. We found captions produced by RL have the most grammar errors (5.64% on the standard split and 4.67% on the robust split), especially the Article Misuse. On the other hand, by approximating the true data distribution of each word in the sentence, rAIRL and MLE have the least grammar errors among all learning methods 70..5%%/0.78 on the standard split and 0.57%/0.57%
7

---

Under review as a conference paper at ICLR 2020
on the robust split)). We also noticed that each method except rAIRL is biased towards particular type of grammar error: agreement error in MLE, article misuse in RL, redundancy in GAN, article misuse in AIRL. On both splits, our rAIRL does not appear to be biased towards a specific type of these grammar errors.
Summary. The proposed rAIRL constantly performs well on both splits of MS COCO and is capable of producing qualitative captions with few grammar errors. As a new adversarial algorithm, rAIRL enhances GAN by disentangling compact reward for each word in the caption and improves AIRL by shifting the stationary point towards Nash equilibrium. In the following sections, we first give ablation studies to see which component of our method explains the performance improvements, and then compare rAIRL with existing methods.
5.4 COMPARISON RESULTS
Table 5: Ablation methods of FAIRI. "terml" I is the constant term in Eq. and "term2" is the conditional term Eq. (16). GE denotes grammar error rate.
Method
Stundard Split
Robust Split
SPICE
CHAIR,
CHAIR,
VC
NN
GE
SPICE
CHAIR,
CHAIR;
VC:
NS
Gi:
rAIRL/wlo terml)
18.8
10.5
12.8
73.5
1.07
17.5
14.1
rAIRL/wlo (erm2)
19.3
9.1
7.4
12.2
71.3
0.83
17.9
18.9
158
13.7
62.4
0.72
rAIRL
20.4
7,2
5.5
13.6
76.1
0.75
18.7
17.1
14.3
15.8
76,5
0.57
Ablation studies. We conducted ablation experiments to understand the importance of each component of our algorithm. Specifically, the constant term in Eq. (9) and the conditional term in Eq. (16) is removed, respectively. Scores of all the evaluation techniques mentioned above are presented in Table [ We found that all the scores drop after removing either one of the terms. Comparing these two terms, the constant term seems more important in recognizing objects and relations in the image since removing it has larger drop on SPICE. The lager drop on vocabulary coverage and ratio of novel sentence in the second row indicates that the conditional term plays significant role in increasing the diversity of the generated captions. More results on using different model architectures included in Appendix [
Table 6: Comparison with existing methods on the handcrafted evaluation metrics.
Learning Method
Model
Standard Split
Robust Split
BLFF14
CIDFr
SPICE
RI FI14
CIDFs
SPICE
Attian
101.3
31.5
MLE
Up-Dawn
NBT
36.2
113.5
20.3
30.1
31.6
31.7
94.1
92.0
18.1
34.6
112.9
20.7
31.1
96.8
19.1
GANSISCST, Co-ail, D)5D)
181.1
RL
Up-Down
Arrzin
36.3
33.3
111.4
120,1
21.4
21.3
30.8
97.9
G-GAN
182
GAN
(SCST, Co-ott, log(D))
97.5
(UP-L(())DDww))
33.8
110.2
20.4
30.2
93.7
18.7
Comparison with existing methods. Based on the learning methods, existing models are divided into three categories in Table 6 and we chose recent proposed methods for comparison: Att2in (Rennie et al. 2017), G-GAN (Dai & Lin, 2017)), NBT (Lu et al. 2018), Up-Down (Anderson et al., 2018) and GAN,, GAN; (Dognin et al.] (019). Although some metries based on r--rrmm overlapping (BLEU4, CIDEr) do not correlate well with human, their results are also reported in Table 6 for fair comparison, Among the adversarial methods (GAN category), our rAIRL performs the best on all metrics. The results on COCO online server are given in Appendix F.
To further demonstrate the generalization ability of our algorithm, we built our algorithm on the non-adversarial based models. The composite models are denoted with rAIRL+MLEE and r rAIRL+RL. In rAIRL+MLE, the conditional term is replaced by the cross-entropy loss of MLE; in rAIRL+RL, the RL loss is added into the loss function of the generator. In Table 6 our rAIRL+MLE further improves the MLE baseline (i.e., Up-Down using MLE loss) on SPICE, whereas rAIRL+RL does not improve the RL baseline (i.e., Up-Down using RL loss) on these evaluation metrics. This is caused by the difficulty of normalizing the learned reward and the handcrafted reward to the same order of magnitude (Shelton Christian] 200T), and we leave this problem to our future work.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
In this paper, we address the reward ambiguity problem in image captioning and propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method that solves the problem by disentangling reward for each word in a sentence. Moreover, it achieves stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium, and mode control technique is incorporated to mitigate mode collapse. It is demonstrated that our method can learn compact reward through extensive experiments on MS COCO.
REFERENCES
Somak Adiyya, Yezhou Yang, Chitta Baral, Yiannis Aloimonos, and Comelia Fermüller. Image understanding using vision and reasoning through scene description graph. Computer Vision and Image Understanding, 2017.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In ECCV, 2016.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
Chen Chen, Shuai Mu, Wanpeng Xiao, Zexiong Ye, Liesi Wu, and Qi Ju. Improving image captioning with conditional generative adversarial nets. In AAAI, 2019.
Xinlei Chen, Hao Fang, Tsung-Y. Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00322 05055 2015.
Yin Cui, Guandao Yang. Andreas Veit, Xun Huang, and Serge Belongie. Learning to evaluate image captioning. In CVPR, 2018.
Bo Dai and Dahua Lin. Contrastive learning for image captioning. NeurlPS, 2017.
Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. Towards diverse and natural image descriptions via a conditional GAN. In ICCV, 2017.
Pierre L. Dognin, Igor Melnyk, Youssef Mroueh, Jerret Ross, and Tom Sercu. Adversarial semantic alignment for improved image captions. CVPR, 2019.
Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy-based based models. In NewrIPS, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. In ICLR, 2018
Ian Goodfellow, Jean Pbaget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Justin Johnson, Ranjay Krishna, Michael Stark, Lijia Li, David A Shamma, Michael S Bernstein, and Li Feifei. Image retrieval using scene graphs. CVPR, 2015.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Nannan Li, Zhenzhong Chen, and Shan Liu. Meta learning for image captioning. In AAAI, 2019a.
Ziming Li, Julia Kiseleva, and Maarten de Rijke. Dialogue generation: From imitation learning to inverse reinforcement learning. In AAAI, 2019b.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Improved image captioning via policy gradient optimization of spider. In ICCV, 2017.

---

Under review as a conference paper at ICLR 2020
Jiasen Lu, Caiming Xiong. Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, 2017.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In CVPR, 2018.
Lars M. Mescheder and Andreas Geiger. The numerics of gans, In NeurlPS, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprini arXiv: 1411 17844, 2014.
Andrew Ng. Daishi Harada, and Stuart Russell, Policy invariance under reward transformations: Theory and application reward shaping. in international conference machine learning. In ICML, 1999.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow. In ICLR, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. Deep reinforcement learning-based image captioning with embedding reward. In CVPR, 2017.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination image captioning. In EMNLP, 2018.
Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. Learning- based composite metrics for improved caption evaluation. In ACL, 2018.
Shelton Christian. Balancing multiple sources of reward in reinforcement learning. NeurIPS, 2001.
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine human captions by adversarial training. ICCV, 2017.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Towards diverse text generation with inverse reinforcement learning. In IJCAI, 2018.
Richard S Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. No metrics are perfect: Adversarial reward learning for visual storytelling. In ACL, 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In JCML, 2015.
Ting Yao. Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, 2018
10

---

Under review as a conference paper at ICLR 2020
A
VISUALIZED RESULTS OF GENERATED CAPTIONS
MLE:
a girl is looking at birthday cake. (8.7)
a city street filled with of traffic. (18.2)
RL: wornan holding
RL: yellow bus driving
cake in a room. (17.4)
down a city street. (18.2)
GAN: a woman getting
man riding
candles
matortycle street. (22.9)
birthday cake. (33.0)
rAIRL: woman
rAIRL: man riding
holding
motorrycle down busy
candles (36.4)
street (25.2)
MLE: a woman is holdling of scissors, (23.1]
MLE: man taking a picture of himself mirror. (30.5) RL: man standiing in bathroom with toilet. (23.5)
RL: a woman holding pair of scissorsin a.
(26.7)
GAN: woman holding
GAN: man standing
pair of scissors, (23..1)
front of a mirror. (26.7)
rAIRL: a smiling woman holding pair scissors. (26.7)
rAIR: man taking selfie
bathroom mirror.
(44.4)
Figure 2: Captions produced by different methods from the test set (standard split). Beside each caption report SPICE score. Captions generated by rATRL correct and human-like in these examples.
a young boy standing in front tv. (27.5)
white sitting desk.
RL: a young boy playing
(30.6)
video game in living
RL: a white cat sitting on top desk
room. (7.7)
GAN: littlc boy standing in a living room. (34.5)
(21.4)
GAN: a cat that sitting on desk. (16.0)
rAILR: a young boy is playing a video game, (14.3)
rAIRL: cat sitting on top of computer desk. (22.2)
Figure 3: Failed examples of rAIRL. The objects and relations are not correctly recognized in these pictures.
MLE: closeup of a plate of
MLE: doscup of plates of food an table.
food.
RL: a plate of food on table with plate.
RL: a plate of food on a table
with a plate.
GAN: plate food on plate.
GAN: a lunch of food on plate.
AIRL: plate of food on plate.
AIRL: a plate
on a plate.
rAIRL: table topped with plates food it.
rAIRL: plate of food on table.
MLE:
a stop sign on the side of road.
MLE: clock street at night night.
STOP
stop sign on the side of street.
RL:
RL: traffic light on the side of street.
GAN: stop sign on a street corner
GAN: stop sign on the street.
AIRL: stop sign on the side of street.
AIRL: a view of stop sign
street.
rAIRL: red stop sign sitting an side of road.
rAIRL: a view of stop sign
lit street at night.
Figure 4: Examples showing diversity of the captions. The left and right columns show pictures with similar content but different details. The proposed rAIRI successfully recognizes these differences and gives diverse captions.
11

---

Under review as a conference paper at ICLR 2020
1) blue and white bus drivinng down a street.
1] an aimlane flyling in the blue sky
2) blue and white bus on a city street.
sky airlann flying through the blue sky.
plane flying
3) public transit bus on city street. city driving down the street.
4) airlane flying the air.
5) a red plane flying in the sly..
5) a bus driving down city street.
Figure 5: Examples the top-5 generated captions of rAIRI. Driven by the compact reward function, generator descrihes a given image with semantically similar words.
B DIAGNOSE AND IMPROVE CAPTIONS
Table 7: Results of rewriting caption from the located position by rAIRL MS COCO standard split. Beside each report its improvement relative to rewriting from random position.
Source Caption
BLEU2
BLEU3
BLEU4
ROUGE.
scure
A
Suore
A
scure
Score
Scure
4
seure
A
scure
A
(40.0)
57.1
(40.)
43.4
(+0.1)
32.6
(40.5)
(40.1)
(+2.0)
20.6
(+0.1)
RL
72.9
(-0.1)
56.7
(40.3)
42.4
(40.5)
31.1
(+0.4)
50.8
(40.2)
104.0
(-0.5)
20.1
(+0.0)
(+1.1)
56.3
(+1.2)
42.0
(+1.0)
30.9
(+0.8)
(40.4)
(42.3)
20.0
(+0.4)
AIRL.
72.6
(+1.0)
56.1
(+1.3)
41.8
(+1.2)
30.5
(+0.9)
50.6
(+0.8)
102.6
(+4.1)
19.9
(+0.8)
Since the proposed rAIRL learns word-wise reward, it's also applicable to diagnose a given caption by finding the wrong word (e.g., the word whose reward decreases sharply compared with that of the previous word) and rewriting the caption to improve its quality. For example, improving a man is playing soccer man and a kid are playing soccer can be done by rewriting the caption from is. Therefore, we choose to rewrite given caption (source caption) from the word whose reward has decrease rate larger than 50%. However, we found that even rewriting the source caption from random position using rAIRL can also improve the evaluation scores. Thus, rewriting from random position selected as the baseline to compare with rewriting from the located position. Table 7 shows results of rewriting from the located position, where the source captions GAN and AIRL. Beside each score we report its improvement relative to rewriting from a random position, whose values are mostly positive. This demonstrates that the proposed rAIRL can diagnose the caption at a word level, and further improves the caption quality by rewriting from the located position.
given by MLE, RL,
Before: black and white bird standing in the sand.
Before: a dog sitting on top a chair,
After: black white standing on the edge of lake.
After: dog is looking out wicker chair.
Figure 6: Examples showing the generated captions from AIRT before and after re-written.
C HUMAN EVALUATION PROCESS
We conducted two types of human studies, one for evaluating the learned reward (in Section (.2)). and the other for examining quality of the generated captions (in Section 5.3). In the first human study experiment (in Section 5.2). we used the human scores the COMPOSITE dataset (Aditya et al. 2017), whose images are subsets from Flickr8k, Flickr30k and MS COCO. The descriptions from this dataset are either ground truth captions or generated sentences by (Aditya et al., 2017 Johnson al. 2015). In the human evaluation process, the AMT worker was asked to give score at range of
httpp::/ pttpss:/iimagadgg wordpress wwt.://sss..cmmmaoaeeeeeeeeee
12

---

Under review as a conference paper at ICLR 2020
Table 8: Full results of the sentence-leww correlation. p-value (not shown) are than 0.001.
Correctness
Throughness
Method
Peason
Spearman
Kendall
Peason
Spearman
Kendall
BLEUI
0.19
0.27
0.19
0.20
0.28
0.20
BLEU4
0.33
0.30
0.22
0.32
0.31
0.22
CIDEr
0.40
0.45
0.37
0.41
0.45
0.36
SPICE
0,44
0.45
0.39
0.45
0.46
0.38
GAN
0.12
0.11
0.15
0.12
0.11
0.15
AIRL
0.04
0.06
0.08
0.05
0.06
0.07
rAIRL
0.43
0.40
0.35
0.40
0.37
0.34
FAIRLABLEUI
0.44
0.41
0.35
0.41
0.39
0,34
0.45
0.43
0.36
0.42
0.42
0.35
TAIRL+CIDET
0.43
0.45
0.38
0.42
0,46
0.37
FAIRL+SPICE
0.47
0.46
0.41
0.46
0.47
0.39
1-5 to evaluate the correctness and throughness each sentence. Captions with length exceeding 20 were discarded, resulting total of 11, 657 sentences. Full results of the correlation is shown in Table SPICE correlates better with human evaluation when compared with other handcrafted metrics, whilst the composite metric rAIRL+SPICE further increases the correlation.
A herd of cattle drinking from pond.
A herd of cows the water.
3: A group of cows standing by river. herd of cows grazing the water.
5: A herd of cattle drinking from a river.
Figure 7: An example of the images shown to the human evaluator. The captions were produced by MLE, GAN, RL, AIRL and rAIRL methods in randomized order.
In the second human study experiment (in Section 5.3
randomly selected 500 test images from the standard split and robust split of MS COCO, respectively. The worker was asked "which caption the best" by given an image with five sentences generated from the adversarial and aoversaars methods, as shown in Figure The worker was allowed but not encouraged to make multiple choices he/she thinks these captions are equally correct. The order of captions produced by different methods was randomized. Following (Shetty et al. 2017), each image in the test set was evaluated by workers.
D ABLATION EXPERIMENTS ON MODEL ARCHITECTURES
Theoretically, our algorithm is model-agnostic since it is independent of the design model architecture. For empirical support of the claim, we show results of using Att2in (Rennie al. 2017) and Up-Down (Anderson et al.) 2018) architectures Table We report the metrics used in the original paper for fair comparison. The proposed rAIRL mainly improves SPICE, which correlates well with human evaluations, on both architectures.
13

---

Under review as a conference paper at ICLR 2020
Table Results using different model architectures our method.
Standard Split
Robust Split
Method
BLEU4
CIDEr
SPICE
BLEU4
CIDEr
SPICE
AttZin
31.0
101.3
31.5
90.6
17.7
TAIRL(At22nn)
31.3
105.2
19.9
30.7
92.5
18,0
Up-Down Down
36.2
113.5
20.3
31.6
92.0
18.1
IAIRL(Up- Down)
33.8
110.2
20.4
30.2
93.7
18.7
E FULL RESULTS ON MS COCO
We adopt SPICE to evaluate content correctness in the paper because has better correlation with human wodgmenns(Anderrsn al. 2016). Table To gives full results of the handerafted metrics Two splits of MS COCO. Comparing the adversarial (GAN, AIRL, rARIL) and non-adversarial (MLE, RL) methods, RL outperforms other methods most metrics. In adversarial methods, the proposed TAIRL performs the best. Table shows results on the MS COCO online test server. The proposed rAIRL improves AIRL on all the metrics.
Table 10: Results of conventional handcrafted metrics MS COCO test split.
Method
Standard Split
Robust Split
BLEU1
BLEU2
BLBUS
BLEU4
ROUGE
CIDbr
SPICE
BLEU
BLEU2
BLEU3
BLEU4
ROUGE: I.
CWDE
SPICE
MLE
14.5
$7.0
41.7
30.3
30.2
104.6
19.0
69:5
52.7
39.2
29.3
48.7
93.4
18.6
RL
75.5
29.2
43.8
35.0
51.6
115.1
20.7
73.5
35.9
40.8
29.9
49.9
95.1
18.1
GAN
67.7
51.9
38.6
28.3
48.7
93.4
18.3
64.7
48.0
34.5
24.6
46.2
78.3
16.8
AIRL
69.9
53.6
39.1
27.5
67.4
17.3
67.6
50.5
36.3
25.9
47.3
79.5
16.7
AAIKL
204
Table 11: Results COCO test server. Methods marked with adopt RL of CIDEr optimization.
BLEU1
BLEU2
BLEU3
BLEU4
METEOR
ROUGE I.
CIDEr
Method
c40
c40
c40
<40
Adaptive Lu al
2017 74.8
92.0
58.4
84.5
44.4
74.4
33.6
63.7
26.4
35.9
55.0
70.5
104.2
105.9
Art2all'
(Remi ot dl 2017
78.1
93.7
61.9
86.0
47.0
75.9
35.2
64.5
27.0
35.5
56.3
70.7
114.7
116.7
Up- Up-Down AAntesoon aie 2018
80.2
95.2
64.1
88.8
49.1
79.4
36.9
68.5
27.6
36.7
57.1
72.4
117.9
120.5
AIRL(Up-Dowa)
72.5
90.4
55.2
83.7
42.3
73.6
30.8
62.6
25.0
34.2
53.5
68.5
81.9
82.6
rAIRL(Up Down)
75.4
93.1
59.8
86.5
45.9
77.2
35.0
66.7
26.1
35.4
55.6
71.1
104.1
105.2
MAIRL=MLEUUpDDoom
75.5
93.3
59.8
86.7
46.2
77.4
35.4
67.0
26.5
36.0
55.8
71.5
105.9
106.2
-AIRL-RL(Up-Down)
79.5
94.1
63.5
88.0
48.3
78.9
36.2
68.5
27.5
36.6
56.2
71.8
112.3
115.1
F DISCUSSION ON LOSS FUNCTIONS
Table 12: Formulas of different loss functions.
Method
Loss Function
MLE
log(nE"")
RL
Σ log(m:
GAN (generator)
Dyen Σ log(mT)
rAIRL (generator)
(DF")) log(m:) -- -Diiee log(x!"u)
We compare the formula of the proposed loss function with existing methods in Table 12 including MLE, RL and GAN. n is the length of sentence. is the handcrafted metric, such as BLEU, CIDEr and SPICE. me is the probability of the in generated word, and n, tue is the probability of the La true
14

---

Under review as a conference paper at ICLR 2020
word. The loss functions rewritten using similar symbols for direct comparison. MLE maximizes the probability of the true data Th whist RL and GAN optimize the reward by sampling from Ti- GAN is different from RL in that its reward is learned from the discriminator adversarially instead of being predefined. GAN is capable of mimicking numan-writted captions by adversarial learning, but the estimated reward function gee of full trajectory and thus is too ambiguous. The proposed rAIRL further disentangles the reward into D each time step t, as well as incorporating the true data for better diversity. From the perspective of loss functions, rAIRL be regarded as integration of GAN and MLE using coefficients t (D and o (D
be explained by multiple optimal policies
G GENERATOR OBJECTIVE
Below we show the deductions details of the generator objective (i.e., Eq. (16) in section 4.2
Taking the integral of V Bu [-] w.r.t. &
Ewe [fa. (w,, st) (TT( + 1] VWlEWarw (we, se) log((Tv) 1]cb (12)
According to Eq.
10
1
V V (N 86) log(Fpp) +1]= (Wr -oog(T))) Thus have
(13)
T
/
1
Bur (w),) log(TT>) + +1]
(fes - log((=))) V
(14)
= (Jo (Wr, 81) - log((T))) log(TT))
Similarly, replacing we with witue we get
Eng wang [J (w: shte) - log(m tree
+1 1] = (So,(wt s log(T log(= (15)
Thus, the generator objective is:
L, (v+) - -.ierT [fe (wre st) log(TT) 1] E Eupeeasee (w (quu0) log log(mtrue)
+
- -Joo 81) - log(T()) log(TT)) - (Jo , - log(=((()) log(m(p))
(16)
15


Reviews:
[{"review_id": "S1eMFynatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose using a recent method for adversarial inverse reinforcement learning (AIRL) for the task for generating high-quality image captions. Leveraging the GAN framework, a discriminator is trained to distinguish real captions from those produced by the generator, while the generator is optimized with policy-gradients (REINFORCE) to maximize the pseudo-reward from the discriminator. The main difference from prior work seems to be that the discriminator acts on a word-level, rather than sentence-level (as done, for instance in Dai et. al. 2017). Correspondingly, the generator policy is updated with the objective of 1-step reward maximization (more like contextual bandits), rather than with a long-term sequential decision-making objective (as done in Dai et. al. 2017). The evaluation is done using 2 data splits – standard and robust, with various metrics such as SPICE, CIDEr, BLEU, CHAIR. Diversity analysis and ablations are also performed to dissect the performance of the proposed approach. \n\nMy 2 main issues with the paper are confusing motivation (in section 1) and various imprecise parts (in section 3 and 4).  \n\n1.\tThe authors argue that current GAN-based captioning models provide ill-defined rewards due to the “reward ambiguity problem”. This problem is not explained or motivated well in the paper, but instead the readers are referred to the AIRL paper. “Reward ambiguity” in inverse-RL arises because there could be many reward functions that yield the same optimal policy. The AIRL algorithm recovers one of these possible reward functions, and since such a recovered reward could be shaped by the environment dynamics, AIRL attempts to disentangle the reward from the dynamics. The motivation there is to use the recovered reward on a new system with different transition dynamics. In the context of this paper though, I would like to understand the angle of reward ambiguity. The authors disentangle the sentence reward into word-wise rewards; however, I’m not sure if there’s any relation between this and the disentanglement done in AIRL for solving reward ambiguity.\n\n2.\tOne of the objectives is learning compact rewards. It is claimed that addition of a constant term to the reward provided to the generator policy results in this, but what’s the intuition behind this? As for evaluation, it needs to be shown that words with similar semantics have similar discriminator score. How do we conclude this from Figure 2.? Also, please include Up-Down method results in Figure 2.\n\n3.\tSection 3 questions\n     a.\tHow is state s_t defined? It is very hard to follow sections 3 and 4 without a clear definition and example for this.\n     b.\t“Finn et. al. 2016 proved that IRL is mathematically equivalent …” --- this is imprecise. Maximum-Entropy-IRL is equivalent to the GAN formulation, not general IRL. \n     c.\tp_theta(a,s) is referred to as “reward distribution”. I don’t think it’s a distribution.\n     d.\tEquation 3. AIRL defines g only as a function of state (s_t) for the disentanglement, and not like what the authors have written.\n     e.\tEquation 4. How is s_t sampled?\n\n4.\tSection 4 questions\n     a.\t4.1 says discriminator “maximizes the divergence”. This doesn’t seem correct.\n     b.\tf is referred to as state-value. This doesn’t seem correct.\n     c.\tShouldn’t the -1 term in Equation 8 disappear under expectation?\n     d.\tDon’t understand how second line of Equation 11 is arrived at.\n\nThere are quite a few other sources of mathematically imprecise writing that I noticed. I would recommend the authors to be more robust in their presentation.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlRe9-RFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a refined Adversarial Inverse Reinforcement Learning (rAIRL) to remedy the reward ambiguity by decoupling the reward for each word in a sentence, while the existing methods that utilize reinforcement learning to optimize evaluation score handle only sentence-level rewards. Furthermore, a conditional term is introduced in the loss function to avoid mode collapse and to increase the diversity of the generated captions. Throughout experiments on MS COCO show that the proposed method achieves state-of-the-art performance with several evaluation scores.\n\nI think this is a good paper. The idea to disentangle the sentence-level reward into word-level ones with Adversarial Inverse Reinforcement Learning (AIRL) is highly motivated. Although the original AIRL has a problem that the convergence is slow, the authors introduce a constant term to shift on of the stationary points. As reported, this refinement surprisingly improves the performance and achieves state-of-the-art performance, while the original AIRL degraded the performance. \n\nAlthough I lean to accept this paper, I have two comments.\n- I would like to recommend that the source code to reproduce the result should be released.\n- As discussed by the authors, the introduced constant term can be regarded as a baseline in REINFORCE. In (Rennie et al., 2017), a self-critical sequence is introduced as a baseline in REINFORCE. Therefore, it is notable that this paper proposes another type of baseline. I would like the authors to compare Att2in (Rennie et al., 2017) to the proposed method, and to discuss why rAIRL outperforms Att2in as shown in Table 5.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xsvvvVtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall this is a good paper.\nSuch compact rewards over image captions are useful for evaluation and diagnosis. \n\n*concerns:\n1. method:\n   1.1 At first, I think section 4 could be re-written to corporate more concepts in image captioning (such as replacing action a to word w) so that it becomes more readable for related readers. \n   1.2 The motivation for adding a constant to shift nash equailibrium could be stated clearer. \n   1.3 what is the state s in the context of image captioning? the hidden vector h ?\n   1.4 since the reward is computed for each pair of (a, s), how to get the reward for the whole sentence? Sum them up? I wonder this because it seems you compute this in Table 1.\n   1.5 in Eq.(5) and Eq.(11), there are expectations. Do you need to compute them using Monte Carlo Rollouts? Or just averaging over mini-batchs? Because G-GAN (Dai et al) also utilizes rollouts to estimate different values for different words. \n\n2. experiments:\n  1.1 the experiment in terms of \"compactness\" may not  reflect well the concept of compactness. While compactness means similar words may obtain similar rewards in the same caption,  top-k captions for an image may be different in multiple words, or different in formats rather than word choices. How about top-k words in a given position of a given caption? Or replace a given word with different words and compare the correlation between reward differences and semantic differences.\n  1.2  qualitative samples in terms of caption diversity could be included. \n  1.3 since the rewards are computed for each word separately,  the authors may include experiments on using such rewards to diagnose captions, such as replacing a word to make the caption better, etc.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a refined AIRL method to deal with the reward ambiguity problem in image captioning, wherein the main idea is to refine the loss function in word level instead in sentence level, and introduce a conditional term in the loss function to mitigate mode collapse problem.  The results show the proposed method improves the performance and achieves state-of-the-art performance.  However there are concerns from the reviewers that the motivation of the work was not well explained and some inprecise parts exist in the paper.  The concept of "reward ambiguity problem" is not properly addressed according the opinion of reviewer2.  I would like to see these concerns be well addressed before the paper can be accepted.  
Decision: Reject