Year: 2019

Abstract:
Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.

text: Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.

heading: 1 INTRODUCTION
text: Image captioning is a task of generating descriptions of a given image in natural language. In a general encoder-decoder structure (Vinyals et al., 2015), image features are encoded in a CNN and decoded into a caption in a word by word manner. Based on the loss function, standard approaches to the problem could be divided into three categories: MLE (Maximum Likelihood Estimation), RL (Reinforcement Learning) and GAN (Generative Adversarial Network).
Early proposed methods were based on MLE function and made improvements by designing specific model structure (Xu et al., 2015). MLE adopts the cross-entropy loss and learns a one-hot distribution for each word in the sentence. By maximizing the probability of the ground truth word whilst suppressing other reasonable vocabularies, the probability distribution learned by MLE tends to be sparse and the generated captions have limited diversity (Dai et al., 2017). On the other hand, RL has advantages in boosting the model performance by optimizing the handcrafted metrics (Rennie et al., 2017; Liu et al., 2017; Chen et al., 2019). However, due to the reward hacking problem, RL maximizes the reward in an unintended way and fails to produce human-like descriptions (Li et al., 2019a). Considering naturalness and diversity of the generated captions, GAN has raised attention in image captioning for its capability of producing descriptions that are indistinguishable from human-written ones (Dai et al., 2017; Shetty et al., 2017; Chen et al., 2019; Dognin et al., 2019).
In image captioning, the generator of GAN learns true data distribution by maximizing the reward function learned from a discriminator, and the discriminator distinguishes the generated sample from the true data. The adversarial training converges to an equilibrium point (i.e., Nash equilibrium) at which both the generator and discriminator cannot improve (Goodfellow et al., 2014). As shown in Figure 1, the learned distribution of GAN is closer to the ground truth distribution than that of other methods (i.e., MLE and RL) on different splits. However, previous work of adversarial networks in image captioning gives one reward function D for a complete sentence consisting of n words. This strategy causes the reward ambiguity problem (Ng et al., 1999) since which word(s) causes the reward to increase or decrease is not accounted for, and thus there are many optimal policies that determine the sentence can explain one reward. An example is that each image in MS COCO has five ground truth captions. Although these captions may vary in formats, each caption has the same reward value in GAN. From the perspective on the system level, learning sentence-level reward from different image-caption pairs is analogous to learning reward of a trajectory from different system dynamics, which makes the discriminator unable to distinguish the true reward functions from those shaped by the environment dynamics (Fu et al., 2018).
Facing the challenge, we adopt AIRL (Fu et al., 2018) to solve the reward ambiguity problem by disentangling reward for each action (i.e., word in a sentence) from different image-caption pairs and learning a compact reward function. compact means words with similar semantics, such as children and kids, correspond to close reward values. Driven by the compact reward function of the discriminator, the generator learns the optimal policy and thus produces qualitative descriptions. However, there are still two major problems to address: 1) AIRL is difficult to converge to Nash equilibrium using policy gradient (See Section 4.2 for details); 2) AIRL is designed without mode control, and thus the outputs have limited diversity, which is a commonly encountered issue called mode collapse (Mirza & Osindero, 2014).
In this paper, we propose a refined AIRL method to learn a compact reward function for each word, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. The refined method makes it possible to reach the equilibrium point for a non-concave model function of the generator. In addition, a conditional term is introduced in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Both the caption evaluator (i.e., discriminator) (Cui et al., 2018; Sharif et al., 2018) and the generator are cast into this unified framework, where the discriminator evaluates captions using a learned compact reward function, and the generator produces qualitative image descriptions. We demonstrate the effectiveness of our method in the experiments.

heading: 2 RELATED WORK
text: Image Captioning. The development of image captioning can be summarized into two directions: model structure design (Lu et al., 2017; Yao et al., 2018) and loss function construction (Rennie et al., 2017; Ren et al., 2017). In the methods based on model structure design, attention mechanism and the fusion of visual and semantic information are the key focus. Lu et al. (2017; 2018) proposed a sentinel gate to learn adaptive attention between visual content and non-visual text. Yao et al. (2018) explored the role of visual relationship in image captioning. On the other hand, methods based on loss function construction focus on optimization of the loss function. Rennie et al. (2017) optimized on non-differentiable evaluation metric using policy gradient, and improved scores of these metrics on various models. Ren et al. (2017) designed an embedding reward under actor-critic reinforcement learning. Similarly, we address the construction of loss functions, and thus our algorithm can be built on existing model structures. See Appendix F for a short discussion about different loss functions.
Adversarial Methods for Image Captioning. Adversarial methods are known for producing plausible samples by training the generator and the discriminator in an adversarial manner (Goodfellow et al., 2014). In image captioning, the discriminator is formed as a binary classifier that distinguishes the generated sentence from the ground truth, while the generator produce captions that can fool the discriminator. Conditional GAN was proposed in (Dai et al., 2017) to improve the naturalness and diversity of generated captions. CNN and RNN based discriminators were introduced in (Chen et al., 2019). However, existing methods estimate a reward function for the complete sentence consisting of n words, where multiple optimal policies that determine the sentences can correspond to one reward
(Ng et al., 1999). Thus the learned reward is ambiguous and ill-defined. We solve this problem by recovering a compact reward function for each word in the sentence under a refined AIRL framework. Although AIRL has been utilized to solve problems in other fields (Wang et al., 2018; Li et al., 2019b; Shi et al., 2018), we are the first to make algorithmic improvements to AIRL such that Nash equilibrium can be reached even for a non-concave model function of the generator, and that diversity of the outputs can be increased.

heading: 3 ADVERSARIAL INVERSE REINFORCEMENT LEARNING
text: Due to the high variance estimate of a full sentence and the reward ambiguity problem, instead of learning reward for a complete sentence, we could learn reward distribution pθ(wt, st) for each word-state pair (wt, st) so that the true reward can be recovered at optimality (Fu et al., 2018). In the following, we introduce how AIRL disentangles reward for each word-state pair (wt, st).
AIRL is an adversarial reward learning algorithm based on Maximum-Entropy-IRL. Finn et al. (2016) first proved that Maximum-Entropy-IRL is mathematically equivalent to GAN under a special form of the discriminator:
Dθ(wt, st) = pθ(wt, st)
pθ(wt, st) + π(wt, st) (1)
pθ(wt, st) = exp {fθ(wt, st)} (2) where pθ(wt, st) is the actual probability distribution estimated by the discriminator. π(wt, st) is the policy distribution produced by the generator, which is also called vocabulary distribution under the context of image captioning.
The goal is to estimate pθ(wt, st) that approximates the true data distribution ptrue(wt, st), as well as to learn an optimal vocabulary distribution π that maximizes the reward. Subsequently, considering reward ambiguity problem, Fu et al. (2018) further extended the theory to AIRL by adding a reward shaping term hϕ into fθ(wt, st), and thus disentangled reward from different system dynamics (i.e., image-caption pairs):
fθ,ϕ(wt, st) = gθ(wt, st; st+1) + γhϕ(st+1)− hϕ(st) (3) where gθ denotes the reward approximator that recovers the true reward up to a constant, and hϕ is the reward shaping term that preserves the optimal π. γ is a constant in range (0, 1].
In the context of divergence minimization, the adversarial process can be represented as a min-max game (Mescheder & Geiger, 2017):
min π max θ,ϕ
Ewtruet ∼ptrue [log ( Dθ,ϕ(w true t , s true t ) ) ] + Ewt∼π[log ( 1−Dθ,ϕ(wt, st) ) ] (4)
where ptrue is the true data distribution and π is the vocabulary distribution learned by the generator. (wtruet , s true t ) is the word-state pair of the true data.
Despite of the capability of AIRL in disentangling reward for each word, it is difficult for the above AIRL algorithm to converge to Nash equilibrium and to produce diverse outputs through adversarial training (See Section 4.2 for details). These issues can result in a non-optimal solution and lack of diversity of the generated descriptions. Aiming to learn the optimal compact reward as well as diverse captions, we refine the loss function to shift the stationary point towards Nash equilibrium and to mitigate mode collapse in the two-player game.

heading: 4 LEARNING COMPACT REWARD FOR IMAGE CAPTIONING
text: To address the problems discussed above, we refine the loss function to: 1) find a compact reward function that is optimal; 2) increase diversity of the generated captions. In particular, a constant term is used to solve 1) by shifting the stationary point to Nash equilibrium, and a conditional term is introduced to solve 2) by utilizing mode control techniques. Our algorithm is detailed in Algorithm 1, where n is the sentence length and N denotes number of iterations.
In the following notations, θ and ϕ are the parameters of the discriminator, and ψ represents the parameter of the generator. wt and st denote the tth word and its corresponding hidden state vector, respectively. For better clarity, policy πψ is hereinafter referred to as vocabulary distribution.
Algorithm 1: Refined ARIL Initialize the vocabulary distribution πψ and discriminator fθ,ϕ. for iteration i in {1, ..., N} do
Obtain caption {wtrue1 , ..., wtruen } from the ground truth. Collect generated caption {w1, ..., wn} using the vocabulary distribution πψ . Dθ,ϕ←sigmoid(fθ,ϕ − log(πψ)) Update (θ, ϕ) via Eq. (5) for the discriminator. Update ψ via Eq. (16) for the generator.
end

heading: 4.1 DISCRIMINATOR
text: The objective of the discriminator is to distinguish the true caption from the generated one. At time t, the discriminator maximizes the divergence in Eq. (4) by
Lt(θ, ϕ) = − log ( Dθ,ϕ(w true t , s true t ) ) wtruet ∼ptrue − log ( 1−Dθ,ϕ(wt, st) ) wt∼πψ
(5)
where ptrue is the distribution of the true caption, and πψ is the vocabulary distribution estimated by the generator. Dθ,ϕ is computed as in Eq. (1) and Eq. (2), where the discriminator learns the reward function fθ,ϕ for Dθ,ϕ and the generator estimates the vocabulary distribution πψ for Dθ,ϕ, respectively.

heading: 4.2 GENERATOR
text: In the following, Dθ,ϕ is represented as below (Fu et al., 2018) using Eq. (1) and Eq. (2): Dθ,ϕ(wt, st) = sigmoid ( fθ,ϕ(wt, st)− log(πψ ) ) (6)
Given word wt that is sampled from the vocabulary distribution πψ, the generator maximizes Dθ,ϕ(wt, st) by
Lt(ψ) = −Ewt∼πψ [log ( Dθ,ϕ(wt, st) ) − log ( 1−Dθ,ϕ(wt, st) ) ]
= −Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ)] (7)
Using REINFORCE algorithm (Sutton & Barto, 1998), the gradient∇ψLt becomes: ∇ψLt = − ∑
πψ
( fθ,ϕ(wt, st)− log(πψ) ) ∇ψπψ+πψ∇ψ ( fθ,ϕ(wt, st)− log(πψ) ) = −
∑ πψ πψ [ 1 πψ ( fθ,ϕ(wt, st)− log(πψ) ) ∇ψπψ +∇ψ ( fθ,ϕ(wt, st)− log(πψ) )] = − 1
πψ
( fθ,ϕ(wt, st)− log(πψ) ) ∇ψπψ −∇ψ ( fθ,ϕ(wt, st)− log(πψ) ) = − 1
πψ
( fθ,ϕ(wt, st)− log(πψ)− 1 ) ∇ψπψ
(8)
When the generator converges (i.e.,∇ψLt = 0), there exists two stationary points: ∇ψπψ = 0 and log(πψ) = fθ,ϕ(wt, st)− 1. If Nash equilibrium can be reached at optimality, Dθ,ϕ should converge to 0.5 when ∇ψLt = 0. Thus it’s only possible for the first point to reach Nash equilibrium since Dθ,ϕ = sigmoid ( fθ,ϕ(wt, st)− log(πψ ) ) = sigmoid(1) 6= 0.5 at the second point. However, even for the first point, Nash equilibrium exists only for a concave πψ, requiring Hessian of the gradient vector filed being positive definite (Mescheder & Geiger, 2017). To relax this constraint, a constant term is added into the expectation in Eq. (7)
Lt(ψ) = −Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1] (9)
∇ψLt = − 1
πψ (fθ,ϕ(wt, st)− log(πψ))∇ψπψ (10)
to expand the feasible region of reaching Nash equilibrium by shifting the second stationary point to Dθ,ϕ(wt) = sigmoid(0) = 0.5. The constant term makes it possible to achieve Nash equilibrium
even for a non-concave πψ, which also makes it easier for the adversarial model to converge . It is noted that the constant term can also be regarded as baseline in REINFORCE, except it is utilized to centralize the stationary point instead of reducing variance of the estimation.
In practice, mode collapse occurs when the generator produces a single or limited modes, which exhibits as little diversity in image captioning. To mitigate mode collapse (Mirza & Osindero, 2014) and increase the diversity of the generated captions, we add ground truth data into the generator as a conditional term:
Lt(ψ) = −Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1]− Ewtruet ∼πtrueψ [fθ,ϕ(w true t , s true t )− log(πtrueψ ) + 1] = − ( fθ,ϕ(wt, st)− log(πψ) ) log(πψ)− ( fθ,ϕ(w true t , s true t )− log(πtrueψ ) ) log(πtrueψ )
(11) where πtrueψ is the approximated true caption distribution in the generator, and Ewtruet ∼πtrueψ [·] is the conditional term. See Appendix G for the deduction details.The coefficient of log(πtrueψ ) is symmetrical to the coefficient of log(πψ) and is updated adaptively in the training process. The conditional term helps in strengthening the generator in the adversarial training. When Dtrue > Dgen, the gradient of the true data becomes larger than that of the generated one (∇πtrueψ Lt > ∇πψLt), and thus the generator further increases the probability of the true word (πtrueψ ). Otherwise (i.e., Dtrue < Dgen), the generator prefers sampling its self-generated words to fool the discriminator. By switching between the true words and the generated ones, the generator maintains informative gradient during the adversarial training (Peng et al., 2019). Note that adding the conditional term does not change the model’s convergence to Nash equilibrium since πψ = πtrueψ at the second stationary point.

heading: 5 EXPERIMENTS
text: In the experiments, we validate the effectiveness of the proposed algorithm by answering three questions: 1) Is the caption evaluator (i.e., discriminator) capable of learning compact reward? 2) Driven by the learned reward, is the caption generator effective to produce qualitative captions? 3) How does our algorithm perform when built on or compared with existing methods?
To answer 1), we first tested the compactness of the learned reward by observing performance of the collected top-k captions. Then we explored the correlation between the learned reward and the human evaluation results. To answer 2), we evaluated the quality of the generated caption on its content, diversity and grammar. To answer 3), we built our algorithm on existing learning methods and compared their performance. We also conducted ablation experiments to demonstrate the importance of each component of our algorithm.

heading: 5.1 IMPLEMENTATION DETAILS
text: We conducted experiments on the well-known benchmark dataset MS COCO (Chen et al., 2015). The dataset has 123,287 labeled images and each image has at least 5 human annotated captions as reference. To assess the robustness of our algorithm, we use two splits of the COCO dataset: standard split (Karpathy & Fei-Fei, 2015) which is created by randomly picking test images, and robust split (Lu et al., 2018) which is organized to maximize difference of the co-occurrence distribution between the training and test set. The robust split is recently proposed and is more challenging due to its distribution difference between the training and test set. The standard split has 113287/5000/5000 train/val/test images and the robust split has 110234/3915/9138 train/val/test images.
We implement our algorithm using Adam optimizer (Kingma & Ba, 2014) with fixed learning rate 10−5. Our vocabulary size is fixed to 10,000 including a special start sign <BOS>and an end sign <EOS>. In the generator, the number of hidden nodes of every layer is 512. For simplicity, the discriminator has the same model structure as the generator except having one additional layer for estimating hϕ. For fair comparison, all the methods in MLE, RL, GAN, AIRL and rAIRL were produced using the same image features and model structure in (Anderson et al., 2018). Specifically, RL is the self-critical algorithm in (Rennie et al., 2017). GAN is the adversarial algorithm in (Dai & Lin, 2017) that learns sentence-level reward. AIRL is the standard adversarial inverse reinforcement learning method in (Fu et al., 2018), and rAIRL is the proposed method. Note that our scores of MLE are lower on the standard split but higher on the robust split than (Anderson et al., 2018) because
1) we used fixed number of the bounding box (i.e., 36) for simplicity; 2) the hyperparameters were tuned to adapt to both splits and thus are not exactly the same with (Anderson et al., 2018).

heading: 5.2 PERFORMANCE OF THE RECOVERED REWARD
text: Compactness. Compactness means that the reward values should be close for similar words and different for distinct words. For example, kid can also be referred to as little boy or little girl, and thus their reward values should be close to each other in the discriminator. To see the correlation between the reward differences and semantic differences, we replace a given word wt in the generated caption with a similar word wsimilar and a distinct word wdistinct, respectively. Specifically, in a generated caption, the first word that belongs to the COCO 80 class1 (Lu et al., 2018) is replaced. A sentence is discarded if no word can be replaced. The words within the same class are considered to be similar (such as bike and bicycle), and the words that belong to difference classes are distinct (such as man and bike). For RL, since it maximizes a handcrafted reward (SPICE (Anderson et al., 2016) in our experiment) instead of learning a reward function, the reward difference is the variation of SPICE before and after replacement. For reward-learning methods, the reward difference is the variation of the learned reward. The semantic difference is the Euclidean distance between the Glove embedding vectors of two words (Pennington et al., 2014). The results are reported in Table 1. Higher correlation indicates better compactness. RL serves as a baseline in that the handcrafted reward SPICE compares n-gram overlapping without considering the semantic difference. The reward differences of rAIRL correlate the best with the semantic differences for both similar words and distinct words, proving the compactness of the learned reward. It’s also noted that due to the reward ambiguity problem, the reward differences of GAN poorly correlate with the semantic differences, especially for similar words.
Correlation with human evaluation. As a caption evaluator, the discriminator learns gθ that recovers the true reward up to a constant at optimality (Fu et al., 2018). We explore the correlation between the recovered reward and the human evaluation scores from the COMPOSITE dataset (Aditya et al., 2017), where the Amazon Mechanical Turk (AMT) workers evaluate two aspects of the captions (i.e., correctness and throughness) at a range of 1-5, see Appendix C for details of the human evaluation process. The correlation is evaluated using Pearson p, Kendall’s τ and Spearman’s r correlation coefficients. In Table 2, the reward of AIRL/rAIRL is the sum of the word-wise reward gθ, and the reward of rAIRL+SPICE is a linear combination of gθ and the SPICE score. Among the reward-learning methods, AIRL poorly correlates with human, whereas the proposed rAIRL improves AIRL on all the correlation metrics, especially on the Pearson correlation (from 0.04 to 0.43). Furthermore, a simple combination of SPICE and the recovered reward leads to an increased correlation with the human scores, which proves the capacity of the discriminator as a caption evaluator.
1https://github.com/jiasenlu/NeuralBabyTalk/blob/master/data/coco/coco_ class_name.txt

heading: 5.3 EVALUATION ON THE GENERATED CAPTIONS.
text: Content correctness. For a comprehensive evaluation of the content correctness, the results of both the handcrafted metrics and human studies are shown in Table 3. For the handcrafted metrics, we report scores of SPICE and the recently proposed CHAIRs and CHAIRi since they correlate well with human (Anderson et al., 2016; Rohrbach et al., 2018). SPICE computes similarity with the ground truth captions based on scene graph whilst CHIAIRs and CHIARi indicate ratio of hallucinated objects. The full results of other handcrafted metrics can be found in Appendix E. Compared with non-adversarial methods (i.e., MLE, RL), existing adversarial net (GAN) does not perform well on SPICE due to the reward ambiguity problem, whereas our rAIRL improves GAN (from 16.8 to 18.7) by disentangling reward for each word, and even outperforms RL (from 18.1 to 18.7) on the robust split. The lowest scores on CHIAIRs and CHIARi suggest that object hallucination is less likely in rAIRL. As for the human evaluation, HE in Table 3 indicates the percentage of captions that are considered the best among the five methods. See Appendix C for details of the human evaluation process. The descriptions generated by our rAIRL are considered the best for over 40% images, whilst RL has the lowest scores that are less than 10%.
Diversity. The diversity of captions is evaluated on a corpus level, indicating to what extent the generated captions of different images have diverse expressions. The results are presented in Table 3. VC indicates vocabulary coverage, which is the number of vocabularies of the generated captions over number of vocabularies of the ground truth captions. NS represents ratio of novel sentence, which is the ratio of sentences that do not appear in the training set. The fact that RL has high ratio of novel sentence (88.5%/87.3%) but low vocabulary coverage (11.4%/12.7%) suggests that it uses high-frequency words (such as “in a”, “of a”) to reconstruct captions that appear to be different from the training set (Li et al., 2019a). See Appendix A for a few examples. Our rAIRL improves AIRL on the diversity metrics and outperforms other learning methods on vocabulary coverage, indicating its capability of generating diverse descriptions on a corpus level.
on the robust split)). We also noticed that each method except rAIRL is biased towards a particular type of grammar error: agreement error in MLE, article misuse in RL, redundancy in GAN, article misuse in AIRL. On both splits, our rAIRL does not appear to be biased towards a specific type of these grammar errors.
Summary. The proposed rAIRL constantly performs well on both splits of MS COCO and is capable of producing qualitative captions with few grammar errors. As a new adversarial algorithm, rAIRL enhances GAN by disentangling compact reward for each word in the caption and improves AIRL by shifting the stationary point towards Nash equilibrium. In the following sections, we first give ablation studies to see which component of our method explains the performance improvements, and then compare rAIRL with existing methods.

heading: 5.4 COMPARISON RESULTS
text: Ablation studies. We conducted ablation experiments to understand the importance of each component of our algorithm. Specifically, the constant term in Eq. (9) and the conditional term in Eq. (16) is removed, respectively. Scores of all the evaluation techniques mentioned above are presented in Table 5. We found that all the scores drop after removing either one of the terms. Comparing these two terms, the constant term seems more important in recognizing objects and relations in the image since removing it has larger drop on SPICE. The lager drop on vocabulary coverage and ratio of novel sentence in the second row indicates that the conditional term plays a significant role in increasing the diversity of the generated captions. More results on using different model architectures are included in Appendix D.
Comparison with existing methods. Based on the learning methods, existing models are divided into three categories in Table 6, and we chose recent proposed methods for comparison: Att2in (Rennie et al., 2017), G-GAN (Dai & Lin, 2017), NBT (Lu et al., 2018), Up-Down (Anderson et al., 2018) and GAN2, GAN3 (Dognin et al., 2019). Although some metrics based on n-gram overlapping (BLEU4, CIDEr) do not correlate well with human, their results are also reported in Table 6 for fair comparison. Among the adversarial methods (GAN category), our rAIRL performs the best on all metrics. The results on COCO online server are given in Appendix E.
To further demonstrate the generalization ability of our algorithm, we built our algorithm on the non-adversarial based models. The composite models are denoted with rAIRL+MLE and rAIRL+RL. In rAIRL+MLE, the conditional term is replaced by the cross-entropy loss of MLE; in rAIRL+RL, the RL loss is added into the loss function of the generator. In Table 6, our rAIRL+MLE further improves the MLE baseline (i.e., Up-Down using MLE loss) on SPICE, whereas rAIRL+RL does not improve the RL baseline (i.e., Up-Down using RL loss) on these evaluation metrics. This is caused by the difficulty of normalizing the learned reward and the handcrafted reward to the same order of magnitude (Shelton Christian, 2001), and we leave this problem to our future work.

heading: 6 CONCLUSION
text: In this paper, we address the reward ambiguity problem in image captioning and propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method that solves the problem by disentangling reward for each word in a sentence. Moreover, it achieves stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium, and mode control technique is incorporated to mitigate mode collapse. It is demonstrated that our method can learn compact reward through extensive experiments on MS COCO.

heading: B DIAGNOSE AND IMPROVE CAPTIONS
text: Since the proposed rAIRL learns a word-wise reward, it’s also applicable to diagnose a given caption by finding the wrong word (e.g., the word whose reward decreases sharply compared with that of the previous word) and rewriting the caption to improve its quality. For example, improving a man is playing soccer to a man and a kid are playing soccer can be done by rewriting the caption from is. Therefore, we choose to rewrite a given caption (source caption) from the word whose reward has a decrease rate larger than 50%. However, we found that even rewriting the source caption from a random position using rAIRL can also improve the evaluation scores. Thus, rewriting from a random position is selected as the baseline to compare with rewriting from the located position. Table 7 shows results of rewriting from the located position, where the source captions are given by MLE, RL, GAN and AIRL. Beside each score we report its improvement relative to rewriting from a random position, whose values are mostly positive. This demonstrates that the proposed rAIRL can diagnose the caption at a word level, and further improves the caption quality by rewriting from the located position.

heading: C HUMAN EVALUATION PROCESS
text: We conducted two types of human studies, one for evaluating the learned reward (in Section 5.2), and the other for examining quality of the generated captions (in Section 5.3). In the first human study experiment (in Section 5.2), we used the human scores in the COMPOSITE3 dataset (Aditya et al., 2017), whose images are subsets from Flickr8k, Flickr30k and MS COCO. The descriptions from this dataset are either ground truth captions or generated sentences by (Aditya et al., 2017; Johnson et al., 2015). In the human evaluation process, the AMT worker was asked to give a score at range of
3https://imagesdg.wordpress.com/image-to-scene-description-graph/
1-5 to evaluate the correctness and throughness of each sentence. Captions with length exceeding 20 were discarded, resulting a total of 11, 657 sentences. Full results of the correlation is shown in Table 8. SPICE correlates better with human evaluation when compared with other handcrafted metrics, whilst the composite metric rAIRL+SPICE further increases the correlation.
In the second human study experiment (in Section 5.3), we randomly selected 500 test images from the standard split and robust split of MS COCO, respectively. The worker was asked “which caption is the best” by given an image with five sentences generated from the adversarial and non-adversarial methods, as shown in Figure 7. The worker was allowed but not encouraged to make multiple choices if he/she thinks these captions are equally correct. The order of captions produced by different methods was randomized. Following (Shetty et al., 2017), each image in the test set was evaluated by 5 workers.

heading: D ABLATION EXPERIMENTS ON MODEL ARCHITECTURES
text: Theoretically, our algorithm is model-agnostic since it is independent of the design of model architecture. For empirical support of the claim, we show results of using Att2in (Rennie et al., 2017) and Up-Down (Anderson et al., 2018) architectures in Table 9. We report the metrics used in the original paper for fair comparison. The proposed rAIRL mainly improves SPICE, which correlates well with human evaluations, on both architectures.

heading: E FULL RESULTS ON MS COCO
text: We adopt SPICE to evaluate content correctness in the paper because it has better correlation with human judgments(Anderson et al., 2016). Table 10 gives full results of the handcrafted metrics on two splits of MS COCO. Comparing the adversarial (GAN, AIRL, rARIL) and non-adversarial (MLE, RL) methods, RL outperforms other methods on most metrics. In adversarial methods, the proposed rAIRL performs the best. Table 11 shows results on the MS COCO online test server. The proposed rAIRL improves AIRL on all the metrics.

heading: F DISCUSSION ON LOSS FUNCTIONS
text: We compare the formula of the proposed loss function with existing methods in Table 12, including MLE, RL and GAN. n is the length of a sentence. r is the handcrafted metric, such as BLEU, CIDEr and SPICE. πt is the probability of the tth generated word, and πttrue is the probability of the tth true
word. The loss functions are rewritten using similar symbols for direct comparison. MLE maximizes the probability of the true data πtruet whist RL and GAN optimize the reward by sampling from πt. GAN is different from RL in that its reward is learned from the discriminator adversarially instead of being predefined. GAN is capable of mimicking human-written captions by adversarial learning, but the estimated reward function Dgen of a full trajectory can be explained by multiple optimal policies and thus is too ambiguous. The proposed rAIRL further disentangles the reward into Dgent at each time step t, as well as incorporating the true data for better diversity. From the perspective of loss functions, rAIRL can be regarded as an integration of GAN and MLE using coefficients σ−1(Dgent ) and σ−1(Dtruet ).

heading: G GENERATOR OBJECTIVE
text: Below we show the deductions details of the generator objective (i.e., Eq. (16) in section 4.2).
Taking the integral of ∇ψEwt∼πψ [·] w.r.t. ψ Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1] = ∫ ψ ∇ψEwt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1]dψ (12)
According to Eq. (10),
∇ψEwt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1] = 1
πψ (fθ,ϕ(wt, st)− log(πψ))∇ψπψ (13)
Thus we have Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1] = ∫ ψ 1 πψ (fθ,ϕ(wt, st)− log(πψ))∇ψπψdψ
= ( fθ,ϕ(wt, st)− log(πψ) ) log(πψ)
(14)
Similarly, replacing wt with wtruet we get
Ewtruet ∼πtrueψ [fθ,ϕ(w true t , s true t )− log(πtrueψ ) + 1] =
( fθ,ϕ(w true t , s true t )− log(πtrueψ ) ) log(πtrueψ ) (15)
Thus, the generator objective is:
Lt(ψ) = −Ewt∼πψ [fθ,ϕ(wt, st)− log(πψ) + 1]− Ewtruet ∼πtrueψ [fθ,ϕ(w true t , s true t )− log(πtrueψ ) + 1] = − ( fθ,ϕ(wt, st)− log(πψ) ) log(πψ)− ( fθ,ϕ(w true t , s true t )− log(πtrueψ ) ) log(πtrueψ )
(16)

Review summaries:
The authors propose using a recent method for adversarial inverse reinforcement learning (AIRL) for the task for generating high-quality image captions. Leveraging the GAN framework, a discriminator is trained to distinguish real captions from those produced by the generator, while the generator is optimized with policy-gradients (REINFORCE) to maximize the pseudo-reward from the discriminator. The main difference from prior work seems to be that the discriminator acts on a word-level, rather than sentence-level (as done, for instance in Dai et. al. 2017). Correspondingly, the generator policy is updated with the objective of 1-step reward maximization (more like contextual bandits), rather than with a long-term sequential decision-making objective (as done in Dai et. al. 2017). The evaluation is done using 2 data splits – standard and robust, with various metrics such as SPICE, CIDEr, BLEU, CHAIR. Diversity analysis and ablations are also performed to dissect the performance of the proposed approach. 

This paper proposes a refined Adversarial Inverse Reinforcement Learning (rAIRL) to remedy the reward ambiguity by decoupling the reward for each word in a sentence, while the existing methods that utilize reinforcement learning to optimize evaluation score handle only sentence-level rewards. Furthermore, a conditional term is introduced in the loss function to avoid mode collapse and to increase the diversity of the generated captions. Throughout experiments on MS COCO show that the proposed method achieves state-of-the-art performance with several evaluation scores.