Title: LEARNING COMPACT REWARD FOR IMAGE CAPTION-ING

Abstract: Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.

1. INTRODUCTION:

Image captioning is a task of generating descriptions of a given image in natural language. In a general encoder-decoder structure (Vinyals et al., 2015) , image features are encoded in a CNN and decoded into a caption in a word by word manner. Based on the loss function, standard approaches to the problem could be divided into three categories: MLE (Maximum Likelihood Estimation), RL (Reinforcement Learning) and GAN (Generative Adversarial Network).

Early proposed methods were based on MLE function and made improvements by designing specific model structure (Xu et al., 2015) . MLE adopts the cross-entropy loss and learns a one-hot distribution for each word in the sentence. By maximizing the probability of the ground truth word whilst suppressing other reasonable vocabularies, the probability distribution learned by MLE tends to be sparse and the generated captions have limited diversity (Dai et al., 2017) . On the other hand, RL has advantages in boosting the model performance by optimizing the handcrafted metrics (Rennie et al., 2017; Liu et al., 2017; Chen et al., 2019) . However, due to the reward hacking problem, RL maximizes the reward in an unintended way and fails to produce human-like descriptions (Li et al., 2019a) . Considering naturalness and diversity of the generated captions, GAN has raised attention in image captioning for its capability of producing descriptions that are indistinguishable from human-written ones (Dai et al., 2017; Shetty et al., 2017; Chen et al., 2019; Dognin et al., 2019) .

In image captioning, the generator of GAN learns true data distribution by maximizing the reward function learned from a discriminator, and the discriminator distinguishes the generated sample from the true data. The adversarial training converges to an equilibrium point (i.e., Nash equilibrium) at which both the generator and discriminator cannot improve (Goodfellow et al., 2014) . As shown in Figure 1 , the learned distribution of GAN is closer to the ground truth distribution than that of other methods (i.e., MLE and RL) on different splits. However, previous work of adversarial networks in image captioning gives one reward function D for a complete sentence consisting of n words. This strategy causes the reward ambiguity problem (Ng et al., 1999) since which word(s) causes the reward to increase or decrease is not accounted for, and thus there are many optimal policies that determine the sentence can explain one reward. An example is that each image in MS COCO has five ground truth captions. Although these captions may vary in formats, each caption has the same reward value in GAN. From the perspective on the system level, learning sentence-level reward from different image-caption pairs is analogous to learning reward of a trajectory from different system dynamics, which makes the discriminator unable to distinguish the true reward functions from those shaped by the environment dynamics (Fu et al., 2018) .

(a) Standard split (Karpathy & Fei-Fei, 2015) (b) Robust split (Lu et al., 2018) Figure 1 : Comparison of word count ratios (Shetty et al., 2017) on two splits of MS COCO. x axis is the {test frequency}/{train frequency} of a word and y axis is the word count of the corresponding ratio. GT represents ground truth distribution.

Facing the challenge, we adopt AIRL (Fu et al., 2018) to solve the reward ambiguity problem by disentangling reward for each action (i.e., word in a sentence) from different image-caption pairs and learning a compact reward function. compact means words with similar semantics, such as children and kids, correspond to close reward values. Driven by the compact reward function of the discriminator, the generator learns the optimal policy and thus produces qualitative descriptions. However, there are still two major problems to address: 1) AIRL is difficult to converge to Nash equilibrium using policy gradient (See Section 4.2 for details); 2) AIRL is designed without mode control, and thus the outputs have limited diversity, which is a commonly encountered issue called mode collapse (Mirza & Osindero, 2014) .

In this paper, we propose a refined AIRL method to learn a compact reward function for each word, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. The refined method makes it possible to reach the equilibrium point for a non-concave model function of the generator. In addition, a conditional term is introduced in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Both the caption evaluator (i.e., discriminator) (Cui et al., 2018; Sharif et al., 2018) and the generator are cast into this unified framework, where the discriminator evaluates captions using a learned compact reward function, and the generator produces qualitative image descriptions. We demonstrate the effectiveness of our method in the experiments.

2. RELATED WORK:

Image Captioning. The development of image captioning can be summarized into two directions: model structure design (Lu et al., 2017; Yao et al., 2018) and loss function construction (Rennie et al., 2017; Ren et al., 2017) . In the methods based on model structure design, attention mechanism and the fusion of visual and semantic information are the key focus. Lu et al. (2017; 2018) proposed a sentinel gate to learn adaptive attention between visual content and non-visual text. Yao et al. (2018) explored the role of visual relationship in image captioning. On the other hand, methods based on loss function construction focus on optimization of the loss function. Rennie et al. (2017) optimized on non-differentiable evaluation metric using policy gradient, and improved scores of these metrics on various models. Ren et al. (2017) designed an embedding reward under actor-critic reinforcement learning. Similarly, we address the construction of loss functions, and thus our algorithm can be built on existing model structures. See Appendix F for a short discussion about different loss functions.

Adversarial Methods for Image Captioning. Adversarial methods are known for producing plausible samples by training the generator and the discriminator in an adversarial manner (Goodfellow et al., 2014) . In image captioning, the discriminator is formed as a binary classifier that distinguishes the generated sentence from the ground truth, while the generator produce captions that can fool the discriminator. Conditional GAN was proposed in (Dai et al., 2017) to improve the naturalness and diversity of generated captions. CNN and RNN based discriminators were introduced in (Chen et al., 2019) . However, existing methods estimate a reward function for the complete sentence consisting of n words, where multiple optimal policies that determine the sentences can correspond to one reward (Ng et al., 1999) . Thus the learned reward is ambiguous and ill-defined. We solve this problem by recovering a compact reward function for each word in the sentence under a refined AIRL framework.

Although AIRL has been utilized to solve problems in other fields (Wang et al., 2018; Li et al., 2019b; Shi et al., 2018) , we are the first to make algorithmic improvements to AIRL such that Nash equilibrium can be reached even for a non-concave model function of the generator, and that diversity of the outputs can be increased.

3. ADVERSARIAL INVERSE REINFORCEMENT LEARNING:

Due to the high variance estimate of a full sentence and the reward ambiguity problem, instead of learning reward for a complete sentence, we could learn reward distribution p θ (w t , s t ) for each word-state pair (w t , s t ) so that the true reward can be recovered at optimality (Fu et al., 2018) . In the following, we introduce how AIRL disentangles reward for each word-state pair (w t , s t ).

AIRL is an adversarial reward learning algorithm based on Maximum-Entropy-IRL. Finn et al. (2016) first proved that Maximum-Entropy-IRL is mathematically equivalent to GAN under a special form of the discriminator:

D θ (w t , s t ) = p θ (w t , s t ) p θ (w t , s t ) + π(w t , s t ) (1) p θ (w t , s t ) = exp {f θ (w t , s t )}

(2) where p θ (w t , s t ) is the actual probability distribution estimated by the discriminator. π(w t , s t ) is the policy distribution produced by the generator, which is also called vocabulary distribution under the context of image captioning.

The goal is to estimate p θ (w t , s t ) that approximates the true data distribution p true (w t , s t ), as well as to learn an optimal vocabulary distribution π that maximizes the reward. Subsequently, considering reward ambiguity problem, Fu et al. (2018) further extended the theory to AIRL by adding a reward shaping term h ϕ into f θ (w t , s t ), and thus disentangled reward from different system dynamics (i.e., image-caption pairs):

EQUATION

where g θ denotes the reward approximator that recovers the true reward up to a constant, and h ϕ is the reward shaping term that preserves the optimal π. γ is a constant in range (0, 1].

In the context of divergence minimization, the adversarial process can be represented as a min-max game (Mescheder & Geiger, 2017) :

EQUATION

where p true is the true data distribution and π is the vocabulary distribution learned by the generator.

(w true t , s true t ) is the word-state pair of the true data. Despite of the capability of AIRL in disentangling reward for each word, it is difficult for the above AIRL algorithm to converge to Nash equilibrium and to produce diverse outputs through adversarial training (See Section 4.2 for details). These issues can result in a non-optimal solution and lack of diversity of the generated descriptions. Aiming to learn the optimal compact reward as well as diverse captions, we refine the loss function to shift the stationary point towards Nash equilibrium and to mitigate mode collapse in the two-player game.

4. LEARNING COMPACT REWARD FOR IMAGE CAPTIONING:

To address the problems discussed above, we refine the loss function to: 1) find a compact reward function that is optimal; 2) increase diversity of the generated captions. In particular, a constant term is used to solve 1) by shifting the stationary point to Nash equilibrium, and a conditional term is introduced to solve 2) by utilizing mode control techniques. Our algorithm is detailed in Algorithm 1, where n is the sentence length and N denotes number of iterations.

In the following notations, θ and ϕ are the parameters of the discriminator, and ψ represents the parameter of the generator. w t and s t denote the t th word and its corresponding hidden state vector, respectively. For better clarity, policy π ψ is hereinafter referred to as vocabulary distribution.

Algorithm 1: Refined ARIL Initialize the vocabulary distribution π ψ and discriminator f θ,ϕ . for iteration i in {1, ..., N } do Obtain caption {w true 1 , ..., w true n } from the ground truth. Collect generated caption {w 1 , ..., w n } using the vocabulary distribution π ψ . D θ,ϕ ←sigmoid(f θ,ϕ -log(π ψ )) Update (θ, ϕ) via Eq. ( 5) for the discriminator. Update ψ via Eq. ( 16) for the generator. end

4.1. DISCRIMINATOR:

The objective of the discriminator is to distinguish the true caption from the generated one. At time t, the discriminator maximizes the divergence in Eq. ( 4) by

L t (θ, ϕ) = -log D θ,ϕ (w true t , s true t ) w true t ∼ptrue -log 1 -D θ,ϕ (w t , s t ) wt∼π ψ (5)

where p true is the distribution of the true caption, and π ψ is the vocabulary distribution estimated by the generator. D θ,ϕ is computed as in Eq. ( 1) and Eq. ( 2), where the discriminator learns the reward function f θ,ϕ for D θ,ϕ and the generator estimates the vocabulary distribution π ψ for D θ,ϕ , respectively.

4.2. GENERATOR:

In the following, D θ,ϕ is represented as below (Fu et al., 2018) using Eq. ( 1) and Eq. ( 2):

EQUATION

Given word w t that is sampled from the vocabulary distribution π ψ , the generator maximizes D θ,ϕ (w t , s t ) by

EQUATION

Using REINFORCE algorithm (Sutton & Barto, 1998) , the gradient ∇ ψ L t becomes:

EQUATION

When the generator converges (i.e.,∇ ψ L t = 0), there exists two stationary points:

∇ ψ π ψ = 0 and log(π ψ ) = f θ,ϕ (w t , s t ) -1.

If Nash equilibrium can be reached at optimality, D θ,ϕ should converge to 0.5 when ∇ ψ L t = 0. Thus it's only possible for the first point to reach Nash equilibrium since D θ,ϕ = sigmoid f θ,ϕ (w t , s t ) -log(π ψ ) = sigmoid(1) = 0.5 at the second point. However, even for the first point, Nash equilibrium exists only for a concave π ψ , requiring Hessian of the gradient vector filed being positive definite (Mescheder & Geiger, 2017) . To relax this constraint, a constant term is added into the expectation in Eq. ( 7)

L t (ψ) = -E wt∼π ψ [f θ,ϕ (w t , s t ) -log(π ψ ) + 1] (9) ∇ ψ L t = - 1 π ψ (f θ,ϕ (w t , s t ) -log(π ψ ))∇ ψ π ψ (10)

to expand the feasible region of reaching Nash equilibrium by shifting the second stationary point to D θ,ϕ (w t ) = sigmoid(0) = 0.5. The constant term makes it possible to achieve Nash equilibrium even for a non-concave π ψ , which also makes it easier for the adversarial model to converge . It is noted that the constant term can also be regarded as baseline in REINFORCE, except it is utilized to centralize the stationary point instead of reducing variance of the estimation.

In practice, mode collapse occurs when the generator produces a single or limited modes, which exhibits as little diversity in image captioning. To mitigate mode collapse (Mirza & Osindero, 2014) and increase the diversity of the generated captions, we add ground truth data into the generator as a conditional term:

EQUATION

) where π true ψ is the approximated true caption distribution in the generator, and E w true

t ∼π true ψ [•]

is the conditional term. See Appendix G for the deduction details.The coefficient of log(π true ψ ) is symmetrical to the coefficient of log(π ψ ) and is updated adaptively in the training process. The conditional term helps in strengthening the generator in the adversarial training. When D true > D gen , the gradient of the true data becomes larger than that of the generated one (∇ π true ψ L t > ∇ π ψ L t ), and thus the generator further increases the probability of the true word (π true ψ ). Otherwise (i.e., D true < D gen ), the generator prefers sampling its self-generated words to fool the discriminator. By switching between the true words and the generated ones, the generator maintains informative gradient during the adversarial training (Peng et al., 2019) . Note that adding the conditional term does not change the model's convergence to Nash equilibrium since π ψ = π true ψ at the second stationary point.

5. EXPERIMENTS:

In the experiments, we validate the effectiveness of the proposed algorithm by answering three questions: 1) Is the caption evaluator (i.e., discriminator) capable of learning compact reward? 2) Driven by the learned reward, is the caption generator effective to produce qualitative captions? 3) How does our algorithm perform when built on or compared with existing methods?

To answer 1), we first tested the compactness of the learned reward by observing performance of the collected top-k captions. Then we explored the correlation between the learned reward and the human evaluation results. To answer 2), we evaluated the quality of the generated caption on its content, diversity and grammar. To answer 3), we built our algorithm on existing learning methods and compared their performance. We also conducted ablation experiments to demonstrate the importance of each component of our algorithm.

5.1. IMPLEMENTATION DETAILS:

We conducted experiments on the well-known benchmark dataset MS COCO (Chen et al., 2015) . The dataset has 123,287 labeled images and each image has at least 5 human annotated captions as reference. To assess the robustness of our algorithm, we use two splits of the COCO dataset: standard split (Karpathy & Fei-Fei, 2015) which is created by randomly picking test images, and robust split (Lu et al., 2018) which is organized to maximize difference of the co-occurrence distribution between the training and test set. The robust split is recently proposed and is more challenging due to its distribution difference between the training and test set. The standard split has 113287/5000/5000 train/val/test images and the robust split has 110234/3915/9138 train/val/test images.

We implement our algorithm using Adam optimizer (Kingma & Ba, 2014) with fixed learning rate 10 -5 . Our vocabulary size is fixed to 10,000 including a special start sign <BOS>and an end sign <EOS>. In the generator, the number of hidden nodes of every layer is 512. For simplicity, the discriminator has the same model structure as the generator except having one additional layer for estimating h ϕ . For fair comparison, all the methods in MLE, RL, GAN, AIRL and rAIRL were produced using the same image features and model structure in (Anderson et al., 2018) . Specifically, RL is the self-critical algorithm in (Rennie et al., 2017) . GAN is the adversarial algorithm in (Dai & Lin, 2017) that learns sentence-level reward. AIRL is the standard adversarial inverse reinforcement learning method in (Fu et al., 2018) , and rAIRL is the proposed method. Note that our scores of MLE are lower on the standard split but higher on the robust split than (Anderson et al., 2018) because 1) we used fixed number of the bounding box (i.e., 36) for simplicity; 2) the hyperparameters were tuned to adapt to both splits and thus are not exactly the same with (Anderson et al., 2018) . Compactness. Compactness means that the reward values should be close for similar words and different for distinct words. For example, kid can also be referred to as little boy or little girl, and thus their reward values should be close to each other in the discriminator. To see the correlation between the reward differences and semantic differences, we replace a given word w t in the generated caption with a similar word w similar and a distinct word w distinct , respectively. Specifically, in a generated caption, the first word that belongs to the COCO 80 class1 (Lu et al., 2018) is replaced. A sentence is discarded if no word can be replaced. The words within the same class are considered to be similar (such as bike and bicycle), and the words that belong to difference classes are distinct (such as man and bike). For RL, since it maximizes a handcrafted reward (SPICE (Anderson et al., 2016) in our experiment) instead of learning a reward function, the reward difference is the variation of SPICE before and after replacement. For reward-learning methods, the reward difference is the variation of the learned reward. The semantic difference is the Euclidean distance between the Glove embedding vectors of two words (Pennington et al., 2014) . The results are reported in Table 1 . Higher correlation indicates better compactness. RL serves as a baseline in that the handcrafted reward SPICE compares n-gram overlapping without considering the semantic difference. The reward differences of rAIRL correlate the best with the semantic differences for both similar words and distinct words, proving the compactness of the learned reward. It's also noted that due to the reward ambiguity problem, the reward differences of GAN poorly correlate with the semantic differences, especially for similar words. Correlation with human evaluation. As a caption evaluator, the discriminator learns g θ that recovers the true reward up to a constant at optimality (Fu et al., 2018) . We explore the correlation between the recovered reward and the human evaluation scores from the COMPOSITE dataset (Aditya et al., 2017) , where the Amazon Mechanical Turk (AMT) workers evaluate two aspects of the captions (i.e., correctness and throughness) at a range of 1-5, see Appendix C for details of the human evaluation process. The correlation is evaluated using Pearson p, Kendall's τ and Spearman's r correlation coefficients. In Table 2 , the reward of AIRL/rAIRL is the sum of the word-wise reward g θ , and the reward of rAIRL+SPICE is a linear combination of g θ and the SPICE score. Among the reward-learning methods, AIRL poorly correlates with human, whereas the proposed rAIRL improves AIRL on all the correlation metrics, especially on the Pearson correlation (from 0.04 to 0.43). Furthermore, a simple combination of SPICE and the recovered reward leads to an increased correlation with the human scores, which proves the capacity of the discriminator as a caption evaluator. Content correctness. For a comprehensive evaluation of the content correctness, the results of both the handcrafted metrics and human studies are shown in Table 3 . For the handcrafted metrics, we report scores of SPICE and the recently proposed CHAIR s and CHAIR i since they correlate well with human (Anderson et al., 2016; Rohrbach et al., 2018) 3 . VC indicates vocabulary coverage, which is the number of vocabularies of the generated captions over number of vocabularies of the ground truth captions. NS represents ratio of novel sentence, which is the ratio of sentences that do not appear in the training set. The fact that RL has high ratio of novel sentence (88.5%/87.3%) but low vocabulary coverage (11.4%/12.7%) suggests that it uses high-frequency words (such as "in a", "of a") to reconstruct captions that appear to be different from the training set (Li et al., 2019a) . See Appendix A for a few examples. Our rAIRL improves AIRL on the diversity metrics and outperforms other learning methods on vocabulary coverage, indicating its capability of generating diverse descriptions on a corpus level. Grammar. We used LanguageTool2 to check grammar of the generated captions. Table 4 shows percentage of sentences that have grammar errors found by LanguageTool: 1) Redundancy means repeated phrases in a sentence; 2) Agreement Error means subject-verb agreement error, such as "people is"; 3) Article Misuse denotes inappropriate usage of indefinite articles, such as using "a" before uncountable nouns or plural words; 4) Incomplete Sentence refers to incomplete sentence that lacks a subject. We found captions produced by RL have the most grammar errors (5.64% on the standard split and 4.67% on the robust split), especially the Article Misuse. On the other hand, by approximating the true data distribution of each word in the sentence, rAIRL and MLE have the least grammar errors among all learning methods (0.75%/0.78% on the standard split and 0.57%/0.57% on the robust split)). We also noticed that each method except rAIRL is biased towards a particular type of grammar error: agreement error in MLE, article misuse in RL, redundancy in GAN, article misuse in AIRL. On both splits, our rAIRL does not appear to be biased towards a specific type of these grammar errors.

5.2. PERFORMANCE OF THE RECOVERED REWARD:



5.3. EVALUATION ON THE GENERATED CAPTIONS.:

Summary. The proposed rAIRL constantly performs well on both splits of MS COCO and is capable of producing qualitative captions with few grammar errors. As a new adversarial algorithm, rAIRL enhances GAN by disentangling compact reward for each word in the caption and improves AIRL by shifting the stationary point towards Nash equilibrium. In the following sections, we first give ablation studies to see which component of our method explains the performance improvements, and then compare rAIRL with existing methods.

5.4. COMPARISON RESULTS:

Table 5 : Ablation methods of rAIRL. "term1" is the constant term in Eq. ( 9) and "term2" is the conditional term in Eq. ( 16). GE denotes grammar error rate. Ablation studies. We conducted ablation experiments to understand the importance of each component of our algorithm. Specifically, the constant term in Eq. ( 9) and the conditional term in Eq. ( 16) is removed, respectively. Scores of all the evaluation techniques mentioned above are presented in Table 5 . We found that all the scores drop after removing either one of the terms. Comparing these two terms, the constant term seems more important in recognizing objects and relations in the image since removing it has larger drop on SPICE. The lager drop on vocabulary coverage and ratio of novel sentence in the second row indicates that the conditional term plays a significant role in increasing the diversity of the generated captions. More results on using different model architectures are included in Appendix D. Comparison with existing methods. Based on the learning methods, existing models are divided into three categories in Table 6 , and we chose recent proposed methods for comparison: Att2in (Rennie et al., 2017) , G-GAN (Dai & Lin, 2017) , NBT (Lu et al., 2018) , Up-Down (Anderson et al., 2018) and GAN 2 , GAN 3 (Dognin et al., 2019) . Although some metrics based on n-gram overlapping (BLEU4, CIDEr) do not correlate well with human, their results are also reported in Table 6 for fair comparison. Among the adversarial methods (GAN category), our rAIRL performs the best on all metrics. The results on COCO online server are given in Appendix E.

To further demonstrate the generalization ability of our algorithm, we built our algorithm on the non-adversarial based models. The composite models are denoted with rAIRL+MLE and rAIRL+RL.

In rAIRL+MLE, the conditional term is replaced by the cross-entropy loss of MLE; in rAIRL+RL, the RL loss is added into the loss function of the generator. In Table 6 , our rAIRL+MLE further improves the MLE baseline (i.e., Up-Down using MLE loss) on SPICE, whereas rAIRL+RL does not improve the RL baseline (i.e., Up-Down using RL loss) on these evaluation metrics. This is caused by the difficulty of normalizing the learned reward and the handcrafted reward to the same order of magnitude (Shelton Christian, 2001) , and we leave this problem to our future work.

6. CONCLUSION:

In this paper, we address the reward ambiguity problem in image captioning and propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method that solves the problem by disentangling reward for each word in a sentence. Moreover, it achieves stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium, and mode control technique is incorporated to mitigate mode collapse. It is demonstrated that our method can learn compact reward through extensive experiments on MS COCO.

A VISUALIZED RESULTS OF GENERATED CAPTIONS Since the proposed rAIRL learns a word-wise reward, it's also applicable to diagnose a given caption by finding the wrong word (e.g., the word whose reward decreases sharply compared with that of the previous word) and rewriting the caption to improve its quality. For example, improving a man is playing soccer to a man and a kid are playing soccer can be done by rewriting the caption from is. Therefore, we choose to rewrite a given caption (source caption) from the word whose reward has a decrease rate larger than 50%. However, we found that even rewriting the source caption from a random position using rAIRL can also improve the evaluation scores. Thus, rewriting from a random position is selected as the baseline to compare with rewriting from the located position. Table 7 shows results of rewriting from the located position, where the source captions are given by MLE, RL, GAN and AIRL. Beside each score we report its improvement relative to rewriting from a random position, whose values are mostly positive. This demonstrates that the proposed rAIRL can diagnose the caption at a word level, and further improves the caption quality by rewriting from the located position.

Figure 6 : Examples showing the generated captions from AIRL before and after re-written.

C HUMAN EVALUATION PROCESS:

We conducted two types of human studies, one for evaluating the learned reward (in Section 5.2), and the other for examining quality of the generated captions (in Section 5.3). In the first human study experiment (in Section 5.2), we used the human scores in the COMPOSITE3 dataset (Aditya et al., 2017) , whose images are subsets from Flickr8k, Flickr30k and MS COCO. The descriptions from this dataset are either ground truth captions or generated sentences by (Aditya et al., 2017; Johnson et al., 2015) . In the human evaluation process, the AMT worker was asked to give a score at range of We compare the formula of the proposed loss function with existing methods in Table 12 , including MLE, RL and GAN. n is the length of a sentence. r is the handcrafted metric, such as BLEU, CIDEr and SPICE. π t is the probability of the t th generated word, and π t true is the probability of the t th true word. The loss functions are rewritten using similar symbols for direct comparison. MLE maximizes the probability of the true data π true t whist RL and GAN optimize the reward by sampling from π t . GAN is different from RL in that its reward is learned from the discriminator adversarially instead of being predefined. GAN is capable of mimicking human-written captions by adversarial learning, but the estimated reward function D gen of a full trajectory can be explained by multiple optimal policies and thus is too ambiguous. The proposed rAIRL further disentangles the reward into D gen t at each time step t, as well as incorporating the true data for better diversity. From the perspective of loss functions, rAIRL can be regarded as an integration of GAN and MLE using coefficients σ -1 (D gen t ) and σ -1 (D true t ).

G GENERATOR OBJECTIVE:

Below we show the deductions details of the generator objective (i.e., Eq. ( 16) in section 4.2).

Taking 

:

https://github.com/jiasenlu/NeuralBabyTalk/blob/master/data/coco/coco_ class_name.txt

https://languagetool.org

https://imagesdg.wordpress.com/image-to-scene-description-graph/

Figures and Tables:

Figure 2: Captions produced by different methods from the test set (standard split). Beside each caption we report SPICE score. Captions generated by rAIRL are correct and human-like in these examples.

Figure 3: Failed examples of rAIRL. The objects and relations are not correctly recognized in these pictures.

Figure 4: Examples showing diversity of the captions. The left and right columns show pictures with similar content but different details. The proposed rAIRL successfully recognizes these differences and gives diverse captions.

Figure 5: Examples of the top-5 generated captions of rAIRL. Driven by the compact reward function, the generator describes a given image with semantically similar words.

Correlation between the reward differences and semantic differences by replacing a given word with a similar word (RP S) and a distinct word (RP D), respectively.

Table content: <table><tr><td>Method</td><td colspan="2">Standard Split</td><td colspan="2">Robust Split</td></tr><tr><td/><td colspan="4">RP S RP D RP S RP D</td></tr><tr><td>RL</td><td>0.03</td><td>0.01</td><td>-0.10</td><td>0.00</td></tr><tr><td>GAN</td><td>0.07</td><td>0.21</td><td>0.04</td><td>0.18</td></tr><tr><td>AIRL</td><td>0.15</td><td>0.11</td><td>0.30</td><td>0.20</td></tr><tr><td>rAIRL</td><td>0.54</td><td>0.30</td><td>0.51</td><td>0.31</td></tr></table>

Sentence-level correlation with human evaluation. All p-value (not shown) are less than 0.001.

Table content: <table><tr><td>Method</td><td/><td>Correctness</td><td/><td/><td>Throughness</td><td/></tr><tr><td/><td colspan="6">Pearson Spearman Kendall Peason Spearman Kendall</td></tr><tr><td>SPICE</td><td>0.44</td><td>0.45</td><td>0.39</td><td>0.45</td><td>0.46</td><td>0.38</td></tr><tr><td>GAN</td><td>0.12</td><td>0.11</td><td>0.15</td><td>0.12</td><td>0.11</td><td>0.15</td></tr><tr><td>AIRL</td><td>0.04</td><td>0.06</td><td>0.08</td><td>0.05</td><td>0.06</td><td>0.07</td></tr><tr><td>rAIRL</td><td>0.43</td><td>0.40</td><td>0.35</td><td>0.40</td><td>0.37</td><td>0.34</td></tr><tr><td>rAIRL+SPICE</td><td>0.47</td><td>0.46</td><td>0.41</td><td>0.46</td><td>0.47</td><td>0.39</td></tr></table>

Evaluation scores on generated captions. The best score is in bold font and the second best score is underlined. SPICE is the handcrafted evaluation metric. CHAIRs and CHAIRi represent the object hallucination ratio at sentence level and instance level, respectively. HE indicates human evaluation. VC indicates vocabulary coverage and NS is the ratio of novel sentences.

Table content: <table><tr><td>Method</td><td/><td colspan="2">Standard Split</td><td/><td/><td/><td/><td>Robust Split</td></tr><tr><td/><td colspan="4">SPICE CHAIR s CHAIR i HE</td><td>VC</td><td colspan="4">NS SPICE CHAIR s CHAIR i HE</td><td>VC</td><td>NS</td></tr><tr><td>MLE</td><td>19.0</td><td>8.3</td><td>6.0</td><td colspan="3">16.1 12.4 49.7</td><td>18.6</td><td>19.1</td><td>16.9</td><td>18.0 12.5 58.8</td></tr><tr><td>RL</td><td>20.7</td><td>11.4</td><td>8.5</td><td colspan="3">8.7 11.4 88.5</td><td>18.1</td><td>25.2</td><td>20.4</td><td>6.4 12.7 87.3</td></tr><tr><td>GAN</td><td>18.3</td><td>7.6</td><td>6.4</td><td colspan="3">19.9 13.4 75.0</td><td>16.8</td><td>17.3</td><td>15.2</td><td>20.2 15.3 75.6</td></tr><tr><td>AIRL</td><td>17.3</td><td>12.7</td><td>10.3</td><td colspan="3">14.0 12.3 67.3</td><td>16.7</td><td>22.7</td><td>18.5</td><td>14.8 15.6 73.8</td></tr><tr><td>rAIRL</td><td>20.4</td><td>7.2</td><td>5.5</td><td colspan="3">41.3 13.6 76.1</td><td>18.7</td><td>17.1</td><td>14.3</td><td>40.6 15.8 76.5</td></tr></table>

. SPICE computes similarity with the ground truth captions based on scene graph whilst CHIAIR s and CHIAR i indicate ratio of hallucinated objects. The full results of other handcrafted metrics can be found in Appendix E. Compared with non-adversarial methods (i.e., MLE, RL), existing adversarial net (GAN) does not perform well on SPICE due to the reward ambiguity problem, whereas our rAIRL improves GAN (from 16.8 to 18.7) by disentangling reward for each word, and even outperforms RL (from 18.1 to 18.7) on the robust split. The lowest scores on CHIAIR s and CHIAR i suggest that object hallucination is less likely in rAIRL. As for the human evaluation, HE in Table3indicates the percentage of captions that are considered the best among the five methods. See Appendix C for details of the human evaluation process. The descriptions generated by our rAIRL are considered the best for over 40% images, whilst RL has the lowest scores that are less than 10%.

Table content: <table/>

Percentage of different grammar errors found in the generated captions. Re represents Redundancy, AE is Agreement Error, AM denotes Article Misuse and IS is Incomplete Sentence.

Table content: <table><tr><td>Method</td><td/><td colspan="3">Standard Split</td><td/><td/><td colspan="2">Robust Split</td></tr><tr><td/><td>Total</td><td>Re</td><td colspan="2">AE AM</td><td>IS</td><td>Total</td><td>Re</td><td>AE AM</td><td>IS</td></tr><tr><td>MLE</td><td colspan="8">0.78 0.04 0.56 0.14 0.04 0.57 0.04 0.26 0.16 0.10</td></tr><tr><td>RL</td><td colspan="2">5.64 0.90</td><td>0</td><td colspan="5">3.36 1.38 4.67 0.19 0.02 3.8 0.69</td></tr><tr><td>GAN</td><td colspan="8">1.24 0.62 0.18 0.06 0.38 2.40 1.10 0.40 0.26 0.63</td></tr><tr><td>AIRL</td><td colspan="8">1.68 0.04 0.62 0.70 0.32 1.20 0.10 0.27 0.72 0.12</td></tr><tr><td>rAIRL</td><td colspan="8">0.75 0.14 0.20 0.21 0.20 0.57 0.14 0.17 0.16 0.10</td></tr></table>

Comparison with existing methods on the handcrafted evaluation metrics.

Table content: <table><tr><td>Learning Method</td><td>Model</td><td/><td>Standard Split</td><td/><td/><td>Robust Split</td><td/></tr><tr><td/><td/><td colspan="6">BLEU4 CIDEr SPICE BLEU4 CIDEr SPICE</td></tr><tr><td/><td>Att2in</td><td>31.3</td><td>101.3</td><td>-</td><td>31.5</td><td>90.6</td><td>17.7</td></tr><tr><td>MLE</td><td>NBT Up-Down</td><td>34.7 36.2</td><td>107.2 113.5</td><td>20.1 20.3</td><td>31.7 31.6</td><td>94.1 92.0</td><td>18.3 18.1</td></tr><tr><td/><td>rAIRL+MLE(Up-Down)</td><td>34.6</td><td>112.9</td><td>20.7</td><td>31.1</td><td>96.8</td><td>19.1</td></tr><tr><td/><td>GAN2(SCST, Co-att, log(D)+5×CIDEr)</td><td>-</td><td>111.1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>RL</td><td>Att2in Up-Down</td><td>33.3 36.3</td><td>111.4 120.1</td><td>-21.4</td><td>--</td><td>--</td><td>--</td></tr><tr><td/><td>rAIRL+RL(Up-Down)</td><td>35.0</td><td>115.7</td><td>21.3</td><td>30.8</td><td>97.9</td><td>19.7</td></tr><tr><td/><td>G-GAN</td><td>20.7</td><td>79.5</td><td>18.2</td><td>-</td><td>-</td><td>-</td></tr><tr><td>GAN</td><td>GAN3 (SCST, Co-att, log(D))</td><td>-</td><td>97.5</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>rAIRL(Up-Down)</td><td>33.8</td><td>110.2</td><td>20.4</td><td>30.2</td><td>93.7</td><td>18.7</td></tr></table>

Results of rewriting caption from the located position by rAIRL on MS COCO standard split. Beside each score we report its improvement relative to rewriting from a random position.

Table content: <table><tr><td>Source Caption</td><td colspan="2">BLEU1</td><td colspan="2">BLEU2</td><td colspan="2">BLEU3</td><td colspan="2">BLEU4</td><td colspan="2">ROUGE L</td><td colspan="2">CIDEr</td><td colspan="2">SPICE</td></tr><tr><td/><td>score</td><td>∆</td><td>score</td><td>∆</td><td>score</td><td>∆</td><td>score</td><td>∆</td><td>score</td><td>∆</td><td>score</td><td>∆</td><td>score</td><td>∆</td></tr><tr><td>MLE</td><td colspan="14">73.3 (+0.0) 57.1 (-0.2) 43.4 (+0.1) 32.6 (+0.5) 51.4 (+0.1) 108.5 (+2.0) 20.6 (+0.1)</td></tr><tr><td>RL</td><td colspan="14">72.9 (-0.1) 56.7 (+0.3) 42.4 (+0.5) 31.1 (+0.4) 50.8 (+0.2) 104.0 (-0.5) 20.1 (+0.0)</td></tr><tr><td>GAN</td><td colspan="14">72.7 (+1.1) 56.3 (+1.2) 42.0 (+1.0) 30.9 (+0.8) 50.6 (+0.4) 103.0 (+2.3) 20.0 (+0.4)</td></tr><tr><td>AIRL</td><td colspan="14">72.6 (+1.0) 56.1 (+1.3) 41.8 (+1.2) 30.5 (+0.9) 50.6 (+0.8) 102.6 (+4.1) 19.9 (+0.8)</td></tr></table>

Results of using different model architectures in our method. adopt SPICE to evaluate content correctness in the paper because it has better correlation with human judgments(Anderson et al., 2016). Table10gives full results of the handcrafted metrics on two splits of MS COCO. Comparing the adversarial (GAN, AIRL, rARIL) and non-adversarial (MLE, RL) methods, RL outperforms other methods on most metrics. In adversarial methods, the proposed rAIRL performs the best. Table11shows results on the MS COCO online test server. The proposed rAIRL improves AIRL on all the metrics.

Table content: <table><tr><td>Method</td><td/><td>Standard Split</td><td/><td/><td>Robust Split</td><td/></tr><tr><td/><td colspan="6">BLEU4 CIDEr SPICE BLEU4 CIDEr SPICE</td></tr><tr><td>Att2in</td><td>31.0</td><td>101.3</td><td>-</td><td>31.5</td><td>90.6</td><td>17.7</td></tr><tr><td>rAIRL(Att2in)</td><td>31.3</td><td>105.2</td><td>19.9</td><td>30.7</td><td>92.5</td><td>18.0</td></tr><tr><td>Up-Down</td><td>36.2</td><td>113.5</td><td>20.3</td><td>31.6</td><td>92.0</td><td>18.1</td></tr><tr><td>rAIRL(Up-Down)</td><td>33.8</td><td>110.2</td><td>20.4</td><td>30.2</td><td>93.7</td><td>18.7</td></tr><tr><td colspan="3">E FULL RESULTS ON MS COCO</td><td/><td/><td/><td/></tr></table>

Results of the conventional handcrafted metrics on MS COCO test split.

Table content: <table><tr><td>Method</td><td/><td/><td colspan="2">Standard Split</td><td/><td/><td/><td/><td colspan="2">Robust Split</td><td/><td/><td/><td/></tr><tr><td/><td colspan="14">BLEU1 BLEU2 BLEU3 BLEU4 ROUGE L CIDEr SPICE BLEU1 BLEU2 BLEU3 BLEU4 ROUGE L CIDEr SPICE</td></tr><tr><td>MLE</td><td>74.5</td><td>57.0</td><td>41.7</td><td>30.3</td><td>50.2</td><td>104.6</td><td>19.0</td><td>69.5</td><td>52.7</td><td>39.2</td><td>29.3</td><td>48.7</td><td>93.4</td><td>18.6</td></tr><tr><td>RL</td><td>75.5</td><td>58.2</td><td>43.8</td><td>35.0</td><td>51.6</td><td>115.1</td><td>20.7</td><td>73.5</td><td>55.9</td><td>40.8</td><td>29.9</td><td>49.9</td><td>95.1</td><td>18.1</td></tr><tr><td>GAN</td><td>67.7</td><td>51.9</td><td>38.6</td><td>28.3</td><td>48.7</td><td>93.4</td><td>18.3</td><td>64.7</td><td>48.0</td><td>34.5</td><td>24.6</td><td>46.2</td><td>78.3</td><td>16.8</td></tr><tr><td>AIRL</td><td>69.9</td><td>53.6</td><td>39.1</td><td>27.5</td><td>49.8</td><td>87.4</td><td>17.3</td><td>67.6</td><td>50.5</td><td>36.3</td><td>25.9</td><td>47.3</td><td>79.5</td><td>16.7</td></tr><tr><td>rAIRL</td><td>73.8</td><td>58.2</td><td>44.6</td><td>33.8</td><td>52.1</td><td>110.2</td><td>20.4</td><td>70.3</td><td>54.1</td><td>40.5</td><td>30.2</td><td>49.4</td><td>93.7</td><td>18.7</td></tr></table>

Results on COCO test server. Methods marked with * adopt RL of CIDEr optimization.(Anderson et al., 2018) 80.2 95.2 64.1 88.8 49.1 79.4 36.9 68.5 27.6 36.7 57.1 72.4 117.9 120.5

Table content: <table><tr><td>Method</td><td colspan="2">BLEU1</td><td colspan="2">BLEU2</td><td colspan="2">BLEU3</td><td colspan="2">BLEU4</td><td colspan="2">METEOR</td><td colspan="2">ROUGE L</td><td colspan="2">CIDEr</td></tr><tr><td/><td>c5</td><td>c40</td><td>c5</td><td>c40</td><td>c5</td><td>c40</td><td>c5</td><td>c40</td><td>c5</td><td>c40</td><td>c5</td><td>c40</td><td>c5</td><td>c40</td></tr><tr><td>Adaptive (Lu et al., 2017)</td><td colspan="14">74.8 92.0 58.4 84.5 44.4 74.4 33.6 63.7 26.4 35.9 55.0 70.5 104.2 105.9</td></tr><tr><td>Att2all  *  (Rennie et al., 2017)</td><td colspan="14">78.1 93.7 61.9 86.0 47.0 75.9 35.2 64.5 27.0 35.5 56.3 70.7 114.7 116.7</td></tr><tr><td>Up-Down  AIRL(Up-Down)</td><td colspan="13">72.5 90.4 55.2 83.7 42.3 73.6 30.8 62.6 25.0 34.2 53.5 68.5 81.9</td><td>82.6</td></tr><tr><td>rAIRL(Up-Down)</td><td colspan="14">75.4 93.1 59.8 86.5 45.9 77.2 35.0 66.7 26.1 35.4 55.6 71.1 104.1 105.2</td></tr><tr><td>rAIRL+MLE(Up-Down)</td><td colspan="14">75.5 93.3 59.8 86.7 46.2 77.4 35.4 67.0 26.5 36.0 55.8 71.5 105.9 106.2</td></tr><tr><td>rAIRL+RL(Up-Down)  *</td><td colspan="14">79.5 94.1 63.5 88.0 48.3 78.9 36.2 68.5 27.5 36.6 56.2 71.8 112.3 115.1</td></tr><tr><td colspan="5">F DISCUSSION ON LOSS FUNCTIONS</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Formulas of different loss functions.

Table content: <table><tr><td>Method</td><td colspan="3">Loss Function</td></tr><tr><td>MLE</td><td>-</td><td>n</td><td>log(π true t )</td></tr><tr><td/><td/><td>t=1</td><td/></tr><tr><td/><td/><td colspan="2">n</td></tr><tr><td>RL</td><td colspan="2">-r</td><td>log(πt)</td></tr><tr><td/><td/><td colspan="2">t=1</td></tr><tr><td/><td/><td/><td>n</td></tr><tr><td>GAN (generator)</td><td colspan="2">-Dgen</td><td>log(πt)</td></tr><tr><td/><td/><td colspan="2">t=1</td></tr><tr><td>rAIRL (generator) -</td><td/><td/><td/></tr></table>

the integral of ∇ ψ E wt∼π ψ [•] w.r.t. ψ E wt∼π ψ [f θ,ϕ (w t , s t ) -log(π ψ ) + 1] = E wt∼π ψ [f θ,ϕ (w t , s t ) -log(π ψ ) + 1]dψ(12)According to Eq. (10),∇ ψ E wt∼π ψ [f θ,ϕ (w t , s t ) -log(π ψ ) + 1] = 1 π ψ (f θ,ϕ (w t , s t ) -log(π ψ ))∇ ψ π L t (ψ) = -E wt∼π ψ [f θ,ϕ (w t , s t ) -log(π ψ ) + 1] -E w true t ∼π true ψ [f θ,ϕ (w true t , s true t ) -log(π true ψ ) + 1] = -f θ,ϕ (w t , s t ) -log(π ψ ) log(π ψ ) -f θ,ϕ (w true t , s true

Table content: <table><tr><td/><td>(14)</td></tr><tr><td>Similarly, replacing w t with w true t</td><td>we get</td></tr><tr><td>E w true t ∼π true</td><td/></tr></table>

Peer review (iclr2020):
This paper proposes a refined Adversarial Inverse Reinforcement Learning (rAIRL) to remedy the reward ambiguity by decoupling the reward for each word in a sentence, while the existing methods that utilize reinforcement learning to optimize evaluation score handle only sentence-level rewards. Furthermore, a conditional term is introduced in the loss function to avoid mode collapse and to increase the diversity of the generated captions. Throughout experiments on MS COCO show that the proposed method achieves state-of-the-art performance with several evaluation scores. I think this is a good paper. The idea to disentangle the sentence-level reward into word-level ones with Adversarial Inverse Reinforcement Learning (AIRL) is highly motivated. Although the original AIRL has a problem that the convergence is slow, the authors introduce a constant term to shift on of the stationary points. As reported, this refinement surprisingly improves the performance and achieves state-of-the-art performance, while the original AIRL degraded the performance. Although I lean to accept this paper, I have two comments. - I would like to recommend that the source code to reproduce the result should be released. - As discussed by the authors, the introduced constant term can be regarded as a baseline in REINFORCE. In (Rennie et al., 2017), a self-critical sequence is introduced as a baseline in REINFORCE. Therefore, it is notable that this paper proposes another type of baseline. I would like the authors to compare Att2in (Rennie et al., 2017) to the proposed method, and to discuss why rAIRL outperforms Att2in as shown in Table 5.