Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a refined Adversarial Inverse Reinforcement Learning (rAIRL) to remedy the reward ambiguity by decoupling the reward for each word in a sentence, while the existing methods that utilize reinforcement learning to optimize evaluation score handle only sentence-level rewards. Furthermore, a conditional term is introduced in the loss function to avoid mode collapse and to increase the diversity of the generated captions. Throughout experiments on MS COCO show that the proposed method achieves state-of-the-art performance with several evaluation scores.

I think this is a good paper. The idea to disentangle the sentence-level reward into word-level ones with Adversarial Inverse Reinforcement Learning (AIRL) is highly motivated. Although the original AIRL has a problem that the convergence is slow, the authors introduce a constant term to shift on of the stationary points. As reported, this refinement surprisingly improves the performance and achieves state-of-the-art performance, while the original AIRL degraded the performance. 

Although I lean to accept this paper, I have two comments.
- I would like to recommend that the source code to reproduce the result should be released.
- As discussed by the authors, the introduced constant term can be regarded as a baseline in REINFORCE. In (Rennie et al., 2017), a self-critical sequence is introduced as a baseline in REINFORCE. Therefore, it is notable that this paper proposes another type of baseline. I would like the authors to compare Att2in (Rennie et al., 2017) to the proposed method, and to discuss why rAIRL outperforms Att2in as shown in Table 5.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors propose using a recent method for adversarial inverse reinforcement learning (AIRL) for the task for generating high-quality image captions. Leveraging the GAN framework, a discriminator is trained to distinguish real captions from those produced by the generator, while the generator is optimized with policy-gradients (REINFORCE) to maximize the pseudo-reward from the discriminator. The main difference from prior work seems to be that the discriminator acts on a word-level, rather than sentence-level (as done, for instance in Dai et. al. 2017). Correspondingly, the generator policy is updated with the objective of 1-step reward maximization (more like contextual bandits), rather than with a long-term sequential decision-making objective (as done in Dai et. al. 2017). The evaluation is done using 2 data splits – standard and robust, with various metrics such as SPICE, CIDEr, BLEU, CHAIR. Diversity analysis and ablations are also performed to dissect the performance of the proposed approach. 

My 2 main issues with the paper are confusing motivation (in section 1) and various imprecise parts (in section 3 and 4).  

1.	The authors argue that current GAN-based captioning models provide ill-defined rewards due to the “reward ambiguity problem”. This problem is not explained or motivated well in the paper, but instead the readers are referred to the AIRL paper. “Reward ambiguity” in inverse-RL arises because there could be many reward functions that yield the same optimal policy. The AIRL algorithm recovers one of these possible reward functions, and since such a recovered reward could be shaped by the environment dynamics, AIRL attempts to disentangle the reward from the dynamics. The motivation there is to use the recovered reward on a new system with different transition dynamics. In the context of this paper though, I would like to understand the angle of reward ambiguity. The authors disentangle the sentence reward into word-wise rewards; however, I’m not sure if there’s any relation between this and the disentanglement done in AIRL for solving reward ambiguity.

2.	One of the objectives is learning compact rewards. It is claimed that addition of a constant term to the reward provided to the generator policy results in this, but what’s the intuition behind this? As for evaluation, it needs to be shown that words with similar semantics have similar discriminator score. How do we conclude this from Figure 2.? Also, please include Up-Down method results in Figure 2.

3.	Section 3 questions
     a.	How is state s_t defined? It is very hard to follow sections 3 and 4 without a clear definition and example for this.
     b.	“Finn et. al. 2016 proved that IRL is mathematically equivalent …” --- this is imprecise. Maximum-Entropy-IRL is equivalent to the GAN formulation, not general IRL. 
     c.	p_theta(a,s) is referred to as “reward distribution”. I don’t think it’s a distribution.
     d.	Equation 3. AIRL defines g only as a function of state (s_t) for the disentanglement, and not like what the authors have written.
     e.	Equation 4. How is s_t sampled?

4.	Section 4 questions
     a.	4.1 says discriminator “maximizes the divergence”. This doesn’t seem correct.
     b.	f is referred to as state-value. This doesn’t seem correct.
     c.	Shouldn’t the -1 term in Equation 8 disappear under expectation?
     d.	Don’t understand how second line of Equation 11 is arrived at.

There are quite a few other sources of mathematically imprecise writing that I noticed. I would recommend the authors to be more robust in their presentation.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Overall this is a good paper.
Such compact rewards over image captions are useful for evaluation and diagnosis. 

*concerns:
1. method:
   1.1 At first, I think section 4 could be re-written to corporate more concepts in image captioning (such as replacing action a to word w) so that it becomes more readable for related readers. 
   1.2 The motivation for adding a constant to shift nash equailibrium could be stated clearer. 
   1.3 what is the state s in the context of image captioning? the hidden vector h ?
   1.4 since the reward is computed for each pair of (a, s), how to get the reward for the whole sentence? Sum them up? I wonder this because it seems you compute this in Table 1.
   1.5 in Eq.(5) and Eq.(11), there are expectations. Do you need to compute them using Monte Carlo Rollouts? Or just averaging over mini-batchs? Because G-GAN (Dai et al) also utilizes rollouts to estimate different values for different words. 

2. experiments:
  1.1 the experiment in terms of "compactness" may not  reflect well the concept of compactness. While compactness means similar words may obtain similar rewards in the same caption,  top-k captions for an image may be different in multiple words, or different in formats rather than word choices. How about top-k words in a given position of a given caption? Or replace a given word with different words and compare the correlation between reward differences and semantic differences.
  1.2  qualitative samples in terms of caption diversity could be included. 
  1.3 since the rewards are computed for each word separately,  the authors may include experiments on using such rewards to diagnose captions, such as replacing a word to make the caption better, etc.
review_title: Official Blind Review #1

Meta-review: The paper proposed a refined AIRL method to deal with the reward ambiguity problem in image captioning, wherein the main idea is to refine the loss function in word level instead in sentence level, and introduce a conditional term in the loss function to mitigate mode collapse problem.  The results show the proposed method improves the performance and achieves state-of-the-art performance.  However there are concerns from the reviewers that the motivation of the work was not well explained and some inprecise parts exist in the paper.  The concept of "reward ambiguity problem" is not properly addressed according the opinion of reviewer2.  I would like to see these concerns be well addressed before the paper can be accepted.  
Decision: Reject