# Image Captioning with Word Gate and Adaptive Self-Critical Learning
## Methodology
### Adaptive Self-Critical Sequence Training  
Similar to [1], we can cast image captioning as an RL problem, in which the \"agent\" (the LSTM) interacts with the external \"environment\" (i.e., words and image features). At each episode of the training process, the agent takes a sequence of \"actions\" (the predictions of the next word, \(w_{t}\)) according to a policy \(p_{\theta}\) (the parameters of the network, \(\theta\)), and observes a \"reward\" \(R\) (e.g., the CIDEr score) at the end of the sequence. The goal of training is to find a policy (the parameters) of the agent that maximizes the following expected reward:  
\[J(\theta)=\mathbb{E}_{w_{s}\sim p_{\theta}}[R(w^{s})], \tag{6}\]  
where \(w^{s}=(w^{s}_{1},...,w^{s}_{T})\) is a sentence sampled from \(p_{\theta}\) and \(w^{s}_{t}\) is the words sampled at time step \(t\). Using the REINFORCE algorithm [13], the gradient can be computed as:  
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{w_{s}\sim p_{\theta}}[R(w^{s})\
abla_{ \theta}\log p_{\theta}(w^{s})]. \tag{7}\]  
In practice, the expectation can be approximated with a single Monte Carlo sample from the following distribution of actions:  
\[\
abla_{\theta}J(\theta)\approx-R(w^{s})\
abla_{\theta}\log p_{\theta}(w^{s}),\;w^{s}\sim p_{\theta}. \tag{8}\]  
However, the Monte Carlo estimation method is not stable, especially when the policy changes in the runtime, which usually causes high variance in the estimated gradient. A common solution is to reduce the following variance by shifting the reward with a \"baseline\" \(B\):  
\[\
abla_{\theta}J(\theta)\approx(R(w^{s})-B)\
abla_{\theta}\log p_{\theta}(w^{ s}),\;w^{s}\sim p_{\theta}. \tag{9}\]  
where \(B\) can be any function that is independent with the action \(w^{s}\). The optimal baseline that yields the lowest variance estimator on \(\
abla_{\theta}J(\theta)\) is the following expected reward:  
\[B^{*}=\mathbb{E}_{w_{s}\sim p_{\theta}}[R(w^{s})]. \tag{10}\]  
Finally, using the REINFORCE algorithm with a baseline \(B\), the gradient of \(o_{t}\) (the input to the softmax) is given as [1]:  
\[\frac{\partial J(\theta)}{\partial o_{t}}\approx(R(w^{s})-B)(p_{\theta}(w_{t} |h_{t})-\mathds{1}_{w^{s}_{t}}). \tag{11}\]  
For the full derivation of the gradients, please refer to [13; 15] and Chapter 13 in [16]. Self-critical sequence training [4] gives a typical implementation of the RL-based method  
converges easily, and has a lower variance. Self-critical sequence training can be more effectively trained on mini-batches of samples with stochastic gradient descent (SGD). The beam search method is a heuristic search algorithm that explores a graph by expanding the most promising node in a limited set. This method is widely used in the natural language generation model. SCST uses greedy decoding to select the current action \(w_{t}\) for the baseline estimation in time step \(t\):  
\[\hat{w}_{t}=arg\max_{w_{t}}p(w_{t}|h_{t}). \tag{13}\]  
This is used as the foundation of the beam search method, and the original training method with cross-entropy loss optimizes the probability of the max. However, this method only considers the single-word probability, while the Monte Carlo sampling considers the probabilities of all words in the vocabulary. This inequivalence is harmful, since the aim of \(R(\hat{w}\) is to give a baseline for \(R(w^{s})\), but its value is computed in a very different way. Furthermore, we found that this greedy strategy causes SCST to be unstable in the training progress. As shown in Figure 4b, in SCST, the greedy reward (\(R(\hat{w})\) has higher variance than the Monte Carlo sampling reward (\(R(w^{s})\)), and their gap is also very unstable. As a result, the CIDEr score in Figure 4a of SCST is lower than that of adaptive SCST.