Experiments ::: Experimental Settings ::: Baselines and proposed models	We first train an MLE model as our baseline, trained on the Conceptual Captions training split alone. We referred to this model as Baseline. For a baseline approach that utilizes (some of) the Caption-Quality data, we merge positively-rated captions from the Caption-Quality training split with the Conceptual Captions examples and finetune the baseline model. We call this model Baseline$+(t)$, where $t \in [0,1]$ is the rating threshold for the included positive captions. We train models for two variants, $t\in \lbrace 0.5, 0.7\rbrace $, which results in $\sim $72K and $\sim $51K additional (pseudo-)ground-truth captions, respectively. Note that the Baseline$+(t)$ approaches attempt to make use of the same additional dataset as our two reinforced models, OnPG and OffPG, but they need to exclude below-threshold captions due to the constraints in MLE.

Experiments ::: Evaluations ::: Side-by-side caption evaluation	In the other type of evaluation, we measure the relative improvement of a model against the Baseline model; Three professional raters are shown the input image and two captions (anonymized and randomly shuffled with respect to their left/right position) side-by-side. One of the captions is from a candidate model and the other always from Baseline. We ask for relative judgments on three dimensions – Informativeness, Correctness and Fluency, using their corresponding questions shown in Table TABREF32. Each of these dimensions allows a 5-way choice, shown below together with their corresponding scores:  Each model is evaluated by the average rating scores from 3 distinct raters. As a result, we obtain 3 values for each model in the range $[-1, 1]$, where a negative score means a performance degradation in the given dimension with respect to Baseline. For every human evaluation, we report confidence intervals based on bootstrap resampling .

Experiments ::: Results ::: Side-by-side evaluations	The results from the side-by-side evaluations are are shown in Table TABREF39. The OffPG method achieves significant improvements on all three different dimensions. This is an important result, considering that we trained the model using a caption ratings dataset that contains single-scalar scores for generic 'goodness' (as opposed to the well-defined dimensions along which the OffPG method scores have improved). These results demonstrate that the single-caption 'goodness' ratings encapsulate a signal for all these dimensions into its scalar value. Note that we observe the same tendency consistently under a variety of hyperparameter settings in our internal experiments. Figure FIGREF44 highlights the way in which the OffPG method achieves its superiority over the Baseline model, compared to the other alternative models (using the 'Corectness' scores). For instance, over 75% of the captions for both Baseline$+(t)$ models receive a 0.0 score (equal quality), and more than half of them are exactly identical to their corresponding Baseline captions.

Experiments ::: Experimental Settings ::: Training details	We train Baseline using the Adam optimizer  on the training split of the Conceptual dataset for 3M iterations with the batch size of 4,096 and the learning rate of $3.2\times 10^{-5}$. The learning rate is warmed up for 20 epochs and exponentially decayed by a factor of 0.95 every 25 epochs. Baseline$+(t)$ are obtained by fine-tuning Baseline on the merged dataset for 1M iterations, with the learning rate of $3.2\times 10^{-7}$ and the same decaying factor. For OnPG, because its memory footprint is increased significantly due to the additional parameters for the rating estimator, we reduce the batch size for training this model by a 0.25 factor; the value of $b$ in Eq. (DISPLAY_FORM12) is set to the moving average of the rating estimates. During OffPG training, for each batch, we sample half of the examples from the Conceptual dataset and the other half from Caption-Quality dataset; $b$ is set to the average of the ratings in the dataset.

Experiments ::: Results ::: Qualitative results	Figure FIGREF46 presents some qualitative example outputs for our models, showcasing the effectiveness of the OffPG method. We observe that the OffPG model is often successful at correcting arbitrary qualifiers present in the baseline outputs (e.g., `half marathon' and `most beautiful' in the second and third examples, respectively).

In contrast, the time and space complexities of OffPG remain the same as Baseline and Baseline$+(t)$, since the only difference is the use of scalar weights ($r(c|I)$ and $\eta $) to gradients of each caption likelihood ($\bigtriangledown _\theta \ln p_\theta (c|I)$), as shown in Figure FIGREF8.

Experiments ::: Evaluations	We run two sets of human evaluation studies to evaluate the performance of our models and baselines, using the T2 dataset (1K images). For every evaluation, we generate captions using beam search (beam size of 5).

In addition to the baselines, we train two reinforced models: one based on the on-policy policy gradient method with a rating estimator (OnPG), and the other based on the off-policy policy gradient method with the true ratings (OffPG). The differences between the methods are shown in Figure FIGREF27.

However, the mismatch between the true rating function and the estimator (depicted by the gap between solid and dashed lines) can degenerate the quality of the resulting captioning model. On the other hand, the off-policy method focuses on the captions with true rating scores in the dataset, by changing the sampling distribution. In contrast to MLE, where each sample is viewed as equally correct and important, the off-policy method weights each caption by its rating, and therefore includes captions with negative feedback, as illustrated in Figure FIGREF8c. Note that, in the off-policy method, the baseline determines the threshold for positive/negative feedback; captions with ratings below the baseline are explicitly penalized, while the others are positively rewarded.

Experiments ::: Results ::: Single-caption evaluation	Table TABREF38 shows the goodness scores from the single-caption evaluation. Both “average” and “voting” metrics clearly indicate that OffPG significantly improves over Baseline, while the other methods achieve only marginal gains, all of which are within the error range. Baseline$+(t)$ models use only 1.5% and 2.2% additional data, at $t=0.7$ and $t=0.5$, respectively, with insignificant impact. Moreover, these methods only maximize the likelihood of the additional captions, which are already generated with high likelihood by previous models trained on the same dataset, which results in self-reinforcement. In contrast, the policy gradient methods are allowed to utilize the negative feedback to directly penalize incorrect captions. However, OnPG fails to improve the quality, most likely because it relies on a noisy caption ratings estimator that fails to generalize well over the large space of possible captions.

In contrast, OffPG makes a strong impact by explicitly penalizing the captions with negative feedback: less than 16% captions are identical to the corresponding Baseline captions. Moreover, we observe a large portion of captions with scores of 1.0 in favor of OffPG, indicating that many captions are significantly enhanced. We observe similar trends in all the three metrics.

Specifically, we use a policy gradient method where an approximated policy gradient is computed using Monte-Carlo sampling, where $\mathbb {E}_\pi $ represents $\mathbb {E}_{I\sim p_\mathcal {D}(I),c\sim p_\theta (c|I)}$, $I_s$ and $c_s$ are image and caption sampled from $p_\mathcal {D}(I)$ and $p_\theta (c|I)$, respectively, and $S$ is the number of samples. In the above equations, we subtract a baseline $b$ from the rating score $r(c_{s}|I_{s})$ to reduce the variance of the estimator while keeping its original bias. Although this formulation is straightforward, there remains a critical challenge to apply this technique to our task, since the dataset $\mathcal {D}$ contains only sparse information about $r(c|I)$ and true ratings for most captions are unknown. Eq. (DISPLAY_FORM12) requires the rating $r(c_s|I_s)$ for a randomly sampled caption which may not be present in the dataset $\mathcal {D}$.

Experiments ::: Experimental Settings ::: Model architecture	As the backbone model for image captioning we adopt the architecture described in , since it provides the highest single-model score in the Conceptual Captions Challenge. Given an image, we extract two types of visual features: 1) ultra fine-grained semantic features using pretrained network  from the entire image and 16 bounding boxes proposed by faster-RCNN , and 2) label embeddings of objects predicted by Google Cloud Vision API. We use these features with an encoder-decoder Transformer Network  to generate the captions. In addition, we train a caption rating estimator for the OnPG method using the Caption-Quality dataset. The rating estimator extracts the same types of visual features as the captioning model above, and embeds the input caption with a pretrained BERT encoder . We concatenate all these features after projecting into a common embedding space and predict the human ratings of the input image/caption pair. To feed the generated captions from the captioning model directly into the rating estimator, we share the vocabulary (but not the token embeddings) between the two models. We fix the pretrained image feature extraction modules in both models during training, as well as the BERT encoder of the rating estimator.

Introduction	Image captioning is the task of automatically generating fluent natural language descriptions for an input image. However, measuring the quality of generated captions in an automatic manner is a challenging and yet-unsolved task; therefore, human evaluations are often required to assess the complex semantic relationships between a visual scene and a generated caption , , . As a result, there is a mismatch between the training objective of the captioning models and their final evaluation criteria. The most simple and frequently-used training objective is maximum likelihood estimation (MLE) , , , , , while other approaches make use of handcrafted evaluation metrics, such as CIDEr , to optimize model parameters using reinforcement learning (RL) , , , . However, these surrogate objectives capture only limited aspects of caption quality, and often fail to guide the training procedure towards models capable of producing outputs that are highly-rated by human evaluators. As a result of the need to understand the performance of the current models, human evaluation studies for measuring caption quality are frequently reported in the literature , , , . In addition to an aggregate model performance, such human evaluation studies also produce a valuable by-product: a dataset of model-generated image captions with human annotated quality labels, as shown in Figure FIGREF1.

We argue that such a by-product, henceforth called a caption ratings dataset, can be successfully used to improve the quality of image captioning models, for several reasons. First, optimizing based on instance-level human judgments of caption quality represent a closer-to-truth objective for image captioning: generating more captions judged as good but fewer ones rated as poor by human raters. Second, while having highly-rated captions as positive examples (i.e., how good captions may look like), a caption ratings dataset also contains captions that are highly-scored by a model but annotated as negative examples (i.e., how model-favored yet bad captions look like), which intuitively should be a useful signal for correcting common model biases. To the best of our knowledge, our work is the first to propose using human caption ratings directly for training captioning models. Our goal is to leverage the signals from a pre-collected caption ratings dataset  for training an image captioning model. We propose a method based on policy gradient, where the human ratings are considered as rewards for generating captions (seen as taking actions) in an RL framework.

Conclusion	In this paper, we describe how to train an improved captioning model by using a caption ratings dataset, which is often a natural by-product in the development process of image captioning models. We show that an off-policy RL technique with an alternative sampling distribution successfully deals with the sparsity of information about the rating function, while an on-policy method has difficulties in obtaining an improved model, due to generalization issues of the ratings estimator. While this conclusion may not be definitive, it is definitely an important result, and it also opens up additional lines of inquiry along the relative merits of these RL techniques.

First, obtaining ratings for automatically generated outputs is significantly cheaper than collecting ground-truth labels, because it requires less rater training and less time spent annotating. Moreover, if human evaluation is performed anyway during a model's development cycle, there is no additional cost associated to using these annotations for further improving the model. In addition to that, it is easy to capture consensus between multiple raters to reduce noise, e.g., by averaging their scores; it is completely non-trivial to achieve a similar effect from multiple ground-truth labels. Last but not least, the examples with a negative rating score provide valuable training signals, as they explicitly penalize the mistakes that appear in model outputs with high-probability; this type of signal is completely lacking in traditional supervised-learning datasets.

Methods ::: Reinforcing Caption Generator using Ratings	Given a caption ratings dataset $\mathcal {D}$ with triplets $(I, c, r(c|I))$, our objective is to maximize the expected ratings of the output captions $\mathcal {J}(\theta )$, which is given by where $p_\mathcal {D}(I)$ is the dataset distribution for $I$ and $p_\theta (c|I)$ is the conditional caption distribution estimated by a model parameterized by $\theta $. Our objective in Eq. (DISPLAY_FORM11) exactly aligns with the reward maximization of RL, and therefore we apply the techniques of RL by configuring the captioning model as the agent, the rating scores as the reward, the input images as the states, and the captions as the actions.

Methods ::: Caption Ratings Dataset	A sample in a caption ratings dataset is comprised of an image $I$, a machine-generated caption $c$, and a human judgment for the caption quality $r(c|I) \in \mathbb {R}$. For each image, multiple captions from several candidate models are available, some of which might be rated higher than others. In the setup used in this paper, the low-rated captions serve as negative examples, because human annotators judged them as bad captions (see examples in Figure FIGREF1). $r(c|I)$ is possibly an aggregate of multiple ratings from different raters. Section SECREF23 provides more details of the caption ratings dataset that we employ. We make a few observations that apply not only to image captioning, but more generally to the principle of generating annotations. Although a human-ratings dataset is usually just a by-product of human evaluations for past models, such a dataset can be valuable for improving models (as we show in this paper). There are several advantageous properties of a ratings dataset over traditional supervised-learning datasets.

Related Work	There have been multiple attempts to define metrics that evaluate the quality of generated captions. Several studies proposed automatic metrics using ground-truth captions. A few of them are adopted from machine translation community and are based on $n$-gram matches between ground-truth and generated captions; BLEU  and ROUGE  measures precision and recall based on $n$-gram matches, respectively, while METEOR  incorporates alignments between $n$-gram matches. In the context of evaluating image caption quality specifically, CIDEr  and SPICE  utilize more corpus-level and semantic signals to measure matches between generated and ground-truth captions. Aside from these handcrafted metrics, a recent study proposes to learn an automatic metric from a captioning dataset , while another uses semantic similarity between object labels identified in the image and the words in the caption . To overcome the limitations imposed by the automatic metrics, several studies evaluate their models using human judgments , , , . However, none of them utilizes the human-rated captions in the model evaluations. In this work, we show how one can utilize such human-rated captions for training better captioning models.

MLE with ground-truth captions has been widely adopted as the standard surrogate objective for training , , , , . Aside from this main thrust, an additional line of research is concerned with optimizing models that maximize some automatic evaluation metric(s) using RL, in an attempt to bridge the mismatch between the training objective and the evaluation criteria , , , . To our knowledge, this is the first study that proposes to optimize test-time scores of human judgment using a dataset generated by a human evaluation process. Another line of related research is focused on learning from human feedback, which has been actively explored in the field of RL. Some approaches use binary human feedback to update an agent , ,  whereas approaches with preference-based RL take human feedback as preferences between two action/state trajectories , , . A common technique adopted in these methods is to learn an estimation model from human feedback to approximate the absent reward function , , . However, these approaches assume that the models receive human feedback iteratively in a training loop; in contrast, our approach uses the caption ratings in an off-line manner, simply as a pre-existing annotation dataset. As a result, our method focuses on existing examples within the dataset, using an off-policy technique.

Methods ::: Reinforcing Caption Generator using Ratings ::: Curriculum learning	As our training conditions, we assume the access to both a captioning dataset and a caption ratings dataset. Under a curriculum learning procedure, we first train a model by MLE on the captioning dataset, and then fine-tune the model with the above methods using the caption ratings dataset. To avoid overfitting during fine-tuning, we add the MLE loss on the captioning dataset as a regularization term. Given the caption labeled dataset $\mathcal {D}_\mathrm {IC}$ and the caption ratings dataset $\mathcal {D}_\mathrm {CR}$, the final gradients w.r.t. the parameters are therefore computed as follows: where $\mathcal {J}_\mathrm {MLE}$ is the average log-likelihood of ground-truth captions in $\mathcal {D}_\mathrm {IC}$, and $\alpha $ is a hyper-parameter that balances the regularization effect.

Experiments ::: Datasets ::: Caption ratings dataset	In our experiments, we use the Caption-Quality dataset , recently introduced for the purpose of training quality-estimation models for image captions. We re-purpose this data as our caption ratings dataset $\mathcal {D}_\mathrm {CR}$. The dataset is divided into training, validation and test splits containing approximately 130K, 7K and 7K rated captions, respectively. Each image has an average of 4.5 captions (generated by different models that underwent evaluation evaluation). The captions are individually rated by asking raters the question “Is this a good caption for the image?”, with the answers “NO” or “YES” mapped to a 0 or 1 score, respectively. Each image/caption pair is evaluated by 10 different human raters, and an average rating score per-caption is obtained by quantizing the resulting averages into a total of nine bins $\lbrace 0, \frac{1}{8} \dots \frac{7}{8}, 1\rbrace $.

Reinforcing an Image Caption Generator Using Off-Line Human Feedback	Human ratings are currently the most accurate way to assess the quality of an image captioning model, yet most often the only used outcome of an expensive human rating evaluation is a few overall statistics over the evaluation dataset. In this paper, we show that the signal from instance-level human caption ratings can be leveraged to improve captioning models, even when the amount of caption ratings is several orders of magnitude less than the caption training data. We employ a policy gradient method to maximize the human ratings as rewards in an off-policy reinforcement learning setting, where policy gradients are estimated by samples from a distribution that focuses on the captions in a caption ratings dataset. Our empirical evidence indicates that the proposed method learns to generalize the human raters' judgments to a previously unseen set of images, as judged by a different set of human judges, and additionally on a different, multi-dimensional side-by-side human evaluation procedure.

Since the dataset provides ratings only for a small set of images and captions, we do not have a generic reward function for random image-caption pairs. Therefore, it is not straightforward to apply policy gradient method that requires a reward for randomly sampled captions. To address this challenge, we use an off-policy technique and force the network to sample captions for which ratings are available in the dataset. We evaluate the effectiveness of our method using human evaluation studies on the T2 test set used for the Conceptual Captions Challenge, using both a similar human evaluation methodology and an additional, multi-dimensional side-by-side human evaluation strategy. Additionally, the human raters in our evaluation study are different from the ones that provided the caption ratings in , thereby ensuring that the results are independent of using a specific human-evaluator pool. The results of our human evaluations indicate that the proposed method improves the image captioning quality, by effectively leveraging both the positive and negative signals from the captions ratings dataset. The main contributions of this paper are the following: We propose to train captioning models using human ratings produced during evaluations of previous models. We propose an off-policy policy gradient method to cope with the sparsity in available caption ratings.

Experiments ::: Datasets ::: Conceptual Captions Challenge T2 dataset	To evaluate our models, we run human evaluation studies on the T2 test dataset used in the CVPR 2019 Conceptual Captions Challenge. The dataset contains 1K images sampled from the Open Images Dataset . Note that the images in the Caption-Quality dataset are also sampled from the Open Images Dataset, but using a disjoint split. So there is no overlap between the caption ratings dataset $\mathcal {D}_\mathrm {CR}$ we use for training, and the T2 test set we use for evaluations.

Experiments ::: Evaluations ::: Single-caption evaluation	In the first type of evaluation, 6 distinct raters are asked to judge each image caption as good or bad. They are shown the image and caption with the “Goodness” question prompt shown in Table TABREF32. The bad or good rating is translated to 0 or 1, respectively. We measure “average” goodness score as the average of all the ratings over the test set. We also report a “voting” score which is the average of the binarized score for each caption based on majority voting. Note that both the “average” and “voting” scores are in the range $[0, 1]$, where higher values denote better model performance.

The alternative caption sampling distribution is defined as: where $p_\mathcal {D}(c|I)$ is the conditional caption distribution in the dataset $\mathcal {D}$, $U(\cdot )$ is the uniform distribution, and $\epsilon \ll 1$ is a small positive weight assigned to the uniform distribution. In all experiments, we sample a single caption per image in the batch. While captions that are not present in the dataset may still be sampled from $U(c)$, we assign a reward $b$ to these captions, in order to prevent incorrect contributions to the gradient computation. In the policy gradient formulation, examples with reward value $b$ are considered to have no information, and their weight $r(c|I)-b=0$ cancels out the entire term corresponding to these examples. Note that the off-policy methods enable experience replay, which is repeating previous experiences with known rewards. In this view, this method is viewed as training a captioning model by replaying the experiences in the ratings dataset.

Methods ::: Comparing two policy gradient methods	Intuitively, the two policy gradient methods described in this section have strong relationships to MLE, since training signals are based on the gradients of caption log-likelihoods. We illustrate the training settings of MLE and the two proposed methods in Figure FIGREF8. In MLE, we train the model using positive captions only and treat all positive captions equally, as illustrated in Figure FIGREF8a: the parameters are updated by the gradients of log-likelihoods of ground-truth captions $c_\mathrm {GT}$. The on-policy policy gradient method (Eq. (DISPLAY_FORM14)) instead computes the gradients of reward-weighted log-likelihoods of sample captions $c_s$ over all possible captions. By sampling from the policy distribution (on-policy), we may sample captions whose true rating scores are not known (not in the dataset). The on-policy method thus approximates the rating function by a rating estimator $\tilde{r}(c|I)$, depicted by the background gradient in Figure FIGREF8b.

Although we can (and did) introduce some heuristics to avoid some of these failure cases in the estimator, we observe that OnPG training would continue to suffer from the estimator failing to generalize well over the vast space of possible captions. This observation is similar to the mode collapsing phenomenon seen when training generative adversarial networks (GANs), but even more severe as the estimator in OnPG is fixed (unlike the discriminators in GANs which are trained simultaneously). Another drawback of OnPG is that it increases the computational complexity significantly during training. In terms of the memory usage, the rating estimator introduces 65% additional parameters, and uses more than double the memory for gradient computation compared to the other models. Also, the sequential caption sampling in OnPG slows down the training procedure, by breaking the parallelism in the Transformer computations, in addition to the time complexity incurred by the rating estimator. Empirically, OnPG is over 10 times slower than the others in processing the same number of examples in training.

Experiments ::: Results ::: On-policy vs. off-policy performance	We compare the OnPG and OffPG methods in more depth, by performing ablation experiments for the $\alpha $ hyper-parameter (the weight for the policy gradient). Figure FIGREF45 shows the results of these ablation experiments, for which we performed side-by-side comparisons over a 200-image subset from the T2 dataset. The results indicate that a very small $\alpha $ limits the impact of the additional signal for both models, since the regularization effect from the original loss term becomes too strong. By allowing updates using policy gradient with a larger $\alpha $ value, OffPG improves the performances along all three dimensions, whereas the performance of OnPG starts degrading at higher $\alpha $ values. At $\alpha =100$, OnPG drastically suffers from mode collapse and ends up generating a single caption for every image. This mode collapse is a result of poor generalization of the rating estimator: the collapsed captions are structurally ill-formed (e.g., an empty string, or a string with simply a period `.'), but they receive high rating estimates ($>0.9$) from the estimator.

In the rest of this section, we present two alternative techniques for this challenge, and discuss the advantages of one alternative versus the other.

Methods ::: Reinforcing Caption Generator using Ratings ::: On-policy policy gradient with rating estimates	One approach to address the sparsity of the rating function is to construct a caption quality estimator, while keeping the sampling process on-policy; this is the method adopted in, e.g., , , . Incidentally, it is also the expressed goal for the effort behind the caption ratings dataset in  that we use in this work. For this purpose, we train a rating estimator $\tilde{r}(c|I;\phi )$ parameterized by $\phi $, by minimizing mean squared error of the true rating scores for the image-caption pairs on the caption ratings dataset. The trained estimator then replaces the true rating function $r(c_s|I_s)$ in Eq. (DISPLAY_FORM12) and the estimated policy gradient is now: This technique allows to obtain rating estimates for any image-caption pairs, including ones that are not present in the dataset $\mathcal {D}$. The training objective with Eq. (DISPLAY_FORM14) is now maximizing the expected rating estimate of captions.

This approach is effective only if the trained rating estimator generalizes well to unseen images and captions, and it is expected to be effective only to the extent to which the rating estimator performs well over the sampled search space. In our work, we have observed artifacts of the ratings estimator that negatively impact the performance of this method, e.g., severely ill-formed captions for which the caption estimator had no training signal but assigned high ratings. We report results for this method in Section SECREF4.

Methods ::: Reinforcing Caption Generator using Ratings ::: Off-policy policy gradient with true ratings	This second method takes an orthogonal approach to address the sparsity of the rating function. We modify the sampling process in such a manner that it allows us to directly utilize the true ratings of the dataset (no estimation involved), while ensuring that the training procedure is not influenced by the captions whose true ratings are not available. More precisely, we adopt an off-policy policy gradient technique that uses an alternative distribution $q(c|I)$, instead of the true policy distribution $p_\theta (c|I)$ for sampling. The policy gradient in Eq. (DISPLAY_FORM12) is approximated as follows: where $\mathbb {E}_\beta $ represents $\mathbb {E}_{I\sim p_\mathcal {D}(I),c\sim q(c|I)}$ with an alternative caption distribution $q(c|I)$, and $\frac{p_\theta (c|I)}{q(c|I)}$ represents the importance weight for sample caption $c_s$ and image $I_s$.

The rating estimator achieves a test performance that is close to the one reported (0.519 Spearman correlation) in ; however, as we will discuss further, its performance on the Caption-Quality test set does not transfer well to the needs of the OnPG method, which needs correct rating estimates for ill-formed captions as well.

Experiments ::: Datasets ::: Image captioning dataset	In the experiments, we use Conceptual Captions , a large-scale captioning dataset that consists of images crawled from the Internet, with captions derived from corresponding Alt-text labels on the webpages. The training and validation splits have approximately 3.3M and 16K samples, respectively.

We present a set of experiments using human evaluations that demonstrates the effectiveness of our approach.

Question: What are the baseline models?
Answer: MLE model, Baseline$+(t)$