Venue: NAACL 2018

# Pragmatically Informative Image Captioning

with Character-Level Inference

Reuben Cohn-Gordon , Noah Goodman , and Chris Potts

###### Abstract

We combine a neural image captioner with a Rational Speech Acts (RSA) model to make a system that is _pragatically informative_: its objective is to produce captions that are not merely true but also distinguish their inputs from similar images. Previous attempts to combine RSA with neural image captioning require an inference which normalizes over the entire set of possible utterances. This poses a serious problem of efficiency, previously solved by sampling a small subset of possible utterances. We instead solve this problem by implementing a version of RSA which operates at the level of characters ("a","b","c",...) during the unrolling of the caption. We find that the utterance-level effect of referential captions can be obtained with only character-level decisions. Finally, we introduce an automatic method for testing the performance of pragmatic speaker models, and show that our model outperforms a non-pragmatic baseline as well as a word-level RSA captioner.

Stanford University

{reubencg, ngoodman, cgpotts}@stanford.edu

## 1 Introduction

The success of automatic image captioning Farhadi et al. (2010); Mitchell et al. (2012); Karpathy and Fei-Fei (2015); Vinyals et al. (2015) demonstrates compellingly that end-to-end statistical models can align visual information with language. However, high-quality captions are not merely _true_, but also _pragmatically informative_ in the sense that they highlight salient properties and help distinguish their inputs from similar images. Captioning systems trained on single images struggle to be pragmatic in this sense, producing either very general or hyper-specific descriptions.

In this paper, we present a neural image captioning system1 that is a _pragmatic speaker_ as defined by the Rational Speech Acts (RSA) model Frank and Goodman (2012); Goodman and Stuhlmuller (2013). Given a set of images, of which one is the _target_, its objective is to generate a natural language expression which identifies the target in this context. For instance, the literal caption in Figure 1 could describe both the target and the top two distractors, whereas the pragmatic caption mentions something that is most salient of the target. Intuitively, the RSA speaker achieves this by reasoning not only about what is true but also about what it's like to be a listener in this context trying to identify the target.

Footnote 1: The code is available at [https://github.com/reubenharry/Recurrent-RSA](https://github.com/reubenharry/Recurrent-RSA)

This core idea underlies much work in referring expression generation Dale and Reiter (1995); Monroe and Potts (2015); Andreas and Klein (2016); Monroe et al. (2017) and image captioning Mao et al. (2016); Vedantam et al. (2017), but these models do not fully confront the fact that the agents must reason about all possible utterances, which is intractable. We fully address this problem by implementing RSA at the level of characters rather than the level of utterances or words: the neural language model emits individual characters, choosing them to balance pragmatic informativeness with overall well-formedness. Thus, the agents reason not about full utterances, but rather only about all possible character choices, a very small space. The result is that the information encoded recurrently in the neural model allows us

Figure 1: Captions generated by literal (\(S_{0}\)) and pragmatic (\(S_{1}\)) model for the target image (in green) in the presence of multiple distractors (in red).



to obtain global pragmatic effects from local decisions. We show that such character-level RSA speakers are more effective than literal captioning systems at the task of helping a reader identify the target image among close competitors, and outperform word-level RSA captioners in both efficiency and accuracy.

## 2 Bayesian Pragmatics for Captioning

In applying RSA to image captioning, we think of captioning as a kind of reference game. The _speaker_ and _listener_ are in a shared context consisting of a set of images \(W\), the speaker is privately assigned a target image \(w^{*}\in W\), and the speaker's goal is to produce a caption that will enable the listener to identify \(w^{*}\). \(U\) is the set of possible utterances. In its simplest form, the _literal speaker_ is a conditional distribution \(S_{0}(u|w)\) assigning equal probability to all true utterances \(u\in U\) and \(0\) to all others. The pragmatic listener \(L_{0}\) is then defined in terms of this literal agent and a prior \(P(w)\) over possible images:

\[L_{0}(w|u)\propto\frac{S_{0}(u|w)*P(w)}{\sum_{w^{\prime}\in W}S_{0}(u|w^{ \prime})*P(w^{\prime})} \tag{1}\]

The pragmatic speaker \(S_{1}\) is then defined in terms of this pragmatic listener, with the addition of a rationality parameter \(\alpha>0\) governing how much it takes into account the \(L_{0}\) distribution when choosing utterances. \(P(u)\) is here taken to be a uniform distribution over \(U\):

\[S_{1}(u|w)\propto\frac{L_{0}(w|u)^{\alpha}*P(u)}{\sum_{w^{\prime}\in U}L_{0}( w|w^{\prime})^{\alpha}*P(w^{\prime})} \tag{2}\]

As a result of this back-and-forth, the \(S_{1}\) speaker is reasoning not merely about what is true, but rather about a listener reasoning about a literal speaker who reasons about truth.

To illustrate, consider the pair of images 2a and 2b in Figure 2. Suppose that \(U=\{\textit{bus},\textit{red bus}\}\). Then the literal speaker \(S_{0}\) is equally likely to produce _bus_ and _red bus_ when the left image 2a is the target. However, \(L_{0}\) breaks this symmetry; because _red bus_ is false of the right bus, \(L_{0}(2\mathsf{a}|\textit{bus})=\tfrac{1}{3}\) and \(L_{0}(2\mathsf{b}|\textit{bus})=\tfrac{2}{3}\). The \(S_{1}\) speaker therefore ends up favoring _red bus_ when trying to convey 2a, so that \(S_{1}(\textit{red bus}|2\mathsf{a})=\tfrac{3}{4}\) and \(S_{1}(\textit{bus}|2\mathsf{a})=\tfrac{1}{4}\).

## 3 Applying Bayesian Pragmatics to a Neural Semantics

To apply the RSA model to image captioning, we first train a neural model with a CNN-RNN architecture (Karpathy and Fei-Fei, 2015; Vinyals et al., 2015). The trained model can be considered an \(S_{0}\)-style distribution \(P(\textit{caption}|\textit{image})\) on top of which further listeners and speakers can be built. (Unlike the idealized \(S_{0}\) described above, a neural \(S_{0}\) will assign some probability to untrue utterances.)

The main challenge for this application is that the space of utterances (captions) \(U\) will be very large for any suitable captioning system, making the calculation of \(S_{1}\) intractable due to its normalization over all utterances. The question, therefore, is how best to approximate this inference. The solution employed by Monroe et al. (2017) and Andreas and Klein (2016) is to sample a small subset of probable utterances from the \(S_{0}\), as an approximate prior upon which exact inference can be performed. While tractable, this approach has the shortcoming of only considering a small part of the true prior, which potentially decreases the extent to which pragmatic reasoning will be able to apply. In particular, if a useful caption never appears in the sampled prior, it cannot appear in the posterior.

### Step-Wise Inference

Inspired by the success of the "emittor-suppressor" method of Vedantam et al. (2017), we propose an incremental version of RSA. Rather than performing a single inference over utterances, we perform an inference _for each step of the unrolling of the utterance_.

We use a character-level LSTM, which defines a distribution over characters \(P(u|\textit{pc},\textit{image})\), where _pc_ ("partial caption") is a string of char

Figure 2: Captions for the target image (in green).

 

acters constituting the caption so far and \(u\) is the next character of the caption. This is now our \(S_{0}\): given a partially generated caption and an image, it returns a distribution over which character should next be added to the caption. The advantage of using a character-level LSTM over a word-level one is that \(U\) is much smaller for the former (\(\approx\!\!30\) vs. \(\approx\!\!20,000\)), making the ensuing RSA model much more efficient.

We use this \(S_{0}\) to define an \(L_{0}\) which takes a partial caption and a new character, and returns a distribution over images. The \(S_{1}\), in turn, given a target image \(w^{*}\), performs an inference over the set of possible characters to determine which is best with respect to the listener choosing \(w^{*}\).

At timestep \(t\) of the unrolling, the listener \(L_{0}\) takes as its prior over images the \(L_{0}\) posterior from timestep \((t-1)\). The idea is that as we proceed with the unrolling, the \(L_{0}\) priors on which image is being referred to may change, which in turn should affect the speaker's actions. For instance, the speaker, having made the listener strongly in favor of the target image, is less compelled to continue being pragmatic.

### Model Definition

In our incremental RSA, speaker models take both a target image and a partial caption _pc_. Thus, \(S_{0}\) is a neurally trained conditional distribution \(S_{0}^{t}(u|w,p_{c_{t}})\), where \(t\) is the current timestep of the unrolling and \(u\) is a character.

We define the \(L_{0}^{t}\) in terms of the \(S_{0}^{t}\) as follows, where _ip_ is a distribution over images representing the \(L_{0}\) prior:

\[L_{0}^{t}(w|u,i\textit{p}_{t},p_{c_{t}})\propto S_{0}^{t}(u|w, p_{c_{t}})*i\textit{p}_{t}(w) \tag{3}\]

Given an \(S_{0}^{t}\) and \(L_{0}^{t}\), we define \(S_{1}^{t}\) and \(L_{1}^{t}\) as:

\[S_{1}^{t}(u|w,i\textit{p}_{t},p_{c_{t}})\propto\\ S_{0}^{t}(u|w,p_{c_{t}})*L_{0}^{t}(w|u,i\textit{p}_{t},p_{c_{ t}})^{\alpha} \tag{4}\]

\[L_{1}^{t}(w|u,i\textit{p}_{t},p_{c_{t}})\propto\\ L_{0}^{t}(w|u,i\textit{p}_{t},p_{c_{t}})*S_{0}^{t}(u|w,p_{ c_{t}}) \tag{5}\]

UnrollingTo perform greedy unrolling (though in practice we use a beam search) for either \(S_{0}\) or \(S_{1}\), we initialize the state as a partial caption _pc\({}_{0}\)_ consisting of only the start token and a uniform prior over the images _ip\({}_{0}\)_. Then, for \(t>0\), we use our incremental speaker model \(S_{0}\) or \(S_{1}\) to generate a distribution over the subsequent character \(S^{t}(u|w,i\textit{p}_{t},p_{c_{t}})\), and add the character \(u\) with highest probability density to \(p_{c}\), giving us _pc\({}_{t+1}\)_. We then run our listener model \(L_{1}\) on \(u\), to obtain a distribution _ip\({}_{t+1}=L_{1}^{t}(w|u,i\textit{p}_{t},p_{c_{t}})\)_ over images that the \(L_{0}\) can use at the next timestep.

This incremental approach keeps the inference itself very simple, while placing the complexity of the model in the recurrent nature of the unrolling.2 While our \(S_{0}\) is character-level, the same incremental RSA model works for a word-level \(S_{0}\), giving rise to a word-level \(S_{1}\). We compare character and word \(S_{1}\)s in section 4.2.

Footnote 2: The move from standard to incremental RSA can be understood as a switching of the order of two operations; instead of unrolling a character-level distribution into a sentence level one and then applying pragmatics, we apply pragmatics and then unroll. This generalizes to any recursive generation of utterances.

As well as being incremental, these definitions of \(S_{1}^{t}\) and \(L_{1}^{t}\) differ from the typical RSA described in section 2 in that \(S_{1}^{t}\) and \(L_{1}^{t}\) draw their priors from \(S_{0}^{t}\) and \(L_{0}^{t}\) respectively. This generalizes the scheme put forward for \(S_{1}\) by Andreas and Klein (2016). The motivation is to have Bayesian speakers who are somewhat constrained by the \(S_{0}\) language model. Without this, other methods are needed to achieve English-like captions, as in Vedantam et al. (2017), where their equivalent of the \(S_{1}\) is combined in a weighted sum with the \(S_{0}\).

## 4 Evaluation

Qualitatively, Figures 1 and 2 show how the \(S_{1}\) captions are more informative than the \(S_{0}\), as a result of pragmatic considerations. To demonstrate the effectiveness of our method quantitatively, we implement an automatic evaluation.

### Automatic Evaluation

To evaluate the success of \(S_{1}\) as compared to \(S_{0}\), we define a listener \(L_{\textit{eval}}(\textit{image}|\textit{caption})\propto P_{S_{0}}(\textit{ caption}|\textit{image})\), where \(P_{S_{0}}(\textit{caption}|\textit{image})\) is the total probability of \(S_{0}\) incrementally generating _caption_ given _image_. In other words, \(L_{\textit{eval}}\) uses Bayes' rule to obtain from \(S_{0}\) the posterior probability of each image \(w\) given a full caption \(u\).

The neural \(S_{0}\) used in the definition of \(L_{\textit{eval}}\) must be trained on separate data to the neural \(S_{0}\) used for the \(S_{1}\) model which produces captions, since otherwise this \(S_{1}\) production model effectively has access to the system evaluating it. As Mao et al. (2016) note, "a model might 'com 

municate' better with itself using its own language than with others". In evaluation, we therefore split the training data in half, with one part for training the \(S_{0}\) used in the caption generation model \(S_{1}\) and one part for training the \(S_{0}\) used in the caption evaluation model \(L_{eval}\).

We say that the caption succeeds as a referring expression if the target has more probability mass under the distribution \(L_{eval}(\textit{image}|\textit{caption})\) than any distractor.

DatasetWe train our production and evaluation models on separate sets consisting of regions in the Visual Genome dataset Krishna et al. (2017) and full images in MSCOCO Chen et al. (2015). Both datasets consist of over 100,000 images of common objects and scenes. MSCOCO provides captions for whole images, while Visual Genome provides captions for regions within images.

Our test sets consist of clusters of 10 images. For a given cluster, we set each image in it as the target, in turn. We use two test sets. Test set 1 (TS1) consists of 100 clusters of images, 10 for each of the 10 most common objects in Visual Genome.3 Test set 2 (TS2) consists of regions in Visual Genome images whose ground truth captions have high word overlap, an indicator that they are similar. We again select 100 clusters of 10. Both test sets have 1,000 items in total (10 potential target images for each of 100 clusters).

Footnote 3: Namely, _man_, _person_, _woman_, _building_, _sign_, _table_, _bus_, _window_, _sky_, and _tree_.

Captioning SystemOur neural image captioning system is a CNN-RNN architecture4 adapted to use a character-based LSTM for the language model.

HyperparametersWe use a beam search with width 10 to produce captions, and a rationality parameter of \(\alpha=5.0\) for the \(S_{1}\).

Footnote 4: [https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning](https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning)

### Results

As shown in Table 1, the character-level \(S_{1}\) obtains higher accuracy (68% on TS1 and 65.9% on TS2) than the \(S_{0}\) (48.9% on TS1 and 47.5% on TS2), demonstrating that \(S_{1}\) is better than \(S_{0}\) at referring.

Advantage of Incremental RSAWe also observe that 66% percent of the times in which the \(S_{1}\) caption is referentially successful and the \(S_{0}\) caption is not, for a given image, the \(S_{1}\) caption is not one of the top 50 \(S_{0}\) captions, as generated by the beam search unrolling at \(S_{0}\). This means that in these cases the non-incremental RSA method of Andreas and Klein (2016) could not have generated the S\({}_{1}\) caption, if these top 50 \(S_{0}\) captions were the support of the prior over utterances.

Comparison to Word-Level RSAWe compare the performance of our character-level model to a word-level model.5 This model is incremental in precisely the way defined in section 3.2, but uses a word-level LSTM so that \(u\in U\) are words and \(U\) is a vocabulary of English. It is evaluated with an \(L_{eval}\) model that also operates on the word level.

Though the word \(S_{0}\) performs better on both test sets than the character \(S_{0}\), the character \(S_{1}\) outperforms the word \(S_{1}\), demonstrating the advantage of a character-level model for pragmatic behavior. We conjecture that the superiority of the character-level model is the result of the increased number of decisions where pragmatics can be taken into account, but leave further examination for future research.

Footnote 5: Here, we use greedy unrolling, for reasons of efficiency due to the size of \(U\) for the word-level model, and set \(\alpha=1.0\) from tuning on validation data. For comparison, we note that greedy character-level \(S_{1}\) achieves an accuracy of 61.2% on TS1.

Variants of the ModelWe further explore the effect of two design decisions in the character-level model. First, we consider a variant of \(S_{1}\) which has a prior over utterances determined by an LSTM language model trained on the full set of captions. This achieves an accuracy of 67.2% on TS1. Second, we consider our standard \(S_{1}\) but with unrolling such that the \(L_{0}\) prior is drawn uniformly at each timestep rather than determined by the \(L_{0}\) posterior at the previous step. This achieves an accuracy of 67.4% on TS1. This suggests that neither this change of \(S_{1}\) nor \(L_{0}\) priors has a large effect on the performance of the model.

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **TS1** & **TS2** \\ \hline Char \(S_{0}\) & \(48.9\) & \(47.5\) \\ Char \(S_{1}\) & \(\textbf{68.0}\) & \(\textbf{65.9}\) \\ Word \(S_{0}\) & \(57.6\) & \(53.4\) \\ Word \(S_{1}\) & \(60.6\) & \(57.6\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy on both test sets.

 

Conclusion

We show that incremental RSA at the level of characters improves the ability of the neural image captioner to refer to a target image. The incremental approach is key to combining RSA with language models: as utterances become longer, it becomes exponentially slower, for a fixed \(n\), to subsample \(n\)% of the utterance distribution and _then_ perform inference (non-incremental approach). Furthermore, character-level RSA yields better results than word-level RSA and is far more efficient.

## Acknowledgments

Many thanks to Hiroto Udagawa and Poorvi Bhargava, who were involved in early versions of this project. This material is based in part upon work supported by the Stanford Data Science Initiative and by the NSF under Grant No. BCS-1456077. This work is also supported by a Sloan Foundation Research Fellowship to Noah Goodman.

## References

* Andreas and Klein (2016) Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1173-1182. Association for Computational Linguistics.
* Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. _arXiv preprint arXiv:1504.00325_.
* Dale and Reiter (1995) Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. _Cognitive Science_, 19(2):233-263.
* Farhadi et al. (2010) Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In _European Conference on Computer Vision_, pages 15-29. Springer.
* Frank and Goodman (2012) Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. _Science_, 336(6084):998.
* Goodman and Stuhlmuller (2013) Noah D Goodman and Andreas Stuhlmuller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. _Topics in Cognitive Science_, 5(1):173-184.
* Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3128-3137.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International Journal of Computer Vision_, 123(1):32-73.
* Mao et al. (2016a) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016a. Generation and comprehension of unambiguous object descriptions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 11-20. IEEE.
* Mao et al. (2016b) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016b. Generation and comprehension of unambiguous object descriptions. pages 11-20.
* Mitchell et al. (2012) Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daume III. 2012. Midge: Generating image descriptions from computer vision detections. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 747-756. Association for Computational Linguistics.
* Monroe et al. (2017) Will Monroe, Robert X. D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. _Transactions of the Association for Computational Linguistics_, 5:325-338.
* Monroe and Potts (2015) Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In _Proceedings of 20th Amsterdam Colloquium_, Amsterdam. ILLC.
* Vedantam et al. (2017) Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. _arXiv preprint arXiv:1701.02870_.
* Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3156-3164