SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4: The  $F_{1}$  scores of coherence relation classifiers with label mapping. The aggregated Weighted scores use the numbers in the first row as weights.
parameters using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of  $10^{-6}$ .
Results Table 4 shows the results of the single-label prediction experiments, where we present the  $F_{1}$  scores over each of the individual relations, as well as an overall weighted average. The USE+GraphRise model using the label mapping achieves the highest performance, with an overall  $F_{1}$  score of 0.57. Next, we describe how we use this classifier's predictions to annotate the training and validation splits of the Conceptual Caption dataset (3.3 million image-captions pairs), in order to train a controllable caption-generation model.
5 Generating Coherent Captions
We use the coherence label predictions on the Conceptual Captions dataset (Section 4) to train a coherence-aware caption generation model.
Figure 5: Coherence-aware image captioning model
Model We model the output caption using a sequence-generation approach based on Transformer Networks (Vaswani et al., 2017). The output is the sequence of sub-tokens comprising the target caption. The input is obtained by concatenating the following features.
Image Features We obtain a 64 dimensional representation for the image using the Graph-RISE (?) feature extractor, which employs a ResNet-101 network to classify images into some 40M classes. We do not fine tune this image encoder model. We use the 64-dimensional feature available immediately before the classification layer, and embed into the Transformer encoder embedding space using a trainable dense layer.
Detected Objects We obtain object labels for the image using Google Cloud Vision API.4 We represent each label using pre-trained 512-dimensional vectors trained to predict co-occurring objects on web pages, in a similar fashion as the word2vec model (Mikolov et al., 2013). We embed each of these into the Transformer encoder embedding space using a trainable dense layer.
Coherence relation label This is an input label fed at training time, for which we use the inferred coherence relation for the image-caption pair; at inference time, the label input is used to control the information in the generated caption. Embeddings for the coherence labels are trainable model parameters. Additionally, the relationship label serves as the start token for the Transformer decoder (Figure 5), i.e., it is made available both for the encoder network and directly for the decoder network. When training and evaluating a coherence-agnostic model, this label is set to a special symbol, such as NONE, essentially running the model without coherence information. For all models described in this paper, the Transformer network has 6 encoder layers, 6 decoder layers, 8 attention heads, and a 512-dimensional embedding space.
6 Results and Evaluation
In what follows, we discuss evidence for our hypotheses: (a) a coherence-aware model presents information that is aligned with the goal of the
4 cloud.google.com/vision
6531
Coherence agnostic	Visible coherence-aware	Subjective coherence-aware	Story coherence-aware	Meta coherence-aware
Visible	52.1%	79.9%	31.7%	25.0%	42.80%
Subjective	11.4%	2.6%	24.4%	2.6%	1.9%
Action	10.7%	10.8%	6.3%	8.8%	11.4%
Story	51.3%	16.0%	45.0%	58.8%	17.34%
Meta	31.2%	32.8%	15.1%	17.7%	46.5%
Irrelevant	12.2%	12.3%	10.7%	9.9%	21.40%
When	9.5%	5.6%	4.1%	17.7%	9.6%
How	21.3%	21.3%	9.6%	25.0%	30.26%
Where	5.3%	8.6%	4.1%	8.8%	16.6%
Table 5: The distribution of coherence relations in image-caption pairs when captions are generated with the discourse-aware model vs the discourse agnostic model (the mode of the distribution in bold).
(a) coherence-aware Meta: A girl in the winter forest.  
coherence-agnostic: beautiful girl in a red dress.
(b) coherence-aware Visible: the pizza at restaurant is seen. coherence-agnostic: the best pizza in the world.
Figure 6: Captions generated by the coherence-aware and coherence-agnostic models. (Photo credits: YesVideo; TinnaPong; Sok Chien Lim; GoPro)
(c) coherence-aware Subjective: beautiful chairs in a room.  
coherence-agnostic: the living room of the home.
(d) coherence-aware Story: how to spend a day. coherence-agnostic: dogs
discourse; and (b) a coherence-aware model can significantly improve caption quality.
Evaluation by expert annotators We train the model described above with the predicted discourse relation labels for image-caption pairs in the Conceptual Captions training and validation sets. The checkpoint with highest CIDEr (Vedantam et al., 2015) score on the validation set is selected for inference and human evaluations. We asked our annotators to annotate a subset of randomly selected image-caption pairs generated by this model. These evaluation images were selected from the Conceptual Captions evaluation set based on their predicted coherence label using the single-label classifier (Section 4) on the captions generated by the coherence-agnostic model (Section 5).
According to our sensitivity power analysis, with a sample size of 1500 image-text pairs, 300 in each category, we are able to detect effect sizes as small as 0.1650 with a power and significance level of  $95\%$ . Table 5 shows the result distributions for the coherence-agnostic and coherence-aware model. Differences greater than  $3\%$  are statistically sig
nificant with  $(p < 0.05, t > 2.5)$ . The ability to control the generated caption using an input coherence relation is clear: when asking for Visible (the column under Visible),  $79.85\%$  of the captions are evaluated to fit the Visible label (non-overlapping), an absolute increase of  $27.7\%$  over the coherence-agnostic model (with only  $52.09\%$  Visible); at the same time, the rate of Story and Subjective captions reduces significantly. This reduction is particularly noteworthy in the light of eliminating potential context hallucinations, which are likely to be found under the Story and Subjective labels.
A similar trend is observed when asking for, e.g., Meta:  $46.49\%$  of the captions are evaluated to fit the Meta label (non-overlapping; the column under Meta), up  $15.3\%$  over the coherence-agnostic model (with  $31.18\%$  Story). A qualitative analysis of the generated captions shows that captions generated under the Meta label include terms such as "screenshot" and "view", while Subjective captions come with adjectives such as "beautiful" or "favorite". Figure 6 shows several examples.
6532
Crowdsourcing and Automatic Metrics For the following experiments, a subset of the Conceptual Captions validation data was selected where the ground-truth captions are labeled as Visible.
To compare the quality of the generated captions using our framework with other models, we follow the same crowdsourcing protocol that Sharma et al. (2018) employed for quality assessment. We asked subjects whether the generated captions are "good" or not.  $86\%$  of the captions generated by the coherence-aware model were selected as "good" captions, whereas only  $74\%$  of the captions generated by the coherence-agnostic model were selected as "good" captions. Note that, based on the human-evaluation data published for the Conceptual Caption Workshop at CVPR 2019, this rate is on average  $67\%$  "good" captions for the participating state-of-the-art models in 2019. Furthermore, in a follow-up experiment we ask subjects to choose between a caption generated by the coherence-aware model and one generated by the coherence-agnostic model:  $68.2\%$  of the time subjects preferred the coherence-aware result, versus  $31.8\%$  for the coherence-agnostic one.
In addition, we study the quality and the relevance of the captions generated by our model as suggested by (van der Lee et al., 2019). On a scale of 0 to 5, the average scores of the quality of the captions generated by the coherence-aware and the coherence-agnostic model are, respectively, 3.44 and 2.83. The average score of the relevance for the coherence-aware and the coherence-agnostic conditions are, respectively, 4.43 and 4.40. Note that subjects rated the quality and the relevance of the captions while seeing the questions on the same page. Screenshots and code for the experiments can be found on our GitHub page.
With the exception of the relevance condition, the results of the other questions that we asked in the crowdsourcing experiments are statistically significantly different  $(p < 0.05, t > |3.1|)$ , which indicates that subjects prefer captions generated by the coherence-aware model. We also mention here that this difference in quality, albeit significant from a human-rating perspective, is not reflected in the CIDEr score computed on the same data (against the available reference captions). The CIDEr score of the captions generated by the coherence-aware and the coherence-agnostic models are, respectively, 0.958 and 0.964. This is not surprising, as
the reference captions used by CIDEr are subject to the same distribution over coherence relations as the rest of the data, and therefore generating caption outputs with a different coherence-relation distribution (Table 5) is unlikely to have a positive impact on reference-driven metrics such as CIDEr.
Representing coherence in image-text presentations can provide a scaffold for organizing, disambiguating and integrating the interpretation of communication across modalities. We show that cross-modal coherence modeling significantly improves the consistency and quality of the generated text with respect to information needs. This is a step forward towards designing systems that learn commonsense inferences in images and text and use that to communicate naturally and effectively with the users. In addition, the presented dataset, Clue, provides opportunities for further theoretical and computational explorations. The experiments described for the coherence relation prediction task set the stage for designing better models for inferring coherence for images-text pairs.
The presented work has limitations that can be addressed in future research. According to the description of the Conceptual Captions dataset, its captions have been hypernymized. However, by studying the examples in the Other category, we discovered an additional coherence relation that exists between an image and caption, in which the caption identifies an object or entity in the image-Identification. Examples of this relation involves a caption that mentions the brand of a product or the name of the person in the image. Identification is easy to annotate but missing from this work due to the properties of the corpus we annotated. Future work should study this additional relation in the context of caption annotation and generation.
The research presented here is supported by NSF Awards IIS-1526723 and CCF-19349243 and through a fellowship from the Rutgers Discovery Informatics Institute. Thanks to Gabriel Greenberg and the anonymous reviewers for helpful comments. We would also like to thank the Mechanical Turk annotators for their contributions. We are grateful to our data annotators, Ilana Torres and Kathryn Slusarczyk for their dedicated work.
<sup>5</sup>http://wwwconceptualcaptions.com/winners-and-data
Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, and Matthew Stone. 2019. Cite: A corpus of image-text discourse relations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 570-575.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936-945, Copenhagen, Denmark. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. 2017. Attend to you: Personalized image captioning with context sequence memory networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2019. Show, control and tell: A framework for generating controllable and grounded captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8307-8316.
Samuel Cumming, Gabriel Greenberg, and Rory Kelly. 2017. Conventions of viewpoint coherence in film. Philosophers' Imprint, 17(1):1-29.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Hanqing Lu. 2019. Mscap: Multi-style image captioning with unpaired stylized text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4204-4213.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Jerry R Hobbs. 1978. Why is discourse coherent. Technical report, SRI INTERNATIONAL MENLO PARK CA.
Jerry R. Hobbs. 1985. On the coherence and structure of discourse. Technical report, Center for the Study of Language and Information, Stanford University.
Julia Hunter, Nicholas Asher, and Alex Lascarides. 2018. A formal semantics for situated conversation. Semantics and Pragmatics.
Da-Cheng Juan, Chun-Ta Lu, Zhen Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. 2020. Ultra fine-grained image semantic embedding. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM 20, page 277285, New York, NY, USA. Association for Computing Machinery.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. 2019. Integrating text and image: Determining multimodal document intent in Instagram posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4614-4624, Hong Kong, China. Association for Computational Linguistics.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4. International Journal of Computer Vision, pages 1-26.
Alex Lascarides and Matthew Stone. 2009. A formal semantic analysis of gesture. Journal of Semantics, 26(4):393-449.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355-368, Tokyo, Japan. Association for Computational Linguistics.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Emily E Marsh and Marilyn Domas White. 2003. A taxonomy of relationships between images and text. Journal of Documentation, 59(6):647-672.
Scott McCloud. 1993. Understanding comics: The invisible art. William Morrow.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Christian Otto, Matthias Springstein, Avishek Anand, and Ralph Ewerth. 2019. Understanding, categorizing and predicting semantic image-text relations. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pages 168-176. ACM.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
B Phillips. 1977. A calculus of cohesion. In Fourth LACUS Forum, Montreal, Canada.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn discourse treebank 2.0. In LREC. Citeseer.
Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5:6-1.
Hannah Rohde, Alexander Johnson, Nathan Schneider, and Bonnie Webber. 2018. Discourse coherence: Concurrent explicit and implicit relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257-2267, Melbourne, Australia. Association for Computational Linguistics.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156.
Philipp Sadler, Tatjana Scheffler, and David Schlangen. 2019. Can neural image captioning be controlled via forced attention? In Proceedings of the 12th International Conference on Natural Language Generation, pages 427-431, Tokyo, Japan. Association for Computational Linguistics.
Deborah Schiffrin. 1980. Meta-talk: Organizational and evaluative brackets in discourse. Sociological Inquiry, 50(34):199-236.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2556-2565.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In *The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.
Una Stojnic, Matthew Stone, and Ernest Lepore. 2013. Deixis (even without pointing). Philosophical Perspectives, 26(1):502-525.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Alakananda Vempala and Daniel Preotiac-Pietro. 2019. Categorizing and inferring the relationship between the text and image of twitter posts. In Proceedings of the 2019 Conference of the Association for Computational Linguistics.
Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th annual meeting of the association for computational linguistics: Short papers-volume 2, pages 90-94. Association for Computational Linguistics.
Bonnie Webber, Alistair Knott, Matthew Stone, and Aravind Joshi. 1999. Discourse relations: A structural and presuppositional account using lexicalised tag. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, pages 41-48. Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78.
6535