SlowGuess/ACL_2020_OCR huggingface dataset:
Visual Genome We use the Visual Genome (Krishna et al., 2017) dataset to pre-train our object detection model. The dataset includes 108K images. To pre-train the object detection model with Faster R-CNN, we strictly follow the setting in (Anderson et al., 2018), taking 98K/5K/5K for training, validation, and testing, respectively. The split is carefully selected to avoid contamination of the MSCOCO validation and testing sets, since nearly 51K Visual Genome images are also included in the MSCOCO dataset.
Implementation Details We use Faster R-CNN (Ren et al., 2015) to identify and localize instances of objects. The object detection phase consists of two modules. The first module proposes object regions using a deep CNN, i.e., ResNet-101 (He et al., 2016). The second module extracts feature maps using region-of-interest pooling for each box proposals. Practically, we take the final output of the ResNet-101 and perform nonmaximum suppression for each object class with an IoU threshold. As a result, we obtain a set of image regions,  $V = \{v_{1}, v_{2}, \dots, v_{n}\}$ , where  $n \in [10, 100]$  varies with input images and confidence thresholds. Each region is represented as a 2,048-dimensional vector obtained from the pool5 layer after the RoI pooling. We then apply a feedforward network with a 1000-dimensional output layer for predicates classification. The network of the same size is also used for feature projection  $(\phi_{o}, \phi_{i})$  and GCN  $(f_{r}, f_{in}, f_{out})$ . In the decoder LSTM, the word embedding dimension is set to be 1,000 and the hidden unit dimension in the top-layer and bottom-layer LSTM is set to be 1,000 and 512, respectively. The trade-off parameter  $\gamma$  in multi-task learning is 0.15. The whole system is trained with the Adam optimizer. We set the initial learning rate to be 0.0005 and mini-batch size to be 100. The maximum number of training epochs is 30 for Cross-entropy and CIDEr-D optimization respectively. For sequence generation in the inference stage, we adopt the beam search strategy and set the beam size to be 3.
We construct object and predicate categories for VRD training. Similar to (Lu et al., 2018), we manually expand the original 80 object categories to
7459
Cross entropy	CIDEr-D optimization
B1	B4	ME	RG	CD	SP	B1	B4	ME	RG	CD	SP
SCST	-	31.3	26.0	54.3	101.3	-	-	33.3	26.3	55.3	111.4	-
LSTM-A	75.4	35.2	26.9	55.8	108.8	20.0	78.6	35.5	27.3	56.8	118.3	20.8
Up-Down (Baseline)	77.2	36.2	27.0	56.4	113.5	20.3	79.8	36.3	27.7	56.9	120.1	21.4
StackCap	76.2	35.2	26.5	-	109.1	-	78.6	36.1	27.4	-	120.4	-
CAVP	-	-	-	-	-	-	-	38.6	28.3	58.5	126.3	21.6
GCN-LSTM	77.3	36.8	27.9	57.0	116.3	20.9	80.5	38.2	28.5	58.3	127.6	22.0
VSUA	-	-	-	-	-	-	-	38.4	28.5	58.4	128.6	22.0
SGAE	77.6	36.9	27.7	57.2	116.7	20.9	80.8	38.4	28.4	58.6	127.8	22.1
This Work (MT-I)	78.1	38.4	28.2	58.0	119.0	21.1	80.8	38.9	28.8	58.7	129.6	22.3
This Work (MT-II)	77.9	38.0	28.1	57.6	117.8	21.3	80.5	38.6	28.7	58.4	128.7	22.4
Table 1: Single-model performances on the MSCOCO dataset (Karpathy split) in both cross-entropy and RL training period. B1, B4, ME, RG, CD, and SP denote BLEU-1, BLEU-4, METEOR, ROUGE, CIDEr-D and SPICE, respectively.
B4	ME	RG	CD
c5	c40	c5	c40	c5	c40	c5	c40
GCN-LSTM*	38.7	69.7	28.5	37.6	58.5	73.4	125.3	126.5
VSUA	37.4	68.3	28.2	37.1	57.9	72.8	123.1	125.5
SGAE	38.5	69.7	28.2	37.2	58.6	73.6	123.8	126.5
Baseline	36.9	68.5	27.6	36.7	57.1	72.4	117.9	120.5
This Work	38.6	70.1	28.6	37.8	58.8	74.5	125.1	126.7
Table 2: The performance on COCO online test server of various methods that incorporate visual relationships. * denotes that their training batch size and epochs are far beyond average setting in (Anderson et al., 2018; Yang et al., 2019).
413 fine-grained categories by utilizing a list of caption tokens. For example, the object category "person" is expanded to a list of fine-grained categories ["boy", "man", ...]. Then for all extracted triples that have both objects appearing in the 413 category list, we select the 200 most frequent predicates as our predicate categories.
4.2 Quantitative Analysis
Model Comparison We compare our models with the following state-of-the-art models: (1) SCST (Rennie et al., 2017) employs an improved policy gradient algorithm by utilizing its own inference output to normalize the rewards; (2) LSTM-A (Yao et al., 2017) integrates the detected image attributes into the CNN-plus-RNN image captioning framework; (3) Up-Down (Anderson et al., 2018) uses both a bottom-up and top-down attention mechanism to focus more on salient object regions; (4) GCN-LSTM (Yao et al., 2018) leverages graph convolutional networks over the detected objects and relations; (5) CAVP (Liu et al., 2018) proposes a context-aware policy network by accounting for visual attentions as context for generation; (6) VSUA (Guo et al., 2019) exploits the alignment
between words and different categories of graph nodes; (7) SAGE (Yang et al., 2019) utilizes an additional graph encoder to incorporate language inductive bias into the encoder-decoder framework.
Our baseline is built on Up-Down (Anderson et al., 2018). We propose two variants of final models using different multi-task blocks, namely MT-I and MT-II shown in Fig 4(b). We conduct extensive comparisons on the dataset with the above state-of-the-art techniques. We also perform detailed analysis to demonstrate the impact of different components of our framework.
Table 1 lists the results of various single models on the MSCOCO Karpathy split. Our model outperforms the baseline model significantly, with CIDEr-D scores being improved from 113.5 to 119.0 and 120.1 to 129.6 in the cross-entropy and CIDEr-D optimization period, respectively. In addition, the model with MT-II shows an advantage over that with MT-I on SPICE, which implies that the proposed adaptive visual attention mechanism works in multi-task block II.
Table 2 compares our model with three models that also incorporate VRG, plus the baseline model, on the MSCOCO online test server. Our model improves significantly from the baseline (from 120.5 to 126.7 in CIDEr-D) and has achieved the best results across all evaluation metrics on c40 (40 reference captions).
Figure 5 shows the effect of taking different weights  $\gamma$  in the multi-task loss item (Equation 19). The results indicate that the weight around 0.15 yields the best performance in both multi-task blocks. Meanwhile, Figure 6 shows the ablation analysis by removing the multi-task caption generation and graph convolution operation, respectively, to check the effect of these components. The results
7460
Figure 5: Test results (cross-entropy optimization) on various  $\gamma$ .
Figure 6: Ablation results (CIDEr-D optimization).
show that both the graph convolution operation and multi-task learning help improve the quality of the generated captions.
Note that the code of our paper has been made publicly available in the webpage provided in the abstract.
Human evaluation We performed human evaluation with three non-author human subjects, using a five-level Likert scale. For each image and each pair of systems in comparison (MT-I vs. Up-Down, MT-I vs. GCN-LSTM, and MT-I vs. SGAE), we show the captions generated by the two systems to the human subjects. We ask each subject if the first caption sentence is: significantly better (2), better (1), equal (0), worse  $(-1)$ , or significantly worse  $(-2)$ , compared to the second.
Following (Zhao et al., 2019), we obtain the subjects' ratings for fidelity (the first caption is superior in terms of making less mistakes?), informativeness (the first caption provides more informative and detailed description?), and fluency (the first caption is more fluent?). For each question asked for an image, we calculate the average of the three subjects' scores. For each pair of models in comparison, we randomly sampled 50 images from the Karpathy testset.
- MT-I vs. Up-Down: For fidelity, MT-I is better or significantly better on  $44\%$  images (where the average of the three human subjects' scores is larger than 0.5), equal to Up-Down on  $46\%$  images (the average is in range  $[-0.5, 0.5]$ ), and worse or significantly worse on  $10\%$  images (average is less than  $-0.5$ ).
For informativeness, MT-I is better or significantly better on  $60\%$  images, equal on  $34\%$ , and worse or significantly worse on  $6\%$ . For fluency, the numbers are  $18\%$ ,  $72\%$ , and  $10\%$ .
- MT-I vs. GCN-LSTM: For fidelity, MT-I is better or significantly better on  $40\%$  images, equal to GCN-LSTM on  $52\%$ , and worse or significantly worse on  $8\%$ . For informativeness, the numbers are  $32\%$ ,  $50\%$ , and  $18\%$ , respectively. For fluency, the numbers are  $12\%$ ,  $76\%$ , and  $12\%$ .
- MT-I vs. SGAE: For fidelity, MT-I is better or significantly better on  $36\%$  images, equal to SGAE on  $56\%$ , and worse or significantly worse on  $8\%$ . For informativeness, the numbers are  $30\%$ ,  $48\%$ , and  $22\%$ , respectively. For fluency, the numbers are  $6\%$ ,  $90\%$ , and  $4\%$ .
Figure 7 shows several specific examples, each including an image, a detected caption guided visual relationship graph, a ground truth sentence, a generated word sequence, and a learned visual relationship composition. We can see that the proposed model generates more accurate captions coherent to the visual relationship detected in the image. Consider the upper middle demo as an example; our model extracts a visual relationship graph covering the critical predicates "filled with" and "in front of" for understanding the image, thus producing a comprehensive description. In addition, we observe that the model generates the triple (table, filled with, food), which is a new composition that has not appeared in the training set.
Figure 8 visualizes the effect of our tag sequence generation process. Specifically, we visualize the tag probabilities of the "object", "predicate", and "none" category in each generation step. Our model successfully learns to distinguish the correct category for each time step, which is in consistent with the tag of the predicted word. For example, for the generated words "flying over", the probability for the "predicate" category is the highest, which is also true for words like "bird" and "water".
This paper presents a novel image captioning architecture that constructs caption-guided visual relationship graphs to introduce beneficial inductive
Figure 7: Several image captioning examples generated by our model.
Figure 8: Examples of generated word and tag sequences.
bias to better utilize captions. The representation is further enhanced with text and visual features of neighbouring nodes. During generation, the network is regularized to take into account explicit object/predicate constraints with multi-task learning. Extensive experiments are performed on the MSCOCO dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under various evaluation metrics. In the near future we plan to extend the proposed approach to several other language-vision modeling tasks.
We would like to thank the anonymous reviewers for their valuable comments. This research of the first and last author is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In ECCV, pages 382-398. Springer.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 6077-6086.
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. 2017. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In CVPR, pages 5659-5667.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dólar, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv.
Bo Dai, Yuqi Zhang, and Dahua Lin. 2017. Detecting visual relationships with deep relational networks. In CVPR, pages 3076-3086.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, pages 2625-2634.
Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dolkar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In CVPR, pages 1473-1482.
7462
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic compositional networks for visual captioning. In CVPR, pages 5630-5639.
Longteng Guo, Jing Liu, Jinhui Tang, Jiangwei Li, Wei Luo, and Hanqing Lu. 2019. Aligning linguistic words and visual semantic units for image captioning. arXiv.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR, pages 770-778.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In CVPR, pages 3668-3678.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR, pages 3128-3137.
Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. 2019. Dense relational captioning: Triple-stream networks for relationship-based captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6271-6280.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao'ou Tang. 2017. Vip-cnn: Visual phrase guided convolutional neural network. In CVPR, pages 1347-1356.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer.
Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu. 2018. Context-aware visual policy network for sequence-level image captioning. arXiv.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In ECCV, pages 852-869. Springer.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, pages 375-383.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In CVPR, pages 7219-7228.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. arXiv.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In CVPR, pages 7008-7024.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Workshop on Vision and Language (VL15), Lisbon, Portugal. ACL.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR, pages 4566-4575.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164.
Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In CVPR, pages 203-212.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In CVPR, pages 10685–10694.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In ECCV, pages 684-699.
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. 2017. Boosting image captioning with attributes. In ICCV, pages 4894-4902.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In CVPR, pages 4651-4659.
7463
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. 2017. Visual translation embedding network for visual relation detection. In CVPR, pages 5532-5540.
Sanqiang Zhao, Piyush Sharma, Tomer Levinboim, and Radu Soricut. 2019. Informative image captioning with external sources of information. arXiv preprint arXiv:1906.08876.