SlowGuess/NAACL_2022_OCR huggingface dataset:
because they hold the current SoTA in VQG for realistic inference regimes. We omit (Xu et al., 2021) and (Xie et al., 2021) from our table of results because these models follow an unrealistic inference regime, requiring an explicit answer during training and inference. Our baseline is an image-only model, without other guiding information or latent variables.
In Section 3 we described the shared aspects of our model variants. The reported scores in Section 5 use the same hyperparameters and model initialisation. A table of hyperparameters and training details can be found in Appendix B. BERT Base (Devlin et al., 2018) serves as our PLM encoder and following Wolf et al. (2020); Scialom et al. (2020), we use a pre-trained BERT model for decoding too. Though typically not used for decoding, by concatenating the encoder inputs with a [MASK] token and feeding this to the decoder model, we are able to obtain an output (e.g.  $\hat{q}_1$ ). This decoded output is concatenated with the original input sequence, and once again fed to the decoder to sample the next token. Thus, we use the BERT model as a decoder in an auto-regressive fashion.
To encode the images based on the Faster-RCNN object features (Ren et al., 2015; Anderson et al., 2018), we use a standard Transformer (Vaswani et al., 2017) encoder. Empirically, we find  $k = 2$  to be the best number of sampled objects.
We present quantitative results in Table 1 and qualitative results in Figure 2. We evaluate the explicit, implicit and variational implicit models in a single-reference setup, as the chosen input concepts are meant to guide the model output towards one particular target reference.
5.1 Quantitative Results
Starting with the explicit variant, as seen in Table 1, we note that our image-only baseline model achieves a BLEU-4 score of 5.95. We test our model with different combinations of text features to identify which textual input is most influential to the reported metrics. We notice that the contribution of the category is the most important text input with respect to improving the score of the model, raising the BLEU-4 score by more than 11 points (image-category) over the aforementioned baseline.
'https://visualqa.org/
Model	BLEU	CIDEr	METEOR	ROUGE	MSJ
1	2	3	4	3	4	5
Comparative
IMVQG (z-path)†	50.1	32.3	24.6	16.3	94.3	20.6	39.6	47.2	38.0	31.5
IMVQG (t-path)	47.4	29.0	19.9	14.5	86.0	18.4	38.4	53.8	44.1	37.2
WBS‡	42.1	22.4	14.1	9.2	60.2	14.9	29.1	63.2	55.7	49.7
DBN	40.7	-	-	-	-	22.6	-	-	-	-
C3VQG	41.9	22.1	15.0	10.0	46.9	13.6	42.3	-	-	-
image-only	25.9	15.9	9.8	5.9	41.4	13.5	27.8	52.2	42.8	36.0
Explicit
image-category	40.8	29.9	22.5	17.3	131	20.8	43.0	64.2	55.5	48.8
image-objects	34.7	25.0	19.1	15.0	130	19.4	36.9	67.4	59.2	52.7
image-guided	46.3	36.4	29.5	24.4	214	25.2	49.0	71.3	63.6	57.3
image-guided-random	23.6	12.1	5.75	2.39	17.6	10.8	24.2	62.3	52.6	45.0
Implicit
image-category	28.4	17.5	11.3	8.5	42.8	13.5	30.7	51.8	42.9	36.4
image-guided	33.8	24.0	18.3	14.2	123	19.1	35.9	66.7	58.9	52.5
image-guided-pred	25.3	14.9	9.1	6.3	27.3	11.6	27.3	52.0	44.0	38.1
image-guided-random	21.3	11.4	6.3	3.6	23.1	10.7	22.2	61.7	52.8	45.9
Variational Implicit
image-guided	33.9	23.5	16.8	12.6	113	18.8	35.6	64.2	56.3	49.8
image-guided-pred	22.6	12.5	6.9	4.1	24.3	11.2	23.0	58.6	49.3	42.4
image-guided-random	19.8	10.7	5.9	3.3	19.6	10.0	21.3	58.8	50	43.4
Table 1: Single reference evaluation results. “*-guided” refers to the combination of category and objects. In the explicit variant only, objects refers to the subset of detected objects and caption keywords, filtered on the target QA pair. † indicates an unrealistic inference regime, using answers as input for question generation. ‡ WBS scores are from single reference evaluation based on the VQA1.0 pre-trained “Im. + Cap.” model provided by the authors.
However, whilst the BLEU-4 for the image-object variant is 2.3 points lower, it outperforms the image-category variant by 3.9 points on the diversity orientated metric MSJ-5 - indicating that the image-category variant creates more generic questions. As expected, the inclusion of both the category and objects (image-guided) outperforms either of the previously mentioned models, achieving a new state-of-the-art result of 24.4 BLEU-4. This combination also creates the most diverse questions, with an MSJ-5 of 57.3.
We also test our hypothesis that guiding produces questions that are relevant to the fed in concepts. This is tested with 'image-guided-random' variant. This variant is the same trained model as 'image-guided', but uses  $k = 2$  random concepts from a respective image instead of using the ground truth question to generate concepts. Our results show that guiding is an extremely effective strategy to produce questions related to conceptual information, with a BLEU-4 score difference of over 20 points. We refer the reader to Section 5.3 for human evaluation which again validates this hypothesis, and Section 3.1 for an explanation of why guiding is valid for evaluating VQG models.
We evaluate the implicit models as follows. The
implicit image-category variant does not predict any objects internally. It uses all image features and object embeddings alongside the category supervision signal as described in Equation 7. The implicit image-guided models use the 'gold' objects at inference (See Section 3.1). If these variants fit the 'gold' objects well, it indicates that their generative abilities are suitable for guiding/conditioning on predicted or random objects. The image-guided-pred variants are evaluated using internally predicted objects - and the model variant that would be used in a real inference setting. Finally, the image-guided-random variants are fed in random object labels at inference.
For implicit guiding to be a valid methodology, we need to validate two criteria: 1) Successfully conditioning the decoder on guiding information; 2) Better than random accuracy of object prediction/selection. Note that intuitively, the implicit model is expected to perform worse than the explicit model in terms of the language generation metrics. This is because of the inherently large entropy of the relevant answer category and the objects given an image. However, if the learned distributions over the categories and objects can capture the relevant concepts of different images,
they may benefit the question generation when compared with image-only.
According to Table 1, by predicting just an answer category and no objects (image-category), the proposed implicit model beats the image-only baseline. The BLEU-4 score difference is less than 1 with the best performing WBS model (Scialom et al., 2020) – which also generates questions without explicit guided information.
As mentioned above, we can evaluate the implicit model by either feeding the 'gold' objects obtained as described in Section 3.1, or by the internally predicted objects as described in Section 3.2. These form the variants image-guided and image-guided-pred respectively. For both the implicit and variational implicit models, image-guided is expected to perform the best. Results validate this, showing a performance of 14.2 and 12.6 BLEU-4 respectively. Importantly, the relatively high scores of these guided models (compared to the comparative approaches) show that these models can successfully be conditioned on guiding information.
We also notice that for both types of implicit models, image-guided-pred outperforms image-guided-random. Specifically for the non-variational implicit, we see a higher BLEU-4 score difference of 2.7. Interestingly, despite this BLEU-4 difference being higher than its variational counterpart, there is a trade-off for the diversity-oriented MSJ metric. This indicates that although generated questions are discretely 'closer' to the ground truth, similar phrasing is used between the generated questions. In fact, an acute case of this phenomena occurs for the image-category variant where the BLEU-4 variant is higher than image-guided-pred or image-guided-random. In this case, qualitative analysis shows us that the higher BLEU-4 score can be attributed to the generic nature of the generated question. Failure cases of automatic evaluation metrics in NLP is discussed further in (Caglayan et al., 2020).
To satisfy the 'better than random accuracy of object prediction/selection' criteria previously outlined, we measure the overlap of the  $k$  predicted objects vs  $k$  'gold' object labels. These 'gold' object labels are obtained similarly to the explicit variant (Section 3.1), however the caption tokens are not fed to the filtering process. Random accuracy for selecting objects is  $12.5\%$ . Our overlap accuracy on implicit image-pred is  $18.7\%$  - outperforming random selection. Variational implicit image-pred
Baseline	Implicit	V-Implicit	Explicit
Experiment 1	34.3% ± 0.1	47.1% ± 0.12	36.7% ± 0.08	44.9% ± 0.08
Experiment 2	95.9% ± 0.03	76.6% ± 0.16	89% ± 0.09	93.5% ± 0.06
Experiment 3	-	-	-	77.6% ± 0.09
Experiment 4	-	-	-	74.1%/40.0% ± 0.07/0.18
Table 2: Human evaluation results (and standard dev.)
failed to outperform random accuracy.
5.2 Qualitative Results
Qualitative results are shown in Figure 2 and Appendix D. Figure 2 depicts how outputs from different model variants compare to ground truth questions. Without any guiding information, the image-only variant is able to decode semantic information from the image, however this leads to generic questions. The implicit variant, for which we also report the predicted category and objects, mostly generates on-topic and relevant questions. Focusing on the explicit variant, we witness high-quality, interesting, and on-topic questions.
Appendix D depicts how well our explicit image-guided variant handles a random selection of detected objects given the image. This experiment intends to gauge the robustness of the model to detected objects which may fall on the low tail of the human generating question/data distribution. To clarify, humans are likely to ask commonsense questions which generally focus on obvious objects in the image. By selecting objects at random for the question to be generated on, the model has to deal with object permutations not seen during training, and categories that are invalid for an image.
Analysing the outputs, when viable categories and objects that are expected to fall in a commonsense distribution are sampled, the model can generate high quality questions. Interestingly, we observe that when the sampled objects are not commonsense (e.g. "ears arms" for the baby and bear picture), the model falls back to using the object features instead of the guiding information. This phenomenon is also witnessed when the sampled category does not make sense for the image (e.g. category 'animal' in image 531086). Despite the category mismatch, the model successfully uses the object information to decode a question.
We ask seven humans across four experiments to evaluate the generative capabilities of our models. Experiment 1 is a visual Turing test: given an image, a model generated question and a ground truth question, we ask a human to determine which ques
85187	63958	447043	82259	38083
Ground Truth	is this in the usa?	is the zebra tall or short?	where is this building?	is this person airborne?	how many containers of strawberries are there?
Baseline	Generated Question:	is this a busy street?	what is the zebra doing?	is this a church?	what is the man doing?	what is the green vegetable?
Implicit	Predicted Category: Predicted Objects:	count street, statues	binary zebra, ground	other building, tower	colour person, people	binary strawberries, strawberry
Generated Question:	how many red poles are there?	is the rock in the background?	what is the weather like?	what is the color of the skier?	how many carrots are there?
Variational Implicit	Predicted Category: Predicted Objects:	binary truck, man	binary legs, shadow	object statue, tower	binary ski pole, poles	count apple, leaves
Generated Question:	is this a new truck?	is this a zoo?	what color is the clock?	is this a ski park?	how many different different types of fruit are there?
Explicit	Given Category: Given Objects:	binary statues, pillar	attribute zebra, tail	other building, tower	binary snow, skis	count strawberries, strawberry
Generated Question:	is this a rural setting?	what pattern is on the zebra's mohawk?	what is the condition of the building?	are these people going to ski down a mountain?	how many different kinds of produce are in the bowl?
Figure 2: Qualitative Examples. The ground truth is the target question for the baseline, implicit and explicit. The examples of explicit variant uses image-guided whereas the implicit is using the non-variational image-pred.
tion they believe is model generated. Experiment 2 attempts to discern the linguistic and grammatical capabilities of our model by asking a human to make a binary choice about whether the generated question seems natural. Experiment 3 shows a human an image alongside a model generated question (explicit variant). Then, we ask the human to make a choice about whether the generated question is relevant to the image (i.e. could an annotator have feasibly asked this question during data collection). Finally, experiment 4 judges whether objects are relevant to a generated question. The experiment is set up with true-pairs and adversarial-pairs. True-pairs are samples where the shown objects are the ones used to generate the question. Adversarial-pairs show a different set of objects than those which generated the question. If more true-pairs are marked correct (i.e. if at least one of the objects is relevant to the generated question) than the adversarial-pairs, then our model successfully generates questions on guiding information.
In experiment 1, a model generating human-level questions should be expected to score  $50\%$ , as a human would not be able to reliably distinguish them from the manually created questions. Our results show the explicit and non-variational implicit model outperforming the variational implicit and baseline variants, fooling the human around  $45\%$  of the time. Whilst not yet at the ideal  $50\%$ , the explicit approach provides a promising step towards beating the visual Turing Test. Experiment 2
evaluates the grammatical plausibility of the generated questions. In general, all models perform extremely well in this experiment, with the baseline variant generating grammatically correct sentences  $96\%$  of the time. This is expected, as the baseline typically falls back to decoding easy/generic questions. Experiment 3, is evaluated on our best performing model (explicit image-guided). Here,  $78\%$  of the generated questions are marked as relevant/on-topic given an image. Finally, experiment 4's results show true-pairs marked as correct vs adversarial-pairs (incorrectly) marked as correct. Since the former is larger than the latter  $-72\%$  vs  $42\%$ , the model can successfully use guiding/object information to create on-topic questions.
We presented a guided approach to visual question generation (VQG), which allows for the generation of questions that focus on specific chosen aspects of the input image. We introduced three variants for this task, the explicit, implicit, and variational implicit. The former generates questions based on an explicit answer category and a set of concepts from the image. In contrast, the latter two discretely predict these concepts internally, receiving only the image as input. The explicit model achieves SoTA results when evaluated against comparable models. Qualitative evaluation and human-based experiments demonstrate that both variants produce realistic and grammatically valid questions.
Lucia Specia, Zixu Wang and Yishu Miao received support from MultiMT project (H2020 ERC Starting Grant No. 678017) and the Air Force Office of Scientific Research (under award number FA8655-20-1-7006).
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Loic Barrault, Fethi Bouguares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304-323, Belgium, Brussels. Association for Computational Linguistics.
Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale. pages 2322-2328.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4159-4170, Minneapolis, Minnesota. Association for Computational Linguistics.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2019. Meshed-Memory Transformer for Image Captioning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 10575-10584.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215-233, Copenhagen, Denmark. Association for Computational Linguistics.
Zhihao Fan, Zhongyu Wei, Siyuan Wang, Yang Liu, and Xuanjing Huang. 2018. A Reinforcement Learning Framework for Natural Question Generation using Bi-discriminators. Technical report.
William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better Text Generation via Filling in the_____. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward Controlled Generation of Text. Technical report.
Unnat Jain, Ziyu Zhang, and Alexander Schwing. 2017. Creativity: Generating Diverse Questions using Variational Autoencoders. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January:5415-5424.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical Reparameterization with Gumbel-Softmax. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings. International Conference on Learning Representations, ICLR.
Krasser and Stumpf. 2020. fairseq-image-captioning. https://github.com/krasserm/ fairseq-image-captioning.
Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2019. Information Maximizing Visual Question Generation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:2008-2018.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguistics.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12375 LNCS:121-137.
Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. 2018. Visual question generation as dual task of visual question answering. CVPR.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1727-1736, New York, New York, USA. PMLR.
Andriy Mnih and Karol Gregor. 2014. Neural Variational Inference and Learning in Belief Networks. 31st International Conference on Machine Learning, ICML 2014, 5:3800-3809.
Andriy Mnih and Danilo J. Rezende. 2016. Variational inference for Monte Carlo objectives. 33rd International Conference on Machine Learning, ICML 2016, 5:3237-3248.
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019. Jointly measuring diversity and quality in text generation models.
Edwin G Ng, Bo Pang, Piyush Sharma, Radu Soricut, and Google Research. 2020. Understanding Guided Image Captioning Performance across Domains.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems, 2017-December:6307-6316.
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent Advances in Neural Question Generation.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.