SlowGuess/NAACL_2022_OCR huggingface dataset:
Badri N. Patro, Sandeep Kumar, Vinod K. Kurmi, and Vinay P. Namboodiri. 2018. Multimodal differential network for visual question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 4002-4012. Association for Computational Linguistics.
Badri N. Patro, Vinod K. Kurmi, Sandeep Kumar, and Vinay P. Namboodiri. 2020. Deep Bayesian Network for Visual Question Generation. Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020, pages 1555-1565.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3982-3992.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Technical report.
Thomas Scialom, Patrick Bordes, Paul-Alexis Dray, Jacopo Staiano, and Patrick Gallinari. 2020. What BERT Sees: Cross-Modal Transfer for Visual Question Generation.
Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 543-553, Berlin, Germany. Association for Computational Linguistics.
Shagun Uppal, Anish Madan, Sarthak Bhagat, Yi Yu, and Rajiv Ratn Shah. 2020. C3VQG: Category Consistent Cyclic Visual Question Generation. arXiv.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need.
Ramakrishna Vedantam, C. L. Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566-4575.
Oriol Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2015. Show and tell: A neural image caption generator. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156-3164.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Jiayuan Xie, Yi Cai, Qingbao Huang, and Tao Wang. 2021. Multiple Objects-Aware Visual Question Generation. MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia, pages 4546-4554.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2048-2057, Lille, France. PMLR.
Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen. 2021. Radial graph convolutional network for visual question generation. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1654-1667.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2016. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. 31st AAAI Conference on Artificial Intelligence, AAAI 2017, pages 2852-2858.
Shijie Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang. 2016. Automatic Generation of Grounded Visual Questions. *IJCAI International Joint Conference on Artificial Intelligence*, pages 4235-4243.
Yue Zheng, Yali Li, and Shengjin Wang. 2018. Intention Oriented Image Captions with Guiding Objects. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:8387-8396.
A Training, testing and inference
Here, using an example, we clarify the inputs to our explicit model (Section 3.1) in the training, testing and inference setups.
- Ground truth question: What is the labrador about to catch?
- Answer: Frisbee
- Category: Object
- Image:  $i \in  {\mathbb{R}}^{{k}_{o} \times  d}$
- {Caption}: A man throwing a frisbee to a dog
- {Objects}: person, dog, frisbee, grass
N.B. {Caption} and {Objects} are both model generated, requiring only an image as input. These inputs are thus available at inference time.
Firstly, we create a set of candidate_concepts (see eq. 1) from the caption and objects: [person, dog, frisbee, grass, man, throwing]  $(\in \mathbb{R}^6)$ . These words are individually embedded. Secondly, we concatenate and embed the set of question and answer tokens  $(\in \mathbb{R}^7)$ .
Then, we construct a matrix which gives us cosine similarity scores for each candidate_concepts token to a QA token  $(\in \mathbb{R}^{6\times 7})$ . We choose  $k = 2$  tokens from the candidate_concepts which are most similar to the words from the QA. Here, "dog" and "frisbee" are likely chosen. Our input to the model is then  $<i$ , "object", "dog", "frisbee">.
Notice that it is possible for these words to be in the QA pair (e.g. "frisbee"). Importantly, these words have not been fed from the QA pair - they have been fed in from model-obtained concepts (\{Object\} and \{Caption\}). Philosophically similar, Krishna et al. (2019) constructed inputs based on target information for use in training and benchmarking.
Testing. Imagine a data labeler creating questions based on an image. They would look at the image, and decide on the concepts to create the question for. Our testing methodology follows this intuition using the strategy outlined above: the  $k = 2$  selected objects from candidate_concepts is a programmatic attempt for selecting concepts which could generate the target question. Note that there can be many questions generated for a subset of
concepts (e.g. 'is the dog about to catch the frisbee?', 'what is the flying object near the dog?' etc.). As outlined above, we are not taking concepts from the target. Rather we use information from the target to emulate the concepts an actor would think of to generate the target question. Because there can be different concepts questions are based on for one image (see ground-truth questions in Appendix D), our strategy allows us to generate questions which might be similar to a singular target question. This leads to an evaluation which fairly uses information a human has access to to generate a question.
Inference. However, in the real world, there is no 'ground-truth' question. In this case, we simply feed image features, and actor selected concepts to our question generator model. The selection process of the actor may be random - in which case a human agent does not need to be involved in the question generation process. The  $k \leq 2$  selected concepts here are a subset of candidate_concepts, which are fully generated from models.
Batch size	128
Learning rate	1e-5
Text model layers	12
Text model dimension	768
Image encoder layers	6
Image encoder dimension	768
Image encoder heads	8
Empirically, for both variants, we find  $k = 2$  to be the best number of sampled objects. All experiments are run with early stopping (patience 10; training iterations capped at 35000) on the BLEU-4 metric. Scores reported (in Section 5) are from the highest performing checkpoint. We use the PyTorch library and train our model on a V100 GPU (1.5 hours per epoch).
C Impact of model size on results
Table 3: Hyperparameters for our model variants.
Model	BLEU	CIDEr	METEOR	ROUGE
1	2	3	4
image-category	38.6	28.4	21.4	16.2	118	19.9	40.1
image-guided	44.5	34.4	27.4	22.1	197	24.6	47
Table 4: Truncated models single reference evaluation results.
Our models use the heavier Transformers than previous SoTA we compare to. For example, (Krishna et al., 2019) use ResNet and RNNs for their image encoder and question generator ( $\sim$ 18M parameters). Our models have between 200-300M parameters. To validate that our results are not purely attributable to model size, we train a truncated version of image-category and image-guided (explicit only). We truncate our models by using only the first and last layers of our BERT based encoders and decoders ( $\sim$ 36M parameters). Our closest model to theirs is the (truncated) explicit image-category, which achieves a BLEU-4 of 16.2 as seen in Table 4 - an improvement of 1.7 BLEU-4 over IMVQG's  $t$ -path. Even if we attribute  $100\%$  of this score improvement to the pre-trained nature of the BERT models we use, our methodology still introduces a 5.9 BLEU-4 increase over the image-category combination (truncated image-guided achieves a BLEU-4 of 22.1).
D More Qualitative Examples.
Examples can be seen in Figure 2 (next page). When examined, we see that the generated question accurately uses the guiding category when the category is valid for the given image. For example, 531086/1 has animal as the sampled category. Because no animal is present in the image, this category isn't valid for the image. The generated question then correctly relies on the object labels and visual modality to generate a valid question given the image. Similarly for 490505/2.
There are some cases where a sampled object/concept is not valid given an image. For example, at least one of the objects in 22929/1, 41276/1, 531086/2, 281711/1, 490505/1 is not valid. In this case the model usually relies on the other available guiding information, prioritising the category information (e.g. 531086/2). In rare cases, the model has failure cases where some of the valid sampled objects may not be used in the generated question (e.g. 293705/2 and 490505/2).
The concept extractor utilises a pre-trained image captioning model and object detector model. This may lead to an accumulation of downstream errors, especially if the data fed into the pre-trained models are from a significantly different data generating distribution than those used to train the model. In this erroneous case, the model will likely fallback to rely on the image modality and category information to produce a generic question
(e.g. 22929/1, 22929/2, 531085/1, 293705/2).
E Responsible NLP Research
Our approach claims to achieve SoTA in Visual Question Generation. However, we are only able to train and test our model on one dataset because it is the only existing dataset which contains answer categories. It is possible that our work may be suitable for use in a zero-shot setting, but we have not evaluated or tested our model in this setup.
E.2 Risks
Our model could be used to generate novel questions for use in Visual Question Answering. This may have a knock-on effect which leads to training more VQA models, thus having a negative impact on the environment.
Our model could be used in downstream tasks such as language learning. There may be incorrectness in the generated questions which has a knock on effect to a user using this model (e.g. the user may gain a wrong understanding of a concept because of a question the model has generated)
22929	41276	531086
Ground Truths (truncated @ 5)	is the bear bigger than the baby? is the baby showing the bear love? what is the child sitting with?	how many planes are there? is this a Boeing 737? is the sky blue? how many propellers on the plane? is this a single engine aircraft? is the landing gear visible?	do the cabinets have handles? what room is this? do they wash dishes by hand? are there any magnets on the fridge? is there a coffee maker in the photo?
Explicit	1. Sampled Category: 1. Sampled Objects: 1. Generated Question: 2. Sampled Category: 2. Sampled Objects: 2. Generated Question:	spatial baby jacket which of the two bears's arms is closer to the camera? activity ears arm what is the baby doing?	material cockpit tail what is the landing gear made of? count wings sky how many clouds are in the sky?	animal door wall what is the only colorful object on the wall? binary book door is there a dishwasher in the picture?
Valid category for image? On topic with category? On topic with objects? Valid question for image?	✓ ✓ ✓ ✘ ✘ ✓ ✓	✓ ✓ ✓ ✓ ✓ ✓ ✘ ✓	✓ ✘ ✓ ✓ ✘ ✓ ✓
281711	490505	Before Singing v. After Singing staged more stage 1/2 stage 3/4 stage 5/6 stage 7/8 stage 9/10 stage 11/12 stage 13/14 stage 15/16 stage 18/19 stage 20/21 stage 23/24 stage 26/27 stage 29/20
Ground Truths (truncated @ 5)	where are the paper towels hanging at? is this a museum? is there a plant? what type of flooring do you see?	what color is not included in the roses? has the envelope been opened? what color is the envelope?	is the electric bill being paid? would you pay money for staging? what color is the sofa?
Explicit	1. Sampled Category: 1. Sampled Objects: 1. Generated Question: 2. Sampled Category: 2. Sampled Objects: 2. Generated Question:	attribute television door is the refrigerator door open or closed? location counter magnet what is on top of the counter?	location ground flowers where is the vase? food flower counter what is the red and white item?	attribute living room, books what pattern is on the couch? shape counter leg what shape is the rug?
Valid category for image? On topic with category? On topic with objects? Valid question for image?	✓ ✓ ✓ ✘ ✘ ✓ ✓	✓ ✘ ✓ ✘ ✓ ✘ ✘ ✓ ✘	✓ ✓ ✓ ✓ ✓ ✘ ✓ ✓
Figure 3: Qualitative outputs from explicit variant being fed random guiding information. Failure cases are also shown.