SlowGuess/NAACL_2022_OCR huggingface dataset:
Nihir Vedd, Zixu Wang, Marek Rei, Yishu Miao, and Lucia Specia  
Imperial College London
{n.vedd19, zixu.wang, marek.rei, y.miao20, l.specia} @imperial.ac.uk
https://github.com/nihirv/guiding-vqg
In traditional Visual Question Generation (VQG), most images have multiple concepts (e.g. objects and categories) for which a question could be generated, but models are trained to mimic an arbitrary choice of concept as given in their training data. This makes training difficult and also poses issues for evaluation - multiple valid questions exist for most images but only one or a few are captured by the human references. We present Guiding Visual Question Generation - a variant of VQG which conditions the question generator on categorical information based on expectations on the type of question and the objects it should explore. We propose two variant families: (i) an explicitly guided model that enables an actor (human or automated) to select which objects and categories to generate a question for; and (ii) 2 types of implicitly guided models that learn which objects and categories to condition on, based on discrete variables. The proposed models are evaluated on an answer-category augmented VQA dataset and our quantitative results show a substantial improvement over the current state of the art (over 9 BLEU-4 increase). Human evaluation validates that guidance helps the generation of questions that are grammatically coherent and relevant to the given image and objects.
In the last few years, the AI research community has witnessed a surge in multimodal tasks such as Visual Question Answering (VQA) (Antol et al., 2015; Anderson et al., 2018), Multimodal Machine Translation (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018; Caglayan et al., 2019), and Image Captioning (IC) (Vinyals et al., 2015; Karpathy and Fei-Fei, 2015; Xu et al., 2015). Visual Question Generation (VQG) (Zhang et al., 2016; Krishna et al., 2019; Li et al., 2018), a multimodal task which aims to generate a question given an image, remains relatively under-researched despite
the popularity of its textual counterpart. Throughout the sparse literature in this domain, different approaches have augmented and/or incorporated extra information as input. For example, Pan et al. (2019) emphasised that providing the ground truth answer to a target question is beneficial in generating a non-generic question. Krishna et al. (2019) pointed out that requiring an answer to generate questions violates a realistic scenario. Instead, they proposed a latent variable model using answer categories to help generate the corresponding questions. Recently, Scialom et al. (2020) incorporated a pretrained language model with object features and image captions for question generation.
In this work, we explore VQG from the perspective of 'guiding' a question generator. Guiding has shown success in image captioning (Zheng et al. (2018) and Ng et al. (2020)), and in this VQG work we introduce the notion of 'guiding' as conditioning a generator on inputs that match specific chosen properties from the target. We use the answer category and objects/concepts based on an image and target question as inputs to our decoder. We propose our explicit guiding approach to achieve this goal. We additionally investigate an implicit guiding approach which attempts to remove the dependency on an external actor (see more below).
The explicit variant (Section 3.1) is modelled around the notion that an actor can select a subset of detected objects in an image for conditioning the generative process. Depending on the application, this selection could be done by a human, and algorithm or chosen randomly. For example, imagine either a open-conversation chat-bot or a language learning app. In the chat-bot case, a human may show the bot a picture of something. The bot may use randomly sampled concepts from the image (e.g. an object-detected tree) to ask a human a question upon. In the language learning case, the human may wish to select certain concepts they want the generated question to reflect. For exam
Human Language Technologies, pages 1640 - 1654
ple, they might select a subset of animal-related objects from the whole set of detected objects in order to generate questions for teaching the animal-related vocabulary in a language learning setting. Alongside the objects, the actor may also provide, or randomly sample, an answer category to the question generator.
The implicit variant (Section 3.2), on the other hand, is motivated by removing the dependency on the aforementioned actor. We provide two methodologies for our proposed implicit variant. The first uses a Gumbel-Softmax (Jang et al., 2016) to provide a discrete sample of object labels that can be used for generating a question. The second method employs a model with two discrete latent variables that learn an internally-predicted category and a set of objects relevant for the generated question, optimised with cross-entropy and variational inference (Kingma and Welling, 2014; Miao et al., 2016).
Human evaluation shows that our models can generate realistic and relevant questions, with our explicit model almost fooling humans when asked to determine which, out of two questions, is the generated question. Our experiments and results are presented in Section 5.
To summarise, our main contributions are: 1) The first work to explore guiding using object labels in Visual Question Generation; 2) A novel generative Transformer-based set-to-sequence approach for Visual Question Generation; 3) The first work to explore discrete variable models in Visual Question Generation; and 4) A substantial increase in quantitative metrics - our explicit model improves the current state of the art setups by over 9 BLEU-4 and 110 CIDEr.
2.1 Visual Question Generation
Zhang et al. (2016) introduced the first paper in the field of VQG, employing an RNN based encoder-decoder framework alongside model-generated captions to generate questions. Since then, only a handful of papers have investigated VQG. Fan et al. (2018) demonstrated the successful use of a GAN in VQG systems, allowing for non-deterministic and diverse outputs. Jain et al. (2017) proposed a model using a VAE instead of a GAN, however their improved results require the use of a target answer during inference. To overcome this unrealistic requirement, Krishna et al. (2019) augmented the VQA (Antol et al., 2015) dataset with answer
categories, and proposed a model which doesn't require an answer during inference. Because their architecture uses information from the target as input (i.e. an answer category), their work falls under our definition of guided generation. More recently, Scialom et al. (2020) investigate the cross modal performance of pre-trained language models by fine-tuning a BERT (Devlin et al., 2018) model on model-based object features and groundtruth image captions. Other work, such as Patro et al. (2018), Patro et al. (2020) and Uppal et al. (2020), either do not include BLEU scores higher than BLEU-1, which is not very informative, or address variants of the VQG task. In the latter case the models fail to beat previous SoTA on BLEU-4 for standard VQG. Recently and (Xu et al., 2021) and (Xie et al., 2021) achieve SoTA in VQG using graph convolutional networks. However, both works follow an unrealistic setup by conditioning their model on raw answers during training and inference - a dependency we attempt to remove.
2.2 Discrete (Latent) Variable Models
Discrete variable models are ideal for tasks which require controllable generation (Hu et al., 2017) or 'hard' indexing of a vector (Graves et al., 2016). Existing literature provides several methods to achieve discretization. NLP GAN literature (such as SeqGAN (Yu et al., 2016) and MaskGAN (Fedus et al., 2018)) commonly use REINFORCE (Williams, 1992) to overcome differentiability issues with discrete outputs. Other discretization methodologies can be found in Variational Auto Encoder (VAE) literature (Kingma and Welling, 2014). Some older methodologies are NVIL (Mnih and Gregor, 2014) and VIMCO (Mnih and Rezende, 2016). However, VAE literature also introduced Concrete (Maddison et al., 2016), Gumbel-Softmax (Jang et al., 2016) and Vector Quantization (Oord et al., 2017) as discretization strategies (technically speaking, Concrete and Gumbel-Softmax are strongly peaked continuous distributions).
In this work, we use a Gumbel-Softmax approach to sample a distribution over objects. At inference time, given a set of object tokens, learning this 'hard' distribution allows the model to internally sample a subset of objects that produce the most informative question. Our variational model additionally learns a generative and variational distribution that allow the model to implicitly learn which objects are relevant to a question and an
Figure 1: Architecture of the explicit model (a) and implicit model (b)
(a) Architecture of our explicit model. Given an image, first an object detection model is used to extract object labels and object features; a captioning model is used to generate relevant captions. Questions and answers are concatenated to filter the conceptual information from generated objects and captions. Next the filtered concepts are combined with the category as the input to the text encoder; the extracted object features are fed into an image encoder. Finally the outputs from the text encoder and the image encoder are fused into the decoder for question generation.
swer pair whilst incorporating non-determinism for diverse outputs.
We introduce the shared concepts of our explicit and implicit model variants, before diving into the variant-specific methodologies (Section 3.1 & 3.2).
For both variants, we keep the VQG problem grounded to a realistic scenario. That is, during inference, we can only provide the model with an image, and data that can either be generated by a model (e.g. object features or image captions) and/or trivially provided by an actor (i.e. answer category and a selected subset of the detected objects). However, during training, we are able to use any available information, such as images, captions, objects, answer categories, answers and target questions, employing latent variable models to minimise divergences between feature representations
of data accessible at train time but not inference time. This framework is inspired by Krishna et al. (2019). In Appendix A, we discuss the differences of input during training, testing and inference.
Formally, the VQG problem is as follows: Given an image  $\tilde{i} \in \tilde{I}$ , where  $\tilde{I}$  denotes a set of images, decode a question  $q$ . In the guided variant, for each  $\tilde{i}$ , we also have access to textual utterances, such as ground truth answer categories and answers. The utterances could also be extracted by an automated model, such as image captions (Li et al., 2020), or object labels and features (Anderson et al., 2018). In our work, answer categories take on 1 out of 16 categorical variables to indicate the type of question asked. For example, "how many people are in this picture?" would have a category of "count" (see Krishna et al. (2019) for more details).
Text Encoder. For encoding the text, we use BERT (Devlin et al., 2018) as a pre-trained lan
guage model (PLM). Thus, for a tokenised textual input  $\tilde{S}$  of length  $T$ , we can extract a  $d$ -dimensional representation for  $\tilde{s}_t \in \tilde{S}$ :  $X = \mathrm{PLM}(\tilde{S}) \in \mathbb{R}^{T \times d}$
Image Encoder. Given an image  $\tilde{i}$ , we can extract object features,  $f\in \mathbb{R}^{k_o\times 2048}$ , and their respective normalized bounding boxes,  $b\in \mathbb{R}^{k_o\times 4}$  with the 4 dimensions referring to horizontal and vertical positions of the feature bounding box. Following the seminal methodology of Anderson et al. (2018),  $k_{o}$  is usually 36. Subsequent to obtaining these features, we encode the image using a Transformer (Vaswani et al., 2017), replacing the default position embeddings with the spatial embeddings extracted from the bounding box features (Krasser and Stumpf, 2020; Cornia et al., 2019). Specifically, given  $f,b$  from image  $\tilde{i}$ :  $i = \mathrm{Transformer}(f,b)\in \mathbb{R}^{k_o\times d}$
Text Decoder. We employ a pretrained Transformer decoder for our task (Wolf et al., 2020). Following standard sequence-to-sequence causal decoding practices, our decoder receives some encoder outputs, and auto-regressively samples the next token, for use in the next decoding timestep. Our encoder outputs are the concatenation (; operator) of our textual and vision modality representation:  $X = [S; i] \in \mathbb{R}^{(T + k_o) \times d}$ , and our decoder takes on the form:  $\hat{q} = \operatorname{Decoder}(X)$ , where  $\hat{q}$  is the predicted question.
In this work, we primarily focus on a set-to-sequence problem as opposed to a sequence-to-sequence problem. That is, our textual input is not a natural language sequence, rather an unordered set comprising of tokens from the answer category, the object labels, and the caption. How this set is obtained is discussed in following section. Due to the set input format, we disable positional encoding on the PLM encoder (Text Encoder in Figure 1).
3.1 Explicit Guiding
As mentioned in Section 1, the explicit variant requires some actor in the loop. Thus, in a real world setting, this model will run in two steps. Firstly, we run object detection (OD) and image captioning (IC) over an image and return relevant guiding information to the actor. The actor may then select or randomly sample a subset of objects which are sent to the decoder to start its generation process. If the actor opts for a random sample strategy, no human is needed during the inference process (see Appendix A for examples).
To enable this setup, we create paired data based
on the guided notion. At a high level, our approach creates this data in three steps: 1) obtain object labels; 2) obtain concepts via IC Formally,
$$
\mathrm {o b j e c t s} = \mathrm {O D} (i) \in \mathbb {R} ^ {k _ {o}}
$$
$$
\mathbf {c a p} = \operatorname {C a p t i o n M o d e l} (i) \in \mathbb {R} ^ {T _ {c a p}}
$$
$$
\mathsf {c a p} = \operatorname {r m S t o p W o r d s} (\mathsf {c a p t i o n}) \in \mathbb {R} ^ {<   T _ {\mathsf {c a p}}}
$$
$$
\text {c a n d i d a t e} = \operatorname {s e t} (\text {o b j e c t s}; \text {c a p}) \in \mathbb {R} ^ {T _ {c c}} \tag {1}
$$
Here, OD stands for an object detector model, rmStopWords is a function which removes the stop words from a list, and set is a function which creates a set from the concatenation (the; operator) of the detected objects and obtained captions. cap stands for caption. The set is of size  $T_{cc} < k_o + T_{cap}$ . Using this obtained candidate_concepts set, we run our filtration process.
Once the set of candidate concepts has been constructed, we filter them to only retain concepts relevant to the target QA pair. After removing stop words and applying the set function to the words in the QA pair, we use Sentence-BERT (Reimers and Gurevych, 2019) to obtain embeddings for the candidate QA pair and candidate_concepts (Eq 1). We subsequently compute a cosine similarity matrix between the two embedding matrices, and then select the top  $k$  most similar concepts. The chosen  $k$  concepts,  $\tilde{S}$ , are always a strict subset of the candidate concepts that are retrieved using automated image captioning or object detection. This process emulates the selection of objects an actor would select in an inference setting when given a choice of possible concepts, and creates paired data for the guided VQG task. We now concatenate an answer category to  $\tilde{S}$ :  $S = \mathsf{PLM}([\tilde{S};\mathsf{category}]) \in \mathbb{R}^{T\times d}$ .
With text encoding  $S$ , we run the model, optimizing the negative log likelihood between the predicted question and the ground truth. Note that the concatenation in the decoder below is along the sequence axis (resulting in a tensor  $\in \mathbb{R}^{T + k_o\times d}$ ).
$$
\hat {q} = \operatorname {D e c o d e r} ([ S; i ]) \tag {2}
$$
$$
\mathcal {L} = \text {C r o s s E n t r o p y} (\hat {q}, q)
$$
3.2 Implicit Guiding
We now introduce our experiments for the implicit variant for VQG. This variant differs from its explicit counterpart as it aims to generate questions using only images as the input, while internally learning to predict the relevant category
and objects. Mathematically, the explicit variant models  $\hat{q} = p(w_t|i,\tilde{S},\text{category},w_0,\dots,w_{t - 1};\theta)$  where  $\tilde{S}$  and category are obtained as described in Section 3.1. During inference, the implicit variant instead attempts to model  $\hat{q} = p(w_t|i,\tilde{e}_{obj},e_{cat},w_0,\dots,w_{t - 1};\theta)$  where  $\tilde{e}_{obj},e_{cat}$  are not explicitly fed in to the model. Rather, they are determined internally as defined in Equation 6.
Given an image, we apply the same object detection model as in the explicit variants to extract object labels, which are then encoded using an embed layer. Formally,
$$
\text {o b j e c t s} = \mathrm {O D} (i) \in \mathbb {R} ^ {k _ {o}} \tag {3}
$$
$$
e _ {o b j} = \operatorname {e m b e d} (\text {o b j e c t s}) \in \mathbb {R} ^ {k _ {o} \times d} \tag {5}
$$
Since we would like the implicit model to learn relevant objects for an image internally, we project each object in  $e_{obj}$  to a real-valued score:
$$
s c o r e s = \operatorname {M L P} \left(e _ {o b j}\right) \in \mathbb {R} ^ {k _ {o}} \tag {4}
$$
Subsequently, we apply a hard Gumbel-Softmax (Jang et al., 2017) to obtain predictions over selected objects. Because Gumbel-Softmax samples from a log-log-uniform distribution, stochasticity is now present in our sampled objects. To sample  $k$  objects, we tile/repeat scores  $k$  times before inputting it into the Gumbel-Softmax.  $\tilde{z}$ , our  $k$ -hot sampled objects vector, is then used to mask object embeddings for use in decoding:
$$
\tilde {z} = \underset {h \times d} {\operatorname {g u m b e l - s o f t m a x}} (\text {s c o r e s}, k) \in \mathbb {R} ^ {k _ {o}} \tag {5}
$$
$$
\tilde {e} _ {o b j} = \tilde {z} * e _ {o b j} \in \mathbb {R} ^ {k _ {o} \times d}
$$
Where  $*$  denotes element-wise multiplication. Categories can also be a strong guiding factor and instead of making it an explicit input, we build a classifier to predict possible categories. In this variant,  $\tilde{e}_{obj}$  is used as an input to both our text encoder, and the MLP responsible for the category prediction:
$$
S = \operatorname {P L M} \left(\tilde {e} _ {o b j}\right) \in \mathbb {R} ^ {k _ {o} \times d}
$$
$$
p \left(\hat {c a t} \mid \tilde {e} _ {o b j}\right) = \operatorname {s o f t m a x} \left(\mathrm {M L P} \left(\tilde {e} _ {o b j}\right)\right) \in \mathbb {R} ^ {k _ {c a t}} \tag {6}
$$
Using the one-hot representation of the predicted category (i.e.  $e_{cat} = \text{one - hot}(p(\hat{cat}|\tilde{e}_{obj}))$ , we can concatenate our image, PLM representation of objects, and predicted category to feed into the decoder:  $\hat{q} = \text{Decoder}([i; S; e_{cat}]) \in \mathbb{R}^{T_{\hat{q}}}$ . However, during training, we teacher force against
the 'gold' set of objects,  $\tilde{S}$  (obtained using candidate_concepts in Equation 1). Training and optimization thus follow:
$$
\hat {q} = \operatorname {D e c o d e r} ([ i; \tilde {S}; e _ {c a t} ]) \in \mathbb {R} ^ {T _ {\hat {q}}}
$$
$$
\mathcal {L} = \text {C r o s s E n t r o p y} (\hat {q}, q) + \tag {7}
$$
$$
C r o s s E n t r o p y (p (\hat {c a t} | \tilde {e} _ {o b j}), c a t) +
$$
$$
\mathrm {S t a r t E n d} (\tilde {e} _ {o b j}, \tilde {S})
$$
where StartEnd is a BERT QA-head style loss (Devlin et al., 2018) that uses binary cross entropy for each  $k$  in  $\tilde{e}_{obj}$ .
Variational Implicit. Hypothesising that ground-truth QA pairs might provide information useful to selecting objects, we additionally attempt to extend our model to incorporate QA pairs to learn a latent variational distribution over the objects. However, since QA pairs can only be used during training to learn a variational distribution, we introduce another generative distribution that is only conditioned on the images and extracted objects. We borrow the idea from latent variable models to minimise Kullback-Leibler (KL) divergence between the variational distribution and generative distribution, where the variational distribution is used during training and the generative distribution is used in inference.
Continuing from Equation 3, we build two matrices,  $M_{gen}$  and  $M_{var}$ . The former is a concatenation of the image features and object embeddings, and the latter the concatenation between the encoded QA pair and  $M_{gen}$ . Depending on whether we're in a training or inference regime, the CLS token of the relevant matrix is used to sample a mask,  $\tilde{z}$ , which is subsequently applied on the aforementioned object embeddings:
$$
M _ {g e n} = \mathrm {e n c o d e} ([ e _ {o b j}; i ]) \in \mathbb {R} ^ {2 k _ {o} \times d}
$$
$$
e _ {q a} = \operatorname {e m b e d} (\mathrm {Q}; \mathrm {A}) \in \mathbb {R} ^ {T _ {q a} \times d}
$$
$$
M _ {v a r} = \mathrm {e n c o d e} ([ e _ {q a}; M _ {g e n} ]) \in \mathbb {R} ^ {(2 k _ {o} + T ^ {q a}) \times d}
$$
$$
q _ {\phi} (z | M _ {g e n}, M _ {v a r}) = \operatorname {M L P} \left(M _ {g e n} ^ {C L S}; M _ {v a r} ^ {C L S}\right) \in \mathbb {R} ^ {k _ {o}}
$$
$$
p _ {\theta} (z | M _ {g e n}) = \mathrm {M L P} (M _ {g e n}) \in \mathbb {R} ^ {k _ {o}}
$$
$$
\tilde {z} = \operatorname {g u m b e l - s o f t m a x} (z, k) \in \mathbb {R} ^ {k _ {o}}
$$
where  $q_{\phi}(z|M_{gen},M_{var})$  is the variational distribution,  $p_{\theta}(z|M_{gen})$  is the generative distribution, and MLP denotes a multilayer perceptron for learning the alignment between objects and QA pairs.
encode is an attention-based function such as BERT (Devlin et al., 2018). From here, our methodology follows on from Equation 6. However, our loss now attempts to minimise the ELBO:
$$
\begin{array}{l} \mathcal {L} = \mathbb {E} [ \log p _ {\theta} (\hat {q} | z, \hat {c a t}) ] \\ - D _ {\mathrm {K L}} \left[ q _ {\phi} \left(z | M _ {g e n} ^ {C L S}, M _ {v a r} ^ {C L S}\right) \right\rvert \left| p _ {\theta} \left(z | M _ {g e n} ^ {C L S}\right) \right] \\ + \log p (\hat {c a t} | M _ {v a r}) \\ \end{array}
$$
We use the VQA v2.0 dataset<sup>1</sup> (Antol et al., 2015) (CC-BY 4.0), a large dataset consisting of all relevant information for the VQG task. We follow the official VQA partition, with i.e. 443.8K questions from 82.8K images for training, and 214.4K questions from 40.5K images for validation. Following Krishna et al. (2019), we report the performance on validation set as the annotated categories and answers for the VQA test set are not available.
We use answer categories from the annotations of Krishna et al. (2019). The top 500 answers in the VQA v2.0 dataset are annotated with a label from the set of 15 possible categories, which covers up the  $82\%$  of the VQA v2.0 dataset; the other answers are treated as an additional category. These annotated answer categories include objects (e.g. "mountain", "flower"), attributes (e.g. "cold", "old"), color, counting, etc.
We report BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015), METEOR (Lavie and Agarwal, 2007), and MSJ (Montahaei et al., 2019) as evaluation metrics. The MSJ metric accounts for both the diversity of generated outputs, and the n-gram overlap with the ground truth utterances.
4.2 Comparative Approaches
We compare our models with four recently proposed VQG models Information Maximising VQG (IMVQG; supervised with image and answer category) (Krishna et al., 2019), What BERT Sees (WBS; supervised with image and image caption) (Scialom et al., 2020), Deep Bayesian Network (DBN; supervised with image, scenes, image captions and tags/concepts) (Patro et al., 2020), and Category Consistent Cyclic VQG (C3VQG; supervised with image and answer category) (Uppal et al., 2020). We follow IMVQG's evaluation setup