Ask Question with Double Hints:  Visual Question Generation with Answer-awareness and Region-reference
Authors: Kai Shen
Lingfei Wu
Siliang Tang
Fangli Xu
Zhu Zhang
Yu Qiang
Yueting Zhuang
Venue:  (2021)
Keywords: Semi-supervised Learning
graph neural network
vision and language
question generation

Abstract: The task of visual question generation~(VQG) aims to generate human-like questions from an image and potentially other side information (e.g. answer type or the answer itself). Despite promising results have been achieved, previous works on VQG either i) suffer from one image to many questions mapping problem rendering the failure of generating referential and meaningful questions from an image, or ii) ignore rich correlations among the visual objects in an image and potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. In particular, we aim to ask the right visual questions with \emph{Double Hints - textual answers and visual regions of interests}, effectively mitigating the existing one-to-many mapping issue. To this end, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework that first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilize a graph-to-sequence model to generate the questions with double hints. Our experiments on VQA2.0 and COCO-QA datasets demonstrate that our proposed model on this new setting can significantly outperform existing state-of-the-art baselines by a large margin. 

Under review as a conference paper at ICLR 2021
ASK QUESTION WITH DOUBLE HINTS: VISUAL QUES- TION GENERATION WITH REGION-REFERENCE
AND
Anonymous authors Paper under double-bling blind review
ABSTRACT
The task visual question generation (VQG) aims to generate human-like ques- tions from an image and potentially other side information (e.g. answer type the answer itself). Despite promising results have been achieved, previous works VQG either i) suffer from one image to many questions mapping problem rendering the failure of generating referential and meaningful questions from an image, ii) ignore rich correlations among the visual objects in image and potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. In particular, we aim to ask the right visual questions with Double Hints lextual answers and visual regions of interests, effectively mitigating the existing me-to--may mapping issue. To this end, we develop a simple methodology self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose new souble-hints guided traph-to-Sequence learn- ing framework that first models them dynamic graph and learns the implicit topology end-to-end, and then utilize a graph-to-sequent model to generate the questions with double hints. Our experiments demonstrate that our proposed model on this new setting can significantly outper- form existing state-of-te oftt-----he-att baselines by large margin.
VQA2.0 and COCO-QA datasets
INTRODUCTION
Visual Question Generation (VQG) is an emerging task in both computer vision (CV) and natural language processing (NLP) fields, which aims to generate human-like questions from image and potentially other side information (e.g. answer type or answer itself). Recent years have seen a surge of interests in VQG because it is particularly useful for providing high-quality synthetic training data for visual question answering (VQA) ) (Li et al., 2018) and visual dialog system (Jain et al. 2018). Conceptually, it is challenging task because the generated questions not only required to be consistent with the image content but also meaningful and answerable to human.
Despite promising results have been achieved, previous works still encounter two major issues. First, all of existing methods significantly suffer from one image to many questions mapping problem rendering the failure of generating referential and meaningful questions from an image. The existing VQG methods can be generally categorized into three classes with respect to what hints are used for generating visual questions: 1) the whole image as the only context input (Mora et al. 2016); 2) the whole image and the desired answers (Li et al. 2018); 3) the whole image with the desired answer types (Krishna et al.) 2019). Since a picture is worth a thousand words, it can potentially mapped to many different questions, leading to the generation of diverse non-informative questions with poor quality. Even with the answer type or desired answer information, the similar one-to-many mapping issue remains, partially because the answer hints often very short side information are often not informative enough for guiding question generation process, rendering the failure of generating referential and meaningful questions from image.
broad. As result, these
The second severe issue for the existing VQG methods is that they ignore the rich correlations among the visual objects in an image and potential interactions between the side information and

---

Under review as a conference paper at ICLR 2021
Double-Hints Guided Multi-Task Auto-Encoder
features
A black watch
BIGRU
Double Guided Space
Task 1: answer prediction
object features
oitaa hinds
position prediction
Masked RCNN
Cross-Modal Alignment
visual hints prediction
ResNet
Double-Hine Guided Craph-to-Sequenee Module
Image feotures
Guide
Topology Learning
Topalogy
Question: What on the man's left wrist
GCN
Space
Language
Node teatures
Graph(Attention
object features
Image
Fusion
A/
Attention LSTM
Figure The overall framework of our proposed model with double hints to guide VQG.
image Krishhaa et al. 2019). Conceptually, the implicit relations among the visual objects (c.g.. spatial, semantic) could be the key generate meaningful and high-quality questions. This is partially because when human annotators ask questions about a given image, they often focus on these kinds of interactions. In addition, another important factor for producing informative and referential questions is about how to make full use of side information to align with the targeted image. Modeling such potential interactions between the side information and an image becomes a critical component for generating referential and meaningful questions.
To address these aforementioned issues, in this paper. we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. More specifically, aim to utilize the referential visual regions of interest hints (denoted as visual hints for simplicity) of the images and the textual answers (denoted answer hints) to faithfully guide question generation. As illustrated in Figure I] by giving an image with visual hints (the region enclosed by the orange rectangle) and answer hints (the answer), the model able to faithfully generate the right question with key entities that reflects the visual hints and answerable to the answer hints. To this end, in order learn these visual hints, we develop a multi-task auto-encoder to learn the visual hints and the unique attributes automatically without introducing any additional human annotations.
Furthermore, to capture the rich interactions between visual and answer hints and the image well as the sophisticated relationships among the visual objects in an image, we propose new Double-Hinte learning framework SOH-Graph2Seq).. The proposed model first models these interactions as dynamic graph and learns the implicit topology end-to-end, and then utilize a Graph2Seq model to generate the questions with double hints. In addition, in the decoder side, we also present visual-hint guided separate attention mechanism to attend image and object graph separately and overlook the non-visual-Be particularly.
guided
In summary, we highlight main contributions follows:
We propose novel learning paradigm to generate visual questions with Double Hints textual answer and visual regions of interests, which could effectively mitigate the existing one-to me-too-many mapping issue. To the best of our knowledge, this is the first time both visual hints and answers hints are used for the VQG task.
Graph-to-Sequence (Graph2Seq) learning problem. We employ graph learning technique to learn the implicit graph topology to capture various rich interactions between and within an image, and then utilize a Graph2Seq model guide question generation with double hints.
We explicitly cast the VQG task
Our extensive experiments on VQA2.0 and COCO-QA datasets demonstrate that our pro- posed model can significantly outperform existing -ate-offthh-art by large margin.
2

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
2.1 VISUAL QUESTION GENERATION
Visual question generation is an emerging task in the visual-language domain. Mora et al. (2016) firstly makes an attempt adapt the CNN-LSTM model to generate image-related question and answer pairs. These works simply map the visual features into textural questions, which lead to abstract and general results (Mostafazadeh al., 2016; Zhang et al., 2016). Jain al. (Jain et al.) 2017) use variational autoencoder together with an LST decoder to generate diverse questions. What's more, dual learning is applied in VQG task (Li et al. 2018 Shah et al.) 2019). They view VQG the dual task of VQA, thus optimizing both of them according to cycle-consistency. Liu ct al. 2018) view this task the inverse of VQA and combining the type-specific partial question with the question-answer pairs produce more reasonable results. Furthermore, Krishna et al. (Krishna et al. 2019) introduce the fine-gre teeeggaaiedd answer type as the guidance to the variational method, which generates goal-driven questions. [u et al. (2020) adapt the graph method to generate meaningful questions with the target of answers. Different from these works, we will generate the questions under the guidance of both visual regions of interest and textual answers, which can generate questions more referential and answerable.
2.2 CRAPH-TO-SSQUENNT LEARNING
Graph neural networks (GNN) (Kipf & Welling 2016 Gilmer et al. (017) has drawn significant amount of attention in recent years. In the NLP domain, graph-to-sequence learning is to generate se- quential results from graph-structured data, which mapping the structural data sequence output (Xul cet al. 2018 Chen et al. 2019c Gao et al. (011)). When coming to the non-graph structured data just like the regions of an image, researchers explore some methods construct objects' (Norcliffe-Brow) et al. 2018) or words' (Chen et al. 2019b topology.
Unlike these previous methods, we propose novel cross-modal graph2seq model which models the relations upon the visual and textural hints. With the two modal's guidance, the model will learn the appropriate embedding which is crucial generate high-quality questions.
3
METHOD
We first introduce novel learning paradigm that generates visual questions with double hints. And then we present the details of each component of our proposed model as shown in Figure
3.1 PROBLEM FORMULATION
The goal of the visual question generation task is to generate human-like and answerable natural language questions based on the given images and potentially other side information, such as textural answer and answer type, where the generated questions need to be consistent with the given images and the hints semantically. In this paper, we will first discuss the traditional answer-based hints and then present our new learning paradigm.
We assume that the raw image is I, and the target answer (answer hint) is a collection of word tokens represented by X" = {ai, a2 ... ). where m denotes the length of the answer. The task of traditional seswer-guided VQG is to generate the natural language question consisting of a sequence of word tokens X" {qi, 92. In) which maximizes the conditional likelihood X = xx L where n is the number of word tokens. To address the proposed issues, we introduce a new setting that focuses not only on the answer-awareness but also on region-reference. Specifically, we formulate it as a joint optimization over visual hints finding and double-hints guided graph2seq learning tasks. The visual hint is a collection of visual object regions of interest in the image which are direct visual clues for question generation. We denote it as V= [21, 02 UN ), where 0; is the bounding box in the image and N is the number of visual hints. Under this setting, the likelihood can be cast particular, the visual hints will be generated in data pre-pros pre-processing (see Sec. [4.) and the model can learn them without human annotations. Thus there are visual hints during inference.
II
P V. I. P(V)) X*)). x)). In
3

---

Under review as a conference paper at ICLR 2021
To make the statement clear, we will give mathematical notations. We denote the input image feature as I Rex Pv, where is the channel number and F. is the visual feature dimension. Each image is given a collection of object regions denoted by V = {ui, U2,,.. )N ) where V is associated with its visual feature r; E (r; ε R), bounding box position vector P: [To, Yo: 21, v/). category attributes (eg. it belongs to animal) c₁ and visual hint indicator y; (ý, is true iff e2 is the visual hint during training stage). ( (#1, 31) are the normalized up-left and bottom-right coordinates. As for the textural answers, adopt GRU module encode the answer embedding X with words to learn the word semantic features X {ai, a2 .... am R Fa,, Fa denotes the answer vector dimension.
3.2 DOUBLE-HING GUIDED MULTI-TASK AUTO- -ETO-ENNODER
The most important point of our first issue located in how to effectively find the visual hints and how to combine the visual features with double hints. In this section, firstly introduce a cross-modal alignment that aligns the cross-modal visual objects and textural answers into the latent space. Then we propose multi-task decoder that can learn the visual hints as well keep the crucial attributes in the latent space.
3.2.1 CROSS-MODAL ALIGNMENTS
To infer which regions of interest are suitable for asking questions, answer clues and visual object features are both crucial. To exploit the fine-graing ene-grained interactions between visual clues and textual answer hints, explicitly model the glohal correlations between them in the emhedding space.
Firstly, we notice that the position and category attributes of objects are indispensable during fine- grained object relation modeling. So we incorporate them by projecting the position vector p; and category attribute C; for object 0; into two embedding space (we assume their dimensions are d). Then we concatenate them with visual feature r.. We overload r; here considering the conciseness, which means in the following content, E R Pv (r; E R. Fy F. + 2 s d) denotes the object features with position and category embedding.
For cach object r and answer word ay, we will calculate the alignment score Sty as follows:
S or. W, aj W.
where W, E Fv W. E Fax and W E Fasion, Fatign is the hidden dimension size. What's more, a denotes the Tanh function. So the alignment score matrix E Nm represents the correlations between the objects and the answer words. Then we align the objects with answer according to
R, x = CONCAT(R: Xatign), (R:tiggn)) Xatign = -38 ,,x",Bi = exp(S
H (
(2)
ES= exp(S.;)
where H is the aligned objects' representation and CONCAT is the row-wise concatenation operation.
3.2.2 MULTI-TAST DECODER
After aligning the visual features with the answers, introduce the multi-task decoder which can infer the visual hints and the unique attributes (i.e object position and answers). By this, the latent embedding absorbs the double hints while retaining the robust features like objects' position clues.
Firstly, we apply MIP with ReLU activation to project H to a F₁. dimension latent space denoted as L. Empirically, it is not proper to infer the attributes directly by L, individually. To infer the visual hints better, we capture the top-down signals (Anderson et al.) 2018) by attending to the image features. In other words, it is more reasonable combine the fine- ame-grained object features with the global coarse- coarse-grained image features. So we apply the attention mechanism between object L; and the whole image
exp(Sij)
Latta = BIW attn Bi,j =
ESEr exp(S.,) Sij o OLW
4

---

Under review as a conference paper at ICLR 2021
where Wi E R FaxPaa W E R W E R Wattn F.XFF, 3 R. Nxc and I E R Fv. Then we use residual connection to combine latent features and global features as Lend (L+ Lottm// 22. In the following paragraph, we will discuss the the multi-task decoder on the latent space Lenh to generate the visual hints and keep the unique attributes in the latent embedding.
Visual hint prediction We apply forard layer and the softmax function to predict the probability of visual hints denoted as Pek- In practice, the number of visual-hint objects are much smaller than -in-iisull- objects, so we combine ceaamcddcroos-entroon loss and focal loss (Lin
et al. 2017) together as follows:
Leh
Npos Es.rah,
1)
Naeg Σ(- - Pon,i)
(4)
where Nneg denotes the number of coo--vsuull-hitee and Npos is the number of visual hints.
Object position prediction For high-quality question generation, the relative spatial relations among objects are important clues. So we want to keep these essential features in the latent space via predicting the coordinates of the objects. Just like the previous visual hints prediction, we apply the feed-forward layer on Lenh and predict the absolute normalized position vector poos- The loss Lpos will be the mean square loss function of Ppos and ground-truie truth p.
Target answer prediction As the answer hint is the guidance of asking questions, we ensure that the latent embedding indeed retains it via regenerating the answer. Technically, we apply a dilated CNN with a max-pooling layer to get the global representation of the latent space, then we predict the answer we expect ask. We denote the answer is t4 E {to, t1, th,,ttt.] tc], where i is the - th sample, and c is the amount of all answers in the training dataset.
L poo! Maxbooo
(5)
Then we apply the feed-forward layer on Looot and softmax function to calculate the probability. Finally, we adopt the cross-entropy loss to calculate the answer prediction loss denoted as Lone-
3.3 DOUBLE HILLSS GUIDED GRAPH CONSTRUCTION AND EMBEDDING
To capture the complex correlations among visual objects the image, we regard the objects as nodes in the object graph G and adopt the paradigm of graph learning. Most common GNN applications often employ a GNN algorithm on structured data to capture the rich structure information and compute the graph node embedding directly. But for VQG, the objects are unstructured, whose relations are not explicitly given. So in this section, we construct the object graph g's topology with the guidance of double hints and then employ GNN model to encode them.
The topology of a graph represents the relations of the nodes. We claim that under our setting, the graph edges should be weighted and learnable (i.e G is dynamic graph) because the relation of two objects should be decided by the double hints. We take the latent embedding Lenn of multi-task auto-encoder exploit such topology since the objects' visual features incorporate the double hints well by the auto-encoder.
Inspired by (Chen et al. 2019a), good similarity metric function is supposed to be learnable and expressively powerful. We first calculate the dense similarity matrix S by multi-head weighted cosine similarity function as follows:
Σ
$
S cos( Wp @ Lenh,i, Wp C Lenh,)), Suy
where wp is the learnable weight, is the Hadamard product, and si,, denotes the similarity between node i and j of head p. The model learns to highlight specific dimensions of the latent embedding space by such formula. is the heads' number. The final similarity results will be the mean of the heads. Because the learned graph similarity matrix S is dense and ranges between [-1, 1], so adopt c-sparsing to make it neg-negative and sparse. Specifically, we mask off St (i.e., set to zero) if it is smaller than the threshold get the adjacency matrix A.
5

---

Under review as a conference paper at ICLR 2021
Then apply a multi-layer graph convolution network (GCN) to effectively learn the node embed- ding from the constructed object graph. Firstly, we align the object feature with the learned double hints. Similarly, the multi-tas task auto-encoder has fused the double hints and predicted the visual hints. Thus it's latent emhedding is the exact guidance we want. The node features will be represented as follows:
X = F(R CONCAT(R; Latent R: X²)) = CONCAT (R; Lenk)
(7)
where Latent is the auto-encoder's latent space representation in Sec. 3.2 And CONCAT is the node-wise concatenation. Then stack layers of classic spectral GCN (Kipf & Welling 2016)
together with residual architecture to aggregate the fine-grained region features. Appendix A is referred to the readers for more details.
3.4 DOUBLE-HING GUIDED QUESTION GENERATION MODULE
For question generation, we adapt the language model from (Lu et al. 2018) for the image, graph with double hints. The model consists of two LSTMs whose initial states are initialized by answer features: the first one for encoding global image feature and the word embedding q, into the hidden state h Rn where m is the dimension and the second one for question words prediction. Between the two LSTMs, the model will attend the image and graph separately by the visual hints. We refer to this procedure as visual-hint-guu hint-guidedd separate attention.
The visual- sail-hiitt-gdeedd separate attention takes in the image and graph and attended by hidden state h separately. Specifically for graph attention, we ignore the nodes which are predicted to be non-visual-hine hint. We name this mechanism as visual attention as follows:
X VisualHintMask((X
h hgraph Attention (Xuh: h1)
where VisualHintMask is to mask off the non-visual -inn objects, X is the graph node embedding and Attention is the classic attention mechanism (Bahdanau et al. 2014). We train with the cross-entropy loss Lgues-
The final loss will be the combination of the previous components.
L Lyues + aLyh + 3 + Lans
(9)
EXPERIMENTS
In this section, we evaluate the effectiveness of our proposed model. The code and data for our model are provided for research purposes'.
4.1 DATSETS AND PRE-PROCESSING
Datasets We conduct the experiments on the VQA2.0 (Antol et al. 2015) and COCO-QA (Ren et al., 2015) datasets. Both of them use images from MS-COCO (Lin al. 2014). And fit our setting, introduce a simple method to generate the visual hints of the original pairs (image, question, answer) without human annotations: (1) we use the Masked-RCNN generate objects (category attributes) V in the image. (2) we use standfordcoreNNLP to find the noun-words in both questions and answers. For each object attributes and noun-words we use the GloVe model initialize them and take an average to get the vector representation denoted by gobj and gnown- The visual-hin hint indicator y; is positive iff its' 12 distance with any gnoun is larger than the threshold μ.. But there are two special cases that can lead to no aligned objects: (1) there are exactly no visual hints (eg. Q: Is there any book? A: No) (2) the error caused by the detection model or the NL.P tools leads to no visual hints. For the first case, will drop them due to the technical drawback. For the second case, will keep them. Morcover, although one image could have multiple answer hints (one image-answer pair generally corresponds to one question), there are small portion of magge-nnswer pair linking to multiple questions. In this case, we will randomly reserve one. After processing, the VQA 2.0 contains 239973 examples for training split and 1 16942 for validation. And COCO-QA contains 53440 for training and 26405 for validation.
Cooe will he released upon the paper acceptance.

---

Under review as a conference paper at ICLR 2021
Table I: Results VQA2.0 and COCO-QA val All accuracies are in %. Notations: B@4- BLEU@4, C-CIDEr, M-METEOR, R- R-ROUGE, QB @ I-Q-BLEU@1.
Dataset
VQA 2.0
COCO-QA
Method
B@4
C
M
R
QB@I 1
B@4
C
M
R
QB @ 1
12Q
9.02
63.21
13.89
35.33
26.32
13,53
95.90
12.61
36.23
31.37
IT2Q
18.41
134.88
19.90
45.71
40.27
17.80
128.64
16.17
43.22
38.88
IMVQG
19.72
149.28
20.43
46.76
40.40
18.43
127.18
17.21
44.07
39.22
Dual
19.90
151.60
20.60
47.00
41.90
18.80
131.10
17.73
44.19
39.92
Radial
20.70
161.90
21.40
48.10
43.50
19.24
139.55
18.19
44.21
40.98
Ours
22.43
180.55
22.57
49.36
45.61
20.84
166.78
19.81
46.80
43.57
We claim that the comparison with other baselines under this setting quite fair since we just use data pre-procsssing to generate the visual hints. Furthermore, the generated visual hints are quite noisy since we align by GloVe cosine similarity instead the exact matching.
Pre-processing We apply pre-trained ResNet101 and Masked-RCNN to extract features. And we truncate questions longer than 20 words and build vocabulary words with at least 3 occurrences. Since the test split is not open for the public, we divide the train set to 90% train split and 10% validation split. Please refer to Appendix C for more detailed pre-processing and implement settings.
4.2 BASELINE METHOSS AND EVALUATION METRICS
Baseline Methods We compare against the following baselines in our experiments: 12Q (means image to question), ii) IT2Q (means images with answer type hints to question), iii) IMVQG* (Krishna al. 2019). iv) Dual* (Li al. 2018). v) Radial* (Xu al.) 2020). The Detailed description of these baselines is provided in Appendix B Experiments on baselines followed by are conducted using the codes originally released by authors.
Evaluation Metrics
Following previous works, we adopt the standard linguistic measures including BLEU (Papineni et al. 2002), CIDEr (Vedantam et al. 2015), METEOR (Banerjee & Lavie 2005). ROUGE-L (Lin) 2004) and Q-BLEU (Nema & Khapra 2018). For visual hints prediction, we adopt Fl metrie. And for answer and object position prediction tasks, we use accuracy and mcan-IOU (Intersection Over Union) metrics. These scores are calculated by official evaluation scripts.
4.2.1 RESULT ANALYSIS AND HUMAN EVALUATION
Table shows the automatic evaluation results comparing our proposed method against other state-of- art baselines. Note that we presents the linguistic evaluation metrics and the visual-hints, answer and object position prediction tasks' performance details in Appendix D We can find that our method consistently outperforms previous methods by a significant margin both datasets. It highlights that our proposed method which asking questions with double hints achieves state-of-art performance, thus address the two issues we highlighted in Sec.
omitted due to limited space, please refer to more
Furthermore, we conduct a human evaluation study to assess the quality of the questions generated by method, the ground truth (GT) and the baselines: Radial and Dual in terms of syntax, semanties, and relevance metrics. The results are shown in Table Details are shown in Appendix F We can that our model outperforms all the strong baseline methods by large margin all metrics. And we observe slightly inferior results on the semantics metric and margin on the relevance metric. So we conduct error analysis which is shown in the following section.
4.3 ABLATION STUDY
In this section, we conduct the ablation studies to demonstrate the effect double hints, double-hints guided graph, visual attention, and multi-task auto-encoder the VQA 2.0. Concretely, will remove one or more components at one time to generate the ablation model as follows: (1) w/o. visual hints (abbr: -visual hints): remove the visual-hints prediction module. (2) w/o. double hints (abbr:
7

---

Under review as a conference paper at ICLR 2021
-iisual/answer hints answer type): remove both the visual and answer hints but the answer- type to further assess the effect of double hints. (3) w/o. double-hints guided graph (abbr: -gnn): we remove the implicit graph construction and gen encoding modules. (4) w/o. visual attention (abbr: -visual attention): we remove the non-visual-hint mask during attention when decoding. (5) w/o. double-hints guided graph and visual attention (abbr: gnn, visual attn): We combine the ablation (3) and (4) together to further assess the effect of them. (6) w/o. position and answer auto-encoder (abbr: -pos/ans predict): we / and y zero for reducing mulli-task auto-encoder visual hints only based auto-encoder.
The ablation study results on VQA2.0 val set's results are shown in Table 2 Firstly, the full model outperforms all the ablation models. It confirms that the visual hints, multi-task auto-encoder, and the graph are useful components.
Specifically, by turning off the visual hints, the model's performance drops nearly 2.8% (METEOR). confirms that the visual hints are indeed helpful for high-quality question generation. And when we further discard the answer hints, the performance continues to drop rapidly by 18.2% (BLEU@4) which further demonstrates that the answer hints are a more helpful signal compared to answer type. These results indicate the importance of double hints.
In addition, by turning off the double-hints guided graph, we observe that the performance drops nearly 1.2% (METEOR). It demonstrates that it is beneficial to exploit the hidden relations by learning a dynamic graph. Furthermore, when we discard the visual attention during decoding, the performance drops by 1.3% (METEOR). This result illustrates that if we force the model to focus on only the predicted visual-hint object regions, it can generate higher quality questions, which further confirms the impact of visual hints. Additionally, if attention, the performance continues to drop, which further confirms that both these two components play an indispensable role in the framework. Finally, we observe that we discard the multi-tasy auto-encoder, the performance drops slightly. It shows that by predicting the object position and target answer, the model can learn more robust visual hint features.
off both the graph and the visual
Table 2: Ablation Results on VQA2.0.
Table 3: Human-study Results on VQA2.0.
Method
B@4
M
QB@1
Method
Syntax
Semantics
Relevance
Full (Duble hints)
22.43
22.57
45.61
Radial
4,25 (0.36)
4.04 (0.45)
3.09 (0.24)
Dual
4.53 (0.40)
4.38 (0.58)
3.24 (0.26)
visual hints
21.67
21.93
44.52
Ours
4.73 (0.35)
4.61 (0.40)
3.58 (0.39)
visual/answer hints
18.53
20.06
40.32
GT
4.52 (0.34)
4.63 (0.41)
4.14 (0.50)
answer type
-gan
22.17
22.31
45.34
visual attn
21.98
22.26
45.11
-gnn, visual attn
21.92
22.21
45.04
pposaans predict
22.34
22.51
45.57
4.4
CASE STUDY AND ERROR ANALYSIS
Ground Truth
Prediction
Answen: 4
Others 14%
Detection Error
GT: How many skateboard wheels visill??
Radial: How many people are in the photo?
Semantic Error 15%
Nsual Hints Prediction Error
17%
Dual: How people picture?
Syntax Error
Reasoning Error
Ours: How many wheels are skateboard?
3%
Figure 2: Case study (left) and error analysis (right) results.
In this section, will further show example to illustrate the quality of our method compared with the same baselines. What's s more, we also dive into the failures of our model by analyzing the emorr cases. The results are shown in Figure 2 for more concrete results please refer Appendix G
8

---

Under review as a conference paper at ICLR 2021
As see, by learning the object graph and generating with the guidance of double hints, can find that our model indeed generates more answer-aware, region-referentill and high-quality questions.
And for error cases, it is very difficult to classify a certain example as a certain error cause, we can only roughly divide it into five categories: 1) visual hint prediction error: It means the error made by visual hints prediction and further mislead the generation. 2) detection error: It means our model detects the objects mistakenly. 3) reasoning error: It means model makes the wrong relation reasoning. 4) syntactic error: It means there are grammar errors. 5) semantic error: It means the question has drawbacks in the semantic view. Further detailed examples and analysis are shown in Appendix H
0.4844
0.4389
0.4133
Accuracy
0.3878
0.3622
0.3367
BUTD BUTD wDH WDHHGGraph2Se
0.2600
Percentage of training data
Figure 3: Performance of VQA model in under different proportions of supervised data.
4.5 DATA AUGMENTATION
One of the most important applications of VQG is providing training data for VQA. In this section, we use the proposed model to generate more questions for training VQA methods. Firstly, adopt the classic Bottom-Up Top-Down (abbr: BUTD) (Anderson et al. 2018) as the VQA benchmark on VQA 2.0 dataset and use top-1 accuracy as the evaluation metric. In order to examine the effect of QG-driven data augmentation of the BUTD baseline with two data augmentation variants, namely, BUTD w/Radial and BUTD Grag-Sga Specifically,
the VQA, compare the performance
split the train-set (the same as the VQG dataset) 2% (0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 1}. The BUTD baseline is trained only on the part, while the other two variants are trained the combination of the golden part and the questions generated by VQG models. What's 's more, to train the VQG model, we further split part to 80%/20% (tran/ddde) The results are shown
From the results, we observe that both VQG models consistently help improve VQA performance. Notably, our DH-Graph2Seq model outperforms the Radial baseline consistently. In particular, when is relatively smaller, which means that under a low-source condition, data-augmentation is more effective.
CONCLUSION
In this paper, we propose new setting with double hints for the visual question generation task. Under the new setting, we further design novel DH-Graph2Seq model which can explore the correlations among the objects in the image generate the questions with answer-awarenes: and region-reference. Our extensive experiments VQA2.0 and COCO-QA datasets demonstrate that our proposed model can significantly outperform existing state-of the-art by a large margin.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077-6086, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprini arXiv: httl::4409.4777 2014.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Deep iterative and adaptive learning for graph neural networks. arXiv preprint 1912.07822 2019aa.
Yu Chen, Lingfei Wu, and Mohammed J Zaki. Graphflow:
Exploiting conversation flow with graph neural networks for conversational machine comprehension. arXiv preprint 1908 2019b.
Yu Chen, Lingfei Wu, and Mohammed J Zaki. Reinforcement learning based graph-to-sequence model for natural question generation. arXiv preprint arXiv: 04942, 2019c.
Yuyang Gao, Lingfei Wu, Houman Homayoun, and Liang Zhao. Dyngraph2seq: Dynamic-graph-4a sequence interpretable learning for health stage prediction in online health forums. arXiv preprint arXiv:1908.000 084449 2019.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 1263-1272. JMLR, org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Unnat Jain, Ziyu Zhang, and Alexander Schwing. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6485-6494, 2017.
Unnat Jain, Svetlana Lazebnik, and Alexander G. Schwing. Two can play this game: Visual dialog with discriminative question generation and answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1itp:1412.99, 2014.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprini arXiv: 1609. 029007 2016.
Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Information maximizing visual question genera- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2008-2018, 2019.
Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. Visual question generation as dual task of visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Chin-Yew Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
l https:: / www. a c web. www.orrg/anthoooggyy///a////
10

---

Under review as a conference paper at ICLR 2021
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8611-8619, 2018.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7219-7228, 2018.
Issey Masuda Mora, Santiago Pascual de la Puente, and X Giro-iNieto Towards automatic generation of question answer pairs from images. CVPRW, 2016.
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. arXiv preprint (.K3v:66330600) 2016.
Preksha Nema and Mitesh M Khapra. Towards better metric for evaluating question generation systems. arXiv preprint arXiv: 808. 10192, 2018.
Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In Advances Neural Information Processing Systems, pp.
2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 111-318. Association for Computational Linguisties, 2002.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in neural information processing systems, pp. 2953-2961, 2015.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-baase based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4575, 2015.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057. 2015.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. Graph2seq: Graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823, 2018
Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen. Radial graph convolutional network for visual question generation. IEEE Transactions Neural Networks and Learning Systems, 2020.
Shijie Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang. Automatic generation of grounded visual questions. arXiv preprint arXiv:1612.000 2016.
11

---

Under review as a conference paper at ICLR 2021
A THE DETAILS OF GRAPH EMBEDDING
We apply multi-layer GCN with residual connection He et al. 2016 to effectively learn the node embedding from the constructed object graph. The basic layer shown follows:
X a(D 1DD11/2AD 1 1/1
X
(10)
v2
where X is the node feature (X in Eq 7 A is the adjacency matrix (A + is the degree matrix of A, W is the trainable weights and a is the ReLU activation function.
B THE DETAILS OF BASELINE MODELS
12Q It means generating the questions without any hints. We adopt the classic image caption show attend and tell method (Xu et al. 2015).
IT2Q It means generating questions with answer types. We modify the show attend and tell (Xu al. 2015) method take input from the joint embedding of the image and answer type. And since there are no answer type annotations in the original datasets, we adopt the same answer-type information as IMVQG which is a baseline we will discuss later.
IMVQG (Krishna et al., 2019) This is a baseline that maximizes the mutual information among the generated questions, the input images, and the expected answers. They use the answer category (type) as hint. For VQA 2.0 dataset, since they just annotate 80% of the original dataset, the rest of them as "other". And for the COCO-QA -A dataset, we find that there are only 430 answers, annotate their type attribute by ourselves just like they do in VQA 2.0.
annotate
Dual al. 2018) This is another competitive baseline that views the VQG task as the dual task of VQA based on MUTAN architecture. They train the VQG task along with VQA enhance both VQG and VQA's performance.
Radial (Xu al. 2020) This is the latest strong baseline for VQG. They use answers to build an answer-radial object graph and learn the graph embedding. Then they use the graph2seq paradigm to generate the questions.
C IMPLEMENTATION DETAILS
C.I DATA PROCESSING
We use pre-trained ResNet-101 model to extract visual image features. And for each image, use a Masked-RCNN detector with ResNeXt-101 backbone to detect 100 object regions (selected by confidence score) and feature extraction (fc6). The ResNet-101 is from torchvision and the Masked- RCNN from Detectron2.
C.2 HYPER-PARE PARAMETER SETTINGS
Overall setting The image feature dimension is 2048 and the object feature dimension 1024. The words' dimension is 512 and their weights are randomly initialized. The hidden size in Eq. 3 128 to save CUDA memory. All the hidden size is 1024 if not otherwise specified. As for the visual hints prediction, the is 4 and À is 2. The overall loss function's 's a is 0.5, 8 is 0.01 and is 0.01 for most cases. But for ablation model "-vh", a O, 3 0.01 and y is 0.01. -positionn, is 0.5, /3 is 0 and 7 is 0.01. And for ablation model "-answer', is 0.5, / is 0.01 and We adopt adam (Kingma & Ba 2014) optimizer with 0.0002 learning rate. The batch size is 120.
ablation model
Graph embedding module setting The multi- cosine similarity metric's The layer of GCN is 2. The sparsing hyper-parameter is 0.75.
12

---

Under review as a conference paper at ICLR 2021
Table 4: Results VQA2.0 val set. All accuracies are in %.
Method
BLEU @4
CIDEr
METEOR
ROUGE
Q-BLEU@1
F1
Accuracy
mIOU
12Q
63.21
13.89
35.33
26.32
IT2Q
18.41
134.88
19.90
45.71
40.27
IMVQG
19.72
149.28
20.43
46.76
40,40
Dual
19.90
151.60
20.60
47.00
41.90
Radial
20.70
161.90
21.40
48.10
43.50
Ours
22.43
180.55
22.57
49.36
45.61
63.24
82.4%
0.752
D THE DETAILS OF RESULTS
See Figure
4 and 5 for full results.
E ANALYSIS OF HYPERPARAMETERS
E.I THE ANALYSIS
22.45
22.40
BLEU@4
22.35
22.30
22.25
0.00
0.50
0.75
0.90
1.00
The value of in GCN module.
Figure 4: The effect of different graph sparsity.
To study the effect of the in sec.
conduct experiments on VQA 2.0 with the varying in [0, 1]. The results are shown in Figure We find that the model achieves the best performance when e ranging [0.5, 0.8). drops rapidly when close to which means the graph too sparse.
E.2 THE μ ANALYSIS
To study the effect of the in sec. we conduct experiments on VQA 2.0 with the varying in (3. 7). When μ is larger, the visual hint will be more accurate, but the reserved questions will be much less. So in order to compare the effect fairly, we use the same test set and evaluate both the DH-Graph2Seq model and DH-Graph2Seq w/o visual hint (abbr: DH-Graph2Seq w/o vh). The results are shown in Figure 5 We can find that the DH- DH--Graph2eeq consistently outperforms the DH-Graph2Seq w/o vh. When is smaller, although the visual hints are more accurate, the processed dataset has less questions which lead to performance drop. And when y is larger, although reserve more questions, the visual hints' quality is lower. So we think a reasonable choice of f is near 5.7.
13

---

Under review as a conference paper at ICLR 2021
22.000
BLEU@4
21.250
20.875
DH- PHt-Graph2Seq w/o. vh DH-Graph2Seq
20.500
3.0
5.7
7.0
The value of p.
Figure 5: The effect of μ.
Table Results on COCO-QA val set. All accuracies are in %.
Method
BLEU@4
CIDEr
METEOR
ROUGE
Q-BLEU@1
Fl
Accuracy
mIOU
12Q
13.53
95.90
12.61
36.23
31.37
IT2Q
17.80
128.64
16.17
43.22
38.88
IMVQG
18.43
127.18
17.21
44.07
39.22
Dual
18.80
131.10
17.73
44.19
39.92
Radial
19.24
139.55
18.19
44.21
40.98
Ours
20.84
166.78
19.81
46.80
43.57
60.81
85.2%
0.713
THE DETAILS OF HUMAN EVALUATION
conduct a small-scale human evaluation on VQA2.0 val set. Concretely, we randomly select 50 examples for each system: 1) the ground-truth results (denoted as GT), 2) our results (denoted as Ours), 3) the 'Radial' baseline's 's results (denoted as Radial), 4) the 'dual' baseline's 's results (denoted Dual).
We ask human evaluators to give feedback on the quality of questions randomly selected in the results of 4 systems. In each example, given triple containing anonymized system's output, they are asked to rate the quality of the output by answering the three questions: a) is the question syntactically correct? is the question semantically correct? the question relevant to the image and the answer pair? For each question, the rating scale is from to The standard is 1. Pool (not acceptable), Marginal, Acceptable, Good, Excellent. We develop software to automatically collect the evaluation results shown in Figure 6 The software will feed the examples and average the scores.
image, target answer, and
G THE DETAILS OF CASE STUDY
See Figure for for qualitative examples. We can find that model generate more complete and vivid questions compared with baseline methods. We think that with the help of double hints, our graph2seq model can find the proper image regions and exploit the rich structure relations better.
14

---

Under review as a conference paper at ICLR 2021
knower
Question
Next
Figure 6: The screenshot of human evaluation software.
H THE DETAILS OF ERROR ANALYSIS
See Figure 8 for eπrог cases of our results. We present one example of cach error
a) means our model recognizes the objects incorrectly, In the picture, the "teddy bear' is next to the refrigerator but our model recognizes it as 'mierowave'.
b) It means our model predicts the visual hints incorrectly, which largely misleads the generation. Indeed, the answer '3' refers the number of humans, but the model picks the horses out and overlooks the men.
It means model infers the relations among the objects incorrectly. In the image, the batter wears the helmet but the expected answer is 'no'.
d) It means our model generates questions which semantically incorrect. In the image, there at least two cars, but in the generated question, the car is in the singular form.
It means model generates questions which syntactically incorrect. The 'How many of is incorrect.
15

---

Under review as a conference paper at ICLR 2021
Ground Truth
Prediction
Answer: white
GT: What color the vehicle in the mirror ?
Radial: What color is the truck?
Dual: What
bus?
Ours: What color
mirror?
Ground Truth
Prediction
Answer: apple
GT: What brand phone is this?
Radial: What brand of computer this using?
Dual: What brand is the laptop?
Ours: What brand the phone?
Ground Truth
Prediction
Answer: umpire
GT: Who the person behind the catcher ?
Radial: What the man black doing?
Dual: What position does this man play?
What the black pants
called?
Ground Truth
Prediction
(d)
Answer:
GT: How many people are wearing sunglasses?
Radial: How many people are wearing hats?
Dual: How many people are the photo?
Ours: How many people are wearing glasses?
Figure 7: The details of case study examples. The red rectangles mean the visual- hint regions.
16

---

Under review as a conference paper at ICLR 2021
a)
Detection Error
A: teddy bear Q: What is sitting on the shelf next to the microwave?
b)
Visual Hints Prediction Error
Q: How many horses are not white?
Reasoning Error
No Q: Is the batter wearing a helmet?
d)
Semantic Error
BON
PURS
A: Red
Q: What color the car?
Syntactic Error
A: 2
Q: How many of bears are there?
Figure 8: The details of error examples.
17


Reviews:
[{"review_id": "gGQrkwS9Iq1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: \nThis submission focuses on the problem of visual question generation and proposes to use two hints: answer and visual regions. With the help of these two hints, a graph representation is built and then a GCN-based graph2sequence generator is applied to generate questions based on given inputs. Experiments on two datasets show its efficiency on question generation quality.\n\nStrengths:\n+ The description of the method (model) is easy to follow.\n+ Improvements of the question generation quality (according to the BLEU and Cider score) is big.\n+ Good ablation studies and human study.\n\nWeakness:\n- The motivation of VQG: I understand VQG is a recently raised task in vision-and-language research, but I can not get any ideas that why this is an important task, at least from this paper. According to my understanding, one reason that we need VQG is that we can use it as a data-augmentation tool to train a better VQA system (which is a more important task). Another reason is that it may lead to a better  (goal-oriented) visual dialogue system, such as GuessWhat?!.  This paper mentioned these two directions very briefly but failed to explain it well and there are no experiments presented in this paper to show whether their question generation model can benefit these two areas. I don't think a better BLEU score can show its potential since it only means this model can generate similar questions as the training data. It is worth to check whether these augmented questions can boost VQA performance.\n\n- The motivation of two hints: It looks like many previous works only use answer types as the hint. I think this is more reasonable since it gives more freedom to the model to generate questions. And I don't think it is an issue that 'one answer/image can be potentially mapped to many different questions'. Instead, I think this provides a diversity of generated questions and further improves the VQA generalisation ability, from a data augmentation view. In this submission, authors choose to prove the answer directly as a hint. There might be two issues: at first, it seems not fair since it requires more annotation. Secondly, how can you get the answer during the inference? Say if one downloads an image from the Internet  (not from a well-annotated VQA dataset) and there are three bears in the image, it might be easy if the answer is 'bear' since you can use a detector, but how can one know the answer is 'three' if we want to generate a question about 'how many?'. In this case, the answer type is more practical.\n\nAccording to the visual hints, if I am right, from section 4.1, they are mined from the questions and answers directly, for both training and testing. This suggests you (authors) already got some info from the target questions (even in the testing!). I can't say it is a fair setting, even it happens in pre-processing step. Please fix me if I am wrong.\n\n- The 3.2.1 is basically an attention mechanism?\n\n- What is the accuracy of the tasks listed in 3.2.2?\n\n- In section 3.3, the \\epsilon seems quite important. It decides the sparsity of the graph. Any ablation studies?\n\n- In section 4.1, another \\epsilon, which decides the quality of the visual hints. Any ablation studies?\n\n--------------------------- After Rebuttal --------------------------------\n\nThe authors did a good job in the rebuttal. Most of my concerns have been addressed so I am happy to raise my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_wFdBUamdE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper proposes a model for the task of Visual Question Generation (VQG) which uses the answer as well as object regions to generate the question. The paper models interactions between various visual entities and the answer tokens using a graph and then use it to generate the question. The proposed approach outperforms existing methods on the VQA and COCO-QA task. \n\nStrengths: \n- The paper emphasizes on using both image-regions for the VQG task. While some of the proposed techniques borrows from existing works, they showed how to combine it to improve over exisitng methods. \n\n- On the VQA and COCO benchmarks, the models outperforms existing methods. On the small scale human study, the proposed approach was rated higher than existing approaches. \n\nWeaknesses: \n- Overall, the novely of the paper is low. The paper is a collection of already popular ideas (use of attented region features for V+L tasks (Shah et al.), use of position embeddings to model spatial relations (ViLBERT, VL-BERT etc). Because, the paper is written as a collection of ideas, the take-away message isn't clear. \n\n- The ablation tables has very subtle performance changes across design choices which are hard to understand. The ablation table suggests that all choices in isolation leads to a drop when compared to the full model. But what happens when you combine these choices one by one. More importantly, what are the most critical components of the proposed method? \n\n- Some of the statements made in the paper are not backed by citations or lack explanation. For instance, the authors claim that the \"amount of objects which are visual hints are much smaller than ones not\" and \"when humans ask questions from the image, we will infer whether the object is important clues by looking around in the image\". \n\n- The paper was hard to follow and it seems like there are a lot of moving parts. In general, this makes reproducing the paper and adopting ideas from the paper difficult. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6b6HzTr-aC9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new model on the task of Visual Question Generation. The model uses cross-modal alignment between the object features, position features and answer hints to find the right subset of relevant visual hints to be used to generate the relevant question. The model also ensures that the latent space features capture the answer and position information by predicting them back from it. Informed by the visual hints, the object and image features are passed to a GCN network that is used to get the final hidden state which is passed to an attention and language lstm similar to BUTD model to generate the final question.\n\nThe cycle consistency used for answer and position features seems to provide grounded question generation for the answer and image. The model is tested on VQA2.0 and COCOQA and achieves better performance compared to the baseline model on automated metrics as well as human evaluation. \n\nOverall, the paper is strong and provides a solid foundation for the intuition and framework behind the model supported with detailed ablation analysis and case studies. What I find missing, is the actual test of how good these questions actually are by using them to train on the actual VQA task. Understanding that and how it performs as extra data on VQA 2.0 would give us a better understanding on how good the generation actually is where it matters. \n\nSome other questions:\n- Is there any specific reason why BERT wasn’t used instead of GloVe for word embeddings?\n- Please add citation for VQA 2.0\n\nOverall, I would like to recommend the paper for acceptance but it is hard to understand the actual value of this work without downstream application on the task of VQA 2.0.\n\nEdit after rebuttal: I have read the author response and I thank the authors for their valuable insights and answers to my questions. It is exciting that this does help in improving the performance on downstream VQA2.0 task though I would have expected the results to be conducted on one of the recent state-of-the-art models instead of very old BUTD model where achieving performance gains is trivial. I would like to keep my rating as it is.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1vXKQxwqhOQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors aim at generating the right questions based on the textual answers and corresponding visual regions of interest (ROIs). The core innovation of the proposed method is leveraging the region information to supervise the question generation, which helps to mitigate the ambiguity of the answers. Correspondingly, a simple method is designed to generate the noisy annotations of the ROIs using the pretrained Masked-RCNN and the questions. \n\nThis core part is mainly divided into two components: 1) aligning the object features with the answer word embeddings using attention mechanism; 2) constructing the object graphs and getting the GCN-refined object features; 3) refining the question embeddings by attending the refined object features and image ones. Experimental results show that the proposed method outperforms the existing approaches significantly. In addition, ablation studies prove the effectiveness of the proposed components. \n\nAlthough the visual regions of interest can help to guide the question generation, the one-to-many mapping issues still exist when generating the visual regions of interest (no additional information is given), which may lead to the same problem in the question generation stage. Nevertheless, the proposed approach is still valuable. Although it helps little in mitigating the one-to-many issues, it helps to learn a better question generation network, as the additional region information can mitigate the issue during the training stage, which will help to avoid learning a generator that prefers to give general questions. Therefore, I think authors should make that clear in the paper that it helps in learning a better question generator instead of generating the specific question because the ROIs are also selected based on the answers only. \n\nAlthough great improvements are achieved, the proposed method is not novel. The main novelty is leveraging the object features in generating questions. However, neither the feature alignment nor the GCN part is new. In addition, the graph construction part does not make sense to me, and no ablation study show that pruning the non-hints objects can help to improve the performance.  \n\nIn addition, the confusing equations and poor writing of the paper cannot make this paper an unaccepted one:\n* In Eq. (2), \\beta X^{a} seems like a matrix, which cannot be concatenated with a vector. \n* In Eq. (3), \\beta_{j} should be \\beta_{i, j}.\n* In Eq. (3), what I_{j} means? Why the image feature has a subscript?\n* In Eq. (3), what does the operation “*” means? The product signs are different in (2) (3) and (4)\n* In Sec. 3.4, it lacks detailed explanations of how the model will attend the image and graph to get a better representation of the question embedding between the two LSTMs, which is one of the core components of the proposed methods. \n* Typos like “size information” should be “side information” in the top paragraph on Page 2, “question tokens” should be “word tokens” in the bottom paragraph on Page 3. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper proposes to generate human-like question for a image by using additional information (hints) such as the textual answer and visual regions of interest (ROIs).  The visual regions are used to guide the question generation so that the model can generate relevant and informative question.  The question generation problem is formulated as a graph-to-sequence problem, starting from an object graph, and using GCNs with attention to align text and visual regions and to generate an appropriate question.

Review Summary: The submission initially mixed reviews (scores from 3 to 6).  While reviewers find the problem interesting and work to be mostly solid, reviewers felt that the novelty of the work was limited (R1, R3) and that some of the presentation was unclear (R1, R3) with some missing details (R2).  R2 was not sure if the VQG was a useful task, and R4 felt that the initial submission was missing a key experiment on whether the generated questions can be useful as data augmentation for VQA. The reviewers were impressed by the extra experiments performed by the authors in the rebuttal and the revised draft, and indicated that most of their concerns were addressed, In particular,  the generated questions were shown to be useful as data augmentation.  Many reviewers increased their scores, ending with 3 scores of 6 (R1,R2,R4) and 1 score of 5 (R3).  

Pros:
- The use of generated questions as data augmentation for VQA is an interesting direction
- Strong empirical results with thorough experiments (and user study)

Cons:
- The technical novelty of the work is still rather limited (R1)
- The paper was difficult to follow (R1,R3) with a lot of moving parts, making it potentially difficult reproduce (R1) - The authors indicated that they will open-source the code.
- The grammar and wording of the writing is poor even after revision and should be improved

Example of poor writing (a full proofreading pass is recommended):
- Section 2.1: "Mora et al. (2016) firstly makes an attempt to adapt" => "Mora et al. (2016) adapted",
"abstract and general results" => "imprecise and generic questions"
- Section 3.2: "The most important point of our first issue located in how to effectively find..." (it's unclear what this means)
- Section 4.1: "standfordcoreNLP" => "Stanford CoreNLP"
- Section 4.5: "shown in 3" => "shown in Figure 3"

Recommendation:
The AC agrees that the work is addresses an interesting area of generating questions as data augmentation for VQA.  Despite the improved reviewer scores, the AC agrees with the initial assessment that the work has limited technical novelty.  The AC also found the writing of the paper to be poor and difficult to follow at places even after revision.  Due the limited novelty, the issues with exposition, and the many changes to paper, the AC believe that the work would benefit from another round of review and should not be accepted at ICLR in its current form.  Given the positive response, the authors are encourage to improve their work and writing and resubmit to an appropriate venue (the AC believes the work would be more appreciated in a vision or language venue).
Decision: Reject