Learning to Generate Visual Questions with Noisy Supervision
Authors: Kai Shen
Lingfei Wu
Siliang Tang
Yueting Zhuang
zhen he
Zhuoye Ding
Yun Xiao
Bo Long
Venue: NeurIPS 2021 Poster (2021)
Keywords: Noisy Supervision
Generative Adversarial Networks
Visual Question Generation

Abstract: The task of visual question generation (VQG) aims to generate human-like neural questions from an image and potentially other side information (e.g., answer type or the answer itself). Existing works often suffer from the severe one image to many questions mapping problem, which generates uninformative and non-referential questions. Recent work has demonstrated that by leveraging double visual and answer hints, a model can faithfully generate much better quality questions. However, visual hints are not available naturally. Despite they proposed a simple rule-based similarity matching method to obtain candidate visual hints, they could be very noisy practically and thus restrict the quality of generated questions. In this paper, we present a novel learning approach for double-hints based VQG, which can be cast as a weakly supervised learning problem with noises. The key rationale is that the salient visual regions of interest can be viewed as a constraint to improve the generation procedure for producing high-quality questions. As a result, given the predicted salient visual regions of interest, we can focus on estimating the probability of being ground-truth questions, which in turn implicitly measures the quality of predicted visual hints. Experimental results on two benchmark datasets show that our proposed method outperforms the state-of-the-art approaches by a large margin on a variety of metrics, including both automatic machine metrics and human evaluation. 

Learning to Generate Visual Questions with Noisy Supervision
Kai Shen", Lingfei Wu Siliang Tang", Yueting Zhuang,, Zhen He Zhuoye Dingt, Yun Xïao, and Bo Long*
ZZejiang University JD.COM
shenkai@zju. edu, cn, lwu@email. um, edu, {siliang, fsiliang,ythuangll yzhuang) edu edu. cn, [b (bjhezhen, dingzuuoye xiaoyun1, bo long]@jd..com
Abstract
The task of visual question generation (VQG) aims to generate human-like neural questions from an image and potentially other side information (e.g.. answer type or the answer itself). Existing works often suffer from the severe image to many questions mapping problem, which generates uninformative and non-referential questions. Recent work has demonstrated that by leveraging double visual and an- swer hints, a model can faithfully generate much better quality questions. However, visual hints not available naturally. Despite they proposed simple rule-based similarity matching method to obtain candidate visual hints, they could be very noisy practically and thus restrict the quality of generated questions. In this paper, we present novel learning approach for double-hints based VQG, which can be weakly supervised learning problem with noises. The key rationale is that the salient visual regions of interest can be viewed as constraint to improve the generation procedure for producing high-quality questions. As result, given the predicted salient visual regions of interest, we can focus on estimating the probability of being ground-truth questions, which in turn implicitly measures the quality of predicted visual hints. Experimental results two benchmark datasets show that our proposed method outperforms the state-of-the-ar approaches by a large margin on variety of metrics, including both automatic machine metrics and human evaluation.
1 Introduction
Recent years have witnessed a surge of interest in the Visual Question Generation (VQG) task. VQG aims generate human-like neural questions from image and potentially other side information (e.g., answer type or the answer itself). VQG is one of the effective data augmentation approaches for providing high-quality synthetic training data for visual question answering (VQA) (28 and visual dialog system [19). Moreover, VQG models are also particularly useful for the few-shot learning or zero-shot learning 36 44 Conceptually, VQG very challenging task since the generated questions are not only required to be consistent with the image content but also meaningful and answerable to humans.
To handle the VQG task, recent approaches explore three directions mainly. One direction is focusing on VQG without any hints. Although more diverse questions be generated, often leads to non-informative questions and hard to find any corresponding answers. Furthermore, in view of data augmentation, successful VQG system should be goal-driven in order to provide valuable additional QA pairs to further enhance generalization capability of its dual VQA task 22).
Both Authors Contributed Equally. This work was conducted when the first author was an intern in JD.COM 8 Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Another direction utilizes answer-type hints for VQG in order to generate more referential questions while exploiting diverse questions. However, given one image. there are often multiple questions with the same question types (i.e., what, how, and how many). If only answer-type is used, the generated questions are often either not referred to any existing answers or not informative questions. As result, these generated questions either cannot be used for training VQA or can be too noisy to degenerate the generalizability of VQA.
The last direction exploits the textual answer information as answer hints for VQG. This is particularly useful when two questions are the same question types but different questions for an image. As result, the answer hint can often greatly reduce the ambiguities when learning to generate visual questions. Therefore, it can train a better question generator for more diverse questions instead of specific questions. Unfortunately, there are still many cases where answer hints are not informative enough to reduce the ambiguities for generating a question, especially when there are multiple visual objects that are associated with answer hint.
Recently, Kai et al. (20) presented novel double-hints approach for VQG, which showed that the model can faithfully generate much better quality questions by leveraging double visual and answer hints. However, visual hints are not available naturally, which requires highly expensive human annotations. To handle this issue, they proposed a simple rule-based similarity matching method to obtain a set of visual hints, which are often too noisy to generate high-quality questions. As result, the performance of VQG could be highly restricted by these low-quality visual hints.
Therefore, an important question is, without any expensive human annotation, can we identify important salient visual regions of interest associated with answer hints to overcome the above deficiencies? In this paper, we present novel learning approach for double-hints based VQG, which can be cast as a weakly supervised learning problem with noises. The key rationale is that, the salient visual regions of interest can be viewed as a constraint to improve the generation procedure for producing high-quality questions. As result, given the predicted salient visual regions of interest, we can focus on estimating the probability of being ground-truth questions, which in turn implicitly measures the quality of predicted visual hints.
To this end, propose a Double- Hints guided Generative Adversarial Networks (DH-GAN) con- sisting of a double-hints based question generator and a queestio--answeraave discriminator. In particular, our generator has visual hints predictor with cross- modsl reasoning module to deter- mine the salient visual regions associated to the corresponding question, and a question generation module with predicted visual hints and textual answer hints. Our discriminator is to distinguish whether sample triplet (i.e., the image, answer, and question) is generated from the generator or ground-truth. Moreover, we design novel hybrid reward function that combines the generator and discriminator so that it encourages the model to perform both exploration of question sample and the visual hints via policy gradient. Experimental results on two benchmark datasets show that our proposed DH-GAN outperforms the state-of- metrics, including both automatic machine metries and human evaluation.
approaches by a large margin on a variety of
2 Double-hints Guided Generative Adversarial Networks for VQG
In this section, we first introduce the double-hints based visual question generation (VQG) problem and then discuss the overall model architecture. Next, we detail each key components of our proposed DH-GAN model for double-hints based VQG in the subsequent sections.
2.1 Problem Formulation and Overall Model Architecture
Problem Formulation. Conceptually, the goal of VQG is to generate natural questions based given form of visual data (i.e., images (22] and videos (46)), together with other side information (i.c., answers 28), answer-types [22], or both textual answers and visual regions 20)). In this paper, we focus on the task of generating high-quality questions from images with double text and visual hints.
Given an image I, we assume an image is represented by a set of visual regions R = fri,r2..... .2..... PN with initial embedding number of visual regions. Similarly, we assume the answer A = {ai, a22 .... .....m] with initial embedding (al, a2; ... am) and the target question Q = {q1, -.. 4n with initial embedding {qi, 92. qn } are collection of words, respectively. Next, the visual hints V = {r1, ..., PT)
[Vi, V2,... vN] via an object detection method (40], where N is the
2

---

Ground truth Question (@) color being thrown... ...
Q
Discriminator
Ground truth Question (0)
Attention
Answer: White
Answer
Quention
Hybrid
Real? Fake?
Generation Quality
Semantic Quality
Sampling Decoding
Greedy Decoding
Faster RCNN
Visual Hint Prodictoo
Predicted Visual Hints
Language LSTM
Cord
Conul
Double- Attention
Dlated-Cana
Vision
Ansuwer:
Policy Gradient
Figure 1: The proposed DH-GAN Model Architecture with Noisy Supervision.
are the visual regions of interest, which are (semantically) associated with the corresponding textual answer. Note that, V is the subset of R (V S R). The double-hints based VQG task aims to generate not only answerable but also referential questions based on the given image I and the double hints A and V, which maximizes the following conditional likelihood
Q = argmax P(QII, =
(1)
where the visual hints V are obtained through the alignments between the image I and the answer A. Note that, since there are no additional human annotations when obtaining target visual hints during pre-processing, is not surprising that they are very noisy, which leads to the non-accurate prediction of V during inference.
The proposed DH-GAN Framework. Despite the potentially useful additional information the visual hints may carry, how identify the salient visual regions of interest from set of visual region candidates without any human annotations a very challenging task. Therefore, it is crucial that proposed model is able learn generate visual questions with noisy supervision. In order to effectively address this challenge, we propose the DH-GAN model, which consists of two key generative model G and discriminative model as shown in Fig.
components:
Formally, we denote the sample triplet as x A1 (I for image, Q for question, and A for answer). We further denote x+ {I, Q f0+ A} as the annan-en sample ("real") and = [I, Q A} as the sample generated by the system ("fake"). Generally speaking. the goal the generator G is to first select the salient visual hints from the input image and the textual answer and then generate the question based on the predicted visual hints and the corresponding textual answers. Since there is direct alignment information between the visual hints and answer, the goal of the discriminator is to implicitly measure the quality of predicted visual hints by judging whether the current sample triplet is "real" or "fake". In other words, the generator G and discriminator D undergo an adversarial process to encourage the generator to identify the salient visual region of interest that the high-quality question can be generated, which indistinguishable from the "real" ones by the discriminator D. Once the Nash Equilibrium is achieved via GAN, the generated question Q would be very close to Q* in the discriminator's embedding space semantically. In what follows, we will discuss each component of the proposed DH-GAN in detail.
2.2 G-step: Double-hints Guided Question Generator
Since visual hints are not available naturally, is unclear how effectively identify the salient visual hints and leverage them for generating the high-quality question. In this section, will introduce two important steps of G-step: identifying visual hints and ii) generating questions with double visual and text hints.

---

2.2.1 Identifying Visual Hints
To exploit the rich correlations of the visual regions, inspired by ResNet 17, we utilize the following relation-aware encoder module:
V out (DiniiiieaateeCnsssCoo (V)) + V
(2)
where V E IR-xxd Vour E xd are input and output features for the visual regions, Conv (-) the CNN layer with kernel size 1, DilatedConvs(-)) is the dilated layer with kernel size 3. Each convolution layer is followed by the batch normalization with a ReLU activation. Empirically, stacking multiple modules typically capture more concise relations between regions. We thus stack k modules to obtain & + number of representations of the visual regions in different subspaces, denoted as {V(0), v(), .... v(k),, J. where V(O) is the original visual features (V) and V is the output of layer I. We assume that each layer can learn the specific visual meaning, which will be used later to align with the corresponding textual answers.
We adopt the bidirectional GRU network to encode the input answer A {[1, 02 .... am } with initial embedding [a1, ag...., am J. We then concatenate the last hidden state of each directions to present the answer
single vector a E Rdd Similarly, to better align with visual regions, we also utilize multiple linear projections to project answer representation a into k + 1 number latent subspaces, denoted as {a("), (1),, a (a)), j. Thus, for each subspace E [0, kd), we employ attention mechanism learn the relevance between each visual region !! (v (!) is the i-th row of v(O) and the answer (!) as follows:
$ T a (W + a1) + b),
where W; Rdxd W RR b RR are the learnable weights, a-)) is the Tanh(.) activation function, and 8 is the similarity score capturing the semantic relevance between the region r; and the answer A. Then we take the average of relevance score in each layer and caleulate the probability of being
visual hint:
Ee Li=0
Puh,i o
k+1
where (-) is the Sigmoid(-) function. Given the target visual hints V, we apply focal loss 32 to optimize this module, which is denoted as Luh as follows:
Lun -1 Ereh,
pens log(pen))) - Σ (1 pon,a) log(1 - Pon.),
(5)
rv
where 1), À are hyper-parameters.
2.2.2 Generating Question with Double Hints
After obtaining the visual hints, the next step is to generate the high-quality question under the guidance of double hints. We employ the attention- based hierarchical sequence decoder from [34] for the double-hints- based question generation step. This module consists of two LSTMs: 1) vision LSTM and 2) language LSTM. The vision LSTM is used encode the global visual features, the input word embedding, and the answer embedding into hidden state hf E Rd. and the language LSTM is used to generate question text sequence. Note that the starting states are initialized by answer embedding. Between the two LSTMs, the attention module is employed guided by the predicted visual hints and the answer hints. Specifically, we first apply a simple pruning strategy to select the candidate visual regions and then identify the predicted visual hints to obtain Voh. E R xd using the predictive model mentioned in the previous section. Next, employ attention visual hints V with h (textural answer hint) to obtain the importance of the salient visual clues for generating the current word token in question. We refer the reader Appendix C for more details.
the predicted
We train with the cross-entropy loss function denoted as Lim- As a result, the supervised loss function for the generator G consists of visual hints prediction loss Loh and language generation loss Lim shown as follows:
L.up = Lim + ahheth
(6)
where a is the coefficient hyper-paramette

---

2.3 D-step: Question-Answer Discriminator
Although the salient visual hints are predicted by our Generator, there is no clear way how to evaluate the quality of selected salient visual hints. In this section, we will introduce the design protocol for an effective discriminator in D-step by leveraging the signals from the generated questions.
In particular, the proposed discriminator is a binary classifier that takes the sample triplet x = (T,Q, A), where Q could be either the generated question the ground-truth question, and then outputs a class label indicating whether the input triplet is from the ground-truit or the proposed generator. Therefore, the discriminator rewards the generator for identifying the most meaningful region of interest in order to leverage these visual hints to generate the best quality of questions.
Specifically, employ bidirectional GRU to encode both question Q and answer A to its vector representations q E R" and a E R', respectively. Then we attend on the visual region features V E R xdd using question vector embeddings q I to obtain the question-ave embeddings Vattn. Next, we concatenate Vattn and answer vector a to generate the probability of being human-annotated triples follows:
visual vector
p= o=o(FVaatto a|a)),
where F(.) : Rdd + R is the linear projection, at-) is the sigmoid function and the simple concatenation. We then apply binary cross-entropy function to calculate the loss denoted as LD.
2.4 Training DH-GAN via Policy Gradient
Our model is not fully differentiable due to the procedure of choosing the next token based on the probability distribution, which makes it infeasible to propagate the gradients from the discriminator the generator. Inspired by the sequence GAN 24 53 8. we adopt the policy gradient mechanism to handle this issue. Concretely, the reward score of the generated sample x=, which is formulated by the discriminator, is used reward signal for the generator. Formally, given sample æ, we define the reward function as R(x)). We adopt the reinforcement learning (RL. algorithm [47 to maximize the expected reward of generated samples x
J(0) = Eq - (1,)) (R(x )(0).
where Q is the question generated by the generator and is the model parameter. The equation can be optimized by the likelihood ratio trick [47 10)
24
g
VJ ~ [
B(x Vog IIQ-
where log P(Q-17, A) is the log-likelihood of the question generator, R(a=) ) is the reward of being "real" given a generated sample x=, and (xx)) ) is the baseline which can reduce the variance of RI. system while keeping unbiased. Obviously, how to select appropriate baseline is one of the keys to learn a useful reward.
we utilize the output of the test-time inference normalize the rewards the generator meets. At each iteration, the generator will output two results: 1) the baseline output Q which is obtained by greedy search, that is, by maximizing the output probability distribution each decoding step, 2) the sampled output Q, generated by multinomial sampling, that is, each word token n. Q is sampled according to the likelihood Pl Ter) predicted by the generator. Following that, the Eq. can be written as:
Inspired by the self-critical sequence training (SCST) 5 42
VJ R [R(U, Q,) R(L1, Q A))J log A)
(10)
Equally important, the exploration of the visual hints could lead to a better question generation. Therefore, for an individual region rú, we determine whether it could be a visual hint according to the probability of Pus.i in Eq. 4 We denote the sampled visual hints as V. It is worth noting that to make the training stable, we apply the temperature on the logits before sigmoid in Eq. 4 Intuitively, a good exploration of the visual hints would be beneficial to improve the quality of the generated questions. Thus we apply the same reward gain in Eq. 10 which can be written as:
VJ (R(TTQ.A) R([T, Q A)) log P(Q)T,A)
(11)
= [R( - R(U, [ P(Q),, 10 A, D) + log P()),
5

---

2.4.1 Reward Function
One of the key factors of Reinforcement Learning is to design a proper reward function. In order to generate syntactically and semantically meaningful questions, we first consider adopting one of the automatic evaluation metrics (i.e. BLEU@4 (77) for measuring the syntactic quality of the generated questions as part of the optimization target. Since well known that the BLEU score cannot faithfully reflect the semantic quality, we further select the discriminator'ss probability output as a reward to measure the semantic quality, Note that, the larger (xx is, the more "real" question Q is. Therefore, define the hybrid reward function follows:
R(x)= BLEU(Q) + cD(z).
(12)
where BI.E(()) is the generation reward, and D(-) is the semantic reward given the ground-truth question, respectively. Moreover, is the coefficient balancing these rewards.
2.4.2 The Loss Functions of DH-GAN for VQG
Generator Loss. The loss function of the generator derived from Eq. 11 can be written as:
F - - [R([), R([I, A))loo P(Q))) P,Q),, A, + 3 log A()],,A)),
(13)
where 9 is the hyper-parameter: log (, the question generation loss with target question Q in Sec. 2.2.2 and log (( the visual hints prediction loss given target visual hints V in Sec. [.2.1 Practically, we find that it is unstable to update the generator by minimizing the loss C... Thus we combine both the teacher-forcing loss Ssup in Eq. and the reinforcement loss as:
LG A,nt + (1 Ceups
(14)
where y is scaling factor controlling the trade-off between teacher-forcing loss and RL loss.
Discriminator Loss After updating the generator for some steps, we will re-train the discriminator to classify the generated and the original samples better as follows:
LD - log D(x )
log(1 D(x ))
(15)
3 Experiments
In this section, we conduct an extensive set of experiments to demonstrate the effectiveness of the proposed method compared with existing state-of-the-art !
Dataset. We conduct the experiments on the VQA2.0 3 and COCO-QA (39 datasets. Following 20). the visual hints are generated by rule-based similarity matching (see Appendix 1.1.1 for more details). Clearly, the original visual hints are too noisy to guide the question generation. To alleviate this issue, introduce simple pruning mechanism to decrease the number of candidates while preserving the key information. After pruning, we will keep at most m visual regions to be visual hints (see Appendix 1.2). After pre-procd proppecesssngg the VQA2.0 has 278707/1 355544, and the COCO-QA dataset has 58979/29017 examples for training/validation split, respectively.
Pre-processing. For each input image, we employ pre-trained Faster-RCNI 44 for visual region detection and feature extraction (fc6) (see Appendix E.1.3). For text data, we truncate the questions longer than 20 words and build the vocabulary the words with least 3 occurrences. Since the test splits (for these two datasets) are not open for the public, we divide the validation set to 10% validation split and 90% test split.
Baseline Methods. We compare against the following baselines our experiments: i) 12Q [49 standard CNN-RNN model in image caption that maps the image to the questions directly. ii IT2Q [22]: a CNN-RNN based model which uses answer type as side information. iii) IMVQG* [2]] variational method that maximizes the mutual information between question, image, and the expected answer. iv) Dual (8]: an advanced model that combines the VQA and VQG tasks by
'The code and data for model are provided for research purposes: DH-GAN for VQG Github Repo

---

Table 1: Results VQA2.0 and COCO-QA val set. All metrics in % and B@4.BLEU@4, C-CIDEr, M-METEOR, R- R-ROUGE, S-SPICE.
Dataset
VQA 2.0
COCO-QA
Method
B@4
C
M
R
@@4
C
M
R
S
12Q
9.02
63.21
13.89
35.33
18.04
14.71
107.90
13.71
38.32
18.65
IT2Q
18.41
134.88
19.90
45.71
22.90
18.04
135.23
17.34
46.76
22.21
IMVQG
19.72
149.28
20.43
47.20
23.10
21.16
156.76
18,93
46.89
24.21
Dual
19.90
151.60
20.60
47.00
23.21
21.48
153.32
18.93
47.03
24.34
Radial
21.87
162.92
22.22
48.65
25.34
22.63
168.29
19.73
47.71
26.71
DH-VQG
22.43
180.55
22.57
49.36
27.40
23.15
175.18
20.04
47.84
27.63
DH-GAN
23.71
191.06
22.91
50.53
28.18
23.52
186.65
20.44
48.61
28.32
dual learning. v) Radial* [50] a strong baseline that evolves an answer-centric radial graph with
GNN encoding to ask questions. vi) DH-VQG [20]: recently proposed baseline that uses very noisy double-hints as side information to generate referential questions. The detailed descriptions of these baselines are provided in Appendix D. Experiments on baselines followed by are conducted using the codes originally released by authors.
Evaluation Metrics. Following previous works [22 50 20
adopt the standard linguistic
measures including BLEU (37], CIDEr [45]. METEOR 4 ROUGE-L [31] and SPICE (1) For visual hints prediction, adopt Fl metric. These scores calculated by officially released evaluation scripts'.
3.1 Results Analysis and Human Evaluation
Table T shows the results comparing our proposed method against other state-of-art baselines on various automatic evaluation metrics. Note that, due limited space, omit the reports of the result performance of the visual-hints only (please see these details in Appendix F). There are two observations worth noting here. First, our proposed DH-GAN method for VQG consistently outperforms other state-of-the-ar methods by a large margin, highlighting the importance of double visual hints and textual answer hints. Second, compared with recently proposed DH-VQG, DH-GAN improves over DH-VQG by a noticeable margin thanks to the proposed GAN method that could effectively identify the salient visual regions associated with the answer hints.
Furthermore, we conduct a human evalua- tion study assess the quality of the ques- tions generated by our proposed DH-GAN method, the ground truth (GT), and the Daseline-Radial on three different dimen- sions in terms of syntax, semanties, and rel- evance metrics, respectively. In addition, to investigate the effectiveness of DH-GAN, we also compare the results with only pre-trained generator without the proposed discriminator (abbr: that our model outperforms all strong
Table 2: Human-study Results VQA2.0.
Method
Syntax
Semantics
Relevance
Radial
4.43 (0.48)
4.44 (0.6)
3.48 (0.59)
Generator
4.67 (0.37)
4.66 (0.48)
3,86 (0.6)
DHI-GAN
4.68 (0.32)
4.65 (0.42)
4 (0.58)
GT
4.7 (0.33)
4.67 (0.44)
4.31 (0.53)
Generator). The results are shown in Table is clear baseline methods on almost all three metrics, especially on relevance score (the closest one to the ground- ground-truth), which highlights how important identify the most important visual region of interest in order to reduce the noises. Our human evaluation details are discussed in Appendix H.
3.2 Ablation Study
We further conduct the ablation study to evaluate the importance of each key component of the proposed DH-GAN method on the VQA 2.0, including visual hints, visual hints sampling, textual answer hints and answer type. Note that, in order to make the comparison consistent and clear, we use the Generator model (from our DH-GAN) as baseline (abbr: Generator) and investigate the performance effect of each component by turning off or adding modules. The ablation study results on the VQA2.0 validation set are shown in Table 3
7

---

Original image
Raw visual hints
Answer:
Detection error 7%
Others
GT: How many people can you see the boat?
9%
Syntactic etror 9%
Radial: How many boats in the picture?
Visual prediction
Generator
Ours
Reasoning error
Generator: How many boats are in the water?
33%
42%
Ours: many people
the boat?
Figure 2: Case study (left) and Error analysis (right)
There several important observations worth highlighting here. First, by turning off the visual hints (- visual hints), the performance of model drops nearly 3.2% (BLEU@4). Clearly, the visual hints are indeed helpful for generating high-quality questions. When we further replace the answer hints with only answer type, the performance continues to drop rapidly by 17.5% (BLEU@4) which indi- cates that the answer hints carry much more helpful information than the answer type.
Table 3: Ablation Results on VQA2.0. Note that "+" denotes having this component and "." means removing this component.
Method
B@4
R
DH-GAN (full model)
23.71
22.91
50.53
GAN
23.53
22.87
50.45
visual hints sampling
Generator
22.60
22.69
49.77
vvisual hints
21.87
22.05
49.14
Second, comparing our DH-GAN (full model) with
the baseline Generator, we can see that the perfor- mance improves significantly by 4.9% BLEU@4)) which demonstrates the effectiveness of the GAN framework for generating informative and referen-
visual/answer hints
18.63
20.01
45.43
answer type
tial questions by identifying the most important salient visual regions. Furthermore, we notice that by removing the visual hints sampling module (+GAN visual hints sampling), the performance drops from 23.71 to 23.53 (BLEU@4), which indicates that the visual hints sampling strategy is beneficial the VQG task by encouraging the model to explore more visual hints.
3.3 Case Analysis and Error Analysis
Next, we perform the case study to illustrate the superiority our method compared with other baselines: 1) Radial, 2) generator without DH-GAN (abbr: Generator), Specifically, we visualize the visual hints: 1) generated with pre-processing visual hints (abbr: Raw visual hints), 2) predicted with only Generator (abbr: Generator), and 3) predicted with full DH-GAN (abbr: Ours). As shown in Fig. 2 (left), we can find that the raw visual hints obtained in pre-processing are quite noisy. Such noisy visual hints fail to provide the clear supervised signal to guide the generator, leading to the imprecise prediction of the generator. In contrast, with the help of DH-GAN, the generator can predict more accurate and referential ones, which faithfully guides the generator predict precise and high-quality questions. Please refer to Appendix for more detailed case studies.
Moreover, we further investigate the failures of our model by pinpointing various bad cases. The results are shown Fig. (right). Since it is very time-consuming to analyze the errors of every example, we qualitatively categorize them into five different error groups. Clearly, even with our DH-GAN method, most the bad cases are caused by the first error type visual hint prediction error, which is not surprising since we have no human annotations on visual hints. The second- largest error group about reasoning error (i.e., yes/no), which indicates the considerable improvement potential by truly understanding the logical polarity in the answer. The remaining errors are mainly from detection error and syntactic error, which can improved with better object detection tools and language models. More detailed examples and analyses are provided in Appendix J.
3.4 Data Augmentation and Zero-shot VQA
To investigate the effect of VQG for real world applications, we apply the VQG to the visual question answering (VQA) under two settings: 1) as a data augmentation method, 2) as a solution for zero-shot VQA.

---

Data augmentation is one of the most important applications of VQG. In this section, we use our proposed DH- BQG method to generate more ques-
tions for training a better VQA model. We em- ploy two existing VQA models: 1) Bottom-Up Top- Down (abbr: BUTD (2) and 2) the Bilinear Attention Network (abbr: BAN [2I]). We compare our DH- VQG with Radial (a strong baseline) to demonstrate the effectiveness of the QG-driven data augmentation.
45911
Clussification Accuracy
64.22
43.03
<2991
Specifically, we split the VQA2.0 train-set (the same as the VQG dataset) z E {0.3, 0.5, 0.7). The VQA models are trained only on the part, while the other VQA variants are trained on the combination of the golden x part and the questions generated by VQG models. The results are shown in Fig. 3 Surprisingly, we observe that the Radial augmented VQA models perform inferior to the single VQA models. We think that the generated questions contain noises, which may mislead the VQA model in some ways [50]. Notably, our DH-GAN model outperforms all the baselines consistently, demonstrating the high- quality of the generated questions.
01888
60.967
BUTD
BUTD WRadia
BUTD
88.089
BAN
BAN Radiil
BAN DHHGAN
35,000
0.3
Percentage training data
Figure Data augmentation
In addition, we introduce the VQG as an effective solution for the zero-shot VQA (ZS-VQA) problem. Following the practice of [50] [29]. we first obtain the zero-shot words and
Table 4: VQG for Zero-shot VQA
Results on VQA2.0.
then construct the zero-shot dataset for VQA 2.0. Concretely,
Method
BUTD
BAN
we first find the same words between the questions and answers in the original VQA dataset (including the training and testing) by filtering out stop words. Then we uniformly sample 10% words from them and obtain the zero-shot words. Secondly. we pick up all samples whose answers contain zero-shot words and construct the ES-VAA testing set. Finally, the ZS- VQA training set which has no overlap with the 7S-VQA testing set can be constructed. Similar to the above data augmentation experiment, we employ two VQA methods: 1) BUTD, 2) BAN, and two VQG methods: 1) Radial, 2) DH- GAN. Technically, for each variant, we use the VQG method to generate the questions and produce additional training data, which can be used to train the VQA model. Following [50), since ZS words appear in the question, they can be found in the images and thus can treat them the target answers to generate questions to help the VQA model to predict the zero-shot answer. As shown in Table 4 we observe that both BUTD and BAN achieve 0% in the ZS-VQA test set. This is because without the help of VQG, both VQA models cannot see the ZS- 2o-words during the inference. However, when we leverage various VQG methods, both the VQA models achieve remarkable improvements. which demonstrate that the VQG can be very beneficial in solving the ZS- VQA problem. Furthermore, we observe that the DH-GAN variants outperform the Radial variants, which proves the superiority of our proposed method in generating high-quality QA pairs.
w/o. VQG.
0%
0%
Radial VQG
18.15%
18.37%
DH-GAN
19.21 %
19.93%
4 Related Work
Visual Question Generation. Visual question generation 355
7 38
25
48 is a promising and
attractive task in the vision and language domain. Jain et al. [88] firstly introduced the variational autoencoder (VAE) based framework to generate diverse questions. Krishna et al. (22) introduced the answer-type as side information based on the variational model generate more goal-driven
results. Furthermore, Li et al. (28), Shah et al. 4 combined the visual question answering (VQA) and visual question generation tasks by dual learning mechanism. Liu al. [33] further formulated the VQG problem as the inverse of the VQA task and regard the type-specific partial question as
side information to guide the training process. Recently, Xu et al. (50) adopted the GNN method to generate answer-specific questions. Kai et al. (20) introduced the double-hints, leveraging both textual answers and visual regions to faithfully guided the generating process, which made a valuable step to address the one-to soc-toomaayy mapping issue.

---

The generative adversarial network has achieved great success in computer vision [2 51 53
Generative Adversarial Network for Text Generation.
6 9 However, compared with images, the language is discrete, making it infeasible for gradients propagating from discriminator to generator. Li al. (24), Yu al. [53] firstly proposed the sequence GAN, which employs the Reinforcement Learning mechanism to provide feedback obtained from the discriminator
27
54
reward g proposed the MaskGAN, which trains the model on a text fill-in- the-blank or in-filling task to reduce the training instability and model drop problem. Despite the RL-based methods [26 5 M.. Kusner and Hemández-Lobato [23 attempted to find continuous approximation of discrete sampling by the Gumbel Softmax mechanism. In addition, Zhang et al. [56] proposed the continuous kernelized discrepancy function to approximate the non-differentiable sampling function in the original GAN.
to the generator. Fedus et al.
Noisy Supervision. Learning from noisy data with noisy supervision has attracted significant Li et al. [30] proposed unified framework to distill the knowledge learned from clean dataset to learn a better model from noisy labels. Han et al. [T5] proposed the "co-teaching" learning diagram for combating with noisy labels. They employ two network and let them teacher each other simultaneously. Han et al. [14] proposed husan-assisted approach called "Masking" which conveys the external rules speculate the noise transition matrix effectively. Ren et al. [4]] further studied the text classification task with rule-induced weak labels. They designed label denoiser to estimate the the source reliability to learn from noisy data. Different from these methods which mostly focus on classification task with noisy target labels, focus on generation task under noisy supervision.
amount of interests in recent years 30
15 14
16 55 52
Conclusion
In this paper, we present novel learning approach for double-hints based VQG, which can be cast as a weakly supervised learning problem with noises. In particular, we propose a Double- Hints guided Generative Adversarial Networks (DH-GAN), which consists of double-hints based question generator and question cnnstiin-answer-awwrr discriminator. Thanks to DH-GAN, the generator and discriminator undergo an adversarial process encourage the generator to identify the salient visual regions of interest so that the high-quality question can be generated, which are indistinguishable from the ground truth questions by the discriminator. Our extensive experiments demonstrate the effectiveness of our proposed model.
Acknowledge
This work has been supported in part by the National Key Research and Development Program of China (2018AAA0101900) Zhejiang NSF (LR21F000004), Chinese Knowledge Center of Engineer- ing Science and Technology (CKCEST).
10

---

References
[1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European conference computer vision, pages 382-398. Springer, 2016.
[2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In The IEEE International Confer- ence on Computer Vision (ICCV), December 2015.
[4] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72,
2005.
[5] Yu Chen, Lingfei Wu, and Mohammed J Zaki. Reinforcement learning based graph-to-sequence model for natural question generation. In The Eighth International Conference on Learning Representations (ICLR 2020), 2020.
[6] Xuefeng Du, Jingfeng Zhang, Bo Han, Tongliang Liu, Yu Rong, Gang Niu, Junzhou Huang, and Masashi Sugiyama. Learning diverse-structured networks for adversarial robustness. arXiv preprint arXiv:2102 01886, 2021.
[7] Zhihao Fan, Zhongyu Wei, Piji Li, Yanyan Lan, and Xuanjing Huang. A question type driven framework to diversify visual question generation. In I/CAI, pages 4048 4084,005,4 2018.
[8] William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the_. arXiv preprint arXiv:1801.07736. 2018.
[9] Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy is aware of adversarial attacks. arXiv preprint 1415, arXiv:2010.1 1415,, 2020.
[10] Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84, 1990.
[11] Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers, pages 3168-3180, 2019.
[12] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 2014.
[13] Jiaxian Guo, Mingming Gong, Tongliang Liu, Kun Zhang, and Dacheng Tao. Ltf: A label transformation framework for correcting target shift. ICML, 2020.
[14] Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. arXiv preprint arXiv: /005 0000000833. 2018.
[15] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv: arXiv:/804.0000 1804.0687, 2018.
[16] Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning, pages 4006-4016. PMLR, 2020.
11

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Unnat Jain, Ziyu Zhang, and Alexander G Schwing. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6485-6494, 2017.
[19] Unnat Jain, Svetlana Lazebnik, and Alexander G. Schwing. Two can play this game: Visual dialog with discriminative question generation and answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[20] Shen Kai, Lingfei Wu, Siliang Tang, Fangli Xu, Zhu Zhang. Yu Qiang, and Yueting Zhuang. Ask question with double hints: Visual question generation with answer-awareness and region- reference, 2021. URL httttttt::t////t httttp:://pperevviiit
[21] Jin-Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bilinear attention networks, arXiv preprint arXiv:1805.07932 2018.
[22] Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Information maximizing visual question generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2008-2011 8, 2019.
[23] Matt J Kusner and José Miguel Hernández-Lobato Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv: 161 1. 04051, 2016.
[24] Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06544 2017.
(25] Juncheng Li, Siliang Tang, Fei Wu, and Yueting Zhuang. Walking with mind: Mental im- agery enhanced embodied qa. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1211-1219, 2019.
[26] Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. Unsupervised reinforcement learming of transferable meta-skills for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 12123-12132, 2020.
[27] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624-639, 2018.
(28) Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. Visual question generation as dual task of visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 61 16-6124, 2018.
[29] Yuanpeng Li, Yi Yang, Jianyu Wang, and Wei Xu. Zero-shot transfer vqa dataset. arXiv preprint 1 2018
[30] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
[31] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics, URL hhtpp::/// wwwp:://ww//wwwwbbbb///b//m/ org/ attp/ooo/oool0000----- 1013
[32] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[33] Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 861 1-8619, 2018
12

---

[34] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7219-7228, 2018
[35] Issey Masuda Mora, Santiago Pascual de la Puente, and X Giro-i Nieto. Towards automatic generation of question answer pairs from images. CVPRW, 2016.
[36] Liangming Pan, Wenhu Chen, Wenhan Xiong, Min- Yen Kan, and William Yang Wang. Unsuper- vised multi-hop question answering by question generation. arXiv preprint arXiv:2010. 12623, 2020.
[37] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics. 2002.
[38] Badri N Patro, Sandeep Kumar, Vinod K Kurmi, and Vinay P Namboodiri. Multimodal differential network for visual question generation. arXiv preprint arXiv: 03986, 2018.
[39] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in neural information processing systems, pages 3953-296.1 2015.
[40] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01499 2015.
[41] Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denoising multi-source weak supervision for neural text classification. arXiv preprint arXiv:2010.04582, 2020.
[42] Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024, 2017.
[43] Meet Shah, Xinlei Chen, Mareus Rohrbach, and Devi Parikh. Cycle-consistency for robust. visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6649-6658, 2019.
[44] Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang. Ramesh Nallapati, and Bing Xiang. End-to-end synthetic data generation for domain adaptation of question answering systems. arXiv preprint Sktiv:2010.06022 2020.
[45] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[46] Yu-Siang Wang, Hung-Ting Su, Chen-Hsi Chang, Zhe- Yu Liu, and Winston H Hsu. Video question generation via cross-modal self-attention networks learning. arXiv preprint 100.000907 2019.
[47] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-255 1992.
[48] Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. De-biased court's 's view generation with causality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 763-780, 2020.
[49] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. International conference on machine learning, pages 2048-2057, 2015.
[50] Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen. Radial graph convolutional network for visual question generation. IEEE Transactions on Neural Networks and Learning Systems, 2020.
13

---

[51] Yanwu Xu, Mingming Gong, Junxiang Chen, Tongliang Liu, Kun Zhang, and Kayhan Bat- manghelich. anneratveediiccimnnati complementary learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 6526-6533 2020.
[52] Jiangchao Yao, Jiajie Wang. Ivor W Tsang, Ya Zhang, Jun Sun, Chenggi Zhang, and Rui Zhang. Deep learning from noisy image labels with quality embedding. IEEE Transactions on Image Processing. 28(4):1909-199 2018.
[53] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[54] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In Interna- tional Conference on Machine Learning, pages 11278 1287. PMLR, 2020.
[55] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. arXiv preprint arX1v:2010.01736. 2020.
[56] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In International Conference Machine Learning, pages 4006-4015. PMLR, 2017.
[57] Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain gener- alization via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
14


Reviews:
[{"review_id": "lhdMx5mcfd0", "reviewer": "Reviewer_noBr", "summary": "The paper proposes to improve visual question generation by using semi-supervised visual grounding information. However, that is not the main contribution of the paper, as it is proposed by a closely related work [19]. Instead, the paper focuses on how to *better* use the information/hint from noisy visual grounding supervision obtained from visual saliency-based approaches. To this end, the paper proposes to use GAN to indirectly supervise the *goodness* of the generated visual region hint by trying to discern real/fake question, answer, image triplet. Experiments are done on the existing VQG dataset show some improvement over [19] and other baselines using both auto-metrics and human judgments.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n**[S1]** Sensible model design: The design of real/fake GAN to deal with noisy visual grounding obtained from [19] is interesting and it is easy to see why that might help the model learn better.\n\n**[S2]** Clearly written with exception of pervasive grammar errors: There are several errors in grammar and phrasing which does need to be rectified, but they do not always make the text difficult to understand. (E.g., in the abstract: \"Despite they proposed a simple rule-based similarity matching method to obtain candidate visual hints, they could be very noisy practically\"). The main techniques, motivations, and experimental results are presented clearly and are easy to understand.\n\n**Weaknesses**\n\n**[W2]** Contributions are sparse and of rather low significance: The idea to use visual grounding information (called double hint) is already explored by the closest work [19]. The main contribution is the use of GAN to deal with noisy supervision which is bound to arise due to the rule-based nature of generating those visual hints in [19]. The formulation of GAN for this purpose is also relatively straightforward and furthermore, it is unclear how much better the proposed method is, compared to [19]. Overall, the novelty and significance of this work is quite low.\n\n**[W1]** Human evaluation is not described well and not meaningful: Firstly, the human evaluation results omit the closest work, DH-VQG which is very concerning. Next, there is next to no detail about the exact instructions given to evaluators, what the criteria \"syntax, semantics, and relevance\" mean (and how they were shown to annotators). There is no justification about why a Likert scale was used instead of head-to-head comparisons etc..We don't even know who the annotators were or how they were evaluated/vetted/compensated. Finally, the final reported numbers are quite close together for baseline and proposed and it is unclear how to interpret that. Overall, the human evaluation adds little to no information due to the numerous problems mentioned above.\n\n**[W3]** VQG is not widely studied and the impact of the work is not clear: VQG, on its own, is not very popular in the community as it remains unclear what its benefits would be (data augmentation?). I know this is not the scope of the paper but the concerns about its significance remain intact. The nearest related work [19] is not published yet (mostly due to low contributions/significance according to open review) and the current work is a slight improvement upon that. As such, I am unsure how important/meaningful the contributions are. The techniques proposed are also unlikely to scale to anything other than VQG.\n\n**Overall:**\nOverall, I think the paper is a good effort and shows promising results. However, the proposed method is only marginally different/better than its nearest related work. Furthermore, the proposed method is likely to be limited in application to VQG, a problem that has not clearly established itself in the field. There are also minor complaints about the reporting of the results and human evaluation. I recommend rejection of this work in its current form.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some concerns about the instructions given, qualifications, and compensation structure for the human annotators. However, I think it is a relatively small-scale experiment and is not of huge concern.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "The posted link to anonymized github repo shows one of the authors as being contributors. While this knowledge has not affected my impartiality, it could potentially be problematic/used as signal to identify the authors."}}, {"review_id": "gtJgFmDFSX9", "reviewer": "Reviewer_6CYj", "summary": "In this paper, the authors proposed a new visual question generation method called Double-hint GAN (DH-GAN). In this method, the authors leveraged both the visual hints and the answer to generate the corresponding question. To make it grounding on the specific visual and answer hints, the authors introduced supervised losses to guide the pointing of visual objects and question decoding. Besides, the authors used a discriminator to learn to differentiate the ground-truth question and generated question, and in turn, the output of the discriminator is used as the reward along with the auto metric BLEU. After training with a fusion of supervised learning and policy gradient REINFORCE, the model demonstrates superior performance on both VQA 2.0 and COCO-QA to previous works. \n\nThe main contribution of this paper is that the authors proposed to use a pipeline similar to the generative adversarial network (GAN) for visual question generation. In the paper, a discriminator was introduced to impose the reward for policy gradient besides the regular supervised training loss and the reward from the automatic metric BLEU. The experimental results show that this method can generate better questions conditioned on the given visual hints and target answers, compared with the prior works.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\n\nThis paper lacks novelty from various perspectives. First, the proposed method is similar to [19] in that they both used similar so-called double hints to generate the questions, which include visual hints and answers. Moreover, the visual hint pointing loss and the question decoding loss used in this paper are also similar. Second, different from [19], this paper used a REINFORCE method to guide the question generation. More specifically, the authors used two rewards, one is from the automatic metric BLEU, and the other one is from the discriminator trained during the whole learning process. Obviously, using the automatic metric as the reward has been used for image captioning [5,38], and the discriminator here is a kind of ad-hoc component to the main pipeline. \n\n[Quality]\n\nPros:\n\n1. This paper proposed a new pipeline called DH-GAN for visual question generation. It consists of two main components, a generator that takes visual hints and answers for question generation, and a discriminator learning to distinguish the ground-truth answer and the generated answer given the image, visual hints, and the answer. \n\n2. The proposed DH-GAN demonstrates superior performance to previous work including DH-VQG on two datasets. Further ablation studies demonstrate that visual hints are important to guide question generation. \n\n3. The authors further tried to use the generated questions to perform data augmentation for visual question answering. This is an interesting and meaningful study.\n\nCons:\n\n1. The visual question generation is somehow an ill-posed problem. Given the image and answer, there are multiple questions that can lead to the same answer. Even though this paper further leveraged the visual hints to reduce the size of candidate space, it is still an ambiguous problem. Forcing the model to predict the correct question is more like an overfitting game.\n\n2. As discussed above, the method proposed in this paper does not have enough novelty from different perspectives. Particularly, this paper is very similar to [19] in terms of the objective function and model pipeline. Though the authors introduced a new discriminator, there is no clear cue about how this discriminator contributes to the final performance, not to say the discriminator is also not new.\n\n3. The authors claimed that the proposed method outperforms the previous method by a large margin. However, to me, it is close to DH-VQA on various metrics in Table 1, which makes sense because they both used visual hints, i.e., detected objects from the image. \n\n4. The ablation study in this paper conveys little information. According to Table 3, the main impression is that the visual hints are important to generate good questions. However, using visual hints is not the novel part of this paper. It would be better to have more ablation studies on how the discriminator helps to question generation.\n\n5. In Sec 3.4, the authors used generated questions for data augmentation. As we can see, the proposed model outperforms the baseline method very marginally, which weakens the contribution of this paper. Note that the model is not aimed at improving VQA during the training. Instead, it always learns to generate the questions which have been already covered in the original dataset. As such, it is more like a heavily conditioned and ill-posed image captioning problem.\n\n[Clarity]\n\n1. The difference between the proposed method and [19] is not clearly discussed. In lines 50-52, the authors mentioned [19] used a simple rule-based method to obtain the visual hints. I do not think this is a correct statement because [19] also used a learnable module to learn to point to the target visual hints. Besides [19], the authors also did not discuss the difference to other methods in the related work section.\n\n2. Many details are missed in the paper. For example, how many objects are extracted from the image? What is the training procedure for the proposed pipeline? How often the discriminator is updated during the training? Is the REINFORCE added at the beginning of the training or later stage? How many visual hints predictor modules are used in the paper and how does it affect the final performance? How to compute the reward D(\\cdot) in Eq. (12)?\n\n3. In Table 1, the authors should add cites for previous works; \"double text and visual hints\" is a weird and annoying term. I do not quite get the usage of \"+\" and \"-\" in Table 3. The different error types shown in Figure 3 are not explained. \n\n[Significance]\n\nThis paper proposed a new method for visual question generation, which demonstrates better performance on VQA 2.0 and COCO-QA. As discussed above, the novelty of this paper is limited. Also, the motivation of this paper is not clear, and the contribution of this paper is not well-justified in terms of the improvements and the benefit to downstream task VQA. Therefore, I think this paper is not ready for this venue. I would suggest the authors focus more on how to leverage the visual question generation for visual question answering or other end tasks. \n\n3. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss the limitations and societal impact. But again, I think the main concern is that the paper lacks novelty and the task of visual question generation stand-alone is ill-posed without the showcase of its notable benefit to downstream tasks, such as VQA.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "ZyPhF-_HRT", "reviewer": "Reviewer_W4CH", "summary": "The work proposes a double hint guided GAN architecture, consisting of a visual and answer hint based generator and question+answer+image triplet discriminator. The work also proposes a policy gradient based reward function to optimize both question generation and visual hint prediction. The proposed method outperforms baselines on several metrics and human evaluation studies. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n- The GAN based VQG architecture achieves better generation result compared to DH-VQG and other baselines on two datasets VQA and COCO-QA.\n- Ablation studies were performed on the different hyper-parameters used for the setup.\n\n\nWEAKNESSES AND REMARKS\n\n- Does VQG task generated questions actually help in improving VQA performance? The premise that using answer hints and visual features help to generate better questions can also result in questions which are very similar to already existing questions in the dataset. Hence addition of such new questions might not improve the VQA task performance significantly. Experiments in Section 3.4 Data Augmentation, it is unclear how any additional questions are added during data augmentation for Radial and DH-GAN results. Interestingly as more questions from the train set are added, the benefits of VQG starts to diminish and all the baselines and DH-GAN starts to perform very same. To some extent that justifies the argument that the generated questions might be just very similar questions that are already in the training set. \n\n- To understand more about the quality of the generated questions, data augmentation experiment is very important. For fair comparison, I would be interested to see how another dual vision+answer hints approaches like DH-VQG work compared to DH-GAN approach. Since the major difference between this work and DH-VQG is the difference in architecture, it is important to see whether the improvement is coming from using visual hints or the GAN based approach.\n\n- Addition of dual visual features and answer hints has already been proposed by DH-VQG[19] paper. This paper applies the same idea but uses a different architecture for question generation. I am unsure how much additional novelty this approach adds. \n\n- How does $k$ (number of modules in the visual hint generator) effect the quality of the visual hints and the eventual generations? This ablation is missing. \n\n- The model architecture is too complex to understand. Latest state-of-the-art multimodal models are using transformer layers to learn attention between different modalities like image features and text, and have been shown to perform better than LSTMs. I would like to understand from the authors if they considered using transformers to learn attention between visual features and answers to generate visual hints over using multiple LSTMs int he generator, that would simplify the architecture. \n\n- Human evaluation is missing DH-VQG[19] baseline, which seems to be the most closely related approach. In addition, human evaluation results are very close to that of only Generator baseline.\n\nI would be interested to see the discussion and authors' replies to the concerns during the response stage in order to consider improving the rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have not added any discussion about societal impact of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "RYlF_bUZDPE", "reviewer": "Reviewer_tJc4", "summary": "This work aims to generate human-like visual questions from an image, the questions and answers in training data, and the visual hints via an ad-hoc method using an object detector where detected visual regions and predicted attributes are available following Kai et al., 2021. The improvement over the previous work, DH-VQG, is to adopt GAN-based training to overcome the noisy supervision of the visual hints. Unfortunately, there are weak evidences for the usefulness of the work and it needs to improve the exposition of the work to be self-contained.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Is it really useful for data augmentation?\n\n1a) As the authors pointed out in L23, the VQG task is “one of the effective data augmentation approaches,” however, in Fig. 3, they showed a minor improvement (<0.2 accuracies around 50% percent of training data) over the baseline. Notice that, for the visual hints, they exploited Stanford CoreNLP (E.1.1) and Faster-RCNN pre-trained on Visual Genome (E.1.3) using its heavy annotations. By the way, one can exploit the visual question answering annotations of Visual Genome for data augmentation to improve the accuracy instead of a hassle of VQG (although the two methods can interfere with each other.)\n\n1b) The automatic evaluation metrics, i.e., BLEU, do not guarantee the usefulness of the generated questions. As the authors pointed out in L206, it measures “the syntactic quality of the generated questions” and “it is well known that the BLEU score cannot faithfully reflect the semantic quality.” Additionally, Table 2 shows that, presumably, the indifference (Could you clarify the confidence interval? What are the numbers in the parentheses?) of the proposed method in the aspect of syntax in the human evaluation. Therefore, Table 1 is supportive evidence, not the ground for the effective evaluation of generated questions.\n\n1c) The BLEU and the human evaluation rely on the ground-truth questions. However, does it promote the usefulness of the generated questions for data augmentation if the generated questions are similar to the ground-truth questions? The proposed method only generated similar expressions of questions sharing the same answer. They do not evaluate the diversity of questions although the authors argue that “to perform both exploration of question sample and the visual hints via policy gradient. (L68)” These issues undermine the purpose of VQG, why VQG is important. \n\n2) Writing and exposition \n\n2a) At the end of Section 2.2.1, the authors do not explain how the target visual hints are obtained. This explanation is critical to understand the proposed method, but the current form is not self-contained. The recommendation is to update a brief mention of the following of Kai et al. (2021) while adding a redirection to E.1.1 and E.1.2 for the details.\n\n3) Minor issues\n\n- The citation of Kai et al., 2021 is missing the name of the conference. (Not limited to this.)\n- In L68, “to perform exploration of *both* question samples and visual hints via policy gradient”? \n- In L85, how do you define the word tokens?\n- In L138, “compute *the* average of relevance score”\n- In L149, why do you initialize the starting states by answer embedding?\n- In L216, you previously define that $\\bar{Q}$ is the sampled (L194) instead of the target question (L216).\n\n----\nI want to let you know that we, the reviewers, have a discussion about the concerns raised by the reviewers. The main concerns of the paper are sparse contributions and low significance. Especially, the authors fail to persuade why VQG is an interesting task and what would meaningfully contribute to the community. The connection of zero-shot and few-shot to VQG is weak and it also seems an astray of the main proposition -- weakly supervised learning with VQG.  \nTherefore, in the revision, I highly recommend strengthening the motivation of VQG and perform rigorous evaluations to show the significance of the proposed idea.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The major limitations come from the evaluation to prove the usefulness of the proposed approach. Please clarify the practicality of data augmentation. ", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend reject after reviewing the author response and discussion.

While the author response addressed some concerns, several critical concerns remained, including:
1) Limited novelty over prior work.
2) Limited focus on the benefit of VQG, although the authors acknowledge the importance in Section 3.4 and the author response. 
2a) Section 3.4. misses comparison to closes competitor DH-VQG and ablations.
2b) The additional results provided in the author response on few / zero-shot are interesting, but again miss comparison to prior work (e.g. DH-VQG) and ablations for this important final task

I thus recommend reject. 
[The authors might want to consider submitting a significantly revised version to a vision or NLP focused venue]
Decision: Accept (Poster)