SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \tilde {\boldsymbol {y}} _ {i} ^ {(t)} = \sigma (\tilde {\boldsymbol {W}} _ {y} [ \tilde {\boldsymbol {a}} _ {i} ^ {(t)}; \boldsymbol {v} _ {i} ^ {(t - 1)} ]) \\ \tilde {\boldsymbol {z}} _ {i} ^ {(t)} = \sigma \left(\tilde {\boldsymbol {W}} _ {z} \left[ \tilde {\boldsymbol {a}} _ {i} ^ {(t)}; \boldsymbol {v} _ {i} ^ {(t - 1)} \right]\right) \tag {6} \\ \end{array}
$$
Finally, we perform the information interaction, where each graph updates its node representations under the control of gates computed by the other graph. More specifically, node representations are updated by:
$$
\begin{array}{l} \boldsymbol {u} _ {i} ^ {(t)} = \tilde {\boldsymbol {z}} _ {i} ^ {(t)} \odot \boldsymbol {u} _ {i} ^ {(t - 1)} + (\boldsymbol {1} - \tilde {\boldsymbol {z}} _ {i} ^ {(t)}) \odot \\ \operatorname {t a n h} \left(\boldsymbol {W} _ {a} \left[ \boldsymbol {a} _ {i} ^ {(t)}; \tilde {\boldsymbol {y}} _ {i} ^ {(t)} \odot \boldsymbol {u} _ {i} ^ {(t - 1)} \right]\right) \\ \end{array}
$$
$$
\begin{array}{l} \pmb {v} _ {i} ^ {(t)} = \pmb {z} _ {i} ^ {(t)} \odot \pmb {v} _ {i} ^ {(t - 1)} + (\pmb {1} - \pmb {z} _ {i} ^ {(t)}) \odot \\ t a n h (\tilde {\boldsymbol {W}} _ {a} [ \tilde {\boldsymbol {a}} _ {i} ^ {(t)}; \boldsymbol {y} _ {i} ^ {(t)} \odot \boldsymbol {v} _ {i} ^ {(t - 1)} ]) \\ \end{array}
$$
The idea of using gates computed by the other graph to update node representations in each graph enables the information in input passage and answers interact more frequently, both of which act as strong constraints to the output questions.
By iteratively performing the three steps for  $T$  times, we get the final representations  $\pmb{u}_i^{(T)}$  and  $\pmb{v}_i^{(T)}$  for  $u_i\in \mathcal{U}$  and  $v_{i}\in \mathcal{V}$
4.6 Decoder
For the  $k$ -th input sentence  $S_{k}$  containing certain answers, our decoder generates the corresponding target-output  $T_{k}$ . As mentioned above, the generation process of all target-outputs are independent. The decoder is based on the Transformer-decoder containing a (masked) multi-head self-attention layer, a multi-head encoder-attention layer, a feedforward projection layer and the softmax layer. To compute keys and values for the multi-head encoder-attention layer, it leverages the outputs from our answer-info encoder, i.e., it uses  $\pmb{H}_{k}^{enc}$  described in Section 4.3 to generate  $T_{k}$  corresponding with the  $k$ -th sentence.
To generate coherent questions, we need to capture the context dependencies between input answers and passages. To this end, both  $\boldsymbol{u}_k^{(T)}$  and  $\boldsymbol{v}_k^{(T)}$ , which comes from the dual-graph interaction process, are used as additional inputs for generating  $T_k$ . First, they are concatenated with the output of each head from both (masked) multi-head self-attention layer and multi-head encoder-attention layer before sending to the next layer. Second, they are concatenated with inputs of the feed-forward projection layer. The two representations are also expected to make generated questions more relevant to given inputs.
4.7 Coarse-To-Fine Generation
Since the semi-autoregressive generation scenario makes it more challenging to deal with coreferences between questions (especially questions in different groups), we perform question generation in a coarse-to-fine manner. The decoder only needs to generate "coarse questions" where all pronouns are replaced by a placeholder "[p]". To get final results, we use an additional pre-trained coreference resolution model to fill pronouns into different placeholders. To make a fair comparison, we use the coreference resolution model (Clark and Manning, 2016) adopted by prior works CoreNQG (Du and Cardie, 2018) and CorefNet (Gao et al., 2019).
Model	BLEU1	BLEU2	BLEU3	ROUGE	METEOR	Length
Seq2seq (Du et al., 2017)	28.72	10.16	6.30	31.75	13.10	5.78
CopyNet (See et al., 2017)	29.40	12.14	6.53	33.71	14.20	5.77
CoreNQG (Du and Cardie, 2018)	33.84	14.69	8.72	34.38	14.05	6.08
VHRED (Serban et al., 2017)	30.51	11.95	6.94	31.93	12.42	4.83
HRAN (Xing et al., 2018)	30.18	12.53	7.65	35.06	12.95	5.02
ReDR (Pan et al., 2019)	30.84	15.17	9.81	35.58	15.41	5.58
CorefNet (Gao et al., 2019)	32.72	16.01	10.97	37.48	16.09	5.96
Ours	35.70	19.64	12.06	38.15	17.26	6.03
In this section, we first introduce the three kinds of baselines. After that, we compare and analyse the results of different models under both automatic and human evaluation metrics.
We compared our model with seven baselines that can be divided into three groups. First, we used three TQG models: the Seq2seq (Du et al., 2017) model which pioneered NN-based QG, the CopyNet (See et al., 2017) model that introduced pointer mechanism, and CoreNQG (Du and Cardie, 2018) which used hybrid features (word, answer and coreference embeddings) for encoder and adopted copy mechanism for decoder. Second, since prior works regarded SQG as a conversation generation task, we directly used two powerful multi-turn dialog systems: the latent variable hierarchical recurrent encoder-decoder architecture VHRED (Serban et al., 2017), and the hierarchical recurrent attention architecture HRAN (Xing et al., 2018). Third, we used prior works mentioned above. For Pan et al. (2019), we adopted the ReDR model which had the best performance. For Gao et al. (2019), we used the CorefNet model. Although a CFNet in this paper got better results, it required additional human annotations denoting the relationship between input sentences and target questions. So it is unfair to compare CFNet with other methods.
It is worth mentioning that when generating questions using the second and third groups of baselines, only previously generated outputs were used as dialog history, i.e., the gold standard questions are remain unknown (in some prior works, they were directly used as dialog history, which we think is inappropriate in practice).
Table 2: Experimental results. In each column, we bold / underline the best performance over all / baseline methods, respectively. Under the evaluation of BLEU, ROUGE-L and METEOR, our model differs from others (except the METEOR score of CorefNet) significantly based on the one-side paired t-test with  $p < 0.05$ .
SQuAD	CoQA	Ours
Passage	117	271	271
Question	10.1	5.5	6.6
Answer	3.2	2.7	3.2
Table 3: Average number of words in passage, question and answer in different datasets.
5.2 Automatic Evaluation Metrics
Following the conventions, we used BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004) and METEOR (Lavie and Agarwal, 2007) as automatic evaluation metrics. We also computed the average word-number of generated questions. As shown in Table 2, our semi-autoregressive model outperformed other methods substantially.
When we focus on the second and third groups of baselines regarding SQG as multi-turn dialog generation tasks, we can find that models from the third group are more powerful since they make better use of information from input passages. Besides, models from the second group tend to generate shortest questions. Finally, similar to the problem that dialog systems often generate dull and responses, these models also suffer from producing general but meaningless questions like "What?", "How?", "And else?".
When we compare the first and third groups of baselines (which are all QG models), it is not surprising that SQG models show more advantages than TQG models, as they take the relationships between questions into consideration. Besides, CorefNet gets better performance among all baselines, especially ReDR. This indicates that comparing with implicitly performing reinforcement learning through QA models, explicitly using target answers as inputs can be more effective.
CoreNQG	CorefNet	Ours
Fluency	2.36	2.51	2.44
Coherence	1.53	2.04	2.17
Coreference	1.15	1.56	1.54
Answerability	1.12	1.18	1.45
Relevance	1.47	1.24	1.62
Note that if we directly compare the performance between SQG task and TQG task under the same model (e.g., the Seq2seq model), evaluation scores for TQG tasks are much higher, which is not surprising since SQG is harder than TQG dealing with dependencies between questions. Another fact lies in the computation of automatic evaluation metrics. As shown in Table 2, questions in SQG datasets are much shorter than TQG. Since our automatic evaluation metrics are based on  $n$ -gram overlaps between generated and gold standard questions, the scores significantly go down with the growth of  $n$  (for this reason, the  $\mathrm{BLEU}_4$  scores are not listed in Table 2). This also illustrates the importance of performing human evaluation.
5.3 Human Evaluation
It is generally acknowledged that automatic evaluation metrics are far from enough for SQG. So we perform human evaluation in five aspects. Fluency measures if a question is grammatically correct and is fluent to read. Coherence measures if a question is coherent with previous ones. Coreference measures if a question uses correct pronouns. Answerability measures if a question is targeting on the given answer. Relevance measures if a question is grounded in the given passage. Since performing human evaluation is rather expensive and time-consuming, we picked up the best TQG model (CoreNQG), SQG model (CorefNet) to compare with our model. We randomly selected 20 passages from the test set with 207 given answers and asked 10 native speakers to evaluate the outputs of each model independently. Under each aspect, reviewers are asked to choose a score from  $\{1, 2, 3\}$ , where 3 indicates the best quality.
The average scores for each evaluation metric are shown in Table 4. We can find that our model gets the best or competitive performance in each metric. When it comes to fluency, all models get high performance, and the CorefNet that outputs
Table 4: Human evaluation results. Scores of each metric ranges between 1 to 3 and larger scores are better.
BLEU3	ROUGE	METEOR
No interact	11.35	37.31	17.05
Uni-graph	9.86	36.44	15.87
Uni-heads	10.33	37.48	16.24
No co2fine	11.75	37.92	17.17
Non-auto	7.79	33.62	14.83
Ours	12.06	38.15	17.26
Table 5: Results for ablation tests.
shortest questions gets the best score. As for coherence, CoreNQG gets poor results since it generates questions independently. When it comes to coreference, our model only slightly lower than CorefNet, which added direct supervision to attention weights by a coreference resolution model. Finally, our model gets the best performance on both answerability and relevance. However, it is worth noticing that all models get rather poor performances under these two aspects, indicating that making a concise question meaningful (i.e., targeting on given answers) with more information from input passage (i.e., performing proper information elimination) is a major challenge in SQG. Besides, as pointed out by Table 3, questions in our SQG dataset are significantly shorter compared with TQG dataset, making subtle errors much easier to be noticed.
6.1 Ablation Test
In this section, we perform ablation test to verify the influence of different components in our model. First, we modify Equation 7 into
$$
\begin{array}{l} \pmb {u} _ {i} ^ {(t)} = \pmb {z} _ {i} ^ {(t)} \odot \pmb {u} _ {i} ^ {(t - 1)} + (\mathbf {1} - \pmb {z} _ {i} ^ {(t)}) \odot \\ t a n h \left(\boldsymbol {W} _ {a} \left[ \boldsymbol {a} _ {i} ^ {(t)}; \boldsymbol {y} _ {i} ^ {(t)} \odot \boldsymbol {u} _ {i} ^ {(t - 1)} \right]\right) \tag {8} \\ \boldsymbol {v} _ {i} ^ {(t)} = \tilde {\boldsymbol {z}} _ {i} ^ {(t)} \odot \boldsymbol {v} _ {i} ^ {(t - 1)} + (\boldsymbol {1} - \tilde {\boldsymbol {z}} _ {i} ^ {(t)}) \odot \\ t a n h \left(\tilde {\boldsymbol {W}} _ {a} \left[ \tilde {\boldsymbol {a}} _ {i} ^ {(t)}; \tilde {\boldsymbol {y}} _ {i} ^ {(t)} \odot \boldsymbol {v} _ {i} ^ {(t - 1)} \right]\right) \\ \end{array}
$$
to get the no interact model, i.e., two graphs are independently updated without any interaction. Second, we build a uni-graph model by removing the passage-info encoder (the remaining rationale graph is updated similarly to Li et al. (2015)). Third, we discard the attention-aware heads in the rationale encoder to get a uni-heads model. Then, we build the no co2fine model without the coarse-to-fine generation scenario. Finally, we build a non-auto model that performs SQG in an non-autoregressive way, i.e., each question is generated in parallel.
Peter was a very sad puppy. He had been inside of the pet store for a very long time. In fact, he had been there for [three months]! Peter had seen many other puppies find a person; he began to wonder why he could not get one. He thought that [maybe his fur was not pretty enough or maybe his bark was not loud enough]2. He tried and tried to please every person who came to the store, but they all picked smaller puppies. However, one day all of this changed. [Sammie]3 came into the store looking for [a golden puppy]4. She wanted a puppy she could snuggle with. It so happened that Peter was very sad and tired that day. Sammie came to hold him. Peter wanted to show off [his bark]5, but he was [too tired]6. He [fell right to sleep]7. Sammie loved him at once and loved holding him in her arms. Sammie took [Peter]8 home that day, and they made lots of fun memories.
Turn	Gold Standard	CorefNet	Ours
1	How long was Peter at pet store?	How long he had been there?	How long was Peter there?
2	Why couldn’t he get someone?	What his fur was?	What did he thought?
3	Who came into the store?	Who came into the store?	Who came into the store?
4	What for?	What was Sammie looking?	Who was she looking for?
5	What did peter wanted to show off?	What Peter wanted show off?	What he show off?
6	Why not?	Why he wanted?	What was he?
7	What did he do with her?	And else?	What did he do?
8	Who did she take?	Who was Sammie took?	What Sammie took that day?
Table 6: Example outputs from different models. We mark the given answers in the passage as blue.
As shown in Table 5, each component in our model plays an important part. Results for the no interact model indicate that compared with independently updating the passage-info graph and answer-info graph, making these information more interacted by our dual-graph interaction scenario is more powerful. Not surprisingly, the uni-graph model removing the passage encoder (i.e., less focusing on context dependencies between sentences from input passage), and the uni-heads model discarding our answer-aware attention mechanism (i.e., less focusing on given answers) get significant worse performance compared with our full model. Besides, our coarse-to-fine scenario helps to better deal with the dependencies between questions since there are widespread coreferences. Finally, although the architecture of non-auto model is a special case of our model where each group only contains a single question, the performance drops significantly, indicating the importance of using semi-autoregressive generation. However, the dual-graph interaction still makes its performance better than the Seq2seq and CopyNet in Table 2.
6.2 Running Examples
In Table 6, we present some generated examples comparing our model and the strongest baseline CorefNet. On the one hand, our model performs better than CorefNet, especially that the output questions are more targeting on given answers (turn 2, 6, 7). It also correctly deals with coreferences (e.g., distinguishing "Peter" and "Sammie"). On the other hand, the generated questions have poor quality when gold standard questions involve more reasoning (turn 2, 6). Besides, the gold standard questions are more concise as well (turn 4, 6).
In this paper, we focus on SQG which is an important yet challenging task. Different from prior works regarding SQG as a dialog generation task, we propose the first semi-autoregressive SQG model, which divides questions into different groups and further generates each group of closely-related questions in parallel. During this process, we first build a passage-info graph, an answer-info graph, and then perform dual-graph interaction to get representations capturing the context dependencies between passages and questions. These representations are further used during our coarse-to-fine generation process. To perform experiments, we analyze the limitation of existing datasets and create the first dataset specially used for SQG containing 81.9K questions. Experimental results show that our model outperforms previous works by a substantial margin.
For future works, the major challenge is generating more meaningful, informative but concise questions. Besides, more powerful question clustering and coarse-to-fine generation scenarios are also worth exploration. Finally, performing SQG on other types of inputs, e.g., images and knowledge graphs, is an interesting topic.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Kevin Clark and Christopher D Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667.
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. arXiv preprint arXiv:1805.05942.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866-874.
Yifan Gao, Piji Li, Irwin King, and Michael R Lyu. 2019. Interconnected question generation with coreference alignment and conversation flow modeling. arXiv preprint arXiv:1906.06893.
Yifan Gao, Jianan Wang, Lidong Bing, Irwin King, and Michael R Lyu. 2018. Difficulty controllable question generation for reading comprehension. arXiv preprint arXiv:1807.03586.
Michael Heilman and Noah A Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617. Association for Computational Linguistics.
Wenpeng Hu, Bing Liu, Jinwen Ma, Dongyan Zhao, and Rui Yan. 2018. Aspect-based question generation.
Hafedh Hussein, Mohammed Elmogy, and Shawkat Guirguis. 2014. Automatic english question generation system based on template driven scheme. International Journal of Computer Science Issues (IJCSI), 11(6):45.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6602-6609.
Vishwajeet Kumar, Ganesh Ramakrishnan, and Yuan-Fang Li. 2019. Putting the horse before the cart: A
generator-evaluator framework for question generation from text. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 812-821.
Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 889-898.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231. Association for Computational Linguistics.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
David Lindberg, Fred Popowich, John Nesbit, and Phil Winne. 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 105-114.
Karen Mazidi and Rodney D Nielsen. 2014. Linguistic considerations in automatic question generation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 321-326.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. arXiv preprint arXiv:1603.06059.
Boyuan Pan, Hao Li, Ziyu Yao, Deng Cai, and Huan Sun. 2019. Reinforced dynamic reasoning for conversational question generation. arXiv preprint arXiv:1907.12667.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Thomas Scialom, Benjamin Piwowarski, and Jacopo Staiano. 2019. Self-attention architectures for answer-agnostic neural question generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6027-6032.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The  $30\mathrm{m}$  factoid question-answer corpus. arXiv preprint arXiv:1603.06807.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Yibo Sun, Duyu Tang, Nan Duan, Shujie Liu, Zhao Yan, Ming Zhou, Yuanhua Lv, Wenpeng Yin, Xiaocheng Feng, Bing Qin, et al. 2019. Joint learning of question answering and question generation. IEEE Transactions on Knowledge and Data Engineering.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. arXiv preprint arXiv:1706.01450.
Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018. Learning to ask questions in open-domain conversational systems with typed decoders.