SlowGuess/ACL_2020_OCR huggingface dataset:
Decoder: Based on the context vector  $c_{t}$ , we exploit another GRU as the decoder. Each question term is yielded by the distribution in Eq.(7), where  $\rho$  is a 1-dim embedding of the reasoning type,  $W_{o}$  and  $b_{o}$  are trainable parameters. We use a copy mechanism (Gu et al., 2016) to tackle unknown words problem, where  $p_{copy}(\cdot)$  denotes the copy distribution. In order to let the questions logically correlate with answers, we guide the decoder by the vector  $c_{g}$ , which encodes the reasoning chain. Accordingly, we regularize the model to adaptively fit the prior pattern represented by the vector  $v_{k}^{m}$ . That can improve the generated quality when the labeled data is insufficient.
$$
\begin{array}{l} p _ {v o c} (y _ {t}) = \operatorname {S o f t m a x} \left(\mathrm {W} _ {o} [ s _ {t}; c _ {t}; c _ {g}; \rho ] + b _ {o}\right) \\ p _ {c o p y} = \sum_ {j = 1} ^ {J} \alpha_ {t j} \times \mathbb {1} \{y = = w _ {j} \} \\ p _ {g} = \operatorname {S i g m o i d} \left(c _ {t}, s _ {t}, y _ {t - 1}\right) \\ s _ {t} = G R U \left(s _ {t - 1}, v _ {k} ^ {m}\right) \\ p \left(y _ {t}\right) = p _ {g} \cdot p _ {\text {v o c}} \left(y _ {t}\right) + \left(1 - p _ {g}\right) \cdot p _ {\text {c o p y}} \left(y _ {t}\right) \tag {7} \\ \end{array}
$$
2.3.4 Learning with Limited Labeled Data
A straightforward solution to train the above QG model is the supervised learning. It minimizes the cross-entropy loss at each generated term by referring to the ground-truth in the labeled data  $D_{L}$ , as  $\mathcal{L}_{sl} = -\frac{1}{n}\sum_{i\in D_L}\sum_{t = 1}^{T_i}\log p(y_{it}|Y_{i; < t},A_i,B_i)$ . However, since  $D_{L}$  only contains a few samples, we would not have enough supervision from  $D_{L}$  to get the best results. While we leverage the unlabeled data  $D_{U}$  to facilitate the training, it is difficult to subtly balance the supervised signal from
6733
$D_{L}$  and the prior pattern learned from  $D_{U}$ . In order to address the problem, we resort to reinforcement learning. It can globally measure the overall quality of the results by minimizing the loss  $\mathcal{L}_{rl} = -\mathbb{E}_{Y^{s} \sim \pi_{\theta}}[r(Y^{s})]$ , where  $Y^{s}$  is a sampled result,  $Y^{*}$  is the ground-truth,  $\theta$  is the parameters of the QG model, and  $\pi$  is the generation policy of the model.  $r(\cdot)$  is a function to evaluate the generated quality. It is the weighted sum of three rewards, including (a) Fluency: we calculate the negative perplexity (Zhang and Lapata, 2017) of  $Y^{s}$  by a BERT-based language model  $p_{LM}$ , that is,  $-2^{-\frac{1}{T} \sum_{t=1}^{T} \log_{2} p_{LM}(y_{t}|Y^{s}<t)}$ ; (b) Answerability: we use a metric  $QBLEU_{4}(Y^{s}, Y^{*})$  (Nema and Khapra, 2018) to measure the matching degree of  $Y^{s}$  and  $Y^{*}$  by weighting on several answer-related factors, including question type, content words, function words, and named entities; (c) Semantics: we employ word mowers distance (WMD) (Gong et al., 2019) to measure the predicted result  $Y^{s}$ , which has different expressive forms but same semantics with gold  $Y^{*}$ , as  $-WMD(Y^{s}, Y^{*}) / Length(Y^{*})$ , where  $Length(\cdot)$  is the length function used as the normalization factor. By considering the metrics are non-differentiable, we exploit the policy gradient method (Li et al., 2017) for optimization. In order to enhance readability, we train the model by a mixed loss, as  $\mathcal{L} = \gamma \mathcal{L}_{rl} + (1 - \gamma) \mathcal{L}_{sl}$ , where  $\gamma$  is a trade-off factor.
3 Evaluations
We extensively evaluate the effectiveness of our approach, including the comparisons with state-of-the-art and the application on a task of MRC-QA.
3.1 Data and Experimental Settings
The evaluations were performed on three typical data sets, including HotpotQA (Yang et al., 2018), ComplexWebQuestions (Talmor and Berant, 2018), and DROP (Dua et al., 2019). These data sets were collected by crowd-sourcing, consisting of  $97\mathrm{k}$ ,  $35\mathrm{k}$ , and  $97\mathrm{k}$  examples, respectively. The HotpotQA data set contained a large proportion of labeled examples. Each comprised of the question, answer, and text with several sentences. Therefore, the HotpotQA data set was suitable to evaluate the multi-hop QG task. The other two data sets contained abundant reasoning questions, but they are not associated with the text and answer. We thus viewed them as the unlabeled data. In order
to simulate the low-resource setting, we randomly sampled  $10\%$  of the HotpotQA train set to learn the models, and evaluated them on the test set with a size of 7k. We verified the generated quality for each evaluated method by comparing the matching degree between the result and gold-standard. We adopted three standard evaluation metrics in the QG task, including BLEU-4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE-L (Lin, 2004). Furthermore, we carried out human evaluations to analyze the generated results. To avoid biases, we randomly sampled 100 cases from the test set and generated questions for each test case by all the evaluated methods. We then invited eight students to give the binary rating on each question independently. The rating was in terms of three metrics, including valid syntax, relevance to input textual sentences, and logical rationality to the answer. We averaged the cumulative scores of the 100 binary judgments as the performances corresponding to the evaluated methods. The resultant scores were between  $0 \sim 100$ , where 0 is the worst, and 100 is the best. We used Randolph's free-marginal kappa (Randolph, 2005) to measure the agreements among the raters.
Model configurations were set as follows. We leveraged 768-dimension pre-trained vectors from the uncased BERT to embed words. The number of states  $K$  and emissions  $L$  in the semi-Markov model was set to 50, 4, respectively. The size of hidden units in both encoder and decoder was 300. The recurrent weights were initialized by a uniform distribution between -0.01 and 0.01 and updated with stochastic gradient descent. We used Adam (Kingma and Ba, 2015) as the optimizer with a learning rate of  $10^{-3}$ . The trade-off parameter  $\gamma$  was set to 0.4. For pattern learning, we parsed every question by the Stanford CoreNLP toolkit (Manning et al., 2014). We then learn better segmentation by forcing the model not to break syntactic elements like the VP and NP. To reduce the bias, we carried out five runs and reported the average performance.
3.2 Comparisons on QG State-of-the-Arts
We compared our approach against five typical and open-source methods. These methods were based on the sequence-to-sequence framework with attention. According to the different techniques used, we summarized them as follows. (a) the basic model with the copy mechanism, i.e.,  $\mathbf{NQG} + +$  (Zhou et al., 2017); (b) ASs2s (Kim et al., 2019), which
6734
encoded the answer separately to form answer-focused questions; (c) CorefNQG (Du and Cardie, 2018) that incorporated linguistic features to represent the inputs better; (d) MaxPointer (Zhao et al., 2018) using gated self-attention to form questions for long text inputs; (e) MPQG+R (Song et al., 2018) that captured a broader context in the text to produce the context-dependent results. In order to understand the effect of unlabeled data, we examined two variants of the proposed model. That is, Ours-Patten which was trained without unlabeled data, and Ours-50% that used  $50\%$  unlabeled data for training. Moreover, we performed empirical ablation studies to gain better insight into the relative contributions of various components in our model, including Ours-Chain that discarded the guidance of the reasoning chain vector and Ours-Reinf that replaced the reinforcement learning with a simple supervised learning.
Table 1: Comparisons of our approach against baselines. Statistically significant with t-test, p-value  $< {0.01}$  .
Methods	BLEU-4	METEOR	ROUGE-L
NQG++	14.55	15.01	31.85
ASs2s	16.89	17.04	34.92
CorefNQG	16.16	16.53	34.30
MaxPointer	17.08	17.34	35.38
MPQG+R	14.90	15.46	32.39
Ours	19.07	19.16	39.41
Ours-50%	18.33	18.36	37.85
Ours-Pattn	17.10	17.35	35.40
Ours-Chain	18.11	18.18	37.37
Ours-Reinf	18.22	18.39	37.48
As reported in Tab.(1), our approach achieved the best performance. We significantly outperformed the best baseline (i.e., MaxPointer) by over  $11.6\%$ ,  $10.5\%$ ,  $11.4\%$  in terms of BLEU-4, METEOR, and ROUGE-L, respectively. From the comparisons among Ours-Pattn, Ours- $50\%$ , and Ours, we found that the performance improves with more unlabeled data. Although we lack an appropriate comparative model based on the unlabeled data, these results can still indicate the effectiveness of our model. With only limited labeled data, our model can effectively leverage unlabeled data to guide the generation. Besides, the ablation on all evaluated components led to a significant performance drop. We may infer that the reasoning chain is crucial for multi-hop QG on the guidance of logical correlations. Also, the reinforcement learning can globally optimize the model by balancing the prior patterns and labeled supervision.
3.3 Human Evaluations and Analysis
Tab.(2) illustrated the results of human evaluation. The average kappa were all above 0.6, which indicated substantial agreement among the raters. Consistent with quantitatively analyzed results in Section 3.2, our model significantly outperformed all baselines in terms of three metrics, where the improvement on the rationality metric was the largest. That showed the satisfied quality of our generated results, especially in terms of multi-hop ability.
Table 2: Human evaluations and kappa agreement. Ration. is short for the rationality metric. Statistically significant with t-test, p-value  $< {0.01}$  .
Methods	Syntax	Relevance	Ration.	Kappa
NQG++	54.3	44.8	50.3	0.61
ASs2s	61.3	50.8	55.8	0.62
CorefNQG	59.0	49.4	54.8	0.64
MaxPointer	61.8	51.8	56.5	0.63
MPQG+R	55.5	47.5	51.3	0.64
Ours	68.3	57.3	62.3	0.65
3.4 Evaluations on Value of Unlabeled Data
We investigated the value of unlabeled data for the overall performance, especially when the labeled data was inadequate. In particular, we randomly sampled  $\{10\%, 40\%, 70\%, 100\}$  of the labeled data, and split the unlabeled data into ten subsets. For each scale on the labeled data, we incrementally added by one subset of unlabeled data to learn the QG model. We used the same training protocol and reported the overall performance on the test set. As shown in Fig.(3), even a small amount of unlabeled data can play a decisive role in improving the performance in terms of three metrics. The ratio of improvement was higher when the scale of the labeled data was small. The results further verified the usefulness of unlabeled data on learning the QG model with a low labeled resource.
3.5 Evaluations on the Mixed Loss Objective
In order to examine the gains of our training approach with the mixed loss objective, we tuned the trade-off parameter (i.e.,  $\gamma$ ) from [0, 1] with 0.1 as an interval. The performance change curve was displayed in Fig.(4). The best performance was obtained at  $\gamma = 0.4$ . The performance dropped dramatically when  $\gamma$  was close to 0 or 1. We would infer that both objectives could help to measure the quality of the outputted results better, and thus train the model efficiently.
6735
Figure 3: Evaluations on effectiveness of unlabeled data under different scales of labeled data.
Figure 4: Evaluations on mixed objective trade-off.
4 Application on the Task of MRC-QA
The task of machine reading comprehension (MRC-QA) aims to answer given questions by understanding the semantics of the text. The mainstream methods are based on the neural network. These methods often need a lot of labeled data for training, but the data is expensive to obtain. Thus, we are inspired to apply our generated results to enrich the training set for the task of MRC-QA. Fig.(5) demonstrates the architecture of this application. Given a case from a small-size labeled set, we first extracted the contents correlated to a specific question from the case's text, including the reasoning chain, reasoning type, answer, evidential entities, and sentences on the entities. We then learned our QG model based on the contents and generated questions as pseudo data to augment the labeled set. For each evaluated case, we could yield approximately  $5\sim 8$  pseudo samples consisted of the text, question, and answer. Later, we trained an MRC-QA model on the augmented labeled set and reported the performance on the test set. By referring to the leaderboard on the HotpotQA website, we
Figure 5: Apply multi-hop QG to support MRC-QA.
chose an open-source model for MRC-QA, named Dynamically Fused Graph Network (DFGN) (Qiu et al., 2019), which achieved the state-of-the-art at the paper submission time. Considering that the size of the training set impacted the model's performance, we ran the entire pipeline with different proportions of the labeled data, so as to verify the proposed model thoroughly. Two evaluation metrics were employed, including exact match  $(EM)$  and  $F1$ . We examined the tasks of answer span extraction, supporting sentence prediction, and the joint task in the distractor setting.
Table 3: Comparison of our QG+QA model against the QA model under different proportions of labeled data.
Labeled Data#	Answer span	Support pred.	Joint
EM	F1	EM	F1	EM	F1
QG + QA(i.e., the DFGN model)
10%	.501	.633	.469	.764	.277	.509
20%	.551	.672	.500	.801	.317	.574
30%	.567	.697	.520	.815	.339	.600
40%	.569	.704	.521	.829	.340	.615
50%	.571	.713	.531	.833	.344	.619
60%	.586	.717	.533	.834	.346	.624
70%	.593	.729	.535	.839	.353	.626
80%	.606	.731	.540	.845	.356	.630
90%	.610	.741	.550	.853	.359	.632
100%	.614	.746	.558	.858	.360	.635
QA(i.e., the DFGN model)
100%	.563	.697	.515	.816	.336	.598
6736
Tab.(3) showed that our  $\mathrm{QG + QA}$  model trained on  $30\%$  labeled data obtained competitive performance against the QA model learned on the  $100\%$  labeled data. When using more labeled data, the performance advantages of our  $\mathrm{QG + QA}$  model continued to grow. Such results showed that our QG model could enlarge the coverage and diversity of the MRC-QA training set given limited labeled data. That could help to learn the state-of-the-art. Moreover, we conducted case studies to understand the generating behavior vividly. As exhibited in Tab.(4), our QG model could generate massive questions on multi-hop reasoning. Contrastively, the gold standard often contained one sample since it was labor-intensive to enumerate all the cases.
Table 4: Case studies on our multi-hop QG model.
Passage: ... (S1) 'The Hard Easy' is the episode written by Thomas Herpich. (S2) He was born in October, 1979 in Torrington, Connecticut, American, along with his twin brother Peter who was a painter and artist. (S3) Thomas is best known for being a storyboard artist on the animated television series 'Adventure Time'. ...
Results of Ours Method
Question: When was the birth time for the writer of the episode 'The Hard Easy'?
Answer: October, 1979
Question: Where is the birthplace for the writer of the episode 'The Hard Easy'?
Answer: Torrington, Connecticut
Question: What nationality was the writer of the episode 'The Hard Easy'?
Answer: American
Question: Who is the twin brother for the writer of the episode 'The Hard Easy'?
Answer: Peter
Question: What is the occupation for the twin brother of the episode writer of 'The Hard Easy'?
Answer: painter and artist
Question: Who is the brother for the writer of the episode
'The Hard Easy'?
Existing models for the QG task include rule-based and neural-based methods. Since the rules are handcrafted, the first method is of low scalability (Chal-i and Hasan, 2015). The researcher turns to the neural model. It can directly map the inputs into questions by using an attention-based sequence-to-sequence framework, which is entirely data-driven with far less labor. Various techniques have been applied to this framework, including answer separately encoding, using linguistic features, capturing border context, reinforcement learning, and em
phasizing on question-worthy contents (Pan et al., 2019). These methods are mainly used to generate simple questions with a single sentence (Yu et al., 2019). They are challenging to generate the reasoning questions accurately due to the lack of fine-grained modeling on the evidential relations in the text. In order to address the problem, Yu et al. (2020) proposed to incorporate a reasoning chain into the sequential framework, so as to guide the generation finely. All the methods are built of the assumption that sufficient labeled data is available. However, labeled data is quite scarce in many real-world applications (Yang et al., 2019). The low-resource problem has been studied in the tasks such as machine translation (Gu et al., 2018), post tagging (Kann et al., 2018), word embedding (Jiang et al., 2018), text generation (Wiseman et al., 2018), and dialogue systems (Mi et al., 2019). To the best of our knowledge, the low-resource multi-hop QG is untouched by existing work. We thus focus on this topic and propose a method to fulfill the gap.
6 Conclusions and Future Works
We have proposed an approach to generate the questions required multi-hop reasoning in low-resource conditions. We first built a multi-hop QG model and guided it to satisfy the logical rationality by the reasoning chain extracted from a given text. In order to tackle the labeled data shortage problem, we learned the structural patterns from the unlabeled data by the hidden semi-Markov model. With the patterns as a prior, we transferred this fundamental knowledge into the generation model to produce the optimal results. Experimental results on the HotpotQA data set demonstrated the effectiveness of our approach. Moreover, we explored the generated results to facilitate the real-world application of machine reading comprehension. We will investigate the robustness and scalability of the model.
This work is supported by the Key-Area Research and Development Program of Guangdong Province (2018B010107005, 2019B010120001), National Natural Science Foundation of China (61906217,61806223,61902439,U1711262,U161 1264,U1711261,U1811261,U1811264,U1911203), National Key R&D Program of China (2018YFB1 004404), Guangdong Basic & Applied Research Foundation (2019B1515130001), Fundamental Research Funds for Central Universities (19lgpy219).
6737
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR, pages 124-131, San Diego, CA, USA.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan.
Yllias Chali and Sadid A. Hasan. 2015. Towards topic-to-question generation. Computational Linguistics, 41(1):1-20.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. In Proceedings of the 2014 EMNLP, pages 1724-1734, Doha, Qatar.
Hanjun Dai, Bo Dai, Yan-Ming Zhang, Shuang Li, and Le Song. 2016. Recurrent hidden semi-markov model. In Proceedings of the ICLR, pages 14-23, San Juan, Puerto Rico.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 NAACL, pages 4171-4186, Minneapolis, Minnesota.
Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proceedings of the 2017 EMNLP, pages 2067-2073, Copenhagen, Denmark.
Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from Wikipedia. In Proceedings of the 56th ACL, pages 1907-1917, Melbourne, Australia.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 NAACL, pages 2368-2378, Minneapolis, Minnesota.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 EMNLP, pages 866-874, Copenhagen, Denmark.
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 NAACL, pages 3168-3180, Minneapolis, Minnesota.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th ACL, pages 1631-1640, Berlin, Germany.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 EMNLP, pages 3622-3631, Brussels, Belgium.
Hafedh Hussein, Mohammed Elmogy, and Shawkat Guirguis. 2014. Automatic english question generation system based on template driven scheme. In International Journal of Computer Science Issues, pages 45-53.
Chao Jiang, Hsiang-Fu Yu, Cho-Jui Hsieh, and Kai-Wei Chang. 2018. Learning word embeddings for low-resource languages by PU learning. In Proceedings of the 2018 NAACL, pages 1024–1034, New Orleans, Louisiana.
Katharina Kann, Johannes Bjerva, Isabelle Augenstein, Barbara Plank, and Anders Søgaard. 2018. Character-level supervision for low-resource POS tagging. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, pages 1-11, Melbourne.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. In Proceedings of the Thirty-Third AAAI, pages 6602-6609, New York City, NY, USA.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR, pages 324-331, San Diego, CA, USA.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of the ICLR, pages 224-231, Banff, AB, Canada.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 EMNLP, pages 2157-2169, Copenhagen, Denmark.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain.
David Lindberg, Fred Popowich, John Nesbit, and Phil Winne. 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 105-114, Sofia, Bulgaria.
Christoper Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of the 52nd ACL, pages 55-60, Baltimore, Maryland.
6738