SlowGuess/NAACL_2022_OCR huggingface dataset:
Unsupervised Domain Adaptation for Question Generation with Domain Data Selection and Self-training
Peide Zhu and Claudia Hauff
{p.zhu-1,c.hauff}@tudelft.nl
Question generation (QG) approaches based on large neural models require (i) large-scale and (ii) high-quality training data. These two requirements pose difficulties for specific application domains where training data is expensive and difficult to obtain. The trained QG models' effectiveness can degrade significantly when they are applied on a different domain due to domain shift. In this paper, we explore an unsupervised domain adaptation approach to combat the lack of training data and domain shift issue with domain data selection and self-training. We first present a novel answer-aware strategy for domain data selection to select data with the most similarity to a new domain. The selected data are then used as pseudo in-domain data to retrain the QG model. We then present generation confidence-guided self-training with two generation confidence modeling methods: (i) generated questions' perplexity and (ii) the fluency score. We test our approaches on three large public datasets with different domain similarities, using a transformer-based pre-trained QG model. The results show that our proposed approaches outperform the baselines, and show the viability of unsupervised domain adaptation with answer-aware data selection and self-training on the QG task. The code is available at https://github.com/zpeide/transfer_qg.
Natural language Question Generation (QG) aims to generate questions from given passages of text. It has been applied to a wide range of applications, such as question answering (Sultan et al., 2020; Fabbri et al., 2020), conversational systems (Gu et al., 2021), and education (Ma and Ma, 2019; Kurdi et al., 2020). Recently, pre-trained language models (LM) have advanced the state-of-the-art across a variety of natural language processing tasks (Devlin et al., 2018). Consequently, by modeling QG as a sequence-to-sequence task and fine
(b)
Figure 1: 2D visualization of average-pool BERT hidden states of data from different domains using t-SNE. (a) Datasets NQ and RACE. (b) NQ and SciQ.
tuning on task-specific data, pre-trained LMs have substantially advanced the state-of-art performance on QG (Dong et al., 2019; Bao et al., 2020).
However, with billions of parameters, the performance of these deep neural models heavily relies on the quantity and quality of available training data. As the manual process of creating high-quality questions is expensive in terms of time and money, compared with abundant unlabeled data, the available data sources containing well-formed questions are insufficient, especially in the educational domain, where a lot of expertise is required to create questions geared towards human learning. To mitigate the lack of labeled training data, one solution is to pre-train models for QG on a data-abundant labeled domain (source domain) and transfer the learned knowledge to the unlabeled target domain, which is known as unsupervised domain adaptation (UDA) (Tan et al., 2018). It is a common challenge in machine learning research to learn knowledge in one domain and apply it in other domains with good generalization performance. One obstacle is the domain shift (Gretton et al., 2006) between the source domain and the target domain, as illustrated in Figure 1, which violates the assumption that the training set and the test set are independent and identically distributed (i.i.d.). This in turn limits the model's generalization and portability. To understand the effect of differences among domains on the performance of downstream QG tasks, fol
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2388 - 2401 July 10-15, 2022 ©2022 Association for Computational Linguistics
lowing previous research (Hu et al., 2019; Aharoni and Goldberg, 2020), we perform a preliminary cross-domain study. We first train the QG model on all domains separately and evaluate them across different domain test sets. As shown in Table 1, the model achieves the best performance on the test set from the same domain and degrades dramatically on test sets of other domains, which poses a great challenge to the transferring task. We argue that based on these numbers further research into domain adaptation methods for QG is needed. There
Dataset	NQ	RACE	SciQ
NQ	29.64	13.76	14.32
RACE	16.59	23.91	12.37
SciQ	17.36	13.02	29.47
Table 1: Impact of domain shift on QG. Each row represents the METEOR score of the UniLM (Dong et al., 2019) model trained on one dataset (the row: NQ, SciQ and RACE) and tested on the test sets (the column).
is a growing interest in applying unsupervised domain adaptation to tackle the domain shift issue in natural language processing tasks, such as question answering (QA) (Rennie et al., 2020; Cao et al., 2020), or neural machine translation (NMT) (Van Der Wees et al., 2017; Rauf et al., 2019; Hu et al., 2019). However, UDA is under-examined in the context of question generation. Unlike the QA task which can be modeled as a multi-label classification problem, QG is a sequence generation problem, where it is hard to model the confidence or quality of generations (Niehues and Pham, 2019). Therefore, UDA methods for QA like pseudo-label generation and filtering cannot be directly extended to the QG area. Moreover, data augmentation UDA methods for the NMT task, such as domain mixing (Britz et al., 2017), back-translation (Sennrich et al., 2015), or target sentences copying (Currey et al., 2017) are not directly applicable to QG.
In this paper, we propose a two-stage unsupervised domain adaptation approach for QG to make use of the labeled source domain data, and abundant unlabeled data. In the first stage, we focus on unsupervised domain data selection. Although the definition of "domain" in QG is ambiguous, including the distribution of vocabulary, stylistic preferences, answer types etc, we first confirm that the learned BERT-based context paragraph representation can be used for robust domain data clustering as shown in Figure 1, and use Gaussian Mixture Models (GMMs) on the learned representations to
find clusters, using methods proposed by Aharoni and Goldberg (2020). We perform domain data selection based on the distance of data example to cluster centers. To mitigate the gap of answer-type distributions, we further propose an answer-type aware data selection method (AADS) for pseudo in-domain data selection. The selected pseudo in-domain data are used to re-train the fine-tuned data to mitigate the domain shift.
In the second stage, we focus on self-training on the unlabeled target domain with the QG model trained in the first stage. The self-training approach is substantially hindered by noisy and low-quality generated pseudo labels. We first propose a normalization method to avoid re-enforcing poorly generated questions. We also explore using sentence perplexity and fluency scores to model the confidence of sequence generation. We filter pseudo labels with low sequence confidence during self-training to prevent the model from being degraded by wrong or low-quality predictions.
We conduct experiments across three domains, including the Natural Question dataset as the source domain, RACE as one target domain of education, and SciQ as the target domain of science. Our results show our proposed approach is effective even when the target domain is substantially different from the source domain and outperforms several baselines including Latent Dirichlet Allocation (LDA) (Druck et al., 2008), BERT discriminator based data selection (Ma et al., 2019), and unsupervised Gaussian mixture model(GMM) clustering on pre-trained language model features (Aharoni and Goldberg, 2020).
In this section, we first present a short review for UDA and question generation, then we briefly discuss how our work is different from recent related research.
The assumption that the training set and the test set are independent and identically distributed (i.i.d.) is a default assumption in many machine learning algorithms. When the underlying distributions do not match, the algorithms face the domain shift problem (Gretton et al., 2008; Ramponi and Plank, 2020), i.e. the source domain and the target domain data are not sampled from the same distribution. This issue happens in real-world scenarios, where
labeled training data are scarce while unlabeled data may be abundant since annotations are time-consuming and costly to acquire. It then translates into performance degradation. Unsupervised domain adaptation provides an elegant and scalable solution for mitigating this issue by learning only from unlabeled target data. In this paper, we focus on the data-centric methods: data selection and pseudo-labeling (Ramponi and Plank, 2020).
Data Selection for Domain Adaptation Not all samples in the source domain are equally important for adaptation. Data selection (Axelrod et al., 2011) aims to select the data that are most related to the target domain. It is attracting more attention, thanks to the abundance of data, and the large pre-trained models (Gururangan et al., 2020). It has been studied for several NLP tasks (Aharoni and Goldberg, 2020; Ma et al., 2019; Guo et al., 2020). Recently, Aharoni and Goldberg (2020) showed that sentence representation learned by pretrained language models such as BERT (Devlin et al., 2018) and Roberta (Liu et al., 2019b) are capable of clustering textual data to domains in an unsupervised way with high precision. In our work, we follow this research and perform domain clustering and selection with BERT.
Self-Training Self-training is a bootstrapping method that has been used for domain adaptation in multiple NLP tasks (McClosky et al., 2006; Chattopadhyay et al., 2012; Bhatt et al., 2015; Sachan and Xing, 2018). The main idea of self-training (Lee et al., 2013) is to predict labels for unlabeled samples with a trained classifier as their 'pseudo' ground-truth, and use the synthetic data for further training.
2.2 Question Generation
Natural Question Generation (QG) aims to generate questions from given passages. Various neural models have been proposed for QG by formulating it as a sequence-to-sequence (Seq2Seq) learning problem (Du et al., 2017; Dong et al., 2019; Bao et al., 2020). QG has been applied to a range of application areas, such as conversational QA (Wang et al., 2018; Gu et al., 2021) and education (Kurdi et al., 2020). Although these approaches have made great strides in improving QG effectiveness, they are trained and tested with data from the same dataset. When there is domain shift between training and test data, the model performance deteriorates considerably. Liao and Koh (2020) explore this using
supervised and semi-supervised domain adaptation but ignore the unsupervised setting.
The most related recent work to ours is by Kulshreshtha et al. (2021), who propose a new training protocol for UDA QG. However, it requires unlabeled questions in the target domain, which is not always available, and we focus on investigating a more effective self-training method. We compare this work in Appendix A.2.
In our work, we close the gap between source and target domain distributions by performing answer-type aware domain data selection.
We now formulate the problem and present our notation. The data in the source domain with ground-truth questions are denoted as  $\mathcal{D}_s = \{(C^s,Q^s)\}$ , while unlabeled data in the target domain is  $\mathcal{D}_t = \{(C^t)\}$ ; here,  $C$  is denoting the context (the passages, and answer spans used for generating questions). The question generation task is then to generate a sequence  $\hat{Q}$  that maximizes the conditional probability of the prediction  $P(Q|C,\theta)$ :
$$
\begin{array}{l} \hat {Q} = \arg \max _ {Q} P (Q | C, \theta) \\ = \arg \min  _ {Q} \sum_ {t = 1} ^ {T} - \log P \left(Q _ {t} \mid C, \theta , Q _ {<   t}\right) \tag {1} \\ \end{array}
$$
where  $\theta$  represents the parameters of the QG model, which is initially learned from training data in the source domain. In our work, we aim to learn to adapt the  $\theta$  from a source domain  $\mathcal{D}_S$  to the target domain  $\mathcal{D}_T$  and achieve optimal performance.
4 Domains
4.1 Source Domain
We use the open-domain question answering corpus Natural Questions (NQ) (Kwiatkowski et al., 2019) as our source domain. It consists of aggregated questions issued to the Google search engine, and answers annotated by crowd-workers from the most related Wikipedia pages. It consists of a large amount of unique passages, and covers a range of topics, which makes it a good source domain for transferring. As there are many examples in NQ with tables as context, to use this dataset for QG, we select a subset which contains 89,453 samples in the training set and 3,726 samples in the test set, from the original NQ dataset.
4.2 Target Domains
Education The first target domain we choose is education, for which we use the RACE (Lai et al., 2017) dataset. RACE is a large dataset consisting of questions, answers and associated passages in English exams for middle-school and high-school Chinese students. Questions in RACE are designed by instructors (i.e. domain experts) for evaluating students' reading comprehension ability. There are three types of questions: cloze, general and specific. Following the practice of EQG-RACE (Jia et al., 2020), we only keep the specific questions in our work. For unsupervised QG, we use 18.6K data examples in the training set. The original dev and test sets are used for evaluation.
Science Our second target domain is science where we make use of the SciQ (Welbl et al., 2017) dataset. SciQ consists of 13.7K crowdsourced multiple-choice science exam questions, including 11.7K questions in the training set, and 1K for dev and test set each. Each SciQ question has an associated passage, the right answer, and the distractors. The SciQ passages are chosen from science study textbooks of different topics including biology, chemistry, earth science and physics. For unsupervised QG, we utilize the support passages in the training set without questions as unlabeled data; we use the original dev and test sets for QG evaluation.
Table 2 lists the basic statistics of our three datasets. On those datasets, we can make a thorough evaluation of the QG model's transfer performance and the effectiveness of proposed approach.
Features	NQ	SciQ	RACE
Question	Search Logs	Crowsourced	Experts
Context	Wikipedia	Textbook	Examinations
Train set	89,453	11,679	18,614
Test set	3,726	1,000	1036
#W/doc	106.27	78.05	318
#Sent./doc	4.43	4.84	17
#W/Sent.	26.81	16.13	17.96
#W/Q	10.20	14.31	10.8
Table 2: Overview of the source domain dataset NQ, and the selected datasets for target domains SciQ and RACE.
5 Domain Data Selection
Not all data are required or even useful for domain adaptation. Irrelevant data samples can add noise,
and affect the learned model's performance and robustness towards cross-domain application considerably (Liu et al., 2019a). A solution to reduce the impact of irrelevant data is domain data selection, i.e. to retrieve the most appropriate data from the source domain data given the target domain data. Most proposed domain data selection approaches consider ranking training examples from  $\mathcal{D}_S$  according to a domain similarity measure and select the top- $n$  examples that are closest to  $\mathcal{D}_T$ .
We encode the context passage at the paragraph level with BERT, and perform average pooling of the last layer hidden state of each token to create its vector representation. To show that this is a robust representation for mapping sentences to domains in an unsupervised, data-driven approach, we first visualize them with t-SNE, as shown in Figure 1. We can observe the encoding vector representation with BERT indeed can cluster data examples to domains. Following the practice of Aharoni and Goldberg (2020), we then perform unsupervised clustering by fitting Gaussian Mixture Models (GMMs) to the vector context representations with  $k$  predefined clusters. We assign each cluster the domain class by measuring its purity (proportion of examples belonging to each domain). We use the Euclidean distance (Lee, 2001) of each example to cluster center as the measure of domain distance. Figure 2 shows the distribution of NQ dataset examples' distance to NQ's, RACE's and SciQ's domain center respectively. We sort source data examples based on their distance to the target domain center and select data examples with most domain similarity as the pseudo-in-domain data.
(b)
Figure 2: Distribution of the distance between each data example to domain cluster center. (a) NQ and RACE. (b) NQ and SciQ.
Table 3 shows the unsupervised domain clustering results. We compare the proposed methods with Latent Dirichlet Allocation-based (LDA) clustering (Druck et al., 2008). We also compare different ways of creating paragraph vector representations, including using BERT [CLS] token
encoding  $(CLS)$ , average pooling of all BERT layer hidden states  $(All)$ , and average pooling of the last hidden states  $(Last)$ . Besides GMM clustering methods, we also compare the GMM method with K-Means (KM). To accelerate the clustering, we perform PCA over the paragraph representation first. Our results show the GMM method with pooling average of the last BERT hidden states to outperform the other methods.
Method	RACE	SciQ
Acc	F1	Rc	Acc	F1	Rc
LDA	0.79	0.76	0.72	0.69	0.61	0.55
KMCLS	0.37	0.35	0.98	0.33	0.25	0.97
KMAll	0.94	0.85	0.99	0.88	0.63	0.89
KMLast	0.97	0.91	0.97	0.91	0.72	0.99
GMMCLS	0.42	0.36	0.97	0.37	0.26	0.94
GMMAll	0.96	0.90	0.95	0.88	0.64	0.89
GMMLast	0.98	0.95	0.96	0.91	0.72	0.99
Table 3: Unsupervised Domain Clustering Results.
5.1 Answer-Type Aware Data Selection
For different application domains, as shown in Figure 3a, the question type distributions vary a lot. For example, in NQ, the 'who' questions account for over  $35\%$  of all questions but in SciQ,  $73.6\%$  of questions have the 'what' type. Traditional data selection methods are based only on the similarity of context passages, which may suffer from unbalanced target label sampling. As there are no questions available in the target domain, it is a challenge to perform data selection according to the distribution of target question types. We first investigate the correlation between the answer types and question types. The question types are identified by the interrogative 'w'-word, such as 'who', 'what', etc. We identify the answer types such as 'time', 'location', etc. using the spacy<sup>1</sup> NER and POS tagger. The correlation matrix (expressed in Pearson correlation coefficient) is shown in Figure 3b. We find question types and answer types are strongly correlated to each other. For example, the correlation coefficient between 'time' and 'when' is 0.67, between 'person' and 'who' it is 0.63. Thus, we propose a heuristic answer-type aware data selection strategy for domain data selection from the source domain with a similar answer type distribution, in order to mitigate the label divergence. Specifically, we first group the data by answer types, and then conduct data selection on each group.
(a)
Figure 3: (a) Question type distributions. (b) Correlation between answer type and question types.
6 Self-Training
When training the QG model with pseudo-labels, it is natural to put more emphasis on the labels that the model is more confident about. An intuitive solution is to weigh each pseudo-token according to its estimated probability in order to avoid re-enforcing poor predictions. Thus, we propose the following normalized training objective for self-training:
$$
\hat {Q} = \arg \min  _ {Q} \sum_ {t = 1} ^ {T} - \log \alpha_ {t} P \left(Q _ {t} ^ {\prime} | C, \theta , Q _ {<   t} ^ {\prime}\right) \tag {2}
$$
where  $Q^{\prime}$  is the pseudo-label, and  $\alpha_{t}$  is the predicted probability of the  $t$ -th word  $Q_{t}^{\prime}$ , and  $T$  is the length of the pseudo-label.
We apply the QG model to generate questions on unlabeled target-domain data, which are then used as 'pseudo' gold labels for further training. The self-training approach is substantially hindered by noisy, low-quality labels. How to deal with noisy pseudo labels is crucial to the final UDA effectiveness. Classical pseudo label generation methods (Mihalcea, 2004; Abney, 2007; Cui and Bollegala, 2019) filter generated labels by their 'confidence' which is the predicted probability of the label in those classification tasks. How to represent confidence of sequence generation in pseudo-labeling is insufficiently explored. Traditionally, confidence estimation has been defined as a task of assessing the quality of the whole sequence of words in the target sentence. Therefore, we propose a question quality guided pseudo labeling method to address this problem, with two confidence metrics: (i) the sentence perplexity, and (ii) the BERT-based fluency score.
Sentence Perplexity The first metric is the perplexity of the generated questions. The generation
with higher confidence should have lower perplexity. Here, perplexity (PPL) is defined as follows:
$$
\mathrm {P P L} (Q) = 2 ^ {- \frac {1}{T} \log \prod_ {1} ^ {T} P \left(Q _ {t} \mid Q _ {<   t}\right)} \tag {3}
$$
BERT-based Fluency Score For our second metric, we use fluency as the question quality metric, which indicates whether the generation follows grammar rules and correct logic. The perplexity of a sentence under a well-trained language model usually serves as a good indicator of its fluency (Yang et al., 2018). We use a fine-tuned BERT language model as evaluator. The fluency metric  $R_{fluency}$  for question  $Q$  is calculated as follows:
$$
R _ {\text {f l u e n c y}} (Q) = \exp \left(- \frac {1}{T} \sum_ {t = 1} ^ {T} \log \operatorname {B E R T} \left(Q _ {t} \mid Q _ {<   t}\right)\right). \tag {4}
$$
During the unsupervised self-training, after each epoch, we perform beam search with the trained model, and the generated questions are ranked according to their fluency score. Only questions with confidence metrics better than the threshold  $\phi$  and  $PPL$  are selected as pseudo-labels. If one data sample got selected in the last epoch, but its generated question's confidence metric in the current epoch is not higher than before, it is removed. In this way, only questions of high quality that improve over time are chosen for training.
Algorithm 1: Self-Training
Input :Target domain data:  $\mathcal{D}_t = \{C^t\}$  .QG model  $\mathcal{M}_{QG}$  with parameters  $\theta$
In this section, we describe the model and the training regime in more detail.
QG Model We use the state-of-art pre-trained transformer-based sequence-to-sequence natural language understanding and generating model UniLM (Dong et al., 2019) for question generation. Specifically, we choose the uncased pretrained unilm1.2-base-uncased model for fine-tuning. It has 12 transformer layers and is jointly pre-trained on large amounts of text, optimized for bidirectional, unidirectional, and sequence-to-sequence language model objectives. We use the s2s-ft package for fine-tuning. To fine-tune our model, the input context passage, the answer, and the generated question are combined together into a sequence: "[CLS] context passage [EOS] answer span [EOS] question [EOS]". Both the input passage and answer are regarded as the first text segment, while the generated question is the second segment in the unified LM.
Training Details The model is trained on a server consisting of 4 GeForce GTX 1080 gpus with a batch size of 32, a mask probability of 0.8, and the label smoothing rate of 0.1. The max_source_seq_length is set 464, the max_target_seq_length is 48. We first fine-tune UniLM with the NQ dataset for 10 epochs. We use the Adam optimizer with  $\epsilon = 1e - 8$ , learning rate is  $1e - 4$  with 500 warmup steps.
Unsupervised Domain Data Clustering We use 4,500 examples randomly selected from NQ, SciQ and RACE for unsupervised data clustering. We set the number of clusters as 2, since we intend to investigate the separability between the source domain and the target domain.