SlowGuess/NAACL_2022_OCR huggingface dataset:
Wanjun Zhong $^{1*}$ , Yifan Gao $^{4*}$ , Ning Ding $^{3}$ , Yujia Qin $^{3}$ , Zhiyuan Liu $^{3}$ , Ming Zhou $^{5}$ , Jiahai Wang $^{1}$ , Jian Yin $^{1}$  and Nan Duan $^{2}$
$^{1}$  Sun Yat-sen University  $^{2}$  Microsoft Research Asia
$^{3}$  Tsinghua University  $^{4}$  Chinese University of Hong Kong  $^{5}$  Langboat Technology  $\{\text{zhongwj25}@mail2, \text{wangjiah}@mail, \text{issjyin}@mail}$ .sysu.edu.cn yfgao@cse.cuhk.edu.hk; liuzy@tsinghua.edu.cn  $\{\text{dingn18}, \text{qyj20}\} \text{@mails.tsinghua.edu.cn}$  nanduan@microsoft.com; zhouming@chuangxin.com
Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.<sup>1</sup>
Question Answering has long been an inspirational challenge in NLP research, and is viewed as the next-generation search engine and an essential tool for human beings to obtain knowledge (Etzioni, 2011). Many distinct datasets (Rajpurkar et al., 2016; Lai et al., 2017; Kwiatkowski et al., 2019; Gao et al., 2021) have been proposed along with the research trend on QA, involving very diverse
question types (e.g., extractive QA, abstractive QA, multiple-choice QA), domains (e.g., finance, daily events), and answer types (e.g., free-formed text, selected option). The majority of previous works focus on tasks with specific question types (Lai et al., 2017; Yang et al., 2018; Gao et al., 2020) or specific domains (Trischler et al., 2017; Kwiatkowski et al., 2019). Recent research on large pre-trained language models (Brown et al., 2020; Bommasani et al., 2021) indicates that there may be tight connections among various tasks, which sheds light on a unified paradigm that can be potentially applied to solve various QA tasks to model their commonality.
This observation motivates us to develop a unified QA model, which can model both the commonly-required QA ability and the difference between various QA tasks within a same paradigm. To achieve this goal, there are several key challenges needed to be addressed: (1) How to model commonalities and enhance transferability among different QA tasks in various domains/formats while reducing the conflict between them? (2) How to construct large-scale QA corpus as the high-quality QA-centric data is scarce for pre-training?
In light of this, we conceive ProQA, a unified QA paradigm, which builds up a general model to solve different QA tasks utilizing a structural prompt and improves commonly-required QA ability via structural prompt-based pre-training.
Firstly, to model the commonalities and distinguish task differences, we adopt a structural prompt to organize the inputs with a unified structurally designed input schema. As illustrated in Fig. 1, given the complex components (e.g., "Domain", "Format", "Task", "Question", "Passage") as inputs, ProQA divides components into multiple key-value pairs, in which a specific component like "Question" denotes a key, and the specific in
* Indicates equal contribution
<sup>1</sup>The code is available at https://github.com/zhongwanjun/ProQA.
Human Language Technologies, pages 4230 - 4243
Figure 1: Approach overview of ProQA. Each box represents a specific instance formulated with the structural prompt, and ProQA is pre-trained with structural prompt-based pre-training. [ ] indicates special key indicator, < > denotes hard prompt, and colored squares denote continuous learnable soft prompts.
stance in this component is taken as the value. In this way, the model can discriminate different input components by key indicators and model the speciality of each task via task-specific values (learnable prompts).
Secondly, to alleviate data sparsity problem and empower the model with transferability to the adaptation of new tasks, we conduct structural prompt-based pre-training. We first build a large-scale synthetic QA corpus automatically from Wikipedia, utilizing only a few seed datasets as the prior supervisions for pre-training corpus construction and finally covering primary QA formats. Then we format the pre-training data with the structural prompt, and teach the model to learn the general purpose QA-centric ability and the functionality of each component in the structural prompt via pre-training.
We evaluate the effectiveness of ProQA on 11 downstream QA benchmarks, and the results show that our system achieves consistent performance boost in full data fine-tuning, few-shot learning, and zero-shot learning settings. Experiments demonstrate that ProQA can better mitigate the
catastrophic forgetting issue during continual learning by restoring the task-specific soft prompts residing in the structural prompt. Further analyses illustrate that our model has better transferability as it can be more quickly adapted to a newly involved task. Ablation studies verify the effectiveness of both the soft prompt and prompt-based pre-training.
- We propose ProQA, a unified QA framework for solving various tasks within a single paradigm, taking an extensible and learnable structural prompt as the bridge.
- We enhance general QA-centric capabilities via structural prompt-based pre-training.
- Comprehensive experiments show that our model consistently improves the performance on 11 QA tasks especially in low-resource settings and exhibits better effectiveness in continual learning and few-shot transfer learning.
Unifying QA formats. Despite vast diversity of current QA tasks in question type, answer type, an-
swer source, and data domain (Zeng et al., 2020), there have been efforts in exploring a unified format for various QA tasks. Some pioneered to demonstrate the generalization and transferability among different QA tasks (Talmor and Berant, 2019; Dua et al., 2019a; Fisch et al., 2019). Another line of works investigate multi-task learning for QA (McCann et al., 2018; Shen et al., 2019; Deng et al., 2019) by jointly training a single encoder to promote knowledge sharing. However, these methods typically require deploying distinct prediction heads for different tasks, which lead to poor scalability and flexibility when confronted with emerging QA tasks of new types.
To this end, inspired by the success of casting multiple tasks into the same text-to-text format (Lewis et al., 2020; Raffel et al., 2020), researchers propose to learn a single model to unify various QA formats, alleviating the labor of task-specific designs (Khashabi et al., 2020b; Tafjord and Clark, 2021). However, these models (1) do not explicitly model the task or component characteristics, thus failing to properly disentangle the difference among QA tasks; and (2) overly rely on supervised data from specific tasks, which may not be available under data-scarce scenarios.
QA-centric pre-training. Numerous efforts have been spent on improving PLMs' reasoning abilities with an intermediate pre-training stage before fine-tuning on target QA tasks, including (1) language modeling adaptation with salient span masking, which trains PLMs to recover randomly chosen (Guu et al., 2020; Wang et al., 2021) or machine-generated (Kang et al., 2020) masked named entities in the raw corpus; (2) training data augmentation (Zhong et al., 2022) with synthetic question-answer-context triples, such as generating (a) pseudo questions through adversarial training (Hosking and Riedel, 2019; Li et al., 2019), knowledge bases (Hu et al., 2021) or machine translation (Lewis et al., 2019; Li et al., 2020), (b) pseudo answers exploiting recurring spans (Ram et al., 2021) or rules based on heuristics (Bian et al., 2021) and (c) pseudo contexts via information retrieval (Glass et al., 2020). Nevertheless, these works largely target at improving a certain reasoning ability for PLMs, and thus cannot be easily generalized to other QA tasks.
Prompts for PLMs. To effectively stimulate the knowledge acquired through pre-training, prompt-
oriented fine-tuning is receiving increasing attention (Liu et al., 2021; Ding et al., 2021), which re-formulates the objective of downstream tasks similar to that of pre-training by inserting manually designed (Schick and Schütze, 2021a,b) or automatically searched (Jiang et al., 2020; Shin et al., 2020) hard prompt tokens into the input text. Considering that discrete prompts may not be an optimal solution in the continuous embedding space, recent works (Li and Liang, 2021; Hambardzumyan et al., 2021) proposed tunable soft prompts. It achieves satisfying performance especially when the model size grows extremely large (Lester et al., 2021). Compared with the cumbersome parameters in PLMs, soft prompts are lightweight and pluggable, which paves the way for our goal of flexible adaptation to a new QA task.
3 ProQA
In this section, we dually describe the whole framework of ProQA for general purpose QA, which solves various QA tasks within the same paradigm.
As shown in Fig. 2, we organize the inputs of various QA tasks with a unified structural prompt (§ 3.2), and adopt a unified model for question answering. Then, to enhance the model in learning the QA-centric ability and the semantics of the structural prompt, we conduct structural prompt-based pre-training with synthetic pre-training corpus formatted with the structural prompt (§ 3.3).
Inspired by Khashabi et al. (2020b) and T5 (Raffel et al., 2020), we solve all downstream QA tasks with a unified text-to-text model. In this work, we mainly adopt T5 as the model backbone. Taking the structural prompt-based model input, the unified model generates the answer of the question.
3.2 Structural Prompt
Here we dually illustrate the design of the structural prompt and its formatted input to the model.
Definition. We organize complex QA task inputs with the structural prompt. As shown in Fig. 2, the structural prompt consists of multiple  $\{key: value\}$  pairs, where the key represents a specific component<sup>2</sup> (e.g., "Task", "Format",
2It is worth noting that "Format" key denotes the format type (e.g., "MultiChoice QA") of the task while the "Task" key denotes a specific dataset (e.g., "SQuAD").
Figure 2: An illustration of the structural prompt. [ ] indicates special key indicator,  $<$  denotes hard prompt, and grey squares indicate continuous soft prompts.
"Question", etc.), and the value has two possible types: (1) textual content (e.g., question, passage, options) of the data instance; (2) task attributes (e.g., format, domain) represented as the combination of a discrete hard prompt and continuous soft prompts. The hard prompt is a predefined discrete description (we adopt a special token here), and the soft prompts are lightweight learnable and pluggable continuous embeddings that are proven to be parameter-effective in task adaptation (Lester et al., 2021). The structural prompt-formatted examples are illustrated in Fig. 1. In the case of the SQuAD dataset, " ⟨Format Prompt⟩", " ⟨Task Prompt⟩", " ⟨Domain Prompt⟩" will be " ⟨Extractive QA⟩", " ⟨SQuAD⟩", " ⟨Wikipedia⟩", respectively.
To enhance the model in discriminating the functional difference between components, we adopt a special key indicator with learnable representation to represent each key. Furthermore, to model the difference between several tasks/domains/formats, we also adopt learnable and storable specific soft prompts as the value to represent their customized characteristics, which makes the model more flexible for task adaptation.
As a result, the structural prompt can empower the model in the following aspects: (1) modeling knowledge generalization of various tasks utilizing a unified input schema; (2) discriminating different components with the special key indicator; (3) customizing the speciality of each task/format/domain with learnable and storable soft prompts as the value under corresponding keys.
Input Representation. Specifically, given a structural prompt-formatted instance, we describe
the specific representation of the model input. We firstly translate  $k^{th}$  key to a key indicator  $D_{k}$  (a special token), which is attached by the tokens  $V_{k}$  of the specific value to form a token sequence. It is further represented as  $E_{k} = \text{Embedding}([D_{k};V_{k}])$ . The representation of  $D_{k}$  is initialized and updated during training. Since we use soft prompts  $P_{\text{task}} / P_{\text{format}} / P_{\text{domain}}$  as the value of the corresponding key and they are commonly required for all the tasks, we prepend them to the input for convenience and concatenate all the  $E_{k}$  to form the final model input  $X$ :
$$
\boldsymbol {X} = \left[ \boldsymbol {P} _ {\text {d o m a i n}}; \boldsymbol {P} _ {\text {f o r m a t}}; \boldsymbol {P} _ {\text {t a s k}}; \boldsymbol {E} _ {1}; \dots ; \boldsymbol {E} _ {k} \right] \tag {1}
$$
It is also worth noting that the representations  $D$  of key indicators and the soft prompts  $P$  are jointly trained with the main model parameters during pretraining for learning the semantics of the structural prompt. Moreover, after being tuned by various tasks, the soft prompts  $P$  can be stored to record the customized task-specific characteristics.
3.3 Structural Prompt-based Pre-training
In this part, we introduce how we conduct structural prompt-based pre-training to help the model in learning commonly-required QA ability and the semantics of the structural prompt during pre-training to facilitate the adaption of the structural prompt to downstream tasks.
Task Formulation. Along with the structural prompt-based paradigm, we manifest various exemplary QA format types (i.e., Extractive QA, Abstractive QA, Multiple-choice QA and Yes/No QA) for pre-training to inject the general QA-centric ability. Given the multi-format QA pre-training corpus, we transform all QA formats according to the proposed structural prompt, which enables joint pretraining while keeping the differences among various formats. Taking a structural prompt-formatted instance as the input and a free-form answer as the output, the task is further tailored to a QA task with the encoder-decoder model.
Pre-training Corpus Construction. When we prepare the QA pre-training corpus, data sparsity problem is extremely severe because (1) it is impractical and laborious to obtain a large-scale high-quality annotated data for pre-training and (2) it is hard to generate QA-centric self-supervised data using rule-based methods (e.g., token masking or sentence reordering). In this work, inspired
by Lewis et al. (2021), we adopt a generation-filtering based corpus construction method to synthesize a large-scale pre-training corpus, based on a large-scale unlabeled Wikipedia corpus with almost 6 million passages.
Typically, the general generation-filtering process consists of the following components:
1. A QA-pair generation model  $g_{qa}(q, a|c)$ : Given a passage  $c$  as input,  $g_{qa}(q, a|c)$  generates  $q$  [SEP]  $a$  as the output sequence including a pair of question  $q$  and its answer  $a$ .
2. A filtering QA language model  $f(a|q, c)$  for filtering the generated QA-pairs to ensure the quality and consistency of the question and the answer.  $f(a|q, c)$  is a conditional-probability-based approach to filter out QA pairs softly. It scores a QA pair  $(q, a)$  with the likelihood of the answer  $a$  conditioned on the passage  $c$  and question  $q$ . The QA-pairs with scores higher than a threshold will be kept for pre-training.
We adopt the same text-to-text pre-trained model T5 described in § 3.1 as the model backbone of both the generation and filtering model.
To ensure the reliability of the generation and filtering models, we inevitably select a few seed datasets (typically one for each QA format type) as the prior supervisions to train these models. It is worth mentioning that, we avoid using more supervised data for corpus construction, because we expect the whole paradigm to have better expandability. In other words, if we want to extend the paradigm for a newly-involved QA format type but with limited supervised data, we can utilize these data to automatically create a synthetic large-scale pre-training corpus.
More specifically, the construction method has little variance for different formats according to their input components. For Extractive QA and Abstractive QA, we adopt the aforementioned general method to synthesize QA-pairs. We also tried to first extract answers using rule-based method (extracted named-entities or key phrases), and only generate questions. We empirically find that this method performs much worse as it involves simple bias of the rule-based method. As the inputs for Multiple-Choice QA involve a new component "Candidate Answers", we adopt a distractor (negative options) generation model  $g_{\text{neg}}(o|c,q,a)$  to generate three negative options  $o$ . For Yes/No QA, we simply generate questions by taking True/False as the corresponding answers. Further details are
Format	Dataset	#Train	#Dev	QA Skills
Extractive QA	SQuAD*	87k	10k	Word Matching
Quoref	22k	2k	Coreference Reasoning
NewsQA	76k	4k	Word Matching
Abstractive QA	NarQA*	65k	21k	Story Understanding
DROP	77k	9k	Discrete Reasoning
NQOpen	79k	3.6k	Multi-passage Understanding
MultiChoice QA	RACE*	87k	4k	Multi-sentence Reasoning
DREAM	6k	2k	Dialog Reasoning
MCTest	1.4k	320	Multi-sentence Reasoning
OBQA	4k	501	Common Knowledge
SIQA	33.4k	2.2k	Commonsense Reasoning
Table 1: Dataset statistics and required language understanding skills. Datasets with * denote seed datasets for preparing pretraining data.
described in Appendix A.
We consider three formats of QA datasets in our experiments: Extractive QA, Abstractive QA and Multiple-Choice  $\mathrm{QA}^3$ . For each QA format, we select one seed dataset for preparing the large-scale pre-training data. The seed dataset is used to train the question-answer generation and filtering models in the process of pre-training corpus construction. In total, the experiments are conducted on 11 QA datasets with three different formats and various language understanding abilities. An overview of datasets used in the experiments and their required QA skills are summarized in Table 1.
Extractive QA. We take SQuAD 1.1 (Rajpurkar et al., 2016) as the seed dataset for extractive style QA. In addition, we consider NewsQA (Trischler et al., 2017) and Quoref (Dasigi et al., 2019) to evaluate the generalization ability of models. The EM (Exact Match) score between the extracted span and the gold answer span is used as the evaluation metric for extractive QA.
Abstractive QA. Narrative QA (NarQA) (Kočisky et al., 2018) is taken as the seed dataset for Abstractive QA. DROP (Dua et al., 2019b) and the open-domain version of NaturalQuestions (NQOpen) (Kwiatkowski et al., 2019) are also considered. Passages for each question in NQOpen are retrieved by the dense passage retriever (Karpukhin et al., 2020) and are concatenated into a sequence.
3We also include Yes/No QA in our pilot study. We do not consider it in our main experiments because datasets in this format are extremely rare. Results on this QA formats are shown in Appendix B.
Setting	Dataset	ExtractiveQA	AbstractiveQA	MultiChoiceQA	Avg
SQuAD	Quoref	NewsQA	NarQA	DROP	NQOpen	RACE	DREAM	MCTest	OBQA	SIQA
Full-Data	T5	83.4	64.9	45.2	49.3	45.0	42.3	67.9	54.8	44.4	49.6	64.1	55.5
UnifiedQA	84.4	74.8	45.3	49.6	45.1	42.5	71.6	67.6	83.1	57.6	64.9	62.4
ProQA (qapair)	84.9	76.6	50.8	49.8	55.0	43.2	73.6	72.9	85.0	61.6	67.5	65.5
ProQA (paq)	85.3	76.8	50.4	50.1	52.5	43.9	73.2	73.3	85.9	61.4	67.2	65.5
Few-Shot	T5	6.7	14.6	20.5	3.4	5.8	11.9	26.2	34.7	38.1	29.0	32.4	20.3
UnifiedQA	82.0	38.2	34.2	49.1	22.2	31.6	53.0	57.4	73.8	41.2	42.8	48.1
ProQA (qapair)	82.9	44.2	41.1	49.1	24.9	33.3	63.4	64.5	82.5	46.2	49.1	52.8
ProQA (paq)	84.4	52.2	42.1	49.2	27.1	36.0	66.5	66.0	84.1	44.8	49.4	54.7
Zero-Shot	T5	0.0	0.0	0.0	3.5	2.0	1.5	24.1	34.2	27.5	21.9	33.2	13.5
UnifiedQA	80.7	27.9	31.4	48.3	18.0	30.9	53.0	57.0	73.4	35.9	40.3	45.2
ProQA (qapair)	80.4	30.5	30.7	48.1	17.0	33.0	62.6	64.3	81.3	36.0	47.2	48.3
ProQA (paq)	81.3	42.1	31.8	48.4	19.7	36.0	65.9	65.2	81.3	38.6	46.7	50.6
Table 2: Main results on 11 downstream QA datasets under full-data fine-tuning, few-show learning, and zero-shot learning settings. Since the supervisions of seeds datasets are used in the pre-training corpus construction which may introduce bias in few-shot and zero-shot settings, results on these corresponding entries are underlined.
We use ROUGE-L (Lin, 2004) metric for NarQA and F1 score for DROP and NQOpen.
Multiple-Choice QA. For multiple choice QA, the following datasets are considered: RACE (Lai et al., 2017) (seed dataset), DREAM (Sun et al., 2019), MCTest (Richardson et al., 2013), OpenBookQA (OBQA) (Mihaylov et al., 2018), Social IQa (SIQA) (Sap et al., 2019). OBQA does not have contexts (reading comprehension passages). The context for DREAM is in the dialogue style and we concatenate them into a sequence as the passage input. We select the option with the highest textual similarity with the generated answer as the final answer. We compute the accuracy of the correct options for all multiple choice QA datasets.
4.2 Approaches
T5 (Raffel et al., 2020) is a unified text-to-text pre-training framework that covers all text-based language problems. We use google/t5-v1_1-base from HuggingFace Transformers (Wolf et al., 2020) that is only pre-trained on C4 excluding any supervised training dataset (e.g., QA datasets).
UnifiedQA (Khashabi et al., 2020b) crosses the format boundaries of different QA tasks by formulating them into text-to-text tasks under T5. It directly concatenates all inputs via  $\backslash n$  into a sequence and feeds it into T5 for predicting the answer. We train our own UnifiedQA model on the combination of three aforementioned seed datasets, namely SQuAD, NarQA, and RACE.
ProQA is our proposed structural prompt-based pre-training approach. ProQA is pre-trained
jointly on three formats of pre-training corpus: Extractive QA, Abstractive QA, and Multiple-Choice QA. This approach using corpus prepared from QA-pair generation-filtering model described in § 3.3 is named as ProQA (qapair). Additionally, we leverage the off-the-shelf large-scale QA pairs from Probably-Asked Questions/PAQ (Lewis et al., 2021), and replace our extractive QA pre-training corpus by a subset of PAQ (abstractive QA and multiple-choice QA corpus remains unchanged). PAQ provides a refined pipeline that introduces learned models on every step of QA pair generation, i.e., passage selection, answer identification, question generation, and filtering. We name this variant as ProQA (paq).
For every downstream QA dataset, we start from the above pre-trained models and conduct experiments under full-data fine-tuning, few-shot learning, and zero-shot learning settings. For few-shot learning, we randomly sample 32 instances from the training set.
5 Results and Analyses
Main results are shown in Table 2, and we have the following observations:
- QA-centric pre-trained models, namely UnifiedQA and ProQA, outperform T5 by a large margin on both seed datasets and non-seed datasets. This is because there is some transferable knowledge across different QA tasks. Once the model is pre-trained by any QA task, the learned knowledge can be generalized to any other datasets.
- ProQA demonstrates better knowledge cus