SlowGuess/NAACL_2022_OCR huggingface dataset:
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231.
Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with macaw. ArXiv preprint, abs/2109.02593.
Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911-4921, Florence, Italy. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Cunxiang Wang, Pai Liu, and Yue Zhang. 2021. Can generative pre-trained language models serve as knowledge bases for closed-book QA? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3241-3251, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, and Jianjun Hu. 2020. A survey on machine reading comprehension—tasks, evaluation metrics and benchmark datasets. Applied Sciences, 10(21):7640.
Wanjun Zhong, Junjie Huang, Qian Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. 2022. Reasoning over hybrid chain for table-and-text open domain qa. ArXiv preprint, abs/2201.05880.
Wanjun Zhong, Duyu Tang, Jiahai Wang, Jian Yin, and Nan Duan. 2021. UserAdapter: Few-shot user learning in sentiment analysis. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 1484–1488, Online. Association for Computational Linguistics.
A.1 Corpus Preparation
In this part, we describe the details of corpus construction.
The current pre-training corpus contains almost 4 million pre-training instances formulated with the structural prompt, including 1 million Multiple-choice QA instances, 2 million Extractive QA instances, and 2 million Abstractive QA instances. When generating questions and answers, we take the context as the input, and the sequence "question [SEP] answer" as the output. In order to train the filtering model, we take the context and question as the inputs, and the answer as the output. During the inference process of QA-pairs filtering, we take the context and the generated question as the model input of the QA model and set generated answer as the label. Then we compute the final soft score with the cross-entropy loss between the label and the answer generated by the QA model, Next, we rerank all the generated QA-pairs according to the soft scores in an ascending order to select the most consistent QA-pairs as the pre-training instances.
Specifically, we employ AdamW as the optimizer for model training. We adopt T5-Large as the model backbone and the seed datasets as supervisions for training both the question-answer pairs generation, and the filtering QA model. We set learning rate as 1e-5, warmup step as 0, batch size as 2 per GPU, and training epochs as 10.
A.2 Details on Pre-training and Task Adaptation.
Pre-training. During pre-training, we jointly train the main model parameters with the representations of the special key indicators and the task/format-specific soft prompts.
Initially, we don't have any specific tasks during pre-training, so we take the three pre-training corpus (i.e., "MultiChoiceQA, Extractive QA, and Abstractive QA") as the three initial tasks, and randomly initialize the task and format specific soft prompts.
Specifically, we use T5-Base as the model backbone, and set learning rate as 1e-4, batch size as 8 per GPU and gradient accumulation steps as 10. We adopt 8 V100 GPUs for pre-training.
Fine-tuning. During fine-tuning, we need to initialize the task/format-specific soft prompts for a specific downstream task. If the task corresponds
Setting	Dataset	BoolQ
Full-Data	T5	62.2
Pro QA	80.6
Few-Shot	T5	0.0
Pro QA	55.4
Zero-Shot	T5	0.0
Pro QA	62.1
Table 5: Result on two Yes/No QA tasks under full-data fine-tuning, few-shot learning, and zero-shot learning settings.
to a specific format participating in the pre-training stage, we use the corresponding soft prompts of this format type to initialize the soft prompts for the current tasks to transfer the learned knowledge. If the task corresponds to a new format, we can randomly initialize the task/format prompts.
Specifically, we use T5-Base as the model backbone, and set learning rate as 1e-4, batch size as 2 per GPU, gradient accumulation steps as 2, and training epochs as 5. We adopt 8 V100 GPUs for fine-tuning.
Few-shot Learning. We adopt a similar way to initialize the task-specific soft prompts for few-shot learning. We use the standard setting which utilizes 32 randomly selected instances for few-shot learning. Specifically, we adopt T5-Base as the model backbone, and set learning rate as 1e-5, batch size as 1 per GPU, gradient accumulation steps as 1, and training steps as 800 for few-shot learning.
Zero-shot Learning Since zero-shot learning does not involve training stage, we just need to initialize the task-specific prompt for inference. Therefore, we initialize the task-specific prompt with the pre-trained task prompts of its corresponding format type.
B Results on Yes/No Pre-training
During our pilot study, we take the BoolQ (Clark et al., 2019) as the seed dataset to construct a large-scale pre-training corpus, and test the full-data, few-shot, zero-shot on top of the pre-trained ProQA. We also take the naturally-perturbed version of this dataset BoolQ-NP (Khashabi et al., 2020a) into account for evaluation. Results are shown in Table 5. We find that the ProQA significantly outperforms T5 baseline on all settings. Note that we take a strict evaluation towards the model's output.
Task A	Task A Model (Few-Shot Results)	Task B	Task B Model (Evaluation on Task A)	Task B Model (w/ Task A Prompt) (Evaluation on Task A)
EX/NewsQA (EM)	42.1	EX/Quoref (EM)	33.7	35.3
AB/DROP (F1)	31.3	33.7
AB/NQOpen (F1)	27.6	34.2
MC/DREAM (Acc)	35.2	37.9
MC/MCTest (Acc)	34.5	36.8
MC/OBQA (Acc)	34.4	35.9
EX/Quoref (EM)	52.2	EX/NewsQA (EM)	41.1	42.8
AB/DROP (F1)	43.0	51.4
AB/NQOpen (F1)	38.0	51.0
MC/DREAM (Acc)	47.2	51.3
MC/MCTest (Acc)	47.8	52.0
MC/OBQA (Acc)	48.5	51.6
AB/NQOpen (F1)	36.0	EX/NewsQA (EM)	32.8	33.9
EX/Quoref (EM)	36.2	35.4
AB/DROP (F1)	34.3	35.8
AB/DROP (F1)	27.1	EX/NewsQA (EM)	22.8	26.3
EX/Quoref (EM)	24.0	26.6
AB/NQOpen (F1)	25.0	25.6
MC/DREAM (Acc)	25.1	26.2
MC/MCTest (Acc)	25.4	26.7
MC/OBQA (Acc)	25.6	26.7
MC/MCTest (Acc)	84.1	EX/NewsQA (EM)	81.6	83.1
AB/DROP (F1)	82.8	83.8
MC/DREAM (Acc)	83.2	83.8
MC/OBQA (Acc)	82.2	82.5
MC/DREAM (Acc)	66.0	EX/NewsQA (EM)	64.0	65.4
AB/DROP (F1)	63.5	65.7
MC/MCTest (Acc)	65.5	65.8
MC/OBQA (Acc)	65.2	65.5
MC/OBQA (Acc)	44.8	EX/NewsQA (EM)	41.4	45.2
AB/DROP (F1)	40.6	44.2
MC/MCTest (Acc)	44.8	45.8
MC/DREAM (Acc)	44.2	46.8
Table 6: Full results on continual learning. For each task, we provide the task format (EX, AB, MC) and its evaluation metrics (EM, F1, Acc). EX: Extractive QA; AB: Abstractive QA; MC: Multiple-Choice QA.
In other words, if the output is not any format of "yes", "no", "true", "false", that prediction will be classified as wrong.
performance when a bunch of tasks arrive.
C Details on Continual Learning
Table 6 provides the full results for the continual learning experiment. The model is firstly trained on task A under few-shot setting, and then fine-tuned on task B. Afterwards, we evaluate the trained "Task B Model" and "Task B Model (w/ Task A Prompt)" on task A to test its continual learning capability. Detailed results on every Task A/Task B combination (33 reported in total) are shown in Table 6. Note that we consider two tasks in continual learning because we also want to investigate the task adaptation to-and-fro the same format (e.g.,  $\mathrm{MC}\rightarrow \mathrm{MC}$ ) or different formats (e.g.,  $\mathrm{AB}\rightarrow \mathrm{EX}$ ). The results shed light on how could we arrange the order of training on tasks to achieve the best overall