Improving Low-resource Question Answering by Augmenting Question Information
Authors: Andong Chen
Yuan Sun
Xiaobing Zhao
Rosella P. Galindo Esparza
Kehai Chen
Yang Xiang
Tiejun Zhao
Min zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Question Answering
Data augmentation
Low-resource domains

Abstract: In the era of large models, low-resource question-answering tasks lag, emphasizing the importance of data augmentation - a key research avenue in natural language processing. The main challenges include leveraging the large model's internal knowledge for data augmentation, determining which QA data component - the question, passage, or answer - benefits most from augmentation, and retaining consistency in the augmented content without inducing excessive noise. To tackle these, we introduce PQQ, an innovative approach for question data augmentation consisting of Prompt Answer, Question Generation, and Question Filter. Our experiments reveal that ChatGPT underperforms on the experimental data, yet our PQQ method excels beyond existing augmentation strategies. Further, its universal applicability is validated through successful tests on high-resource QA tasks like SQUAD1.1 and TriviaQA.

Improving Low-resource Question Answering by Augmenting Question
Information
Andong Chen Yuan Sun
Xiaobing Zhao Rosella P. Galindo Esparza .
Kehai Chen",, Yang Xiang* Tiejun Zhao Min zhang!
Harbin Institute of Technology, Harbin, China
< Minzu University of China, Beijing, China
Pengeheng Laboratory, Shenzhen, China
Brunel University Uxbridge, London, UK
Harbin Institute of Technology (Shenzhen), Shenzhen, China
Abstract
How can we strike balance between maintaining the consistency of the augmented content and not introducing excessive noise? Despite significant advancements in the field, many of these questions remain unanswered. For instance, The multi-modal QA benchmark, MAQA (Li et al., 2023), accentu- ates the need for data augmentation using negation examples in QA tasks but doesn't focus leverag- ing the knowledge of large models. The fill-in-the- blank QA framework, Gotta (Chen et al., 2023b), leverages generative, prompt-based augmentation methods to enhance learning. However, it does not identify the most beneficial segments of the QA data for improvement. Lastly, the label-guided data augmentation framework PROMPTDA (Chen and Shu, 2022) utilizes label semanties for data aug- mentation in few-shot learning, however, over- looks the task of balancing content consistency with noise introduction. Existing augmentation techniques, such as template- based question gener- ation (Ali et al., 2010: Heilman and Smith, 2009; Chali and Hasan, 2015), and the broad application of Seq2Seq models in natural language process- ing (Bahdanau et al., 2014; Du et al., 2017), have made certain progress but also show similar limita- tions.
In the of large models, low-resource question-answering answering tasks lag, emphasizing the importance data augmentation. The main challenges include leveraging the large model's internal knowledge for data augmentation, de- termining which QA data component the ques- tion, passage, or answer benefits most from augmentation and retaining consistency the augmented content without inducing excessive noise. To tackle these, we introduce PQQ, an innovative approach for question data augmen- tation consisting of Prompt Answer, Question Generation, and Question Filter. Our experi- ments reveal that ChatGPT underperforms on the experimental data, yet our PQQ method ex- cels beyond existing augmentation strategies. Further, its universal applicability is validated through successful tests on high-resource QA tasks like SQUADI.1 and TriviaQA !
1 Introduction
Low-resource issues pose significant challenge within the field of Question-Answering (QA) tasks. These problems largely arise from data scareity and lack of domain-specific training, which often results in underfitting. Even though we are now in the era of large models, which have consider- ably advanced the field of natural language process- ing, these low-resource problems in QA tasks per- sist (Wanjawa al., 2022; Sun et al., 2021b; Chen et al., 2023a). Consequently, addressing these low- resource issues has become an essential research focus, with data augmentation strategies emerging as common and effective approach.
To address these challenges, we propose novel and straightforward question data augmentation method, termed PQQ. Firstly, we construct prompt templates to generate answers that are related to the original ones but expressed differently (Prompt An- swer). Then, a diverse set of questions is produced by question generation model (Question Gener- ation). Lastly, using a question filter, we retain only those questions that are logically related to the original QA data as the augmented data (Question Filter). Our experimental findings indicate dis- tinct underperformance of ChatGPT on our ex- perimental dataset, while our proposed methodol- ogy, PQQ, exhibits a superior performance, outper- forming contemporary mainstream data augmen-
However, current research in data augmentation presents several unanswered questions: How can the internal knowledge of large models be har- nessed to enhance the quality of augmented data? Which component of QA data the question, pas- sage, or answer benefits most from augmentation?
*Conresponding author. https://gthhhb.com///dnnnggeeee

---

Discard
Original Passage
Wher doea Beyonce parcontt
<passage, answer, question>
[CL-gpuustion & Whal arses Bepenco corpeete
Talent
X
Question
growing Pastge is:
Vocal
Generstion
great? What Btyonce's
quaraags. anamer
[MASK] anring dinccnng) compertions as onio.. (SEP)
Artali
Model
026022n7
noW questions
Prompled Answers
Original Answer
Quesiion
Augumented Dataset
Stsp I: Prompt Answer
Step2: Question Generation
Step3: Question Filter
Figure I: An overview of our approach where the Bert-large output Top k prompted answers based original QA data, using QG model to obtain generated questions by prompted answers and corresponding passage, the Bert is employed as question filter to retain questions related to the original dataset.
tation techniques. We further validate the univer- sality of our PQQ method, investigating its effi- cacy on high-resource question-answering (QA) tasks such SQUADI.1 and TriviaQA (Rajpurkar al., 2016; Joshi al., 2017). Remarkably, by ploying Bert-base model for QA tasks, PQQ- augmented upproach surpasses the performance of a Bert- model without PQQ augmentation. demonstrating the potent efficacy of our methodol- ogy.
enabling the creation of contextually varied ques- tions.
2.2 Questions Generation
Once the Prompt Answer is obtained, the next step involves constructing a question generation model to generate a diverse range of question data. Since different QA tasks encompass varying do- main knowledge (e.g., TechQA includes education domain knowledge while PolicyQA focuses le- gal domain knowledge), it necessary to fine-tune a specific question generation model for each QA dataset. In our study, we selected the base model (Raffel et al., 2019) as the question gen- cration model.
2 PQQ Apporach
This section succinetly outlines the PQQ data aug- mentation approach. This process, illustrated in Figure 1, consists of three main steps.
To fine-tune the question generation model, we utilize passages and answers from the QA dataset as inputs the T5 model. The format of the input as follows: Input <answer></>> <passagg>>>.s>>>>> During the fine- tne-tunnng process of the T5 model, the loss function employed is negative log-likelihood estimate.
2.1 Prompt Answer
Implementing prompt-basek learning in re- search, we transform the QA dataset into a fill- in-the-blank task by substituting answer slots with [MASK] special tokens. Then, we uti- lize the Bert-Large model, which aligns with our task, to deliver complete phrase outputs. A prompt template, Input = (CLs]Question is:
2.3 Question Filter
is: passage directs question and passage information to the model. Answer replacements, input starts, and question- passage separators denoted by [MASK], [CLS], and [SEP], respectively. When inputs exceed the 51 512-character limit, they're truncated from either end based on [MASK] length.
Figure suggests a possible disconnect between new questions, generated from the Prompt Answer and passage, and the original QA dataset. To tackle this, repurpose the natural language inference task into question-filtering task to evaluate the relevance of generated questions the original data. Using the concept of Bert's pre-trained Next Sentence Prediction task (Devlin et al., 2018; Sun et al., 2021a), we design the Sentence Correlation Prediction (SCP) task to fine-tune the relevance evaluation, which encompasses filter input and out- put construction.
The Bert-Large model generates various Prompt Answers, with the volume and diversity of these outputs determining the subsequent quantity, diver- sity, and relevance of questions. For instance, the input (CLS) Question is: What areas did Beyonce compete in when she was growing up?[SEP] Pas- sage Beyonce performed various [MASK] competitions as a child... generates four diverse re- sponses( (polens, Artistic, Vocal, and International),
The SCP fine-tuning task proceeds in two steps. The filter input, Step 1, involves the combination of passages and generated questions, following the format Tingut

---

Here, Sen- tencel is the generated question, Sentence2 is the sentence containing the answer from the pas- sage, with [CLS] and [SEP] marking the sen- tence's final hidden state and sentence separation. Step 2, filter output, vectorizes the [CLS] output. The SCP header then processes this to calculate the relevance probability PM (n:, Impud,) exp Pi.q.) En (x)pp_)))) Here, 8 = W (CLS) is the SCP head, built via an MLP network using h[ci.s] vec- tor as input. This results in the class probability distribution, with and q denoting passages and questions, and the relevance label. The associ- ated loss function is NNSP log PM (n x))-
CLSSSennene//SSSPPPSeeeeeeee
instances and evaluated them using the test set. Ap- pendix ?? provides more examples of the data after augmentation."
4 Experiment Result
In this section, we provide a comprehensive analy- sis of the results to thoroughly evaluate the perfor- mance of the proposed approach.
4.1 Validity Experiments in Low-resource Datasets
As depicted in Table 1, our data augmentation strat- egy, referred to as PQQ, has been evaluated on two distinct low-resource Question Answering (QA) datasets: PolicyQA and TechQA The results high- light the significant advantage of PQQ over com- parative methods. Specifically, PQQ achieved EM score of 58.9 and Fl score of 29.7 on the PolicyQA dataset, and an EM score of 32.6 and an Fl score of 59.8 on the TechQA dataset, outper- forming all comparison methods. The remarkable performance of PQQ can be primarily attributed to its ability to generate high-quality question data. In terms of data augmentation comparison meth- ods, have chosen approaches that augment either <Question, Answer> pairs(Schmidr et al., 2022; Alberti al., 2019) or independent <An- swer> data( Van et al., 2021; Chen et al., 2023b). However, even when augmenting the same data, these strategies remain weaker than the PQQ method, thus indicating that augmenting question data exclusively is better choice.
PAlicyAA
TechQA
Method
EM
F1
EM
F1
Deep learning methods
Bert-base
base
Bert-hase
57.3
26.9
24.6
51.7
Bert-Large
61.1
28.7
28.8
53.9
RoBBRTa-base
58.1
27.1
24.4
48.3
SummAug (2021)
57.7
27.0
28.6
53.2
LambadaAug(2020)
57.9
26.8
24.9
49.8
QusAnsAuu(2019)
58.3
27.1
29.4
54,9
AnsAug (2021)
58.0
27.9
30.1
55.1
MulStaAug (2022)
58.6
28.2
31.9
58.3
PromptAug(2023b)
28.8
30.8
57.9
Large Language Model & Our Approach
ChatGPT(one-shot)
20.8
11.5
20.4
31.2
Our Approach
58.9
29.7
32.6
59.8
Table The validity experiment results in low- resourcee datasets. The highest scores of unsupervised methods are in bold.
3 Dataset and Experimental Design
assessing our method's effectiveness on low- resource QA datasets, we chose PolicyQA and TechQA, with 25,107 and 600 training instances respectively, focusing on privacy policies and tech- nology (see Appendix A. We conducted two comparative experiments, one applying traditional and SOTA data augmentation techniques in the QA domain, and another employing inference via a large-scale pre-training language model, Chat- GPT (details in Appendix A.2). To ascertain the impact of these different approaches, we adopted two evaluation metrics: Exact Match (EM) and Fl scores (Appendix A.3). We trained with Bert- Large and T5 as the prompt answer/question fil- ter and question-generaation models respectively. For the ultimate QA tasks, we opted for fine- tuning approach using a Bert-base model on QA data (training specifics in Appendix A.4). We stan- dardized the augmented data for both datasets to an additional 45,000 <Passage, Answer, Question>
It is noteworthy that significant performance gap was observed when testing with ChatGPT2, large language model trained on substantial cor- pus. Specifically, on the PolicyQA dataset, Chat- GPT achieved an EM score of 20.8 and an score of 1 1.5,, while on the TechQA dataset, it scored an EM of 20.4 and an Fl 31.2. These results empha- size the necessity of fine-tuning low-resource QA data, regardless of the use of broad and diversified training corpora.
PolicyQA
TechQA
EM
FI
EM
F]
Bert+PQQ
58.9
29.7
32.6
59.8
Bert+PQQ-P
58.1
29.3
31.9
59.2
Bert+PQQ-Q
57.8
28.4
30.9
58.1
Bent+PQQ-P.Q
57.1
27.6
29.4
57.7
Table 2: Results of ablation experiments. P iiddicaees the -Prompt strategy and Q indicate the -Question Filter.
TThe ChatGPT used in this work is gpt-3.5-turbo API.

---

4.2 Ablation Experiments
4.3
Validity Experiments in High-resource Datasets
The experimental results, as shown in Table 2. demonstrate the significance of Prompt answers and the Question Filter in the PQQ approach, which determined through ablation studies conducted by Eliminating the Prompt answers template and the Question Filter step resulted in -Prompt and -Question Filter scenarios, respectively.
As shown in Table 3, we applied the PQQ ap- proach to evaluate its effectiveness on both low- resource and high-resource QA datasets, specifi- cally SQuADI.1(Rajpure
et al., 2016) and Trivi- (See Appendix A. The approach tech- nique surpassed baseline measures and demon- strated significant improvement in evaluation met- rics. Specifically, when compared the traditional BiDAF model (Seo et al., 2016), the PQQ approach resulted in an increase of 1.4 points in EM and 1.1 points in FI for SQuADI.1. Moreover, training Bert-base with PQQ yielded superior results com- pared to training Bert-lange on the original dataset. Similar enhancements were seen with TriviaQA, where the Bert-base model trained using PQQ out- performed Bert-large trained on the original dataset and surpassed the AugMulStages method (Schmidt et al., 2022) by points in EM and 3.5 points in F1.
Results highlighted that every component of the PQQ framework adds positively to the augmenta- tion results, most notably when employing the full PQQ approach with the Bert-base-based QA model. The Question Filter was particularly influential, as its removal led to I to 2 point reduction in F1 and EM scores. Moreover, the worst results were observed when both -Prompt and -Question Fil- ter were removed, confirming their crucial roles within the PQQ framework. The experiment re- sults showed more significant decrease in perfor- mance when the -Question Filter was removed, demonstrating the effectiveness of using the Ques- tion Filter to reduce noise in the augmented data.
SQUADI.I
TriviaQA
Models
EM
EM
FI
Augmentation
BiDAF
66.7
Bert-base
81.2
65.1
71.2
F1
Bert-large
84.2
91.1
67.9
74.8
EM
LLaMA2(one shot)
19.3
18.8
31.2
90
1.LMA2(1.2 1. epochs)
27.9
38.1
27.2
55.1
88
Augmentation
Mug
66.9
77.6
86
our upproach+
68.1
78.4
AugMulStages++
81.9
89.0
67.2
73.4
84
our approach+
86.4
92.6
69.2
76.9
Table 3: SQUAD1.1 and TriviaQA experimental results. The Bert-base model and the BiADF(Seo al., 2016) model are represented by + and A respectively. The version of LLaMA2 LLaMA2 SaaMMA2-hhtt-BB... Chat-7B.Th The model representation of the subsequent experiments is similar. The highest scores unsupervised methods are in bold.
Size of Promoted Numberlk)
(a) Experimental result SQuAD1.1
4.4 Prompt Answer Quantity Analysis
60
55
The impact of Prompt Answers quantity on PQQ approach performance, using the Fl metric within SQuAD1.1 and PolicyQA datasets, was investi- gated. Figure 2 shows that the F1 metric for the high-resource SQuADI. dataset plateaus at 92 with 9 Prompt Answers, suggesting diminishing re- turns beyond this point. In contrast, the Fl score for the low-sresource PolicyQA dataset escalates past 12 Prompt Answers, implying that more prompts further bolster QA performance in low-resource scenarios. This highlights the crucial influence of Prompt Answers on data augmentation efficiency, with the ideal count differing due to dataset nature
F1 EM
45
40
35
0
4
8
10
12
Size of Promoted Numberlk)
(b) Experimental result on PolicyQA
Figure 2: Experimental results on PolicyQA and SQUADI.I using different amounts of prompted an- swers with comparison of K-EM and K-F1 scores

---

and resource provision. The consistent improve- ment with more Prompt Answers in low-resource datasets underscores the potential of prompt-based learning in enhancing QA data quality in large pre- trained models.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic qa corpora generation with roundtrip consistency. arXiv preprint urXiv:1906.054/1
Husam Ali, Yllias Chali, and Sadid A Hasan. 2010. Automatic question generation from sentences. Actes de la 17e conférence sur le Traitement Automa- tique des Langues Naturelles. Articles courts, pages 213-218.
5 Conclusion
Our work presents a new data augmentation method for low-resource question-answering tasks. We first use prompt-based learning to gather vari- answers. Then, model generates diverse ques- tions, and filter retains only those relevant to the original QA data as augmented data. Our approach excels in low-resource contexts and shows broad applicability in the general QA field. Future work will employ large pre-trained models to enhance data quality.
Ateret Anaby-Tavor Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling, 2020. Do not have enough data? deep learning the rescue! In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 7383-7390.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
6 Limitations
Vittorio Castelli, Rishav Chakravarti, Saswati Dana, An- thony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, et 2019. The techqa dataset. arXiv preprint arXiv: 1911902884.
The proposed method is effective for extractive question-answering tasks, but has several limita- tions. First, its potential in situations where the answers aren't 't in the provided text remains unex- plored. Second, as answer length increases, the employed masking mechanism struggles to cover all words, impacting data quality. This issue awaits further investigation and solution development. Fi- nally, the method's performance hasn't been fully tested on specialized or resource-limited question- answering datasets, necessitating further research understand its potential in these areas.
Yllias Chali and Sadid A Hasan. 2015. Towards topic- to to-question generation. Computational Linguisties, 41(1):1-20.
Andong Chen, Feng Yao, Xinyan Zhao, Yating Zhang, Changlong Sun, Yun Liu, and Weixing Shen. 2023a. Equals: A real-world dataset for legal question an- swering via reading chinese laws. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 71-80.
Canyu Chen and Kai Shu. 2022. Promptda: Label- guided data augmentation for prompt-based few shot learners. arXiv preprint 2xxi::2205.09222
7 Acknowledgements
We want to thank all the anonymous reviewers for their valuable comments. We also thank Yang Xiang, Xuefeng Bai for their great help and in- sightful suggestions when polishing this paper. This work was partially supported by the Na- tional Natural Science Foundation of China (Grant No.62376075, No.62276077, No.61972436, and No.621061 1.5) and by Shenzhen College Stability Support Plan (Grant GXWD202208 (1 170358002 and GXWD202208117 17131100021) This work was also partially supported by the International Coop- eration Project of PCL (PCL2022D01).
Xiusi Chen, Yu Zhang, Jinliang Deng. Jyun- Yu Jiang. and Wei Wang. 2023b. Gotta: Generative few-shot question answering by prompt- cloze data aug- mentation. CoRR, abs/2306.04101.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn ing to ask: Neural question generation for reading comprehension. arXiv preprini 1rXXvv1705.000
Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, Maamegie-Mellooe Univ Pittsburgh pa language technologies insT.
References
Wasi Uddin Ahmad, Jianfeng Chi, Yuan Tian, and Kai- Wei Chang. 2020. Policyqa: A reading compre- hension dataset for privacy policies. arXiv preprint arXiv:2010.02557.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint arXiv: 035.0.

---

Patrick Lewis. Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Miqa: Eval- uating cross-lingual extractive question answering. arXiv preprint 19100779..
A Experiment Setup
A.1 Datasets
To evaluate the effectiveness of our proposed ap- proach on low-resource QA datasets, selected two representative datasets:
Judith Yue Li, Aren Jansen. Qingqing Huang, Joonseok Lee, Ravi Ganti, and Dima Kuzmin. 2023, Maqa: A multimodal benchmark for negation. arXiv preprint arXi::2301 230.00338...
PolicyQA (Ahmad et al., 2020): This low- resource question-and-answe dataset focuses on privacy policies. consists of 25, 107 passages derived from the privacy policies of 1 115 websites, covering various aspects of privacy.
Chenyang Lyu, Lifeng Shang. Yette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization- informed question generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. pages 4134-4148, Association for Computa- tional Linguistics.
TechQA (Castelli et al., 2019); This low- resource QA dataset revolves around the field of technology. The data is collected from real tech- nical forums and comprises 600 training samples, 310 validation samples, and 490 test samples.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
SQuAD1.1: This is a well-known large QA dataset that contains over 100,000 question-answer pairs from more than 500 Wikipedia articles.
Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2019. Exploring the limits of transfer learning with unified text-to-text trans- former. arXiv preprint Aiv: 10683.
TriviaQA: This is massive and challenging QA dataset with total of 650,000 QA pairs, surpassing the size of SQuAD 1.1.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.052500
A.2 Comparison Methods
In the realm of natural language processing, we set out to evaluate the effectiveness of our proposed data augmentation approach by comparing it to several existing benchmark strategies. These en- compass both deep learning- based techniques such as the augmentation of question data and answer- question pairs.
Maximilian Schmidt, Andrea Bartezzaghi, Jasmina Bo- gojeska, A Cristiano Malossi, and Thang Vu. 2022. Improving low-resource question answering using active learning in multiple stages. arXiv preprint arXiv:2211.14880.
Minjoon Seo, Aniruddha Kembhavi. Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint
Methods like SummAug(Lyu et al., 2021) and LambadaAug(AnnabyTTaaaaa et al., 2020) generate fresh questions derived from the original answers. These methods employ language models to en- sure the quality of the newly generated data. Al- ternatively, QusAnsAug(Albers et al., 2019) and AnsAug( Van et al., 2021) opt to extract and gen- erate novel answers from the original question- answer datasets, providing different angle of data augmentation.
arXiv:1611.01603.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang. Junyuan Shang. Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021a. Ernie 3.0: Large-scale knowledge enhanced pre-training for lan- guage understanding and generation. arXiv preprint arXiv:2107.0213:
Yuan Sun, Chaofan Chen, Andong Chen, and Xiaobing Zhao. 2021b. Tibetan question generation based on sequence to sequence model. Computers, Materials & Continua, 68(3).
Another noteworthy approach Sta.uu(SShhmit et al., 2022), which synergizes active learning and data augmentation through the generation of question question-answer answer pairs. The aim of this method is to improve question-answering in low-resouree environments. Furthermore, Promp- tAug(Chen al., 2023b) utilizes generative prompt-based method for foundational cloze data augmentation bolster the learning process. This approach employs a cloze task that involves understanding the context, filling in the blanks,
Mul-
Hoang Van, Vikas Yadav, and Mihai Surdeanu. 2021. Cheap and good? simple and effective data augmen- tation for low resource machine reading. In Proceed- ings of the 44th International ACM SIGIR Confer- ence Research and Development in Information Retrieval, pages 21 211221..
Barack Wanjawa, Lilian Wanzare, Florence Indede. Owen McOnyango, Lawrence Muchemi, and Ed- ward Ombui. 2022, Kenswquad-a question answer- ing dataset for swahili low resource language. arXiv preprint 2205 02364..

---

and subsequently generating answers, simulating the human reasoning process.
As for the question generation model, the T5 model is initially fine-tuned on the QA dataset for augmentation. During the fine-tuning process, the model is trained using 2 V100 GPUs for epochs, with a learning rate of 7e-4 and a batch size of 8.
In order to ensure a fair evaluation, all of these methods were assessed under identical conditions. We used 45k augmented question-answer instances to level the playing field and negate any bias asso- ciated with data volume differences.
For the question filter model, we have opted for the Bert-large-hhole-woon wow.ool-wwrd--mkkiinn masking (Bert-Large) model. Step 2 of fine-tuning the question filter model involves the construction of a three-layered Multilayer Perceptron (MLP) network, functioning as the SCP head, with an intermediate layer dimen- sion of 1024. The Bert-large model is fine-tuned on the QA dataset. As for dataset partitioning, we have allocated 50% of the passage-question pairs from the training set positive data. The remaining 50% designated as negative data, accomplished by ran- domly replacing the question data the <passage, question> pairs across the entire dataset. The fine- tuning process involves training the model on V100 GPU for 5 epochs, with a learning rate of 5e-4 and batch size of
A.3 Evaluation Measures
In the evaluation of the two low-resource QA datasets, quantitative evaluation metrics were em- ployed, namely EM (Exact Match) and Fl scores (Lewis et al., 2019). These metrics were utilized assess the effectiveness of the data augmentation.
EM (Exact Match): EM calculates whether the predicted outcome exactly matches the standard answer. The EM score is computed using the fol- lowing formula:
Nreal Nau
EM
Nreal represents the number of predicted answers that exactly match the true answer, while Nau rep- resents the total number of true answers.
The fine-tuning process for the Bert-base model applied to the QA data was conducted using spe- cific parameters. This training regimen entailed a ten-epoch training cycle implemented on sin- gle V100 GPU, with reduced learning rate set at le-5. The batch size was established at eight, and the training configured to handle maximum sentence length of 328.
Fl Score: Fl score measures the word-level match between the predicted outcome and the stan- dard answer. The calculation of the Fl score in- volves the following steps:
N Noveriop
P
N Nallanswer
B supplementary experiments
Noverlap
R Niruthanswer
B.1 PQQ with LLM
2 x P R.
F1
To verify the impact of the current large model on our method, this experiment analyzes the PQQ approach by replacing various components with ChatGPT.FFr each experimental group, the exper- iments augmented the training data by 1k on the TechQA dataset. The experimental results are as follows:
P+ R
Nowerlop represents the number of words/characters predicted correctly, indicating the lexical overlap between the predicted answer and the true answer. Nallanswer denotes the number of words/characters in the predicted answer, while Nirathanswer de- notes the number of words/characters in the true answer.
The results still prove the effectiveness of our methods. Although ChatGPT has the capability to substitute all PQQ components, its use in shot setting results in heightened data noise and duced performance. However, employing ChatGPT solely to replace the question filtering component yields a notable boost in performance, indicating its effectiveness for logic consistency classification tasks.
A.4 Training Details
To obtain the Prompt Answer, the first step in- volves constructing the prompt input template as de- scribed in detail in Section 2.1. On the model side, (Bert-Large) model provided by Hugging Face? is used for in- ference to obtain the Prompt Answer.
the largel hert-large-whole-wwee wwhle-word-maane
hhtp://wwugnnggsocccooeernnnnmmm masking
'ntpss//huggginffccc...tt../. baa.aa/gaiggaaccccooooo-aabeee

---

TechQA
respectively.
Technique
EM
Fl
For every length group, we selected 8,935 ran- dom QA pairs and employed consistent PQQ method to produce 10k augmented data points. Dis- tinct bert-base models were trained using each aug- mented dataset and then assessed on the SQUAD validation set. The outcomes revealed:
BERT-base
24.6
51.7
PQQ
26.1
53.6
PQQ-replace
24.5
51.8
PQQ-replace QG
24.9
51.7
PQQ-replace QF
25.3
52.8
PQQ-replace ALL
24.1
51.2
SQUAD
Table 4: P represents Prompt Answer, QG represents Question Generation, and QF represents Question Fil- tering.
Range
EM
F1
[1-3]
64.31
69.96
(3-7)
63.98
69.31
(7-43)
63.04
68.47
B.2 Augmented Data Volume and Performance Correlation
Table 6: Evaluation of Answer Generation Method with Varying Masking Lengths
To further validate the effect of increasing the num- ber of context-question-answw triples, we initi- ated an additional experiment. Maintaining the same experimental framework as in the Appendix A, the expanded 45,000 samples were divided into batches of 155, 25k, 35k, 45k, and further extended to 55k, 60k, and 65k. A distinct bert-base model trained using each subset and subsequently as- sessed on the SQUAD validation set. The outcomes are presented below:
These findings indeed validate the method's sen- sitivity. We plan to explore this aspect in greater detail in upcoming research.
SQUAD
PolicyQA
Augmented Datas
EM
F1
EM
F1
15k
81.5
89.0
26.1
55.6
25k
84.8
90.5
27.7
56.8
35k
85.2
91.5
28.5
57.4
45k
86.4
92.6
29.7
58.9
55k
86.9
93.1
29.9
59.1
60k
87.0
93.2
30.6
59.5
65k
87.0
93.2
31.1
60.1
Table 5: A Comparative Analysis of Augmented Data Volume and Correlation with Performance.
These results are consistent with the conclusions presented in Figure of our paper. Moreover, this experiment also further validates the effectiveness of our work.
B.3
Answer Generation Sensitivity Masking Length
To further investigate this matter, carried out experiments for validation. Initially, analyzed the distribution of answer lengths (which equate to mask lengths) in the SQUAD dataset, segmenting them into three groups: [1-3], (3-7), and (7-43), with sample counts of 52,578, 26,086, and 8,935


Reviews:
[{"review_id": "hnvxIWXmvv", "reviewer": "Reviewer_xYud", "summary": "", "questions": "- There are experimental results about one-shot Chat-GPT. However, it is possible to replace each fine-tuned model of each step in PQQ to Chat-GPT. In other word, (1) prompt answers using chat-gpt, (2) generate questions using chat-gpt, (3) question filtering using chat-gpt. Can you provide additional experimental results with PQQ (w/ Chat-GPT)?\n- Can you provide experimental results when applied to larger language models such as LLaMA-7B?\n- If possible, please provide more examples of augmented dataset in the revised version.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for Reviewer xYud's valuable comments! Here is the response to Reviewer xYud.\n\n**Q1**: The target model to train is limited.\n\n**A1**: Based on the situation you've raised, our understanding may not be comprehensive enough due to the lack of context. We currently interpret this situation from two perspectives.\n\nFirstly, we speculate that after PQQ obtains augmented training data, the QA model may suffer from insufficient training. In fact, we train BERT on different tasks until convergence and save the best-performing model on the validation set for final evaluation. Additionally, we will release the source code, model weights, training logs, and augmented QA data available for researchers to refer to.\n\nSecondly, it is considered that the QA task in this work is limited to extractive models (like BERT) rather than generative models (such as LLaMA), thus imposing limitations. Regarding this perspective, we conducted two supplementary experiments where we replaced various components of PQQ with ChatGPT and other LLM models (LLaMA2-Chat-7) on QA data. The results of both experiments have proven that current extractive models have advantages in the QA tasks mentioned in the paper.\n\nIf there are aspects of your comment that we may not have fully grasped, please let us know. We look forward to your feedback to help improve this work.\n\n**Q2**: There are experimental results about one-shot Chat-GPT. However, it is possible to replace each fine-tuned model of each step in PQQ to Chat-GPT. In other word, (1) prompt answers using chat-gpt, (2) generate questions using chat-gpt, (3) question filtering using chat-gpt. Can you provide additional experimental results with PQQ (w/ Chat-GPT)?\n\n**A2**: This is a very interesting idea, and we have conducted experiments according to your recommendation. For each scheme, we augmented 1k QA data (TechQA comprises 600 training and 490 samples) and separately trained BERT models for validation on the test set, with the results as follows:\n| Technique      | EM  | F1  |\n|----------------|-----|-----|\n| BERT-base      | 24.6| 51.7|\n| PQQ            | 26.1| 53.6|\n| PQQ-replace *P*  | 24.5| 51.8|\n| PQQ-replace *QG* | 24.9| 51.7|\n| PQQ-replace *QF* | 25.3| 52.8|\n| PQQ-replace ALL| 24.1| 51.2|\nNote: *P* represents Prompt Answer, *QG* represents Question Generation, and *QF* represents Question Filtering.\n\nExcitingly, the results still prove the effectiveness of our methods. Additionally, we gained some insights. First, although ChatGPT can replace all PQQ components, using it in a one-shot setting leads to increased data noise and decreased performance. Secondly, when ChatGPT is only used to replace the question filtering component, there is a significant improvement in performance, suggesting that it is useful for logic consistency classification tasks.\n\n**Q3**: Can you provide experimental results when applied to larger language models such as LLaMA-7B?\n\n**A3**: Thank you for your suggestion. We have successfully applied it to low-resource (TechQA) and high-resource (SQUAD) tasks in LLaMA2-Chat-7B. The experimental results are as follows:\n\n|                   | EM   | F1   |\n|-------------------|------|------|\n| SQUAD (one-shot)   | 19.34| 28.90|\n| SQUAD (1.2 epochs) | 27.89| 38.12|\n| TechQA (one-shot)  | 18.78| 31.17|\n| TechQA (1.2 epochs)| 27.19| 55.10|\n| | | |\n\nThese results align with those presented in Table 1 of our paper. The findings underscore the importance of fine-tuning on these QA datasets. The complete set of results will be furnished in the final version of the paper.\n\n**Q4**: If possible, please provide more examples of augmented dataset in the revised version. seems to be quite important.\n\n**A4**: We appreciate your feedback. To better understand augmented data, here are some additional examples:\n| Passage | Original answer | Original question | Prompted answer | Generated question |\n|---------|----------------|-------------------|-----------------|--------------------|\n| [...]It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. [...] | { \"text\": [ \"Saint Bernadette Soubirous\" ], \"answer_start\": [ 515 ] } | To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? | Bernadette | To whom did the Virgin Mary reputedly appear in the grotto at Lourdes, France in 1858, according to the description of the architectural features of the school? |\n| [...]It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. [...] | { \"text\": [ \"Saint Bernadette Soubirous\" ], \"answer_start\": [ 515 ] } | To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? | Soubirous        | What is the last name of Saint Bernadette, to whom the Virgin Mary reputedly appeared in Lourdes, France in 1858? |\n| [...]It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. [...] | { \"text\": [ \"Saint Bernadette Soubirous\" ], \"answer_start\": [ 515 ] } | To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? | Someone          | To whom did the Virgin Mary reputedly appear at the grotto in Lourdes, France, according to the description of the school's architectural features? |\n| | |  |\n\nFrom the examples, you can see that the prompted answer is related to the original answer, and the generated questions are logically consistent with the original QA data. Due to space limitations, we will list more examples in the final appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- There exist several trials to augment or generate nlp dataset using large language models [1,2]. The proposed QQA suggests that one-shot LLM based dataset generation may underperform when applied to QA data augmentation, showing the value of studying algorithm for efficient and effective QA data augmentation.\n- Table 2 is nice.\n\n\n[1] SQUARE: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration, Lee et al., ACL 2023\n\n[2] WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation, Liu et al., EMNLP Findings 2022", "reasons_to_reject": "- The target model to train is limited. \n- I think more ablations are needed. Please check questions.", "questions_to_authors": "- There are experimental results about one-shot Chat-GPT. However, it is possible to replace each fine-tuned model of each step in PQQ to Chat-GPT. In other word, (1) prompt answers using chat-gpt, (2) generate questions using chat-gpt, (3) question filtering using chat-gpt. Can you provide additional experimental results with PQQ (w/ Chat-GPT)?\n- Can you provide experimental results when applied to larger language models such as LLaMA-7B?\n- If possible, please provide more examples of augmented dataset in the revised version.", "paper_topic_and_main_contributions": "- This paper aims to augment low-resource qa dataset by using language models.\n- The proposed data augmentation method PQQ, consisting of Prompt Answer, Question Generation, and Question Filter, generates additional valid triplets (question, passage, answer) for given data.\n- The empirical results show the superiority of PQQ when we train BERT model in a low-resource qa dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7h9KJ9COHy", "reviewer": "Reviewer_N3fX", "summary": "", "questions": "A. [CLS] and [SEP] are not the conventional special tokens for T5 models. Are they typos in line 137? If not, have they caused degraded performance?\n\nB. Are the results reported in Tables 1~3 the maximum, mean, etc., or just from a single run?\n\nC. Will you release the code and trained models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for Reviewer N3fX's valuable comments! Here is the response to Reviewer N3fX.\n\n**Q1**: It remains unclear how the number of available context-question-answer triples used for the answer prompt template would affect the performance of the proposed method.\n\n**A1**: We are delighted to receive this comment. To better understand the work, we initially outline the process of PQQ. Specifically, we generate prompted answers using prompt templates and then use these prompted answers, further refined by a question filter, to augment the question data.\n\nConsequently, line 174 in our paper states that we used the full set of context-question-answer triples for data augmentation, and Section 4.4 discusses the impact of different numbers of prompted answers on performance. \n\nLastly, to further validate the effect of increasing the number of context-question-answer triples, we conducted another experiment. Keeping the experimental setup consistent with our paper, the augmented 45,000 samples were subdivided into sets of 15k, 25k, 35k, 45k, and further extended to 55k, 60k, and 65k. Each subset was used to train an independent bert-base model, which was then evaluated on the SQUAD validation set. The results are as follows: \n\n|       | SQUAD  |       | PolicyQA |      |\n|-------|--------|-------|----------|------|\n|       |   EM   |  F1  |    EM    |  F1  |\n|  15k  |  81.5  | 89.0 |   26.1   | 55.6 |\n|  25k  |  84.8  | 90.5 |   27.7   | 56.8 |\n|  35k  |  85.2  | 91.5 |   28.5   | 57.4 |\n|  45k  |  86.4  | 92.6 |   29.7   | 58.9 |\n|  55k  |  86.9  | 93.1 |   29.9   | 59.1 |\n|  60k  |  87.0  | 93.2 |   30.6   | 59.5 |\n|  65k  |  87.0  | 93.2 |   31.0   | 60.1 |\n| | | | | |\n\nThese results are consistent with the conclusions presented in Figure 2 of our paper. Moreover, this experiment also further validates the effectiveness of our work.\n\n**Q2**: The proposed answer generation method is very sensitive to answer/masking length due to the nature of filling masked tokens. (Although the authors list it as future research direction, it could be better if the authors can include some preliminary analysis of this issue in the paper or its appendix)\n\n**A2**: We are delighted that you are concerned about this phenomenon. To further investigate this matter, we conducted experiments to validate it. We initially examined the distribution of answer lengths (equivalent to the mask lengths) within the SQUAD dataset, categorizing them into three tiers: [1-3], (3-7], and (7-43], with corresponding sample sizes of 52,578, 26,086, 8,935.\n\nFor each length category, we randomly sampled 8,935 QA instances and used a fixed PQQ method to generate 10,000 augmented data points. Independent bert-base models were trained on each set of augmented data and were subsequently evaluated on the SQUAD validation set. The results are as follows:\n| Range  |   EM   |   F1   |\n|--------|--------|--------|\n| [1-3]  | 64.31  | 69.96  |\n| (3-7]  | 63.98  | 69.31  |\n| (7-43] | 63.04  | 68.47  |\n| | | |\n \nThe experimental results indeed confirmed the sensitivity. In future work, we will further discuss this phenomenon.\n\n**Q3**: [CLS] and [SEP] are not the conventional special tokens for T5 models. Are they typos in line 137? If not, have they caused degraded performance?\n\n**A3**: Thank you for pointing out this detail. In our paper, we used 
 as the delimiter for passages and answers. We will make the appropriate corrections in the new version of the paper. Before settling on this scheme, we did try different approaches and found that various delimiters have some impact on the T5 model's question-generating capability. We fine-tuned the T5 model on SQUAD's training data and used BLEU for evaluation. Previously, we had tried three different separation methods: <1> `w/ 
`, the structure is `answer
passage
`; <2> `w/o special tokens`, the format is `answer is: {answer}. passage is {passage}`; <3> `Prompt`, we used `Generate a question related to the answer {answer} within the context of the passage {context}. The experimental results are as follows:\n|                 | BLEU |\n|-----------------|------|\n| w/ 
          | 21.2 |\n| w/o special tokens | 20.9 |\n| Prompt           | 20.1 |\n| | |\n\nBased on these results, we ultimately chose to use 
 as the delimiter. \n\n**Q4**: Are the results reported in Tables 1~3 the maximum, mean, etc., or just from a single run?\n\n**A4**: In our experiments, we conducted multiple runs to ensure the robustness and credibility of the results. The results reported in Tables 1~3 are the averages obtained from 3 independent runs for each configuration. This approach helps to mitigate the effects of randomness and provides a more robust estimation of the PQQ's performance. We are willing to provide further details in the appendix to facilitate readers and researchers in reproducing our experiments. \n\n**Q5**: Will you release the code and trained models?\n\n**A5**: We will release the trained models, code, and training logs, along with the filtered augmented QA data.\n\n**Q6**: It could be better if the K-EM and K-F1 shown in figure 2 are plotted with different scales.\n\n**A6**: This suggestion greatly aids in enhancing the readability of the paper. We have made the modification in the new version of the figure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Comprehensive QA pairs augmentation baselines that are both strong and recent.\n- Evaluation was conducted with both high-resource QA datasets (SQUAD1.1 and TriviaQA) as well as low-resource QA datasets (PolicyQA and TechQA)\n- The experiment results warrant the effectiveness of PQQ. Table 2 and Figure 2 provide some additional insights for the understanding of the proposed method.", "reasons_to_reject": "- It remains unclear how the number of available context-question-answer triples used for the answer prompt template would affect the performance of the proposed method.\n- The proposed answer generation method is very sensitive to answer/masking length due to the nature of filling masked tokens. (Although the authors list it as future research direction, it could be better if the authors can include some preliminary analysis of this issue in the paper or its appendix)", "questions_to_authors": "A. [CLS] and [SEP] are not the conventional special tokens for T5 models. Are they typos in line 137? If not, have they caused degraded performance?\n\nB. Are the results reported in Tables 1~3 the maximum, mean, etc., or just from a single run?\n\nC. Will you release the code and trained models?", "paper_topic_and_main_contributions": "This paper proposes an effective framework PQQ for augmenting QA pairs. The proposed method first generates diverse candidate answers by filling masked answer slots and then generates associated questions accordingly. The generated questions are further filtered by an NLI based relevance evaluation module.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It could be better if the K-EM and K-F1 shown in figure 2 are plotted with different scales."}, "extra_text": {}}, {"review_id": "ZdtplqCijf", "reviewer": "Reviewer_KiMC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for Reviewer KiMC's valuable comments! Here is the response to Reviewer KiMC.\n\n**Q1**: This paper is not well-written and lacks some key details\n\n**A1**: Thank you for taking the time to review our paper. Regarding the writing issues you've mentioned, I have already sought revisions from native English speakers to improve the quality of the writing in the new version of the paper.\n\nThis work focused on introducing a novel data augmentation technique (PQQ) specifically designed for low-resource question-answering tasks. Through prompt-based learning and model-generated queries, our method demonstrates excellent performance in low-resource settings and holds broad applicability across the general QA domain.\n\nIn the experiments of the paper, we first explored the effectiveness of PQQ in low-resource (TechQA/PolicyQA) and high-resource (SQUAD/TriviaQA) tasks. Then, through ablation studies, we verified that all components of PQQ have a positive impact. Finally, we examined the influence of the \"Prompted answer\" hyperparameter on performance in PQQ.\n\nDuring the review period, we first presented more examples of data augmentation to more clearly display the work's details. We then added comparison experiments with LLMs models and PQQ methods, further proving the effectiveness of our work. Lastly, we conducted a preliminary experiment to analyze the sensitivity of the PQQ method to masking length, thus providing guidance for future research. Additionally, we will release the trained models,  training logs, and code, along with the filtered augmented QA data.\n\nIf you have any other concerns, we hope to receive your feedback. We will continue to explain and verify, thereby further improving this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Their experiments reveal that ChatGPT underperforms on the experimental data, the proposed PQQ method excels beyond existing augmentation strategies.", "reasons_to_reject": "This paper is not well-written and lacks some key details", "questions_to_authors": "", "paper_topic_and_main_contributions": "To improve Low-resource Question Answering, this paper introduces a approach for question data augmentation consisting of Prompt Answer, Question Generation, and Question Filter.  Firstly, it construct prompt  templates to generate answers that are related to the original ones but expressed differently (Prompt Answer). Then, a diverse set of questions is produced by a question generation model . Lastly, use a question filter to  filter those questions that are logically related to the  original QA data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles to improve QA performance by generating synthetic question-answer pairs and training on them. This method is demonstrated to improve two low-resource QA datasets as well as two high-resource QA datasets.

Reviewers pointed out that experiments support the effectiveness of the method well (xYud, N3fX, KiMC), baselines are comprehensive (N3fX) and analysis is insightful (xYud, N3fX).

Reviewers raised concerns on the sensitivity of the method (N3fX) and the presentation (KiMC). Concerns on lack of ablations were also raised (xYud) which are addressed in the author responses.

During the reviewer discussion period, reviewers raised additional concerns on limited novelty, given that the general pipeline of answer identification -> question generation -> filtering for data augmentation in QA has been done in prior work [1, 2, 3]. However, the paper still made a small-but-non-trivial contribution on improving answer identification which goes beyond simply using linguistic rules and encourages the diversity of the answers, which is well-supported by a set of experiments and additional ablations provided during the rebuttal period.

[1] https://arxiv.org/pdf/2004.11546.pdf
[2] https://arxiv.org/pdf/2010.12643.pdf
[3] https://arxiv.org/pdf/2102.07033.pdf
Decision: Accept-Findings