KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP
Authors: Yufei Wang
Jiayi Zheng
Can Xu
Xiubo Geng
Tao Shen
Chongyang Tao
Daxin Jiang
Venue: ICLR 2023 poster (2023)
Keywords: Data Augmentation
Low-Resource NLP

Abstract: This paper focuses on data augmentation for low-resource NLP tasks where the training set is limited. The existing solutions either leverage task-independent heuristic rules (e.g., Synonym Replacement) or fine-tune general-purpose pre-trained language models (e.g., GPT2) using the limited training instances to produce new synthetic data. Consequently, they have trivial task-specific knowledge and are limited to yielding low-quality synthetic data. To combat this issue, we propose Knowledge Mixture Data Augmentation Model (KnowDA), a Seq2Seq language model pretrained on a mixture of diverse NLP tasks under a novel framework of Knowledge Mixture Training (KoMT). The goal of KoMT is to condense diverse NLP task-specific knowledge into the single KnowDA model
(i.e., all-in-one). The resulting KnowDA could utilize these knowledge to quickly grasp the inherent synthesis law of the target task through limited training instances. Specifically, KoMT reformulates input examples from various heterogeneous NLP tasks into a unified text-to-text format and employs denoising training objectives in different granularity to learn to reconstruct partial or complete samples. To the best of our knowledge, we are the first to attempt to apply 100+ NLP multi-task training for data augmentation. Extensive experiments show that i) the synthetic data produced by KnowDA successfully improves the performance of the strong pre-trained language
models (i.e., Bert, ALBert and Deberta) by a large margin on the low-resource NLP benchmark FewGLUE, CoNLL’03 and WikiAnn; ii) KnowDA successful transfer the task knowledge to NLP tasks whose types are seen and unseen in KoMT.

Published as a conference paper at ICLR 2023
KNOWDA: ALL-IN- KNOWLEDGE MIXTURE MODEL FOR DATA AUGMENTATION IN LOW-RESOURCE NLP TASKS
Yufei Wang Jiayy Zheng Can Xu 3, Xiubo Geng 3, Tao Shen 3, Chongyang Tao 3, Daxin Jiang Macquarie University, Sydney, Australia Peking University, Beijing, China Microsoft Corporation, Beijing, China
yufei. wangestudense mq. edu au, 2hengjiay11900012 com {caxu, xiubo. geng, shentao, chongyang taa, djiang wtw.:a/.iiinngeeoooooooooo...... omt.l/aott.m.cooooo..ttoooooooo.
com
ABSTRACT
This paper focuses the data augmentation for low-resource NLP tasks where the training set is limited. The existing solutions either leverage iask-independent heuristic rules (e.g., Synonym Replacement) or fine-tune general-purpose pre- trained language models (c.g.. GPT2) using the limited training instances to produce new synthetic data. Consequently, they have trivial task seek-speiffic knowledge and are limited to yielding low- low-quality synthetic data. To combat this issue, we propose Knowledge Mixture Data Augmentation Model (KnowDA) which an Seq2Seq language model pretrained on mixture of diverse NLP tasks under novel framework of Knowledge Mixture Training (KoMT). The goal of KoMT is to condense diverse NLP task-specific knowledge into the single KnowDA model (i.c., all-in-one) such that KnowDA could utilize these knowledge to quickly grasp the inherent synthesis law of the target task through limited training instances. Specifically, KoMT reformulates input examples from various heterogeneous NLP tasks into unified text-to-text format, and employs denoising training objectives different granularity to learn to reconstruct partial or complete samples. To the best of our knowledge, we are the first attempt to apply 100+ NLP multi-task training for data augmentation. Extensive experiments show that i) the synthetic data produced by KnowDA successfully improves performance of the strong pre-trainn pre-trained language models (i.e., Bert, ALBert and Deberta) by large margin on the low- low-resource NI.P benchmark GLUE, CoNLL'OS 0003 and WikiAnn; ii) KnowDA successfully transfer the task knowledge to NLP tasks whose types are seen and unseen in KoMT. [
INTRODUCTION
Neural NLP models require extensive supervised training data to achieve superior performance (Bow man et 2015). However, due to the enormous cost annotating data, developers could only use limited labeled data for training in common real-world uses of neural NLP models. This problem has attracted considerable attention recently. Many researchers (Kumar et al., 2020 Wang al., 2022 Zhou et al. 2021) resort to data augmentation techniques to generate more synthetic samples to boost the performance of low-resource NLP tasks.
The existing NLP data augmentation (DA) methods either leverage task-independent heuristic rules, such as Synonym Replacement (Zhang et al., 2015) and Random Swap (Wei & Zou| 2019a). or fine-tune cenneral-puppose pre-trained language models by using the handful training examples of target tasks, such as GPT2 (Radford et al., 2019) in LAMBADA (Anaby- produce new synthetic data. Consequently,
et al.) 2020) and
T5 (Raffel al. 2020) in PromDAA (Wang et al. 2022). these DA methods have trivial target task knowledge and are limited to yielding low-quality synthetic data (e.g., either irrelevant extremely similar to the training data). In addition, these DA methods
"Work done during the internship at Microsoft STCA. "orresponding author: Daxin (aannidddia@@mmroot.t.cmm
'The source code is released in http:////////t////o//////////a//

---

Published as a conference paper at ICLR 2023
often applied to simple NLP tasks where the inputs only include single and short sentences. Recently, Zhou et al. (2021) demonstrate that these DA methods perform even worse on tasks with complicated structures (c.g.. SuperGLUE). These issues prevent the existing DA methods from practical usage for various low-resource NI.P tasks.
Motivated by this, in this paper, propose Knowledge Mixture Data Augmentation Model (KnowDA) to tackle these two issues. To enrich the task knowledge for KnowDA, we propose Knowledge Mixture Training (KoMT) to represent various heterogeneous NLP tasks in unified and scalable manner. Specifically, in KoMT, arbitrary NLP task instances are represented as the key-value list format where the key is typically a short phrase indicating the feature function and the value is a string representation of the feature content. We further employ the denoising training objectives in different granularity of the NI.P task instances. That is, during KoM', we randomly mask subset of values (e.g., an input document or a question) and train KnowDA to reconstruct those masked ones. With the dynamic multi- graiiiaraalarttyy masking mechanism and the unified format, we successfully scale Knowledge Mixture Training (KoMT) of LM to about 100 NLP tasks without much human effort. Compared with previous attempts in unified multi-task learning works (e.g., TO (Wei et al. 2022) and FLAN (Wei et al. 2022)). KoMT is more scalable and comprehensive because those works heavily rely on the tuman--caftte prompts and are only trained to improve the NLP task performance (i.e., only generating correct output labels). Furthermore, previous data augmentation methods only focus on simple NL.P tasks, such as single-sentence classification. However, modern NLP tasks, such as NLI and QA, have much more complicated structures (i.e., multiple input text and long documents). To enable KnowDA to handle these NLP tasks with complicated structures, we propose a novel auto-regressive generation framework for KnowDA. At each step, we either fine-tune or directly use (i.e., zero-shot) a separate copy of KnowDA to generate a textual feature in an NLP instance. The feature generation order is based on task- spe--secifii feature dependency. As KnowDA is trained generate arbitrary input text features, we find it beneficial to generate long text in a zero-shot manner directly using the KnowDA checkpoint. Under this scenario, we control the outputs of KnowDA using relevant feature keys and full example demonstrations from the target tasks.
For evaluation, we conduct experiments on the challenging FewGLUE benchmark (Schick & Schüizze 2020) with 32 training examples. We also verify the effectiveness of KnowDA two Sequence Labeling tasks, CoNLL'OS '3 (Sang & De Meulder 2003) and WikiAnn (Pan et al. 2017), whose task types are held-out during KoMT. KnowDA successfully outperforms recently proposed state-of-the-an data augmentation algorithms such as DA and PromDA (Wang et al.] (022). We further compare the quality of generated synthetic data from KnowDA and FlipDA, confirming that KnowDA produces synthetic data with a higher level of diversity and better quality verified by humans.
The contributions of this paper the following:(1) To the best of knowledge, we the first work to scale the number of tasks to 100+ in multitask pretraining for data augmentation; (2) We propose novel multitask pre-training approach KoMT for data augmentation, resulting in new pre- pre-trained model, KnowDA and (3) KnowDA outperforms state-of-the of-te-of-the-art data augmentation methods on the low-resource setting of popular benchmarks FewGLUE, CoNLL'O3, and WikiAnn.
2
METHOD
In this section, our setting is introduced in Sec. 2.1 The details of KnowDA, including the design of KoMT and the auto-regressive data augmentation procedure, are discussed in Sections 2.3 and 2.4
2.1 DATA AUGMENTATION FOR LOW-RESOURCE NLP
In the low-resource NLP tasks, only a handful of labeled training data T = (x y:) are available. Data Augmentation generates synthetic data Tyym = { ( M:) ; from the original labeled training data T using language models, where m is allowed to be much larger than n. The goal is that NLP models trained using T Tsyn outperform the ones only trained using T.
2.2 OVERVIEW OF KNOWDA
KnowDA is an encoder-decoder generative language model that generates relevatl and diverse synthetic data from scratch. It is initialized from existing pre-trained eecoder-decoder language
2

---

Published as a conference paper at ICLR 2023
model. We further apply Knowledge Mixture Training (Sec. 2.3 inject diverse NLP task task-specific knowledge into KnowDA Finally, we fine-tune or directly use KnowDA to produce synthetic data to improve strong NLP baseline models in various low-resource NLP tasks (Sec. 2.4)
2.3 KNOWLEDGE MIXTURE TRAINING
To recap, previous data augmentation methods lack task-specific knowledge. Currently, the research community has accumulated large amount of various human-generated datasets. The main motiva- tion of KnowDA is to transfer the task knowledge in the existing large amount of NLP supervisions the generated synthetic data for other NLP tasks. The resulting models should learn to generate task-related data examples merely from a handful training samples of the target task. However, existing artificially generated NLP tasks have variety of complex structures (e.g., sentence pairs classification, multiple-choice QA) that makes learning unified data generator quite challenging. We convert the generation data examples from various heterogeneous NLP tasks into a unified text- to-text format and employ denoising objectives in different granularity to simulate the reconstruction of partial or whole instances. We call this training paradigm Knowledge Mixture Training (KoMT). We will elaborate KoMT in detail from following aspects: task collection, unified text-to-tex. to-tttttttttt format, demonstration examples and the denoising objectives.
Task Collection The success of KoMT is built upon resource with sufficient number of tasks, covering a diverse range of NLP applications. Similar to Ye et al. (2021). ). we select English monolingual datasets with open access in the Huggingface Datasets (Lhoest al. [021]. The tasks in KoMT broadly belong to the following task families: Text Classification, Natural Language Inference, Reading comprehension, Question Answering, Summarization, and other NLP applications. Training examples from each task are sampled proportionate to the size of individual NLP dataset. Each NLP dataset contribute at most 300k instances during KoMT.
Demonstration
[Premise]: car starts up [Hypothesis]: A man driving down lanaly road
people
[Premise]: |: soccer game with multiple males playing [Hypothesis]: Some men are playing [Tag]: Entailment
[Teg]: Contradiction
Masked Instance
Masked Instance
[Premise]: A man inspects the uniform of figure in some East country.
[Premise]: <MASKO>
[Hypothesis]: The man sleeping [Tag]: <MASK1>
[Hypothesis]: The man is sleeping [Tag]: <MASK0>
<MASK0> À man inspects the uniform of figurre in some East Asian country.
<MASKD> Contradiction
<MASKI> Contradiction
Knowledge Mixture Training
[Premise]: man inspects the uniform o figure in some East Asian country. [Hypothesls]: sleeping
[Tag]: Contradiction
Original Training Instance
Figure 1: running example of the masked training instances in the Knowledge Mixture Training. Keywords within the brackets are the task key.
Unified Text-to-Ter Format Although there many NLP tasks with various structures, all of these tasks include one or more input features and an output result. For example, we have an input sentencepaangrapp/ddmmett and an output label in the sentence classification tasks. Each feature, including the output result, could be described as a key-value pair. The key is typically short phrase indicating the feature function, and the value content. That is, given an instance (I,, yi) from arbitrary NLP tasks, we could always represent it unified key-value list format (length n):: (æ,, y/)) [[K] [[f].e.)... ).),,... (K. ef'] where each (k) either corresponds to an input feature or the output result y: Unlike previous works (Wei et al. 2022)
string representation of the feature
Sanh et al. 2022) requiring exhausting human-crafted prompts perform multi-rask scaling,
3

---

Published as a conference paper at ICLR 2023
key-value design only needs the feature name in the original NLP dataset. Therefore, proposed unified format could quickly extend KoMT to arbitrary tasks without much human effort.
Denoising Objectives Based on the unified key-value list format, our denoising objective is defined as follows: given training instance, we randomly mask K values in the key-value list, where number K is sampled from range [1 to ni with equal probability and n is the total number of key-value pairs. Then, KnowDA is trained with the objective to reconstruct those masked values using the remaining information. In addition, we will prepend I. examples as demonstrations in the front of the input KnowDA during KoMT, where L is sampled from the range [0 to m] with equal probability and m is the hyperparameter of how many demonstrations to put at most. Figure [ shows two masked training instances from the CB task (De Marneffe et al. 2019) where Premise and Hypothesis are input features and Tag is the output result. This unified format only requires simple cross-entropy loss, avoiding extra task-specific effort for loss design, loss scaling, or explicit gradient accumulation (Aghajanyan et al., (021). This strategy trains KnowDA to reconstruct NLP task instances from multiple granularities, potentially satisfying different needs in the data augmentation tasks. Our dynamic random masking mechanism also encourages KnowDA to fully use the task- specific keys and demonstration examples. When most of the values are masked for prediction, they are the only reliable information for KnowDA to generate meaningful synthetic data. For when both "Premise" and "Hypothesis" are masked, KnowDA could only obtain
example, in Figure [ task-relevant information from keys and demonstrations.
Demonstration Exemplars The unified task representations and denoising objectives allow KnowDA to encode various task knowledge in the model parameters, implicitly. When handling the low-resource target tasks that are dissimilar to the existing training tasks in KoMT, such implicit task knowledge transfer could become less effective. To better represent the semantics of the low-resource target tasks, we add demonstration section in our text-to-text format like GPT-3, As shown in Figure [] during KoMT, we randomly select a few instances for the demonstration section from the identical tasks and never take any value masking. To allow KnowDA to cope with different number of demonstration exemplars, the number of selected instances is random number between 0 and 16. We also discover that demonstration exemplars play an important role when KnowDA generates long text in a zero-shot fusion (Details See Sec. 2.4
2.4 GENERATING SYNTHETIC DATA USING KNOWDA
After KoMT, KnowDA has already captured various NLP task knowledge. It could be further adapted to generate synthetic data for low-resource NLP tasks. As discussed in Sec. 2.3 a NLP training instance could be represented as list of key-value pairs [[k] ,,').-. (k*", a where the key k is the feature name and e² is the textual content. To generate NLP training instances with arbitrary complicated structures from scratch, we propose an aulo-regressive data augmentation framework. We first analyze the task-specific feature dependency, then generate each feature value sequentially.
Task-specific Feature Dependency Different NLP tasks have their own dependencies for input features. These dependencies are mostly based on the annotation procedure of the target task. For example, as shown in Figure 2 in the task of Adversarial QA (Bartolo et al. 2020a), given a passage (i.e.. context), human annotator first produces question, and then creates the corresponding answer span for that question. Given this annotation procedure, we generate context, question, and answer retooregressively. To maintain a high diversity of the generated synthetic data in the low-resource setting, we train a different copy of KnowDA for generation at each stage. This is conceptually similar to the Prompt ensemble used in Wang et al. (2022). Note that the fine-tuning cost (i.e., training time) of KnowDA at each stage is relatively low. Practically, we only need to store the generated synthetic data and safely discard the fine-tuned KnowDA parameters. The detailed data augmentation process used in this paper can be found in Appendix A.5
Generating Text For Features For short nd (e.g., less than 250 tokens), directly fine-tuning KnowDA with the low-resource training data is a straightforward solution. However, it is tremen- dously difficult to handle long vd. In our preliminary experiments, we find that fine-tuned KnowDA memorizes some textual spans from the training examples and injects these memorized textual spans into the generated text. As result, the generated text is inconsistent across different sentences and of
4

---

Published as a conference paper at ICLR 2023
Context
Question
Answer
[Context]:
<MASK>
[Context]: Value LQuestion]:
[question]: Value
[ConLexL]: Value lue
<MASIC
[Answer]: <MASK>
Figure 2: Feature Dependency in the task of Adversarial QA.
low quality. To avoid this fine-tuning issue, directly use KnowDA, without further fine-tuning, to produce long text because KnowDA is pre-trained to generate many long texts. However, this could result in irrelevant outputs compared to the given handful of training examples in the target task. To combat this issue, we transfer task knowledge from the pre-training tasks in KoMT and the target task. Specifically, we select relevant feature keys based on the similarity (e.g., length and topie) between the target task and the pre-training tasks. To fully use the small set of training data in the target task, randomly select few of them and use them full demonstration examples the inputs. Both strategies could effectively control the contents of generated text, achieving satisfying data augmentation performance. We show the importance of these strategies in Sections 3.3.] and 3.3.2
EXPERIMENTS
In this paper, to show the effectiveness KnowDA, we compare KnowDA with recently proposed state-of-the-art data augmentation algorithm FlipDA (Zhou et al. 2022) and PromDA (Wang et al.
2022 in Sections 3.1 and 3.2 We also present an in-depth analysis of KnowDA in See 3.3
Evaluation Benchmark
Following PromDA and FlipDA, we conduct low-source experiments on the FewGLUE (Schick & Schütze, 2020), CoNLL'03 (Sang & De Meulder 2003). and WikiAnn (Pan al., 2017) benchmarks. For each benchmark, we run our experiments multiple times with different random seeds, templates and data splits and report the averaged results with the corresponding standard deviation. We write the standard deviation as the subscript of mean results (e.g., 80.01.o means the averaged performance is 80.0 with standard deviation 1.0). To avoid any form of data leakage, we exclude all above evaluation tasks from KoMT. We further filter out any training instances containing evaluation task input features (e.g.. input sentences) from KoMT. Finally, 1 114 diverse NLP tasks (See Appendix
A.3 3 for details) are selected for KoMT.
KnowDA Details KnowDA initialized from the -.--ree model (Raffel et al. 2020 We train KnowDA for 100k steps with a maximum sequence length of 512 and batch size 2048 in Linux environment with 16 x A100 GPU (40G). Fine-tuning KnowDA carried out only using a single A100 GPU (40G). We use Adam as the optimizer to train KnowDA. More details see Appendix A.I
3.1 DATA AUGMENTATION ON FEWGLUE
We use FewGLUE to evaluate KnowDA's 's ability on low-resource data augmentation. Each challeng- ing FewGLUE task only has labeled instances.
Setting In this experiment, we largely follow the settings in FlipDA (Zhou et al. baseline models, the input templates and the random seeds. We compare KnowDA with SR (Zhang al. 2015). EDA (Wei & Zou 2019b), T5-MLM (Zhou et al. 2021), and FlipDA (Zhou et al. 2021 The PET templates and random seeds are identical to FlipDA. Tables D and show the experiment results of FLUG. As FlipDA outperforms other baseline models by large margin, rerun the official code of FlipDA to compare its mean and standard deviation results with KnowDA. Results of baseline, SR. EDA and T5-MLM taken from Zhou al. (202T). More details see Appendix A.2
2021). including the
Result Adding KnowDA's 's synthetic data improves the baseline models in out of 8 FewGLUE tasks, with average improvement of 5.2 and 4.1 in the ALBERT and DeBERTa baseline models, respectively. On average, KnowDA outperforms previous state-of-the art FlipDA by 1.6 and 2.2 on the ALBERT and DeBERTa models, respectively. KnowDA consistently exceeds FlipDA on out
5

---

Published as a conference paper at ICLR 2023
of 8 tasks, which include QA (BoolQ, MultiRC), NLI (RTE, CB), and Word Sense Disambiguation (WiC). This shows that KoMT is capable of transferring prior task-specific knowledge to similar tasks (i.e., QA, NLI), well as novel tasks (i.e., Word Sense Disambiguation). Finally, KnowDA performs worse than FlipDA in the task of COPA and ReCoRD. This could be because KnowDA is unable to produce sufficient correct output labels for these two tasks.
Table 1: Performance Baseline and KnowDA on FewGLUE with ALBERT-xxlarge-22 model.
BOOLQ
RTE
CB.
WiC Acc.
WSC Aoc.
MuliRC EM/F1a
COPA Acc.
ReCoRD Acc/F]
Mothod
Acc.
Acc.
Acc./FI
avg
Baseline
72.5
61.4
82.7/74.8
51.3
77.0
33.0/7..6
88.3
86.2/86.8
71.2
83.3/78.1
51.3
34.1/75.6
85.6/86.1
71.6
EDA
72.7
58.3
81.1/73.6
51.8
75.9
28.7/73.1
84.5
85,4/86.0
69.6
TS-MLM
73.9
83,5/75.0
51.1
79.2
33.8/74.1
87.3
85.2/85.7
71.5
FipDA
77.0..3
70.72.8
86.3g.s/ 82.5s.
53.41.0
78.74.3
36.42.1/ 16.21.1
89.21.2
86.0p.2/86.6ml 80.2
74.8
KnowDA
78.20.s
78.7o.9
89.61.v/85-42.0
55.91.6
77.95.6
36.8g,
89.02.1
85, 25 86.
76.4
Table 2: Performance of Baseline and KnowDA FewGLUE with DeBERTa-v2-xxlare model.
BOOLQ
WiC Acc.
MaliRC EM/Fla
COPA Acc.
ReCoRD Acc/F1
Method
Acc.
Acc.
Acc./F1
Acc.
avg
Baseline
78.3
82.0
85.4/79.3
58.7
80.1
40.4/78.1
87.7
90.2/90.8
77.4
SR
77.4
76.3
87.2/80.3
58.9
80.9
35.7/76.3
87.0
89.1/89.6
76.2
83.6/76.2
59.3
37,0/77.1
88. 1/88.6
75.1
TS-MLM
77.4
81.2
83,0/73.7
60.7
82.4
35.0/75.0
88.2
89,7/90.3
76.7
RlipDA
81.6,.1
83.01.s
88 2/8.
61.21.4
77.65.0
13.61.
79.999988 &
86.5..5
90. 90.7o.2/91.3pp
79.3
KnowDA
83.91.6
86.81.0 92. 92.11.6/89.21.
66.53.4
79.52.s 46.82.5/81
90.75.0
89.7a.a /90.30.3
81.5
3.2 DATA AUGMENTATION FOR SEQUENCE LABELING TASKS
In Sec. 3.1 KnowDA shows success in handling NLP tasks with complicated structures from Super- GLUE. In this section, we further verify the effectiveness of KnowDA in the Sequence labeling Tasks which are excluded from KoMT.
Setting Following (Wang et al. 2022), we take BERT-base (Devlin et al. 2019) as the baseline
model. We compare with several strong data augmentation models, including SDANER (Dai & Adel
2020 LAMBADA (Anaby- Tavor et al. 2020). MetaST (Wang et al.) 2021) and PromDA (Wang al. 2022). We conduct the shot- 10 setting with 40 samples for CoNLL'33 and 30 samples for WikiAnn. There are two stages in generating synthetic data for sequence labeling tasks: 1) generating synthetic input sentences; labeling entities on the synthetic input sentences. We use low-resource training data to fine-tune KnowDA for both stages. We find that feeding the generated synthetic data back to KnowDA can improve its entity labeling performance and produce better synthetic data. We, thus, iterate this process times four times. Following (Wang et al., 2022), repeat the experiment five times with different random seeds and data splits and report the averaged results in Table 3 We also show the entity labeling performance of KnowDA and and the performance of BERT fine-tuned with the corresponding synthetic data in each iteration in Table 4 More details see Appendix A.4
Result Table 3 illustrates BERT labeling performance on the CoNLL'03 and WikiAnn Benchmark after using synthetic data from various DA methods. Compared to the BERT baseline models, BERT with KnowDA's synthetic data achieves 12.9 and 16.1 Fl score improvements. Compared to the state-of-the-art PromDA method, KnowDA's synthetic data gains an improvements of 7.8 points and points on CONLL''3 and WikiAnn, respectively. Both PromDA and KnowDA are based on the T5-Large checkpoints. The major difference is that PromDA only receives mask-agnostic pre-training on sub-set of C4 corpus (Raffel et al.. 2020), while KnowDA receives KOMT. This implies the effectiveness of using NLP prior task knowledge for data augmentation. As shown in Table adding KnowDA's 's synthetic data substantially improves the entity labeling performance of KnowDA and BERT (i.e., To vs. T.). Both models further improved the iteration continues. Although KnowDA could be directly used to perform sequence labeling tasks, the BERT model trained with generated synthetic data consistently outperforms KnowDA (i.e., in CONLL'03. 85.0 vs. 85.4; in

---

Published as a conference paper at ICLR 2023
WikiAnn, 64.9 vs. 66.3). This shows that the BERT model successfully learns the knowledge embedded in the synthetic data from KnowDA.
Table 3: Performance of DA on the sequence labeling benchmarks.
Table 4: The entity labeling performance for KnowDA and BERT after each iteration (from To Tạ).
Method
CONIL:03
Wikiann
CoNLL'03
WikiAnn
Iter.
BERT- base
72.54.6
50.82.8
KnowDA
BERT
KnowDA
BERT
SDANER
73.94.3
51.73.3
To
80.11.9
72.55.1
62.52.a
50.23.6
LAMBADA
74.93.7
52.91.9
Ti
83.01.3
84.00.9
64.23.1
65.42.6
MetaST
76.71.2
56.62.3
T2
84.91.2
84.80.5
65.42.9
65.92.8
PromDA
77.53.5
59.02.9
T3
84.90.8
85.
65.32.9
66.03.0
KnowDA
85.40.s
66.32.9
T1
85.01.1
85.40.8
64.93.0
66.32.9
3.3 DISCUSSION
In this section, we conduct an in-depth analysis of KnowDA. We investigate the impact of key choices and demonstrations for KnowDA in Sec. 3.3.1 We present performance comparison of KnowDA s fine-tune and zero-shot versions for generating long context in section Sec 3.3.2] We also compare KnowDA with other multi-task pre-training generative models (e.g.. TO and T5) on data augmentation in Sec 3.3.3
We further compare the synthetic data from KnowDA and FlipDA in Sec 3.3.4
KnowDA Replace Template
Demonstration Replace Template
89.6
85.00
78.7
78.5
76.7
74.9
75
73.5
72.6
70
RTE
CB Acc.
CB F1
Figure 3: The impact of different choices of key and demonstrations. w/o Demonstration, no demonstration examples used. In Replace Templates, different keys to generate synthetic data. We report accuracy for the RTE task, and both accuracy and Fl-score for the CB task.
3.3.1 1 TASK KNOWLEDGE TRANSFER
KoMT has two important components to transfer task knowledge: 1) Task-specific key list: 2) Full demonstration examples. To show the importance of key choices and demonstrations, choose the challenging RTE and CB tasks, which involve unsupervised long document generation steps. The best template for RTE and CB long document generation is "Premise [MASK]" and "Text [MASK]", respectively. In the Replace Template, we replace the key Premise with Context in RTE and replace the key Text with Document in CB. We generate separated three settings (Replace Template, w/o Demonstrations, and both). We follow the settings in Table and feed these sets of synthetic data into the PET ALBERT model separately. Figure 3 shows that both key and demonstrations have considerable impact on the generated synthetic data: RTE drops up to 6. accuracy, and CB score drops up 8. 8.7%. Notably, the impact of the key is larger than the demonstrations in the RTE task, while this is opposite in the CB task. We hypothesize that this could be because keys and demonstrations activate different knowledge from KnowDA, which have different impacts the downstream tasks. This is further confirmed by the fact that applying both changes would further negatively impact the PET ALBERT model performance. In summary, keys and demonstrations have similar and positive impacts KnowDA in data augmentation. They effectively transfer appropriate and different prior tpsc-ic knowledge to the ongoing task. More analysis can be found in Appendix A.7
of synthetic data under
7

---

Published as a conference paper at ICLR 2023
3.3.2 LONG TEXT GENERATION
As discussed in Sec. 2.4 zero-shot learning applied when KnowDA generates long text for the task BoolQ, RTE, and CB. These three tasks include long text in their labelled instances (e.g., the passage BoolQ). We conduct an ablation study and show the importance of this design choice by directly fine-tuning KnowDA to generate the long text. The experiment is based on the ALBERT model. As shown in T fine-tuning negatively affects the quality of synthetic data and the corresponding PET performance becomes worse than using the zero-shot version.
Table 5: Ablation Study of zero-shot generation for long text the ALBERT-xXlargee model.
Method
BOOLQ (Acc.)
RTE (Acc.)
CB. (Acc./11)
avg
KnowDA Fine-Tuning)
73.3p.7
68.51.8
86.32.s / 80.8 77.2
KnowDA (Original)
78.2p.s
78.7u.9
89.61.9 / 85.62.9
83.0
3.3.3 COMPARISON WITH TO AND RAw T5
In this section, we compare KnowDA with Raw T5-Large model (i.e., KnowDA w/o KoMT) and TO, T5-3B model trained with large and diverse set of human-annotated prompts to solve different NL.P tasks. We select the task of WiC, WSC, and COPA for comparison because none of KnowDA, Raw T5 and TO have seen these three tasks during pre-training. The experiment is based the ALBERT model. As shown in Table 6 both raw T5 and TO perform worse than KnowDA. Although TO achieves good performance in solving those NLP tasks, TO is never trained produce synthetic data. When applying TO to generate synthetic data (e.g., generating a full sentence given a class label), its data augmentation performance is sub-optimal.
Table 6: Comparison with Raw T5 and TO model the ALBERT-xxlarge-2 model.
Method
WiC
WSC
COPA
avg
TO
52.12.9
71.1a.s
88,02.s
70.4
Raw T5
51.31.1
70.24.7
87.72.0
69.8
KnowDA
55.91.5
77.93.6
89.02.1
74.2
3.3.4 SYNTHETIC DATA QUALITY ANALYSIS
Table and have confirmed that the synthetic data from KnowDA can effectively improve low- resource performance in FewGLUE We further examine the quality of generated synthetic data from KnowDA and compare that with FlipDA, including diversity analysis and human evaluation.
85.4
Table Diversity analysis on Synthetic Data.
67.1
Model FlipDA 67.9 44.0
Self Belf-BLEUL
Novel Entity
42.1
758.5
FlipDA
KnowDA
KnowDA
1048.3
Figure 4: Human Evaluation Results.
Diversity Analysis We compare the diversity of the generated synthetic data from KnowDA and FlipDA. We sample 200 synthetic data from each FewGLUE task. Following Wang et al. (2022), we calculate Novel Entity (i.e., number of entity mentions or keywords not appearing in the training data) and Self-BLEU score (Zhu et al. 2018) for each FewBLUE task separately. We report the averaged score across all tasks to quantify the overall synthetic data diversity. As shown Table 7 synthetic data from FlipDA is less diverse than the one from KnowDA. Its Self-BLEU score increases from 44.0 to 67.9, and Novel Entity score decreases from 1048.3 to 758.5. This explains the advantages of training signals in the generated synthetic data.
KnowDA because it provides
8

---

Published as a conference paper at ICLR 2023
Human Evaluation We further conduct human evaluation to analyze the quality of the synthetic data. We sample 50 instances from ground trunh training data (GT), KnowDA's 's synthetic data (KnowDA), and FlipDA's synthetic data (FlipDA) in each FewGLUE task, resulting in 1200 samples in total. We mask the data source of these samples and assign them to three annotators who are only told that there are slightly more machine-generated samples than human-generated ones. They asked to judge whether each sample is generated by humans. Figure 4 shows the human evaluation results. 85% of GT are correctly classified, which shows that is relatively easy to identify the human-genee generated samples. 67.1% of KnowDA's synthetic data are identified as human- -enaeratdd samples, while this number decreases to 42. 1% for FlipDA's synthetic data. This clearly exhibits that, compared to FlipDA, the synthetic data from KnowDA are more similar to the human-crafted data.
RELATED WORK
Data Augmentation for Low-resource NLP We divided data augmentation methods for low- resource NLP into two categories according to the complexity of the tasks that need to be augmented. The works of the first category focus on simple tasks, e.g., single-sentence classification and sequence labeling task that only has sentence input, and one sequence as output. Early, researchers propose word substitution based methods such as KNN replacement (Vijayaraghavan et al., 2016 Wang & Yang. 2015), Synonym Replacement (Zhang al. 2015), TF-IDF replacement (Xie et al. 2020) and EDA (Wei & Zou| 2019b) integrated with muluple base replacements. Later, large generative models have been used for data augmentation, such as back translation (Fadace al.) Sennrich et al., 2015) utilizes machine translation models to synthesize new data samples, LAMBADA (Anaby- Tavor et al.] 2020) finetunes a GPT-2 model to generate augmented data, GPT3Mix YYo et al. 20211 uses GPT-3 along with hard prompting to yield augmented examples, PromDA (Wang et al. 2022) leverages soft prompting to perform efficient learning from few shots. The previous work (Zhou et al.) 202 )) demonstrates that these methods are hard to generate proper synthetic data for tasks with more complicated structures (c.g.. SuperGLUE), Another line of low- resource data augmentation is exploring augmentating tasks with complicated structures (i.e., long sequences or multiple sentences). FlipDA (Zhou et al. 202T) has conducted preliminary attempts on these tasks by applying local phrase- edits over the original training data. Our method also make efforts on this setting. The main difference between KnowDA and FlipDA lies in that our KnowDA could generate the whole augmented examples from scratch.
Multi-task Pre-training Recent research efforts have emerged in unifying multiple NLP tasks as a unified Sequence-to-Sequene text generation task. Paolini et al. (2021) proposes to tackle various structured prediction language tasks using a single text-to-text generation model. /eei et al. (2021), |anh et al. (2022), Aribandi et al. (2022), and Xu et al.) (2022) further demonstrate that adding these unified Sequence-to-Sequemee can further improve the model performance. Our model is broadly inspired by this long line of previous works with following major differences. First, we focus on utilizing more human- human-annotunn annotated NLP task datasets to learn how to generate synthetic samples of a given task rather than just learning how to complete the task. Second, most of previous works rely on the human-crafted task-specific hard prompt the unified format do multi-task pre-training. Our unified key-value format does not require extra human efforts and allows KoMT to easily scale to many NLP tasks.
NLP training instances to pre-trained language models
CONCLUSION AND FUTURE WORK
This paper explores multi-task learning paradigms at a massive scale for data augmentation in low-resource NLP tasks for the first time. We demonstrate that the proposed Knowledge Mixture training enables pre-trained language models the capability of generating proper synthetic instances from scratch for complicated tasks (i.e., the data sample has long sequences or multiple sentences). Experiments verified the effectiveness of our KnowDA, and KnowDA outperforms state-of-the-art data augmentation approaches on the popular benchmarks FewGLUE, CoNLL'O3, and WikiAnn. We also perform ablation studies indicating the importance of including demonstrations and the impact of different keys. Moreover, increasing the size of multi-task scaling and investigating more advanced training objectives for data augmentation is still a promising direction worthy of long-term exploration.

---

Published as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5799-5811 1, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1t1S5///122021..mmmmmaaaaaa 2021 wmnlp- 468
URL
Tiago A Almeida, José María Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium Document engineering, pp. 259-262, 2011.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Kedzis-Kedziosski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-basel formalisms.
arXiv preprint arXiv:1905. 13319, 2019.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlo- mov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! Proceedings of the AAAI Conference on Artificial Intelligence, 34(05))7383-33900 Apr. 2020. doi: a0.16699aaai..3440000 v34105.6233, URL
org. http//iddxxxxe httpp/pppppppppppa.aaapa /AA/AAT//
article/view/623 6233
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022. URL
id=Vzh1BFUGi1X
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the
AI: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguisticss, 8:662 678, 2020a. doi: 10.1162/tacl 000388. URL https::/aaclant loghhhoooo....oo tttpr///222...ll1....1
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-677 678, 2020b.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL http::://aalaaaaollllot http::/hho.oooo.oooo111111-111--
wwt.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning, arXiv preprint 19Xiv:190.005733 2019.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, pp. 7432-7439, 2020.
Jan A Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. Learning split and rephrase from wikipedia edit history. arXiv preprint arXiv:1808, 09468, 0806009688 2018.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2015. Association for Computational Linguistics. doi: 10. 1/D18///1//11 URL https
Lisbon, Portugal, September
7/aclanthology htwpoloooyorg//i1511 1075
Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. SemEval- 2019 task 3: EmoContext contextual emotion detection in text. Proceedings of the 13th International Workshop Semantic Evaluation, pp. 49,48, Minneapolis, Minnesota, USA,
10

---

Published as a conference paper at ICLR 2023
June 2019. Association for Computational Linguistics. doi: 10.18653/v1/$19-2005 URL
httts:///aeaanthooooy.oo2//////
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Mocha: A dataset for training and evaluating generative reading comprehension metrics. arXiv preprint arXiv:20100036366 2020.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:I909.022 19t.199999.114,, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. (8X3vv8803.0544 803.00557v4 2018.
Arman Cohan, Waleed Ammar, Madeleine Van Zuylen, and Field Cady. Structural scaffolds for citation intent classification in scientific publications. arXiv preprint arXiv: 1904. 104.01608, 2019.
Xiang Dai and Heike Adel. An analysis of simple data augmentation for named entity recognition. arXiv preprint arXiv:2010. 1683, 2020.
Pradeep Dasigi, Nelson Liu, Ana Marasović, Noah A Smith, and Matt Gardner. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. arXiv preprint arXiv: 055003 2019.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM 17, pp. 512-515, 2017.
Ona De Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. Hate speech dataset from a white supremacy forum arXiv preprint arXiv:1809.0444 2018.
Carri-Catheerne De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. volume 23, pp. 107-124, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/+1NN19-4422, URL https
//aclanthology htwp:hhhoooooooo/1111113. 1423
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims, 2020.
Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from search engine. arXiv preprint 17X4v:170..05179, 2017.
Ondicj Dušck, Jekaterina Novikova, and Verena Rieser. Evaluating the state-of-the-art of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59:123-156, 2020.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.186533v1//11--1100 URL https: /
org/P19 1102
Marzich Fadace, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural 2017.
machine translation. arXiv preprint
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: 2019.
Long form question answering. arXiv preprint

---

Published as a conference paper at ICLR 2023
Manaal Faruqui and Dipanjan Das. Identifying Well-formed Natural Language Questions. In Proc. of EMNLP, 2018.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Sunmarization, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1
URL.tttss: //attts:://all anellanthha //tth///oolllllllgogooo
org/D19-5409
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009.
Giovanni Grano, Andrea Di Sorbo, Francesco Mercaldo, Corrado A Visaggio, Gerardo Canfora, and Sebastiano Panichella. Android apps and user feedback: a dataset for software evolution and quality improvement. In Proceedings of the 2nd ACM SIGSOFT international workshop on app market analytics, pp. 8 1-,1,22 2017.
Harsha Gurulingappa. Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann- and Luca Toldo. Development of benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of biomedical informatics, 45(5): 885-892, 2012.
Luheng He, Mike Lewis, and Luke Zettlemoyer. Question answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 643-653, 2015.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In arXiv: (ee:: 909.00077... 2019.
Kelvin Jiang, Dekun Wu, and Hui Jiang. Freebaseqa: A new factoid qa data set matching trivia-style question-answer pairs with freebase. In NAACL-HLT (1), pp. 318 088-323,22 2019.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Shioty-Second AAAI Conference on Artificial Intelligence, 2018.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8082-8090, 2020.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks. arXiv preprint arXiv: 1.00783, 1.000,8811.00788, 2018.
Neema Kotonya and Francesca Toni. Explainable automated fact-checking for public health claims. arXiv preprint arXiv:2010.09926, 2020.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. arXiv preprint urXiv:2003.02245. 2020.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017, Association for Computational Linguistics. doi: 10.18663//1/DD1--1822 URL https //aclanthology wtt.:/gyyygg///11/1111100
Rémi Lebret, David Grangier, and Michael Auli. Generating text from structured data with application to the biography domain. CoRR, abs/1603.07771, 2016. URL ltttp://axxiv.xx//
0.03.07771
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195. 2015.
12

---

Published as a conference paper at ICLR 2023
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computationnl Linguistics. doi: 10.18653. v1/2021 v1/2021.emnlp- .emnlp-demo.21. URL https: httpp//////aaaaaooooooooooo//e/ 2021. wttp://orr.........emee....e..ee emnlp-dene 21
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: constrained text generation challenge for generative commonsense reasoning. arXiv preprint 1rXiv:191.003770 2019a.
Bill Yuchen Lin, Seyeon Lec, Rahul Khanna, and Xiang Ren. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of EMNLP, 2020. to appear.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. arXiv preprint arXiv:1908.00 100.00820000,,, 2019b.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. ACL, 2017.
Annie Louis, Dan Roth, and Filip Radlinski. i'd rather just go to bed": Understanding indirect answers. arXiv preprint arXiv:2010.03450, 2020.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http 7/ w/www. ac web. ww.. ant wwwhhhoooyyy/ea wwgg//11-100000
Malo, A. Sinha, P. Korhonen, Wallenius, and Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Irene Manotas, Ngoc Phuoe An Vo, and Vadim Sheinin. Limit: The literal motion in text dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 991-1000, 2020.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216-223, 2014.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. arXiv preprint arXiv:2012.10288 2020.
Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Effective transfer learning for identifying similar questions: matching user questions to covid-19 faqs. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 3458-3465, 2020.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 1809.08000000000 027892 2018.
13

---

Published as a conference paper at ICLR 2023
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. Semeval- 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 31-41, 2016.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium,
2018. Association for Computational Linguistics, doi: 10.18653/v1/ 118-1206. URL battss://aclanthottt
1206
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial
NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
Online, July 2020.
Association for Computational Linguistics. doi: 10. 18653/111/22200 ael-main.441. URL https //aclanthology. org
main.
Achraf Othman and Mohamed Jemni. English-ast gloss parallel corpus 2012: Aslg-pe12. In Sth Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC, 2012.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1946-1958, 2017.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. Structured prediction as translation between augmented natural languages. In 9th International Conference on Learning Representa- fions, ICLR 2021, 2021.
Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. Biomre: A dataset for biomedical machine reading comprehension. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pp. 140-149, 2020.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 18.18653/v1/200 10/553/v12221.mm// wtt.:022..aammmmi....m URL https
//aclanthologs wtt.org//2222 naacl main. 200
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http:://mmi http:///mmm...o//pppeeeeee www.pppers.//ppoppepooooopeeeeea /tteers//v1/20 -221/220074.2ttm 074.htm.
Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: the winograd schema challenge. Proceedings of the 2012 Joint Conference on Empirical Methods Natural Language Processing and Computational Natural Language Learning. pp. 777-789. Association for Computational Linguistics, 2012.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906. 02360, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 2392, Austin, Texas, November 2016. Association
14

---

Published as a conference paper at ICLR 2023
for Computational Linguistics. doi: 10.1863//1/D16--222 1264. URL httpss//ccaaathhhlo.t org/D16-1264
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: new benchmark and dataset. In ACL, 2019.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to ai complete question answering: A set of prerequisite real tasks. In Proceedings of the AAAI conference artificial intelligence, pp. 8722-8731, 2020.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, doi: 1/116553/1/115-1444 URL http: : / /xx. www..oo........oo. 10
18653/v1/D15-DD
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC:
Towards Complex Language Understanding with Paraphrased Reading Comprehension. Meeting of the Association for Computational Linguistics (ACL), 2018.
Erik F Sang and Fien De Meulder. Introduction the conll-2003 shared task: Saiguugg--indpenddee named entity recognition. arXiv preprint cs/0306050, 2003.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak. De- bajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL http:: http://///pppeeeeeeeeeeeeeeeeeee wwtp::/weeeeoootoede//ee/er (or.adavvvrrbooooooo
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. Carer: Contextu- alized affect representations for emotion recognition. In Proceedings of the 2018 conference on empirical methods natural language processing, pp. 3687-3697, 2018
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint 0rXiv:2009.07116 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv: 2015.
Emily Sheng and David Uthus. Investigating societal biases in a poetry composition system. arXiv preprint arXiv:201 1. ..06666, 2020.
Damien Sileo, Tim Van-De-Cruys Camille Pradel, and Philippe Muller. Mining discourse markers for unsupervised sentence representation learning. arXiv preprint
2019.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7063-7071, 2019a.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. Quartz: An open-domain dataset of 2019b.
qualitative relationship questions. arXiv preprint
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge. arXiv preprint 18/ 2018.
Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. Wiqa: A dataset for" what if... " reasoning over procedural text. arXiv preprint arXiv:
2019.
15

---

Published as a conference paper at ICLR 2023
Sowmya Vajjala and Ivana Lučié. Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pp. 297-304, 2018.
Prashanth Vijayaraghavan, Ivan Sysoev, Soroush Vosoughi, and Deb Roy. Deepstance at semeval- 2016 task 6: Detecting stance in tweets using character and word-level cnns, arXiv preprint arXiv:1606.05594 2016.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, 2019.
William Yang Wang. " liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv: 00007005500044 2017.
William Yang Wang and Diyi Yang. That's so annoying!!!: lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2557-2563, 2015.
Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Hassan Awadallah. Meta self-training for few-shot neural sequence labeling. In Pro- ceedings of the 27th ACM SICKDD Conference on Knowledge Discovery & Data Mining, pp. 1737-1747, 2021.
Yufei Wang. Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. PromDA: Prompt-base based data augmentation for low-resource NLU tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4242-4255, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 10.18653/ v1/2022.acl-long.22 292. URL htttttts:///ac /ttp::/s///....ttoooolo......../ wwtp:ooloooor.//22222 ac acl- 292
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6382-6388. Hong Kong, China, November 2019a. Association for Computational Linguistics. doi: 10. 11.11666//1//1111100 URL httpt:://aaa aaelllnnthhlyoo
orq/D19-1670
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11100 2019b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2/09.0/0 (55.2 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL htt.t:::///ppeeeieee.. www.://wwwwwweeeeeoru.r..
Id=gEZ=GCozdqq
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: meee::0070000000 2017.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 2020.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 62625666268, 2020.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. Tweetqa: A social media focused question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
16

---

Published as a conference paper at ICLR 2023
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Zeroprompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint wrXiv:22010699 2022.
Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2013-2012 2015.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NL.. Proceedings of the 2021 Conference on Empirical Methods in Natural Langwage P Processing, pp. 7163-7189, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.11853/v1/200 http///.e/....elpmmmee/e malp-mmaan.5222 URL NEtps://aclanthont
org/ 2021 emnlp ma main........................mm 572
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:210.008826 2021.
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv: 1809, 08887, 2018 8.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint 1808. 053266 2018.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Rui Zhang and Joel Tetreault. This email could save your life: Introducing the task of email subject
line generation, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL https:: sttpss://proceedingg www.eeiiiiinnnn..nnn.eeeee www..pps.cccppppare///// /0//paper/211//fleeee www333333cccbbbo Papp pdf
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv: arXiv: 1904.0/130 2019.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00100 2017.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. goong on vacation" takes longer than" going for walk": A study of temporal commonsense understanding. arXiv preprint arXiv: 190.999990055 2019.
Jing Zhou, Yanan Zheng, Jie Tang, Jian Li, and Zhilin Yang. Flipda: Effective and robust data augmentation for fho- leamiin. arXiv preprint arXiv:2108.06333 2021.
Jing Zhou, Yanan Zheng, Jie Tang, Li Jian, and Zhilin Yang. FlipDA: Effective and robust data augmentation for few-shot learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8646-8665, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1866//1//022.add long.59.. URL htwpehhhooloyyrr/222222
org/ www.b/a..ccloooooa.. 592
Yaoming Zhu, Sidi Lu, Lei Zheng. Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097-1100, 2018.
17

---

Published as a conference paper at ICLR 2023
A APPENDIX
In this Appendix, provide more experiment details about KnowDA in Sec. A. We introduce the diverse NLP tasks used in KoMT in Sec. A.3] We add additional experiment description about our data
A.3.
augmentation experiments in Sec. A.2
and A.4 We present the detailed data augmentation procedure in Sec. A.5 Finally, in Sec. A.6 we showcase representative FewGLUE examples generated by
KnowDA
A.1 EXPERIMENT DETAILS
Demonstration Selection in KoMT As we discussed above, we use demonstration in KoMT to allow KnowDA to transfer task-specific knowledge. Given the training NLP instance T,, we first conduct random masking to it and treat the resulting string as initial input string. We then select 16 instances from the same NLP task and check to see the maximum number of instances (i.e., m) can be inserted into the input string within the maximum input length. Lastly, we will randomly select k instances demonstration where & is also random number between 0 and m, which avoids KnowDA from relying on the demonstration too much. The only requirement for the demonstrations that they should come from the same NLP task as the masked NLP task instance. In addition, even in KoMT, long NLP instance (i.e., input length close to maximum input length) may never be accompanied by any full demonstration examples.
Soft Prompt for Data Augmentation We observe that, when KnowDA trained to generate input text features, KnowDA generates relatively long text (e.g., a document for QA); when KnowDA is trained to generate output labels, normally generates relatively short text (e.g., label phrases). To disentangle the different skills required for different roles, during KoMT, prepend two sets of Soft Prompt (i.c, mundomly-initiliize trainable vectors) at the beginning of each layer. One for input text generation and another one for output text generation. Different from the standard prompt tuning, we train the parameters of KnowDA and these Soft Prompt together. We add 5 vectors per layer, resulting * 24 * 1024 * = 245,760 additional parameters (only 0.003% of the original T5-Large model). When applying KnowDA to downstream tasks, we should use the appropriate Soft Prompt for unsupervised generation or fine-tuning. Our preliminary experiments show that using different Soft Prompt has an significant impact on the overall performance.
A.2 FEWGLUE
In this section, we add more implementation details for the FewGLUE data augmentation experiments Sec. 3.T Appendix A.5 shows the details of data augmentation procedure for each FewGLUE task. In all FewGLUE tasks, when need to update the parameters of KnowDA, we simply fine-tune KnowDA (i.e., updating all parameters) with batch size 12 with learning rate of 5e - for 500 steps. For fair comparison with FlipDA, we directly produce synthetic data and feed into the ALBERT and DeBERTa model for further training. Unlike FlipDA which creates synthetic instances from the few-shot instances, KnowDA produces synthetic instances from scratch. Consequently, there is no explicit linkage between the synthetic instances and the few-shot instances. We therefore use the ALBERT and DeBERTa model for Consistency filtering, which only keeps synthetic instances that have the consistent output results from KnowDA and the ALBERT and DeBERTa model. We find such filtering policy works as well as the complicated label-based filtering strategies proposed in FlipDA Zhou et al. (2021).
In this experiment, we use PET (either based on the ALBERT or DeBERTa model) as the base model. PET is a semi-supervised training algorithm that uses language- based patterns to convert the input training instances into cloze-style. For example, to predict whether "Mia likes pie" and "Mia hates pie" are entailed or not, PET covert them into "Mia likes pie Mia hates pie". The training objective is to train the PET model to fill "Yes" or "No". Note that given a target task, there could be multiple patterns. PET works follows: 1) we train separate PET model for each pattern the same small training data; 2) The ensemble of all the above models is then combined to annotate a large unlabeled dataset; 3) the labeled dataset above is finally used train a final classifier.
18

---

Published as a conference paper at ICLR 2023
A.3 KOMT TASK DETAILS
Table 8 and list all supervision NLP tasks in KoMT. They are all available from Huggingface Dataset Lhoest et al. (2021). Popular task types include QA, Multiple Choices, Text Classification, Text Summarization and Generation and Machine Reading comprehensive.
Table 8: All of the training NI.P task datasets used in KoMT.
Dataset(s)
Description
Train Datasets
[
Cittion
ADECorpus V2
Text Clusification
23,316
Curvlingapos al. 2012
Adversarial QA
Natural Lunguage
$0,000
AESLC
Email Summarization
Zhang Ter
2019
AG-News
Text Classiiiationn
120,000
Zhang ct 8.
2013
A12-ARC
Multiple Chuice
1.119
ANII
Adverserial NI.1
Nie 41211
AppRoviows
Text Scoring
288.065
Gradk 21
201
Aqui-RAT
Multiple Choice
97.467
2017
ART
NLI
169.654
Bisgaalal 2l al
2019
ASLG-PCI2
Text Generation
87,710
Ulbmmin & Jemn
2012
BinMRC
Machine Reading Comprehension
700.000
21:00
Bicak-Data
Question Deconposition Meaning Represcutations
61,824
Wolssn 2 A1
2120
Circa
Text Clussiiication
34,268
L003 21 15 2020
Climate-Feve
Text Scoting
1,535
20200
Codah
Multiple Choice
1,665
Common-Gen
Text Gencration
67,389
Lin et al. 2019
Commonsense-
Open Domain QA
9.741
201
COS-E
Text Generation
9,741
Kapai A1V
Cosmos-QA
Multiple Choice
25,262
Himing et m1 2019
DBpedial4
Text Classification
560,000
Definite-roooun- Promoun- Pow.oo--eoooooiiniinn
Precoun resolution
1,322
Kal-man & Np 2012
Discovery
Text Classification
1,566,000
Sileo et 2019
Dream
Multiple Choice
6,116
DurRC
Abstractive QA
60,721
Stba
2018
E2E- Cleaned
Text Generation
33,525
Disck et 2022
E1.15
Absrractive
502,957
2019
EMO
Text Classification
30, 30,160
Chilerje 2t al
2019
Emotion
Text Clussificationn
16,000
Saravia et 2013
Empathetic- Diaaoogui
Open Domain QA
Kasikin
Financial Parasebank
Text Classification
2264
Maloeeal 2014
PreehaseQA
Open Domain QA
20,358
Tians al
2019
Gigaword
Text Summarization
3,803,957
GLUE
General Language Understanding
949_101
Wang et ak.
2019
Wellfereerrr
Text Scoring
17,500
Fartinal Line
2018
HateSpeoch18
Text Classification
2018
HusSpee:hOfeasiiee
Text Classification
24.783
Diviuson 0t 33.
2017
Hatexplain
Text Classificationn
15,383
Malnew 21 10. 2020
HealthFact
Text Classification
9,832
Koinaya
20b0
Hellaswag
Text Geacration
39,905
Zellas et 3),
2219
HotporQA
QA
90,447
Yong ct 2018
IMDb Reviews
Text Classification
Maas cl
leopurdy
Natural langunge QA
216.930
KILT
Knowledge-Intentt Language Tasks
2,791,679
Petroni et
2021
Fake News Detection, Text Classification
Limit
Text classification
23,559
MiRoss et 4.
2020
MatQQA
QA
29,837
Ammii 2 2119
MC-Taco
Multiple Choice
9,442
Medical Questions- Pains
Text Classification
3048
MRL Retty
d
4020
Mocha
QA
$1,099
m
21.07
Multi-Nows
Text Summarization
2019
NumerSense
Numerical commonserse reasoning prohing
10.444
Ji d ml.
211.0
Onestop-Enng English
Text Classifiaation
567
2018
OpenBookQA
Open
QA
4,957
Mihaylov
2018
PAWS
Text Classification
49, 39.401
Zhang et d.. 21199
PIQA
Multiple Choice
16,000
A1 2170
Poem- Sentiment
Text Clussiiication
2020
QA-SRL
Multiple Choice
6,414
Fe dl. 2013
QASC
Multiple Choice
8,134
Kline 2 ml 60.0
QUAIL
Multiple Choice
2720
QUAREL
Multiple Choice
1,941
Tarped & al. 2019
QUARTZ
Multiple Choice
2,696
Tarped 25 m 20196
QUOREP
Disigl
RACE
School QA (MCQ)
87, 8/6
mi 2
401
Reddä-Tifu
Text Summarization
42.139
2018
Ropes
Extractive- QA
2019b
Rotten- Rotten-Tomatres
Text Classinication
8,530
Pany &
12
22005
SamSum
Text Summarization
14,372
Girws e 2019
Sentiment
Text Classification
1,600,000
SCIGit
Text Classification
8.194
Lohun at 2019
SCIQ
Multiple Choice
11.679
Webbi or ai 2011
SCITail
23,596
Klock
2013
Searcb-QA
QA
151.295
Donin e 2i 2017
Sick
Text Clussifcation
4,439
Marelli
2014
19

---

Published as a conference paper at ICLR 2023
Table 9: All of the training datasets used in KoMT Part II.
Dataset(s)
Description
No. Train Datasets
[D]
Citation
SMS-Spam
Text Classification
5,574
Almeida al. (0011)
Social-I-QA
QA
33,410
Sap ct al. 2019
Spider
Text Generation
7,000
Yu
2018
SQuAD
QA (context)
87,599
Rajpurkar
2016
Swag
Text Classification
73,546
Zellers ct al.
2018
Tab-Fact
Text Classification
92,283
Chen
20199
TREC
Text Classification
5,452
& Roth
2002
TweetEval
Text Classification
120,104
Mohammad ct al.
2016
TweetQA
Natural Language QA
10,692
Xiong
2019
WebQuestions
QA (open)
3,778
Berant 20113
WIQA
QA
29,808
Tandon et al.
2011
Wiki-Bio
Text Generation
582,659
Lebret
2016
Wiki-QA
QA
20,360
Yang et al. 2015
Wiki-Split
Text Split
989,944
Botha ct al.
2018
WikiSQL
NLI
56,355
Zhong
2017
XSum
Text Summarization
203,577
Narayan et al.
[2018
Velp-Polarity
Text Classification
560,000
Zhang et al.
2015
Full
Text Classification
650,000
Zhang el
2015
Total
114
A.4 SEQUENCE LABELING TASKS
In this section, we add more implementation details for the sequence labeling data augmentation experiments in Sec. 3.3, including data augmentation procedure, training technologies and iteration- based training.
Data Augmentation Procedure KnowDA produces synthetic data via two steps:
Generating input sentences (without any entity annotations) via the sequence of named entity tags. This is similar to the Output View described in Wang et al. (2022).
2. Use KnowDA label named entities for the generated input sentences.
For example, given a training sequence labeling data instance:
[Org All Fishermen 's Association] secretary [Per N.J. Bose] said the strike would continue indefinitely.
where Org corresponds to the entity label Organization and Per corresponds to the entity label Person. In the Stage 1, the format of training instances is:
Input: [Output Tags]: Organization and Person [Sentence]: (MASK)
Target: All Fishermen 's Association secretary N.J. Bose said the strike would continue indefinitely.
As sequence labeling tasks mostly involves with short sentences, we train copy of KnowDA to complete this step. In the Stage 2, following Arriaadi et al. (2022), the format of a training instances is:
Input: [Output Tags]: (MASK) [Sentence]: All Fishermen 's Association secretary N.J. Bose said the strike would continue indefinitely. Target: Organization All Fishermen 's Association; Person N.J. Bose.
Given the target outputs, we can easily map the entity strings back to the input sentences and obtain word-level BIO tags. Although this approach may fail when entity appears multiple times in the input sentence, we find such case rarely impact on the final labeling performance. We train a separated copy of KnowDA complete this step.
20

---

Published as a conference paper at ICLR 2023
Training KnowDA for Data Augmentation Interestingly, similar to Wang al. (2022). find it beneficial to add Soft Prompt (i.e., trainable vectors prepended to each layer of PLM) when training KnowDA to label entities. In this paper, directly use mandomly-initaaiize than the pre-trained Soft Prompt in PromDA). When we train KnowDA to label entities in Stage 2, frozen all parameters of KnowDA and only train the newly added Soft Prompt parameters with learning rate of Le - When we train KnowDA to generate input sentences, we simply fine-tune all KnowDA parameters with a learning rate of 5e-6
Soft Prompt (rather
peraationbbase Training Given the low-resource training data, we first train KnowDA generate synthetic input sentences without entity annotations, then train another copy of KnowDA to label entities over the synthetic input sentences. Interestingly, feeding these complete synthetic data, combined with few-shot training data, back to KnowDA can improve its entity labeling performance. KnowDA with stronger entity labeling performance can be used to annotate the synthetic input sentences again and produce better synthetic data. We thus iterate this process four times. In every iteration, the synthetic input sentences are fixed and we only update the entity annotations these sentences. The iteration results, including KnowDA's 's entity labeling performance and the performance of the BERT model trained with the corresponding synthetic data are shown Table 4 These results are all averaged across five different random seeds and data splits.
A.5 FEWGLUE DATA AUGMENTATION PROCEDURE
In the task BoolQ, RTE, CB, MultiRC and ReCoRD, we use KnowDA to generate long text without any further finettuning (i.e., zero-shot). All of these zero-shot components in Blue with the snow mark. All components with fine-tuning are Orange with the fire mark.
BoolQ
As shown in Figure the data augmentation procedure for the task of BoolQ has three steps: i) generating article; ii) generating question and iii) generating the final answer (i.e., True or False). As article in BoolQ are often relatively long, we generate it directly from KnowDA without any fine-tuning ge. zero-shot). We further separately fine-tune KnowDA for the rest two steps.
Article
Question
Answer
[Artide]:
Value
[Article]: Value
[Artide]:
<MASK>
[Questions] : Value
[Questions] <MASK>
[Answex]:
<MASK>
Figure 5: The Data Augmentation Procedure for the task of BoolQ.
As shown in Figure 6 the data augmentation procedure for the task of RTE has three steps: i) generating premise; ii) generating hypothesis and iii) generating the final relation label (i.e., whether premise and hypothesis are entailment). As premise in RTE are often relatively long, generate directly from KnowDA without any fine-tuning (e.g., zero-shot). We further separately fine-tune KnowDA for the rest two steps.
RTE
Premise
Hypothesis
Answer
[Premise]:
Value
[Premise] : Value
[Premise]:
<MASK>
[Hypothesis]:
Value
[Hypothesis] <MASK>
[Answer]:
<MASK>
Figure 6: The Data Augmentation Procedure for the task of RTE.
21

---

Published as a conference paper at ICLR 2023
As shown in Figure the data augmentation procedure for the task of CB has three steps: generating premise; ii) generating hypothesis and iii generating the final relation label (i.e., whether premise and hypothesis directly from KnowDA without any fine-tuning (e.g., zero-shot). The premise text CB are often dialogue-based, which is different from commonly seen premise in other NLI tasks. We therefore use the key "text" and full demonstrations to generate synthetic data for CB premise. We further separately fine-tune KnowDA for the rest two steps.
CB
entailment). As premise in CB often relatively long, generate
Premise
Hypothesis
Answer
[Premise]:
Value
[Premise] : Value
[Text]
<MASK>
[Hypothesis]
Value
[Hypothesis] <MASK>
[Answer]:
<MASK>
Figure 7: The Data Augmentation Procedure for the task of CB.
MultiRC As shown in Figure 8 the data augmentation procedure for the task of MultiRC has four steps: generating document; generating question: iii) generating answer and iv) generating label (i.e., whether the article, question and answer are correct). As document in MultiRC are often relatively long, we generate directly from KnowDA without any fine-tuning (e.g., zero-shot). We further separately fine-tune KnowDA for the rest three steps.
Document
Question
Answer
Label
[Document] : Value
Document]
<MASK>
[Question]:
Document]: : Value
[Document] Value [Quertion]: [Answex]
[Answer]: Value
[Question]: Value
<NASK>
[Label]: <NASK>
Figure 8: The Data Augmentation Procedure for the task of MultiRC.
WIC As shown in Figure the data augmentation procedure for the task of WiC has two steps: i) generating the disambiguation word as well the sentence pair containing the above word for selection; ii) generating final result (i.e., whether the words in the sentence pair have the same meaning). We fine-tune KnowDA for both steps.
WiC Instance
Answer
[Word]:
<MASKO> <MASKI>
[Word]:
Value
[Sentence1]:
[Sentence1]:
Value
[Sentence2]:
<MASK2>
[Sentence2]:
Value
[Answer]:
[MASK]
Figure 9: The Data Augmentation Procedure for the task WiC.
WSC
As shown in Figure I0 the data augmentation procedure for the task of WSC has two steps:
i) generating the sentences that include the noun (e.g., person name) and pronoun which refers the previous noun; ii) generating final result (i.e., whether the noun and pronoun refer to the same entity, such specifie person). We fine-tune KnowDA for both steps.
22

---

Published as a conference paper at ICLR 2023
WSC Input Text
Answer
[Text] :
<MASK>
[Text]
Value [MASK]
[Answer]
Figure 10: The Data Augmentation Procedure for the task of WSC.
As shown in Figure [1] the data augmentation procedure for the task COPA has two steps: i) generating new premise text given the options and questions. This is because given the premise and question, there could many possible options to generated. We find those generated synthetic data contributes little to the ALBERT and DeBERTa model performance in our preliminary experiments; ii) generating final answer (i.c., which option is correct given the premise and question). We fine-tune KnowDA for both steps.
CОРA
Premise
Answer
[Premise]:
<MASK>
[Premise]:
Value
[Question]:
Value
[Question]:
Value
[Solution1]:
Value
[Solution1]:
Value
[Solution2]:
Value
[Solution2]:
Value
[Answer]:
<MASK>
Figure 11: The Data Augmentation Procedure for the task of COPA.
ReCoRD As shown in Figure 12 the data augmentation procedure for the task of ReCoRD has four steps: generating document; ii) generating entity list; iii) generating query text and iv) verifying the answer by generating the entity. As document in ReCoRD are often relatively long, we generate it directly from KnowDA without any fine-tuning (e.g., zero-shot). We further separately fine-tune KnowDA for the rest three steps. When generating query, we generating full sentences without any placeholder. We will create placeholder for the query text if there is an entity appearing in the query text and the entity list.
Document
Entities
Query
Query
[Document]: [Entities]:
Value
[Document]: [Quexy]:
Value <NASK>
[Passage]:
[Document]:
<MASK>
[Entities]: : [Quexy]:
Value
<NASI>
Figure 12: The Data Augmentation Procedure for the task of ReCoRD.
A.6
CASE STUDY
In this section, we select eight representative examples from KnowDA the FewGLUE benchmark (i.e., one example for each task). In the task of BoolQ, RTE, CB, MultiRC and ReCoRD, we use KnowDA to generate long text without any further fine-tuning (i.e., zero-shot). We show that KnowDA could produce high-quality text with similar styles in this challenging setting. In the task of COPA. WSC and WiC, we find that after task-specfic fine- tinettunng, KnowDA could also generate appropriate text.
23

---

Published as a conference paper at ICLR 2023
BoolQ In the original BoolQ training data, there some instances about biography of famous athletes. KnowDA successfully produce another detailed, coherent and long athlete biography from the keys and demonstration examples, showing the success of KoMT.
Article: Selena Grace Anderson (born December 9, 1987) is an American football wide receiver for the University of Alabama Birmingham (UAB). She was drafted by the New Orleans Saints in the third round of the 2011 NFL Draft. She played college football at Syracuse. Questions: Is Selena Grace a basketball player? Answer: False
RTE Similar to the BoolQ task, many instances in the RTE task have long value in Premise. KnowDA successfully produce long and high-quality text for Premise only using key and demonstra- tions.
Hypothesis: Fort Lauderdale is part of the United States. Premise: The Millennium State Highway (the New Millennium) is west-bound U.S. highway that runs for 183 miles (195 km) between the cities of Fort Lauderdale and Fort Lauderdale-Davia Florida. Developed by the Federal Highway Administration 1956, one of the major arterial roads in the U.S. Answer: entailment
CB In the CB task, unlike above Wikipedia-style text, some instances are drawn from special domains, such as fiction (i.e., from British National Corpus) and spoken dialogue (i.e., from Switch- board). KnowDA still generate long text following these special styles.
Hypothesis: Jools Holland was the director of the movies. Premise: And who is the director of the movies in England? said Jools Holland who said he hoped we'd get some film or television. Answer: contradiction
CОРA In this example, the original premise in the training is "The customer filed complaint with the store manager.". Given options and question, KnowDA could generate different but coherent premise text.
The customer was upset. Solutionl: The sales associate undercharged the customer. Solution2: The sales associate acted rude to the customer. Question: What is the cause for this? Answer: 2
Premise:
WSC In the WSC task, KnowDA directly generates the whole Premise text with the noun and pronoun work marked with "** The generated text are coherent: the pronoun "he" clearly refers to "Toby". This shows that KnowDA has strong few-shot ability because it could quickly learn this pattern from only examples.
Premise: Stan's son *Toby* was playing football with his friends when *he* was injured
Answer: True
WIC In the WiC task, KnowDA directly generates word and the two sentences. The generated results follow the WiC data format: the sentence pair needs to include the word hand. This shows that KnowDA has strong few-shot ability because it could quickly learn this pattern from only 32 examples.
sentencel: You look like someone hand you a book. sentence2: pushed hand against a table. word: hand Answer: False
MultiRC KnowDA further demonstrate its ability in generating long text. The style of the article is similar to the training data in MultiRC. KnowDA also generates high-quality questions given the long article. This shows that KnowDA could make use the same skills for different tasks (e.g., in BoolQ, KnowDA also generates good questions).
The women and two were abducted on Thursday by students their way to class in the Cambridge St Andrews Catholic Primary School. The child had been released in April after getting school clothes, but police believe there was much blood her
Article:
24

---

Published as a conference paper at ICLR 2023
leg. The condition of the pupils is not known. The Catholic school, which opened in 1962, has a special focus on education. Sisters and brothers from the school remain trapped in the school, a service which has been called miracle. It is the first such ever celebration modern education. Question: What school opened 1962? Answer: Cambridge St. Andrews Catholic Primary School Label: True
ReCoRD In the ReCoRD task, there are a few instances talk about Ukraine. Interestingly, through the demonstration, KnowDA successfully generates another high quality piece of text regarding this topic.
Passage Ukraine's 's prime minister has said it is not encouraging tense talks between President Vladimir Putin and the opposition in order to expel the dissident republican leader. Ukraine's chief prosecutor accused Mr Putin of refusing to cooperate with the independent opposition, alleging that he had been building a grudge against the republican group. He was confronted in Chechnya on Sunday, but was not allowed to hold talks with the opposition. Ukraine was moved to emergency status as result of its violations of the ceasefire with Russia after protests in Kiev and Shenyang over the halt to other steps. President Vladimir Putin denies intelligence evidence that Kiev has been preparing a terror campaign to combat the rebel group. Ukraine said in the statement on Monday that it was preparing a prosecution, though there is no evidence linking the attacks to what is known main parties in the region. The two sides' treatment of each other Kiev is said to have been hostile act, with officers accusing Putin of triggering war, while the opposition, which not a ally the Russians, denies any involvement. Query Mr Putin not allowed to hold talks with the opposition in @ @placeholder because it would violate the ceasefire, Yuriy Yatsenyuk said. Answer Chechnya
parallel deal between the two
A.7 THE IMPACT OF THE NUMBER OF DEMONSTRATION EXEMPLARS
In this experiment, we expand the analysis in Section 3.3. to investigate the impact of the number of demonstration exemplars. Table 10 shows the PET ALBERT model performance in the task of RTE and CB when using & demonstration exemplars where & 0,1,2,3, The performance gap between the KnowDA model without exemplars and the one with exemplars is much larger than the KnowDA models using a different number of exemplars. Thus, is important to use exemplars in the KnowDA model, but its data augmentation performance is relatively robust to the choices of the number of exemplars.
Table 10: The impact of the number of demonstration exemplars in the PET ALBERT model.
Method
RTE (Acc.)
CB. (Acc./Fl)
0 exemplar
74.9
85,0/ 77.3
exemplar
78.1
87.7 83,9
2 exemplars
78.7
89.6 / 85,4
3 exemplars
78.9
89.1 / 85.0
25


Reviews:
[{"review_id": "nk7iWJm7DJ", "reviewer": "Reviewer_Fev8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the current paper, the authors present a new framework to perform data augmentation on low-resource NLP tasks. The core innovation is the idea of Knowledge mixture training (KoMT), that allows representing a large variety of tasks in a uniform way, in the form of keys and values pairs. This allows training the LM in a large range of NLP tasks through a denoising objective, where different combinations of full keys or values are masked. Through additional tools, such as the usage of demonstrators, and by also harnessing the zero-shot capabilities of the large LM model used as initial checkpoint (T5), the authors show the capabilities of KnowDA on tasks that require generating long text sequences. Furthermore, at generation time, the authors propose an autoregressive scheme, through some fine-tuning of the model after KoMT, to better adapt to the structure of each task, by establishing a dependency relation between the different features of the task at hand.\n\nThrough all the proposed innovations, the authors are able to provide significant improvement on many low-resource NLP tasks, more notably the fewGLUE benchmark, when compared to previous approaches such as FlipDA. Besides, they perform several ablation studies to underline the significance of the proposed methods.", "summary_of_the_review": "I believe the authors confront a challenging problem, not from the point of view of a reduced set of tasks, but from a really holistic perspective, tackling a large set of challenging NLP problems. The results reported are quite competitive, outperforming substantially previous approaches. Besides, the paper is really well written, and provides a clear intuition of the advancements implemented, understandable also for the readers non-versed on this particular field of NLP. For these reasons, I believe the paper paper is quite relevant, and should be accepted as it is. ", "strength_and_weaknesses": "The paper presents interesting and meaningful methodologies for the important problem of tackling low-resource NLP tasks. All the ideas proposed, even though based on previous well-known works, still offer a substantial degree of advancement, and move in reasonable and intuitive directions, which is always appreciated when discussing large LMs. The core idea of KnowDA, of harmonizing tasks' representation and training objectives, is clear, well-argued and nicely implemented. Besides, some additional tricks, such as the zero-shot plus demonstrators for long text generation, and the autoregressive approach for better generation, are quite reasonable and grounded to tackle some of the most complicated and challenging tasks.\n\nFurthermore, the authors do an excellent job in terms of exhaustively benchmarking their methodology, both from the point of view of the specific tasks considered, varied and challenging, and the number of baselines they compare against. And in most of these tasks, the methods proposed outperform previous ones. And when it is not the case, the authors try to give some intuition on why the methodology might down-perform in these cases, which is appreciated. From my point of view, the results are quite good, and prove the proposed methodology advances in the right direction in the problem of low-resource NLP.\n\nFinally, I also appreciate the extensive work done for the supplementary material, as it helps clarify many of the doubts that might arise during the main manuscript, and provide further valuable results and information to better understand the value of the current method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZVUnRc3Cdk", "reviewer": "Reviewer_CnYA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores a data augmentation method, called KnowDA, for low-source NLP tasks, aiming to capture task knowledge to improve the relevance and diversity of augmented data. The method first attempts to pre-train text-to-text generation model that will be used to generate augmented data for NLP tasks for data augmentation. To learn from diverse NLP tasks, the method select multiple NLP tasks with public datasets in Huggingface. Afterward, a key-value list format is introduced as a unified framework to represent instances of different NLP tasks. The key idea is to use keys to indicate feature functions (e.g., premise, hypothesis, tag) and values to capture string representations of feature content. KnowDA then uses denoising objectives to pre-train the model over converted instances of the NLP tasks (i.e., via masking selected values in the key-value list format for reconstruction from remaining information). In addition, KnowDA adds some selected demonstrations/instances from the same tasks to the input for the text-to-text formulation during pre-training. To leverage KnowDA for data generation, task-specific feature dependency is used to provide an order to generate keys and values to create new training data for a target task. To generate long values (e.g., documents), KnowDA uses zero-shot learning framework. The method is evaluated over low-resource experiments using FewGLUE, CoNLL'03 and WikiAnn (for sequence labeling). Experiments show better performance of KnowDA over several selected baselines. The paper also conduct some analysis to demonstrate the benefits of KnowDA, including ablation study, long text generation, data diversity, and human evaluation.\n", "summary_of_the_review": "The fairness of the comparison can be improved as KnowDA uses training data from multiple external NLP tasks that is not employed in the baselines. A text-to-text baseline can be considered to show the benefits of data augmentation.", "strength_and_weaknesses": "Strength:\n\n+The key-value list format to unify different NLP tasks for multi-task learning is interesting and seems helpful.\n\n+KnowDA achieves strong performance on the benchmark datasets for low-resource learning.\n\nWeakness:\n\n-Although interesting, the key-value list format is similar to the text generation formulation of NLP tasks that is studied extensively recently (e.g., [1]). These work should be discussed to highlight the model's benefits.\n\n-Compared to the baselines (e.g., FlipDA) that do not require training data of external NLP tasks, KnowDA needs to use datasets of multiple NLP tasks to pre-train its model. This might make the comparison less compatible and convincing in this work.\n\n-Given the training data of multiple NLP tasks for pre-training KnowDA, another possible baseline is to formulate the NLP tasks into the text-to-text problems using the key-value list format. Given a target NLP task, a text-to-text model can be trained over the external task data and the provided training data to directly solve the task (i.e., not using data augmentation). It might be helpful to discuss or evaluate this baseline to show the benefits of data augmentation given training data from external tasks.\n\n[1] Paolini et al. Structured prediction as translation between augmented natural language, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JR-FTcDkqQ", "reviewer": "Reviewer_U7D8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel data augmentation model and technique for improving performance of few shot and low resource NLP tasks. Based on a seq-to-seq pre-trained language model (T5 large), this paper first present a multi-task pre-finetuning on 100+ NLP task using a new Knowledge Mixture Training (KoMT) framework, where all the datasets are represented in an unified  key-value pair format, adding some demonstration exemplars at the beginning, and training using a de-noising objective. For augmenting a low resource NLP task, the new samples are generated autoregressively according to task specific feature dependency. To accomplish this, different copies of the pre-finetuned model are trained for each auto-regressive step. The proposed method is evaluated on benchmark low resource NLP datasets (FewGLUE, CoNLL’03, and WikiAnn) and shown improvements over previous SOTA baselines.", "summary_of_the_review": "The paper develops a novel multi-task pre-training technique for large language models, trained over 100+ NLP tasks, which can be subsequently used for data augmentation in a new unseen low resource NLP task. The paper demonstrates improved performance of the proposed data augmentation method on benchmark low resource NLP datasets. These new findings could be beneficial to the community.", "strength_and_weaknesses": "Strengths:\n \nThe paper is well written and easy to follow;\nThe proposed KoMT pre-finetuning technique is novel and the paper claims to be the first work to perform large scale multi-task pre-training on over 100+ NLP tasks;\nThe model is thoroughly evaluated, and shows improved data augmentation performance on low-resource NLP tasks. Human evaluation demonstrates that the generated data samples are closer to human generated samples than previous SOTA models.\n \nWeaknesses:\n \n1. Despite the improved data generation performance, several proposed techniques make the overall system less practical for real applications. For example, the auto-regressive generation based on feature-dependency requires training multiple copies of the same pre-trained model;\n\n2. It is unclear how the authors define/choose different task specific feature dependencies. Is it done randomly or using some domain specific knowledge? How will it be chosen for a brand new task? There are no experiments on the how sensitive is the performance on the choice of such feature dependency;\n \n3. Many key details are missing:  What is meant by using KnowDA as task solver? Does it mean the pre-trained seq-to-seq model is directly being finetuned on the sequence labeling task? If so, is this model different than the one being used to generated the augmentation samples?\nHow many exemplars are chosen (per sample) for each task during pre-training and fine-tuning? Is one exemplar sufficient? If more than one is needed then how sensitive is the model to this choice?\n\nTable 4 and its description on page 6 is not clear.\nFigure 3 caption: Please add what is meant by T and D. Figure captions should be self-contained and understandable.\n \nTypos:\n \nPage 2: “.. KoMT is more scalable and comprehensive because those works heavily reply on the human-crafted prompts ..” -> heavily rely on\nPage 2: “To summarize, contributes of this paper are following” -> contributions of this paper\nPage 6: “We conduct the shot-10 setting where 40 samples for CoNLL’03 and 30 samples for WikiAnn”  -> setting with 40 samples", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster