GDA: Grammar-based Data Augmentation for Text Classification using Slot Information
Authors: Joonghyuk Hahn
Hyunjoon Cheon
Elizabeth Grace Orwig
Su-Hyeon Kim
Sang-Ki Ko
Yo-Sub Han
Venue: EMNLP 2023 Findings (2023)
Keywords: data augmentation
rules of grammar
text classification
TL;DR: Our paper presents GDA, a grammar-based data augmentation strategy in text classification using slot information by ensuring semantically accurate and diverse sentence generation.

Abstract: Recent studies propose various data augmentation approaches to resolve the low-resource problem in natural language processing tasks. Data augmentation is a successful solution to this problem and recent strategies give variation on sentence structures to boost performance. However, these approaches can potentially lead to semantic errors and produce semantically noisy data due to the unregulated variation of sentence structures. In an effort to combat these semantic errors, we leverage slot information, the representation of the context of keywords from a sentence, and form a data augmentation strategy which we propose, called GDA. Our strategy employs algorithms that construct and manipulate rules of context-aware grammar, utilizing this slot information. The algorithms extract recurrent patterns by distinguishing words with slots and form the "rules of grammar"---a set of injective relations between a sentence's semantics and its syntactical structure---to augment the dataset. The augmentation is done in an automated manner with the constructed rules and thus, GDA is explainable and reliable without any human intervention. We evaluate GDA with state-of-the-art data augmentation techniques, including those using pre-trained language models, and the result illustrates that GDA outperforms all other data augmentation methods by 19.38%. Extensive experiments show that GDA is an effective data augmentation strategy that incorporates word semantics for more accurate and diverse data.

GDA: Grammar-based Data Augmentation for Text Classification using Slot Information
Joonghyuk Hahn'
Hyunjoon Cheon'
Elizabeth Orwig'
Su-Hyeon Kim² Sang-Ki Ko
Yo-Sub Han'
Yonsei University Kangwon National University UUniverstty of Seoul
(greghahn, hyun orwig, emmous] @yonsei.ac e kr tngus387207kkaangwa aa..aac.... ac.kr
sangkiko@uos. ar.kr
Abstract
Methods
Original:
soug Mars my playlist Bruto a song Bruno by Uptowa my playlist.
Recent studies propose various data augmen- tation approaches to resolve the low-resource problem natural language processing tasks. Data augmentation is a successful solution to this problem and recent strategics give variation on sentence structures boost performance. However, these approaches can potentially lead to semantic errors and produce semantically noisy data due to the unregulated variation of sentence structures. In an effort to combat these semantic errors, we leverage slot information, the representation of the context of keywords from sentence, and form a data augmentation strategy which we propose, called GDA. Our strategy employs algorithms that construct and manipulate rules of context-aware grammar, uti- lizing this slot information. The algorithms tract recurrent patterns by distinguishing words with slots and form the "rules of grammar" set of injective relations between sentence's 's semantics and its syntactical structure-to aug- ment the dataset. The augmentation is done in an automated manner with the constructed rules and thus, GDA is explainable and reliable without any human intervention. We evaluate GDA with tate-offthhe-as data augmentation techniques, including those using pre-trained language models, and the result illustrates that GDA outperforms all other data augmentation methods by 19.38%. Extensive experiments show that GDA is effective data augmenta- tion strategy that incorporates word semantics for more accurate and diverse data.
EDA
a Mars
my plaplist
SSMBA
my playlis
Mars io song by Mirs,
Add a song my playlis Bruno Mans liiis by Bruno Mars,
ALP
Adil a wong by Brano Mans to my Bruno Mars His playfie. Add a song by Bruno Mar to ay playlist Bruso Marses Hits,
GDA
Please add Bruno Mars to playlist Bruno Mars His..
Iteest a Bruno Mars Brano Mars His.
Table 1: Example sentences generated by GDA and state-of-the-ar data augmentation methods. Note the logical errors generated by EDA, SSMBA, and ALP as opposed GDA which generates contextually equiva- lent sentences with variations.
Language models often fail to learn the data distribution the (sampled) dataset is too small to characterize the population (low-resource prob- lem). Recent studies propose many techniques that alleviate the low-rcsource problem, such as data augmentation-a brooodn accepted technique in the field of NLP. Many data augmentation approaches such as EDA (Wei and Zou, 2019) and ALP (Kim et al., 2022) focus on syntactic variation to produce diverse data and have shown a definitive perfor- mance improvement. However, these strategies are not always reliable because they can lead to semantic errors the variation might negatively impact the meaning of sentence. We present some examples of sentences with logical errors in Table 1. As result, the augmented sentences, which are potentially nonsensical, may be harmful to a model degrading classifier performance. On the other hand, UDA (Xie et al., 2020) also shows definitive performance improvement, but employs back-translation and word replacement strategies in order to prevent semantic errors from occurring, limiting the production of syntactically versatile augmented sentences.
1 Introduction
Text classification is popular application of deep learning (Minace et al., 2021), and BERT-based language models achieve promising performance for the task (Devlin et al., 2018; Liu al., 2019; Lan et al., 2019). These popular language models, however, often exhibit a critical performance drop when the labeled data are insufficient.
In text classification, a data sample pairs a nat-

---

ural language sentence with its class, and model learns to determine the classes from given sen- tences. Several models utilize relations between the words to understand the given sentences, such as entities, part-of-speech (POS), and role seman- tics (Wu et al., 2018; Sood et al., 2020; Zhang et al., 2022). We can use this information to augment the training data and improve the performance of lan- guage models.
Original Sentences
Adele My Favorites
Add a song by Bruno Mars to my playlist This Bruno Mars Insert song of ABBA to The Greatest Pop Songs
Please add IU playlist Modern K POP
Rules Grammar
sartist splaylist
Add a song by sertiss to my playlist #playlist Insert song of sartist to splaylist
Please add sortist to my playlist aplaylist
Recent studics represent each sentence with a slot sequence in an effort to enhance sentence clas- sification (Goo et al., 2018; Wu et al., 2020; Qin et al., 2021). These studies, however, show limi- tations utilizing slot information because they only apply this information during the training phase of the models. We leverage slot informa- tion enrich the diversity of generated sentences data augmentation.
Manipulation GDA(combined)
Iplease [BLK]] (add) Insert) la song by a song of [BLK]) fartist to (my playlist [BIx]) iolaylist
Augmented Sentences GDA
Add song of IU to My Favorites Please insert song by ABBA to My Favorites Please add Bruno Mars my playlist The Greatest Pop Songs Insert to my playlist Hits 2010s
Figure overview of GDA. constructs RoGs by distinguishing keywords with slot information. We em- ploy manipulation for rules in more broad forms and then, generate sentences. The symbols preceding dollar sign (S) are substitution variables for slots.
Given sentence S = Wh W2 Wh. there is sequence of slots 8 8182 8m of S, which con- tains contextual information of words in S where S. n, and there class-label I of S, in which I represents the perceived intention of the sentence. See Figure 1 for an example.
with symbols representing slots. After analysis, the authors conclude the approach effective for low-resource problems.
= Please add Iris DeMent to my playlist this is Selena playlist
artist
PadToPlaylist
In similar vein, we propose grammar-based data augmentation (GDA) using slots. We first construct rules of grammar (RoGs) by substituting keywords in the sentences with their coinciding slots. We then apply heuristics inspired by formal language to generalize these RoGs. Such manipula- tions provide variations in sentence structures and augment syntactically propitious sentences. We then produce a variety of sentences from the RoGs and augment them to the training data. Figure 2 describes the overview of GDA. Our definition of GDA consists of three parts: RoG construction. rule manipulation, and sentence generation.
Figure 1: Example of sentence S, its slot sequence and class I. Words labeled by "-' denote the words with no corresponding contextual information.
While data augmentation in recent studies de- pends heavily words present in the sentences, we focus more on the implicit properties of the words. For instance in Figure 1, the slot 'art ist' indicates that Tris DeMent' describes name of an 'artist'. The other underlined phrase 'this is Selena' could be understood as an individual sen- tence. However, the slot 'play plaaylitt associated with the phrase implies that the phrase indicates name of playlist, rather than sentence by itself. These properties aid the model in comprehending such sentences precisely and act as a crucial compo- nent for interpreting the sentences. Thus, we utilize these implicit properties to gain more appropriate data for each class.
GDA is a powerful data augmentation method when conducted on sentences with slots. It shows a maximum of 19.38% improvement terms of pre- diction accuracy in the 5-shot experiment of SST-2, We also expand our method to more general types of datasets where slots are not initially available. With intuition that named entities contain con- textual information of the sentences (Mohit, 2014), conduct named entity recognition (NER) the sentences and infer RoGs by substituting words with their named entities. The experiment com- pares state-of-te of-th-o--he-arr data augmentation techniques
Slots work by partitioning sentence into mean- ingful word phrases. Hahn al. (2021) utilize the characteristics of slots in semi-supervised learn- ing for lseull-labeling procedures. Their work infers set of rules by substituting word phrases

---

GDA and demonstrates that GDA is successful implementation on typical text classification.
fies each of the inferred rules R to form a larger of rules R' that represents common structures of sentences. We present a detailed overview of the manipulation phase of GDA in Figure 3. this phase, we mix the rules carefully while main- taining their semantics. GDA then generates new sentences represented by the rules It',
2
Related works
Data augmentation solution to the data scarcity problem which works by generating artificial data from the original dataset, with the key objective producing accurate and diverse data. Traditional data augmentations involve synonym replacements random word insertion, deletion, or substitution. More advanced augmentation techniques, such as symbolic augmentation, utilize alternative repre- sentations of natural language sentences (Zhang et al.. 2020; Wei and Zou, 2019). For example, EDA (Wei and Zou, 2019) applies predefined oper- ations (rules) to modify sentences and Coulombe (2018) proposes utilizing regular expressions to filter and clean noisy data while augmenting it. Another line of research in data augmentation focuses on conditional generation using pre-trained language models-thh -te results of which have been promising, especially given their significant perfor- mance improvements (Devlin et al., 2018; Raffel et al.. 2020). Large pre-trained language models like GPT-3 (Brown et al., 2020) have gained popu- larity in data augmentation. However, the resource requirements for these models well as the pro- cess of fine-tuning such large models for specific datasets is often infeasible and poses a challenge for individual users and researchers.
3.1 Rule inference
An initial ruleset R for given sentences is context- free grammar and is constructed by replacing every keyword with variable for its slot. Context-free grammar G (V, Vo) is generative gram- mar that consists of two disjoint sets, V and T. of variables and terminals respectively, set S V x (V U TT) of production rules and a start vari- able Vo E V. We refer to the items in V U T by 'tokens'. A production p E P pairs a vari- able A and sequence X of tokens, which we denote A + X. We refer to the variable and the sequence in a production p as substitution variable and substitution sequence respectively. We often denote several productions A + X1, A + X2. . - Xx, having the same substitu- tion variable in an alternation sequence (or, alterna- tion) A + ses to denote a substitution sequence in single pro- duction instead of in separated productions for con- ciseness. For instance, 'A + (add insert) song' is a short notation of these three productions: 'A > A' a song'. 'A' + add', and 'A' + insert'. A ruleset R = (V, T, P. Vo) generates sentence by starting from Vo and replaces substitution vari- able with one of its substitution sequences in P until there are no variables left.
| We also use parenthe-
Recent studies explore the use of word proper- ties from given sentences, such as slot information in task-specific datasets, like ATIS and Snips, or named entities. In our work, we utilize this con- textual information to overcome the challenges of employing large language models (LLMs) and then propose rule-based data augmentation ap- proach based on context-free grammar.
Wh .. Wn in class I with its slot sequence s = $182 5m. for every us where its corresponding slot s, contains contextual information (i.e.. sy is not -), we replace such woy 's with its slot sj. Then, we pair the class label 7 and this modified sentence S' into a production. Finally, pair the start variable Vo and every class label T. Thus, we build context-free grammar G (V,T, Vo) where V is a set of slots, class labels, and other variables and T is a set words.
Given a sentence
3 Methodologies
GDA works three phases: inference, manipula- tion of the RoGs, and sampling sentences over the rules. Given a set S of sentences, we first infer the set R of RoGs from S. In the inference phase, GDA identifies keywords- word phrases to which slot is attached of sentences in S and replaces these phrases with variable representing the slots, thus contextually equivalent phrases are replaced with the same variable. For instance, 'Bruno Mars' and 'Adele' are replaced by "$artist'. Then in the manipulation phase, GDA combines and modi-
These initial rules effective but scarcely pro- duce sentences with high syntactic variation be- cause they maintain the structure of underlying sen- tences. This limitation of the initial rules, similar to the limitation of self self-train training, results in possibly

---

Initial Rules Grammar
Combined
#artist to #playlist
add add song
Add song fartist: my playlist &playlist
playlist
$playlist
Insert a song of Sartist to splaylist
insert song
Please add #artist my playlist &playlist
please add
playlist
Distance-basee
Keyword-baa based add
add a song Insert song of
to
to
add
playlist
add a song by insert song of please add
to my playlist
splaylist
#artist
splayliat
to
to
please add
playlist
to my playlist
(a) Distance-based ($3.2.1):
(please| [BLX]] adi fartist to (my [BLK]] (playlist [BLK]] $playlist (add insert) a song (by of) #artist to (my [BLK]] (playlist [BCX]] splaylist
(b) Keyword-based ($3.2.2):
(add add song insert song of please add) #artist (to to playlist) splaylist
Combined ($3.2.3):
(please [BLX]] (add Insert) (a song by song [BGX]] sartist (my playlist [BLK]) splay]
Figure 3: An overview of GDA manipulation methods. Details of manipulation strategies in the yellow box and the resulting rules by manipulations are in the blue box. [=Lx] denotes an empty word.
many rules there sentences in the class. This limitation potentially hinders the diversity of generated rules and thus the generation of varying sentences. For example, two rules 'add $song to playlist' and 'insert the song $song to playlist' ask to add the song denoted by the slot "$song' to some playlist, but the rules fail to generate sen- tence like S = 'add the song dynamite to playlist' the word phrase 'add the song' is not recognized by both rules. However, generate S by allowing the second rule to have "add' place of 'insert' or by allowing the first rule to generate 'the song' between 'add' and "$oong' On the other hand, we allow the second rulc to generate 'remove" in place of 'insert', the seman- tics of the generated sentence diverges from the class label (and the initial sentences). Thus it is important to preserve underlying semantics and grammar structure while transforming the initial rules represent various sentences.
sections briefly explain how these two rule manip- ulation algorithms work.
3.2.1 Distance-based rule manipulation
Edit-distance d( B) a dissimilarity metric over two substitution sequences and B, defined as the minimum cost of editing A into B, word-by-word We use three types of rules to edit sequences: in- serting, deleting, or replacing word. They have unit cost and are often referred to edit opera- tions. For instance, words such as 'IU" or 'Michael Jackson' qualify for word-to-word edits but a slot, such as 'sartis'', does not. Thus we expect that distance-based manipulation preserves the seman- tics, since d(A, B) anchors the sentence around the slots (the main keywords) and modifies the surrounding words. The syntactic structure is pre- served since distance-based manipulation does not reorder the structure of the rules.
make these rules
One concern with the distance function d(A, B) is that counts the edit operations, which is often proportional to the length of A and B, where the length of a rule is defined as the number of tokens in the rule. This leads to the false implication that only the number of different words is important. For example, if there exist two rules with dif- ferent words and length of 50 each, as well two rules with different words and length of 10 each, the dissimilarity of these rules would be misinterpreted when compared by counting edit
3.2 Rule manipulation
With the initially inferred rules, we propose tech- niques based on formal structures to extend the rules' coverage. We first introduce a distance-based rule manipulation algorithm which utilizes edit- distance extend the structure of rules. We then introduce a keyword-based rule manipulation algo- rithm which employs slot information to generate more contextually diverse sentences. The following

---

operations. Thus we use normalized distance func- tion d' (A, B) d(A, B) max {|A], } instead d(A,, A, B), where | and [B] are the lengths of A and B respectively.
(isis n) to build merged rule r. In DIST-MERGE, (z, y) denotes an alignment of two items x and yy. When an alignment pair consists of the same item, the algorithm uses the item J. On the other hand, we use the alternation of two target items (aly) so that the inference step can choose one of the possible target items. For in- stance in Figure 3(a), 'add' and 'insert' are aligned and the alternation of two words 'addlinsert' is written where 'add' and 'insert' appear.
Algorithm 1 Distance- manipulation. The inputs R and O in DIST-MANIPULATIONS )) are set of rules and upper threshold for normalized edit-distance, respectively.
1: function FDIT-SEQUENCFFA A, H)
2: return a sequence ([u1, bi),.... (an, bn]] of pairs
where A = 0102" Onn B = byby -. bas and Σ.= c(a,, b.) = d(A, B).
3.2.2 Keyword-based manipulation
The second approach gives priority the seman- tics of given RoGs. Since keywords imply the main intention of sentence, we hypothesize that inspect- ing keywords every rule can aid in categorizing rules with similar contexts. Under this assumption, we use the slots keywords for each sentence. Keyword-based manipulation replaces the words and expands the representative power of rules contain several new sentences different forms. Algorithm 2 describes the keyword- beseed manip- ulation and Figure 3(b) shows running example of keyword manipulation on four rules.
3: end function
4: function DIST-CLUSTER(R, e (0,1)
5:
i<
6: while R do
7:
Choose R random. C, + {x E R | d'(r,,) I e and, r3 and x have the same class lahel)
9:
R R
10:
iti+1
11:
end while
12: return (C1, C2....,
13:
end function
14:
function DIST-MERGE(C: = (ri, P., Baa)))
15:
T+-r
16:
for E do
17:
seq EDIT-SEQQEENC((S ris)
18:
r' [RT.K]
19:
for (ær, v) seq do
Algorithm 2 Keyword-basee manipulation. R is set of RoGs.
20:
if y then
21:
r'
P append the
KENAMANPPUAATTOOOOOO ATTLLLTIOOH R) 2: P partition R by the slots in rules
1: function KEY-MANIPUI
22:
clse
23:
- r' (xl9)
append an alternation
3:
for P E do
24:
end if end for
4: Extract maximal word sequences in P
25:
Construct new rule by adding alternations
26: y
of extracted sequences and slots
27: end
6:
end for
28: return r
7: return r
29:
end function
8: end function
30:
function DIST-MANIPULATIONNE (0, I))
31:
CI.CC....,,C C. < DIST-CLUSTER(I, @)
32:
-.-r DiST-MERGE(C.)
Note that the example rules in Figure 3(b) show different results from that of distance-basee ma- nipulation in Figure 3(a): Keyword-based manip- ulation merges set of sequence fragments, while distance-based manipulation generates alternations composed only of words.
33:
return
34: end function
Algorithm shows a pseudocode for the distance-based manipulation method. The main function is DIST-MANIPLLATIONN.. θ) (line 30) that takes two inputs, the set R of all rules and threshold O (0, 1] on normalized edit-distance. DIST-MANIPULAATION calls two functions DIST- CLUSTER and DIST-MERGE.
3.2.3 Combination strategy
We also present a third approach, a hybrid strat- egy consisting of both the aforementioned manip- ulation methods in order to generalize the repre- sentative power of RoGs further. This approach constructs rules that have more syntactic and mantic variation by combining the advantages of both manipulation strategies. For example, ap- plying keyword-based manipulation followed by the distance- based manipulation restricts the edit- distance to work only in the alternations. This
DIST-CIUSTER (line 4) partitions the given rules R. into several clusters {CihissN. Based on randomly chosen rule rı. DIST-CLUSTER builds cluster C₁ by collecting rules whose minimum nor- malized edit-distance from r; at most O DIST- MERGE (line 14) then analyzes the editing sequence the representative rule 7; and the other rules ri.j

---

reduces the syntactic errors of the rules by merging the word phrases grouped in an alternation well as individual words.
Dataset (without slots) #Class
#Train
#Valid
#Test
AGNews
120,000
7,600
IMDB
25,000
25,000
SST-2
67,349
872
Yahoo
10
1,400,000
60,000
3.3 Sentence generation
Dataset (with slots)
#Class
#Train
#Valid
#Test
ATIS
4,502
500
893
We generate sentences from the manipulated rules r = 21 .. In by substituting slots with their representing words or word phrases and vary- ing the surrounding words. An alternation or a slot often represents several word phrases. Thus, we can generate several distinct sentences. Algo- rithm 3 illustrates the generation procedure.
Snips
7
13,084
700
700
FR-MTO
30,521
4,181
8,621
MultiDoGO (6 subcategories)
Airline
11
15,438
2,053
4,063
Fastfood
14
15,391
2,101
4,485
Finance
19
14,493
2,212
4,421
Insurance
10
12,234
2,039
4,102
Media
16
13,551
1,747
3,488
Software
16
13,084
1,840
3,858
Table 2: Summary of original datasets.
Algorithm 3 Rule-based sampling algorithm for augmentation. All selections are at random.
1: function RETRIEVE-WORD(a)
The three datasets: AGNews, IMDB, and Yahoo! answers (Yahoo) do not contain validation set. We therefore split the initial train set into train and validation sets which are disjoint each other. On the other hand, the SST-2 dataset does not contain a labeled test set, and due to this we use the validation set its stead, splitting the train set to generate a replacement validation set.
2: return word or a word phrase having slot x.
3:
end function
4: function GENERATE(r = x12))
S: for ιi E do
6: if z;; is a slot (Sxxx) then
7:
00, + RETRIEVE- (UTRIEVE-WORD(E.) WO.D(a.)
8:
else if lr; is an alternation (as [ai[a2]- |a2|--1) then Choose oy
9:
10:
+ GENERATE(A;)
11:
clse
12:
t-
& should be word
13:
end if
4.2 Baseline
14:
end for
15:
return sentence UP2 Wa-
We use SlotRefine (Wu et al., 2020) and a co- interactive transformer (Qin et al., 2021) as two baseline models to evaluate GDA, as well as three tfate-of-the-an augmentation strategies: EDA (Wei and Zou, 2019), SSMBA (Ng et al., 2020), and ALP (Kim et al., 2022). These models utilize slot information for sentence class prediction and thus we use them as a basis for comparison. Also for proper analysis on the relative effectiveness com- pared with LLMs, we employ Bard (Thoppilan et al., 2022) and fine-tune BERT (Devlin et al., 2018) on each dataset for data augmentation.
16:
end function
4 Experiment settings
Following the few-shot settings of recent data aug- mentation research (Lin et al., 2023), we train the baseline models on 5- and 10-shot train sets sam- pled from each target dataset to simulate few-shot environment. We generate 500 sentences per class on each combination strategy. We also measure the prediction performance on the original, full dataset to estimate the best performance the model on each dataset. We present the performance of the full datasets that have slots in Appendix B.
4.3 Evaluation metrics
For the datasets without slots, we simply report the accuracy score. However, for the datasets with slots, we evaluate the effectiveness of our strategy with the average of F1 scores in order to decouple the score for each class label from the number of sentences for each class in the test dataset. We report scores of all combined strategies for each dataset in Appendix B.
4.1 Dataset
We target two types of datasets: those which con- tain slot information and those which do not. For datasets without slot information, we label enti- ties for each word in substitution for slots using pre-trained BERT in Stanford CoreNLP (Manning al., 2014) for NER task. Table summarizes the details of datasets used the experiment. A set of classes of a dataset a set of distinct labels cach sentence in the dataset.
Results and discussion
We employ GDA to augment datasets from Sec- tion 4.1 and use the baseline classifiers from Sec-

---

AGNews
IMDB
SST-2
Yahoo
5
10
5
10
5
10
5
10
EDA
78.89
80.72
60.32
69.80
56.22
53.96
55,49
63.12
SSMBA
78.65
84.68
66.43
63.36
56.34
59.05
53.17
61.50
ALP
82.30
86.18
67.05
71.29
63.40
69.72
55.19
64.16
GDA: without manipulation 82.66
82.93
55.21
76.60
69.95
75.57
47.19
52.00
GDA: combined manipulation
83.13
84.22
67.32
76.60
75.69
78.78
49.32
53.06
Table 3: Performance comparison of GDA and other baseline data augmentation strategies (EDA, SSMBA, and ALP). The accuracy (%) on each setting is reported and bold scores are highest scores on each setting. The results EDA, SSMBA, and ALP are from Kim et al. (2022).
tion 4.2 to evaluate the performance of augmen- tation. We illustrate the effectiveness of GDA in Tables and 4. We report scores of all combined strategies in Appendices A and B.
performance of GDA.
GDA achieves promising results, however, its performance is largely dependent on the existence of slot information. In other words, GDA is less effective in sentences where keywords are scarce. On the assumption that dataset contains more keywords if it is more domain-specific, pute the Kullback- Leibler divergence and observe that the IMDB and Yahoo datasets contain more common terms with scarce keywords than the other two datasets, i.e., the IMDB and Yahoo datasets are domain-unspecific. The major difference between the IMDB and Yahoo datasets is that almost half the sentences in the Yahoo dataset do not contain even one slot. Because GDA leverages slot information to give semantic and syntactic variation, sparsity or lacking slots in the dataset leads to relatively poor performance of our model. We present this Table 3, as the performance of GDA on the Yahoo dataset is relatively less sufficient in comparison to the other three datasets. We present a table of statistics and detailed analysis of how GDA's us- age of RoGs is less effective for the Yahoo dataset in Appendix D.
5.1 Comparison of data augmentation strategies
com-
Table 3 compares GDA with the baseline data augmentation models: EDA, SSMBA, and ALP. By utilizing named entities as slot information, GDA achieves compelling performance compared baseline augmentation models.
Among the baselines from Section 4.2, ALP the state-oftthe-se model which utilizes POS tag- ging for data augmentation (Kim et al., 2022). uses a formal conception probabiiistic context- free grammar (PCFG), the rules of which introduce hierarchical structure variation while preserving the context of the initial data. This syntactic variation leads decent improvements in the performance of text classification. On the other hand, since PCFG reflects the distribution of syntactic variation in the original dataset, it is hard to provide sentences with novel syntactic forms. We observe the limita- tions of ALP empirically in Table 3. Our approach, GDA, achieves more advanced scores for AGNews, IMDB, and SST-2 compared ALP. GDA is es- pecially effective in SST-2 showing performance increase by 19.38% for 5-shot and 13.00% for 10- shot from that of ALP.
5.2 Ablation study
From the initial rules of GDA, we construct a larger of rules with distance- and keyword ma- nipulation algorithms in Section 3.2.1 and 3.2.2. We present the results of the different combina- tions of GDA strategies in Table 4, along with the results of the hybrid strategy utilizing both rule ma- nipulations. However, LLMs have been regarded as effective solutions for NLP tasks, thus prompt- ing our investigation to whether not GDA is more effective than LLMs for data augmentation. We employ both Bard (Thoppilan et al., 2022) and BERT (Devlin al., 2018), widely used LLMss, combine with GDA for augmentation and compare the performance.
GDA produces slot-aware rules representing shared syntactical structures across all the initial sentences sharing the same class. By applying both distance- and keyword- hased manipulations, GDA generates more semantic and syntactie variations opposed using only one manipulation strategy. The scores of GDA(combined) in Table 3 represent the performance of GDA applied with both manip- ulation strategies. From this, we can observe that these rule manipulations definitively enhance the

---

ATIS
Snips
FB-MTO
MultiDoGO
5
10
5
10
5
10
5
10
Vanilla
32.01
49.86
59.58
79.55
51.73
69.90
48.49
59.98
BERT
38.74
53.17
70.12
83.95
46.40
54.59
47.87
55.84
Bard
30.01
43.32
58.10
70.38
48.54
54.54
55.35
60.75
GDA: without manipulation
49,22
69.86
74.16
80.59
64.19
66.74
54.45
62.72
GDA(distance): distance manipulation
51.83
69.36
79.25
75.01
63.01
70.32
55.28
62.05
GDA(key): + keyword manipulation
50.68
67.26
76.50
76.90
58.76
69.29
54.12
61.20
GDA(combined): both manipulation
52.60
71.92
78.28
79.69
64.45
70.31
56.35
63.25
GDA(combined) BERT
50.57
59.83
75.13
82.84
57.17
64.15
49.82
57.30
GDA(combined) Bard
47.40
56.23
70.88
80.38
57.99
70.27
55.98
61.74
Table Performance comparison of GDA, pre-trained language models (Bard, BERT), and their combined strategies. Each represents RoG-based, model-based, and mixed augmentation, respectively. The average of Fi-scores (%)
is reported and bold scores are the highest scores on each setting. The score without augmentation (Vanilla) is provided for estimating the effect of augmentation strategies.
Table exemplifies that the strategies of GDA are generally more effective than LLMs. For ex- ample, GDA outperforms the fine-tuned BERT by over 25% for ATIS and FB-MTO datasets. For MultiDoGO, GDA shows 1.8% performance im- provement for 5-shot and 4. 1 for 10-shot com- pared to Bard. This proves that, the performance GDA surpasses that of widely used LLMs. How- ever, GDA may not always suited for datasets such as Snips, as the results of 10-shot experiments prove less performant than that of fine-tuned BERT.
properties of a dataset.
Another experiment that was performed involves the combinations of GDA and LLMs, in which we combine both the generated sentences from GDA and from the LLMs to then train the baselines. While the combination appears promising for the 10-shot experiments of the Snips dataset, no sig- nificant improvements are observed on the rest of the datasets, as the overall performance generally declines from that of GDA(combined).
With regard to the manipulation strategies of GDA, GDA(distance) and GDA(key) represent GDA when it employs distance-based manipu- lation and keyword-based manipulation, respec- tively. GDA(combined) denotes our hybrid strat- egy comprised of both manipulation methods. We observe that GDA(combined) outperforms sin- gle manipulation approaches in general, an intu- itive result given that GDA(combined) produces sentences of more semantic and syntactic diver- sity, leveraging the performance improvement of both distance- and keyword-based manipulation techniques. GDA(combined), however, does not demonstrate a significant improvement for all datasets as combining manipulation strategies does not always benefit learning performance. The performance of GDA(combined) is approximately equivalent to that of GDA(distance) for the 10- shot experiment of the FB-MTO dataset. Similarly, for the 5-shot experiment of the Snips dataset, observe that GDA(distance) is more performant than GDA(combined). This suggests that while GDA(combined) generally performs well, its ac- tual performance varies depending on the specific
Conclusions
We have proposed novel approach, GDA which leverages the power of RoGs and slot informa- tion and demonstrated how GDA can effectively enhance data augmentation for text classification, particularly focusing on few-shot settings. GDA solves a major issue among recent data augmen- tation techniques in which they potentially gener- ate sentences with incorrect syntax or semantics, thereby limiting their effectiveness in improving performance. Our experimental results have show- cased the superiority of GDA over the state-of- the-art data augmentation methods, highlighting its performance gains. Furthermore, we have con- ducted a qualitative analysis of the manipulation strategies inherent to GDA, specifically examining the distance- and keyword- based manipulations. Through these empirical experiments and analyses, have confirmed that hybrid approach combin- ing both manipulation strategies yields the most effective results. Our approach has provided in- sights on the utilization of slot information and RoGs for data augmentation in text classification,

---

particularly within the challenging few-shot set- tings.
Joonghyuk Hahn, Hyunjoon Cheon, Kyuyeol Han, Cheongjae Lee, Junseok Kim, and Yo-Sub Han. 2021. Self-training using rules of grammar for few-shot NLU. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 4576-4581.
Limitations
While GDA has demonstrated promising results and outperformed other data augmentation meth- ods, it primarily targets few-shot settings. In one- shot learning scenarios, GDA is less effective as GDA generates sentences based the initial RoGs. Similarly, it does not have direct upplicability to zero-shot learning. Another limitation is the avail- ability of the slot information as this information is usually limited in general datasets. We have employed language models to mitigate these lim- itations; however, further research into RoGs and extracting yield significant advancements.
Hazel H. Kim, Daecheol Woo, Scong Joon Oh, Jeong- Won Cha, and Yo-Sub Han. 2022. ALP: Data aug- mentation using lexicalized PCFGs for few-shot text classification. In Proceedings of the AAAI-22 Confer- ence on Artificial Intelligence, pages 10894-10902
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Poyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint. 1909. 1942.
Yen Ting Lin, Alexandros Papangelis, Scokhwan Kim, Sungjin Lec, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani-Tur, 2023, Selective in-context data augmentation for intent de- tection using pointwise V-information. In Proceed- ings of the 17th Conference of the European Chap- of the Association for Computational Linguistics, pages 1455-1468.
Acknowledgements
This research was supported by the NRF grant (RS- 0023-00208094) and the AI Graduate School Pro- gram (No. 2020-0-01361) funded by the Korean government (MSIT). Han is a corresponding au- thor. The first two authors contributed equally to this work.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint, 1907.11692.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, pages 1877-1901.
Christoper D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc- Closky. 2014. The Stanford CoreNI. natural lan- guage processing toolkit. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60.
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Nar- jes Nikzad, Meysam Chenaghlu, and Jiangeng Gao. 2021. Deep learning- based text classification: A comprehensive review. ACM Computing Surveys, 54(3):62:1-62:40.
Behrang Mohit. 2014. Named entity recognition. In Natural Language Processing of Semitic Languages, pages 221- Springer Berlin, Heidelherg.
Claude Coulombe. 2018. Text data augmentation made simple by leveraging NLP cloud APIs. arXiv preprint, 1812.04718.
Nathan Ng. Kyunghyun Cho, and Marzyeh Ghassemi. 2020. SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robust- ness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 1268 11883
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 BERT: Pre-training of deep bidirectional transformers for language under- standing. arXiv preprint, 1810.04805.
Chih-Wen Goo, Guang Gao, Yun-Kar Kai Hsu, Chih-Li Huo, Tsung-Chie Chich Chen, Keng- Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 753-757.
Libo Qin. Tailu Liu, Wanxiang Che. Bingbing Kang. Sendong Zhao, and Ting Liu. 2021. A co-interactive transformer for joint slot filling and intent detection. In Proceedings of the 2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing. pages 8193-8197.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, (140)):1-67
Rongzhi Zhang, Yue Yu, and Chao Zhang. 2020. Se- qMix: Augmenting active sequence labeling via se- quence mixup. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing, pages 8566-8579.
Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang, Xiaofeng He, and Jun Huang. 2022. DKPLM: Decomposable pre-trained language model for natural language un- derstanding. In Thirty-Sixth AAAI Conference Artificial Intelligence, pages 11703-11711.
Ekta Sood, Simon Tannert, Philipp Müller, and Andreas Bulling. 2020. Improving natural language process- ing tasks with human gaze-guided neural attention. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, pages 6327-6341.
wwwwlldge-enhaneee
The BNC Consortium, 2005. The BNC Sampler, XML Version. Distributed by Oxford University Com- puting Services on behalf the BNC Consortium.
http:/ http://ww.tttt.... nww.wra.ccr....om ox ac. .k//
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzz Reze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrac Lec, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma. Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Deier-Hellsterrs Meredith Ringel Morris Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lec, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. 2022. LaMDA: Language models for dia- log applications. arXiv preprint, 2201.08238 082239.
Jason Wei and Kai Zou. 2019, EDA: Easy data aug- mentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing, pages 6382-6388
Di Wu, Liang Ding, Fan Lu, and Jian Xie. 2020. SlotRe- fine: fast non-autoreeresssvv model for joint intent detection and slot filling. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing, pages 1932-1937.
Lijun Wu, Fei Tian, Li Zhao, Jianhuang Lai, and Tie- Yan Liu. 2018. Word attention for sequence to se- quence text understanding. Secony-Seoond AAAI Conference Artificial Intelli- gence, pages 5578-554 5585.
Proceedings of the
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmenta- tion for consistency training. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020. NeurIPS 2020, pages 6256-6268.

---

A Full results of GDA
Table 5 demonstrates the performance of full strategies of GDA for the AGNews, IMDB, SST-2, and Yahoo datasets. Intuitively, named entities, used as slots, have potential noises. We confirm that the performance of the keyword-based manipulation strategy, GDA(keyword) is relatively poor compared to other rule manipulation and we can observe the result in Table 5. While the distance-based manipulation strategy, GDA(distance) shows more promising results than GDA(keyword), GDA(initial) is the most effective average. Our results indicate that employing manipulation strategies such GDA(keyword) and GDA(distance) for RoGs would produce more noise, resulting lower performance.
5, 10: applied GDA on (5, 10] (sh-thot train set.
Dataset
AGNews
IMDB
SST-2
Yahoo
k-shot
10
10
10
5
10
GDA(initial)
82.66
82.93
55.21
76.60
69.95
75.57
47.19
52.00
GDA(distance)
82.66
83.96
60.02
69.48
70.87
77.18
47.00
52.50
GDA(keyword
80.99
83.58
55.18
64.16
69.38
71.33
45.86
52.62
GDA(keyword)+ GDA(ed)
81.03
83.47
62.57
69.00
68.12
73.51
46.94
52.99
GDA(combined)
83.13
84.22
67.32
76.60
75.69
78.78
49.32
53.06
Table 5: Full result. Accuracy (%) is reported.
B Performance comparisons of all experiment settings
Tables and show an overview of the experiment results we conducted. The red lines each graph, marked by 'Full', indicate the F1 scores of models trained with the full dataset,
The augmentation settings on the scores we report are as follows:
Vanilla: The or 10-shot dataset. No augmentation is done at all.
Initial: Vanilla 500 sentences generated from the RoGs after the inference phase.
Keyword, Distance, Combined: Vanilla 500 sentences generated from RoGs keyword-based, distance-based, and combined manipulation strategies, respectively.
Keyword+Distance: Vanilla 500 sentences (in total) of:
250 sentences generated from RoGs keyword-based manipulation and
250 sentences generated from RoGs on distance-based manipulation.
(Keyword,
Disttaic,,ttnninnnnnn + Bard); Vanilla + 500 sentences (in total) of:
250 sentences generated from RoGs on keyword-based (distance-based and combined, respec- tively) strategy and
250 sentences generated from BERT or Bard.
Full: The full, original train set. This denotes the maximum score of the model on each dataset.

---

Datasets Baselines
Performances
ATIS
SlotRefine
Full
5
80
1000
60
40
Vanilla
Initial
Keyword
Distance
Keyword+ Keyword+Distanee Distance
Combined
BEERT
Keyword+ wardd Bard
Distance+ Bard
Combined Bard
co-interactive
5
transformer
80
Full
I010
60
40
Vanilla
Inirial
Keyword word
Distance
Keyword Neyword+Dishe Distance
Combined
Keyword BERT
Distance- BERT
Combined Mambined+BER BERT
Keyword++he -Bard
Distance Band
Snips SlotRefine
100
I15 5
Full
1110
80
60
Vanilla
Initial
Keyword
Distance
Meyword+Distane Distance
Combined
Keyword+ BERT
DistaneeBEER
Combined+BFR1
Keyword+ Bard
Distance Bard
Combined+Bal Bard
co-interactive
100
I 5
transformer
Full
90
Iu
80
70
60
Vanilla
Initial
Keyword
Distance
Keyword Deyword-Distames Distance
Combined
Keyword+BEs BERT
Distunce BERT
Combined+ BERT
BERT
Keyword+ii woood+BBad Bard
Distance+ Bard
Combined+ Bard
Table 6: Performance overview of the datasets with slots attached (1/2). The blue and orange bars denote the Fi score (%) of each strategy on 5- and 10-shot source datasets, respectively. The red line (Full) denotes the Fi score (%) of the original full dataset on each model as the expected best performance. The scores for MultiDoGO is an average of scores for six datasets.

---

Datasets
Baselines
Performances
100
FB-MTO
SlotRefine
5
Full
Im10
80
60
Vanilla
Initial
Keyword
Distance
Weydordd Distance
Combined
Keyword Keyword-BERR BERT
-BRR
Keyword Bard
Combinest + Bard
Distunee C
Distance+Ban
co-interactive
100
l15 5
transformer
Full
IU10
80
60
Vanilla
Initial
Keyword
Distance
Keyward+ Distance
Distance Co
Keyword-Bar Bard
DistaceeBard
Keyword-BEi
MultiDoGO SlotRefine
80
I15
Full
70
In
60
50
Vanilla
Initial
Keyword
Distance
Combined
Reyword-Disha Distance
Distance BERT
Combined+ BERT
Distance+ Keyword+ Reyword+Bard Bard
Combined+ Bard
Keyword+BER
Keyword
co-interactive
80
115
Full
transformer
I
70
50
Vanilla Initial
Keyword
Distance
Keyword+Dista Distance
Combined
Keyword+Biin BERT
Distance+BERs BERT
Combined BERT
Key ward+ Bard
Distance+ Bard
Combine+
Table 7: Performance overview of the datasets with slots attached (2/2). The blue and orange bars denote the
Fi score (%) of each strategy on 5- and 10-shot source dataset, respectively. The red line (Full) denotes the Fi score (%) of the original full dataset on each model as the expected best performance. The scores for MultiDoOO are the averages scores for six datasets.

---

C Distance threshold on distance-based manipulations
Figure illustrates the change of Fi scores on each training setting by three values of a threshold è for normalized edit-distance. In the distance-based manipulation algorithm (Algorithm 1), DIST-CLUSTER uses this threshold value è to select rules whose normalized edit-distance is less than θ and group these rules into a cluster.
Given two rules A = 'add song by $artis ist to $playlist' and B 'insert song of Sart ist my playlist $playl i ss'', compute the edit-distance by finding token-by-toker alignment that has the least number of pairs that contain different tokens. For example, we align A and B as in Figure
add a song by Sartist to [BLK]
[BLK]
Splaylist
insert
a song of Sartist to my playlist
Splaylist
Figure 4: An example alignment of two rules.
Since the alignment Figure has the fewest number of mismatched pairs (including the pairs having [BLK] token), 4 is the edit-distance between A and B. Then, since the normalized edit-distance is the edit-distance divided by the maximum length of either rules, 4/9 = 0.444 is the normalized edit-distance between A and B. If θ is greater than 4/9, these two rules will be in the same cluster.
Among the three thresholds, θ 0.3 shows the best performance as opposed the other threshold candidates. We presume that this value provides good balance between interpolating the syntactic and semantic structures and introducing grammatical errors in the rules. We adopt threshold value 0.3 when we experiment on the distance-based manipulation.
observe from Figure 5 that, in the combined manipulation settings,
80
Fi score (%)
70
60
50
0.3
0.5
0.8
Normalized edit-distance
ATIS 5-shot SlotRefine
ATIS 10-shot SlotRefine
ATTS 5-shot co-interactive x ATIS 10-shot co-interactive
Snips 5-shot SlotRefine
Snips 10-shot SlotRefine
Snips 5-shot co-interactive
Snips 10-shot co-interactive
Figure 5: Performance comparison edit-distance threshold for combined manipulation
D Analysis on low performance of the Yahoo dataset
The performance of GDA on the Yahoo dataset is inferior compared to the other datasets in Table 3. Among the datasets, AGNews (news), IMDB (movie review), and SST-2 (sentiment analysis) are specialized datasets for their respective purpose, but the Yahoo dataset contains general question-and andd-nswer sentences and is not domain-specific at all. We hypothesize that domain-specific datasets would show different distributions in the usage of special terms and exploit the term frequencies of each dataset. For two distributions P and Q, the Kullback-Leibhe divergence (KL--divergence) (R Q) = P(x) the expectation the log- ligeliied from P to Q, over P. estimates the difference Σ, P(x) log Q())

---

of the distribution P from the reference distribution Q. The value is always non-negative, and closer O when P and Q are similar distributions. The reference corpus we use is the British national corpus (BNC) (The BNC Consortium, 2005). BNC one of the reference corpora for British English, which contains 100M words of written (90%) and spoken (10%) English from the 1980s to the 1990s. Since BNC would contain the typical (British) English at that time, this corpus is a reasonable reference for computing the term frequencies of general English usage.
Table 8 shows the KL-divergence value that estimates the difference from the term frequency distribution of BNC to those of datasets for each strategy.
Dataset
1--hoot
Augmentation strategies
Vanilla
GDA(initial)
GDA(combined)
AGNews
3.184
3,933
3,934
2.736
3,303
3,280
IMDB
1.865
2.218
2.214
10
1.404
1,876
1.832
SST-2
5
3.511
4.165
4.211
10
3.313
3.753
4.137
Yahoo
5
1.682
2.178
2.115
10
1.579
1.957
2.023
Table 8: KL--dieereecce of term frequency distribution from BNC (The BNC Consortium, 2005) to each training dataset. We consider all sets with the same combination of six MultiDoGO datasets at once. 'Vanilla" column shows the performance of k-shot dataset before the augmentation. 'GDA(initial)" and 'GDA(combine)) columns show the performance of models trained by the sentences generated from the RoGs after the inference phase and after combined manipulation, respectively.
Table shows that the AGNews and SST-2 datasets have more dissimilar term frequency distribution from the reference corpus than the IMDB and Yahoo datasets. In other words, the first two datasets are more specialized in their respective topics than the others. The following examples show augmentation results on the IMDB and Yahoo datasets.
IMDB
I saw this film at SXSW with the director in attendance. ... I've seen at this festival Frownland is among the worst. ... Given that Frownland is Captain Beefheart song maybe you'l 'd have be able enjoy Trout Mask Replica on heavy rotation appreciate this film...
Yahoo
looking for fixer u property in Togo that will be sold auctioned or under HUD for extremely cheap. Gleen Falls
Here's 's the situation My dad is currently in the hospital looking at losing his leg. ... He is diabetic I don't 't know if that is what caused this The question is is there anything else that can be done (No slots in this text.)
Table 9: Samples of augmentation results the IMDB and Yahoo datasets. Applied the combined strategy on the 10-shot dataset. Some punctuations are removed after normalizing the dataset. Spacing is manually adjusted for easier reading.
The BERT NER model (Manning et al., 2014) we used, as mentioned in Section 4.1, recognizes three types of keywords: names of people (PER), places and locations (LOC), and other proper nouns (MISC). Considering that the purpose of the IMDB dataset is to predict the evaluation from its review, the text often contains several such keywords, for example, the name of directors, actors, and characters in the movie. Since GDA replaces these keywords with other words word phrases having the same contextual information. The augmented sentences are likely be distinct from the original sentence. On the other hand, the Yahoo dataset rarely contains keywords that NER easily recognizes (See Table 9). As result, the Yahoo dataset has 2060 sentences out 5100 (10 classes, 10 original sentences + 500 augmented for each class) which have no slot labels at all. Note that all 1020 sentences in the IMDB dataset have at least

---

one slot. This results in overfitting of the baseline models and inferior performance of the Yahoo dataset compared other text classification datasets.


Reviews:
[{"review_id": "e8oNOi8VrH", "reviewer": "Reviewer_LAee", "summary": "", "questions": "- Question A: Can you elaborate on the details of how BERT and Bard were used to generate sentences? It would be helpful to understand Table 4 better, which shows an unexpected result where an older / smaller model like BERT outperforms a much larger and newer model like Bard on half the datasets.\n- Question B: In Figure 6 and for SlotRefine on Snips, did you investigate the unexpected result of the 10-shot setting performing worse than the 5-shot setting for several techniques?  \n- Question C: What is the time efficiency of each step in the GDA algorithm? \n- Question D: Are the training results robust (the average of multiple runs)? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the detailed comments and suggestions. We appreciate the productive remarks that would improve our paper. We promise to incorporate your feedback in our final version.\n\n**1. Details on the usage of context-free grammar**\n\n>Reasons to Reject #1-1)  Unclear why / how the authors base their approach on a context-free grammar when natural language is context-dependent. For example, in the sentiment analysis dataset IMDB, there may be a \\\\$director slot. It appears that GDA treats all directors as interchangeable even though there are a range of reputations that might influence the label semantics. “This movie was made by \\\\$director, so you know what to expect!” would likely entail positive sentiment if \\\\$director $\\rightarrow$ Steven Spielberg, likely negative if \\\\$director $\\rightarrow$ M. Night Shyamalan. In the conclusion, the paper claims that GDA “solves” issues with semantics, but it doesn’t seem like this can be true, at least not fully.\n\n**[Response]** We would first like to state that formal grammar, such as context-free grammar (CFG), is widely used in NLP, where numerous studies pay testament to its usefulness in various NLP tasks [1-5]. Context-free grammar is present in numerous facets of NLP, and in particular, is becoming more popular in neuro-symbolic approaches for interpretability. For example, Amazon recently published results and analyses of Alexa’s efficiency with regard to generating sentences based on formal grammar [6]. At the cross-section of natural and formal language, CFG, derived from several types of formal grammar, composes the underlying framework in which sentences are structured. Phrase structuring and sentence diagramming rules also [7]  utilize CFG to express natural language sentences. In [5], the researchers engage the self-training by using context-free grammar to pseudo-label the data and show meaningful performance improvements.\n\nThe underlying rationale of GDA is based on rule sets derived from context-free grammar, but with the addition of slot information and rule manipulations, we are able to maintain the context of the original sentence. Our intention, which we stated in the abstract, but should have more thoroughly introduced in the methodologies section, is to make use of ‘context-aware grammar’, an extension of context-free grammar, as we are using the slot information as variables and restrict the grammar to generate sentences with semantic and syntactic similarity.\n\n>Reasons to Reject #1-2) For example, in the sentiment analysis dataset IMDB, there may be a \\\\$director slot. It appears that GDA treats all directors as interchangeable even though there are a range of reputations that might influence the label semantics. “This movie was made by \\\\$director, so you know what to expect!” would likely entail positive sentiment if \\\\$director $\\rightarrow$ Steven Spielberg, likely negative if \\\\$director $\\rightarrow$ M. Night Shyamalan. In the conclusion, the paper claims that GDA “solves” issues with semantics, but it doesn’t seem like this can be true, at least not fully.\n\n**[Response]** In regard to the influence of label semantics, this is an interesting and important facet of natural language, however we deemed it out of scope for our research as we aim to cover sentence semantics and syntax, but not necessarily sentiment. As we are dealing with data augmentation and producing sentences that are natural sounding and semantically and syntactically correct, we concluded that maintaining sentence sentiment does not impact the grammatical correctness of the augmented sentences. We do, however, find the idea of maintaining sentence sentiment while augmenting data to be intriguing and particularly interesting and would like to consider this as future work.\n\n\n[1] Valvoda et al., Benchmarking Compositionality with Formal Languages, Coling’22.\n\n[2] Igor Buzhinsky, Formalization of natural language requirements into temporal logics: a survey, INDIN’19.\n\n[3] Aydin et al., Domain Knowledge Representation Languages and Methods for Building Regulations, EBF’19.\n\n[4] Ziqi et al., Tab-CoT: Zero-shot Tabular Chain of Thought, ACL Findings’23.\n\n[5] Hahn et al., Self-training using rules of grammar for NLU, EMNLP Findings’21.\n\n[6] Tools for generating synthetic data helped bootstrap Alexa’s new-language releases: https://www.amazon.science/blog/tools-for-generating-synthetic-data-helped-bootstrap-alexas-new-language-releases.\n\n[7] An Intro to Phrase Structure Rules: https://www.linguisticsnetwork.com/an-intro-to-phrase-structure-rules.\n\n**2. Explanation of syntactic and semantic issues**\n\n>Reasons to Reject #2) In general, the claim of “solving” syntactic and semantic issues present in other augmentation techniques is left unquantified empirically, nor substantiated theoretically (likely not possible).\nFor syntax, authors could use a grammar checker like LanguageTool (https://github.com/languagetool-org/languagetool) to demonstrate that GDA does not introduce any new syntax issues.\nFor semantics, authors could use an approach like cleanlab (https://github.com/cleanlab/cleanlab) to demonstrate that GDA does not introduce any new label issues brought on by shifts in augmented semantics.\n\n**[Response]** The syntactic and semantic issues that we are claiming to solve relate to the possible logical errors (known syntactic and semantic issues) of current data augmentation (DA) techniques. Rather than focusing on logically correct sentence structure, we find that state-of-the-art DA methods use syntactic variation which begets logical errors. For instance, in Figure 1, ‘Add my playlist Bruno Mars Hits to a song by Bruno Mars’, generated by SSMBA, contains the logical error of adding a playlist to a song. Such errors can be potentially harmful to model performance, and thus are part of the impetus of our research\nOn the other hand, GDA, based on CFG, restricts the structure of generated sentences through the usage of rules. For richer syntactic and semantic diversity, we also propose the rule manipulations depicted in Figure 3. Figure 2 also showcases the rules and manipulations which produce the diversified sentences generated by GDA.\n\nWhile it’s true that there is currently a tradeoff between focusing on sentence correctness and generating diverse sentences, we would like to affirm that GDA’s sentence correctness, especially grammatically, is competitive. However, the reviewers’ comments and proposed tools for analyzing the grammatical correctness of generated sentences are valuable. We’ll provide a detailed analysis of the grammatical accuracy in the final version.\n\n**3. Details on the experimental setting**\n\n&nbsp; **a.** How we used BERT and Bard to generate sentences\n\n>Question A) Can you elaborate on the details of how BERT and Bard were used to generate sentences? It would be helpful to understand Table 4 better, which shows an unexpected result where an older / smaller model like BERT outperforms a much larger and newer model like Bard on half the datasets.\n\n**[Response]** Given a 5-shot and a 10-shot dataset, we fine-tune BERT (stated in 4.2 last sentence) and then generate sentences with the fine-tuned BERT. For Bard, we designed a prompt that generates sentences when given a 5-shot or a 10-shot dataset. For instance when conducting 5-shot learning, we take 5 sentences sampled from the initial training set and ask Bard to generate semantically similar sentences.\nWhile BERT is fine-tuned with the training dataset, Bard is only used when prompted. Although Bard is the better-performing language model, fine-tuning plays a more critical role for generating more appropriate sentences - thus creating the performance difference between BERT and Bard.\n\n&nbsp; **b.** Details on the Snips dataset\n\n>Question B) In Figure 6 and for SlotRefine on Snips, did you investigate the unexpected result of the 10-shot setting performing worse than the 5-shot setting for several techniques?\n\n**[Response]** We presented our full experimental result in Appendix B regarding the combinations of our rule manipulations and hybrid strategies of language models and GDA. In Table 6, the result of SlotRefine on Snips dataset is questionable as 5-shot performance generally shows higher scores than the 10-shot performance.\nUnfortunately we did not investigate this unexpected result, however we can extrapolate from the data in Table 6 along with the dataset characteristics that the reason for this is that rule manipulation, as employed by GDA, generates syntactic and semantic variants of the original rules. These variants generally perform well, but variants of the Snips dataset induce noise that degrades the performance of SlotRefine. While the 10-shot performance of various combinations of GDA achieves lower than those of 5-shot, the initial grammar of 10-shot achieves the higher performance than that of 5-shot. Furthermore, for 10-shot, the performance of initial grammar shows a higher score than the performance of keyword, distance, and combined manipulations. Thus, for SlotRefine, rule manipulations on Snips dataset are less performant than the other datasets.\n\n&nbsp; **c.** Details on the learning efficiency and robustness\n\n>Question C) What is the time efficiency of each step in the GDA algorithm?\n\n>Question D) Are the training results robust (the average of multiple runs)?\n\n**[Response]** With regard to Question C - GDA works by 1) constructing an initial ruleset R from the given training dataset, 2) applying rule manipulations on R, and 3) generating sentences. The time complexity of the GDA algorithm depends on the number of sentences and their corresponding lengths. When we denote the number of sentences of a given dataset as N and the maximum length of sentences as L, the time complexity of the algorithm is O(NL). Thus, it takes a negligible amount of time to generate sentences by using GDA. We will update the time efficiency of GDA in the final version. If the reviewer still has open concerns regarding the time complexity of GDA, we are happy to provide a detailed explanation.\nWith regard to Question D - We have multiple combinations of employing GDA depicted in Appendices A and B. We calculate the average scores of 5 runs for each combination.\n\nWe appreciate your comments and eruditions, including new outlooks for evaluating syntactic and semantic correctness of GDA. We hope that our rebuttal clarifies your concerns of our approach in terms of its soundness. If you consider our responses inadequate, please let us know. We are delighted to supply you with the proper details for your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- GDA explicitly considers both diversity and naturalness (syntactic / semantic fidelity) of the generations whereas many existing techniques yield ungrammatical, implausible, or incoherent texts. Some would argue that low-quality synthetic texts are acceptable because they still improve downstream model performance and generalization. However, in the real world, people often place significant weight on readability and having a technique that improves performance while aiming to be natural can help overcome objections from human decision makers and increase the likelihood of use. \n- The evaluation setup acknowledges the limitations of the augmentation to low resource settings and sufficiently demonstrates its benefits relative to several existing baselines.\n", "reasons_to_reject": "- Unclear why / how the authors base their approach on a context-free grammar when natural language is context-dependent. For example, in the sentiment analysis dataset IMDB, there may be a \\\\$director slot. It appears that GDA treats all directors as interchangeable even though there are a range of reputations that might influence the label semantics. “This movie was made by \\\\$director, so you know what to expect!” would likely entail positive sentiment if \\\\$director --> Steven Spielberg, likely negative if \\\\$director --> M. Night Shyamalan. In the conclusion, the paper claims that GDA “solves” issues with semantics, but it doesn’t seem like this can be true, at least not fully. \n- In general, the claim of “solving” syntactic and semantic issues present in other augmentation techniques is left unquantified empirically, nor substantiated theoretically (likely not possible). \n    - For syntax, authors could use a grammar checker like LanguageTool (https://github.com/languagetool-org/languagetool) to demonstrate that GDA does not introduce any new syntax issues.\n    - For semantics, authors could use an approach like cleanlab (https://github.com/cleanlab/cleanlab) to demonstrate that GDA does not introduce any new label issues brought on by shifts in augmented semantics. ", "questions_to_authors": "- Question A: Can you elaborate on the details of how BERT and Bard were used to generate sentences? It would be helpful to understand Table 4 better, which shows an unexpected result where an older / smaller model like BERT outperforms a much larger and newer model like Bard on half the datasets.\n- Question B: In Figure 6 and for SlotRefine on Snips, did you investigate the unexpected result of the 10-shot setting performing worse than the 5-shot setting for several techniques?  \n- Question C: What is the time efficiency of each step in the GDA algorithm? \n- Question D: Are the training results robust (the average of multiple runs)? ", "paper_topic_and_main_contributions": "This paper proposes an augmentation scheme that takes the following steps. Given a set of source texts, (1) construct rules of grammar (RoG), which roughly corresponds to replacing NER tagged spans with placeholder slot variables like $artist, (2) alter the RoGs with one or more manipulation strategies to improve their contextual diversity, and finally (3) generate new sentences by sampling fill-values for the slots / contextual alternations. Experiments on several benchmark datasets show the benefits of GDA, especially for the lower resource settings and for datasets with more specific NER taggable tokens. Authors also went to great lengths to understand a key limitation of their technique, namely that it does not work on texts lacking NER taggable tokens. \n\nContributions include a new, well-performing augmentation pipeline that generates contextually diverse texts suitable for general NLP tasks. The analysis of why GDA fails to improve training for certain datasets represents a potentially useful diagnostic technique for future research in text augmentation, especially any techniques dependent on NER tagging / slot information. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Line 170 - missing citations for ATIS and Snips datasets\n- Line 386 - missing citations for AGNews, IMDB, and Yahoo datasets\n- Line 390 - missing citation for SST-2 dataset"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Line 196 - “... is context-free grammar” --> missing “a” before context-free\n\nPresentation:\n- Table 3 - recommend clarifying that the 5 / 10 row is for the shot-count. \n- Lines 551-554 - the conclusions are stated rather strongly and without qualifications. For example, GDA appears to significantly worsen performance on datasets like Yahoo where there are not a lot of opportunities for slot annotation. \n\nDifficult to Follow:\n- Line 177 - the 3 phases of GDA do not match the ones described earlier on line 125. Recommend consistent terminology. Why does inference == RoG construction? Inference suggests a model is being used. Is it the NER tagging model? If so, how does GDA come up with the exact slot names like \\\\$artist and \\\\$song, which are more specific than person, place, and misc from NER tagging?\n- Lines 196-232 - the terminology is not easy to follow and rule inference is unclear. For example, why do G and R share the same expression? Ultimately, \n- Lines 400-402 - “These models…” is a confusing continuation from the previous sentence where the most recent items discussed are augmentation strategies.  \n- Line 468 - Unclear how IMDB is domain-unspecific since it exclusively features movie reviews and likely contains a lot of opportunities for slots like \\\\$actor, \\\\$movie, etc. In line 816, the paper appears to contradict itself by agreeing that IMDB is specialized. \n- Lines 493-496 - Unclear how Bard and BERT were used in conjunction with GDA. Also it's a bit strange now to consider BERT an LLM with less than a billion params. \n- Lines 823-827 - Unclear why a British corpus is being used to compare against datasets that tend to learn more towards American English. However, I still can see the value in performing this comparison to illustrate the point that Yahoo is more “generic” than other studied datasets. This point might have also been possible to make via perplexity or Salazar 2020’s LM scores, which are biased towards scoring general texts more highly. \n    - Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.\n"}}, {"review_id": "5nforeIbzZ", "reviewer": "Reviewer_2Dha", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the extensive comments and recommendations. We value constructive remarks that could enhance our paper. We will make sure to include your feedback in the revised version.\n\n**1. Dependence on slot information (e.g., named entities, slots)**\n\n>Reasons to Reject #1) The limitations of this paper include the dependence on slot information, making it less effective for datasets lacking such information. There's also a risk that the presented hybrid strategy may not consistently outperform single manipulation approaches, leading to variable performance across different datasets. The reliance on named entities extracted by a language model for slot information could introduce noise or inaccuracies, affecting the overall effectiveness of the technique.\n\n**[Response]** Slot information plays an important role in the distance- and keyword-based manipulation strategies employed by GDA when constructing rules. However, there is not a dependency on designated slot information in the data. We use a pre-trained BERT to extract named entities, which are then utilized as slot information, and show how GDA performs well with general datasets such as AGNews and IMDB in Table 3.\n\nAs noted by the reviewer, it is true that the hybrid approach of using language models to extract named entities might produce unexpected noises, an issue that we illustrate in Section 7. However, while our approach is useful for both datasets with and without slot information, we are cautious of employing language models solely for the purpose of yielding slot information. Thus, as described in Section 7, we aim to incorporate rule-based algorithms to solve the potential danger of employing language models for non-slot information related tasks.\n\n**2. The limitation of GDA for zero- and one-shot learning**\n\n>Reasons to Reject #2) Additionally, the approach may not be suitable for one-shot learning scenarios and zero-shot learning, potentially limiting its applicability.\n\n**[Response]** Recent research on few-shot text classification is mainly focused on 5- and 10-shot settings as securing several sentences for each class are guaranteed in this field of research [1-3]. Our paper also aims for the few-shot text classification where several sentences for each class are given. Thus, zero- and one-shot learning are not prioritized in this paper. We do, however, acknowledge this limitation in Section 7. We cannot solely use GDA for zero- and one-shot learning, but we can employ a language model for a few initial sentences and then use GDA in some sort of hybrid strategy. We’ll issue a more detailed explanation on how to use GDA for zero- and one-shot learning in the final version.\n\n[1] Kim et al., ALP: Data Augmentation Using Lexicalized PCFGs for Few-Shot Text Classifcation, AAAI’22.\n\n[2] Sun et al., Hierarchical Attention Prototypical Networks for Few-Shot Text Classification, EMNLP’19.\n\n[3] Lyu et al., Few-Shot Text Classification with Edge-Labeling Graph Neural Network-Based Prototypical Network, Coling’20.\n\nOverall, we acknowledged the limitations of GDA that the reviewer expressed concerns about in Section 7.\nWe are grateful for your valuable insights and perspectives, particularly the detailed observations on GDA. We aim to address your concerns regarding the robustness of our approach through our rebuttal. If you find our responses insufficient, please let us know, and we would be pleased to provide additional information to address any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Enhanced Data Augmentation Strategies: The paper introduces manipulation strategies that enhance both semantic and syntactic diversity in augmented data. This innovation is particularly important in preventing incorrect syntax and semantics in generated sentences, improving the overall quality of the augmentation process.\n\n- The approach demonstrates significantly improved performance compared to other existing techniques, essentially presenting a enhanced iteration of EDA (Easy Data Augmentation).", "reasons_to_reject": "The limitations of this paper include the dependence on slot information, making it less effective for datasets lacking such information. Additionally, the approach may not be suitable for one-shot learning scenarios and zero-shot learning, potentially limiting its applicability. There's also a risk that the presented hybrid strategy may not consistently outperform single manipulation approaches, leading to variable performance across different datasets. The reliance on named entities extracted by a language model for slot information could introduce noise or inaccuracies, affecting the overall effectiveness of the technique. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a novel approach called \"Guided Data Augmentation\" (GDA) for enhancing data augmentation techniques in the context of text classification, particularly focusing on few-shot learning scenarios. The main problem addressed by the paper is the limitations of existing data augmentation methods in generating sentences with accurate syntax and semantics, which hinders their effectiveness in improving text classification performance, especially in scenarios with limited training data.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8N7gJdAfal", "reviewer": "Reviewer_VFyi", "summary": "", "questions": "A. line 200/201 and after: the use of the word \"variables\" here is, in my experience, unusual. I would suggest instead talking about \"terminals\" and \"non-terminals\", in keeping with traditional CFG terminology.\n\nB. lines 255-257: in what way does this approach ensure that underlying semantics and grammatical structure are preserved? \n\nC. section 3.2.1 is confusing to me: the edits proposed seem to be okay, but I find this description confusing and somewhat misleading. it's not easy to see why these should be called \"distance-based\", nor what edit-distance has to do with producing these types of rules. Is the idea that you're taking two sequences/sentences that have a similar ordering of slots (and perhaps the same intent?) and generating a rule that could produce either sequence, using traditional edit operations? This part of the approach is not as clearly explained as most other parts. \n\nD. section 3.2.2: what is the definition of \"keyword\" being used here? It's a big claim to say that \"keywords imply the main intention of a sentence\", especially without explaining what counts as a keyword in general. (I understand that slots are taken to be keywords under this approach, but that is an approximation - how would you define \"keyword\" more generally?)\n\nE. Algorithm 2: do the rules grouped here need to share the same class? \n\nF. line 465: over what are you computing KL divergence? More detail would be helpful here.\n\nG. line 555: the conclusion mentions qualitative analysis, but I don't see any qualitative analysis in the body of the paper. I'm sure I missed something - to what does this refer?\n\nFinally, thanks for your good work!", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to begin by expressing our appreciation for the thorough comments and proposals. We are thankful for the productive remarks that could refine our paper. We assure you that we will incorporate your feedback in the final version.\n\n**1. Details on different types of experiments in the paper**\n\n>Reasons to Reject #1) Inconsistency of experimental settings and reporting of experimental results.\nThe paper claims that GDA, which is designed specifically to use slot information encoded in existing datasets, outperforms other similar data augmentation approaches (specifically, EDA, SSMBA, and ALP), while avoiding \"semantic errors\" and inappropriate changes to sentence structure. The experiments presented, though, only compare to these systems on datasets without original slot information, and for which slot information has been approximated using automated named entity labeling. There is a second set of experiments over the claimed target data type - datasets with slot information labeled - but these experiments do not compare to the other DA methods described in the paper. Instead, they compare to use of LLMs for generating new data. There is no explanation in the paper of why the second set of experiments does not include comparisons to EDA, SSMBA, and ALP, and I cannot myself come up with a reasonable justification for this omission.\nIn addition, the results that do show improvements over comparable SOA methods (those reported in Table 3) report performance with respect to accuracy rather than F1, again without a clear motivation. These factors combined reduce my confidence in the results reported, because they are not motivated. A more minor point - I would question some of the claims made about GDA outperforming other systems, as no significance testing seems to have been done. For example, the paper claims that GDA outperforms ALP for the IMDB dataset, but the difference is tiny - 67.32 accuracy compared to 67.05.\n\n**[Response]** There are two sets of datasets: those without slot information and those with slot information. Our approach achieves high performance for the datasets with slot information, but we are also concerned whether our approach is also applicable for general text classification datasets with no designated slot information.\nThe purpose of the experiment of the datasets without slot information is to inspect whether GDA can achieve state-of-the-art performance even on the general datasets for text classification that do not contain designated slot information. Table 3 highlights the performance improvements compared to the baseline DA models.\nOn the other hand, experiments on the datasets including slot information primarily focus on how GDA, the rule-based data augmentation method, performs better than the language models. We also inspect various combinations of GDA from this set of datasets. The focus of experiments for the datasets with slot information is more centered on the hybrid strategies of GDA in combination with the language models.\nWe did not think it fair to compare the baseline DAs to GDA because they do not make use of slot information. However, we will make sure to clearly describe the different intentions of the experiments for the sets of datasets including and excluding slot information in the final version.\n\n**2. Explanation of strengths of GDA in syntactic and semantic diversity & error correction in a linguistic manner**\n\n>Reasons to Reject #2) Overly strong argumentation regarding elimination/reduction of semantic errors.\n\n\nStarting from the abstract, the paper claims that GDA, by incorporating grammar rules, reduces the occurrence of certain semantic errors common to comparable DA approaches. The only support for this claim is through the examples shown in Table 1. Even with these carefully selected examples, the errors shown are not all semantic errors, and generally are not due to changes in sentence structure. Instead, they're because of either ill-formed NPs ('Bruno Hits Mars') or bad selectional restrictions ('add my playlist to a song') -- even in the latter case, the structure of the sentence doesn't change. The Table 1 errors are claimed to be logical errors, but they are not at all consistent error types. The two ALP examples are semantically fine, in my opinion. SSMBA has one logical error (can't add a playlist to a song) and one PP-attachment error. EDA has one factual error (not equivalent to a logical error) and one clunky (but possible) NP structure.\n\n\nOn another point, the abstract claims that the \"rules of grammar\" produced by the GDA approach represent \"relations between a sentence's semantics and its syntactical structure\". I can only read this as true with a very loose definition of what is meant by both \"semantics\" and \"syntactical structure\" -- as a linguist, this wording leads me to expect a representation that actually incorporates syntactic constraints (or at least syntactic classes) and some sort of semantic representation. The only semantic representation I see, though, is the use of (rather domain-specific) informational categories as slot labels (e.g. artist | playlist). The rules produced look like sophisticated templates for generating text; they do not represent either syntactic or semantic information in the usual sense (again, I'm speaking from the perspective of linguistics).\n\n>Reasons to Reject #3-a,b) Some claims not supported by results.\na. Claim about types of errors made by GDA vs. those made by other comparable approaches. The paper claims several times that GDA reduces semantic errors, but does not demonstrate that. Around lines 448-457, the paper claims that GDA generates more semantic and syntactic variations, but this is not demonstrated, at least not that I see. It would be interesting to perform a study of the sets of sentences generated by the various approaches based on the same input data!\nb. Claim that Table 3 demonstrates improvements of GDA over ALP. This feels like a stretch - performance of GDA is better for some data sets, but only some (mostly just SST-2, where evaluation is happening on a fairly small set. Also, not clear how this strategy really differs from ALP. If ALP also uses CFGs, this should have been noted much earlier - the paper leans toward presenting use of CFG-like structures as something novel.\n\n**[Response]** While it’s true that the sentences in Table 1 do not completely exemplify grammatical or semantic error, such logical errors still critically downgrade the performance of baselines and are thus harmful to the classification performance. The reviewer pointed out that the sentence examples of ALP are not sufficient for showing semantic errors of ALP. Given the following 5 sentences, I’ll provide the sentences that ALP and GDA generate:\n\n* i'd like flight information from dallas fort worth to boston on tuesday,\n\n* list all afternoon flights on united airlines from san francisco to denver,\n\n* is there an atlanta flight to denver connecting,\n\n* i want to fly boston to dallas,\n\n* tell me the last flight from atlanta to philadelphia.\n\nThe following examples are the sentences that ALP generates:\n\n* Dallas Fort Worth to Boston flight information on Tuesday,\n\n* List all afternoon flights on United Airlines Dallas Fort Worth to Boston,\n\n* Is there a direct flight from Atlanta to Denver,\n\n* Fly Boston to Dallas,\n\n* Atlanta flight to Denver connecting.\n\nThe above sentences might look contextually right in the perspective of humans, but the grammatical errors in sentences such as “Fly Boston to Dallas” and “Atlanta flight to Denver connecting” are unlikely to be golden data.\n\nThe following examples are the sentences that GDA generates:\n\n* list all afternoon flights on united airlines from atlanta to philadelphia,\n\n* I'd like flight information from atlanta to denver connecting,\n\n* Tell me the flight from boston to dallas,\n\n* Is there flight information from dallas fort worth to boston,\n\n* I want to fly san francisco to denver on tuesday.\n\nWhile the above sentences maintain the grammatical correctness, they enrich the syntactic representation, contributing to the strong performance of GDA.\n\nWhile we make the claim that each baseline has its own corresponding weakness, we also state the semantic and syntactic diversity that GDA provides. We are grateful that the reviewer understands the core of our initial rule generation. We depend on the slot information (either represented with slot labels or named entities) for rule generation. The initial ruleset, as the reviewer pointed out, lacks the representation of semantic and syntactic information. Thus, we use rule manipulations to enhance the weakness of the initial ruleset.\nFrom rule manipulations, we first find similar sentences both in their structures and semantics. Then, by distance-based manipulation, we align words based on the edit-distance metric and group the sequences of words that can be used to replace one another. Example rules for distance-based manipulation are shown in Figure 3(a). When utilizing keyword-based manipulation, we group the sequences based on the slot information. This example rule for keyword-based manipulation is shown in Figure 3(b). The rules show the syntactic information of possible variants that GDA can generate. We believe the syntactic information of GDA rules guarantees the corresponding syntactic diversity that enhances classification performance. However, as we are approaching this from a formal language and computer science background we would be grateful if the reviewer would comment about a more sophisticated representation of syntactic information and the connotation of syntactic diversity from a linguistic perspective.\n\nWe find slot information is a good representation of the semantic information that rules carry.\nIf the reviewer disagrees, it would be helpful and we would appreciate it if the reviewer would provide which representation would be adequate for semantics.\n\n\n**3. Clarification on the interpretation of the result**\n\n>Reasons to Reject #3-a,c) Some claims not supported by results.\na. Claim about types of errors made by GDA vs. those made by other comparable approaches. The paper claims several times that GDA reduces semantic errors, but does not demonstrate that. Around lines 448-457, the paper claims that GDA generates more semantic and syntactic variations, but this is not demonstrated, at least not that I see. It would be interesting to perform a study of the sets of sentences generated by the various approaches based on the same input data!\nc. No significance testing - some of the differences claimed to show better performance, whether compared to another model or comparing across different datasets, seem to be very very small.\n>Question F) line 465: over what are you computing KL divergence? More detail would be helpful here.\n\n**[Response]** We use Tables 3 and 4 to highlight and empirically confirm that GDA is effective, and also present sentences generated by the baselines and GDA in Table 1 for comparison. We agree that case studies on sentence generation would be helpful to intuitively understand the result of GDA, however due to the space limit, we did not present such a study regarding the comparison of actual data. We appreciate your suggestion and will make sure to add an additional page presenting a case study on comparing sentences generated by the baselines and GDA in the final version of our paper.\n\nWe appreciate the reviewer’s comment on the performance in Table 3. ALP performs better for the 10-shot experiment on AGNews and experiments of Yahoo while GDA performs better for IMDB, SST-2, and 5-shot of AGNews. GDA performs especially well for SST-2, however,2, however, the performance of GDA for Yahoo shows a large performance gap even compared to other baselines. The reason for this is that sentences in the Yahoo dataset scarcely contain named entities. Through analysis of KL divergence on the distribution of total words and slot information (named entities for the Yahoo dataset) from the Yahoo dataset, we observed that the sentences from the dataset scarcely contain words with named entities. We present a detailed case analysis of KL divergence for the Yahoo dataset in Appendix D.\n\n\n**4. Detailed description of our rule inference and manipulation**\n\n>Question B) lines 255-257: in what way does this approach ensure that underlying semantics and grammatical structure are preserved?\n\n>Question C) section 3.2.1 is confusing to me: the edits proposed seem to be okay, but I find this description confusing and somewhat misleading. it's not easy to see why these should be called \"distance-based\", nor what edit-distance has to do with producing these types of rules. Is the idea that you're taking two sequences/sentences that have a similar ordering of slots (and perhaps the same intent?) and generating a rule that could produce either sequence, using traditional edit operations? This part of the approach is not as clearly explained as most other parts.\n\n>Question D) section 3.2.2: what is the definition of \"keyword\" being used here? It's a big claim to say that \"keywords imply the main intention of a sentence\", especially without explaining what counts as a keyword in general. (I understand that slots are taken to be keywords under this approach, but that is an approximation - how would you define \"keyword\" more generally?)\n\n>Question E) Algorithm 2: do the rules grouped here need to share the same class?\n\n>Question G) line 555: the conclusion mentions qualitative analysis, but I don't see any qualitative analysis in the body of the paper. I'm sure I missed something - to what does this refer?\n\n**[Response]** The key distinction between ALP and GDA is rooted in the usage of slot information and rule manipulation.  Our approach may look similar to ALP, as our ruleset is based on context-free grammar (CFG), however diverging from the basic form of the rules, our main focus is the slot information and rule manipulation.\nWe use slot information (e.g., slot labels and named entities) to group the words with the same type of information. For instance from Figure 2, \\\\$artist represents identifying slot information for the artists (e.g., Adele, Bruno Mars, IU). The initial ruleset consists of 1) rules that map slot information to the corresponding words and 2) rules for the structure of each sentence - words replaced with slot information (or named entities).\n\nWhile the initial ruleset is not unlike the initial sentences - save the slot information representation, we use distance- and keyword-based manipulation to enrich the ruleset. The distance-based rule manipulation is based on the edit-distance metric. We take into account the length of each sentence in order to normalize the edit-distance metric (normalized ED) and calculate the similarity between sentences. The similarity between sentences is calculated by using normalized ED and then by the predefined threshold, we can group the similar rules as exemplified in Algorithm 1. Algorithm 1 details how we calculate normalized ED and align the grouped sentences to merge the rules. For instance, in Figure 3 when we apply distance-based manipulation for the four initial rules, we first align sequences of words that minimize normalized ED and merge the four rules to make a new superset-like rule. The resulting rule is in Figure 3(a).\n\nKeyword-based rule manipulation is less complex. We gather the sentences with the same type and order of slot information. We then use slot information which are present in the form of slot labels for datasets such as ATIS, Snips, FB-MTO, and MultiDoGO, and named entities for datasets such as AGNews, IMDB, SST-2, and Yahoo. For instance in Figure 3, the four initial rules have \\\\$artist and \\\\$playlist as slot information, and \\\\$artist occurs first and then \\\\$playlist occurs. Thus, with the same type and order of slot information, we group the four initial rules by aligning the slot information. The resulting rule can be seen in Figure 3(b).\n\nSection 5.2 provides ablation studies and detailed analyses on manipulations of GDA and hybrid strategies of GDA and language models. Providing the full experimental results in Appendix B, we supply more details of the analyses in Appendices C and D. If the reviewer finds our analyses insufficient, we would appreciate getting feedback so that we might answer your concerns.\n\n**5. Clarifying general issues**\n\n&nbsp; **a.** The terminology of CFG\n\n>Question A) line 200/201 and after: the use of the word \"variables\" here is, in my experience, unusual. I would suggest instead talking about \"terminals\" and \"non-terminals\", in keeping with traditional CFG terminology.\n\n**[Response]** We follow the CFG terminology as utilized in theory of computation [1], which uses 'variable'. However, we are grateful of the reviewer’s detailed concerns of the terms.\n\n[1]: Michael Sipser, \"Introduction to the theory of computation\", 2012.\n\n&nbsp; **b.** The grammar and rules of GDA\n\n>Comments #1) Line 175 (compared to abstract): the abstract talks about using \"context-aware grammar\", but the rest of the paper talks about using \"context-free grammar\". The rules and approach described are certainly context-free - is there a reason for calling this approach \"context-aware\" in the abstract? (or maybe just a typo or missed edit...)\n\n**[Response]** We use the term, ‘context-aware grammar’, to indicate that the rules of GDA extend from context-free grammar to represent the context of sentences, thus making them ‘context-aware’. While the basic format of a ruleset generated by GDA is CFG, we design the rules of GDA to be aware of context. The usage of slot information in rule manipulation maintains the abstractive context of rules and restricts the generation of rule variants that may harm the original intention of the rules. We will make sure to be clear in describing the ruleset of GDA, and try to refrain from using unclear terminology in order to avoid any confusion.\n\n&nbsp; **c.** Discussion of the results\n>Comments #2) Section 5: I don't understand why there is so little discussion of the results in Table 4, and so much discussion of results in Table 3. The main contribution of the paper is a DA method that relies on slot information, and yet the bulk of the discussion of results focuses on the experiments on datasets that do not include slot information. This is confusing.\n\n**[Response]** We provide comparison results with DA baselines for general datasets without slot information (1) and comparison results of language models and various combinations in GDA for datasets with slot information (2).\n\nTable 3 shows the performance of (1) and we present an analysis of the experiments in Section 5.1. Table 4 shows the performance of (2) and we present an analysis of the experimental results in Section 5.2. We aim to provide the intentions of the analyses and provide detailed descriptions for comprehensible interpretation of results for both experiments. Thus, we strive to present a similar amount of analysis of experiments for both (1) and (2) in the paper. We would appreciate it if the reviewer would provide details of what the analysis of (2) is lacking. If we need more explanation of Table 4 for the readers to understand, we will use an additional page for the analysis in the final version.\n\nWe value your insights and expertise, which encompass novel perspectives on the comprehensive feedback concerning GDA. Our rebuttal aims to address any reservations about the validity of our approach. Should you find our explanations insufficient, please feel free to inform us. We are enthusiastic about providing the necessary information to address your queries effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of this paper is the development of a CFG-based approach to building sentence generation templates for performing data augmentation for text classification, particularly for datasets which containing slot information. This approach has low computational requirements and seems to compare favorably to other approaches (though this result is not adequately demonstrated).\n\nI very much like the motivation of this approach, and I believe it has promise to be an effective and accessible data augmentation method, which could be demonstrated through additional experiments and analyses. \n\nIn addition, the paper is well-organized and mostly well-written.", "reasons_to_reject": "I see three main weaknesses to this paper - these are described in more detail below. In my opinion, the weaknesses are strong enough to outweigh the strengths described above.\n\n1. Inconsistency of experimental settings and reporting of experimental results.\n\nThe paper claims that GDA, which is designed specifically to use slot information encoded in existing datasets, outperforms other similar data augmentation approaches (specifically, EDA, SSMBA, and ALP), while avoiding \"semantic errors\" and inappropriate changes to sentence structure. The experiments presented, though, only compare to these systems on datasets without original slot information, and for which slot information has been approximated using automated named entity labeling. There is a second set of experiments over the claimed target data type - datasets with slot information labeled - but these experiments do *not* compare to the other DA methods described in the paper. Instead, they compare to use of LLMs for generating new data. There is no explanation in the paper of why the second set of experiments does not include comparisons to EDA, SSMBA, and ALP, and I cannot myself come up with a reasonable justification for this omission. \n\nIn addition, the results that do show improvements over comparable SOA methods (those reported in Table 3) report performance with respect to accuracy rather than F1, again without a clear motivation. These factors combined reduce my confidence in the results reported, because they are not motivated. A more minor point - I would question some of the claims made about GDA outperforming other systems, as no significance testing seems to have been done. For example, the paper claims that GDA outperforms ALP for the IMDB dataset, but the difference is tiny - 67.32 accuracy compared to 67.05.\n\n2. Overly strong argumentation regarding elimination/reduction of semantic errors.\n\nStarting from the abstract, the paper claims that GDA, by incorporating grammar rules, reduces the occurrence of certain semantic errors common to comparable DA approaches. The only support for this claim is through the examples shown in Table 1. Even with these carefully selected examples, the errors shown are not all semantic errors, and generally are not due to changes in sentence structure. Instead, they're because of either ill-formed NPs ('Bruno Hits Mars') or bad selectional restrictions ('add my playlist to a song') -- even in the latter case, the structure of the sentence doesn't change. The Table 1 errors are claimed to be logical errors, but they are not at all consistent error types. The two ALP examples are semantically fine, in my opinion. SSMBA has one logical error (can't add a playlist to a song) and one PP-attachment error. EDA has one factual error (not equivalent to a logical error) and one clunky (but possible) NP structure.\n\nOn another point, the abstract claims that the \"rules of grammar\" produced by the GDA approach represent \"relations between a sentence's semantics and its syntactical structure\". I can only read this as true with a very loose definition of what is meant by both \"semantics\" and \"syntactical structure\" -- as a linguist, this wording leads me to expect a representation that actually incorporates syntactic constraints (or at least syntactic classes) and some sort of semantic representation. The only semantic representation I see, though, is the use of (rather domain-specific) informational categories as slot labels (e.g. $artist or $playlist). The rules produced look like sophisticated templates for generating text; they do not represent either syntactic or semantic information in the usual sense (again, I'm speaking from the perspective of linguistics). \n\n3. Some claims not supported by results.\n\na. Claim about types of errors made by GDA vs. those made by other comparable approaches. The paper claims several times that GDA reduces semantic errors, but does not demonstrate that. Around lines 448-457, the paper claims that GDA generates more semantic and syntactic variations, but this is not demonstrated, at least not that I see. It would be interesting to perform a study of the sets of sentences generated by the various approaches based on the same input data!\n\nb. Claim that Table 3 demonstrates improvements of GDA over ALP. This feels like a stretch - performance of GDA is better for some data sets, but only some (mostly just SST-2, where evaluation is happening on a fairly small set. Also, not clear how this strategy really differs from ALP. If ALP also uses CFGs, this should have been noted much earlier - the paper leans toward presenting use of CFG-like structures as something novel. \n\nc. No significance testing - some of the differences claimed to show better performance, whether compared to another model or comparing across different datasets, seem to be very very small.", "questions_to_authors": "A. line 200/201 and after: the use of the word \"variables\" here is, in my experience, unusual. I would suggest instead talking about \"terminals\" and \"non-terminals\", in keeping with traditional CFG terminology.\n\nB. lines 255-257: in what way does this approach ensure that underlying semantics and grammatical structure are preserved? \n\nC. section 3.2.1 is confusing to me: the edits proposed seem to be okay, but I find this description confusing and somewhat misleading. it's not easy to see why these should be called \"distance-based\", nor what edit-distance has to do with producing these types of rules. Is the idea that you're taking two sequences/sentences that have a similar ordering of slots (and perhaps the same intent?) and generating a rule that could produce either sequence, using traditional edit operations? This part of the approach is not as clearly explained as most other parts. \n\nD. section 3.2.2: what is the definition of \"keyword\" being used here? It's a big claim to say that \"keywords imply the main intention of a sentence\", especially without explaining what counts as a keyword in general. (I understand that slots are taken to be keywords under this approach, but that is an approximation - how would you define \"keyword\" more generally?)\n\nE. Algorithm 2: do the rules grouped here need to share the same class? \n\nF. line 465: over what are you computing KL divergence? More detail would be helpful here.\n\nG. line 555: the conclusion mentions qualitative analysis, but I don't see any qualitative analysis in the body of the paper. I'm sure I missed something - to what does this refer?\n\nFinally, thanks for your good work!", "paper_topic_and_main_contributions": "This paper proposes a new approach to data augmentation, designed specifically to take advantage of slot information, where that exists. The approach is called \"Grammar-based Data Augmentation\", where \"grammar\" refers to the use of a generative, CFG-style set of production rules to - in essence - transform original input sentences and produce new, similar sentences with similar semantic content to the originals. The process of generating data for augmentation involves three steps: 1) collecting a set of production rules from the slot-tagged input data (the authors refer to this as \"inference\", but from my perspective they are harvesting rules, not inferring them); 2) building up a set of substitution rules/templates, by perform various manipulations on the production rules collected in step #1; and 3) generation of new sentences according to the templates produced in step #2. \n\nThe paper presents two sets of experiments, with different evaluation metrics and comparisons. Both sets of experiments include ablations comparing GDA with and without the two different types of manipulations described as part of step #2:\n\nSet A (results in Table 3): application of GDA to four datasets without original slot labeling; NER is used as a stand-in for slot information (as the steps described above crucially rely on having slot information). In these experiments, GDA is compared to three SOA data augmentation strategies. Performance is reported in accuracy of the relevant text classification task.\n\nSet B (results in Table 4): application of GDA to four datasets with original slot labeling. In these experiments, GDA is compared to data augmentation using pretrained LLMs (both BERT and Bard), but is not compared to the three SOA data augmentation strategies from the Set A experiments. Performance is reported in F1 for the relevant text classification task.\n\nOverall, some of the results presented here do seem to achieve better performance than the state-of-the-art, and the method proposed can easily be applied to any dataset with existing slot information, with rather low computational overhead. That said, it is difficult to draw convincing conclusions, given the differences in experimental settings, which are not, in my opinion, convincingly motivated. See detailed points below.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* abstract: spell out full form of 'GDA'\n\n* line 175 (compared to abstract): the abstract talks about using \"context-aware grammar\", but the rest of the paper talks about using \"context-free grammar\". The rules and approach described are certainly context-free - is there a reason for calling this approach \"context-aware\" in the abstract? (or maybe just a typo or missed edit...)\n\n* section 4.1: the paper would benefit from more discussion of the datasets used, and in particular what their respective tasks are, and what the set of possible classes looks like for each. the different DA approaches discussed in the paper use different strategies (for example, changes in sentence structure vs. changes in the structure of NPs vs. choice of verb), and different strategies may be more or less useful for different tasks.\n\n* Section 5: I don't understand why there is so little discussion of the results in Table 4, and so much discussion of results in Table 3. The main contribution of the paper is a DA method that relies on slot information, and yet the bulk of the discussion of results focuses on the experiments on datasets that do not include slot information. This is confusing."}}, {"review_id": "nOeScmsY7K", "reviewer": "Reviewer_YPDx", "summary": "", "questions": "* I have problems with the use of terms such as \"grammar-based\", \"Rule of Grammar\", etc. in this work. I see no [formal] grammar here at all, what I see are just patterns, lexical patterns of some complexity that indirectly account for some syntatic and semantic variation, but lexical patterns after all. Like some kind of word-level RegExps.  \n\n* I'm concerned about the possible domain-dependendence of the approach. It reminds me to some classic  Information Extraction (IE) algorithms for automatically generating extraction patterns through bootstrapping.\n\n* Why are doing the ablation analysis (Table 4) on different datasets that the otiginal ones used for performance comparison (Table 3). The logical action would be to use the same datasets for both studies. \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are appreciative of the reviewer's extensive remarks and proposals. We value the beneficial criticism that will upgrade our paper. We shall ensure that your input is incorporated in the revised version.\n\n**1. Contribution of GDA**\n\n>Reasons to Reject) I see no breakthrough of possible interest for the NLP community.\n\n**[Response]** In a few sentences, GDA constructs rules and by the proposed rule manipulations, generates sentences which enrich the syntactic and semantic diversity - contributing to data augmentation and a few-shot text classification research field. Furthermore, our grammar-based approach acts as a helpful tool for neuro-symbolic approaches, enhancing the interpretability of the result to deep learning models. We find our results interesting and well-performing enough to be an impactful contribution in the NLP community. We would appreciate more detailed explanations of why the reviewer thinks our paper has no significant breakthroughs.\n\n**2. Clarification on the formal grammar used in the paper**\n\n>Question #1) I have problems with the use of terms such as \"grammar-based\", \"Rule of Grammar\", etc. in this work. I see no [formal] grammar here at all, what I see are just patterns, lexical patterns of some complexity that indirectly account for some syntatic and semantic variation, but lexical patterns after all. Like some kind of word-level RegExps.\n\n**[Response]** The formal grammar used in the paper is context-free grammar (CFG). Our ruleset of GDA is based on CFG. The distance-based rule manipulation is also based on the edit-distance metric, a popularly used similarity measure in the theory of computation. Roughly speaking, every formal grammar is based on patterns. So if the reviewer disagrees, I respectfully ask the reviewer what their definition of formal grammar is if CFG does not fall into this category. We would be happy to provide additional information on the definition of formal grammar.\n\n**3. Dependence on slot information and how we stated that our approach also gains improvements on non-domain-specific general dataset**\n\n>Question #2) I’m concerned about the possible domain-dependendence of the approach. It reminds me to some classic Information Extraction (IE) algorithms for automatically generating extraction patterns through bootstrapping.\n\n**[Response]** We acknowledged our own concerns regarding GDA’s dependence on slot information. As a result, we not only conduct experiments on datasets with slot information but also on datasets without slot information. Table 3 presents the performance result of GDA on datasets without designated slot information. We use a pre-trained BERT for NER task to extract named entities and use them in substitution of slot information. The result in Table 3 shows that GDA gains objective improvements on general datasets.\nFurthermore, our rules and manipulations require sophisticated computational steps depicted in Algorithms 1-3, which are not possible solely with just IE algorithms. If the reviewer finds it lacking, we would like to know the detailed comments on this concern. We would be happy to respond to your feedback.\n\n**4. Clarification on how the purpose of the two experiments is different**\n\n>Question #3) Why are doing the ablation analysis (Table 4) on different datasets that the otiginal ones used for performance comparison (Table 3). The logical action would be to use the same datasets for both studies.\n\n**[Response]** Our experiments consist of 1) comparison of baseline data augmentation (DA) models for general datasets without slot information and 2) comparison results of language models and various combinations in GDA for datasets with slot information.\nOur approach achieves high performance for experiment 2, but we are also concerned as to whether our approach is also applicable for general text classification datasets with no designated slot information.\nThe purpose of the experiment 1 is to inspect whether GDA can achieve state-of-the-art performance even on the general datasets for text classification that do not contain designated slot information.Table 3 shows the performance improvements compared to the baseline DA models.\n\nOn the other hand, experiment 2 primarily focuses on how GDA, the rule-based data augmentation method, performs better than the language models. We also inspect various combinations of GDA from this set of datasets. The focus of experiment 2 is more centered on the hybrid strategies of GDA in combination with the language models.\nWe will make sure to clearly describe the different intentions of both experiments in the final version of our paper.\n\n\nWe highly appreciate your valuable feedback and insightful perspectives, which offer new angles to consider in relation to the detailed comments on GDA. We are confident that our response will successfully alleviate any concerns you might have about the robustness of our approach. Should you find our explanations lacking, please let us know. We are eager to provide you with the essential information needed to comprehensively address your inquiries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Data augmentation is a hot topic. It's difficult to find or create datasets good and large enough for current data-based NLP approaches.\n", "reasons_to_reject": "* I see no breakthrough of possible interest for the NLP community. \n", "questions_to_authors": "* I have problems with the use of terms such as \"grammar-based\", \"Rule of Grammar\", etc. in this work. I see no [formal] grammar here at all, what I see are just patterns, lexical patterns of some complexity that indirectly account for some syntatic and semantic variation, but lexical patterns after all. Like some kind of word-level RegExps.  \n\n* I'm concerned about the possible domain-dependendence of the approach. It reminds me to some classic  Information Extraction (IE) algorithms for automatically generating extraction patterns through bootstrapping.\n\n* Why are doing the ablation analysis (Table 4) on different datasets that the otiginal ones used for performance comparison (Table 3). The logical action would be to use the same datasets for both studies. \n", "paper_topic_and_main_contributions": "The authors propose a new data augmentation method based in automatically generated generation patterns built around slot variables. This proposal is named Grammar-based Data Augmentation (GDA) and consists in 3 phases:\n\n(1) Inference of \"Rules of Grammar\" (RoGs) from the training dataset. Ideally, the training dataset should have the slots already identified but, if not, a Named Entity Recognition (NER) process could be applied so the entities identified could be converted into slot variables, thus producing a slot-based dataset as needed. Taking these texts as input, basic ROGs are extracted, mere literal patterns of the examples found involving the slots.\n\n(2) Manipulation (I would say \"Combination\") of the RoGs. Those patterns are combined into more general classes and the patterns themselves are combined/mixed/joined using classical lexical operations like optionality and alternation. This way, we obtain generation rules with a higher degree of variability. Three techniques are proposed for this: the use of word-level edit distance; using the slot variables as keywords to identify and combine rule groups; a mix of both.\n\n(3) Generation. Using the rules obtained in the previous step.\n\nThe authors claim that their approach enrich the diversity of generated sentences, both at syntactic and semantic level.\n\nTheir approach is evaluated against state-of-art data augmentation methods and LLMs, also including an ablation study.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Abstract is excesively long, it must be reduced.\n* Sect.1: this section  should be re-structured, probably split in two. Basically, all content in the right column of page 2 (including Figure 2) is redundant or goes beyond the contents we expect from this section. \n* Sect. 1: I also miss a short paragraph describing the structure of the paper. \n\n* Sect. 3.1: I see no reason for explaining CFGs, NLP community knows them very well.\n\n* Sect. 4.1: A brief description of the nature and domain of the documents of those datasets may be of interest to the reders. In my case, surely it is.\n\n* Table 3: I suggest saying in the caption that those results correspond to the datasets WITHOUT slots. The contrary for Table 4.\n\n* Sect. 5.1, par. 1: It is not clear that \"GDA achieves compeling performance compared to baseline augmentation models\". I see no clear superiority with respect to ALP, for example. \n\n* References: I suggest not listing the whole list of authors in the case of the entries (Brown et al., 2020) and (Thoppilan et al 2022). Just list part of them foillowed by \"et al.\"\n\n"}}]

Author rebuttal:

Meta-review: This paper proposes a novel data augmentation technique for textual data, named GDA, which yields diverse and faithful samples. The methodology consists of inducing a grammar extended with slot information and diversifying it through rule manipulations. The results are especially promising in low-resource settings. Another strength of the paper is that it highlights its own limitations, as it mostly benefits tasks with NER-taggable tokens. However, the choice of CFG as the underlying grammar is questionable, as it restricts the space of sentences that can be successfully modelled by GDA. Another issue that remains not fully addressed is how to avoid error propagation in the proposed pipeline (e.g., incorrect NER slot predictions when these are not available in the data). To make the results more convincing, the Authors should compare the performance of DA baselines on datasets where slot annotation is available, possibly tone down their claims on reducing semantic errors which need more quantitative evidence, and add significance testing to the reported figures.
Decision: Accept-Findings