PMixUp: Simultaneous Utilization of Part-of-Speech Replacement and Feature Space Interpolation for Text Data Augmentation
Authors: Hyeon Soo Kim
Hyejin Won
Kyung Ho Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: text augmentation
part-of-speech
feature space interpolation

Abstract: Data augmentation has become a de facto technique in various NLP tasks to overcome the lack of a large-scale, qualified training set. The previous studies presented several data augmentation methods, such as replacing tokens with synonyms or interpolating feature space of given text input. While they are known to be convenient and promising, several limits exist. First, prior studies simply treated topic classification and sentiment analysis under the same category of text classification while we presume they have distinct characteristics. Second, previously-proposed replacement-based methods bear several improvement avenues as they utilize heuristics or statistical approaches for choosing synonyms. Lastly, while the feature space interpolation method achieved current state-of-the-art, prior studies have not comprehensively utilized it with replacement-based methods. To mitigate these drawbacks, we first analyzed which POS tags are important in each text classification task, and resulted that nouns are essential to topic classification, while sentiment analysis regards verbs and adjectives as important POS information. Contrary to the aforementioned analysis, we discover that augmenting verbs and adjective tokens commonly improves text classification performance regardless of its type. Lastly, we propose PMixUp, a novel data augmentation strategy that simultaneously utilizes replacement-based and feature space interpolation methods. We examine that they are new state-of-the-art in nine public benchmark settings, especially under the few training samples. 

Under review as a conference paper at ICLR 2023
PMIXUP: SIMULTANEOUS UTILIZATION OF PART-OF- SPEECH REPLACEMENT AND FEATURE SPACE INTERPOLATION FOR TEXT DATA AUGMENTATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Data augmentation has become de facto technique in various NLP tasks to over- come the lack of a large-scale, qualified training set. The previous studies pre- sented several data augmentation methods, such as replacing tokens with syn- onyms or interpolating feature space of given text input. While they are known to be convenient and promising, several limits exist. First, prior studies sim- ply treated topic classification and sentiment analysis under the same category of text classification while we presume they have distinct characteristics. Sec- ond, previously proposed replacement-boe based methods bear several improvement avenues as they utilize heuristics or statistical approaches for choosing synonyms. Lastly, while the feature space interpolation method achieved current state-of- the-art, prior studies have not comprehensively utilized with replacement- based methods. To mitigate these drawbacks, we first analyzed which POS tags are important in each text classification task, and resulted that nouns are essential to topic classification, while sentiment analysis regards verbs and adjectives as im- portant POS information. Contrary to the aforementioned analysis, we discover that augmenting verbs and adjective tokens commonly improves text classifica- tion performance regardless of its type. Lastly, we propose PMixUp, a novel data augmentation strategy that simultaneously utilizes replacement-batee based and feature space interpolation methods. We examine that they are new state-of-the-ar in nine public benchmark settings, especially under the few training samples.
perpoouslyprrposee
INTRODUCTION
Background and Motivation Recent improvements in deep neural networks have empowered re- markable advancements in various Natural Langauge Processing (NLP) tasks such as text classifica- tion (Minaee et al.] 202T). question answering (Rogers et al.) 2021), and natural language inference (Bowman & Zhu 2019). .. However, these supreme performances rely large, qualified training sets under the supervised regime. Under the circumstance where the machine learning practitioners have limited number of training samples, the models tend to suffer from overfitting and fail function with the expected performance. As acquiring large-scale, qualified training samples requires a par- ticular amount of resource consumption, there have been numerous studies to escalate the model's performance under the limited number of training samples in NLP tasks (Hedderich et al.) 2020). One promising approach is data augmentation, which generates new training samples by modifying original training samples through transformations (Chen et al. 2021). An underlying motivation of data augmentation to augment original training samples while the transformed text data sustain the original sample's overall semantics (Bayer et al. (021))
Several studies on data augmentation techniques proposed back-translation (Sennrich et al., 2015) and word replacements with predictive language models (Anaby-Tavor et al.) 2020] Yang et al.) 2020b). As these approaches require the massive cost of implementation (i.e., a well-trained transla- tion model for back-translation), academia sought more lightweight augmentation methods. Several studies presented replacement- based augmentation methods where tokens are replaced with syn- onyms fetched from large-scale dictionaries (i.e.. WordNet) as the synonyms are less likely to affect the semantics of original training samples (Bayer et al. 2021]] Feng et al.) 20200; Wei & Zou (019). While replacement-baane based augmentation methods escalate the model's performance, risks creating

---

Under review as a conference paper at ICLR 2023
unrealistic or non-conforming samples. This drawback motivated the development of feature space interpolation methods that transform a given imput in a feature space rather than directly changing tokens (Sun al. 2020; Chen et al. 2020b; Shen al. 2020). It has become state-of-the-ar in data augmentations on text classification.
Main Idea and Its Novelty When we scrutinize replacement-basee approaches, figure out that most of them did not precisely consider Part-of-Speech (POS) information. We presume there exists particular POS that highly correlates to the text classification performance, but it was actively considered in past studies. Furthermore, we hypothesize that essential POS information would dif- fer along with text classification types (topic classification and sentiment analysis) presented in (Sun) et al. 2019). As these tasks have varying characteristics among themselves, we expect the augmen- tation strategies should be considered different respective each task. But, was not considered in the prior works. Lastly, replacement-based methods and feature space interpolation approaches have not been simultaneously considered. The replacement- providing more contextual augmentation results as they directly transform the word token with its synonyms, On the other hand, feature space interpolation methods are also effective because they create infinite amount of new augmented data samples they do not transform the token, but instead, add perturbations to the feature vector. Then. the following question becomes our primary motivational question: What if simultaneously utilize both eeplacement-basee and feature space interpolation methods?
methods have an advantage in
To this end, we propose novel data augmentation method denoted as Part-of-speech MixUp (PMixUp), which replaces tokens belonging to the particular POS tags and applies feature space interpolation in sequential order. We presume simultaneous utilization of both replacement-basee and feature space interpolation methods would further escalate the classification performance. As a preliminary analysis, we firstly examine which POS information influences two text classifica- tion types the most and validate whether augmenting tokens that do not belong to important POS tags helps maintain the original sentence's overall semantics. We then examined its effectiveness in nine benchmark datasets under the various training sample sizes per class and resulted in the pro- posed PMixUp accomplishing new ofate-of-the-art in given public benchmark settings. We hereby highlight our work's ss novelty in the following aspects. First, our work firstly analyzes how POS information becomes important following each classification type. Second, to the best of our knowl- edge, study is the first attempt to simultaneously utilize replacement-basee and feature space interpolation methods for text augmentation.
Key Contributions
We empirically discovered that nouns serve as critical label determinants in topic classifi- cation while verbs and adjectives play essential roles in sentiment analysis.
We figured out that replacing tokens that do not belong to the aforementioned important POS tags at both text classification tasks contributes to maintaining the original sentence's meaning and elevating the test classification performance. Instead, we discovered a com- mon trend across the two tasks that replacing tokens under verbs and adjectives POS tag escalated the test performance at most, and we analyze the reason for this phenomenon as the low interference with core semantics of original text data when replaced.
We present Pp,xUp, a novel data augmentation technique that combines POS- POS-guided re- placement and feature space interpolation methods. Upon nine public benchmark datasets, we examined the proposed PMixUp's 's supreme performance of classification performance escalation rather than prior works; thus, it becomes a new tftte--f-the-ar in given settings.
We discovered that PMixUp is especially effective under a few training samples per class. We further observed that every data augmentation method's beneficial impact decreases when there are many training samples.
Lastly, we scrutinized the supremacy of PMixUp derives from larger knowledge capacity, that our method makes the model acquire more fruitful understanding given data.
2 RELATED WORKS
Replacement-basee Data Augmentation The replacement- augmentation strategies gener- ate text samples by replacing particular tokens words with synonyms (Wei & Zou| 2019) This
2

---

Under review as a conference paper at ICLR 2023
augmentation technique presumes that the overall semantic information of sentence remains un- changed as long as particular words are replaced with their synonyms. Synonyms of the tokens to replace typically matched from pre-defined source such as WordNet (Miller) 1995), Thesaurus (Jungiewicz & Smywinski-Pohi 2019), or from a pre-trained Language Model such as BERT (De- vlin et al.) 2018). (Kolomiyets al. 201T) presented headword replacement technique, which creates augmented training samples by substituting temporal expression words with synonyms from WordNet and Latent Words Language Model. (Wei & Zou 2019) proposed random replacement technique that randomly chooses N words from a given sentence and substitutes them with ran- domly chosen synonyms. (Feng et al., 2020) suggested GenAug technique which utilizes RAKE score (Rose et al. 2010) for keyword extraction and substitutes these keywords with synonyms at WordNet. Recently, several studies proposed POS-based replacement techniques. These studies chose particular POS (i.e., nouns, verbs) as replacement target because they have higher probabil- ity of having synonyms than the others (Marivate & Sefara, 2020, Jungiewicz & Smywinski -ihil 2019). While these approaches share similar motivation with our study of replacing particular POS, our study differs from them because we postulate having a higher probability of having syn- onyms does not simply implicit effective augmentation performance. Our study hypothesizes there certainly exists a particular POS along with text classification's types, and considering this under- standing correlates to precise augmentation performance.
Feature Space Interpolation Feature space interpolation techniques transform a given text sample's feature representation to generate augmented training samples. An underlying motivations of feature space interpolation techniques are follows: producing synthetic samples from the input level poses the risk of dealing with unrealistic samples that leads to performance drop; thus, there should be an alternative augmentation method that does not directly transform original sample (ries & Taylor 2017). Upon this motivation, (Ozair & Bengio 2014; Bengio et al. 2013) showed that augmenting in feature space leads to higher chance of producing realistic examples compared to that in data space or input level. Such augmentation strategies include adding noise, erasing. or interpolating instances of input text. (Kurata et al. 2016) proposed an approach that augments text by randomly perturbing the encoded vectors of input. (Shen et al.] (2020) suggested CutOff, which erases part of the feature vector to produce numerous new samples with noise. Interpolation combines two different training samples to create a new one. (Kumar et al.] 2019) tried joining the
first [CLS] tokens of two training examples to form new instance. (Sun et al. 2020) introduced Mixup Transformer, which interpolates the vectors obtained from the last layer of the transformer architecture. Similarly, (Chen al. 20206) introduced TMix, which interpolates at specific layers within transformer architecture, and showed that combining hidden vectors at the 7th, 9th, and 12th layers containing the syntactic and semantic information improved the preceding interpolating methods. With respect to the motivation of feature space interpolation techniques, we presumed utilizing these methods as well as replacement- augmentation techniques might further escalate the classification performance.
3 PRELIMINARIES
Text Classification and Datasets We categorize the datasets into two types: topic classification and sentiment analysis, following the taxonomy and definition proposed by (Sun et al., 2019). For topic classification, we utilized the datasets SverkOverFlow, BANKING, DBPedia, r8, AGNews and Ohsumed, while using IMDB, Amazon Polarity Reviews and Yelp Reviews for sentiment analysis. The detailed summaries are as below:
StackOverFlow Dataset published in Kaggle.com. We follow the works of (Xu al., 2015) in using a processed version with 20 classes and 1,000 samples for each class.
BANKING A dataset under BANKING domain, consisting of 13,083 customer service queries with 77 intent classes (Casanueva et al. 2020).
DBPedia An ontology classification dataset (Zhang et al. 2015)
* r8: Dataset consisting of news documents from 8 most popular classes of Reuters-21578 corpus (Debole & Sebastiani 2005)
AGNews A large-scale collection of news articles with 4 classes (Zhang et al. 2015
3

---

Under review as a conference paper at ICLR 2023
Ohsumed
A subset of the MEDLINE database. The dataset is composed of 23 Medical Subject Headings classes of cardiovascular diseases group, and we use the commonly used subset from (aoo et al. 2017).
IMDB: A movie review dataset of 50k full length reviews (Maas et al. 2011).
Amazon Polarity Reviews: Dataset consisting of reviews from Amazon, with binary polarity labels. /Zhang et al. 2015)
Yelp Reviews The polarity
Solarity-labbled labeled version of dataset obtained from Yelp Dataset Chal-
lenge in 2015. (Zhang al. 2015
Classifier and Evaluation We utilize pre-trained BERT (Devlin et al. 2018 as backbone language model, and we used implementations provided in Huggingface (Wolf et al., 2019). For training the classifier, we set the learning objective with cross-entropy loss. We trained the model for 15 epochs with AdamW W optimizer (Loshchilov & Hutter (017) under the learning rate of 4e-5, Following the evaluation method utilized in various studies regarding text classification and data augmentations (Fujino et al.] 2008 Mishu & Rafiuddin [016 Ollagnier & Williams, 2020), we employed the FI score at the test set as an evaluation metric. For the reprodueibility of our study, we publicize the code implementations in the supplementary materials.
DISCOVERING KEY FACTORS OF TEXT CLASSIFICATION
Setup As first and foremost analysis, we investigate key factors that contribute to classifying the intent or label of given text at each text classification type. We expect effective data augmenta- tion technique shall sustain a given text's general semantics after replacing particular tokens (Chen) al. 2021) and consider text classification type's characteristies; thus, aim discover which factor is relevant to the text classification performance after eliminating or replacing it. To discover an answer to the aforementioned questions, measured the text classification performance after eliminating or replacing particular factors of given text. Supposing a particular factor is impor- presume model trained with the training where this
tant for text classification performance, factor is removed would acquire lower test classification performance. Therefore, we hypothesize a huge drop in test performance under the transformed training samples (removed factors) implies the importance of that factors.
We utilized three factors of text classification following the prior study (Garg Ramakrishnan 2020 Sun & 2020)) Probability-based Important Token, Attention-based Important Token, POS, and Syntax. For Probability-based Important Token, we scored the importance of each token following the prior work of measuring token importance (Garg & Ramakrishnan 2020). We measured the class probability change after removing every token and regarded particular token as an important one if the removal causes the largest change among every token. For Attention-based Important Token, we observed each token's 's attention scores (Sun & Lu 2020) on a given sentence and checked the token with the highest and lowest attention scores. For POS,
utilized NLTK toolkit, a conventional POS-tager tagger proposed in (Loper & Bird 2002) to acquire POS information each token. To select target POSs in study, we extract various POS tags and count their ratio over total tokens in each dataset. The result is shown in Table [ Among various POS tags, we selected nouns, verbs, and adjectives as target POSs they frequently exist in every dataset. At the same time, the other POSs (i.e., prepositions, determiners) are absent or present with such low frequency, and we empirically expect these POS tags to influence less the text classification; thus, we skipped these scarce POS tags in the analysis. Lastly, for Syntax, we randomly mixed the order of tokens in sentence to provide noise on the syntax of the given sentence. following the prior works of noising syntax (Wei & Zou 2019). We removed and replaced these factors with training samples for each dataset, trained the classifier, and measured the test performance. The results are shown in Table 2
Composition of Important Tokens Before we analyze important POS tags at each text classification type, we scrutinize the characteristics of the tokens that were considered important. We measured the ratio of different POS types existing in the important tokens followed by Probability- approach. Note that we only described the composition of important tokens based on Probability- based approach as Attention-based approach showed very similar results to the aforementioned one. We only considered nouns, verbs, and adjectives they were the majority of the POS tags. The
4

---

Under review as a conference paper at ICLR 2023
Table 1: The ratio of Top-12 POS tags exists in 9 benchmark datasets. Among various POS tags in the dataset, we utilize Top-3 POS tags (Nouns, Verbs, and Adjectives) in our study as they occur across every dataset with high frequency.
AGNews 0,3615
Amazon
BANKING
DEFedia
IXLD5
Ohsumed
18
StackOverl low
Yelp
AVERAGE
Nouns
0,2139
0.3722
0.3557
0,2891
0/4519
0,4700
0,4745
0-1389
0,3781
Verbs
0,1189
0.1638
02224
0.166
0.1529
0.2338
0.2350
0,2302
0.2275
0.2022
Adjectives
0.1199
0.1266
0.1319
(Conama)
0.0320
0A852
019644
006227
0.0434
0.0649
0.3644
00049
00639
0.0609
(pperiod)
0.0336
0,0333
0.0044
0.0525
0.0418
0.0625
0.9620
0.0620
0.0844
0.0607
Determiners
0,4444
049931
00133
0.799
0.0854
0.0005
0.0013
0,0122
0,0435
Prepocition
Cadinall Digit
0.0188
0,0997
0,096
0.125
0.0492
0.0014
0.0006
0.0002
0.091
0.0178
Conrdinating C Conjanction
030226
010011
00192
0.0297
0
03RITT
(00022
00033
0.01
(culon)
020099
0.0089
00008
O1TORG
00004
Adverbs
0,021
03226
000061
000000
0.0173
0
0
0,000
0.000
0,0088
Table 2: The classification performance after removing tokens with each respective condition. The values highlighted in bold indicate the greatest performance drop from the Baseline without any removal, also understandable as the most contributing factor in the task of text classification.
Topic Classilication DRPedla
Sentiment Analysis Amazon Polarity
StackO low
BANKING
r8
AGNews
Ohsumed
IMDB
Yelp
Buseline Removal)
0.8995
0.9554
0.9.69
0.6338
0,9522
0.9529
Probability-hased
0.7405
0.6929
0.7906
0.2113
0.8301
0.4335
0.6448
0.8774
0.9507
Attention-based
0.7806
0,7023
0.8162
0.7345
0.8623
0.4330
0.6991
0,8338
0.8938
Nouns
0,2941
0.8238
0.9068
0.5282
0,9472
0.9687
Verbs
0,8890
0.8331
0.9553
0.8165
0.9339
0.6145
0.3333
0,9462
0.9637
Adjectives
0.8763
0.8613
0.9517
0.8885
0.9344
0.5903
0.9218
0,9501
0.9699
Syntax
0,8990
0.9200
0.9359
0,7410
0.9280
0.5881
0.9350
0.9069
0,9469
Table 3: Ratio of the three POS tags in mrobability-basee important tokens for each dataset. The values highlighted in bold indicate the POS tag that covers the greatest percentage. analysis Verbs and Adjectives where Nouns take the same ratio as Adjectives. We presume the reason as the high frequency of words such as price, diner, or restaurant names.
sentiment more frequently considered important, except for the case in Yelp,
Topic Classification
Sentiment Analysis
FlekOwe
BANKING
DBPedia
AGNews
obsumed
IMDB
Polarity
Yelp
Nouns
0.5940
0,4468
0.6374
0,6459
0.5369
0.5078
0.3352
0.3571
0.3743
Verbs
0.2334
0.4054
0.2417
0.1511
0.2963
0.2623
0.2681
0.3980
0.2513
Adjs
0.1725
0.1209
0.2029
0.1668
0.2298
0.3967
0.2449
0.3743
results summarized in Table 3 We could observe that while nouns the most dominant POS tags for datasets under topic classification, verbs and adjectives become more critical for datasets for sentiment analysis.
Analysis Considering experiment results in Table 2 and the composition of important tokens at Ta- B we discovered that important POS information differs along with text classification types. For topic classification tasks, nouns created the most significant performance drop when removed, indi- cating that nouns play an essential role in the classification. On the other hand, verbs and adjectives contributed to a huge performance drop for sentiment analysis tasks. In the case of IMDB and Ama- zon Polarity datasets, tokens under the verb group created a huge drop when removed, and tokens under the adjective group made the largest drop when removed on the Yelp dataset. In nutshell, we figured out that topic classification and sentiment analysis take different important POS types while they are conventionally grouped as text classification task. Our study utilized this discovery as concrete milestone in data augmentation techniques described in the following sections.
5 Do AUGMENTING IMPORTANT POS ELEVATE CLASSIFICATION PERFORMANCE?
Setup We hypothesize that words belonging to important POS tags contribute more to building the semantics of given text input. Also. replacing such words holds higher risk of disrupting the label of the sentence. We. therefore, believe that replacing words that do not belong to important POS tags would yield a better classification performance escalation. For detailed description of replacement-
5

---

Under review as a conference paper at ICLR 2023
based augmentation, we followed the method proposed in (Wei & Zou 2019). Given single input text, we created n augmented samples by replacing m tokens belonging to the particular POS tag (nouns, verbs, and adjectives) with synonyms from WordNet. We followed the same configuration (number of n, m) proposed in (Wei & Zou 2019) as it is known to achieve the best performance in their experiment settings. Given training samples of every dataset, we trained the classifier by adding augmented training samples, For the target POS, we select important and non-important tokens for each dataset following the analyses in the prior section (which are denoted as Important token Replacement and Non- con-ipporantt token Replacement, respectively). Moreover, we utilize three POS types (Noun, Verb, and Adjective) as target POS information (which are denoted as Noun Replucement, Verb Replacement, and Adjective Replacement, respectively.) Except for augmenting training samples, we utilized the same experiment setting with the procedures in section 4 The experiment result is shown in Table
Table 4: Classification performances based on augmentation of each condition, where the values highlighted in bold indicate the highest improvement compared to the baseline performance. Note that both Important token replacement and Non-important token replacement options are sidered with Probability-base approach presented in section
Tupic Classification DBPedda
Sentiment Amalvsis
Stack(werFlow
BANKING
AGNeus
ohsumed
IMDB
Amazon Polarity
Baseline Angmentation)
0.9554
0.9372
Important token Replacemeent
0.8238
0.9229
0,9485
0.9516
09382
0.7128
0.9144
0.8478
0.413
Non-Important loken Replavement
0.893
0,9319
0,9500
0,4430
0.9399
0.7173
0.9331
0,9688
0.611
Noun Keplacement
0,8956
0.9192
0,9500
0.7645
09300
0.5098
0.9384
0.9529
03992
Replacement
0,9581
0,9006
0.661
0.9364
Adjective Replavement
0,9023
0.213
0.9577
0.877
0.324
0.636
0.322
0.9628
0.962
Analysis We analyze that the replacement of tokens that do not belong to important POS tags does help in enhancing the classification performance for both classification tasks. The replacement of verbs instead of nouns produced the best results although the noun was the most important POS in a topic classification task. This is because the topic usually exists within the sentence in the form of noun. An example of such a case would be as such: An original text from the AGNews dataset "Soaring crude prices plus worries about the economy" can be augmented as "Soaring crude terms plus worries about the thriftiness"" The original text has the label "Economy", but the augmented sentence loses the factor to classify it under the "Economy" label. Furthermore, certain datasets possess dopain-specifie terms. For example, an example of a sentence from the StackOverFlow dataset is "Use Oracle 6 from ASP.NT application". After noun replacement, the sentence becomes "Use prophet from ASP.NET application.". The original label for the text is "Oracle", and under the consensus that the term "Oracle" falls under named entity that holds technical meaning instead of the commonly known synonym of "prophet", the overall semantics of the sentence is completely altered.
However, we further discovered in sentiment analysis tasks that while verbs and adjectives are label determinants for the task, replacing such tokens does not reduce the classification performance but outputs the best results. We explain that the underlying reason for such behavior lies in the preserva- tion of semanties before and after replacing token. Unlike topic classification, the replacement of such words with their synonyms is less likely to alter the semantics of the word itself or the sentence as whole. For example, the sentence "One of the other reviewers has mentioned that after watching just 1 Oz episode you'll 'll be hooked" can be augmented as "One of the other reviewers has remarked that after seeing just 1 Oz episode, you'll 'lll be hooked", with the replaced word in italics. It can be understood that the sentiment information of the sentence is not affected as much as in the case of topic classification after replacing important POS tags, which are verbs and adjectives.
Consequentially, we discover that replacing verbs and adjectives (which are key factors in sentiment analysis tasks) enhances the classification performance in both topic classification and sentiment analysis at the same time. We analyze this result occurs because replacing verbs and adjective tokens is less likely to ruin the semantics of original text input when replaced with a synonym.
6 OUR APPROACH: PMIXUP
Description From prior analyses, we scrutinized that replacing verbs and adjectives with synonyms creates an effective data augmentation impact regardless of text classification tasks. We further

---

Under review as a conference paper at ICLR 2023
presume combining the feature space interpolation augmentation method with the aforementioned replacing verbs and adjectives would escalate classification performance rather than utilizing only either one of the methods. Therefore, we hereby present Part-of-speech replacement and Mixup (PMixUp); a novel data augmentation strategy that uses both synonym replacement and feature space interpolation. Please refer Figure for an overall architecture of our PMixUp. Our study is inspired by the works of (Wei & Zoul 2019) and (Chen et al., (020b), but differs from them as combined replacement-basse
and feature space interpolation methods for further escalation of test performance rather than using either of them. The proposed PMixUp merges two augmentation strategies: 1) replacing tokens belonging verbs and adjectives with their synonyms in Word- Net, 2) TMix, which is a state-of-the-art feature space interpolation technique under cireumstances using supervised learning with labeled data only (Chen et al. 2020b). In the following sections, we describe a series of empirical analyses to examine the effectiveness of PMixUp compared to replacement-bese based and feature space interpolation augmentation techniques.
Batch
Text 1
Text
Text 2
POS Replacement
Backbone Modal (BERT)
MixUpp
Classifier
Text 2"
Text N
Yielded Reprasentation
N'
Text
N"
Fixed Companint
POS Augmented Text
Represematton
Learrable Companent
Figure 1:
Overview of our PMixUp
Experiment Setup Following the setups in TMiX study (Chen et al. 20206), we measured the test classification performance under the different number of original training samples. We aim to amine whether the proposed PMixUp contributes to the escalation of test performance under the circumstance where a few original training samples exist. We set the number of original training samples per class as 10, 200, and 2500 if every class of the dataset includes more than 2500 training samples (AGNews, DBPedia. Amazon Polarity, Yelp, IMDB). For the datasets where each label includes less than 2500 samples, we heuristically set the number of training samples per class dif- ferently to describe a circumstance in which few original training samples exist (StackOverFloww, BANKING, r8, Ohsumed).
Throughout the experiment, we utilized three data augmentation techniques: Synonym Replace- ment, Feature Space Interpolation, and PMixUp. For synonym replacement, we measured the test classification performance under the replacement-based augmentation method proposed in sec- tion [5) which replaces tokens belonging nouns, verbs, and adjectives with synonyms in Word- Net. For the feature space interpolation method (which is currently state-of-the-au
we followed the same configuration in the original publication of TMix. We interpolated the latent represen- tation at one of the 7th, 9th, and 12th layers of the model, and we denoted both the reproduced performance and the reported performance in the original publications. Lastly, for PMixUp, the augmented data from synonym replacement is passed to the TMiX module to apply feature space interpolation into the given sample. Furthermore, in order obtain a detailed understanding of the effect of POS- based augmentation, we compare the results with those of augmentation through ran- dom token replacement without utilzing any POS information. Note that we skipped experiments on mreviously-proposee ferior to TMiX in the public benchmark settings. The experiment results following above setups are summarized in Table and T
replacement-basee and feature space interpolation methods as they were in-
Analysis From the experiment results shown in Table 5 and Table 6 we figured out the proposed Pp mostly outperforms the classification performance rather than the case when only either of replacement- based and feature space interpolation method is used. Upon this supremacy of Pp, figure out several takeaways. First and foremost, the effectiveness of PMixUp follows the trend analyzed in section 5 The replacement based augmentation was most effective when we trans- formed tokens of verbs or adjectives regardless of text classification types. Accordingly, PMixUp
7

---

Under review as a conference paper at ICLR 2023
also performed best when we augmented tokens under verbs and adjectives before interpolating feature space with TMix.
Furthermore, PMixUp is especially effective when extremely few training samples exist per class. We presume this result implies that PMixUp creates augmented training samples with the lowest interference on the core semanties of given original training samples. Furthermore, we interpret the benefit of data augmentation (PMixUp as well as the other methods) as weakened when many train- ing samples exist per class. A large number of training samples would be enough learn the core semantics of each intent; thus, the augmentation would become less effective in creating benefits for the model. Accordingly, we discovered several cases where the solely-applied replacement- bbased augmentation method performs better than PMixUp. We evaluate the performance gap between the PMix Up and the best augmentation method as small (less than I or percent). Nevertheless, we resulted in PMixUp becoming a state-of-the-art data augmentation method to escalate the text clas- sification performance in general cases in our experiment settings. Thus, it would benefit machine learning practitioners and the NLP community to tackle the challenge of lack of training samples when they train their text classifiers.
Table 5: Performance comparison on topic classification tasks. Note that Important Token option is considered with Probability-baset approach presented in section 4
DHPedia
SuccUwerrlow
BANKING
Olsumed
10
200
1500
10
300
2500
10
300
660
30
10
32
8
Baseline (Na Anpmneaiatina
0,7055
0.8822
119380
0.9520
0.9661
1.9000
0.3665
0.5755
0.9012
0.3015
0.9175
0.0322
0 0.9400
0:6812
Noun
0.6607
0.7864
0.9080
0.8300
0.9654
0.9882
0.5610
0.5795
0.9020
0.3455
0.9157
0.9511
0.7645
0.6973
Reploccment
Synooyas
Important Tokens
Adjective
Verth
0-6846
0,7023
0.7129
017500
(17892
0.7892
119009
0.900
119000
0.926
0.9433
0.9249
0.884
0.999)
0.9600
0.923
119900
0.5418
0 8.5202
0.6800
0
0.5651
0.8778
0.8978
0.8010
019021
0.334
0.8077
0.8955
0.5549
0.915
0.9201
0.9408
0.9621
0.9678
03559
0.3401
01905
0.063
0.7112
0.6680
Ramdum Tokens
6.7050
0,818
09121
0.5018
0.9633
09909
0.4087
0.8762
0.8088
0.8818
0.9790
0.6677
cooporial)
0:7410
0.8310
01100
03560
00030
19303
TMIs (rcpmiaacd))
0.7647
09100
0.9611
0.9644
09900
0.0633
0.9010
0.7041
0.3991
0.9858
0.016
0.5611
Noun
6.7796
0.7913
09179
0.9622
0.8677
0.9933
0.339
0.5775
0.9919
0.9012
0.9652
0.9430
0.6712
0.8014
01913
0.7841
0.5797
0.9212
0,9889
0.9617
0.7109
PM (OURS)
Adjective
Importunt Tokens
0,8025
0.9000
0.7546
0.2504
0,9419
0.8911
0.6708
Random Toleens
0,7762
09115
019653
0.646
0
0.611
0.5763
0
0.9652
0.6379
Table 6: Performance comparison on sentiment analysis tasks. The highlighted values indicate the highest scores achieved a particular setting.Nor that Important Token option is considered with Probability-based approach presented in section
Amazon Polarity
Yelp
IMDB
10
200
2500
10
200
2500
10
200
2500
Baseline (No Augmentation)
0.6148
0.869
0.8887
0.7128
0.8928
0.9122
0.6430
0.879
0.9132
Noun
0.6088
0.879
0.9078
0.7078
0.8796
0.9112
0.6498
0.8836
0.9222
Verb
0.6122
0.8836
0.9120
0.7388
0.8922
0.9239
0.6328
0.8854
0.9198
Synonym Replacement
Adjective
0.7173
0.8854
0.9198
0.7412
0.8854
0.9121
0.6328
0,8860
0.9202
Important Tokens
0.6022
0.8860
0.9202
0.7128
0.8860
0.9009
0.6440
0.7960
0,9000
Random Tokens
0.6276
0,8760
0.9017
0.7284
0.8845
0.9138
0.6262
0,8829
0.9220
TMix (reported)
0.6930
0.8740
0.9030
TMix (reproduced)
0.5142
0,7960
0,5000
0.7821
0.9411
0.9132
0.6986
0,8712
0.9076
Noun
0.6768
0.8740
0.9030
0.7901
0.9511
0.9222
0.6340
0.8746
0.9110
Verb
0.7221
0,8712
0.9076
0.8109
0.9619
0.9114
0.6988
0.8886
0.9108
PMixUp (OURS)
Adjective
0.6809
0.882
0.9110
0.7589
0.9617
0.91 14
0.6854
0,8840
0.9116
Important Tokens
0,7138
0.8655
0.9108
0.7645
0.9408
0.9089
0.6455
0.8230
0,8978
Random Tokens
0.5649
0.8778
0.8999
0.6625
0.9180
0.9209
0.6450
0,8780
0.9079
WHAT DRIVES PMIXUP'S EFFECTIVENESS?
Setup Upon discovering the effectiveness of PMixUp from above sections, we further investigate the underlying reason for its superior performance. We hypothesize that a better learning paradigm would fully utilize the capacity of the model, which means that models achieving better inference performance contain more fruitful information within its layers. In order to justify our hypothesis, we utilize Centered Kernel Alignment (CKA) (Kornblith et al. 2019). a method which returns the similarity between two models in scale between 0 and I. We compare the representation similar- ities at different layers when the model is trained with different augmentation strategies. A higher similarity among layers within the same network would mean shared knowledge embraced be- tween the layers, further implying that the model has a low knowledge capacity and a lack of ability to describe more abundant aspects of a given text data. Note that among various methods to quantify
8

---

Under review as a conference paper at ICLR 2023
representation similarities, such CCA (Hardoon al. 2004) SVCCA RRghhu al. 2017 we utilize CKA which presented state-of ootie--o--heharr performances in their domain's 's benchmark settings (Kornblihh et al. 2019). For implementation details, measured the layer-wise similarity within model by extracting the feature vectors obtained after normalization of each fine-tuned BERT layers. We compared the results from baseline model without any augmentation, TMix, and our proposed PMixUp. The test dataset was used for the CKA process, and the results are shown in Figure
10
10
08
0.6
0.6
0.4
0.4
04
2
0.2
0.2
0.0
10
0.0
(a) Baseline
(b) TMiX
(c) PMixUp
Figure 2: Visualized representation similarities yielded by CKA on BANKING dataset.
Analysis From the experiment results, we could confirm our initial hypothesis and observe that model trained with the proposed PMixUp has the highest knowledge capacity among different aug- mentation strategies. While the other two setups showed higher similarities among layers, PMixUp resulted in lower representation similarity. The difference especially noticeable higher lev- els of the model, which can connect to the prior works of (Rogers et al.) 2020) and (Ethayarajh) 2019); while higher layers not only contain task-specific information but also high-level contex- tual understandings of texts, a clearly low similarity in such layers imply that a model trained with PMixUp is more capable of understanding the contextual information of given text. We can hence conclude that model trained with the PpixUp approach able to illustrate given data in a more fruitful manner, and therefore leading to high classification performance.
DISCUSSIONS AND CONCLUSION
We present series of analyses discover key factors of text classification and effective data augmentation methods. First, we figure out that important POS information differs in topic classification and sentiment analysis; nouns are critical determinants in topic classification, whereas verbs or adjectives are essential in sentiment analysis. Furthermore, we extended these observations by questioning whether replacing these important POS would escalate classification performance when it gets augmented. Throughout experimental analyses. we discovered that replacing verbs and adjectives was commonly effective both topic classification and sentiment analysis tasks. Upon these findings, we further introduce PMixUp, novel data augmentation strategy that utilizes both synonym replacement and feature space interpolation. We prove that method not only matches but outperforms previously dominant methods through experi- ments on nine benchmark datasets, achieving state-of-the-art results, especially a case where few training samples exist per class. While our study proposes concrete data augmentation practices, several improvement avenues still exist. While the PMixUp replaces particular POS tokens with synonyms in WordNet. future works might utilize other methods of finding syn- onyms (i.e., Thesaurus (Roget's 21st Century Thesaurus 2013), mytheas (LibreOffice) (Zhang al., (0155). Future works might improve our PMixUp with the other feature space interpola- tion techniques such as Local Additivity based Data Augmentation (LADA) (Chen et al. (2000a)
Furthermore, our PMixUp could be examined under harsher circumstances, such class imbalance,
check its robustness. Lastly, would examine the effectiveness of PMixUp in other NLP tasks such as Question Answering (Rogers el al.) 2021) Named Entity Recognition (Yadav & Bethard 2019), or translation (Yang al. 2020a).. We expect our study serves effective guideline for data augmentation, and recommend applying our works on various text classification tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ateret Anaby-Tue Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning the rescue! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7383-7390, 2020.
Markus Bayer, Marc-André Kaufhold, and Christian Reuter. A survey data augmentation for text classification. arXiv preprint arXiv:2107.03/58, 2021.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen- tations. International conference on machine learning, pp. 552-560. PMIR, 2013.
Samuel Bowman and Xiaodan Zhu. Deep learning for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 6-8, 2019.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003. 04800, 2020.
Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. Local additivity based data augmentation for semi-supervised ner. arXiv preprint arXiv:2010.01677, 2020a.
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-inforned interpolation of hidden space for semi-supervised text classification. arXiv preprint arXiv:2004. 122399,,00.12339, 2020b.
Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An empirical survey of data
augmentation for limited data learning in nlp. arXiv preprint arXiv:2/06007499 2021.
Franca Debole and Fabrizio Sehastiani. An analysis of the relative hardness of reuters-21578 subsets. Journal of the American Society for Information Science and technology, 56(6):584-555 2005.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00880, 2018.
Terrance Veieies and Graham W Taylor. Dataset augmentation feature space. arXiv preprint arXiv:1702.05538, 2017.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.005:: 2019.
Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. Genaug: Data augmentation for finetuning text generators. arXiv preprint arXiv:2010.01794, 2020.
Akinori Fujino, Hideki Isozaki, and Jun Suzuki. Multi-label text categorization with model combina- tion based on fl-score maximization. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-1I, 2008.
Siddhant Garg and Goutham Ramakrishnan. Bac: Bert-based adversarial examples for text classifi- cation. arXiv preprint arXiv:2004.01977 019970, 2020.
David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):269-2264, 2004.
Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. A survey recent approaches for natural language processing in low-resource scenarios. arXiv preprint arXiv:2010.12309, 2020.
Michat Jungiewicz and Aleksander Smywinski-Poohe Towards textual data augmentation for neural networks: synonyms and maximum loss. Computer Science. 20, 2019.
Oleksandr Kolomiyets, Steven Bethard, and for textual temporal analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, volume 2, pp. 271-276. ACL; East Stroudsburg, PA, 2011.
Moens. portaiioytyyylity experiments
10

---

Under review as a conference paper at ICLR 2023
Simon Kornblith, Mohammad Norouzi, Honglak Lee. and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR. 2019.
Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and William Campbell. A closer look feature space data augmentation for few-shot intent classification. arXiv preprint UrXiv:1910.4410 2019.
Gakuto Kurata, Bing Xiang, and Bowen Zhou. Labeled data generation with encoder-decoder Istm for semantic slot filling. In INTERSPEECH, pp. 725-729, 2016.
Edward Loper and Steven Bird. Nitk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110.1711.05 2017.
Andrew Maas, Raymond Daly, Peter Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Vukosi Marivate and Tshephisho Sefara. Improving short text classification through global augmen- tation methods. In International Cross-Doman Conference for Machine Learning and Knowledge Extraction, pp. 385-399. Springer. 2020.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(1)): 39-41, 1995.
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. Deep learning-based text classification: comprehensive review. ACM Computing Surveys 54(3):1111 2021.
Sadia Zaman Mishu and SM Rafiuddin. Performance analysis of supervised machine learning algo- rithms for text classification. In 2016 19th International Conference on Computer and Information Technology (ICCIT), pp. 409-413. IEEE, 2016.
Anaïs Ollagnier and Hywel TP Williams. Text augmentation techniques for clinical case classifica- tion. In CLEF (Working Notes), 2020.
Sherjil Ozair and Yoshua Bengio. Deep directed generative autoencoders. 1rXiv:1410.0630, 2014.
arXiv preprint
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
Anna Rogers, Matt Gardner, and Isabelle Augenstein. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. arXiv preprint arXiv:2107.12708, 2021.
TE Roget's 21st Century Thesaurus. Thesaurus. com. Retrieved May, 9:2013, 2013.
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. Automatic keyword extraction from individual documents. Text mining: applications and theory, 1:1-20, 2010.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprini a 1511606709 2015.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough- to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009 20.3:/0009/////88 2020.
11

---

Under review as a conference paper at ICLR 2023
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How fine-tune bert for text classification? In China national conference on Chinese computational linguistics, pp. 194-206, Springer, 2019.
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S Yu, and Lifang He. Mixup- transformer: dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394, 2020.
Xiaobing Sun and Wei Lu. Understanding attention for text classification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3418-3428, 2020.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance text classification tasks. arXiv preprint arXiv:1901.11191 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: Statc-of-the-ant natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. Short text clustering via convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 62-69, 2015.
Vikas Yadav and Steven Bethard. A survey recent advances in named entity recognition from deep learning models. arXiv preprint 19X0v: 1470, 2019.
Shuoheng Yang, Yuxin Wang, and Xiaowen Chu. A survey of deep learning techniques for neural machine translation. arXiv preprint arXiv:2002.07526, 2020a.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. arXiv preprint arXiv:2004. 11546, 2020b.
Liang Yao, Yin Zhang. Baogang Wei, Zhe Jin, Rui Zhang, Yangyang Zhang, and Qinfei Chen. Incorporating knowledge graph embeddings into topic modeling. In Thirty-first AAAI conference on artificial intelligence, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
12


Reviews:
[{"review_id": "I5ohbZ-9Nic", "reviewer": "Reviewer_iN19", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes PMxiUp, a new way to add to text data augmentation by replacing parts of speech (POS) and interpolating features. The proposed method swaps out tokens that belong to a particular POS tag and uses feature space interpolation in order. The author discovered through experimentation that nouns are essential in determining how to label a topic. At the same time, verbs and adjectives are paramount in figuring out how someone feels about something. In the reported experiments on nine public benchmark datasets to test PMixUp's best performance on classification performance rather than previous works, it becomes a new state-of-the-art in specific settings.", "summary_of_the_review": "Good: This paper is of interest to the *ICLR audience and could be published, but might not be appropriate for a top-tier publication.", "strength_and_weaknesses": "An exciting new text method is presented in text data augmentation, focusing on using different POS correlates to the performance of topic classification and sentiment analysis; nouns are critical determinants in topic classification, whereas verbs or adjectives are essential in sentiment analysis. The results show that the proposed model using CKA during training performs well when comparing the representation similarities at different layers when the model is trained with different augmentation strategies scaling between 0 and 1.\n\nSome assertions are made that are not entirely supported by the results obtained. Maybe this is how I felt because the results did not thoroughly analyze how the proposed method would perform with the class imbalance dataset and whether it would be inadequate for a robust model.\n\nMany why questions instead of the how ones are addressed in this article.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FU00o7nuI_", "reviewer": "Reviewer_Rbzx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes PMixUp that combines two augmentation methods (replacement-based augmentation and feature space augmentation) for the topic classification and sentiment analysis. They emphasis the importance of POS tags for those NLP tasks based on their study, and apply the feature space interpolation to swap out tokens corresponding to those POS tags. The results indicate that nouns are matter to some extent when selecting to label the topics.  ", "summary_of_the_review": "The method of combining two augmentation of techniques is reasonable. The results show the types of POS can impact for the model performance in some circumstances. However, it is not clear to demonstrate the settings for all the experiments. ", "strength_and_weaknesses": "Strengths:\n* The paper reads well and it is in general easy to follow.\n* The idea is neat and easy to implement the method of combining the two augmentations is clear.\n\nWeaknesses:\n* The novelty of the proposed method is relatively limited, especially in terms of theoretical contribution. \n* Determining which types of POS are relevant to model performance on push tasks is still somewhat unclear. It would be great to further explore the error analysis based on the tokens and sentences.\n* The impact of model performance by augmenting Nouns, Verbs and Adjectives for those NLP tasks are limited from the results shown in Table 3 and Table 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ufJDoRCPRm", "reviewer": "Reviewer_mExN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new data augmentation method, PMixUp. Specifically:\n\n* This paper performs a relatively comprehensive study of the importance of POS tags in topic classification and sentiment analysis.\n* This paper proposes a new data augmentation method, PMixUp, that utilizes both replacement-based and feature space interpolation methods.\n* The authors tests their new method on nine public benchmark settings.\n\n", "summary_of_the_review": "I am leaning towards acceptance but would like to see some revisions to the paper.", "strength_and_weaknesses": "Strengths:\n\n* This paper combines the replacement-based method and the feature space interpolation method together to increase the performance.\n* This paper answers a lot of why questions instead of how questions.\n\nWeaknesses:\n\n* I don't think our field put topic classification and sentiment analysis into the same category.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2uQp1pcB__w", "reviewer": "Reviewer_8rka", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript combines two reasonable augmentation strategies to improve topic classification and sentiment analysis.  The first is replacement based augmentation and TMix style feature space augmentation.\n\nThe findings suggest that using Part of speech (POS) to drive replacement augmentation matters, and its impact is different based on the task.  Second, these two strategies combine effectively.", "summary_of_the_review": "The approach of combining TMix and replacement augmentation is sensible.  The particular claims about the relationship between POS classes and sentiment analysis are not as clean as would be expected from the presentation of the work.  It's not clear how this would be extended to some other NLP/NLU task without a similar precursor POS analysis.  Additionally the value of POS sensitive replacement is not obviously necessary for either task; while there is some improvement, it is typically quite modest and not entirely consistent across tasls .\n\nTable 6 includes a typo 9017 should be 0.9017.  \nTable 5 why is StackOverflow 10, TMix (reproduced) only 0.0663 while even the no augmentation baseline is 0.2668?", "strength_and_weaknesses": "Strengths\n* This is a reasonable combination of existing techniques.\n* Decent analyses of different POS influence on performance of different tasks on a variety of corpora.\n\nWeaknesses\n* The identification of which POS is relevant for which task remains very manually derived. It would be interesting to see the conditioning on POS as part of an objective function that could be learned as part of the training operation. This would eliminate the need for a user to perform a POS base analysis in order to bring this framework to a new task.\n* The reported claims of NOUNS for topic classification and VERBS and ADJECTIVES for sentiment is not as clearly supported by the data as the abstract would suggest.  While the role of nouns in topic classification is reasonably well supported, Table 3 suggests a more nuanced situation in the Sentiment Analysis corpora with less consistency across corpora.  Similarly Table 4 suggests that there are corpora differences again in the value of replacing different POS classes.  While the syntactic class as a space for augmentation.\n* The reported differences in Tables 5 and 6 are small.  Some significance testing would go a long way to understanding the relevance of these findings.  Specifically, it's not clear that the value of PMixUp compared to TMix is substantial enough to warrant the additional training and risk of poor replacement generation. \n* The findings from Tables 1-3 seem to be inconsistent with those in Table 5.  The replacement based augmentation benefits from NOUNS for topic classification somewhat strongly.  However, in Table 5, this suggests that VERB replacement is better when replacement augmentation is combined with TMIX.  This relationship is surprising and suggests that initial analyses based on, say replacement, may not be useful to designing a system.  The two approaches aren't entirely complementary", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject