Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes PMixUp that combines two augmentation methods (replacement-based augmentation and feature space augmentation) for the topic classification and sentiment analysis. They emphasis the importance of POS tags for those NLP tasks based on their study, and apply the feature space interpolation to swap out tokens corresponding to those POS tags. The results indicate that nouns are matter to some extent when selecting to label the topics.  
strength_and_weaknesses: Strengths:
* The paper reads well and it is in general easy to follow.
* The idea is neat and easy to implement the method of combining the two augmentations is clear.

Weaknesses:
* The novelty of the proposed method is relatively limited, especially in terms of theoretical contribution. 
* Determining which types of POS are relevant to model performance on push tasks is still somewhat unclear. It would be great to further explore the error analysis based on the tokens and sentences.
* The impact of model performance by augmenting Nouns, Verbs and Adjectives for those NLP tasks are limited from the results shown in Table 3 and Table 4.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
The paper is clear and the content is easy to read.

Novelty:
The novelty of the proposed method is relatively limited, especially in terms of theoretical contribution.

Reproducibility: 
Due to the lack of details of the experimental setup, it might be difficult to reproduce the experiment.
summary_of_the_review: The method of combining two augmentation of techniques is reasonable. The results show the types of POS can impact for the model performance in some circumstances. However, it is not clear to demonstrate the settings for all the experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript combines two reasonable augmentation strategies to improve topic classification and sentiment analysis.  The first is replacement based augmentation and TMix style feature space augmentation.

The findings suggest that using Part of speech (POS) to drive replacement augmentation matters, and its impact is different based on the task.  Second, these two strategies combine effectively.
strength_and_weaknesses: Strengths
* This is a reasonable combination of existing techniques.
* Decent analyses of different POS influence on performance of different tasks on a variety of corpora.

Weaknesses
* The identification of which POS is relevant for which task remains very manually derived. It would be interesting to see the conditioning on POS as part of an objective function that could be learned as part of the training operation. This would eliminate the need for a user to perform a POS base analysis in order to bring this framework to a new task.
* The reported claims of NOUNS for topic classification and VERBS and ADJECTIVES for sentiment is not as clearly supported by the data as the abstract would suggest.  While the role of nouns in topic classification is reasonably well supported, Table 3 suggests a more nuanced situation in the Sentiment Analysis corpora with less consistency across corpora.  Similarly Table 4 suggests that there are corpora differences again in the value of replacing different POS classes.  While the syntactic class as a space for augmentation.
* The reported differences in Tables 5 and 6 are small.  Some significance testing would go a long way to understanding the relevance of these findings.  Specifically, it's not clear that the value of PMixUp compared to TMix is substantial enough to warrant the additional training and risk of poor replacement generation. 
* The findings from Tables 1-3 seem to be inconsistent with those in Table 5.  The replacement based augmentation benefits from NOUNS for topic classification somewhat strongly.  However, in Table 5, this suggests that VERB replacement is better when replacement augmentation is combined with TMIX.  This relationship is surprising and suggests that initial analyses based on, say replacement, may not be useful to designing a system.  The two approaches aren't entirely complementary
clarity,_quality,_novelty_and_reproducibility: Clarity: Mostly clear.  Additional contextualization for the analyses would be helpful.  The claims made in the abstract and introduction are less consistently supported by the data than would be expected.

Quality: Reasonably high

Novelty: Modest.  Both TMix and replacement augmentation have been previously developed. Their combination is being addressed in this work, with additional analysis for the tasks the paper focuses on.

Reproducibility: High


summary_of_the_review: The approach of combining TMix and replacement augmentation is sensible.  The particular claims about the relationship between POS classes and sentiment analysis are not as clean as would be expected from the presentation of the work.  It's not clear how this would be extended to some other NLP/NLU task without a similar precursor POS analysis.  Additionally the value of POS sensitive replacement is not obviously necessary for either task; while there is some improvement, it is typically quite modest and not entirely consistent across tasls .

Table 6 includes a typo 9017 should be 0.9017.  
Table 5 why is StackOverflow 10, TMix (reproduced) only 0.0663 while even the no augmentation baseline is 0.2668?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes PMxiUp, a new way to add to text data augmentation by replacing parts of speech (POS) and interpolating features. The proposed method swaps out tokens that belong to a particular POS tag and uses feature space interpolation in order. The author discovered through experimentation that nouns are essential in determining how to label a topic. At the same time, verbs and adjectives are paramount in figuring out how someone feels about something. In the reported experiments on nine public benchmark datasets to test PMixUp's best performance on classification performance rather than previous works, it becomes a new state-of-the-art in specific settings.
strength_and_weaknesses: An exciting new text method is presented in text data augmentation, focusing on using different POS correlates to the performance of topic classification and sentiment analysis; nouns are critical determinants in topic classification, whereas verbs or adjectives are essential in sentiment analysis. The results show that the proposed model using CKA during training performs well when comparing the representation similarities at different layers when the model is trained with different augmentation strategies scaling between 0 and 1.

Some assertions are made that are not entirely supported by the results obtained. Maybe this is how I felt because the results did not thoroughly analyze how the proposed method would perform with the class imbalance dataset and whether it would be inadequate for a robust model.

Many why questions instead of the how ones are addressed in this article.
clarity,_quality,_novelty_and_reproducibility: The proposed approach is nuanced, but the algorithm for replicating this is unclear. If possible, perhaps the author can briefly discuss how PMxiUp can be reproduced. Maybe very difficult to implement compared to Emix, a data augmentation method that uses interpolations of word embeddings and hidden layer representations to construct virtual examples (https://aclanthology.org/2020.coling-main.611/).
summary_of_the_review: Good: This paper is of interest to the *ICLR audience and could be published, but might not be appropriate for a top-tier publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new data augmentation method, PMixUp. Specifically:

* This paper performs a relatively comprehensive study of the importance of POS tags in topic classification and sentiment analysis.
* This paper proposes a new data augmentation method, PMixUp, that utilizes both replacement-based and feature space interpolation methods.
* The authors tests their new method on nine public benchmark settings.


strength_and_weaknesses: Strengths:

* This paper combines the replacement-based method and the feature space interpolation method together to increase the performance.
* This paper answers a lot of why questions instead of how questions.

Weaknesses:

* I don't think our field put topic classification and sentiment analysis into the same category.
clarity,_quality,_novelty_and_reproducibility: This paper is relatively clear and the content is original. It might be hard to replicate the experiments since the details are fuzzy.
summary_of_the_review: I am leaning towards acceptance but would like to see some revisions to the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper combines three augmentation methods (replacement-based augmentation which is enabled using POS tags, feature space augmentation and a Mixup style) to support training for text classification tasks.  They did empirical analysis to understand how important tokens of different POS tags were for classification.  They also tested their approach on a wide variety of data sets including sentiment analysis.  This is good work but contested by reviewers.

The novelty is relatively limited.  The relative role of nouns, verbs and adjectives has almost certainly been studied before in text classification, but the results (Table 4) are mixed other than the favorability of verbs. The use of POS with augmentation is the key novelty.  The empirical evaluation of Tables 1&2 was inconsistent with the eventual results, Table 5&6 so less useful.

In text classification, a common standard for significance testing is average of 5 or 10.  10 is adequate to do hypothesis testing per data set.  With less you need one hypothesis test for the full experiment.   [1] is a reference for augmentation but not a classic text classification work.  Also, the model is developed (i.e., the POS sets to use), i.e., as shown in Table 4, using the same data sets used in the final evaluation, i.e., Tables 5 & 6.   The final Tables 5&6 should only have Verb, the best performing, it seems poor to present 5 different methods and then report the best performing.

There was a substantial difference between reviewers.  In interpreting this, reviewers confidence, experience and publications were taken into account in arriving at the final Reject.
justification_for_why_not_higher_score: Reviewers experience was taken into account in arriving at the final Reject.  The one Accept had less confidence and substantially less publication experience than the one Reject.  On close inspection of the reviews, 8rka was found to have good points that weren't well rebutted.
Decision: Reject