Is ChatGPT the ultimate Data Augmentation Algorithm?
Authors: Frédéric Piedboeuf
Philippe Langlais
Venue: EMNLP 2023 Findings (2023)
Keywords: Data augmentation
ChatGPT
GPT-3.5
classification
T5

Abstract: In the aftermath of GPT-3.5, commonly known as ChatGPT, research have attempted to assess its capacity for lowering annotation cost, either by doing zero-shot learning, generating new data, or replacing human annotators. Some studies have also investigated its use for data augmentation (DA), but only in limited contexts, which still leaves the question of how ChatGPT performs compared to state-of-the-art algorithms. In this paper, we use ChatGPT to create new data both with paraphrasing and with zero-shot generation, and compare it to seven other algorithms. We show that while ChatGPT performs exceptionally well on some simpler data, it overall does not perform better than the other algorithms, yet demands a much larger implication from the practitioner due to the ChatGPT often refusing to answer due to sensitive content in the datasets.

Is ChatGPT the ultimate Data Augmentation Algorithm?
Frédéric Piedboeuf
Philippe Langlais
RALI, Diro
RALI, Diro
Université de Montréal
Université de Montréal
frederic. itteericcppeddbeee@uutttrreeee
felipe@iro. umontreal.ca
Abstract
the performance of ChatGPT and GPT-4 classi- fication tasks of medium size (500 examples), com- paring zero shot data generation and fow-shot+data augmentation to crowdsourcing annotation, but not to other DA techniques. Dai et al. (2023) compare the use of DA with ChatGPT for large datasets (sev- eral thousand examples), focussing on bio-medical data. Their algorithm however does not isolate ChatGPT for data augmentation, but instead com- bines with pre-traue pre-training, preventing an objective evaluation the capacities of ChatGPT for DA.
In the aftermath of GPT-3.5, commonly known as ChatGPT, research has attempted to assess its capacity for lowering annotation cost, either by doing zero-shot learning, generating new data, or replacing human annotators. Some studies have also investigated its use for data augmentation (DA), but only in limited con- texts, which still leaves the question of how ChatGPT performs compared to state-of-the- art algorithms. this paper. we use Chat- GPT create new data both with paraphrasing and with zero-shot generation, and compare it to seven other algorithms. We show that while ChatGPT performs exceptionally well on some datasets, it overall does not perform bet- ter than the other algorithms, yet demands much larger implication from the practitioner due to ChatGPT often refusing to answer due sensitive content the datasets.
In this paper, we compare the use of ChatGPT for paraphrasing well as for zero-shot data gen- eration, on five classification datasets (three binary and two multiclass ones), seven algorithms that have shown good performances the past. We show that the performance of ChatGPT is highly dependent on the dataset, due mainly to poorly defined tasks, which makes prompting difficult. These tasks were chosen because they are stan- dard on the textual DA literature, and such these biases are important point out. With efficient prompting, generating new data with ChatGPT mains the best way to perform textual DA.
1 Introduction
Textual data augmentation (DA) rich and com- plicated field, with the goal of finding ways to gen- erate the most informative artificial examples to add to a training set without additional labelling cost. Many techniques have been developed and thoroughly tested in order to create informative ar- tificial data. Recently, the new model ChatGPT has put into question much of what was known for dataset construction, with researchers even dering if its capacities meant the end of human labelling (Kuzman et al., 2023).
2 Related Work
There are roughly three main types of DA tech- niques: word-level augmentation, paraphrase, and generative methods.'
In word-level DA, operations modify genuine sentences' words to create variations. Commonly, the operation word substitution, replacing with a synonym (Wei and Zou, 2019; Liesting et al., 2021), a neighboring word in pre-trained embed- ding (Marivate and Sefara, 2020), or by masking and predicting with neural network (Kobayashi. 2018; Wu et al., 2019; Kumar et al., 2020).
While extensive studies have been done on the use of ChatGPT for many Natural Language Under- standing tasks, its use for DA has been surprisingly little investigated yet. To our knowledge, only two papers have looked into this, namely (Møller et al., 2023) and (Dai al., 2023). Both however ex- plore limited settings, which makes it still unclear whether ChatGPT is a good tool for data augmenta- tion or not. In particular, Møller et al. (2023) study
For full review of the literature surrounding textual data augmentation, we refer to (Shorten and Khoshgoftaar, 2019; Feng ct al., 2021).

---

Paraphrasing techniques attempt to create para- phrases from the available sentences. The most seminal technique of this family Back- Translation (BT), a technique in which a sentence is translated to a pivot language and then back into English (Hayashi et al., 2018; Yu et al., 2018; Edunov et al., 2018; Corbeil and Ghadivel, 2020; AlAwawdeh and Abandah, 2021). Neural networks directly generate paraphrases have also been used, with specialized decoding techniques for RNN (Kumar et al., 2019), or by using a BART model trained on a corpus of paraphrases generated from BT (Okur et al., 2022).
test the capacities of ChatGPT we select variety of techniques to compare DA objectively: EDA, AEDA, CBERT, CBART, CGPT, BT, TS-Tapaco, ChatGPT-Pal and ChatGPT-Desc. We briefly de- scribe each of those algorithms, and refer to the code for the full details of the implementations and hyper-parameters.
EDA and AEDA are two simple word-level al- gorithms that achieved great performances the past. In EDA, one of four operations is chosen (in- sertion of related words, swapping words, deleting words, and replacing words by synonyms) and ap- plied to a percentage of the words in the sentence." In AEDA (Karimi et al., 2021), punctuations are randomly inserted in the sentence (among "?", ".", n." "!" and "."), the number of insertion being RANDINT(1, len (enentence)//))
Generative methods learn the distribution of the training data and generate new data from it. While the obvious advantage is that data should be more diverse, generative models are often more compli- cated to train and fine-tune correctly. Examples of this family of methods includes using GPT-2 for generating new data (Kumar ct al., 2020; Liu et al., 2020; Queiroz Abonizio and Barbon Jumior, 2020), other generative models such as VALss (Ma- landrakis al., 2019; Qiu et al., 2020; Piedbocuf and Langlais, 2022) or conditional VAEs gen- erate examples conditionned on the class (Zhuang et al., 2019; Malandrakis al., 2019; Rizos et al., 2019; Wang ct al., 2020).
CBERT and CBART have very similar method- ologies. We prepend the class of the example to all genuine sentences, mask fraction of the tokens, and fine-tune the model on the available training set to predict the masked words. For generation, we then give the modified sentence (masked and with the class prepended) and pass it through the trans- formers. The main difference between CBERT and CBART is that the latter can predict spans instead of tokens, which allows more flexibility.
Finally, there have also been interest into the of proprictury models for DA. Both Yoo et al. (2021) and Sahu al. (2022) show that GPT-3 is able generate excellent new data, either by com- pleting a list of sentences from one class, or by asking to generate both new sentences and their labels. ChatGPT has also been studied to gen- erate new data, asking it to paraphrase existing data (Dai et al., 2023) for few-shot learning, but the authors first fine-tune the classifier with large dataset from the same distribution, making it hard isolate the impact of the generated sentences. Fi- nally, Møller et al. (2023) look at the performance ChatGPT and GPT-4 for data augmentation com- pared to human annotation, and conclude that for simple datasets (such as review analysis of prod- ucts), ChatGPT is better, but otherwise human an- notation outperforms data generated by ChatGPT. As they do not compare other DA techniques, is also hard to know how ChatGPT performs.
CGPT also works by prepending the class the sentence, which then allows GPT-2 learn generate conditionally it. For generation, we give the class as well as the separator token and let GPT-2 generate new sentences. 4
In BT, first translate the sentence to pivot language and retranslate it in English, creating para- phrases. We use the FSMT model from hugging face", with the intermediary language heing Ger- man, which has been shown to obtain good perfor- mances (Edunov et al., 2018).
Okur et al. (2022) propose to fine-tune BART corpus of in-domain paraphrases created with BT. We found in our experiments that we could get results just as good by using T5-small-Tapal which the T5 model fine-tuned on the corpus of
"Cdde available at https: //go:///iii www.tctt/b..oee.eooooooooooeaoat wwt.s/oooi//.l..../oooooooommmme
DatakugmentationENN..
"We affect 10% of the words, as recommended in the paper of Wei and Zou (2019)
"We GPT2-large with top_p of 0.95 and no repeat n gra.grmm_siii size of three.
3 Algorithms
"https://huggingface hoteeecooooo//taasoorereeee model_doc/fsmt
It is not clear from the literature which DA algo- rithms perform best, and so in order to thoroughly
t5-small-tapaco ctt.aacccoctpppadaaaal

---

paraphrases TaPaCo (Scherrer, 2020).
papers, we found DA to have little effect in this set- ting. As such we do not discuss them in the paper. but we include them in Appendix C.
Finally, we test the use ChatGPT, either by ask- ing for paraphrases of genuine sentences (ChatGPT- Par), or by giving a short description of the task and classes and asking for novel sentences (CharGPT- Desc). We give the exact prompts in Appendix A. Because part of the experiments were done before the API became publicly available, we use in this paper the Pro version of the Web interface of Chat- GPT and leave further fine-tuning for future work?.
Based on results reported in the literature as well as our experiments, we use larger ratio of generated-to-gentee
sentences for small datasets and a smaller one for larger datasets. Concretely, we generate 10 new sentences per genuine sen- tences for the dataset sizes of 10 and 20, and one for 500 and 1000. As a classifier, we use BERT, and we fine-tune it using the development set, using ctrly-stopping. We run each experiment 15 times, reporting accuracy for binary tasks, and macro-fl for multiclass tasks.
4 Datasets and Methodology
We test on five datasets with various complexity and characteristic to fully assess the performance of the algorithms. We use SST-2 (Socher et al., 2013), a binary dataset of movie reviews classification, FakeNews dataset of news to classify into real news or fake ones, Irony and IronyB (Van Hee et al.. 2018). a binary and multiclass version of a task consisting into classifying tweets as ironic or not, and which kind of irony for the multiclass version (polarity clash, situational irony, other irony), and TREC6 (Li and Roth, 2002), a multiclass dataset where the goal is to classify questions into six cate- gories (abbreviation, description, entities, human beings, locations, and numeric values). More infor- mation is available in Appendix C.
5 Results
Table shows results for the dataset sizes of 10 and 20 and Table 2, for the sizes of 500 and 1000. For small dataset sizes, we observe that the per- formance of ChatGPT-Par is on par with the best algorithms, but doesn't beat them by significant margin on average. While ChatGPT-Desc per- forms exceptionally well, its performance comes almost exclusively from SST-2, for which gener- ates highly informative data. For other datasets, it mostly either brings no gain in the performance or very small one. Overall, all the algorithms pro- vide reasonable augmentation of the performance. except maybe CGPT, which performs poorly. Ex- cluding ChatGPT-Desc, BART and T5-TaPaCo ob- tain some of the best performance, although not by much. Given the generative nature of ChatGPT- Desc and its performance, one could also wonder if it would perform better if we generated more sentences. In Appendix C we show that this is not the case, and that the performance for all datasets plateau quickly.
These datasets were chosen to get a spread of tasks, and because they are commonly used in the literature in data augmentation. SST-2 and TREC6 are both fairly standard in DA research, being used for example in (Kumar et al., 2020; Quteineh et al., 2020; Regina et al., 2021; Kobayashi, 20. The Trony datasets are also used quite regularly, for ex- ample in (Liu et al., 2020: Turban and Kruschwitz, 2022; Yao and Yu, 2021). Finally, while FakeNews has not been used in DA to our knowledge, it is still commonly used for Fake News detection, for example in (Verma et al., 2023; Chakraborty et al., 2023; Iceland, 2023).
For larger training sets, ChatGPT performs bet- ter, while BART and T5-TaPaCo degrade the per- formance. We believe that this is due to these algo- rithms creating paraphrases which are closer to the original sentences, leading to less diversity overall. While on few-shot learning this is not problem be- cause the goal is to give the neural network enough relevant data to learn, on larger datasets diversity seems to become a prevalent factor. Nevertheless, the overall effect of augmentation on moderately
We test data augmentation on two settings: few shot learning (10 or 20 starting examples), and classification with dataset sizes of 500 and 1000 ex- amples. While sampling the starting set, we make sure to balance the classes to be able to observe the performance of data augmentation without the ad- ditional factor of imbalanced data. We also tested the process on the full dataset but, similarly to other
This most notably means that we couldn't 't play with pa- rameters such as the lemperature, which affects the diversity the sentences.
"While not factor much studied in the DA literature, extensive experiments shows that at larger training set sizes, larger ratio is detrimental since it makes the neural network forget genuine examples.
Phttps: http://wwwwkk/lee// kaggle. com/c/fakes neteeeewww///////v/eeeeeee

---

Table Average metric over 15 rums for the training set sizes of (left) and 20 (right) with a ratio of 10. We report
accuracy for binary tasks and macro-fl for multiclass ones. STDs between 1.5 and depending the dataset.
SST2
FakeNews
Irony
IronyB
Trec6
Average
Baseline
56.4/60.6
52.2/53.3
52.5/58.3
23.6/23.4
27.9/34.7
42.5/46.1
EDA
59.4/63.2
55.0/56.6
53.7/57.3
25.3/27.8
30.8/43.7
44.9/49.7
AEDA
59.3/64.6
53,4/55.3
54.1/56.0
25,4/27.1
26.5/44.2
43.7/49.5
BT
59.0/64.6
55.1/56.2
54.3/56.5
25.4/26.7
32.9/46.3
45.4/50.1
CBERT
57.6/63.1
54.5/55.5
55.1/57.4
25.3/29.5
29.0/40.6
44.3/49.2
CGPT
55.6/61.2
52.4/54.7
51.9/53.1
22.9/25.1
23.3/38.1
41.2/46.4
CBART
60.3/64.9
55.8/57.2
54.8/56.6
25.5/28.0
34.1/46.1
46.2/50.6
T5
60.4/64.6
54.5/56.6
53.2/56.7
23.5/27.1
34.0/46.3
45.1/50.3
GPT3.5-Par
62.5/69.0
53.8/54.9
54.2/57.5
24.4/27.8
31.3/44.8
45.3/50.8
GPT3.5-Dest 78.6/82.6
51.5/52.8
53.2/54.1
27.1/27.7
31.4/42.9
48.3/52.0
Table 2: Average metric over 15 runs for the training sizes of 500 (left) and 1000 (right) with ratio of 1. We
report accuracy for binary tasks and macro-fl for multiclass ones. STDs are between 0.6 and 3.0, depending on the dataset.
SST2
FakeNews
Irony
IronyB
Trec6
Average
Baseline EDA
87.7/88.8
73.3/77.0
65.6/68.1
42.4/45.2
81.0/85.4
70.0/72.9
87.9/88.9
73.7/77.6
65.8/68.8
43.1/46.6
81.3/86.1
70.4/73.6
AEDA
88.0/89.0
73.5/77.6
65.7/69.2
42.8/46.1
82.7/86.4
70.5/73.7
BT
88.2/89.1
73.6/77.4
66.2/69.2
42.4/46.0
81.7/86.1
70.4/73.5
CBERT
87.5/88.3
73.6/77.5
65.8/68.1
40.6/45.5
80.9/85.4
69.7/72.9
CGPT
87.8/88.7
73.2/77.6
65.2/68.8
42.8/45.2
82.1/87.2
70.2/73.5
CBART
87.7/88.6
73.9/77.9
65.9/68.8
42.7/45.0
78.6/83.4
69.8/72.7
T5
87.9/88.7
73.8/73.3
65.3/68.2
43.1/45.3
79.9/85.0
70.0/72.1
GPT3.5-Par
88.2/89.1
73.8/77.7
66.8/69.3
42.8/45.9
82.4/87.1
70.8/73.8
GPT3.5-Desc
87.4/88.9
71.9/75.9
64.1/66.9
41.1/44.4
79.8/84.0
68.9/72.0
sized datasets are very small, which brings the ques- tion whether data augmentation is relevant at all in these cases.
discussed in other papers, namely the need for data reparation. ChatGPT-Par quite often refuses to cre- ate paraphrases, especially for the FakeNews and Irony/IronyB datasets which contain occasional mentions of rape and sex. In these cases, we had to manually find the "bad" element of the batch and rerun it without it, adding considerable complexity the data augmentation algorithm. Another op- tion would be to simply not correct them, but our preliminary studies indicate that this degrades the performance DA significantly."
6 Discussion
Of all the algorithms used here, T5 and ChatGPT present the greatest novelty as well as show some of the best performances. As such, we center our discussion these two algorithms. When we ob- serve T5 sentences (see Appendix B), can that they are not as good one would expect, of- ten being ungrammatical badly formed. Still, has been noted before that having correctly formed sentences is not an important criterion for the ef- ficiency of a DA algorithm, which might explain why its performance is high (Karimi et al., 2021). ChatGPT-Dese often has difficulty generating sentences of the desired class, accounting for its overall poor performance, while ChatGPT-Par cre- ates excellent paraphrases, bringing diversity to the dataset while maintaining class coherence. Never- theless, there is a hidden cost that we found was not
6.1 Poorly defined tasks and dataset biases.
Despite the description strategy being able to add something akin to external data, our experi- ments show that ChatGPT underperforms with this method, the performance often being worse than when paraphrasing the existing data. This raises many questions adding more diverse data should augment performance.
As we see in Appendix C, running with smaller ratio for few shot learning reduces the performance by a large amount.

---

We found that for most part, the poor perfor- mance of ChatGPT was related to the poor health of the datasets. Except for SST-2, we found that FakeNews, Trony, IronyB, and TREC6 have poorly defined labels in relation to the task, and that ex- amples in the datasets were often ambiguous to human eyes. Under these conditions, it is difficult to expect ChatGPT to perform well. We underline these problems here because poor dataset health is not a rare phenomenon.
seemed difficult and interesting task, but while analyzing it, we found biased in a sense similar to Irony and IronyB. From what little information
we could find, news were gathered from various sources and split into real or fake news based on the journal they came from. This causes biases because while some journals may have tendency to sprout fake news, it does not mean all of its news are fake. Furthermore, we found strange choices of labelling. For example, all articles from Breitbart are labelled as real news even if it receives a mixed score of factual reporting" and articles from consortium news, which receives the overall same score, are labelled as f
Irony and IronyB are two datasets of the Se- mEVAL 2018 competition. Data was collected from Twitter by collecting tweets that contained some specific hashtag such as #irony, #not, or #sar- casm, which were then removed to form the data of the ironic class. The non-ironic class was then formed by collecting other tweets. This creates a heavy bias in the dataset which shift the task from predicting if the tweet is ironic to predicting if there was a #irony hashtag coming with it. Without the clue that the hashtags give us, it is often impossible to know if the tweet is ironic or not, and we show in Appendix C some examples of ironic tweets which, from manual labelling, we found were ambiguous in their classes.
By refining prompting, we can augment the TREC6 dataset to go to 68.6, which still underper- forms when compared to BERT! We found Chat- GPT to have difficulty understanding the concept of "Entity" and "Human" questions, often labelling them instead as "Description".
Conclusion
Data augmentation is seminal technique to lower annotation cost and keep good performances, but even today it is difficult to figure out which tech- nique works best. particular, the use of ChatGPT has not been correctly assessed for data augmenta- tion, leading to the unknown factor for industries of whether it is worth the price.
TREC6 is a dataset of the Text REtrieval Con- ference and consists in classifying the questions into six categories. While all data was manually annotated, we found inconsistencies in the anno- tation. For example, "What is narcolepsy?" is la- belled as description but "What is fear of motion?" Entity. Other inconsistencies are "What is the oldest profession?" and "What team did base- ball 's St. Louis Browns become?" labelled as Human vs "What do you call a professional map drawer" as Entity, or "Where did Indian Pudding come from?" being labelled as Description but "Where does chocolate come from?" as Location. Given that the same mislabelling remains in the test set (ex "Where does dew come from?" being la- belled as location), ChatGPT generating sentences of the correct class won't 't help the classifier much. It is to note that these issues were already noted by Li and Roth (2002), who advise using multi- label classification to reduce biases introduced in the classifier. In all of its usage for DA however, we found it used as regular classification problem, with all the ambiguity problems it entails.
In this paper, we study nine data augmentation techniques, including novel one using a pre- trained T5 system for paraphrasing, and show that while ChatGPT achieves among the best results, it doesn't 't outperform the other algorithm by sig- nificant margin. This, coupled with the fact that using ChatGPT costs both time and money when compared to the other algorithms, brings us to a dif- ferent conclusion than what previous studies using ChatGPT for DA found, namely that it might not be worth it depending on the task. We further found that while zero-shot generation of data could give outstanding results, it was often hindered by biased datasets, which prevented efficient prompting of ChatGPT.
"https://nedi abiadfiibbiissssa assaccthhck www.booocotteeeeccc tbartt// 1htttps://medaaiises www.sffctchcccckkk.cckk com/ consort tiuntuwwwsww//
Finally, FakeNews a Kaggle dataset which has been thereafter used in many papers for fake news detection. We decided to use this dataset because
"Because FakeNews and the Irony datasets contains a lot of examples breaking CharGPT term of services, we could not attempt to refine the prompting for those datasets

---

Limitations
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023. On the Possibilities of Al-Generated Text Detection. ArX1v:2304.04336 [cs].
This paper explores the use of ChatGPT for DA. comparing it to other algorithms from the literature. Technical limitations of this paper include limited fine-tuning for some of the algorithms, including ChatGPT for which used the Web interface and therefore could not finetune the hyperparameters. While the other algorithms have been fine-tuned on some parameparameters fine-tuning was built on some supposition (such as the use of German as a pivot language for BT), which may not be the best.
Dean-Philippe Corbeil and Hadi Abdi Ghadivel. 2020. BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Iden- tification Context. ArXiv:2009. 12452 [es].
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Quanzheng Li. Dinggang Shen, Tiamming Liu, and Xiang Li. 2023. ChatAug: Leveraging ChatGPT for Text Data Augmentation. ArXiv:2302.1300 [cs].
This paper also focuses on English language and textual classification for short sentences, both as- sumptions which do not hold for many other tasks. As such, we do not guarantee the results are appli- cable to other tasks, especially for languages which are low-resource (such as Inuktitut or Swahili) or for longer texts, for which most of the algorithms used would most likely perform poorly due to lack of training data/limited context for input.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-Translation at Scale. ArXiv: 1808 008.09381 [cs].
Steven Y. Feng. Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A Survey of Data Augmentation Approaches for NLP. In Findings of the Association for Computational Linguistics; ACL-IJCNLP 2021. pages 968-988, Online. Association for Computa- tional Linguistics.
Ethics Statement
Use of pre-trained language models, and espe- cially of "very large language models", come with a plethora of ethical problems which have been well discussed in the litterature, including the en- vironmental cost (Schwartz et al., 2019; Bender et al., 2021) and environmental racism (Rillig et al., 2023), the repetition of learned biases against mi- norities (Singh, 2023), and concerns over data pri- vacy (Li et al., 2023).
T. Hayashi. S. Watanabe, Y. Zhang, T. Toda, T. Hori, R. Astudillo, and K. Takeda. 2018. Back Transsaaanttion- Style Data Augmentation for end-to-end ASR. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 426-433.
Matthew Iceland. 2023. How Good Are SOTA Fake News Detectors. ArXiv:2308.0271 02772777777 [cs].
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. AEDA: An Easier Data Augmentation Technique for Text Classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 9748-2754, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
A big concern with the most recent models is the effect it will have employement, but believe this paper mitigates this effect by showing limitation of ChatGPT, especially in the context of data annotation and dataset creation.
Sosuke Kobayashi. 2018. Contextual Augmentation: Data Augmentation by Words with Paradigmatic Re- lations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computa- tional Linguistics.
References
Shorouq M. AlAwawdeh and Gheith A. Abandah. 2021. Improving the Accuracy of Semantic Similarity Pre- diction Arabic Questions Using Data Augmen- tation and Ensemble. In 2021 IEEE Jordan Inter- national Joint Conference on Electrical Engineer- ing and Information Technology (JEFIT), pages 272- 277.
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhan- dari, and Partha Talukdar. 2019. Submodular Optimization-basee Diverse Paraphrasing and its Ef- fectiveness in Data Augmentation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3609 6609-3619, Minneapolis, Min- nesota. Association for Computational Linguistics.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major. and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Mod- els Be Too Big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pages 610-622 Virtual Event Canada. ACM.

---

Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data Augmentation using Pre-trained Trans- former Models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18 18-26, Suzhou, China. Association for Com- putational Linguistics.
Frédéric Piedboeuf and Philippe Langlais. 2022. Effec- tive Data Augmentation for Sentence Classification Using One VAE per Class. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 3454-3464 Gyeongju. Republic of Korea. International Committee on Computational Linguistics.
Taja Kuzman, Igor Mozetič, and Nikola Ljubešié. 2023. ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre 2Xiv:2303.03955: [es].
Siyuan Qiu, Binxia Xu, Jie Zhang, Yafang Wang, Xi- aoyu Shen, Gerard de Melo, Chong Long, and Xi- aolong Li. 2020. Easy An Automatic Textual Data Augmentation Platform for Classification Tasks, In Companion P Proceedings of the Web Conference 2020, pages 249-252. Association for Computing Machinery, New York, NY, USA.
Identification.
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fampu Meng, and Yangqiu Song. 2023. Multi-step Jailbreaking Privacy Attacks on ChatGPT.
ArXiv:2304.05197 [es].
Hugo Queiroz Abonizio and Sylvio Barbon Junior. 2020. Pre-trained Data Augmentation for Text Classifica- tion. In Intelligent Systems, Lecture Notes in Com- puter Science, pages 551 Cham. Springer Inter- national Publishing.
Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Proceedings of the 19th international confer- ence on Computational linguistics- +, volume 1, pages 1-7, Taipei, Taiwan. Association for Computational Linguistics.
Husam Quteineh, Spyridon Samothrakis, and Richard Sutcliffe. 2020. Textual Data Augmentation for Effi- cient Active Learning Tiny Datasets. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7400- 1400,7710, Online. Association for Computational Linguisties.
Tomas Liesting, Flavius Frasincar, and Maria Mihaela Truşcä 2021. Data augmentation in a hybrid ap- proach for aspect-based sentiment analysis. In Pro-
ceedings of the 36th Annual ACM Symposium on Applied Computing, SAC '21, pages 828-835, New York, NY, USA. Association for Computing Machin- ery.
Mehdi Regina, Maxime Meyer, and Sébastien Goutal. 2021. Text Data Augmentation: Towards better de- tection of spear- spear-phishing emails. ArXiv:2007.025 020033 [cs].
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma. Lili Wang, and Soroush Vosoughi. 2020. Data Boost: Text Data Augmentation Through Reinforce- ment Learning Guided Conditional Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9041-9 Online. Association for Computa- tional Linguisties.
Matthias C. Rillig, Marlene Agerstrand, Mohan Bi, Kenneth A. Gould, and Uli Sauerland. 2023. Risks and Benefits of Large Language Models for the En- vironment. Environmental Science & Technology, 376)):366-3466.
Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, and Angeliki Met- allinou. 2019. Controlled Text Generation for Data Augmentation in Intelligent Artificial Agents. In Pro- ceedings of the 3rd Workshop on Neural Generation and Translation, pages 90-98, Hong Kong. Associa- tion for Computational Linguistics.
Georgios Rizos, Konstantin Hemker, and Björn Schuller. 2019. Augment to Prevent: Sexrt-Text Data Augmen- tation in Deep Learning for Speech Classifica- tion. In Proceedings of the 28th ACM International Conference on Information and Knowledge Manage- ment, CIKM '19, pages 991-1000, New York, NY, USA. Association for Computing Machinery.
Vukosi Marivate and Tshephisho Sefara. 2020. Im- proving Short Text Classification Through Global Augmentation Methods. In Machine Learning and Knowledge Extraction, Lecture Notes in Computer Science, pages 385-399, Cham. Springer Interna- tional Publishing.
Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data Augmentation for Intent Clas- sification with Off- the-shelf Large Language Models. ArXiv:2204.01959 [cs].
Anders Giovanni Møller, Jacob Aarup Dalsgaard, Ari- anma Pera, and Luca Maria Aiello. 2023. Is prompt and a few samples all you need? Using GPT-4 for data augmentation in low-resource classification tasks. ArXiv:2304. 13861 [physics].
Yves Scherrer. 2020. TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AL. ArXiv:1907.10599 [cs. stat].
Eda Okur, Saurav Sahay, and Lama Nachman. 2022. Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System. ArXiv:2205.0405 040006 [cs].
Connor Shorten and Taghi M. Khoshgoftaar, 2019. A survey on Image Data Augmentation for Deep Learn- ing. Journal of Big Data, 6(1):60.

---

Sahib Singh. 2023. Is ChatGPT Biased? A Review. Publisher: OSF Preprints.
Peiye Zhuang, Alexander G. Schwing, and Oluwasanmi Koyejo. 2019. FMRI Data Augmentation Via Synthe- sis. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 1783-1787. ISSN: 1945-8452.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631 1-442. Seattle, Washington, USA. Asso- ciation for Computational Linguistics.
A Prompts used for ChatGPT
We use three types of prompts for querying Chat- GPT, two for the paraphrasing technique and one for the description technique, in order to get the best possible results while minimizing the num- ber of queries. For the paraphrasing, if the ratio was more than one, then the query is of the type: "Create X paraphrases of the following sentence:
Christoph Turban and Udo Kruschwitz. 2022. Tackling irony detection using ensemble classifiers. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1976-6984.
Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in Englishh Tweets. In Proceedings of The 12th Interna- tional Workshop on Semantic Evaluation, pages 39- 50, New Orleans, Louisiana. Association for Compu- tational Linguistics.
If the ratio is one, then we process the examples in batch, with the query: CCrette paraphrase for each of the following sentences: 1. 2. [..-]"
Finally, for the description strategy the tem- plate we use is : "Generate 10 new sentences that you haven't 't generated before for dataset of DATASET DESCRIPTION which would be CLASS DESCRIPTION". We found that speci- fying "new sentences that you haven't generated before" helped ChatGPT generate more diverse sen- tences. The given dataset descriptions are "movie review", "headline Fake/Real news classification", "Ironic tweet detection"' and "Question Classifica- tion".
Pawan Kumar Verma, Prateek Agrawal, Vishu Madaan, and Radu Prodan. 2023. MCred: multi-modal mes- sage credibility for fake news detection using BERT and CNN. Journal of Ambient Intelligence and Hu- manized Computing, 14(8): (111):10611-1062,,
Qian Wang, Fanlin Meng, and T. Breckon. 2020. Data Augmentation with VAE for Unsupervised Do- main Adaptation. ArXiv.
Jason Wei and Kai Zou. 2019. EDA: Easy Data Aug- mentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
The class values are "negative or somewhat neg- ative" or "positive or somewhat positive" for SST- 2, "Real" and "Fake" for FakeNews, "Non Ironic Tweets" and "Ironic Tweets" for Irony, "Tweets ironic by polarity contrast, where the polarity is in- verted between the literal and intended evaluation", "Tweets ironic by Situational Irony, where situa- tion fails to meet some expectation", "Tweets ironic by Other type of Irony, where the Irony is neither by Polarity Contrast or by Situational Irony", and "Tweets that are not ironic" for IronyB, and finally for TREC6 we use "Question about an abbrevia- tion" , "Question about an entity (event, animal, language, etc)", "Question concerning a descrip- tion (of something. definition, a reason. etc)", "Question about human (description of someone, an individual, etc)", "Question about location", and "Question about something numerical (weight, price, any other number)".
Xing Wu, Shangwen Lv. Liangjun Zang. Jizhong Han, and Songlin Hu. 2019. Conditional BERT Contextual Augmentation. In Computational Science ICCS 2019, Lecture Notes in Computer Science, pages 84- 95, Cham. Springer International Publishing.
Zonghai Yao and Hong Yu. 2021. Improving Formality Style Transfer with Context-Aware Rule Injection. ArXiv:2106.00210 [cs].
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Wor Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging Lcrge-scale Language Models for Text Aug- mentation. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2225- 2239, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Adams Wei Yu, David Dohan, Minh- Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining Local Convolution with Global Self-Attenion for Reading Comprehen- sion. arXiv: 09541 [cs]. ArXiv: 1804.09541.
We referred to the description given in the orig- inal papers of cach dataset to craft informative prompts.

---

B Example of generated sentences
Table 4: The tasks tackled in this study. The length of the sentences defined by the number of tokens when tokenized at white spaces.
We give in Table examples of generated sen- tences for the SST-2 dataset and the negative class, with the starting sentence "makes joke out of car chases for hour and then gives us half an hour of car chases." for the algorithms that takes sentence input (all except CGPT and ChatGPT-Desc). When fine-tuning needed, we training
Name
SST2
rrnny FakeNews
Irony B
TREC6
Iclassesi
2
2
2
6
1Duiin
6920
2683
12799
2681
5452
len. sents,
19.3
13.7
12.5
13.7
10.2
size 20.
Algo
Generated sentence
ChatGPT-DesC
T5 baseline
EDA
makes joke out of chases for hour and then gives us half an hour of car chases
AEDA
makes, a joke out of? car chases for hour and, then gives us! half an hour of: car chases turns car chases into joke for an hour and then gives us half an hour of car chases.
Avorage
BT
CBERT
makes two breakfast out of car wash for a, then gives us half an inch car wash.
Figure 1: Average metric Ratio for a dataset size of 10 and ChaiGPT-Dese, AEDA, and T5.
CGPT
'makes a joke out of car chases for an hour and then gives us half an
Figure shows the performance as we increase the ratio for the ChatGPT-Desc strategy, compared AEDA and T5, and with dataset size of 10. As we can observe, the performance plateau quickly for all algorithms. Given that Chat-GPTDese per- forms much better on SST-2 than the other datasets, we also give in Figure the results while exclud- ing SST-2. We leave for future work to investigate whether the plateauing for ChatGPT-Desc is due to lack of fine-tuning simply the limit of ChatGPT when comes to generate diverse sentences.
CBART
Stays out of car chases for an hour and then gives up an hour's worth' n a'tt-i---f--ar- chases
T5-Tapaco
The car chases for half hour is joke.
ChatGTT-Par
turns car chases into a comedic spectacle for an entire hour, fol- lowed by another 30 minutes of non-stop car action.
ChatGPT-Desc
The film was a major disappoint- ment, lacking any coherent plot engaging characters.
D Technical details
All final hyperparameters are detailed in the github, and we show a summary which hyperparameters we fine-tuned Table For fine-tuning the classi- fiers, we changed the number of epochs while leav- ing the other parameters fixed. For fine-tuning the algorithms, we played with the hyperparameters detailed in Table 7, exploring random combinations around the hyperparameters recommended in the original papers. To correctly assess the capacities of the DA methods on the different datasets, we keep the same hyperparameters for a given dataset size across all datasets. Experiments were run on NVIDIA GeForce RTX 3090 with 24G of memory.
Table 3: Examples of generated sentences for each al- gorithm for SST-2 dataset and with a dataset size of 20.
C Supplementary Results
In this section we give supplementary results to the paper. Table 4 gives some information about the datasets, and Table 5, the results of data aug- mentation on the full training set, with a ratio of generated-to cenerated-to-genuine one.

---

Table Average metric over 15 runs (accuracy and macro-F1) for the full training set and for datasets. STDs are between 0.3 and 3.3, depending the dataset.
SST2 FakeNews Irony IronyB
Trec6
Average
Baseline
87.7
73.3
65.6
42.4
81.0
70.0
EDA
87.9
73.7
65.8
43.1
81.3
70.4
AEDA
88.0
73.5
65.7
42.8
82.7
70.5
BT
88.2
73.6
66.2
42.4
81.7
70.4
CBERT
87.5
73.6
65.8
40.6
80.9
69.7
CGPT
87.8
73.2
65.2
42.8
82.1
70.2
CBART
87.7
73.9
65.9
42.7
78.6
69.8
T5
87.9
73.8
65.3
43.1
79.9
70.0
GPT3.5-Par
88.2
73.8
66.8
42.8
82.4
70.8
GPT3.5-Desc
87.4
71.9
64.1
41.1
79.8
68.9
EDA
ratios ins, del, swap, syn
AEDA
BT num. beams (en - de, de + en),
CBERT Ir, epochs, bs, ratio mask
CGPT
Ir, epochs, bs,
CBART Ir, epochs, bs, ratio mask
T5 top_p
ChatGPT
Average Motni
02
Table 7: Hyperparameters fine-tuned during experi- ments. stands for learning rate, bs for batch size, and ratio mask refers to the percentage of words that are masked.
AEDA
T5
baseline
39
20
500
1000
EDA Omls 0mls Om4s 0m6s
10
20
30
40
50
Rablo
AEDA
Om0s 0m0s Om0s Om0s
BT
0m7s 0ml Is 0m27s
0m52s
Figure 2: Average metric vs Ratio dataset size of 10 and ChatGPT-Dese, AEDA, and T5, excluding the SST-2 datasets from the average.
CBERT
Omls 0mls 0m8s
0m15s
CGPT
0m15s 0m22s 2m 1s 5m38s
T5
Om6s
0m3s 0m18 8s 0m36s
Table 8: Running time for the different DA algorithms
and SST-2, excluding the time of the classifier.
In Table 8, show the running time for aug- mentation part of the process, for SST-2 and the different dataset sizes. We do not include the train- ing time of the classifier.
Shoutout to my mom for being Ironic
hella supportive of me Luv this
Ironic
@alyssaanicolel. this Friday lit Non-Ironic they don't sing live, but they sure are hella good looking #smh Table Examples of the Irony datasets which ual examination found to be ambiguous.
Non-Ironic


Reviews:
[{"review_id": "2gmsFfsjHh", "reviewer": "Reviewer_FG9i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments. \n\nRegarding reason n.1 for Rejection, section 3 is dedicated to this effect (line 128 to 183, 148-162 for CBERT, CBART, and CGPT). Short formats only allow for general discussion, but we will review our descriptions to make them clearer.\n\nRejection reason n.2: we cited work using ChatGPT for data augmentation (line 37, line 114-127), to our knowledge the only two existing at the time of writing, and explained their conclusions. If we are missing any important references that we didn't see during our literature review, please do not hesitate to point them out.\n\nRejection reason n.3:   We attempted many ratios of generated-to-genuine, and found that for larger seed size, there were no benefits to have larger ratios. This is because DA is generally ineffective on larger dataset size except for a weak regularization effect, which can be attained with few data (compared with few shot learning where DA acts to prevent overfitting). This inefficiency has been well documented in the literature. We agree however that it can be misleading and will better explain our reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The author selected representative groups of existing DA methods and provided a reasonable amount of experiments. Also, the paper is well-structured. ", "reasons_to_reject": "1. I recommend some brief explanations for how the selected algorithm works, such as CBERT, CBART, and CGPT since they are not discussed in the related work and no detailed explanation in the algorithm part. \n2. In the related work, I suggest discussing how's the performance of the DA by GPT or other methods. \n3. I don't see the point to generate 1 augmentation for the larger training size. I would suggest keeping the same ratio between augmented data and original data. Based on the results, there's nothing much the 1 augmented example can do and all differences may come from the randomness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author compared the ability of ChatGPT in paraphrasing and data augmentation and tested the performance on 5 classification datasets with 7 other augmentation algorithms. The results showed that GPT augmentation can be limited when it comes to sensitive topics, and overall it is not the best augmentation method even though it has good performance on simpler datasets. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Some works refer to GPT augmentation as weak supervision. Some discussion there would be helpful."}, "extra_text": {}}, {"review_id": "IDtgxaSjWO", "reviewer": "Reviewer_WQnu", "summary": "", "questions": "Please refer to Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments. We agree that we may get better performances by combining DA with another form of learning. However, we feel this falls outside the scope of the paper where we aimed to objectively assess the capacity of ChatGPT for DA, as DA alone is something that is still heavily researched. \n\nRegarding «Soundness»: We do believe working on DA alone is worth the investigation, but this is of course arguable.  \n\nRegarding «Reproducibility»: We understand your concern: this is a short paper, and since we compare many techniques, it is hard to provide a self-contained description of all of them. Still, we work on public datasets, and we provide a documented code necessary to reproduce all the results we produced in the supplementary materials (code, hyperparameters, and data). We propose to add a complete description of each method in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The author's experiments may indeed provide valuable insights and inspiration to the GPT community. They highlight the potential of using GPT for text augmentation and demonstrate its impact on existing tasks. While the results may not show significant improvements in isolation, they can spur further research and encourage exploration of innovative ways to combine GPT with other techniques to enhance various natural language processing tasks.\n\n\n\n\n", "reasons_to_reject": "The author's conclusion may be one-sided. The data obtained by using GPT to enhance prediction should be integrated with existing methods to achieve better results, such as combining with comparative learning.\n\n", "questions_to_authors": "Please refer to Reasons To Reject.", "paper_topic_and_main_contributions": "The author discussed the potential of using GPT for text augmentation and concluded that while GPT can lead to some improvement in existing tasks, the overall enhancement is not significant. Moreover, the high cost of training GPT makes it less suitable for text augmentation tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X0bcDittP7", "reviewer": "Reviewer_Fjbw", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments. We agree that the contribution is limited to the ability of ChatGPT, but we would like to argue that it is an essential contribution in a time when many papers are claiming the superiority of ChatGPT in data creation/curating tasks. Showing that it possesses very real limits and that it does not fair well above others, carefully constructed, and most importantly, free algorithms is we believe an interesting contribution, at least for a short paper. \n\nWe fully agree on the need of more analysis, and we will provide more in depth analysis if the paper is accepted. \n\nFor the citations missing on line 140, we cited all relevant material in the related work section, but have no problem to duplicate the citation. Thanks for suggesting it. We will as well add an introduction to the baseline method and acronyms.\n\nAll hyperparameters (as well as the data we used) are available in the supplementary zip file, which contains the code for running the experiments. But definitely, we will add hyperparameters in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper explored ChatGPT's performance on DA, which has not been well investigated.\n2. The experiments setting is reasonable and sufficient.\n3. The paper identified limitations of both applying DA for text classification tasks, as well as limitations of applying ChatGPT for DA, which can be useful for future research.", "reasons_to_reject": "1. The paper is generally an evaluation of ChatGPT's ability, which makes the contribution limited.\n2. The analysis in this paper is rather shallow. We encourage the authors to conduct detailed and in-depth analysis on the augmentated examples, and support the conclusion with more visible and stronger evidence.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper evaluated the potential of applying ChatGPT for data augmentation, which can be essential for dataset construction and improving model performance. This paper conducted experiments on five classification tasks by comparing data augmentation by ChatGPT for paraphrasing and zero-shot data generation with seven baselines. The paper identified several limitations of applying ChatGPT for DA such as difficulty generating sentences of the desired class and the need for data reparation. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Line 140, there should be references to papers about EDA, CBERT, CBART, and CGPT.\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Grammar issue:\n1. In the Section Algorithms, there should be some brief introduction to the baseline methods and acronyms, which would make it easier to follow.\n2. Table 5 reports the hyperparameters fine-tuned during experiments, however, the value of these paprameters are not reported."}}]

Author rebuttal:

Meta-review: The authors used ChatGPT to create new data both with paraphrasing and with zero-shot generation, and compare it to seven other algorithms.
The 2 out of 3 reviewers selected "Good" for soundness and reproducibility seem to have a problem.
Decision: Accept-Findings