Low-Resource Comparative Opinion Quintuple Extraction by Data Augmentation with Prompting
Authors: Qingting Xu
Yu Hong
Fubang Zhao
Kaisong Song
Yangyang Kang
Jiaxiang Chen
Guodong Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Comparative opinion quintuple extraction
Low-resource
Data augmentation
Large language models
Sentiment analysis

Abstract: Comparative Opinion Quintuple Extraction (COQE) aims to predict comparative opinion quintuples from comparative sentences. These quintuples include subject, object, shareable aspect, comparative opinion, and preference. The existing pipeline-based COQE method fails in error propagation. In addition, the complexity and insufficient amounts of annotated data hinder the performance of COQE models. In this paper, we introduce a novel approach called low-resource comparative opinion quintuple extraction by Data Augmentation with Prompting (DAP). Firstly, we present an end-to-end model architecture better suited to the data augmentation method from triplets to quintuples and can effectively avoid error propagation. Additionally, we introduce a data-centric augmentation approach that leverages the robust generative abilities of ChatGPT and integrates transfer learning techniques. Experimental results over three datasets (Camera, Car, Ele) demonstrate that our approach yields substantial improvements and achieves state-of-the-art results. The source code and data are publicly released at: https://github.com/qtxu-nlp/COQE-DAP.

Low-Resource Comparative Opinion Quintuple Extraction by Data Augmentation with Prompting
Qingting Xu', Yu Hong"; Fubang Zhao?, Kaisong Song2?, Yangyang Kang",
Jiaxiang Chen!, Guodong Zhou!
'School of Computer Science and Technology, Soochow University, Suzhou, China
DAMO Academy, Alibaba Group, China Northeastern University, China (qtxu0801, tianxianer, xxf256 y @maal. commiil.cmm.
(fubang. z zfb, kaisong sks, yangyang kangyy @a Dalibaba-aa wwaaaabaann...... com gdzhou@suda www.ssuuaaaddudo.oooo cn
Abstract
Liu, 2006b; Wang et al., 2010; Ma et al., 2020), sentiment analysis (Schouten and Frasincar, 2015;
Comparative Opinion Quintuple Extraction (COQE) aims to predict comparative opinion quintuples from comparative sentences. These quintuples include subject, object, shareable as- pect, comparative opinion, and preference. The existing pipeline-based COQE method fails in error propagation. In addition, the complexity and insufficient amounts of annotated data hin- der the performance of COQE models. this paper, introduce a novel approach called low-resource comparative opinion quintuple ex- traction by Data Augmentation with Prompting (DAP). Firstly, we present an end-to-end model architecture better suited to the data augmen- tation method from triplets to quintuples and can effectively avoid error propagation. Addi- tionally, we introduce a data-centric augmenta- tion approach that leverages the robust genera- tive abilities of ChatGPT and integrates transfer learning techniques. Experimental results over three datasets (Camera, Car, Flle demonstrate that our approach yields substantial improve- ments and achieves state-of-the-art results. The source code and data are publicly released at: http://github..c https//gihhub..om//uuuuut cow.uuunppouopooCC----.
Zhang et al., 2022; Aftab et al., 2022), and cus- tomer satisfaction estimation (Ando et al., 2022).
Existing pipeline- based method (Liu et al., 2021) suffers from error propagation. The heavy reliance on extensive annotated data poses a bottleneck the training process. To address the aforementioned issues, we propose data augmentation method with prompting for low-resource COQE. Firstly, we propose BERT-based (Devlin et al., 2018) end-to- end deep learning model our backbone avoid error propagation. Although existing LLMs such as ChatGPT possess rich linguistic knowledge and impressive generative capabilities, they encounter difficulties in generating satisfactory quintuple ex- amples due to the inherent complexity of COQE, In this paper, we propose to develop a lightweight data augmentation, where the triple examples are required to be generated for augmentation, instead of the unabridged quintuple examples. It is rela- tively easy for ChatGPT produce some qualified triple examples rather than quintuple. Addition- ally, we leverage these generated triple examples to warm up the end-to-end extraction model before training it over the benchmark quintuple dataset. To summarize, the main contributions of our work follows:
1 Introduction
COQE is an essential subfield of Natural Language Processing (NLP). Its primary objective is to ex- tract five specific components from comparative sentences, namely: subject, object, shareable as- pect, comparative opinion, and preference, as de- fined in (Liu et al., 2021). For example, in the sentence "Like the viewfinder, the Nikon D80 has the same sensor as the D200.", "Nikon D80" and "D200" are respectively the subject and object enti- ties, the aspect term is "sensor", the opinion word "same", and the comparative preference is "Equal". COQE plays crucial role in various applications, such as comparative opinion mining (Jindal and
We introduce an end-to-end model framework to suit data augmentation methods better and avoid error propagation. Additionally, we propose a two- stage data augmentation approach for low-resource COQE, leveraging the generative capabilities of ChatGPT and the transfer learning method.
Experimental results demonstrate that our ap- proach yields substantial improvements compared to the baseline and the current oftteoff-hheart model. resulting in new highest performance on three COQE datasets. Furthermore, we conduct further analyses and supplementary experiments to verify the effectiveness our approach.
The authory contributed equally to this work and there- fore are considered as co-corresponding authors.

---

(rompt]: Subject and eject refer to the subject and object ontilies heing compured, aspect term term of two
[Prompt I] triplet
Cameraa COQE
denotes the comparraiive compared entilies. For the given aspect term the camera domain. please generate three puirs of comppred entities.
[Prompt subSkobij
New Datasct
Camera- COQE
[Prompt 3] for subskas
Step1
Step4
Steps
Stup2
Stepa
Obtain all aspect torms
Generate three pains of tuplets
Genernte sentences
Data filtering and processing
Transfer leammin
Generate triplet data
Transfer learning
Figure I: Overview of our approach. ChatGPT logo indicates steps requiring ChatGPT implementation.
2 Approach
the extraction probabilities for the quadruple using formula (4),
2.1 Task Definition
COQE aims to predict quintuples {sub, obj, ppe from the given sentence X {i1, x2 .... 2n) (n denotes the number of tokens). sub and obj represent entities that serve the sub- ject and object in a comparison relation, respec- tively. asp refers the shareable aspect term of two compared entities. Additionally, op denotes comments or opinions expressed regarding the as- pect term of subject and object entities. There are four categories of preference (pre) to be considered, including Better, Equal, Different and Worse.
Pe = softmar +b)
(3)
p softmar(V; tawh:WanhhWWa W,,H)
(4)
where We. We. Wh. b and V are all trainable pa- rameters. of is the i-th embedding output by the final decoder layer.
We optimize the combined objective function during training. Leotal comprises classification and extraction loss using the cross-entry loss function.
N
K=8 log
Liotal
log
P
(5)
2.2 End-to-end Model
We utilize BERT (Devlin et al., 2018) as the sen- tence encoder. Using a BPE tokenizer (Sennrich et al., 2016), we obtain context-aware representa- tions for each token in the input sentence X.
where N denotes the number of initialized Q, K denotes the number of loss functions computations required for a quadruple, which is equal 8.
2.3 Data Augmentation for Transfer
H = BERT(X)
(1)
Currently, there is lack of datasets that are specif- ically annotated for subject, object and aspect comparative sentences. In this paper, we intro- duce a data-centric method to leverage the rich linguistic knowledge within ChatGPT and further enhance COQE performance. ChatGPT generates dataset containing triplets (sub, obj, asp). In this section, we take the Camera dataset as an exam- ple. multi-stage approach is needed generate proper sentences for automatic annotation the dataset. Building the triplet dataset involves four steps, as depicted in Figure 1.
We employ non- an-auutoregressvve (Guo et al., 2019) decoder to generate the quintuples, which remove the dependence on previous target tokens from the input of the decoder. Specifically, we randomly initialize a vector representation Q to represent quintuple. In each decoder layer, we update the representation of Q using the formula (2). In this paper, utilize l-layer transformer decoder for non-autoregressive generation, and in our experiments, we set / 3.
Q Decoder Q1
(2)
Obtaining Aspect Terms Firstly, we count the unique aspect terms separately from each dataset. The number of distinct aspect terms for threc datasets is provided in Table 2.
Given the output of the final decoder layer, employ a classifier and four pointer networks to extract quintuples. Each pointer network is respon- sible for identifying the start and end position of one element within quadruple. We calculate the classification probabilities using formula (3) and
Generating Triplets We generate triplets based on the aspect terms obtained from step 1. Specifi- cally, we invoke the ChatGPT API and design an appropriate prompt to generate three triplets.

---

Kind
Prompt
Please generate new comparative sentence that compares describes subject and object based the given aspect term.
(sub, obj, asp)
Table 1: The prompt for generating comparative sentences.
Camera
Car
Ele
Models
Camera
Car
Ele
Dataset
#Sent
#Asp
#Sent
#Asp
#Sent
#Asp
MS
3,46
5.19
4.07
Train
2,114
510
2,269
500
2,304
421
MSCRF
4.88
8.65
4.71
Dev
529
160
568
163
576
141
Test
661
190
710
205
720
170
MSLSTM
9.05
10.28
14.90
BSBERT
13.36
29.75
30.73
Table 2: Statistics in the training, dev and test sets.
E2E
19.21
35.02
36.66
"Sent" and "Asp" indicate the total number of sentences and unique aspect terms for each dataset, respectively.
DAP
21.06
36.13
39.24
Table Fl-scores for various COQE methods using exact strategy. The best scores are in bold.
Generating Sentences Based on the statistics from the Camera dataset, 221 sentences contain triplets sub, obj, asp). In contrast, there are 202 sentences with only binary combinations such as (sub, obj), }. 139 sentences with (sub, asp), and 52 sentences with {obj, asp). To ensure diversity in priori-
ditional annotations data points regarding com- parative opinions and preferences. The statistics the three datasets are shown in Table 2.
Evaluation Metrics We evaluate all models through Precision (P), Recall (R), and Fl metries. Additionally, employ three matching strategies to evaluate prediction performance: exact-match (Ex), proportional match (Pr) and binary-math (Bi) evaluation. The details of the three matching strate- gies are as follows:
the generated sentences by ChatGPT, tize the first three scenarios. We design specific prompts for each scenario. For example, in the first scenario, we establish the prompt as shown in Table 1.
Data Filtering and Processing Despite the pro- vided constraints, ChatGPT may still generate some samples that do not meet the specifications. Therefore, before automatic labeling, matching the generated sentences with the given triplets is essen- tial. Only the sentences that successfully match the triplets should be labeled automatically.
0
3 (pr + gi)
Ex
(6)
otherwise
(
We train the backbone model using newly con- structed triplet data to obtain feature representa- tions. Subsequently, we employ transfer learning techniques to fine-tune the gold quintuples based the obtained representations.
Pr
Σ len(gipp))) 2, lenlg.)
(7)
otherwise
o (
Bi
(8)
1 otherwise
3 Experiments
where, 9: and p; denote the i-th element in the gold and predicted quintuple result, respectively. The index ranges from to 5. We report the average performance based three runs, utilizing shuffled random seeds for each run.
3.1 Datasets and Evaluation Metrics
Datasets We evaluate our method on three datasets. Camera an English corpus (Liu et al., 2021). builds upon the prior work of Kessler et al.(Kessler and Kuhn, 2014) by providing additional annota- tion for comparative sentences with comparative opinions and preferences. Besides, Liu al. (Liu al., 2021) construct two Chinese datasets specifi- cally designed for comparative opinion quintuple extraction. They extend the COAE (Songbo Tan, 2013) dataset by building upon it and providing ad-
3.2 Compared Models
We compare with the following models:
MSsvM+CRF first propose comparative sentence identification. They utilize SVM (Cortes and Vap- nik, 1995) for identifying comparative sentences and CRF (Lafferty et al., 2001) for extracting com- parative elements (Jindal and Liu, 2006a).

---

SOU TAR
Metric
Model
P
F1
Dataset
SST
MSBERT
23:44
Camera
8,5F-03
7.8
DAP
33.19
27.10
29.83 (16.99)
Car
3,4E-03
2.5
MSarar
30.49
Fle Car
Pr
Ele
1.2E-02
47.84
42.99 (112.50)
31,64
DAP
50.50
41.23
45,39 (113.75)
Table 5: Results of two significance tests.
MSBERT
24.42
Ex
DAP
34.31
28.91
31.38 (76.90)
MSnEer
Car Fll
Pr
DAP
52.70
44.42
48,2 (136.00)
superior cross-domain experimental results surpass- ing previous COQE approaches, owing to our ap- proach's ' generalization of data transfer.
MSBERT
32.85
DAP
55.97
47.18
51.20 (1 18.35)
Table 4: Results of our proposed model in the cross- domain setting, The mark =_" indicates the results were not presented in (Liu et al., 2021)3s work.
3.5 Significance Test
We perform a statistical significance test (abbr., SST) to validate the reliability of our method. The sampling-based P-value (Johnson, 1999) used as the metric for measuring significance levels. On the other hand, to provide a comprehensive insight into the significance, we conduct the practical sig- nificance test (abbr., PST) (Zhu et al., 2020), which is more reliable than SST. Cohen's D-value is used as the metric of PST. It is noteworthy that, in SST, the reported P-value below 0,05 (i.c., 5.0E-02) indi- cates significant improvement, otherwise insignif- icant (Dror et al.. 2018). Similarly, in PST, the reported Cohen's D-value exceeds 1 indicates a significant improvement.
MSCRF employ CRF-based model for com- parative sentence identification and comparative element extraction (Wang et al., 2015).
MSLSTM introduce multi-stage framework uti- lizing LSTM as the text encoder. Firstly, they iden- tify comparative sentences and extract comparative elements. Subsequently, they combine and filter these elements. Finally, they classify valid quadru- ples into four categories Liu et al. (2021).
MSBERT is modified version of MS For MSBERT, Liu et al. (2021) choose BERT the model'ss s text encoder.
3.3 Main Results
The results of the SST and PST are presented Table It can be found that the P-values of SST are lower than the threshold, while Cohen's D-value of PST is higher than the threshold. This demonstrates that DAP yields significant improvements.
We show the performance over three test sets in Table 3. It can be observed that DAP yields sub- stantial improvements on three datasets compared to all the current SoTA or our backbone (E2E). In particular, even E2E is superior to the performance of BSBERT- This also proves that the end-to-end model can effectively avoid error propagation. Be- sides, compared the current SOTA results, DAP leads to F1 score improvements of 7.70%, 6.38%, and 8.51% % on the Camera, Car, and Ele datasets, respectively. This shows that the performance of COQE can be effectively improved by introducing external knowledge contained in LLM.
4 Related Work
4.1 Comparative Sentence Analysis
Comparison-oriented information extraction has at- tracted considerable research interest. Jindal and Liu (2006a) first introduce the concept of compara- tive sentences and implement comparative sentence discrimination based on rules and SVM (Cortes and Vapnik, 1995). Park and Blake (2012) explore an extensive set of syntactic and semantic features, and employ three different classifiers to identify comparative sentences. The recent studies con- centrate on fine-grained component analysis and parsing upon comparative sentences (Kessler and Kuhn, 2013; Arora al., 2017; Ma al., 2020). In particular, Liu al. (2021) propose a novel task called comparative opinion quintuple extraction, which aims to extract quintuples from the given comparative sentences.
3.4
Cross-Domain Experiments
We follow Liu et al. (2021) to evaluate the cross- domain generalization ability of our method. We conduct cross-domain experiments on two Chinese datasets, where cross-domain refers to using train- ing and validation sets from the souree domain (SOU) and test from the target domain (TAR). In Table 4, "Ele + Car" denotes electronic domain serves as the SOU, car domain the TAR. can be observed that our method DAP yields

---

4.2 Data Augmentation
of the International Conference on Acoustics, Speech and Signal Processing, pages 6912-6916. IEEE.
Data augmentation is a technique that expands and diversifies training datasets by applying var- ious transformation or modification approaches to the existing data. Fadaee et al. (2017) use back- translation data augmentation method for gen- erating synthetic parallel sentences, thereby hancing the performance of low-resource neural machine translation systems. Wei and Zou (2019) use synonym replacement, random insertion, ran- dom deletion and synonym swapping to increase the diversity of training data. The approaches con- tribute to enhancing the robustness of text classifier. Chen and Qian (2020) develop a prototype gen- erator for data augmentation, where internal and external prototypes are adopted.
Atsushi Ando, Yumiko Murata, Ryo Masumura, Satoshi Suzuki, Naoki Makishima, Takafumi Moriya, Takanori Ashihara, and Hiroshi Sato. 2022. Cus- tomer satis faciifatiinn estimation using unsupervised rep- resentation learning with multi-format prediction loss. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pages 8497- 8501.
Jatin Arora, Sumit Agrawal, Pawan Goyal, and Sayan Pathak. 2017. Extracting entities of interest from comparative product reviews. In Proceedings of the 2017 ACM Conference Information and Knowl- edge Management, pages 1975-1978.
Zhuang Chen and Tieyun Qian. 2020. Enhancing aspect term extraction with soft prototypes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2107-2117
5 Conclusion and Future Work
To avoid error propagation, design end-to- end model. Additionally, we propose data-centric augmentation upproach using the powerful gener- ative capability of ChatGPT. The performance on three datasets achieves SoTA. Future work will fo- cus on integrating existing annotated triplet data with automatically generated domain-specific data,
Corinna Cortes and Vladimir Vapnik. 1995. Support- vector networks. Machine learning, 20:273-297.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04800
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language process- ing. Proceedings of the 56th annual meeting of the association for computational linguistics, pages 1383-1392.
Limitations
Despite achieving a new state-of-the-art perfor- mance, our model still has several limitations. One obvious limitation of the method is that the origi- nal three data sets contain certain proportion of multiple comparison sentences, making predictions more difficult. In this paper, we only improve per- formance by introducing external knowledge and extracting the quintuple from the difficult to the easy. Future work can concentrate on tackling the COQE problem specifically from the perspective of multiple comparative sentences
Marzich Fadace, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, pages 567-573.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019, Non von-autoregressive neural ma- chine translation with enhanced decoder input. In Proceedings of the AAAI conference on artificial in- telligence, volume 33, pages 3723-3730.
A Acknowledgements
Nitin Jindal and Bing Liu. 2006a. Identifying compar- ative sentences in text documents. In Proceedings of the 29th annual international ACM SIGIR confer- ence on Research and development in information retrieval-SIGIR '66, pages 244-251. ACM Press.
The research is supported by National Key R&D National
Program of China (2020YFB1313)) Natural Science Foundation of China (62376182, 62076174, 62 62106039).
Nitin Jindal and Bing Liu. 2006b. Mining conpara- tive sentences and relations. In Proceeding of 2 National Conference Artifitial Intelligence, 2006. pages 133 1-366.
References
Arya Aftab, Alireza Morsali, Shahrokh Ghaem- maghami, and Benoit Champagne. 2022. Light- sernet: lightweight fully convolutional neural net- work for speech emotion recognition. In Proceedings
Douglas H Johnson. 1999. The insignificance of sta- tistical significance testing. The journal of wildlife management, pages 763-772.

---

Wiltrud Kessler and Jonas Kuhn. 2013. Detection of product comparisons-oor how far does an out-of-the-boe of-tthe-oxx semantic role labeling system take you? In Proceed- ings of the 2013 conference on empirical methods in natural lunguage processing, pages 1892-1897.
Conference on Natural Language Processing, pages
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022. A survey aspect-based senti- ment analysis: tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering. pages 1-20.
Wiltrud Kessler and Jonas Kuhn. 2014, corpus of
comparisons in product reviews. In Proceedings of the Ninth International Conference on Language Re- sources and Evaluation, pages 2242-2243 2248.
Haotian Zhu, Denise Mak, Jesse Gioannini. and Fei Xia. 2020. Nipstattest: A toolkit comparing nlp system performance. In Proceedings of the 1st Con- ference the Asia- Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing: System Demonstrations, pages 40-46.
John Lafferty. Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data.
Ziheng Liu, Rui Xia, and Jianfei Yu. 2021. Comparative opinion quintuple extraction from product reviews, In Proceedings the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 3955-3965.
Nianzu Ma, Sahisnu Mazumder, Hao Wang, and Bing Liu. 2020. Entity-aware dependency- based deep graph attention network for comparative preference classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5782-5788.
Dae Hoon Park and Catherine Blake. 2012. Identifying comparative claim sentences in full-text scientific articles. In Proceedings the workshop on detecting structure in scholarly discourse, pages 1-9.
Kim Schouten and Flavius Frasincar: 2015. Survey on aspect-level sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, pages 813-
830.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, pages 1715-1725, Association for Compu- tational Linguistics.
Kang Liu. Songbo Tan. 2013. Overview of chinese opinion analysis evaluation 2013.
Suge Wang, Hongxia Li, and Xiaolei Song. 2010. Auto- matic semantic role labeling for chinese comparative sentences based on hybrid patterns. 2010 interna- tional conference on artificial intelligence and com- putational intelligence, volume 1, pages 378-382. IEEE.
Wei Wang, Tiejun Zhao, Guodong Xin, Yuyong Xu. 2015. Extraction of comparative elements using conditional random fields. Acta Automatica Sinica, 41(8):1355-139)
Jason Wei and Kai Zou. 2019. Eda: Easy data aug- mentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint


Reviews:
[{"review_id": "C3ydOyEWnZ", "reviewer": "Reviewer_wHff", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and promise to improve this paper in the revised version.\n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The Data Augmentation with prompting (DAP) technique is not clearly explained how it compares with the benchmark and outperforms subsequently. \n\n\n**Response to #1:** I sincerely sorry for any inconvenience caused by our failure to adequately clarify the Data Augmentation with prompting (DAP) technique in the paper. In fact, we introduce  two main innovations in this paper: 1) Compared with existing pipeline methods which fail in the issue of error propagation, we propose an end-to-end extraction framework (i.e., **E2E**). 2) Faced with the tricky low-resource problem in COQE, we introduce a data-centric augmentation approach that leverages the robust generative abilities of ChatGPT and integrates transfer learning techniques (i.e., **DAP**). COQE is a new task which was proposed recently [1]. As a result, it hasn’t yet been sufficiently studied. To our best knowledge, the only methodology published prior to this submission comes from Liu et al. (2021)’s work [1]. Therefore, we concentrate on the comparison among our E2E model, our enhanced version DAP and Liu et al. (2021)’s Pipeline model [1]. We show the experimental result in Appendix behind these replies (Table 1). It can be concluded that the data augmentation method (DAP) proposed in this paper not only demonstrates improvements over the baseline model (E2E) but also exhibits significant advancements when compared to the current state-of-the-art (**Pipeline**) methods.\n\n\n**Comment #2:** The organization of the paper sections could have been a little better.\n\n\n**Response to #2:** We acknowledge that the organization of the paper could be improved for better clarity and coherence. Based on your suggestions, we intend to restructure our paper sections. For instance, we propose the incorporation of a new Section 2.2, dedicated to offering a comprehensive overview of the overall architecture of DAP. We appreciate your suggestions and  promise to make the necessary revisions in the final version.\n\n----------Appendix----------\n\n\n\n**Table 1. F1-scores for various COQE methods.**\n\n\n| Models           | Camera | Car   | Ele   |\n| ---------------- | ------ | ----- | ----- |\n| Pipeline         | 13.36  | 29.75 | 30.73 |\n| E2E  | 12.56  |35.02 | 36.66 |\n| DAP              | 14.96  | 36.13 | 39.24 |\n\n\nWe sincerely appreciate your thoughtful insights and all the constructive suggestions. We will strictly follow your suggestions to improve the paper.\n\n\n\n----------Reference----------\n\n\n[1] Ziheng Liu, Rui Xia, and Jianfei Yu. Comparative opinion quintuple extraction from product reviews. In EMNLP 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a novel concept to predict the triplets - subject, object and aspect. It depicts the comparisons between ChatGPT and fine-tuned BERT model while obtaining the triplets\n\n", "reasons_to_reject": "The Data Augmentation with prompting (DAP) technique is not clearly explained how it compares with the benchmark and outperforms subsequently.\n\nThe organization of the paper sections could have been a little better.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about Comparative Opinion Quintuple Extraction (COQE) where a two-stage data augmentation strategy is proposed. Furthermore, experiments were conducted to verify the effectiveness of the method on 3 available COQE datasets", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mlBSIdznvA", "reviewer": "Reviewer_Azi6", "summary": "", "questions": "A. Line 128 - Where is Wt used?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments. \n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The significance of the data augmentation is shown via experiments but I am not convinced by the method.\n\n\n**Response to #1:** We realize your concern and will respond to it from two aspects, including our motivation and the corresponding solution.\n\n\n* Motivation: There is a limited annotated resource for COQE task, which leads to a problem of data sparsity. Data augmentation is potentially effective for addressing the issue. However, the existing large language models (LLM) like ChatGPT fail to generate satisfactory quintuple examples for augmentation due to the high complicacy of COQE. As a result, the conventional data augmentation approaches cannot be applied directly for COQE.\n\n\n* Solution: To tackle the issues, we propose to develop a lightweight data augmentation, where the triple examples are required to be generated for augmentation, instead of the unabridged quintuple examples. It is relatively easy for LLM to produce some qualified triple examples rather than quintuple. Our approach uses the generated triple examples to warm up the end-to-end extraction model before training it over the benchmark quintuple dataset. In Section 2.3, we detail the instruction and prompt that we used for generating triple examples. In Table 2 (lines 200-201) of our paper, we compare the SoTA model (Pipeline) with our two models, including the end-to-end backbone (referred by E2E) and the enhanced version by augmentation (referred by DAP). It can be observed that substantial improvements are obtained by DAP, compared to E2E. This demonstrates the positive effectiveness of triple-example based data augmentation. \n\n\nIn addition, we realize that your concern may attribute to the reliability of P-value based statistical significance test. Therefore, in order to provide a comprehensive insight into the significance, we conduct the practical significance test (PST) [2], which is more reliable than statistical significance test (SST) [1]. We list the test results of both PST and SST as below (namely Table 1 and Table 2), and hope they facilitate the verification of significance. It is noteworthy that, in SST, the reported P-value corresponds to a significant improvement only if it is smaller than 0.05, while in PST, the reported Cohen’s D-value corresponds to a significant improvement only if it is a much higher value than 1. \n\n\n**Comment #2:** Moreover, there is only one baseline that is considered. I would like to see more baselines being compared.\n\n\n**Response to #2:** We recognize your perspective and consider it to be the justification for including just one state-of-the-art approach for comparison. In fact, COQE is a new task which was proposed recently [1]. As a result, it hasn’t yet been sufficiently studied. To our best knowledge, the only methodology published prior to this submission originates from the work of Liu et al. (2021) [1]. Therefore, we concentrate on the comparison among our E2E (end-to-end) model, our enhanced version by data augmentation and Liu et al. (2021)’s Pipeline model [1]. We acknowledge that the COQE performance of some other neural models (e.g., LSTM and CRF) has been reported, though they are reported by Liu et al. (2021) themselves with the role of possible baselines. Due to the significantly worse performance of these conventional models compared to Liu et al. (2021)’s Pipeline model, we chose to omit them in our study. We feel sorry to simplify the comparison process, and provide the unabridged comparison results in Appendix behind these replies (Table 3). We also promise that we will expand the comparison experiments in the extended paper, after investigating the related work that may emerge recently.\n\n\n**Comment #3:** Can you also add the generated data to the \"pipeline\" and report its performance to see the improvement?\n\n\n**Response to #3:** We understand your concern and assure you that we did consider the possibility of adding such data upon the Pipeline model. However, our data augmentation strategy cannot be directly applied upon it. It is because of the following reasons.\n\n* Reason 1: First, we generate triple examples instead of quintuples, as explained in the response to Comment #1. Though, such triple examples aren’t friendly to the Pipeline model. In other words, the input of the Pipeline model is fixed, which cannot be dynamically changed. \n\n* Reason 2: Second, we did work on the enhancement of the pipeline model by adding the generated quintuples. However, the model performance decreases by 2.5% in average due to a large amount of noises and redundant information in the generated quintuples. We will add more discussions about this in the extended version of our paper and explore more extensible data augmentation strategy in the future.\n\n\n\n----------Response to Questions ----------\n\n\n**Question #1:** Line 128 - Where is Wt used?\n\n\n**Response to #1:** In line 128, \"Wt\" should be corrected as \"Wc\". We will correct this typo.\n\n\n\n----------Response to Typos Grammar Style And Presentation Improvements ----------\n\n\n**Suggestion #1:** Line 206-209 We train the backbone model using newly constructed data.\n\n\n**Response to #1:** Thanks you for reminding us about the presentation. What we intend to present in the sentence is about “We train the backbone model using the newly constructed triple data”. We will modify this description, so as to make is easy to follow.\n\n\n\n----------Appendix----------\n\n\n**Table 1. The results of SST (P-value is considered).**\n\n\n|            | Camera | Car    | Ele   |\n| ---------- | ------ | ------ | ----- |\n| E2E VS DAP | 0.0085 | 0.0034 | 0.012 |\n\n\n**Table 2. The results of PST (Cohen's D-value is considered).**\n\n\n|            | Camera | Car  | Ele  |\n| ---------- | ------ | ---- | ---- |\n| E2E VS DAP | 4.32   | 1.31 | 2.62 |\n\n\n**Table 3. F1-scores for various COQE methods.**\n\n\n| Models           | Camera | Car   | Ele   |\n| ---------------- | ------ | ----- | ----- |\n| Multi-Stage-CRF  | 3.46   | 5.19  | 4.07  |\n| Joint-CRF        | 4.88   | 8.65  | 4.71  |\n| Multi-Stage-LSTM | 9.05   | 10.28 | 14.90 |\n| Pipeline         | 13.36  | 29.75 | 30.73 |\n| DAP              | 14.96  | 36.13 | 39.24 |\n\n\n\n----------Reference----------\n\n\n[1] Douglas H Johnson. The insignificance of statistical significance testing. In the journal of wildlife management 1999.\n\n[2] Haotian Zhu, Denise Mak, Jesse Gioannini, Fei Xia. NLPStatTest: A Toolkit for Comparing NLP System Performance. In ACL 2020.\n\n[3] Rotem Dror, Gili Baumer, Segev Shlomov, and RoiReichart. The hitchhiker’s guide to testing statistical significance in natural language processing. In ACL 2018. \n\n[4] Ziheng Liu, Rui Xia, and Jianfei Yu. Comparative opinion quintuple extraction from product reviews. In EMNLP 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Easy to understand\n- Positive results", "reasons_to_reject": "This paper presents two things:\n1. A model that does Comparitive Opinion Quintuple Extraction\n2. Data augmentation pipeline using ChatGPT.\n\nA. The significance of the data augmentation is shown via experiments but I am not convinced by the method. \nB. Moreover, there is only one baseline that is considered. I would like to see more baselines being compared.\nC. Can you also add the generated data to the \"pipeline\" and report its performance to see the improvement?\n", "questions_to_authors": "A. Line 128 - Where is Wt used?\n", "paper_topic_and_main_contributions": "This paper presents a data augmentation method using ChatGPT for comparitive opinion quintuple extraction. They also introduce an end-to-end model that benefits from such data augmentation. Although experiments show that generated data is clearly beneficial, the proposed model is not much convincing.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 206-209 We train the backbone model using newly constructed data.."}, "extra_text": {}}, {"review_id": "fzgGXeFJ1y", "reviewer": "Reviewer_ttGM", "summary": "", "questions": "1) How about the performance of ChatGPT? ChatGPT is not good at COQE, so why use it to generate data?\n2) How about the performance of other data augmentation metheds and COQE methods?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and promise to improve this paper in the final version. \n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The existing baselines for Comparative Opinion Quintuple Extraction and data augmentation are missing in the experiments.\n\n\n**Response to #1:** We realize your concern and regard it as the reason why there is only one SoTA approach involved for comparison. In fact, COQE is a new task which was proposed recently [1]. As a result, it hasn’t yet been sufficiently studied. To our best knowledge, the methodology that was published before this submission only comes from Liu et al. (2021)’s work [1]. Therefore, we concentrate on the comparison among our E2E (end-to-end) model, our enhanced version by data augmentation and Liu et al. (2021)’s Pipeline model [1]. We acknowledge that the COQE performance of some other neural models (e.g., LSTM and CRF) has been reported, though they are reported by Liu et al. (2021) themselves with the role of possible baselines. Due to the significantly worse performance of these conventional models compared to Liu et al. (2021)’s Pipeline model, we chose to omit them in our study. We feel sorry to simplify the comparison process, and provide the unabridged comparison results in Appendix behind these replies (Table 1). We also promise that we will expand the comparison experiments in the extended paper, after investigating the related work that may emerge recently.\n\n\n**Comment #2:** The motivation of this paper is not clear. \n\n\n**Response to #2:** We realize your concern and will respond to it from two aspects as below.\n\n\n* Error propagation: The pipeline model works in the way of progressively detecting and extracting the elements of each quintuple, step by step. This easily causes the error propagation problem. To address the issue, we propose to fulfill the quintuple extraction within an end-to-end framework, and develop the corresponding model.\n\n\n* Data sparsity: There is a limited annotated resource for COQE task, which leads to a problem of data sparsity. Data augmentation is potentially effective for addressing the issue. However, the existing large language models (LLM) like ChatGPT fail to generate satisfactory quintuple examples for augmentation due to the high complicacy of COQE. As a result, the conventional data augmentation approaches cannot be applied directly for COQE. To address the issues, we propose to develop a lightweight data augmentation, where the triple examples are required to be generated for augmentation, instead of the unabridged quintuple examples. It is relatively easy for LLM to produce some qualified triple examples rather than quintuple. Our approach uses the generated triple examples to warm up the end-to-end extraction model before training it over the benchmark quintuple dataset. \n\n\n**Comment #3:** Why do they use ChatGPT to generate augmentation data? How about other data augmentation methods?\n\n\n**Response to #3:** Data sparsity continues to be a significant concern, mainly due to the limited availability of annotated resources for the COQE task. ChatGPT has strong generative ability and easily produce high-quality data without the requirement of heavy manual efforts. The data generated by ChatGPT can introduce new knowledge and enhance the diversity of the dataset. Building on this, we utilize the data generated by ChatGPT in conjunction with transfer learning to enhance the performance of our model.\n\nCommon data augmentation techniques, such as random insertion, deletion, and replacement, often introduce changes to the original semantics of sentences. Diverging from conventional data augmentation techniques, we adopt the approach of Karimi et al. [2] by randomly inserting a specified proportion of punctuation marks into sentences (referred by AEDA). This method maintains the original word order while altering the positions of words within the sentence, effectively enhancing the model's robustness. We show the performance of employing the AEDA method on the COQE task in Appendix behind these replies (Table 2). Compared to the popular AEDA method, our method achieves 1.1% improvements in average on Car and Ele datasets. \n\n\n**Comment #4:** Ablation studies are missing.\n\n\n**Response to #4:** Actually, we have conducted the ablation study. We propose a basic end-to-end model E2E and a data-augmentation based full model DAP. The results have been displayed in Table 2 (see lines 200-201 in our paper). These results are equivalent to the ablation experiments. We will highlight this part in our final version.\n\n\n\n----------Response to Questions----------\n\n\n**Question #1:** How about the performance of ChatGPT? \n\n\n**Response to #1:** We conduct an analysis of directly employing ChatGPT for the COQE task, the results are illustrated in Figure 2 (lines 270-271). As you can see from Figure 2, even given well-designed prompt and instruction, implementing COQE tasks directly with ChatGPT has much lower performance than our DAP approach.\n\n\n**Question #2:** ChatGPT is not good at COQE, so why use it to generate data?\n\n\n**Response to #2:** Although ChatGPT has demonstrated success in other extraction tasks, its performance in COQE is notably inadequate. We attribute it to the fact that COQE consists of the product-domain data which are not fully used by ChatGPT. Additionally, COQE contains comparative opinions and proves to be more challenging than than aspect-based sentiment analysis. Generating quintuples directly poses a challenge, while obtaining triples is comparatively easier. This is the first work that discusses the methodology of utilizing ChatGPT in COQE, which paves the way for further research in NLP community. \n\n**Question #3:** How about the performance of other data augmentation methods and COQE methods?\n\n\n**Response to #3:** We conduct experiments comparing our approach with other data augmentation method AEDA (citation [2]), and the results are presented in Table 2. From the results, we can find our DAP outperforms AEDA because of COQE-specific data augmentation strategy.\n\n\n\n----------Appendix----------\n\n\n**Table 1. F1-scores for various COQE methods.**\n\n\n| Models           | Camera | Car   | Ele   |\n| ---------------- | ------ | ----- | ----- |\n| Multi-Stage-CRF  | 3.46   | 5.19  | 4.07  |\n| Joint-CRF        | 4.88   | 8.65  | 4.71  |\n| Multi-Stage-LSTM | 9.05   | 10.28 | 14.90 |\n| Pipeline         | 13.36  | 29.75 | 30.73 |\n| DAP              | 14.96  | 36.13 | 39.24 |\n\n\n**Table 2. Comparison results with other data augmentation methods.**\n\n\n| Model    | Car   | Ele   |\n| -------- | ----- | ----- |\n| Pipeline | 29.75 | 30.73 |\n| AEDA     | 35.02 | 38.13 |\n| DAP      | 36.13 | 39.24 |\n\n\nThanks again for all your constructive suggestions and we promise to solve all these problems in the final version.\n\n\n\n----------Reference----------\n\n\n[1] Ziheng Liu, Rui Xia, and Jianfei Yu. 2021. Comparative opinion quintuple extraction from product reviews. In EMNLP.\n\n[2] Akbar Karimi, Leonardo Rossi, Andrea Prati. 2021. AEDA: An Easier Data Augmentation Technique for Text Classification. In EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors propose an LLMs-based data augmentation strategy for low-resource comparative opinion quintuple extraction.\n2) This paper is well organized and easy to follow.", "reasons_to_reject": "1) The existing baselines for Comparative Opinion Quintuple Extraction and data Augmentation are missing in the experiments.\n2) The motivation of this paper is not clear. Why do they use ChatGPT to generate augmentation data? How about other data augmentation metheds?\n3) Ablation studies are missing.", "questions_to_authors": "1) How about the performance of ChatGPT? ChatGPT is not good at COQE, so why use it to generate data?\n2) How about the performance of other data augmentation metheds and COQE methods?", "paper_topic_and_main_contributions": "This paper proposes an LLMs-based model for low-resource comparative opinion quintuple extraction. The author design a two-stage data augmentation strategy to improve the performance of COQE. Particularly, they use ChatGPT to generate triplet datasets and then use transfer learning. The experiment on three benckmark datasets shows their model outperforms the baseline.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pMI6V87XjZ", "reviewer": "Reviewer_1sq6", "summary": "", "questions": "Question A: The author did not provide a definition of what K represents in formula (5).\n\nQuestion B: On page 3, line 202, I am not clear about the use of transfer learning, I think it is to conduct pre-training on the dataset generated by ChatGPT and then fine-tune on the COQE dataset. Please correct me if I have misunderstood.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments and questions. \n\n\n\n----------Response to Questions ----------\n\n\n**Question #1:** The author did not provide a definition of what K represents in formula (5).\n\n\n**Response to #1:** Thank you for your suggestions. In our work, we employ a classifier and four pointer networks to extract quintuples. Each pointer network is responsible for identifying the start and end position of one element within a quadruple. For each pointer network, the value of K is set to 2. In formula (5), K denotes the number of loss functions computations required for a quadruple, which is equal to 8 (2*4) .\n\n\n**Question #2:** On page 3, line 202, I am not clear about the use of transfer learning, I think it is to conduct pre-training on the dataset generated by ChatGPT and then fine-tune on the COQE dataset. Please correct me if I have misunderstood.\n\n**Response to #2:** We appreciate your opinion, and it is absolutely correct. We first conduct pre-training on the dataset generated by ChatGPT, and then utilize transfer learning to fine-tune our model on the original COQE dataset.\n\n\nThanks again for your constructive suggestions, and we will add more descriptions to solve any confusion in the extended version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In this paper, they propose a novel two-stage data augmentation method that leverage generative capabilities of ChatGPT and the transfer learning, which realizes the effect of data augmentation on COQE task. This has some inspiration on how to better combine other tasks with LLMs capabilities to improve the performance of target tasks.", "reasons_to_reject": "no", "questions_to_authors": "Question A: The author did not provide a definition of what K represents in formula (5).\n\nQuestion B: On page 3, line 202, I am not clear about the use of transfer learning, I think it is to conduct pre-training on the dataset generated by ChatGPT and then fine-tune on the COQE dataset. Please correct me if I have misunderstood.\n", "paper_topic_and_main_contributions": "This paper introduces a novel Data Augmentation with Prompting approach to address two issues in comparative opinion quintuple extraction (COQE), namely applying Large Language Models (LLMs) directly to COQE has resulted in underwhelming performance and the complexity and insufficient amounts of annotated data hinder the performance of COQE models. Firstly, they propose an end-to-end model architecture better suited to the data augmentation method from triplets to quintuples. Additionally, they propose a two-stage data augmentation strategy combining the LLMs (ChatGPT) and transfer learning to further improve the performance of low-resource COQE. The experimental results validate the effectiveness and robustness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors present the low-resource comparative opinion quintuple extraction by Data Augmentation with Prompting (DAP). Authors use a two-stage data augmentation strategy which first guides ChatGPT to generate triplet datasets and then employs transfer learning. 

One of the 4 reviewers did not acknowledge the reading of the rebuttal.
For the soundness, reviewers give 2 (borderline), 3(good), 3(good), and 4(strong). For the excitement, 3 reviewers give 3 (ambivalent) and 1 gives 4 (strong). Some discussions authors did with the reviewer giving 2 were post-rebuttal. The response from authors has a reasonable explanation for the post-rebuttal question, as well as to the other questions. I have taken all these issues into account.
Decision: Accept-Findings