UniCOQE: Unified Comparative Opinion Quintuple Extraction as a Set
Year: 2023

Abstract:
Comparative Opinion Quintuple Extraction (COQE) aims to identify comparative opinion sentences in product reviews, extract comparative opinion elements in the sentences, and then incorporate them into quintuples. Existing methods decompose the COQE task into multiple primary subtasks and then solve them in a pipeline manner. However, these approaches ignore the intrinsic connection between subtasks and the error propagation among stages. This paper proposes a unified generative model, UniCOQE, to solve the COQE task in one shot. We design a generative template where all the comparative tuples are concatenated as the target output sequence. However, the multiple tuples are inherently not an ordered sequence but an unordered set. The pre-defined order will force the generative model to learn a false order bias and hinge the model’s training. To alleviate this bias, we introduce a new “predict-and-assign” training paradigm that models the golden tuples as a set. Specifically, we utilize a set-matching strategy to find the optimal order of tuples. The experimental results on multiple benchmarks show that our unified generative model significantly outperforms the SOTA method, and ablation experiments prove the effectiveness of the set-matching strategy.

heading: 1 Introduction
text: As an essential branch of opinion mining, comparative opinion mining aims to explore the information of comparisons in product reviews. Jindal and Liu (2006b) first proposed the concept of comparative opinion mining and introduces two primary subtasks: Comparative Sentence Identification (CSI) and Comparative Element Extraction (CEE). The former aims to identify whether a given sentence is a comparative sentence, while the latter aims to extract all the comparative elements in a given comparative sentence. Panchenko et al. (2019) further
∗Corresponding author.
proposed the Comparative Preference Classification (CPC) task, which seeks to predict the comparative preference (BETTER, WORSE, or NONE) of a provided comparative sentence.
In order to integrate various subtasks of comparative opinion mining, Liu et al. (2021) first proposed the Comparative Opinion Quintuple Extraction (COQE) task (shown in Fig. 1). COQE aims to identify comparative opinion sentences in product reviews and extract five comparative opinion elements in the sentences, i.e., comparative subject (sub), comparative object (obj), comparative aspect (ca), comparative opinion (co) and comparative preferences (cp), and then incorporate them into a quintuple (sub, obj, ca, co, cp). Liu et al. (2021) adopted a multi-stage model, decomposing the COQE task into primary sub-tasks (CSI, CEE, and CPC formerly mentioned), and then solving them one by one in a pipeline manner. However, the pipeline model ignores the internal connection between multiple subtasks of comparative opinion mining, and the error propagation between each stage heavily strains the model’s performance.
To this end, we employ a generative extraction model called UniCOQE for the first time on the COQE task. We utilize T5 (Raffel et al., 2020) as the backbone and propose a generative template to adapt to the COQE task, identifying comparative
12229
sentences and extracting all quintuples once and for all.
In the generation paradigm, we concatenate all the golden comparative tuples together as the target output sequence of the model. However, multiple tuples are essentially not an ordered sequence but an unordered set. If a pre-defined order is imposed, it will introduce an order bias, forcing the generative model to learn the bias, which hinders the model’s training. Taking Fig. 1 as an example, there are four target tuples: t1, t2, t3, and t4. Theoretically, A44 = 24 types of permutations of target tuples are all correct. During training, the model would get “confused”: Why t1; t2; t3; t4 is correct but t4; t3; t2; t1 is unacceptable?
In order to alleviate this order bias problem, we introduce a “predict-and-assign” training paradigm to the generative model. During the training phase, we first let the model autoregressively predict comparative tuples in the given sentence. Subsequently, we model the golden tuples as a set and use the Hungarian algorithm (Kuhn, 1955) to match the set of golden tuples with the predicted sequence to find the optimal order of golden tuples.
Finally, we validate the performance of our approach on three COQE benchmarks. Experimental results show that our model significantly outperforms SOTA methods, and the effectiveness of the set-matching strategy is demonstrated through ablation experiments.
The contributions of this paper can be summarized as follows:
• We propose a generative comparative opinion quintuple extraction model to solve the error propagation problem of previous multi-stage models.
• We introduce the “predict-and-assign” training paradigm based on a set-matching strategy to alleviate the order bias of the generative model during training.
• Our model significantly outperforms previous SOTA models, and ablation experiments verify the effectiveness of the set-matching strategy.

heading: 2 Related Works
text: As an important subtask of opinion mining, the task of comparative opinion mining was first proposed by Jindal and Liu (2006a,b), which aims to identify
comparative sentences in product reviews and extract all the comparative opinion elements (entities, features, and comparative keywords). Specifically, it used class sequential rules(Hu and Liu, 2006) to identify comparative sentences and label sequential rules to extract comparative elements.
Some subsequent studies concentrated on the comparative sentence identification (CSI) task. Huang et al. (2008) used diverse features (e.g., keywords and sequential patterns) to recognize comparative sentences. Park and Blake (2012) exploited semantic and grammatical features to explore the task of identifying comparative sentences in scientific texts. Liu et al. (2013) recognized comparative sentences on Chinese documents based on keywords, sentence templates, and dependency analysis.
On the comparative element extraction (CEE) task, Hou and Li (2008) used semantic role labeling (SRL) to analyze the structure of comparative sentences and trains a conditional random field (CRF) to extract comparative features. Some studies (Song et al., 2009; Huang et al., 2010; Wang et al., 2015a) also used CRF as the extraction model. Kessler and Kuhn (2013) further explored the application of existing SRL methods to comparative element extraction. Arora et al. (2017) proposed applying deep learning methods to comparative opinion mining, mainly using an LSTM-CRF framework to extract comparative elements.
Considering the early comparative opinion mining tasks did not include the author’s comparative preference, Ganapathibhotla and Liu (2008) proposed the Comparative Preference Classification (CPC) task for the first time, aiming to predict which entity is preferred given a comparative sentence and its comparative elements. It utilized a keyword-based approach to identify comparative preferences. Panchenko et al. (2019) used a pretrained encoder to encode sentences and classified sentences’ comparative preference based on XGBoost (Chen and Guestrin, 2016). Ma et al. (2020) employed a graph attention network to model the syntactic parsing information of comparative sentences to better predict comparative preferences. Nevertheless, the premise of the CPC task is that the two entities to be compared are annotated in advance, which is challenging to apply in real-world scenarios.
Liu et al. (2021) first introduced the task of comparative opinion quintuple extraction (COQE),
which aims to extract quintuples (comparative subject, comparative object, comparative aspect, comparative opinion, comparative preference). Specifically, it utilized a multi-stage model based on BERT (Devlin et al., 2019) performing CSI, CEE, and CPC tasks at each stage. Although this method serialized multiple subtasks of comparative opinion mining in a pipeline manner, the error propagation across multiple stages undermined the model’s performance.
In addition to subtasks such as CSI, CEE, CPC, and COQE, some research directions are also closely related to comparative opinion mining. Comparative question answering system (Alhamzeh et al., 2021; Chekalina et al., 2021) allows the machines to automatically answer the comparative question “Is X better than Y with respect to Z?’. Opinion tuple extraction (Jian et al., 2016; Peng et al., 2020) and quadruple (Cai et al., 2021) extraction in traditional aspect-based sentiment analysis aim to extract fine-grained opinion information in the text.
Several studies have also explored the use of setmatching strategies for generative models. In the keyphrase extraction task, Ye et al. (2021) concatenate all the keyphrases as target outputs of Transformer (Vaswani et al., 2017) without predefining an order. In the event argument extraction task, Ma et al. (2022) introduces a scheme for optimal span
assignments of BART (Lewis et al., 2020). These studies demonstrate the effectiveness of set matching strategies in generative models, highlighting their potential for improving the performance of generative LMs.

heading: 3 Methodology
text: This section introduces the UniCOQE framework in detail (as shown in 2). In this framework, we model the COQE task as a natural language generation task. We use the generative pre-trained language model T5 (Raffel et al., 2020) as the backbone model and adopt a generation template to directly identify comparative sentences and output the comparative quintuples therein in an end-to-end manner. To further alleviate the order bias problem of the generative models, we introduce the
“predict-and-assign” training paradigm.

heading: 3.1 Task Formulation
text: We first formulate the COQE task as follows: Given a product review sentence X = { x1, ..., xn } containing n tokens, COQE aims to identify whether it is a comparative sentence and (if so) extract all comparative quintuples in it:
SX = { tup1, ..., tupk }
= { (sub1, obj1, ca1, co1, cp1), ...,
(subk, objk, cak, cok, cpk) } (1)
where k is the number of comparative quintuples extracted from comparative sentence X . tup = (sub, obj, ca, co, cp) is an extracted quintuple, where sub is the subject entity, obj is the object entity, ca is the aspect being compared, co is the opinion of the author reflecting a comparative preference. cp ∈ {WORSE, EQUAL, BETTER, DIFFERENT} is the comparative preference of the author.

heading: 3.2 COQE with Generative Paradigm
text: In this section, we introduce the generative paradigms for the COQE task. We design a T5 generation template for end-to-end extraction of quintuples. Examples are as follows:
Input: Canon’s optics and battery are more reliable than those of Sony and Nikon. Target: (Canon, Sony, optics, more reliable, BETTER); (Canon, Sony, battery, more reliable, BETTER); (Canon, Nikon, optics, more reliable, BETTER); (Canon, Nikon, battery, more reliable, BETTER)
Input: Canon’s optics and battery are so great. Target: (unknown, unknown, unknown, unknown, unknown)
In the generative paradigm, k golden quintuples are concatenated with “ ; ” as the target sequence of the model. If a comparison element does not exist, it is padded with the word “unknown”. If the target sequence is“(unknown, unknown, unknown, unknown, unknown)”, the corresponding input sentence X is then considered a non-comparative sentence. We call this approach the Vallina generative paradigm.
Still, a problem exists with the Vallina generative paradigm: The k target tuples are essentially an unordered set, rather than an ordered sequence. The training of the generative model is fundamentally based on the cross-entropy loss, depending heavily on the order of the target text sequence. In multituple scenarios, artificially predefining an order can introduce a false order bias during training, undermining the model’s performance.

heading: 3.3 Improving Generative COQE with Predict-and-Assign Paradigm
text: To address the order bias problem, we introduce a “predict-and-assign” training paradigm. The paradigm incorporates two steps: predicting step and assigning step.

heading: 3.3.1 Predicting Stage
text: For the input sentence X = { x1, ..., xn } , during the training phase, we temporarily turn off the gradient backpropagation of the model and send X into the T5-encoder to get the latent representation of the sentence :
henc = Encoder(X) (2)
We then used T5-decoder to predict all the comparative quintuples autoregressively. At the cth moment of the decoder, henc and the previous output tokens: t1:c−1 are utilized as the input into the decoder:
hdecc = Decoder(h enc, t1:c−1) (3)
The conditional probability of token tc is defined as follows:
P (tc|t1:c−1, X) = Softmax(hdecc W + b) (4)
where W ∈ Rdh×|V|, b ∈ R|V|. V here refers to the vocabulary size of T5. Then the final predicted sequence of tuples is:
Tpred = t1:m = {t1, ..., tm} (5) where m is the length of the predicted sequence. We split Tpred with the semicolon symbol “ ; ” to get a set of comparative quintuple predicted by the model: Qpred = {tuppred1 , ..., tuppredl }.

heading: 3.3.2 Assigning Stage
text: Given two tuples: p and g, we define the similarity score between p and g as follows:
sim(p, g) = 1
n
n∑
k=1
IoU ( p(k), g(k) ) (6)
where n is the number of elements in tuples. In our case, n = 5 constantly for we have five elements(i.e., sub, obj, ca, co, and cp) in the comparative quintuples. IoU here refers to the “intersection over union” of the two token sequences, and k refers to the index of the element (e.g., k = 3 for ca). Therefore, IoU(p(k), g(k)) calculates the IoU score of the k-th element of both tuples. We eventually take the average IoU score of all five elements as the similarity score of two tuples. For example, in Fig. 3, we have tuple p1 = (Canon, Nikon, sensors, less stable, WORSE), and g2 = (Canon, Sony, sensors, less stable, WORSE), the element-wise IoU scores are 1, 0, 1, 1, and 1, respectively. So the similarity score between p1 and g2 is 0.8.
We then define the assignment cost between p and q:
cost(p, g) = 1− sim(p, g) (7) For the ground-truth tuple set Qgold = {tupgold1 , ..., tupgoldK }, we aim to find a permutation π̂ of Qgold, so that π̂(Qgold) is the most similar sequence to the tuples predicted by the model in predicting stage (Section.3.3.1). This is essentially an assignment (a.k.a binary matching) problem.
Formally, to find an optimal order of groundtruth tuples Qgold, we search for a permutation π̂ that minimizes the total assignment cost:
π̂ = arg min π∈Π(K)
Cmatch ( π∗(Qpred), π(Qgold) ) (8)
where K is the number of tuples in Qgold. Π(K) is the space of permutations of K tuples in Qgold. π∗(Qpred) is the predicted sequence of tuples in Formula (5). This process of finding the optimal assignment can be solved efficiently by Hungarian algorithm (Kuhn, 1955). Cmatch(π∗, π̂) is the total pair-wise matching cost between permutation π∗ and permutation π̂. The assignment cost can be defined as follows:
Cmatch ( π∗(Qpred), π(Qgold) )
= s∑
i=1
cost ( π∗(Qpred)i, π(Qgold)i ) (9)
where s = min(|Qpred|, |Qgold|) is the minimum number of tuples between Qpred and Qgold.
π∗(Qpred)i and π(Qgold)i refer to the ith tuple in π∗(Qpred) and π(Qgold) respectively.
After assigning the new order of the golden tuples, we take the new order as the training target of the model and re-open the gradient backpropagation to restart training.

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments on three COQE datasets released by Liu et al. (2021): Camera-COQE, CarCOQE, and Ele-COQE:
• Camera-COQE contains English product reviews in the camera domain. This dataset is based on Kessler and Kuhn (2014), completing the annotations of comparative opinions (co) and comparative preferences (cp).
• Car-COQE contains Chinese product reviews in the automobile domain. This dataset is based on the Car dataset in the
COAE2012/2013 (Tan et al., 2013), supplemented with annotations of comparative opinions (co) and comparison preferences (cp).
• Ele-COQE similarly derives from the electronic product review dataset in COAE2012/2013 (Tan et al., 2013), which contains Chinese comparative product reviews of electronic products.
The statistics of the three datasets are demonstrated in Table 1. Each dataset contains both noncomparative and comparative sentences. #Comparative indicates the number of comparative sentences, and #Non-Comparative refers to the number of non-comparative sentences. #MultiComparisons is the number of comparative sentences containing multiple comparisons.

heading: 4.2 Experimental Setup
text: We employ T5 as the backbone model. We utilize T5 for the English dataset and Multilingual T5 (mT5) (Xue et al., 2021) for the Chinese datasets. We did not choose the Chinese T5 model because there are multiple non-Chinese characters (i.e., product names and versions) in the Car-COQE and Ele-COQE. We employ T5-base and mT5-base provided by Huggingface1 library for experiments. For T5 and mT5, we set the batch size to 24 and 10, respectively. The learning rates of both models are set to 3e-4. We train T5 for 60 epochs and mt5 for 30 epoches.

heading: 4.3 Evaluation Metrics
text: Following the setting of Liu et al. (2021), for the comparative sentence identification (CSI) task, We report the Accuracy metric. For the COQE task, we consider three matching strategies: Exact Match, Proportional Match, and Binary Match. These three metrics measure the F1 scores to varying degrees on the predicted tuples by the models.
1https://github.com/huggingface/transformers
Specifically, for the three metrics, we define #correcte, #correctp, #correctb as follows:
#correcte = { 0, ∃(gk ̸= pk) 1, otherwise
(10)
#correctp =
{ 0, ∃(gk ̸= pk = ∅)∑
k len(gk∩pk)∑ k len(pk) , otherwise
(11)
#correctb = { 0, ∃(gk ∩ pk = ∅) 1, otherwise (12)
where gk is the kth element of a golden comparison quintuple, and pk is the kth element of a predicted comparison quintuple. len(·) represents the length of the comparison element.

heading: 4.4 Baseline Models
text: We take the following baseline models for comparison :
Multi-StageCSR-CRF (Jindal and Liu, 2006a) uses an SVM based on CSR features to identify comparative sentences and uses a CRF to extract comparative elements.
JointCRF (Wang et al., 2015b) uses CRF to jointly extract comparative sentences and comparative elements.
Multi-StageLSTM (Liu et al., 2021) utilizes an LSTM as a text encoder. The method decomposes the COQE task into three subtasks: comparative sentence identification, comparative element extraction, and comparative preference classification, and solves these subtasks successively in a pipeline manner.
Multi-StageBERT (Liu et al., 2021) is a variant of Multi-StageLSTM, specifically, replacing the text encoder with BERT.

heading: 4.5 Main Results
text: In Table.2, we report the performance of all five methods on the two tasks of CSI and COQE on
the three datasets: Camera-COQE, Car-COQE, and Ele-COQE. For CSI, we report the Accuracy metric. All indicators are in the case of Exact Match.
Experimental results show that the UniCOQE model achieves the best performance on all three datasets on both the CSI task and the COQE task. The two CRF-based methods generally yield the lowest performance on both tasks. MultiStageLSTM achieves relatively better performance. On the CSI task, Multi-StageBERT has already achieved rather satisfactory results of Accuracy: 93.04, 97.39, and 98.31 on three datasets. However, it is notable that our UniCOQE model still outperforms Multi-StageBERT by 2.17, 0.89, 0.10 percent.
On the COQE task, the UniCOQE model
achieves 18.59, 6.80, and 4.73 percent of improvement on the Camera-COQE Car-COQE and EleCOQE datasets, respectively. It is worth noting that the advantage of our UniCOQE model over other models is more evident on the English dataset than on the Chinese datasets. One possible explanation is that mT5, a multilingual version of the T5, involves the pre-training of multiple languages and has a more expansive vocabulary list, which would weaken the model’s performance on monolingual datasets.

heading: 4.6 Influence of the Set-Matching Strategy
text: In Table.3, we show the impact of the set-matching strategy over the generative model. The experimental results show that compared with the Vallina generative model, the set-matching strategy has improved the model’s performance on CameraCOQE, Car-COQE, and Ele-COQE datasets under all three metrics. It reveals that the set-matching strategy indeed finds a better order of tuples, helping the model better learn the data distribution.

heading: 4.7 Multi-Tuple Scenarios Results
text: To measure the model’s effectiveness on the multituple data, we only use the multi-tuple data in the test set for evaluation. We demonstrate the multituple scenario results in Table.4. The experimental results show that the set-matching strategy has considerably improved the model’s performance on multi-tuple data. Taking the Exact match metric as an example, compared to the Valiina generative
paradigm, UniCOQE obtains 3.87, 1.74, and 1.70 percent of improvements on the Camera-COQE, Car-COQE, and Ele-COQE, respectively.

heading: 4.8 Exchanges of Multi-Tuples
text: Fig. 4 exhibits the number of exchanges of multituples during the training process of UniCOQE. During the first ten epochs, the number of tuple exchanges keeps on increasing. Around the 11th epoch, all three datasets reach their peak, and the number becomes stabilized. The number of tuple exchanges on Camera and Car is both stabilized at around 140. In contrast, the Electronic dataset is stabilized at around 60, for the Electronic domain has fewer multi-tuple data.

heading: 4.9 Case Study
text: In Fig. 5, we illustrate the effect of the tuplematching strategy on T5’s training procedure. Taking Example.1 as an instance, we can observe that at the very beginning of the model’s training (epoch 1), if we follow the default “golden” sequence order, the calculated cross-entropy loss will be 1.453. However, if we assign a new tuple order according to our set-matching strategy, the new loss will become 0.598. The phenomenon is more evident as the training epoch increases. As demonstrated in Example 2, at epoch 15, the default tuple order would end up with a loss of 2.244, whereas the loss of the newly assigned order is much smaller: 0.032.

heading: 5 Conclusion
text: In this paper, we investigate the task of comparative opinion quintuple extraction. To overcome
the error propagation problem of previous pipeline models, we propose an extraction model based on the generative paradigm. We further introduce a set-matching strategy based on the Hungarian algorithm to alleviate the order bias of the generative model during training. The experimental results show that our model significantly outperforms the SOTA models, and we verify the effectiveness of the set-matching strategy through in-depth experiments.

heading: 6 Limitations and Future Works
text: We summarize the limitations of our work as follows:
• We only validate the effectiveness of the setmatching strategy for generative models on the COQE task.
• We observe that the scale of the COQE datasets is quite small and has caused the model’s overfitting problem.
In the future, we will conduct further research from the following perspectives:
• Explore further application of the setmatching strategy in multiple research directions, such as information extraction, sentiment analysis, etc.
• Utilize unsupervised data to better help the models mine comparative opinion information.
• Design data augmentation methods to relieve the data sparsity problem.
Ethics Statement
We perform experiments on three datasets formerly established by (Liu et al., 2021), namely CameraCOQE, Car-COQE, and Ele-COQE. These datasets do not include personal information or contain any objectionable content that could potentially harm individuals or communities. It’s important to note that certain product reviews may include subjective comparisons between products given by anonymous customers, which do not necessarily reflect the preferences of this study.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of China (No. 62076133, 62006117, and 72001102), and the Natural Science Foundation of Jiangsu Province for Young Scholars (No. BK20200463) and Distinguished Young Scholars (No. BK20200018).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes. We discuss the limitations of our work in Section 6: Limitations and Future Works.

heading: A2. Did you discuss any potential risks of your work?
text: Not applicable. The main theme of our work is to mine comparative opinions in publically available product reviews, and all the datasets utilized in this paper are also open source.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes. Abstract and Section 1: Introduction summarize the paper’s main claims. We demonstrate the topic, challenge, and contributions of our paper in both sections.
7 A4. Have you used AI writing assistants when working on this paper? No.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.

heading: B1. Did you cite the creators of artifacts you used?
text: Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.

heading: B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Not applicable. We do not use or create scientific artifacts in this paper.
C 3 Did you run computational experiments? Yes. In section 4: Experiments.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes. In section 4.2: Experimental Setup.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes. In section 4.2: Experimental Setup.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes. In section 4.5 Main Results.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes. In section 4.2: Experimental Setup.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Not applicable. We do not involve human annotations in this paper.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Not applicable. We do not involve human annotations in this paper.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Not applicable. We do not involve human annotations in this paper.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Not applicable. Not applicable. We do not involve human annotations in this paper.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Not applicable. Not applicable. We do not involve human annotations in this paper.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Not applicable. We do not involve human annotations in this paper.

heading: Future Work
text: We discuss the limitations of our work in Section 6: Limitations and Future Works