Aspect-aware Unsupervised Extractive Opinion Summarization
Year: 2023

Abstract:
Extractive opinion summarization extracts sentences from users’ reviews to represent the prevalent opinions about a product or service. However, the extracted sentences can be redundant and may miss some important aspects, especially for centroid-based extractive summarization models (Radev et al., 2004). To alleviate these issues, we introduce TokenCluster – a method for unsupervised extractive opinion summarization that automatically identifies the aspects described in the review sentences and then extracts sentences based on their aspects. It identifies the underlying aspects of the review sentences using the roots of noun phrases and adjectives appearing in them. Empirical evaluation shows that TokenCluster improves aspect coverage in summaries and achieves strong performance on multiple opinion summarization datasets, for both general and aspectspecific summarization. We also perform extensive ablation and human evaluation studies to validate the design choices of our method. The implementation of our work is available at https://github.com/leehaoyuan/TokenCluster.

heading: 1 Introduction
text: In the internet era, online reviews are important for both customers and businesses. Customers use the reviews to help them make better choices while businesses gather feedback from the reviews. However, the large number of reviews for a product can make it time-consuming to go through all of them. Opinion summarization aims to tackle this problem by creating a concise summary of all the reviews. Recently, there has been significant progress in the supervised summarization of single (Liu and Lapata, 2019; Zhong et al., 2020; Liu et al., 2022) and multiple (Fabbri et al., 2019; Pasunuru et al., 2021; Xiao et al., 2022) documents. Unfortunately, opinion summarization cannot directly benefit from the progress as collecting human-written summaries for large-scale review sets is expensive. Therefore,
most opinion summarization techniques resort to unsupervised approaches.
There are two major paradigms for opinion summarization: abstractive summarization, which generates summaries using novel phrasing, and extractive summarization, where the summary is selected from a subset of the input sentences. For abstractive summarization, previous works usually generate summaries using aggregated sentence representations (Chu and Liu, 2019; Iso et al., 2021a) or train a supervised model on synthetic datasets (Amplayo et al., 2021a; Amplayo and Lapata, 2020). For extractive summarization, recent works use techniques that consists of two primary components: (a) a model for learning representations for review sentences, and (b) an inference algorithm that uses these representations to select summarizing sentences (Angelidis and Lapata, 2018; Angelidis et al., 2021; Chowdhury et al., 2022a). Our work focuses on the inference algorithm for extractive opinion summarization.
12662
Extractive summarization can benefit from aspect identification since extracted sentences should cover all salient aspects within a review set. Aspects in user reviews usually focus on specific features or attributes of an entity (e.g., room, cleanliness, and food for a hotel). Most opinion summarization systems are centroid-based and extract sentences closest to the centroid as the summary. The extraction process can result in redundant summaries that do not cover all aspects. Therefore, it is beneficial to identify aspects of review sentences in order to generate more diverse extractive summaries (as illustrated in Figure 1).
Previous works (Angelidis and Lapata, 2018; Zhao and Chaturvedi, 2020) use aspect seed words to identify salient aspects in reviews. Aspect seed words are sets of words related to the aspect of interest (e.g. ‘food’, ‘restaurant’, ‘breakfast’ for the food aspect). These works have shown that using aspect seed words can reduce redundancy and improve the informativeness of summaries. However, extraction of aspect seed words either requires manual annotation (Angelidis and Lapata, 2018) or external data (Zhao and Chaturvedi, 2020). Moreover, aspects for different products or entities may vary based on the domain, requiring additional effort while scaling such techniques to different domains.
Motivated by this observation, we propose TokenCluster, an inference algorithm to automatically identify aspects of sentences and extract summarizing sentences based on their aspects. TokenCluster identifies the aspect of a sentence using the noun phrases and adjectives that appear in it. After that, it uses a novel inference algorithm for extracting sentences that summarize the general opinion of each aspect. It eventually orders the extracted sentences to improve readability. TokenCluster is independent of the underlying sentence representation model and can perform general as well as aspect-specific summarization. Our experiments show that TokenCluster outperforms competitive baselines on the Space (Angelidis et al., 2021) and Amazon (Bražinskas et al., 2020) datasets using various sentence representation models. To summarize, our contributions are:
• We propose a novel approach to automatically identify aspects of review sentences. • We design a novel inference method that extracts summaries covering the most salient aspects. • TokenCluster shows strong performance using both automatic and human evaluation metrics.
• We perform extensive analysis experiments to validate the design of TokenCluster.

heading: 2 Related Work
text: Recent opinion summarization methods could be divided into two categories: abstractive and extractive summarization. For abstractive summarization, previous works generate abstractive summaries in two ways. Some of these works (Chu and Liu, 2019; Iso et al., 2021b; Isonuma et al., 2021) use aggregated sentence representations from autoencoders to generate summaries. Others (Bražinskas et al., 2020; Amplayo and Lapata, 2020; Amplayo et al., 2021a,b; Wang and Wan, 2021; Ke et al., 2022) generate synthetic datasets to train generation models in a supervised setting. For extractive summarization, previous works usually follow a two-step approach. In the first step, they learn representations for the review sentences. In the second step (the inference step), they use the learned representations to define relevance scores based on distance from the mean (Chowdhury et al., 2022a,b), distance from the aspect representation (Angelidis et al., 2021) or aspect-specificity and sentiment polarity (Zhao and Chaturvedi, 2020; Angelidis and Lapata, 2018). This work proposes a novel inference algorithm that is independent of the underlying sentence representation model and performs summarization by leveraging the aspects present in a review sentence.
Previous works usually identify aspects in two ways. Some (Angelidis and Lapata, 2018; Zhao and Chaturvedi, 2020; Amplayo et al., 2021a) of them identify aspects using aspect seed words. They generally require manual annotation or external data to obtain aspect seed words. Others (Amplayo et al., 2021b; Wang and Wan, 2021; Ke et al., 2022) use autoencoders to identify the aspect automatically. They do not require human annotation, but they can be noisy since only a portion of words contributes towards the aspect. TokenCluster tries to combine the advantages of both approaches. Motivated by Hu and Liu (2004, 2006); Lu et al. (2009), we identify a set of words that are likely to describe the aspect and cluster them into several aspects.
Our work is also related to a contemporary but unpublished work by Bhaskar et al. (2022). Like our method, their method also clusters review sentences based on the distance between a single keyword of each sentence and the aspect seed words. Our method differs from their
work in two major aspects. First, in contrast to their approach, TokenCluster extracts multiple aspect-related words from each sentence since a sentence can mention several aspects. Second, TokenCluster does not rely on human-annotated aspect seed words.

heading: 3 TokenCluster
text: In this section, we describe our proposed approach, TokenCluster. We first describe the problem setup (Sec. 3.1). Then, we describe the three components involved of TokenCluster’s general summarization algorithm: (a) aspect identification in review sentences (Sec. 3.2), (b) salient sentences identification using aspect information (Sec. 3.3), and (c) sentence ordering to produce the final summaries (Sec. 3.4). Finally, we describe how TokenCluster generates aspect-specific summaries (Sec. 3.5).

heading: 3.1 Problem Definition
text: To perform opinion summarization, we consider an entity e (e.g., a product or hotel) for which a set of reviews are available. Each review contains multiple sentences. We use Se = {s1, s2, . . .} to denote the set of sentences from all reviews for e. The goal is to extract some sentences from Se in order to summarize the prevalent opinion (general summarization) or to summarize opinions about a particular aspect (aspect-specific summarization).

heading: 3.2 Aspect Identification
text: In this section, we describe how we identify the aspects of review sentences. The aspect identification workflow is shown in Figure 2. To achieve this, we try to locate aspect-related words within a sentence, that describe a certain feature of the product or entity being reviewed. Inspired by Hu and Liu (2004, 2006), we consider the roots of noun phrases as aspect-related words since they can directly name the features. For example, for the noun phrase – ‘front desk staff’, we consider its root – ‘staff’. We do not consider pronouns (as it is difficult to perform coreference resolution on user reviews) and proper nouns. We further observe that adjectives can also be indicative of the aspects of sentences. Based on this observation, we incorporate adjectives that do not belong to any noun phrases into the set of aspect-related words. We exclude the adjectives belonging to noun phrases since their aspect information has been covered already by the roots
phrases are highlighted, roots of those noun phrases are underlined, and adjectives (not part of noun phrases) are shown in blue. Second, we cluster these aspect-related words to identify latent aspects. Finally, we assign latent aspects to review sentences based on the presence of cluster-specific words.
of corresponding noun phrases. We showcase an example of this identification process in Figure 2. Eventually, the set of aspect-related words for an entity e, W(e)asp = {w1, w2, ...}, includes the roots of noun phrases and adjectives (not belonging to noun phrases) from all sentences in Se.
Next, we obtain the contextual representation for each aspect-related word in W(e)asp using a pretrained BERTbase model (Devlin et al., 2019). We use contextual representations of aspect-related words because words with similar surface forms could describe different aspects. For example, people could use ‘nice’ to describe that the staff is kind or use it to say ‘food’ was good. Let wi ∈ W(e)asp be the k-th word of a sentence s ∈ Se containing L words. We feed the sentence s into the pre-trained encoder, Encoder(·):
e = Encoder(s) ∈ RL×d, (1)
where d is the representation dimension. We use wi = e[j], the j-th element of e as the representation for word wi. Finally, we obtain W (e) asp = {w1,w2, . . .}, which denotes the set of representations of words in W(e)asp.
Despite focusing on noun phrases and adjectives, the set of aspect-related words can contain irrelevant words that do not provide much information about the aspects. To exclude outliers and represent the prevalent aspects, we filter out word representations that are far away from the centroid of word
representations c = E[W(e)asp] ∈ Rd. Specifically, we filter out wi if ∥wi − c∥2 are ranked among the top γ of W(e)asp.
Next, we want to cluster words that talk about the same aspect together. For example, words like ‘friendly’, ‘nice’, and ‘attentive’ that describe the ‘staff’ aspect should be grouped together. For this, we cluster representations in W(e)asp into k clusters using Ward hierarchical clustering algorithm (Ward Jr, 1963). We assume that each cluster captures an underlying aspect within the review set. Thereby, an entity has a set of k salient aspects, Ae = {a1, a2, . . . , ak}.
So far, we have identified the aspect-related words in the review set. Then, we have clustered these words into k groups that capture underlying aspects. Next, we identify the aspects of a sentence based on the cluster assignment of the aspect-related words within it. In this setup, a review sentence can be assigned multiple aspects. For an entity e, S(e)j denotes the set of sentences mentioning the aspect aj .

heading: 3.3 Sentence Extraction
text: Once we assign every review sentence to one or more aspects, we design an algorithm to extract sentences based on their aspect information. For this, we need to obtain a representation si, for each sentence, si. We can use any sentence representation model to obtain si. In our experiments, we use SemAE (Chowdhury et al., 2022a) to retrieve si ∈ RD since it achieves state-of-the-art performance in extractive opinion summarization.
After we retrieve the sentence representations, we calculate the relevance between a sentence and each of the aspects it belongs to (as determined in Sec. 3.2). We define the relevance score R(si, aj) between sentence si, and aspect aj , which is calculated using KL-divergence:
R(si, aj) = { −KL(si,aj), if sj ∈ S(e)j −∞, otherwise , (2)
where aj is the mean representation of all review sentences containing the j-th aspect, S(e)j :
aj = E[{si|si ∈ S(e)j }] ∈ RD. (3) A higher R(si, aj) score indicates that si is more representative of the aspect aj . We use this score to extract summary sentences. However, apart from
Algorithm 1 Aspect-aware Sentence Extraction 1: Input: Aspect set Ae, relevance score function
R(·, ·), summary budget B 2: Uasp = Ae ▷ Initializing uncovered aspects 3: Oe = {} ▷ extracted sentences 4: while Uasp ̸= ∅ ∧ |Oe| < B do 5: rbest = −∞ 6: for a ∈ Uasp do 7: sa = argmaxs{R(s, a)|a ∈ asp(s)} 8: if R(sa, a) > rbest then 9: s∗ = sa ▷ pick sa with best score
10: rbest = R(sa, a) 11: end if 12: end for 13: Oe = Oe ∪ s∗ ▷ add to summary 14: Casp = ∪s∈Oeasp(s∗) 15: Uasp = Uasp \ Casp ▷ update Uasp 16: end while 17: return Oe
representing diverse aspects, we want the extracted sentences to be concise and convey as much information as possible within a length budget. To ensure this, we modify the relevance score by introducing a length penalty term to penalize the number of tokens per aspect.
R(si, aj) = R(si, aj)− β log ( |si| |asp(si)| ) , (4)
where β is the weight of length penalty term, |si| is the number of tokens in si, asp(si) is the set of aspects that are covered in si. We use the log(·) function because sentence lengths usually follow a long-tail distribution.
Next, we use the relevance scores to extract sentences that would form the summary. We extract sentences iteratively and ensure the extracted sentences cover all the salient aspects. The extractive summarization routine is described in Algorithm 1. We denote the set of uncovered aspects as Uasp at any point. Uasp is initialized as the entire aspect set Ae (Line 2). At each time step, our goal is to extract one sentence representative of one of the uncovered aspects. For this, we identify one representative sentence, sa, per uncovered aspect, a, with a corresponding relevance score, R(sa, a), and then among all uncovered aspects, extract the sentence with the highest relevance score (to its aspect) for the summary. Specifically, at each time step, we iterate over the uncovered aspects a ∈ Uasp (Line 6) and identify the sentence sa
with the highest R(s, a) score (Line 7). Then, we pick the sentence with the highest overall score among all aspects (Line 8, 9, 10) and add it to the summary, Oe (Line 13). We update the uncovered aspects Uasp by removing aspects covered by the extracted sentence, s∗ (Line 14, 15). We repeat this process until all the aspects are covered or the summary budget (B) is reached (Line 4). Finally, Oe is the sequence of sentences that form the summary for entity e (Line 17).

heading: 3.4 Sentence Ordering
text: The above-mentioned extraction process does not consider the order of sentences, which might hurt the readability of the summary. For instance, we want sentences mentioning the same aspect to appear together to prevent abrupt context switches between aspects. We also want general sentences to appear first. Ideally, we want the order of extracted sentences to mimic the writing style in human reviews. To achieve these goals, we train a state-ofthe-art textual coherence model, REBART (Basu Roy Chowdhury et al., 2021), on user reviews. We use the trained REBART to reorder the sentences in Oe. For an entity e, the extractive summary is the concatenation of sentences in Oe after reordering.

heading: 3.5 Aspect-specific Summarization
text: TokenCluster can also perform aspect-specific summarization. In this setup, the user provides a query aspect r, specified by a set of aspect seed words Qr. We obtain a set of sentences, S (e) r , that contain at least one aspect seed word from Qr. Then, we obtain the set of aspect-related words for the subset S(e)r using the aspect identification procedure (described in Sec. 3.2). However, while filtering the aspect-related words, we do not compute the centroid (c) using all aspect-related words (W(e)asp). We calculate c only using the representations of aspect-seed words:
c = E[{wi|wi ∈ Qr}] ∈ Rd. (5)
This filtering helps reduce the noise and capture aspect-specific information. We follow the same subsequent steps as described in Sec. 3.3 and 3.4.

heading: 4 Experimental Setup
text: In this section, we describe the details of our experimental setup including the datasets, hyperparameters, and baselines.

heading: 4.1 Datasets
text: We perform the experiments on the Space hotel reviews (Angelidis et al., 2021) and the Amazon product reviews (Bražinskas et al., 2020). For the Space dataset, we use the data preprocessed by Angelidis et al. (2021). For the Amazon dataset, we preprocess the data following the instructions of Bražinskas et al. (2020) and exclude the reviews that are not in English using Compact Language Detector 2.1 The statistics of the datasets are shown in Table 1. The development and test sets of both datasets contain three human-written general summaries per entity. Space dataset also contains three human-written aspect-specific summaries for the following six aspects: building, cleanliness, food, location, rooms, and service.
1https://github.com/CLD2Owners/cld2

heading: 4.2 Implementation Details
text: We use spaCy (Honnibal et al., 2020) to identify noun phrases and adjectives. We use BERTbase (Devlin et al., 2019) to get the contextual representations for aspect-related words. Since BERT uses subword tokenization, we average the subwords’ representations to get the corresponding word’s representation. For ordering the extracted sentences, we trained two versions of REBART (Basu Roy Chowdhury et al., 2021) on reviews of opinion summarization datasets for 3 epochs.
We train our representation learning model SemAE on Space and Amazon datasets using the same hyper-parameters mentioned in Chowdhury et al. (2022a). The number of clusters, k = 6 for the Space and k = 8 for Amazon. The aspectrelated word filtering percentage, γ = 40% for Space and γ = 20% for Amazon. The weight of length penalty term, β = 5× 10−3 for Space and β = 10−2 for Amazon. All hyperparameters are tuned on the development set.

heading: 4.3 Baselines
text: We compare our method, TokenCluster, with three types of summarization systems: Single Review Systems. These systems select a single review as the output summary. We compare TokenCluster with the following: (a) Random randomly samples one review from the set of reviews for an entity, (b) CentroidBERT selects the review closest to the centroid of all reviews formed
using BERT representation, (c) Oracle selects the review with the highest ROUGE score overlap with human summaries. Extractive Systems. These systems extract sentences from reviews to form summaries. We compare TokenCluster with LexRank (Erkan and Radev, 2004) using BERT representations, QT (Angelidis et al., 2021), AcesumEXT (Amplayo et al., 2021a), and SemAE (Chowdhury et al., 2022a). Abstractive Systems. These systems generate summaries using novel phrasing. We compare with MeanSum (Chu and Liu, 2019), Copycat (Bražinskas et al., 2020), PlanSum (Amplayo et al., 2021b), AceSum (Amplayo et al., 2021a), TranSum (Wang and Wan, 2021) and COOP (Iso et al., 2021b).

heading: 4.4 Results
text: We measure the performance of summarization models using ROUGE-F1 (Lin, 2004). We report ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL) for general summarization on the Space dataset in Table 3 and on the Amazon dataset in Table 2. On the Space dataset, TokenCluster achieves state-of-the-art performance on all metrics. On the Amazon dataset, TokenCluster performs the best among all extractive systems and outperforms some abstractive systems like MeanSum, CopyCat, and PlanSum in ROUGE-1 and ROUGE-2. These results show that TokenCluster can effectively select sentences from a large pool of reviews. We attribute the improvement to the enhanced aspect awareness during inference. The improvement over SemAE is statistically significant on all metrics (p < 0.05 using paired bootstrap resampling (Koehn, 2004)).
We evaluate the aspect awareness – the degree to which a general summary covers different aspects. We report RLASP (Angelidis et al., 2021), which is the average ROUGE-L score using gold aspect-specific summaries as the reference. A higher RLASP suggests better aspect coverage. We also report the average aspect coverage,
E[NASP ] (Chowdhury et al., 2022a). NASP measures the aspect coverage of the generated summaries based on the presence of aspect seed words. A higher E[NASP ] indicates that the method covers more aspects. The results are reported in Table 4. We observe that TokenCluster can cover more aspects and cover them better.
For aspect-specific summarization, we report the ROUGE-L for each aspect and the average ROUGE-1 (R1), ROUGE-2 (R2), and ROUGEL (RL) among all aspects. The results are shown in Table 5. TokenCluster achieves the best performance on average ROUGE-1 and ROUGE2. We observe that the performance gain of TokenCluster over the baselines on aspect summarization is relatively small compared to general summarization. We believe it is because the sentences obtained after filtering using aspect seed words are less diverse. Therefore, the clusteringbased extraction strategy becomes less effective. Since the aspect seed words are not very exclusive when querying the specific aspects, some clusters of words might correspond to irrelevant aspects.

heading: 4.5 Ablation Study
text: We investigate the functioning of TokenCluster by ablating several design choices within it. Length Penalty. We investigate the effect of the
length penalty term on the relevance score. For this, we compare TokenCluster with its variant without length penalty term (in Table 2, 3). On Amazon, there is a slight performance drop without the length penalty term. On Space, the ROUGE-1 increases while ROUGE-2 and ROUGE-L drop. In Table 4, there is a drop in E[NASP ], which suggests that shorter sentences can cover more aspects. We show the median sentence lengths for the two datasets in Table 6. It shows that the length penalty term encourages TokenCluster to select shorter, and hence more concise sentences as summaries. Token-level Clustering. To assign aspects for each sentence, we cluster the aspect-related words (see Sec. 3.2). Another possible way of assigning aspects for each sentence is to generate (aspectrelated) sentence representations and cluster them. In this section, we compare word-level clustering (TokenCluster’s method) with sentence-level clustering. The major difference between these two settings is that a sentence can only contain one aspect when using sentence-level clustering while a
sentence can contain multiple aspects using wordlevel clustering. We consider 3 types of sentence representations for clustering:
• BERTASP generates the aspect-related sentence representation for a sentence by averaging the BERT contextual embeddings of all aspectrelated words the sentence contains. • BERTall averages the BERT contextual representations of all words in a sentence to generate the final representation. • SemAE uses representations from SemAE as sentence representations.
We report the results of this experiment in Table 7.2 We observe that TokenCluster outperforms all sentence-level clustering variants by a significant margin except for BERTASP on Amazon, where the performances are comparable. Besides, BERTASP outperforms BERTall, which suggests that aspectrelated words are more important than all words when identifying the aspects of sentences. We also notice that BERTall outperforms SemAE but not TokenCluster. It shows that the improvement is not entirely brought by the BERT model. These results suggest that sentences can contain multiple aspects and explicitly modeling them is more effective for extractive summarization. Variants of aspect-related words. We consider the roots of noun phrases and adjectives excluding proper nouns as aspect-related words (Sec. 3.2). In this section, we compare this definition with the following five variants: TokenCluster (w/ NProot)
2For simplicity, we evaluate TokenCluster without using sentence ordering in this setting.
TokenCluster
considers only roots of noun phrases excluding adjectives. TokenCluster (w/ NN+Adj.) includes all nouns and adjectives as aspect-related words without consideration of noun phrases. TokenCluster (w/ NProot+Adj.+NNP) additionally includes proper nouns. TokenCluster (w/ all) uses all words as aspect-related words. TokenCluster (w/ aspect) uses the aspect terms extracted by Snippext (Miao et al., 2020), an Aspect-based Sentiment Analysis model. For Amazon dataset, Snippext is finetuned on the laptop domain. For Space dataset, Snippext is finetuned on the hotel domain. results are reported in Table 8. We observe that (w/ NN+Adj.) and (w/ NProot) outperform SemAE, which shows our method can provide some improvements in summarization with simpler rules of extracting aspect-related words. We also notice that the inclusion of proper nouns leads to a significant performance drop. The potential reason is that it could be difficult for the model to relate the root of proper noun phrases to a certain aspect when the phrases are long and rare. For example, the root of the proper noun phrase, ‘Canon 70-200 2.8L’ is ‘2.8L’. It would be difficult to assign this proper noun phrase into the cluster of lenses or cameras. Overall, TokenCluster with the original set of aspect-related words outperforms others. Sentence Ordering. In this section, we explore the effect of sentence ordering on the summary quality in Table 2. We observe that sentence ordering could improve the ROUGE-L score of general summarization on both datasets. However, it leads to a performance drop on RLASP and aspect sum-
marization. We believe the reason is that we train REBART on the reviews, which are more similar to general summaries than aspect-specific summaries in terms of content diversity. One example of ordered sentences is shown in Figure 9. We observe that ordering the sentences helps TokenCluster put general sentences at the beginning (shown in green) and avoid abrupt context switches between aspects (shown in red), improving the readability.
Sentence Representation Ablations. As mentioned before, TokenCluster is independent of the sentence representation model. In our implementation, we used SemAE’s sentence representation. In this section, we evaluate TokenCluster using SimCSE (Gao et al., 2021) and QT’s (Angelidis et al., 2021) sentence representation. For SimCSE, we consider two inference methods as baselines: (i) ‘Mean’ extracts sentences whose representations are close to the average sentence representation obtained from SimCSE, (ii) ‘SentCluster’ clusters sentence representations and extracts sentences close to the center of each cluster. For QT, we use the inference method used by Angelidis et al. (2021). When calculating relevance scores (Equation 2), we use Euclidean distance for SimCSE and multi-head cosine similarity for QT. We do not change other hyperparameters and design choices. In Table 10, we observe that TokenCluster significantly outperforms all baseline inference methods except on the Amazon dataset using SimCSE. The results show that TokenCluster can improve extractive summarization performance regardless of the underlying sentence representation models.

heading: 4.6 Human Evaluation
text: We perform the human evaluation on 25 general summaries from the Space dataset using Amazon Mechanical Turk (AMT). We ask the annotators to compare generated summaries from QT, SemAE, and TokenCluster based on three criteria: informativeness, coherence, and conciseness, in a pairwise manner. Each pair of summaries is annotated by three annotators. We compute the scores using Best-Worst Scaling (Louviere et al., 2015). In Table 11, we observe that TokenCluster achieves the best coherence score, and comparable informativeness scores with QT. However, it slightly falls behind QT in terms of conciseness. We observe that QT often generates short repetitive sentences with low aspect coverage, which makes it appear compact. This phenomenon is already captured in our earlier results in Table 4. We report the human evaluation results for aspect-specific summarization in Appendix A.1.

heading: 5 Conclusion
text: We present TokenCluster, a sentence extraction algorithm to automatically identify aspects of sentences and extract sentences based on their aspects. The aspect information helps TokenCluster cover more aspects in summaries. TokenCluster is independent of underlying sentence representation models and shows strong performance on Space and Amazon datasets using multiple sentence representation models. Our extensive analysis shows that our choice of token-based clustering and aspectrelated words is optimal compared to its variants. Moreover, using sentence ordering, TokenCluster can generate summaries with better readability. However, TokenCluster can be sensitive to noisy data and may not work well when underlying aspects are not well represented by unique words. Future research can focus on improving the robustness of aspect identification in TokenCluster by incorporating external data and expanding its applicability to diverse domains and languages.

heading: 6 Limitations
text: We propose TokenCluster, a sentence extraction algorithm that can automatically identify aspects in user reviews and leverage that for performing extractive summarization. Being a data-driven approach, it is susceptible to noisy data. Therefore, a limitation of TokenCluster is that the clusters of aspect-related words can be noisy and imbalanced if data is noisy. Future works can focus on more robust ways of extracting and partitioning aspect-related words. This can possibly be achieved using external data. Another limitation is that TokenCluster is computationally more expensive compared to more simple sentence extraction approaches. This is because TokenCluster clusters aspect-related words during inference, which can be restrictive when the review set is large.

heading: 7 Ethical Consideration
text: We do not expect any ethical risks caused by our work. The datasets we use are all publicly available. We do not annotate any data on our own. All our datasets have user reviews in the English language. We performed human evaluation experiments on Amazon Mechanical Turk (AMT). The human annotators were compensated at a rate of $15 per hour. During the evaluation, human annotators were not exposed to any sensitive or explicit content.

heading: 8 Acknowledgments
text: This work was supported in part by NSF grant DRL2112635. Any opinions, findings, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF. We also thank anonymous reviewers for their constructive reviews.

heading: A Appendix
text: A.1 Human Evaluation
The human annotators are required to be in the United States, have HIT Approval Rate greater than 98, and be masters. The screenshot of the human evaluation interface is shown in Figure 4.
We further perform the human evaluation on 150 aspect-specific summaries in the same manner as human evaluation on general summaries. For aspect-specific summaries, we conduct human evaluation based on two criteria: aspect informativeness and aspect specificity. The results are shown in Table 12. We observe that TokenCluster does not outperform SemAE and QT on aspect-specific summarization.
Figure 3 shows the best-worst scaling scores of TokenCluster and SemAE per aspect. TokenCluster outperforms SemAE on the location and building aspects but performs poorly on the cleanliness, food, rooms, and service aspects. We compare the aspect-specific summaries of these two models and find two potential reasons for the suboptimal performance on these aspects. First, for the service aspect and the room aspect, most sentences containing corresponding aspect seed words describe the same thing. For example, the retrieved sentences for the service aspect repeat ‘The staff are friendly and helpful’. The extracted sentences of TokenCluster and SemAE are very similar. Therefore, the clustering-based extraction strategy becomes less effective. Second, for the food aspect and the cleanliness aspect, the aspect seed words are too general when querying these aspects. Some clusters of aspect-related words might correspond to irrelevant aspects. For example, ‘good’ is an aspect-seed word for the food aspect. Some sentences containing ‘good’ are not related to the food aspect and might constitute irrelevant clusters during clustering. The same problem also exists for the cleanliness aspect since ‘nice’ is an aspect-seed word for the cleanliness domain. Since clustering-based extraction is less
robust to these unrelated sentences compared to centroid-based extraction, the generated summaries of TokenCluster is less specific compared to the generated summaries of SemAE. On the contrary, the sentences retrieved by the aspect-seed words of the building aspect and the location aspect are more diverse. For example, the retrieved sentences of the building aspect can describe a pool, balcony, or lounge. Centroid-based extraction strategy suffers from repetition in this situation. Besides, for these two aspects, the corresponding seed words are more specific and do not include general words like ‘good’ or ‘nice’. Therefore, TokenCluster outperforms SemAE on these two aspects.
A.2 Qualitative Examples Table 13 shows the clusters of aspect-related words and the summaries generated by TokenCluster. For general summaries, we observe that most clusters corresponds to certain aspects, which helps TokenCluster generate summaries cover more aspects. For the aspect-specific summary, we observe that some clusters are not closely related to the query, which undermines TokenCluster QT’s performance on aspect-specific summarization.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.1
3 B1. Did you cite the creators of artifacts you used? 4.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. 4.1, 7
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4.1
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 4.1,7
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4.1,7
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1
C 3 Did you run computational experiments? 4
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We use the existing models. Model structure is not the focu of paper
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix, 4.2
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4.6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 7
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We use public dataset.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. We use public dataset.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? It is not availabel on mturk.

heading: Future Work
text: Future works can focus on more robust ways of extracting and partitioning aspect-related words. This can possibly be achieved using external data. Another limitation is that TokenCluster is computationally more expensive compared to more simple sentence extraction approaches. This is because TokenCluster clusters aspect-related words during inference, which can be restrictive when the review set is large