Painsight: An Extendable Opinion Mining Framework for Detecting Pain Points Based on Online Customer Reviews
Year: 2023

Abstract:
As the e-commerce market continues to expand and online transactions proliferate, customer reviews have emerged as a critical element in shaping the purchasing decisions of prospective buyers. Previous studies have endeavored to identify key aspects of customer reviews through the development of sentiment analysis models and topic models. However, extracting specific dissatisfaction factors remains a challenging task. In this study, we delineate the pain point detection problem and propose Painsight, an unsupervised framework for automatically extracting distinct dissatisfaction factors from customer reviews without relying on ground truth labels. Painsight employs pre-trained language models to construct sentiment analysis and topic models, leveraging attribution scores derived from model gradients to extract dissatisfaction factors. Upon application of the proposed methodology to customer review data spanning five product categories, we successfully identified and categorized dissatisfaction factors within each group, as well as isolated factors for each type. Notably, Painsight outperformed benchmark methods, achieving substantial performance enhancements and exceptional results in human evaluations.

heading: 1 Introduction
text: The thriving e-commerce market has rendered online customer reviews an indispensable factor in influencing the purchasing decisions of potential consumers (Zhu and Zhang, 2010; Kwahk and Kim, 2017; Dellarocas et al., 2007). These reviews offer invaluable insights for businesses, empowering them to refine their products and services (Alibasic and Popovic, 2021). The analysis of copious customer reviews to comprehend customer needs and pain points is of paramount importance for augmenting service quality and heightening customer satisfaction (Plotkina and Munzel, 2016; Eslami and Ghasemaghaei, 2018; Wu and Chang, 2020;
Berger et al., 2020). Pain points pertain to specific difficulties or problems encountered by customers while utilizing a product or service (Lee, 2014). These encompass emotional challenges arising from psychological demands and the incongruities between customers’ actual perceptions and their expectations of products and services. Identifying and addressing pain points serves as a crucial initial step in enhancing the quality of products and services (East et al., 2008; Ho-Dac et al., 2013; Geetha et al., 2017).
The increasing importance of customer reviews has spurred a wealth of research into comprehending their effects through the application of natural language processing methodologies, encompassing opinion mining, sentiment analysis, topic modeling, and keyword extraction (Eslami and Ghasemaghaei, 2018; Heng et al., 2018; De Geyndt et al., 2022). Nonetheless, the identification of specific attributes, such as pain points in customer review data, necessitates annotation tailored to each product and service. Owing to this constraint, prior studies have predominantly concentrated on devising methods for extracting keywords, a notion more expansive than pain points. Wang et al. (2018) extracted attributes from product specifications, while Klein et al. (2022) employed a BERT-based BIO tagging (Ramshaw and Marcus, 1995) model to extract aspect and opinion terms. More recently, researchers have shifted their focus toward aspectbased sentiment analysis (Bu et al., 2021), striving to analyze customers’ opinions at the granular aspect level, as opposed to mere classification into being positive or negative (Zhang et al., 2022). Moreover, Wu and Chang (2020) utilized Latent Dirichlet Allocation (LDA) to differentiate review types and explore topic-related negative emotions.
Existing research has mainly focused on extracting keywords and aspect terms from reviews or simply analyzing review types; nonetheless, such methodologies exhibit limitations. Firstly, pain
215
points are not only more specific but also contingent upon the product or service in question, in contrast to keywords or aspect terms. This is because customers may articulate disparate pain points utilizing identical keywords. Consequently, a comprehensive definition of pain points must be established prior to their detection (Forman et al., 2008; De Bonte and Fletcher, 2014; Wang et al., 2016). Secondly, even when equipped with a definition for pain points, the unique nature of pain points across products and services necessitates the ongoing annotation of new products, an endeavor that is both labor-intensive and costly (Saura et al., 2021). To overcome these limitations, an automated framework for the definition, extraction, and analysis of pain points from customer reviews is requisite, which is applicable to any product or service. In this study, we propose Painsight, an extendable opinion-mining framework for pain point detection, composed of a series of modules. Painsight implements a pipeline that conducts sentiment analysis and topic modeling using pre-trained language models, subsequently extracting pain points based on gradient-based attribution. When applied to customer reviews encompassing five categories of home appliances, Painsight effectively classified pain points emerging in diverse product groups. The extracted pain points exhibited substantial enhancements in performance, both quantitatively and qualitatively, in comparison to the results procured by the baseline model. The main contributions of this study can be summarized as follows:
• We propose Painsight, an automated and scalable opinion-mining framework explicitly tailored for pain point analysis.
• Painsight encompasses a comprehensive pipeline that executes sentiment analysis, topic modeling, and task-specific gradientbased attribution, drawing on a pre-trained language model.
• Painsight demonstrates both quantitatively and qualitatively exceptional performance in the accurate identification of pain points concerning sentiment and topic across different product groups.

heading: 2 Problem Statement
text: We aim to detect pain points in user-generated content (UGC), such as customer reviews, to identify customer discomforts and challenges (Cheng et al.,
2021). However, the absence of standardized criteria for determining what constitutes pain points, contingent on the target product or service, poses a challenge because pain points exhibit variation in scale (Humphreys and Wang, 2018). As a result, the precise definition of pain points pertinent to the target product or service is crucial.

heading: 2.1 Definition of Pain Point
text: Pain points arise from emotions customers experience while utilizing products and services, typically characterized by keywords reflecting negative opinions. However, not all negative keywords constitute pain points; rather, they signify complaints addressable through functional or procedural enhancements (Homburg and Fürst, 2007; Rawson et al., 2013). For instance, while "I tried vacuum A, and it’s not good" conveys dissatisfaction without actionable insight, "I used vacuum B, and the battery drains too fast" distinctly identifies "battery" as an area for potential improvement. The scope of pain points can be determined based on their prevalence among users, with a focus on addressing common concerns to develop product and service improvement strategies, while simultaneously uncovering infrequent, personalized demands for diversification. In this study, we concentrate on detecting pain points affecting the majority of users, with the prospect of future methodology advancements catering to minority-specific pain points.

heading: 2.2 Task Formulation
text: We first formulate pain point detection, a novel task proposed in this study. Given n customer review texts x = (x1,x2, . . . ,xn), the model aims to identify k pain points p1, . . . , pk within a specific product group review dataset X (x ∈ X ). The set of pain points in X is denoted as P(X ), with each pain point pk ∈ P(X ) comprising contiguous tokens xi, . . . , xi+t that form part of review text x. In real-world settings, ground truth pain points for each product group are not predefined. Consequently, extracting suitable pain point candidates from review data and selecting product-specific pain points is essential. However, executing pain point extraction in a fully unsupervised manner presents significant challenges. Therefore, we assume the existence of relevant tasks with strong dependencies on pain point detection, with supervision y provided for learning these tasks on the dataset. These relevant tasks serve as weak supervision, approximating pain point detection by mod-
eling contexts of product-specific review and incorporating valuable knowledge. Given that each review is grounded in customer perception, words with substantial influence on relevant task predictions can be understood as pain points of actual customers.
In this study, we utilized sentiment analysis and iterative topic modification (ITM) based on topic modeling as relevant tasks for pain point detection. Moreover, we aimed to extract keyword-based pain points using token attribution scores derived from these tasks. In both tasks, token attribution scores are computed through the following procedure. Given review x and its corresponding embedding sequence e = ( e1, e2, . . . , e|x| ) , we define the relevant task classifier f task, which takes the embedding sequence as input. The input gradient of each token can be used to evaluate the influence of the token for the target task, represented as a normalized gradient attribution vector a = ( a1, a2, . . . , a|x| ) . As described by Wang et al. (2020), the attribution at position i can be expressed as:
ai = |∇xiL · xi|∑ j ∣∣∇xjL · xj ∣∣ , (1)
where L denotes the loss generated by classifier f task, and ai is calculated through the dot product between the gradient of L and the embedding ei. Gradient-based attribution a (Sundararajan et al., 2017; Ross et al., 2017) represents each token’s influence on the final prediction and can thus approximate word importance (Feng et al., 2018).

heading: 3 Painsight
text: The primary objective of the proposed framework is to model the entire process of automatically detecting pain points in real-world scenarios. Specifically, this study aims to address the following two practical research questions:
• Q1: How do customers perceive products in general?
• Q2: What types of discomfort do customers experience?
Considering the significance of both perspectives from prior works, a framework capable of generating accurate and diverse output, covering a wide range of distinct pain points, is necessary. The architecture of Painsight, depicted in Figure 1, features a parallel structure to incorporate these two research questions.

heading: 3.1 Pre-processing & Filtering
text: We constructed a dataset by collecting various customer reviews from the web. As real-world customer reviews contain diverse noise, such as grammatical errors and outliers (i.e., irrelevant reviews), we applied three pre-processing steps for data refinement: i) spacing correction, ii) keyword dictionary construction, and iii) data filtering. Review data often consists of colloquial styles and may contain grammatical and spacing errors. We first utilized a pre-trained spacing model1 to correct spacing errors and employed regular expressions to fix common typos. Furthermore, as mentioned in Section 2.1, filtering out sentences without pain
1We employed a character-level convolution neural network-based spacing model.
points is essential. To achieve this, we constructed a stopwords list and a keyword dictionary to filter out sentences expressing sentiments irrelevant to pain point extraction. We considered words frequently appearing in negative reviews and not included in the stopwords list as important keywords. Additionally, we filtered out duplicate reviews or those with fewer than ten tokens.

heading: 3.2 Sentiment-Aware Pain Point
text: The Sentiment-Aware pain point approach focuses on negative emotions to address the question of "How do customers perceive products in general?". To achieve this goal, we conducted sentiment analysis on customer reviews using a classifier denoted as fsent. The sentiment label y is defined as y ∈ {positive, negative}, and ŷ represents the predicted class, expressed as ŷ = argmaxŷ f
sent(ŷ | x). Reviews predicted as ‘negative’ by the trained fsent were considered likely to contain pain points.

heading: 3.3 Topic-Aware Pain Point
text: The Topic-Aware pain point approach focuses on identifying specific types of discomfort that customers experience, addressing the question "What types of pain points do customers encounter?". We began by analyzing negative reviews to identify various types of complaints. To detect pain points by topic, we further refined sentences in outlier topics to properly segment them. The Topic-Aware pain point approach consists of three stages: i) topic modeling, ii) topic merging, and iii) ITM, as illustrated in Figure 2.
Topic modeling The construction of TopicAware pain points necessitates establishing ground
truth for each product group, which includes the number of topics and review-topic matching information. However, such labels are typically absent in customer reviews available on the web. Even when predefined pain points exist, detecting new pain points from real-time collected review data remains challenging. To address this issue, we first employed topic modeling to determine the initial topic class. Specifically, we utilized BERTopic (Grootendorst, 2022) to identify significant clustered topics. However, we observed that over 50% of the data were classified as outliers, defined in this study as reviews devoid of any distinct topic, with more than 100 topics extracted. Misclassified topic modeling results can impede interpretability and provide users with incorrect pain points. To tackle this problem, we merged topics to relabel misclassified ones and performed ITM to accurately distinguish outlier reviews with low relevance to pain points.
Topic merging In the above paragraph, we addressed the issue of reviews with similar topics being assigned to different clusters in the BERTopic output. To resolve this challenge, we propose a procedure for selecting representative words for each topic and determining which topics should be merged. Initially, we extracted nouns with the highest c-TF-IDF scores for each topic and designated them as representative words. Words with scores at least s were considered keywords for the product group.2 Subsequently, we searched for topics containing these keywords and merged them into a single topic. Following the topic merging process, we observed that semantically similar topics were combined, resulting in roughly 20 to 30 merged topics.
2We set the hyperparameter s to 0.1 based on experimental results.
We also adjusted minor topic classes, which consisted of 5 to 10 reviews. However, the described topic merging approach primarily focuses on lexical matching assigned to the topics. Therefore, incorporating additional improvements that consider semantic aspects is crucial for refining the results.
Iterative topic modification Despite the improvements in topic modeling results through merging, over 50% of reviews still remain outliers. To address this problem, it is essential to assign initial topics that may be misclassified and to distinguish between reviews containing pain points and outlier reviews in the data. Consequently, we propose an ITM algorithm to enhance topic modeling results by updating the predicted topic with a confidence threshold above a certain level if it differs from the existing topic class during the training process. Our ITM algorithm is inspired by curriculum pseudo labeling (CPL) (Zhang et al., 2021), a method designed for semi-supervised learning that flexibly adjusts the threshold based on the learning difficulty for each class. Similar to CPL, the ITM algorithm adjusts the number of updated data points by varying the threshold according to the classification difficulty for each topic during the training process. As a result, this approach enables the modification of misclassified topic classes and induces additional topic merging.
Let the classifier for ITM be denoted as f itmt , where t represents the current time-step of the classifier, and f itmt (yt | x) represents the prediction probability of yt. yt is the topic class at time t for review x and belongs to the topic set C = topic1, topic2, . . . , topicM . We used the merged topic output assigned to each review as the initial topic y0. τ denotes a pre-defined threshold3, and the threshold, T (yt), for label modification based on f itmt can be defined as:
T (yt) = R(yt) · τ. (2)
T (yt) is a flexible threshold for topic class yt at time-step t, and R(yt) is a relative recall value representing topic class difficulty. In CPL, accuracy is used as a measure of difficulty; however, accuracy can lead to biased estimates due to class imbalance in our topic modeling. As an alternative, we utilize recall, a sensitivity measure, to define the difficulty of the topic class. High recall indicates an easy class with high T (y), while low recall implies a
3τ is a hyperparameter set experimentally within the search space [0.4, 0.5, 0.6, 0.7].
difficult class with low T (y). Class difficulty is defined as:
R(yt) = recall(yt)
maxyt∈C(recall(yt)) . (3)
We train f itmt to maximize log-likelihood based on the topic at t−1 and calculate the difficulty of each class T (yt) at every time-step4. If the predicted probability f itmt (yt | x) is greater than T (yt), we modify the topic to yt. yt is defined as:
yt = { yt if f itmt (yt | x) > T (yt), yt−1, otherwise.
(4)
Upon initializing the training with initial topics, we terminated the process when topics no longer merged or reached a satisfactory state5. We then considered the final prediction of ITM as the topic for each review.

heading: 3.4 Gradient-based Attribution
text: In this study, we aim to extract word importance related to pain points from two interdependent tasks: sentiment analysis and topic modeling. We employed gradient-based attribution, using token attribution scores for each task’s prediction. Normalized attribution vectors asenti and a itm i for individual tokens are derived from trained classifiers fsent and f itm, as shown in Eq. (1):
asenti = |∇xiLsent · xi|∑ j ∣∣∇xjLsent · xj ∣∣ , (5) aitmi = |∇xiLtopic · xi|∑ j ∣∣∇xjLtopic · xj ∣∣ . (6)
We computed asenti for ‘negative’ reviews and a itm i for reviews in M−1 topics, excluding outlier topics. We then selected the top-g asenti and a itm i with the highest attribution vectors and extracted words with index i. These words form the pain point candidate set C(X ). The pain point set P(X ) is a subset of C(X ). We employed Captum (Kokhlikyan et al., 2020)’s integrated gradient method for axiomatic attribution calculation.

heading: 3.5 Post-Processing
text: The derived C(X ), when used as the final result for pain points, has certain limitations. Since asenti and aitmi are token attribution scores contributing to
4In this study, we set each epoch as a time-step. 5Meaning two out of three evaluation metrics no longer
show improvement.
each class prediction, they tend to focus on words describing "states," such as verbs and adjectives like "not good" and "bad." To include both the subject and object related to the predicate’s action, additional post-processing is necessary. We addressed this issue by refining the results using a dependency parsing model (Heo et al., 2021), a method that captures relationships between words in a sentence.
First, we sorted the top g tokens with the highest attribution scores from ftask and extracted the corresponding words wi. Then, we analyzed word relationships within sentences using dependency parsing. If wi belongs to a noun phrase (NP), we define it as C(X )′; if it belongs to a verb phrase (VP), we add the related NP to C(X )′. If the related word is a VP, we search for an NP to supplement C(X )′. We calculate word frequencies in the stopword-filtered C(X )′ and define the top N words as the pain points P(X ) for each product group6. Examples of post-processing can be found in Figure 3.

heading: 4 Experiment setup

heading: 4.1 Dataset
text: We utilized Korean customer reviews on five home appliance categories gathered from various web sources between January 2020 and November 2021. The target categories included dryers, stylers, washing machines, vacuum cleaners, and robotic vacu-
6We set g to 3, the number of P(X )[sent to 30, and the number of P(X )topic to 10 for each topic.
ums, obtained from 17 sites spanning five source types: retail, blog, cafe, community, and news. The dataset, originally collected by a home appliance company in Korea, had sentiment labels assigned based on the company’s internal sentiment analysis logic within their voice of customer (VOC) analysis system. Data distribution and statistics for each product group are illustrated in Figure 4 and Table 1. Detailed distribution of product categories can be found in Appendix A.

heading: 4.2 Baselines

heading: 4.2.1 Sentiment Analysis
text: TextCNN (Kim, 2014) utilizes 1D convolutions to capture variable-length local features and n-grams. Hierarchical Attention Network (HAN) (Yang et al., 2016) consists of a single-layer Gated Recurrent Unit (GRU) that utilizes attention mechanisms at the word, sentence, and document levels to extract important information effectively. BERT (Devlin et al., 2019)is a bidirectional pretrained language model 7. BERT + AVocaDo (Hong et al., 2021) is an extension of BERT with AVocaDo technique. It is designed to handle product-specific review data by performing vocab expansion on a domain-specific corpus. The expansion process is based on tokenizer training and calculation of the sub-word segmentation ratio for the domain corpus, with contrastive learning between the original tokenizer and the expanded one during fine-tuning. BERT + Vocab Expansion manually adds frequently occurring vocabulary from the review to the BERT model. High-frequency words were iden-
7The ‘klue/bert-base’ version was employed.
tified using a count-based approach, and those not present in BERT vocabulary were added.

heading: 4.2.2 Topic Modeling
text: LDA (Blei et al., 2003) is a method inferring topics from word occurrence patterns in a corpus. While LDA provides multiple topics for each review, for comparison with Painsight, we considered the highest probability topic as the representative. BERTopic is a BERT-based topic model 8 to generate document embeddings, clustering and creates representations using the c-TF-IDF procedure.

heading: 4.2.3 Pain Point Detection
text: spaCy (Honnibal and Montani, 2017) is baseline using part-of-speech tagging to extract nouns within sentences. The same post-processing method as Painsight was applied.

heading: 5 Experiment Results

heading: 5.1 Sentiment Analysis
text: Table 2 presents the sentiment analysis performance across the five product groups. Given that review data is domain-specific, the language model can benefit from a vocabulary expansion process (Hong et al., 2021). We aimed to enhance performance by applying AVocaDo and Vocab Expansion to the original BERT model. The experimental results showed that the BERT + Vocab Expansion model achieved superior performance and was thus selected as the final model for calculating attribution scores.

heading: 5.2 Topic Modeling
text: Table 3 presents the topic modeling performance for the five product groups. We evaluated the performance of LDA, BERTopic, and ITM using
8In this study, we used the "sentence-transformers/xlm-r100langs-bert-base-nli-stsb-mean-tokens" model.
coherence-based clustering metrics such as NPMI. LDA generates the desired number of topics without considering outliers, resulting in significantly lower performance compared to BERTopic and ITM. Conversely, BERTopic designates, on average, 57% of the reviews as outliers and assigns topics only to the remaining data. This results in BERTopic being evaluated with a smaller number of reviews compared to other methods, which could create a more favorable environment for BERTopic’s performance measurements. In contrast, our ITM algorithm effectively assigns reviews initially deemed as outliers to suitable topics, achieving comparable or superior performance to BERTopic in most cases. These results demonstrate that ITM successfully reduced the average proportion of outliers to 32%, even under relatively
challenging experimental conditions.

heading: 5.3 Pain Point Detection
text: As discussed in Section 2.2, real-world customer reviews lack gold labels for pain points. Thus, to validate the effectiveness of Painsight in detecting sentiment-aware and topic-aware pain points, we conducted human evaluations. We designed experiments for three tasks, each addressing the following research questions:
• Task 1: Can the sentiment-aware pain point module extract appropriate pain point candidates from each sentence?
• Task 2: Can the sentiment-aware pain point module identify suitable pain points for each product category?
• Task 3: Can the topic-aware pain point module detect appropriate pain points for specific topics within each product category?
We conducted human evaluations for the three tasks across five product groups, with eight unique human judges participating in each experiment. In Task 1, we randomly selected 100 sentences per product category and used spaCy and Painsight to identify pain point candidates in each sentence. Judges assessed the appropriateness of the extracted candidates on a scale from 1 to 5, where 1 signifies ‘all words are extracted incorrectly’ and 5 indicates ‘all words are extracted correctly’. In Task 2, we provided 100 randomly sampled example sentences for each product category and examined the adequacy of the final pain points identified for each category. Judges evaluated each pain point word, assigning a score of 0 for unsuitable and 1 for suitable. For Task 3, we randomly selected 20 example sentences per product category according to topics and assessed the appropriateness of the detected pain points for the corresponding topics. Judges assigned a score of 0 for unsuitable and 1 for suitable pain point words.
Table 4 presents the human evaluation results across the three tasks. First, in Task 1 (Table 4 - (a)), which concentrates on extracting pain point candidates from each review sentence, Painsight’s approach — extracting words contributing to ‘negative’ predictions based on high attribution scores — outperforms spaCy’s noun extraction, with an average improvement of 0.65 points. In Task 2 (Table 4 - (b)), the performance of the final sentimentaware pain points is evaluated. These pain points
are obtained by sorting the pain point candidates from Task 1 by frequency across all product categories. Assessing the output for each word reveals an average improvement of 0.17 points across the five product categories. Lastly, Task 3 (Table 4 - (c)) examines the final results of the topic-aware pain points by extracting pain points for each topic within the product categories. This assessment considers the relevance of the final pain points for each topic, and Painsight records the highest performance across the five product categories. Our method exhibits an average performance improvement of 0.35 points over LDA and 0.14 points over BERTopic. The results in Table 4 demonstrate the effectiveness of ITM, which could not be solely assessed using the NPMI metrics in Table 3. By employing Tasks 1, 2, and 3, the Painsight pipeline, which extracts pain points based on relevant tasks, also records higher performance compared to the baseline in human evaluation results. This validates the appropriateness of Painsight as an automatic framework for pain point detection.

heading: 6 Conclusion
text: In this study, we propose Painsight, a novel framework for automatically extracting and evaluating pain points from customer reviews. We address the under-explored problem of pain point detection and
present a practical pipeline for real-world scenarios. By employing sentiment analysis and topic modeling, we identify sentiment-aware and topic-aware pain points that reflect customer perceptions and various types of discomfort. The final output is obtained by extracting the most important words or features from the data using a gradient-based attribution score. This score enables us to determine which words or features are most critical in influencing the model’s decision-making process and utilize this information in the post-process to recognize more accurate and meaningful pain points. Experimental results demonstrate that Painsight outperforms existing models on five product group reviews, with human evaluation results indicating a high level of agreement compared to the baseline. Future work could involve incorporating diverse customer feedback and constructing a high-quality benchmark dataset to further validate and enhance the proposed approach.
Limitations
In prior research (Salminen et al., 2022), several challenges have been identified in this field, such as noisy or low-quality data, semantic ambiguity, absence of standards, social desirability bias, and the requirement for human intervention. Our study aimed to tackle the challenge of detecting pain points and devised various strategies for managing noisy real-world reviews. Nonetheless, to fully unlock the potential of the Painsight, additional research is necessary to explore the wide range of emotional polarities beyond the generic ‘negative’ sentiment. Furthermore, customer reviews often show mixed sentiments, which calls for addressing semantic ambiguity. Lastly, the performance of Painsight assessment was constrained to five product categories, highlighting the need for a comprehensive, high-quality benchmark encompassing diverse domains and performance evaluations across distinct categories.
Ethics Statement
Throughout our human evaluation, we collected demographic details such as name, age, gender, and highest education level, after securing participants’ consent and assuring them that their information would be exclusively utilized for research purposes. The results from the human evaluation were anonymized to protect participant confidentiality. The authors meticulously examined all customer
reviews employed in the assessment, verifying the absence of any offensive or biased material. Participants took part in the evaluation for an estimated 40 minutes. They were compensated with a 5,000 KRW (equivalent to 3.7 USD) gift card, which was marginally above the Korean minimum wage during that period.

heading: Acknowledgements
text: We express our gratitude to all DSBA Lab members for their support. We also thank Keonwoo Kim, Joonwon Jang, Minjin Jeon, and Hyowon Cho for their insightful discussions and valuable feedback that greatly contributed to the refinement of this paper

heading: A Data Distribution
text: The distribution of site types used for collecting review data for five product categories (Dryer, Styler, Washing Machine, Vacuum Cleaner, and Robotic Vacuum) is presented in Figure 5. The data collection involved a diverse range of site types across product categories, with the highest number of reviews collected from cafe and community sites. The sentiment-aware pain point detection experiments utilized train, valid, and test datasets in an 8:1:1 ratio. However, the topic modeling in topicaware pain point detection did not involve splitting the dataset.

heading: B Training Details
text: The training process of Painsight was conducted using NVIDIA RTX 2080 Ti (for sentiment analysis and topic modeling) and RTX A6000 (for gradientbased attribution score and ITM). The PyTorch 9 library was performed for model training, and postprocessing was carried out using Pororo 10 and Captum 11. The hyperparameters used in Painsight are described in detail below:
9https://pytorch.org/ 10https://github.com/kakaobrain/pororo 11https://captum.ai/
B.1 BERT + Vocab Expansion
The model was trained using the transformers library (Wolf et al., 2019) with a BERT classifier. The batch size was set to 32, and the optimizer used was AdamW with a learning rate of 2e-05. The model was trained for 10 epochs, with 62 new vocabularies added. The maximum length of input sequences was set to 128.
B.2 ITM
The initial topic labels were generated using BERTopic, and the transformers library with a BERT classifier was used for classification. The optimizer used was Adam, and the model was trained for 1 epoch with a validation iteration of 10 and a maximum iteration of 100. Early stopping was used with patience 2. The maximum length of input sequences was set to 128, and the batch size was 64 with a learning rate of 3e-5.

heading: C Evaluation Metric
text: We employed Normalized Pointwise Mutual Information (NPMI) (Stevens et al., 2012) as an evaluation metric to measure the performance of ITM. NPMI is a widely used measure of the correlation between two words, which is computed by normalizing Pointwise Mutual Information (PMI). PMI measures the probability of two words occurring together, taking into account the frequency of their individual occurrences. However, PMI tends to overestimate the importance of infrequent words. To address this issue, NPMI normalizes PMI by considering the probability of the respective words. Through this normalization process, NPMI can more accurately measure the correlation between two words and ranges between -1 and 1. NPMI is often used in topic modeling and is computed using the following formula:
PMI(wi, wj) = P (wi, wj)
P (wi)P (wj) , (7)
NPMI(wi, wj) = PMI(wi, wj)
− log(P (wi, wj)) . (8)
Here, P (wi, wj) denotes the probability of words wi and wj co-occurring, while P (wi) and P (wj) represent their individual probabilities. The numerator normalizes the probability of the two words occurring together by dividing it by the product of their individual occurrence probabilities. The denominator uses the log value of their co-occurrence probability to obtain PMI. Using NPMI, we can extract sets of related words in topic classes classified through ITM and evaluate if each topic has coherence. Therefore, we evaluated the consistency of each topic in LDA, BERTopic, and ITM with a set of related reviews using NPMI.

heading: D Human Evaluation
text: The evaluation instructions provided to annotators for each task in human evaluation are as follows, and the example is shown for the dryer.
D.1 Task 1: Sentiment-aware pain point evaluation (Pain point candidates)
Each sheet contains three items for each product group:
• Consumer reviews of appliances for each product group collected online
• Keywords extracted for each review (2-4 per review)
Please rate how well the keywords were extracted from each customer review:
• 1: All keywords were extracted incorrectly. • 2: Keywords were generally not extracted.
• 3: Keywords were extracted at an average level (50% of all keywords).
• 4: Keywords were generally well extracted.
• 5: All keywords were extracted well.
D.2 Task 2: Sentiment-aware pain point evaluation (Final pain points)
Evaluate the suitability of pain points identified as frequently appearing in negative reviews of the product category after reviewing 100 example sentences. To clarify the concept of "pain point," we provide example sentences (Figure 6):
Scoring criteria:
• 0: This pain point cannot be considered a pain point for the product category.
• 1: This pain point can be considered a pain point for the product category.
Notes:
• Pain points may be composed of morphemes or spaced units rather than conventional word structures, selected based on their meaning.
• There may be cases where a pain point cannot be extracted according to the logic. In such cases, you can give a score of 1.
D.3 Task 3: Topic-aware pain point evaluation
The following example contains three elements:
• Consumer reviews on dryers collected from online community
• Review examples for each topic class
• Key pain points for each topic class
20 reviews are provided for each topic class. Please assess the reviews in detail and evaluate whether the pain point can be used as a representative pain point for the topic class, using the following scores:
• 0: The pain point cannot represent the topic class.
• 1: The pain point can represent the topic class.
Notes:
• Pain points can be constructed at the morpheme level, rather than a common word structure, depending on their meaning.
• Do not consider overlaps in meaning or form between pain points. Please only judge the representativeness of each pain point for the reviews on the topic.
• If the topic modeling performance is low, the topics of each review may not match. If the meaning of the topic cannot be identified through the sampled sentences for each topic class, you can assign 0 points to all the pain points.
• Each topic often includes more than 500- 1,000 sentences. Some pain points may be not contained in the sampled review. If similar words to that pain points, however, they could be suitable pain points for its cluster. ([ex] "I tried it myself and my wrist hurts", pain point (‘arms’: considered correct))

heading: Future Work
text: Future work could involve incorporating diverse customer feedback and constructing a high-quality benchmark dataset to further validate and enhance the proposed approach. Limitations
In prior research (Salminen et al. , 2022), several challenges have been identified in this field, such as noisy or low-quality data, semantic ambiguity, absence of standards, social desirability bias, and the requirement for human intervention. Our study aimed to tackle the challenge of detecting pain points and devised various strategies for managing noisy real-world reviews. Nonetheless, to fully unlock the potential of the Painsight, additional research is necessary to explore the wide range of emotional polarities beyond the generic ‘negative’ sentiment. Furthermore, customer reviews often show mixed sentiments, which calls for addressing semantic ambiguity. Lastly, the performance of Painsight assessment was constrained to five product categories, highlighting the need for a comprehensive, high-quality benchmark encompassing diverse domains and performance evaluations across distinct categories