Venue: NAACL 2022

# _What Do Users Care About?_

Detecting Actionable Insights from User Feedback

Kasturi Bhattacharjee\({}^{1}\), Rashmi Gangadharaiah\({}^{1}\), Kathleen McKeown\({}^{1,2}\), Dan Roth\({}^{1,3}\)

###### Abstract

Users often leave feedback on a myriad of aspects of a product which, if leveraged successfully, can help yield useful insights that can lead to further improvements down the line. Detecting actionable insights can be challenging owing to large amounts of data as well as the absence of labels in real-world scenarios. In this work, we present an aggregation and graph-based ranking strategy for unsupervised detection of these insights from real-world, noisy, user-generated feedback. Our proposed approach significantly outperforms strong baselines on two real-world user feedback datasets and one academic dataset.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Collecting vast amounts of user feedback on products and services is a common practice these days for a multitude of companies. This can prove to be a rich resource for product owners to improve the quality of their product offerings, correct failures and gauge the general performance of their product from both implicit and explicit signals that might be present in the feedback. However, in most cases, including the one we address in this work, the feedback is unstructured and voluminous, and can therefore remain underutilized for the most part. In our particular use-case, we receive thousands of textual feedback daily on average 1, and it is time-consuming and laborious for product owners to manually extract actionable insights from them.

Footnote 1: We take our responsibility to protect customer content extremely seriously. For this research, we followed Amazon’s Customer Content policy guidelines.

Users leave feedback on a variety of aspects they experience in the course of using the product (Table 1). These consist of functionalities they find useful (_calculate total size_), issues they encounter when attempting to perform an action (_scroll sideways_), requests around enabling certain features (_sort by Date Modified_), and so on. In this work, our goal is to capture short informative phrases, or _themes_ which reflect _actionable_ insights in the feedback. In capturing these actionable insights, we wish to focus on desired actions that users want to be able to carry out (e.g. _scroll_ sideways).

Owing to the influx of this data in large amounts, and the cost of annotations, it often remains _unlabeled_. Therefore, in this work, we present an _unsupervised_ framework to detect actionable insights from such data. In order to capture these insights from an aggregated view of the data, we propose the following two-step approach: a) _aggregating_ similar feedback such that each cluster represents coherent insights; b) _detecting themes_ from clusters that are pertinent to actionable insights. For instance, in Figure 1, the red cluster consists of feedback expressing users' need to be able to download several files at one time, for which a _possible_ theme could be _download files_. As seen in these examples, the desired themes may consist of non-contiguous tokens appearing in text and typically contain a verb mentioning the action. Prior work (see Section 2) uses keyphrase extraction which extracts contiguous words within a noun phrase and thus, cannot capture the kind of actionable insights we find in our data. We utilize unsupervised clustering algorithms for the aggregation step, and a graph-based ranking strategy for the theme detection step.

[TABLE:S4.T1][ENDTABL



**Contributions of the paper:**

* We propose a novel approach of identifying actionable user insights in an unsupervised manner. We do so by framing the problem as a clustering and cluster theme detection problem.
* Our approach is unique in the utilization of graph-based ranking in the identification of cluster themes, especially focused on capturing actions that users want to perform.
* We find our method to significantly outperform baselines on two real-world datasets and one academic dataset.

## 2 Related Work

There is limited work that focuses on uncovering insights from _real-world_ user feedback data. Most of the work involves labeled academic datasets, requiring approaches that involve some form of supervision. For instance, Lin et al. (2012) involves a weakly supervised joint sentiment-topic model that detects sentiment and topic simultaneously from text, applied to two labeled academic datasets. Approaches that are unsupervised Qiu et al. (2021); Liu et al. (2010) are largely based on Latent Dirichlet Allocation (LDA) Blei et al. (2003), which yields distribution of unigrams as topics. Although Qiu et al. (2021) adopt an unsupervised strategy, it has not been explored on user feedback data. These approaches are not directly applicable to our problem setting since we require short phrases focusing on the performance of an _action_ by the user.

Approaches such as TextRank Kazemi et al. (2020), SingleRank Wan and Xiao (2008), ExpandRank Wan and Xiao (2008), TopicRank Bougouin et al. (2013), TopicalPageRank Sterckx et al. (2015), PositionRank Florescu and Caragea (2017), Bi-LSTM-CRF Sequence Labeling Alzaidy et al. (2019), FACE Chau et al. (2020), and MultipartiteRank Boudin (2018) have been applied to the task of key phrase extraction from documents as opposed to theme detection from clusters. For cluster-labeling, there has been work around using keyword extraction from clusters, utilizing WordNet synsets to expand the keywords, followed by a selection procedure to assign the final label Poostchi and Piccardi (2018); Chang and McKeown (2019). None of these approaches focus on the extraction of short _actionable_ phrases, a.k.a. _themes_, as is our use case. In most of these approaches, candidate key phrases are assumed to appear in contiguous positions in a document and are concatenated to form phrases. As described earlier (Figure 1), it is unrealistic to make such assumptions on real-world user feedback data and our proposed approach considers non-contiguous candidate phrases as well.

## 3 Data

In this work, we use two internal unlabeled datasets containing user feedback in English on two product offerings. The feedback collection pipeline has opt-in and opt-out mechanisms in place to allow the user to decide whether their data can be used for further analysis. User-specific information has been removed from these datasets for privacy and confidentiality reasons. These datasets vary in content based on the specific product they contain feedback about, and in the number of feedback documents contained in each. We refer to them as \({\bf Prod}_{1}\) and \({\bf Prod}_{2}\). \({\bf Prod}_{1}\) received orders of magnitude more feedback than \({\bf Prod}_{2}\). For exploration purposes we sampled about 10K documents for \({\bf Prod}_{1}\) and 1.5K documents for \({\bf Prod}_{2}\). In addition,

\begin{table}
\begin{tabular}{|c|c|c|} \hline \multirow{2}{*}{**Split**} & \multirow{2}{*}{**\# of samples**} & \multicolumn{2}{l|}{**\# of samples**} \\  & & **per intent** \\ \hline \hline Train & 15K & 100 \\ \hline Dev & 3K & 20 \\ \hline Test & 4.5K & 30 \\ \hline \end{tabular}
\end{table}
Table 2: CLINC150 Data Statistics

Figure 1: Our pipeline is illustrated above. User feedback documents are first grouped into various clusters. Here, two clusters (blue and red), each representing different insights are shown. For each cluster, we automatically identify a “theme” (right part of the figure) that concisely captures the desired actions expressed in the cluster, e.g. _download files_ for the red cluster; _view graphs_ for the blue cluster. <_MASK_> tokens are inserted to maintain anonymity when displaying the examples above.



we conduct evaluations and report results on an intent classification dataset - **CLINC150** (CC 3.0) Larson et al. (2019); Zhang et al. (2020) that contains English utterances labeled with one of 150 intents, thereby containing _document-level_ labels. The data contains utterances from 10 domains, e.g. Banking, Travel, Kitchen & Dining etc. Table 3 contains utterance examples. The intent labels in this data (e.g. _find phone, book hotel, schedule maintenance, etc._) are similar in form to the cluster themes we aim to discover from our product feedback data, which makes it a good candidate for evaluating our approach. We obtain proxy ground-truth labels (details in Section 5.3) from the data to evaluate and report metrics on this dataset. **Note that the labels were used for the sole purpose of evaluation and not training, since the real-world use case is in an unsupervised setting.** We use the same train/dev/test splits provided with this dataset, statistics of which are reported in Table 2.

## 4 Methodology

In this section, we describe our proposed approach for extracting themes that help in discovering insights from user-generated text (outlined in Figure 2). In order to discover coherent emerging insights from the vast amounts of user-generated data, we first aggregate semantically similar feedback using clustering algorithms, and extract a _representative set_ of documents per cluster. This is described in Section 4.1. Thereafter, themes for these clusters are generated as detailed in Section 4.2. The entire pipeline is unsupervised.

### Document Aggregation & Representative Set Selection

Documents are embedded using Sentence-BERT (SBERT) Reimers and Gurevych (2019), which we find to capture semantic similarity well even for our internal datasets. k-means MacQueen et al. (1967) is selected as our clustering approach of choice. Additionally, we explore the use of DNN-based clustering approaches such as Deep Embedded Clustering (**DEC**) Xie et al. (2016) which has been shown to outperform k-means for a few academic text classification datasets (e.g. REUTERS Lewis et al. (2004)). When applied to our real-world user-generated text, we find DEC to perform well on the larger dataset, Prod\({}_{1}\) but not on the smaller dataset Prod\({}_{2}\) (Table 4), while k-means performs well across both datasets. Thus, for the remainder of the paper, we report results using k-means as the clustering strategy.

Based on the hypothesis that the centroid of a cluster is representative of the overall cluster itself, we rank documents based on their proximity to the centroid for each cluster. 10 documents closest to the centroid per cluster are subsequently considered for cluster label detection. We chose 10 documents as it provided a good balance between having good representative members of the cluster while also ensuring that we have very few noisy cluster members, if any. Henceforth, we refer to these as the **representative set** of a cluster.

### Unsupervised Cluster Theme Identification

Here, we describe the procedure for cluster theme identification. As previously mentioned, our themes could consist of non-contiguous tokens. We begin by extracting candidate themes using a dependency parsing approach, followed by a graph-based ranking strategy to assign a theme per cluster.

Figure 2: Outline of proposed approach.

\begin{table}
\begin{tabular}{c|p{85.4pt}}
**Intent Label** & **Document** \\ \hline \hline findphone & _i need your help finding my lost phone_ \\ \hline book hotel & _i’m inquiring about the availability of a room that fits 10 people from monday to tuesday in manhattan_ \\ \hline schedule maintenance & _please find someone who specializes in cars, my check engine light has turned on_ \\ \end{tabular}
\end{table}
Table 3: Examples with various intent labels from **CLINC150** Dataset. **Note that labels are utilized merely for evaluation purposes.

 



#### 4.2.1 Cluster Theme Candidate Extraction

* **Noun Phrase Extraction:** Nouns, Proper Nouns and Noun Phrases are extracted from the representative set per cluster. All pronouns that occur in the beginning/end of noun phrases are removed, e.g. _my check engine light_ converted to _check engine light_. This is done to capture more generic noun phrases. Those that occur > 2 times throughout the ranked set are retained.
* **Dependency Parsing:** We run a Dependency Parser and extract all verbs for which any of the above selected set of nouns and noun phrases are a _nominal subject_ or _object_. For instance in Fig. 3, _tire pressure_ is an object for verb _tell_.
* **Candidate Theme Extraction:** Phrases of the form <VERB, NOUN> are thus constructed. For cases where the Noun is part of a previously selected noun phrase, we expand the phrases to <VERB, NP>. These act as _candidate themes_ for a given cluster.

#### 4.2.2 Graph-based Ranking for Theme Identification

Graph-based ranking algorithms are often employed in approaches for unsupervised document summarization to measure the importance of a sentence for inclusion in a summary Erkan and Radev (2004); Zheng and Lapata (2019). We apply a similar approach for cluster theme detection. Inspired by these approaches, we construct a graph per cluster, where the nodes consist of the candidate themes and the edge weights capture the semantic similarity between pairs of themes, obtained using cosine similarity between SBERT embeddings of corresponding themes. We then use a graph-based ranking strategy, PageRank Brin and Page (1998) and assign the phrase with the highest rank as the cluster theme.

## 5 Experiments

In this Section, we describe the experimental details of the pipeline, and provide details on the baselines we compare with.

### Data Processing

Since user-generated text tends to be noisy in nature, we preprocess our internal datasets beforehand. This includes converting to lowercase, removing URLs, special characters, and removing text consisting only of digits. For CLINC150, the only pre-processing performed is to replace underscores in the intent labels with spaces, i.e. _book_hotel_ converted to _book hotel_, since the generated themes are of a similar form.

### Baselines

We use two baselines to compare with, which are described below.

Poostchi and Piccardi (P&P)This work proposes an approach for cluster labeling by leveraging word embeddings and the synonymy and hypernymy relations in the WordNet Miller (1995) lexical ontology. Similar to Chang and McKeown who adapt this method for their clusters, we perform the following steps for each of our clusters. We extract keywords using RAKE Rose et al. (2010) from the representative set per cluster, to ensure a fair comparison with our methodology. Hypernyms of the component words (restricted to Nouns) of these keywords are obtained, expanded by synonyms (via _synsets_, WordNet's synonym sets). We use _CentHyp_ - the best strategy as per Poostchi and Piccardi, to assign the final cluster label. This selects hypernyms that are most central w.r.t. the centroid of the cluster. SBERT embeddings are used for this purpose, to ensure a fair comparison with our approach. Since we could not find official code for this work, we used our own implementation.

Random baselineWe also compare with a random baseline in which the cluster theme is selected at random from the generated set of cluster themes using our proposed approach (Section 4.2.1).

\begin{table}
\begin{tabular}{|c|c|c|} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c|}{**Mean Accuracy Score (Human Eval)**} \\ \cline{2-3}  & **k-means** & **DEC** \\ \hline \hline Prod\({}_{1}\) & 90.0 & 80.0 \\ \hline Prod\({}_{2}\) & 65.0 & 45.0 \\ \hline \end{tabular}
\end{table}
Table 4: k-means and DEC clustering algorithms compared on the internal datasets. Annotators provide a score of 0 or 1 to the clusters, based on whether they agree with the quality. Average accuracy per annotator is computed. Scores reported in this table are an average of the annotator scores.

Figure 3: Example of dependency parsing output on CLINC150 utterance.





### Evaluation Strategies & Metrics Reported

CLINC150For this dataset, we leverage the document-level intent labels to generate a _proxy_ groundtruth label per cluster. The same representative set (as in Section 4.1) is selected per cluster and the cluster theme is determined by a majority vote over the intent labels of these documents. For the rare case where there is no clear majority, we consider the label of the centroid to be the cluster label. We compute METEOR Denkowski and Lavie (2014) and BERTScore Zhang et al. (2019) metrics w.r.t. the model outputs and proxy labels for our baselines as well as the proposed approach.

Internal DatasetsSince \({\text{Prod}}_{1}\) and \({\text{Prod}}_{2}\) do not contain labels either on the document or cluster level, we utilize human annotations to evaluate the efficacy of our methodology w.r.t. the baselines. We employ 2 internal annotators who are presented with the cluster themes generated on both datasets, and the representative set of documents per cluster that the themes were detected from. An annotator votes 1 if they completely agree with the cluster theme, 0 if they completely disagree. The final scores are an average of the scores of both annotators. IAA (Cohen's kappa) score is 0.72.

### Modeling Details

We use Stanford Stanza Qi et al. (2020) for POS tagging and dependency parsing, and scikit-learn's Pedregosa et al. (2011) k-means implementation for clustering. Using input number of clusters to be the same as the number of intent labels \(k=150\) yields the best results for CLINC150. For our internal datasets, \(k=150\) for \({\text{Prod}}_{1}\) and \(k=25\) for \({\text{Prod}}_{2}\) are used. For PageRank, we use networkx's Hagberg et al. (2008) package, with maximum number of iterations set to 100. CPU-based computing instances are used for both baselines and our methodology.

## 6 Results

Model performance on CLINC150Table 5 illustrates the performance of the baseline models and our proposed approach on the dev and test splits of the CLINC150 dataset, over 3 experimental runs.

Dev setStatistical significance tests conducted show that on the dev set, our methodology significantly outperforms random baseline on METEOR

\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Dataset** & **Method** & \multicolumn{1}{l|}{
\begin{tabular}{c} **Human** \\ **Evaluation Score** \\ \end{tabular} } \\ \hline \hline \multirow{3}{*}{\({\text{Prod}}_{1}\)} & P\&P & 32.89 \\  & Random & 29.70 \\  & **Ours** & **72.60** \\ \hline \multirow{3}{*}{\({\text{Prod}}_{2}\)} & P\&P & 16.67 \\  & Random & 8.00 \\ \cline{1-1}  & **Ours** & **40.00** \\ \hline \end{tabular}
\end{table}
Table 6: Performance of the proposed approach (Ours) against baselines, P&P Poostchi and Piccardi) and RAND on our internal datasets.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline
**Data** & **Method** & **METEOR** & **BERTScore** \\ \hline \hline \multirow{4}{*}{Dev} & P\&P & 21.30 \\  & & \(\pm 0.69\) & \(\pm 0.0033\) \\  & Random & 21.12 \\  & & \(\pm 0.27\) & \(\pm 0.0021\) \\  & **Ours** & **25.79** & **0.8962** \\  & & \(\pm 0.55\) & \(\pm 0.0023\) \\ \hline \multirow{4}{*}{Test} & P\&P & 20.94 \\  & & \(\pm 1.37\) & \(\pm 0.0034\) \\  & Random & 19.59 & 0.8955 \\  & & \(\pm 1.47\) & \(\pm 0.0023\) \\  & **Ours** & **25.31** & **0.9026** \\  & & \(\pm 0.69\) & \(\pm 0.0018\) \\ \hline \end{tabular} 
\begin{tabular}{|c|c|c|} \hline
**(Proxy) Cluster** & \multicolumn{1}{l|}{\begin{tabular}{c} **Baseline** \\ **(P\&P)** \\ **Prediction** \\ \end{tabular} } & \multicolumn{1}{l|}{\begin{tabular}{c} **Our** \\ **Prediction** \\ \end{tabular} } \\ \hline pay bill & electric bill & **pay bill** \\ \hline calendar & check & \multicolumn{1}{l|}{\begin{tabular}{c} **check** \\ **calendar** \\ \end{tabular} } \\ \hline meeting schedule & today & \multicolumn{1}{l|}{\begin{tabular}{c} **schedule** \\ **meeting** \\ \end{tabular} } \\ \hline insurance change & insurance policy & \multicolumn{1}{l|}{\begin{tabular}{c} **update** \\ **insurance** \\ **policy** \\ \end{tabular} } \\ \hline shopping list & \multicolumn{1}{l|}{\begin{tabular}{c} **shopping** \\ **list** \\ \end{tabular} } & buy milk \\ \hline change accent & \multicolumn{1}{l|}{
\begin{tabular}{c} **change** \\ \end{tabular} } & change voice \\ \hline \end{tabular}
\end{table}
Table 7: Comparing cluster theme predictions from P&P Poostchi and Piccardi) & our approach on CLINC150 test set. Themes in bold are those that are qualitatively most similar to the proxy cluster theme.



score by 4.67 points on average (p-value 0.0004). On mean BERTScore, we outperform the random baseline by 0.0041 points (p-value 0.1449). Compared with [25], we find our methodology to yield a significantly better performance on METEOR score - an increase of 4.49 points (p-value 0.003). Further, we outperform Poostchi and Piccardi (2018) on BERTScore by 0.007 points (p-value 0.148).

Test set:We significantly outperform both baselines - random and Poostchi and Piccardi (2018), by 5.72 points (p-value 0.0076) and 4.37 points (p-value 0.0159) respectively, on METEOR score. Performance gains obtained using our method over both baselines for BERTScore are also statistically significant - an increase of 0.0071 points (p-value 0.0293) w.r.t. random baseline and that of 0.0114 points (p-value 0.0132) w.r.t. Poostchi and Piccardi (2018).

Model performance on Internal datasetsTable 6 demonstrates the significant boost in performance our proposed approach provides over the baselines, as measure by human evaluation scores. We find our approach to outperform both baselines by a large margin for both datasets. On Prod\({}_{1}\), we improve upon the random baseline by 42.9 points and upon Poostchi and Piccardi (2018) by 39.71 points. For Prod\({}_{2}\), we obtain an improvement of 32 points w.r.t. the random baseline and 23.33 points as compared to Poostchi and Piccardi (2018).

Error AnalysisIn Tables 7 and 8, we present examples comparing the cluster theme predictions from Poostchi and Piccardi (2018) with ours on the CLINC150 test set, and our internal datasets, respectively. Our method is able to yield more descriptive phrases as cluster themes, that help capture the action being performed. In comparison, the baseline captures shorter and less descriptive phrases. For instance, our method generates themes such as **pay bill** and **update insurance policy** on clusters from the CLINC150 test set, where the corresponding baseline themes are _electric bill_ and _insurance policy_, respectively. Similarly, for a cluster from the internal dataset (Prod\({}_{1}\)), the theme assigned by our proposed approach is **show alarms**, whereas the theme detected by the baseline method is _alarms_.

## 7 Conclusion & Future Work

This work addresses the problem of discovering actionable insights from unlabeled real-world user feedback data, in an unsupervised fashion. Data is clustered into groups containing coherent insights, followed by theme detection per cluster using a graph-based ranking approach. Experiments conducted on two real-world user feedback datasets as well as an academic dataset show our proposed approach to significantly outperform baselines by a large margin. In the future, we would expand the scope of our work to datasets with other characteristics and distributions (e.g. review datasets) to study

\begin{table}
\begin{tabular}{c|c|c} \hline  & **Representative set of documents per cluster** & **P\&Pours** \\ \hline \hline _- The new monitoring tab is very poor. It doesn’t show enough data_ & & \\ _and you can’t click on graphs to get detailed views._ & & \\ _- I cannot see the monitoring of 2 <MASK> <MASK> side by side._ & & \\ _Unreliable reporting of metrics, graphs are sometime unavailable._ & & \\ _- i can’t even select two <MASK> and view their monitoring graphs at the same time._ & & \\ \hline _- Need to see the <MASK> alarms setup per <MASK> like before instead of loosing this functionality in a new <MASK>._ & & \\ _- I miss the alarm status ” ”button” that shows all alarms connected_ & & \\ _to ¡MASK> <MASK>_ & & \\ _- On the new ¡MASK> dashboard all alarms connected to ¡MASK>_ & & \\ _!MASK? do not show as it does in the old console. On the old console you get a direct view if any of your ¡MASK> has any issues while we on the new get that there are no alarms for the ¡MASK¿._ & & \\ \hline \end{tabular}
\end{table}
Table 8: Comparing cluster themes from the baseline P&P method [25] vs ours. The documents are in ascending order of proximity to centroid. Text in bold highlights themes that best capture the action being performed by a user.

 

the applicability of our approach to those use-cases. Further, we would explore the use of generative models to obtain abstractive themes from data.

## References

* Alzaidy et al. (2019) Rabah Alzaidy, Cornelia Caragea, and C. Lee Giles. 2019. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The World Wide Web Conference, WWW '19, New York, NY, USA. Association for Computing Machinery.
* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. _J. Mach. Learn. Res._, 3(null):993-1022.
* Boudin (2018) Florian Boudin. 2018. Unsupervised keyphrase extraction with multipartite graphs. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 667-672, New Orleans, Louisiana. Association for Computational Linguistics.
* Bougouin et al. (2013) Adrien Bougouin, Florian Boudin, and Beatrice Daille. 2013. TopicRank: Graph-based topic ranking for keyphrase extraction. In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_, pages 543-551, Nagoya, Japan. Asian Federation of Natural Language Processing.
* Brin and Page (1998) Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual web search engine. _Computer networks and ISDN systems_, 30(1-7):107-117.
* Chang and McKeown (2019) Serina Chang and Kathleen McKeown. 2019. Automatically inferring gender associations from language. _arXiv preprint arXiv:1909.00091_.
* Chau et al. (2020) Hung Chau, Igor Labutov, Khushboo Thaker, Daqing He, and Peter Brusilovsky. 2020. Automatic concept extraction for domain and student modeling in adaptive textbooks. _International Journal of Artificial Intelligence in Education_, pages \(1-27\).
* Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the EACL 2014 Workshop on Statistical Machine Translation_.
* Erkan and Radev (2004) Gunes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. _Journal of artificial intelligence research_, 22:457-479.
* Florescu and Caragea (2017) Corina Florescu and Cornelia Caragea. 2017. PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1105-1115, Vancouver, Canada. Association for Computational Linguistics.
* Hagberg et al. (2008) Aric Hagberg, Pieter Swart, and Daniel S Chult. 2008. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States).
* Kazemi et al. (2020) Ashkan Kazemi, Veronica Perez-Rosas, and Rada Mihalcea. 2020. Biased textrank: Unsupervised graph-based content extraction.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. _arXiv preprint arXiv:1909.02027_.
* Lewis et al. (2004) David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. 2004. Rev1: A new benchmark collection for text categorization research. _Journal of machine learning research_, 5(Apr):361-397.
* Lin et al. (2012) Chenghua Lin, Yuan He, Richard Everson, and Stefan Ruger. 2012. Weakly supervised joint sentiment-topic detection from text. _IEEE Transactions on Knowledge and Data Engineering_, 24(6):1134-1145.
* Liu et al. (2010) Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010. Automatic keyphrase extraction via topic decomposition. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 366-376, Cambridge, MA. Association for Computational Linguistics.
* MacQueen et al. (1967) James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_, volume 1, pages 281-297. Oakland, CA, USA.
* Miller (1995) George A. Miller. 1995. Wordnet: A lexical database for english. _Commun. ACM_, 38(11):39-41.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Poostchi and Piccardi (2018) Hanieh Poostchi and Massimo Piccardi. 2018. Cluster labeling by word embeddings and WordNet's hypernymymy. In _Proceedings of the Australasian Language Technology Association Workshop 2018_, pages 66-70, Dunedin, New Zealand.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
* Qiu et al. (2021) John Xi Qiu, Adam Faulkner, and Aysu Ezen Can. 2021. Towards theme detection in personal finance questions.

 

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. _arXiv preprint arXiv:1908.10084_.
* Rose et al. (2010) Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic keyword extraction from individual documents. _Text mining: applications and theory_, 1:1-20.
* Sterckx et al. (2015) Lucas Sterckx, Thomas Demeester, Johannes Deleu, and Chris Develder. 2015. Topical word importance for fast keyphrase extraction. In _Proceedings of the 24th International Conference on World Wide Web_, WWW '15 Companion, page 121-122, New York, NY, USA. Association for Computing Machinery.
* Volume 2_, AAAI'08, page 855-860. AAAI Press.
* Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In _International conference on machine learning_, pages 478-487. PMLR.
* Zhang et al. (2020) Jian-Guo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. _arXiv preprint arXiv:2010.13009_.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zheng and Lapata (2019) Hao Zheng and Mirella Lapata. 2019. Sentence centrality revisited for unsupervised summarization. _arXiv preprint arXiv:1906.03508_.