Abstract: Short text clustering is a challenging problem due to its sparseness of text representation. Here we propose a flexible Self-Taught Convolutional neural network framework for Short Text Clustering (dubbed STC^2), which can flexibly and successfully incorporate more useful semantic features and learn non-biased deep text representation in an unsupervised manner. In our framework, the original raw text features are firstly embedded into compact binary codes by using one existing unsupervised dimensionality reduction methods. Then, word embeddings are explored and fed into convolutional neural networks to learn deep feature representations, meanwhile the output units are used to fit the pre-trained binary codes in the training process. Finally, we get the optimal clusters by employing K-means to cluster the learned representations. Extensive experimental results demonstrate that the proposed framework is effective, flexible and outperform several popular clustering methods when tested on three public short text datasets.

Introduction: Short text clustering is of great importance due to its various applications, such as user profiling BIBREF0 and recommendation BIBREF1 , for nowaday's social media dataset emerged day by day. However, short text clustering has the data sparsity problem and most words only occur once in each short text BIBREF2 . As a result, the Term Frequency-Inverse Document Frequency (TF-IDF) measure cannot work well in short text setting. In order to address this problem, some researchers work on expanding and enriching the context of data from Wikipedia BIBREF3 or an ontology BIBREF4 . However, these methods involve solid Natural Language Processing (NLP) knowledge and still use high-dimensional representation which may result in a waste of both memory and computation time. Another way to overcome these issues is to explore some sophisticated models to cluster short texts. For example, Yin and Wang BIBREF5 proposed a Dirichlet multinomial mixture model-based approach for short text clustering. Yet how to design an effective model is an open question, and most of these methods directly trained based on Bag-of-Words (BoW) are shallow structures which cannot preserve the accurate semantic similarities. Recently, with the help of word embedding, neural networks demonstrate their great performance in terms of constructing text representation, such as Recursive Neural Network (RecNN) BIBREF6 , BIBREF7 and Recurrent Neural Network (RNN) BIBREF8 . However, RecNN exhibits high time complexity to construct the textual tree, and RNN, using the hidden layer computed at the last word to represent the text, is a biased model where later words are more dominant than earlier words BIBREF9 . Whereas for the non-biased models, the learned representation of one text can be extracted from all the words in the text with non-dominant learned weights. More recently, Convolution Neural Network (CNN), as the most popular non-biased model and applying convolutional filters to capture local features, has achieved a better performance in many NLP applications, such as sentence modeling BIBREF10 , relation classification BIBREF11 , and other traditional NLP tasks BIBREF12 . Most of the previous works focus CNN on solving supervised NLP tasks, while in this paper we aim to explore the power of CNN on one unsupervised NLP task, short text clustering. We systematically introduce a simple yet surprisingly powerful Self-Taught Convolutional neural network framework for Short Text Clustering, called STC INLINEFORM0 . An overall architecture of our proposed approach is illustrated in Figure FIGREF5 . We, inspired by BIBREF13 , BIBREF14 , utilize a self-taught learning framework into our task. In particular, the original raw text features are first embedded into compact binary codes INLINEFORM1 with the help of one traditional unsupervised dimensionality reduction function. Then text matrix INLINEFORM2 projected from word embeddings are fed into CNN model to learn the deep feature representation INLINEFORM3 and the output units are used to fit the pre-trained binary codes INLINEFORM4 . After obtaining the learned features, K-means algorithm is employed on them to cluster texts into clusters INLINEFORM5 . Obviously, we call our approach “self-taught” because the CNN model is learnt from the pseudo labels generated from the previous stage, which is quite different from the term “self-taught” in BIBREF15 . Our main contributions can be summarized as follows: This work is an extension of our conference paper BIBREF16 , and they differ in the following aspects. First, we put forward a general a self-taught CNN framework in this paper which can flexibly couple various semantic features, whereas the conference version can be seen as a specific example of this work. Second, in this paper we use a new short text dataset, Biomedical, in the experiment to verify the effectiveness of our approach. Third, we put much effort on studying the influence of various different semantic features integrated in our self-taught CNN framework, which is not involved in the conference paper. For the purpose of reproducibility, we make the datasets and software used in our experiments publicly available at the website. The remainder of this paper is organized as follows: In Section SECREF2 , we first briefly survey several related works. In Section SECREF3 , we describe the proposed approach STC INLINEFORM0 and implementation details. Experimental results and analyses are presented in Section SECREF4 . Finally, conclusions are given in the last Section.

Question: What were their performance results?
Answer: On SearchSnippets dataset ACC 77.01%, NMI 62.94%, on StackOverflow dataset ACC 51.14%, NMI 49.08%, on Biomedical dataset ACC 43.00%, NMI 38.18%