Self-Taught Convolutional Neural Networks for Short Text Clustering

Abstract: Short text clustering is a challenging problem due to its sparseness of text representation. Here we propose a flexible Self-Taught Convolutional neural network framework for Short Text Clustering (dubbed STC^2), which can flexibly and successfully incorporate more useful semantic features and learn non-biased deep text representation in an unsupervised manner. In our framework, the original raw text features are firstly embedded into compact binary codes by using one existing unsupervised dimensionality reduction methods. Then, word embeddings are explored and fed into convolutional neural networks to learn deep feature representations, meanwhile the output units are used to fit the pre-trained binary codes in the training process. Finally, we get the optimal clusters by employing K-means to cluster the learned representations. Extensive experimental results demonstrate that the proposed framework is effective, flexible and outperform several popular clustering methods when tested on three public short text datasets.

Short text clustering is of great importance due to its various applications, such as user profiling BIBREF0 and recommendation BIBREF1 , for nowaday's social media dataset emerged day by day. However, short text clustering has the data sparsity problem and most words only occur once in each short text BIBREF2 . As a result, the Term Frequency-Inverse Document Frequency (TF-IDF) measure cannot work well in short text setting. In order to address this problem, some researchers work on expanding and enriching the context of data from Wikipedia BIBREF3 or an ontology BIBREF4 . However, these methods involve solid Natural Language Processing (NLP) knowledge and still use high-dimensional representation which may result in a waste of both memory and computation time. Another way to overcome these issues is to explore some sophisticated models to cluster short texts. For example, Yin and Wang BIBREF5 proposed a Dirichlet multinomial mixture model-based approach for short text clustering. Yet how to design an effective model is an open question, and most of these methods directly trained based on Bag-of-Words (BoW) are shallow structures which cannot preserve the accurate semantic similarities.
Recently, with the help of word embedding, neural networks demonstrate their great performance in terms of constructing text representation, such as Recursive Neural Network (RecNN) BIBREF6 , BIBREF7 and Recurrent Neural Network (RNN) BIBREF8 . However, RecNN exhibits high time complexity to construct the textual tree, and RNN, using the hidden layer computed at the last word to represent the text, is a biased model where later words are more dominant than earlier words BIBREF9 . Whereas for the non-biased models, the learned representation of one text can be extracted from all the words in the text with non-dominant learned weights. More recently, Convolution Neural Network (CNN), as the most popular non-biased model and applying convolutional filters to capture local features, has achieved a better performance in many NLP applications, such as sentence modeling BIBREF10 , relation classification BIBREF11 , and other traditional NLP tasks BIBREF12 . Most of the previous works focus CNN on solving supervised NLP tasks, while in this paper we aim to explore the power of CNN on one unsupervised NLP task, short text clustering.
We systematically introduce a simple yet surprisingly powerful Self-Taught Convolutional neural network framework for Short Text Clustering, called STC INLINEFORM0 . An overall architecture of our proposed approach is illustrated in Figure FIGREF5 . We, inspired by BIBREF13 , BIBREF14 , utilize a self-taught learning framework into our task. In particular, the original raw text features are first embedded into compact binary codes INLINEFORM1 with the help of one traditional unsupervised dimensionality reduction function. Then text matrix INLINEFORM2 projected from word embeddings are fed into CNN model to learn the deep feature representation INLINEFORM3 and the output units are used to fit the pre-trained binary codes INLINEFORM4 . After obtaining the learned features, K-means algorithm is employed on them to cluster texts into clusters INLINEFORM5 . Obviously, we call our approach “self-taught” because the CNN model is learnt from the pseudo labels generated from the previous stage, which is quite different from the term “self-taught” in BIBREF15 . Our main contributions can be summarized as follows:
This work is an extension of our conference paper BIBREF16 , and they differ in the following aspects. First, we put forward a general a self-taught CNN framework in this paper which can flexibly couple various semantic features, whereas the conference version can be seen as a specific example of this work. Second, in this paper we use a new short text dataset, Biomedical, in the experiment to verify the effectiveness of our approach. Third, we put much effort on studying the influence of various different semantic features integrated in our self-taught CNN framework, which is not involved in the conference paper.
For the purpose of reproducibility, we make the datasets and software used in our experiments publicly available at the website.
The remainder of this paper is organized as follows: In Section SECREF2 , we first briefly survey several related works. In Section SECREF3 , we describe the proposed approach STC INLINEFORM0 and implementation details. Experimental results and analyses are presented in Section SECREF4 . Finally, conclusions are given in the last Section.
In this section, we review the related work from the following two perspectives: short text clustering and deep neural networks.
There have been several studies that attempted to overcome the sparseness of short text representation. One way is to expand and enrich the context of data. For example, Banerjee et al. BIBREF3 proposed a method of improving the accuracy of short text clustering by enriching their representation with additional features from Wikipedia, and Fodeh et al. BIBREF4 incorporate semantic knowledge from an ontology into text clustering. However, these works need solid NLP knowledge and still use high-dimensional representation which may result in a waste of both memory and computation time. Another direction is to map the original features into reduced space, such as Latent Semantic Analysis (LSA) BIBREF17 , Laplacian Eigenmaps (LE) BIBREF18 , and Locality Preserving Indexing (LPI) BIBREF19 . Even some researchers explored some sophisticated models to cluster short texts. For example, Yin and Wang BIBREF5 proposed a Dirichlet multinomial mixture model-based approach for short text clustering. Moreover, some studies even focus the above both two streams. For example, Tang et al. BIBREF20 proposed a novel framework which enrich the text features by employing machine translation and reduce the original features simultaneously through matrix factorization techniques.
Despite the above clustering methods can alleviate sparseness of short text representation to some extent, most of them ignore word order in the text and belong to shallow structures which can not fully capture accurate semantic similarities.
Recently, there is a revival of interest in DNN and many researchers have concentrated on using Deep Learning to learn features. Hinton and Salakhutdinov BIBREF21 use DAE to learn text representation. During the fine-tuning procedure, they use backpropagation to find codes that are good at reconstructing the word-count vector.
More recently, researchers propose to use external corpus to learn a distributed representation for each word, called word embedding BIBREF22 , to improve DNN performance on NLP tasks. The Skip-gram and continuous bag-of-words models of Word2vec BIBREF23 propose a simple single-layer architecture based on the inner product between two word vectors, and Pennington et al. BIBREF24 introduce a new model for word representation, called GloVe, which captures the global corpus statistics.
In order to learn the compact representation vectors of sentences, Le and Mikolov BIBREF25 directly extend the previous Word2vec BIBREF23 by predicting words in the sentence, which is named Paragraph Vector (Para2vec). Para2vec is still a shallow window-based method and need a larger corpus to yield better performance. More neural networks utilize word embedding to capture true meaningful syntactic and semantic regularities, such as RecNN BIBREF6 , BIBREF7 and RNN BIBREF8 . However, RecNN exhibits high time complexity to construct the textual tree, and RNN, using the layer computed at the last word to represent the text, is a biased model. Recently, Long Short-Term Memory (LSTM) BIBREF26 and Gated Recurrent Unit (GRU) BIBREF27 , as sophisticated recurrent hidden units of RNN, has presented its advantages in many sequence generation problem, such as machine translation BIBREF28 , speech recognition BIBREF29 , and text conversation BIBREF30 . While, CNN is better to learn non-biased implicit features which has been successfully exploited for many supervised NLP learning tasks as described in Section SECREF1 , and various CNN based variants are proposed in the recent works, such as Dynamic Convolutional Neural Network (DCNN) BIBREF10 , Gated Recursive Convolutional Neural Network (grConv) BIBREF31 and Self-Adaptive Hierarchical Sentence model (AdaSent) BIBREF32 .
In the past few days, Visin et al. BIBREF33 have attempted to replace convolutional layer in CNN to learn non-biased features for object recognition with four RNNs, called ReNet, that sweep over lower-layer features in different directions: (1) bottom to top, (2) top to bottom, (3) left to right and (4) right to left. However, ReNet does not outperform state-of-the-art convolutional neural networks on any of the three benchmark datasets, and it is also a supervised learning model for classification. Inspired by Skip-gram of word2vec BIBREF34 , BIBREF23 , Skip-thought model BIBREF35 describe an approach for unsupervised learning of a generic, distributed sentence encoder. Similar as Skip-gram model, Skip-thought model trains an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded sentence and released an off-the-shelf encoder to extract sentence representation. Even some researchers introduce continuous Skip-gram and negative sampling to CNN for learning visual representation in an unsupervised manner BIBREF36 . This paper, from a new perspective, puts forward a general self-taught CNN framework which can flexibly couple various semantic features and achieve a good performance on one unsupervised learning task, short text clustering.
Assume that we are given a dataset of INLINEFORM0 training texts denoted as: INLINEFORM1 , where INLINEFORM2 is the dimensionality of the original BoW representation. Denote its tag set as INLINEFORM3 and the pre-trained word embedding set as INLINEFORM4 , where INLINEFORM5 is the dimensionality of word vectors and INLINEFORM6 is the vocabulary size. In order to learn the INLINEFORM7 -dimensional deep feature representation INLINEFORM8 from CNN in an unsupervised manner, some unsupervised dimensionality reduction methods INLINEFORM9 are employed to guide the learning of CNN model. Our goal is to cluster these texts INLINEFORM10 into clusters INLINEFORM11 based on the learned deep feature representation while preserving the semantic consistency.
As depicted in Figure FIGREF5 , the proposed framework consist of three components, deep convolutional neural network (CNN), unsupervised dimensionality reduction function and K-means module. In the rest sections, we first present the first two components respectively, and then give the trainable parameters and the objective function to learn the deep feature representation. Finally, the last section describe how to perform clustering on the learned features.
In this section, we briefly review one popular deep convolutional neural network, Dynamic Convolutional Neural Network (DCNN) BIBREF10 as an instance of CNN in the following sections, which as the foundation of our proposed method has been successfully proposed for the completely supervised learning task, text classification.
Taking a neural network with two convolutional layers in Figure FIGREF9 as an example, the network transforms raw input text to a powerful representation. Particularly, each raw text vector INLINEFORM0 is projected into a matrix representation INLINEFORM1 by looking up a word embedding INLINEFORM2 , where INLINEFORM3 is the length of one text. We also let INLINEFORM4 and INLINEFORM5 denote the weights of the neural networks. The network defines a transformation INLINEFORM6 INLINEFORM7 which transforms an input raw text INLINEFORM8 to a INLINEFORM9 -dimensional deep representation INLINEFORM10 . There are three basic operations described as follows:
Wide one-dimensional convolution This operation INLINEFORM0 is applied to an individual row of the sentence matrix INLINEFORM1 , and yields a resulting matrix INLINEFORM2 , where INLINEFORM3 is the width of convolutional filter.
Folding In this operation, every two rows in a feature map are simply summed component-wisely. For a map of INLINEFORM0 rows, folding returns a map of INLINEFORM1 rows, thus halving the size of the representation and yielding a matrix feature INLINEFORM2 . Note that folding operation does not introduce any additional parameters.
Dynamic INLINEFORM0 -max pooling Assuming the pooling parameter as INLINEFORM1 , INLINEFORM2 -max pooling selects the sub-matrix INLINEFORM3 of the INLINEFORM4 highest values in each row of the matrix INLINEFORM5 . For dynamic INLINEFORM6 -max pooling, the pooling parameter INLINEFORM7 is dynamically selected in order to allow for a smooth extraction of higher-order and longer-range features BIBREF10 . Given a fixed pooling parameter INLINEFORM8 for the topmost convolutional layer, the parameter INLINEFORM9 of INLINEFORM10 -max pooling in the INLINEFORM11 -th convolutional layer can be computed as follows: DISPLAYFORM0 
where INLINEFORM0 is the total number of convolutional layers in the network.
As described in Figure FIGREF5 , the dimensionality reduction function is defined as follows: DISPLAYFORM0 
where, INLINEFORM0 are the INLINEFORM1 -dimensional reduced latent space representations. Here, we take four popular dimensionality reduction methods as examples in our framework.
Average Embedding (AE): This method directly averages the word embeddings which are respectively weighted with TF and TF-IDF. Huang et al. BIBREF37 used this strategy as the global context in their task, and Socher et al. BIBREF7 and Lai et al. BIBREF9 used this method for text classification. The weighted average of all word vectors in one text can be computed as follows: DISPLAYFORM0 
where INLINEFORM0 can be any weighting function that captures the importance of word INLINEFORM1 in the text INLINEFORM2 .
Latent Semantic Analysis (LSA): LSA BIBREF17 is the most popular global matrix factorization method, which applies a dimension reducing linear projection, Singular Value Decomposition (SVD), of the corresponding term/document matrix. Suppose the rank of INLINEFORM0 is INLINEFORM1 , LSA decompose INLINEFORM2 into the product of three other matrices: DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are the singular values of INLINEFORM2 , INLINEFORM3 is a set of left singular vectors and INLINEFORM4 is a set of right singular vectors. LSA uses the top INLINEFORM5 vectors in INLINEFORM6 as the transformation matrix to embed the original text features into a INLINEFORM7 -dimensional subspace INLINEFORM8 BIBREF17 .
Laplacian Eigenmaps (LE): The top eigenvectors of graph Laplacian, defined on the similarity matrix of texts, are used in the method, which can discover the manifold structure of the text space BIBREF18 . In order to avoid storing the dense similarity matrix, many approximation techniques are proposed to reduce the memory usage and computational complexity for LE. There are two representative approximation methods, sparse similarity matrix and Nystr INLINEFORM0 m approximation. Following previous studies BIBREF38 , BIBREF13 , we select the former technique to construct the INLINEFORM1 local similarity matrix INLINEFORM2 by using heat kernel as follows: DISPLAYFORM0 
where, INLINEFORM0 is a tuning parameter (default is 1) and INLINEFORM1 represents the set of INLINEFORM2 -nearest-neighbors of INLINEFORM3 . By introducing a diagonal INLINEFORM4 matrix INLINEFORM5 whose entries are given by INLINEFORM6 , the graph Laplacian INLINEFORM7 can be computed by ( INLINEFORM8 ). The optimal INLINEFORM9 real-valued matrix INLINEFORM10 can be obtained by solving the following objective function: DISPLAYFORM0 
where INLINEFORM0 is the trace function, INLINEFORM1 requires the different dimensions to be uncorrelated, and INLINEFORM2 requires each dimension to achieve equal probability as positive or negative).
Locality Preserving Indexing (LPI): This method extends LE to deal with unseen texts by approximating the linear function INLINEFORM0 BIBREF13 , and the subspace vectors are obtained by finding the optimal linear approximations to the eigenfunctions of the Laplace Beltrami operator on the Riemannian manifold BIBREF19 . Similar as LE, we first construct the local similarity matrix INLINEFORM1 , then the graph Laplacian INLINEFORM2 can be computed by ( INLINEFORM3 ), where INLINEFORM4 measures the local density around INLINEFORM5 and is equal to INLINEFORM6 . Compute the eigenvectors INLINEFORM7 and eigenvalues INLINEFORM8 of the following generalized eigen-problem: DISPLAYFORM0 
The mapping function INLINEFORM0 can be obtained and applied to the unseen data BIBREF38 .
All of the above methods claim a better performance in capturing semantic similarity between texts in the reduced latent space representation INLINEFORM0 than in the original representation INLINEFORM1 , while the performance of short text clustering can be further enhanced with the help of our framework, self-taught CNN.
The last layer of CNN is an output layer as follows: DISPLAYFORM0 
where, INLINEFORM0 is the deep feature representation, INLINEFORM1 is the output vector and INLINEFORM2 is weight matrix.
In order to incorporate the latent semantic features INLINEFORM0 , we first binary the real-valued vectors INLINEFORM1 to the binary codes INLINEFORM2 by setting the threshold to be the media vector INLINEFORM3 . Then, the output vector INLINEFORM4 is used to fit the binary codes INLINEFORM5 via INLINEFORM6 logistic operations as follows: DISPLAYFORM0 
All parameters to be trained are defined as INLINEFORM0 . DISPLAYFORM0 
Given the training text collection INLINEFORM0 , and the pre-trained binary codes INLINEFORM1 , the log likelihood of the parameters can be written down as follows: DISPLAYFORM0 
Following the previous work BIBREF10 , we train the network with mini-batches by back-propagation and perform the gradient-based optimization using the Adagrad update rule BIBREF39 . For regularization, we employ dropout with 50% rate to the penultimate layer BIBREF10 , BIBREF40 .
With the given short texts, we first utilize the trained deep neural network to obtain the semantic representations INLINEFORM0 , and then employ traditional K-means algorithm to perform clustering.
We test our proposed approach on three public short text datasets. The summary statistics and semantic topics of these datasets are described in Table TABREF24 and Table TABREF25 .
SearchSnippets. This dataset was selected from the results of web search transaction using predefined phrases of 8 different domains by Phan et al. BIBREF41 .
StackOverflow. We use the challenge data published in Kaggle.com. The raw dataset consists 3,370,528 samples through July 31st, 2012 to August 14, 2012. In our experiments, we randomly select 20,000 question titles from 20 different tags as in Table TABREF25 .
Biomedical. We use the challenge data published in BioASQ's official website. In our experiments, we randomly select 20, 000 paper titles from 20 different MeSH major topics as in Table TABREF25 . As described in Table TABREF24 , the max length of selected paper titles is 53.
For these datasets, we randomly select 10% of data as the development set. Since SearchSnippets has been pre-processed by Phan et al. BIBREF41 , we do not further process this dataset. In StackOverflow, texts contain lots of computer terminology, and symbols and capital letters are meaningful, thus we do not do any pre-processed procedures. For Biomedical, we remove the symbols and convert letters into lower case.
We use the publicly available word2vec tool to train word embeddings, and the most parameters are set as same as Mikolov et al. BIBREF23 to train word vectors on Google News setting, except of vector dimensionality using 48 and minimize count using 5. For SearchSnippets, we train word vectors on Wikipedia dumps. For StackOverflow, we train word vectors on the whole corpus of the StackOverflow dataset described above which includes the question titles and post contents. For Biomedical, we train word vectors on all titles and abstracts of 2014 training articles. The coverage of these learned vectors on three datasets are listed in Table TABREF32 , and the words not present in the set of pre-trained words are initialized randomly.
In our experiment, some widely used text clustering methods are compared with our approach. Besides K-means, Skip-thought Vectors, Recursive Neural Network and Paragraph Vector based clustering methods, four baseline clustering methods are directly based on the popular unsupervised dimensionality reduction methods as described in Section SECREF11 . We further compare our approach with some other non-biased neural networks, such as bidirectional RNN. More details are listed as follows:
K-means K-means BIBREF42 on original keyword features which are respectively weighted with term frequency (TF) and term frequency-inverse document frequency (TF-IDF).
Skip-thought Vectors (SkipVec) This baseline BIBREF35 gives an off-the-shelf encoder to produce highly generic sentence representations. The encoder is trained using a large collection of novels and provides three encoder modes, that are unidirectional encoder (SkipVec (Uni)) with 2,400 dimensions, bidirectional encoder (SkipVec (Bi)) with 2,400 dimensions and combined encoder (SkipVec (Combine)) with SkipVec (Uni) and SkipVec (Bi) of 2,400 dimensions each. K-means is employed on the these vector representations respectively.
Recursive Neural Network (RecNN) In BIBREF6 , the tree structure is firstly greedy approximated via unsupervised recursive autoencoder. Then, semi-supervised recursive autoencoders are used to capture the semantics of texts based on the predicted structure. In order to make this recursive-based method completely unsupervised, we remove the cross-entropy error in the second phrase to learn vector representation and subsequently employ K-means on the learned vectors of the top tree node and the average of all vectors in the tree.
Paragraph Vector (Para2vec) K-means on the fixed size feature vectors generated by Paragraph Vector (Para2vec) BIBREF25 which is an unsupervised method to learn distributed representation of words and paragraphs. In our experiments, we use the open source software released by Mesnil et al. BIBREF43 .
Average Embedding (AE) K-means on the weighted average vectors of the word embeddings which are respectively weighted with TF and TF-IDF. The dimension of average vectors is equal to and decided by the dimension of word vectors used in our experiments.
Latent Semantic Analysis (LSA) K-means on the reduced subspace vectors generated by Singular Value Decomposition (SVD) method. The dimension of subspace is default set to the number of clusters, we also iterate the dimensions ranging from 10:10:200 to get the best performance, that is 10 on SearchSnippets, 20 on StackOverflow and 20 on Biomedical in our experiments.
Laplacian Eigenmaps (LE) This baseline, using Laplacian Eigenmaps and subsequently employing K-means algorithm, is well known as spectral clustering BIBREF44 . The dimension of subspace is default set to the number of clusters BIBREF18 , BIBREF38 , we also iterate the dimensions ranging from 10:10:200 to get the best performance, that is 20 on SearchSnippets, 70 on StackOverflow and 30 on Biomedical in our experiments.
Locality Preserving Indexing (LPI) This baseline, projecting the texts into a lower dimensional semantic space, can discover both the geometric and discriminating structures of the original feature space BIBREF38 . The dimension of subspace is default set to the number of clusters BIBREF38 , we also iterate the dimensions ranging from 10:10:200 to get the best performance, that is 20 on SearchSnippets, 80 on StackOverflow and 30 on Biomedical in our experiments.
bidirectional RNN (bi-RNN) We replace the CNN model in our framework as in Figure FIGREF5 with some bi-RNN models. Particularly, LSTM and GRU units are used in the experiments. In order to generate the fixed-length document representation from the variable-length vector sequences, for both bi-LSTM and bi-GRU based clustering methods, we further utilize three pooling methods: last pooling (using the last hidden state), mean pooling and element-wise max pooling. These pooling methods are respectively used in the previous works BIBREF45 , BIBREF27 , BIBREF46 and BIBREF9 . For regularization, the training gradients of all parameters with an INLINEFORM0 2 norm larger than 40 are clipped to 40, as the previous work BIBREF47 .
The clustering performance is evaluated by comparing the clustering results of texts with the tags/labels provided by the text corpus. Two metrics, the accuracy (ACC) and the normalized mutual information metric (NMI), are used to measure the clustering performance BIBREF38 , BIBREF48 . Given a text INLINEFORM0 , let INLINEFORM1 and INLINEFORM2 be the obtained cluster label and the label provided by the corpus, respectively. Accuracy is defined as: DISPLAYFORM0 
where, INLINEFORM0 is the total number of texts, INLINEFORM1 is the indicator function that equals one if INLINEFORM2 and equals zero otherwise, and INLINEFORM3 is the permutation mapping function that maps each cluster label INLINEFORM4 to the equivalent label from the text data by Hungarian algorithm BIBREF49 .
Normalized mutual information BIBREF50 between tag/label set INLINEFORM0 and cluster set INLINEFORM1 is a popular metric used for evaluating clustering tasks. It is defined as follows: DISPLAYFORM0 
where, INLINEFORM0 is the mutual information between INLINEFORM1 and INLINEFORM2 , INLINEFORM3 is entropy and the denominator INLINEFORM4 is used for normalizing the mutual information to be in the range of [0, 1].
The most of parameters are set uniformly for these datasets. Following previous study BIBREF38 , the number of nearest neighbors in Eqn. ( EQREF15 ) is fixed to 15 when constructing the graph structures for LE and LPI. For CNN model, the networks has two convolutional layers. The widths of the convolutional filters are both 3. The value of INLINEFORM0 for the top INLINEFORM1 -max pooling in Eqn. ( EQREF10 ) is 5. The number of feature maps at the first convolutional layer is 12, and 8 feature maps at the second convolutional layer. Both those two convolutional layers are followed by a folding layer. We further set the dimension of word embeddings INLINEFORM2 as 48. Finally, the dimension of the deep feature representation INLINEFORM3 is fixed to 480. Moreover, we set the learning rate INLINEFORM4 as 0.01 and the mini-batch training size as 200. The output size INLINEFORM5 in Eqn. ( EQREF19 ) is set same as the best dimensions of subspace in the baseline method, as described in Section SECREF37 .
For initial centroids have significant impact on clustering results when utilizing the K-means algorithms, we repeat K-means for multiple times with random initial centroids (specifically, 100 times for statistical significance) as Huang BIBREF48 . The all subspace vectors are normalized to 1 before applying K-means and the final results reported are the average of 5 trials with all clustering methods on three text datasets.
In Table TABREF43 and Table TABREF44 , we report the ACC and NMI performance of our proposed approaches and four baseline methods, K-means, SkipVec, RecNN and Para2vec based clustering methods. Intuitively, we get a general observation that (1) BoW based approaches, including K-means (TF) and K-means (TF-IDF), and SkipVec based approaches perform not well; (2) RecNN based approaches, both RecNN (Ave.) and RecNN (Top+Ave.), do better; (3) Para2vec makes a comparable performance with the most baselines; and (4) the evaluation clearly demonstrate the superiority of our proposed methods STC INLINEFORM0 . It is an expected results. For SkipVec based approaches, the off-the-shelf encoders are trained on the BookCorpus datasets BIBREF51 , and then applied to our datasets to extract the sentence representations. The SkipVec encoders can produce generic sentence representations but may not perform well for specific datasets, in our experiments, StackOverflow and Biomedical datasets consist of many computer terms and medical terms, such as “ASP.NET”, “XML”, “C#”, “serum” and “glycolytic”. When we take a more careful look, we find that RecNN (Top) does poorly, even worse than K-means (TF-IDF). The reason maybe that although recursive neural models introduce tree structure to capture compositional semantics, the vector of the top node mainly captures a biased semantic while the average of all vectors in the tree nodes, such as RecNN (Ave.), can be better to represent sentence level semantic. And we also get another observation that, although our proposed STC INLINEFORM1 -LE and STC INLINEFORM2 -LPI outperform both BoW based and RecNN based approaches across all three datasets, STC INLINEFORM3 -AE and STC INLINEFORM4 -LSA do just exhibit some similar performances as RecNN (Ave.) and RecNN (Top+Ave.) do in the datasets of StackOverflow and Biomedical.
We further replace the CNN model in our framework as in Figure FIGREF5 with some other non-biased models, such as bi-LSTM and bi-GRU, and report the results in Table TABREF46 and Table TABREF47 . As an instance, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models. From the results, we can see that bi-GRU and bi-LSTM based clustering methods do equally well, no clear winner, and both achieve great enhancements compared with LPI (best). Compared with these bi-LSTM/bi-GRU based models, the evaluation results still demonstrate the superiority of our approach methods, CNN based clustering model, in the most cases. As the results reported by Visin et al. BIBREF33 , despite bi-directional or multi-directional RNN models perform a good non-biased feature extraction, they yet do not outperform state-of-the-art CNN on some tasks.
In order to make clear what factors make our proposed method work, we report the bar chart results of ACC and MNI of our proposed methods and the corresponding baseline methods in Figure FIGREF49 and Figure FIGREF53 . It is clear that, although AE and LSA does well or even better than LE and LPI, especially in dataset of both StackOverflow and Biomedical, STC INLINEFORM0 -LE and STC INLINEFORM1 -LPI achieve a much larger performance enhancements than STC INLINEFORM2 -AE and STC INLINEFORM3 -LSA do. The possible reason is that the information the pseudo supervision used to guide the learning of CNN model that make difference. Especially, for AE case, the input features fed into CNN model and the pseudo supervision employed to guide the learning of CNN model are all come from word embeddings. There are no different semantic features to be used into our proposed method, thus the performance enhancements are limited in STC INLINEFORM4 -AE. For LSA case, as we known, LSA is to make matrix factorization to find the best subspace approximation of the original feature space to minimize the global reconstruction error. And as BIBREF24 , BIBREF52 recently point out that word embeddings trained with word2vec or some variances, is essentially to do an operation of matrix factorization. Therefore, the information between input and the pseudo supervision in CNN is not departed very largely from each other, and the performance enhancements of STC INLINEFORM5 -AE is also not quite satisfactory. For LE and LPI case, as we known that LE extracts the manifold structure of the original feature space, and LPI extracts both geometric and discriminating structure of the original feature space BIBREF38 . We guess that our approach STC INLINEFORM6 -LE and STC INLINEFORM7 -LPI achieve enhancements compared with both LE and LPI by a large margin, because both of LE and LPI get useful semantic features, and these features are also different from word embeddings used as input of CNN. From this view, we say that our proposed STC has potential to behave more effective when the pseudo supervision is able to get semantic meaningful features, which is different enough from the input of CNN.
Furthermore, from the results of K-means and AE in Table TABREF43 - TABREF44 and Figure FIGREF49 - FIGREF53 , we note that TF-IDF weighting gives a more remarkable improvement for K-means, while TF weighting works better than TF-IDF weighting for Average Embedding. Maybe the reason is that pre-trained word embeddings encode some useful information from external corpus and are able to get even better results without TF-IDF weighting. Meanwhile, we find that LE get quite unusual good performance than LPI, LSA and AE in SearchSnippets dataset, which is not found in the other two datasets. To get clear about this, and also to make a much better demonstration about our proposed approaches and other baselines, we further report 2-dimensional text embeddings on SearchSnippets in Figure FIGREF58 , using t-SNE BIBREF53 to get distributed stochastic neighbor embedding of the feature representations used in the clustering methods. We can see that the results of from AE and LSA seem to be fairly good or even better than the ones from LE and LPI, which is not the same as the results from ACC and NMI in Figure FIGREF49 - FIGREF53 . Meanwhile, RecNN (Ave.) performs better than BoW (both TF and TF-IDF) while RecNN (Top) does not, which is the same as the results from ACC and NMI in Table TABREF43 and Table TABREF44 . Then we guess that both ”the same as” and ”not the same as” above, is just a good example to illustrate that visualization tool, such as t-SNE, get some useful information for measuring results, which is different from the ones of ACC and NMI. Moreover, from this complementary view of t-SNE, we can see that our STC INLINEFORM0 -AE, STC INLINEFORM1 -LSA, STC INLINEFORM2 -LE, and STC INLINEFORM3 -LPI show more clear-cut margins among different semantic topics (that is, tags/labels), compared with AE, LSA, LE and LPI, respectively, as well as compared with both baselines, BoW and RecNN based ones.
From all these results, with three measures of ACC, NMI and t-SNE under three datasets, we can get a solid conclusion that our proposed approaches is an effective approaches to get useful semantic features for short text clustering.
With the emergence of social media, short text clustering has become an increasing important task. This paper explores a new perspective to cluster short texts based on deep feature representation learned from the proposed self-taught convolutional neural networks. Our framework can be successfully accomplished without using any external tags/labels and complicated NLP pre-processing, and and our approach is a flexible framework, in which the traditional dimension reduction approaches could be used to get performance enhancement. Our extensive experimental study on three short text datasets shows that our approach can achieve a significantly better performance. In the future, how to select and incorporate more effective semantic features into the proposed framework would call for more research.
We would like to thank reviewers for their comments, and acknowledge Kaggle and BioASQ for making the datasets available. This work is supported by the National Natural Science Foundation of China (No. 61602479, No. 61303172, No. 61403385) and the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDB02070005).

Figure and table captions:
Figure 1: The architecture of our proposed STC2 framework for short text clustering. Solid and hollow arrows represent forward and backward propagation directions of features and gradients respectively. The STC2 framework consist of deep convolutional neural network (CNN), unsupervised dimensionality reduction function and K-means module on the deep feature representation from the top hidden layers of CNN.
Figure 2: The architecture of dynamic convolutional neural network [11]. An input text is first projected to a matrix feature by looking up word embeddings, and then goes through wide convolutional layers, folding layers and k-max pooling layers, which provides a deep feature representation before the output layer.
Table 1: Statistics for the text datasets. C: the number of classes; Num: the dataset size; Len.: the mean/max length of texts and |V |: the vocabulary size.
Table 3: Coverage of word embeddings on three datasets. |V | is the vocabulary size and |T | is the number of tokens.
Table 6: Comparison of ACC of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.
Table 7: Comparison of NMI of our proposed methods and some other non-biased models on three datasets. For LPI, we project the text under the best dimension as described in Section 4.3. For both bi-LSTM and bi-GRU based clustering methods, the binary codes generated from LPI are used to guide the learning of bi-LSTM/bi-GRU models.
Figure 3: ACC results on three short text datasets using our proposed STC2 based on AE, LSA, LE and LPI.
Figure 4: NMI results on three short text datasets using our proposed STC2 based on AE, LSA, LE and LPI.
Figure 5: A 2-dimensional embedding of original keyword features weighted with (a) TF and (b) TF-IDF, (c) vectors of the top tree node in RecNN, (d) average vectors of all tree node in RecNN, (e) average embeddings weighted with TF, subspace features based on (f) LSA, (g) LE and (h) LPI, deep learned features from (i) STC2-AE, (j) STC2-LSA, (k) STC2-LE and (l) STC2-LPI. All above features are respectively used in K-means (TF), K-means (TF-IDF), RecNN (Top), RecNN (Ave.), AE (TF), LSA(best), LE (best), LPI (best), and our proposed STC2-AE, STC2-LSA, STC2-LE and STC2-LPI on SearchSnippets. (Best viewed in color)

Questions about this paper:
What were their performance results?
By how much did they outperform the other methods?

Answers:
On SearchSnippets dataset ACC 77.01%, NMI 62.94%, on StackOverflow dataset ACC 51.14%, NMI 49.08%, on Biomedical dataset ACC 43.00%, NMI 38.18%
on SearchSnippets dataset by 6.72% in ACC, by 6.94% in NMI; on Biomedical dataset by 5.77% in ACC, 3.91% in NMI