DEIM Forum 2020 G4-2

Experiments on Unsupervised Text Classification
based on Graph Neural Networks
Haoxiang SHI† , Cen WANG†† , and Tetsuya SAKAI†† Faculty of Science and Engineering, Waseda University
3-4-1 Okubo, Shinjuku-ku, Tokyo, 169-8555, Japan†† KDDI Research Inc.,
2-1-15 Ohara, Fujimino-shi, Saitama-ken, 356-0003, Japan
E-mail: †hollis.shi@toki.waseda.jp, ††ce-wang@kddi-research.jp, †††tetsuya@waseda.jp
Abstract Traditional text classification requires labelled data for training the models, but obtaining a suﬃcientamount of training data is costly and often not practical. Recently, an unsupervised approach to text classification
has been proposed, which is based on a novel neural network architecture that represents textual relationships bygraphs. The present study reports on an experiment to validate this approach: our model achieves 42.5% in averageclassification accuracy and outperforms a basic autoencoder baseline with an accuracy of 31.3%.
Key words Text Classification, Graph Neural Network, Unsupervised Deep Learning.

1 Introduction

solution to realize contextualization [6]. A common way toconvert a text into a graph is to use co-occurrence window

Text classification is a critical language task that can sup-(e.g. TextRank [7]). Graph auto-encoder (GAE) neural net-

port multiple applications, such as fake news identification,

work is an advanced approach that combines the advantage

spam detection, sentiment analysis and opinion mining [1].

of AE and graph representation. Nevertheless, GAE can only

Recently, supervised deep learning methods have made great

encode text graph into few dimensions, which may cause in-progress in text classification tasks. The majority of super-suﬃcient text feature representation to pursue accurate clas-

vised deep learning methods focus on the design of textualsification. To deal with such a problem, inspired by a task

encoder [2][3]. After proper encoding, classification task canof citation link prediction [8], variational graph autoencoder

obtain high accuracy [4]. These procedures may require mass(VGAE) neural network has been applied to text classifica-labeled data (i.e. corpus) to train the deep learning model.

tion. Unlike conventional graph auto-encoder, VGAE pro-However, data in real world is often unlabeled. To label datavides a probabilistic manner for representing an observation(i.e. to build a corpus) is costly and ineﬃcient. In order toin the latent space of a text. Thus, the encoder can be formu-

make natural language processing (NLP) models versatile,

lated to represent a probability distribution for each latent

the eﬀectiveness of unsupervised models should be explored.

attribute of a text. Upon high-dimensional latent space representation, more accurate unsupervised text classification

Traditionally unsupervised text classification models use

could be achieved.

word frequency or spare word vectors to represent text, andthen text classification is done according to the features lying

In this paper, using the 20newsgroup dataset [9], we havein the corresponding representations. These simple represen-evaluated the eﬀectiveness of VGAE in a text classifica-

tation approaches cannot preserve context. As a result, thetion experiment. The results suggest that VGAE outper-

text classification performance may deteriorate. To furtherforms state-of-the-art unsupervised deep learning methods,

promote eﬀectiveness, it is necessary to find a contextual-

namely, AE and GAE. The code of our work can be foundization representation without a supervised pre-training en-in https://github.com/wcdtom/Underan/.

coder.

2 Related Work

Auto-encoder (AE) neural networks [5] can generate a

Unsupervised text classification has a long history. The ba-

powerful automatic representation in an unsupervised man-sic idea to classify text in an unsupervised manner, in fact,

ner. On the other hand, graph representation is a betteris to cluster texts with similar features. In the early years,

simple features such as word frequency (e.g. TF-IDF [10])and topic words (e.g. TextRank) have been used.
!"#$"%$&'"()*#"+,)-.%&/0'1&20#)3!*-45

With the development of deep learning, text can be represented as deep latent features. The state-of-the-art neural6(7/8259!"#$%&'()*#'+%!

;5<"$'#+,'-)%.#(/0+%".=>123)#24*)2"5%12#$ %' ( #): % * +)1'<"$'#&?/)272@'#+,-./$01

network architectures composed by convolutional neural net3"5works (CNNs) or long-short term memory (LSTM) cells can-not naturally support relativity representations. To this end,A(B%)'-)%%2

there is training data. It is because CNN and LSTM can-*#"+,)

"$ !

'!*-4

60+#070'%"%$&'

3 .41
3 .51
9("77$:$0#3 .61

8&2.(0;C(8*()2"53 .71 8 )99) 23%)0'%<8(33%3/(<'not represent latent features with semantic relativity unlessin supervised learning, graph-based text representation and3;5classification methods have been proposed in recent years.

&29D%E%,'-)%<8(332F2<()2"5%4G%7'(53%"F%H.I;

Fu et al. [6] have proposed the GraphRel that uses graphconvolutional neural network (GCN) to realize relation ex-Figure 1 Text classification by means of VGAE

traction. Then, Yao et al. [1] have implemented text classification based on GCN, and achieved the highest accuracy
when comparing to CNN and LSTM -based approach.

As aforementioned, basic AE can only represent an input
into a single point (i.e. usually 2D or 3D). Hence, VAE adds
latent variables to the AE. The main idea of variational is

In unsupervised learning, graph-based text representationand classification have also received much attention. Several studies focus on building more complex graph to describe text. For example, they add special edges to indicate
grammatical relation between two vertexes (i.e. words) [11].
Then, they use unsupervised text classification to verify themodified graph representation. Hinton et al. [5] have pro-to restrict the learned parameters from a known distribution.
For any sampling of the latent distributions, the decoder isexpected to accurately reconstruct the input. The statistical
distribution helps to force a continuous, smooth latent space
representation. Namely, the values in distributions are nearto one another in latent space, which could contribute a moresimilar reconstruction.

posed the epoch-making AE, which enables the unsupervised
working manner of the neural network. Kingma et al. [12]proposed the variational autoencoder (VAE) neural network
to enable more generative representations. Meanwhile, basedon graph-like dataset and utilizing powerful GCN from su-

While GAE is a variant of AE, the input of GAE is a graph.
The encoder of the GAE is GCN. Using the encoded featuresof the GCN, output graph can be regenerated via decoder.
The procedures of GAE can be described as follows:

pervised models, Kipf et al. [13] proposed GAE to do thesemi-supervised classification task. Following that, Wang et

Z = GCN (A, X)

al. [14] have compared unsupervised text classification un-Â = sigmoid(ZZ⊤ )der multiple kinds of auto-encoder neural networks. The
contrastive results shows the limitation of the models of AE,

where, A is the input graph, X is the feature matrix of theVAE and GAE. Further, Kipf et al. [6] proposed VGAE

nodes in graph A (the specific form of X will be detailed

which combines the advantages of GAE and VAE methods.

later). The Z is the representations via the encoder. Â is aVGAE has been successfully utilized on citation link pre-regeneration of graph A. The learning approaches (i.e. thediction. Inspired by Kipf’s work, we want to investigateloss function) of a GAE model can be the cross entropy.

whether the VGAE performs well in other language tasks.
Specifically, the present study considers the task of text classification.

VGAE is a combination of GAE and VAE. The calculationof VGAE is depicted in Fig .1(a). The encoder of VGAE isas follows:

3 Text Classification via VGAE
µ = GCNµ (A, X)
VGAE inherits the architecture of AE, which includes encoder and decoder parts. The encoder part represents theinputs into latent features. And decoder part uses the latentfeatures to generate the outputs, which can be regarded aslog σ = GCNσ (A, X)∏N
i=1 q(zi |A, X)

q(Z|A, X) =q(zi |A, X) = N (zi |µi , diag(σi2 ))the regeneration of the inputs. The learning feedback can bewhere, µ is the set of each matrix µi , σ is the set of eachbased on the diﬀerence between inputs and outputs.

matrix σi . A two-layer GCN is defined as:0.9

where, Ã = D−1/2 AD1/2 is the symmetrically normalized

0.8

adjacent matrix, and Relu(·) = max(0, ·).

0.7

The decoder of VGAE is given as:p(A|X) =Loss

GCN (A, X) = ÃRelu(ÃXW0 )W1∏N ∏Ni=1

j=1 p(aij |zi , zj )VGAEGAEAE

0.6
0.5p(aij = 1|zi , zj ) = sigmoid(z⊤i zj )0.4
20

where, aij is an element of the graph A.
The loss function of the VGAE is:4060
Epoch80

100

Figure 2 The training loss of each unsupervised model

L = Eq(Z|A,X) [log p(A|Z] − KL[q(Z|A, X)||p(Z)]1 layer
2 layers

40

structing loss and the latent variable restriction loss. Constructing loss is to measure to what extent the constructedadjacency matrix is similar to the input one; the other loss

ACC (%)The loss function during learning consists of two parts: con30
20
10

is to apply KL-divergence [15] to measure how similar thedistribution of the latent variable and a normal distribution

0VGAE

are.
Upon the VGAE, the procedure of the unsupervised text

GAE
Unsupervised model

AEFigure 3 The ACC of VGAE, GAE, AE with one-layer and twolayerclassification is described in Fig. 1(b). Texts are firstly embedded into graphs. In a single text graph A, words andtexts are regarded as a node, and the node feature is a one-The training loss (with one-layer model) of the AE, GAE

hot vector. All the vectors form the text feature matrix X.

and VGAE are depicted in Fig. 2. It can be found that theMatrices A and X are inputted into VGAE, then the latent

training loss of AE reduced very fast. It may be because thatfeature Z is retrieved by the classifier. Text classification isthe architecture of the model is simple. On the other hand,to cluster all the feature matrices. Thus, classifier can be im-

the GAE and VGAE converge slowly, which is caused by theplemented by any clustering method, such as k-means, SVM

more complex inner structure of graph neural network.

and spectral clustering [16].

4 Experimental Results

The ACCs of AE, GAE and VGAE are shown in Fig. 3.
Both one-layer model and two-layer model are evaluated.

In order to evaluate the eﬀectiveness of VGAE in text clas-

The results suggest that the two-layer VGAE can achievesification, we take 18,821 texts in the 20newsgroup dataset

the highest ACC, and VAE is the next, and both VGAE andas inputs of experiment. After the VGAE training, 16 di-VAE are more eﬀective than the basic AE for unsupervised

mensions of latent feature are obtained. Then the k-means

text classification.

method is selected to classify the texts according to the latent features.

5 Conclusion
In this paper, we have verified the eﬀectiveness of VGAE

We use classification accuracy as our evaluation measure,∑ni=1 δ(cti ,map(ĉti ))nwhere cti is the real label of text i,i = 1, 2, . . . , n, and ĉti
is the predictive label of text i. If cti = map(ĉti ), thenδ(cti , map(ĉti )) = 1. The function map(·) indicates a permutation mapping that best matches the predictive clusteringlabels to real labels.

In the experiment, wecompared VGAE to other two methods, GAE and AE. In

which is defined as:
ACC =in text classification empirically.

terms of accuracy in text classification, VGAE outperforms
GAE, which in turn outperforms AE. In addition, two-layer
networks outperform their one-layer counterparts for every
model.
For future work, we would like to use other datasets toevaluate the VGAE text classification model. Additionally,
the fine-tuning on this model can be further investigated.

Acknowledgement
We would like to thank Mr. Zhaohao Zeng (Ph.D Candidate)for his valuable advice.

References[1] Yao Liang, Chengsheng Mao and Yuan Luo. Graph convolutional networks for text classification , Proc. of the AAAI,vol. 33, 2019.
[2] K. S. Tai, R. Socher and C. D. Manning, Improved semantic representations from tree-structured long short-term
memory networks , Proc. of ACL, pp. 1556–1566, 2015.
[3] J. Pennington, R. Socher and C. Manning, Glove: Globalvectors for word representation , Proc. of the EMNLP, pp.
1532–1543, 2014.
[4] G. Wang, C. Li, et al. Joint embedding of words and labels
for text classification , Proc. of the ACL, pp. 2321–2331,2018.
[5] E. Geoﬀrey Hinton and Salakhutdinov Ruslan Reducing
the dimensionality of data with neural networks , Science
vol. 313 5786, pp. 504-507, 2006.
[6] Tsu-Jui Fu, Peng-Hsuan Li and Wei-Yun Ma. GraphRel:
Modeling text as relational graphs for joint entity and relation extraction , Proc. of the ACL, 2019.
[7] R. Mihalcea and P. Tarau, TextRank: Bringing order intotexts , Proc. of ACL, pp. 404-411, 2004.
[8] Thomas N. Kipf, and Max Welling,
Variational Graph
Auto-Encoders , in Bayesian Deep Learning Workshop ofthe NIPS, 2016.
[9] [Online Available.] 20newsgroup: http://qwone.com/ jason/20Newsgroups/[10] C. S. Saranyamol and L. Sindhu. A survey on automatic
text summarization , Int. J. Comput. Sci. Inf. Technol, vol.
5, no. 6, pp. 7889-7893，2014.
[11] Esteban Castillo, et al. Analysis Using Diﬀerent GraphBased Representations , Computacion y Sistemas, Vol. 21,
No. 4, pp. 581–599, 2017.
[12] [Online Available] P. Diederik P. and Welling Max, AutoEncoding Variational Bayes :https://arxiv.org/abs/1312.6114.
[13] T. N. Kipf and M. Welling. Semi-supervised classification
with graph convolutional networks , Proc. of ICLR, pp.
1-14, 2017.
[14] Shiping Wang, et al. An Overview of Unsupervised Deep
Feature Representation for Text Categorization , IEEE
Transactions on Computational Social Systems, vol. 6, no.
3, pp. 504-517, 2019.
[15] Severin Klingler, et al. Eﬃcient Feature Embeddings forStudent Classification with Variational Auto-Encoders ,Proc. of the 10th International Conference on Educational
Data Mining, pp.73-79, 2017.
[16] [Online Available] KL-divergence:https://en.wikipedia.org/wiki/Kullback%E2%80%93...
...Leibler divergence.