言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Improving Concept Representations for Short Text
Classiﬁcation
Sijie Tao
Tetsuya Sakai
Waseda Universitytsjmailbox@ruri.waseda.jp tetsuyasakai@acm.org

Abstract
Short text classiﬁcation is a challenging task in Natural Language Processing (NLP). Compared with documents, short texts are more sparse and ambiguous
due to lack of context. In order to overcome this diﬃculty, recent work tends to take an approach of combining neural language models with external linguistic resources such as knowledge bases. In this kindof approaches, concepts obtained from a knowledge
base are usually mapped to an implicit space andrepresented as a vector. However, how to eﬀectivelyrepresent concepts in a neural language model is notwell studied yet. Hence, in this study, we construct
several formulae for concept embedding and comparethem in a short text classiﬁcation task. Our experimental results show that utilizing proper concept
embeddings can slightly improve the performance ofshort text classiﬁcation.

1Introduction

With the remarkable development of the Internet,
the amount of short texts is booming in many web
application areas, such as short messages, InstantMessages, online Chat Logs, Bulletin Board System
Titles, Web Logs Comments, Internet News Comments, SMS, Twitter etc[6]. Processing short texts
becomes a crucial task in Natural Language Processing (NLP). For example, tweets related to inﬂuenza
can be classiﬁed into positive or negative for disease
surveillance[1]. However, short text classiﬁcation isconsidered as a challenging task. The diﬃculty ofthis task originates from the sparsity and the ambiguity of short texts. Short texts contain less context
information than documents or paragraphs. Moreover, short texts are more ambiguous as it is hard
to identify the meaning of polysemes with limited
context.
To overcome the characteristics of short texts, recent work [9][11][2] tends to tackle short text classiﬁcation by combining neural language models withexternal linguistic resources such as knowledge bases.
In this kind of approaches, a knowledge base obtains

a set of concepts related to a word, and the concepts
are mapped to a neural language model space andrepresented as a vector. The word embedding andthe concept representation are then concatenated toenrich the text representation.
Looking into the results of previous work, it is obvious that combining a neural language model anda knowledge base can enrich the text representationand improve the performance of classiﬁcation. However, as far as we are aware, how to eﬀectively represent concepts in a neural language model is not wellstudied yet.
Thus in this paper, we construct several formulae for concept embedding and compare them in ashort text classiﬁcation task. By evaluating the performances of classiﬁcation, our experimental results
show that diﬀerent types of concept representation
can eﬀect the performance of short text classiﬁcation and utilizing proper concept embedddings canslightly improve the performance.
The remainder of this paper is organized as follows. Section 2 introduces related work. Section 3
explains the details of the proposed method. The
information of experimental settings is provided inSection 4. Section 5 and 6 give the analysis of theexperimental results and the conclusion of this study
respectively.

2

Related WorkOne of the eﬀective methods to overcome the diﬃculty of short text classiﬁcation is to improve text
representations via associate short texts with external explicit language resources, such as a knowledge
base. The knowledge retrieved from a knowledge
base contains rich semantic information and has beenused in several previous work.
Wang et al.[9] adopted a large-scale knowledge
base Probase[10] to retrieve relevant concepts andassociate them with short texts. Given a short text
as an input, the conceptualization API of Probase
can obtain a concept vector including relevant concepts and weights that show the relevance of theconcepts related to the text. The concepts are― 251 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

directly mapped to word embeddings from a pretrained model and the embedding of the concepts
is represented as a weighted average of the word embeddings.
In the work of Xu et al.[11], concepts obtained
from Probase are ﬁrst represented as the weighted
average of all the words which belong each concept,
rather than being directly transformed to word embeddings. Chen et al.[2] took full advantages ofknowledge bases. Multiple knowledge bases such as
YAGO[8], Probase, and CN-Probase[3] are appliedin this work.

3

Proposed Method

In this section, we describe the details of proposed
methods. First, given an input word x, the knowledge base returns a list C which contains the top-k
concepts related to the word: C = {c1 : w1 , c2 :w2 , ..., ck : wk }, where c1 , c2 , ..., ck are the concepts
and w1 , w2 , ..., wk are scores that represent the relavance of the concept with the given word. Second,every concept c is then transformed to vectors vc using a formula of concept embedding. Next, the conceptual representation of word x is calculated as theweighted average using the concept embeddings andtheir weights. At last, the word embedding of x isconcatenated with its conceptual representation as
the ﬁnal text representation for classiﬁcation. The
following sections presents the formulae of conceptembedding.
In this study, we use Microsoft Concept
Graph[10]1 to obtain concept knowledge and a pretrained Word2Vec model2 to obtain word embeddings. For out-of-vocabulary words, their embeddings are initialized with zero vectors.

3.1

Simple Average Embedding

Given a concept c, and the words {x1 , x2 , ..., xn } belong to it, the concept embedding vc can be represented as the average of the word embeddings:∑nex
vc = i=1 i(1)nwhere exn is the word embedding of xn obtained fromthe pre-trained word embedding model. We call thisformula the Simple Average Embedding.

3.2

word-concept pair (x, c). Based on the Simple Average formula, we can construct a formula to represent
concept incorporating the weights. In other words,
a concept can be represented as the weighted average of the word embeddings. Previous work [9][11]utilized this formula to represent concepts in experiments. In this study, this formula is named as theWeighted Average Embedding.
vc =3.3∑ni=1 ri exi∑ni=1 ri(2)

Relation-aware Average Embedding

In this formula, we assume that when a concept cis mapped to a implicit space, its representation vc
should be similar to the word embedding of itself
ec . Based on this assumption, we can represent therelation between the concept and the words belong
to it as:v ri = e c − e x i(3)Then the concept embedding is calculated by thesum of average of word embeddings and the weighted
average of relation representations. We name thisformula the Relation-aware Average Embedding.
∑nvc =

4i=1 exin∑ni=1 ri vri+ ∑ni=1 ri(4)Experiments

In order to evaluate the formulae of concept embedding, we establish a short text classiﬁcation task andcompare them. This section describes the experiment setup of this work. The environment of experiments is shown in Table 1.
Table 1: Environment of Experiments
CPUMemory SizeGPUGPU Memory SizeIntel Core i7-6700K
32GB
Nvidia GeForce RTX 2080Ti
11GB

Weighted Average Embedding

In a knowledge base, knowledge is stored as (x, c, r),where r is a relation score shows the relavance of the1 https://concept.research.microsoft.com/
2 https://code.google.com/archive/p/word2vec/

4.1

Datasets

In our experiments, we adopt two widely knowndatasets: The Question Classiﬁcation Dataset of Liand Roth, and AG’s corpus of news articles.

― 252 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Table 2: Hyper Parameters
Parameters
Valueskernel size
3, 4, 5, 6
embedding dimension
300
hidden layer dimension
100
dropout rate
0.5
learning rate
0.001
4.1.1

The Question Classiﬁcation Dataset ofLi and Roth

Table 3: Accuracy on the test sets
FormulaQCD AG’s news
Simple91.40
83.80
Weighted
92.00
84.54
Relation-aware 93.20
85.13
Figure 1: Cosine similarity distribution of the Simple
Average Embeddings

The Question Classiﬁcation Dataset of Li andRoth[5]3 (hereafter reﬀered to as QCD) contains
questions which are classiﬁed into six categories. The
training set has 5,452 questions and the test set has
500 questions.
4.1.2

AG’s corpus of news articles

AG’s corpus of news articles4 (hereafter reﬀered to as
AG’s news) is a dataset that includes title, description, category, etc. of over 1 million news articles. Inorder to construct a short text classiﬁcation task, weonly use the titles and the categories. The dataset
used in this study is a subset of AG’s news and it isadopted from the work of Zhang et al.[12]. The number of classes is four and there are 120,000 training
samples and 7,600 test samples.

4.2

Figure 2: Cosine similarity distribution of the
Weighted Average Embeddings

Text Classiﬁer

After obtaining the text representations, we build atext classiﬁer with Convolutional Neural Networks(CNN) to test the performances of diﬀerent input
embeddings. The classiﬁer includes a convolution
layer, a max-pooling layer and a hidden layer withdropout[7]. Inspired by previous work[13], we set
multiple kernel sizes for the convolution layer to capture diﬀerent scales of freature maps. The activation
funcitons are set to ReLU and tanh for convolution
layer and hidden layer respectively. The loss funciton is set to categorical cross entropy and we adopt
Adagrad[4] to optimize the training process. Following the work of Wang et al.[9], the hyper parameters
are set as shown in Table 2.
The training process is continued until the loss on
the test set stops to change for two epochs. The
highest accuracy on test set, which is calculated as
the percentage of the samples are correctly predicted,
is used for the evaluation of the model to make acomparison between the diﬀerent embeddings.
3 https://cogcomp.seas.upenn.edu/Data/QA/QC/
4 http://groups.di.unipi.it/ gulli/AG_corpus_of_~
news_articles.html

Figure 3: Cosine similarity distribution of theRelation-aware Average Embeddings

5

Results and Discussions

The experimental results on both datasets are shownin Table 3. We can see that the performance ofthe model using the Relation-aware Average Embedding outperforms those with the Simple Average Em-― 253 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

bedding and the Weighted Average Embedding by
1.8% and 1.2% on QCD, 1.13% and 0.59% on AG’snews, which indicates that the Relation-aware Average Embedding can slightly improve the performanceof text classiﬁcation on these datasets. The reason
is that the Relation-aware Average Embeddings canhelp acquire features from the concepts eﬀectively byadopting the representation of the relations between
concepts and words. As a result, the Relation-aware
Average Embedding is able to enrich the representation of short texts better than the other formulae.
In order to ﬁgure out the diﬀerence between thethree concept embedding formulae, we calculated
the cosine similarity between the concept embeddings and the word embeddings of the concept words.
Except the words that do not have an embedding in the pre-trained Word2Vec model (Out-ofVocabulary words), 6,337 pairs of concept embeddings and word embeddings are collected and theirdistributions of cosine similarity are presented in Figure 1, 2 and 3. From the graphs, we can clearly ﬁndthat the Relation-aware Average Embeddings havehigher similarity with the word embeddings while
the other two types do not. It also indicates that
the Relation-aware Average Embeddings meet theassumption that the concept embeddings are similar
to their word embeddings.

6Conclusion

In this paper, we construct three diﬀerent formulae ofconcept embedding and compare them in a short textclassiﬁcation task. The experimental results showthat the Relation-aware Average Embedding formulacan slightly improve the performance of text classiﬁcation because it can eﬀectively represent concepts
by considering relations between concepts and words.
For future work, we would like to test the formulaeon more datasets, with more classiﬁers in diﬀerent
architectures.

References[1] Eiji Aramaki, Sachiko Maskawa, and Mizuki
Morita. Twitter catches the ﬂu: Detecting inﬂuenza epidemics using twitter. In EMNLP,2011.
[2] Jindong Chen, Yizhou Hu, Jingping Liu,
Yanghua Xiao, and Haiyun Jiang. Deep shorttext classiﬁcation with knowledge powered attention. In AAAI, 2019.
[3] Jindong Chen, Ao Wang, J. Bradley Chen,Yanghua Xiao, Zhendong Chu, Jingping Liu, Jiaqing Liang, and Wei Wang. Cn-probase: A

data-driven approach for large-scale chinese taxonomy construction. 2019 IEEE 35th International Conference on Data Engineering (ICDE),
pages 1706–1709, 2019.
[4] John C. Duchi, Elad Hazan, and Yoram Singer.
Adaptive subgradient methods for online learning and stochastic optimization. J. Mach.
Learn. Res., 12:2121–2159, 2010.
[5] Xin Li and Dan Roth. Learning question classiﬁers. In COLING 2002: The 19th International
Conference on Computational Linguistics, 2002.
[6] Ge Song, Yunming Ye, Xiaolin Du, Xiaohui
Huang, and Shifu Bie. Short text classiﬁcation:A survey. Journal of Multimedia, 9:635–643,
2014.
[7] Nitish Srivastava, Geoﬀrey Hinton, Alex
Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov. Dropout: A simple way toprevent neural networks from overﬁtting. J.
Mach. Learn. Res., 15(1):1929 ‒ 1958, January
2014.
[8] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: A Core of Semantic
Knowledge. In 16th International Conference
on the World Wide Web, pages 697–706, 2007.
[9] Jin Wang, Zhongyuan Wang, Dawei Zhang, andJun Yan. Combining knowledge with deep convolutional neural networks for short text classiﬁcation. In IJCAI, 2017.
[10] Wentao Wu, Hongsong Li, Haixun Wang, andKenny Q. Zhu. Probase: A probabilistic taxonomy for text understanding. In Proceedings ofthe 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD 12,
page 481 ‒ 492, New York, NY, USA, 2012. Association for Computing Machinery.
[11] Jingyun Xu and Yi Cai. Incorporating contextrelevant knowledge into convolutional neuralnetworks for short text classiﬁcation. In AAAI,2019.
[12] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks fortext classiﬁcation. In NIPS, 2015.
[13] Ye Zhang and Byron C. Wallace. A sensitivity
analysis of (and practitioners’ guide to) convolutional neural networks for sentence classiﬁcation. In IJCNLP, 2015.

― 254 ―

Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.