DEIM Forum 2019 B4-4

Weakly-Supervised Neural Categorization of Wikipedia articles
Xingyu Chen†

Mizuho Iwaihara‡†Graduate School of Information, Production and System, Waseda University
Kitakyushu 808-0135 Japan
E-mail: †chenxingyu@akane.waseda.jp, ‡iwaihara@waseda.jp
Abstract Wikipedia is the largest online encyclopedia, in which articles are organized into hierarchical categories. One
category in Wikipedia consists of its member articles and subcategories. The task of Wikipedia article categorization is thatgiven a new article, finding which category it should belong to. Deep neural models are gaining increasing popularity for many
NLP task, due to their strong expressive power and less requirement for feature engineering. However, neural models always
need a large amount of labeled training data, while one Wikipedia category does not have enough member articles. In thispaper, we utilize a weakly-supervised neural document classification model for article categorization. We examine
classification accuracy on different levels of categories, and find that the model performs well on the top-level category, butwhen we descend the categories, the performance declines. Besides, we also perform experiments on finding the limitation of
document length for the model’s component RNN model. As for the limitation of document length for RNN model, we propose
a method that first extracts the key sentences of documents then applies to train the RNN model.
Keyword Weakly-supervised learning, Text classification, Wikipedia category, Neural classification, Key sentence extraction

1. Introduction

which category it should belong to. Our task is a kind of

Wikipedia is a free multi-lingual online encyclopedia

document categorization and our method can be applied towhich is constructed by volunteers , having over 5.7

any categorization. In this paper, we consider the depth

million articles in English.

from the top category as a parameter of hardness of

Wikipedia Category Structure In Wikipedia, articles are

categorization.

organizeda

Neural Networks In recent years, neural networks havecategory in Wikipedia has three kind s of forms: (1)been widely applied to NLP tasks such as machine

category consists of member articles, (2) category consists

translation, relation extraction, word embedding, etc,

of subcategories, and (3) category consists of member

showing strong expressive power. But neural networks

articles and sub categories.

need a large volume of labeled training corpus to train aintohierarchicalcategories.

Generally,neural network, but member articles of one Wikipedia
category are not sufficient to train a neural network. Yu et
al. [6] proposed a weakly supervised neural model , called
WSTC, which utilizes CNN [3] (Convolutional Neural
Networks)

and

HAN

[8]

(Hierarchical

Attention

Networks) can work well on text classification, even if
only a subset have class labels. HAN is a type of RNN,
and it uses sequences of words and sentences, but thelength of documents may exceed the learning ability of
RNN. Therefore, in this paper we explore how the
document length influences the performance of WSTC.
Considering that a large amount of Wikipedia articles arelong but one of the WSTC component models— the HAN
Figure 1 A category consists of subcategories and member

model has limitation on handling long articles, we propose

articlesa method that we first utilize TextRank [5] to extract key

Wikipedia Article Categorization the task of Wikipedia

sentences for each document, then app ly them to thecategorization is that given a Wikipedia article, finding

model.

Top-level categories, topic of each category is qui te

top-level category is a category named Contents, which

diverse in topics, such as sports, arts, and law, which are

has subcategories of various types of encyclopedia

relatively simple to distinguish. But as we descend thecontents, such as articles and glossaries, content s thatcategory level, the topics of categories become detailed

assist navigation in the encyclopedia, such as indexes andand similar to each other. Thus we also examine theportals, as well as pages related to the maintenance of theclassification accuracy on different levels of categories.

encyclopedia. However, since we focus only on the

Our work makes four main contributions: First, we

articles of Wikipedia, we select Articles, which is theutilize apply the weakly-supervised neural model WSTC

highest level category for all articles of Wikipedia, as our[6] to Wikipedia article categorization problem; second,

top category. Articles have a subcategory named Main

we

the

topic classification which can reach any article andperformance of WSTC; and third, we examine thecategories and reflecting different fields of knowledge.

classification accuracy on different level of categories ;Therefore, category Main topic classification is a list ofand fourth, we propose a method that combines key

Wikipedia’s major topic classifications and contains 28

sentence extraction and the WSTC model and we find this

topics in total —Arts, Concepts, Business, Culture and somodel outperformed all other models .

on.

study

on

how

document

lengthinfluence

The rest of this paper is organized as follows. Section 2
covers related work. Section 3 describes Wikipedia
category structures and how we collect the dataset.
Section 4 discusses combination of WSTC and text
extraction. Section 5 present result and finding of result.
Section 6 concludes this paper.

2. Related Work
There exist several researches related to Wikipedia
article categorization. Aniket et al. [2] provide the first

Figure 2 Overview of category systemcomprehensive quantitative mapping of the distribution of

When a node is a child of other node means this category

topicsin

Wikipedia

based

on

Wikipedia

is subcategory of the other category

category.

Mohamed et al. [1] explore “is a” taxonomy from

3.2 Data Collection

Wikipedia category graph. There has been a large amountIn Section 3.1 we have discussed that Wikipedia article sof research on neural text classification and weakly

are divided into the 28 top topics. But, those topics are notsupervised classification. Siwei et al. [4] proposed

independent of each other. Also, certain topics are

Recurrent Convolutional Neural Networks model which

subcategories of multiple topics such as Arts, Language,combine CNN and RNN model for text classification.

Society, and Culture. Furthermore, certain topics share the

Yangqiu et al. [7] take advantage of category name andsame part of subcategories of Culture, and Humanities

project the name and document into the same sematicshares subcategories with Arts, and Books. In our case, wespace. Then they perform classification based on thewantsemantic similarity between the category surface name andrelationship,

the documents.

overlapping member articles. As a result, we extract thetoextractnocategories

that

overlapping

have

no

subcategories

inclusion
andnofollowing five categories as top level categories – Arts,

3. Wikipedia

Category

Structure

and

Data

Collection
In this section, we discuss the Wikipedia category
structure and how we collect our data for experiments.
3.1 Wikipedia category structure
In Section 1, we have introduced the three forms of
categories — consisting of member articles, consisting of
subcategories and consisting of member articles andsubcategories. In this section, we describe the features of
the top-level structure of Wikipedia categories. The

Mathematics,

Law,

LanguageandSports.

BecauseWikipedia needs to control the size of member articles of
one category, articles in a large category are often moved
to subcategories. So, we extract its subcategory’s member
articles as this category’s member article s. Finally, we
obtained 7261 articles.
To examine whether or not the depth of the categories
can cause degradation of classification accuracy, we
extract articles from different levels of categories.

Dataset name

Table1 Number of articles for each category
Category(No. of articles)

Total

L1 Dataset

Arts(843), Mathematics(2236),Law(1437),Language(1310),Sports(1427)

7261

L2 Dataset

Architecture ( 2390 ) , Communication design(888), Culinary Arts(268),
Fashion design(1089), Performing arts(1202),Perfumery(540)
Advertising(1024), Document(1452), Graphic design(1258),
Heraldry(766), Information visualization (201), Multimedia(642),
Political communication(453)6377

L3 Dataset

Category Arts, which is a member of the top level

5823

4.1 Wikipedia Article Categorization Problem

categories, can divide into seven sub-topics: Architecture,

The Wikipedia article categorization problem is that

Communication design, Culinary Arts, Fashion design,

given a set of Wikipedia articles D, target categories C, we

Performing arts, Perfumery. We regard those six topics as

assign to each Wikipedia article D j a category C i .

the second level categories. Then the total number of

4.2 Weakly Supervised Text Classification Modelarticles is 6377.

Here, we discuss how we use the WSTC model to our

Similarly, Communication design can be divided into

task. The WSTC model can be divided into seven steps:

seven sub-topics – Advertising, Documents, Graphic

Extracting class related keywords. First we train word

design, Heraldry, Information visualization, Multimedia,

embeddings for each corpus. After we obtain word

Political communication. We regard those seven topics asembeddings,

the third level. The total number of the articles is 5823.

seed information is label surface name L, then for each

Notice that, for each category, we extract this category’sclass j, the embedding of surface name L j of each category

member articles, its subcategories’ member articles,

C j is used to retrieve top-t nearest words in the semantic

subcategories of its subcategories member articles as its

space, and these t words are adopted as class related

final member articles.

keywords.

4. Weakly Supervised Text Classification Neural

If the seed information is a small amount of Wikipedia

we extract class related keywords. If thearticles whose category labels are known, we first extract t

Model
text

representative keywords from the labeled articles by usingclassification model which can apply deep learning to text

tf-idf weighting, then adopt these words as class-relatedclassification problems, even there are only limited seed

keywords.

information such as class labels, and/or labeled articles. In

Modeling the semantic distribution of each class as

this paper, we utilize this model to our Wikipedia

von Mise Fisher (vMF) After extracting class related

article categorization problem. In this section, we
first give a formulation of our task and thenintroduce the weakly supervised text classification(WSTC) neural model, then we discuss about theneural models – CNN and HAN that we use; finally
we introduce how we combine TextRank and the
WSTC-HAN model to deal with classification of long
documents.
Table 2 Notation table

keywords, we model the word distribution of each class by

Yu et at. [6] propose a weakly supervised

von

Mise

Fisher(vMF),

such

that

the

probability

distribution of each class is given by:𝑇𝑓(𝑥; 𝜇, 𝑘) = 𝑐𝑝 (𝑘)𝑒 𝑘𝜇 𝑥
Here, k>0, || μ||=1, p ≥2, and 𝑐𝑝 (𝑘)(1)
is normalization

given by:𝑐𝑝 (𝑘) =𝑝
𝑘 2 −1𝑝
2𝜋 2 𝐼𝑝−1 (𝑘)2(2)Where I r (.) is the modified Bessel function of the first

Notation

Description

kind at order r. Let X be a set of vectors for keywords on

D={D 1 ,D 2 ,…, D n }A set of Wikipedia
articlesm target categories

unit sphere,

Surface name for eachcategory
A set of l labeled
documents in category.

calculated

C={C 1 ,C 2 ,..C m }L={L 1 ,L 2 ,..L m }D j L ={D j ,1 ,…D j, l }X ={x i | x i drawn from 𝑓(𝑥; 𝜇, 𝑘),1 ≤i≤ t}
vMF

needs
by

two

parametersthefollowingkandmaximum

estimattion:𝜇=∑𝑡𝑖=1 𝑥𝑖|| ∑𝑡𝑖=1 𝑥𝑖 ||𝜇 ,where

(3)

𝜇 is

likelihood

and

CNN model to the sentence classification problem. In our
𝐼𝑝/2 (𝑘)
𝐼𝑝

2 −1

(𝑘)

=

|| ∑𝑡𝑖=1 𝑥𝑖 ||𝑡case, the input of CNN is a Wikipedia article of length 𝑙𝑒(4)

which is represented by a concatenation of word vectors:
d = 𝑥1 ⊕ 𝑥2 ⊕ … .⊕ 𝑥𝑙𝑒 ,Generating pseudo documents. In order to pre-train(9)

the neural model, we first generate pseudo documents

where x i ∈ 𝑅𝑝 is the p-dimensional word vector of the i t hfrom the vMF distribution. To generate pseudo documents,

word. A feature f i is generated with window size s:
𝑓𝑖 = 𝑓(𝑤 ⋅ 𝑥𝑖:𝑖+𝑠−1 + 𝑏)

a document vector is chosen from the class probability

ℎ𝑝

(10)

is filter operating, b ∈ 𝑅(𝑥; 𝜇, 𝑘) , then pseudo document 𝑑𝑖 is generated by

Where w∈ 𝑅

drawing words w from the probability distribution:

After the filter operation, a feature map is generated as

𝛼𝑃𝐵 (w)𝑃(𝑤|𝑑𝑖 ) = {𝛼𝑃 (w) + (1 − 𝛼)𝐵𝑓 = [𝑓1 , 𝑓2 , … 𝑓𝑙𝑒−𝑠+1 ]w ∉ 𝑉𝑑𝑖exp(𝑑𝑖 𝑥𝑤 )∑𝑤′ ∈𝑉 exp(𝑑𝑖 𝑥 ′ )𝑑𝑖𝑤

is bias terms.

w ∈ 𝑉𝑑𝑖 (5)(11)

Finally, max-pooling operation is performed for eachfeature map to obtain final feature and the fully connected

Here, 𝑃𝐵 (𝑤) is the background distribution of word 𝑤,

layer of those final features is the input of a softmax layer,

𝛼 is background distribution weight , and 𝑉𝑑𝑖 is thefrom which the final output is generated.

keywords consisting of the most similar𝜉 words in the

4.3.2 HAN Model. Zichao et al.[8] proposed the HAN

semantic space for document 𝑑𝑖 . For each category 𝛽

modeldocuments are generated.

architecture of documents by using a sequence encoder

Neural model pre-training In order to overcome the

which

takes

advantageofthehierarchicaland attention layer. The input article is represented by aoverfit problem, this model creates pseudo labels forsequence of sentences s i , i∈ [1, 𝐿]

pseudo documents rather than using one-hot encoding

represented by a sequence of words w i t , t∈ [1. 𝑇]. For the 𝑡method. The pseudo label ℓ𝑖 for pseudo documents D i *-th word, the GRU calculates the new states by:ℎ𝑡 = (1 − 𝑧𝑡 ) ⊙ ℎ𝑡−1 + (𝑧𝑡 ) ⊙ ℎ𝑡∗ (12)is given as:ℓ𝑖𝑗 = {and each sentence is(1 − 𝛼) +𝛼𝐷𝑖∗ is generate from class j𝑚𝛼otherwise𝑚

𝑟𝑡 is the reset gate vector :𝑟𝑡 = 𝜎(𝑊𝑟 𝑥𝑡 + 𝑈𝑟 ℎ𝑡−1 + 𝑏𝑟 )

(6)

(13)

𝑧𝑡 is the update gate vector :𝑧𝑡 = 𝜎(𝑊𝑧 𝑥𝑡 + 𝑈𝑧 ℎ𝑡−1 + 𝑏𝑧 )Then the neural model is pre-trained by minimizing the
KL loss between output Y and pseudo labels L. The KL

ℎ𝑡∗

(14)

is the candidate state vector :ℎ𝑡∗ = 𝑡ℎ𝑎ℎ(𝑊ℎ 𝑥𝑡 + 𝑟𝑡 ⊙ 𝑈𝑟 ℎ𝑡−1 + 𝑏ℎ ),loss is given as:
KL(𝐿||𝑌) = ∑𝑖 ∑𝑗 𝑙𝑖𝑗 𝑙𝑜𝑔

𝑙𝑖𝑗
𝑦𝑖𝑗

(7)

(15)

where 𝑥𝑡 is the 𝑡-th sequence vector. After encoding
words, we apply the attention layer to generate the

Neural model self-training After pre-train the model,

sentence level representation. Similarly, after the sentence

we get the Pre-WSTC model, we use unlabeled articles tolevel, the attention layer generates the document levelself-train the Pre-WSTC model. It first uses the Pre-WSTC

representation, which becomes the input for the softmax

to categorize the unlabeled articles, and current output Y islayer.

obtained. Then the pseudo labels are updated by equation

layer.

(8):

4.4 Combination of WSTC-HAN and TextRank𝑙𝑖𝑗 = ∑2𝑦𝑖𝑗′𝑗′(𝑦𝑖𝑗 /𝑓𝑗 ),(8)

The final output is generated from the softmax

RNN-based models such as HAN have difficulties indealing with long sequences, so one method to resolve thiswhere 𝑓𝑖 = ∑𝑖 𝑦𝑖𝑗 is the soft frequency for class j. Pseudo

problem is to compress long articles into short version.

labels are updated iteratively while minimizing the KL

The simplest way to compress an article is to use theloss.

When the number of changes in the category

leading sentences of one article. But import ant contents

assignment is less than 𝜎, this iterative process will st op

may appear in any position of the article, so only using theand we can get our WSTC model.

leading

4.3 Neural Models

information. So, in this paper, we consider another way to

In the neural model part, we use two kind s of neuralcompress one article. We use TextRank to compress one

models—CNN and HAN.
4.3.1 CNN Model. Yoon al. [6] proposed applying thesentences

may

not

catch

certainimportarticle by extracting key sentences.
TextRank [5] is first proposed by Mihalcea, which can beTable 3 Dataset Statistics
Category name(no. of articles)
train dataset
test dataset

nameL1 dataset

Arts(843), Mathematics(2236),Law(1437),
Language(1310), Sport(1427)

5808

1453

Averagedocument
length(words)
1325

L2 dataset

Architecture(2390),
Communication design(888),
Culinary arts(268),
Fashion design(1089),
Performing arts(1202),
Perfumery(540),
Advertising(1024),
Documents(1452),
Graphic design(1258),
Heraldry(766),
Information visualization(201),
Multimedia(642),
politic communication(453)

5104

1273

976

42

4661

1162

969

41

L3 dataset

Average document
length(sentences)52

used to extract keywords and key sentence s from a single

the F1 score of each class then regard the average of F1

document without supervised learning.

score as F1-macro score.

TextRank also

has been applied to a wide range of areas such as text

5.2 Parameter setting

summarization. In this paper, we utilize TextRank toFirst, the number of labeled articles for each class is 20,

extract

if the seed information is labeled article, the number t ofkey

sentences.

TextRank

isagraph -basedalgorithm, where each sentence is represented as a graphextracted

node, and an edge between two nodes is weighted bybackground word distribution weight 𝛼 is 0.2. The

similarity:

number of pseudo articles for each category 𝛽 is 500. The

𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑠𝑖 , 𝑠𝑗 ) =|{𝑤𝑘 |𝑤𝑘 ∊𝑠𝑖 &𝑤𝑘 ∊𝑠𝑗 }|

(16)

|𝑠 |

𝑙𝑜𝑔|𝑠𝑖 | +𝑙𝑜𝑔 𝑗𝑤𝑖𝑗 = 𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑠𝑖 , 𝑠𝑗 )(17)

And score of each sentence is calculated by:
𝑆𝑐𝑜𝑟𝑒(𝑉𝑖 ) = (1 − 𝑑) + 𝑑 ∗ ∑𝑣𝑗∊𝐼𝑛(𝑉 ) ∑
𝑖

𝑤𝑖𝑗
𝑣

𝑘∊𝑜𝑢𝑡(𝑉𝑗)𝑤𝑗𝑘

𝑆𝑐𝑜𝑟𝑒(𝑣𝑗 ) (18)

keywords

for

each

category

is

10.

The

self-training stop condition is 𝜎 = 0.0001. The dimensionof word embeddings is 100.
5.3 Dataset Statistics
In Section 3 we have discussed about how we collect thedatasets – top-level dataset, second level dataset, and thethird level dataset. For each dataset, we split it into

Where 𝐼𝑛(𝑣𝑖 ) is the set of vertexes that point totraining dataset and test dataset, according to the radio of𝑣𝑖 (predecessors), 𝑂𝑢𝑡(𝑣𝑖 ) is the set of vertexes that 𝑣𝑖

8:2. Besides, as the category level goes down, the topics

points to(successors). Then calculate each vertex ’s score

between them become more similar, so th ere exist several

until convergence.

Wikipedia articles that belong to multiple categories. But

After we calculate the score for each sentence, we rank

we only need Wikipedia articles that only belong to one

sentences of one article according to the score and finallycategory, so we remove articles which belong to two or

we choose top-t sentences as the compressed version of

more categories. In Table 4 we will show the number of

one article. As the result, we use compress version of

overlapped articles in each dataset.

documents for pre-training and self-training WSTC-HAN

Table 4 Overlap articles statistic (L2 dataset)
Name of each category
Overlap articles for each
class
Architecture0model.

5. Experiments
5.1 Evaluation method
In this paper, we use F1-Micro and F1-Macro score toevaluate the performance of the model. Suppose TP
represent the true positive, FP means false positive , FN
means false negative, TN means true negative.
F1-micro first calculate all classes’ TP, FP, FN and TN,then calculate F1 score whereas F1-macro first calculate

Communication design
Culinary arts
Fashion design
Performing arts
Perfumery
total

2
0
4
2
2
8

arms, coat, heraldry, royal …

Table 5 Overlap articles statistic (L3 dataset)
Name of each category
Overlap articles for each
class
Advertising0
Documents
1
Graphic design
10
Heraldry
7
Information visualization
2
Multimedia
9
Political communication
19
Total
48
Besides, in this paper, we utilize two kinds of seedheraldryinformation—category

the results.

surface

names

and

labeledinformation
visualization,

computer, graphics, research,visualization …
Political,
committees,
agreement

Political,
communication
multimedia

tv, digital, channels, video …

5.4 Experimental Results
In this section, we report our results and what we find in

Wikipedia articles. If the seed information is category

5.4.1 Effect of category level. In this paper, we apply thename, we directly regard the category surface name as

WSTC model on datasets L1, L2 and L3 of three different

class related keywords, if the seed information is labeled

category levels. The results are shown in Figures 3 and 4.

articles, we extract t=10 keywords for each class andDataset L1, extracted from the top-level category, shows

regard them as class related keywords.

better results than L2 and L3 from lower categories, forboth CNN and HAN models, and in both macro - andTable 6 class-related keyword of different seed
Category
nameartsinformation for L1 dataset
surface Labeled Wikipedia article

mathematics
law
languagesportsart, video, design, work, artist…
filter,permutation,
crank,
additive …
grant, monarch, felony, equity,
legislaturelearner, jargon, pronunciations,progmatics …
footbag, polo, basketball, rink …

Table 7 class-related keyword of different seed
information for L2 dataset
Category surface name Labeled Wikipedia article
architecturebuildings,
architectural,
Audubon, townhouses, …
communication, design webpage, messaging, telephonysmartphone …
culinary, arts
foods, chicken, fried, seafood
dishes…
fashion, design
sarafpour,wendi,
turunen,womenswear …
performing, arts
demonstrations,theatricality,
ipcny, musicology …
perfumeryfragrances, gourmand, scents,
perfume …
Table 8 class-related keyword of different seed
information for L3 dataset
Category surface name Labeled Wikipedia article
advertising
advertising,ad,campaign,
television…
documents
letters, cards, recipient, …
graphic, design

graphics,
lines,…

visual,micro-averages.

One explanation is that when the

category level descends, the topics between each category
become more similar. For example, for extracted keywords
for each category of dataset L1, there is no overlapping
word, but for dataset L3, the keywords for categorygraphic design and information visualization have acommon keyword – “graphics”, indicating that the topics
betweenthem

becomemoresimilar,makingtheclassification task harder.
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6

0.90276

0.88582
0.85283

0.85230.80911

0.79379
0.749290.74234
0.67194

0.67047

F1-macro(CNN)

0.6704F1-macro(HAN)
L1-data

F1-micro(CNN)

L2-data

0.6704

F1-micro(HAN)

L3-dataFigure 3 Classification results on different levels of category
hierarchy. The seed information is labeled articles.
0.9
0.8
0.7
0.6

0.85272

0.83805
0.653230.56408

0.5

0.48362
0.482290.61351
0.58434

0.49018

0.4

0.328740.244310.3

0.560220.2
F1-macro(CNN)

F1-macro(HAN)

L1-dataL2-data

F1-mIcro(CNN)

F1-micro(HAN)

L3-dataFigure 4 Classification results on different level s category
hierarchy. The seed information is category surface name .

typography,
5.4.2 Effect of article length. In this paper, we apply HAN

which is a variation of RNN models that takes advantageof text sequences. The HAN model updates a stateaccording to the state of one previous state, so if thelength of article is too long, the information which iscarried by the beginning of the article will contribute lessto the model. Besides, as the length of articles increase,
the training time will also become longer. So, it isnecessary to find a proper length of articles which can(a)maximize the F1 score at the same time decreasing
training time. So, we explore how varying lengths ofarticles affect the performance of the model. We explore
seven different lengths – the leading 10, 15, 20, 25, 30, 35
sentences, and the whole documents. In the experiments
we find that when the document length is trimmed toaround leading 15 sentences, the best F1 score is achieved
while the training time is much smaller than the time used
for the whole documents.

Also, Figure 5 shows when the(b)seed information is category name, we can see the best

FigureF1-micro and -macro scores at length 15, for both L1 and(b) F1-macro score on different lengths of documents, where

L2. Figure 6 shows when the seed information is labeled

the seed information is labeled articles .

articles, the best F1-micro and F1-macro scores at length

Figure 7 shows the training time at varying lengths of

15, for both L1 and L2.

articles, where the seed information is labeled documents(a) F1-macro score on different lengths of documents;

and the dataset is top-level L1. As the length of articles
increases, the training time is increasing. When the article
length is 15 sentences, training time is 3260.77 sec. On theother hand, if we use the whole article, the training time is4735.79 sec, which takes 1474.33 sec more than when
document length is 15 sentences.

(a)Figure 7 Training time for different length of documentsfor L1 dataset. X-axis is document length, and Y-axis istraining time.
(b)
Figure 5 (a) F1-macro score on different lengths of documents

5.4.3 Influence of word dimension for WSTC-HAN. In the(b) F1-micro score on different lengths of documents, whereprevious experiments, we perform the experiments where

the seed information is category name.

word dimension is 100. In this experiment, we show theresults on varying word dimension and varying document
length, for the WSTC-HAN model. The document length isvaried within the leading 5, 10, 15, 20, 25, 30

Table 9 Influence of different word dimensions of different documen t length for WSTC-HAN model
Document lengthWorddimensionF1-score

5

10

15

20

25

30

all

50

F1-Macro
F1-Micro
F1-Macro
F1-Micro
F1-Macro
F1-Micro
F1-Macro
F1-Micro

0.85150.86376
0.873
0.889
0.858270.87655
0.86635
0.88332

0.861130.876960.86777
0.88483
0.861450.879310.86935
0.887280.86883
0.88367
0.88605
0.902760.854210.875860.86662
0.88298

0.85432
0.87956
0.87935
0.894480.857070.87172
0.865430.882780.855130.872380.859430.87310.85397
0.871030.856120.872630.85258
0.87235
0.857180.872410.8536
0.871030.85258
0.87235

0.820290.83108
0.81675
0.828280.830440.832590.829310.83592

100
200
300

Model

Table 10 F1 score for all methods on three datasets. The seed information is labeled documents.
L1 dataset
L2 dataset
L3 dataset

WSTC-CNN
WSTC-HAN
WSTC-HAN(leading 15 sentences)

F1-Macro
0.85230.81675
0.88582

F1-Micro
0.852830.828280.90276

F1-Macro
0.749290.72070.79397

F1-Micro
0.74234
0.71799
0.80911

F1-Macro
0.67047
0.62344
0.67194

F1-Micro
0.67040.62198
0.6704WSTC-HAN(TextRank)

0.894

0.90966

0.80686

0.81697

0.682210.68675

sentences and all the sentences of documents. The seedclassification problem becomes harder and the accuracy

information is labeled documents. We find that the best

declines. We also explored how document lengths affectF1-score is achieved when the word dimension is 50 andthe results, and find that when a proper length for

100, and the document length is first 15 sentences. When

document curtailing is used, it can both increase thethe word dimension is 200 and 300, the best result isaccuracy and decrease the training time. Finally, weachieved with the document length 10 sentences. Besides,

proposed a method that combines summarization byword dimension 100 is showing the overall best result,

TextRank

surpassing dimensions 50, 200, and 300.

models.

5.4.4 Results on combination of WSTC-HAN and TextRank.

harder as category topics become similar to each other. So

In this experiment, we compare the results of different

for future work, we try to deal with overlapping phrases.

and

WSTC-HAN,

Generally,

document

outperforming

all

other

categorization

becomesmodels where the seed information is labeled documen ts.
Table 10 shows that when TextRank is used to extract top

References

15 key sentences, WSTC-HAN outperforms all the other[1] M. B. Aouicha, M. Ali, H. Taieb and M. Ezzeddine,“Derivation of ‘is a ’ taxonomy from Wikipedia
Category Graph” Eng.Appl.Artif.Intell. pp.265-286
2016[2] A. Kittur, “What’s in Wikipedia? Mapping Topics andConflict Using Socially Annotated Category
Structure”, ACM. pp.1509-1512,2009[3] Y. Kim, “Convolutional Neural Network for Sentence
Classification”, EMNLP. pp. 1746-1751, 2014[4] S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent
Convolutional
NeuralNetworksfor
Text
Classification”. AAAI. Vol.333, 2015
[5] R. Mihalcea and P. Tarau, “TextRank: Bridging Order
into Texts”. EMNLP. pp.404-411, 2004
[6] Y. Meng ， J. Shen, C. Zhang and J. Han,
“Weakly-Supervised Neural Text Classification”,
CIKM 2018.
[7] Y. Song and D. Roth, “On Dataless Hierarchical Text
Classification”. AAAI, 2014[8] Z. Yang, D. Yang, C. Dyer, X. He, A. J. Smola, and E.
H. Hovy. “Hierarchical Atthention Neyworks forDocument Classification” HLT-NAACL 2016

models. If we only adopt the leading 15 sentences as thecompressed version of one article, WSTC-HAN cannot
catch important information in the middle or end ofarticles.

On

the

other

hand,

TextRank

identifies

connections between various sentences, and promotes
sentencesthat

containwords

frequently

mentioned

throughout the article. So, instead of only using theleading part of articles, we use TextRank to capture
sentences from the whole article, that are effective forgenerating pseudo documents for self-training of neural
document categorization.

6. Conclusion and Future WorkIn this paper, we discussed weakly supervised neuralmodels for categorizing long articles of Wikipedia.

We

revealed

the

that

as

the

category

level

descends,