Automatic Classification of Wikipedia Articles byUsing Convolutional Neural Network
Keita Tsuji 1
1 Faculty of Library, Information and Media Science, University of Tsukuba, Japan

Abstract: Wikipedia has emerged as an important source of information for
university students. It has been reported that the students tend to start their search with
Google that leads to Wikipedia articles even in university libraries. Recent researchfindings indicate that relatively few students actually search and read books. Within thiscontext, we are now developing a system that recommends books based on the articles
Wikipedia users read in libraries (the system will be added to web browsers of libraries’desktop PCs). Such a system aims to encourage students to read library books as a morereliable source of information rather than relying on Wikipedia articles. Nippon DecimalClassification (NDC) categories are found to be an effective machine learning method
for book recommendation. Therefore, if NDC categories could be assigned to Wikipedia
articles, they might be used as an effective tool for book recommendation. Accordingly,
we developed a method to automatically assign NDC categories to Wikipedia articles byusing convolutional neural network (CNN), which is one of the representative methodsof deep learning. We found that the accuracy of assigning top-level (i.e. Main Class) andsecond-level (i.e. combination of Main Class and Division) of NDC reached 87.7% and74.7%, respectively. These results were achieved by using titles and categories of
Wikipedia articles as input to CNN, while the accuracies obtained by other combinations
such as titles, categories, and main texts were relatively poor.
Keywords: automatic classification; Wikipedia; convolutional neural network;
Nippon Decimal Classification; NDC; category assignment

1. Introduction
Wikipedia has become an important source of information that has been widelyused by university students. It has been reported that the students tend to start
their search with Google that leads to Wikipedia articles even in university
libraries. The recent research findings indicate that relatively few students
actually search and read books (Anbiru et al. 2010). Within this context, we are
now developing a system that recommends books based on the articles
Wikipedia users read in libraries (the system will be added to web browsers oflibraries' desktop PCs). Such system aims to encourage students to read librarybooks as a more reliable source of information rather than relying on Wikipedia
articles. Nippon Decimal Classification (NDC) categories are found to be aneffective machine learning method for book recommendation (Tsuji et al. 2013,2014, 2015). Therefore, if NDC categories could be assigned to Wikipedia
articles, they might be used as an effective tool for book recommendation.
Based on this background, we developed a method to automatically assign
NDC categories to Wikipedia articles by using convolutional neural network(CNN), which is one of the representative methods of deep learning. Whilesome studies have focused on automatically assigning NDC categories toreference records in libraries (Arai & Tsuji 2015), relatively few studies havefocused on Wikipedia articles.

As a first step toward automatic assignment of NDC categories to
Wikipedia articles, the present study focused on NDC categories of books citedin the articles. We believe that Wikipedia articles fall into the same (or similar)
NDC categories as those of books cited in these articles. For instance, if article
X cites many books whose NDC categories are “324,” X is likely to belong to
the “civil law” category. Based on this assumption, we extracted articles thatwere citing many books whose NDC categories were mostly the same and usedsuch articles as training and testing data for CNN. We considered the mostpopular NDC category in each article as a label that CNN should output for that
article (supervisory signals). For instance, if article X cites many books whose
NDC categories are “324,” we assume that CNN should output “324” for article
X.
For input to CNN, information obtained from titles, categories, and main
texts of articles were used. More specifically, we used distributed
representation of nouns contained in titles, categories, and main texts that wereobtained through word2vec. We examined which combination would produce
the best accuracy. The tested combinations were (1) titles only, (2) categories
only, (3) main texts only, (4) titles and categories, (5) titles and main texts, (6)categories and main texts, and (7) titles, categories and main texts.

2. Related Studies
Kim (2014) reported the results of various experiments on CNN textclassification. Texts used were movie reviews, various products reviews,
subjectivity and opinion polarity datasets, Stanford Sentiment Treebank andTREC. Wang et al. (2015) proposed to use CNN for short text classification.
Texts used were Google Snippets and TREC. Johnson & Zhang (2015)
examined effective use of word order for CNN text classification. Texts usedwere movie reviews, electronic product reviews and Reuters news articles.
However, few studies have been conducted on classifying Wikipedia articles.

3. Method
In this section, we will explain the training and testing data sets used in thisstudy, matrix input to CNN, and CNN settings.

3.1 Training and Testing Data
The steps below were followed to obtain the required data for training andtesting purposes:(1) The dump of the Japanese Wikipedia was downloaded fromhttps://dumps.wikimedia.org/jawiki/ (as of February 21, 2015).
(2) The pages whose namespace (ns) tag was “0,” which means that the page
was “Main/Article” (henceforth referred to as “articles”) were extracted.
(3) Word2vec in gensim was used to obtain 200-dimension distributed
representation for each noun in the Wikipedia articles (we used mecab
0.996 for morphological analysis). We set the parameters window andmin_count for Word2Vec module in gensim to 5 and 10, respectively.

(4) The articles extracted in Step (2) were used to extract the articles, whichcontain “References.”(5) Articles with “References” that contained bibliographies of books showing
their ISBNs were extracted.
(6) The correct and consistent books bibliographies were searched by using theISBNs in OpenSearch that provided by the National Diet Library of Japan.
(7) Bibliographies of books that did not contain triple-digit NDCs (i.e. which
consisted of Main Class, Division, and Section) or titles were excluded.
(8) The number of NDCs in each Wikipedia article was counted to identify themost popular NDC and its frequency in the bibliography. As for the level or
depth of NDC, two types were used i.e. Main Class only and the
combination of Main Class and Division. For instance, if an article
contained six books and their NDCs were 324, 324, 324, 325, 367, and 369,
NDC “32” (Main Class is “3” and Division is “2.” Incidentally, “32”represents “Law” and “36” represents “Society”) was considered as themost popular combination in terms of Main Class and Division, and its
frequency was four. When the Main Class was only considered, “3” was themost popular, and its frequency was six.
(9-1) The articles whose Main Class numbers of NDCs obtained through Step(8) were no less than three and the ratios of the most popular ones were noless than 0.8 were extracted. Hence, the example article mentioned in Step(8) was extracted because the number of citing books is six (>=three) andthe ratio is 1.0 (>=0.8). This set of articles will be referred to as L1.
(9-2) Similarly, the articles whose combinations numbers of Main Class andDivision of NDCs obtained through Step (8) were no less than three and theratios of the most popular ones were no less than 0.8 were extracted. Thus,
the example article mentioned in Step (8) was NOT extracted because theratio is 0.67 (=4/6 < 0.8). This set of articles will be referred to as L2.
(10) It is assumed that the most popular Main Class of NDC in each article in
L1 is the one, which should be assigned to that article by CNN. For
instance, the Main Class of NDC that should be assigned to the example
article mentioned in Step (9-1) is “3.” Accordingly, a set of pairs of theabovementioned Main Class of NDCs and correspondent articles weredeveloped as the training and testing data for CNN. Similarly, a set of pairsof the abovementioned combination of Main Class and Division of NDCs
and correspondent articles concerning L2 set were developed.
The total number of articles in the Japanese Wikipedia is 2,689,050 articles,
while the number of articles obtained through Step (2) was 1,104,962. The
number of nouns in Step (3) input to word2vec was 3,645,356. The number ofarticles obtained through Steps (4) and (5) were 95,194 and 28,154,
respectively. The number of ISBNs contained in 28,154 articles, and thus usedin Step (6) was 66,295 (including duplicates). The number of bibliographies
obtained through Step (7) was 61,173 (including duplicates). The number ofarticles that contained at least one such bibliography was 26,879. Therefore,
NDCs were obtained from 28.2% (= 26,879 / 95,194) of articles. For moredetailed information, see Tsuji (2016).
The numbers of articles of L1 and L2 sets are 3,985 and 3,091, respectively.
These were divided into training and testing data sets. The number of articles inthe testing data set from both L1 and L2 sets were 300. While the rest were
included in the training data set, i.e. 3,685 (=3,985-300) for L1 and
2,791(=3,091-300) for L2 set, respectively.

3.2 Matrix Input to CNNFirst, the titles from each article in L1 (and L2) set were extracted andrepresented by 5 times 200 matrix whose i-th row is the 200-dimension vector(obtained through Step (2)) for i-th noun in title. If the number of nouns in acertain title was less than five, the corresponding row was set to zero vector. If
the number of nouns was more than five, only the first five nouns would be
adopted in the title. Henceforth, this matrix was referred to as tl.
Then, the first five categories from each article in L1 (and L2) set wereextracted and represented by 25 times 200 matrix whose (i times j)-th row is the
200-dimension vector (obtained through Step (2)) for j-th noun in i-th category.
If the number of nouns in a certain category was less than five, thecorresponding row was set to zero vector. If the number of nouns was morethan five, only first five nouns in each category would be adopted. Henceforth
this matrix was referred to as cg.
Finally, ten nouns from the main text of each article in L1 (and L2) set
whose TF-IDFs were the highest were extracted. TF-IDF of noun X in article Y
is defined as “the frequency of X in the main text of article Y” times “log (thenumber of articles/the number of articles whose main texts contain X)”. These
were represented by 10 times 200 matrix whose i-th row is the 200-dimension
vector (obtained through Step (2)) for noun whose TF-IDF was the i-th highest
in the main texts. Henceforth this matrix was referred to as tx.
Seven types of matrices were tested as the input to convolutional neuralnetwork. They were (1) tl, (2) cg, (3) tx, and their vertically concatenatedmatrices, as follows: (4) tl+cg, (5) tl+tx, (6) cg+tx and (7) tl+cg+tx (where “+”represents the vertical concatenation). The numbers of their rows are 5, 25, 10,30, 15, 35, and 40, respectively. They all have the same number of columns, i.e.
200.

3.3 CNN Settings
The convolutional neural network is well-known for its ability to automaticallyclassify images. The previously mentioned matrices can be considered as (2D)
images. Similar to visual recognition, five sets of heights of filters were tested,
i.e. (a) {1, 2, 3}, (b) {3, 4, 5}, (c) {1, 2, 3, 4, 5}, (d) {3, 4, 5, 6, 7}, and (e) {1,
2, 3, 4, 5, 6, 7}. The widths of all filters were the same, i.e. 200. Furthermore,
three kinds of number of filters for each filter size for L1 set were tested, i.e. (i)20, (ii) 30 and (iii) 40. While for L2 set, we tested 100, 150, 200, 250, 300, 350,
400, 450, and 500. Therefore, for instance 150 (=5 times 30) filters were used
when the combination of (d) and (ii) were tested for L1 set.
Tensorflow was used to form and train CNN. The strides and padding fortf.nn.conv2d function were set to [1, 1, 1, 1] and VALID, respectively. Rectified
linear units (tf.nn.relu) were used after tf.nn.conv2d. Then we used max
pooling. The ksize, strides and padding for tf.nn.max_pool function were set to[1, IH-FH+1, 1, 1], [1, 1, 1, 1] and VALID, respectively, where IH and FH
represent the height of input matrix and that of the filter. Softmax function wasused in the final output layer. The size of mini batches, the number of epochs,
dropout rate, and learning rate were 300, 300, 0.5, and 0.0001, respectively.
The data flow is illustrated in Figure 1.

Figure 1. Data Flow of the Developed CNN

4. Results
The results are shown in Tables 1 and 2, in which “FS” represents the filter
sizes. For instance, “1 to 5” in FS column indicates that the used filter sizes
were 1, 2, 3, 4, and 5. While “NF” represents the number of filters. “－”represents that we could not apply filters whose heights were more than five totl matrix because the number of row (height) of tl was just five.
As shown in Table 1, the results obtained by using the combination of both
titles and categories (tl+cg) are better compared to those of the othercombinations. Note that using all the titles, categories and main texts(tl+cg+tx) does not outperform using only titles and categories (tl+cg). Using
the main texts of Wikipedia entries might have caused problem determining
their NDC categories. However, considering that only ten nouns with thehighest TF-IDFs were used, thus there might be other effective ways to utilize
main texts.
Based on the preliminary experiment results of L2 set, it is found that better
results were obtained by the filter size set {1, 2, 3, 4, 5} compared to the othersets. The results obtained by this filter size set are shown in Table 2. It can beseen in Table 2 that tl+cg produces higher accuracy compared to the other setswith an accuracy reaches 74.7% when the NF was set at 300.

Table 1. Accuracies of NDC Categories Assignment of L1 Set
FS
1 to 33 to 5

1 to 53 to 71 to 7

NF
100
150
200
250
300
350
400
450
500

NF
20
30
40
20
30
40
20
30
40
20
30
40
20
30
40

tl

cg

tx

70.7
71.0
70.3
70.7
68.7
69.0
68.7
68.3
68.7
－
－
－
－
－
－

80.7
81.0
81.0
81.3
82.3
81.7
82.0
82.3
84.3
81.3
83.3
84.0
82.0
84.0
84.3

79.3
79.7
78.3
77.7
79.0
79.3
78.0
78.0
80.0
78.7
80.7
80.3
79.7
80.3
80.7

tl+cg
84.3
84.3
84.3
83.7
84.3
86.7
84.7
87.7
87.3
84.7
87.7
86.3
87.0
85.7
86.0

tl+tx
78.3
78.0
79.0
78.0
78.7
77.3
78.0
79.3
81.7
78.7
81.0
81.0
79.0
81.0
80.7

tx+cg

tl+cg+tx

82.3
82.7
81.7
80.3
82.3
82.0
82.3
83.7
83.7
83.3
84.7
83.7
82.3
84.3
83.3

82.3
81.3
81.3
80.7
82.0
82.0
83.0
83.0
83.0
81.3
83.3
82.3
83.3
83.3
82.3

Table 2. Accuracies of NDC Categories Assignment of L2
tl
cg
tx
tl+cg
tl+tx
tx+cg tl+cg+tx
52.7
53.7
54.3
54.0
53.3
53.7
53.3
53.3
53.3

67.0
66.7
68.0
72.3
68.3
68.3
68.7
68.3
68.3

65.0
65.0
64.7
65.0
65.7
68.7
69.7
68.0
65.3

72.7
73.7
74.0
74.0
74.7
74.0
73.7
74.0
73.3

64.0
65.3
66.0
66.7
66.7
71.0
68.3
67.7
68.0

70.7
72.3
72.7
72.3
70.3
73.0
71.0
72.0
70.0

70.3
71.7
71.7
72.3
72.0
71.0
71.3
71.7
72.3

The numbers of epochs and accuracies concerning tl+cg and tl+cg+tx with
NF={30, 40, 60} for L1 are shown in Figure 2. We can see in Figure 2 thateven if we increase the number of epochs (to 500), the accuracies do notincrease and the accuracies of tl+cg are higher than those of tl+cg+tx.

Figure 2. Numbers of Epochs and Accuracies

The numbers of NDCs which should be assigned (i.e. label) and those of
NDCs actually assigned by CNN with tl+cg, NF=30 and FS={1, 2, 3, 4, 5} for
L1 are shown in Table 3. Table 3 also shows the accuracy for each NDC label.
We can see in Table 3 that accuracies for NDC labels (i.e. which should be
assigned) which appeared no less than 20 times in our testing data were no lessthan 90.0% (i.e. “2” (History), “4” (Natural Sciences), “5” (Technology), and“7” (The Arts)). On the other hand, the accuracy for “3” (Social Sciences) was
relatively low (50%) although the number of their samples was not extremely
small (i.e. 16). Among the eight articles which the CNN failed to assign “3”were “Western costume (16th century),” “... (15th century),” “... (14thcentury),” “... (13th century),” and “Western costume (11th and 12th
centuries).” The last one was assigned “7” (The Arts) and the rests wereassigned “2” (History). The articles on Western costume in these centuries are
related to history and probably the arts. It might not cause a serious problem if
we recommended books whose NDCs were “2” or “7” to these Wikipedia
articles.
Table 3. Assigned NDCs and Accuracies
NDCs Assigned by CNN with tl+cg, NF=30 and FS={1, 2, 3, 4, 5}

0 (General Works)
1 (Philosophy)
2 (History)
NDCs
3 (Social Sciences)which
4 (Natural Sciences)
should5 (Technology)beAssigned 6 (Industry)
7 (The Arts)
8 (Language)
9 (Literature)
Total

0
0
0
0
0
0
0
0
0
0
0
0

1
0
8
1
0
0
0
0
0
0
1
10

2
1
3
153
6
2
0
0
0
0
2
167

3
1
1
5
8
0
1
1
0
0
0
17

4
1
0
2
1
37
1
0
0
0
0
42

5
0
0
3
0
0
18
1
0
0
0
22

6
0
0
0
0
0
0
0
0
0
0
0

7
0
0
1
1
0
0
0
39
0
1
42

8
0
0
0
0
0
0
0
0
0
0
0

9
0
0
0
0
0
0
0
0
0
0
0

Total

Accuracy3
0.0
12 66.7
165 92.7
16 50.0
39 94.9
20 90.02
0.0
39 100.0
0－4
0.0
300 87.7

4. Conclusions
Based on the assumption that recommending books to Wikipedia reader inlibraries is an effective method to encourage students read books, we developed
a method to automatically assign NDC categories to Wikipedia articles. The
study results showed that using a combination of titles and categories was more
effective than using other combinations such as titles, categories and main
texts. The accuracy of L1 set (i.e. Main Class of NDC) reached 87.7%, which isconsidered significant to help determine which books should be recommended.
Accordingly, our future research will focus on developing a method to actually
determine the books to be recommended using the present study’s results.

References
Anbiru, T. et al. (2010). Information Seeking Behavior. Proceedings of the Spring
Meeting of the Japan Society of Library and Information Science, 87–90. (text inJapanese).
Arai, S. and Tsuji, K. (2015). Automatically Assigning NDC Categories to Reference
Service Records by Using Machine Learning Methods. Journal of Japan Society ofInformation and Knowledge, Vol. 25, No. 1, 23–40. (text in Japanese).
Johnson, R. and Zhang, T. (2015). Effective Use of Word Order for Text Categorization

with Convolutional Neural Networks. Proceedings of Human Language
Technologies: The 2015 Annual Conference of the North American Chapter of the
ACL, 103–112.
Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification.
Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing (EMNLP), 1746–1751.
Mecab. <http://taku910.github.io/mecab/>.
Tensorflow. <https://www.tensorflow.org/>.
Tsuji, K. et al. (2013). Book Recommendation based on Library Loan Records andBibliographic Information. Proceedings of the 3rd International Conference on
Integrated Information (IC-ININFO 2013), 8p. (No Pagination).
Tsuji, K. et al. (2014). Book Recommendation Using Machine Learning Methods Basedon Library Loan Records and Bibliographic Information. Proceedings of the 5thInternational Conference on E-Service and Knowledge Management (ESKM 2014),
76–79.
Tsuji, K. et al. (2015). Book Recommendation Using Machine Learning Methods Basedon Library Loan Records and Bibliographic Information. International Journal of
Academic Library and Information Science, Vol. 3, No.1, 7–23.
Tsuji, K. (2016). Books Cited in Wikipedia: Possibility to Use their Nippon DecimalClassification Categories for Book Recommendation. Proceedings of the 7thInternational Conference on E-Service and Knowledge Management (ESKM 2016),
1196-1197.
Wang, P. et al. (2015). Semantic Clustering and Convolutional Neural Network for
Short Text Categorization. Proceedings of the 53rd Annual Meeting of the
Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing, 352–357.
Word2vec. <https://radimrehurek.com/gensim/models/word2vec.html>.