DEIM Forum 2019 I2-3

Measuring Beginner Friendliness of Chinese Web Pages
explaining Academic Concepts
using Deep Learning and Text/HTML Features
Bingcai HAN† , Hayato SHIOKAWA† , Shintaro OKADA†† , Chiharu HIROHANA† , Takehito
UTSURO††† , Yasuhide KAWADA†††† , and Noriko KANDO†††††† Grad. Sch. of Systems and Information Engineering, University of Tsukuba, Tsukuba, 305-8573, Japan†† College of Engineering Systems, Sch. of Sci. and Eng., University of Tsukuba, Tsukuba, 305-8573, Japan††† Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba, 305-8573, Japan†††† Logworks Co., Ltd., Tokyo 151-0053, Japan††††† National Institute of Informatics, Tokyo, 101-8430, Japan
Abstract Search engine is an important tool of modern academic study, but the results are lack of measurementof beginner friendliness. For improving the eﬃciency of using search engine for academic study, it is necessary
to ﬁnd a method of measuring the beginner friendliness of Web pages explaining academic concepts and to build
an automatic measurement system. In this thesis, we ﬁrst formalize the measurement of beginner friendliness by
several individual factors, including deﬁnition, formula and so on. We collect about 2,000 Web pages for manual
measurement based on the individual factors and build a reference dataset. Then, we analyze the HTML data of thecollected dataset, and extract speciﬁc features for measuring beginner friendliness. And we use a modiﬁed VGG16
model (a convolutional neural network model for image classiﬁcation) to measure the layout of Web pages we havecollected. The results are taken as features for further measurement. All the features are evaluated using SVM
and the performance is shown in a recall-precision curve. Finally, we test about 300 Web pages and evaluate theperformance of diﬀerent features of HTML data and CNN measurement results of Chinese Web pages. The resultof this thesis would be an important reference for further work of a practical assistance system on Web learning.
Key words Web pages explaining academic concepts, beginner friendliness, search engine, deep learning, support
vector machine (SVM), convolutional neural network (CNN)

1 Introduction

beginner friendly Web pages among those ranked at N -th or
higher (N = 1, . . . , 10). With this evidence, we claim that

A search engine is a quite important tool for obtaining

those search engines do not have any systematic criterion on

concerned knowledge when it comes to the study of aca-

measuring beginner friendliness of Web pages explaining aca-

demic concepts. However, if we want to select the beginner

demic concepts. Therefore, it comes up with us to invent afriendly pages during using a search engine, it is necessary

method of measuring the beginner friendliness of Web pages

to compare and to manually select beginner friendly pages.

explaining academic concepts automatically, and ﬁnally to

The reason of the ineﬃcient manual comparison is that thebuild a practical assistance system for promoting academic

state-of-the-art search engines are not designed to selectively

study using a search engine, which would improve the eﬃ-

rank beginner friendly Web pages high in the searched re-ciency of Web learning.

sults. For example, for Chinese Web pages explaining aca-

This thesis presents a method of formalizing the measure-demic concepts, Figure 1 shows an evidence of non-existence

ment of beginner friendliness considering several individualof such systematic criterion on measuring beginner friendli-

factors and automatic measurement of beginner friendlinessness of Web pages ranked at 10th or higher by Baidu.com

using SVM based on HTML features and CNN measurement

and Google.hk, in the case of the overall 56 queries of aca-

results. For collecting reference dataset, we analyze the Web

demic terms from the seven academic ﬁelds of linear algebra,

pages explaining academic concepts based on HTML data,physics, biology, programming, IT, statistics, and chemistry

mainly the structures. And after collecting data, we extract(listed in Table 1). In the ﬁgure, we plot the rates of thespeciﬁc features from HTML data for classiﬁer training. As

(a) IT, statistics, linear algebra, and physics(b) programming, biology, and chemistry

Figure 1 Rate of beginner Friendly Web Pages explaining Academic Concepts ranked at
10th or Higher in the Results of Search Engines     

       



   

 

 

 (a) Overall Measurement as Beginner Friendly(b) Overall Measurement as Beginner Unfriendly

Figure 2 Examples of Web pages that explains an academic concept

a supplement, a modiﬁed CNN model based on VGG16 [10]and results. Section 6 introduces the related work of thisis used for measurement of Web page layout, which is one

research. Finally, Section 7 concludes this thesis.

of individual factors for the overall measurement. The features consist of HTML part and CNN part. At last, we evaluate the performance of feature combinations using recallprecision curves.

2 Factors of Beginner Friendliness of Web
Pages Explaining Academic Concepts
The measurement of individual factors and overall mea-

We use data from Baidu.com and Google.hk, which are

surement are binary decision, and the rules are modiﬁed ac-

two search engines mostly used for academic search in Chi-

cording to the measurement results of same Web pages by 3

nese. The formalization of manual measurement of beginner

members in our group.

friendliness is described with details in Section 2. Section 3

2.1 Individual Factors

shows the details about reference dataset of Web pages ex-

We determine several individual factors to formalize theplaining academic concepts. Section 4 describes the details

measurement of beginner friendliness of Web pages explain-of building a modiﬁed CNN model and the measurement

ing academic concepts. After prior investigation, we abstract

procedure. Section 5 describes the procedure of evaluation

six individual factors including deﬁnition, formula, ﬁgure, example, beginner friendliness of text and Web page layout [1].

Table 1 The Details of Reference Dataset

Each factor is judged based on measurement by manual work

sured as positive for both ﬁgure and example.

according to several rules.

(e) Beginner friendliness of text: measured as positive(a) Deﬁnition: measured as positive when a Web page

when the text of Web page is considered as beginner friendly

contains a correct and precise deﬁnition of the explained aca-by the annotator. The measurement should not take otherdemic concept.

factors into consideration.

(b) Formula: measured as positive when a Web page con-(f) Web page layout: measured as positive when the lay-tains any formula whether in text or ﬁgures. The formulas

out of the Web page is considered as easy to read by theshould be relevant to the academic concept explained in the

annotator. The measurement should not take other factors

Web page.

into consideration.

(c) Figure: measured as positive when a Web page contains ﬁgures or pictures relevant to the academic concept
explained in the Web page, except when the ﬁgure shows

2.2 Overall Measurement considering Individual
Factors
The overall measurement of the beginner friendliness ofany formula only.

Web pages explaining academic concepts is performed by(d) Example: measured as positive when a Web page con-each of our group members. A Web page could be measured

tains examples relevant to the explained academic concept,

as beginner friendly as in the case of the example shown inincluding examples of application, proof, explanation and so

Figure 2(a), or beginner unfriendly as in the case of the ex-on. When the examples shown in ﬁgures, it would be mea-

ample shown in Figure 2(b). The overall measurement isTable 2 Details of Manual Measurement Rules

performed not only based on the combination of individual

data of top 10 Web pages（注 1） for each query determined be-

factors but also based on which factor is measured as pos-fore. Here, the inaccessible and unrelated Web pages would

itive. A Web page explaining academic concept would be

be ruled out.

measured as beginner friendly when the result of the mea-

Among the factors listed in Section 2, those other thansurement of individual factors can be recognized as showing

Web page layout, beginner friendliness of text, and overallthat the page contains necessary and enough information

measurement, we measure the individual factors as described

helpful for beginner learning.

in Section 2. For Web Page layout, beginner friendliness of3 Reference Dataset of Web Pages Explaining Academic Concepts
In this section, we collect the dataset of Web pages ex-

text, and overall measurement, we consider several absoluteand optional rules, whose details are shown in Table 2. The
measurement process using absolute and optional rules areshown below.

plaining academic concepts and develop a reference dataset(a) Beginner friendliness of text: For positive measure-by annotating them with beginner friendliness.

ment, the absolute rules must be all satisﬁed, and any one

3.1 Academic Fields and Concepts for Studyof three optional rules must be satisﬁed. The absolute rule 2

After prior investigation, we determine that the Web

is subjective. If its measurement is not so clear, it should be

pages are collected on 7 academic ﬁelds, including statis-

determined by the optional rules 1 and 2, which means thetics, physics and so on. For each ﬁeld, we choose 8 queries

structure of text is considered as clear if the optional rules 1

for academic concepts based on the materials of high schooland 2 are satisﬁed.

and collage in China, and then we collect the top 10 pagesof each query from the two search engines Baidu.com and
Google.hk. Table 1 shows the details about the collection.
3.2 Procedure（注 1）

Out of those top 10 Web pages, we exclude those from the Web

sites baike.baidu.com, www.baike.com, zh.wikipedia.org, wiki.mbalib.
com, and zh.wikihow.com, and we recollect top 11 to 20 searched Web

For building the reference dataset, we choose the two

pages. Those Web sites share many top ranked Web pages across thosesearch engines Baidu.com and Google.hk for collecting Chi-

7 academic ﬁelds. Each site mostly has Web pages of the same Web

nese Web pages. For each search engine, we collect the URL

page layout. So, it is necessary to exclude Web pages of those ﬁve Web
sites, since Web pages of those ﬁve Web sites tend to have mostly thesame Web page layout, where those Web pages may appear both in
the training data and the development/evaluation data.

Figure 3 Procedure of Modifying VGG16 for Our Task(b) Web Page layout: For positive measurement, the ab-layout of Web pages explaining academic concepts. VGG16

solute rules must be all satisﬁed, and any one of the two

is a CNN model which has won the second prize for image

optional rules must be satisﬁed.

classiﬁcation and ﬁrst prize for single-object location in Im-(c) Overall measurement of beginner friendliness: For

ageNet Large Scale Visual Recognition Challenge(ILSVRC)

positive measurement, the absolute rules must be all satis-in 2014 [6]. This model consists of 13 layers of convolutionalﬁed, and any two of the three optional rules must be satisﬁed.

layers, 5 layers of max pooling layers, 3 fully connected lay-

3.3 Reference Dataseters (the ﬁrst two layers have 4,096 units each and the last

The reference dataset of Web pages explaining academic

layer is a soft-max classiﬁcation layer with 1,000 units rep-concepts contains over 1,000 Web pages. During the mea-

resenting the 1,000 ImageNet classes), and output layer. For

surement, some Web pages showing only academic papers or

the trained VGG16 model, the model is trained with thebooks are considered as unmeasurable and ruled out from theImageNet 2014 dataset for the 1,000 ImageNet classiﬁcation

ﬁnal dataset. The results of manual measurement are shown

task, which has been publicly available. It is also known thatin Table 1. We use the dataset and collected HTML ﬁles forthis trained model can be widely used for other tasks. In thisextracting features, classiﬁer training and evaluation, which

thesis, the experiment and evaluation are carried out using

are described in next section.

the model published in Keras（注 2）, which is a deep learning

4 Measurement of Web Page Layout of
Web Pages Explaining Academic Concepts by Deep Learning

library of Python.
4.2 Procedure of Automatic Measurement of Web
Page Layout
In this thesis, we measured the Web page layout of

4.1 VGG16 [10]Web pages explaining academic concepts using the trained

In recent years, deep learning shows quite high perfor-

VGG16 model as a base model. For building the dataset

mance and reliability in the tasks of various ﬁelds. Especiallyfor VGG16 model training, we take the screen shots of Web

in the ﬁeld of image recognition, the appearance of convolu-

pages we have collected . The details of the collected dataset

tional neural network (CNN) and large scale datasets suchare shown in Table 1. After preparing the dataset, we moved

as ImageNet [5,6] has demonstrated remarkable performance

to prepare the CNN model for the measurement task. As

in various tasks. It is also known that the CNN parameters

shown in Figure 3, we use the trained VGG16 model as a baselearned using large datasets such as ImageNet can be used inmodel. Since the model is trained for 1,000 ImageNet clas-

diﬀerent domain tasks as high performance feature extrac-

siﬁcation task, it is necessary to modify the trained model.

tors. In this thesis, VGG16 model is used as a basic feature
extractor dealing with the task of measurement of Web page

（注 2）

https://keras.io/ja/

First, we remove the 2 fully connected layers and output

small. This feature is multi-valued.

layer which are designed and applicable for 1,000 classiﬁca-(c) HTML tag attribute: the total number of times oftion task, and replace them with new 2 fully connected layers

use of the HTML tag attributes in the page. The measured

and output layer for 2 classiﬁcation task which is to measuretag attributes include index, allowtransparency, data-layout,a Web page in good layout or not in this parer. And then weframeborder, method, data-screen-name and onload.

modify the whole model into a new model, so called modelfeature is multi-valued.

Thiswith ﬁne-tuned VGG16. For this model, we keep the original

As supplement, the features of URL are also used. Here,

10 convolutional layers and ﬁrst 4 max pooling layers with

we use a feature which represents whether the URL includes

their parameters as trained for ImageNet, and then we train

the string “news”. This is a binary feature.

the remain 3 convolutional layers, 1 max pooling layer andfully connected layers, then set new parameters with our prepared dataset. The training and test data details are shownin Table 1.

5 Classiﬁer Learning using Features fromHTML Data and CNN Model and Evaluation
In this section, we use the collected HTML data and mea-

5.2. 2 Features Extracted from Text Contents
We also extract features from text contents which are contained in HTML structure. The details are shown below.
(a) Text Character: the diﬀerence in the number of characters, which equals the result of the number of Chinese characters minus the number of alphabets in each page’s maintext. This feature is multi-valued.
(b) Include “応用 (application)”: the frequency of strings“応用 (application)” in the HTML of each page. This featuresurement results by modiﬁed VGG16 of Web pages explain-is multi-valued.

ing academic concepts for training the classiﬁer and evaluate(c) Include “如何 (how)”: the frequency of characters “如the performance of various features.

何 (how)” in the HTML of each page. This feature is multi-

5.1 Whole Procedure of Evaluation

valued.

Since we would take the CNN measurement results as a(d) Include “· · · 法 (principle)”: the frequency of string

feature, which would be taken in SVM training and eval-patterns “· · · 法 (principle)” in the HTML of each page. Thisuation, we ﬁrst train the modiﬁed VGG16 model with thefeature is multi-valued.

collected dataset（注 3）. After the preparation of dataset with(e) Include “首頁 (home)”: whether there is a string “首頁

all the features built in, the evaluation would be performed(home)” in the HTML of each page. This feature is a binary

by single feature and feature combination separately for com-feature.

prehensiveness and reliability.
5.2 Features

5.2. 3 Features Obtained from CNN Model Measurement
Results

The features used for classiﬁer learning are described as

From the measurement results of section 4.2, we could ob-

below, including two parts as features from HTML data andtain the conﬁdence value of each page on factor of Web page

measurement results form the CNN model. And for features

layout. The conﬁdence value is within the range of [0,1].

from HTML data, there are two source of features as HTMLTaking 0.5 as a midpoint naturally, the page has higher pos-

structure and text contents.

sibility in good Web page layout when the conﬁdence value

5.2. 1 Extracted from HTML Structures

is close to 1, and the page has higher possibility in not good

The features extracted from HTML structures include

Web page layout when the conﬁdence value is close to 0. The

three parts as height of Web page (in pixel), HTML tag andconﬁdence value would be taken as a feature for the overall

HTML tag attribute. The details of the features are shown

measurement.

as below.

5.3 Detailed Procedure of SVM Training(a) Height of Web page: the height of body contents ofFor classiﬁer, we used SVM (sklearn.SVM.SVC tool) [4] in

Web page (in pixel). This feature is multi-valued.

the scikit-learn package. For features that take multi-valued(b) HTML tag: the total number of times of use of thevalues of three or more values, we have evaluated various

HTML tag in the page. The measured tags include media,

types of method for setting value as binary feature with con-iframe, div, sizes, meta, nav, required, td, h1, h2, h3 andsideration of the distribution of value, such as binary features
taken from multiple ranges without overlap, binary features（注 3）

Out of the ﬁve academic ﬁelds for training, we use four for train-taken from multiple ranges with overlap and etc. As a re-ing the modiﬁed VGG16 model and the remaining one for obtaining

sult, a feature setting method is used, which sets left end

the SVM feature by applying the modiﬁed VGG16 model to the remaining one academic ﬁeld. We repeat this procedure ﬁve times andﬁxed and right end with 20 to 40 diﬀerent ranges, since thisobtain the SVM feature for all the ﬁve academic ﬁelds.

setting method achieves the best performance. We use the

1
0.95

HTML Feature

0.9

precision

0.85

CNN Feature

0.8
0.75
0.7
0.65

Text Feature

0.6
0.55
0.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall(a) Performance of Single Features(b) Performance of Various Feature Combinationsw

Figure 4 Evaluation Results

RBF kernel as the kernel function of the SVM and optimize

And we evaluate the performance of using each feature only

the cost parameter C (1 or 10) and the parameter γ (0.01,

to build a baseline for further evaluation. The results are

0.001 and 0.0001) of the RBF kernel by grid search, withshown in Figure 4(a) below. And it is seen that the feature

the area of the recall-precision curves taken as the objectivefrom HTML structure has achieved the best performance

function of optimization. Then we trained the SVM with theamong the single features.

labeled data of overall measurement results.
5.4 Evaluation Results
For now, we have obtained features sorted into 3 categories

To evaluate the best performance of our method, we evaluate the following feature combinations.
(a) All features

as shown below.

(b) All features − CNN feature(a) HTML features: features obtained from the HTML(c) All features − Text feature

structure, mainly the tags.

(d) All features − HTML feature(b) Text features: features obtained from text contents in

Evaluation results are shown in Figure 4(b). It is seen thatthe HTML structure.

when all the features are used, the method achieves the best(c) CNN feature: taken from the CNN measurement re-performance. Among the three of CNN, Text, and HTMLsults as the feature directly.

features, the HTML feature has the most inﬂuence on theperformance since the combinations (a) and (d) have themaximum diﬀerence.

6 Related Work[6]

As a concerned ﬁeld with this research, the community
Question-Answering selection is focused [2, 7], where methods of evaluating and selecting good answers in Question-[7]

Answering system are studied, which is related to the measurement of beginner friendliness discussed in this thesis to acertain extent. For Japanese Web pages explaining academic[8]

concepts, related methods are studied by other members in
our group, including using SVM to perform the measurement[9]

based on HTML structures [3] and image recognition using
deep learning for measuring the visual intelligibility of the
Web page layout [8, 9].
[10]

7 Conclusion
This thesis presents a method of automatic measurementof beginner friendliness of Web pages explaining academic
concepts using SVM, and evaluates the performance of various features from HTML data and CNN measurement results. The evaluation results of SVM show that it has thebest performance when all the features are used for the classiﬁer learning. And it is proved that HTML data is applicable for measuring the beginner friendliness of Web pages
explaining academic concepts. The use of HTML structureis quite important for the automatic measurement according to the comparison ﬁgure. And the image recognition by
deep learning is a reliable method to solve the measurementof Web page layout. It does beneﬁt the overall measurementof beginner friendliness of Web pages explaining academic
concepts. The method presented in this thesis would be abase idea of measuring the beginner friendliness of Web pages
automatically, and it would also be an important reference
for further work of building a complete assistance system foracademic study using a search engine.
References[1] B. Han, H. Shiokawa, K. Kawaguchi, T. Utsuro, and
Y. Kawada. Measuring beginner friendliness of Chinese Web
pages explaining academic concepts using HTML structures. 第 32 回人工知能学会全国大会論文集, 2018.
[2] 石川大介, 酒井哲也, 関洋平, 栗山和子, 神門典子. コミュニティ
QA における良質回答の自動予測. 情報知識学会誌, Vol. 21,
No. 3, pp. 362–382, 2011.
[3] 春日孝秀, 塩川隼人, 韓炳材, 宇津呂武仁, 河田容英. HTML 構
造上の特徴を利用した学術用語解説ウェブページの分かり易さの自動評定. 第 10 回 DEIM フォーラム論文集, 2018.
[4] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn:
Machine learning in Python. Journal of Machine Learning
Research, Vol. 12, pp. 2825–2830, 2011.
[5] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,

S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet large scale visual
recognition challenge. International Journal of Computer
Vision, Vol. 115, No. 3, pp. 211–252, 2015.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein,
A. C. Berg, and F.-F. Li. ImageNet large scale visual recognition challenge. CoRR, Vol. abs/1409.0575, , 2014.
T. Sakai, D. Ishikawa, N. Kando, Y. Seki, K. Kuriyama,
and C.-Y. Lin. Using graded-relevance metrics for evaluating community QA answer selection. In Proc. 4th WSDM,
pp. 187–196, 2011.
塩川隼人, 春日孝秀, 川口輝太, 韓炳材, 宇津呂武仁, 河田容英.
異種の素性を併用した学術用語解説ウェブページの分かり易さの自動評定. 第 32 回人工知能学会全国大会論文集, 2018.
H. Shiokawa, K. Kawaguchi, B. Han, T. Utsuro, Y. Kawada,
M. Yoshioka, and N. Kando. Measuring beginner friendliness of Japanese Web pages explaining academic concepts
by integrating neural image feature and text features. In
Proc. 5th NLPTEA, pp. 143–151, 2018.
K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In Proc. 3rd
ICLR, 2015.