DEIM Forum 2019 F7-2

Enhancing Focused Crawler through Genre Detection
Jiayi QIAN†

Tanaphol SUEBCHUA‡

Hayato YAMANA‡†Waseda University, 3-4-1 Ookubo, Shinjuku-ku, Tokyo, 169-8555 Japan
E-mail : †ula666@fuji.waseda.jp, ‡{tanaphol.su, yamana}@yama.info.waseda.ac.jp
Abstract Web crawlers that attempt to download pages related to a pre-defined topic are called focused crawler or topicalcrawlers. A general problem in focused crawler is to predict the relevancy of web pages with a given topic before downloading.
To challenge the problem, in this research, we hypothesize that genre of a web page, such as news and blog, is a useful feature
to estimate the relevancy of the web page. Thus, we propose to utilize the genre of the source web pages to prioritize the unvisited
web pages in the priority queue. According to our experiment on finding web pages related to Baseball and Gaming topic, thegenre feature could enhance the crawling efficiency of the traditional Best-First focused crawler 10%, at most.
Keyword

Focused Crawler，Genre Detection，Web Crawling

1. Introduction

To solve the problem, in this paper, we have an idea to

Focused Crawler is a web crawler that attempts toadopt genre of web pages, i.e., the functional usage of web

download only web pages that are relevant to a pre-defined

pages, to predict their relevancy. For example, web pages

topic. The main problem in focus crawling is to predict thefrom blog and news websites can be considered as different

similarity of the contents of a web page to a pre-defined

genres. We here hypothesize that the probability of a web

topic before downloading. This prediction process enables

page being relevant depends on the genre of its source web

focused crawler to selectively gather relevant web pages as

pages. Suppose that 1) our target topic is informative topic,

much as possible while downloading irrelevant web pages

e.g., education or medical, 2) the crawler downloaded two

as less as possible. It consequently makes a focused

web pages, and 3) the first downloaded web page belongs

crawler to utilize resources, e.g., bandwidth and time, more

to image gallery website, whereas another one belongs toefficiently than general web crawlers such as Breadth-First

blogs website. In this case, the relevant probability of thesearch [4].

links extracted from the image-genre web page should be

To predict whether an unvisited web page is relevant

lower than blog-genre web pages since the image-genre

or not, recent researches use various mechanisms, such as

web pages are not likely to cite informative web pages.

heuristic rule or machine learning, to analyze the content

Based on this idea, we propose a new focused crawlingof the downloaded web pages. The analysis prioritizes themethod that assigns the priority score of each unvisitedunvisited URLs that are out-links of the downloaded web

destination web page in accordance with the genre of thepage. Usually, the URL extracted from high relevant web

downloaded source web page.

pages will have high priority to be downloaded. However,

The outline of this paper is as followed: In section 2,

the prioritization method based on the relevancy of web

we introduce the related work of our research. Our

pages has its own limitation. Suppose that our given topic

proposed method is explained in section 3. Experimental

is chemistry and our crawler visits an entry web page ofprocess and results are shown in section 4, and we conclude

the chemistry department in collage. In this case, the entry

our research in section 5.

page is classified as high relevant web page to the given
topic. However, most of links on this web pages point to

2. Related work

many irrelevant web pages, such as web pages of other

2.1. Focused crawler

departments and collage profiles. In this case, it can be

Chakrabarti et al. [4] first introduced a topic-specificseen that relevancy feature is not enough for prioritize thefocused crawler which specified a topic by documents

relevancy.

instead of using keywords. In their approach, they first use
a classifier to predict the relevancy of the downloaded web
pages. Then, they utilize a module named distiller. Thismodule utilizes the estimated relevancy and linkage

words “image”, “写 真 ” and “画 像 ” in the text of web

between downloaded web pages and unvisited ones tocalculate the priority score. The unvisited web page that is

page.
7)

likely to be an access point to many relevant web pages
will be given more priority than others.

in the web page.
8)

In Rungsawang et al. [2], they defined website segment

Words related to Blog: number of appearances of thewords “blog”, “ブログ” and “日 記 ”

as a group of web pages whose longest directory paths are
the same. For example, http://a.com/gaming/dragonq.html

Image tag: number of elements representing image tagin the text of

web page.
9)

Words related to News: number of appearances of theand http://a.com/gaming/zeldom.html belong to the samewords “news”, “ニ ュ ー ス ” and “新 聞 ” in the text ofwebsite segment. With the definition of website segment,

web page.

the authors proposed to train a machine learning classifier
for finding relevant website segments instead of relevant

These features are used to train several classification

web pages. The web pages belonging to the most probablealgorithms.

Accordingtotheexperimental

results,

relevant web pages will be downloaded at once in each

Decision Tree, Logistic Tree and J48 Tree models achieve

crawling attempt. T hrough testing in real web space, their

better performance than others.

proposed method shows better performance than both
Breadth-First crawler and Best-First focused crawler.

Simaki et al. [5] proposed their study to identify genre
from different text types. They classified texts into fourtypes: short stories, news articles, recipes and dictionaries.

2.2. Genre classificationfor

A genre clue indicator is built to indicate characteristics ofclassifying genre of web pages into the following four

genre within the text, and the results show high accuracy

types:of detecting different genre clue from each other.

1)

Fiol-Roig

et

al.

[1]

proposedanapproach

In Berger et al. [6], the authors aim to detect genre of2)(onlineweb pages in four classes: blog, forum, portals and original

representation of traditional newspapers) that are

websites. They trained the classifier with a combination ofclose to real time.

textual content, structural three groups of features: n-

Blog: a regularly updated web page (a personal online

grams, structural properties and quantitative properties

journalwith a set of 80 features and SVM parameters. Finally, their

News:

a

web

with

page

providingreflections)

which

news

is

orderedchronologically.
3)
4)

Image: a photo gallery on a website.

experiment achieves a high precision of 87.5%.
De Assis et al. [7] aims to distinguish syllabus web

Video: a web page providing videos or a venue forpages from content web pages. In this paper, author defines

sharing videos among every person on the Internet.

web pages containing syllabus as “Other” genre and could
be selected by the crawler.

In their classification approach, they extracted 9

Simaki et al. [5] proposed their study to identify genre

numeric features from the HTML code of web page. The

from different text types. They classified text into four

extracted features are as follows:

types:

Shortstories,News

articles,RecipesandDictionaries. A genre clue indicator is built to indicate
1)
2)Page’s text length: number of text characters in thecharacteristics of genre within text, and the results show

web page.

high accuracy of detecting different genre clue from each

Internal links: number of links belong to the domain

other.

of source web page.
3)
4)
5)
6)

External links: number of links belong to any domain

3. Proposed method

other than the domain of source web page.

3.1. Overview

Words related to Video: number of appearances ofIn this paper, we hypothesize that, the probability of awords “video” and “動 画 ” in the text of web page.

web page being relevant web page depends on the genre ofMultimedia Objects: number of elements representing

its source web pages. Under this assumption, we propose avideos and flash player tag in the web page.

focused crawler that assigns the priority score of each

Words related to Image: number of appearances of theunvisited web page based on both relevancy and genre ofthe downloaded source web page. The structure of ourand genre classifiers, respectively. 𝑤; and 𝑤> represent

proposed focused crawler is shown in Figure 1. Two

the weights. The value of𝑤> is equal to 𝑤> =1-𝑤; .

classifiers are adopted in this crawler; one is a relevant web
relevancy classifier is the same classifier that was proposed

𝑃A> shows the probability of a web page being relevant
pages depending on its downloaded source web pages’in Rungsawang et al.’s work [2]. As for the genre classifier,genre G. 𝑃A> is calculated by using the following equation.

page classifier, another is a web page genre classifier. The

we adopted Fiol-Roig et al.’s one [1]. For each downloaded

𝑃A> =web page, we first utilize both classifiers to predict the

𝑁;
𝑁D

(1)

𝑁; is the total number of links to relevant

relevancy and genre of the web page. The results obtained,wherefrom both classifiers are as follows:

destination pages where the relevancy and genre class ofthe downloaded source web pages belong to class I and G.

I: The predicted relevancy class. Because the web pageclassifier is trained to classify whether the downloaded
web page is relevant web page or not. Thus, I∈{Relevant, Irrelevant}.
𝑅: The probability indicating the relevancy of an input web
page with a given topic. This result is obtained from

As for 𝑁D , it is the total number of source web pages
belong to G.
It can be seen that we have to find out every 𝑃A> forall possible classes I and G. This can be achieved byanalyzing the linkage structure of the subset of the web.
The analysis process is later described in Section 4.3.

relevant web page classifier.
𝐺: The predicted class of genre obtained from genre
classifier. In this research, G ∈ {News, Blog, Image,Video, Other}.

4. ExperimentAccording to our investigation on Yahoo! Japan
Directory,

gamingandbaseball

topic

have

highest

𝐶: The probability indicating a downloaded web page

popularity on the Internet because they have the mostbelongs to its predicted genre class in 𝐺. This output isnumber of URLs in the directory. Hence, in this work, we

also obtained from genre classifier.

select gaming and baseball topics to evaluate our crawler.
Section 4.1 introduces the web page genre classifier

We use all prediction results to calculate the priority

training followed by relevant web page classifier training

score of the URLs extracted from the downloaded web pagein section 4.2. In section 4.3, we calculate 𝑃A> . The

later. The details on our priority scoring function isweighting values are calculated in section 4.4. Section 4.5

described in the following section 3.2.

shows our experimental result.

4.1. Training Web Page Genre Classifier
We classify the genre of web pages based on FiolRoig’s approach [1]. Besides that four genres defined
above, we define “Others” genre that contains any othergenre type’s web pages.
Among all of the classification methods, we selectRandom Forests due to its short run-time and ability to deal
with unbalanced data. Usually, Random Forest starts withdecision tree, a dataset is input at the top of the tree and as
Figure 1: Flow chart of genre based focused crawler

it goes down the tree, the data gets into smaller sets thatare easily to be analyzed. Random Forests consist of a

3.2. Scoring function
In this research, we design the priority scoring
function as follows:
𝑆 = 𝑤; ×𝑅 + 𝑤> ×(𝐶×𝑃A> )group of decision tree classifiers on different sub samplesof the dataset and it uses average number to enhance thepredictive accuracy.
Based on the definition of web page genre, we

In the above equation, as introduced in Section 3.1, R

manually picked up 100 URLs from each category ‘新 聞 ’

and C are the probability obtained from relevant web page

(news), ‘ブログ’ (blog), ‘写 真 ’ (image), ‘動 画 ’ (video),
and randomly picked up 100 URLs from other categories

under Yahoo Japan Directory as the training dataset for

two hops from seed URLs. The number of collected web‘others’ group. With these 500 datasets, we trained apages is shown in Table 1. For gaming topic, the total

Random forest classifier under the test mode with 5-foldnumber of linkages between web pages is 2,033,538 links.

cross validation, and the accuracy of this web page genre

and for baseball topic, the number of linkage is 2,675,069

classifier was 85.4%.

links. The number of links is much bigger than web pages
due to intricate relationship among links in this dataset, for4.2. Training Relevant Web Page Classifier
In this work, we follow Rungsawang et al. [2] to build

instance, one destination web page has several source web
pages.

a web page classifier based on Naïve Bayes Multinomial

We run a Best-first crawler that starts from the seedclassifier to calculate the similarity of the input web page

URLs shown in Table 1 to build a web graph. In thisand the given topic. In addition, we also use this classifiercrawler, we use the relevant web pages and genre classifierto estimate the efficiency for performance evaluation.

to decide the relevancy and genre of each web page in the

For relevant webpage classifier, the text appeared in

web graph. With the linkage structure in a web graph andthe following HTML Tag of an input web page are

the output from both classifiers, we can calculate the 𝑃A>

extracted:

for every classes I and G. The values of

𝑃A>

are

summarized in Table 2 and Table 3.
1)

TITLE

2)

BODY

3)

ANCHORWe manually examine and select URLs from thefollowing categories of Yahoo! Japan Directory for two
topics: ‘ゲーム’ (gaming) and ‘ 野 球 ’ (baseball), andTable 1: Structure of web graphDatasetNumber of web
pages (Baseball)Number of web
pages (Gaming)

82
4,868

156
5,914

37,49742,724

Seed URLs
st
1 hop
nd
2
hop

launch the Breadth-First search (BFS) crawler to collect at
most 300 web pages from each website. All of the web

Table 2: Probability of detecting relevant web pages forpages have been classified by using language classifier“Gaming”

derived from the LangDetect library [3] to filter out nonJapanese web pages. After examined all of the downloaded

𝑃A>
Genre

web pages manually, we got 13,401 relevant web pages

from relevant
sourceform irrelevant
sourcegrouped in positive samples and 15,303 irrelevant web

Blog

89.2%

10.9%

pages grouped in negative samples for gaming topic, and

Image

70.1%

2.7%

30,002 positive and 31,237 negative samples for baseball

News

71.2%

2.3%

topic. We use these datasets to build the web page classifier

Video

64.7%

0.3%

model for each topic by using the 10-fold cross-validation.

Others

87.3%

1.1%

However, it can be seen our dataset is imbalance. Thus, we

ave.

82.9%

1.9%

apply the under-sampling technique to every training foldgenerated by the cross-validation process. Finally, we got
two web page classifier models for classifying the gaming

Table 3: Probability of detecting relevant web pages for“Baseball”

and baseball with 97.84% and 94.07% respectively.

4.3. Calculating 𝑷𝑰𝑮
As mentioned in Section 3.2, we have to find out every

𝑃A>
Genre
Relevant source

Irrelevant source

Blog

76.4%

3.3%

Image

36.7%

1.5%

News

52.7%

4.6%

the URLs of web page that we previously used to trained

Video

69.5%

1.8%

classifiers were removed. We then launched the Breadth-

Others

23.6%

0.4%

First crawler to download and follow the links within in

Total

57.6%

2.6%

𝑃A> for all possible classes I and G. To achieve this task,for each topic, we first select out seed URLs from Yahoo!
Japan Category as our seeds. In our seed selection process,Coverage rate represents the ratio of downloaded
From Table 2 and Table 3, we can see “Blog” genre

relevant web pages with total relevant web pages. In

has the highest probability detecting relevant links for both

baseball dataset, the number of total relevant web pages isgaming and baseball topic. “Other” genre has the lowest

7,124, and for gaming topic, there are 6,901 relevant web

probability for baseball topic, but for gaming topic, it ispages in total. Due to no discard in the simulating crawling

the second highest one. Also, in baseball topic, theprocess, each score function achieves 100% of coveragevariation of probability for each genre has a huge range

rate in final. However, it is difficult to distinguish whichfrom 23.6% to 76.4%. However, in gaming topic, thescore function shows better performance from this figure

probability to detect each genre relevant pages are similar

macroscopically, we use Area Under Curve(AUC) method

to each other. To find out the reason of high probability into calculate the area each score function reaches. Figure 3“Other” genre, we examined on the dataset. We found thatshows the result.

a crowd of web pages that are mostly the official gaming
web site was classified in “Other” genre.

4.4. Finding optimal Weight (𝒘𝑹 and 𝒘𝑮 )As mentioned in Section 3.3, there are two factors in
our priority scoring function, i.e., 𝑅 and (𝐶 × 𝑃MN ). Eachfactor has its own weight, i.e.,𝑤; and 𝑤> . To find out theweights that make the crawler achieves the best result, we
simulate the crawling process on labeled web dataset(mentioned in section 4.3) with score priority scorefunction 𝑆 = 𝑤; ×𝑅 + 𝑤> ×(𝐶×𝑃MN ) in 20 combinations of
Figure 3: Area under curve for each parameter 𝑤;

𝑤; and 𝑤> from 𝑤; equals to 1 to 0 with interval 0.05,
and record the coverage rate of each score function with its
weight. Area Under Curve is used to compare the efficiency

For baseball topic, the area declines gradually asof each function. When 𝑤; equals to 1, priority scoreparameter 𝑤; decreases from 0.95 to 0.05, when 𝑤;

function 𝑆 = 𝑅 represents the baseline, which means genre

equals to 0.95, the area reaches its maximum of 36,595. On

feature does not make any effect in sorting the priority

the contrary, the area of gaming topic shows an unstableorder of web pages and our crawler turns into a Best-firstchange as the variation of w R , the maximum of area iscrawler. Figure 2 shows the coverage rate of baseball topic.

34,814 when w R equals to 0.85. For both two topics, mostHere is the calculation function of coverage rate:of score functions that added genre feature show better
performance than baseline (𝑤; equals to 1).
(2)

4.5. Evaluation on the Internet
For each evaluated topic, we choose the priority
function that produces the best result in our crawling
simulation and test in the real web space as shown in Table
4.
Table 4: Testing score functions for two topics
Priority scorefunction(Baseball)Figure 2: Coverage rate of twenty score function for“Baseball”

Best-First crawler(baseline)
Genre enhanced
crawlerPriority scorefunction(Gaming)
𝑆=𝑅

𝑆 =0.95×𝑅 +0.05×(𝐶×𝑃 )𝑆 =0.90×𝑅 +0.10×(𝐶×𝑃 )For each topic, we launched two crawlers withdifferent

priorityscore

function

under

the

same

environment: initiate crawling at same time with 7 same

seed URLs that are different from our training dataset,
which are selected from top google searching results fromthekeyword“ 野 球 ”(baseball).

The

limitation

According to the result shown in Table 5, our genre

ofenhanced crawler for baseball topic achieves the highest

downloading web pages from same website is set to 300

harvest rate 78.8% after downloading 10,000 web pages.

web pages to prevent duplicated download of same

The baseline crawler shows the worst result among thesecontents. We evaluate the harvest rate of 10,000 web pages

four crawlers. For gaming topic, harvest rates aredownloaded by each crawler. Here is the calculation

approximately 70%, which are higher than the baselinefunction of harvest rate:

crawler achieved 65.8% which utilizes only the relevancy
feature.
(3)

From the Internet evaluation, we can conclude that
focused crawler utilized genre feature shows better

Result is shown in Figure 4 and Figure 5. Table 5
concludes the harvest rate of the results of baseball andperformance than normal best-first crawler on baseball andgaming topic.

gaming topic.

5. Conclusion
In this paper, we proposed a new topic-specificfocused crawling method that incorporates the genre of the
web pages, i.e., the type of web pages, to enhance theharvest rate. We assume that the probability, that we willfind the links to relevant web pages, also depends on thegenre of the source web page. Our method consists of two
web page classifier models; The first one is a web pageclassifier for classifying the relevant web page, whereas
the other classifier is used for classifying the five genre of
web pages. Our experiments on the Baseball and Gaming
Figure 4: Harvest rate of four different crawlers for

topics shows that our genre enhanced Best-First crawler

baseball topic

outperforms the simple Best-First focused crawler around
9.7%, relatively.
Althoughourproposedmethodachieves

bettercrawling results, there are still some place worth for further
improvement. Firstly, in our method, it is not able to adopt
the web page genre classifier without pre-analyzing on the
topic. Secondly, there are a certain number of relevant web
pages exist in web pages belong to “Other” genre. Hence,adding variety of genre is a direction of improvement of
our approach. Furthermore, to ensure the generality of our
proposed method, we plan to evaluate it on more topics.

References
Figure 5: Harvest rate of four different crawlers for[1] G. Fiol-Roig, M. Miró-Julià, and E. Herraiz, “Datagaming topic

mining techniques for web page classification,” Proc. of
9th International Conference on Practical Applications ofTable 5: Harvest rate of Baseball and Gaming topic

Best-First crawler(baseline)
Genre enhanced
crawler

focused crawler
Harvest rate
(Baseball)

Harvest rate
(Gaming)

71.8%

65.8%

78.8%

70.5%

Agents and Multiagent Systems, pp. 61-68, 2011.
[2] A. Rungsawang, T. Suebchua, and B. Manaskasemsak,“Thai related foreign language-specific website segment
crawler,” Proc. of 28th International Conference on
Advanced

Information

Networking

Workshops, pp. 293–29, 2014.

andApplications

[3] N. Shuyo, “Language detection library for java,”
http://code.google.com/p/language-detection, 2010.
[4] S. Chakrabarti, M. van den Berg, and B. Dom, “Focused
crawling: A new approach to topic-specific web resource
discovery,” Proc. of the 8th International World Wide Web
Conference, pp. 1623–1640, 1999.
[5] V. Simaki, S. Stamou, and N. Kirtsis, “Empirical Text
Mining for Genre Detection,” Proc. of International
Conference on Web Information Systems and Technologies(WEBIST), pp. 733-737, 2012.
[6] P. Berger, P. Hennig, M. Schoenberg, and C. Meinel,“Blog, forum or newspaper? Web genre detection using
SVMs,” Proc. of Web Intelligence and Intelligent Agent
Technology (WI-IAT), pp. 64–68. 2015.
[7] G.T. De Assis, A.H. Laender, M.A. Gonçalves, and A.S.
Da Silva, “A genre-aware approach to focused crawling,”
Proc.

of

the

18th

International

Conference, pp. 285-319, 2009.

World

Wide

Web