2009
2009IEEE/WIC/ACM
IEEE/WIC/ACMInternational
InternationalJoint
JointConferences
Conference onon Web
Web IntelligenceIntelligence andand Intelligent
Intelligent Agent
Agent Technologies
TechnologyAutomated Web Site Evaluation – An Approach Based on Ranking SVM
Peng Li
Tokyo Institute of Technology
liorlee@nii.ac.jp

Seiji Yamada
National Institute of Informatics
seiji@nii.ac.jp

Abstract

usability, accessibility and security of web sites start toappear, the evaluation tasks are basically carried out byhand. There are limitations to evaluate enormous
amount of web sites manually, so an automatedevaluation system is significantly necessary.
In this paper, we propose a method to evaluate web
sites rely on categories automatically. Specifically, wetake the following 2 steps. First, we focus on the web
sites that are classified into categories and establish anevaluation criterion for each category by applying
machine learning technique dealing with ranking
problems. Then, collect the features of the evaluation
objectives and do the evaluation based on thesefeatures and the criterion of the corresponding
category.
To establish the evaluation criteria, it’s important tomanage to get the evaluation data from the generalpublic automatically on a regular basis. Until recently,
the evaluation tasks are carried out by questionnaire
survey. However, the situation has changed with theavailability of the registration ranking sites. A
registration ranking site is a real-time web site ranking
service based on users’ recommendations and votes. It
has developed significantly during this year. The mostpopular ones like SiteRank TM have hundreds ofthousands of web sites registered, and users aremaking categories, registering new sites, voting andwriting comments frequently. In this research, we use
these categorized ranking information of registration
ranking sites to establish the evaluation criteria of web
sites. In other words, we treat automated web site
evaluation as a classification learning problem whichlearns the discriminant functions from the categorized
ranking information as training data. Since we need an
appropriate learning algorithm to solve this problem,
we use Ranking SVM [1] as our learning algorithm
due to its high accuracy of learning totally orderedmulti-class discriminant functions.

This paper proposes an automated web site
evaluation approach using machine learning to copewith ranking problems. Evaluating web sites is asignificant task for web service because evaluated websites provide useful information for users to estimate
sites’ validation and popularity. Although many
practical approaches have been taken to present ameasuring stick for web sites, their evaluation
functions are set up manually. Thus, we develop amethod to obtain evaluation function using Ranking
SVM and automatically rank web sites with the learnedclassifier. Also we conducted experiments andconfirmed the effectiveness of our approach and its
potential in performing high quality web site
evaluation.

1. Introduction
The estimated Internet user is over 1.5 billion andthe number of web sites has been growing at an
exponential rate. In the April 2009 survey, there areabout 231 million web sites available to users. Web
sites are rich information sources with high update
rates. It's extremely easy, cheap and fast to publish on
the Internet. But due to the fundamental nature of theweb, questionable information is being used recklessly,
without adequately assessing its authority and good
information is being disregarded, because trust in theinformation is lacking. This leaves a large
responsibility on the user to carefully and criticallyevaluate the web sites.
As is widely alleged, there are no unified criteriathat can be fully trusted for web site evaluation. Each
user has different criteria, and it’s almost impossible toget a positive result using the same evaluation criteria
to the web sites belong to different categories.
Moreover, even for the same web site, the evaluation
changes by time. Although many tools in evaluating
978-0-7695-3801-3/09 $26.00 © 2009 IEEE
DOI 10.1109/WI-IAT.2009.224

34

Table 1. Evaluation features classified in
10 fields.

2. Related WorkWeb site evaluation demands appropriate
evaluation measures. Researches such as Web
WISDOM [2] and WebMac [3] use the index achieved
from the findings of library and information science asmeta-information to make the evaluation measures.
But these researches are not aimed as establishingevaluation criteria automatically.
Meanwhile, aside from the approach ofquestionnaire survey, Velayathan and Yamada [4]proposed an approach to automatically evaluate web
pages based on user browsing behavior. However, theevaluation objectives in this research are web pages,
not web sites, and the discriminant functions achieved
can only handle 2 classes such as “interested” and “notinterested” which are not appropriate for web site
evaluation.
Moreover, many tools focus on the evaluation ofusability, accessibility and security of web sites
become available recently. These tools can be divided
into 5 categories: 1. Analysis of server performance; 2.
Analysis of usage(based on log data); 3. Check ofguideline compliance; 4. Analysis of navigation text; 5.
Navigation simulation by cyber agent. Unfortunately,
tools in the comprehensive evaluation of web sites
have not been found.

Field
Top page’s global link popularity
Freshness
Indexable text information
Multi-media contents
Accuracy of spelling and grammar
Accuracy of HTML documents
Contents security
Contents constitution
DesignOthers

3.1. Ranking SVM
Fig. 1 shows the graphical representation of theranking mechanism. The distance from thehyperplane w of a datapoint x is mapped to a one
dimensional space. In this space, T1 ,..., T K 1 are thedifferent thresholds against which the distance iscompared. Data point x having rank i will satisfy
Ti 1  wT x j  Ti . On the contrary, checking which
section the value of wT x j go into will help determine
the rank of x.

3.2. Evaluation Features
In this research, we use the ranking information

T1

D2
T 2 T3

D K 2T k 2

12
1
4
1
4
5
45
4

from registration ranking sites as training data. Eachdata is composed of multiple evaluation features andtheir values, and the data value is the rank determinedby the ranking sites. Meanwhile, using the discriminantfunctions learned from the training data, we can rank
the unranked web sites according to their features.
Data of these unranked web sites are test data.
The traditional evaluation method of questionnaire
survey can deal with both user’s subjective andobjective features. But in this research, our goal is toautomate the evaluation process. Thus the features weneed have to be objective and can be obtained
automatically from web servers. Table. 1 shows the listof such 31 features categorized in 10 fields which areused in our research at the current stage. Detailed
information on each feature is described below.
Global Link Popularity. Global link popularity shows
the popularity of a web page by examining the link
structure of the web. We use Google PageRank here.
Freshness. The freshness of a web site is the contrastof new information and old information. We use
update frequency and changed amount of text
information as a guide.
Indexable Text Information. It’s the amount ofvisible HTML text information obtained by calculating
the size of HTML source file with tag information
removed.
Multi-media Contents. The number of files including
images, videos, audios and flashes are used here.
Accuracy of Spelling and Grammar. The number of
misspellings is counted at the current stage.
Accuracy of HTML Documents. Several features
such as whether the character code and the size ofimages are specified or not are checked.
Contents Security. Features related to the control ofcache, scripts and web robots are checked here.

3. Automated Web Site Evaluation

D1

Number offeatures

T K 1Fig. 1. Ranking mechanism.
35performance. In statistics, the mean absolute error is aquantity used to measure how close forecasts orpredictions are to the eventual outcomes.

Ranking Inform ation

Evaluation Features

Evaluation Criteria

Features of theEvaluation Objective

4.2. Data Preprocessing

web robot

Ranking SVM

The values of evaluation features vary from 0 tomillions, so before the experiment, we scaled the datato [0,1]. Then we extended data vector to n  K  2
dimensions to apply Ranking SVM. Here K is 5, so we
did a 3-dimension extension. Most of the ranking
information is obtained in the form of totally orderedrank. We transformed the ranking information fromtotal order into partial order by dividing data into 5
equally sized groups and labels are assigned to themranging from [1,5].
LibSVM provides four kinds of kernels: linear,polynomial, radial basis function and sigmoid. We dida preliminary experiment to investigate the accuracy of
all these kernels. As a result, we decided to use theRBF kernel in the future experiment due to its highest
accuracy. RBF kernel has two parameters, c and J .
We did a grid search using cross-validation to find thebest c and J for each of the seven categories.

Rank of the Evaluation Objective

Fig. 2. System construction.
Contents Constitution. Proportions of multi-media
contents to text information are used as a guide.
Design. Information about BGM, background image,frames, style sheets, etc are considered here.
Others. Features such as the number of links on thetop page, the number of links overall, whether theauthor of the page is specified and whether thedescription of the page is specified are used here.

3.3. Web Site Evaluation Based on Ranking
SVM
Fig.2 shows the structure of our web site evaluation
system based on Ranking SVM. At the learning phase,
categorized ranking information obtained fromregistration ranking sites and evaluation featurescollected by web robots are input to Ranking SVM tolearn categorized discriminant functions. At the test
phase, unranked web sites get predicted ranks
automatically according to their features and thecorresponding discriminant function.

4.3. Experimental Result
Fig.3 shows the experimental result of ourevaluation experiment. 10-fold Cross-validation wasconducted for each category in order to avoid
influence of biased data. The average of MAE is 0.78,
and the standard deviation is 0.3. That is, in most of
the cases, the predicted value is either equal to the true㪈㪅㪍

4. Experiments㪈㪅㪋

We conducted experiments to verify theeffectiveness of the automated web site evaluation
based on Ranking SVM. The data we used were from a
total of 735 web sites classified into seven categories.
As discussed before, 31 evaluation features areselected. We use libSVM2.84 as a SVM tool.
In addition, we compare the performance ofPerceptron-based ranking algorithm (Pranking [5]),multi-class SVM and Ranking SVM to verify whetherRanking SVM is adapted to our research.

㪤㪘㪜㪈㪅㪉㪈㪅㪇㪇㪅㪏㪇㪅㪍㪇㪅㪋㪘

㪙㪚㪛Category㪜㪝㪞

4.1. Evaluation Measures

Fig. 3. Result of evaluation experiment.

We evaluate web sites with a 5-level evaluation: 1(Excellent) to 5 (Not expected). We use the mean
absolute error (MAE) to evaluate the system

The y-axis represent the values of MAE, and the xaxis correspond to the categories A: computerscience, B: music, C: health and diet, D: movies, E:sports, F: games, G: fashion.
36

due to the change of user group and the biased qualityof web sites. With the development of registration
ranking sites, enough data will become available froma specific site, and the improvement of evaluation
accuracy can be expected.
In addition, in the registration ranking sites,
categories tend to be more and more segmentalized.
It’s believed that the evaluation accuracy will grow if
the categories become more specialized. Furthermore,as mentioned before, with the improvement of web
robot and the development of web technology, moreevaluation features will become available. With more
adapted features acquired, things will change for thelow-accuracy categories.

Table 2. Result of comparative experiment.
MAE

Minimum number of adjacenttranspositions

Ranking SVM

0.78

132.7

Multi-class SVM

0.93

217.4

Pranking

1.12

264.5

value or has a shift of 1 rank. We investigated several
studies which also use MAE with 5-level evaluationand found that it’s considered to be sufficiently
effective if MAE is held under 1.
In addition to the perceptron-based ranking
algorithm, ranking can be treated as a multi-classclassification problem. Table.2 shows the result ofcomparative experiment of the three algorithms. Notethat to get more detailed information, besides MAE,
the performance of the three algorithms was analyzed
by computing the minimum number of adjacenttranspositions needed to bring the predicted value tothe ground truth. It is clear from Table.2 that Ranking
SVM significantly outperformed perceptron-basedranking algorithm and multi-class SVM. Hence itmeans that Ranking SVM is more adapted to ourresearch than other ranking algorithms. It also impliedthat the ranking problem in this research is a nonlinear
problem.

6. Conclusions
In this paper, we proposed an automated approach toevaluate web sites. Ranking SVM was used to learn
discriminant functions through categorized ranking
information obtained from registration ranking sites
and evaluation features collected by web robots.
Unranked web sites can be evaluated automaticallyaccording to their evaluation features and thecorresponding discriminant function. The experimental
results confirmed the effectiveness of our approachand verified that Ranking SVM is an adapted ranking
algorithm to our research.
This paper shows a practical approach to give anautomatic and comprehensive evaluation to any website on the web. It also indicates the potential of thisapproach in performing high quality web site
evaluation.

5. Discussion
From the experimental results, we can see that thegaps of the evaluation accuracy between categories are
not small. The reason why categories with low
accuracy exist can be considered from two aspects.
First, it’s difficult to set unified evaluation criteria forsome categories because of their characteristics. For
example, for health and diet category, the purpose of
the users visiting the web sites may be very different
from that of users for movie and music categories.
They may be looking for health advice, foods, recipes
or even body building equipments. Naturally, different
purposes lead to different evaluation criteria. Second,
it’s possible that one or more important features which
are key to the establishment of evaluation criteria are
not acquired in the experiment. For instance, ifbeautiful images with high resolution are preferred in
fashion category, then it’s difficult to set a highaccuracy criterion without feature describes it.
Data used in this research are from over 700 web
sites. In order to get enough data for each category, we
collected data from multiple registration ranking sites.
This may lead to the decrease of evaluation accuracy

7. References[1] S. Rajaram, A. Garg, X.S. Zhou, and T.S. Huang,“Classification approach towards ranking and sortingproblems”, Proc. of ECML, Cavtat-Dubrovnik, Croatia, 2003,
pp. 301-312.
[2] J.E. Alexander, and M.A. Tate, “Web WISDOM: How toevaluate and create information quality on the Web”,
Lawrence Erlbaum, Hillsdale, NJ, 1999.
[3] M.P. Arnone, and R.V. Small, “WWW motivation
mining finding treasures for teaching evaluation skills grade
1-6”, Linworth Publishing, Worthington, OH, 1999.
[4] G. Velayathan, and S. Yamada, “Behavior-based Web
page evaluation”, Journal of Web Engineering, 2007, pp.
222-243.
[5] K. Crammer, and Y. Singer, “Pranking with ranking”,
Proc of NIPS, MIT Press, 2002, vol. 1, pp. 641-647.

37