Applying Outlier Deletion to Analogy Based Cost
Estimation
Masateru Tsunoda

Akito Monden

Mizuho Watanabe1

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

masate-t@is.naist.jp

akito-m@is.naist.jp

mizuho.watanabe01@is.naist.jp

Takeshi Kakimoto

Ken-ichi Matsumoto

Kagawa National College ofTechnology
355 Chokushicho, Takamatsu-shi,
Kagawa 761-8058 Japan

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japankakimoto@t.kagawa-nct.ac.jp

matumoto@is.naist.jp

ABSTRACTIn this research, we apply outlier deletion methods to analogy
based software effort estimation to evaluate their effects. We employed existing deletion methods (Cook’s distance based deletion,and Mantel’s correlation based deletion) and new deletion method
proposed in this research. While existing deletion methods eliminates outliers from entire dataset before estimation, our method
identifies and eliminates outliers from neighborhood projects of
estimation target. Our method treats a project as an outlier whenthe effort of the project is extremely higher or lower than otherneighborhood projects. In the experiment, our method showed
highest performance among applied deletion methods, and average BRE (Balanced Relative Error) indicated 20.8% improvementby our method.

Categories and Subject Descriptors
D.2.9 [Software Engineering]: Management – Cost estimation,
K.6.1 [Computing Milieux]: Project and People Management –Staffing

General Terms(neighborhood projects) which are similar to the estimated projectfrom past project dataset, and estimates effort based on similar
projects’ effort. One of the advantages of analogy based estimation is that estimation results are comprehensible for estimatorssuch as project managers [22], because they can confirm neighborhood projects used for estimation. Although ordinary estimation models like linear regression model estimate various target
projects’ effort by one model, analogy based estimation does notmake such a model, and estimates effort by neighborhood projects’ effort. So analogy based estimation can reflect individualityof each target project in estimation.
Past project dataset sometimes includes project data which should
not be used for estimation [18]. For example, projects where exceptional amount of reworks were occurred have larger effort thanother same scale projects. Additionally, when effort was inaccurately collected or recorded, recorded effort is different from actual effort. These projects should be eliminated from dataset before
estimation, because they lessen estimation accuracy. However, toidentify these projects is not easy because inside details of eachproject are usually not recorded in the dataset. Even if such details
can be grasped, it is difficult to settle elimination criteria (Forexample, to settle criterion of abnormal amount of reworks).

Management, Measurement, Economics, Experimentation.

Keywords
Case based reasoning, effort prediction, abnormal value, project
management, productivity.

1. INTRODUCTION
To achieve success of software development project, it is important to estimate development effort accurately, and therefore
many quantitative estimation methods have been proposed[2][17][20]. Recently, analogy based estimation [19] gets attention, and many proposals and case studies have been reported[6][7][13][21][22]. Analogy based estimation selects projects
1

Presently with IBM Japan, Ltd.

Thus, such projects are often eliminated by statistical outlier deletion methods. Outlier deletion methods identify projects as outliers when specific variants’ values are extremely large or combination of variants’ values (effort, system size, or duration) is fairly
different from other projects’ one, and remove them from dataset.
Cook’s distance is widely used as outlier deletion method when
applying linear regression analysis. In addition to Cook’s distance,some outlier deletion methods for effort estimation [6][18] havebeen proposed. However, there are very few case studies which
apply outlier deletion methods to analogy based estimation, andevaluate effect of outlier deletion methods toward analogy based
estimation.
In this research, we apply outlier deletion methods to analogy
based estimation to evaluate their effects. Two types of outlier
deletion methods (Cook’s distance based deletion, and Mantel’scorrelation based deletion) are applied to ISBSG dataset [5], anddevelopment effort is estimated by analogy based estimation.
ISBSG dataset includes many project data which are collectedfrom software development companies, and it is widely used inmany researches.
Also, we propose new outlier deletion method considering characteristics of analogy based estimation, and compare its effect toothers. In analogy based estimation, when variance of actual effortof neighborhood projects is large, estimation accuracy gets low[16]. Our method identifies a project as an outlier when the effortof the project is extremely higher or lower than other neighborhood projects, and excludes it from computation of estimatedeffort. Actual effort of neighborhood projects is normalized by Zscore computation [9], and when the normalized value is larger
than threshold, the project is identified as an outlier. While existing deletion methods eliminates outliers from entire dataset before
estimation, our method does from neighborhood projects.
In what follows, Section 2 explains analogy based estimation.
Section 3 explains outlier deletion methods, and Section 4 describes experimental settings. Section 5 shows results of the experiment and discusses it, and Section 6 concludes the paper witha summary.

2. ANALOGY BASED ESTIMATIONThe origin of analogy based estimation is CBR (case based reasoning), which is studied in artificial intelligence field. Shepperd
et al. [19] applied CBR to software development effort estimation.
CBR selects a case similar to current issue from accumulated pastcases, and applies solution of the case to the issue. CBR assumes
similar issues can be solved by similar solution. Analogy based
estimation assumes neighborhood (similar) projects (For example,
development size and used programming language is similar) have
similar effort, and estimates effort based on neighborhood projects’ effort. Although ready-made estimation models such as
COCOMO [2] can make estimation without stored software project dataset, analogy based estimation cannot estimate without it.
It is a weak point of analogy based estimation, but it can be overcome by using public dataset.
Analogy based estimation uses m×n matrix shown in Table 1. In
the matrix, Proji is i-th project, Metricj is j-th variable, xij is avalue of Metricj of Proji, fpi is the development size (e.g. function
point) of Proji, and yi is the actual effort of Proji. We presume
Proja is estimated project, and ŷa is the estimated value of ya.
Procedures of analogy based estimation consist of the three steps
described below.
Step 1: Since each variable Metricj has different range of value,
this step makes the ranges [0, 1]. The value x´ij, normalized thevalue of xij is calculated by:xij 

xij  min Mertic j 

max Metric j   min Metric j (1)

In the equation, max(Metricj) and min(Metricj) denote the maximum and minimum value of Metricj respectively.
Step 2: To find projects which are similar to estimated project
Proja (i.e. identifying neighborhood projects), distance between
Proja and other projects Proji is calculated. Although various

Table 1. Dataset used by analogy based estimation
Effort Size Metric1 Metric 2 … Metric j … Metric n
Proj1

y1

fp1

x11

x12

…

x1j

…

x1n

Proj2

y2

fp2

x21

x22

…

x2j

…

x2n

…

…

…

…

…

Proji

yi

fpi

xi1

xi2

…

…

…

…

…

Projm

ym

fpm

xm1

xm2

…
…

xij

…
…

…
…

xmj

xin
…

…

xmn

measures (e.g. a measure directly handling nominal variables) areproposed [1], we applied Euclidean distance measure because it is
widely used [21]. In the measure, short distance indicates two
projects are similar. Distance Dist(Proja, Proji) between Proja andProji is calculated by:DistProja , Proji  m x   x  
h 1

2

ah

(2)

ih h

Step 3: The estimated effort ŷa of project Proja is calculated byactual effort yi of k neighborhood projects. While average ofneighborhood projects’ effort is generally used, we adopt size
adjustment method, which showed high estimation accuracy insome researches [7][13][22]. Size adjustment method assumes
effort yi is s times (s is real number greater than 0) larger whendevelopment size fpi is s times larger, and the method adjusts
effort yi based on ratio of estimated project’s size fpa and neighborhood project’s size fpi. Adjusted effort adjyi is calculated byequation (3), and estimated effort ŷa is calculated by equation (4).
In the equation, Simprojects denotes the set of k neighborhood
projects which have top similarity with Proja.
adjyi  yi 

fp a
fp i

 adjyhyˆ a  hSimprojects
k

(3)

(4)

3. OUTLIER DELETION METHOD
Outlier deletion method examines whether a case (project) indataset is an outlier or not, and eliminates it from dataset when it
is identified as an outlier. When software development effort isestimated, Cook’s distance based deletion is widely applied before
building a linear regression model to eliminate outliers (e.g., [12]).
However, there are few researches which analyzed effects of outlier deletion methods [18], or proposed outlier deletion method
suitable for analogy based estimation [6].
Seo et al. [18] proposed that LTS (least trimmed squares) based
deletion and k-means based deletion are applied before effort
estimation, and evaluated their effects by estimating development
effort with linear regression model, neural network, and Bayesian
network. However, they did not use analogy based estimation witha deletion method. Keung et al. [6] proposed Mantel’s correlation
based deletion. Although they analyzed which projects were elim-inated, they did not compare it with other deletion method, estimating development effort with cross validation. Outlier deletion
methods used in our research are explained below.

3.1 Cook’s Distance Based Deletion
Cook’s distance based deletion is used with multiple linear regression analysis, and identifies an outlier when the case greatly varies
coefficient of the regression model. Cook’s distance indicates how
much residual of all cases varies when a certain case is omitted
from model building. Large Cook’s distance means the case greatly affects the model. A case is eliminated from dataset when
Cook’s distance is larger than 4 / n (n is the number of cases inthe dataset). Although Cook’s distance based deletion is used
when linear regression model is built, we applied it to analogy
based estimation, because it is widely used in many effort estimation researches.

3.2 Mantel’s Correlation Based Deletion
Mantel’s correlation based deletion identifies an outlier when aset of independent variables’ values is similar, but dependent
variable’s value is not similar to other cases. The method is originally proposed in Analogy-X method [6] designed for analogy
based estimation. Analogy-X method is (1) delivering a statisticalbasis, (2) detecting a statistically significant relationship and reject non-significant relationships, (3) providing simple mechanism for variable selection, (4) identifying abnormal data point(project) within a dataset, and (5) supporting sensitivity analysis
that can detect spurious correlations in a dataset. We applied function (3) as outlier deletion method.
While ordinary correlation coefficient like Pearson’s correlation
denotes strength of relationship between two variables, Mantel’scorrelation does between two set of variables (i.e. a set of independent variables and a dependent variable). Mantel’s correlation
clarifies whether development effort (dependent variable) is similar or not, when project attributes like duration or development
size (a set of independent variable) is similar. To settle Mantel’scorrelation, Euclidean distance based on independent variables
and Euclidean distance based on dependent variable is calculated,
and then correlation coefficient of them is calculated.
Mantel’s correlation based deletion identifies outliers by the following procedure.
1.

For all projects, Mantel’s correlation ri is calculated by excluding i-th project.

2.

Average of ri ( r ), and standard deviation of ri (rs) are calculated.

3.

Leverage metric lmi, impact of i-th project on r is calculated by the following equation:lmi  ri  r
4.

(5)

lmi is divided by rs, and when the value (standard score) islarger than 4, the project is eliminated from dataset.

3.3 Neighborhood’s effort based deletion
Our method, neighborhood’s effort based deletion identifies anoutlier when effort of a project is extremely higher or lower thanother neighborhood projects. As stated in section 2, procedure ofanalogy based estimation consists of range normalization (step 1),
neighborhood projects selection (step 2), and estimated effort

computation (step 3). When effort of neighborhood projects is nothomogeneous in step 2, estimation accuracy gets low [16]. Focusing on the issue, our method identifies an outlier in step 2. Analogy based estimation assumes when characteristics (independent
variables’ values) of project is similar, effort (dependent variable’s value) is also similar. Our method treats a project as an outlier when the project is not fit to the assumption. While existing
deletion methods eliminates outliers from entire dataset before
estimation, our method eliminates outliers after selecting neighborhood projects.
To identify an outlier, effort of each neighborhood project is compared to average of neighborhood projects’ effort. However, when
variance of neighborhood projects’ effort is large, each project’s
deviation from the average effort is also large. So Z-score computation [9] is applied to standardize each neighborhood’s effort
before the comparison. In more detail, our method eliminates
outliers from k neighborhood projects as follows. Note that although neighborhood project’s effort is denoted by yi, yi signifies
size adjusted effort, not actual effort when using size adjustment
method.
1.

Average of yi ( y ) and standard deviation of yi (ys) is calculated.

2.

Standardized effort y’i is calculated by the following equation (Z-score):

yi 
3.

yi  yys(6)

A project is identified as an outlier and eliminated whenabsolute value of y’i is greater than threshold th (i.e. when
deviation from y is greater than th times ys). Note that if allneighborhood projects are identified as outliers, no project
is eliminated. Estimated effort ŷ a is calculated by the following equation:yˆ a 

hEliminatedprojects

k d

yh

(7)

In the equation, Eliminatedprojects denotes a set of k neighborhood projects excluded d outlier projects. We set th as
1.65 (one sided 5% of standard normal distribution).
Some researches pointed out that when productivity (development
size / effort) of neighborhood projects is not homogeneous, estimation accuracy with size adjustment method gets low [7][13].
Our method with size adjustment is regarded to eliminate outliers
based on productivity. When size adjustment is used, our method
eliminates a project whose adjusted effort adjyi is extremely higher or lower. From equation (3), adjyi is calculated by multiplying
estimated project’s size fpa by the reciprocal productivity yi / fpi(yi is effort of a neighborhood project, and fpi is development sizeof its). fpa is same for all neighborhood projects, and therefore it isregarded as a constant. Therefore, adjyi is regarded as productivity
in our method.
Our method is totally different from neighborhood selection suchas distance-based neighborhood selection. When neighborhoods
are selected, only independent variables are used and dependent
variable is not used, because the value of dependent variable ofthe target project is unknown. On the contrary, our method is used
independent variable.

4. EXPERIMENT
4.1 Dataset

4.3 Experimental Procedure

To evaluate existing deletion methods and our method, we compared estimation accuracy of analogy based estimation when eachoutlier deletion method is applied. We used ISBSG dataset [5],which is provided by International Software Benchmark Standard
Group (ISBSG). It includes project data collected from softwaredevelopment companies in 20 countries, and the projects werecarried out between 1989 and 2004.

1.

Dataset is randomly divided into two equal set. One is treated as fit dataset, and the other is treated as test dataset. Fit
dataset is used to compute estimated effort (regarded as past
projects), and test dataset is used as estimation target (regarded as ongoing projects).

2.

Outlier deletion method is applied to fit dataset, to eliminate
outliers from fit dataset.

3.

To decide neighborhood size k, estimation for fit dataset is
performed, changing k from 1 to 20. After estimation, residual sum of square (It is same as sum of squares of AE, andwidely used for estimation model selection [10]) is calculated, and k which shows smallest residual sum of square isadopted.

4.

Estimation for test dataset is performed. k which is settled atstep 3 is used.

5.

Evaluation criteria are calculated by actual effort of test
dataset and estimated effort.

6.

Step 1 to 5 is repeated 10 times (As a result, 10 sets of fit
dataset, test dataset, and evaluation criteria are made).

Experimental procedure for existing deletion methods is follows:

We assumed estimation point is the end of project plan phase. So,
only variables whose values were fixed at the point were used as
independent variables, although 99 variables are recorded in thedataset. The independent variables are same as the previous study[11] (unadjusted function point, development type, programming
language, and development platform). Development type, programming language, and development platform were transformed
into dummy variables, because they are nominal scale variables.
ISBSG dataset includes low quality project data (Data quality
ratings are also included in the dataset). So we extracted projects
based on the previous study [11] (Data quality rating is A or B,
and function point was recorded by IFUPG method, and so on).
Also, we excluded projects which included missing values. As aresult, we used 593 projects.

4.2 Evaluation criteria

Experimental procedure for our method is follows:

To evaluate accuracy of effort estimation, we used average andmedian of AE (Absolute Error), MRE (Magnitude of Relative
Error) [4], MER (Magnitude of Error Relative to the estimate) [8],and BRE (Balanced Relative Error) [14].

1.

Dataset is randomly divided into two equal set. One is treated as fit dataset, and the other is treated as test dataset.

2.

Estimation for fit dataset is performed, changing k from 1 to
20. After estimation, residual sum of square is calculated,and k which shows smallest residual sum of square is adopted.

(8)

3.

Estimation for test dataset is performed with our method. kwhich is settled at step 2 is used.

(9)

4.

Evaluation criteria are calculated by actual effort of test
dataset and estimated effort.

5.

Step 1 to 4 is repeated 10 times.

When x denotes actual effort, and x̂ denotes estimated effort,
each criterion is calculated by the following equations:AE  x  xˆ

MRE 
MER 

x  xˆxx  xˆxˆ

 x  xˆ, x  xˆ  0 xˆBRE  ˆxx, x  xˆ  0 


 xˆ


(10)

5. RESULTS AND DISCUSSION(11)

Lower value of each criterion indicates higher estimation accuracy.
Intuitively, MRE means relative error to actual effort, and MERmeans relative error to estimated value. However, MRE and MER
have biases for evaluating under and over estimation [3][10].
Maximum MRE is 1 even if terrible underestimate is occurred(For instance, when actual effort is 1000 person-hour, and estimated effort is 0 person-hour, MRE is 1). Similarly, maximum
MER is smaller than 1 when overestimate is occurred. So in addition to MRE and MER, we adopted BRE whose evaluation is notbiased both MRE and MER [15]. We did not use Pred(25) [4]which is sometimes used as an evaluation criterion, because
Pred(25) is based on MRE and it has also bias for evaluating under estimation.

Experimental results are shown in Table 2 and Table 3. Table 2
denotes the values of evaluation criteria and deletion ratio wheneach outlier method is applied. Deletion ratio is defined as thenumber of deleted projects / the number of all projects. When our
method is applied, it is defined as average of the number of deleted projects / neighborhood size. The values of evaluation criteria
are average for 10 test dataset. Table 3 denotes differences of
evaluation criteria between when each outlier deletion method isapplied and not applied. Negative values mean evaluation criteria
got worse by applying outlier deletion method. Table 3 also shows
statistical test results for the difference by Wilcoxon signed-rank
test (The values of evaluation criteria did not have normal distribution). Significant level was set as 5%, and italicized figures inthe table signify there were significant differences.
When our method was applied, average MER got worse, but othercriteria got better (Average AE, median AE, and median MRE
showed significant difference). Especially, average BRE showed
20.8% improvement and median BRE did 5.3%. In case of Man-

Table 2. Evaluation criteria of each outlier deletion method
Outlier deletion
methodAverage
AEMedianAE

Average
MRE

MedianMRE

Average
MER

MedianMER

Average
BRE

MedianBRE

DeletionrationNot applied

3680

1607

166.93%59.59%

91.27%

58.05%

210.36%91.69%

0.00%

Neighborhood’s effort

3252

1374

138.07%55.98%

97.35%

56.67%

189.56%86.36%

7.75%

Mantel’s correlation

3549

1603

162.40%60.40%

92.61%

58.48%

207.17%92.01%

1.21%

Cook’s distance

3371

1611

154.69%62.50%

110.35%58.80%

216.35%99.05%

4.65%

Table 3. Difference of evaluation criteria of each outlier deletion method
Outlier deletion
methodAverage
AEMedianAE

Average
MRE

MedianMRE

Average
MER

MedianMER

Average
BRE

MedianBRE

Neighborhood’s effort

Difference
p-value

428
0.03

234
0.00

28.9%
0.06

3.6%
0.03

-6.1%
0.43

1.4%
0.70

20.8%
0.08

5.3%
0.06

Mantel’s correlation

Difference
p-value

131
0.08

4
0.58

4.5%
0.01

-0.8%
0.25

-1.3%
0.43

-0.4%
0.74

3.2%
0.19

-0.3%
0.84

Cook’s distanceDifference
p-value

309
0.01

-4
0.70

12.2%
0.06

-2.9%
0.20

-19.1%
0.06

-0.8%
0.85

-6.0%
0.43

-7.4%
0.77

tel’s correlation based deletion, four evaluation criteria out ofeight were improved, and lowering of the other criteria were
smaller than 1.3%. However, the extent of improvement was 3.2%
on average BRE, and it was very small on median BRE (Onlyaverage MRE showed significant difference). When Cook’s distance based deletion was applied, only two evaluation criteria outof eight were improved, and moreover, degradation of average
BRE and median BRE were more than 6%.
We should carefully understand the results because we used only
one dataset, but at least, our method shows high performance toeliminate outliers toward a certain kind of dataset (i.e. ISBSG
dataset), and we could say that our method is promising. The effect of Mantel’s correlation based deletion is not very strong whenapplied to ISBSG dataset. The result does not means Mantel’scorrelation based deletion is always not effective to eliminate
outliers, but means it is not very effective to a certain kind of dataset. Applying Cook’s distance based deletion made estimation
accuracy lower. Cook’s distance based deletion may be effective
to other dataset, but it is not fit to a certain kind of dataset, andtherefore we should consider whether Cook’s distance based deletion apply or not before using analogy based estimation.
While existing deletion methods identifies outliers from wholedataset, our method does from neighborhood projects of the estimated project. The characteristics of our method may be effective
especially when it is applied to dataset like ISBSG dataset which
includes various projects.

6. CONCLUSIONSIn this research, we applied outlier deletion methods to analogy
based software development effort estimation, and evaluated theireffects. Also, we propose new outlier deletion method for analogy
based estimation. While existing deletion methods eliminates
outliers from entire dataset before estimation, our method does

after neighborhood projects are selected by analogy based estimation. In our method, when the effort of the project is extremely
higher or lower than other neighborhood projects, it is not used
for effort estimation. In the experiment, we estimated development effort using ISBSG dataset, and in the results, our method ismost effective, Mantel’s correlation based deletion is not very
effective, and Cook’s distance based deletion made estimation
accuracy lower. As future work, we will apply other deletion
methods to other dataset and compare their effects to enhancereliability of our research.

7. ACKNOWLEDGMENTS
This work is being conducted as a part of the StagE project, The
Development of Next-Generation IT Infrastructure, and Grant-inaid for Young Scientists (B), 22700034, 2010, supported by the
Ministry of Education, Culture, Sports, Science and Technology.
We would like to thank Dr. Naoki Ohsugi for offering the collaborative filtering tool.

8. REFERENCES[1] Angelis, L., and Stamelos, I. 2000. A Simulation Tool for
Efficient Analogy Based Cost Estimation, Empirical Software Engineering, 5, 1, 35-68.
[2] Boehm, B. 1981. Software Engineering Economics. PrenticeHall.
[3] Burgess, C., and Lefley, M. 2001. Can genetic programming
improve software effort estimation? A comparative evaluation. Journal of Information and Software Technology, 43,
14, 863-873.
[4] Conte, S., Dunsmore, H., and Shen, V. 1986. Software Engineering, Metrics and Models. Benjamin/Cummings.

[5] International Software Benchmarking Standards Group(ISBSG). 2004. ISBSG Estimating: Benchmarking and research suite. ISBSG.

[14] Miyazaki, Y., Terakado, M., Ozaki, K., and Nozaki, H. 1994.
Robust Regression for Developing Software Estimation
Models. Journal of Systems and Software, 27, 1, 3-16.

[6] Keung, J., Kitchenham, B., and Jeffery, R. 2008. Analogy-X:
Providing Statistical Inference to Analogy-Based Software
Cost Estimation. IEEE Trans. on Software Eng. 34, 4, 471484.

[15] Mølokken-Østvold, K., and Jørgensen, M. 2005. A Comparison of Software Project Overruns-Flexible versus Sequential
Development Models, IEEE Trans. on Software Eng. 31, 9,
754-766.

[7] Kirsopp, C., Mendes, E., Premraj, R., and Shepperd, M.
2003. An Empirical Analysis of Linear Adaptation Techniques for Case-Based Prediction. In Proc. of International
Conference Case-Based Reasoning, Trondheim, Norway,
June 2003, 231-245.

[16] Ohsugi, N., Monden, A., Kikuchi, N., Barker, M., Tsunoda,
M., Kakimoto, T., and Matsumoto, K. 2007. Is This Cost Estimate Reliable? - the Relationship between Homogeneity of
Analogues and Estimation Reliability. In Proc. of the International Symposium on Empirical Software Engineering and
Measurement (ESEM), Madrid, Spain, September 2007, 384392.

[8] Kitchenham, B., MacDonell, S., Pickard, L., and Shepperd,
M. 2001. What Accuracy Statistics Really Measure. In Proc.
of IEE Software, 148, 3, 81-85.
[9] Larsen, R., and Marx, M. 2000. An Introduction to Mathematical Statistics and Its Applications. Prentice Hall.
[10] Lokan, C. 2005. What Should You Optimize When Building
an Estimation Model?, In Proc. of International Software
Metrics Symposium (METRICS), Como, Italy, Sep. 2005, 34.
[11] Lokan, C., and Mendes, E. 2006. Cross-company and singlecompany effort models using the ISBSG Database: a further
replicated study, In Proc. of the International Symposium on
Empirical Software Engineering (ISESE), Rio de Janeiro,
Brazil, Sep. 2006, 75-84.
[12] Mendes, E., Martino, S., Ferrucci, F., and Gravino, C. 2008.
Cross-company vs. single-company web effort models using
the Tukutuku database: An extended study. The Journal of
Systems and Software, 81, 5, 673-690.
[13] Mendes, E., Mosley, N., and Counsell, S. 2003. A Replicated
Assessment of the Use of Adaptation Rules to Improve Web
Cost Estimation. In Proc. of the International Symposium on
Empirical Software Engineering (ISESE), Rome, Italy, September 2003, 100-109.

[17] Selby, R., and Porter, A. 1988, Learning from examples:
generation and evaluation of decision trees for software resource analysis. IEEE Trans. on Software Eng. 14, 12, 743757.
[18] Seo, Y., Yoon, K., and Bae, D. 2008. An Empirical Analysis
of Software Effort Estimation with Outlier Elimination. In
proc. of the international workshop on Predictor models insoftware engineering (PROMISE), Leipzig, Germany, May
2008, 25-32.
[19] Shepperd, M., and Schofield, C. 1997. Estimating software
project effort using analogies. IEEE Trans. on Software Eng.
23, 12, 736-743.
[20] Srinivasan, K., and Fisher, D. 1995. Machine Learning Approaches to Estimating Software Development Effort. IEEE
Trans. on Software Eng. 21, 2, 126-137.
[21] Tosun, A., Turhan, B., and Bener, A. 2009. Feature
weighting heuristics for analogy-based effort estimation
models. Expert Systems with Applications, 36, 7, 1032510333.
[22] Walkerden, F., and R. Jeffery. 1999. An Empirical Study of
Analogy-based Software Effort Estimation. Empirical Software Engineering, 4, 2, 135-158.