Incorporating Expert Judgment into Regression
Models of Software Effort Estimation
Masateru Tsunoda

Akito Monden

Faculty of Information Sciences and Arts
Toyo University
Saitama, Japantsunoda@toyo.jp

Graduate School of Information Science
Nara Institute of Science and Technology
Nara, Japanakito-m@is.naist.jp

Jacky Keung

Kenichi Matsumoto

Department of Computing
Hong Kong Polytechnic University
Kowloon, Hong Kong
Jacky.Keung@comp.polyu.edu.hk

Graduate School of Information Science
Nara Institute of Science and Technology
Nara, Japan
matumoto@is.naist.jp

Abstract—One of the common problems in building an effort
estimation model is that not all the effort factors are suitable aspredictor variables. As a supplement of missing information inestimation models, this paper explores the project manager’s
knowledge about the target project. We assume that the experts
can judge the target project’s productivity level based on his/her
own expert knowledge about the project. We also assume that
this judgment can be further improved, because using theexpert’s judgment solely could incur subjective perception. This
paper proposes a regression model building/selection method toaddress this challenge. In the proposed method, a fit dataset formodel building is divided into two or three subsets by projectproductivity, and an estimation model is built on each data subset.
The expert judges the productivity level of the target project andselects one of the models to be used. In the experiment, we used
three datasets to evaluate the produced effort estimation models.
In the experiment, we adjusted the error rate of the judgment
and analyzed the relationship between the error rate and theestimation accuracy. As a result, the judgment-incorporating
models produced significantly higher estimation accuracy thanthe conventional linear regression model, where the expert’s
error rate is less than 37%.
Keywords: Software Effort Estimation, Project Management,
Expert Judgment, Stratification, Productivity, Estimation error

I.

INTRODUCTION

In today’s rapid software development environment,
software systems grow in size and complexity. Software
project management activities such as staffing, scheduling andproject progress management are becoming increasingly
important to avoid project failure (cost overrun and/or delayeddelivery). As a basis of project management, effort estimation
plays a fundamental role in the decision making process;therefore, accurate effort estimation is essential to softwaredevelopment project success.
To date, various estimation models that use past projects’
historical data have been proposed. One of the most commonlyused estimation models is a linear regression model, which
represents the relationship between the dependent variable (i.e.
effort) and independent variables such as functional size,architecture, programming language, and so on.
A major challenge in building an accurate estimation modelis that not all of the effort estimators are available as predictor
variables. For example, non-functional requirements greatly
affect the ultimate development effort; however, suchinformation is often not available in the historical project
datasets. To provide supplement of missing information inestimation models, this paper focuses on the project manager’s
knowledge and perception about the target project. Generally,
experts have more complimentary information than the datasets,such as non-functional requirements, which are not available inquantitative forms in the datasets. In this paper, we assume thatthe experts are able to determine the target project’s
productivity level (either high or low, or either high, middle orlow) using his/her own knowledge about the project. On theother hand, we also assume that this subjective judgment canbe invalid, and this paper also attempts to improve theestimation accuracy using regression models.
To incorporate the expert’s judgment on productivity level
into the model, this paper proposes the following modelbuilding/estimation procedure. First, the dataset of past projects
is divided into two or three subsets by productivity, and an
estimation model is built on each data subset. Then, the expert
judges the productivity level of the estimation target projectand chooses one of the models to be used. In this respect, it is
not clear how the error rate of expert’s judgment affects the
final estimation accuracy. So, we experimentally change theerror rate of the judgment and analyze the relationship between
the error rate and the estimation accuracy.
Section 2 explains our proposed effort estimation method.
Section 3 describes the experimental setting, results of theexperiment and discussion. Section 4 explains related work,
and Section 5 concludes the paper.

size are denoted as x1, x2, and x3, the effort estimation model
based on the linear regression model is denoted as:

Original
Datasety = β0 + β1x1 + β2x2 + β3x3 + ε

Step 1Low
productivity
subsetHigh
productivity
subset(1)

Step 2Model for highModel for low
productivity projects
productivity projects

In the equation, β0 is an intercept, β1, β2, β3 are partial
regression coefficients, and ε is an error term. Logarithmictransformation is often used when an effort estimation model isbuilt. For example, when logarithmic transformation is applied
to x1, x2 and y, and not to x3, the model is denoted as:

Step 4y = x1β1 x2β2 e β x + β + εHighproductivity
project

Low
productivity
project

Step3

3 3

Figure 1. Procedure of the expert judgment incorporated
estimation (two-level model)

II. ESTIMATION INCORPORATING EXPERT JUDGMENT
The proposed method uses regression model and the
productivity level given by a project manager. Figure 1
illustrates a procedure of our method. Our method consists offour steps. In the first step, a fit dataset (for model building) is
divided into subsets by productivity level. In the second step,an estimation model is built on each subset. In the third step, a
project manager judges the productivity level of the target
project. Lastly, the model is then selected based on the
judgment and the selected model estimates the effort. The
detail of this procedure is as follows:
1) Dividing dataset: The dataset used to build an
estimation model is divided into subsets by productivity. The
productivity is defined as the ratio of functional size to effort.
We propose two-productivity-level model and threeproductivity-level model (i.e. dividing a dataset into two
subsets or three subsets). This is because we presume judging
the productivity in two or three levels is not difficult for a
project manager, but by more than three levels is difficult.
When using the two-level model, median of productivity ofthe dataset is used, and low productivity subset yields highproductivity subset are being made. When using the threelevel model, first quartile and third quartile of productivity isused, and this resulting low productivity, medium productivity
and high productivity subsets are being made. We do not use
33 percentile and 66 percentile because we presume judging
particulary low (first quartile) or high (third quartile)
productivity project is convinient than judging slightly low (33percentile) or high (66 percentile) productivity project.
2) Building estimation models: An estimation model isbuilt on the each subset made in step 1. For instance, when the
two-level model is applied, two estimation models (anestimation model on the high productivity subset and a modelon the low productivity subset) are built. We assume that theestimation model is built using linear regression. When effort
is denoted as y, and independent variables such as functional(2)

The equation is then transformed as:
y = x2β2 e β x + β x1β1 eε3 3Projectmanager00(3)Where functional size is denoted as x1, and x2β2 e β3x3 + β0 implies
productivity. The model refined by productivity makes thevariance of productivity smaller. It helps to infer thecoefficients β2 and β3, and the intercept β0 accurately.
Additionally, the accurate inference of them helps the inferenceof the coefficient β0.
3) Judging productivity: When a project manager judges
productivity of a target project, the two-level model is applied,
he/she provides the productivity of the project to be high or low
level, and compared with past projects. Similarly, when thethree-level model is applied, he/she judges the productivity ofthe project to be high, middle or low. In judgment, the project
manager determines the factors seems to affecting productivity
the most such as non-functional requirements of the system.
4) Estimating effort: Based on the judgment in step 3, oneof the estimation models built in step 2 is selected, and theeffort of the target project is estimated. For example, when the
two-level model is applied, and the expert selected the highproductivity model, the effort is estimated by the selectedmodel.
Instead of dividing the dataset by productivity, the variablewhich denotes productivity level is applicable, to incorporate
the expert’s judgment into the estimation model. We applied
dividing the dataset because more flexible model is made by
the divided dataset than the variable.
III.

EXPERIMENTFirst, to evaluate the effect of incorporating expert
judgment into the estimation model, we experimentally clarify
the estimation accuracy of our judgment-incorporating models,comparing to the conventional model, i.e. an effort estimation
model based on linear regression without incorporating expert
judgment. Then, to evaluate the influence of misjudgment, wechange the error rate of expert judgment, and compare theaccuracy with the conventional model.
A. Datasets
We used the ISBSG [7], Kitchenham [12], and Desharnais
datasets [6]. We assume that the estimation phase is at the endof the project planning. So, only variables whose values were
fixed at this point were used as independent variables.
Logarithmic transformation was applied to the effort and thefunction point in the datasets. Nominal scale variables weretransformed into dummy variables (e.g. if the variable has n
categories, it is transformed into n - 1 dummy variables).
The ISBSG dataset is provided by the International
Software Benchmark Standard Group (ISBSG), and it includes
project data collected from software development companies in
20 countries [7]. The dataset (Release 9) includes 3026 projects
that were carried out between 1989 and 2004, and 99 variables
were recorded. The ISBSG dataset includes low quality project
data (Data quality ratings are also included in the dataset).
We extracted projects based on the previous study [14](Data quality rating is A or B, function point was recorded by
the IFPUG method, and so on). Also, we excluded projects that
included missing values (listwise deletion). As a result, weused 593 projects. The variables used in our experiment are FP,
language type, development type, and development platform.
They are same as the previous study [14].
The Kitchenham dataset includes 145 projects of a single
outsourcing company, shown by Kitchenham et al. in theirpaper [12]. We selected 135 projects that do not include
missing values. FP and Development type were chosen as theindependent variables, and inadequate variables for effort
estimation (e.g. estimated effort by a project manager) were
eliminated. Development type was transformed into dummy
variables.
The Desharnais dataset includes 88 projects of 1980’s,
collected from a single Canadian company by Desharnais [6].
The dataset is available at the PROMISE Repository [3]. We
used 77 projects that do not have missing values. FP,
adjustment factor, experience of team, experience of manager,
and language were used as independent variables, anddevelopment year and duration were not used. Also, theadjusted function point, the number of transactions, and thenumber of entities were not used to avoid multicollinearity.
Programming language was transformed into dummy variables
which reflect different development environments.
The variance of productivity of the ISBSG dataset(collected from multi companies) was the largest, and thevariance of the Desharnais dataset (collected from a single
company) was the smallest in the datasets.
B. Evaluation criteria
To evaluate the accuracy of effort estimation, we used theconventional metrics such as MRE (Magnitude of Relative
Error) [5], MER (Magnitude of Error Relative to the estimate)[11], and BRE (Balanced Relative Error) [15]. Especially, MRE
is widely used to evaluate the effort estimation accuracy [18](The residual sum of squares is not widely used for theevaluation). A lower value of each criterion indicates higher
estimation accuracy. Intuitively, MRE means error relative toactual effort, and MER means error relative to estimated effort.
However, MRE and MER have biases for evaluating under andover estimation [4][13]. The maximum MRE is 1 even if an
extreme underestimate occurs (For instance, when the actual
effort is 1000 person-hour, and the estimated effort is 0 personhour, MRE is 1). Similarly, maximum MER is smaller than 1
when an overestimate occurs. So we employed BRE whose

evaluation is not biased as is both MRE and MER [16], and weevaluated the judgment-incorporating models based on mainly
BRE (MRE and MER were used for reference).
C. Procedure of Experiment
In the proposed method, we assume and allow misjudgment
occurs, that is, the expert’s judgment of productivity isincorrect and thus he/she cannot select the right model. In theexperiment, we generate n% misjudgment in estimation, andcompute the estimation accuracy when the percentage varies.
We assume that judging the productivity to be high or low iseasy for a project manager when the actual productivity isextremely high or low. On the contrary, when the actual
productivity is close to the border line that divides the high andlow productivity classes, then misjudgment can be easily
occurred.
Figure 2 explains the misjudged projects generated in our
experiment. The fit dataset is used to build estimation models(regarded as past projects), and the test dataset is used as theestimation target (regarded as ongoing projects). In the figure,projects in datasets are ordered by productivity, and the number
in parentheses indicates productivity of each project. Based on
the assumption explained in the previous paragraph,misjudgment of the target project in the test dataset occurs
when productivity of the project is close to the borderline ofproductivity. To generate n% misjudgment of target projects,
we choose top n% projects whose productivity is close to theborderline in the test dataset, and consider they are misjudged.
Figure 3 shows the experimental procedure when the twolevel model is used. Details of the procedure are as follows:
1.
2.
3.

4.

5.

A dataset is randomly divided into five equal sets.
One subset is treated as a test dataset, and the others
are treated as a fit dataset (five-fold cross validation).
Productivity (the ratio of functional size to effort) ofprojects included in the fit dataset is computed, andthe fit dataset is divided into subsets by the
productivity.
Estimation model based on linear regression is built
on each subset. When building the model, stepwise
variable selection based on AIC is applied.
Productivity levels of all projects in the test dataset are
Fit dataset

Test dataset

f1 (0.024)

t1 (0.030)
Modelfor low
productivity

fi-1 (0.063)
fi (0.065)
fi+1 (0.065)

tj-1 (0.061)
Borderline

fi+2 (0.066)

tj (0.064)
tj+1 (0.066)

Misjudged
projects

tj+2 (0.069)
Modelfor high
productivity

fk (0.481)

tl (0.373)

Figure 2. Misjudgment of target projects

1

Fit
datasetHigh productivity
subsetModel
43

Estimated
Effort5,6
7
Evaluation
criteria

Low productivity
subset
Model
4

2
3

Test
dataset2
Original
dataset8

Figure 3. Experimental procedure (two-level model)

settled by their productivity and the misjudgment rate
(initially the misjudgment rate is zero.)
6. For each projects in the test dataset, one of the models
in step 4 is selected based on the productivity levels
settled in step 5, and the effort of the project isestimated.
7. Evaluation criteria are calculated based on the actual
effort and the estimated effort.
8. Increase misjudged projects in the test dataset, andSteps 5 to 7 are repeated
9. Steps 2 to 8 are repeated, selecting other subset as fit
dataset and test dataset.
10. Steps 1 to 9 are repeated four times, to increase thenumber of trials.
We also compute the estimation accuracy of theconventional model (without incorporating expert judgment),by skipping step 3, 5 and 8 (The reason for skipping these steps
is that the conventional model does not handle productivity,
and therefore the judgment and selection steps are not needed).
In the experiment, we call it no-level model. To compare theaccuracy of our method with the no-level model statistically,
we use Wilcoxon signed-rank test at a significance level of
0.05. In the following tables, italic face means the difference
was statistically significant at 0.05 level, and the number inparentheses indicates p-value of the test.
D. Results and Discussion
Table 1 shows the accuracy of the no-level model and the
proposed models. In the table, no-level indicates theconventional model. There is significantly difference between
the no-level model and the proposed models. As shown in thetables, all criteria were greatly improved and there arestatistically different. We conclude that if there is no
misjudgment, the estimation accuracy can be greatly improvedby incorporating expert judgment into the estimation model.
Next, we further explored the influence of misjudgment rate.
Table 2 shows the limit rate when all criteria of the judgmentincorporating models were smaller than the no-level model.
When the misjudgment rate was smaller than the limit rate, theaccuracy of the judgment-incorporating models was higher
than the no-level model. Although all evaluation criteria were
not statistically different, if we were consider all criteria, whenthe misjudgment rate was smaller than 36% (average of limit
rate on three datasets), performance of the two-level models

was higher than the no-level model on average. Similarly,
when the misjudgment rate was smaller than 36%, performanceof the three-level model was higher the no-level model.
As explained in the evaluation criteria section, we consider
that the BRE is the most preferable measure in this experiment.
So we analyzed the limit rate, by focusing on the BRE. The
limit rate is shown in Table 3. In the table, “smaller(significant)” means that the average or median BRE of thejudgment-incorporating models was significantly smaller(higher accuracy) than the no-level model, when misjudgment
rate was smaller than the limit rate. “Smaller (not significant)”
means average or median BRE of the judgment-incorporating
models was smaller than the no-level model, whenmisjudgment rate was smaller than the limit rate, but there wasno statistically difference. “Average” is the average of the limit
rate on the three datasets.
On average, when the misjudgment rate on the two-level
model was smaller than 37%, performance of the model was
higher than no-level model because median BRE was not lower
than the no-level model, and average BRE was statisticallysmaller. Similarly, when the misjudgment rate on the threelevel model was smaller than 43%, performance of the model
was higher. So, the answer of RQ2 is that when using the twolevel model, the acceptable error rate is 37% on average, and it
is 43% on average when using the three-level model.
Finally, we compared the two-productivity-level modelwith the three-level model. Figure 4 shows the change of
average BRE and median BRE on each dataset. When the
misjudgment rate was same, three-level model showed higher
accuracy than two-level model, except for median BRE on
Desharnais dataset, when misjudgment rate was high. However,
the result does not mean performance of the three-level model
is higher than the two-level model, because there is noevidence that the misjudgment rate of the two-level model andthe three-level model are same. So, as our future work, we needto clarify actual misjudgment rate of project managers.
IV.

RELATED WORK

Apart from the model based effort estimation, expert
judgment based effort estimation is proposed, and its practicalguidelines are discussed [8]. However, while benefits of themodel based estimation and expert judgment based estimation
are different, using both approaches is preferable to avoid risks
in estimation [9]. Our approach exploits the both benefits ofexpert judgment and the model based estimation, to enhancethe estimation accuracy. Some model based estimation (such as
COCOMO [2]) uses variables based on expert judgment suchas team skill. However, there is no model that includes highly
project-specific but important information such as a serious
conflict between the project leader and a best programmer [9].
Our approach can include in the model any kind of informationthat affects the productivity as far as the project manager isaware of it.
There is a research which uses expert judgment to build amodel. Baker [1] proposed an estimation tool which supports
experts to build an effort estimation model. However, it uses
the conventional estimation model, and therefore it does notdirectly include the expert’s knowledge in the model.

TABLE I.

ACCURACY OF THE JUDGMENT-INCORPORATING MODELS(a) ISBSG dataset

Levels
No
Two
Three

Average MRE
1.04
0.67 (0.000)
0.45 (0.000)

Median MRE Average MER
0.51
0.84
0.39 (0.000)
0.57 (0.000)
0.32 (0.000)
0.42 (0.000)

Levels
No
Two
Three

Average MRE
0.69
0.44 (0.000)
0.31 (0.000)

Median MRE Average MER
0.38
0.58
0.26 (0.000)
0.38 (0.000)
0.20 (0.000)
0.28 (0.000)

Levels
No
Two
Three

Average MRE
0.36
0.24 (0.000)
0.22 (0.000)

Median MRE Average MER
0.28
0.38
0.20 (0.000)
0.26 (0.000)
0.18 (0.000)
0.23 (0.000)

Median MER
0.57
0.42 (0.000)
0.30 (0.000)

Average BRE
1.45
0.89 (0.000)
0.58 (0.000)

Median BRE
0.86
0.54 (0.000)
0.38 (0.000)

Median MER Average BRE
0.41
0.92
0.27 (0.000)
0.55 (0.000)
0.21 (0.000)
0.38 (0.000)

Median BRE
0.51
0.32 (0.000)
0.23 (0.000)(b) Kitchenham dataset(c) Desharnais dataset

TABLE II.

Median MER
0.29
0.19 (0.000)
0.18 (0.000)

Average BRE
0.47
0.30 (0.000)
0.27 (0.000)

Median BRE
0.34
0.21 (0.000)
0.20 (0.000)

LIMIT OF THE M ISJUDGMENT RATE (BASED ON ALL CRITERIA)(a) ISBSG dataset

LevelsTwo
Three

Misjudgment rate
Average MRE
33%
0.78 (0.000)33%
0.57 (0.000)

Median MRE Average MER
0.48 (0.231)
0.74 (0.000)
0.43 (0.001)
0.82 (0.701)

LevelsTwo
Three

Misjudgment rate
Average MRE
41%
0.54 (0.000)41%
0.48 (0.000)

Median MRE Average MER
0.36 (0.202)
0.57 (0.452)
0.35 (0.064)
0.52 (0.027)

LevelsTwo
Three

Misjudgment rate
Average MRE
33%
0.30 (0.000)33%
0.31 (0.030)

Median MRE Average MER
0.27 (0.261)
0.34 (0.002)
0.24 (0.014)
0.33 (0.008)

Median MER Average BRE
0.50 (0.014)
1.11 (0.000)
0.38 (0.000)
1.02 (0.000)

Median BRE
0.67 (0.002)
0.53 (0.000)

Median MER Average BRE
0.38 (0.165)
0.77 (0.000)
0.36 (0.123)
0.68 (0.000)

Median BRE
0.48 (0.729)
0.44 (0.105)

Median MER
0.26 (0.083)
0.23 (0.007)

Median BRE
0.31 (0.123)
0.27 (0.006)(b) Kitchenham dataset(c) Desharnais dataset

Similar notion with our assumption that subjective
judgment includes error is pointed out by Kitchenham et al.
[10]. They pointed out five sources of estimation uncertainty.
Assumption error is one of the sources, and indicates error of amodel’s input parameters. For example, difference of estimatedfunctional size and actual size is the assumption error. The
subjective judgment is regarded as the assumption error.
In our method, dividing the dataset by productivity issomewhat akin to the analogy-based estimation (ABE) [17],which estimates effort using similar projects. Whereas ABE
uses only available information in the historical project dataset,
our method uses not only the available information but also
expert judgment. Our method is also applicable to ABE, and it
may improve estimation accuracy of ABE, because it wasdifficult to predict productivity of a target project by ABE in
our preliminary analysis.
V.

CONCLUSIONSThis work incorporates the expert judgment method tocompliment software effort estimation model. We proposed acomposite approach of model building, which has four steps.
The first step uses a fit dataset divided into two or three subsets
by productivity. The second step builds an estimation model oneach subset. A project manager judges the productivity level ofthe target project in the third step. Lastly, the model is selectedbased on the expert judgment, and the selected model produce

Average BRE
0.39 (0.000)
0.40 (0.000)

better effort estimates, which has been evaluated using anempirical experiment using real project datasets.
In the experiment, three dataset were used and compared
the accuracy of the judgment-incorporating models with theconventional model. We changed the error rate of the judgment
and analyze the relationship between the rate and the accuracy.
The result showed that when the two-productivity-level modelis used, the acceptable error rate is 37% on average, and when
the three-productivity-level model is used, the rate is 43% on
average. We conclude that the judgment-incorporating models
are useful for effort estimation given its improved prediction
performance. One of important findings of this work is thatincorporating the expert judgment is effective to improve theaccuracy of the effort estimation model, even if misjudgment
occurred to some extent. Future studies will be carried out toclarifying actual misjudgment rate of project managers, andapplying the judgment-incorporating models to otherestimation method such as the analogy-based estimation. This
work is important to the research of combining expert-based
software effort estimation in software engineering.
REFERENCES
[1]
[2]
[3]

D. Barker, A Hybrid Approach to Expert and Model Based Effort
Estimation, , Master Thesis, West Virginia University, 2007.
B. Boehm, Software Engineering Economics, Prentice Hall, 1981.
G. Boetticher, T. Menzies, and T. Ostrand, PROMISE Repository of
empirical software engineering data, West Virginia University,
Department of Computer Science, 2007.

TABLE III.

LIMIT OF THE MISJUDGMENT RATE (BASED ON BRE)(a) Two-level model(b) Three-level model

ISBSG Kitchenham Desharnais Average
48%
39%
41%
Average Smaller
43%(0.002)(0.036)(0.006)BRE(Significant)
46%
52%
39%
Smaller46%(0.216)(0.006)(Not significant) (0.368)
33%
37%
26%
Median Smaller
32%(0.002)(0.021)(0.001)BRE(Significant)
38%
41%
33%
Smaller37%(0.729)(0.123)(Not significant) (0.430)
1.30

3.50

2.50

No subset
Two subset
Three subset

1.10

1.00

No subset
Two subset
Three subset

0.60

0.70

1.50

0.40

0.50

1.00

0.30

0.50

0.20

0% 11% 22% 33% 44% 56% 67%

0% 13% 26% 38% 51% 64% 77% 89%

Misclassification rate

1.00

Median BRE

1.50
1.00
0.50
0.00

0.80

(a) ISBSG dataset

39%

59%

78%

(c) Desharnais dataset
0.65

No subset
Two subset
Three subset

No subset
Two subset
Three subset

0.55
0.45

0.60

0.35

0.40

0.25

0.20
0% 13% 26% 38% 51% 64% 77% 89%

Misclassification rate

20%

Misclassification rate(b) Kitchenham dataset

2.50
No subset
Two subset
Three subset

0%

Misclassification rate(a) ISBSG dataset

Median BRE

No subset
Two subset
Three subset

0.80

0.90

2.00

2.00

ISBSG Kitchenham Desharnais Average
52%
33%
48%
44%
(0.048)(0.012)(0.000)
54%
63%
46%
54%
(0.409)(0.985)(1.000)
41%
37%
33%
37%
(0.015)(0.001)(0.006)
48%
41%
39%
43%
(0.784)(0.105)(0.349)Median BRE

Average BRE

3.00

Average Smaller
BRE(Significant)Smaller(Not significant)Median Smaller
BRE(Significant)Smaller(Not significant)

0.15

0% 11% 22% 33% 44% 56% 67%

0%

20%

Misclassification rate(b) Kitchenham dataset

39%

59%

78%

Misclassification rate(c) Desharnais dataset

Figure 4. Transitions of average BRE and median BRE on each dataset[4]

C. Burgess, and M. Lefley, “Can genetic programming improve software
effort estimation? A comparative evaluation,” Journal of Informationand Software Technology, vol. 43, no. 14, pp. 863–873, 2001.
[5] S. Conte, H. Dunsmore, and V. Shen, Software Engineering, Metrics andModels, Benjamin/Cummings, 1986.
[6] J. Desharnais, Analyse Statistique de la Productivitie des Projets
Informatique a Partie de la Technique des Point des Function, Master
Thesis, University of Montreal, 1989.
[7] International Software Benchmarking Standards Group (ISBSG), ISBSG
Estimating: Benchmarking and research suite, ISBSG, 2004.
[8] M. Jørgensen, “Practical Guidelines for Expert-Judgment-Based
Software Effort Estimation,” IEEE Software, vol. 22 no. 3, pp. 57–63,
2005.
[9] M. Jørgensen, B. Boehm, and S. Rifkin, “Software Development Effort
Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol.
26 no. 2, pp. 14–19, 2009.
[10] B. Kitchenham, and S. Linkman, “Estimates, Uncertainty, and Risk,”
IEEE Software, vol. 14 no. 3, pp. 69–74, 1997.
[11] B. Kitchenham, S. MacDonell, L. Pickard, and M. Shepperd, “What
Accuracy Statistics Really Measure,” In Proceedings of IEE Software,
vol. 148, no. 3, pp. 81–85, 2001.

[12] B. Kitchenham, S. Pfleeger, B. McColl, and S. Eagan, “An Empirical
Study of Maintenance and Develop-ment Estimation Accuracy,” Journalof Systems and Software, vol. 64, no. 1, pp. 57–77, 2004.
[13] C. Lokan, “What Should You Optimize When Building an Estimation
Model?” In Proceedings of International Software Metrics Symposium(METRICS), pp. 34, Como, Italy, Sep. 2005.
[14] C. Lokan, and E. Mendes, “Cross-company and single-company effort
models using the ISBSG Database: a further replicated study,” In
Proceedings of the International Symposium on Empirical Software
Engineering (ISESE), pp. 75–84, Sep. 2006.
[15] Y. Miyazaki, M. Terakado, K. Ozaki, and H. Nozaki, “Robust
Regression for Developing Software Estimation Models,” Journal of
Systems and Software, vol. 27, no. 1, pp. 3–16, 1994.
[16] K. Mølokken-Østvold, and M. Jørgensen, “A Comparison of Software
Project Overruns-Flexible versus Sequential Development Models,”IEEE Transaction on Software Engineering, vol. 31, no. 9, pp. 754–766,
2005.
[17] M. Shepperd, and C. Schofield, “Estimating software project effort using
analogies,” IEEE Transaction on Software Engineering, vol. 23, no. 12,
pp. 736–743, 1997.
[18] F. Walkerden, and R. Jeffery, “An Empirical Study of Analogy-based
Software Effort Estimation,” Empirical Software Engineering, vol. 4, no.
2, pp. 135-158, 1999.