Empirical Evaluation of Missing Data Techniques for Effort Estimation
Koichi Tamura1, Takeshi Kakimoto2, Koji Toda1, Masateru Tsunoda1,
Akito Monden1, Ken-ichi Matsumoto1
1
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 630-0192, Japan{koichi-t, koji-to, masate-t, akito-m, matumoto}@is.naist.jp
2
Graduate School of Information Science and Technology, Osaka University
1-3 Machikaneyama, Toyonaka, Osaka, 560-8531, Japankakimoto@ist.osaka-u.ac.jp

Abstract
Multivariate regression models have been commonlyused to estimate the software development effort to assist
project planning and/or management. These models
require a complete data set that has no missing values formodel construction. The complete data set is usually built
either by using imputation methods or by deleting
projects and/or metrics that have missing values (we callthis RC deletion). However, it is unclear which method is
the most suitable for the effort estimation. In this paper,using the ISBSG data set of 706 projects (containing 47%
missing values) collected from several companies, weapplied four imputation methods (mean imputation,
pairwise deletion, k-NN method and CF method) and RC
deletion to build regression models. Then, using a data
set of 143 projects (with no missing values), we evaluatedthe estimation performance of models after applying each
imputation or the RC deletion. The result showed that thesimilarity-based imputation method (k-NN method and
CF method) showed better performance than othermethods in terms of MdMAE, MdMRE, MdMER andPred(25).

1. Introduction
In a software development project, software costestimation is necessary for management of schedule andresources. So far, various quantitative estimation methods
using a historical project data set have been proposed andused [3] [12] [14]. Among these methods, the regression
model has been most widely used for its convenience [1][9] [15].
One of the practical problems in using estimation
methods is that the historical project data usually contain

substantial numbers of missing values [1] [11]. One
reason is that different divisions in an organization might
have different policies on data collection, i.e. one project
collects a particular metric while other projects do not.
Even if the organization has a unified policy, not all
metrics are collected in each project due to the pressing
development schedule. However, not only regression
models but also many other estimation models need a data
set with no missing values to build models.
One approach to solve this problem is to delete themetrics and projects with missing values from the data set(we call this RC deletion). This method is easy to use;
however, deletion might remove useful information for
effort estimation. In addition, there are unclear trade-offs
between deletion of metrics and deletion of projects.
Another commonly used approach is using imputation
methods [6] [8] [11]. Imputation does not reduce theinformation; however, it might introduce noise to the data
set. Yet another approach is pairwise deletion, which isapplicable to regression models [15]. While all thesemethods are considered useful, it is unclear which is thebest. It is important for an engineer to know which is
better: (a) to avoid losing information despite introducing
noise or (b) to avoid noise despite losing information.
Our primary goal is to clarify which missing datatechnique shows the best performance for building theregression model. In this paper, using a data set of 706
projects (containing 47% missing values) collected by theInternational Software Benchmarking Standards Group(ISBSG) [5], we applied four imputation methods (meanimputation, pairwise deletion, k-NN method and CF
method) and RC deletion to build estimation models using
stepwise multiple regression analysis [2]. Then, using adata set of 143 projects (with no missing values), also
from the ISBSG data set, we evaluated the estimationperformance of models after applying each imputation or
the RC deletion. The CF method is based on collaborative
filtering and applying it to missing data imputation is a
first-time approach.

2. Imputation and deletion methods

denote the maximum and minimum value in {vx,j | px Pj }respectively.
Step 2 (computation of similarity between projects): In
this step, similarity sim(pa, pi) between the target projectpa and other projects pi is computed. Formally, we candefine the sim(pa, pi) between the target project pa andother projects pi as:

In this paper, the following methods were applied tothe data set before building estimation models.

sim p a , p i 

2.1. Mean imputation

This method fills each missing value with the mean ofobserved values [8] [11] [15].
2.2. Pairwise deletion
This method is particularly associated with theregression analysis [8] [15]. While calculating acorrelation matrix to build a regression model, acorrelation between each pair of variables is calculated
from all cases (projects) that have non-missing value on
those two variables. This method is widely used instatistical analysis tools such as SPSS.
2.3. Similarity-based imputation (k-NN method)
This method uses existing metrics values of k mostsimilar projects to fill missing values of the target project.
The degree of similarity between two projects is computedby Euclidean distance [1] [6].
2.4. Similarity-based imputation (CF method)
This method is an alternative similarity-based
imputation based on collaborative filtering [12]. Although
this method is used for cost estimation, we newly apply it
to missing data imputation. The method uses cosine
similarity to compute the similarity instead of Euclidean
distance. The three-step procedure of the CF method is
described below.
Step 1 (normalization of metrics): Since each metric has
a different value range, this first step normalizes values of
metrics so that the value range becomes [0, 1]. Here, wedenote that pi is i-th project, mj is j-th metric, and vi,j is thevalue of metric mj observed in project pi. The normalized
value v´i,j of vi,j (of project pi) is calculated by thefollowing equation:vi, j 

vi , j  min Pj 

max Pj   min Pj (1)where Pj denotes a set of projects in which the value of
metric mj was observed (collected), max(Pj) and min(Pj) v   md m  v   md m jM a  M i

a, j

j

 v   md m i, j v   md m 

2jM a  M i

a, j

j

j

(2)

2

jM a  M i

i, j

j

where Ma and Mi denote a set of metrics observed inproject pa and pi respectively, m´j denotes the normalized
value of mj, and md(m´j) denotes the median of m´j.
The metrics that are higher than md(m´j) show positive
values and the metrics that are lower than md(m´j) show
negative values by subtracting md(m´j). The value range ofsim(pa, pi) is [-1, 1] for this computation. Note that sim(pa,pi) shows low or negative value (i.e., the computed
similarity shows low value) if the difference of metrics
between pa and pi is great.
Step 3 (computation of estimation): This step calculates
an estimated value vˆa , b of the metric mb on the target
project pa using sim(pa, pi) calculated in the previous step.
The estimated value is computed as the sum of themetrics’ values given by the other projects similar to pa.
Each value is weighted by the corresponding amplifier(pa,pi) and sim(pa, pi) between pa and pi. Formally, we candefine the estimated value as:vˆ a ,b  v  amplifier p , p  sim p , p i ,bik  nearestPro
jects

a

i

 sim p a , pi 

a

i

(3)

ik  nearestPro
jects

where k-nearestProjects denotes a set of k projects (calledneighborhoods) that have the highest similarity with pa.
The neighborhoods must have mj as an observed metric.
Generally, the neighborhood size k affects the estimation
accuracy (This point applies to the k-NN method as well).
To improve accuracy of the estimation, theamplifier(pa, pi) calculates an approximate value of the vi,bwith comparing the sizes of projects pa and pi, i.e. theamplifier indicates what times pa’s value is pi’s value. The
amplifier derived from the fact that the pa’s value isseveral times larger (or smaller) than the pi’s value whenpi is a similar to pa. It’s because the similarity is computedby vector operation but not Euclidean distance. sim(pa, pi)
is computed by comparing tendencies of the values,
whereas Euclidean distance is computed by comparing

absolute values. Formally, we can define the amplifier(pa,pi) as:f
amplifier p a , pi   afiTable 1. Metrics used in the experiment
Name

(4)

where fa denotes the function points of project pa.
2.5. Row-column deletion method (RC deletion)
This method deletes the projects and/or metrics withmissing values from a data set to build a complete data set.
Listwise deletion [8] [15] is a subset of this method, which
only deletes projects with missing values. The RC deletion
allows deleting metrics to reduce the projects to bedeleted. There exists a trade-off between deletion of
metrics and deletion of projects.

3. Experiment3.1. Overview
Using the ISBSG data set, we experimentally compare
missing data techniques (imputation methods and the RC
deletion) by evaluating the prediction performance ofeffort estimation models after applying the techniques.
We cannot evaluate the accuracy of imputed (filled)
values by comparing true values with imputed values
since we use the data set that originally contained missing
values. Therefore, using a test data set with no missing
values, we indirectly evaluate the performance ofimputation methods by evaluating the performance ofeffort estimation for the test data.

3.2. Dataset
In the experiment, we used the ISBSG data set,collected from 20 nations’ software development
companies [5] and with many missing values (missing
value ratio was 58%). Furthermore, the ISBSG data set
has been widely used for past empirical experimentations
that evaluated various missing data techniques andestimation methods [9] [10] [13].
We assumed that to estimate the effort is the end of thedesign phase. The extracted data set of 849 projects(missing value ratio was 39%) whose summary work
effort (estimation target) was recorded (i.e. not missing),
the count approach of FP was <IFPUG>, the development
type was <new development> and the data quality rating
was <A> or <B> [9].
Table 1 presents metrics contained in the ISBSG data set.
We chose the four metrics to estimate the objective
variable. Although <Project elapsed time> is an actually
measured value, we included it as a predictor variable

Predictor
variables
Objective
variable

Missing
Value Ratio
Function points
0%
Project elapsed time [month]
8.8%
Effort plan [person-hours]
77.0%
Effort specify [person-hours]
71.2%
Summary work effort
0%[person-hours]

since (1) it probably affects productivity and cost, (2) theplanned value of the project elapsed time is not collected,(3) the project duration is usually fixed in the initial stageof a project and <Project elapsed time> is usually notwidely different from its planned value. Mean imputationand similarity-based imputation focus on filling thenumerical variable, and then we removed categoricalvariables such as development platform, language typeand business area type. Furthermore, the number ofprojects to build regression models falls if we include too
many metrics in predictor variables. For these reasons, wechose the four metrics as predictor variables.
We divided the data set according to missing values
into the fit data set of 706 projects (containing 47%
missing values) and the test data set of 143 projects (withno missing value). The fit data set is used for building
estimation models and the test data set is for evaluation of
the estimation performance of built models.
The fit data set has at least one missing value in all
projects, so RC deletion first deletes the metrics withmany missing values and then deletes the projects withmissing values in the fit data set. According to the metrics
to be deleted, we built three data sets: by deleting <Effort
plan> (72 projects), by deleting <Effort specify> (28projects) and by deleting both <Effort plan> and <Effort
specify> (631 projects). We used k = 3 in the k-NN
method and k = 8 in the CF method whose residual meansquare showed the minimum when we built regression
models.

3.3. Evaluation criteria
We used four evaluation criteria: magnitude ofabsolute error (MAE), magnitude of relative error (MRE),
magnitude of error relative (MER) [3], and Pred(25) [14].
MRE and MER are defined as (5) and (6) respectively as
follows (where X = actual effort, X̂ = predicted effort):
MRE 

MER 

Xˆ  X(5)

X

Xˆ  X.
Xˆ

(6)

Table 2. Estimation performance when each method was used
MdMAE
Mean imputation
Pairwise deletion
Similarity-based method (k-NN method)
Similarity-based method (CF method)
RC deletion (P deletion)
RC deletion (S deletion)
RC deletion (P, S deletion)
P is Effort plan and S is Effort specify

MRE and MER are criteria to evaluate overestimation
and underestimation respectively [3]. Using either of themis insufficient because even if the MRE of a model issmall, the model might overestimate if MER is muchgreater than MRE. Pred(25) is the percentage ofpredictions that fall within 25 percent of the MRE.

3.4. Experimental procedure
The experimental procedure is as follows.
Step 1. We apply each missing data technique to the fit
data set.
Step 2. Using the fit data set, we apply a stepwise
regression analysis to build an estimation model whose
objective variable is <Summary work effort>.
Step 3. Considering that the summary work effort of thetest data set is unknown, we estimate the summary work
effort of the test data set with built models and calculate
the value of each evaluation criterion.

4. Results and discussion
4.1. Overall results
Table 2 shows the median of MAE, MRE and MER(MdMAE, MdMRE and MdMER), and Pred(25) of
estimations when we applied each missing data technique.
Fig. 1 and Fig. 2 show box plots of the MRE and MER
values when each technique is applied respectively. In
Table 2, Fig. 1 and Fig. 2, “P deletion” stands for a data
set built by deleting Effort plan, and “S deletion” forEffort specify. The result showed that the similarity-based
imputation methods (k-NN method and CF method)
showed better performance than other methods in all
criteria.
We ascertain whether each evaluation criterion is
statistically significant. The value of MAE, MRE, MER is
not normally distributed, so we use the non-parametric
Wilcoxon matched-pairs signed-ranks test to assess theirstatistical differences. And we use a chi-square test forPred(25). The level of significance is 0.05 in each test.
The result of each test is described in the following
section.

2648
1036
760
829
1050
1463
1875

MdMRE
MdMER
Pred(25)
0.818
1.112
20%
0.461
0.609
28%
0.304
0.268
43%
0.274
0.295
46%
0.458
0.416
28%
0.479
0.526
27%
0.555
0.563
18%

4.2. Comparison of similarity-based imputationand other imputation
Among three imputation methods, similarity-based
imputation methods (k-NN method and CF method) weremuch better than the mean imputation (see Table 2). This
result follows the past research using artificial missing
values [15]. Similarity-based methods were also muchbetter than the pairwise deletion. As shown in Fig. 1 andFig. 2, inter-quartile range (IQR) of similarity-based
imputation methods was narrower than mean imputationand pairwise deletion, i.e. the variability of the errors islower. Furthermore, each test of the result of similaritybased imputation and the other imputation showed thateach evaluation criterion was statistically significant.

4.3. Comparison of similarity-based imputation
and RC deletion
As shown in Table 2, similarity-based imputation
methods were also better than all RC deletions (P deletion,
S deletion and P, S deletion). In the RC deletions,although there was no significant difference between P
deletion and S deletion in MdMRE and Pred(25), P
deletion was better than S deletion in MdMAE and
MdMER. Although 631 projects remained in the P, S
deletion data set, it seems deleting both P (Effort plan)and S (Effort specify) removed useful information for theestimation. On the other hand, it can be considered that
the P deletion data set (72 projects) and the S deletion
data set (28 projects) contained too few projects to build
an accurate model, i.e. the confidence interval might
become wide.
As shown in Fig. 1 and Fig. 2, IQR of the similaritybased imputation method (k-NN method) was narrower
than three RC deletions. IQR of the CF method was
narrower than three RC deletions in MRE. Although IQRof the CF method was wider than P deletion, the positionof the boxplot of the CF method was lower than three RC
deletions. Therefore, the CF method was better than three
RC deletions. Furthermore, each test of the result of thesimilarity-based imputation and RC deletion showed that

Fig. 1. MRE values when each method was used

Fig. 2. MER values when each method was usedeach evaluation criterion was statistically significant
except for MER between the CF method and RC deletion(P deletion).
This indicates that the large project data set with a lotof missing values is worthier than the small project data
set with no missing value. We believe engineers should
not be afraid of usage of “incomplete” project data since it
is useful enough for the effort estimation. In addition,interestingly, all RC deletions still showed better
performance than the mean imputation. This indicates thatthe incomplete data set is useful only if missing values areproperly filled in.

collected by the actual industrial organizations. Other
experimental simulation would have to be performed on adifferent data set for improving reliability of the study.
Our experimentation focused on stepwise multiple
regression analysis as the modeling technique. This is oneof the most popular techniques, such as ordinary leastsquares regression analysis, for building cost estimation
models. However, it is possible that other modelingtechniques would require different missing datatechniques.

4.4. Comparison of similarity-based imputation
methods

Jonsson et al. [6] focused on filling in missing data
accurately. They artificially deleted some recorded values
at random and compared filled in values with originallyrecorded values. The result showed that the k-NN method
using incomplete case strategy and k-value of the squareroot of complete cases performed well.
Sentas et al. [13] and Strike et al. [15] compared
several imputation methods and listwise deletion, but theyused a data set whose values were artificially deleted at
random. In [13], multinomial logistic regression showed
better performance than listwise deletion, mean
imputation, expectation maximization and regression
imputation for estimating categorical missing values. In[15], they have compared listwise deletion, mean
imputation and eight different types of hot-deck
imputation. The results showed that the k-NN method(Euclidean distance) and a z-score standardization showed
the best performance.
Although the data set whose values were deleted at
random was used in the experiment ([6] [13] [15]), themissing data pattern is realistically not random but burst.
There are several reasons of the burst missing. The burst
missing is generated if several data sets collected bydifferent business organizations having different policies
on data collection are merged. Even with the sameorganization, the burst missing is generated if its policy isdifferent from each period.

Comparing two similarity-based imputation methods,
the k-NN method was better than the CF method in termsof MdMAE and MdMER, while this became the opposite
in terms of MdMRE and Pred(25) (see Table 2). As
shown in Fig. 1 and Fig. 2, IQR of the k-NN method was
narrower than the CF method in MRE and MER.
Furthermore, each test of the result of the CF method andk-NN method showed that MAE and MER were
statistically significant.
From these results, there is not a big difference
between the k-NN method and CF method in theirprediction performance. However, we could say the k-NN
method tends to build an overestimate model, while the
CF method tends to build an underestimate model.
Therefore, it is preferable for an estimator to use bothmethods to confirm that there is no big difference in theirestimates.

4.5. Threats to validity
We evaluated using only one data set; however, there
are many conditions (i.e., various amounts anddistributions of the missing values) in the data set

5. Related work

Cartwright et al. [1] and Myrtveit et al. [11] compared
imputation methods by evaluating the goodness of fit ofeffort estimation models built after imputations wereapplied to a fit data set containing missing values.
Therefore, they did not evaluate the prediction
performance of the models. But it remains possible thatthe built models too much fit (i.e., overfitting [4]) the fit
data set, and then the models might not perform well when
they evaluate the estimation accuracy to another data set.
In this paper, we have done imputation or RC deletion
to a data set containing missing values naturally andprepared a test data set with no missing values to do thisevaluation. Furthermore, none of the past researchcompared imputation methods with RC deletion, which isoften used in industry.

6. Conclusion
In this paper, we experimentally compared missing
data techniques (mean imputation, pairwise deletion, kNN method, CF method and RC deletion) by evaluating
the prediction performance of effort estimation models
after applying the techniques. Our findings include: Similarity-based imputation methods (k-NN method
and CF method) showed better performance than all
RC deletions. This indicates that a large project data
set with a lot of missing values is worthier than asmall data set with no missing value.
 The mean imputation was worse than all three RC
deletions. This indicates that an incomplete data set
becomes useful only if its missing values are properly
filled in.
 The pairwise deletion was worse than similaritybased imputation.
 The k-NN method tends to build an overestimate
model, while the CF method tends to build anunderestimate model. We recommend an estimator touse both methods to confirm there is no big
difference in their estimates.
Our future work will be to use other data sets toincrease the validity of the results. Furthermore, we willdevelop similarity-based imputation method to improve
imputation performance and make comparison of the costestimation performances with other methods given
missing data (e.g. optimized set reduction).

7. Acknowledgments
This work is being conducted as a part of the StagE
Project, the Development of Next Generation IT
Infrastructure, supported by Ministry of Education,
Culture, Sports, Science and Technology.

8. References[1] Cartwright, M., Shepperd, M.J. and Song, Q., “Dealing
with Missing Software Project Data,” Proc. 9th IEEE
International Software Metrics Symposium (Metrics'03),
Sydney, Australia, pp.154-165, 2003.
[2] Draper, N. R., Smith, H., Applied Regression Analysis, 3rded, John Wiley and Sons, New York, 1998.
[3] Foss, T., Stensrud, E., Kitchenham, B. and Myrtveit, I., “A
Simulation Study of the Model Evaluation Criterion
MMRE,” IEEE Transactions on Software Engineering,
Vol.29, No.11, pp.985-995, 2003.
[4] Hawkins, D.M., “The Problem of Overfitting,” Journal ofChemical Information and Modeling, Vol.44, No.1, pp.112, 2004.
[5] ISBSG Estimating, Benchmarking and Research Suite
Release 9, International Software Benchmarking Standards
Group, http://www.isbsg.org/[6] Jonsson, P. and Wohlin, C., “An evaluation of k-nearest
neighbour imputation using likert data,” Proc. 10th IEEE
International Software Metrics Symposium (Metrics'04),
Chicago, Illinois, pp.108-118, 2004.
[7] Kromrey, J. and Hines, C., “Nonrandomly missing data in
multiple regression: An empirical comparison of common
missing-data treatments,” Educational and Psychological
Measurement, Vol.54, No.3, pp.573-593, 1994.
[8] Little, R.J.A. and Rubin, D.B., Statistical Analysis with
Missing Data, 2nd ed., John Wiley and Sons, New York,
2002.
[9] Mendes, E., Lokan, C., Harrison, R. and Triggs, C. “A
Replicated Comparison of Cross-company and Withincompany Effort Estimation Models using the ISBSG
Database,” Proc. 11th IEEE International Software Metrics
Symposium (Metrics'05), Como, Italy, pp.36, 2005.
[10] Moses, J. and Farrow, M., “Assessing Variation in
Development Effort Consistency Using a Data Source with
Missing Data,” Software Quality Journal, Vol.13, No.1,
pp.71-89, 2005.
[11] Myrtveit, I., Stensrud, E. and Olsson, U. H., “Analyzing
data sets with missing data: An empirical evaluation of
imputation methods and likelihood-based methods,” IEEE
Transactions on Software Engineering, Vol.27, No.11,
pp.999-1013, 2001.
[12] Ohsugi, N., Tsunoda, M., Monden, A., and Matsumoto, K.,“Effort Estimation Based on Collaborative Filtering,” 5thInternational Conference on Product Focused Software
Process Improvement (Profes2004), Kyoto, Japan,
Vol.3009, pp.274-286, 2004.
[13] Sentas, P. and Angelis, L., “Categorical missing data
imputation for software cost estimation by multinomial
logistic regression,” The Journal of Systems and Software,
Vol.79, No.3, pp.404-414, 2006.
[14] Shepperd, M. and Schofield, C., “Estimating software
project effort using analogies,” IEEE Transactions on
Software Engineering, Vol.23, No.12, pp.736-743, 1997.
[15] Strike, K., El Eman, K. and Madhavji, N., “Software costestimation with incomplete data,” IEEE Transactions on
Software Engineering, Vol.27, No.10, pp.890-908, 2001.