Analysis of Software Maintenance Efficiency
Focused on Process Standardization
Masateru Tsunoda

Akito Monden

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

masate-t@is.naist.jp

akito-m@is.naist.jp

Ken-ichi Matsumoto

Tomoki Oshino

Nara Institute of Science andTechnology
Kansai Science City, 630-0192
Japan

Economic Research Institute,
Economic Research Association
Higashi-Ginza Mitsui Bldg. 5-13-16,
Ginza, Chuo-ku, Tokyo, 104-0061 Japan

matumoto@is.naist.jp

er352@zai-keicho.or.jp

ABSTRACTIn this research, to establish a benchmark for software maintenance efficiency, we analyzed factors affecting the work efficiency
and showed reference value stratified the factors. We analyzed
dataset of software maintenance collected from 83 organizations.
Attributes recorded in the dataset are standardization status oforganization, system architecture, the number of engineers, thenumber of base modules, the number of modified modules, and soon. In the analysis, we regarded modified modules per engineer asa work efficiency index, and clarified relationships to other attributes. As a result, we identified that standardization status of organization is important for work efficiency.

Categories and Subject Descriptors
K.6.3 [Management of Computing and Information Systems]:
Software Management – Software maintenance, Management ofComputing and Information Systems]: general – Economics.

General Terms
Management, Measurement, Economics.

Keywords
Module modification, productivity,
benchmarking.

cross-company dataset,

1. INTRODUCTION
Recently, a number of software users (companies or organizations) contract with software developers (companies) for maintenance of enterprise software, and therefore the agreement of software maintenance becomes more important. Software maintenance does not mean only removing faults found after software
release. Software needs extensions or modifications of its functions due to changes in a business environment, and software
maintenance also indicates them. ISO/IEC 14764 classifies software maintenance into followings:

1.

Corrective maintenance: modifications of faults found after
software release.

2.

Preventive maintenance: corrective modifications before
potential faults become actual faults, after software release.

3.

Adaptive maintenance: modifications to keep software
availability against environmental changing after software
release.

4.

Perfective maintenance: modifications for conservation or
improvement of software performance or maintainability after software release.

In this research, we try to establish a benchmark (reference values
to compare an organization’s work efficiency with others [5]) ofwork efficiency for software maintenance contract. To establish
the benchmark, factors affecting work efficiency (e.g. system architecture) are clarified first, and then the dataset is stratified by
the factors, using dataset collected from various organizations(cross-company dataset). When using the benchmark, compare
work efficiency with a reference value whose factor (e.g. systemarchitecture) is correspond to the target. We focus on the standardization of software maintenance process in the analysis. Whenstandardization status (process is standardized or not) has strong
relationships to work efficiency, reference values stratified standardization status can be used to confirm difference of work efficiency between standardized organizations and not standardized
ones. That is useful to decide whether software maintenance process should be standardized or not. Major contribution of ourresearch is to clarify relationships between standardization status
and maintenance efficiency using cross-company dataset.
The dataset used in the analysis was collected from 83 organizations in 2007 by Economic Research Association [2]. We analyzed relationships between work efficiency and other attributes
such as system architecture, in addition to standardization status.
The dataset does not have enough cases to use analysis results as
rigorous benchmark. So, values shown in the research should beTable 1. Description of dataset
Attribute

Description
Status of standardization of software maintenance process (process is standardized, standardization is work in
Process standardization
progress, or maintenance process is not standardized)
System architecture
System architecture on which the software runs (client–server system, Web based system, mainframe system)
Number of base modules Total number of modules included in the software
Number of modified
The number of modified modules in the maintenance activity in a year
modulesNumber of engineers
The number of contractor’s resident engineers
Modified modules per
The number of modified modules / number of engineers
engineer
Modified modules per
The number of modified modules / the number of engineers / the number of base modules
engineer and base
Human factor
Degree of difficulties about size of project (or organization) and level of skill
Problem factor
Degree of difficulties about type, importance, relationships, restriction, and ramification of problems
Process factor
Degree of difficulties about programming language and software development methodology
Product factor
Degree of difficulties about reliability, size, control structures, and complexity of the software
Resource factor
Degree of difficulties about hardware, duration, and budget
Tool factor
Degree of difficulties about library, complier, test tool, maintenance tool, and reverse engineering tool
used casually. Nevertheless, we think our result is effective because there are very few researches or reports which analyzed
software maintenance efficiency using cross-company dataset.
In the analysis, we regarded modified modules per engineer as awork efficiency index. That is, the number of engineers was considered as inputs, and the number of modified modules in a year
was considered as outputs (High modified modules per engineer
indicates high work efficiency). It is more precise that a work
efficiency index is defied using software maintenance effort andmodified lines of code (or function point). However, modifiedlines of code (and function point) in the dataset used includes
many missing values, and software maintenance effort is not recorded. In addition, it is easier for software uses (companies ororganizations) to measure the number of modified modules thanmodified lines of code or function point (Actually, the dataset isalmost collected from software uses). Similarly, it is easier tomeasure the number of engineers than software maintenance effort.
So using the number of modified modules and the number of engineers makes it easy for software users to refer the benchmark.
In what follows, Section 2 explains dataset used in the analysis.
Section 3 shows analysis results. Section 4 introduces related
works, and Section 5 concludes the paper with a summary.

2. DATASET
The dataset used in the analysis includes 83 cases of software
maintenance agreement which were collected from 83 organizations in 2007 by Economic Research Association [2]. 78 cases arebusiness software, and rest cases are factory automation softwareand other software. 46 cases are fixed price contract (Software
maintenance is performed during certain period by fixed price [6]).
The cases were collected mainly from software uses (companies
or organizations). Each case is representative software maintenance agreement in each organization (Each organization provided one case), and number of modified module was collected in ayear. The number of analyzed cases was different for attributes,
because each attribute includes missing values. Attributes ana-lyzed in this research are described in Table 1, and the followings
are detailed explanations for some attributes.
Process standardization status does not represent the statusof a case included in the dataset, but the status of the entire
organization which offered the case (Values of other attributes were collected in each software maintenance agreement).

When process is standardized, standard process of software
maintenance (sequence of activities such as analyzing, reviewing, documenting, and approval) is explicitly defined.

Some cases have multiple system architectures (for example,
a case of system architecture includes both mainframe andweb based system).

Modified modules per engineer and base is explained in
section 3.2.

Attributes from human factor to tool factor (We call themproductivity factors) are defined based on [6], and they were
evaluated on a three-point scale (Low value indicates severe
condition, i.e., productivity may be decreased).

3. ANALYSIS RESULTS
3.1 Analysis Procedure
In the analysis, we analyzed influences of system architecture,
productivity factor, the number of base modules, and the numberof engineers to work efficiency (modified modules per engineer),in addition to influence of process standardization. When an attributes has considerable influence, the dataset was stratified by
the attribute, to eliminate the influence of the attribute. For example, if system architecture had influence to work efficiency, thedataset was stratified by it, and a relationship between process
standardization and work efficiency was analyzed. The dataset has
missing values, so the number of analyzed cases is different ineach analysis.

Table 2. Relationships between size attributes
Attribute
Number of basemodulesNumber of modified
modulesNumber of engineers
Modified modules per
engineerρp-value
Number of cases
ρp-value
Number of cases
ρp-value
Number of cases
ρp-value
Number of cases

Number ofbase modules
1.00
35
0.70
0.00
25
0.23
0.30
22
0.60
0.02
15

When analyzing a relationship of ratio (or ordinal) scale attributes,
we used Spearman's rank correlation coefficient to avoid influenceof outliers. In what follows, “correlation” and ρ indicate theSpearman correlation. We did not apply a multivariate regression
model because there are many missing values in the dataset. Instead of applying it, we stratified the dataset as mentioned above.
We used a box plot to analyze relationships between nominal
scale attribute and ratio scale attribute. In a box plot, the bold line
in each box indicates the median value. Small circles indicate
outliers, that is, values that are more than 1.5 times larger than the
25%-75% range from the top of the box edge. Stars indicate extreme outliers, whose values are more than 3.0 times larger than
this range. Some outliers do not include figures to improve readability of them. Mann–Whitney U test was applied to confirm median is statistically different or not between two attributes. We set
significance level as 5%.

Number of modified
modules
0.70
0.00
25
1.00
25
0.56
0.03
15
0.83
0.00
15

Number ofengineers
0.23
0.30
22
0.56
0.03
15
1.00
35
0.03
0.92
15

Modified modules per
engineer
0.60
0.02
15
0.83
0.00
15
0.03
0.92
15
1.00
15

relationships are illustrated in Figure 1. Observations are as follows:

There is a positive correlation between the number of modified modules and the number of engineers (ρ = 0.56). So,
when the number of engineers is large, the number of modified modules is also large.

The correlation between the number of engineers and thenumber of base modules is not large (ρ = 0.23). Therefore,
size of entire software does not greatly affect the number ofengineers.



The correlation between the number of engineers and modified modules per engineer is very small (ρ = 0.03). This does
not mean fewer engineer increases modified modules per
engineer.

We analyzed relationships between modified modules per engineer, the number of base modules, and the number of engineers,
to confirm whether modified modules per engineer can be used aswork efficiency index or not. It may be helpful in comprehending
the analysis to regard the number of base modules as function
point, the number of engineers as development effort, and modified modules per engineer as productivity.

The number of base modules has significant correlation to modified modules per engineer (ρ = 0.60) and the number of modified
modules (ρ = 0.70). It may be likely that when the number of basemodules is large, the number of modified modules is also large,
and as a result, modified modules per engineer becomes large. So
we defined modified modules per engineer and base (the numberof modified modules / the number of engineers / the number ofbase modules), to eliminate influence of the number of base module, and used it to strengthen analysis results.

Table 2 shows correlations of the relationships, and the major

3.3 Relationship to System Architecture

3.2 Relationships between Size Attributes

0.60

Total number ofmodules

0.23

0.70

Number ofmodified
modules

0.56

Modified modules
per engineer(Work efficiency)

0.03

Number ofengineers

Figure 1. Major correlations between the number of basemodules, number of engineers, and other attributes

Figure 2 and Figure 3 show boxplots of modified modules per
engineer and modified modules per engineer and base for threedifferent system architectures. In modified modules per engineer(Figure 2), mainframe system is the highest, and Web based system is the lowest. The median value of Web based system is 2.2
times smaller than client-server system, and 6.9 times smaller thanmainframe system, as shown in Table 3. However, they are notstatistically different. There are not enough cases in the dataset(see Table 3), and it may affect statistical test results.
Also, in modified modules per engineer and base, mainframe system is the highest while the distribution is wide, as shown in Figure 3. The distribution (position and size of the box in the figure)of Web based system is almost same as client-server system. The
median value of Web based system is same as client-server system,

Table 3. Median values of modified modules per engineer (and base) stratified by system architecture

Modified modules per
engineer

Modified modules per
engineer and base

Systemarchitecture

Number ofcasesMedianClient-server
Web Based
Mainframe
Total
Client-server
Web Based
Mainframe
Total

9
11
8
28
9
11
8
28

40.0
18.0
125.0
55.0
0.007
0.007
0.011
0.008

p-value(difference fromWeb based)
88%
21%
82%
27%
-

p-value(difference fromclient-server)
88%
37%
82%
37%
-

Table 4. The number of base modules, the number of modified modules, and the number of engineers
stratified by system architecture
System architecture
Client-server
Web Based
Mainframe
Total

MedianNumber of cases
MedianNumber of cases
MedianNumber of cases
MedianNumber of cases

Number ofbase modules
3000
25
3000
23
4628
21
3000
69

and 1.6 times smaller than mainframe system (Table 3). Note that
there is no statistically difference among them.
In the results, modified modules per engineer (and base) is larger
when system architecture is mainframe system. Although weshould carefully understand it because there is no statisticallydifference, it suggests system architecture should be consideredwhen comparing work efficiency (modified modules per engineer).

Number ofengineers
10
26
10
21
10
20
10
67

programming languages.
We counted the number of base modules, the number of modified
modules, and the number of engineers by system architecture(Table 4). Observations are as follows:Median values of the number of engineers are same amongsystem architectures.



In median values of the number of modified modules, mainframe system is 2.2 times larger than Web based system, and3.2 times larger than client-server system.

In median values of the number of base modules, mainframe
system is 1.8 times larger than others.

In mainframe systems, it may be probable that the number of base

Modified modules
per engineer

Modified modules
per engineer and base

We examined programming language used in each system. In the
result, the most used programming language is Visual Basic inclient-server systems, it is SQL in Web based systems, and it is
COBOL in mainframe systems (Note that multiple programming
languages are used in each system). Difference of work efficiency
between system architectures may be affected by characteristics ofNumber ofmodified modules
125
16
180
17
400
14
250
47

ClientWeb
Mainframe
serverbased
System architecture

Figure 2. Modified modules per engineer stratified
by system architecture

ClientWeb
Mainframe
serverbased
System architecture

Figure 3. Modified modules per engineer and base stratified
by system architectureTable 5. Median values of modified modules per engineer (and base) stratified by process standardization
Process
standardization
Modified modules per
engineer

Modified modules per
engineer and base

Number ofcasesMedian

6
8
1
15
6
8
1
15

118.3
14.0
7.1
40.0
0.026
0.005
0.007
0.008

Standardized
Work in progress
Not standardized
Total
Standardized
Work in progress
Not standardized
Total

p-value(difference fromstandardized)
11%
29%
2%
57%
-

p-value(difference fromnot standardized)
29%
89%
57%
100%
-

Table 6. The number of base modules, the number of modified modules, and the number of engineers
stratified by process standardization
Process standardization
Standardized
Work in progress
Not standardized
Total

MedianNumber of cases
MedianNumber of cases
MedianNumber of cases
MedianNumber of cases

modules increased the number of modified modules (see section
3.2). However, relative difference of the number of base modules
between mainframe system and others is smaller than the numberof modified modules. This reinforces the analysis result that work
efficiency is different for system architecture.

3.4 Relationship to Process Standardization

Number ofmodified modules
400
10
100
12
5
3
180
25

Number ofengineers
6
11
10
17
6
7
10
35

ferent.
Same tendency was observed in modified modules per engineer
and base. Figure 5 shows a relationship between process standardization and modified modules per engineer and base, and Table 5
presents median values of them and statistical test results. In Figure 5, the values of “standardized” are larger than “work in progress.” In Table 5, the median value of “standardized” is 5.2 times
larger than “work in progress,” and there is statistically difference
between them. The results indicate process standardization isrelated to work efficiency.
We checked the number of base modules, the number of modified
modules, and the number of modules, stratifying the dataset according to process standardization. In Table 6, between “standardized” and “work in progress,” there is 0.6 times difference in the

Modified modules
per engineer

Modified modules
per engineer and base

Figure 4 illustrates a relationship between process standardizationand modified modules per engineer, and Table 5 shows medianvalues of them and statistical test results for their differences. In
Figure 4, when process is standardized, modified modules per
engineer is the highest, although the distribution is wide. The
median value of “standardized” is 8.5 times larger than “work inprogress,” as shown in Table 5. But they are not statistically dif-

Number ofbase modules
3000
13
4000
15
1000
7
3000
35

Standardized Work in
Not
progress standardized
Process standardization

Figure 4. Modified modules per engineer stratified
by process standardization

Standardized Work in
Not
progress standardized
Process standardization

Figure 5. Modified modules per engineer and base stratified
by process standardization

Table 7. Relationships to productivity factors

Modified modules
per engineer
Modified modules
per engineer andbaseρ
p-value
Number of cases
ρp-value
Number of cases

Humanfactor
-0.09
76%
13
-0.26
39%
13

Problemfactor
0.38
20%
13
-0.10
74%
13

Process
factor
0.20
51%
13
-0.23
46%
13

Product
factor
0.32
29%
13
-0.01
97%
13

Resource
factor
-0.02
96%
13
0.06
85%
13

Tool
factor
0.07
81%
13
0.00
100%
13

number of engineers, and 0.75 times difference in the number ofbase modules. Nevertheless, there is 4.0 times difference in thenumber of modified modules. So in this case, we do not have tocare the positive correlation between the number of base modules
and the number of modified modules (see section 3.2). This also
supports the result that process standardization is related to work
efficiency.

and Industry used the cross-company dataset, and showed work
efficiency (maintenance cases per engineer) stratified by businesssector [3]. However they did not clarified the relationship between
process standardization and work efficiency. Yokota [7] showed
standardizing maintenance process is effective for work improvement, based on questionnaire data. But quantitative data analysis
was not performed.

When stratifying the dataset by system architecture, both modified
modules per engineer and modified modules per engineer andbase are higher in process standardized organizations (except for
modified modules per engineer in mainframe systems). Similarly,
when stratifying the dataset by process standardization, they arehigher in mainframe systems. Although we should be care that
there are a few cases in each group after stratifying, both systemarchitecture and process standardization are considered to affectwork efficiency.

5. CONCLUSIONS

3.5 Relationship to Productivity Factors
Table 7 shows correlations between productivity factors and modified modules per engineer (and base). When a value of productivity factor is smaller, condition is more severe. That is, positive
correlation indicates work efficiency decreases when the conditionof the factor is severe.
Although the correlation between problem factor and modified
modules per engineer is relatively higher (ρ = 0.38; not significant), the correlation to modified modules per engineer and baseis low (ρ = -0.10). So, we cannot conclude problem factor affects
work efficiency. Other factors have low correlation, and they are
not statistically significant. In addition, their correlations are inconsistent between modified modules per engineer and modified
modules per engineer and base. Namely, we did not observe explicit influences of productivity factors on work efficiency.

4. RELATED WORKS
Except for process standardization, some researches analyzed
work efficiency factors on software maintenance. Jørgensen [4]analyzed software company dataset, and showed that work efficiency is not affected by the number of base modules and programming language. Ahn et al. [1] used variables which are similar to the productivity factors in a software maintenance effort
estimation model. However, past researches did not clarified effect of process standardization. This is because they used dataset
collected from a few companies, and it made the analysis of process standardization effect difficult.
There are very few reports or researches which analyzed crosscompany software maintenance dataset. Japan Users Associationof Information Systems (JUAS) and Ministry of Economy, Trade

In this research, to establish a benchmark for software maintenance efficiency, we analyzed software maintenance data collected from 83 organizations. Modified modules per engineer is regarded as work efficiency index. We compared work efficiency bystratifying the dataset, and clarified process standardization statusis related to work efficiency (modified modules per engineer). Our
future work is collecting more data and analyzing it to enhancereliability of the results.

6. ACKNOWLEDGMENTS
This work is being conducted as a part of the StagE project, The
Development of Next-Generation IT Infrastructure, and Grant-inaid for Young Scientists (B), 22700034, 2010, supported by the
Ministry of Education, Culture, Sports, Science and Technology.

7. REFERENCES[1] Ahn, Y., Suh, J., Kim, S., and Kim, H. 2003. The software
maintenance project effort estimation model based on function points. Journal of Software Maintenance: Research andPractice, 15, 2, 71-85.
[2] Economic Research Association. http://www.zaikeicho.or.jp/about/english.php[3] Japan Users Association of Information Systems 2008. Software Metric Survey 2008. Japan Users Association of Information Systems.
[4] Jørgensen, M. 1995. Experience With the Accuracy of Software Maintenance Task Eﬀort Prediction Models. IEEE
Transactions on Software Engineering, 21, 8, 674-681.
[5] Lokan, C., Wright, T., Hill, P, and Stringer, M. 2001. Organizational Benchmarking Using the ISBSG Data Repository.
IEEE Software, 18, 5, 26-32.
[6] Software Evolution Research Consortium 1995. Software
Evolution Research Consortium Report, Software Evolution
Research Consortium.
[7] Yokota, T. 2003. An Assessment for Software Maintenance
Environment and its Improvement Examples. Journal of theSociety of Project Management, 5, 2, 40-44.