2023 30th Asia-Pacific Software Engineering Conference (APSEC)

Cost-Beneﬁt Analysis for Modernizinga Large-Scale Industrial System
Kazuki Yokoi∗ , Eunjong Choi† , Norihiro Yoshida‡ , Joji Okada∗ and Yoshiki Higo§∗ NTT DATA Group Corporation, Japan, {Kazuki11.Yokoi, Joji.Okada}@nttdata.com† Kyoto Institute of Technology, Japan, echoi@kit.ac.jp‡ Ritsumeikan University, Japan, norihiro@fc.ritsumei.ac.jp§ Osaka University, Japan, higo@ist.osaka-u.ac.jp

engaged in the modernization of real software projects in theircooperation.
Although several studies have been conducted to estimate
the cost-beneﬁt of refactoring [4]–[8], to the best of our knowledge, the cost-beneﬁt estimation for incremental modernization has not yet been explored. To bridge this gap, we present
a case study that conducts a cost-beneﬁt estimation analysis
on a large-scale industrial system, speciﬁcally focusing on
incremental modernization. The goal of this study is to present
a case study that estimates the cost-beneﬁt of incremental modernization in a large-scale industrial system. To achieve thisgoal, in the case study, we apply the cost-beneﬁt estimation
approach, which employs dependency analysis, to a core system within a ﬁnancial institution. Then, we compare the actualand estimated cost-beneﬁt values in the context of incremental
modernization. Consequently, we conﬁrm the effectiveness ofthe cost-beneﬁt estimation approach for estimating the costbeneﬁt associated with incremental modernization.
The main contributions of this study are as follows:• We have preliminary estimated cost-effectiveness, whichis essential in system modernization for a large-scaleﬁnancial system.
• We developed a model for calculating a cost-beneﬁt forincremental modernization for stable system modernization.
• We have analyzed the difference between the preestimated cost-beneﬁt and the actual cost-beneﬁt anddiscussed why the difference occurred.
The remainder of this paper is organized as follows. Section
II presents the background of this study. Section III describesa system that underwent incremental modernization, which isthe focus of this study. Section IV explains the estimation
approach used in this study. Section V presents the analysis.
Section VII introduces the related work. Finally, Section VIII
concludes the paper and discusses future challenges.

Abstract—Legacy systems pose signiﬁcant challenges to companies. Software modernization approaches have been proposed
to address this issue. However, a lack of standardization andreliance on ad hoc processes often lead to software modernization
failures. Incremental modernization, a strategy that improves
software systems in a step-by-step manner rather than attempting
to simultaneously overhaul the entire system, aims to mitigate therisk of failure. However, this approach can increase costs owing
to the complexity of integrating legacy and modernized products.
In this paper, we present a case study that employs a costbeneﬁt estimation analysis in a large-scale industrial project that
underwent incremental modernization in the past. We compare
the actual and estimated cost-beneﬁt values in the context ofincremental modernization. As a result, we conﬁrmed that thecost estimates were valid, but we could not judge whether the
beneﬁt estimates were valid.
Index Terms—software maintenance, dependency analysis, costestimation

I. I NTRODUCTION
Various software modernization approaches have been proposed to address critical challenges posed by legacy systems[1]. Many companies demonstrate a strong demand for software modernization, motivated by the necessity of rejuvenating legacy systems for agile responses to business changes andcost-effective maintenance. Frequently, the systems targeted
for modernization are critical, allowing for minimal tolerance
for failure. However, the lack of standardization and relianceon ad-hoc processes often leads to failures in software modernization. Incremental modernization, which is an approachthat improves software systems in a step-by-step manner rather
than attempting to overhaul the entire system simultaneously,
aims to mitigate the risks of failure [2], [3].
Software modernization approaches can increase costs because of the complexity of integrating legacy and modernizedprojects [2]. Therefore, estimating the cost-beneﬁt at the planning phase of modernization is crucial, especially in industry.
However, it is challenging to estimate the cost-beneﬁt ofincremental modernization in the planning phase. For example,
the lack of design documents hinders the accurate estimationof development costs and beneﬁts. Furthermore, it is difﬁcult
to predict the main risk factors for incremental modernizationin the planning phase; as a result, the actual costs may
exceed the estimated costs. The ﬁrst and fourth authors of
this study frequently encountered such scenarios when they2640-0715/23/$31.00 ©2023 IEEE
DOI 10.1109/APSEC60848.2023.00055

II. BACKGROUND
A. Modernization
Several modernization approaches have been proposed over
the years, including the “big bang” approach [1] and incremental modernization [3]. Figure 1 shows the overview of“big bang” and incremental modernization. The left side of441

development costs for the entire system.
B. Cost Estimation
Cost estimation is an empirical process used to estimate
the effort and time costs of any software project before its
development [10]. In software development, these estimatedcosts are often inadequate, causing a discernible gap between
the estimated and actual effort. This gap tends to widen, particularly during the planning phase of software development,
owing to many ambiguous elements (e.g., requirements anddesign). However, cost estimation during the planning phaseis crucial for many companies because it facilitates budget
allocation and overall planning. Therefore, it is necessary torecognize inherent uncertainty embedded in the planning phaseof cost estimation, communicate this to all stakeholders, andre-estimate the associated costs.
Several cost estimation approaches have been proposed,
including analogy-based, bottom-up, and parametric. An
analogy-based estimation was derived through comparison
with similar past projects. Although this approach is straightforward and suitable for early-stage system development estimates, its application becomes challenging when the background, constraints, and features of past project performances
are unclear.
Bottom-up estimation involves identifying the components
of a target project, aggregating the effort estimates for eachcomponent, and calculating the estimated value for the entire
project. While this approach tends to be more accurate, it
requires breaking down a target project into smaller, moremanageable tasks using a structure such as a Work Breakdown
Structure (WBS) and pre-identifying components such as subsystems and components as part of the deliverables. Therefore,
adopting this method in the initial stages of a project ischallenging because of inherent ambiguity.
The parametric approach involves representing factors suchas effort as the dependent variable and attributes such as
size and factors as independent variables through mathematical functions. This approach primarily employs an equation
effort in which effort and size exhibit direct proportionality(E = α × Size) or size is proportional through a powerfunction (E = α × Sizeβ ). The latter, in which the effort
is proportional to the exponent of size, was also employedin the COCOMO method [11]. In this study, we adopted thisapproach for cost estimation.

Fig. 1. “big bang” approach and incremental modernization

Figure 1 shows the “big bang” approach, which simultaneously
reconstructs the entire system. However, this approach carries
a signiﬁcant risk of migration failure. To mitigate this risk,
server approaches including incremental modernization [3] andthe “Chicken-Little” strategy [2] have been proposed. These
approaches involve dividing the system improvement or migration process into smaller and more manageable components to
reduce the risk. The right side of Figure 1 illustrates the coreidea of the “Chicken-Little” strategy to construct a composite
system consisting of both legacy and modernized products.
Communication between these legacy and modernized products occurs through a gateway. In this strategy, a segmentof the legacy product is initially replaced by a modernizedproduct, and the gateway directs requests to the modernizedproduct. Over time, the proportion of the legacy product being
replaced by the modernized product gradually increases. This
process leads to a step-by-step expansion of the modernizedproduct.
While the “Chicken-Little” strategy mitigates the risk of
migration failure, it introduces increased risks due to thecomplexity of integrating legacy and modernized projects.
These risks include:
1) Complex communication control between the legacy andmodernized products.
2) Ensuring data consistency between the legacy and modernized products becomes challenging.
To mitigate these risks, the integration of legacy and modernized products requires a shell API [9]. Shell API refers
to codes or scripts that enable interoperability between different software components or systems [9]. This enables thecollaboration between independently developed components or
systems. In the context of incremental modernization, integrating different technology stacks or platforms between legacy
and modernized products requires communication control anddata consistency management through a shell API. Consequently, incremental modernization incurs additional costs fordeveloping shell API, resulting in an overall increase in theIII. TARGET P ROJECT
In this section, we introduce the target project for incremental modernization in this case study. The target project was
the core project of a ﬁnancial institution in Japan, and thissystem has undergone incremental modernization. The project
was a COBOL core system, and the corporation where the ﬁrst
and fourth authors of this paper are currently employed has
been responsible for maintaining and developing the projectfor many years. A summary of the system is presented in Table
I. It was a substantial legacy system with inherent issues of low
maintainability prior to modernization. The primary objective442

Fig. 2. Estimation approach overview

systems. To answer the above-mentioned RQs, we present anapproach for estimating the cost-beneﬁt in the planning phaseof incremental modernization and assess the validity of theestimation approach by applying it to the target project.

TABLE IS UMMARY OF TARGET S YSTEM
System Overview
System Type
Language
SizeCore System of ﬁnancial institutions
Batch and Online System
Open COBOL
About 4.6 MLOC (Only COBOL)

IV. E STIMATION A PPROACH
In this section, we introduce an approach for estimating thecost-beneﬁts of incremental modernization. Figure 2 presents
an overview of this estimation approach. As can be seen in theﬁgure, the approach employs a dependency graph constructed
through dependency analysis and then estimates the costbeneﬁt of incremental modernization based on this graph. In
the following sections, we describe the dependency graph,followed by explanations of cost and beneﬁt estimations.

of the incremental modernization of the target system was toenhance its maintainability.
Through the ﬁrst incremental modernization, approximately
870 KLOC were extracted from the target system and restructured into Java programs. Unfortunately, the complexityand potential risks associated with incremental modernization
were insufﬁciently considered during the planning phase ofmodernization. Consequently, the actual size of the target
system exceeded the estimated size by 27.5%.
Although the primary objective of incremental modernization was to improve maintainability, a comparison of thedevelopment efforts before and after modernization revealed
an uncertain enhancement in maintainability. Moreover, owing
to the absence of deﬁned metrics to measure this improvement,
it remains unclear whether the project yielded any beneﬁts
after the modernization.
This experience indicates that the corporation, involving theﬁrst and fourth authors of this paper, encountered challenges
in estimating the cost-beneﬁt of incremental modernization
during the planning phase. Moreover, apart from the project
discussed here, the corporation is involved in several otherprojects in which incremental modernization is being considered. Therefore, this issue is very relevant for the corporation.
However, even though several existing studies have estimatedthe cost-beneﬁt of refactoring [4]–[8], cost-beneﬁt estimatesfor incremental modernization have yet to be explored, to thebest of our knowledge. To bridge this gap, we propose thefollowing Research Questions (RQs):
RQ1 Is it possible to estimate more valid costs during theplanning phase of incremental modernization?
RQ2 Is it possible to estimate more valid beneﬁts during
the planning phase of incremental modernization?
This study aims to present a case study estimating the costbeneﬁt of incremental modernization in large-scale industrial

A. Dependency GraphFirst, we discuss the dependency graph used in this approach. A dependency graph is a graph structure that represents the relationships among software components. In
this graph, the nodes depict code entities (e.g., methods orfunctions) and data entities (e.g., database tables or global
variables). The relationships among these nodes are illustrated
as directed edges. These edges represent various dependencies
between the software components, including call dependencies, where one method or function calls another, or data
dependencies, where a piece of code relies on a speciﬁc data
entity. These dependencies are categorized as “relation types”[12]. These edges include various types of dependencies,
such as call dependencies and data dependencies, which arereferred to as ”relation types.” Figure 3 shows an example
of a dependency graph. In this ﬁgure, the nodes represent
functions and database tables, and the edges indicate call anddata dependencies.
B. Cost Estimation
The cost estimation calculates the person-hours for incremental modernization. This involves determining the estimatedwork hours by multiplying the calculated person-hours by themonthly rate per person, resulting in a monetary value.
In the cost estimation, the size of the development forincremental modernization was calculated. In the case ofmodernization of the entire system, the effort is calculated443

Fig. 3. Example of dependency graphby multiplying the size of the entire system by productivity.
During incremental modernization, part of the system was
extracted and rebuilt. However, in some cases, by simply
multiplying the size of the extracted program by productivity, the estimated effort may be less than the actual effort.
This is because the estimate of the shell API effort was
missing. Therefore, we propose an approach for estimating
the development size of shell API to obtain the shell API
development effort. Dependency graphs are used to estimate
the development size of the shell API.
1) Procedure for Cost Estimation: The development effort
E for the shell API can be determined as follows:

Nd Ld
(1)
E =α×

Fig. 4. Examples of functions and tables used for cost estimation
TABLE IIE XAMPLE OF EFFORT ESTIMATION FOR EACH DEPENDENCY
Dependency
Type
Call
Data

Number of
Dependency
20
10

LOC per
Dependency Productivity
Effort
40
855
0.93
100
855
1.17
Sum of Effort
2.10
Note1: LOC per dependency and productivity are tentative value.
Note2: Productivity is measured in LOC/person-month, and effort ismeasured in person-month.

d∈DHere, Nd denotes the number of dependencies d spanningthe legacy and modernized products, Ld represents the shell
API implementation line number per dependency d, andα signiﬁes the productivity (person-months/size). These aresummed up for each type of dependency d.
When a modernized product is extracted from a legacy
product, the dependencies between the legacy and modernized products are disconnected. For example, dependencies
connected by function calls before extraction change to communication via the network after extraction. Therefore, the development of components to convert communication systems
requires considerable time and effort.
The number of lines of the shell API implemented for eachtype of dependency depends on the architecture of the legacy
and modernized products. Therefore, estimates of these values
are determined by analogy from past development results orby developing a shell API as a proof of concept and collecting
data.
2) Example of Cost Estimate Calculation: As illustrated in
Figure 4, if only function B is decomposed from the others,
communication between functions A and B, functions B andC, and function B and table 1 occurs via the shell API.
Therefore, for incremental modernization, the development of
a new shell API is necessary. From Table II, the number of
dependencies via the shell API is 20 for Call Dependency
and 10 for Data Dependency. Thus, the size to implement
one Call Dependency with the shell API is 40 LOC, and thesize to implement one Data dependency with the shell API is

100 LOC, which are tentative values. When multiplied by the
inverse of productivity shown in Table II, the shell API effort
is estimated to be 2.10 person-months.
C. Beneﬁt Estimation
Modernization has several beneﬁts. In this study, however,
we estimated the beneﬁt of the reduction in maintenanceeffort. A reduction in maintenance effort results in shorter
maintenance periods and lower maintenance costs. As a result,the maintenance period per project is shortened, and development agility is improved. Furthermore, reducing maintenancecosts will promote investment in other areas, emphasizing the
beneﬁts of modernization.
In this study, we estimated the beneﬁts of a reduction inmaintenance effort per project after incremental modernization.
Previous research has shown that software maintenancerequires signiﬁcant effort to understand and test [13]–[15],and it is evident that this effort is signiﬁcantly inﬂuenced
by change impact [16], [17]. Therefore, the reduction rate ofmaintenance effort can be affected by the reduction rate of thechange impact.
Impact scope refers to the range within which changes to apart of a program can affect other programs. A broader change
impact means a more comprehensive range to understandand test, increasing the maintenance effort. By decomposing
systems through incremental modernization, we estimated the444reduction rate of this change impact and its effect on reducing
the maintenance effort.
1) Procedure for Estimating Beneﬁt: The reduction rateof maintenance effort, R, is determined by the following
Equation:
Pm CIm
R = 1 − m∈M
(2)
m∈M Pm CIm
In this context, M represents the set of all modules inthe entire system. Pm denotes the probability that modulem (within the module set M ) will be modiﬁed during amaintenance activity. CIm and CIm
denote the change impact
for module m before and after modernization, respectively.
The change impact for a module is measured by the lines ofcode to be tested when the module is modiﬁed.
Equation (2) calculates the expected change impact acrossthe entire system by multiplying the change probability of amodule with its change impact. The expected change impact
was calculated because not all modules will be modiﬁed in asingle maintenance project. For instance, modules that are frequently modiﬁed will have a higher effect when reducing theirchange impact, whereas less-modiﬁed modules will have alesser effect. Therefore, Equation (2) considers the probabilityof a change for each module. This study determines the change
probability Pm based on the proportion of the module m size
to the entire system size. This is based on the assumptionthat larger modules have a higher likelihood of modiﬁcationand smaller modules have a lower likelihood. Furthermore,
the change impact CIm is calculated by summing the lines ofcode for module m and all the modules that depend on module
m. This is because the modules that depend on the changed
module must be retested for change impact. These concepts
are supported by [4].
The expected change impact after modernization is calculated by assuming that dependencies spanning legacy andmodernized products will be eliminated. For instance, if amodule had a dependency through a function call before
decomposition, but communicates via the network after decomposition, the coupling between modules weakens, making
testing more accessible. Consequently, the expected change
impact is reduced.
Based on the above, we can estimate the maintenance effort
reduction rate from the reduction ratio of the expected change
impact between before and after modernization.
2) Example of Beneﬁt Estimate Calculation: As shown in
Figure 5, if a maintenance activity modiﬁes module D, all codein modules A, B, C, and D must be retested. The probabilityof a maintenance event changing module D is the ratio of thecode size to the total module size (40% from Table III). The
contribution to the mean system change impact is the sum of
the change impact of the module multiplied by the probabilityof occurrence. The contributions from all modules were thensummed to provide the expected value of the change impact
for the system.
When modules C and D are decomposed from modules A
and B, communication between modules B and C occurs via

Fig. 5. Examples of modules used for beneﬁt estimation
TABLE IIIE XAMPLE OF EXPECTED CHANGE IMPACT BEFORE MODERNIZATION
Module
A
B
C
D

CIm
LOC
Pm
10
10%
10
20
20%
30
30
30%
60
40
40%
100
Expected Change Impact

Contrib. to
Expected
1
6
18
40
65

TABLE IVE XAMPLE OF E XPECTED C HANGE I MPACT AFTER MODERNIZATION
Module
A
B
C
D

CIm
LOC
Pm
10
10%
10
20
20%
30
30
30%
30
40
40%
70
Expected Change Impact

Contrib. to
Expected
1
6
9
28
44

the shell API. Therefore, the shell API prevents the expansionof the change impact. Tables III and IV show that the change
impact of module C was reduced from 60 to 30 and that ofmodule D from 100 to 70 before and after modernization.
Consequently, the expected change impact was reduced from65 to 44, resulting in a 32.3% reduction in the maintenanceeffort.
V. A NALYSISThis analysis aimed to conﬁrm the effectiveness of the
proposed estimation approach. In the following, we refer tothe estimation approaches described in Section IV as proposed
approaches.
A. RQ1
To address RQ1, we applied the cost estimation approachto the target system described in Section III. We estimated thedevelopment size and analyzed the gap from the actual value.
First, we analyzed the dependencies of the target systemand created a dependency graph. The vertices of the graphrepresent the source ﬁles, and the edges represent the callrelationships. In general, COBOL programs are compiledﬁle-by-ﬁle, and other programs are called using the CALL
statement. Figure 6 shows an example of the COBOL calldependency between two COBOL programs PRG001.cob

445

IDENTIFICATION DIVISION.
PROGRAM-ID. PRG001.
*> omitPROCEDURE DIVISION.
CALL 'PRG002' USING VAR1 VAR2.
*> omitFirst, a dependency analysis was performed on the target
system, a dependency graph was created, and programs wereclassiﬁed into those extracted and reconstructed as modernizedproducts and those left as legacy products. This process was
the same as that described in Section V-A. We estimate theexpected change impact before and after modernization using
the formula denoted Equation (2).
This study presents the development performance statistics
before and after incremental modernization in Table V. We
focused on 21 cases of maintenance activity before modernization and 10 cases after modernization. We excluded
cases where the production size, test size, and development
productivity were incorrectly recorded.
For development performance, we aggregated both the mean
and median for the following three items:• Code to be tested in IT1:
Lines of code to be tested in integration test 1 (IT1) phase.
IT1 is closer to a unit test as it is on a smaller level thanIT2.
• Code to be tested in IT2:
Lines of code to be tested in integration test 2 (IT2) phase.
IT2 is closer to a system test as it is on a larger level than
IT1.
• Productivity:
Development person-hours per production size. The unit
is KLOC/person-month. Development hours are the sumof the hours from basic design to system testing.
A smaller line of code to be tested indicates a smaller change
impact and suggests improved maintainability. A more signiﬁcant value for the development productivity indicates thatthe number of implementation lines per person-month is moresigniﬁcant, suggesting improved maintainability. In addition,
the development requirements before and after modernization
did not match. Therefore, in this study, we compare thedevelopment productivity by dividing the development hours
by the production size.
With the median of the IT2 Size showing a positivechange rate and both the mean and median of Productivity
showing negative change rates, we were unable to conﬁrm
that maintainability necessarily improved due to incremental
modernization.
We analyzed the gap estimation from the development performance shown in Table V. We estimated the performance after modernization by multiplying the reduction rate determinedby Equation (2). We then compared the actual development
performance after modernization with our estimates to assessthe degree of the gap. This gap was calculated using the same
formula for the cost estimation MRE.

IDENTIFICATION DIVISION.
PROGRAM-ID. PRG002.
DATA DIVISION.
LINKAGE SECTION.
01 PARAM1PIC X(10).
01 PARAM2
PIC 9(5).
*> omitFig. 6. Example of COBOL call dependency

and PRG002.cob. By performing dependency analysis, practitioners can determine which COBOL programs are calledfrom other programs.
After creating a dependency graph, the programs were categorized into those to be extracted and rebuilt as modernizedproducts, and those to be left as legacy products. Subsequently,
the number of dependencies between programs rebuilt for themodernized product and those remaining in the legacy productwas tallied.
Finally, the overall development size of the system was determined by summing the development sizes of the modernizedproduct, legacy product, and shell API (see Section II-A).
The development sizes of legacy and modernized products
reuse estimated results from the planning phase of the target
project. During the planning phase, developers were unawareof the need to develop shell API in incremental modernization.
Consequently, the estimated size of the planning phase wasreused in this analysis. The development size for the shell
API was estimated using the formula described in Equation(1) in Section IV-B. Note that productivity α is excluded fromthe calculation in this analysis because we evaluated the size
estimation instead of the effort estimation.
The accuracy was evaluated by examining the deviationof the cost model results from the actual development scaleof 3,691 KLOC. Compared to the 2,675 KLOC estimated at
the planning stage of the target system, we evaluated which
method had a lower deviation rate from the actual values,
the proposed approach, or the actual values. This gap was
calculated using the Magnitude of Relative Error (MRE), as
shown in the following Equation:
M RE =|(actual) − (estimated)|(actual)(3)

B. RQ2
To answer RQ2, we applied the cost estimation approachto the target system described in Section III. We estimatedthe reduction in LOC to be tested and then analyzed its gap
from the actual values. Because the cost of the testing phase isrelatively high during the development process, reducing theeffort of the testing phase leads to a considerable reduction in
the overall effort. We estimated the reduction in the test scale
based on the assumption that the number of person-hours in
the testing process increases or decreases proportionally withthe test scale [16], [17].

VI. R ESULTS AND L ESSON L EARNED
A. RQ1
The number of call dependencies from the modernizedproduct to the legacy product was 3,431, and from the legacy
to the modernized was 80, totaling 3,511 inter-system callrelationships. The LOC size per program was 343.3 LOC.

446

TABLE V
D EVELOPMENT P ERFORMANCE S TATISTICS B EFORE AND A FTER I NCREMENTAL M ODERNIZATION

Number of Cases
MeanMedianCode to be tested in IT1 [KLOC]
Code to be tested in IT2 [KLOC]
Productivity [KLOC/person-month]
Code to be tested in IT1 [KLOC]
Code to be tested in IT2 [KLOC]
Productivity [KLOC/person-month]

Before Modernization
21
22.75
24.04
0.27
14.78
17.77
0.24

Estimation in this analysis
Estimation at the planning phase
Actual Value

Change Rate
-7%
-3%
-30%
-8%
+21%
-21%

B. RQ2TABLE VIC OST E STIMATION : ACTUAL AND ESTIMATED D EVELOPMENT SIZE
Development Size
3,880 KLOC
2,675 KLOC
3,691 KLOC

After Modernization
10
21.24
23.28
0.19
13.53
21.49
0.19

In the target project, the expected change impact before
incremental modernization was 7.99 KLOC, whereas, after
incremental modernization, it became 6.99 KLOC. From
Equation (2), it is estimated that the maintenance effort could
be reduced by 13%.
Table VII compares the actual value after incremental modernization; the estimated value reduced by 13% from the actual
value before incremental modernization. From Table VII, theestimated value displayed MRE ranging from 5% to 68%
compared to the actual values after incremental modernization.
Although the degree of MRE varied, the actual values weremore signiﬁcant than the estimated values for the code tested
in IT1 and IT2. Furthermore, productivity had smaller actual
values. This indicates that no development performance was
as effective as estimated.

Answer to RQ2

MRE
5.1%
27.5%
-

Therefore, the development size of the shell API was estimatedto be 1,205 LOC. By adding the estimated size of the shell
API to the previously estimated size in the planning phase(excluding the shell API) of 2,675 KLOC, the estimated total
development size for the entire system was 3,880 KLOC.
A comparison between the estimated and actual values isshown in Table VI. The development size estimated using
the proposed approach was 3,880 KLOC. Compared with theactual development size of 3,691 KLOC, the MRE was 5.1%.
The estimation in the planning phase had a gap of 27% fromthe actual value, suggesting that the proposed approach yieldeda closer estimation of the actual results. In the planning phase,the factors contributing to the increased costs of incremental
modernization were unrecognized, and the development size ofthe shell API was excluded from the estimation. Therefore, byadding the development size of the shell API to the estimation
during the planning phase, we obtained an estimation closer
to the actual value.

It is impossible to determine whether the beneﬁt estimation is valid because the maintenance effort has remained
the same or increased since modernization.
The lesson learned from these results is that incremental
modernization only occasionally improves maintainability. The
frequency of change requests, bugs, and future modiﬁcations
should be considered when extracting programs. Although the
beneﬁts of performing incremental modernization have been reported, these beneﬁts are only fully realized if the approAnswer to RQ1
priate functionality is extracted. Therefore, an approach toquantitatively estimate the effect of improved maintainability
Cost estimation can provide more valid estimates because
in modernization is still needed in the future. In addition,
they have fewer errors with actual values than estimatesusing the proposed estimation formula, the estimated result
made at the planning stage.
 of maintenance effort after restructuring was always smaller
than the maintenance effort before. Therefore, it is necessary to
The lesson learned from these results is that incremental
study a model to estimate the deterioration in maintainability.
modernization, including shell API development size in theestimate using dependency analysis, produces an estimated
C. Threats to Validity
result close to the actual development size. The development
In the cost-beneﬁt calculation model, this analysis estimatedsize of the shell API accounts for a large percentage of thethe lines of code to implement per dependency from theentire system development size. Therefore, if the development
average number of lines per program. By conducting proof
size of the shell API is excluded from the estimate, the effort
of concept (PoC), it is possible to determine the lines ofrequired exceeds the estimate. Before the analysis, we werecode to implement each dependency more precisely. However,concerned that MRE would be little changed or be worse thanconducting PoC during the planning phase is challenging. As
that in the planning phase. However, it improved by 22.4%,
shown in Table VI, even with the estimation approach usedthat is, from 27.5% to 5.1%. This indicates that the proposedin this analysis, the MRE is smaller than the estimates at theapproach is reasonable.

447

TABLE VII
B ENEFIT ESTIMATION : ACTUAL AND ESTIMATED DEVELOPMENT PERFORMANCE

MeanMedianCode to be tested IT1 [KLOC]
Code to be tested IT2 [KLOC]
Productivity [KLOC/person-month]
Code to be tested IT1 [KLOC]
Code to be tested IT2 [KLOC]
Productivity [KLOC/person-month]

Actual Value(After Modernization)
21.24
23.28
0.19
13.53
21.49
0.19

Estimated Value

MRE

19.79
20.91
0.32
12.86
15.46
0.27

7%
10%
68%
5%
28%
42%

[5]. Their ﬁndings enabled estimations of modiﬁcation work
costs for each type of dependency. However, their method did
not consider system extraction, which distinguishes it from thecost estimation in this study.
Xiao et al. investigated the correlation between architectural
technical debt and maintenance development costs [8]. Theirﬁndings enable the estimation of increased maintenance costs
owing to existing technical debt. Nevertheless, they did not
estimate the reduction in maintenance effort owing to system
extractions, setting their method apart from the cost estimationin this study.
Rebêl et al. explored the relationship between design stability and maintenance tasks by contrasting aspect-oriented extractions with object-oriented designs [6]. Their work allowed
for the estimation of maintenance cost reductions owing to
aspect-oriented extractions. However, their focus on aspectoriented extractions instead of incremental modernization distinguishes their beneﬁt estimation from the beneﬁt estimationin this study.
Kobayashi et al. proposed a metric to quantify the propagation model of changes to improve system failure prediction
accuracy [12]. They used dependency graphs to judge thechange impact, similar to the proposed method. However,their primary focus on enhancing failure prediction accuracy,
rather than estimating the reduction in development effort,
distinguishes their approach from the beneﬁt estimation in thisstudy.

planning phase, indicating that the estimation accuracy using
the average size per program is acceptable.
The beneﬁt estimation has no baseline for setting a thresholdfor the MRE. Therefore, no upper or lower limit of the
MRE can be considered to indicate the accuracy of the effectestimation model. Therefore, determining whether the beneﬁt
estimation is accurate is challenging.
Moreover, other metrics exist for evaluating the accuracyof the estimates apart from the MRE. In this study, we
answered the RQ based on the MRE from the actual values.
However, different results could be obtained if other metrics
were adopted.
In this analysis, we analyzed just a single case. All that
became established was that the estimated cost of this project
was close to the actual cost. From this, we determined thatthe cost estimate was valid. However, increasing the numberof systems under analysis and conducting cross-validation toenhance the reliability of the analysis is necessary.
Additionally, as seen in Table V, the change rate for the code
to be tested in IT2 (median) is positive, and productivity (bothmean and median values) is negative. This implies that thesituation worsened after incremental modernization. Hence,
despite the expected reduction in test scope and maintenanceeffort, we are concerned that expectations will fail after theincremental modernization. This suggests the possibility ofvalidating from the premises.
VII. R ELATED W ORK
Leitch et al. proposed a method for estimating the costbeneﬁt of refactoring using dependency analysis [4]. Theirapproach calculates the possible maintenance cost savings
from refactoring based on control and data dependencies anduses COCOMO II [11] to estimate the implementation cost of
refactoring. However, Leitch et al. focused only on evaluating
the cost-beneﬁt of refactoring within a system, differentiating
it from incremental modernization.
Cai et al. developed a framework for evaluating the costbeneﬁt of refactoring based on the source code, architectural
information, and version histories [7]. Although their framework shares similarities with the proposed approach, they
did not detail the exact estimation techniques. Their effectestimation also diverges from the proposed approach because it
does not focus on the change impact and relies on correlations
with past refactoring tasks.
Cui et al. investigated the correlation between methods ofmodifying dependencies and the costs of these modiﬁcations

VIII. C ONCLUSION
Adopting an incremental modernization strategy can reduce
the risk of software system modernization failures. However,modernization costs tend to increase compared to the estimatedcosts in the planning phase. Furthermore, quantifying the
beneﬁts of incremental modernization is challenging, leading
to delayed decisions to invest in modernization.
In this study, for systems in which incremental modernization was conducted, we applied a cost-beneﬁt estimation
approach using dependency analysis and compared the gap
with the actual value. As a result of this analysis, we conﬁrmedthat the cost estimates are valid estimates, but we cannot judge
whether the beneﬁt estimates are valid.
Future works include the following three points:• Improving the reliability of analysis results: In thisanalysis, the analysis has been conducted on just a single
case. Increasing the number of systems under analysis

448and conducting cross-validation to enhance the reliabilityof the analysis is future work.
• Expansion of dependency relationships: In this analysis, we analyzed only the call dependencies, but otherdependencies exist (e.g., access to databases). Expanding
the dependency types and analyzing whether the accuracyof the estimated improves is future work.
• Application to other languages: In this analysis, estimations were conducted targeting the COBOL language. In
this analysis, we performed our estimation on COBOL
programs. We are applying the proposed approach tolanguages other than COBOL and analyzing whether
accurate estimation results can be obtained in future work.

[15] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings,
S. Sinha, S. A. Spoon, and A. Gujarathi, “Regression test selection
for java software,” in Proc. of OOPSLA. Association for Computing
Machinery, 2001, p. 312–326.
[16] A. Ko, H. H. Aung, and B. Myers, “Eliciting design requirements formaintenance-oriented ides: a detailed study of corrective and perfective
maintenance tasks,” in Proc. of ICSE, 2005, pp. 126–135.
[17] L. Briand, Y. Labiche, and G. Soccar, “Automating impact analysis andregression test selection based on uml designs,” in Proc. of ICSM, 2002,
pp. 252–261.

ACKNOWLEDGMENT
This study beneﬁted greatly from discussions with Katsuro
Inoue (Nanzan University, Japan), Makoto Matsushita (Osaka
University, Japan), and Eiju Ueda (NTT DATA Group, Japan).
R EFERENCES[1] J. Bisbal, D. Lawless, R. Richardson, D. O’Sullivan, B. Wu, J. B.
Grimson, and V. P. Wade, “A survey of research into legacy system
migration,” in Proc. of the International Conference on Informatics andAnalytics, 2007.
[2] M. L. Brodie and M. Stonebraker, “Darwin: On the incremental migration of legacy information systems,” GTE Laboratories, Inc, Tech. Rep.,
1993.
[3] S. Comella-Dorda, G. Lewis, P. Place, D. Plakosh, and R. Seacord,“Incremental modernization of legacy systems,” Carnegie Mellon University, Software Engineering Institute’s Digital Library, Tech. Rep.,
2001.
[4] R. Leitch and E. Stroulia, “Assessing the maintainability beneﬁts ofdesign restructuring using dependency analysis,” in Proc. of METRICS,
2003, p. 309.
[5] D. Cui, L. Fan, S. Chen, Y. Cai, Q. Zheng, Y. Liu, and T. Liu, “Towards
characterizing bug ﬁxes through dependency-level changes in apachejava open source projects,” Science China Information Sciences, vol. 65,
2022.
[6] H. Rebêl, R. M. F. Lima, U. Kulesza, M. Ribeiro, Y. Cai, R. Coelho,
C. Sant’Anna, and A. Mota, “Quantifying the effects of aspectual
decompositions on design by contract modularization: a maintenancestudy,” Int. J. Softw. Eng. Knowl. Eng., vol. 23, no. 7, pp. 913–942,
2013.
[7] Y. Cai, R. Kazman, C. V. Silva, L. Xiao, and H.-M. Chen, “Chapter 6- a decision-support system approach to economics-driven modularity
evaluation,” in Economics-Driven Software Architecture.
MorganKaufmann, 2014, pp. 105–128.
[8] L. Xiao, Y. Cai, R. Kazman, R. Mo, and Q. Feng, “Detecting the locations and predicting the maintenance costs of compound architectural
debts,” IEEE Trans. Softw. Eng., vol. 48, no. 9, pp. 3686–3715, 2022.
[9] H. Michael Ayas, P. Leitner, and R. Hebig, “An empirical study of thesystemic and technical migration towards microservices,” Empir. Softw.
Eng., vol. 28, no. 85, 2023.
[10] A. Arifoglu, “A methodology for software cost estimation,” SIGSOFT
Softw. Eng. Notes, vol. 18, no. 2, p. 96–105, 1993.
[11] B. W. Boehm, C. Abts, A. W. Brown, S. Chulani, B. K. Clark,
E. Horowitz, R. Madachy, D. J. Reifer, and B. Steece, Software Cost
Estimation with COCOMO II. Prentice Hall, 2000.
[12] K. Kobayashi, A. Matsuo, K. Inoue, Y. Hayase, M. Kamimura, andT. Yoshino, “Impactscale: Quantifying change impact to predict faults
in large software systems,” in Proc. of ICSM, 2011, pp. 43–52.
[13] T. A. Corbi, “Program understanding: Challenge for the 1990s,” IBM
Syst.J., vol. 28, no. 2, pp. 294–306, 1989.
[14] C. L. McClure, The three Rs of software automation : re-engineering,
repository, reusability. Prentice Hall, 1992.

449