Analysis of Information System Operation Cost
Based on Working Time and Unit Cost
Masateru Tsunoda

Akito Monden

Nara Institute of Science and Technology
Nara, Japan
Department of Informatics, Kindai University
Osaka, Japantsunoda@info.kindai.ac.jp

Graduate School of Natural Science and Technology
Okayama University
Okayama, Japanmonden@okayama-u.ac.jp

Kenichi Matsumoto

Sawako Ohiwa, Tomoki Oshino

Graduate School of Information Science
Nara Institute of Science and Technology
Nara, Japan
matumoto@is.naist.jp

Economic Research InstituteEconomic Research Association
Tokyo, Japan{er421, er352}@zai-keicho.or.jp

Abstract—Recently, information system operation becomes
more important because of increasing size of information systemand outsourcing the system operation. However, it is not easy forcustomers to judge the validity of the system operation cost. To
provide information which helps the judgment, we analyzed
factors which affect system operation cost. Working time ofsystem operation service provider has the strong relationship tothe cost. So, if customers know the working time, they estimate
the cost properly. However, it is difficult for customers to know
the working time generally. So, we assumed that customers
estimate unit cost and working time, to speculate total operation
cost roughly. To help the estimation, we analyzed factors affectedworking time and unit cost. The analysis results show thatworking time is settled based on the software size and thenumber of users, and the unit cost of the engineers increases
when network range of the system is wide.
Keywords—benchmarking; it operations; work efficiency;
regression analysis

I. INTRODUCTION
Recently, size of information system gets large, and theinformation system operation is often outsourced. Given thesituation, the information system operation becomes more
important. The information system consists of computers,
network and software. In the information system operation,
operators manage the computers and the network, e.g., they
replace updated software and recover the system when asystem failure occurs. To operate the system, standardizationof system operation process such as ITIL (Information
Technology Infrastructure Library) [1] and ISO20000 get more
attention.
Cost is one of the important factors when the system
operation is outsourced. However, it is not easy for thecustomers (system users) to judge whether the operation cost isvalid or not. This is because generally, customers do not know
the operation cost in other customers’ companies, and therefore

they cannot compare the cost with others to judge the validityof the cost. The goal of our study is for customers to support
the judgement of the validity of the cost.
As explained in the section 2, the relationship between theoperation cost and working time of operation service supplier(outsourced operators) was very strong. So, if the working
time of the supplier is known, operation cost can be estimated,
and the estimated cost can be used to judge the validity of theactual cost. However, it is not easy for the customers to know
the working time of the supplier. So, we focus on information
which the customers can know (e.g., size of software), toestimate the working time. In the analysis, first, we clarified
the factors which affect the working time. Then, data wasstratified based on the factors, to show the benchmark ofworking time. Using the benchmark, customers estimate theworking time roughly.
To estimate the operation cost, the estimated working time
only is insufficient. The unit cost of outsourced operators is
also needed. That is, operation cost can be estimated, by
estimated working time multiplied by the estimated unit cost.
To estimate the unit cost roughly, we clarified factors which
affect the unit cost, and showed the benchmark of the unit cost,
stratifying the dataset based on the factors.
Major contribution of the study is not benchmarking work
efficiency of system operation, but benchmarking the system
operation cost. Although the former is useful for the service
suppliers, it is not utilized for the customers to judge thevalidity of the cost. In contrast, the latter is useful for thejudgement. As long as we know, there is no study whichanalyzed system operation cost for the customers. In ourprevious study [5], we analyzed factors which affects thesystem operation cost. In the study, we regarded that the cost is
derived by the number of operators, the number of computers,
and the unit cost of outsourced operators. This is because in thedataset used in [5], the working time was not recorded in mostdata points. Instead of the time, we used the number ofOperation cost

operators and the number of computers. The analysis of thispaper is more proper than the previous study, since we used theworking time directly.
The dataset used in the study was collected in 2011 and2013 by Economic Research Association. It includes 179 data
points collected from both customers and operation service
suppliers. The size of the systems recorded in the dataset isvarious (i.e., both small systems’ data and large systems’ data
are included). Therefore, the analysis results derived from thedataset would have generality to some extent.
II. RELATIONSHIP BETWEEN OPERATION COST AND WORKINGTIME
In the analysis, we clarified factors which decide thesystem operation cost. The system operation cost means thetotal cost required to the operation in a year. Note that the costmeans the contract price, and not the real cost in the service
supplier. Most of the system operation cost is considered to be
based on the labor cost of the operators. In other words, thecost is assumed to relate strongly to the working time of theoperators of the service supplier. Note that the system isoperated by not only the service supplier but also the customer.
So, working time of the service supplier should be
distinguished from the working time of the customer, whenanalyzing the operation cost.
We analyzed the relationship between the operation cost
and the working time of the service supplier, using theregression analysis. In the regression analysis, the dependent
variable was the operation cost, and the independent variable
was the working time of the service supplier. Before building
the model, log-transformation was applied to the variables. As
a result, the built model of R2 was 0.91. This means theoperation cost mostly depends on the labor cost. In other words,
if the customers know the working time of the service supplier,the operation cost can be estimated.
As explained above, the cost is the contract price, and theprice includes the profit and expenses of the service supplier.
They are one of the factors which affect the operation cost.
Also, the unit cost of the operators of the service supplier is
one of the factors. However, their influence is not very strong,since the model of R2 was very large (0.91).
Fig. 1 shows the relationship between the operation cost
and the working time of the service supplier. To avoid abusing
the result, we do not show the operation cost on the verticalaxis. Although there is two outliers at the upper left of the
figure (their cost is higher than others, considering the workingtime), the relationship between the cost and the time was strong
in other data points. The result is not always natural in software
engineering field. For example, in the software maintenanceactivity, the relationship between the maintenance cost and theworking time was relatively weak [6].
III. FACTORS AFFECTING WORKING TIME
In the followings, the sum of working time of thecustomers and the service supplier in a year is called as total
working time. As explained in section II, the information
system is operated by the customers, not only the service

Working time of service supplierFig. 1

Relationship between the operation cost and service
supplier’s working time per year.

suppliers. The ratio of the operators of the customers to theservice suppliers was not same in each data point. So, using
working time of the service supplier is not appropriate in theanalysis. Instead of it, we used the total working time in thefollowing analysis.
To benchmark the total working time, we defined work
efficiency as the number of programs divided by the total
working time. In the next subsection, we explain why the
number of programs is used in the definition. In the following
analysis, we show relationship between the work efficiency
and other factors using box plots. To estimate the working time
roughly, customers multiply the work efficiency shown in theboxplot by the number of programs.
We used multiple regression analysis, to analyze factors
affecting the total working time. Before building the model, allvariables of the ratio scale were log-transformed. In the model,
the dependent variable was the total working time.
A. Relationship to System Size
As explained in section I, main activities of the system
operation is to manage computers and network on the system.
So, the system size such as software size and the number ofcomputers are considered to be main factor of the total working
time. As software size, we used the number of programs,
instead of the functional size of software on the system. Since
in the number of programs, the number of missing values was
relatively small. The dataset has various variables such as thenumber of Windows servers and the number of mainframes.
Based on the preliminarily analysis, we used the number ofWindows servers as hardware size of the system.
When the number of the users increases, answering time tothe users may also increases. So, we also analyzed the
maximum number of users on the system. Using the multiple
linear regression analysis, we analyzed the influence of thefactors to the total working time. In the model, independent
variables were the number of programs, the number ofWindows servers, and the maximum number of the users. We
applied variable selection to build the regression model.
Table I shows standardized partial regression coefficients inthe built model. The number of Windows servers was removed

TABLE I.

REGRESSION MODEL USING SYSTEM SIZE

Variable

p-value
0.0%
0.0%

Maximum number of users
Number of programs
Process Standardization

Relationship between the maximum number of users andservice supplier’s working time.

p-value
0.0%
0.0%
9.3%

Process standardization

Fig. 4

Relationship between process standardization and work
efficiency.

Working time of service supplier

B. Relationship to Process Standardization
In the system operation, the standard procedure of theoperation is sometimes defined, to enhance the efficiency ofthe operation and to suppress system trouble. We assumed thatwhen the operation process is standardized, the work efficiency
is improved, and it affects the total working time.

Number of programs

Fig. 3

Std. partial reg.
coefficient.
0.47
0.47
-0.16

Work Efficiency
Maximum number of users

Fig. 2REGRESSION MODEL USING PROCESS STANDARDIZATION

Variable

Working time of service supplier

Maximum number of users
Number of programs

Std. partial reg.
coefficient.
0.41
0.49

TABLE II.

Relationship between the number of programs and service
supplier’s working time.

from the model by the variable selection. This means thenumber of Windows servers is not indispensable to estimate
the total working time. The partial regression coefficient of thenumber of programs was larger than the maximum number ofthe users. So, we used the number of programs to define thework efficiency. Note that the latter was not small, and hence
the influence of the maximum number of the users was notignorable. Hence, we also used the maximum number of theusers to build regression models in the followings.
Adjusted R2 was 0.53, and this means that the total working
time is explained by the number of programs and the maximum
number of the users, since Adjusted R2 was larger than 0.5. Fig.
2 and Fig. 3 show the relationships between total working time
and the variables. In the figures, data points are scattered, andthese suggest the estimation is difficult, using the number ofprograms or the maximum number of the users only. So, in thefollowing analysis, we focused on other factors which wereconsidered to affect the total working time.

We applied multiple regression analysis, to clarify the
relationship between process standardization and the total
working time. In the analysis, independent variables were thenumber of programs, the maximum number of the users, andthe extent of the process standardization. The extent is signified
by three levels. Level U1 means process is standardized, levelU2 means process standardization is work in progress, andlevel U3 means process is not standardized. We made adummy variable which denotes the process standardization islevel U1 or not, because the distribution of level U1 wasdifferent from other levels (see Fig. 4).
Standardized partial regression coefficients of the model
are shown in Table II. Adjusted R2 of the model was 0.54, andit was almost same as the model without the process
standardization built in section III.A, although the regression
coefficients of the process standardization was not very small.
P-value of the partial regression coefficient of the process
standardization was smaller than 10%. Therefore, the process
standardization has the relationship to the total working time.
The regression coefficient of the process standardization was
negative. So, when the process is standardized (i.e., level U1),
the total working time decreases, and work efficiency becomeshigh.
Fig. 4 shows the relationship between the work efficiency
and the process standardization. In the figure, there were many
data points whose work efficiency were high in level U1,
although the variance of level U1 was large. Based on theboxplot and the regression analysis, the process standardizationis regarded to have the relationships to the work efficiency.

TABLE III.

REGRESSION MODEL USING SOCIAL IMPACT

Variable

To analyze the influence of the social impact to the total
working time, we applied multiple regression analysis. In theanalysis, independent variables were the number of programs,
the maximum number of the users, and the extent of the social
impact of the system. The social impact is denoted by threelevels. Level C1 denotes the social impact of the system isalmost nothing, level C2 denotes the system has the social
impact, but it is limited, and level C3 denotes the social impact
is very large. We made a dummy variable which denotes thesocial impact is level C1 or not, because level C3, the numberof data points was few.
Table III shows standardized partial regression coefficientsof the model. Adjusted R2 of the model was 0.54, and it wasalmost same as the model built in section III.A. However, theregression coefficients of the process standardization was notvery small. P-value of the coefficient of the social impact was
relatively small, although it was larger than 10%. Therefore, itis probable that the social impact has the relationship to thetotal working time. The regression coefficient of the social
impact was negative. Hence, when the social impact is small(i.e., level C1), the total working time decreases, and the work
efficiency becomes higher.
The relationship between the work efficiency and the social
impact is shown in Fig. 5. In the level C3, the number of data
points was two. So, we disregard level C3. Although themedian of level C1 and C2 is almost same, variance of C1 waslarge, and in level C2, there were few data points whose work
efficiency were high. That is, when the social impact is not
small, work efficiency tends to be lower.
D. Analysis of Working Time Considering Multiple Factors
If the process standardization and the social impact has therelationship, one strongly relates to the work efficiency, but theother may not relate very much. To identify the major factorsof the total working time, we built a multiple regression model.
In the model, independent variables were the number ofprograms, the maximum number of the users, the process
standardization, and the social impact.
Variable selection was applied to build the model. As aresult, the social impact was removed from the model. Thismeans the total working time is explained by the number ofprograms, the maximum number of the users, the process
standardization. So, the relationship between the total working
time and the social impact was weaker than others. Note that
we do not show the built model because it is same as the model
built in section III.B.

Maximum number of users
Number of programs
Social impact

p-value
0.0%
0.0%
10.6%

Work Efficiency

C. Relationship to Social Impact
Some information systems are used as the infrastructure of
the society. If some trouble occurs on the system, many social
activities are affected by the trouble. We call the influence as
the social impact of the system. When an information system
has high social impact, the system should be operated carefully.
It may need more working time than the system which has
small social impact.

Std. partial reg.
coefficient.
0.37
0.44
-0.16

Social Impact

Fig. 5

Relationship between social impact and work efficiency.

IV. FACTORS AFFECTING UNIT COST
To estimate the operation cost, the estimated working time
only is insufficient. The unit cost of outsourced operators is
also needed, as explained in section I. We defined the unit costof operators as the total cost divided by the working time ofservice supplier. If there is a factor which affects the operation
cost, in addition to the working time, the factor is considered toaffect the unit cost. So, we analyzed factors affecting theoperation cost, using multiple regression analysis. In the model,
the dependent variable was the operation cost, and all variablesof the ratio scale were log-transformed.
A. Relationship to Network range
Information system sometimes connects to other systems
using the network. When the network is complex, system
operators who have high network skills are needed to operate
the system, and it may increase the unit cost of operators. So,
we analyzed the relationship between the network range andthe operation cost, i.e., the influence of the network range tothe unit cost. The network range is signified by five levels.
Level N1 means the network is between customers and otherorganizations, level N2 means it is between customers andother branch offices, level N3 means it is within a branch
office, level N4 means it is within a department, and level N5means process the system does not connect to other system.
In the multiple regression analysis, independent variables
were the working time of service supplier and the network
range. We treated the network range as ordinal scale,considering the relationship between the unit cost and the
network range (see Fig. 6). We replaces level N5 as level N3,
since the number of the data point was one, and there is no data
point whose network range was level N4.
Standardized partial regression coefficients of the model
are shown in Table IV. Adjusted R2 of the model was 0.92. Pvalue of the partial regression coefficient of the processTABLE IV.

REGRESSION MODEL USING NETWORK RANGE

Variable

p-value
0.0%
2.0%

TABLE V.

REGRESSION MODEL USING CONTRACT TYPE

Variable
Working time of service supplier
Contract type K1Contract type K2Contract type K3

Std. partial reg.
coefficient.
0.97
-0.07
-0.01
-0.12

p-value
0.0%
19.1%
87.9%
4.5%

Unit cost

Unit cost

Working time of service supplier
Network range

Std. partial reg.
coefficient.
0.95
-0.12

Network Range

Fig. 6

Relationship between network range and unit cost.

standardization was smaller than 5%. Therefore, the network
range has the relationship to the operation cost. The regression
coefficient of the process standardization was negative. So,
when the network range is narrow (e.g., level N3), theoperation cost decreases, and the unit cost becomes low. Fig. 6
shows the relationship between the network range and the unit
cost. As explained above, the number of the data point of levelN1 was one, and there is no data point whose network range
was level N4. Currency unit of the vertical axis is Japanese yen.
The figure shows also when the network range is narrower, theunit cost is lower.
B. Relationship to Contract Types
There are some contract types of the system operation.
Contract type means the working type of the operators. The
contract type has four types. Type K1 is dispatched workers,
type K2 is quasi-mandate workers, K3 is contract workers, typeK4 is mixing quasi-mandate and contract workers, and type K5
is others. When the contract type is different, the unit cost of
the operators may also different.
To clarify the relationship between the contract type andthe operation cost, we applied multiple regression analysis. In
the analysis, independent variables were the working time ofservice supplier and the contract type. We made dummy
variables which denote the contract type. The dummy variablesof type K5 was not used, because the number of data points ofthem was one. Note that we did not make the dummy variablesof type K4, since it can be denoted by the variables of type K1,
K2 and K3.
Table V shows standardized partial regression coefficientsof the model. Adjusted R2 of the model was 0.91. P-value ofthe coefficient of the contract type (type K3) was smaller than
5%. Therefore, the contract type of K3 has the relationship to
the operation cost. The regression coefficient of the value wasnegative. Hence, when the contract type is K3, the operation
cost decreases, and the unit cost also becomes low. The
relationship between the operation cost and the contract type isContract typeFig. 7

Relationship between contract type and unit cost.

shown in Fig. 7. The number of data points of type K1 and K5
was few. So, we disregard type K1 and K5. The figure showsthat type K3 is the lowest unit cost.
C. Relationship to SLA
When the system operation is outsourced, SLA (Service
Level Agreement) is sometimes made. SLA assures service
level of the operation such as system recovery time in advance.
While it ensures quality of the system operation, it may alsoenhance the unit cost of the operators. So, we applied multiple
regression analysis to clarify the influence of SLA to theoperation cost (i.e., the unit cost). The independent variables
were the working time of service supplier and SLA status. SLA
status denotes SLA of each data point. Status S1 means SLA is
made (goals are guaranteed), status S2 means SLA is made(goals are not guaranteed), and status S2 means SLA is not
made. We made dummy variables which denote status S1 andS2.
Standardized partial regression coefficients of the model
are shown in Table VI. Adjusted R2 of the model was 0.91. Pvalue of the coefficient of the SLA status (status S1) was
relatively small, although it was larger than 10%. The
regression coefficient was positive. This means when the SLA
is made (i.e., status S1), the operation cost increases, and theunit cost becomes high. Fig. 8 shows the relationship between
the SLA status and the unit cost. In the figure, although themedian of status S3 was lower than others, the distribution was
not very different. When SLA is not made, the unit cost may below. However, the relationship between SLA status and theunit cost was not explicit.
D. Analysis of Operation Cost Considering Multiple Factors
There may be relationships between the network range, thecontract type, and SLA status. If there is the relationships, afactor strongly affects the operation cost, but the others mayTABLE VI.

REGRESSION MODEL USING SLA
Std. partial reg.
coefficient.
0.93
-0.09

Variable
Working time of service supplier
SLA statusp-value
0%
10%

To our knowledge, there is no study which treats
quantitative IT operations staff (cost) estimation. ITIL explains
cost estimation based on cost types such as equipment cost ororganization (staff) cost [1]. However, it does not tell how toestimate the number of staff quantitatively.
VI. CONCLUSIONS

Unit costTo support for customers to validate the operation cost of
information system. Using a cross-company dataset, weclarified the factors which affect the working time, and the unit
cost of the operators. The dataset is stratified based on thefactors, to estimate the working time and the unit cost roughly.
 The operation cost is settled based on the working timeof operators
 The process standardization relates to the working timeof operators.
 The network range and the contract type relates to theunit cost of operators.

SLA status

Fig. 8TABLE VII.

Relationship between SLA and unit cost.

REGRESSION MODEL USING NEWWORK RANGE AND CONTRACT
TYPE

Variable
Working time of service supplier
Network range
Contract type K3

Std. partial reg.
coefficient.
0.97
-0.10
-0.09

p-value
0.0%
4.6%
5.6%

not affect it very much. To clarify the major factors of theoperation cost, we applied multiple regression analysis. In theanalysis, independent variables were the working time ofservice supplier, the network range, the contract type, and SLA
status.
Table VII shows standardized partial regression
coefficients of the model. Variable selection was applied tobuild the model, and SLA status was removed from the model.
So, the relationship between the operation cost and SLA status
was weaker than others. On the contrary, the network range
and the contract type is not ignorable to estimate the unit cost.

Main contribution of the study is to benchmark the system
operation cost, and not to benchmark the work efficiency of thesystem operation. The benchmark of the work efficiency isuseful for the service suppliers, but it is not useful forcustomers in most cases. In contrast, the benchmark of the cost
is useful for customers. As long as we know, there is no study
which analyzed the system operation cost, considering thecustomers. One of our future works is to analyze whether thesimilar tendency is observed or not in the emerging Web-based
or cloud-based systems.
ACKNOWLEDGMENT
This research was partially supported by the Japan Ministryof Education, Science, Sports, and Culture [Grant-in-Aid for
Scientific Research (C) (No. 25330090)]
REFERENCES[1][2]

V. RELATED WORK
While there are some studies treating IT operations, theymainly focus on the process of IT operations. For instance,
Brooks [2] showed metrics for evaluation of IT operations
process, but the metrics are not effective for IT operations costestimation. Pollard et al. [4] performed case study of theUnited States and Australian companies’ IT operations, andidentified some critical success factors (CSF) for successful
ITIL implementations. The research also focused on IT
operations process, and did not treat cost estimation.
Mannino et al. [3] analyzed efficiency of data warehouse
operations by data envelopment analysis (DEA). Although data
warehouse operations is one part of IT operations, their resultscannot apply to staff or cost estimation.

[3]

[4]

[5]

[6]

J. Bon, ed.: Foundations of IT Service Management: based on ITIL, Van
Haren Publishing, 2005.
P. Brooks, “Metrics for IT Service Management,” Van Haren Publishing,
2006.
M. Mannino, S. Hong, and I. Choi, “Efficiency evaluation of data
warehouse operations, Decision Support Systems,” vol.44, no.4, pp.883898, 2008.
C. Pollard, and A. Cater-Steel, “Justifications, Strategies, and Critical
Success Factors in Successful ITIL Implementations in U.S. andAustralian Companies: An Exploratory Study,” Information System
Management, vol.26, no.2, pp.164-175, 2009.
M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino,“An Empirical Analysis of Information Technology Operations Cost,”
Proc. of International Workshop on Software Measurement(IWSM/Metrikon/Mensura), pp.571-585, 2010.
M. Tsunoda, A. Monden, K. Matsumoto, S. Ohiwa, and T. Oshino,“Benchmarking Software Maintenance Based on Working Time,” In
Proc. of Applied Computing and Information Technology (ACIT),
pp.20-27, 2015.