An Empirical Analysis of Information Technology Operations Cost

An Empirical Analysis of Information Technology Operations Cost
Masateru Tsunoda1, Akito Monden1, Ken-ichi Matsumoto1, Akihiko Takahashi2,
and Tomoki Oshino2
1

Graduate School of Information Science, Nara Institute of Science and Technology{masate-t, akito-m, matumoto}@is.naist.jp
2

Economic Research Institute, Economic Research Association{er102, er352}@zai-keicho.or.jp

Abstract:
When outsourcing IT (Information Technology) operations, it is important to estimate operations cost accurately. To help IT operations cost estimation, we analyzed
cross-organization data which includes 73 IT operation cases collected from Japanese companies/organizations. Operations cost is roughly decided by the number ofoperations staff and unit cost of staff. So we clarified attributes which have relationships to the number of staff or unit cost of staff to help analogous estimating ofthem. The result showed that the number of computer terminals and network range
are major factors of the number of staff. Also, the number of UNIX server andprocess standardization are major factors of unit of staff. In addition, we tried tobuild estimation models for the number of staff and total cost of operations. The
number of staff estimation model is used when IT operations service providers estimate operations cost, and the operations cost estimation model is assumed to be
used by IT operations service purchasers. In the experiment, the number of staff estimation model showed moderate accuracy, but the total operations cost estimation
model showed low accuracy.
Keywords
Estimation, staff, cost, cross organization data, linear regression

1

Introduction

Recently, IT (Information Technology) operations (or software system operations)are often outsourced. After software is released, IT operations are necessary tosustain availability of the software system. Activities of IT operations consist of
manipulations such as daily backup, monitoring software and hardware status,
supporting software and hardware recovery when incidents occurred, and so on.
When outsourcing IT operations, it is important to estimate operations cost accurately. For IT operations service providers, accurate cost estimation is the basis
for generating appropriate profits. For IT operations service purchasers (systemusers), accurate cost estimation by them is useful to validate operations cost offered by IT operations service providers.

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino
Operations cost is roughly decided by the number of operations staff and unit costof the staff (per hour). To help analogous estimating of the number of staff andunit cost of staff, we clarify attributes which have relationships to them. Also, webuilt linear regression models to estimate the number of staff and total cost of operations (per year). The number of staff estimation model is used when IT operations service providers estimate operations cost. By the model, operations cost
can be decided because IT operations service providers know unit cost of staff.
The operations cost estimation model is assumed to be used by IT operations service purchasers.
In the analysis, we used 73 cases of cross-organization data collected from Japanese companies/organizations. Analysis results using cross-organization data is
expected to have more generality than using single-organization data.
In what follows, Section 2 explains dataset used in the analysis. Section 3
describes procedure of the analysis. Section 4 shows results of the analysis anddiscussion of them. Section 5 introduces related works. In the end, Section 6
concludes the paper with a summary.

2

DatasetDataset includes 108 IT operations cases which were collected from 107 Japanesecompanies/organizations in 2007 by Economic Research Association [6]. In thedataset, the scope of IT operations is defined as operation process of ISO/IEC
12207 [8], and operations staff means people who engage in both service supportand service delivery defined in ITIL (Information Technology Infrastructure Library) version 2 [3]. Each case is the most typical IT operations case in a company/organization. Although the cases were collected from both IT operations
service purchasers and IT operations service providers, we selected 73 cases collected from only the purchasers, because total cost of operations was not collectedfrom the providers. Note that process standardization, ITIL status, and SLA (Service License Agreement) status signify the status of each company/organization,
not each IT operations case (For example, process standardization indicates
whether the company’s process is standardised or not).
Attributes of the cases are as follows.
Numerical attributes
The number of {mainframes, Windows servers, UNIX servers, computer
terminals}: the number of equipments included in the system.

The number of users: the number of people using the system.

The number of program: the number of program of software on the system.
The attribute was used instead of software size (function point) in the analysis.

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost

Operation time: total system operation time per week.



Unit cost of staff: unit cost of providers’ operations staff per hour (Japanese
yen).



The number of staff: total number of staff of the IT operations service provider and the purchaser engaged in IT operations.



Total cost: total contract cost for IT operations per year (Japanese yen).

The percentage of {service request, incident, problem, change, release,configuration, service level, capacity, IT service continuity, availability}
management staff, the percentage of financial management staff for IT
services: assignment of operations staff for each management area. The classifications of management areas are based on ITIL version 2 [3].

Nominal attributes
Process standardization: denote operation process is standardized, standardization is work in progress, or operation process is not standardized.



ITIL status: denote ITIL is applied whole operations, ITIL is applied part ofoperations, adoption of ITIL is under review, or ITIL is not applied.



SLA status: denote SLA is applied all agreements, SLA is applied part ofagreements, adoption of SLA is under review, or SLA is not applied.

Business sector: business sector of the company/organization where IT operations are performed.

System service time: system service providing time represented by 5 days, 5days and 24 hours, 7 days, or 7 days and 24 hours.



Network range: denote the system is connected to other companies/organizations’ system, the system is connected to branch offices’ system,or the system is connected to local systems only.



Using {Microsoft SQL Server, Oracle, open source database}: The database is used or not on the system.

3

Analysis procedure

First, we focus on unit cost of staff and the number of staff, and analyze
relationships to other attributes, to help analogous estimation of unit cost of staff
and the number of staff. When analyzing relationships between them and othernumerical attributes, we use Spearman's rank correlation coefficient, to avoid
influences of outliers. When the analysis results are suspected to include spurious
relationships, we apply partial correlation to correct bias.

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino
We use adjusted variance explained (  2 ) of one-way ANOVA [17] whenanalyzing relationships between them and other nominal attributes. The value
range of adjusted variance explained is [0%, 100%], and a large value indicates
there is a strong relationship between attributes. The value of adjusted varianceexplained is calculated using the following equation.

SSB  (k  1) MSE(1)
SST  MSE
When the analysis results are suspected to include spurious relationships, weapply analysis of covariance (ANCOVA) to correct bias.
We draw boxplots when strong relationships are observed by adjusted varianceexplained, because it does not denote whether there is positive relationship or
negative relationship, unlike correlation coefficient. In boxplots, the bold line ineach box indicates the median value. Small circles indicate outliers, that is, values
that are more than 1.5 times larger than the 25%-75% range from the top of thebox edge. Stars indicate extreme outliers, whose values are more than 3.0 times
larger than this range.
Next, we build the number of staff estimation model and the operations costestimation model. The operations cost estimation model does not include unit costof staff and the number of staff as independent variable, because the model isassumed to be used by IT operations service purchasers, and they do not know
actual values of them.
To build the models, we use linear regression model and stepwise variableselection. Attributes which show some relationship to unit cost of staff or thenumber of staff are regarded as major factors of operations cost, and they are used
as independent variables for the operations cost estimation model. Also, attributes
which show some relationship to the number of staff are used as independent
variables for the number of staff estimation model. Note that stepwise variableselection with all attributes did not work well, and consequently low accuracy
models were made.
Before building the model, log transformation is applied to numerical attributes,
because distributions of numerical attributes are skewed. Also, nominal attributes
are transformed into dummy variables (A nominal attribute is transformed into
multiple variables whose variables are 0 or 1). Leave-one-out cross-validation isapplied when building the models.
As evaluation criteria of the models, we use average and median of AE (absolute
error), MRE (magnitude of relative error) [5], and MER (Magnitude of Error
Relative to the estimate) [10]. These evaluation criteria are calculated asfollowing equations (where X is actual value, and X̂ is estimated value):2 Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost
AE  X  X

MRE 
MER 

XX
X
XX
X

(2)
(3)
(4)

MRE and MER are criteria to evaluate overestimation and underestimation
respectively [7]. Using either of them is insufficient because even if the MRE of amodel is small, the model might overestimate if MER is much greater than MRE.

4

Results

4.1

Attributes related to the number of staff

Table 1 shows correlation coefficient of the number of staff and other (numerical)
attributes (Table 1 shows attributes whose p-value of correlation coefficient issmaller than 10%). Considering tasks of operations staff, it is reasonable that the
number of computer terminals has the highest correlation coefficient to thenumber of staff.
Also, as shown in Table 1, the number of computer terminals has high correlation
coefficient to other attributes, except for the number of mainframes. There is a
probability that only the number of computer terminals has actual relationship to
the number of staff, and other attributes have spurious relationships. So weapplied partial correlation to eliminate influence of the number of computer
terminals. Table 2 shows partial correlation of attributes included in Table 1. The
number of Windows servers, mainframes, and UNIX servers have moderate
relationships to the number of staff. So in addition to the number of computer
terminals, the number of server machines should be considered when the numberof staff is estimated. It is reasonable that server machine operations needs
additional staff.
Other observations are described below.


Although the number of users seems to have moderate relationship to thenumber of staff in Table 1, it has weak relationship to the number of staff
when applying partial correlation (Table 2). Hence, correlation of the numberof users and the number of staff shown in Table 1 is regarded as spurious correlation. A large number of users may not increase workload of operations
staff directly.

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino# of computer
terminals# of staff
Attributes
Correlation
coefficientp-value

Correlation
coefficientp-value# of computer terminals

0.69

0%

-

-

# of users

0.63

0%

0.89

0%

# of Windows servers

0.55

0%

0.80

0%

# of mainframes

0.47

2%

0.27

17%

% of financial management
staff for IT services

0.32

8%

0.52

0%

# of UNIX servers

0.29

9%

0.74

0%

Table 1: Correlation coefficient to the number of staff (p-value < 10%) andthe number of computer terminals

Percentages of operations staff assignment (That is, the percentage of service
request management staff and the percentage of other management staff) do
not have relationship to the number of staff. The result suggests that when estimating the number of staff, it is not necessary to consider operations staff
assignment, and modification of operation task balance would not need tochange the number of staff.

The percentage of financial management staff for IT service has moderate relationship to the number of computer terminals (Relationship to the number ofstaff is spurious correlation). It may because when operations cost is higher(That is, the number of computer terminals is larger), more financial management staff is required.



The number of program has low correlation coefficient to the number of staff(Correlation coefficients is 0.28). Roughly, the number of program denotes
software size. It is natural that the number of program (software size) does notaffect the number of the staff, because operations staff do not modify softwareon the system, and therefore workload of operations staff is not affected bysoftware size (Large size software does not always need many staff).



Operation time has low correlation coefficient to the number of staff (Correlation coefficients is 0.14). Although it seems plausible that long operation time

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost
needs shift work and it increases the number of staff, operation time does notaffect the number of staff. So basically, we do not have to care operation time
when estimating the number of staff.
Partial correlation
coefficient

Attributes

p-value# of Windows servers

0.44

1%

# of mainframes

0.44

4%

# of UNIX servers

0.44

1%

% of financial management
for IT service

-0.24

18%

# of users

-0.13

43%

Table 2: Partial correlation eliminating influence of the number of computer
terminals# of staff# of computer
terminals

Adjusted
variance
explained

p-value

Adjusted
variance
explained

p-value

Network range

35%

0%

0%

44%

ITIL status

28%

0%

7%

10%

SLA status

25%

0%

8%

8%

Using MS SQL Server

7%

5%

13%

0%

Attributes

Table 3: Adjusted variance explained to the number of staff (p-value < 10%)and the number of computer terminals

Table 3 shows adjusted variance explained (of nominal attributes) to the numberof staff (Table 3 shows attributes whose p-value of adjusted variance explained issmaller than 10%) and the number of computer terminals. Although adjusted

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshinovariance explained to the number of computer terminals is not large, their p-value
are almost less than 10%, except for network range. There is a probability that
only the number of computer terminals has actual relationship to the number ofstaff, and other attributes have spurious relationships.
We applied ANCOVA to eliminate influence of the number of computer terminals.
The result showed that only network range has statistically significant
relationship to the number of staff. As shown in Figure 1, when network range isbroader, more operations staff is needed. The reason would be that operation tasks
for network needs additional staff, separately from operations staff for computer
terminals.
ITIL status, SLA status, and using Microsoft SQL Server have some relationships tothe number of computer terminals (Relationships to the number of staff are
spurious). It may because when operations cost is higher (That is, the number ofcomputer terminals is larger), ITIL and SLA are needed. Microsoft SQL Server
seems to be used when the number of computer terminals is large.
From the results, we conclude that the number of staff has relationships to
network range, and the number of computer terminals, Windows servers,
mainframes, and UNIX servers.

Figure 1:

4.2

Relationship between the number of staff and network range

Attributes related to unit cost of staff

Table 4 shows correlation coefficient of unit cost of staff and other (numerical)
attributes. (Table 4 shows attributes whose p-value of correlation coefficient issmaller than 10%). In the analysis, we assumed that allocation of staff of the IT
operations service provider is almost same as the purchaser. Although there is a

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost
probability that all configuration management staff was the purchasers’ one forexample, almost in half cases, 80% of staff was the providers’ one, and therefore
we regarded the probability as low.
Observations are described below.


The number of UNIX server and the percentage of configuration management
staff have moderate relationships to unit cost of staff, and the results suggestthat unit cost of UNIX server operations staff and configuration management
staff are higher than other operations staff. The reason of higher price would
be that these staff are required certain skill level, and therefore their market
value is higher in Japan. Note that the results may be different from countries
due to supply and demand of the market.

The percentage of staff of financial management for IT services has positive
correlation to unit cost of staff. The result would not suggest that unit cost of
staff of financial management for IT services is high, but suggest that whenunit cost of staff is higher, financial management staff is needed.

The number of computer terminals has low correlation coefficient to unit costof staff (Correlation coefficients is -0.06). Hence, we do not have to care thenumber of computer terminals when estimating unit cost of staff.
Attributes

Correlation coefficientp-value# of UNIX server

0.53

1%

% of configuration management staff

0.50

2%

% of financial management staff for IT
service

0.37

9%Table 4:

Correlation coefficient to unit cost of staff (p-value < 10%)Attributes

Adjusted variance explained

p-value

Process standardization

41%

0%

Business sector

38%

1%

Using open source database

21%

1%

System service time

17%

8%

Table 5:Adjusted variance explained to unit cost of staff (p-value < 10%)

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino
Table 5 shows adjusted variance explained (of nominal attributes) to unit cost of
staff (Table 5 shows attributes whose p-value of adjusted variance explained issmaller than 10%). The relationships between unit cost of staff and otherattributes are shown in Figure 2-5. Descriptions of the observations are as follows.


As shown in Figure 2, when process is not standardized, unit cost of staff ishigher. The reason would be that operations staff who have higher skills areneeded when process is not standardized, and that makes unit cost of staff
higher.



Unit cost of staff is different by business sector as shown in Figure 3. Intensity of competition of IT operations service may be different from each business sector, and that would make the differences of unit cost of staff.

Figure 2:

Relationship between unit cost of staff and process standardization

Figure 3:

Relationship between unit cost of staff and business sector

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost

Figure 4:

Figure 5:

Relationship between unit cost of staff and using open source database

Relationship between unit cost of staff and system service timeWhen using open source database, unit cost of staff is higher as shown in
Figure 4. This would be that the staff are required certain skill level, andtherefore their market value is higher in Japan.



As shown in Figure 5, when system service time is 7 days and 24 hours, unit
cost of staff is highest (Note the number of cases of 5 days and 24 hours isonly two, so the cases are better to omit). The reason would be that 24 hours
operation needs shift work, and that makes unit cost of staff higher.

From the results, we conclude that the number of UNIX server, the percentage ofconfiguration management staff, process standardization, business sector, using

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino
open source database, and system service time have relationships to unit cost of
staff.
4.3

Staff and cost estimation models

We built the staff estimation model using the number of computer terminals, thenumber of Windows servers, the number of mainframes, the number of UNIX
servers, and network range (attributes which have relationships to the number ofstaff as shown in section 4.1). Accuracy of the model is shown in Table 6. The
model has moderate accuracy, and therefore the number of staff can be estimatedby system configurations such as the number of computer terminals.
The total cost estimation model was built using the number of computer terminals,
the number of Windows servers, the number of mainframes, the number of UNIX
servers, network range, the percentage of configuration management staff,
process standardization, business sector, using open source database, and systemservice time (attributes which have relationships to the number of staff as shownin section 4.1 or unit cost of staff as shown in section 4.2). Accuracy of the modelis shown in Table 6. Accuracy of the model is very low. This result suggests that itis difficult to estimate total cost by mathematical models using attributes shownin this paper.
From the results, we conclude that IT operations service providers can estimate
operations cost by mathematical models only, using estimated number of staff by
the models and actual unit cost of staff which they already know. That would help
enhancing reliability of cost estimation. On the contrary, it is difficult for IT
operations service purchasers to estimate operations cost by mathematical models
only. Operations cost estimation by the purchasers needs both mathematicalmodels for number of staff estimation and analogous estimation of unit cost of
staff. The analogous estimation is performed based on similarity of major factorsof unit cost shown in section 4.2.

Dependent variable

Mean
AE

Median MeanAE
MREMedian Mean
MRE
MERMedianMER

Number of staff

18.3

5.4

76%

57%

116%

47%

Total cost

242.4

62.5

221%

65%

172%

80%

Table 6:

Accuracies of staff estimation and cost estimation

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost
5

Related work

While there are some studies treating IT operations, they mainly focus on theprocess of IT operations. For instance, Brooks [4] showed metrics for evaluationof IT operations process, but the metrics are not effective for IT operations costestimation. Pollard et al. [12] performed case study of the United States andAustralian companies’ IT operations, and identified some critical success factors(CSF) for successful ITIL implementations. The research also focused on IT
operations process, and did not treat cost estimation.
Mannino et al. [11] analyzed efficiency of data warehouse operations by dataenvelopment analysis (DEA). Although data warehouse operations is one part of
IT operations, their results cannot apply to staff or cost estimation.
To our knowledge, there is no study which treats quantitative IT operations staff(cost) estimation. ITIL explains cost estimation based on cost types such as
equipment cost or organization (staff) cost [3]. However, it does not tell how toestimate the number of staff quantitatively. The concept of our research is similar
to software development effort (cost) estimation [2, 13, 14, 15, 16], or software
maintenance effort estimation researches [1, 9].

6Conclusion

To help estimating IT (Information Technology) operations cost, we analyzed
cross-organization dataset of IT operations, focusing on the number of operations
staff and unit cost of the staff, because IT operations cost is basically decided bythem. Our findings include the followings:
The number of staff has relationships to network range, and the number ofcomputer terminals, Windows servers, mainframes, and UNIX servers.



Unit cost of staff has relationships to the number of UNIX Servers, the percentage of configuration management staff, process standardization, businesssector, using open source database, and system service time.

The findings are useful to perform analogous estimation of the number of staff
and unit cost of staff.
Also, we tried to build the number of staff estimation model and the IT operations
cost estimation model. As a result, the number of staff model showed moderate
accuracy, but the IT operations cost model showed low accuracy. The result
suggests that although IT operations service providers can estimate operations
cost by mathematical models only, IT operations service purchasers should
combine both mathematical estimation and analogous estimation to speculate IT
operations cost.

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino
Our future work is collecting more data and analyzing it to enhance reliability of
the analysis results.

Acknowledgement
This work is being conducted as a part of the StagE project, The Development of
Next-Generation IT Infra-structure, and Grant-in-aid for Young Scientists (B),
22700034, 2010, supported by the Ministry of Education, Culture, Sports,
Science and Technology.
References
1. Y. Ahn, J. Suh, S. Kim, and H. Kim: The software maintenance project effort estimation model based on function points, Journal of Software Maintenance: Research andPractice, vol.15, no.2, pp.71-85, 2003.
2. B. Boehm: Software Engineering Economics, Prentice Hall, 1981.
3. J. Bon, ed.: Foundations of IT Service Management: based on ITIL, Van Haren Publishing, 2005.
4. P. Brooks: Metrics for IT Service Management, Van Haren Publishing, 2006.
5. S. Conte, H. Dunsmore, and V. Shen: Software Engineering, Metrics and Models,
Benjamin/Cummings, 1986.
6. Economic Research Association.
On-line < http://www.zai-keicho.or.jp/about/english.php>
7. T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit: A Simulation Study of theModel Evaluation Criterion MMRE, IEEE Transactions on Software Engineering,vol.29, no.11, pp.985-995, 2003.
8. ISO/IEC 12207:1995: Information technology - Software life cycle processes, International Organization for Standardization, 1995.
9. M. Jørgensen: Experience With the Accuracy of Software Maintenance Task Effort
Prediction Models, IEEE Transactions on Software Engineering, vol.21, no.8,
pp.674-681, 1995.
10. B. Kitchenham, S. MacDonell, L. Pickard, and M. Shepperd: What Accuracy Statistics Really Measure, In Proc. of IEE Software, vol.148, no.3, pp.81-85, 2001.
11. M. Mannino, S. N. Hong, and I. J. Choi: Efficiency evaluation of data warehouse operations, Decision Support Systems, Vol.44, no.4, pp.883-898, 2008.
12. C. Pollard, and A. Cater-Steel: Justifications, Strategies, and Critical Success Factors
in Successful ITIL Implementations in U.S. and Australian Companies: An Exploratory Study, Information System Management, vol.26, no.2, pp.164-175, 2009.
13. L. Putnam: A general empirical solution to the macro software sizing and estimatingproblem, IEEE Trans. on Software Eng., vol.4, no.4, pp.345-361, 1978.

Software Measurement Conference

An Empirical Analysis of Information Technology Operations Cost
14. R. Selby, and A. Porter: Learning from examples: generation and evaluation of decision trees for software resource analysis, IEEE Trans. on Software Eng., vol.14, no.12,
pp743-757, 1988.
15. M. Shepperd, and C. Schofield: Estimating software project effort using analogies,
IEEE Trans. on Software Eng., vol.23, no.12, pp.736-743, 1997.
16. K. Srinivasan, and D. Fisher: Machine Learning Approaches to Estimating Software
Development Effort, IEEE Trans. on Software Eng., Vol.21, No.2, pp.126-137, 1995.
17. B. Winer, D. Brown, and K. Michels: Statistical Principles in Experimental Design(3rd edn.), McGraw-Hill, 1991.

IWSM/MetriKon 2010

M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, and T. Oshino

Software Measurement Conference