# Reliability Evaluation of Service-Oriented Architecture Systems Considering Fault-Tolerance Designs

Authors: Kuan-Li Peng; Chin-Yu Huang
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Service-oriented architecture (SOA) provides an elastic and automatic way to discover, publish, and compose individual services. SOA enables faster integration of existing software components from different parties, makes fault tolerance (FT) feasible, and is also one of the fundamentals of cloud computing. However, the unpredictable nature of SOA systems introduces new challenges for reliability evaluation, while reliability and dependability have become the basic requirements of enterprise systems. This paper proposes an SOA system reliability model which incorporates three common fault-tolerance
strategies. Sensitivity analysis of SOA at both coarse and fine grain levels is also studied, which can be used to efficiently identify the critical parts within the system. Two SOA system scenarios based on real industrial practices are studied. Experimental results show that the proposed SOA model can be used to accurately depict the behavior of SOA systems. Additionally, a sensitivity analysis that quantizes the effects of system structure as well as fault tolerance on the overall reliability is also studied. On the whole, the proposed reliability modeling and analysis framework may help the SOA system service provider to evaluate the overall system reliability effectively and also make smarter improvement plans by focusing resources on enhancing reliability-sensitive parts within the system.


---

## Body

## 1. Introduction

Service-oriented architecture (SOA) has become a major distributed computing framework [1]. With characteristics like standardized interfaces, loosely coupled structure, cross-platform as well as elastic service discovery, deployment, and reuse capabilities, SOA opens a new door to faster integration of existing software components from different parties, especially in the scheme of Web services (WS). Legacy components may still live within the system via service adapters [2], which is good for enterprises which prefer system upgrades in gentle and stable way.It is noted that SOA also makes fault-tolerance (FT) techniques feasible for building reliable systems. Since it is difficult to build failure-free useful systems under limited development costs and the pressure of time to market, software fault tolerance [3], whose concepts originated from hardware reliability assurance, was proposed as an effective way to utilize redundancy to mask software failures and recover to normal operational states in a long running system. However, the extra costs of bringing out alternative software designs (redundancy) basically limit the applications of software fault tolerance to the fields that require ultrahigh reliabilities such as military, transportation, and aerospace. The emergence of service-oriented computing (SOC) helps lower the costs of making redundant software logic by reusing similar services published by different parties [4].The unpredictability of open distributed environments where SOA systems are exposed, such as service fail-outs, service modification, linkage failures, and traffic congestion, threatens the dependability of the overall system. To ensure sufficient SOA system dependability, suitable reliability assessment and improvement methodologies are needed. Reliability modeling has been studied extensively in the field of software engineering, and many elegant solutions have emerged [5, 6], among which the component-based or architecture-based models [7, 8] appear to be most conceptually suited to be mapped to SOA systems. However, one of the major distinctions between traditional software system reliability models and SOA system reliability models are that the former models usually assume reliable communication between components and high transparency of system operation information, which generally is not applicable for SOA systems.Based upon our previous research [9], in this paper, we further investigate the sensitivity analysis of each of the system’s subparts, which may be useful in making critical system maintenance plans for resource allocation. Note that the proposed model is an extended version of the work by Wang et al. [10], whereas a large portion of changes and extensions have been made for SOA systems such as the incorporation of service-based concepts and more detailed considerations of fault-tolerance mechanisms. The major contributions of this paper are twofold. (
1
) The first is elegant SOA system reliability modeling considering unreliable services, unreliable communication links, and internal mechanisms of three typical fault-tolerance strategies. Note that existing reliability modeling methods tend to ignore the unreliabilities of communication links and oversimplify the FT mechanisms, which may deviate from the real situations. This paper is also among the earliest SOA reliability studies that adopt Markov-based analysis rather than popular path-based analysis, and therefore the complexities of tricky path analysis involving various branches and loops are eliminated. (
2
) The second is sensitivity analysis of SOA at two grain levels, which helps the system service provider to identify reliability-critical subparts and make smarter system improvement plans.The remainder of the paper is organized as follows. Section2 presents some work on the reliability modeling and the realization of FT on service-based systems. Section 3 describes the proposed system reliability model for SOA systems and presents the reliability sensitivity analysis. Section 4 is the experiment results and analysis. Then in Section 5, there are discussions of factors that should receive attention when the results of this research are extended to other systems. Finally, Section 6 concludes the paper. Supplements such as RcB reliability formulation, sensitivity analysis derivation, and manual path-based reliability computation are presented in the appendixes.


## 2. Related Works

SOA systems generally run in open and distributed environments, which introduces new sources of failures against traditional software systems such as interface changes, workflow inconsistency, time-outs, service-level agreement (SLA) constraints, and QoS constraints. Chan et al. [11] presented a fault taxonomy for service-oriented systems. In addition to the basic enabling protocols for SOA such as SOAP [12] and WSDL [13], protocols to enhance reliability on SOA systems have been proposed. WS-ReliableMessaging [14] allows SOAP messages to be delivered between services reliably. WS-coordination [15] describes a framework enabling coordination of transaction and workflow to operate in a heterogeneous environment. However, these protocols have not yet formed a complete SOA reliability solution and sometimes it is inevitable to interact with services without the support of those protocols. As mentioned in the preceding section, fault tolerance is suitable for applications that require high reliability, and various FT implementations under service-oriented environments are available such as [16–18], which makes it a viable option for SOA system engineers.It is also noted that some models for evaluating the reliability of SOA systems have been proposed. Some focus on the reliability assessment of a single Web service. For example, Zheng et al. [19] designed a framework to retrieve the information of a large number of Web services worldwide, autogenerate testing modules, make testing invocations, and finally analyze the feedback from Web services to retrieve their reliability values. Other work, such as [20], evaluates the reliability of a composite service based on its structural information or considers basic FT behaviors on SOA systems, such as [4]. Existing models are helpful in understanding the QoS of service-oriented systems though improvements in the SOA reliability models are still desired, including more detailed considerations of FT mechanisms as well as uncertainties of communication links. This paper covers three major fault-tolerance designs, namely, Recovery Block (RcB), N-version Programming (NVP), and Retry Block (RtB), without assuming the implementation details. Basic introduction considerations of their exceptional behavior are briefly presented in the following.Recovery block (RcB) [3] is a backward recovery technique that uses the acceptance test (AT) to check the outputs of a module; the next alternative is invoked once the former alternative fails the AT. The executive in RcB is responsible for checkpoint establishment, checkpoint restoration, invocation of the alternatives, and successful return of RcB. RcB is vulnerable to failure under conditions such as when (
1
) checkpoint establishment fails, (
2
) checkpoint restoration or invocation of the next alternative fails, or (
3
) the AT itself fails or no alternative passes the AT.N-version programming (NVP) [3] is a forward recovery technique that uses the decision maker (DM) to vote for the outputs of all the involved alternative modules. The executive is responsible for input distribution and successful return of the NVP block. NVP is vulnerable to failure under conditions such as when (
1
) input distribution to the alternatives fails, (
2
) less than ⌊
n
/
2
+
1
⌋ consistent and correct results successfully reach the DM, or (
3
) the DM itself fails.Retry block (RtB) [3] is a backward recovery technique that also uses AT like RcB to check the outputs of a module. Contrary to RcB, RtB retries the same module rather than using another module once the AT evaluation fails. The original design of RtB requires a data reexpression algorithm (DRA) to change the form of inputs before reusing the same module [3], but since in the Web applications or SOA systems failures may be caused by temporary service busy or network congestion (Heisenbug) and it is not always possible to tailor the DRA for the application, some designs eliminate DRA and use the same inputs on retry, and this simplified retry mechanism is widely supported in modern SOA execution environments. This paper considers the latter designs. RtB is vulnerable to failure under conditions such as when (
1
) checkpoint establishment fails, (
2
) checkpoint restoration fails, or (
3
) the AT itself fails or the retry limit is exceeded before the AT passes.It should be noted that although the main function is protected by the redundant designs/executions in the fault-tolerance blocks, the executive and the AT/DM are not protected and may be a potential reliability bottleneck. Several modified designs of RcB and NVP such as [21, 22] also introduce redundancy at these points, resulting in even more complex fault-tolerance design and higher costs. Therefore such improvements are not considered in this paper.


## 3.  SOA System Reliability Modeling and Analysis

Before the introduction of the proposed model, several definitions of SOA systems and reliability modeling are listed here.


### 3.1. Framework of the Model

Without loss of generality, an SOA system may be viewed as a flow of services (calledworkflow) and be depicted by a BPMN diagram [23] as in Figure 1, which contains the single start event (the thin-lined circle), n abstract services (round rectangles), message transmission between services (arrows), branching points (diamonds), and the single end event (the thick-lined circle). Specifically, this research takes the viewpoint of on-demand business process rather than long running process, where at the request of a client, the system starts at the virtual start event, passes control to the next abstract service along the message path, and finally stops normally and transfers to the success state when the control reaches the virtual end point. The workflow may branch and converge at some points.Figure 1
SOA service flow graph.In the realization of the SOA system, an abstract service is often fulfilled by one or morephysical services, which could be an atomic Web service or a composite Web service which invokes one or more atomic/composite services internally, and messages travel in the network under the protocols such as SOAP [12]. Services may also be protected by the FT techniques in the form of a composite Web service. It should be noted that in the real world, failures (or exceptions) may originate from any service, control point, or communication path. Unhandled failures interrupt the normal process flow and turn the system into a failure state.A number of notations in this paper are defined as follows.Reliability. This paper views the operation of SOA systems as an on-demand business process, and the reliability is the probability of successful executions. This kind of time-invariant reliability definition is used in traditional software reliability estimation [7, 24] and later widely adopted in existing research on SOA reliability modeling that also considers on-demand cases, such as [1, 25, 26].System Reliability (R). It is the probability of a system execution that finally reaches the success state, estimated by R
=
1
-
f
/
t, where t is the total number of system executions and f is the number of system executions that ended up in failure states.Service Reliability (
r
i
). It is the probability of successful executions of physical service i, estimated by r
i
=
1
-
f
i
/
t
i, where t
i is the total number of service invocations and f
i is the number of service executions that threw unhandled exceptions.Link Reliability (
l
i
;
j
). It is the probability of successful messages passing from physical service i to physical service j, estimated by l
i
,
j
=
1
-
f
i
,
j
/
t
i
,
j, where t
i
,
j is the total number of messages passing from service i to service j and f
i
,
j is the total number of unsuccessful messages passing from service i to service j.System Reliability Model. The SOA system reliability model is denoted by a 5-tuple (
Q
,
δ
,
s
1
,
s
k
,
M
) [9, 10], where(i)
Q: a finite set of k states {
s
1
,
s
2
,
…
,
s
k
};
(ii)
δ: state transition mapping Q
×
E
→
Q, and E is a set of triggering events;
(iii)
s
1: the virtual initial state with reliability 1, which transits to all the real initial states with their corresponding initiating probabilities, and if there is exactly one initial state, s
1 can be replaced by that state for simplicity;
(iv)
s
k: the virtual success final state with reliability 1, which is reached from all the real final states with their corresponding terminating probabilities;
(v)
M: a k
×
k Markov transition matrix for the states in Q, whose entries are defined by
(1)
m
i
,
j
=
{
p
i
,
j
,
if
∃
e
∈
E
s
.
t
.
δ
(
s
i
,
e
)
=
s
j
,
0
,
otherwise
,
wherep
i
,
j is the transition probability from state i to state j.To estimate the reliability of an SOA system, one may apply the following steps (Figure2).(1)
Identify the workflow of the system and the physical services that realize each abstract service within the workflow. This may be obtained from process specifications, such as WS-BPEL [27] documents.
(2)
Determine the reliability of each physical service and message link within the system. This may be estimated from the service access logs if they are available or be collected and estimated from user feedback.
(3)
Determine the transition probabilities (or frequency) between abstract services. Existing techniques for building operational profiles [28] may apply here.
(4)
Identify the internal states and construct the Markov transition matrixM from the information collected in Steps 1–3. This is explained further in Section 3.2.
(5)
Derive the reliability estimation of the whole SOA system.
(6)
Optionally, perform further analysis (such as sensitivities) and action based on the results of Step 5.Figure 2
Framework of the model.Note that, as reported by Zheng and Lyu [1], different users may actually experience different performance from the same service, since the service performance is influenced by the communication links substantially, and some SOA system may provide different localized physical services depending on the users’ preferences or location. Therefore, in Steps 2 and 3, the information may be collected together (from the viewpoint of the system operator) or be classified in groups of users individually (from the viewpoint of specific user groups) depending on the purpose of the reliability analysis.


### 3.2. Construction of Markov Transition MatrixM

In the system reliability model, the internal states are derived from abstract services. Each abstract service is mapped to a distinct macro-state. A macro-state consists of only one microstate unless its corresponding abstract service is guarded by fault tolerance, in which more than one tightly coupled microstate may be generated for that service. It should be noted that each microstate belongs to exactly one macro-state; each macro-state has exactly one entry micro-state and one exit micro-state and the two may overlap. Also remember that macro-state 1 is the initial state while the last macro-state is the final state as stated in Section3.1. The transition matrix is obtained by
(2)
M
=
[
m
i
,
j
]
=
{
s
r
i
s
p
i
,
j
s
l
i
,
j
,
if
∃
A
,
B
s
.
t
.
i
=
ω
A
∧
j
=
α
B
,
f
(
i
,
j
)
,
else
if
∃
A
s
.
t
.
i
∈
A
∧
j
∈
A
,
0
,
otherwise
.
In (2), s
r
i, s
p
i
,
j, and s
l
i
,
j denote state reliability, state transition probability, and state message link reliability, respectively; the entry and exit micro-states of a macro-state K are denoted by α
K and ω
K; and the inclusion relation of microstate i to macro-state K is denoted by i
∈
K.In the first condition of (2), assuming that the corresponding physical services to micro-states i and j are services m and n respectively, then s
r
i
=
r
m (It is the default case unless redefined later in this section.), s
p
i
,
j
=
p
m
,
n, and s
l
i
,
j
=
l
m
,
n. The method for retrieving the p
i
,
j and l
i
,
j values has been briefly explained earlier, and [9] has the interpretation of l
i
,
j in detail under two major service composition schemes in SOA systems. The major problem left is to define f
(
i
,
j
) for the macro-states whose corresponding abstract services are realized in more complicated way, especially in various FT blocks.Generally, for nonfault-tolerant services, either atomic or composite, it is intuitive to define a macro-state with a single micro-state, which reduces to the first condition and it is therefore not required to definef
(
i
,
j
) for them.For any abstract service implemented in RcB, two approaches may be used. One way is to define micro-states separately for the executive, the AT, and each alternative. Assume that there aren alternatives in total, and both the microstate numbers as well as the corresponding physical service numbers for (executive, alternatives from the first to last, AT) are (d, d
+
1
⋯
d
+
n, d
+
n
+
1). Then within the domain d
≤
i
,
j
≤
d
+
n
+
1, f
(
i
,
j
) is defined as follows:
(3)
f
(
i
,
j
)
=
{
r
d
,
i
=
d
,
j
=
d
+
1
,
(
1
-
S
d
,
i
,
d
+
n
+
1
)
r
d
r
d
+
n
+
1
,
d
+
1
≤
i
<
d
+
n
,
j
=
i
+
1
,
S
d
,
i
,
d
+
n
+
1
,
d
+
1
≤
i
≤
d
+
n
,
j
=
d
+
n
+
1
,
0
,
otherwise
.Here a convenient substitutionS
a
,
b
,
c
=
l
a
,
b
r
b
l
b
,
c is used to simplify the formula. Further note that s
r
d
+
n
+
1
=
r
d
+
n
+
1. The second approach is to equivalently define a single microstate k with the state reliability computed by
(4)
s
r
k
=
∑
i
=
1
n
‍
(
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
.The first approach is more intuitive while the second one results in more compact transition matrix. Refer to AppendixA for more on the RcB reliability formulation.For any abstract service implemented in NVP, assuming that there aren alternatives in total and the corresponding physical service numbers for (executive, alternatives from the first to last, DM) are (e, e
+
1
⋯
e
+
n, e
+
n
+
1), then one can also define a single microstate k with the state reliability computed by
(5)
s
r
k
=
r
e
r
e
+
n
+
1
∑
F
c
∈
2
n
‍
∏
1
≤
i
≤
n
‍
S
e
,
e
+
i
,
e
+
n
+
1
F
c
,
i
×
(
1
-
S
e
,
e
+
i
,
e
+
n
+
1
)
1
-
F
c
,
j
[
|
F
c
|
>
n
2
]
,
where [
X
] is an indicator function such that for condition X,
(6)
[
X
]
=
{
1
,
if
X
is
true
,
0
,
otherwise
,
and F
c denotes a configuration set c of n binary values representing the outcomes F
c
,
i of each alternative i defined as follows:
(7)
F
c
,
i
=
{
1
,
if
alternative
i
successfully
passes
DM
,
0
,
otherwise
;
|
F
c
|
=
∑
i
‍
[
F
c
,
i
=
1
]
.
For any abstract service implemented in RtB, assuming that the retry limit is γ and the service numbers for (executive, the retry alternative, AT) are numbered (h, h
+
1, h
+
2), then its operation is equivalent to an RcB where there are γ duplicates of the alternative h. One may create γ
+
2 micro-states and define f
(
i
,
j
) as in (3) or otherwise define a single microstate k with the state reliability computed by
(8)
s
r
k
=
∑
i
=
1
γ
‍
(
S
h
,
h
+
1
,
h
+
2
(
1
-
S
h
,
h
+
1
,
h
+
2
)
i
-
1
r
h
i
r
h
+
2
i
)
.Finally, once the transition matrixM is completed, the system reliability R is computed by
(9)
R
=
(
-
1
)
k
+
1
|
(
I
-
M
)
k
,
1
|
|
I
-
M
|
,
according to [7], where I is the identity matrix of dimension k
×
k and (
I
-
M
)
k
,
1 is the minor matrix eliminating the last row and the leading column of I
-
M.


### 3.3. Sensitivity Analysis

Based on the previous reliability modeling, this section tries to explore the system more deeply and identifies the critical services with respect to their impact on the overall reliability of an SOA system.Here the sensitivity analysis is made at two levels [10]: a coarse-grained level (L1, denoted by sc) and a fine-grained level (L2, denoted by sf). L1 involves the sensitivity of each macro-state within the reliability model, while L2 involves the sensitivity of each service within the SOA system. Substantial modifications of the original sensitivity analysis have been made for SOA systems, and link reliabilities are also included in this study. For convenience of analysis, here it is assumed that each macro-state consists of exactly one microstate and no physical services are shared among different abstract services.A number of notations are defined as follows:(i)
W
=
[
w
i
,
j
]
=
I
-
M,
(ii)
E
=
[
e
i
,
j
]
=
(
I
-
M
)
k
,
1,
(iii)
α
i
,
j: cofactor of w
i
,
j,
(iv)
β
i
,
j: cofactor of e
i
,
j.In L1, (9) can be rewritten as
(10)
R
=
(
-
1
)
k
+
1
|
E
|
|
W
|
,
(11)
|
W
|
=
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
,
where θ
1
,
i
=
α
i
,
i, θ
2
,
i
=
-
α
i
,
i
s
p
i
,
i
s
l
i
,
i, and θ
3
,
i
=
-
∑
1
≤
j
≤
k
,
j
≠
i
‍
α
i
,
j
s
p
i
,
j
s
l
i
,
j. It is noted that none of θ
1
,
i, θ
2
,
i, or θ
3
,
i is a function of s
r
i. In addition,
(12)
|
E
|
=
σ
1
,
i
+
(
σ
2
,
i
+
σ
3
,
i
)
s
r
i
,
where σ
1
,
i
=
β
i
,
i
-
1, σ
2
,
i
=
-
β
i
,
i
-
1
s
p
i
,
i
s
l
i
,
i, and σ
3
,
i
=
-
∑
2
≤
j
≤
k
,
j
≠
i
‍
β
i
,
j
-
1
s
p
i
,
j
s
l
i
,
j for 1
<
i
<
k. It is also noted that none of σ
1
,
i, σ
2
,
i, or σ
3
,
i is a function of s
r
i.Then,(13)
R
=
(
-
1
)
k
+
1
s
r
k
σ
1
,
i
+
(
σ
2
,
i
+
σ
3
,
i
)
s
r
i
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
,
for
1
<
i
<
k
,
and the sensitivity for microstate i is
(14)
s
c
i
=
∂
R
∂
s
r
i
=
(
-
1
)
k
+
1
s
r
k
θ
1
,
i
(
σ
2
,
i
+
σ
3
,
i
)
-
(
θ
2
,
i
+
θ
3
,
i
)
σ
1
,
i
(
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
)
2
,
for
1
<
i
<
k
.
s
c
i is only defined over 1
<
i
<
k because macro-states 1 and k are virtual states. Related derivations of |
W
| and |
E
| are given in Appendix B.Next in L2, the sensitivity analysis is made by different categories of the abstract services connected to each macro-state.For a nonfault-tolerant servicek, since there is only one physical service p associated with the macro-state, the state sensitivity is consistent with the service sensitivity. That is,
(15)
s
f
p
=
s
c
k
.For a servicek implemented in RcB, assuming the same service number settings in Section 3.2 and recalling the state reliability for RcB in (4), then the sensitivity of the executive d is
(16)
s
f
d
=
∂
R
∂
r
d
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
=
s
c
k
∑
i
=
1
n
‍
(
i
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
-
1
r
d
+
n
+
1
i
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
,
the sensitivities of the alternatives d
+
1
⋯
d
+
n are
(17)
s
f
d
+
a
=
∂
R
∂
r
d
+
a
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
+
a
=
s
c
k
l
d
,
d
+
a
l
d
+
a
,
d
+
n
+
1
(
A
+
B
)
,
where
(18)
A
=
r
d
a
r
d
+
n
+
1
a
∏
j
=
1
a
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
B

=

-
∑
a
<
i
≤
n
‍


(
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
∏
1
≤
j
<
i
,
j
≠
a


‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)


)
,
and the sensitivity of the AT d
+
n
+
1 is
(19)
s
f
d
+
n
+
1
=
∂
R
∂
r
d
+
n
+
1
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
+
n
+
1
=
s
c
k
∑
i
=
1
n
‍
(
i
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
-
1
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
.For a servicek implemented in NVP, note that (5) is not in a simple closed form. To simplify the discussion, here it is assumed that 3 alternatives in total are in the NVP block. Equation (5) then reduces to
(20)
s
r
k
=
r
e
r
e
+
4
[
S
e
,
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
(
1
-
S
e
,
e
+
3
,
e
+
4
)
+
S
e
,
e
+
1
,
e
+
4
(
1
-
S
e
,
e
+
2
,
e
+
4
)
S
e
,
e
+
3
,
e
+
4
+
(
1
-
S
e
,
e
+
1
,
e
+
4
)
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
+
S
e
,
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
]
.The sensitivity of the executivee is
(21)
s
f
e
=
∂
R
∂
r
e
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
=
s
c
k
s
r
k
r
e
.The sensitivities of the alternativese
+
1
⋯
e
+
3 are
(22)
s
f
e
+
a
=
∂
R
∂
r
e
+
a
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
+
a
,
where
(23)
∂
s
r
k
∂
r
e
+
1
=
r
e
r
e
+
4
[
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
(
1
-
S
e
,
e
+
3
,
e
+
4
)
-
l
e
,
e
+
1
l
e
+
1
,
e
+
4
(
1
-
S
e
,
e
+
2
,
e
+
4
)
S
e
,
e
+
3
,
e
+
4
+
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
+
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
]
and ∂
s
r
k
/
∂
r
e
+
2 as well as ∂
s
r
k
/
∂
r
e
+
3 can be derived similarly.The sensitivity of the DMe
+
4 is
(24)
s
f
e
+
4
=
∂
R
∂
r
e
+
4
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
+
4
=
s
c
k
s
r
k
r
e
+
4
.Equations for the sensitivities of the RtB services are similar to those for the RcB services.


## 3.1. Framework of the Model

Without loss of generality, an SOA system may be viewed as a flow of services (calledworkflow) and be depicted by a BPMN diagram [23] as in Figure 1, which contains the single start event (the thin-lined circle), n abstract services (round rectangles), message transmission between services (arrows), branching points (diamonds), and the single end event (the thick-lined circle). Specifically, this research takes the viewpoint of on-demand business process rather than long running process, where at the request of a client, the system starts at the virtual start event, passes control to the next abstract service along the message path, and finally stops normally and transfers to the success state when the control reaches the virtual end point. The workflow may branch and converge at some points.Figure 1
SOA service flow graph.In the realization of the SOA system, an abstract service is often fulfilled by one or morephysical services, which could be an atomic Web service or a composite Web service which invokes one or more atomic/composite services internally, and messages travel in the network under the protocols such as SOAP [12]. Services may also be protected by the FT techniques in the form of a composite Web service. It should be noted that in the real world, failures (or exceptions) may originate from any service, control point, or communication path. Unhandled failures interrupt the normal process flow and turn the system into a failure state.A number of notations in this paper are defined as follows.Reliability. This paper views the operation of SOA systems as an on-demand business process, and the reliability is the probability of successful executions. This kind of time-invariant reliability definition is used in traditional software reliability estimation [7, 24] and later widely adopted in existing research on SOA reliability modeling that also considers on-demand cases, such as [1, 25, 26].System Reliability (R). It is the probability of a system execution that finally reaches the success state, estimated by R
=
1
-
f
/
t, where t is the total number of system executions and f is the number of system executions that ended up in failure states.Service Reliability (
r
i
). It is the probability of successful executions of physical service i, estimated by r
i
=
1
-
f
i
/
t
i, where t
i is the total number of service invocations and f
i is the number of service executions that threw unhandled exceptions.Link Reliability (
l
i
;
j
). It is the probability of successful messages passing from physical service i to physical service j, estimated by l
i
,
j
=
1
-
f
i
,
j
/
t
i
,
j, where t
i
,
j is the total number of messages passing from service i to service j and f
i
,
j is the total number of unsuccessful messages passing from service i to service j.System Reliability Model. The SOA system reliability model is denoted by a 5-tuple (
Q
,
δ
,
s
1
,
s
k
,
M
) [9, 10], where(i)
Q: a finite set of k states {
s
1
,
s
2
,
…
,
s
k
};
(ii)
δ: state transition mapping Q
×
E
→
Q, and E is a set of triggering events;
(iii)
s
1: the virtual initial state with reliability 1, which transits to all the real initial states with their corresponding initiating probabilities, and if there is exactly one initial state, s
1 can be replaced by that state for simplicity;
(iv)
s
k: the virtual success final state with reliability 1, which is reached from all the real final states with their corresponding terminating probabilities;
(v)
M: a k
×
k Markov transition matrix for the states in Q, whose entries are defined by
(1)
m
i
,
j
=
{
p
i
,
j
,
if
∃
e
∈
E
s
.
t
.
δ
(
s
i
,
e
)
=
s
j
,
0
,
otherwise
,
wherep
i
,
j is the transition probability from state i to state j.To estimate the reliability of an SOA system, one may apply the following steps (Figure2).(1)
Identify the workflow of the system and the physical services that realize each abstract service within the workflow. This may be obtained from process specifications, such as WS-BPEL [27] documents.
(2)
Determine the reliability of each physical service and message link within the system. This may be estimated from the service access logs if they are available or be collected and estimated from user feedback.
(3)
Determine the transition probabilities (or frequency) between abstract services. Existing techniques for building operational profiles [28] may apply here.
(4)
Identify the internal states and construct the Markov transition matrixM from the information collected in Steps 1–3. This is explained further in Section 3.2.
(5)
Derive the reliability estimation of the whole SOA system.
(6)
Optionally, perform further analysis (such as sensitivities) and action based on the results of Step 5.Figure 2
Framework of the model.Note that, as reported by Zheng and Lyu [1], different users may actually experience different performance from the same service, since the service performance is influenced by the communication links substantially, and some SOA system may provide different localized physical services depending on the users’ preferences or location. Therefore, in Steps 2 and 3, the information may be collected together (from the viewpoint of the system operator) or be classified in groups of users individually (from the viewpoint of specific user groups) depending on the purpose of the reliability analysis.


## 3.2. Construction of Markov Transition MatrixM

In the system reliability model, the internal states are derived from abstract services. Each abstract service is mapped to a distinct macro-state. A macro-state consists of only one microstate unless its corresponding abstract service is guarded by fault tolerance, in which more than one tightly coupled microstate may be generated for that service. It should be noted that each microstate belongs to exactly one macro-state; each macro-state has exactly one entry micro-state and one exit micro-state and the two may overlap. Also remember that macro-state 1 is the initial state while the last macro-state is the final state as stated in Section3.1. The transition matrix is obtained by
(2)
M
=
[
m
i
,
j
]
=
{
s
r
i
s
p
i
,
j
s
l
i
,
j
,
if
∃
A
,
B
s
.
t
.
i
=
ω
A
∧
j
=
α
B
,
f
(
i
,
j
)
,
else
if
∃
A
s
.
t
.
i
∈
A
∧
j
∈
A
,
0
,
otherwise
.
In (2), s
r
i, s
p
i
,
j, and s
l
i
,
j denote state reliability, state transition probability, and state message link reliability, respectively; the entry and exit micro-states of a macro-state K are denoted by α
K and ω
K; and the inclusion relation of microstate i to macro-state K is denoted by i
∈
K.In the first condition of (2), assuming that the corresponding physical services to micro-states i and j are services m and n respectively, then s
r
i
=
r
m (It is the default case unless redefined later in this section.), s
p
i
,
j
=
p
m
,
n, and s
l
i
,
j
=
l
m
,
n. The method for retrieving the p
i
,
j and l
i
,
j values has been briefly explained earlier, and [9] has the interpretation of l
i
,
j in detail under two major service composition schemes in SOA systems. The major problem left is to define f
(
i
,
j
) for the macro-states whose corresponding abstract services are realized in more complicated way, especially in various FT blocks.Generally, for nonfault-tolerant services, either atomic or composite, it is intuitive to define a macro-state with a single micro-state, which reduces to the first condition and it is therefore not required to definef
(
i
,
j
) for them.For any abstract service implemented in RcB, two approaches may be used. One way is to define micro-states separately for the executive, the AT, and each alternative. Assume that there aren alternatives in total, and both the microstate numbers as well as the corresponding physical service numbers for (executive, alternatives from the first to last, AT) are (d, d
+
1
⋯
d
+
n, d
+
n
+
1). Then within the domain d
≤
i
,
j
≤
d
+
n
+
1, f
(
i
,
j
) is defined as follows:
(3)
f
(
i
,
j
)
=
{
r
d
,
i
=
d
,
j
=
d
+
1
,
(
1
-
S
d
,
i
,
d
+
n
+
1
)
r
d
r
d
+
n
+
1
,
d
+
1
≤
i
<
d
+
n
,
j
=
i
+
1
,
S
d
,
i
,
d
+
n
+
1
,
d
+
1
≤
i
≤
d
+
n
,
j
=
d
+
n
+
1
,
0
,
otherwise
.Here a convenient substitutionS
a
,
b
,
c
=
l
a
,
b
r
b
l
b
,
c is used to simplify the formula. Further note that s
r
d
+
n
+
1
=
r
d
+
n
+
1. The second approach is to equivalently define a single microstate k with the state reliability computed by
(4)
s
r
k
=
∑
i
=
1
n
‍
(
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
.The first approach is more intuitive while the second one results in more compact transition matrix. Refer to AppendixA for more on the RcB reliability formulation.For any abstract service implemented in NVP, assuming that there aren alternatives in total and the corresponding physical service numbers for (executive, alternatives from the first to last, DM) are (e, e
+
1
⋯
e
+
n, e
+
n
+
1), then one can also define a single microstate k with the state reliability computed by
(5)
s
r
k
=
r
e
r
e
+
n
+
1
∑
F
c
∈
2
n
‍
∏
1
≤
i
≤
n
‍
S
e
,
e
+
i
,
e
+
n
+
1
F
c
,
i
×
(
1
-
S
e
,
e
+
i
,
e
+
n
+
1
)
1
-
F
c
,
j
[
|
F
c
|
>
n
2
]
,
where [
X
] is an indicator function such that for condition X,
(6)
[
X
]
=
{
1
,
if
X
is
true
,
0
,
otherwise
,
and F
c denotes a configuration set c of n binary values representing the outcomes F
c
,
i of each alternative i defined as follows:
(7)
F
c
,
i
=
{
1
,
if
alternative
i
successfully
passes
DM
,
0
,
otherwise
;
|
F
c
|
=
∑
i
‍
[
F
c
,
i
=
1
]
.
For any abstract service implemented in RtB, assuming that the retry limit is γ and the service numbers for (executive, the retry alternative, AT) are numbered (h, h
+
1, h
+
2), then its operation is equivalent to an RcB where there are γ duplicates of the alternative h. One may create γ
+
2 micro-states and define f
(
i
,
j
) as in (3) or otherwise define a single microstate k with the state reliability computed by
(8)
s
r
k
=
∑
i
=
1
γ
‍
(
S
h
,
h
+
1
,
h
+
2
(
1
-
S
h
,
h
+
1
,
h
+
2
)
i
-
1
r
h
i
r
h
+
2
i
)
.Finally, once the transition matrixM is completed, the system reliability R is computed by
(9)
R
=
(
-
1
)
k
+
1
|
(
I
-
M
)
k
,
1
|
|
I
-
M
|
,
according to [7], where I is the identity matrix of dimension k
×
k and (
I
-
M
)
k
,
1 is the minor matrix eliminating the last row and the leading column of I
-
M.


## 3.3. Sensitivity Analysis

Based on the previous reliability modeling, this section tries to explore the system more deeply and identifies the critical services with respect to their impact on the overall reliability of an SOA system.Here the sensitivity analysis is made at two levels [10]: a coarse-grained level (L1, denoted by sc) and a fine-grained level (L2, denoted by sf). L1 involves the sensitivity of each macro-state within the reliability model, while L2 involves the sensitivity of each service within the SOA system. Substantial modifications of the original sensitivity analysis have been made for SOA systems, and link reliabilities are also included in this study. For convenience of analysis, here it is assumed that each macro-state consists of exactly one microstate and no physical services are shared among different abstract services.A number of notations are defined as follows:(i)
W
=
[
w
i
,
j
]
=
I
-
M,
(ii)
E
=
[
e
i
,
j
]
=
(
I
-
M
)
k
,
1,
(iii)
α
i
,
j: cofactor of w
i
,
j,
(iv)
β
i
,
j: cofactor of e
i
,
j.In L1, (9) can be rewritten as
(10)
R
=
(
-
1
)
k
+
1
|
E
|
|
W
|
,
(11)
|
W
|
=
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
,
where θ
1
,
i
=
α
i
,
i, θ
2
,
i
=
-
α
i
,
i
s
p
i
,
i
s
l
i
,
i, and θ
3
,
i
=
-
∑
1
≤
j
≤
k
,
j
≠
i
‍
α
i
,
j
s
p
i
,
j
s
l
i
,
j. It is noted that none of θ
1
,
i, θ
2
,
i, or θ
3
,
i is a function of s
r
i. In addition,
(12)
|
E
|
=
σ
1
,
i
+
(
σ
2
,
i
+
σ
3
,
i
)
s
r
i
,
where σ
1
,
i
=
β
i
,
i
-
1, σ
2
,
i
=
-
β
i
,
i
-
1
s
p
i
,
i
s
l
i
,
i, and σ
3
,
i
=
-
∑
2
≤
j
≤
k
,
j
≠
i
‍
β
i
,
j
-
1
s
p
i
,
j
s
l
i
,
j for 1
<
i
<
k. It is also noted that none of σ
1
,
i, σ
2
,
i, or σ
3
,
i is a function of s
r
i.Then,(13)
R
=
(
-
1
)
k
+
1
s
r
k
σ
1
,
i
+
(
σ
2
,
i
+
σ
3
,
i
)
s
r
i
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
,
for
1
<
i
<
k
,
and the sensitivity for microstate i is
(14)
s
c
i
=
∂
R
∂
s
r
i
=
(
-
1
)
k
+
1
s
r
k
θ
1
,
i
(
σ
2
,
i
+
σ
3
,
i
)
-
(
θ
2
,
i
+
θ
3
,
i
)
σ
1
,
i
(
θ
1
,
i
+
(
θ
2
,
i
+
θ
3
,
i
)
s
r
i
)
2
,
for
1
<
i
<
k
.
s
c
i is only defined over 1
<
i
<
k because macro-states 1 and k are virtual states. Related derivations of |
W
| and |
E
| are given in Appendix B.Next in L2, the sensitivity analysis is made by different categories of the abstract services connected to each macro-state.For a nonfault-tolerant servicek, since there is only one physical service p associated with the macro-state, the state sensitivity is consistent with the service sensitivity. That is,
(15)
s
f
p
=
s
c
k
.For a servicek implemented in RcB, assuming the same service number settings in Section 3.2 and recalling the state reliability for RcB in (4), then the sensitivity of the executive d is
(16)
s
f
d
=
∂
R
∂
r
d
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
=
s
c
k
∑
i
=
1
n
‍
(
i
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
-
1
r
d
+
n
+
1
i
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
,
the sensitivities of the alternatives d
+
1
⋯
d
+
n are
(17)
s
f
d
+
a
=
∂
R
∂
r
d
+
a
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
+
a
=
s
c
k
l
d
,
d
+
a
l
d
+
a
,
d
+
n
+
1
(
A
+
B
)
,
where
(18)
A
=
r
d
a
r
d
+
n
+
1
a
∏
j
=
1
a
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
B

=

-
∑
a
<
i
≤
n
‍


(
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
∏
1
≤
j
<
i
,
j
≠
a


‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)


)
,
and the sensitivity of the AT d
+
n
+
1 is
(19)
s
f
d
+
n
+
1
=
∂
R
∂
r
d
+
n
+
1
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
d
+
n
+
1
=
s
c
k
∑
i
=
1
n
‍
(
i
S
d
,
d
+
i
,
d
+
n
+
1
r
d
i
r
d
+
n
+
1
i
-
1
∏
j
=
1
i
-
1
‍
(
1
-
S
d
,
d
+
j
,
d
+
n
+
1
)
)
.For a servicek implemented in NVP, note that (5) is not in a simple closed form. To simplify the discussion, here it is assumed that 3 alternatives in total are in the NVP block. Equation (5) then reduces to
(20)
s
r
k
=
r
e
r
e
+
4
[
S
e
,
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
(
1
-
S
e
,
e
+
3
,
e
+
4
)
+
S
e
,
e
+
1
,
e
+
4
(
1
-
S
e
,
e
+
2
,
e
+
4
)
S
e
,
e
+
3
,
e
+
4
+
(
1
-
S
e
,
e
+
1
,
e
+
4
)
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
+
S
e
,
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
]
.The sensitivity of the executivee is
(21)
s
f
e
=
∂
R
∂
r
e
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
=
s
c
k
s
r
k
r
e
.The sensitivities of the alternativese
+
1
⋯
e
+
3 are
(22)
s
f
e
+
a
=
∂
R
∂
r
e
+
a
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
+
a
,
where
(23)
∂
s
r
k
∂
r
e
+
1
=
r
e
r
e
+
4
[
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
(
1
-
S
e
,
e
+
3
,
e
+
4
)
-
l
e
,
e
+
1
l
e
+
1
,
e
+
4
(
1
-
S
e
,
e
+
2
,
e
+
4
)
S
e
,
e
+
3
,
e
+
4
+
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
+
l
e
,
e
+
1
l
e
+
1
,
e
+
4
S
e
,
e
+
2
,
e
+
4
S
e
,
e
+
3
,
e
+
4
]
and ∂
s
r
k
/
∂
r
e
+
2 as well as ∂
s
r
k
/
∂
r
e
+
3 can be derived similarly.The sensitivity of the DMe
+
4 is
(24)
s
f
e
+
4
=
∂
R
∂
r
e
+
4
=
∂
R
∂
s
r
k
∂
s
r
k
∂
r
e
+
4
=
s
c
k
s
r
k
r
e
+
4
.Equations for the sensitivities of the RtB services are similar to those for the RcB services.


## 4. Experiments and Discussion

In this paper, results and discussions for a vehicle rental system adapted from industrial practices [29] are presented. The workflow diagram (Figure 3) for the system has sequences, branching, and looping, common in real-world projects. According to the diagram, when a customer uses the service, a rental agreement is prepared in the system, and then, based on the customer’s preferences, the system searches for vehicle choices from the database. The customer may either accept a choice, call for more choices, or drop the service directly. Upon accepting the rental choice, the system then assists the user in contacting the insurance company and paying the security deposit through a third-party payment service. Finally, the system activates the rental agreement and the customer is guided to take his car.Figure 3
BPMN diagram.Two scenarios based upon this workflow specification are incorporated: scenario no. 1 is the fault-tolerance-free version where each abstract service is fulfilled by one physical (either internal or external) service, while scenario no. 2 uses RcB to ensure higher reliability for the payment operation. For each scenario, Table1 displays the physical services involved and their reliabilities, Table 2 displays the transition (branching) probabilities between the abstract services, and Table 3 displays the link reliabilities between the physical services.Table 1
Service configuration.
Abstract srv.
Scenario no. 1
Scenario no. 2
Physical srv.
Srv rel.
Physical srv.
Srv rel.
1. RA init.
Int srv (1)
0.99
Int srv (1)
0.99
2. Veh. listing
Int DB srv (2)
0.97
Int DB srv (2)
0.97
3. Higher-grp.-veh. Asgmnt.
Int DB srv (3)
0.97
Int DB srv (3)
0.97
4. Insurance order
Ext srv (4)
0.91
Ext srv (4)
0.91
5. Payment
Ext srv (6)
0.91
Int exec (5)
0.99
Ext alt I (6)
0.91
Ext alt II (7)
0.88
Ext alt III (8)
0.85
Int AT (9)
0.99
6. RA activation
Int DB srv (10)
0.97
Int DB srv (10)
0.97Table 2
Abstract service transition probability.
Abstract srv trans.
Prob
2
.
→
3.
0.25
2
.
→
4.
0.40
2
.
→
End
0.35Table 3
Physical service link reliability.
Physical srv lnk
Rel
Start
→
(
1
)
1.00
(
1
)
→
(
2
)
1.00
(
2
)
→
(
3
)
1.00
(
2
)
→
(
4
)
0.98
(
2
)
→
End
1.00
(
3
)
→
(
2
)
1.00
(
4
)
→
(
5
)
1.00
(
4
)
→
(
6
)
0.98
(
6
)
→
(
10
)
0.98
(
5
)
→
(
6
)
0.98
(
5
)
→
(
7
)
0.98
(
5
)
→
(
8
)
0.96
(
6
)
→
(
9
)
0.98
(
7
)
→
(
9
)
0.98
(
8
)
→
(
9
)
0.96
(
9
)
→
(
10
)
1.00
(
10
)
→
End
1.00


### 4.1. Reliability Modeling

Applying the Steps in Section3.2, one can derive the transition matrix for scenario no. 1(25)
M
scn
1
=
[
0
0
.
99
0
0
0
0
0
0
0
0
.
242
0
.
38
0
0
0
.
339
0
0
.
97
0
0
0
0
0
0
0
0
0
0
.
892
0
0
0
0
0
0
0
0
.
892
0
0
0
0
0
0
0
0
.
97
0
0
0
0
0
0
0
]
and the transition matrix for scenario no. 2
(26)
M
scn
2


=


[
0
0
.
99
0
0
0
0
0
0
0
0
0
0
0
0
.
242
0
.
38
0
0
0
0
0
0
0.339
0
0
.
97
0
0
0
0
0
0
0
0
0
0
0
0
0
0.91
0
0
0
0
0
0
0
0
0
0
0
0.99
0
0
0
0
0
0
0
0
0
0
0
0.124
0.874
0
0
0
0
0
0
0
0
0
0
0.845
0
.
152
0
0
0
0
0
0
0
0
0
0
0
0.99
0
0
0
0
0
0
0
0
0.783
0
0
0
0
0
0
0
0
0
0
0
0
0
0.97
0
0
0
0
0
0
0
0
0
0
0
]
.Ten simulations of the system where each run contained 100,000 system calls were executed for each scenario. Comparing the experimental results and the values from the proposed model in Table4. In Table 4, the 1st row is the reliability estimation by the proposed model, the 2nd row is from the manual probability computation as explained in Appendix C, the 3rd row contains the statistics of simulation, and the last row is the relative error between the simulation results and the theoretical values in the 3rd row. It can be seen that the proposed SOA reliability model has identical values to those computed from probability theory, and the average simulation results are very close to the theoretical values. Comparing the two scenarios, it can also be seen that the introduction of RcB in the payment service (abstract service 5) improves the local service reliability by 11.36% (Considering both the service invocation and the service execution, the payment service reliability is 0.874 in scenario no. 1 and is 0.973 in scenario no. 2.) and improves the overall system reliability by 5.2%.Table 4
System reliability model validation results.
Scenario no. 1
Scenario no. 2
SOA reliability model
0.819
0.862
Probability computation
0.819
0.862
Simulation results
0.819
0.865
Simulation relative error
0.00%
0.35%It is also clarified that the 0.00% simulation relative error in Table4 is only the statistical value for certain scenario in certain precision level and should not be otherwise interpreted as 100% simulation accuracy.


### 4.2. Sensitivity Analysis

Results of the sensitivity analysis for both scenarios are displayed in Tables5 and 6, respectively. The sensitivity value of a state/service indicates its impact on the overall system reliability. It could also be observed from Figure 4 that abstract services with higher execution probabilities (such as abstract services 1 and 2) have higher sensitivity values. In abstract service 5 in scenario no. 2, it is also noted that the executive service and the AT service actually dominate the reliability of RcB, as one observes the sub-graph within Figure 4. Such result is reasonable since both services are always on the execution paths for all the alternatives. On the other hand, the sensitivity values of the alternatives are significantly reduced (94% for physical service 6) in the RcB because their failures are largely masked by invoking the succeeding alternatives. Sensitivity analysis effectively quantifies the importance of each part within the system, which may be very useful for system engineers in determining reliability bottlenecks of the system and making further system improvement plans.Table 5
Sensitivity analysis for scn. no. 1.
Level 1
Level 2
State no.
Val.
Physical srv no.
Val.
1.
0.827
(1)
0.827
2.
1.104
(2)
1.104
3.
0.260
(3)
0.260
4.
0.417
(4)
0.417
5.
0.417
(6)
0.417
6.
0.391
(10)
0.391Table 6
Sensitivity analysis for scn. no. 2.
Level 1
Level 2
State no.
Val.
Physical srv no.
Val.
1.
0.871
(1)
0.871
2.
1.162
(2)
1.162
3.
0.273
(3)
0.273
4.
0.465
(4)
0.465
5.
0.434
(5)
0.485
(6)
0.023
(7)
0.012
(8)
0.007
(9)
0.485
6.
0.436
(10)
0.436Figure 4
Component sensitivities. The main figure shows component sensitivity comparisons of scenarios no. 1 and no. 2. The subfigure shows the RcB internal sensitivities of the payment service in scenario no. 2.


## 4.1. Reliability Modeling

Applying the Steps in Section3.2, one can derive the transition matrix for scenario no. 1(25)
M
scn
1
=
[
0
0
.
99
0
0
0
0
0
0
0
0
.
242
0
.
38
0
0
0
.
339
0
0
.
97
0
0
0
0
0
0
0
0
0
0
.
892
0
0
0
0
0
0
0
0
.
892
0
0
0
0
0
0
0
0
.
97
0
0
0
0
0
0
0
]
and the transition matrix for scenario no. 2
(26)
M
scn
2


=


[
0
0
.
99
0
0
0
0
0
0
0
0
0
0
0
0
.
242
0
.
38
0
0
0
0
0
0
0.339
0
0
.
97
0
0
0
0
0
0
0
0
0
0
0
0
0
0.91
0
0
0
0
0
0
0
0
0
0
0
0.99
0
0
0
0
0
0
0
0
0
0
0
0.124
0.874
0
0
0
0
0
0
0
0
0
0
0.845
0
.
152
0
0
0
0
0
0
0
0
0
0
0
0.99
0
0
0
0
0
0
0
0
0.783
0
0
0
0
0
0
0
0
0
0
0
0
0
0.97
0
0
0
0
0
0
0
0
0
0
0
]
.Ten simulations of the system where each run contained 100,000 system calls were executed for each scenario. Comparing the experimental results and the values from the proposed model in Table4. In Table 4, the 1st row is the reliability estimation by the proposed model, the 2nd row is from the manual probability computation as explained in Appendix C, the 3rd row contains the statistics of simulation, and the last row is the relative error between the simulation results and the theoretical values in the 3rd row. It can be seen that the proposed SOA reliability model has identical values to those computed from probability theory, and the average simulation results are very close to the theoretical values. Comparing the two scenarios, it can also be seen that the introduction of RcB in the payment service (abstract service 5) improves the local service reliability by 11.36% (Considering both the service invocation and the service execution, the payment service reliability is 0.874 in scenario no. 1 and is 0.973 in scenario no. 2.) and improves the overall system reliability by 5.2%.Table 4
System reliability model validation results.
Scenario no. 1
Scenario no. 2
SOA reliability model
0.819
0.862
Probability computation
0.819
0.862
Simulation results
0.819
0.865
Simulation relative error
0.00%
0.35%It is also clarified that the 0.00% simulation relative error in Table4 is only the statistical value for certain scenario in certain precision level and should not be otherwise interpreted as 100% simulation accuracy.


## 4.2. Sensitivity Analysis

Results of the sensitivity analysis for both scenarios are displayed in Tables5 and 6, respectively. The sensitivity value of a state/service indicates its impact on the overall system reliability. It could also be observed from Figure 4 that abstract services with higher execution probabilities (such as abstract services 1 and 2) have higher sensitivity values. In abstract service 5 in scenario no. 2, it is also noted that the executive service and the AT service actually dominate the reliability of RcB, as one observes the sub-graph within Figure 4. Such result is reasonable since both services are always on the execution paths for all the alternatives. On the other hand, the sensitivity values of the alternatives are significantly reduced (94% for physical service 6) in the RcB because their failures are largely masked by invoking the succeeding alternatives. Sensitivity analysis effectively quantifies the importance of each part within the system, which may be very useful for system engineers in determining reliability bottlenecks of the system and making further system improvement plans.Table 5
Sensitivity analysis for scn. no. 1.
Level 1
Level 2
State no.
Val.
Physical srv no.
Val.
1.
0.827
(1)
0.827
2.
1.104
(2)
1.104
3.
0.260
(3)
0.260
4.
0.417
(4)
0.417
5.
0.417
(6)
0.417
6.
0.391
(10)
0.391Table 6
Sensitivity analysis for scn. no. 2.
Level 1
Level 2
State no.
Val.
Physical srv no.
Val.
1.
0.871
(1)
0.871
2.
1.162
(2)
1.162
3.
0.273
(3)
0.273
4.
0.465
(4)
0.465
5.
0.434
(5)
0.485
(6)
0.023
(7)
0.012
(8)
0.007
(9)
0.485
6.
0.436
(10)
0.436Figure 4
Component sensitivities. The main figure shows component sensitivity comparisons of scenarios no. 1 and no. 2. The subfigure shows the RcB internal sensitivities of the payment service in scenario no. 2.


## 5. Threats to Validity

Users who adopt the proposed SOA reliability model and sensitivity analysis technique should be aware of the potential threats to validity. Factors limiting the generalizability of the results include the following.Reliability Model. The proposed reliability model assumes independent failure occurrences between services. The Markovian property [30] also implies history independent system behavior. Possible bias should be taken into account when these assumptions do not hold in application.Sensitivity Analysis. The derivative-based sensitivity analysis adopted in this paper is efficient to calculate sensitivity values at the current computing point. The major limitation of such technique lies on the narrow parameter input space, and higher-order sensitivity indices are also not explored [31]. Care must be taken when there are uncertainties of model inputs or when there is strong interaction/dependency between the constituent services within the SOA system.Subject Systems. The experiments are performed on a set of scenarios based on real-world project presented in [29], and the settings of service reliability, link reliability, and transmission probability are constructed in this research. To the best of our knowledge, no public reliability data source for SOA systems is available. Researches on SOA reliability modeling such as [4, 8, 20, 25] generated their own scenarios for validation either by simulation or by operating their own example systems. Therefore, care must be taken in applying the model to other subject systems or system configuration.Performance Evaluation. The restrictions of the proposed model also apply in the experiments. Additionally, failures are generated binomially in the simulation. The evaluation results are valid only with respect to the conditions and system scales in the experiments.


## 6. Conclusions

This paper presents a Markov-based system reliability model for SOA systems. Starting from the workflow specification of an SOA system, we have shown how to map each part in the workflow to the model. The proposed reliability model also considers the internal mechanisms of three well-known fault-tolerance strategies such that the execute node, the AT/DM node, the alternatives, and the interactions between them are well reflected in the reliability model. Sensitivity analysis at two grain levels is also incorporated in this paper, which enables the SOA system engineers to identify the reliability critical blocks or internal services effectively and efficiently. Experimental results show that the proposed SOA model and methods give very close results to theoretical and simulation values and how the sensitivity analysis quantizes the effects of system structure as well as fault tolerance on the overall reliability.



---

*Source: 160608-2014-01-09.xml*