# The Assessment and Foundation of Bell-Shaped  Testability Growth Effort Functions Dependent System  Testability Growth Models Based on NHPP

Authors: Tian-Mei Li; Cong-Qi Xu; Jing Qiu; Guan-Jun Liu; Qi Zhang
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

This paper investigates a type of STGM (system testability growth model) based on the nonhomogeneous Poisson process which incorporates TGEF (testability growth effort function). First, we analyze the process of TGT (testability growth test) for equipment, which shows that the TGT can be divided into two committed steps: make the unit under test be in broken condition to identify TDL (testability design limitation) and remove the TDL. We consider that the amount of TGF (testability growth effort) spent on identifying TDL is a crucial issue which decides the shape of testability growth curve and that the TGF increases firstly and then decreases at different rates in the whole life cycle. Furthermore, we incorporate five TGEFs: an Exponential curve, a Rayleigh curve, a logistic curve, a delayed S-shape curve or an inflected S-shaped curve which are collectively referred to as Bell-shaped TGEFs into STGM. Results from applications to a real data set of a stable tracking platform are analyzed and evaluated in testability prediction capability and show that the Bell-shaped function can be expressed as a TGF curve and that the logistic TGEF dependent STGM gives better predictions based on the real data set.


---

## Body

## 1. Introduction

Generally, system testability is defined as the probability of fault detection or isolation for a specific period of time in a specified environment, which is quantified by various testability indexes, such as FDR, FIR, and FAR [1–3]. Over the last several decades, many testability test technologies, especially testability demonstration test for equipment, have been researched [4–15]. However, some research work indicates that the result of testability demonstration test departures from the actual value of testability greatly and makes the result of testability demonstration test unauthentic [4]. The root cause for this is that the vacancy of TGT in tracking and measuring the growth of testability as equipment is being developed.In general, any fault diagnosis system similar to reliability for large-scale and complicated equipment like a missile may be premature when the fault diagnosis system has been developed, in which many TDLs hide, such as nonexpectant failure, test vacancy, ambiguity group, fuzzy point, improper resistance tolerance, or threshold. And then, the designers analyze and judge the root cause of TDL and further track to the design of equipment testability, like UUT, redesign of test equipment or interface equipment, circuit of BIT, fault diagnosis software, and test program of ATE. Thus, almost all the fault diagnosis systems for large-scale and complicated equipment need a certain extent of time period to develop TGT to identify and remove the TDLs to attain the expected value under contract.The TGT is an important and expensive part of equipment testability development. A TDL in UUT leads to an output that differs from specifications and requirements. The aim of TGT is to identify and remove the TDL and further to increase the fault detection/isolation probability that a designed testability will work as intended in the hands of the designers. The testability growth test phase aims at identifying and removing the TDLs. Typically, TGT is more constructive and has high confidence level for system testability rather than the testability demonstration test at the acceptance stage only.A common approach for measuring system testability is by using an analytic model whose parameters are generally estimated from available data on fault detection/isolation. In this paper, the analytic model is referred to by us as STGM. A STGM provides a mathematical relationship between the number of TDLs removed and the test time during the whole life cycle and generally is used as a tool to estimate and predict the progress of system testability. At the same time, the STGM can be used for planning and controlling all test resources during development and can assure us about the testability of equipment. There is an extensive body of literature emphasizing the importance of TGT, but no depth of research is done on STGM [3, 8].In the context of TGT, the basic objective is to identify the TDLs and remove them one by one. Therefore, the process of TGT contains not only a TDL identification process but also a TDL removal process and that the key issues are the TGE which are consumed to identify the TDL and the effectiveness of TDL removal, respectively. In general, the TGE can be represented as man-hour, TGT cost, the times of fault injection, and so forth. The functions that describe how a TGE is distributed over the TGT phase are referred to by us as TGEF. The shape of the observed testability growth curve depends strongly on the time distribution of the TGE. Due to the randomness of TGT and consider the designer’s capability and familiarity to UUT synchronously, the other key issue (i.e., the effectiveness of TDL removal) can be referred to by us as a constant TDL identification rate and a constant TDL removal rate, respectively.There is an extensive body of literature [16–40] on software reliability growth model describing the relationship between the test time and the amount of test-effort expended during that time, in which the test effort was often described by the traditional Exponential [19, 21], Rayleigh [16, 19, 21], logistic [29, 30, 37, 38], delayed-shaped [19, 39] or inflected S-shaped [18, 39] curves. Thus, to address the issue of TGEF, this paper will use the above five test effort curves which are collectively referred to as Bell-shaped TGEFs to describe the relationship between the test time and the amount of TGE expended during that time by analyzing the consumption rule of TGE. Sometimes the TGE can be represented as the number of faults injected or occurred naturally instead of man-hours or TGT cost. Similarly, to address the issue of effectiveness of TDL removal, we assume that the TDL identification rate and the TDL removal rate are constant.In this paper, the process of TGT is decomposed into fault injection or occurrence, TDL identification, and TDL removal processes which are all based on NHPP. We show how to integrate time dependent TGEF, constant TDL identification rate, and constant TDL removal rate into STGM to find the framework of STGM. We further pay main attention on the consumption pattern of TGE based on fault injection collection of fault occurring naturally and then discuss how to integrate five Bell-shaped TGEFs into STGM. A method to estimate the model parameter is provided. Experimental results from a stable tracking platform are analyzed and the five STGMs based on the above five Bell-shaped TGEFs are compared with each other to show which STGM can give better prediction.The remainder of the paper is organized as follows. In Section2, we found a framework of STGM considering TGEF. The fault occurrence, TDL identification and TDL removal process based on NHPP are also described and analyzed in this section. In Section 3, we pay main attention to analyzing the consumption rule of TGEF which is represented by fault injection or collection of fault occurred naturally at design & development stage and trial & in-service stage, respectively. At the same time, we show how to incorporate the five Bell-shaped TGEFs into STGM. Parameters of the proposed STGMs as estimated by the method of LSE are discussed in Section 4. Finally, Section 5 discusses the goodness of the above five STGMs on prediction with the application of these models to a real data set.


## 2. A General Framework of STGM Based on NHPP

In general, the precondition for testability growth test is that the UUT is in failed condition. That is to say, there is one or more faults occurred in the UUT. Under the precondition, TGT is the process of running a testability diagnostic program and capturing the fault detection and isolation data to identify the TDLs and remove them. In other words, the TGT aims at identifying the TDLs and removing them one by one under the precondition of fault occurred. Therefore, the process of removing TDLs can roughly be divided into three steps: fault occurrence, TDL identification, and TDL removal. At the same time, each of the above three steps can be described effectively as a counting process based on NHPP.In practice, the most important factor which affects the STGM’s evaluation and predication accuracy is TGE. TGE is the cumulative testability growth effort consumed in TGT which can be measured by the number of fault injection, test cost, staff, and so on. The consumed TGE indicates how TDLs are identified effectively in the UUT and can be modeled by different curves considering the consumption pattern in TGT. In this section, a framework for foundation of STGM with TGEF is proposed.


### 2.1. Fault Occurrence, TDL Identification, and Removal

TDL identification may be realized under the condition that the UUT must be broken. Once the UUT is in broken condition which usually can be realized by fault injection or collection information of fault occurred naturally, the design of testability begins to run to identify the TDL and analyze the root cause of the TDL. When the specific root cause is recognized, the designers can remove the TDL accordingly.We can use the number of residual TDLs as a measure of system testability. The great the number of TDLs, the lower of testability level would be. So, the process of TGT is the process to identify and remove the TDLs existing in the UUT. In other words, the process of TGT is to identify and remove the TDL one by one. So, the process of TGT can be described by Poisson process considering the randomness of fault injection, fault detection and isolation, TDL identification, and removal etc. Here, we give an apt shown in Figure1 to describe the process of TGT.Figure 1
An apt sketch map of TGT.The fives processes given by Figure1 about TGT and fault detection and isolation are (a) the process of fault occurred naturally or injected of the UUT, (c) the process of identifying that TDL exists in the UUT, (d) the process of removing TDL exist in the UUT, (b) and (e) depict the fault detection and isolation before and after the TGT, respectively.In (a),f
1
,
f
2
,
…
,
f
i
,
…
,
f
M is the failure mode set which is obtained by FMEA and M is the total number of failures in the UUT. With the progress of TGT, the time-dependent failure phenomena which can be realized by fault injection or collection of fault information occurred naturally at random time serial t
1
,
t
2
,
…
,
t
i
,
…
,
t
M.In (b), When some failures occurred at random time serialt
1
,
t
2
,
…
,
t
i
,
…
,
t
M, the system testability is activated to detect and isolate the failures accordingly. If a failure cannot be detected and isolated successfully, a TDL is identified. Practically, some failures may be easy to detect and isolate, and some others are not. For example, f
1 and f
4 cannot be detected and isolated correctly, so two TDLs are identified and are noted as T
D
L
1 and T
D
L
2, respectively.With the progress of TDL identification, increasing TDLs are identified gradually; thus we have the process (c) which describes the counting process of TDL identified.In (d), the designers analyze the root cause of the TDL and try their best to modify the design of testability and remove the TDL accordingly. But, not all the TDLs identified can be removed successfully because of designer’s familiarity to UUT. For example, theT
D
L
j presented in process (c) still cannot be removed successfully after an infinite amount of test time and test effort consumed. All the TDLs which can be identified but cannot be removed successfully form the process (d). In TGT, we should pay more attention to the expected average number of TDLs identified and removed successfully.By comparison with process (b), process (e) gives the growth effect of fault detection and isolation after TGT apparently.


### 2.2. NHPP in TGT

There is an extensive body of literature [12, 32] on NHPP used for description of counting process effectively, such as the probabilistic failure process in software and equipment. In this paper, the NHPP is introduced to describe the TDL detection and removal process. N
(
t
) is a nonnegative integer and a time-dependent nondecreasing function, which describes the cumulative number of TDLs identified and removed up to time t. If s
<
t, N
(
s
)
-
N
(
t
) is the counts of the TDL identified and removed in the interval (
s
,
t
).The following characters ofN
(
t
) can be derived from the assumption that “the probability that two or more faults occurred synchronously is wee and can be neglected in testability engineering” [3].(1)
N
(
0
)
=
0.
(2)
The process has independent increment.
(3)
lim
h
→
0
⁡
P
N
(
t
+
h
)
-
N
(
t
)
=
1
=
λ
(
t
)
h
+
o
(
h
).
(4)
lim
h
→
0
⁡
P
N
(
t
+
h
)
-
N
(
t
)
≥
2
=
o
(
h
).Thus, the process of{
N
(
t
)
,
t
≥
0
} follows a Poisson process with a parameter λ
(
t
), where λ
(
t
) is the TDL intensity of m
r
(
t
) (i.e., number of TDLs identified and removed correctly per unit time). The counting process with a constant λ
(
t
) has a smooth increment, which is named homogeneous Poisson process. Otherwise, the counting process with a time-dependent λ
(
t
) has a fluctuant increment, which is named nonhomogeneous Poisson process. HPP can be considered as a special case of NHPP, and NHPP is an extend case of HPP.Makem
r
(
t
)
=
E
[
N
(
t
)
]; then m
r
(
t
) is the expected mean number of TDLs identified and removed successfully in time (
0
,
t
], so m
r
(
t
) can be expressed as(1)
m
r
t
=
∫
0
t
λ
u
d
u
.So(2)
λ
t
=
m
r
′
t
=
l
i
m
Δ
t
→
0
E
N
t
+
Δ
t
-
N
t
Δ
t
.To anyt
≥
0, s
≥
0, N
(
t
+
s
)
-
N
(
t
) follows the Poisson distribution with parameters m
r
(
t
+
s
)
-
m
r
(
t
), so we have(3)
P
N
t
+
s
-
N
t
=
n
=
e
x
p
-
m
r
t
+
s
-
m
r
t
·
m
r
t
+
s
-
m
r
t
n
n
!
.We can use either the number of TDLs identified and removed or the number of remaining TDLs as a measure of testability quality. Here, we use the number of TDLs identified and removed successfully (i.e.,m
r
(
t
)) as a measure of system testability quality. Parallel to the analysis of Figure 1, m
r
(
t
) is critical for both the mean and variance of testability estimation and prediction. Hence, calculation of m
r
(
t
) is our main focus in the following parts of this paper.


### 2.3. A General Framework of STGM Considering TGEF

A STGM provides a mathematical relationship between the number of TDLs identified and removed successfully (m
r
(
t
)) from the UUT and the test time. STGM can be used as a tool to estimate and predict the progress of system testability. A TDL inherent in the UUT leads to an output that the fault occurred or injected cannot be detected and isolated correctly. The TGT aims at identifying these TDLs and removing them. At any time during the TGT phase, the basic process of TGT includes four steps which are shown in Figure 2.Figure 2
The basic process of TGT.In Figure2, T
e is the target testability level which has been fixed at the beginning of TGT. But it is frequently realized that this target may not be achievable for a number of reasons, like inadequacy of TGE or inefficiency of the test team. So, TGT is a repetitive work of “test-identification-correction-test” generally until the conditional expression T
(
t
0
)
<
T
e is satisfied. T
(
t
0
) is the estimation value of testability at time t
0.Firstly, we need to estimateT
(
t
0
) at time t
0. From the above analysis, the main cause leading to T
(
t
0
)
<
T
e is the TDL existing in the UUT. In this paper, we assume that the removal of a TDL from the UUT includes two phases. In the first phase the TDL identification team, primarily consisting of test personnel, identifies a TDL by making the UUT be in failed condition. Concretely, the identification team can either collect the fault occurred naturally or injected to verify the level of testability. During the TGT phase, much testability growth effort is consumed. The consumed TGE indicates how the TDLs are identified effectively in the UUT and can be modeled by different curves. Actually, the system testability is highly related to the amount of testability growth effort expenditures spent on identifying TDLs. In this step, the work in the process of TDL identification is calculated by W
(
t
), such as man-hour, TGT cost, and the times of fault injection. W
(
t
) signifies the cumulative TGE consumed in identifying the TDLs up to the given time t. Then another team, primarily consisting of designers, analyzes the reason for the TDL and modifies the design of testability to remove the TDL by redesigning of UUT or ATE and the corresponding interface, optimizing the circuit of BIT, debugging the diagnostic software and rewriting the diagnostic program of ATE, etc. All the work about removing the TDLs will be done in the third step of Figure 2 by the designers of system testability, which is described by a constant b
2. Finally, all the teams evaluate the value of system testability after TGT noted as T
(
t
) and give a test conclusion that if T
(
t
)
≥
T
e, the TGT can be stopped, else the TGT will be continued until the condition expression T
(
t
)
≥
T
e is satisfied. Apparently, testability can be enhanced if TGT is done for a prolonged period, that is, T
e
≥
T
(
t
)
>
T
(
t
0
).The STGM developed below is based on the following assumptions.(1)
The TDL identification and its removal processes all follow NHPP.
(2)
The failure of fault detection/isolation at random times is caused by the TDLs which are inherent in the UUT.
(3)
All TDLs are mutually independent and have the same contribution to the failure of fault detection/isolation occurred in the system.
(4)
There is a one-to-one correspondence between a failure and the corresponding test.
(5)
Two phases can be observed within the TGT process: TDL identification and TDL removal. There is no time lag between the TDL identification and its removal. Whenever a TDL is identified, the goal of TDL removal is to analyze the root cause of the TDL and to remove it. At the same time, no new TDLs are introduced.
(6)
The mean number of TDL identified in the time interval[
t
,
t
+
Δ
t
] by the current TGE is proportional to the mean number of TDLs unidentified in the UUT. The mean number of TDL identified and removed in the time interval [
t
,
t
+
Δ
t
] by the current testability growth effort is proportional to the mean number of TDLs uncorrected in the UUT. The proportions are constants and expressed by b
1 and b
2, respectively.
(7)
The consumption of TGE is modeled byW
(
t
).Letm
i
(
t
) be the MVF of the expected number of TDL identified and let m
r
(
t
) be the MVF of the expected number of TDLs identified and removed in time (
0
,
t
]. Then, according to the above assumptions, we describe the STGM based on TGEF as follows:(4)
d
m
i
t
d
t
×
1
w
t
=
b
1
a
-
m
i
t
,
d
m
r
t
d
t
×
1
w
t
=
b
2
m
i
t
-
m
r
t
.Note that here we assume b
1
≠
b
2. Solving (4) under the boundary condition m
i
(
0
)
=
m
r
(
0
)
=
0 we have(5)
m
i
t
=
a
×
1
-
exp
⁡
-
b
1
W
∗
t
,
(6)
m
r
t
=
a

×

1
-
b
1
exp
⁡
-
b
2
W
∗
t

-

b
2
exp
⁡
-
b
1
W
∗
t
b
1
-
b
2
,
(7)
a
r
e
m
a
i
n
i
n
g
=
a
-
m
r
(
∞
)
=
a
b
1
exp
⁡
-
b
2
W
∗
∞
-
b
2
exp
⁡
-
b
1
W
∗
∞
b
1
-
b
2
.Due to the space limitations, here we only propose the testability growth model of FDR in general. FDR is defined as the capability to detect fault occurred in the UUT, and FDR is also used to describe the average detection probability to the failure mode set which is gained from FMEA. The mathematical model of FDR can be formulated as [1–3](8)
F
D
R
=
N
D
M
,where M is the total number of failures. N
D is the number of failures which can be detected accurately by the system testability design.According to the above assumption (4), we can have that the number of failures which can be detected accurately by the system testability design is(9)
N
D
=
M
-
a
t
=
M
-
a
-
m
r
t
.Substituting it into (8), we obtain(10)
q
t
=
M
-
a
-
m
r
t
M
.Substitute m
r
(
t
) (in (6)) into (10); the q
(
t
) can be rewritten as(11)
q
t
=
M
-
a
b
1
exp
⁡
-
b
2
W
∗
t
-
b
2
exp
⁡
-
b
1
W
∗
t
b
1
-
b
2
-
1
M
-
a
=
a
i
i
×
b
1
-
b
2
-
1
M
-
1
.Depending on how elaborate a model one wishes to obtain is, one can usew
(
t
) to yield more complex or less complex analytic solutions for q
(
t
). Different w
(
t
) reflect various assumption patterns of TGE in TGT.Equation (11) is the modeling framework which considers TGEF into the NHPP STGM. At the same time, (11) can provide designers or testers with an estimation of the time needed to reach a given level of FDR. And that, it may also be used to determine the appropriate ending time for TGT, and can provide useful information for making decision on turn phase.


## 2.1. Fault Occurrence, TDL Identification, and Removal

TDL identification may be realized under the condition that the UUT must be broken. Once the UUT is in broken condition which usually can be realized by fault injection or collection information of fault occurred naturally, the design of testability begins to run to identify the TDL and analyze the root cause of the TDL. When the specific root cause is recognized, the designers can remove the TDL accordingly.We can use the number of residual TDLs as a measure of system testability. The great the number of TDLs, the lower of testability level would be. So, the process of TGT is the process to identify and remove the TDLs existing in the UUT. In other words, the process of TGT is to identify and remove the TDL one by one. So, the process of TGT can be described by Poisson process considering the randomness of fault injection, fault detection and isolation, TDL identification, and removal etc. Here, we give an apt shown in Figure1 to describe the process of TGT.Figure 1
An apt sketch map of TGT.The fives processes given by Figure1 about TGT and fault detection and isolation are (a) the process of fault occurred naturally or injected of the UUT, (c) the process of identifying that TDL exists in the UUT, (d) the process of removing TDL exist in the UUT, (b) and (e) depict the fault detection and isolation before and after the TGT, respectively.In (a),f
1
,
f
2
,
…
,
f
i
,
…
,
f
M is the failure mode set which is obtained by FMEA and M is the total number of failures in the UUT. With the progress of TGT, the time-dependent failure phenomena which can be realized by fault injection or collection of fault information occurred naturally at random time serial t
1
,
t
2
,
…
,
t
i
,
…
,
t
M.In (b), When some failures occurred at random time serialt
1
,
t
2
,
…
,
t
i
,
…
,
t
M, the system testability is activated to detect and isolate the failures accordingly. If a failure cannot be detected and isolated successfully, a TDL is identified. Practically, some failures may be easy to detect and isolate, and some others are not. For example, f
1 and f
4 cannot be detected and isolated correctly, so two TDLs are identified and are noted as T
D
L
1 and T
D
L
2, respectively.With the progress of TDL identification, increasing TDLs are identified gradually; thus we have the process (c) which describes the counting process of TDL identified.In (d), the designers analyze the root cause of the TDL and try their best to modify the design of testability and remove the TDL accordingly. But, not all the TDLs identified can be removed successfully because of designer’s familiarity to UUT. For example, theT
D
L
j presented in process (c) still cannot be removed successfully after an infinite amount of test time and test effort consumed. All the TDLs which can be identified but cannot be removed successfully form the process (d). In TGT, we should pay more attention to the expected average number of TDLs identified and removed successfully.By comparison with process (b), process (e) gives the growth effect of fault detection and isolation after TGT apparently.


## 2.2. NHPP in TGT

There is an extensive body of literature [12, 32] on NHPP used for description of counting process effectively, such as the probabilistic failure process in software and equipment. In this paper, the NHPP is introduced to describe the TDL detection and removal process. N
(
t
) is a nonnegative integer and a time-dependent nondecreasing function, which describes the cumulative number of TDLs identified and removed up to time t. If s
<
t, N
(
s
)
-
N
(
t
) is the counts of the TDL identified and removed in the interval (
s
,
t
).The following characters ofN
(
t
) can be derived from the assumption that “the probability that two or more faults occurred synchronously is wee and can be neglected in testability engineering” [3].(1)
N
(
0
)
=
0.
(2)
The process has independent increment.
(3)
lim
h
→
0
⁡
P
N
(
t
+
h
)
-
N
(
t
)
=
1
=
λ
(
t
)
h
+
o
(
h
).
(4)
lim
h
→
0
⁡
P
N
(
t
+
h
)
-
N
(
t
)
≥
2
=
o
(
h
).Thus, the process of{
N
(
t
)
,
t
≥
0
} follows a Poisson process with a parameter λ
(
t
), where λ
(
t
) is the TDL intensity of m
r
(
t
) (i.e., number of TDLs identified and removed correctly per unit time). The counting process with a constant λ
(
t
) has a smooth increment, which is named homogeneous Poisson process. Otherwise, the counting process with a time-dependent λ
(
t
) has a fluctuant increment, which is named nonhomogeneous Poisson process. HPP can be considered as a special case of NHPP, and NHPP is an extend case of HPP.Makem
r
(
t
)
=
E
[
N
(
t
)
]; then m
r
(
t
) is the expected mean number of TDLs identified and removed successfully in time (
0
,
t
], so m
r
(
t
) can be expressed as(1)
m
r
t
=
∫
0
t
λ
u
d
u
.So(2)
λ
t
=
m
r
′
t
=
l
i
m
Δ
t
→
0
E
N
t
+
Δ
t
-
N
t
Δ
t
.To anyt
≥
0, s
≥
0, N
(
t
+
s
)
-
N
(
t
) follows the Poisson distribution with parameters m
r
(
t
+
s
)
-
m
r
(
t
), so we have(3)
P
N
t
+
s
-
N
t
=
n
=
e
x
p
-
m
r
t
+
s
-
m
r
t
·
m
r
t
+
s
-
m
r
t
n
n
!
.We can use either the number of TDLs identified and removed or the number of remaining TDLs as a measure of testability quality. Here, we use the number of TDLs identified and removed successfully (i.e.,m
r
(
t
)) as a measure of system testability quality. Parallel to the analysis of Figure 1, m
r
(
t
) is critical for both the mean and variance of testability estimation and prediction. Hence, calculation of m
r
(
t
) is our main focus in the following parts of this paper.


## 2.3. A General Framework of STGM Considering TGEF

A STGM provides a mathematical relationship between the number of TDLs identified and removed successfully (m
r
(
t
)) from the UUT and the test time. STGM can be used as a tool to estimate and predict the progress of system testability. A TDL inherent in the UUT leads to an output that the fault occurred or injected cannot be detected and isolated correctly. The TGT aims at identifying these TDLs and removing them. At any time during the TGT phase, the basic process of TGT includes four steps which are shown in Figure 2.Figure 2
The basic process of TGT.In Figure2, T
e is the target testability level which has been fixed at the beginning of TGT. But it is frequently realized that this target may not be achievable for a number of reasons, like inadequacy of TGE or inefficiency of the test team. So, TGT is a repetitive work of “test-identification-correction-test” generally until the conditional expression T
(
t
0
)
<
T
e is satisfied. T
(
t
0
) is the estimation value of testability at time t
0.Firstly, we need to estimateT
(
t
0
) at time t
0. From the above analysis, the main cause leading to T
(
t
0
)
<
T
e is the TDL existing in the UUT. In this paper, we assume that the removal of a TDL from the UUT includes two phases. In the first phase the TDL identification team, primarily consisting of test personnel, identifies a TDL by making the UUT be in failed condition. Concretely, the identification team can either collect the fault occurred naturally or injected to verify the level of testability. During the TGT phase, much testability growth effort is consumed. The consumed TGE indicates how the TDLs are identified effectively in the UUT and can be modeled by different curves. Actually, the system testability is highly related to the amount of testability growth effort expenditures spent on identifying TDLs. In this step, the work in the process of TDL identification is calculated by W
(
t
), such as man-hour, TGT cost, and the times of fault injection. W
(
t
) signifies the cumulative TGE consumed in identifying the TDLs up to the given time t. Then another team, primarily consisting of designers, analyzes the reason for the TDL and modifies the design of testability to remove the TDL by redesigning of UUT or ATE and the corresponding interface, optimizing the circuit of BIT, debugging the diagnostic software and rewriting the diagnostic program of ATE, etc. All the work about removing the TDLs will be done in the third step of Figure 2 by the designers of system testability, which is described by a constant b
2. Finally, all the teams evaluate the value of system testability after TGT noted as T
(
t
) and give a test conclusion that if T
(
t
)
≥
T
e, the TGT can be stopped, else the TGT will be continued until the condition expression T
(
t
)
≥
T
e is satisfied. Apparently, testability can be enhanced if TGT is done for a prolonged period, that is, T
e
≥
T
(
t
)
>
T
(
t
0
).The STGM developed below is based on the following assumptions.(1)
The TDL identification and its removal processes all follow NHPP.
(2)
The failure of fault detection/isolation at random times is caused by the TDLs which are inherent in the UUT.
(3)
All TDLs are mutually independent and have the same contribution to the failure of fault detection/isolation occurred in the system.
(4)
There is a one-to-one correspondence between a failure and the corresponding test.
(5)
Two phases can be observed within the TGT process: TDL identification and TDL removal. There is no time lag between the TDL identification and its removal. Whenever a TDL is identified, the goal of TDL removal is to analyze the root cause of the TDL and to remove it. At the same time, no new TDLs are introduced.
(6)
The mean number of TDL identified in the time interval[
t
,
t
+
Δ
t
] by the current TGE is proportional to the mean number of TDLs unidentified in the UUT. The mean number of TDL identified and removed in the time interval [
t
,
t
+
Δ
t
] by the current testability growth effort is proportional to the mean number of TDLs uncorrected in the UUT. The proportions are constants and expressed by b
1 and b
2, respectively.
(7)
The consumption of TGE is modeled byW
(
t
).Letm
i
(
t
) be the MVF of the expected number of TDL identified and let m
r
(
t
) be the MVF of the expected number of TDLs identified and removed in time (
0
,
t
]. Then, according to the above assumptions, we describe the STGM based on TGEF as follows:(4)
d
m
i
t
d
t
×
1
w
t
=
b
1
a
-
m
i
t
,
d
m
r
t
d
t
×
1
w
t
=
b
2
m
i
t
-
m
r
t
.Note that here we assume b
1
≠
b
2. Solving (4) under the boundary condition m
i
(
0
)
=
m
r
(
0
)
=
0 we have(5)
m
i
t
=
a
×
1
-
exp
⁡
-
b
1
W
∗
t
,
(6)
m
r
t
=
a

×

1
-
b
1
exp
⁡
-
b
2
W
∗
t

-

b
2
exp
⁡
-
b
1
W
∗
t
b
1
-
b
2
,
(7)
a
r
e
m
a
i
n
i
n
g
=
a
-
m
r
(
∞
)
=
a
b
1
exp
⁡
-
b
2
W
∗
∞
-
b
2
exp
⁡
-
b
1
W
∗
∞
b
1
-
b
2
.Due to the space limitations, here we only propose the testability growth model of FDR in general. FDR is defined as the capability to detect fault occurred in the UUT, and FDR is also used to describe the average detection probability to the failure mode set which is gained from FMEA. The mathematical model of FDR can be formulated as [1–3](8)
F
D
R
=
N
D
M
,where M is the total number of failures. N
D is the number of failures which can be detected accurately by the system testability design.According to the above assumption (4), we can have that the number of failures which can be detected accurately by the system testability design is(9)
N
D
=
M
-
a
t
=
M
-
a
-
m
r
t
.Substituting it into (8), we obtain(10)
q
t
=
M
-
a
-
m
r
t
M
.Substitute m
r
(
t
) (in (6)) into (10); the q
(
t
) can be rewritten as(11)
q
t
=
M
-
a
b
1
exp
⁡
-
b
2
W
∗
t
-
b
2
exp
⁡
-
b
1
W
∗
t
b
1
-
b
2
-
1
M
-
a
=
a
i
i
×
b
1
-
b
2
-
1
M
-
1
.Depending on how elaborate a model one wishes to obtain is, one can usew
(
t
) to yield more complex or less complex analytic solutions for q
(
t
). Different w
(
t
) reflect various assumption patterns of TGE in TGT.Equation (11) is the modeling framework which considers TGEF into the NHPP STGM. At the same time, (11) can provide designers or testers with an estimation of the time needed to reach a given level of FDR. And that, it may also be used to determine the appropriate ending time for TGT, and can provide useful information for making decision on turn phase.


## 3. STGM with Bell-Shaped TGEF

Actually, the system testability is highly related to the amount of testability growth effort expenditures spent on identifying the TDLs. In this section we develop some STGMs in which five Bell-shaped TGEFs: an Exponential TGEF, a Rayleigh TGEF, a logistic TGEF, a delayed S-shaped TGEF, and an inflected S-shaped TGEF are taken into account. The forms of TGEF are all discussed in the literature about software reliability growth model [16–39]. Here, we choose appropriate forms of TGEF in view of the TGT’s specialty.


### 3.1. Bell-Shaped TGEF

In TGT, one of the key factors is the TGE which can be represented as the number of man-hours, the times of fault injection, and the cost of TGT, and so forth. The functions that describe how an effort is distributed over the testability growth phase are referred to by us as TGEF. TGEF describes the relationship between the test time and the amount of TGE expended in the test time. Actually, fault injection is one of the best effective measures to make the UUT be in broken condition and to identify the inherent TDLs in the UUT further. There is an extensive body of literature on fault injection used for identifying TDL in testability test [1–3, 41–43]. Therefore, we will use the times of fault injection as a measurement of TGE, which has an advantage that it is very intuitive and that testability growth effort can be quantified exactly at the same time.In general, the whole life-cycle of equipment can be divided into three stages: demonstration stage, design & development stage, and trial & in-service stage. But the empirical analysis shows that the effective and credible stages to conduct TGT are design & development stage and trial & in-service stage considering the cost effectiveness in which fault injection and collection information for naturally occurred faults are done, respectively [3, 6]. At the same time, TGE are calculated by times of fault injection and the number of naturally occurred faults at design & development stage and trial & in-service stage, respectively.Actually, the more faults we inject or collect, the more confidence we obtain in the estimation of system testability. Unfortunately, TGT with redundant fault injections or a large number of failure data may lead to excessive cost and too much time consumption. It is impracticable with the constraint of test cycle and total cost.To give an accurate and reasonable description of TGE, we need to know not only how many faults need to inject at design & development stage but also how many failure data need to collect at trial & in-service stage. That is to say, we need to analyze the TGE’s consumption pattern in TGT by combining Figure3 which gives an abridged general view of testability growth process in the whole life-cycle.Figure 3
The testability growth chart in the whole life-cycle.In general, the TGT time can be represented by the serial number of testability growth test stage instead of the actual execution time. The assumptions which are shown in Figure3 and will be used for description of TGEF are as follows.(1)
The TGT stage is expressed as the time span[
0
,
t
e
] and is composed by the design & development stage time span [
0
,
t
s
] and the trail & in-service stage time span [
t
s
,
t
e
], respectively.
(2)
The times of TGT based on fault injection at design & development stage arem, and the maximum number of fault injection at the design & development stage is marked as N
D
D. At the design & development stage, we can observe n data pairs in the form (
n
k
,
f
k
), 1
≤
k
≤
m, n
k is the number of fault injection at the kth TGT stage, and f
k is the number of TDLs identified successfully at the kth TGT stage.
(3)
After a period of time especially themth TGT at design & development stage, the UUT is put into the trail & in-service stage at which the number of TGT based on fault occurred naturally is m
S and the maximum number of failures collecting is marked as N
S. In a similar way, the same data pairs in the form (
n
j
,
f
j
),  m
<
j
≤
m
S are observed by collecting the failure data occurred naturally. n
j is the number of failures by collecting the naturally occurred fault and f
j is the number of TDLs identified successfully at the jth TGT stage.
(4)
With the progress of TGT,q
i,  1
≤
i
≤
m
S is the true value of FDR at the ith TGT stage, and that q
1
<
⋯
q
i
⋯
<
q
m
S. The TGT can be stopped only when the Q
E is achieved, in which Q
E is the FDR’s desired value of TGT shown in Figure 3.Based on the above assumptions, we will analyze the TGE consumption pattern from two perspectives; one is fault injection based TGE consumption at design & development stage; the other is naturally occurred fault based TGE consumption at trail & in-service stage.At the beginning of the design & development stage, FDR level is relatively lower; hence, a certain number of TDLs will be identified as long as we inject less fault. In order to identify a certain number of TDLs at the design & development stage, the number of faults need to inject grow up with the level of FDR, which can be formulated as [1–3](12)
q
k
=
n
k
-
f
k
n
k
,
1
≤
k
≤
m
.For example, with reference to (12), the number of faults which need to be injected with the change of FDR level and number of TDLs, respectively, are shown in Figure 4. As seen from Figure 4, we find that at the same number of TDLs, the FDR level needs to achieve increase as time goes on, and the number of faults needs to inject increases accordingly. At the same level of FDR, the greater number of TDLs needs to identify, the greater number of faults needs to inject. Consequently, TGEF is a time-dependent increasing function at design & development stage.Figure 4
The number of faults needs to be injected with the change of FDR level and number of TDLs, respectively.Collection of failure which occurred naturally is also an effective measure to identify TDL When the UUT is put into the trail & in-service stage. Actually, the number of faults at trail & in-service stage is “small sample” under several constraints such as high reliability requirements or specified test cycle. Considering the growth of reliability, the expected number of faults at regular intervals will decrease gradually. That is to say, at trail & in-service stage, TGEF is a time-dependent decreasing function which has the similar variation tendency shown in Figure5.Figure 5
The diagrammatic drawing of TGEF at trail & in-service stage.In conclusion, at the whole TGT phase, TGEF increases firstly and then decreases at different rate. Based on the increase-decrease characteristic, we will use a kind of Bell-shaped function which increases firstly and then decreases to fit the practical growth rate of TGEF. Bell-shaped function, just as its name implies, draws a time-dependent curve like a bell which has a similar shape to the curve shown in Figure6.Figure 6
A schematic of Bell-shaped current TGE consumption.It should be noted that Figure6 is just a schematic used to depict the Bell-shaped function; the concrete functional expression depends on the actual TGE data set. Because actual testability growth effort data represent various expenditure patterns, sometimes the testability growth effort expenditures are difficult to describe only by a Bell-shaped curve. In this paper, the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF have been used to explain the testability growth effort, which can be derived from the assumption that “the efforts that govern the pace of identifying the TDL for the UUT increase firstly and then decrease gradually.”


#### 3.1.1. The Exponential TGEF

Yamada et al. [21] found that the TEF in software reliability growth could be described by a Weibull-type distribution with two cases: the Exponential curve and the Rayleigh curve.The Exponential TGEF over time period(
0
,
t
] can be expressed as(13)
W
E
t
=
N
1
1
-
exp
⁡
-
β
1
t
.The current TGE expenditure rate at timet is(14)
w
E
t
=
N
1
β
1
exp
⁡
-
β
1
t
,where W
E
(
t
)
=
∫
0
t
w
E
(
τ
)
d
τ.The Exponential curve is used for process that declines monotonically to an asymptote.Substitute theW
E
∗
(
t
) in (13) into (7); we obtain(15)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
E
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
E
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
1
-
b
2
exp
⁡
-
b
1
N
1
b
1
-
b
2
.


#### 3.1.2. The Rayleigh TGEF [16, 19, 21, 37]

Another alternative of the Weibull-type distribution is the Rayleigh curve. It has been empirically observed that software development projects follow a life-cycle pattern described by the Rayleigh curve.The Rayleigh TGEF over time period(
0
,
t
] can be expressed as(16)
W
R
t
=
N
2
1
-
exp
⁡
-
β
2
2
t
2
.The current TGE expenditure rate at timet is(17)
w
R
t
=
N
2
β
2
t
exp
⁡
-
β
2
2
t
2
,where W
R
(
t
)
=
∫
0
t
w
R
(
τ
)
d
τ.Therefore,w
R
(
t
) is a smooth Bell-shaped curve and reaches its maximum value at time(18)
t
m
a
x
=
1
β
2
.The Rayleigh curve first increases to a peak, and then decreases at a decelerating rate.Substitute theW
R
∗
(
t
) in (16) into (7), we obtain(19)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
R
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
2
-
b
2
exp
⁡
-
b
1
N
2
b
1
-
b
2
.


#### 3.1.3. The Logistic TGEF

Logistic TEF was originally proposed by Parr [16] about software reliability growth model. It exhibits similar behavior to the Rayleigh curve, except during the early part of the project. Huang et al. [28, 30, 37] proposed that a logistic testing effort function can be used instead of the Weibull-type curve to describe the test effort patterns during the software development process. In some two dozen projects studied in the Yourdon 1978–1980 project survey, the Logistic TEF appeared to be fairly accurate in describing the expended test efforts [44].The logistic TGEF over time period(
0
,
t
] can be expressed as(20)
W
L
t
=
N
3
1
+
A
exp
⁡
-
β
3
t
.The current TGE expenditure rate at timet is(21)
w
L
t
=
N
3
A
β
3
exp
⁡
-
β
3
t
1
+
A
exp
⁡
-
β
3
t
2
,where W
L
(
t
)
=
∫
0
t
w
L
(
τ
)
d
τ.Therefore,w
L
(
t
) is a smooth Bell-shaped function, and reaches its maximum value at time(22)
t
m
a
x
=
1
β
3
ln
⁡
A
.Substitute theW
L
∗
(
t
) in (20) into (7), we obtain(23)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
∞
-
b
2
exp
⁡
-
b
1
W
R
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
3
/
1
+
A
-
b
2
exp
⁡
-
b
1
N
3
/
1
+
A
b
1
-
b
2
.Ohba [17, 18] found that the test effort function in software reliability growth could be described by S-shaped distribution with two cases: the delayed S-shaped curve and the inflected S-shaped curve.


#### 3.1.4. The Delayed S-Shaped TGEF [17, 40]

The delayed S-shaped TGEF over time period(
0
,
t
] can be expressed as(24)
W
D
S
t
=
N
4
1
-
1
+
β
4
t
exp
⁡
-
β
4
t
.The current TGE expenditure rate at timet is(25)
w
D
S
t
=
N
4
β
4
2
t
exp
⁡
(
-
β
4
t
)
,where W
D
S
(
t
)
=
∫
0
t
w
D
S
(
τ
)
d
τ.Therefore,w
D
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(26)
t
m
a
x
=
1
β
4
.Substitute theW
D
S
∗
(
t
) in (24) into (7); we obtain(27)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
D
S
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
D
S
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
4
-
b
2
exp
⁡
-
b
1
N
4
b
1
-
b
2
.


#### 3.1.5. The Inflected S-Shaped TGEF [18, 40]

An inflected S-shaped TGEF over time period(
0
,
t
] can be expressed as(28)
W
I
S
t
=
N
5
1
-
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
.The current TGE expenditure rate at timet is(29)
w
I
S
t
=
N
5
β
5
1
+
φ
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
2
,where W
I
S
(
t
)
=
∫
0
t
w
I
S
(
τ
)
d
τ.Therefore,w
I
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(30)
t
m
a
x
=
1
β
5
ln
⁡
φ
.Substitute theW
I
S
∗
(
t
) in (28) into (7); we obtain(31)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
I
S
∗
∞
-
b
2
exp
⁡
-
b
1
W
I
S
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
5
-
b
2
exp
⁡
-
b
1
N
5
b
1
-
b
2
.With reference to the above five Bell-shaped TGEFs, the instantaneous TGEs which are shown in Figures7, 8, 9, 10, and 11 decrease ultimately during the testability growth life-cycle because the cumulative TGEs approach a finite limit. As shown from (15), (19), (23), (27), and (31), we find that not all the inherent TDLs in the UUT can be fully removed, even after a long TGT period because the total amount of TGE to be consumed during the TGT phase is limited to W
m
a
x. This assumption is reasonable because no system testability design company will spend infinite resources on TGT considering the restricted cost & development cycle.Figure 7
An Exponential curve with fixed parameters.Figure 8
A Rayleigh curve with fixed parameters.Figure 9
A logistic curve with fixed parameters.Figure 10
A delayed S-shaped curve with fixed parameters.Figure 11
An inflected S-shaped curve with fixed parameters.


### 3.2. Bell-Shaped TGEF Dependent STGM

In this section, we will use the above five TGEFs in the foundation of STGM. One of these can be substituted into (10) to have five kinds of STGM which are formulated as follows.


#### 3.2.1. EX-STGM

Substitute (13) into (11); the EX-STGM can be formulated as(32)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
1
1
-
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
M
-
a
=
a
W
i
i
-
b
2
exp
⁡
-
b
1
N
1
1
-
exp
⁡
-
β
1
t
M
-
a
=
a
W
i
i
×
b
1
-
b
2
-
1
M
-
1
.


#### 3.2.2. RA-STGM

Substitute (16) into (11); the RA-STGM can be formulated as(33)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
exp
⁡
-
b
1
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
×
b
1
-
b
2
-
1
M
-
1
.


#### 3.2.3. LO-STGM

Substitute (20) into (11); the LO-STGM can be formulated as(34)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
3
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
w
i
-
b
2
exp
⁡
-
b
1
N
3
1
A
exp
⁡
(
-
α
t
)
=
M
-
a
+
w
1
+
A
+
W
i
×
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
i
i
exp
⁡
1
A
exp
⁡
-
α
t
×
b
1
-
b
2
-
1
M
-
1
.


#### 3.2.4. DS-STGM

Substitute (24) into (11); the DS-STGM can be formulated as(35)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
=
M
-
a
W
i
-
b
2
exp
⁡
-
b
1
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
=
m
-
a
i
W
i
×
b
1
-
b
2
-
1
M
-
1
.


#### 3.2.5. IS-STGM

Substitute (28) into (11); the IS-STGM can be formulated as(36)
q
t
=
M
-
a
b
1
exp
⁡
-
b
2
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
=
i
-
b
2
exp
⁡
-
b
1
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
i
exp
⁡
-
β
2
t
φ
exp
⁡
-
β
2
t
×
b
1
-
b
2
-
1
M
-
1
.


## 3.1. Bell-Shaped TGEF

In TGT, one of the key factors is the TGE which can be represented as the number of man-hours, the times of fault injection, and the cost of TGT, and so forth. The functions that describe how an effort is distributed over the testability growth phase are referred to by us as TGEF. TGEF describes the relationship between the test time and the amount of TGE expended in the test time. Actually, fault injection is one of the best effective measures to make the UUT be in broken condition and to identify the inherent TDLs in the UUT further. There is an extensive body of literature on fault injection used for identifying TDL in testability test [1–3, 41–43]. Therefore, we will use the times of fault injection as a measurement of TGE, which has an advantage that it is very intuitive and that testability growth effort can be quantified exactly at the same time.In general, the whole life-cycle of equipment can be divided into three stages: demonstration stage, design & development stage, and trial & in-service stage. But the empirical analysis shows that the effective and credible stages to conduct TGT are design & development stage and trial & in-service stage considering the cost effectiveness in which fault injection and collection information for naturally occurred faults are done, respectively [3, 6]. At the same time, TGE are calculated by times of fault injection and the number of naturally occurred faults at design & development stage and trial & in-service stage, respectively.Actually, the more faults we inject or collect, the more confidence we obtain in the estimation of system testability. Unfortunately, TGT with redundant fault injections or a large number of failure data may lead to excessive cost and too much time consumption. It is impracticable with the constraint of test cycle and total cost.To give an accurate and reasonable description of TGE, we need to know not only how many faults need to inject at design & development stage but also how many failure data need to collect at trial & in-service stage. That is to say, we need to analyze the TGE’s consumption pattern in TGT by combining Figure3 which gives an abridged general view of testability growth process in the whole life-cycle.Figure 3
The testability growth chart in the whole life-cycle.In general, the TGT time can be represented by the serial number of testability growth test stage instead of the actual execution time. The assumptions which are shown in Figure3 and will be used for description of TGEF are as follows.(1)
The TGT stage is expressed as the time span[
0
,
t
e
] and is composed by the design & development stage time span [
0
,
t
s
] and the trail & in-service stage time span [
t
s
,
t
e
], respectively.
(2)
The times of TGT based on fault injection at design & development stage arem, and the maximum number of fault injection at the design & development stage is marked as N
D
D. At the design & development stage, we can observe n data pairs in the form (
n
k
,
f
k
), 1
≤
k
≤
m, n
k is the number of fault injection at the kth TGT stage, and f
k is the number of TDLs identified successfully at the kth TGT stage.
(3)
After a period of time especially themth TGT at design & development stage, the UUT is put into the trail & in-service stage at which the number of TGT based on fault occurred naturally is m
S and the maximum number of failures collecting is marked as N
S. In a similar way, the same data pairs in the form (
n
j
,
f
j
),  m
<
j
≤
m
S are observed by collecting the failure data occurred naturally. n
j is the number of failures by collecting the naturally occurred fault and f
j is the number of TDLs identified successfully at the jth TGT stage.
(4)
With the progress of TGT,q
i,  1
≤
i
≤
m
S is the true value of FDR at the ith TGT stage, and that q
1
<
⋯
q
i
⋯
<
q
m
S. The TGT can be stopped only when the Q
E is achieved, in which Q
E is the FDR’s desired value of TGT shown in Figure 3.Based on the above assumptions, we will analyze the TGE consumption pattern from two perspectives; one is fault injection based TGE consumption at design & development stage; the other is naturally occurred fault based TGE consumption at trail & in-service stage.At the beginning of the design & development stage, FDR level is relatively lower; hence, a certain number of TDLs will be identified as long as we inject less fault. In order to identify a certain number of TDLs at the design & development stage, the number of faults need to inject grow up with the level of FDR, which can be formulated as [1–3](12)
q
k
=
n
k
-
f
k
n
k
,
1
≤
k
≤
m
.For example, with reference to (12), the number of faults which need to be injected with the change of FDR level and number of TDLs, respectively, are shown in Figure 4. As seen from Figure 4, we find that at the same number of TDLs, the FDR level needs to achieve increase as time goes on, and the number of faults needs to inject increases accordingly. At the same level of FDR, the greater number of TDLs needs to identify, the greater number of faults needs to inject. Consequently, TGEF is a time-dependent increasing function at design & development stage.Figure 4
The number of faults needs to be injected with the change of FDR level and number of TDLs, respectively.Collection of failure which occurred naturally is also an effective measure to identify TDL When the UUT is put into the trail & in-service stage. Actually, the number of faults at trail & in-service stage is “small sample” under several constraints such as high reliability requirements or specified test cycle. Considering the growth of reliability, the expected number of faults at regular intervals will decrease gradually. That is to say, at trail & in-service stage, TGEF is a time-dependent decreasing function which has the similar variation tendency shown in Figure5.Figure 5
The diagrammatic drawing of TGEF at trail & in-service stage.In conclusion, at the whole TGT phase, TGEF increases firstly and then decreases at different rate. Based on the increase-decrease characteristic, we will use a kind of Bell-shaped function which increases firstly and then decreases to fit the practical growth rate of TGEF. Bell-shaped function, just as its name implies, draws a time-dependent curve like a bell which has a similar shape to the curve shown in Figure6.Figure 6
A schematic of Bell-shaped current TGE consumption.It should be noted that Figure6 is just a schematic used to depict the Bell-shaped function; the concrete functional expression depends on the actual TGE data set. Because actual testability growth effort data represent various expenditure patterns, sometimes the testability growth effort expenditures are difficult to describe only by a Bell-shaped curve. In this paper, the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF have been used to explain the testability growth effort, which can be derived from the assumption that “the efforts that govern the pace of identifying the TDL for the UUT increase firstly and then decrease gradually.”


### 3.1.1. The Exponential TGEF

Yamada et al. [21] found that the TEF in software reliability growth could be described by a Weibull-type distribution with two cases: the Exponential curve and the Rayleigh curve.The Exponential TGEF over time period(
0
,
t
] can be expressed as(13)
W
E
t
=
N
1
1
-
exp
⁡
-
β
1
t
.The current TGE expenditure rate at timet is(14)
w
E
t
=
N
1
β
1
exp
⁡
-
β
1
t
,where W
E
(
t
)
=
∫
0
t
w
E
(
τ
)
d
τ.The Exponential curve is used for process that declines monotonically to an asymptote.Substitute theW
E
∗
(
t
) in (13) into (7); we obtain(15)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
E
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
E
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
1
-
b
2
exp
⁡
-
b
1
N
1
b
1
-
b
2
.


### 3.1.2. The Rayleigh TGEF [16, 19, 21, 37]

Another alternative of the Weibull-type distribution is the Rayleigh curve. It has been empirically observed that software development projects follow a life-cycle pattern described by the Rayleigh curve.The Rayleigh TGEF over time period(
0
,
t
] can be expressed as(16)
W
R
t
=
N
2
1
-
exp
⁡
-
β
2
2
t
2
.The current TGE expenditure rate at timet is(17)
w
R
t
=
N
2
β
2
t
exp
⁡
-
β
2
2
t
2
,where W
R
(
t
)
=
∫
0
t
w
R
(
τ
)
d
τ.Therefore,w
R
(
t
) is a smooth Bell-shaped curve and reaches its maximum value at time(18)
t
m
a
x
=
1
β
2
.The Rayleigh curve first increases to a peak, and then decreases at a decelerating rate.Substitute theW
R
∗
(
t
) in (16) into (7), we obtain(19)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
R
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
2
-
b
2
exp
⁡
-
b
1
N
2
b
1
-
b
2
.


### 3.1.3. The Logistic TGEF

Logistic TEF was originally proposed by Parr [16] about software reliability growth model. It exhibits similar behavior to the Rayleigh curve, except during the early part of the project. Huang et al. [28, 30, 37] proposed that a logistic testing effort function can be used instead of the Weibull-type curve to describe the test effort patterns during the software development process. In some two dozen projects studied in the Yourdon 1978–1980 project survey, the Logistic TEF appeared to be fairly accurate in describing the expended test efforts [44].The logistic TGEF over time period(
0
,
t
] can be expressed as(20)
W
L
t
=
N
3
1
+
A
exp
⁡
-
β
3
t
.The current TGE expenditure rate at timet is(21)
w
L
t
=
N
3
A
β
3
exp
⁡
-
β
3
t
1
+
A
exp
⁡
-
β
3
t
2
,where W
L
(
t
)
=
∫
0
t
w
L
(
τ
)
d
τ.Therefore,w
L
(
t
) is a smooth Bell-shaped function, and reaches its maximum value at time(22)
t
m
a
x
=
1
β
3
ln
⁡
A
.Substitute theW
L
∗
(
t
) in (20) into (7), we obtain(23)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
∞
-
b
2
exp
⁡
-
b
1
W
R
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
3
/
1
+
A
-
b
2
exp
⁡
-
b
1
N
3
/
1
+
A
b
1
-
b
2
.Ohba [17, 18] found that the test effort function in software reliability growth could be described by S-shaped distribution with two cases: the delayed S-shaped curve and the inflected S-shaped curve.


### 3.1.4. The Delayed S-Shaped TGEF [17, 40]

The delayed S-shaped TGEF over time period(
0
,
t
] can be expressed as(24)
W
D
S
t
=
N
4
1
-
1
+
β
4
t
exp
⁡
-
β
4
t
.The current TGE expenditure rate at timet is(25)
w
D
S
t
=
N
4
β
4
2
t
exp
⁡
(
-
β
4
t
)
,where W
D
S
(
t
)
=
∫
0
t
w
D
S
(
τ
)
d
τ.Therefore,w
D
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(26)
t
m
a
x
=
1
β
4
.Substitute theW
D
S
∗
(
t
) in (24) into (7); we obtain(27)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
D
S
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
D
S
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
4
-
b
2
exp
⁡
-
b
1
N
4
b
1
-
b
2
.


### 3.1.5. The Inflected S-Shaped TGEF [18, 40]

An inflected S-shaped TGEF over time period(
0
,
t
] can be expressed as(28)
W
I
S
t
=
N
5
1
-
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
.The current TGE expenditure rate at timet is(29)
w
I
S
t
=
N
5
β
5
1
+
φ
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
2
,where W
I
S
(
t
)
=
∫
0
t
w
I
S
(
τ
)
d
τ.Therefore,w
I
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(30)
t
m
a
x
=
1
β
5
ln
⁡
φ
.Substitute theW
I
S
∗
(
t
) in (28) into (7); we obtain(31)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
I
S
∗
∞
-
b
2
exp
⁡
-
b
1
W
I
S
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
5
-
b
2
exp
⁡
-
b
1
N
5
b
1
-
b
2
.With reference to the above five Bell-shaped TGEFs, the instantaneous TGEs which are shown in Figures7, 8, 9, 10, and 11 decrease ultimately during the testability growth life-cycle because the cumulative TGEs approach a finite limit. As shown from (15), (19), (23), (27), and (31), we find that not all the inherent TDLs in the UUT can be fully removed, even after a long TGT period because the total amount of TGE to be consumed during the TGT phase is limited to W
m
a
x. This assumption is reasonable because no system testability design company will spend infinite resources on TGT considering the restricted cost & development cycle.Figure 7
An Exponential curve with fixed parameters.Figure 8
A Rayleigh curve with fixed parameters.Figure 9
A logistic curve with fixed parameters.Figure 10
A delayed S-shaped curve with fixed parameters.Figure 11
An inflected S-shaped curve with fixed parameters.


## 3.1.1. The Exponential TGEF

Yamada et al. [21] found that the TEF in software reliability growth could be described by a Weibull-type distribution with two cases: the Exponential curve and the Rayleigh curve.The Exponential TGEF over time period(
0
,
t
] can be expressed as(13)
W
E
t
=
N
1
1
-
exp
⁡
-
β
1
t
.The current TGE expenditure rate at timet is(14)
w
E
t
=
N
1
β
1
exp
⁡
-
β
1
t
,where W
E
(
t
)
=
∫
0
t
w
E
(
τ
)
d
τ.The Exponential curve is used for process that declines monotonically to an asymptote.Substitute theW
E
∗
(
t
) in (13) into (7); we obtain(15)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
E
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
E
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
1
-
b
2
exp
⁡
-
b
1
N
1
b
1
-
b
2
.


## 3.1.2. The Rayleigh TGEF [16, 19, 21, 37]

Another alternative of the Weibull-type distribution is the Rayleigh curve. It has been empirically observed that software development projects follow a life-cycle pattern described by the Rayleigh curve.The Rayleigh TGEF over time period(
0
,
t
] can be expressed as(16)
W
R
t
=
N
2
1
-
exp
⁡
-
β
2
2
t
2
.The current TGE expenditure rate at timet is(17)
w
R
t
=
N
2
β
2
t
exp
⁡
-
β
2
2
t
2
,where W
R
(
t
)
=
∫
0
t
w
R
(
τ
)
d
τ.Therefore,w
R
(
t
) is a smooth Bell-shaped curve and reaches its maximum value at time(18)
t
m
a
x
=
1
β
2
.The Rayleigh curve first increases to a peak, and then decreases at a decelerating rate.Substitute theW
R
∗
(
t
) in (16) into (7), we obtain(19)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
R
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
2
-
b
2
exp
⁡
-
b
1
N
2
b
1
-
b
2
.


## 3.1.3. The Logistic TGEF

Logistic TEF was originally proposed by Parr [16] about software reliability growth model. It exhibits similar behavior to the Rayleigh curve, except during the early part of the project. Huang et al. [28, 30, 37] proposed that a logistic testing effort function can be used instead of the Weibull-type curve to describe the test effort patterns during the software development process. In some two dozen projects studied in the Yourdon 1978–1980 project survey, the Logistic TEF appeared to be fairly accurate in describing the expended test efforts [44].The logistic TGEF over time period(
0
,
t
] can be expressed as(20)
W
L
t
=
N
3
1
+
A
exp
⁡
-
β
3
t
.The current TGE expenditure rate at timet is(21)
w
L
t
=
N
3
A
β
3
exp
⁡
-
β
3
t
1
+
A
exp
⁡
-
β
3
t
2
,where W
L
(
t
)
=
∫
0
t
w
L
(
τ
)
d
τ.Therefore,w
L
(
t
) is a smooth Bell-shaped function, and reaches its maximum value at time(22)
t
m
a
x
=
1
β
3
ln
⁡
A
.Substitute theW
L
∗
(
t
) in (20) into (7), we obtain(23)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
R
∗
∞
-
b
2
exp
⁡
-
b
1
W
R
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
3
/
1
+
A
-
b
2
exp
⁡
-
b
1
N
3
/
1
+
A
b
1
-
b
2
.Ohba [17, 18] found that the test effort function in software reliability growth could be described by S-shaped distribution with two cases: the delayed S-shaped curve and the inflected S-shaped curve.


## 3.1.4. The Delayed S-Shaped TGEF [17, 40]

The delayed S-shaped TGEF over time period(
0
,
t
] can be expressed as(24)
W
D
S
t
=
N
4
1
-
1
+
β
4
t
exp
⁡
-
β
4
t
.The current TGE expenditure rate at timet is(25)
w
D
S
t
=
N
4
β
4
2
t
exp
⁡
(
-
β
4
t
)
,where W
D
S
(
t
)
=
∫
0
t
w
D
S
(
τ
)
d
τ.Therefore,w
D
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(26)
t
m
a
x
=
1
β
4
.Substitute theW
D
S
∗
(
t
) in (24) into (7); we obtain(27)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
D
S
∗
(
∞
)
-
b
2
exp
⁡
-
b
1
W
D
S
∗
(
∞
)
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
4
-
b
2
exp
⁡
-
b
1
N
4
b
1
-
b
2
.


## 3.1.5. The Inflected S-Shaped TGEF [18, 40]

An inflected S-shaped TGEF over time period(
0
,
t
] can be expressed as(28)
W
I
S
t
=
N
5
1
-
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
.The current TGE expenditure rate at timet is(29)
w
I
S
t
=
N
5
β
5
1
+
φ
exp
⁡
-
β
5
t
1
+
φ
exp
⁡
-
β
5
t
2
,where W
I
S
(
t
)
=
∫
0
t
w
I
S
(
τ
)
d
τ.Therefore,w
I
S
(
t
) is a smooth Bell-shaped function and reaches its maximum value at time(30)
t
m
a
x
=
1
β
5
ln
⁡
φ
.Substitute theW
I
S
∗
(
t
) in (28) into (7); we obtain(31)
a
r
e
m
a
i
n
i
n
g
=
a
b
1
exp
⁡
-
b
2
W
I
S
∗
∞
-
b
2
exp
⁡
-
b
1
W
I
S
∗
∞
b
1
-
b
2
≈
a
b
1
exp
⁡
-
b
2
N
5
-
b
2
exp
⁡
-
b
1
N
5
b
1
-
b
2
.With reference to the above five Bell-shaped TGEFs, the instantaneous TGEs which are shown in Figures7, 8, 9, 10, and 11 decrease ultimately during the testability growth life-cycle because the cumulative TGEs approach a finite limit. As shown from (15), (19), (23), (27), and (31), we find that not all the inherent TDLs in the UUT can be fully removed, even after a long TGT period because the total amount of TGE to be consumed during the TGT phase is limited to W
m
a
x. This assumption is reasonable because no system testability design company will spend infinite resources on TGT considering the restricted cost & development cycle.Figure 7
An Exponential curve with fixed parameters.Figure 8
A Rayleigh curve with fixed parameters.Figure 9
A logistic curve with fixed parameters.Figure 10
A delayed S-shaped curve with fixed parameters.Figure 11
An inflected S-shaped curve with fixed parameters.


## 3.2. Bell-Shaped TGEF Dependent STGM

In this section, we will use the above five TGEFs in the foundation of STGM. One of these can be substituted into (10) to have five kinds of STGM which are formulated as follows.


### 3.2.1. EX-STGM

Substitute (13) into (11); the EX-STGM can be formulated as(32)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
1
1
-
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
M
-
a
=
a
W
i
i
-
b
2
exp
⁡
-
b
1
N
1
1
-
exp
⁡
-
β
1
t
M
-
a
=
a
W
i
i
×
b
1
-
b
2
-
1
M
-
1
.


### 3.2.2. RA-STGM

Substitute (16) into (11); the RA-STGM can be formulated as(33)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
exp
⁡
-
b
1
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
×
b
1
-
b
2
-
1
M
-
1
.


### 3.2.3. LO-STGM

Substitute (20) into (11); the LO-STGM can be formulated as(34)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
3
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
w
i
-
b
2
exp
⁡
-
b
1
N
3
1
A
exp
⁡
(
-
α
t
)
=
M
-
a
+
w
1
+
A
+
W
i
×
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
i
i
exp
⁡
1
A
exp
⁡
-
α
t
×
b
1
-
b
2
-
1
M
-
1
.


### 3.2.4. DS-STGM

Substitute (24) into (11); the DS-STGM can be formulated as(35)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
=
M
-
a
W
i
-
b
2
exp
⁡
-
b
1
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
=
m
-
a
i
W
i
×
b
1
-
b
2
-
1
M
-
1
.


### 3.2.5. IS-STGM

Substitute (28) into (11); the IS-STGM can be formulated as(36)
q
t
=
M
-
a
b
1
exp
⁡
-
b
2
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
=
i
-
b
2
exp
⁡
-
b
1
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
i
exp
⁡
-
β
2
t
φ
exp
⁡
-
β
2
t
×
b
1
-
b
2
-
1
M
-
1
.


## 3.2.1. EX-STGM

Substitute (13) into (11); the EX-STGM can be formulated as(32)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
1
1
-
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
M
-
a
=
a
W
i
i
-
b
2
exp
⁡
-
b
1
N
1
1
-
exp
⁡
-
β
1
t
M
-
a
=
a
W
i
i
×
b
1
-
b
2
-
1
M
-
1
.


## 3.2.2. RA-STGM

Substitute (16) into (11); the RA-STGM can be formulated as(33)
q
t
=
M
-
a
×
b
1
exp
⁡
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
exp
⁡
-
b
1
N
2
1
-
exp
⁡
-
β
2
2
t
2
=
M
-
a
×
W
W
-
b
2
N
2
1
-
exp
⁡
-
β
2
2
t
2
×
b
1
-
b
2
-
1
M
-
1
.


## 3.2.3. LO-STGM

Substitute (20) into (11); the LO-STGM can be formulated as(34)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
3
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
w
i
-
b
2
exp
⁡
-
b
1
N
3
1
A
exp
⁡
(
-
α
t
)
=
M
-
a
+
w
1
+
A
+
W
i
×
1
1

+

A
exp
⁡
(
-
α
t
)
-
1
1

+

A
=
M
-
a
+
i
i
exp
⁡
1
A
exp
⁡
-
α
t
×
b
1
-
b
2
-
1
M
-
1
.


## 3.2.4. DS-STGM

Substitute (24) into (11); the DS-STGM can be formulated as(35)
q
t
=
M

-

a
b
1
exp
⁡
-
b
2
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
b
1
-
b
2
-
1
=
M
-
a
W
i
-
b
2
exp
⁡
-
b
1
N
4
1
-
1

+

β
1
t
exp
⁡
(
-
β
1
t
)
=
m
-
a
i
W
i
×
b
1
-
b
2
-
1
M
-
1
.


## 3.2.5. IS-STGM

Substitute (28) into (11); the IS-STGM can be formulated as(36)
q
t
=
M
-
a
b
1
exp
⁡
-
b
2
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
=
i
-
b
2
exp
⁡
-
b
1
N
5
1
+
exp
⁡
(
-
β
2
t
)
1
+
φ
exp
⁡
(
-
β
2
t
)
=
M
-
a
W
i
exp
⁡
-
β
2
t
φ
exp
⁡
-
β
2
t
×
b
1
-
b
2
-
1
M
-
1
.


## 4. Estimation of STGM Parameters

Fitting a proposed model to actual TGT data involves estimating the model parameters from the real TGT data set. Two popular estimation techniques are MLE and LSE [45]. The MLE estimates parameters by solving a set of simultaneous equations. However, the equation set may be very complex and usually must be solved numerically. The LSE minimizes the sum of squares of the deviations between what we actually observe and what we expect. In order to avoid the solution of complex simultaneous equations of MLE, in this section, we employ the method of LSE to estimate the parameters of the above five TGEFs. Using the estimated TGEFs, the other parameters a, b
1, b
2 in (5), (6) can also be estimated by LSE.Due to the limitations of paper size, reference to the above five TGEFs, only the parametersN
1, β
1 of the Exponential TGEF in (13) are estimated by the method of LSE. At the same time, the parameters a, b
1 given in (5) and b
2 given in (6) can all be estimated by LSE. For the method of LSE, the evaluation formula S
E
(
N
1
,
β
1
), S
M
c
(
a
,
b
1
) and S
M
r
(
m
c
,
b
2
) are as follows:(37)
M
i
n
i
m
i
z
e
S
E
N
1
,
β
1
=
∑
k
=
1
n
W
E
k
-
W
E
t
k
2
,
M
i
n
i
m
i
z
e
S
M
c
a
,
b
1
=
∑
k
=
1
n
M
C
k
-
M
C
t
k
2
,
M
i
n
i
m
i
z
e
S
M
r
m
c
,
b
2
=
∑
k
=
1
n
m
r
k
-
m
r
t
k
2
.DifferentiatingS
E, S
M
i, S
M
c with respect to N
1
,
β
1, a
,
b
1, and m
i
,
b
2, respectively, setting the partial derivatives to zero, and rearranging these terms, we can solve this type of nonlinear least square problems. For a simple illustration, we only consider the generalized Exponential TGEF.Take the partial derivatives ofS
E with respect to N
1, β
1; we get(38)
∂
S
E
∂
N
1
=
∑
k
=
1
n
-
2
W
E
k
-
N
1
1
-
exp
⁡
-
β
1
k
A
=
b
×
[
1
-
exp
⁡
-
β
1
k
]
=
0
.Thus, the least squares estimatorN
1 is given by solving the above equation, yielding(39)
N
1
=
∑
k
=
2
n
W
k
[
1
-
exp
⁡
(
-
β
1
k
)
]
∑
k
=
2
n
[
1
-
exp
⁡
(
-
β
1
k
)
]
2
.Next, we have(40)
∂
S
E
∂
β
1
=
∑
k
=
1
n
-
2
N
1
k
W
k
-
N
1
1
-
exp
⁡
-
β
1
k
A
=
b
×
exp
⁡
-
β
1
k
=
0
.The parameterβ
1 can also be obtained by substituting the least squares estimator N
1 into the above equation.Similarly, the parametersN
2, β
2 of the Rayleigh TGEF in (16), N
3, A, β
3 of the logistic TGEF in (20), N
4, β
4 of the delayed S-shaped TGEF in (24), N
5, β
5 of the inflected S-shaped TGEF in (28) can also be estimated by LSE.


## 5. Data Analysis and STGM Comparisons

### 5.1. Data Description

To validate the proposed STGMs with the above five Bell-shaped TGEFs, TGT on a stable tracking platform have been performed. The TGT data set employed (listed in Table1) was from the testability laboratory of National University of Defense Technology for a stable tracking platform which can isolate the movement of moving vehicle, such as car, ship, and aircraft. FMEA of the stable tracking platform had been done and had gained that the stable tracking platform consisted of approximately 350 functional circuit level failures. Over the course of 12 weeks at the design & development stage, 72 TDLs were identified by injecting 203 functional circuits level failures. Failures were injected by 1553B fault injection equipment, ARINC 429 fault injection equipment, RS232/422 fault injection equipment, CAN bus fault injection equipment, and the like. Further, testability designers analyzed the root cause of the TDLs and had tried their best to modify the design of testability and removed 46 TDLs successfully. On the other hand, over the course of 12 weeks at the trial & in-service stage, 110 functional circuit level failures had occurred naturally. 14 TDLs were identified, in which 7 TDLs were removed successfully.Table 1
Data set for a stable tracking platform.
Test time (weeks)
Cumulative TGE (fault times)
Cumulative TDLs identified
Cumulative TDLs removed
Test time (weeks)
Cumulative TGE (fault times)
Cumulative TDLs identified
Cumulative TDLs removed
1
4
3
2
13
223
74
48
2
9
6
4
14
241
76
49
3
16
11
7
15
256
77
50
4
25
17
11
16
268
79
51
5
38
25
16
17
278
80
52
6
55
33
21
18
286
80
52
7
74
41
27
19
294
82
53
8
95
50
33
20
301
83
54
9
119
57
37
21
306
84
54
10
145
63
41
22
310
85
55
11
173
68
44
23
312
85
55
12
203
72
46
24
313
86
55


### 5.2. Criteria for Model Comparison

A STGM can generally be analyzed according to its estimation capability, fitness capability, and predictive capability. That is to say, a STGM can be analyzed according to its ability to reproduce the identified TDL and behavior of the UUT and to predict the future behavior of the UUT from the observed TDL data. In this paper, the STGMs are compared with each other based on the following three criteria.


#### 5.2.1. The Accuracy of Estimation Criterion [39]

For practical purposes, we will use AE to calculate the accuracy of estimation. AE is defined as(41)
A
E
=
m
i
-
a
m
i
.m
i is the actual cumulative number of identified TDL after the TGT, and a is the estimated number of initial TDLs. m
i is obtained from system testability TDL tracking after TGT.


#### 5.2.2. The Goodness-of-Fit Criteria

To quantitatively compare long-term predictions, we use MSE because it provides a well-understood measure of the difference between actual and predicted values. The MSE is defined as [30, 37, 45](42)
M
S
E
=
∑
i
=
1
n
m
r
(
t
i
)
-
m
r
i
2
n
.A smaller MSE indicates a smaller fitting error and better performance.


#### 5.2.3. The Predictive Validity Criterion

The capability of the model to predict TDL identification and removal behavior from present & past TDL behavior is called predictive validity. This approach which was proposed by Musa et al. [45] can be represented by computing RE for a data set(43)
R
E
=
m
r
(
t
a
)
-
e
e
.Assuming that we have identified and removede TDLs by the end of TGT time t
e, we employ the TDL data up to time t
a (t
a
≤
t
e) to estimate the parameters of m
r
(
t
). Substituting the estimates of these parameters in the MVF yields the estimate of the number of TDLs m
r
(
t
e
) by time t
e. The estimate is compared with the actual number e. The procedure is repeated for various values of t
a. We can check the predictive validity by plotting the relative error for different values of t
a. Numbers closer to zero imply more accurate prediction. Positive values of error indicate overestimation and negative values indicate underestimation.


### 5.3. STGM Performance Analysis

In this section, we present our evaluation of the performance of the proposed STGMs when applied to DS listed in Table1.All the parameters of the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF are also estimated by LSE. Firstly, the two unknown parametersN
1, β
1 of the Exponential TGEF are solved by LSE, giving the estimated values N
1
=
2592.7 (fault times) and β
1
=
0.0060/week. Correspondingly, the estimated parameters of the Rayleigh TGEF are N
2
=
329.57 (fault times), and β
2
=
0.0123/week. In similarly, the estimated parameters of the logistic TGEF, the delayed S-shaped TGEF, the inflected S-shaped TGEF are N
3
=
311.18 (fault times), A
=
39.36, β
3
=
0.3524/week, N
4
=
406.06 (fault times), β
4
=
0.1313/week, N
5
=
314.44 (fault times), φ
=
27.09, and β
5
=
0.3232/week. In order to clearly show the comparative effectiveness of the fitness for the observed TGE data, the comparisons between the observed five current TGE data and the estimated five current TGE data are illustrated graphically in Figures 12 and 13, respectively. Similarly, the comparisons between the observed five cumulative TGE data and the estimated five cumulative TGE data are illustrated graphically in Figures 14 and 15, respectively.Figure 12
Observed and three estimated current TGEFs for DS.Figure 13
Observed and two estimated current TGEFs for DS.Figure 14
Observed and three estimated cumulative TGEFs for DS.Figure 15
Observed and two estimated cumulative TGEFs for DS.In order to check the performance of the five above TGEFs and make comparisons with each other, here we select some comparison criterions for the estimation of TGEFs [30, 37, 45](44)
P
E
i
=
A
c
t
u
r
a
l
o
b
s
e
r
v
e
d
i
-
P
r
e
d
i
c
t
e
d
e
s
t
i
m
a
t
e
d
i
,
B
i
a
s
=
1
n
∑
i
=
1
n
P
E
i
,
V
a
r
i
a
t
i
o
n
=
∑
i
=
1
n
P
E
i
-
B
i
a
s
2
n
-
1
.The PE, Bias, and Variation for the above five TGEFs are listed in Table2. From Table 2, we see that the inflected S-shaped TGEF has lower values of PE, Bias, and Variation than the other four TGEFs. On average, the inflected S-shaped TGEF yields a better fit for this data set, which can also be drown from Figures 12 to 15 approximately.Table 2
Comparison results for different TGEFs applied to DS.
TGEF
Bias
Variation
PEend of TGT
Exponential TGEF
4.26
25.55
34.71
Rayleigh TGEF
1.27
7.07
7.03
Logistic TGEF
0.70
3.78
−4.40
Delayed S-shaped TGEF
2.73
13.86
20.91
Inflected S-shaped TGEF
0.18
2.28
−2.29Table3 lists the estimated values of parameters of different STGMs. We also give the values of AE, RE, and MSE in Table 3. It is observed that the STGM with inflected S-shaped TGEF (i.e., IS-STGM) has the smallest value of MSE when compared with other STGMs.Table 3
Estimated parameters values and model comparisons for DS.
STGMs
a
b
1
b
2
AE (%)
MSE
MRE
EX-STGM
114.39
0.0046
0.0090
33.1
—*
0.3156
RA-STGM
95.60
0.0068
0.0076
11.16
83.91
0.1623
LO-STGM
87.35
0.0096
0.0070
1.57
85.51
0.1063
DS-STGM
101.30
0.0059
0.0080
17.79
90.09
0.2213
IS-STGM
90.66
0.0081
0.0071
5.42
86.13
0.1226
∗The symbol “—” indicates that the value is too large to characterize the MSE.From Table3 we see that the LO-STGM has lower values of AE and RE than the other four Bell-shaped TGEF dependent STGMs and that the RA-STGM has lower value of MSE than the other four Bell-shaped TGEF dependent STGMs. On average, the logistic TGEF dependent STGM yields a better fit for this data set.Finally, Figures16, 17, 18, 19, and 20 depict the RE curve for different selected STGMs.Figure 16
RE curve of EX-STGM compared with actual DS.Figure 17
RE curve of RA-STGM compared with actual DS.Figure 18
RE curve of LO-STGM compared with actual DS.Figure 19
RE curve of DS-STGM compared with actual DS.Figure 20
RE curve of IS-STGM compared with actual DS.Substitute the estimated value ofa, b
1, b
2 listed in Table 3 into (32), (33), (34), (35), (36), we can find five TGEF dependent STGMs of FDR. Figures 21, 22, 23, 24, and 25 depict the growth curve of FDR at the whole TGT stage.Figure 21
FDR curve based on EX-STGM.Figure 22
FDR curve based on RA-STGM.Figure 23
FDR curve based on LO-STGM.Figure 24
FDR curve based on DS-STGM.Figure 25
FDR curve based on IS-STGM.Finally, the performance of STGM strongly depends on the kind of data set. If a system testability designer plans to employ STGM for estimation of testability growth of UUT during system development processes, the testability designers need to select several representative models and apply them at the same time. Although models sometimes give good results, there is no single model that can be trusted to give accurate results in all circumstances.From our results, we can draw the following conclusions.(1)
The Bell-shaped TGEF may be a good approach to providing a more accurate description of resource consumption during TGT phase. Particularly, the inflected S-shaped TGEF has the smallest value of Bias, Variation, and PE compared with the other four TGEFs when applied to DS for a stable tracking platform.
(2)
By incorporating the Bell-shaped TGEF into the structure of STGM, the STGMs with different TGEFs are very powerful and flexible for various test environments. The RA-STGM has the smallest value of MSE compared with the other four STGMs when applied to DS for a stable tracking platform. Similarly, the LO-STGM has the smallest value of AE and RE compared with the other four STGMs.


## 5.1. Data Description

To validate the proposed STGMs with the above five Bell-shaped TGEFs, TGT on a stable tracking platform have been performed. The TGT data set employed (listed in Table1) was from the testability laboratory of National University of Defense Technology for a stable tracking platform which can isolate the movement of moving vehicle, such as car, ship, and aircraft. FMEA of the stable tracking platform had been done and had gained that the stable tracking platform consisted of approximately 350 functional circuit level failures. Over the course of 12 weeks at the design & development stage, 72 TDLs were identified by injecting 203 functional circuits level failures. Failures were injected by 1553B fault injection equipment, ARINC 429 fault injection equipment, RS232/422 fault injection equipment, CAN bus fault injection equipment, and the like. Further, testability designers analyzed the root cause of the TDLs and had tried their best to modify the design of testability and removed 46 TDLs successfully. On the other hand, over the course of 12 weeks at the trial & in-service stage, 110 functional circuit level failures had occurred naturally. 14 TDLs were identified, in which 7 TDLs were removed successfully.Table 1
Data set for a stable tracking platform.
Test time (weeks)
Cumulative TGE (fault times)
Cumulative TDLs identified
Cumulative TDLs removed
Test time (weeks)
Cumulative TGE (fault times)
Cumulative TDLs identified
Cumulative TDLs removed
1
4
3
2
13
223
74
48
2
9
6
4
14
241
76
49
3
16
11
7
15
256
77
50
4
25
17
11
16
268
79
51
5
38
25
16
17
278
80
52
6
55
33
21
18
286
80
52
7
74
41
27
19
294
82
53
8
95
50
33
20
301
83
54
9
119
57
37
21
306
84
54
10
145
63
41
22
310
85
55
11
173
68
44
23
312
85
55
12
203
72
46
24
313
86
55


## 5.2. Criteria for Model Comparison

A STGM can generally be analyzed according to its estimation capability, fitness capability, and predictive capability. That is to say, a STGM can be analyzed according to its ability to reproduce the identified TDL and behavior of the UUT and to predict the future behavior of the UUT from the observed TDL data. In this paper, the STGMs are compared with each other based on the following three criteria.


### 5.2.1. The Accuracy of Estimation Criterion [39]

For practical purposes, we will use AE to calculate the accuracy of estimation. AE is defined as(41)
A
E
=
m
i
-
a
m
i
.m
i is the actual cumulative number of identified TDL after the TGT, and a is the estimated number of initial TDLs. m
i is obtained from system testability TDL tracking after TGT.


### 5.2.2. The Goodness-of-Fit Criteria

To quantitatively compare long-term predictions, we use MSE because it provides a well-understood measure of the difference between actual and predicted values. The MSE is defined as [30, 37, 45](42)
M
S
E
=
∑
i
=
1
n
m
r
(
t
i
)
-
m
r
i
2
n
.A smaller MSE indicates a smaller fitting error and better performance.


### 5.2.3. The Predictive Validity Criterion

The capability of the model to predict TDL identification and removal behavior from present & past TDL behavior is called predictive validity. This approach which was proposed by Musa et al. [45] can be represented by computing RE for a data set(43)
R
E
=
m
r
(
t
a
)
-
e
e
.Assuming that we have identified and removede TDLs by the end of TGT time t
e, we employ the TDL data up to time t
a (t
a
≤
t
e) to estimate the parameters of m
r
(
t
). Substituting the estimates of these parameters in the MVF yields the estimate of the number of TDLs m
r
(
t
e
) by time t
e. The estimate is compared with the actual number e. The procedure is repeated for various values of t
a. We can check the predictive validity by plotting the relative error for different values of t
a. Numbers closer to zero imply more accurate prediction. Positive values of error indicate overestimation and negative values indicate underestimation.


## 5.2.1. The Accuracy of Estimation Criterion [39]

For practical purposes, we will use AE to calculate the accuracy of estimation. AE is defined as(41)
A
E
=
m
i
-
a
m
i
.m
i is the actual cumulative number of identified TDL after the TGT, and a is the estimated number of initial TDLs. m
i is obtained from system testability TDL tracking after TGT.


## 5.2.2. The Goodness-of-Fit Criteria

To quantitatively compare long-term predictions, we use MSE because it provides a well-understood measure of the difference between actual and predicted values. The MSE is defined as [30, 37, 45](42)
M
S
E
=
∑
i
=
1
n
m
r
(
t
i
)
-
m
r
i
2
n
.A smaller MSE indicates a smaller fitting error and better performance.


## 5.2.3. The Predictive Validity Criterion

The capability of the model to predict TDL identification and removal behavior from present & past TDL behavior is called predictive validity. This approach which was proposed by Musa et al. [45] can be represented by computing RE for a data set(43)
R
E
=
m
r
(
t
a
)
-
e
e
.Assuming that we have identified and removede TDLs by the end of TGT time t
e, we employ the TDL data up to time t
a (t
a
≤
t
e) to estimate the parameters of m
r
(
t
). Substituting the estimates of these parameters in the MVF yields the estimate of the number of TDLs m
r
(
t
e
) by time t
e. The estimate is compared with the actual number e. The procedure is repeated for various values of t
a. We can check the predictive validity by plotting the relative error for different values of t
a. Numbers closer to zero imply more accurate prediction. Positive values of error indicate overestimation and negative values indicate underestimation.


## 5.3. STGM Performance Analysis

In this section, we present our evaluation of the performance of the proposed STGMs when applied to DS listed in Table1.All the parameters of the Exponential TGEF, the Rayleigh TGEF, the logistic TGEF, the delayed S-shaped TGEF, and the inflected S-shaped TGEF are also estimated by LSE. Firstly, the two unknown parametersN
1, β
1 of the Exponential TGEF are solved by LSE, giving the estimated values N
1
=
2592.7 (fault times) and β
1
=
0.0060/week. Correspondingly, the estimated parameters of the Rayleigh TGEF are N
2
=
329.57 (fault times), and β
2
=
0.0123/week. In similarly, the estimated parameters of the logistic TGEF, the delayed S-shaped TGEF, the inflected S-shaped TGEF are N
3
=
311.18 (fault times), A
=
39.36, β
3
=
0.3524/week, N
4
=
406.06 (fault times), β
4
=
0.1313/week, N
5
=
314.44 (fault times), φ
=
27.09, and β
5
=
0.3232/week. In order to clearly show the comparative effectiveness of the fitness for the observed TGE data, the comparisons between the observed five current TGE data and the estimated five current TGE data are illustrated graphically in Figures 12 and 13, respectively. Similarly, the comparisons between the observed five cumulative TGE data and the estimated five cumulative TGE data are illustrated graphically in Figures 14 and 15, respectively.Figure 12
Observed and three estimated current TGEFs for DS.Figure 13
Observed and two estimated current TGEFs for DS.Figure 14
Observed and three estimated cumulative TGEFs for DS.Figure 15
Observed and two estimated cumulative TGEFs for DS.In order to check the performance of the five above TGEFs and make comparisons with each other, here we select some comparison criterions for the estimation of TGEFs [30, 37, 45](44)
P
E
i
=
A
c
t
u
r
a
l
o
b
s
e
r
v
e
d
i
-
P
r
e
d
i
c
t
e
d
e
s
t
i
m
a
t
e
d
i
,
B
i
a
s
=
1
n
∑
i
=
1
n
P
E
i
,
V
a
r
i
a
t
i
o
n
=
∑
i
=
1
n
P
E
i
-
B
i
a
s
2
n
-
1
.The PE, Bias, and Variation for the above five TGEFs are listed in Table2. From Table 2, we see that the inflected S-shaped TGEF has lower values of PE, Bias, and Variation than the other four TGEFs. On average, the inflected S-shaped TGEF yields a better fit for this data set, which can also be drown from Figures 12 to 15 approximately.Table 2
Comparison results for different TGEFs applied to DS.
TGEF
Bias
Variation
PEend of TGT
Exponential TGEF
4.26
25.55
34.71
Rayleigh TGEF
1.27
7.07
7.03
Logistic TGEF
0.70
3.78
−4.40
Delayed S-shaped TGEF
2.73
13.86
20.91
Inflected S-shaped TGEF
0.18
2.28
−2.29Table3 lists the estimated values of parameters of different STGMs. We also give the values of AE, RE, and MSE in Table 3. It is observed that the STGM with inflected S-shaped TGEF (i.e., IS-STGM) has the smallest value of MSE when compared with other STGMs.Table 3
Estimated parameters values and model comparisons for DS.
STGMs
a
b
1
b
2
AE (%)
MSE
MRE
EX-STGM
114.39
0.0046
0.0090
33.1
—*
0.3156
RA-STGM
95.60
0.0068
0.0076
11.16
83.91
0.1623
LO-STGM
87.35
0.0096
0.0070
1.57
85.51
0.1063
DS-STGM
101.30
0.0059
0.0080
17.79
90.09
0.2213
IS-STGM
90.66
0.0081
0.0071
5.42
86.13
0.1226
∗The symbol “—” indicates that the value is too large to characterize the MSE.From Table3 we see that the LO-STGM has lower values of AE and RE than the other four Bell-shaped TGEF dependent STGMs and that the RA-STGM has lower value of MSE than the other four Bell-shaped TGEF dependent STGMs. On average, the logistic TGEF dependent STGM yields a better fit for this data set.Finally, Figures16, 17, 18, 19, and 20 depict the RE curve for different selected STGMs.Figure 16
RE curve of EX-STGM compared with actual DS.Figure 17
RE curve of RA-STGM compared with actual DS.Figure 18
RE curve of LO-STGM compared with actual DS.Figure 19
RE curve of DS-STGM compared with actual DS.Figure 20
RE curve of IS-STGM compared with actual DS.Substitute the estimated value ofa, b
1, b
2 listed in Table 3 into (32), (33), (34), (35), (36), we can find five TGEF dependent STGMs of FDR. Figures 21, 22, 23, 24, and 25 depict the growth curve of FDR at the whole TGT stage.Figure 21
FDR curve based on EX-STGM.Figure 22
FDR curve based on RA-STGM.Figure 23
FDR curve based on LO-STGM.Figure 24
FDR curve based on DS-STGM.Figure 25
FDR curve based on IS-STGM.Finally, the performance of STGM strongly depends on the kind of data set. If a system testability designer plans to employ STGM for estimation of testability growth of UUT during system development processes, the testability designers need to select several representative models and apply them at the same time. Although models sometimes give good results, there is no single model that can be trusted to give accurate results in all circumstances.From our results, we can draw the following conclusions.(1)
The Bell-shaped TGEF may be a good approach to providing a more accurate description of resource consumption during TGT phase. Particularly, the inflected S-shaped TGEF has the smallest value of Bias, Variation, and PE compared with the other four TGEFs when applied to DS for a stable tracking platform.
(2)
By incorporating the Bell-shaped TGEF into the structure of STGM, the STGMs with different TGEFs are very powerful and flexible for various test environments. The RA-STGM has the smallest value of MSE compared with the other four STGMs when applied to DS for a stable tracking platform. Similarly, the LO-STGM has the smallest value of AE and RE compared with the other four STGMs.



---

*Source: 613170-2015-06-08.xml*