Reliability Evaluation Optimal Selection Model of Component-Based System

If the components in a component-based software system come from different sources, the characteristics of the components may be different. Therefore, evaluating the reliability of a component-based system with a fixed model for all components will not be reasonable. To solve this problem, this paper combines a single reliability growth model with an architecture-based reliability model, and proposes an optimal selecting approach. First, the most appropriate model of each component is selected according to the historical reliability data of the component, so that the evaluation deviation is the smallest. Then, system reliability is evaluated according to both the relationships among components and the using frequency of each component. As the approach takes into account the historical data and the using frequency of each component, the evaluation and prediction results are more accurate than those of using a single model.

section_content: Component-based software reliability evaluation relates to the rationality of system design and the success of software systems. The current evaluation approaches include black box and white box approaches. Black-box approaches regard the system as a whole, only consider the system interaction with the external environment, and do not consider the internal architecture of the system. White box approaches [1] [2] [3] [4] [5] consider the system architecture. These approaches disassemble the system into components, establish the relationship among the components, and then evaluate the entire system reliability. On the contrary, black-box approaches don't not consider the internal architecture of the system and usually assume that the numbers of system errors are proportional to the failure rate, so black-box approaches are more suitable for the systems developed independently. 

Supported by the National Natural Science Foundation of China (Grant No. 61073052) and Research Fund for the Doctoral Program of Higher Education of China (Grant No. 20092302110040 ). 

However, the using frequencies of the components in a system is different from each other in fact. If a component contains more errors than the other components in the system, but the using frequency of the component is lower, the effect of the component to the system reliability would not be larger. Black-box approaches overestimate this part effect to the reliability of software system, so in the practical application the approaches are subject to certain restrictions. Component-based white-box approaches specifically consider the using frequency of each component, and no longer have the assumptions that system errors are proportional to the system failure rates. So these approaches can be more accurate in the evaluation of system reliability. 

With the progress of software development technology, components have been commercialized. As many largescale systems have been developed with the COTS (Commercial-Off-The-Shelf), component-based software system reliability evaluation is becoming more and more important. However, most existing componentbased evaluation approaches usually assume that the reliability of each component to be a fixed and known value without considering the historical data of each component. That will cause a bigger deviation in evaluating system reliability [6]. Feng and Zou et al. [7, 8] have presented comprehensive models respectively, but these models are for the entire system, rather than the individual components. 

This paper proposes a component-based optimal selec-tion model (OSM). This approach bases on the system architecture, fully utilizes the historical data of each component, and takes into account the software reliability growth. OSM combines the reliability growth model of every single component with the architecture-based reliability model. It selects the proper evaluation model of component according to the specific characteristic of each component, and then synthetically evaluates the system reliability according to the relationships among these components in the overall system. As the proposed approach selects the most suitable component evaluation model according to the characteristics of each component when evaluating component reliability, the evaluation result of each component is more accurate. As a result, the reliability evaluation result of the system is also more accurate. 

The contribution of this paper is that the OSM is proposed. The OSM combines a single reliability growth model with an architecture-based reliability model. As the new model takes into account the historical data and the using frequency of each component, the evaluation and prediction results are more accurate. 

The organization of this paper is organized as follows. In Section 2, we will give a survey of existing component reliability models and component-based system reliability models. In Section 3, the OSM operation process is described. In Section 4, approach of selecting component reliability model is presented. In Section 5, the software reliability evaluation approach is discussed. In Section 6, case analysis and verification is demonstrated. Finally, the conclusion remarks are given in Section 7.
section_name: Introduction

section_content: A component usually refers to executable modules that encapsulate data and function. Components can be independently produced or developed by the third parties. Usually these approaches that evaluate the system reliability considering the system as a whole [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] can be used to evaluate a single component. We can select a proper model according to the component's historical data. We can also use a certain model by translating the component's historical data into the form that the model can be used. According to different historical reliability data that the models can use, the component reliability models can be divided into two categories, i. e. the time between failure (TBF) models [9] [10] [11] [12] and the failure count (FC) models [13] [14] [15] [16] [17] [18]. TBF models use time-between-failure data, while FC models use the failure count data. 

A system constructed by components with logical relation is called component-based system. Component-based system reliability evaluation models include pathbased approach [1, 2, 19], state-based approach [3] [4] [5] 20, 21], and additive model [22]. Path-based approaches [1] [2] 19] evaluate system reliability by considering all the possible execution paths of the software. A sequence of all execution paths are gained by algorithm, experiment or simulation. The reliability of each path is estimated, and then the system reliability is estimated by calculating the average reliability of all paths. State-based models [4, 5, 20, 21] regard the execution of the software as a state transfer process [6]. This class of models apply stochastic process theorem to the analysis of the system reliability. Some typical approaches adopt the Markov process theory, which assume that the system change process with the state of Markov. The approaches include the discrete time Markov model (DTMC) [5, 20] and the time continuous Markov model (CTMC) [4] and semi-Markov process (SMP) [21]. Additive models [22] are mainly used in the software testing stage. It assumes that the reliability of each component can be modeled by nonhomogeneous Poisson process (NHPP), and thus the failure behavior of the system will also be a NHPP [23]. 

Path-based and state-based models typically assume that the reliability of components is known, and no longer consider the problem of the reliability evaluation for each component. Additive models concentrate on system failure-sensitive issues of time concerned by evaluating the component failure, but no longer explicitly consider the system architecture. To establish an effective component-based reliability evaluation model, we should take into account the evaluation approach of each component in a system. Only by doing this, can we get a more reasonable evaluation result.
section_name: Component Reliability Model and Component-Based System Reliability Model

section_content: This paper gives the following relevant definitions as a basis of OSM. 

Definition 1 Failure interval time data(TBF). It is a tri-ple<F,T,S>, where F is one-dimensional vector representing the failure order, T is the one-dimensional vector representing interval between the two adjacent failure, S is the one-dimensional vector representing the severity of the failure. For any element f in F, there is only one element tf in T, and sf in S to correspond to it, which <f,tf,sf> constitutes a TBF data. 

Definition 2 Failure count data FC. It is a four tuple <I, T 0,C, S>, where I is one-dimensional vector representing the test number, T 0 is the time interval, C is one-dimensional vector composed of errors during the interval T 0, S is one-dimensional vector composed of the severity of the error during the interval T 0. <i,T 0,c i,s i > is a failure count data, where i is an element in I, c i is an element in C, s i is an element in S. The OSM operation process of a component-based system is shown in Figure 1. 

The first step of OSM is that all the components of the system are determined. These components are obtained from the requirements analysis and design documents, and then assorted. The types of reliability-related historical data may be different, because the components come from different sources, and their specific implementation techniques, methods and testing process may be different. The components are divided into two categories: TBF components and FC components, mainly based on the type of the components' historical data. Then, the candidate reliability model sets of the component are determined. 

According to the categories of components and the application scope of the selected model, we determine the candidate reliability model sets of the component. For TBF components, all models in the candidate model set should belong to the class of TBF. For FC components, all models in the candidate model set should belong to be the class of FC. As each component is considered as a whole and its internal structure is no longer considered, black-box evaluation approaches are used. According to the software life-cycle, the classes of software reliability models include the analysis phase model, the design phase model, the implementation phase model, the testing phase model and the validation phase model. For the purpose of reliability evaluation of component-based software systems, the test phase model for each component is used to determine the component's reliability in OSM. 

After that, the evaluation criteria are applied. At present there is not any universal component reliability model, which means no model can be suitable for the reliability evaluation of all kinds of components. In order to evaluate the reliability of each component more effectively, evaluation criteria are used. There are many types of evaluation criteria. We can adopt a single criterion or a combination of several evaluation criteria. The single evaluation criterion means that the final component evaluation model is determined by a single formula. The combination evaluation approach is a combination of multiple evaluation criteria, such as the statically weighted combination. According the computing result of every component by the combination approach, the evaluation model of every component is determined. The complexity and practicality of the algorithm should be taken into account in determining the criteria. If the evaluation approach is too complicated, its practicality will be reduced. It is not that the more complex the selection method is, the better it is. The details of the evaluation criteria and selection methods will be given in part 5. 

Next, the evaluation model and the prediction model are obtained. We should consider from the two sides of evaluation model and prediction model when we select the component reliability model, because our ultimate goal is to obtain system reliability. System reliability evaluations include the assessment and prediction. System reliability assessment usually refers to the reliability the system can be achieved when the development of the software system is completed, and the system reliability prediction is to predict the system failure at a future point when the system is running. So the system reliability evaluation includes the assessment and the prediction. The optimal model for assessment is not necessary the optimal model for prediction, so the assessment model and prediction model should be selected respectively. 

Finally, based on the prediction and assessment model, the system reliability is obtained according to the relationship among the components.
section_name: OSM Operation Process

section_content: In order to accurately assess and predict the reliability of each component, we should use appropriate method and criteria to select the most suitable model of the component reliability.
section_name: Approach of Selecting Component Reliability Model

section_content: Let

Let C is the components set in which its system reliability will be evaluated. 

). If t is the optimal evaluation model of component, the value st satisfies the conditions:

,"optim" is min or max according to the specific indicators. The candidate model and selection criteria will be described specifically later.
section_name: Principle of Selecting Approach

section_content: There are many factors affecting software reliability. The affect of the same factor is different to different software. Many factors have the stochastic characteristic timerelated, so the software reliability models are mostly modeled the form of random process. A number of assumptions are given in advance for the establishment of models. Under the assumption we establish systems reliability evaluation model.  FC models 1) Schick-Wolverton model [9] The state when the software is tested is same as its actual operation. All errors have the same probability to be detected. The expectations of failure occur at any interval are proportional to the number of errors contained in the software and the duration after the last occurrence of failure. All failures have the same severity and independently one another. Faults are corrected as soon as they are found and there is no new fault to be introduced. [24] 2) Non-Homogeneous Poisson for FC [10] The state when the software is tested is same as its ac-tual operation. The number of faults detected in each time is independent one another. All faults have the same probabilities to be detected. The total number of faults detected in any time fallows a Poisson distribution with mean m(t). The mean value of faults is a bounded non-decreasing function related to m(t) and its max value is a. There is no new fault to be introduced when the detected faults are corrected. [24] 3) Schneidewind [11] The state when the software is tested is same as its actual operation. The system failure caused by faults is stochastic. The occurrences of all failures have the same probability and independent of one another. The correction rate of fault is proportional to the found faults. The detected failures mean value decrease with the continuous test. The total number of failures has an upper bound. Every test has the same period interval. The faults detection rate is proportional to the number of system faults. There is no new fault to be introduced when the detected faults are corrected. [24] 4) Yamada S-shaped model [12] The state when the software is tested is same as its actual operation. The system failure caused by faults is stochastic. The number of initial faults in the software system is a random variable. The time between failures (k -1) and k depends on the time to failure (k -1). Only one failure occurs each time and the fault will be corrected immediately. There is no new faults are introduced when the detected fault is removed. The total number of failures has an upper bound. [24]  TBF models 1) Geometric model [13] The test process is same as its actual operation. There is no upper bound on the total number of failures. The probability of all faults detection is equal. The detections of all faults are independent each another. The failure detection rate is geometric distribution and is constant between failure occurrences. [24] 2) Jelinski-Moranda model [14] The condition of the software run is same as its actual operation. The faults detection rate is proportional to the number of the faults. The severity of all failures is equal. The failure rate between the two failures is a constant value. Faults are removed immediately and no new fault is introduced. The total number of faults has an up bound. [24] 3) Littlewood-Verrall model [15] The condition of the software run is same as its actual operation. The successive time between two failures is independent random variable and is an exponential distribution. The mean of i-th failure distribution is 1/(i), (i) has  distribution with parameter  and (i). Where (i) has the form (0)+(1) * i or (0)+(1) * i 2. There is no upper bound on the total number of failures. [24] 4) Musa Basic model [16] The test process is same as its actual operation. The detection of faults is independent one another. All faults of the system will be detected. The intervals between the successive failure are piecewise exponential distribution. The failure rate is proportional to the number of faults. The correction rate is proportional the fault detection rate. There is no new fault to be introduced when the fault is corrected. [24] 5) Musa-Okumoto model [17] The test process is same as its actual operation. The detection of faults is independent one another. The mean of the number of failures is a logarithmic function of time. The intensity of failure decreases exponentially with the experience of failures. The total of faults has an up bound. [24] 6) Non-Homogeneous Poisson Process model [18] The condition of the software run is same as its actual operation. All faults have the same probability to be observed. The cumulative number of failure M(t) follows a Poisson distribution with mean value m(t). The mean function of the number of failure is a bounded non-decreasing. The debug process will not introduce new failure. [24] The models mentioned-above are to assumptions, therefore there are some limitations, no model can replace another.
section_name: Component Candidate Evaluation Model

section_content: The effect is different when one model is applied on different data sets, one may be good but others may be bad. We can select the most suitable evaluation model according to a certain criterion using a certain data set. The key is the determination of the criterion when we select the suitable model. The commonly criteria are deviation of Bias, mean square error MSE, mean absolute error MAE, MSE and sum squire error SSE, chi-square test (Chi-Square), CKolmogorov-Smirnov test) prequential likelihood(PL). 

Chi-square test is a common method that measures model goodness of fit. The smaller result means a better goodness of fit, The formula is shown as (2). 

The Kolmogorov-Smirnov test (K-S test) is a test of goodness of fit. It is used to study if the distribution of sample observations agrees with the specified theoretical distribution. The two-sample K-S test is one of the most useful and general nonparametric methods for comparing two samples, as it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. The K-S statistic quantifies a distance between the empirical distribution function of the sample and the cumulative distribution function of the reference distribution, or between the empirical distribution functions of two samples. The formula is (3), where F(x) and E(x) are the theoretical and actual distribution of the sample respectively. 

The formulas mentioned above are mainly used to select component assessment model. As to the selection of reliability predictive model of component, we often use prequential likelihood. Prequential likelihood values are used to indicate if one model is more applicable to the failure data than the other models. Let j t is the next time of component failure, the failure distribution density function estimated by previous failure data is

, according to the density function we can estimate the next system failure time t j, the expectation of t j is:

For the 1-step prediction, the prequential likelihood value for 1 2,,,

This paper will use K-S test and Chi-square test as the selection criteria of reliability assessment model of component. They are commonly used in project. This paper will use prequential likelihood value as the selection criterion of reliability predictive model of component.
section_name: Determination of the Selection Criteria

section_content: We will determine the interaction among the components of the system after the determination of the evaluation and prediction models of the component. We can model path-based system component diagram, state-based system components diagrams or other system component interaction diagram. It will depend on the actual specific situation. If the software continuously runs 24 h every day, it should be modeled as Continuous Time Markov Chains (CTMCs). If it is operated by user's idea, it should be modeled as Discrete Time Markov Chains (DTMCs). At the same time we should also consider the available data, the different phase of the software life-cycle and what kink of assumptions we made [25]. Table 1 lists the state-based Markov models and applicable conditions [25]. 

As the space limited, please refer to related literature about how to establish relationship among the components of system. After the system model established, the system reliability can be predicted and assessed. As the assessment and prediction reliability value of each component are optimal according to the criteria, and therefore the assessment and prediction reliability of the system will also be optimal. We will prove this point as follows.
section_name: System Reliability Calculation According to the Relationship among Components

section_content: Let system composed by n components, the reliability of components computed by OSA are 1 2

, the exact value of the components reliability are 1 2 respectively. The system reliability calculated by a certain model using the exact value of each component reliability 1 2 is. The system reliability calculated by the model using the evaluation value of each component reliability

is the difference between and

is the smaller is, then we can say the system reliability calculated using the component models selected by OSA is more accurate than other component models. It is not difficult to see from Table 1 that the system reliability models listed in Table 1 is consistent with the hypothesis mentioned above. We will use CTMC_7 as a sample to demonstrate this point. The formula of model CTMC_7 is showed as (
section_name: Discussion of Software Reliability Evaluation Approach

section_content: is the system reliability calculated by exact value 1 2 using formula (

), is the system reliability calculated by 1 2 using formula ( 0 R, n R R R 6). The difference between R and R′ is:

From (7) we can see the smaller the error i R   of each component is, the smaller the error of the system.
section_name: Let

section_content: As the models of components are selected using OSA, there is

so that we can get more accurate system reliability by the approach proposed in this paper.
section_name: R 

section_content: We use following system shown in Figure 2 Table 2 shows these components historical reliability data. 

According to the approach presented in 5. 1 and the criteria in 5. 2, the results are shown as Table 3. 

From the results in Table 3 we can see the best goodness of fit reliability assessment models of components 4. The assessment value refers to the reliability when the system development is completed, the prediction value refers to the system reliability at a certain time T in the future. Here we set T to be 50. T can be changed to predict the system reliability at other time point.
section_name: Case Analysis and Verification

section_content: An optimal selection approach to improve the evaluation accuracy of component-based system reliability is proposed in this paper. The techniques of how to select the most appropriate assessment and prediction model for each component according to historical data based on certain criteria, and how to assess and predict the system reliability according to the selected model and the relationships among the various components, are presented. This approach considers not only the architecture of the system but also the historical data related to each component reliability growth. As a result, its results are more effective and practical than those of the other existing approaches.
section_name: Conclusions