Subsampling Tests of Parameter Hypotheses and Overidentifying Restrictions with Possible Failure of Identification We introduce a general testing procedure in models with possible identification failure that has exact asymptotic rejection probability under the null hypothesis. The procedure is widely applicable and in this paper we apply it to tests of arbitrary linear parameter hypotheses and to tests of overidentifying restrictions in time series models given by unconditional moment conditions. The main idea is to subsample classical tests, like for example the Wald or the J test. More precisely, instead of using these tests with critical values based on asymptotic theory, we compute data-dependent critical values based on the subsampling technique.We show that the resulting tests have exact asymptotic rejection probabilities under the null hypothesis independent of identification failure. Furthermore, the subsampling tests of parameter hypotheses are shown to be consistent against fixed alternatives and to have the same local power as the original tests under full identification. The subsampling test of overidentifying restrictions is shown to be consistent against model misspecification.An algorithm is provided that automates the block size choice needed to implement the subsampling testing procedure. A Monte Carlo study shows that the tests have reliable size properties and often outperform other robust tests in terms of power.JEL Classification: C12, C15, C52. Since Phillips ' (1989) seminal paper on the consequences of identification failure on the distribution of point estimators and test statistics a vast literature on partially or weakly identified models has developed. 1 A major finding of this literature is that in models with possible identification failure, point estimates can be severely biased and classical tests of parameter hypotheses or overidentifying restrictions can be extremely size distorted in finite samples.. In response to the unreliability of classical tests of parameter hypotheses in models with identification failure, such as Wald or likelihood ratio tests, several new tests for simple full vector parameter hypotheses have recently been introduced whose rejection probabilities under the null hypothesis are (asymptotically) unaffected by identification failure. 2 However, to the best of our knowledge, no test of overidentifying restrictions, that is consistent against model misspecification and robust to identification failure, has been introduced in the literature. Furthermore, generalizations in the literature of the above mentioned tests of simple full vector parameter hypotheses to more general parameter hypotheses, either require additional assumptions or are only conservative. For example, Kleibergen's (2001 Kleibergen's ( , 2004 test can be used to test simple subvector hypotheses under the additional assumption that the parameters not under test are strongly identified. Dufour (1997) suggests a projection-based testing procedure for general parameter hypotheses that works without additional assumptions but leads to conservative tests. Furthermore, in general the projection idea is computationally cumbersome. 3 In this paper, we address the need for robust tests of general linear hypotheses and overidentifying restrictions. We introduce a general testing procedure in models with possible identification failure that has exact asymptotic rejection probability under the null hypothesis. We then apply the procedure to tests of arbitrary linear parameter hypotheses and to tests of overidentifying restrictions in time series models given by unconditional moment restrictions. The main idea of our procedure is to apply 1 See among others, Nelson and Startz (1990) , Choi and Phillips (1992) , Dufour (1997) , Staiger and Stock (1997) , Stock and Wright (2000) , Forchini and Hillier (2003) , and for recent reviews of the weak identification literature see Stock et al. (2002) and Startz et al. (2004) . A recent paper by Chao and Swanson (2005) brings together the many (Bekker, 1994) and weak instruments literature.. 2 Besides the early contribution of Anderson and Rubin (1949) see among others, Stock and Wright (2000) , Kleibergen (2001 Kleibergen ( , 2002 , Caner (2003) , Guggenberger (2003) , Guggenberger and Smith (2003, 2005) , Moreira (2003) , Otsu (2003) and Dufour and Taamouti (2004a, 2004b) . Andrews et al. (2004) investigate robust hypothesis testing with optimal power properties when the instrumental variables might be weak.. 3 One exception is the Anderson and Rubin (1949) statistic for scalar linear hypotheses where a closed form solution is available, see Dufour and Taamouti (2004a, 2004b). subsampling to classical tests. More precisely, instead of using the tests with critical values based on asymptotic theory, we compute data-dependent critical values based on the subsampling technique. The test statistic under consideration is evaluated on all (overlapping) blocks of the observed data sequence, where the common block size is small compared to the sample size. The critical value is then obtained as an empirical quantile of the resulting block (or subsample) test statistics.. We first introduce a general definition of identification failure that brings together Phillips ' (1989) notion of partial identification with Stock and Wright's (2000) notion of weak identification. We then apply the subsampling method to the Wald and the J test (Hansen (1982) , also see Newey (1985) ) and show that the resulting tests have exact asymptotic rejection probabilities under the null hypothesis independent of identification failure. Furthermore, we show that the subsampling version of the Wald test is consistent against fixed alternatives under full identification and has the same local power as the Wald test. The subsampling version of the J test is shown to be consistent against model misspecification. Our analysis is done in time series models given by nonlinear moment conditions. Throughout the paper, we use the linear single equation instrumental variables model as an illustrative example of the general time series model. Our parameter tests can be applied to general linear hypotheses without additional identification assumptions. In particular, unlike Kleibergen (2001 Kleibergen ( , 2004 , no additional identification assumptions are required for subvector testing. Also, in a linear single equation instrumental variables model we can test simultaneous hypotheses on the coefficients of the exogenous and endogenous variables. A further advantage of the subsampling approach considered here is its robustness with respect to the model assumptions. For example, we show that the sizes of the subsampling tests are not affected (asymptotically) by instrument exclusion in the reduced form of a linear single equation instrumental variables model. This last advantage also holds true for the Anderson and Rubin (1949) statistic (in the case of a simple full vector hypothesis) but not for the tests by Kleibergen (2002 Kleibergen ( , 2004 or Moreira (2003) ; see Dufour and Taamouti (2004b) .. We assess the finite sample performance of several parameter subvector tests in a Monte Carlo study using Dufour and Taamouti's (2004b) linear design with two endogenous variables on the right side of the structural equation. We find that their projected Anderson and Rubin test is dominated in terms of power by Kleibergen's (2003) test across every single scenario. In all scenarios, where the parameter not under test is only weakly identified, our subsampled Wald test is the clear winner among the three statistics and the power gains can be dramatic in these cases. If this parameter is strongly identified, then Kleibergen's (2003) test typically has slightly better power properties than our test. In an additional Monte Carlo experiment we assess the power loss of the subsampling procedure in a scenario where subsampling does not enjoy a comparative advantage, namely when testing a simple full vector hypothesis in a linear i.i.d. model. We find that even in this disadvantageous setup, subsampling still performs competitively but is often outperformed in terms of power by Moreira (2003) . Lastly, we conduct an experiment to assess the size properties of tests of overidentifying restrictions. Again, our Monte Carlo results are consistent with our theory: While the classical J test oftentimes severely overrejects, we find that its subsampling version has generally very reliable size properties.. Besides all the advantages of the subsampling technique mentioned above, there are also general drawbacks. Firstly, compared to tests that are given in closed form, a relative disadvantage of the subsampling approach is its computational burden. However, this disadvantage is shared with other popular resampling methods, such as the bootstrap and the jackknife. Secondly, an application of the subsampling method requires the choice of a block size b, which can be considered a model parameter.. To overcome that problem, we provide a data-dependent method to automate this choice. Thirdly, under full identification, weak regularity conditions, and one-sided alternatives, the error in rejection probability under the null for tests based on subsampling is typically of order O p (b −1/2 ) compared to the faster O p (n −1/2 ) of standard approaches, where b and n denote the block and sample size, respectively, see Politis et al. (1999, chapter 10.1) . This slower rate of convergence under full identification is the price that has to be paid for making the procedure robust to identification failure. Lastly, in our specific application, the finite sample power function of a subsampling version of a test is oftentimes below the one of the original test in strongly identified situations. However, compared to other tests that are robust to identification failure, our Monte Carlo study indicates that oftentimes there can be tremendous power gains of the subsampling approach.. In the Econometrics literature, subsampling has now been suggested in a variety of situations for the construction of confidence intervals or hypotheses tests where it is at least questionable whether the bootstrap would work. Some recent examples include, Romano and Wolf (2001) who use subsampling to construct confidence intervals for the autoregressive coefficient in an AR(1) model with a possible unit root. Andrews (2003) introduces a subsampling-like testing method for structural instability of short duration. Choi (2005) uses subsampling for tests of linear parameter constraints in a vector autoregression with potential unit roots and Gonzalo and Wolf (2005) suggest subsampling for the construction of confidence intervals for the threshold parameter in threshold autoregressive models with potentially discontinuous autoregressive function.. Related to our paper is Hall and Horowitz (1996) who suggest the bootstrap for [3] improved critical values for certain GMM tests. However, their paper is not concerned with the possibility of identification failure. Kleibergen (2003) The remainder of the paper is organized as follows. In Section 2, the model is introduced, the testing problems are described, and a general definition of identification failure is provided. In order to be self contained, in Section 3 we first briefly review the basic theory of subsampling for time series data. We then derive the asymptotic distribution of some classical test statistics under the general asymptotic framework of identification failure to show that the tests are generally size distorted under identification failure. We then apply subsampling to those tests in subsections 3.2 (overidentifying restrictions) and 3.3 (general linear parameter hypotheses) to cure the problem of size distortion. In Section 4 we provide a data-driven choice of the block size needed to implement the subsampling procedure. Some further robustness properties of the subsampling method are discussed in Section 5. Section 6 describes the simulation results. All proofs are relegated to Appendix (C) while Appendices (A) and (B) contain some additional discussion of our assumption on identification failure and contiguity, respectively.. The following notation and terminology is used in the paper. The symbols " → d ", " → p ", and " ⇒ " denote convergence in distribution, convergence in probability, and weak convergence of empirical processes, respectively. For the latter, see Andrews (1994) for a definition. For "with probability 1" we write "w.p.1" and "a.s." stands for "almost surely". By C i (A, B) we denote the set of functions f : A → B that are i times continuously differentiable. If B = R, the set of real numbers, we simply write C i (A) for C i (A, B). By id we denote the identity map and by O(i) the group of orthogonal i × i matrices. By e j ∈ R p we denote the p-vector (0, ..., 1, ..., 0) 0 with 1 appearing at position j. For a matrix M, M > 0 means that M is positive definite and [M ] i,j denotes the element of M in row i and column j. By I i we denote the [4] i-dimensional identity matrix. Furthermore, vec(M ) stands for the column vectorization of the k × i matrix M , that is, if M = (m 1 , ..., m i ) then vec(M ) = (m 0 1 , ..., m 0 i ) 0 . By P M we denote the orthogonal projection onto the range space of M. Finally, ||M || equals the square root of the largest eigenvalue of M 0 M and "⊗" denotes the Kronecker product.. 2 The Model, Tests, and Identification Failure We introduce new subsampling based tests of parameter hypotheses and overidentifying restrictions that are robust to weak identification. The tests are applicable in a time series context given by unconditional moment restrictions. To the best of our knowledge, there are no other tests of overidentifying restrictions in the literature that are robust to weak identification and consistent against model misspecification. Furthermore, there are no other tests of general linear parameter hypotheses in the literature that are consistent under full identification and have exact (asymptotic) rejection probabilities under the null; for example, projection based tests are only conservative and our Monte Carlo study indicates that they typically have poor power properties under weak identification compared to the subsampling approach. In a linear single equation model, our approach can be used to simultaneously test hypotheses on the coefficients of the exogenous and endogenous variables. On the other hand, this can not be done with test procedures where the exogenous variables are [30] projected out in a first step, see for example, Kleibergen (2002 Kleibergen ( , 2004 .. Besides the applicability of the tests in a general time series context, a further advantage of the subsampling tests seems to be their robustness with respect to the model assumptions. As an example, we show that in the linear single equation model the tests have exact (asymptotic) rejection probability under the null even under instrument exclusion.. Roughly speaking, what is required for the subsampling approach to work, is that asymptotically, under the null hypothesis, the subsampling test statistic obeys a continuous limit law. Given this weak assumption, it seems very likely that the subsampling tests would also be robust to the so called "many instrument problem", see Bekker (1994) , Hahn and Inoue (2002), and Hansen et al. (2004) . This question is currently under our investigation. The subsampling method is very general and could be applied to other testing problems in the context of weak and or many instruments, for example, to tests of exogeneity, see Staiger and Stock (1997, p.567) . The linear model serves as a motivating example for Assumption ID. Example 2.1 (cont.; based on Phillips (1989, p. 185-6) ): In the above linear model simple calculations using E(. where we set. and assume that the matrix Q has full rank. In the linear model the rank condition for identification is that Π has full column rank. Indeed, θ 0 is identified if and only if Π has full column rank. In the polar opposite case, Π = 0, β 0 is totally unidentified, while certain linear combinations of γ 0 may still be identified depending on the rank of the matrix Φ. More precisely, if P = (P 1 , P 2 ) ∈ O(v 2 ) and P 1 spans the null space of Φ 0 , then P 0 1 γ 0 is identified while P 0 2 γ 0 is totally unidentified; for example, if Φ = 0, then γ 0 is fully identified! Similarly, for general rank of Π, choose S = (S 1 , S 2 ) ∈ O(v 1 ) such that S 2 spans the null space of Π. Then S 0 1 β 0 is identified. Therefore, in the partially identified model the identifiable linear combinations of θ 0 can be retrieved after a rotation of the coordinate system. Stock and Wright's (2000) Assumption C is a special case of the partially identified model in the sense that, according to C, in the original coordinates the components of θ 0 are either identified or (asymptotically) unidentified. Putting it differently, the matrix (∂Eb g/∂θ 0 ) ∈ R k×p in Phillips (1989) can be of non-maximal rank without necessarily being of the particular form (∂Eb g/∂θ 0 ) = (0, M), where M is a matrix of maximal rank. On the other hand, such a decomposition into (0, M) is implied by Assumption C in Stock and Wright (2000) as n → ∞ (for M = (∂m 2 /∂θ 0 2 ) under the weak technical assumption that n −1/2 (∂m 1n /∂θ 0 )(θ) → 0 uniformly), where in the limit the derivative of Eb g(θ) with respect to the weakly identified variables has to be constantly equal to 0. In the linear model (∂Eb g/∂θ 0 ) = −QF has a decomposition into (0, M) ∈ R k×(p 1 +p 2 ) if and only if the first p 1 columns of F equal zero which holds if and only if the first p 1 columns of Π and Φ are zero. This is one particular case of, but does not account for the general case of identification failure. Therefore, Assumption ID is motivated by Phillips' (1989) treatment of the linear model that allows for more general forms of rank deficiency of Π and Θ. We finally discuss the [32] coordinate change T in the example of the linear model. Example 2.1 (cont.): In the linear partially identified model choose a matrix T = (T 1 , T 2 ) ∈ O(p 1 + p 2 ) such that T 1 spans the null space of QF. Then. m 2 (θ 2 ) := QF T 2 (θ 02 − θ 2 ), and m 1n ≡ 0. It follows that M 2 (θ 2 ) ≡ −QF T 2 has full column rank and that m 2 (θ 2 ) = 0 if and only if θ 2 = θ 02 . The orthogonal map T transforms the coordinate system in R p 1 +p 2 in such a way that in the decomposition θ 0 := (θ 01 , θ 02 ) of the new variables, the first components θ 01 are unidentified while the remaining components θ 02 are identified.. (B) Some Words on Contiguity:. The notion of contiguity is a very useful tool to compute the limiting power of statistical tests against a certain class of "local" alternatives. Consider two sequences of probability measures {P n } and {Q n } defined on a common probability space. Then the sequence {Q n } is contiguous to the sequence {P n } if P n (E n ) → 0 implies Q n (E n ) → 0 for every sequence of (measurable) events {E n }. Therefore, contiguity can be considered as an asymptotic version of one probability measure being absolutely continuous with respect to another one. Assume one knows the limiting distribution of a sequence of test statistic D n under P n but the behavior of D n under Q n is also required. Contiguity provides a means of performing the required calculation.. To verify contiguity in a particular setting, several high level conditions are available. One of them, and of particular interest to us, is the following. Consider a parametric family {P θ : θ ∈ Θ} with corresponding densities p θ (·) with respect to a σ-finite measure. Assume θ 0 is an interior point of Θ and let θ n = θ 0 +h/ √ n for some h ∈ R p . Denote by P n θ the joint distribution of {z 1 , . . . , z n } when the z i are i.i.d. from P θ . Then, under general smoothness conditions, {P n θn } is contiguous to {P n θ 0 }. One such sufficient condition is that the parametric family {P θ : θ ∈ Θ} be differentiable in quadratic mean in a neighborhood of θ 0 . For example, this condition is satisfied by most exponential families, including the multivariate normal distribution.. For a detailed treatment of contiguity, differentiability in quadratic mean, and corresponding applications to compute the limiting power of tests against "local" alternatives, the reader is referred to Hájek et al. (1999, Chapter 7) , van der Vaart (1998, Chapters 6, 7, 14, and 15) , and Lehmann and Romano (2005, Chapter 12) .. Contiguity, arguably, provides the most elegant tool to compute the limiting power against "local" alternatives, even though sometimes the calculations can be performed by direct means. However, in doing so one has to account for the fact that the probability mechanism changes with n (using Lindeberg's central limit theorem, say).. (C) Proofs: Proof of Theorem 3.1. See Theorem 3.5.1 of Politis et al. (1999) . ¥ Proof of Theorem 3.2. (i) We apply the proof of Theorem 1 in Stock and Wright (2000) to the model in new coordinates. 23 Set S n (θ) := n||A n (e n (T (θ))) 1/2 b g(θ)|| 2 and note that the sequence. By assumption, the function. There exists a random element ( e θ 1 , e θ 2 ) ∈ Θ 1 × R p 2 such that a.s. θ 02 + n −1/2 e θ 2 ∈ Θ 2 and S( e θ 1 , e θ 2 ) < inf. for every open set G in Θ 1 ×R p 2 that contains ( e θ 1 , e θ 2 ). (This condition is needed when applying Lemma 3.2.1 in van der Vaart and Wellner (1996) in Stock and Wright's (2000) Theorem 1(ii).) Therefore, using ID and PE, by the proof of Theorem 1 in Stock and Wright (2000) it follows that ( b. 2 ), where. (ii) The J statistic expressed in new coordinates reads J n = S n ( b θ n ) = S n ( b θ n ). Therefore, the statement follows from careful inspection of Corollaries 4(i) and (j) in Stock and Wright (2000) applied to S n ( b θ n ). ¥ 23 For notational simplicity we write from now on Ψ(θ1, θ02) for Ψ((θ 0 1 , θ 0 02 ) 0 ) and similarly in other expressions. S n (θ) plays the role of S T (θ; θ T (θ)) and S(θ 1 , b) the role of S(α, b; θ(α, β 0 )) in Stock and Wright (2000) . For simplicity, we do not index these expressions by an argument that indicates which GMM estimator, two-step or CU, is used. Note that e n (T (θ)) equals θ in the case of CU and en for two-step GMM. Note that Stock and Wright's (2000) proof can be adapted to our slightly different definition of the GMM estimator as only an approximate (up to order o p (1)) minimizer; all that changes is that on the right hand side of their equation (A.1) we have an op(1) term rather than 0.. Proof of Corollary 3.1. By Theorem 3.2(ii) and assumption, the test statistic D n = J n has a continuous limit distribution J * both under full identification and identification failure. So the proof of (i) follows from part (i) of Theorem 3.1.. To prove (ii), let β = 1 and d n = J n /n in (3.7). Using Assumption MM, Newey and McFadden (1994, Theorem 2.1) implies that b θ n is consistent for e θ. Therefore by Assumption MM, d n converges in probability to d(P ) := ||A(e( e θ)) 1/2 Eg i ( e θ)||. Clearly, d(P ) > 0 under H 1 . On the other hand, under H 0 , e θ, as the unique minimizer of ||A(e(θ)) 1/2 Eg i (θ)||, has to satisfy Eg i ( e θ) = 0 and therefore d(P ) = 0. Part (ii) of Theorem 3.1 therefore proves the result. ¥ Proof of Lemmas 3.1 and 5.1. The proof of Lemma 5.1 contains the proof of Lemma 3.1 as the subcase in which Π * = 0. Therefore, we only deal with the general case of model (5.17). We have. , and 7.27) and note that by the chain rule and linearity of T we have b. ) is chosen such that T 1 spans the null space of Q * F * and b G n is defined in (3.13). Clearly, in the linear model b G * n does not depend on b θ n . First, we show the validity of the FCLT in WS(i). By the proposition on p. 2251 in Andrews (1994) , a sufficient condition for the FCLT is finite dimensional (fidi) convergence and stochastic equicontinuity. For the latter, we have to show that ∀ε > 0 and η > 0, ∃δ > 0 such that. Because b G * n does not depend on θ we have. Given the i.i.d. and moment assumptions, a CLT can be applied to each component of. showing that these expressions are [35] in O p (1). This immediately establishes (7.28). Regarding fidi convergence, note that we do not have to subtract off the mean in the first kp 1 components of Φ n (θ), because. has mean zero by construction. Fidi convergence to a multivariate normal distribution follows by a standard CLT from the i.i.d. and moment assumptions.. Next, we show WS(ii). We have. depend on b θ n , and which, by the weak law of large numbers, converges in probability to (E∂g i /∂θ 0 )T 2 = Q * F * T 2 . The latter matrix has maximal column rank because. ) was chosen such that T 1 spans the null space of Q * F * . We next show WS(iii). The proof is similar to the one in Guggenberger and Smith (2005, proof of Theorem 4) . From WS(ii) we know that the last p 2 columns of the limit of b G * n N equal M 2 which has full column rank by assumption and does not depend on θ. Define O := {o ∈ R kp 1 : ∃e o ∈ R k×p 1 , such that o = vec(e o) and the k ×p-matrix (e o, M 2 ) has linearly dependent columns}. Clearly, O is closed and therefore Lebesgue-measurable. Furthermore, O has empty interior and thus has Lebesguemeasure 0. The first p 1 columns of the limit of b G * n N stacked as a vector equal the first kp 1 components of Φ, Φ| kp 1 say, which by assumption is normally distributed with full rank covariance matrix (which in the linear model does not depend on θ). This implies that for any measurable set O * ⊂ R kp 1 with Lebesgue-measure 0, it holds that Pr(Φ| kp 1 ∈ O * ) = 0, in particular, for O * = O. This proves the full rank claim.. Finally, note that by the weak law of large numbers Λ(θ) = lim n→∞ Eg i (θ)g i (θ) 0 in the linear model and convergence of n −1 P n i=1 g i (θ)g i (θ) 0 to Λ(θ) is uniform by compactness of the parameter space. ¥ Proof of Theorem 3.3. In new coordinates, Assumption ID implies that (in the terminology of Stock and Wright (2000) ) the first p 1 components of the parameter vector θ 0 are weakly identified while the remaining p 2 components are strongly identified and are root n-consistently estimated as shown in Theorem 3.2. Therefore, when deriving the asymptotic distribution of the Wald statistic, we have to renormalize certain expressions that have different convergence rates than others. To do that we use two matrices N ∈ R p×p defined in (3.15) and M ∈ R r×r . To define M, let L ⊂ R r be the linear subspace spanned by the first p 1 columns of the matrix RT ∈ R r×p . Set p 11 := rank(L), for which p 11 ≤ min(r, p 1 ), and assume w.l.o.g. that the first p 11 columns of RT form a basis B 1 of L. Because RT has maximal rank r, there are r −p 11 columns among the last p−p 11 columns of RT that together with B 1 form a basis for R r . W.l.o.g. assume the last r − p 11 columns of RT can be taken and call them B 2 . Let (B ⊥ 2 , B ⊥ 1 ) be another basis of R r , B ⊥ 2 ∈ R r×p 11 , B ⊥ 1 ∈ R r×(r−p 11 ) such that the columns of B ⊥ i are orthogonal to the columns in B i , i = 1, 2. Define M := (n −1/2 B ⊥ 2 , B ⊥ 1 ) ∈ R r×r . Note that in the case of full identification M = I r . Using n −1 P n i=1 (∂g i /∂θ 0 )( b θ n ) = b G n T , the Wald statistic (3.11) under the null hypothesis reads in renormalized form and in new coordinates where, for b G * n defined in (7.27), we set. A n : = A n (e n (T ( b θ n ))) ∈ R k×k .. By construction of M it follows that (the nonstochastic matrix) NT 0 R 0 M ∈ R p×r converges to a matrix that has maximal rank r; namely, the first p 11 and last r − p 11 rows of NT 0 R 0 M span R r0 and do not depend on n because the normalizations n 1/2 from N and n −1/2 from M cancel out. In the other rows of NT 0 R 0 M the normalizations have either cancelled out each other as well or the components of NT 0 R 0 M are in O(n −1/2 ), that is, they converge to zero. Therefore, R * n ∈ R r×p converges to a matrix that has maximal rank r. Regarding the second factor in ξ * n , Theorem 3. i Ω * n e j ; for example, by WS(iv) we have e 0 i Ω * n e j → d Φ 0 i (θ 12 )AΛ(θ 12 )AΦ j (θ 12 ) if both i, j ≤ p 1 . Note that all the above limits hold jointly.. Denote by ξ * , R * , B * , and Ω * the just described limits of ξ * n , R * n , B * n , and Ω * n . Then by the continuous mapping theorem the asymptotic distribution of the Wald statistic is given by the distribution of W * if we can show that R * n B * −1 n Ω * n B * −1 n R * 0 n converges to a random variable (with realizations in R r×r ) whose realizations are invertible a.s.. We have shown already that [37] R * has full rank and it thus remains to show that B * n and Ω * n converge to random variables that have full rank a.s.. But this holds by Assumptions WS(iii)-(iv) and PE(ii) that establish that Λ(θ 12 ), A, and the limit of b G * n N ∈ R k×p have full rank a.s.. ¥ Proof of Corollary 3.2. By Theorem 3.3 and assumption, the test statistic D n = W n has a continuous limit distribution W * both under full identification and identification failure. So the proof of (i) follows from part (i) of Theorem 3.1.. To prove (ii), assume full identification. In (3.7) let β = 1 and d n = W n /n. Then d n converges in probability to d(P ) := (Rθ 0 −q) 0 [RB * −1 Ω * B * −1 R 0 ] −1 (Rθ 0 −q), where B * := plim( b B n ) and Ω * := plim( b Ω n ), defined before (7.29), are the probability limits under full identification. Obviously, d(P ) = 0 if and only if the null hypothesis is true. Now apply part (ii) of Theorem 3.1.. To prove (iii), assume full identification and consider a sequence of alternatives P n that are contiguous to P ∈ P 0 . Denote by θ n the parameter corresponding to P n and by θ 0 the parameter corresponding to P . We have Rθ 0 = q. A test based on the Wald statistic W n has exact asymptotic rejection probability under P when the critical value is the 1 − α quantile of the χ 2 (r) distribution, denoted by χ 2 1−α (r). So in the notation of Theorem 3.1, C(P ) = χ 2 (r). Now let W be a random variable whose distribution is the limiting distribution of W n under P n . Then the asymptotic power of the Wald test is given by P rob{W > χ 2 1−α (r)}. An application of part (iii) of Theorem 3.1 with D n = W n , D = W , and c(1 − α, P ) = χ 2 1−α (r), now implies that the limiting power of the subsampling test against the sequence P n is also given by P rob{W > χ 2 1−α (r)}. ¥

B O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B B O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O B B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O


Mean field variational Bayesian inference for support vector machine classification Abstract A mean field variational Bayes approach to support vector machines (SVMs) using the latent variable representation on Polson & Scott (2012) is presented. This representation allows circumvention of many of the shortcomings associated with classical SVMs including automatic penalty parameter selection, the ability to handle dependent samples, missing data and variable selection. We demonstrate on simulated and real datasets that our approach is easily extendable to non-standard situations and outperforms the classical SVM approach whilst remaining computationally efficient. Support vector machines (SVMs) and its variants remain one of the most popular classification methods in machine learning and has been successfully utilized in many applications. Such applications include image classification, speech recognition, cancer diagnosis, natural language processing, forecasting, bio-informatics and as such these methods are likely to remain popular for many years to come. The strengths of SVMs derive from its formulation as an elegant convex optimization problem which can be efficiently solved, has few tuning parameters and whose solution only depends on a subset of the input samples, called support vectors.. Despite such popularity standard SVMs suffer from several shortcomings. Section 10.7 of Hastie et al. (2009) summarize these as: (i) natural handling data of mixed type, (ii) handling of missing values (iii) robustness to outliers in input space (iv) insensitive to monotonic transformations of inputs (v) computational scalability to large sample sizes, (vi) inability to deal with irrelevant inputs and (vii) intepretability. To this list we would add (viii) the inability to deal with correlation within samples. In this paper we aim to address (ii), (vi) and (viii) . This paper is not the first to consider these problems. Missingness has been considered by Smola et al. (2005) , Pelckmans et al. (2005) and Nebot-Troyano and Belanche-Muñoz (2010) . Dealing with irrelevant inputs via variable/feature selection in SVMs has been considered by many authors including Weston et al. (2000) , Tipping (2001) , Guyon et al. (2002) , Zhu et al. (2003) , Gold et al. (2005) and Chu et al. (2006) . On the other hand, very few papers consider modification of SVMs to handle dependent or nonidentically distributed data. Notable exceptions include Dundar et al. (2007) , Lu et al. (2011) , Pearce and Wand (2009) and Luts et al. (2012) . However, these problems are dealt with in isolation and using different approaches, rather than in a unified manner and it is difficult to see how these approaches could be adapted to multiple complications, e.g., missingness and variable selection.. In the paper we follow the earlier work of Boser et al. (1992) , Bishop and Tipping (2000) , Gao and Wong (2005) and Polson and Scott (2011) who propose various latent variable representations of the SVM loss function and reformulate the problem in a (pseudo-) Bayesian framework. This provides a unified approach which releases SVMs from many of the above problems including allowing efficient penalty parameter selection, correlation within samples, variable selection and missing data via well developed Bayesian methodology. Typically such Bayesian models are fit via Markov chain Monte Carlo (MCMC) methods. Unfortunately, MCMC methods can be notoriously slow when applied to large or complex models and can be rendered unsuitable in applications where speed is essential. These situations are precisely the same situations where SVMs are typically popular.. Our approach to this problem is to apply mean field variational Bayes (VB) methods to the models we propose. The main advantage of this approach is a streamlined and computationally efficient framework for handling to many of the problems associated with the classical SVM approach. In tandem with these algorithms we also develop Gibbs sampling approaches to these methods to facilitate comparisons with an "exact" approach to these models.. In Section 2 we provide the framework for our approach. In Section 3 we consider various extensions including automatic penalty parameter selection, group correlations, variable selection and missing predictors respectively. In Section 4 we show how our approach offers several computational advantages over the classical SVM approach. In Section 5 we conclude. Appendices contain details of our MCMC samplers. We have developed a VB approach to SVM classification. We have shown that the approach is a unified framework for dealing with a variety of complications typically difficult to deal with within a standard SVM framework. For the examples that we present here our VBSVM methods have as good or better classification performance than the standard SVM approach whilst remaining computationally efficient. In this section we present a VB approach to a Bayesian SVM classification formulation for binary classification problems. After introducing Bayesian SVMs and VB methodology we describe the latent variable SVM representation of Polson and Scott (2011) which gives rise to our basic VBSVM approach.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Explore Biological Pathways from Noisy Array Data by Directed Acyclic Boolean Networks We consider the structure of directed acyclic Boolean (DAB) networks as a tool for exploring biological pathways. In a DAB network, the basic objects are binary elements and their Boolean duals. A DAB is characterized by two kinds of pairwise relations: similarity and prerequisite. The latter is a partial order relation, namely, the on-status of one element is necessary for the on-status of another element. A DAB network is uniquely determined by the state space of its elements. We arrange samples from the state space of a DAB network in a binary array and introduce a random mechanism of measurement error. Our inference strategy consists of two stages. First, we consider each pair of elements and try to identify their most likely relation. In the meantime, we assign a score, s-p-score, to this relation. Second, we rank the s-p-scores obtained from the first stage. We expect that relations with smaller s-p-scores are more likely to be true, and those with larger s-p-scores are more likely to be false. The key idea is the definition of s-scores (referring to similarity), p-scores (referring to prerequisite), and s-p-scores. As with classical statistical tests, control of false negatives and false positives are our primary concerns. We illustrate the method by a simulated example, the classical arginine biosynthetic pathway, and show some exploratory results on a published microarray expression dataset of yeast Saccharomyces cerevisiae obtained from experiments with activation and genetic perturbation of the pheromone response MAPK pathway. O ne great challenge of postgenomic research is to identify complex biological networks and pathways from genomewide data such as DNA sequences and expression profiles. This includes metabolic pathways, protein-protein interaction networks, gene regulatory pathways, etc. Along with biological methods such as phylogenetic profile and Rosetta Stone (see Eisenberg et al. [2000] and McGuire and Church [2000] ), computational methods have been developed as powerful data-mining tools in the study of genomics. Our inference strategy consists of two stages. First, we consider each pair of elements and try to find their most likely relation. In the meantime, we assign a score, s-p-score, to this relation. Second, we rank the s-p-scores obtained from the first stage. We expect that those relations with smaller s-p-scores are more likely to be true, and those with larger s-p-scores are more likely to be false.

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Learning pseudo-tags to augment sparse tagging in hybrid music recommender systems Items in 'OpenAIR@RGU', Robert Gordon University Open Access Institutional Repository, are protected by copyright and intellectual property law. If you believe that any material held in 'OpenAIR@RGU' infringes copyright, please contact openair-help@rgu.ac.uk with details. The item will be removed from the repository while the claim is investigated. "NOTICE: this is the author's version of a work that was accepted for publication in Artificial Intelligence. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in ARTIFICIAL INTELLIGENCE, [VOL 219, (2015) This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. Online recommender systems are an important tool that people use to find new music. To generate recommendations, many systems rely on tag representations of music. Such systems however suffer from tag sparsity, whereby tracks lack a strong tag representation. Current state-of-the-art techniques that reduce this sparsity problem create hybrid systems using multiple representations, for example both content and tags. In this paper we present a novel hybrid representation that augments sparse tag representations without introducing content directly. Our hybrid representation integrates pseudotags learned from content into the tag representation of a track, and a dynamic weighting scheme limits the number of pseudo-tags that are allowed to contribute. Experiments demonstrate that this method allows tags to remain dominant when they provide a strong representation, and pseudo-tags to take over when tags are sparse. We show that our approach significantly improves recommendation quality not only for queries with a sparse tag representation, but also those that are well-tagged. Our hybrid approach has potential to be extended to other music representations that are used for recommendation, but suffer from data sparsity, such as user profiles. During the past decade a massive shift has occurred in the way that people store, listen to and discover music. Traditionally, either a single or album was stored on external media such as cassette or CD. With advances in hardware and the growth of the world wide web, this storage has moved to large music collections on personal hard-drives and websites offering massive catalogues of music.. An interesting new problem that has arisen as a consequence of this shift, is how to find and discover new music. Radio stations and music magazines continue to offer advice but, with such a large amount of music readily available, many people now look to recommender systems provided by online services. Users provide a query, such as an example track, and a recommender system will then return a set of recommendations to the user.. Core to this task of providing recommendations is the representation of tracks. Each track in the collection must have a corresponding representation, which describes the key facets of the track. Many state-of-the-art music recommender systems make use of tag-based representations, for example the hugely popular Last.fm service [1] . The tags typically consist of two components: standardised meta-data, such as artist, track title and genre; and free-text that users provide. Free-text tags are extremely useful to a recommender system, providing a wealth of different information, such as genres, artists, topics, opinions and contexts [2, 3] . Many of these tags provide useful information for recommendation, for example, tags relating to genres, artists and topics. Context such as location can be used to improve personalised recommendations, such as when it is known a user is working in a given location [4, 5] .. Uninformative tags can be problematic for tag-based recommender systems, but a far larger problem is when very few tags exist at all. This is a problem commonly known as cold-start and arises from a large popularity bias in music listening habits [6] . As a result, many tracks are under tagged; a track is tagged if it is listened to, but a track is only recommended, and therefore listened to, if it already has tags. Figure 1 displays the general pattern of results found in the experiments, and here is used to illustrate the cold-start problem in query-by-example tag-based recommender systems. The horizontal axis shows the number of tags a query has, and the vertical axis shows recommendation quality. The dotted curve in Figure 1 represents average recommendation quality in a tag- In this paper we have presented a method to improve music recommendation by tackling the problem of sparse tagging. We propose a dynamic hybrid representation that combines tags and pseudo-tags to reduce the sparsity of a track-tag representation matrix of a given dataset. There are two main contributions: the generation of pseudo-tags learned from similar tracks; and Pseudo-tags are learned by aggregating tags from similar tracks; i.e. tracks whose audio content is similar according to the MFS music inspired texture representation. We note that pseudo-tags are not learned from content, they are learned from content and tags for other tracks. Tags capture human interpretations of low level music features as well as social/contextual concepts, and frequently used tags in musical neighbourhoods can provide a consensus of high level and social/contextual information. For tracks that are weakly tagged, this has the effect that pseudo-tags enable them to inherit human interpretation of content and social/contextual information, from the tags of similar tracks. In this way useful tag information is spread to neighbouring tracks in the content space.. The dynamic combination of these pseudo-tags with existing tags creates a hybrid representation which assigns more or less influence to pseudo-tags depending on how well-tagged a track is already. Sparsely tagged tracks attract more pseudo-tags, and these new tags are more equally weighted with existing tags; well-tagged tracks use existing tags to dominate the representation. Our Hybrid recommender system generates recommendations using these combined tags and pseudo-tags. Synergy from a hybrid system should be able to outperform its component parts, regardless of the sparseness of tagging. Our results demonstrate that Hybrid provides improved recommendations compared to Tag and Content individually. The benefit from the social and contextual information in this combined meta-data can be seen in good recommendations when tracks are sparsely tagged. There is also improved recommendation quality for well-tagged tracks, indicating that the small influence of pseudo-tags is not harmful but adds some diversification to the representation.. Sparsely tagged tracks suffer from the cold-start problem in music recommendation. Since learning pseudo-tags is based on content-based retrieval, pseudo-tagging does not suffer from cold-start, unless all similar tracks are untagged. For extreme cases of cold-start, when a track has no tags, pseudotags provide a stronger alternative to content. We have shown that pseudotags with high precision may be learned by taking advantage of the order of tracks in content-based retrieval.. In future work, it will be interesting to investigate a similar approach with different tasks/domains. For example, in a recommender system which uses collaborative filtering, it may be possible to reduce cold-start by learning pseudo-profiles for new users. In this case the hybrid representation would potentially consist of real user track plays, and pseudo-user track plays. There are other domains with dual representations that could benefit from using our dynamic approach to creating a hybrid representation; e.g. image retrieval with pixel content and incomplete captioning could exploit pseudocaptions; product recommendation with some available reviews could learn key comments for pseudo reviews. Tag-based representations, and the knowledge they describe, are very important to music recommender systems. For certain recommendation problems, tags are able to offer greater performance than either content-based or collaborative filtering methods [7] . Whilst a lot of progress has been made in how this knowledge can best be used, cold-start continues to cause problems in many tag-based systems [2] .. To try and understand why cold-start remains a problem, Turnbull et al. [8] compare five approaches to collecting tags, in terms of both scalability and quality. The two approaches they discuss that are relevant to our work are social tagging and auto tagging. Social tagging systems allow users to label tracks with any text, and allows the representation to benefit from social wisdom and context. The main weakness the authors highlight however is that social tags tend to be sparse for unpopular tracks, a phenomenon often referred to as the long tail.. Investigating the level of tagging among various websites, the authors of [9] confirm that this long tail is present in each. This is further illustrated for music tags by the Last.fm contribution to the million song dataset [10, 11] . To compound the problem further, the authors of [6, 12] also find evidence of popularity bias in music listening habits. The combination of the long-tail in tagging, and popularity bias in listening habits, results in many tracks being very under tagged; the cold-start problem.. Two simple approaches to reducing cold-start in tag-based representations are to collect them by survey [13] or through an online game [14, 15] . Two of the most successful approaches are the Tag-a-Tune game [15] and MajorMinor game [16] , collecting tags for 30, 237 and 2, 300 tracks respectively. Both of these approaches however suffer from scalability problems, and as such risk limiting tagging to popular tracks, rather than tagging the popularity long-tail where it is needed.. A further approach to reducing cold-start in tag-based recommender systems is auto-tagging [17] . The goal of auto-tagging is to learn a set of tags which are relevant to a track; these tags may then be used as a representation of the track. In general, most auto-tagging systems follow a similar approach; the content of tracks is modelled, and a training set of tags created. A multi-class classification algorithm is then used to predict the top n most likely classes, or tags, for any given track.. To generate auto-tags, the content of each track is modelled from contentbased representations. The most common content-based representations used include chroma, rhythm and texture [18] , which are typically described as a bag-of-features. For auto-tagging, a growing trend has been to model these bags-of-words using Gaussian Mixture Models (GMM) [13] , which describe the bag-of-words as a mixture of Gaussian distributions. Tags may be modelled in a similar manner, in which case auto-tagging can be achieved using the weights of correlated distributions. This method achieved the top f-measure and AUC ROC-tag score in the MIREX (Music Information Retrieval Evaluation eXchange) 2008 auto-tagging competition [19] .. A further GMM based approach was adopted by Tardieu et al. in their MIREX 2011 auto-tagging submission [20] . A Universal Background Model (UBM) GMM is trained using the expectation maximisation algorithm. The model for each track is then learned by adapting the UBM to fit the track content data. This is achieved by modifying only the mean vectors of the UBM, and thus each track can be represented by the mean vectors of its Gaussian components. In MIREX 2011, this method achieved the top tag classification accuracy, negative example accuracy, precision, and f-measure.. The authors in [21] argue that such classification approaches to autotagging are too simple, and that further statistical modelling is required. They refine the approach by modelling concepts, not tracks, using GMMs.. These models are then refined using a set of SVM classifiers. The approach is competitive with other methods, but does not justify the extra computational complexity that is introduced.. We take inspiration for our work from a more simple approach, which has been used for style and mood annotation [22, 23] , and for propagating semantic information [24] . The authors in each case use content-based retrieval to provide a set of ordered retrievals. It is the order and number of these retrievals which should be used for annotation, not tag correlations with content. This general idea has been used successfully in other related domains, including playlist generation [25] , and image tagging systems [26] .. Further approaches to reducing cold-start make use of a hybrid system, many of which rely on content, not auto-tags. In our previous work we investigated generalising concatenated tag-content representations [27] . Using this generalisation we were able to increase the cold-start discovery of tracks, while maintaining a recommendation quality comparable to a tag-based recommender system. A similar hybrid approach was investigated by Levy and Sandler [28] , but using a representation they call muswords. In this hybrid system the authors first cluster the content representation so that it is comparable with tags, and then concatenate the representations.. This notion of combining musical representations, and then generalising them is a growing trend in hybrid recommendation systems. Combinations of representations include: audio features and user-ratings [29] ; audio symbols and user-ratings [30] ; user-tags and user-ratings [31] ; and tags and audio [27, 28] . The intuition behind such approaches is that sparsities in some representations (tags, user-ratings) can be overcome using content. In each case, the way that content is used is based on its correlation with another representation. One issue with these approaches however is that content does not always correlate with other representations. For example, tags can describe non musical properties, and so looking for non-existent correlations may introduce noise into the hybrid system. This problem of correlating content with other representations is avoided in [31] . The authors look for correlations in user-tags and user-ratings; the user provides a direct link and so correlations are meaningful. To reduce the sparsity, and therefore cold-start problem, the authors investigate propagating user-tags based on content-based similarity. The authors describe this as a way of combining content with other features, which slightly obscures the insight of their method.. It is known that content on its own does not perform well compared to other methods [27] . Based on the previous work in hybrid recommender systems, we propose that content should not be used for recommendation directly at all. Content should be used to strengthen other representations, but should not be integrated into these representations. Our approach attempts to follow these lessons from auto-tagging and hybrid recommendation, and we propose a method where content is used to define a pseudo-tag representation, which is then used in our hybrid recommender system.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study Abstract-Conventional testing methods often fail to detect hidden flaws in complex embedded software such as device drivers or file systems. This deficiency incurs significant development and support/maintenance cost for the manufacturers. Model checking techniques have been proposed to compensate for the weaknesses of conventional testing methods through exhaustive analyses. Whereas conventional model checkers require manual effort to create an abstract target model, modern software model checkers remove this overhead by directly analyzing a target C program, and can be utilized as unit testing tools. However, since software model checkers are not fully mature yet, they have limitations according to the underlying technologies and tool implementations, potentially critical issues when applied in industrial projects. This paper reports our experience in applying Blast and CBMC to testing the components of a storage platform software for flash memory. Through this project, we analyzed the strong and weak points of two different software model checking technologies in the viewpoint of real-world industrial application-counterexample-guided abstraction refinement with predicate abstraction and SAT-based bounded analysis.Index Terms-Embedded software verification, software model checking, bounded model checking, CEGAR-based model checking, flash file systems. A MONG the various storage platforms available, flash memory has become the most popular choice for mobile devices. Thus, in order for mobile devices to successfully provide services to users, it is essential that the storage platform software of the flash memory operate correctly. However, as is typical of embedded software, conventional testing methods often fail to detect bugs hidden in the complex storage platform software. This deficiency incurs significant cost to the manufacturers.. Conventional testing has limitations in verifying whether a target software satisfies given requirement specifications since testing does not provide complete coverage. Furthermore, significant human effort is required to generate effective test cases that provide a certain degree of coverage. As a result, subtle bugs are difficult to detect by testing and can cause significant overhead after the target software is deployed. In addition, even after detecting a violation, debugging requires much human effort to step-by-step replay and analyze the factors leading up to the scenario where the violation occurred. These limitations were manifest in many industrial projects, including the unified storage platform (USP) software for Samsung's OneNAND flash memory [3] . For example, a multisector read (MSR) function was added to USP to optimize the reading speed (see Section 8) . This function, however, caused errors, despite extensive testing and debugging efforts, to the extent that the developers seriously considered removing the feature.. Model checking techniques [26] have been proposed to alleviate the aforementioned weaknesses of the conventional testing methods by automatically exploring the entire state space of an abstract target model. If a violation is detected, the model checker generates a concrete counterexample through which the bug can be conveniently identified. However, model checking techniques are not widely applied in industry since discrepancies exist between the target software and its abstract model. Furthermore, significant additional efforts are required to create an abstract target model, which is not affordable for most industrial projects. Modern software model checkers [34] , [25] remove this overhead by directly analyzing a target C program through either automated abstraction (e.g., counterexample-guided abstraction refinement (CEGAR) [18] with predicate abstraction [31] ) [6] , [21] , [8] , [14] or limiting the range of analysis (e.g., bounded analysis [10] ) [19] . However, since software model checkers are not fully mature yet, they have limitations according to the underlying technologies and tool implementations, which can constitute critical issues when applied to real industrial projects. Therefore, developing strategies for effective use of software model checkers through concrete case studies is very important for successful industrial application. In addition, it is challenging to verify a flash storage platform since it makes heavy use of complex data structures to manage linked lists of physical units and corresponding sector allocation maps (see Section 3.2 and Section 8), while most embedded systems are control-oriented without complex data structures.. In this project, we applied two software model checkers to find hidden bugs in the USP for Samsung's OneNAND flash memory, namely, Blast [8] and CBMC [19] , which utilize CEGAR with predicate abstraction and SAT-based bounded analysis, respectively. We performed an in-depth comparative study of these two different software model checking techniques and summarized the lessons learned regarding the advantages and weaknesses of software model checking for industrial projects.. We selected these two software model checkers for the following reasons. First, Blast and CBMC are relatively stable compared to other tools. The development of both tools has progressed for almost a decade, and both have user communities. Thus, it is likely that many of the bugs and limitations in the original versions have been fixed and USP can therefore be verified using these tools without much difficulty. Second, the source code of CBMC and that of Blast are publicly available. Although model checking technique claims to be a "push-button" technique, a user can improve the verification performance or bypass the underlying limitations of the tool through knowledge of the internal behaviors of the model checking tools, which is difficult to obtain without the tool source code. In addition, it is occasionally necessary to customize the tool depending on the needs of the projects in an industrial setting. Finally, we have expertise in the area of both Blast and CBMC through several years of experience. Hence, we can conduct comparative experiments in a fair and technically sound manner. Therefore, given that the goal here is to determine practical strengths and weaknesses of software model checking techniques on actual industrial software (i.e., flash storage platform software), Blast and CBMC can adequately serve this purpose.. This paper is organized as follows: Section 2 discusses related work. Section 3 describes background of the OneNAND flash memory and USP. Section 4 provides an overview of CEGAR with predicate abstraction and SATbased bounded analysis techniques. Section 5 explains the scope and methodology used in the project. Sections 6-8 present the experiments of analyzing the USP using Blast and CBMC. Section 9 summarizes the lessons learned from this project, and Section 10 concludes with future work. In addition, for the convenience of the readers, Table 4 at the end of the paper explains the list of acronyms in the domain of flash storage platforms. In this project, we successfully applied Blast and CBMC, which utilize CEGAR with predicate abstraction and SATbased bounded analysis, respectively, to detect bugs in the USP for Samsung's OneNAND flash memory. These bugs included incomplete handling of the semaphore exceptions and a logical bug that did not store the current status of an erase operation that was preempted by a prioritized read operation; they had not been previously detected by Samsung. In addition, we established confidence in the correctness of the complex multisector read function through CBMC. Although current software model checking technologies are not yet scalable to verify an entire C program, it is still beneficial to apply software model checkers to analyze units of a target program, as demonstrated in this project.. In future work, we will continue to apply software model checking techniques to embedded software domains to identify practical issues in which these techniques can be applied in an industrial setting. We hope that we can apply more recent techniques such as SATABS [20] , [21] and COGENT [22] to industrial projects in the future.. We also plan to compare software model checking techniques to the concolic testing approach [49] , [30] , [12] thoroughly; software model checking and concolic testing have several goals in common but have different trade-offs between their verification accuracy, verification time, and applicability. A preliminary result of applying concolic testing to MSR was reported in [38] . This study found that CREST [1], which is an open-source concolic testing tool, can be used as an effective bug-finding tool; however, its analysis speed was much slower than that of CBMC since CREST invoked a constraint solver a few million times to generate test cases for all possible paths. Finally, we plan to carry out a systematic construction of an environment model for software model checking to automate the analysis of embedded software further. 9 . The validity of the result produced by Blast in Section 7.1 was not investigated further since CBMC produced the same result. Hotae Kim received the BS degree in computer science from KAIST and the MS degree in computer science and engineering from POST-ECH. He is currently a researcher at Samsung Electronics and his recent project involvement has focused on the LiMo software platform and embedded software for flash memory devices such as file systems and databases. His research interests include software development and evolution applying various software engineering practices such as formal design and verification, model checking, and software product line.. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib. This section discusses related case studies in which software model checkers (SMCs) are applied to achieve high reliability of the target systems. Although SMC was originally proposed nearly a decade ago, this promising technique has not been widely adopted by industries. One main reason is that it is difficult for field engineers to estimate how successfully an SMC will detect bugs in their target programs and how much effort is required to apply an SMC to the projects. As many SMC research papers only demonstrate the strengths of an SMC and do not describe their limitations or efforts related to actual projects, detailed industrial case studies can persuade practitioners to adopt an SMC with an understanding of the pros and cons of the application of the SMC to their projects.. There have been several studies in which an SMC was applied to various target systems such as automotive software [47] , microcontroller programs [48] , Linux device drivers [45] , [46] , file systems [52] , network filters [13] , a protocol stack [40] , and server applications [41] . For a flash storage platform, which is our main target domain, one recent study [32] analyzes flash file systems as an effort of the mini-challenge [35] . However, the majority of verification research [36] , [28] , [11] on flash file systems focus on the specifications of the file system design and not on an actual instance of implementation.. We focus on the related case studies (as shown in Table 1 ) which report the application of off-the-shelf SMCs to various target systems, 1 as the goal here is to determine the applicability of an SMC to industrial projects in which the development of custom tools optimized for a project is often not feasible due to tight time-to-market restrictions.. Groce et al. [32] applied various verification techniques to their flash file systems. In particular, they applied Blast, CBMC, and MAGIC [14] to the pathname canonizer of the file systems. It was reported that Blast and MAGIC failed to extract a model from the pathname canonizer and it failed as well to find a proof or a counterexample. In contrast, CBMC was able to analyze the pathname canonizer with pathnames containing up to six characters. The authors in [32] suspected that heavy use of complex data structure in the file system might cause failures to SMCs, which was similarly observed in our case study. Kolb et al. [40] used Blast to verify the components of the Object linking and embedding for Process Control Unified Architecture (OPC UA) [42] protocol stack with regard to lock policy, message context management, and encoder management. They reported that several bugs were detected in a few seconds to a few minutes, but tool failures and false alarms were observed as well. Ku et al. [41] applied SATABS [21] to detect buffer overflows on the 300 sample codes of server applications such as apache and sendmail. Although 90 percent of the sample codes are less than 100 lines long, they could analyze the sample codes with only minimal buffer sizes (1 and 2) to avoid explosion of analysis time. They reported that SATABS found buffer overflows in 71.4 percent of the sample codes in 600 seconds. Mü hlberg and Luttgen [45] and Post and Kü chlin [46] verified the same Linux device drivers with Blast and CBMC, respectively. They analyzed eight device drivers with regard to use-after-free (UAF) bugs and other eight device drivers with regard to the lock policies. An interesting observation was made that neither Blast nor CBMC outperformed in these two case studies. For example, both tools detected the UAF bugs in five out of eight device drivers. However, Blast and CBMC failed on different targets. Mü hlberg and Luttgen [45] failed to detect the UAF bug in dpt-i2p.c but found the bug in pciehp_ctrl.c. In contrast, Post and Kü chlin [46] failed to detect the UAF bug in pciehp_ctrl.c but found the bug in dpt-i2p.c. Mü hlberg and Luttgen [45] reported that the lock policy bugs were detected in a few seconds to minutes, but no performance information was available on the experiments of [46] .. There are several important issues in the related case studies. First, the related case studies often fail to describe real effort required in the application of an SMC to a target system. For example, most case studies (implicitly) assume that requirement properties can be easily identified and written in assertion statements; this is not true in actual industrial projects in which a user has to understand and capture the requirement as assertions. For example, Post and Kü chlin [46] mentioned that they extracted API conformance rules from the Linux documentation, but no additional explanation was given in the paper. In contrast, we describe the effort required to apply an SMC to the flash storage platform software in an industrial setting (see Section 5) .. Second, apart from Groce et al. [32] , few case studies compare different SMCs on target systems. The selection of the SMC has a crucial impact on the verification project. However, it is difficult to decide which SMC to use in a project as different SMCs have their own strengths and weaknesses (see [45] and [46] ). This paper compares Blast and CBMC on the flash storage platform in a fair and technical manner.. Finally, although success stories exist regarding the application of SMCs in industrial projects, the limitations of SMCs should be considered as well. For example, in spite of the well-known success of the CEGAR-based SMC on Windows device drivers [5] , there are case studies that report the limitations of a CEGAR-based SMC in which the inaccurate handling of arrays [41] and pointers [40] , [45] is outlined (see Section 8.2 and Section 9.2).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I B I I B I I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A content-based dynamic load-balancing algorithm for heterogeneous web server cluster According to the different requests of Web and the heterogeneity of Web server, the paper presents a content-based loadbalancing algorithm. The mechanism of this algorithm is that a corresponding request is allocated to the server with the lowest load according to the degree of effects on the server and a combination of load state of server. Besides, apply a method of random distributing base-probability to assign each request to an appropriate server in terms of their weight. All the parameters that will be used in the algorithm can be acquired by simulated test. Experimental results suggest that this algorithm can balance the load of web server clusters effectively, make full use of the existing source of software and hardware, highly improve the server's performance, and even make the best use of the web server.Zhang Lin, Master, lecturer; research fields: network security, software engineering.LI Xiao-ping, Ph.D., professor; research fields: network security, information processing, multimedia and image processing. Su Yuan, Master; research fields: network security. A Web cluster refers to a Web site that uses two or more server machines housed together in a single location to handle user requests. Although a large cluster may consist of dozens of Web servers and back-end servers, it uses one hostname to provide a single interface for users [2] . Request distribution and load balance are key technological means to realize web server clusters, but most existing request distribution algorithms are static ones and they don't have any optimization strategy and don't synthetically consider the dynamic parameters such as the utilization ratio of CPU and memory. With the apply of Jsp、database and multimedia, the request coming from different client will bring different load on web server. So those existing equilibrium algorithms are inefficient and erratic and don't fit to construct heterogeneous Web server clusters. This paper considers the heterogeneous characteristic of hardware and software on every node and puts forward a content-based load-balancing algorithm of application layer-QSC-Load Balancing algorithm. Experimental results suggest that it can improve the response time and throughput of the system. This paper points out a developed strategy on the basis of the context distribution contra pose to the different requesting content. Show the load condition of every server in the web class more currently through the evaluation of the web content and load server. Load lays the foundation in order to distribute the request balance; introduce the concept of probabilityspace within the process of distribution to make the load more balance; to eliminate the period related effect due to the adoption of the fixed proportion model. Thus enhance its performance.

O O O B O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O